The GMB union has called on the Referees Association to stand up to intimidation from players and managers by threatening legal action.
The football pitch is the only workplace left where bullying is still tolerated and it needs to be stamped out, said Paul Kenny, a senior official at the GMB.
"We're saying this has gone on long enough. Nobody should tolerate this sort of behaviour in the workplace," he said. "The Referees Association needs to take a lead and provide support for its members at work. It needs to say it will take legal action against people who abuse their members."
Players such as Wayne Rooney and Patrick Vieira have been heavily criticised in recent weeks for persistent swearing at referees, while Swedish referee Anders Frisk announced his retirement from the game 11 days ago after receiving death threats from Chelsea fans.