The 52-year-old, who hung up his whistle at the end of last season, reckons only eight of the current crop are good enough and believes referees' chief Mike Riley and his assistant Keren Barratt are part of the problem.
He told the Sun: "Don't get me wrong, there are some really top referees out there.
"I reckon currently there is a 'great eight', in no particular order: Mark Clattenburg, Howard Webb, Mike Dean, Andre Marriner, Lee Probert, Martin Atkinson, Phil Dowd and Chris Foy.
"But as for the other 10 on the elite list, well...some are just not up to it while others need more experience, better nurturing and coaching."
That puts referees including Mike Jones, Kevin Friend and Lee Mason on Halsey's list of those who are not good enough.
Halsey believes much of the blame lies with the body that oversees referees - Professional Game Match Officials Ltd (PGMOL) - as well as their general manager Riley and select group manager Barratt.
"[PGMOL] are running the whole show on an apparent power trip while creating an atmosphere of fear and paranoia," Halsey said.
"That is not helping the weaker officials improve and the system often confuses and frequently frustrates the best referees who get mixed messages about what is expected of them."
Halsey believes the wrong officials are often picked for big games and blames that on Riley and Barratt.