Abstract
Abstract
Safety evaluations of large language models increasingly rely on automated labelling of model responses, and the two dominant instruments are the LLM-as-judge and hand-authored rule-based classifiers. The literature argues, correctly, that an LLM judge introduces a circular dependency: the evaluator shares the alignment vulnerabilities of the system it evaluates. The proposed remedy—replace the judge with a deterministic rule-based classifier—has been widely adopted, but has not itself been validated as a measurement instrument. We audit both instruments against an independent, hand-labelled benchmark of 32 responses drawn from a 550-trial corpus. Neither instrument is accurate. A 1.5B-parameter LLM judge reaches 65.6% accuracy (95% CI 48.3--79.6) against human labels and recalls none of the four genuinely unsafe responses; both responses it did label unsafe were in fact safe refusals. The rule-based classifier reaches 53.1% (CI 36.4--69.1), recalls two of four unsafe responses, and labels six clean refusals unsafe. On the full corpus the two instruments agree on 61.6% of items, but the chance-corrected agreement is kappa=0.115: the raw agreement is almost entirely explained by the fact that 63% of the corpus is a single class, and the judge collapses to that majority class on 84 of 99 items. We further show that the rule classifier's own unsafe count is unstable across its own versions (40 versus 8 of 550 responses), and that neither version was ever validated. We conclude that rule-based classifiers are not a neutral default but measurement instruments with unmeasured error, and we give a minimal validation protocol: report per-class recall and kappa, not agreement; validate every instrument against a held-out hand-labelled benchmark; and report the instrument's own error profile alongside any safety figure it produces.