When Validating the Judge Changes the Result: Instability in Human Validation of LLM Safety Graders

Abstract Safety verdicts from an automated judge built on a large language model are usually checked against human labels before anyone trusts them. We show that this check can return very different answers depending on how it is run. We study one grader, one rubric and one corpus: 300 matched prompt pairs over 25 harmful intents, each written once in formal benchmark style and once in casual first-person style, answered by six instruction- tuned models. A keyword refusal metric finds a casual-minus-formal gap of 0.263. For actionable harm, the answer depends on the validation. One unblinded annotator implies 0.258. Two unblinded annotators given the same 442 items imply−0.036 and +0.274, agree at Cohen’s κ = 0.09, and split 95 times on whether the model refused at all. A pre-registered, blinded audit with three label sets implies 0.035. Regenerating all 3,600 responses without a length cap and grading them at five budgets moved neither automated gap. Across instruments, the gap shrinks from 0.263 to 0.035 as the decision boundary moves from refusal to actionability, with each instrument close to what its own boundary predicts. Neither factor explains why the human audits disagree. What is left is the val- idation procedure, although we cannot say which part of it is responsible. On the same responses, the protocol alone decides whether a model passes a release threshold. We argue that validating a judge is a measurement experiment in its own right, to be specified, blinded and reported per class, and propose a reporting checklist.