Abstract
Abstract
A safety certificate for a clinical AI promises that no more than a stated fraction of the answers it releases carry a defect that would change care. That promise is computed from past cases a clinician marked safe, and clinicians miss things. A missed defect is not filed as a disagreement, it is filed as a safe case, so the certificate rests on a record cleaner than the ward ever was and has to be widened to compensate. Widening it correctly takes one number: how often that clinician misses a defect the model produced. We show that number cannot come from the only test a hospital can run on a working reviewer, planting a known defect and counting the catches. A planted defect is as hard to spot as whoever planted it made it, a real defect is as hard to spot as the model happened to make it, and comparing the two needs the very number being sought. No safety factor recovers it, because one wide enough for every miss rate releases nothing. The failure also runs the dangerous way, because corrections that turn noise into one rate need the mistakes to be unrelated to the case, whereas a defect is missed because it read well, so the risk on record sits below the true risk at every threshold. Across four simulated reviewers that condition never holds, and testing it on a separately drawn sample is affordable only where the answer was never in doubt: a reviewer missing 85% of defects costs 185 adjudicated cases against 147,938 for one missing 3%. Adjudicating the reviewer’s own cases instead removes that test and most of that cost, 498 cases for the careful reviewer, so modelling the noise costs more than measuring it. Our library therefore refuses the widened certificate unless the condition is confirmed on adjudicated data, and fails closed otherwise.