Abstract
Abstract
Background: A guarded language-model pipeline can select a clinical calculator, extract inputs, execute deterministic code, and check whether an alternate representation returns the same value. Such agreement may verify arithmetic execution without establishing that the selected calculator fits the case.
Objective: To test whether CIBB-Guard's internal consistency check can pass after an inappropriate calculator choice and to compare the guarded configuration with direct prompting.
Methods: One frozen local Llama 3.1 8B model was evaluated on a prespecified 1,043-record benchmark and, post hoc, on 77 human-adjudicated de-identified case reports classified as eligible or ineligible for the evaluated calculators. Results: On 33 ineligible cases, CIBB-Guard correctly declined 21 but accepted 12 inappropriate answers; 11 of those 12 passed genuine alternate-representation re-verification. On 44 eligible cases, it produced 4 correct answers versus 0 with direct prompting. On the benchmark, all-record correctness was 631/1,043 (60.5%) for CIBB-Guard and 101/1,043 (9.7%) for direct prompting; all accepted confirmatory answers were synthetic.
Conclusions: Internal execution consistency does not establish calculator applicability. This one-model benchmark and case-series study supports separate applicability checks and does not establish clinical safety or deployment readiness.