Abstract
As digital self-triage becomes more widely used, quality assurance requires evaluation of both emergency recognition and overall agreement with clinical judgement, including how these outcomes differ across generations of large language models. The Benchmark Self-Triage (BeST) project intends to provide a framework to compare systems for digital self-triage against a medical 'gold standard' in Germany. In a first step developing the method and demonstrating its potential we developed a representative set of patient vignettes and compared two regulated medical-device triage systems, SmED and Infermedica, with five structured OpenAI configurations spanning multiple model generations: gpt-4.1, gpt-5.4, gpt-5.6-luna, gpt-5.6-sol, and gpt-6-astra. An LLM-based patient actor covered 480 physician-curated vignettes derived from German routine triage data. A pool of seven physicians provided preliminary reference classifications and consensus for 480 vignettes. Recommendations were mapped to three urgency categories 'emergency', 'acute', and 'not urgent' and evaluated using winsorised weights derived from routine triage frequencies. Weighted emergency sensitivity ranged from 51.3% (95% CI 35.5-66.3%) for gpt-4.1 to 99.9% (99.7-100.0%) for gpt-6-astra. Weighted exact three-category agreement (accuracy) ranged from 54.4% to 68.8%; gpt-4.1 achieved the highest agreement despite the lowest emergency sensitivity. No system performed best across all urgency categories. The later-generation OpenAI configurations showed a more precautionary triage profile resembling SmED, with higher sensitivity for emergency cases; gpt-6-astra showed the strongest shift. This cross-sectional comparison across model generations suggests a shift towards more precautionary triage behaviour, without establishing a longitudinal improvement in accuracy. These findings highlight the limitations of overall agreement for evaluating self-triage and support category-specific assessment. Broader expert consensus and evaluation, as well as further steps to reduce dependence on SmED-derived source data, are needed and planned before drawing conclusions about clinical safety.