Abstract
Firms increasingly route customer requests with language models, and a new class of decision models (Jev, Clef and Perplexity Decider) returns a probability for every allowed option instead of generated text. We benchmarked these models on 10,580 customer-service requests from BANKING77, Bitext and CLINC150. The comparison set comprised the probability readout of Qwen3.8-27B, the backbone from which Clef and Perplexity Decider are post-trained; two generative large language models (LLMs) with verbalized confidence; and trained classifiers. Clef raised accuracy over this backbone readout by 14.9 percentage points on BANKING77 and 9.3 points on CLINC150, but not on Bitext (−1.5 points), the dataset least likely to have been a vendor benchmark target. Perplexity Decider added 1.2–3.0 points. The backbone readout was already well calibrated (expected calibration error 0.019–0.041), whereas verbalized confidence was not (0.048–0.166), so decision post-training gave small and inconsistent calibration gains. With an explicit out-of-scope option, Clef detected 93.7% of out-of-scope requests at a 0.2% false-alarm rate, while trained classifiers caught 56–64%. Reordering the options changed 0.8–11.2% of the decisions of every model except Clef, which is order-invariant by construction. Hosted decision APIs cost US$0.02–0.07 per 1,000 decisions. We derive guidance on model choice, automation thresholds and out-of-scope handling.