Abstract
A class of compact models does not generate text but returns typed decisions: a probability for a yes/no question, a distribution over a set of choices, or an expected level on an ordered rubric. These models are marketed on latency and calibration and are evaluated almost entirely by their own vendors. We present JevBench, an open evaluation framework that runs any model exposing this interface on identical inputs, and apply it to 12 slices of public datasets covering medicine, examinations, banking intent, sentiment and agent safety. We evaluate three models: Jev 1.13, a hosted commercial model; djev-0.1, an open model on a DiffusionGemma 26B-A4B backbone; and Laya, a 421M-parameter local model. Three findings hold across the suite. First, the models rank inputs well, but their probabilities sit away from the default 0.5 threshold: on prompt-injection detection the commercial model reaches an AUROC of 0.978 yet only 73.3% accuracy at 0.5, and moving the threshold alone raises this to 94.8% (thresholds chosen in-sample), so accuracy at 0.5 understates every model we tested. Second, sensitivity to option order differs by an order of magnitude: 2.6-5.4% of matched items flip under reordering for the commercial model, against 12.0-33.6% for the other two. Third, the 421M model is indistinguishable from random guessing on four of six set-of-choices slices, yet outperforms djev-0.1 on prompt-injection detection, so no single quality ordering holds. We also report the first evaluation we are aware of for the image path of an open typed-decision model: its accuracy on image-grounded multiple choice (82.8%) is close to that on text-only rows of the same dataset (84.7%). We release the framework and all 24,799 predictions with their full probability vectors.