Typed Decision Models under Prevalence Shift: Decision Training, Calibration, and Implicit Priors

Abdul Rahman Iqbal, Muhammad Fawad Nasim, Amir Saleem

Typed decision models (TDMs) return a probability over caller-defined options, and Jev is described by its vendor as calibrated. But in deployment class frequencies change, so calibration measured on a fixed test set need not hold where thresholds are applied. Existing evaluations leave two gaps: the gains of vendor decision training have not been separated from those of the backbone under an identical readout, and TDMs have been tested only at the test set's class distribution. We ask whether decision training improves a model over its own backbone, and whether a TDM's calibration and thresholds withstand prevalence (label) shift as well as a fine-tuned classifier's. We compare three TDMs (Jev, Tev1-4B, Nimble-9B), their base models and two trained classifiers on identical decisions from five public datasets under simulated shift, with 35 paired tests fixed before any test was computed. First, decision training is not uniformly beneficial: Nimble-9B, which differs from Qwen3.5-9B only by a LoRA adapter, is better calibrated on intent tasks but 12.0 Macro-F1 points worse on financial sentiment. Second, under moderate shift Jev's expected calibration error (ECE) and threshold risk…