Calibration-Constrained Diagnostics for Evaluating LLM Self-Evaluation

Jae Oh Woo, Mengdie Flora Wang, Rahul Ghosh, Baishali Chaudhury, Mun Young Kim

We introduce a reusable evaluation artifact for diagnosing whether LLM self-reported confidence can support confidence-gated act-or-defer decisions. The artifact is designed for evaluation rather than safety certification: it provides a black-box protocol, prompts, judge-reference labels, model outputs, reproduction scripts, and calibration-constrained diagnostic metrics for auditing LLM self-evaluation across math, factual, and commonsense reasoning tasks. Our central finding is that aggregate uncertainty summaries can produce deployment-misleading rankings: a model that appears strongest under standard aggregate calibration metrics can fail under category-level calibration constraints, while a model with worse aggregate calibration can exhibit stronger calibration-constrained behavior in the positive-margin region where high self-selection accuracy and tight calibration coexist. To expose this failure mode, we define PVC and C-PVC as category-level diagnostics over an accuracy-by-calibration surface: PVC counts categories whose self-evaluation accuracy exceeds a threshold, while C-PVC additionally requires category-level calibration error below a tolerance. We aggregate these…