RadRef: A Reference-Free Metric for Radiology Report Generation
AI-generated radiology reports are commonly evaluated against radiologist-authored references, limiting the applicability of standard reference-based metrics when no reference is available. We introduce RadRef, a reference-free metric that scores report claims against image-derived beliefs over a clinical finding vocabulary, weighted by relevance to the clinical indication. By evaluating findings beyond the report's stated claims, RadRef assesses omissions as well as unsupported assertions. On ReXVal, RadRef achieves $73.1\%$ agreement with radiologist rankings among non-tied same-radiograph report pairs, with $61.8\%$ pair coverage and $64.3\%$ tie-inclusive agreement. Its non-tied agreement is close to GREEN ($73.9\%$) and RadFact ($72.6\%$), with its strongest relative results on omissions. We further evaluate $100$ model-generated and $100$ radiologist-authored reports from $100$ MIMIC-CXR studies, independently annotated by two board-certified radiologists. On same-image omission rankings, RadRef attains Kendall's $τ_b = 0.295$, compared with $0.386$ between the readers. In separate paired comparisons across generated reports, no evaluated comparator significantly…
Introduces RadRef, a reference-free metric for radiology report generation.
- arXiv
- 2611.13168
- Authors
- Shelly Schwartz, Alan Ropp, Michael B. Sneider, Nur Yildirim, Cree M. Gaskin and 1 more
- Categories
- cs.LG