One Reference or Many? Rethinking Multi-Reference Training for Intermediate-Entropy Text Generation

Maitreya Prafulla Chitale, Ketaki Mangesh Shetye, Yash More, Harshit Gupta, Manav Chaudhary +2

Natural language generation (NLG) ranges from highly constrained tasks, such as machine translation, to open-ended tasks like dialogue generation. Between them lie peer-review generation and image captioning, where one input admits several valid, overlapping outputs. We measure this intermediary region with Von Neumann entropy ($\hat{H}\approx0.6$ vs. 0.15 translation, 0.92 dialogue). Single-reference training discards valid perspectives, while multi-reference training introduces high-variance, unaligned targets. To tackle this, we propose target-side consolidation, that is, merging all references for an input into a single target which preserves every distinct information unit. To this end, we introduce MERC-67K, pairing 36,537 papers and 31,014 images with their original and consolidated references. In human evaluation, models trained using consolidated reviews obtain an average rank of 1.70 (out of 4) for coverage against 2.51-2.98 for baselines, and win 63.5-74.0% of pairwise comparisons. Models trained using consolidated captions obtain an average rank of 1.87 for overall caption quality (accuracy and naturalness) against 2.64-2.82 for baselines and obtain the highest…