When Do External Priors Help Low-Resource Structured Prediction? A Ranking-Level Alignment Study

Qiye Lu, Jiang Ji, Liang Zhang

Cold-start credit scoring (scarce labels, immature features, or minimal capacity) motivates injecting external priors (teacher models, LLMs, or domain experts) into training. We study Ranking Prior Alignment (RPA), a model-agnostic objective that aligns a model's ranking distribution to an external prior via a temperature-scaled KL loss with an annealed weight, $\mathcal{L} = \mathcal{L}_{\text{task}} + γ(t)\,\text{KL}(P_{\text{agent}} \| P_{\text{model}})$, and requires no external model at inference. Across tree-ensemble and multiple-instance-learning (MIL) instantiations on the public Amex dataset and an industrial merchant-credit dataset (1.5M+ merchants), we ask when such priors genuinely transfer information rather than merely act as a regularizer. Our central finding: the prior's quality $π$ (its rank correlation with the label) governs transfer only when the alignment granularity matches the granularity at which $π$ is measured. On cross-sample tree alignment (Amex), a high-$π$ teacher yields $+0.028$ AUC over the no-alignment baseline (paired bootstrap 95% CI $[0.022, 0.035]$) whereas zero-information priors (uniform, random) yield none, and $Δ$AUC tracks $π$ (XGBoost…