Where Groups Split: A Probe Prior for Nonzero Within-Group Variance in RL Post-Training

Jonah Ellison, Mateo Ruiz, Owen Pell, Tomas Duncun, Hanae Fujita +2

Group relative policy optimization learns from a prompt only when the rewards in the sampled group disagree. With a binary verifier, a group that is entirely correct or entirely incorrect has within-group variance zero and a zero advantage. Uniform sampling continues to spend rollouts on prompts that are unanimous now and stay unanimous as training goes on. Pass rate is a weak forecast of that future. Two prompts at the same pass rate can move apart: one into the range where a finite group splits, and one toward a ceiling where the group becomes unanimous again. Aperture estimates the later pass rate from a short probe. A small subset of the pool is trained with the same recipe, every prompt is scored on the base model and on the probe model, and the two pass rates are plugged into the probability that a group of the training size is all wrong or split. The resulting weight is a static prior. It can be used alone or multiplied by an online scheduler, and the GRPO update is left as it is. On mathematical and logical reasoning with 1.7B and 4B models, the prior improves greedy accuracy over uniform GRPO and raises the fraction of groups with a nonzero advantage. The same prior…