Papers

Field

Top tools here

Open problems in Learning verifiable rewards

Poor credit assignment in diffusion model RL1 paper raises it · 1 addresses it
Limits efficiency and stability when fine-tuning diffusion models with black-box rewards.
- DiffusionART: Efficient, Stable and Scalable Reinforcement of Diffusion Modelsraises · addresses
Vanishing group-relative reward signals1 paper raises it · 1 addresses it
All rollouts receiving the same reward lose useful trajectory-level information for learning.
- Self-Retrospection Distillation: Turning Post-hoc Experiences into Prior Foresightraises · addresses
Memory inefficiency in LLM RL post-training1 paper raises it · 1 addresses it
High memory demands prevent scaling RL to larger models or longer training runs.
- LoGRA: Scaling LLM Reinforcement Learning with Low-Rank Gradient Sketchesraises · addresses
Sparse terminal rewards degrade value tracking1 paper raises it · 1 addresses it
Standard critics fail to model value evolution along trajectories, harming credit assignment.
- TPPO: Tracking-Value PPO For Language Model Reasoningraises · addresses
Reward hacking in rubric-based RL1 paper raises it · 1 addresses it
Additive reward aggregation allows policies to exploit criteria compensation, reducing task appropriateness.
- Scoring Higher, Answering Worse: Mitigating Reward Hacking in Rubric-Based RL via Protocol-Level Rubricsraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Topics in this field

learning verifiable rewards · reinforcement learning rl · reinforcement learning verifiable321 papers59 this week
optimal regret · near optimal · bandits212 papers38 this week
offline reinforcement learning · offline reinforcement · reinforcement learning rl174 papers30 this week
reinforcement learning rl · reinforcement learning · markov decision processes103 papers17 this week
policy gradient · reinforcement learning rl · reinforcement learning verifiable91 papers17 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Coming with more sources: repos

Activity

Recent papers

All in Research →

arXiv cs.LGOct 13
Follow the Perturbed Leader for Linear Bandits: $\sqrt{T}$ Regret over General Convex Sets
Chen-Yu Wei
We give the first adversarial linear bandit algorithm based on Follow-the-Perturbed-Leader (FTPL) that achieves $\sqrt{T}$ regret over general convex action sets. Our algorithm attains $\widetilde{\mathcal{O}}(d^{5/4}\sqrt{T})$ regret using only linear optimization oracles, and the near-optimal $\widetilde{\mathcal{O}}(d\sqrt{T})$ regret when an optimal-design exploration distribution is further available. This improves on the $T^{2/3}$ regret of the FTPL algorithm of Dani and Hayes (2006) for general convex sets, and generalizes the $\sqrt{T}$ regret of Lévy et al. (2026) over two special action sets (hypercube and Euclidean ball). Compared to previous oracle-efficient linear bandit algorithms based on continuous exponential weights, the FTPL approach generally has better computational efficiency, and is much simpler to implement. The key algorithmic element is a perturbation distribution that adapts to the past reward estimators. Its simplest form, an adaptive Gaussian perturbation, is straightforward to implement and already attains the regret bound above. We further extend it to two general constructions, random-weighting and elliptical perturbations, each covering…
PDF arXiv
arXiv cs.CLCodeOct 13▲ 70
Many Right Answers, One Training Example: Removing Responder Noise from GRPO for LLM Recommenders
Xin Li, Hao Jiang, Jinghao Guo, Weisi Lin, Chau Yuen
A recommendation request on Reddit draws several replies, each naming different titles. When an LLM recommender is trained on such threads with reinforcement learning (RL), which replies should a generated list be rewarded against? The state-of-the-art recipe, Rank-GRPO, picks one reply per training example, so the advantage mostly records which reply was drawn (85% of its variance) rather than how good the list is. We argue that RL should train on the request, not on each answer, and propose ReMaRk: it scores every list once against the union of all replies' titles and samples each request equally. This makes RL cheaper. On seven backbones and three datasets, ReMaRk reaches the validation recall of 2,000 per-reply RL steps in 25 to 53% of the steps and leads at that budget; trained until neither improves, it is never significantly behind Rank-GRPO and in four of seven pairs reaches Rank-GRPO's best with fewer GPU-hours. The same reward helps on ProtoQA, where about 100 people answer each question, which points to a simple rule for RL with several valid human answers: reward them jointly.
PDF arXiv Code
arXiv cs.LGOct 13
Explore with Memory, Learn without It: In-Context Exploration and Self-Distillation for LLM Reasoning
Yutong Chen, Zhichao Xu, Rui Liu, Kunpeng Liu
Exploration bounds learning efficiency in reinforcement learning with verifiable rewards (RLVR). Current methods explore mostly through independent sampling, without fully using the information gathered along the way, such as known wrong answers and environment feedback. We propose LEAF, which injects a dynamic, problem-specific memory into the context before every rollout. The memory contains the most recent response, its reward, and other environment feedback. It is updated after every verifier call. We decouple policy updates from rollouts and update the policy through self-distillation. On four benchmarks and two open models, LEAF accelerates exploration by about 4x compared with a random sampling baseline. On unseen test sets, it improves accuracy over the strongest baseline by 1.59 and 1.83 percentage points on the two models, while using less compute than any RL baseline. These results highlight the potential of LEAF in improving both exploration and learning efficiency. Code is available at: https://github.com/on1262/LEAF
PDF arXiv
arXiv cs.LGOct 13
CoCA: Criterion-Aware Credit Assignment for Rubric-Based Reinforcement Learning
Guangcheng Zhu, Yingfan Ma, Xing Zheng, Haobo Wang, Shenzhi Yang +5
Rubric-based reinforcement learning (RL) evaluates responses against explicit criteria to construct rewards, offering a viable way to train large language models on open-ended tasks. However, standard methods compress criterion-level feedback into scalar rewards and assign uniform token advantages, leaving structured rubric supervision underutilized. Recent work seeks finer-grained guidance by treating rubrics as privileged information to derive self-distillation targets, but the resulting learning signal is not grounded in criterion outcomes and can lead to reward degradation. This motivates us to reframe the problem as criterion-aware credit assignment, where privileged rubric information serves as a relevance cue for credit allocation rather than a source of imitation targets. Accordingly, we propose CoCA, which derives criterion-level credits from group-relative outcomes and allocates them across tokens based on criterion-induced distribution shifts, yielding both outcome-grounded and fine-grained learning signals. Empirically, CoCA achieves an average gain of +4.2% over standard rubric-based RL across medicine and science benchmarks, demonstrating more effective learning on…
PDF arXiv
arXiv cs.AIOct 13
Adaptive-Group Process Rewards for GUI Agents
Zirui Shang, Yang Liu, Xin Shu, Zhi Gao, Xinxiao Wu +1
Training graphical user interface (GUI) agents with reinforcement learning suffers from sparse outcome rewards, leading to inefficient learning. Process rewards offer a promising direction for credit assignment to intermediate actions; however, accurately evaluating an action requires identifying evidence across long-horizon GUI trajectories, as its value may depend on information beyond its immediate state and observation. In this paper, we propose an adaptive-group process reward framework that dynamically partitions a GUI trajectory into semantically coherent subgoal groups, enabling process reward estimation at both the action and group levels. At the action level, a critic evaluates each action based on the visual changes and subsequent actions within the group for action correctness and contribution to the subgoal. At the group level, the framework aggregates action-level evaluations to assess subgoal completion and contribution to the overall task, and further refines these assessments by leveraging the context of all groups in the trajectory. The process reward of each action is obtained by combining its action-level evaluation with the refined group-level evaluation. We…
PDF arXiv
arXiv cs.LGCodeOct 13▲ 34
DeltaTTG: Training Multi-Turn Agents with a Token-to-Go Progress Meter
Xuyan Ye, Xiyan Fu, Shengda Fan, Jiawei Cui, Yankai Lin +1
Reinforcement learning with verifiable rewards (RLVR) for multi-turn agents relies on sparse terminal rewards that say little about individual steps. Within-group step comparisons depend on reliable state matching and sufficiently diverse sampled outcomes, while direct step scoring may require costly external evaluators, human annotations, or task-specific heuristics. We introduce DeltaTTG to learn a reusable progress signal through a shared progress meter, using low-cost supervision from diverse policies' recorded trajectories without task-specific progress rules. Its supervision target, Token-to-Go (TTG), the number of tokens remaining to be generated until task completion, serves as an observable proxy for remaining completion effort. During actor training, changes in the frozen meter's estimates redistribute outcome-derived advantages across steps without altering the reward, preserving each trajectory's advantage sign and token-average scale. On ALFWorld, PaperSearchQA, and BFCLv3, DeltaTTG improves the average score over GRPO by 5.7 and 7.2 points with Qwen3-4B and Llama-3.1-8B-Instruct, respectively, outperforms all compared RL baselines on average, and keeps…
PDF arXiv Code

Papers

Topics in this field

learning verifiable rewards · reinforcement learning rl · reinforcement learning verifiable321 papers59 this week
optimal regret · near optimal · bandits212 papers38 this week
offline reinforcement learning · offline reinforcement · reinforcement learning rl174 papers30 this week
reinforcement learning rl · reinforcement learning · markov decision processes103 papers17 this week
policy gradient · reinforcement learning rl · reinforcement learning verifiable91 papers17 this week

Active contributors · 90 days

Recent papers

All in Research →

arXiv cs.LGOct 13
Follow the Perturbed Leader for Linear Bandits: $\sqrt{T}$ Regret over General Convex Sets
Chen-Yu Wei
We give the first adversarial linear bandit algorithm based on Follow-the-Perturbed-Leader (FTPL) that achieves $\sqrt{T}$ regret over general convex action sets. Our algorithm attains $\widetilde{\mathcal{O}}(d^{5/4}\sqrt{T})$ regret using only linear optimization oracles, and the near-optimal $\widetilde{\mathcal{O}}(d\sqrt{T})$ regret when an optimal-design exploration distribution is further available. This improves on the $T^{2/3}$ regret of the FTPL algorithm of Dani and Hayes (2006) for general convex sets, and generalizes the $\sqrt{T}$ regret of Lévy et al. (2026) over two special action sets (hypercube and Euclidean ball). Compared to previous oracle-efficient linear bandit algorithms based on continuous exponential weights, the FTPL approach generally has better computational efficiency, and is much simpler to implement. The key algorithmic element is a perturbation distribution that adapts to the past reward estimators. Its simplest form, an adaptive Gaussian perturbation, is straightforward to implement and already attains the regret bound above. We further extend it to two general constructions, random-weighting and elliptical perturbations, each covering…
PDF arXiv
arXiv cs.CLCodeOct 13▲ 70
Many Right Answers, One Training Example: Removing Responder Noise from GRPO for LLM Recommenders
Xin Li, Hao Jiang, Jinghao Guo, Weisi Lin, Chau Yuen
A recommendation request on Reddit draws several replies, each naming different titles. When an LLM recommender is trained on such threads with reinforcement learning (RL), which replies should a generated list be rewarded against? The state-of-the-art recipe, Rank-GRPO, picks one reply per training example, so the advantage mostly records which reply was drawn (85% of its variance) rather than how good the list is. We argue that RL should train on the request, not on each answer, and propose ReMaRk: it scores every list once against the union of all replies' titles and samples each request equally. This makes RL cheaper. On seven backbones and three datasets, ReMaRk reaches the validation recall of 2,000 per-reply RL steps in 25 to 53% of the steps and leads at that budget; trained until neither improves, it is never significantly behind Rank-GRPO and in four of seven pairs reaches Rank-GRPO's best with fewer GPU-hours. The same reward helps on ProtoQA, where about 100 people answer each question, which points to a simple rule for RL with several valid human answers: reward them jointly.
PDF arXiv Code
arXiv cs.LGOct 13
Explore with Memory, Learn without It: In-Context Exploration and Self-Distillation for LLM Reasoning
Yutong Chen, Zhichao Xu, Rui Liu, Kunpeng Liu
Exploration bounds learning efficiency in reinforcement learning with verifiable rewards (RLVR). Current methods explore mostly through independent sampling, without fully using the information gathered along the way, such as known wrong answers and environment feedback. We propose LEAF, which injects a dynamic, problem-specific memory into the context before every rollout. The memory contains the most recent response, its reward, and other environment feedback. It is updated after every verifier call. We decouple policy updates from rollouts and update the policy through self-distillation. On four benchmarks and two open models, LEAF accelerates exploration by about 4x compared with a random sampling baseline. On unseen test sets, it improves accuracy over the strongest baseline by 1.59 and 1.83 percentage points on the two models, while using less compute than any RL baseline. These results highlight the potential of LEAF in improving both exploration and learning efficiency. Code is available at: https://github.com/on1262/LEAF
PDF arXiv
arXiv cs.LGOct 13
CoCA: Criterion-Aware Credit Assignment for Rubric-Based Reinforcement Learning
Guangcheng Zhu, Yingfan Ma, Xing Zheng, Haobo Wang, Shenzhi Yang +5
Rubric-based reinforcement learning (RL) evaluates responses against explicit criteria to construct rewards, offering a viable way to train large language models on open-ended tasks. However, standard methods compress criterion-level feedback into scalar rewards and assign uniform token advantages, leaving structured rubric supervision underutilized. Recent work seeks finer-grained guidance by treating rubrics as privileged information to derive self-distillation targets, but the resulting learning signal is not grounded in criterion outcomes and can lead to reward degradation. This motivates us to reframe the problem as criterion-aware credit assignment, where privileged rubric information serves as a relevance cue for credit allocation rather than a source of imitation targets. Accordingly, we propose CoCA, which derives criterion-level credits from group-relative outcomes and allocates them across tokens based on criterion-induced distribution shifts, yielding both outcome-grounded and fine-grained learning signals. Empirically, CoCA achieves an average gain of +4.2% over standard rubric-based RL across medicine and science benchmarks, demonstrating more effective learning on…
PDF arXiv
arXiv cs.AIOct 13
Adaptive-Group Process Rewards for GUI Agents
Zirui Shang, Yang Liu, Xin Shu, Zhi Gao, Xinxiao Wu +1
Training graphical user interface (GUI) agents with reinforcement learning suffers from sparse outcome rewards, leading to inefficient learning. Process rewards offer a promising direction for credit assignment to intermediate actions; however, accurately evaluating an action requires identifying evidence across long-horizon GUI trajectories, as its value may depend on information beyond its immediate state and observation. In this paper, we propose an adaptive-group process reward framework that dynamically partitions a GUI trajectory into semantically coherent subgoal groups, enabling process reward estimation at both the action and group levels. At the action level, a critic evaluates each action based on the visual changes and subsequent actions within the group for action correctness and contribution to the subgoal. At the group level, the framework aggregates action-level evaluations to assess subgoal completion and contribution to the overall task, and further refines these assessments by leveraging the context of all groups in the trajectory. The process reward of each action is obtained by combining its action-level evaluation with the refined group-level evaluation. We…
PDF arXiv
arXiv cs.LGCodeOct 13▲ 34
DeltaTTG: Training Multi-Turn Agents with a Token-to-Go Progress Meter
Xuyan Ye, Xiyan Fu, Shengda Fan, Jiawei Cui, Yankai Lin +1
Reinforcement learning with verifiable rewards (RLVR) for multi-turn agents relies on sparse terminal rewards that say little about individual steps. Within-group step comparisons depend on reliable state matching and sufficiently diverse sampled outcomes, while direct step scoring may require costly external evaluators, human annotations, or task-specific heuristics. We introduce DeltaTTG to learn a reusable progress signal through a shared progress meter, using low-cost supervision from diverse policies' recorded trajectories without task-specific progress rules. Its supervision target, Token-to-Go (TTG), the number of tokens remaining to be generated until task completion, serves as an observable proxy for remaining completion effort. During actor training, changes in the frozen meter's estimates redistribute outcome-derived advantages across steps without altering the reward, preserving each trajectory's advantage sign and token-average scale. On ALFWorld, PaperSearchQA, and BFCLv3, DeltaTTG improves the average score over GRPO by 5.7 and 7.2 points with Qwen3-4B and Llama-3.1-8B-Instruct, respectively, outperforms all compared RL baselines on average, and keeps…
PDF arXiv Code

Activity · 12 weeks

Coming with more sources: repos

Open problems in Learning verifiable rewards

Poor credit assignment in diffusion model RL1 paper raises it · 1 addresses it
Limits efficiency and stability when fine-tuning diffusion models with black-box rewards.
- DiffusionART: Efficient, Stable and Scalable Reinforcement of Diffusion Modelsraises · addresses
Vanishing group-relative reward signals1 paper raises it · 1 addresses it
All rollouts receiving the same reward lose useful trajectory-level information for learning.
- Self-Retrospection Distillation: Turning Post-hoc Experiences into Prior Foresightraises · addresses
Memory inefficiency in LLM RL post-training1 paper raises it · 1 addresses it
High memory demands prevent scaling RL to larger models or longer training runs.
- LoGRA: Scaling LLM Reinforcement Learning with Low-Rank Gradient Sketchesraises · addresses
Sparse terminal rewards degrade value tracking1 paper raises it · 1 addresses it
Standard critics fail to model value evolution along trajectories, harming credit assignment.
- TPPO: Tracking-Value PPO For Language Model Reasoningraises · addresses
Reward hacking in rubric-based RL1 paper raises it · 1 addresses it
Additive reward aggregation allows policies to exploit criteria compensation, reducing task appropriateness.
- Scoring Higher, Answering Worse: Mitigating Reward Hacking in Rubric-Based RL via Protocol-Level Rubricsraises · addresses

Named by an AI from these papers' abstracts.

Fields · AI · updated 38m ago

Learning verifiable rewards

Found by the topic model: learning verifiable rewards, optimal regret, offline reinforcement learning.

161papers this week−16.1% vs last week

All papers · 901 With code · 27

Newest Most relevant

857 papers this month