Papers

Field

Top tools here

Open problems in Learning verifiable rewards

Poor credit assignment in diffusion model RL1 paper raises it · 1 addresses it
Limits efficiency and stability when fine-tuning diffusion models with black-box rewards.
- DiffusionART: Efficient, Stable and Scalable Reinforcement of Diffusion Modelsraises · addresses
Vanishing group-relative reward signals1 paper raises it · 1 addresses it
All rollouts receiving the same reward lose useful trajectory-level information for learning.
- Self-Retrospection Distillation: Turning Post-hoc Experiences into Prior Foresightraises · addresses
Memory inefficiency in LLM RL post-training1 paper raises it · 1 addresses it
High memory demands prevent scaling RL to larger models or longer training runs.
- LoGRA: Scaling LLM Reinforcement Learning with Low-Rank Gradient Sketchesraises · addresses
Sparse terminal rewards degrade value tracking1 paper raises it · 1 addresses it
Standard critics fail to model value evolution along trajectories, harming credit assignment.
- TPPO: Tracking-Value PPO For Language Model Reasoningraises · addresses
Reward hacking in rubric-based RL1 paper raises it · 1 addresses it
Additive reward aggregation allows policies to exploit criteria compensation, reducing task performance.
- Scoring Higher, Answering Worse: Mitigating Reward Hacking in Rubric-Based RL via Protocol-Level Rubricsraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Topics in this field

learning verifiable rewards · reinforcement learning rl · reinforcement learning verifiable336 papers49 this week
offline reinforcement learning · offline reinforcement · reinforcement learning rl190 papers33 this week
optimal regret · near optimal · bandits221 papers32 this week
reinforcement learning rl · reinforcement learning · markov decision processes108 papers16 this week
policy gradient · reinforcement learning rl · reinforcement learning verifiable94 papers14 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Activity

Recent papers

All papers in Research

arXiv cs.AISep 30
Divide and Collapse: MAPF-Collapse via Exact Decomposition into Independent Sub-Instances
Oren Salzman
In this work we study the problem of MAPFC, a post-optimization step for Multi-Agent Path Finding (MAPF) plans where we are given a feasible plan produced by a modern MAPF solver and are tasked with removing avoidable moves while preserving feasibility. This NP-hard problem naturally arises when using learning-based state-of-the-art (SOTA) solvers which construct plans that contain redundant moves that can be removed. Recently, Tang et al. presented Judgelight, which uses Integer Linear Programming (ILP) to solve MAPFC. Importantly, the ILP is constructed over all agents jointly, so its cost is governed by the full instance rather than by the small coupled residue that actually requires joint reasoning. Our key insight, motivating this work, is that MAPFC instances naturally decompose into independent sub-problems, most of which involve a single agent and can be solved without any inter-agent reasoning. To this end, we first identify which agents need to coordinate their motion and partition the instance into sub-problems accordingly. For the cases where no coordination is required, we introduce an extremely lightweight solver that is $\approx\!1{,}900\times$ faster than…
PDF arXiv
arXiv cs.LGSep 30
T2SPO: Trajectory-to-Step Policy Optimization for Agentic Reinforcement Learning
Bo-Wen Zhang, Junwei He, Maoqi Liu, Feiran Li, Song-Lin Lv +4
Reinforcement learning enables large language model (LLM) agents to learn multi-step behaviors through interaction with their environments. However, rewards in many interactive tasks reflect only the final outcome, providing limited guidance on which intermediate decisions advance the task. Successful training trajectories contain intermediate states that can provide supervision for subsequent interactions. We introduce Trajectory-to-Step Policy Optimization (T2SPO), a method that uses past interaction trajectories to provide step-level feedback for policy learning. T2SPO derives remaining-distance targets from successful trajectories and pairs them with representations of the states visited along the way. Conditioned on these examples, a pretrained TabPFN regressor estimates the remaining distance to success at each state of a new rollout. Changes in this distance estimate across consecutive states yield auxiliary credit for agent steps alongside task-level supervision. As training proceeds, newly completed trajectories refresh the estimator's context, incorporating new experience without updating its parameters. Experiments with 1.5B and 7B language models on ALFWorld and…
PDF arXiv
arXiv cs.AISep 30
Advancing Entropy-Level Credit Assignment in RLVR via Proximal Entropy Policy Optimization
Yun Kim, Nojun Kwak
Value-model-free RLVR methods such as GRPO assign uniform advantages to all tokens in a rollout, ignoring that tokens contribute unequally. Recent methods use token entropy as an importance proxy but compute it globally across the batch, conflating importance with prompt difficulty and positional trends. We argue that importance should instead be measured relative to the local context of each token. We introduce proximal entropy, a local measure of token importance relative to neighboring tokens, and prove it is invariant to both confounders. Proximal Entropy Policy Optimization (PEPO) uses it to weight per-token advantages and outperforms GRPO and entropy-based baselines on mathematical reasoning across Qwen3-1.7B, Qwen3-4B, and Llama-3.2-3B-Instruct. We also show the formulation generalizes to other algorithms where substituting proximal entropy into existing methods improves, and applying it to single-stream RL succeeds where global entropy fails.
PDF arXiv
arXiv stat.MLSep 30
Towards Optimal Inventory Control under Censored Demand: A Biased Sample-Average Approximation Approach
Yuxuan Han, Xiaoyu Fan, Jiawei Zhang, Zhengyuan Zhou
We study data-driven multi-period lost-sales inventory control under censored demand, where a stockout reveals only that demand exceeded the stocking level. We develop a unified, model-based framework for policy learning from censored data, built on a new cost decomposition for base-stock policies and a biased sample-average approximation (SAA) approach. The cost decomposition allows us to propose a new coverage condition under which censored observations are informative enough for sample-efficient policy learning. Guided by this coverage condition, we design two biased SAA algorithms: an upper-biased one that achieves near-optimal sample complexity under the offline coverage condition, and a lower-biased one that actively generates the required coverage and achieves near-optimal regret online. More broadly, this biased SAA approach provides a general principle for implementing pessimism and optimism under censored feedback, which may be of independent interest.
PDF arXiv
arXiv cs.LGSep 30
FAER: Auditable Utility-Aligned Trajectory Replay for Language Model Post-Training
Miaobo Hu, Shuhao Hu, Xiaobo Guo, Xin Wang, Bokun Wang +3
Replay selectors often rank cached trajectories by format feedback, confidence, freshness, or response length, although cache-level correctness and downstream learner utility are distinct objectives. We formalize this selection-to-learning gap and introduce FAER as an auditable full-trajectory replay framework. Its training-free fixed selector is a protocol baseline; FAER-UTILITY is the learner-aware selector fitted on disjoint calibration blocks. The normalized gradient alignment is reported as a baseline, while a disposable optimizer-aware virtual update supplies a magnitude-aware utility surface. The audit contract freezes observed fields and replay traces before evaluation labels are joined. On GSM8K with Qwen2.5-1.5B-Instruct, the matched learner study reports quality 0.6329 for the fixed selector, compared with 0.5482 for uniform and 0.6037 for format-feedback under 128 updates. Metadata-only cross-fitted calibration reaches $0.6476\!\pm\!0.0139$ over eight seeds (median 0.6481; paired 95% interval $[+0.079,+0.122]$) at 63,276 target-run tokens; its recorded full cost is 189,642 tokens and 3.48 GPU-hours including calibration. The completed FAER-UTILITY row reaches 0.6624…
PDF arXiv
arXiv cs.AISep 30
On the Complexity of Preference-Based Bandits
Ahmed Ben Yahmed, Marc Abeille, Clément Calauzènes
We study preference-based bandits with general reward function classes, where a learner sequentially selects pairs of arms and observes binary preference feedback governed by the Bradley--Terry model. This setting naturally arises in applications such as recommender systems, tournament ranking, and learning from human feedback, where relative preferences are easier to elicit than absolute rewards. The observation model inherits the logistic bandit challenge of handling the problem-dependent constant $κ$, which accounts for the non-linearity of the link function and can grow arbitrarily large. Moreover, prior work has predominantly focused on linear or kernelized reward models, precluding the use of richer function classes. To address these limitations, we consider general reward function classes and introduce the \emph{locally sensitive eluder dimension}, a novel complexity measure tailored to the logistic structure of preference feedback that yields fine-grained regret guarantees without unfavorable dependence on $κ$. Building on this notion, we propose \textbf{GINOP} (Generic INformative OPtimism), an algorithm that constructs log-loss confidence sets and jointly selects arm…
PDF arXiv

Papers

Topics in this field

learning verifiable rewards · reinforcement learning rl · reinforcement learning verifiable336 papers49 this week
offline reinforcement learning · offline reinforcement · reinforcement learning rl190 papers33 this week
optimal regret · near optimal · bandits221 papers32 this week
reinforcement learning rl · reinforcement learning · markov decision processes108 papers16 this week
policy gradient · reinforcement learning rl · reinforcement learning verifiable94 papers14 this week

Active contributors · 90 days

Recent papers

All papers in Research

arXiv cs.AISep 30
Divide and Collapse: MAPF-Collapse via Exact Decomposition into Independent Sub-Instances
Oren Salzman
In this work we study the problem of MAPFC, a post-optimization step for Multi-Agent Path Finding (MAPF) plans where we are given a feasible plan produced by a modern MAPF solver and are tasked with removing avoidable moves while preserving feasibility. This NP-hard problem naturally arises when using learning-based state-of-the-art (SOTA) solvers which construct plans that contain redundant moves that can be removed. Recently, Tang et al. presented Judgelight, which uses Integer Linear Programming (ILP) to solve MAPFC. Importantly, the ILP is constructed over all agents jointly, so its cost is governed by the full instance rather than by the small coupled residue that actually requires joint reasoning. Our key insight, motivating this work, is that MAPFC instances naturally decompose into independent sub-problems, most of which involve a single agent and can be solved without any inter-agent reasoning. To this end, we first identify which agents need to coordinate their motion and partition the instance into sub-problems accordingly. For the cases where no coordination is required, we introduce an extremely lightweight solver that is $\approx\!1{,}900\times$ faster than…
PDF arXiv
arXiv cs.LGSep 30
T2SPO: Trajectory-to-Step Policy Optimization for Agentic Reinforcement Learning
Bo-Wen Zhang, Junwei He, Maoqi Liu, Feiran Li, Song-Lin Lv +4
Reinforcement learning enables large language model (LLM) agents to learn multi-step behaviors through interaction with their environments. However, rewards in many interactive tasks reflect only the final outcome, providing limited guidance on which intermediate decisions advance the task. Successful training trajectories contain intermediate states that can provide supervision for subsequent interactions. We introduce Trajectory-to-Step Policy Optimization (T2SPO), a method that uses past interaction trajectories to provide step-level feedback for policy learning. T2SPO derives remaining-distance targets from successful trajectories and pairs them with representations of the states visited along the way. Conditioned on these examples, a pretrained TabPFN regressor estimates the remaining distance to success at each state of a new rollout. Changes in this distance estimate across consecutive states yield auxiliary credit for agent steps alongside task-level supervision. As training proceeds, newly completed trajectories refresh the estimator's context, incorporating new experience without updating its parameters. Experiments with 1.5B and 7B language models on ALFWorld and…
PDF arXiv
arXiv cs.AISep 30
Advancing Entropy-Level Credit Assignment in RLVR via Proximal Entropy Policy Optimization
Yun Kim, Nojun Kwak
Value-model-free RLVR methods such as GRPO assign uniform advantages to all tokens in a rollout, ignoring that tokens contribute unequally. Recent methods use token entropy as an importance proxy but compute it globally across the batch, conflating importance with prompt difficulty and positional trends. We argue that importance should instead be measured relative to the local context of each token. We introduce proximal entropy, a local measure of token importance relative to neighboring tokens, and prove it is invariant to both confounders. Proximal Entropy Policy Optimization (PEPO) uses it to weight per-token advantages and outperforms GRPO and entropy-based baselines on mathematical reasoning across Qwen3-1.7B, Qwen3-4B, and Llama-3.2-3B-Instruct. We also show the formulation generalizes to other algorithms where substituting proximal entropy into existing methods improves, and applying it to single-stream RL succeeds where global entropy fails.
PDF arXiv
arXiv stat.MLSep 30
Towards Optimal Inventory Control under Censored Demand: A Biased Sample-Average Approximation Approach
Yuxuan Han, Xiaoyu Fan, Jiawei Zhang, Zhengyuan Zhou
We study data-driven multi-period lost-sales inventory control under censored demand, where a stockout reveals only that demand exceeded the stocking level. We develop a unified, model-based framework for policy learning from censored data, built on a new cost decomposition for base-stock policies and a biased sample-average approximation (SAA) approach. The cost decomposition allows us to propose a new coverage condition under which censored observations are informative enough for sample-efficient policy learning. Guided by this coverage condition, we design two biased SAA algorithms: an upper-biased one that achieves near-optimal sample complexity under the offline coverage condition, and a lower-biased one that actively generates the required coverage and achieves near-optimal regret online. More broadly, this biased SAA approach provides a general principle for implementing pessimism and optimism under censored feedback, which may be of independent interest.
PDF arXiv
arXiv cs.LGSep 30
FAER: Auditable Utility-Aligned Trajectory Replay for Language Model Post-Training
Miaobo Hu, Shuhao Hu, Xiaobo Guo, Xin Wang, Bokun Wang +3
Replay selectors often rank cached trajectories by format feedback, confidence, freshness, or response length, although cache-level correctness and downstream learner utility are distinct objectives. We formalize this selection-to-learning gap and introduce FAER as an auditable full-trajectory replay framework. Its training-free fixed selector is a protocol baseline; FAER-UTILITY is the learner-aware selector fitted on disjoint calibration blocks. The normalized gradient alignment is reported as a baseline, while a disposable optimizer-aware virtual update supplies a magnitude-aware utility surface. The audit contract freezes observed fields and replay traces before evaluation labels are joined. On GSM8K with Qwen2.5-1.5B-Instruct, the matched learner study reports quality 0.6329 for the fixed selector, compared with 0.5482 for uniform and 0.6037 for format-feedback under 128 updates. Metadata-only cross-fitted calibration reaches $0.6476\!\pm\!0.0139$ over eight seeds (median 0.6481; paired 95% interval $[+0.079,+0.122]$) at 63,276 target-run tokens; its recorded full cost is 189,642 tokens and 3.48 GPU-hours including calibration. The completed FAER-UTILITY row reaches 0.6624…
PDF arXiv
arXiv cs.AISep 30
On the Complexity of Preference-Based Bandits
Ahmed Ben Yahmed, Marc Abeille, Clément Calauzènes
We study preference-based bandits with general reward function classes, where a learner sequentially selects pairs of arms and observes binary preference feedback governed by the Bradley--Terry model. This setting naturally arises in applications such as recommender systems, tournament ranking, and learning from human feedback, where relative preferences are easier to elicit than absolute rewards. The observation model inherits the logistic bandit challenge of handling the problem-dependent constant $κ$, which accounts for the non-linearity of the link function and can grow arbitrarily large. Moreover, prior work has predominantly focused on linear or kernelized reward models, precluding the use of richer function classes. To address these limitations, we consider general reward function classes and introduce the \emph{locally sensitive eluder dimension}, a novel complexity measure tailored to the logistic structure of preference feedback that yields fine-grained regret guarantees without unfavorable dependence on $κ$. Building on this notion, we propose \textbf{GINOP} (Generic INformative OPtimism), an algorithm that constructs log-loss confidence sets and jointly selects arm…
PDF arXiv

Activity · 12 weeks

Fields · AI · updated 27m ago

Learning verifiable rewards

Found by the topic model: learning verifiable rewards, optimal regret, offline reinforcement learning.

144papers this week−29.4% vs last week

All papers · 949 With code · 29

Newest Most relevant

879 papers this month

Open problems in Learning verifiable rewards

Poor credit assignment in diffusion model RL1 paper raises it · 1 addresses it
Limits efficiency and stability when fine-tuning diffusion models with black-box rewards.
- DiffusionART: Efficient, Stable and Scalable Reinforcement of Diffusion Modelsraises · addresses
Vanishing group-relative reward signals1 paper raises it · 1 addresses it
All rollouts receiving the same reward lose useful trajectory-level information for learning.
- Self-Retrospection Distillation: Turning Post-hoc Experiences into Prior Foresightraises · addresses
Memory inefficiency in LLM RL post-training1 paper raises it · 1 addresses it
High memory demands prevent scaling RL to larger models or longer training runs.
- LoGRA: Scaling LLM Reinforcement Learning with Low-Rank Gradient Sketchesraises · addresses
Sparse terminal rewards degrade value tracking1 paper raises it · 1 addresses it
Standard critics fail to model value evolution along trajectories, harming credit assignment.
- TPPO: Tracking-Value PPO For Language Model Reasoningraises · addresses
Reward hacking in rubric-based RL1 paper raises it · 1 addresses it
Additive reward aggregation allows policies to exploit criteria compensation, reducing task performance.
- Scoring Higher, Answering Worse: Mitigating Reward Hacking in Rubric-Based RL via Protocol-Level Rubricsraises · addresses

Named by an AI from these papers' abstracts.