Papers

Field

Top tools here

Open problems in Learning verifiable rewards

Poor credit assignment in diffusion model RL1 paper raises it · 1 addresses it
Limits efficiency and stability when fine-tuning diffusion models with black-box rewards.
- DiffusionART: Efficient, Stable and Scalable Reinforcement of Diffusion Modelsraises · addresses
Vanishing group-relative reward signals1 paper raises it · 1 addresses it
All rollouts receiving the same reward lose useful trajectory-level information for learning.
- Self-Retrospection Distillation: Turning Post-hoc Experiences into Prior Foresightraises · addresses
Memory inefficiency in LLM RL post-training1 paper raises it · 1 addresses it
High memory demands prevent scaling RL to larger models or longer training runs.
- LoGRA: Scaling LLM Reinforcement Learning with Low-Rank Gradient Sketchesraises · addresses
Sparse terminal rewards degrade value tracking1 paper raises it · 1 addresses it
Standard critics fail to model value evolution along trajectories, harming credit assignment.
- TPPO: Tracking-Value PPO For Language Model Reasoningraises · addresses
Reward hacking in rubric-based RL1 paper raises it · 1 addresses it
Additive reward aggregation allows policies to exploit criteria compensation, reducing task appropriateness.
- Scoring Higher, Answering Worse: Mitigating Reward Hacking in Rubric-Based RL via Protocol-Level Rubricsraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Topics in this field

learning verifiable rewards · reinforcement learning rl · reinforcement learning verifiable329 papers58 this week
offline reinforcement learning · offline reinforcement · reinforcement learning rl186 papers40 this week
optimal regret · near optimal · bandits217 papers36 this week
reinforcement learning rl · reinforcement learning · markov decision processes105 papers17 this week
policy gradient · reinforcement learning rl · reinforcement learning verifiable94 papers17 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Activity

Recent papers

All papers in Research

arXiv cs.GTOct 5
Parallelism or Concession? Concurrency-Aware Procurement Negotiation for Agentic Commerce
Xiaolin Xu, Donghao Zhu
Agentic buyers can cheaply fork a procurement task into many parallel negotiations, but concurrency is not free: every thread consumes resources, and simultaneous agreements create cancellation and commitment risk. We study a one-unit post-order sourcing problem with a single hard-deadline negotiation window, in which a planner jointly chooses the number of seller-facing negotiators and a common procurement price cap. The model combines a product-specific acceptance curve with fulfillment loss, per-thread cost, and excess-commitment cost. We establish three structural results. First, holding the per-thread acceptance target fixed, the marginal value of another negotiator decays geometrically, yielding a conditional concurrency threshold. Second, under a convex quantile curve, parallelism substitutes for concession: more concurrent negotiators imply a weakly lower per-thread acceptance target and price cap. Third, when prices are more dispersed, Agentic buyers benefit by searching harder for bargains, but suffer when they instead try to guarantee procurement by offering higher prices. We operationalize these results in the Concurrency-Aware Negotiation Optimizer (CANO), a…
PDF arXiv
arXiv cs.LGOct 5
Learning While Scheduling Jobs under Context-Dependent Service Rates: An Anytime Rate-Optimal Algorithm
Seoungbin Bae, Dabeen Lee
We study contextual queueing bandits, where a learner schedules jobs while learning unknown service rates modeled by logistic functions of job-server features. Performance is measured by queue length regret, the expected excess queue length at round $t$ relative to an oracle that knows the service rates. Existing decaying-regret guarantees either have a suboptimal decay rate or require a known fixed horizon. They also assume context-wise slack and a strictly positive minimum eigenvalue of the feature covariance. In this paper, we propose WISE (Widest Interval Selection with Elimination), achieving rate-optimal $\widetilde{\mathcal O}(t^{-1/2})$ queue length regret at every sufficiently large time without knowing the horizon. We assume capacity slack, meaning that expected incoming workload under best-server service is below service capacity, and impose no covariance lower bound. Our analysis uses a workload potential measuring the expected service attempts needed by waiting jobs on their best servers. Its drift on nonempty rounds combines a negative term ensured by capacity slack with errors from suboptimal service choices. Then an elliptical potential count bounds how often WISE…
PDF arXiv
arXiv cs.LGOct 5
Reachability-Aware Diffusion Policy Optimization
Hikmet Simsir, Kutay Demiray, Ozgur S. Oguz
Diffusion policies provide expressive action distributions for continuous-control reinforcement learning. However, safety-aware online diffusion policy optimization remains underexplored, particularly methods that use predictive reachability information without an explicit dynamics model. We propose Reachability-Aware Diffusion Policy Optimization (RADPO), a model-free method that combines predictive first-hit safety estimation with cumulative-cost budget feedback. RADPO learns a discounted first-hit reachability value that captures the discounted risk of a cost event, assigns larger weight to events that occur sooner, and uses this signal to shape the reward. A separate dual-like multiplier adjusts the shaping strength according to realized episodic costs relative to a prescribed budget. The diffusion actor improves through weighted denoising regression on candidate actions scored by the reward critic. Our approach requires neither a learned dynamics model, action gradients through the critics, nor differentiation through the reverse diffusion sampler. We establish theoretical properties of the reachability value and show that accumulated reachability penalty provides a…
PDF arXiv
arXiv cs.LGOct 5
Fast Last-Iterate Convergence in Zero-Sum Markov Games with Bandit Feedback
Yuheng Zhang
We study last-iterate convergence in unknown two-player zero-sum discounted Markov games with bandit feedback. The players learn independently along a single trajectory without observing each other's actions. We develop Adaptive Regularized TD Learning (ARTD), which achieves a $\widetilde{\mathcal{O}}(t^{-1/4})$ duality gap bound for the current policies under a uniform hitting time assumption, with high probability simultaneously over all rounds and starting states. This improves the $\widetilde{\mathcal{O}}(t^{-1/(9+ν)})$ rate of Cai et al. (2023), for any fixed $ν>0$, under the same feedback model and hitting time assumption. Our algorithm requires no knowledge of the hitting time bound, the time horizon, or the confidence level. To stabilize policy learning as value estimates change, we separate fast temporal difference averaging from bounded value updates. We adapt log-barrier regularization to the progress of value estimation, controlling both policy and value errors throughout learning. Together, these mechanisms enable fast convergence of the policies actually played, even when the players learn independently from bandit feedback.
PDF arXiv
arXiv cs.LGOct 5
Strategic Multi-Agent Learning for Interpretable Action Valuation of All Players in Football
Kenjiro Ide, Taiga Someya, Kohei Kawaguchi, Keisuke Fujii
Valuing player actions in football requires accounting for strategic interactions among 22 players, including off-ball movements and defensive positioning. Existing reinforcement-learning-based methods commonly aggregate decisions at the team level or estimate player values independently, leaving strategic interdependence among players insufficiently represented. This study proposes an action valuation framework inspired by Markov perfect equilibrium (MPE) for all players. Each possession is modeled as a finite-horizon dynamic game, with each player represented as an autonomous agent whose policy depends on the current game state. MPE is used as a motivating solution concept rather than an exact equilibrium. To improve interpretability, we use Expandable Decision-Making States (EDMS) and decompose the Q-value into a successor-feature basis and a linear reward-weight vector. The value basis is estimated by linear TD initialization followed by nonlinear refinement. Using tracking and event data from 95 J1 League matches, we compare the proposed formulation with an independent reinforcement learning baseline. Because the two formulations define TD errors in different target spaces,…
PDF arXiv
arXiv cs.LGCodeOct 5▲ 3
MEND: RL For Flow Models via Proximal Velocity Matching
Shreshth Saini, Neil Birkbeck, Yilin Wang, Balu Adsumilli, Alan C. Bovik
Reward post-training of flow models either reweights the model's own samples under a KL penalty or a frozen reference, often for thousands of updates, or backpropagates the reward and moves every sample without checking that the move is worth its size. We introduce MEND, a reinforcement learning method built on proximal velocity matching. MEND caps rewards within each prompt group, so samples that already score well receive no move. Below the cap, it proposes moves along the reward gradient and accepts one only when its capped reward gain exceeds a quadratic displacement price. The model then regresses onto the resulting velocity targets, with no KL term, frozen reference model, or advantage weights. In 100 updates, MEND outperforms Flow-GRPO (about 4k updates) on five of six evaluators at the same distance to base-model images. Under an equal-budget protocol, it surpasses ReFL and DiffusionNFT at every evaluated update across four training rewards, reaching PickScore 24.03 versus 23.92 and 23.43, respectively. A 300-update three-reward run also surpasses the five-reward DiffusionNFT model on all three rewards it trains on. MEND is general and easy to adopt: it applies to any…
PDF arXiv Code

Papers

Topics in this field

learning verifiable rewards · reinforcement learning rl · reinforcement learning verifiable329 papers58 this week
offline reinforcement learning · offline reinforcement · reinforcement learning rl186 papers40 this week
optimal regret · near optimal · bandits217 papers36 this week
reinforcement learning rl · reinforcement learning · markov decision processes105 papers17 this week
policy gradient · reinforcement learning rl · reinforcement learning verifiable94 papers17 this week

Active contributors · 90 days

Recent papers

All papers in Research

arXiv cs.GTOct 5
Parallelism or Concession? Concurrency-Aware Procurement Negotiation for Agentic Commerce
Xiaolin Xu, Donghao Zhu
Agentic buyers can cheaply fork a procurement task into many parallel negotiations, but concurrency is not free: every thread consumes resources, and simultaneous agreements create cancellation and commitment risk. We study a one-unit post-order sourcing problem with a single hard-deadline negotiation window, in which a planner jointly chooses the number of seller-facing negotiators and a common procurement price cap. The model combines a product-specific acceptance curve with fulfillment loss, per-thread cost, and excess-commitment cost. We establish three structural results. First, holding the per-thread acceptance target fixed, the marginal value of another negotiator decays geometrically, yielding a conditional concurrency threshold. Second, under a convex quantile curve, parallelism substitutes for concession: more concurrent negotiators imply a weakly lower per-thread acceptance target and price cap. Third, when prices are more dispersed, Agentic buyers benefit by searching harder for bargains, but suffer when they instead try to guarantee procurement by offering higher prices. We operationalize these results in the Concurrency-Aware Negotiation Optimizer (CANO), a…
PDF arXiv
arXiv cs.LGOct 5
Learning While Scheduling Jobs under Context-Dependent Service Rates: An Anytime Rate-Optimal Algorithm
Seoungbin Bae, Dabeen Lee
We study contextual queueing bandits, where a learner schedules jobs while learning unknown service rates modeled by logistic functions of job-server features. Performance is measured by queue length regret, the expected excess queue length at round $t$ relative to an oracle that knows the service rates. Existing decaying-regret guarantees either have a suboptimal decay rate or require a known fixed horizon. They also assume context-wise slack and a strictly positive minimum eigenvalue of the feature covariance. In this paper, we propose WISE (Widest Interval Selection with Elimination), achieving rate-optimal $\widetilde{\mathcal O}(t^{-1/2})$ queue length regret at every sufficiently large time without knowing the horizon. We assume capacity slack, meaning that expected incoming workload under best-server service is below service capacity, and impose no covariance lower bound. Our analysis uses a workload potential measuring the expected service attempts needed by waiting jobs on their best servers. Its drift on nonempty rounds combines a negative term ensured by capacity slack with errors from suboptimal service choices. Then an elliptical potential count bounds how often WISE…
PDF arXiv
arXiv cs.LGOct 5
Reachability-Aware Diffusion Policy Optimization
Hikmet Simsir, Kutay Demiray, Ozgur S. Oguz
Diffusion policies provide expressive action distributions for continuous-control reinforcement learning. However, safety-aware online diffusion policy optimization remains underexplored, particularly methods that use predictive reachability information without an explicit dynamics model. We propose Reachability-Aware Diffusion Policy Optimization (RADPO), a model-free method that combines predictive first-hit safety estimation with cumulative-cost budget feedback. RADPO learns a discounted first-hit reachability value that captures the discounted risk of a cost event, assigns larger weight to events that occur sooner, and uses this signal to shape the reward. A separate dual-like multiplier adjusts the shaping strength according to realized episodic costs relative to a prescribed budget. The diffusion actor improves through weighted denoising regression on candidate actions scored by the reward critic. Our approach requires neither a learned dynamics model, action gradients through the critics, nor differentiation through the reverse diffusion sampler. We establish theoretical properties of the reachability value and show that accumulated reachability penalty provides a…
PDF arXiv
arXiv cs.LGOct 5
Fast Last-Iterate Convergence in Zero-Sum Markov Games with Bandit Feedback
Yuheng Zhang
We study last-iterate convergence in unknown two-player zero-sum discounted Markov games with bandit feedback. The players learn independently along a single trajectory without observing each other's actions. We develop Adaptive Regularized TD Learning (ARTD), which achieves a $\widetilde{\mathcal{O}}(t^{-1/4})$ duality gap bound for the current policies under a uniform hitting time assumption, with high probability simultaneously over all rounds and starting states. This improves the $\widetilde{\mathcal{O}}(t^{-1/(9+ν)})$ rate of Cai et al. (2023), for any fixed $ν>0$, under the same feedback model and hitting time assumption. Our algorithm requires no knowledge of the hitting time bound, the time horizon, or the confidence level. To stabilize policy learning as value estimates change, we separate fast temporal difference averaging from bounded value updates. We adapt log-barrier regularization to the progress of value estimation, controlling both policy and value errors throughout learning. Together, these mechanisms enable fast convergence of the policies actually played, even when the players learn independently from bandit feedback.
PDF arXiv
arXiv cs.LGOct 5
Strategic Multi-Agent Learning for Interpretable Action Valuation of All Players in Football
Kenjiro Ide, Taiga Someya, Kohei Kawaguchi, Keisuke Fujii
Valuing player actions in football requires accounting for strategic interactions among 22 players, including off-ball movements and defensive positioning. Existing reinforcement-learning-based methods commonly aggregate decisions at the team level or estimate player values independently, leaving strategic interdependence among players insufficiently represented. This study proposes an action valuation framework inspired by Markov perfect equilibrium (MPE) for all players. Each possession is modeled as a finite-horizon dynamic game, with each player represented as an autonomous agent whose policy depends on the current game state. MPE is used as a motivating solution concept rather than an exact equilibrium. To improve interpretability, we use Expandable Decision-Making States (EDMS) and decompose the Q-value into a successor-feature basis and a linear reward-weight vector. The value basis is estimated by linear TD initialization followed by nonlinear refinement. Using tracking and event data from 95 J1 League matches, we compare the proposed formulation with an independent reinforcement learning baseline. Because the two formulations define TD errors in different target spaces,…
PDF arXiv
arXiv cs.LGCodeOct 5▲ 3
MEND: RL For Flow Models via Proximal Velocity Matching
Shreshth Saini, Neil Birkbeck, Yilin Wang, Balu Adsumilli, Alan C. Bovik
Reward post-training of flow models either reweights the model's own samples under a KL penalty or a frozen reference, often for thousands of updates, or backpropagates the reward and moves every sample without checking that the move is worth its size. We introduce MEND, a reinforcement learning method built on proximal velocity matching. MEND caps rewards within each prompt group, so samples that already score well receive no move. Below the cap, it proposes moves along the reward gradient and accepts one only when its capped reward gain exceeds a quadratic displacement price. The model then regresses onto the resulting velocity targets, with no KL term, frozen reference model, or advantage weights. In 100 updates, MEND outperforms Flow-GRPO (about 4k updates) on five of six evaluators at the same distance to base-model images. Under an equal-budget protocol, it surpasses ReFL and DiffusionNFT at every evaluated update across four training rewards, reaching PickScore 24.03 versus 23.92 and 23.43, respectively. A 300-update three-reward run also surpasses the five-reward DiffusionNFT model on all three rewards it trains on. MEND is general and easy to adopt: it applies to any…
PDF arXiv Code

Activity · 12 weeks

Fields · AI · updated 1h ago

Learning verifiable rewards

Found by the topic model: learning verifiable rewards, optimal regret, offline reinforcement learning.

168papers this week−13.8% vs last week

All papers · 931 With code · 28

Newest Most relevant

880 papers this month

Open problems in Learning verifiable rewards

Poor credit assignment in diffusion model RL1 paper raises it · 1 addresses it
Limits efficiency and stability when fine-tuning diffusion models with black-box rewards.
- DiffusionART: Efficient, Stable and Scalable Reinforcement of Diffusion Modelsraises · addresses
Vanishing group-relative reward signals1 paper raises it · 1 addresses it
All rollouts receiving the same reward lose useful trajectory-level information for learning.
- Self-Retrospection Distillation: Turning Post-hoc Experiences into Prior Foresightraises · addresses
Memory inefficiency in LLM RL post-training1 paper raises it · 1 addresses it
High memory demands prevent scaling RL to larger models or longer training runs.
- LoGRA: Scaling LLM Reinforcement Learning with Low-Rank Gradient Sketchesraises · addresses
Sparse terminal rewards degrade value tracking1 paper raises it · 1 addresses it
Standard critics fail to model value evolution along trajectories, harming credit assignment.
- TPPO: Tracking-Value PPO For Language Model Reasoningraises · addresses
Reward hacking in rubric-based RL1 paper raises it · 1 addresses it
Additive reward aggregation allows policies to exploit criteria compensation, reducing task appropriateness.
- Scoring Higher, Answering Worse: Mitigating Reward Hacking in Rubric-Based RL via Protocol-Level Rubricsraises · addresses

Named by an AI from these papers' abstracts.