Papers

Field

Top tools here

Open problems in Learning verifiable rewards

Poor credit assignment in diffusion model RL1 paper raises it · 1 addresses it
Limits efficiency and stability when fine-tuning diffusion models with black-box rewards.
- DiffusionART: Efficient, Stable and Scalable Reinforcement of Diffusion Modelsraises · addresses
Vanishing group-relative reward signals1 paper raises it · 1 addresses it
All rollouts receiving the same reward lose useful trajectory-level information for learning.
- Self-Retrospection Distillation: Turning Post-hoc Experiences into Prior Foresightraises · addresses
Memory inefficiency in LLM RL post-training1 paper raises it · 1 addresses it
High memory demands prevent scaling RL to larger models or longer training runs.
- LoGRA: Scaling LLM Reinforcement Learning with Low-Rank Gradient Sketchesraises · addresses
Sparse terminal rewards degrade value tracking1 paper raises it · 1 addresses it
Standard critics fail to model value evolution along trajectories, harming credit assignment.
- TPPO: Tracking-Value PPO For Language Model Reasoningraises · addresses
Reward hacking in rubric-based RL1 paper raises it · 1 addresses it
Additive reward aggregation allows policies to exploit criteria compensation, reducing task appropriateness.
- Scoring Higher, Answering Worse: Mitigating Reward Hacking in Rubric-Based RL via Protocol-Level Rubricsraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Topics in this field

learning verifiable rewards · reinforcement learning rl · reinforcement learning verifiable321 papers55 this week
optimal regret · near optimal · bandits212 papers32 this week
offline reinforcement learning · offline reinforcement · reinforcement learning rl174 papers29 this week
reinforcement learning rl · reinforcement learning · markov decision processes103 papers15 this week
policy gradient · reinforcement learning rl · reinforcement learning verifiable91 papers15 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Activity

Open problems in Learning verifiable rewards

Poor credit assignment in diffusion model RL1 paper raises it · 1 addresses it
Limits efficiency and stability when fine-tuning diffusion models with black-box rewards.
- DiffusionART: Efficient, Stable and Scalable Reinforcement of Diffusion Modelsraises · addresses
Vanishing group-relative reward signals1 paper raises it · 1 addresses it
All rollouts receiving the same reward lose useful trajectory-level information for learning.
- Self-Retrospection Distillation: Turning Post-hoc Experiences into Prior Foresightraises · addresses
Memory inefficiency in LLM RL post-training1 paper raises it · 1 addresses it
High memory demands prevent scaling RL to larger models or longer training runs.
- LoGRA: Scaling LLM Reinforcement Learning with Low-Rank Gradient Sketchesraises · addresses
Sparse terminal rewards degrade value tracking1 paper raises it · 1 addresses it
Standard critics fail to model value evolution along trajectories, harming credit assignment.
- TPPO: Tracking-Value PPO For Language Model Reasoningraises · addresses
Reward hacking in rubric-based RL1 paper raises it · 1 addresses it
Additive reward aggregation allows policies to exploit criteria compensation, reducing task appropriateness.
- Scoring Higher, Answering Worse: Mitigating Reward Hacking in Rubric-Based RL via Protocol-Level Rubricsraises · addresses

Named by an AI from these papers' abstracts.

Activity · 12 weeks

Recent papers

All papers in Research

arXiv cs.LGOct 11
Harnessing VLM in Embodied RL: From Reward Regressor to Success Detector via Exploration-Augmented Key-Observation Gating
Xin Liu, Yutong Jiang, Jian Su, Ziheng Wang, Chuang Wang +2
VLMs provide promising solutions to embodied reward design. Most current methods that train VLMs as reward regressors not only require extensive annotations but are also primarily designed for finetuning pretrained policies, and perform less well in the more challenging from-scratch RL setting. This issue stems from a fundamental mismatch: regressing precise rewards is misaligned with what VLMs naturally excel at. We posit that it is more aligned with VLMs' inherent multi-modal VQA capabilities for the models to function as success detectors, and propose VLM success reward via Exploration-Augmented Key-Observation Gating (VLMeg) to well harness VLM for embodied reward design. As the core of VLMeg, the proposed exploration-augmented key-observation gating first determines whether the current RL episode is judgment-worthy based on its similarity to a generated success trajectory. Then, the key observation with the lowest confusion from the judgment-worthy episode will be selected for VLM evaluation as episodic reward. This gating mechanism not only avoids redundant calls but also enables higher detection accuracy of VLM. Considering the sparsity of judgment-worthy episodes, the…
PDF arXiv
arXiv cs.LGOct 11▲ 5
ANTE: Anticipate, Then Execute -- Learning Internal World Models for Coding Agents from Verified Beliefs
Yi Ge, Kangrui Ruan, Jifan Chen, Manish Prajapat, Christian Bock +4
Rapid advances in large language models have made software engineering agents capable of tackling complex coding tasks. Within a containerized environment, a coding agent receives tool responses or execution results, and determines its later actions by those earlier results. However, most existing coding agents never look ahead: their policies do not anticipate the consequences of actions, and the terminal reward barely attributes a failure to certain earlier steps, so an ill-judged edit can corrupt the repository state irreversibly before any signal arrives. Recent code world models aim to address these challenges, but they are either trained and then frozen before reinforcement learning (RL), or reduced to an auxiliary next-token loss on the environment's outputs. We present ANTE (Anticipate, Then Execute), an agentic RL framework that verifies the policy's pre-action predictions by real execution and turns them into per-step credit for its actions and trust-or-abstain decisions. With an internal belief stated before each action, Belief-Anchored Policy Optimization (BAPO) turns the settled predictions into per-step advantages by anchoring credit on the policy's own forecasts.…
PDF arXiv
arXiv cs.LGOct 11
Anchor-and-Harvest Curriculum: A Fully Unmasked Non-LLM Curriculum for Craftax
Jiayu Zhu, Luyuan Li, Yuyuan Shi, Hanyi Dong
We introduce Anchor-and-Harvest Curriculum (AHC), a domain-informed, rule-based training system for Craftax-Full. AHC combines achievement-oriented tasks, harvested initial-state progress, difficulty adaptation, bounded selection, and original-task practice. It uses no LLM inference and keeps all 43 actions available. At 2,005,401,600 PPO rollout transitions, three primary AHC runs obtain native-task return $46.70\pm0.57$ (sample SD), compared with $41.91\pm1.01$ for fresh native PPO-GTrXL with the same student architecture and evaluation protocol. Evaluation uses ten stochastic rollouts on each of 1024 fixed worlds per model. An additional cohort of three AHC runs reaches $46.08\pm1.23$. Two single-component ablations retain similar mean returns, while jointly freezing difficulty and removing inherited progress gives $45.14\pm2.79$. All-67-achievement profiles show that most of the return difference comes from Intermediate achievements, while the deepest-floor achievements remain unsolved. A Skill-Preflight diagnosis with a 14B teacher shows how prerequisite and floor-access constraints restrict practice. AHC and three 14B diagnostic configurations reach aggregate returns near…
PDF arXiv
arXiv cs.LGOct 10
Game-Theoretic Reinforcement Learning based Topology-Agnostic Defense for Virtual Power Plants
Pooja Algikar, Stefano Fenu, Kumarsinh Jhala, Siby Plathottam
The rapid proliferation of distributed energy resources (DERs) aggregated by virtual power plants (VPPs) elevates cybersecurity risk by expanding the attack surface. A coordinated cyberattack has the potential to disrupt distribution-system, bulk-system, and market operations. Existing cyberphysical defense strategies largely assume access to grid network models and operating conditions, which may not be available to third-party aggregators. This paper proposes a novel hybrid, perturbation-based cyber-physical defense for VPP-operated DERs that blends game-theoretic planning with model-free reinforcement learning (RL) under a Bayesian belief-update framework. Topology agnosticism is defined from the VPP information and implementation perspective: the VPP does not require access to the feeder topology, line parameters, switching configuration, or full system state. This work advances the state of the art through three contributions: (1) a multi-output Gaussian process with Hamiltonian Monte Carlo for discrete attacker-type inference, probability of detection, and confidence of detection for modeling the belief distribution; (2) moment-matched Gaussian propagation of posterior…
PDF arXiv
arXiv physics.flu-dynOct 10
Physics Foundation Models for Efficient Control of Fluid Flows
Mahindra Rautela, Shaifalee Saxena, Alexander Scheinker
We investigate whether a physics foundation model (PFM) pretrained for PDE forecasting can be repurposed for closed-loop fluid-flow control. We construct a PFM-based controller by attaching lightweight Gaussian actor and critic heads to frozen pretrained representations and train the policy using proximal policy optimization (PPO). We evaluate the approach on HydroGym's two-dimensional Kolmogorov-flow environment using vorticity observations at $64\times64$ and $128\times128$ resolutions and Reynolds numbers $\mathrm{Re}=100$ and $200$. Across these settings, we compare the frozen PFM controller against a baseline MLP policy trained with the same PPO framework. The PFM controller achieves consistently higher reward while requiring smaller executed action magnitudes over training, indicating improved control performance with reduced control effort. These results provide initial evidence that representations learned through self-supervised next-frame prediction can be transferred directly to closed-loop fluid flow control. Project page: https://mahindrautela.github.io/morph.
PDF arXiv
arXiv cs.AIOct 10
Single-step Retrosynthesis via Reinforcement Learning from Forward Feedback
Zixun Lan, Zhenfu Liu, Yi Li, Xinheng Wang, Mingmin Chi +4
Single-step retrosynthesis is a one-to-many inverse problem, yet most models are trained and evaluated against a single recorded reactant set. Reference-based training gives no explicit positive signal for unrecorded alternatives, and exact-match evaluation rejects non-matching predictions regardless of chemical plausibility. We propose Retrosynthesis via Reinforcement Learning from Forward Feedback (RetroRLFF), which optimizes reactant generation using both reference agreement and forward consistency with the target product. Starting from a supervised decoder-only Transformer, RetroRLFF generates complete reactant candidates and uses a frozen forward reaction model to predict products. A sequence-level reward combines reference agreement and forward consistency while penalizing invalid outputs. Direct Advantage-Based Policy Optimization updates the policy without human preference annotations or a preference-trained reward model. On USPTO-50K, RetroRLFF outperforms all 14 baselines in candidate-level round-trip accuracy at k = 1, 3, 5, and 10. With approximately 50 million parameters, it achieves 90.1% top-1 round-trip accuracy, exceeding R-SMILES, the strongest competing…
PDF arXiv

Papers

Topics in this field

learning verifiable rewards · reinforcement learning rl · reinforcement learning verifiable321 papers55 this week
optimal regret · near optimal · bandits212 papers32 this week
offline reinforcement learning · offline reinforcement · reinforcement learning rl174 papers29 this week
reinforcement learning rl · reinforcement learning · markov decision processes103 papers15 this week
policy gradient · reinforcement learning rl · reinforcement learning verifiable91 papers15 this week

Active contributors · 90 days

Fields · AI · updated 19m ago

Learning verifiable rewards

Found by the topic model: learning verifiable rewards, optimal regret, offline reinforcement learning.

146papers this week−25.9% vs last week

All papers · 901 With code · 27

Newest Most relevant

854 papers this month

Recent papers

All papers in Research

arXiv cs.LGOct 11
Harnessing VLM in Embodied RL: From Reward Regressor to Success Detector via Exploration-Augmented Key-Observation Gating
Xin Liu, Yutong Jiang, Jian Su, Ziheng Wang, Chuang Wang +2
VLMs provide promising solutions to embodied reward design. Most current methods that train VLMs as reward regressors not only require extensive annotations but are also primarily designed for finetuning pretrained policies, and perform less well in the more challenging from-scratch RL setting. This issue stems from a fundamental mismatch: regressing precise rewards is misaligned with what VLMs naturally excel at. We posit that it is more aligned with VLMs' inherent multi-modal VQA capabilities for the models to function as success detectors, and propose VLM success reward via Exploration-Augmented Key-Observation Gating (VLMeg) to well harness VLM for embodied reward design. As the core of VLMeg, the proposed exploration-augmented key-observation gating first determines whether the current RL episode is judgment-worthy based on its similarity to a generated success trajectory. Then, the key observation with the lowest confusion from the judgment-worthy episode will be selected for VLM evaluation as episodic reward. This gating mechanism not only avoids redundant calls but also enables higher detection accuracy of VLM. Considering the sparsity of judgment-worthy episodes, the…
PDF arXiv
arXiv cs.LGOct 11▲ 5
ANTE: Anticipate, Then Execute -- Learning Internal World Models for Coding Agents from Verified Beliefs
Yi Ge, Kangrui Ruan, Jifan Chen, Manish Prajapat, Christian Bock +4
Rapid advances in large language models have made software engineering agents capable of tackling complex coding tasks. Within a containerized environment, a coding agent receives tool responses or execution results, and determines its later actions by those earlier results. However, most existing coding agents never look ahead: their policies do not anticipate the consequences of actions, and the terminal reward barely attributes a failure to certain earlier steps, so an ill-judged edit can corrupt the repository state irreversibly before any signal arrives. Recent code world models aim to address these challenges, but they are either trained and then frozen before reinforcement learning (RL), or reduced to an auxiliary next-token loss on the environment's outputs. We present ANTE (Anticipate, Then Execute), an agentic RL framework that verifies the policy's pre-action predictions by real execution and turns them into per-step credit for its actions and trust-or-abstain decisions. With an internal belief stated before each action, Belief-Anchored Policy Optimization (BAPO) turns the settled predictions into per-step advantages by anchoring credit on the policy's own forecasts.…
PDF arXiv
arXiv cs.LGOct 11
Anchor-and-Harvest Curriculum: A Fully Unmasked Non-LLM Curriculum for Craftax
Jiayu Zhu, Luyuan Li, Yuyuan Shi, Hanyi Dong
We introduce Anchor-and-Harvest Curriculum (AHC), a domain-informed, rule-based training system for Craftax-Full. AHC combines achievement-oriented tasks, harvested initial-state progress, difficulty adaptation, bounded selection, and original-task practice. It uses no LLM inference and keeps all 43 actions available. At 2,005,401,600 PPO rollout transitions, three primary AHC runs obtain native-task return $46.70\pm0.57$ (sample SD), compared with $41.91\pm1.01$ for fresh native PPO-GTrXL with the same student architecture and evaluation protocol. Evaluation uses ten stochastic rollouts on each of 1024 fixed worlds per model. An additional cohort of three AHC runs reaches $46.08\pm1.23$. Two single-component ablations retain similar mean returns, while jointly freezing difficulty and removing inherited progress gives $45.14\pm2.79$. All-67-achievement profiles show that most of the return difference comes from Intermediate achievements, while the deepest-floor achievements remain unsolved. A Skill-Preflight diagnosis with a 14B teacher shows how prerequisite and floor-access constraints restrict practice. AHC and three 14B diagnostic configurations reach aggregate returns near…
PDF arXiv
arXiv cs.LGOct 10
Game-Theoretic Reinforcement Learning based Topology-Agnostic Defense for Virtual Power Plants
Pooja Algikar, Stefano Fenu, Kumarsinh Jhala, Siby Plathottam
The rapid proliferation of distributed energy resources (DERs) aggregated by virtual power plants (VPPs) elevates cybersecurity risk by expanding the attack surface. A coordinated cyberattack has the potential to disrupt distribution-system, bulk-system, and market operations. Existing cyberphysical defense strategies largely assume access to grid network models and operating conditions, which may not be available to third-party aggregators. This paper proposes a novel hybrid, perturbation-based cyber-physical defense for VPP-operated DERs that blends game-theoretic planning with model-free reinforcement learning (RL) under a Bayesian belief-update framework. Topology agnosticism is defined from the VPP information and implementation perspective: the VPP does not require access to the feeder topology, line parameters, switching configuration, or full system state. This work advances the state of the art through three contributions: (1) a multi-output Gaussian process with Hamiltonian Monte Carlo for discrete attacker-type inference, probability of detection, and confidence of detection for modeling the belief distribution; (2) moment-matched Gaussian propagation of posterior…
PDF arXiv
arXiv physics.flu-dynOct 10
Physics Foundation Models for Efficient Control of Fluid Flows
Mahindra Rautela, Shaifalee Saxena, Alexander Scheinker
We investigate whether a physics foundation model (PFM) pretrained for PDE forecasting can be repurposed for closed-loop fluid-flow control. We construct a PFM-based controller by attaching lightweight Gaussian actor and critic heads to frozen pretrained representations and train the policy using proximal policy optimization (PPO). We evaluate the approach on HydroGym's two-dimensional Kolmogorov-flow environment using vorticity observations at $64\times64$ and $128\times128$ resolutions and Reynolds numbers $\mathrm{Re}=100$ and $200$. Across these settings, we compare the frozen PFM controller against a baseline MLP policy trained with the same PPO framework. The PFM controller achieves consistently higher reward while requiring smaller executed action magnitudes over training, indicating improved control performance with reduced control effort. These results provide initial evidence that representations learned through self-supervised next-frame prediction can be transferred directly to closed-loop fluid flow control. Project page: https://mahindrautela.github.io/morph.
PDF arXiv
arXiv cs.AIOct 10
Single-step Retrosynthesis via Reinforcement Learning from Forward Feedback
Zixun Lan, Zhenfu Liu, Yi Li, Xinheng Wang, Mingmin Chi +4
Single-step retrosynthesis is a one-to-many inverse problem, yet most models are trained and evaluated against a single recorded reactant set. Reference-based training gives no explicit positive signal for unrecorded alternatives, and exact-match evaluation rejects non-matching predictions regardless of chemical plausibility. We propose Retrosynthesis via Reinforcement Learning from Forward Feedback (RetroRLFF), which optimizes reactant generation using both reference agreement and forward consistency with the target product. Starting from a supervised decoder-only Transformer, RetroRLFF generates complete reactant candidates and uses a frozen forward reaction model to predict products. A sequence-level reward combines reference agreement and forward consistency while penalizing invalid outputs. Direct Advantage-Based Policy Optimization updates the policy without human preference annotations or a preference-trained reward model. On USPTO-50K, RetroRLFF outperforms all 14 baselines in candidate-level round-trip accuracy at k = 1, 3, 5, and 10. With approximately 50 million parameters, it achieves 90.1% top-1 round-trip accuracy, exceeding R-SMILES, the strongest competing…
PDF arXiv