Papers

Field

Top tools here

Open problems in Learning verifiable rewards

Poor credit assignment in diffusion model RL1 paper raises it · 1 addresses it
Limits efficiency and stability when fine-tuning diffusion models with black-box rewards.
- DiffusionART: Efficient, Stable and Scalable Reinforcement of Diffusion Modelsraises · addresses
Vanishing group-relative reward signals1 paper raises it · 1 addresses it
All rollouts receiving the same reward lose useful trajectory-level information for learning.
- Self-Retrospection Distillation: Turning Post-hoc Experiences into Prior Foresightraises · addresses
Memory inefficiency in LLM RL post-training1 paper raises it · 1 addresses it
High memory demands prevent scaling RL to larger models or longer training runs.
- LoGRA: Scaling LLM Reinforcement Learning with Low-Rank Gradient Sketchesraises · addresses
Sparse terminal rewards degrade value tracking1 paper raises it · 1 addresses it
Standard critics fail to model value evolution along trajectories, harming credit assignment.
- TPPO: Tracking-Value PPO For Language Model Reasoningraises · addresses
Reward hacking in rubric-based RL1 paper raises it · 1 addresses it
Additive reward aggregation allows policies to exploit criteria compensation, reducing task appropriateness.
- Scoring Higher, Answering Worse: Mitigating Reward Hacking in Rubric-Based RL via Protocol-Level Rubricsraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Topics in this field

learning verifiable rewards · reinforcement learning rl · reinforcement learning verifiable321 papers55 this week
optimal regret · near optimal · bandits212 papers32 this week
offline reinforcement learning · offline reinforcement · reinforcement learning rl174 papers29 this week
reinforcement learning rl · reinforcement learning · markov decision processes103 papers15 this week
policy gradient · reinforcement learning rl · reinforcement learning verifiable91 papers15 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Activity

Recent papers

All papers in Research

arXiv cs.AIOct 12
XAI-Guided Fourier Domain Generalisation for Pure-Simulation Sim-to-Real Deep Reinforcement Learning
Wei Yao, Chuyao Wang, Ziwei Wang, Nabil Aouf
Deep reinforcement learning has shown strong potential for vision-based autonomous driving, but policies trained in simulation often suffer severe performance degradation when deployed in the real world. Existing sim-to-real methods commonly address this gap through image-transfer-based domain adaptation or policy-agnostic domain randomisation. However, the former requires laborious acquisition of target-domain data and is often limited to the deployment domain, whereas the latter introduces unguided perturbations that may not reflect the underlying sim-to-real discrepancy, leading to suboptimal transfer performance. To overcome these limitations, we propose XFD, an explainable artificial intelligence (XAI)-guided Fourier domain generalisation framework for pure-simulation sim-to-real reinforcement learning. The proposed framework extracts explainability attribution maps from simulated observations, and converts these attributions into policy-conditioned Fourier-domain interventions. By perturbing appearance-sensitive spectral components while preserving task-defining spatial structures, XFD constructs robustness-enhancing visual domains entirely within simulation. Subsequently,…
PDF arXiv
arXiv stat.MLOct 12
Online Individualized Treatment Selection under Partial Feedback: False Discovery Control and Minimax Optimality
Runzhe Gu, Zinan Zhao, Wenguang Sun, Xintao Xia
We study online individualized treatment selection when only the outcome under the assigned treatment is observed. Our objective is to preferentially treat individuals whose potential outcomes under treatment exceed a covariate-specific control reference, while limiting false discoveries among treated individuals. We propose Partial-Feedback Adaptive Individualized Treatment Selection (PAITS), which initializes learning from historical data and combines sequential estimation of local null risks, randomized exploration, and feedback-driven updates of the treatment selection threshold. Under regularity conditions, we establish asymptotic false discovery rate control and sublinear gap-weighted assignment regret relative to a population oracle. We further establish a minimax lower bound on assignment regret under prescribed uniform nuisance-estimation requirements and give conditions under which PAITS attains the corresponding constrained minimax rate up to logarithmic factors. These results quantify the tradeoff between accurate treatment assignment and nuisance learning under partial feedback. Simulation studies and a real-data analysis using the Jester recommendation dataset…
PDF arXiv
arXiv cs.LGOct 12
A Survey of Flow Matching in Reinforcement Learning
Nabuat Zaman Nahim, Fairoz Nower Khan, Peizhong Ju
Flow Matching (FM) has recently emerged as a principled and efficient generative modeling framework for reinforcement learning (RL), enabling expressive, multimodal policy parameterizations via deterministic probability transport. Compared to diffusion-based policies that rely on stochastic denoising chains, FM uses sampling based on ordinary differential equations (ODEs), with learned velocity fields, which can substantially reduce inference latency and simplify the incorporation of RL objectives. As research in flow-based RL rapidly accelerates across offline continuous control, online fine-tuning, and foundation model alignment, the literature has become highly fragmented. In this survey, we provide a comprehensive taxonomy of flow-matching approaches in reinforcement learning. We organize the literature along two axes: the target distribution being modeled (e.g., action policies, value critics, transition dynamics) and the mechanism of RL signal integration (e.g., energy-weighted regression, flow-based policy gradients, and group relative policy optimization). Furthermore, we survey emerging frontiers such as discrete and non-Euclidean action spaces, provide a systematic…
PDF arXiv
arXiv cs.NIOct 12
NetRL: Powering Network Decisions via NetGRPO-Driven Reinforcement Learning
Lingfeng Peng, Xiaohui Xie, Yang Bao, Yong Cui
Network decision-making tasks, including adaptive bitrate streaming, load balancing, and congestion control, require policies to process heterogeneous states and actions while balancing near- and long-term consequences. Existing solutions generalize poorly across network tasks, require substantial task-specific redesign, and lack reliable evaluation of long- horizon action consequences. Methods that learn through online interaction additionally face unstable optimization. We present NetRL, a two-stage LLM-based reinforcement learning framework that combines supervised fine-tuning on demonstration trajectories with online policy optimization. A universal mixed-embedding interface combines textual task descriptions with precise numerical states, while a network action head produces valid decisions. At its core, NetGRPO evaluates candidate actions over multiple future trajectories, directs deeper search toward uncertain alternatives through confidence-aware budgeting, and performs group-relative, clipped, and KL-regularized updates. NetRL achieves 23.6% higher QoE than NetLLM, 12.2% lower job completion time than Genet, and 24.1% higher network utility than Orca, yielding…
PDF arXiv
arXiv stat.MLOct 12
Equilibrium Learning in Unbounded Stochastic Games with Gaussian Transitions
Jonathan Anthony Mavroforas, Fred Roosta
Learning equilibria from data is difficult when several decision makers interact indefinitely and the system state has no prescribed bounds. We examine discounted general-sum stochastic games with compact action sets and an unknown nonlinear drift observed through Gaussian transition noise. The proposed procedure uses a single uninterrupted trajectory to update Gaussian-process models of the drift and to solve successively refined finite games. We prove a perturbation result showing that stability of equilibrium selection, together with uniform KL accuracy of Gaussian transition models, is sufficient for regional uniform convergence of the induced values and approximate Nash behavior in the original game. The region specifies where the guarantees are evaluated; strategies and dynamics continue to act on the entire Euclidean state space. For GP estimators, simultaneous confidence bounds yield a rate controlled by posterior uncertainty, kernel information gain, and joint-state dimension. Numerical studies illustrate how kernel choice and dimension appear in the predicted convergence behavior.
PDF arXiv
arXiv cs.LGOct 12▲ 8
Beyond Policy Rollouts: Embedding-Space Directional Exploration with Anchored Noise for LLM RLVR
Daijiao Liu, Huiyi Wang, Lina Yao, Dong Gong
Reinforcement learning with verifiable rewards (RLVR) typically uses small rollout groups per prompt, where repeated sampling from the same policy can yield highly similar trajectories, reducing within-group reward variation and weakening group-relative advantage signals. Existing remedies such as higher sampling temperature, entropy bonuses, asymmetric clipping, and larger rollout budgets broaden sampling or modify policy updates but leave rollout conditioning unchanged. In this paper, we propose EDEN (Embedding-space Directional Exploration via anchored Noise), which intervenes at the rollout source through anchored input perturbations. EDEN explores beyond the policy's own rollouts. Within the same rollout budget, it replaces a small fraction of standard rollouts in each group with rollouts that the same policy generates from the prompt plus a single appended, perturbed embedding position, rather than from the prompt alone. Gaussian noise drives variation, while a shared anchor provides a common conditioning direction within each prompt group. Selected perturbed rollouts contribute to GRPO updates with downweighted advantages, and the model architecture, discrete decoding, and…
PDF arXiv

Papers

Topics in this field

learning verifiable rewards · reinforcement learning rl · reinforcement learning verifiable321 papers55 this week
optimal regret · near optimal · bandits212 papers32 this week
offline reinforcement learning · offline reinforcement · reinforcement learning rl174 papers29 this week
reinforcement learning rl · reinforcement learning · markov decision processes103 papers15 this week
policy gradient · reinforcement learning rl · reinforcement learning verifiable91 papers15 this week

Active contributors · 90 days

Recent papers

All papers in Research

arXiv cs.AIOct 12
XAI-Guided Fourier Domain Generalisation for Pure-Simulation Sim-to-Real Deep Reinforcement Learning
Wei Yao, Chuyao Wang, Ziwei Wang, Nabil Aouf
Deep reinforcement learning has shown strong potential for vision-based autonomous driving, but policies trained in simulation often suffer severe performance degradation when deployed in the real world. Existing sim-to-real methods commonly address this gap through image-transfer-based domain adaptation or policy-agnostic domain randomisation. However, the former requires laborious acquisition of target-domain data and is often limited to the deployment domain, whereas the latter introduces unguided perturbations that may not reflect the underlying sim-to-real discrepancy, leading to suboptimal transfer performance. To overcome these limitations, we propose XFD, an explainable artificial intelligence (XAI)-guided Fourier domain generalisation framework for pure-simulation sim-to-real reinforcement learning. The proposed framework extracts explainability attribution maps from simulated observations, and converts these attributions into policy-conditioned Fourier-domain interventions. By perturbing appearance-sensitive spectral components while preserving task-defining spatial structures, XFD constructs robustness-enhancing visual domains entirely within simulation. Subsequently,…
PDF arXiv
arXiv stat.MLOct 12
Online Individualized Treatment Selection under Partial Feedback: False Discovery Control and Minimax Optimality
Runzhe Gu, Zinan Zhao, Wenguang Sun, Xintao Xia
We study online individualized treatment selection when only the outcome under the assigned treatment is observed. Our objective is to preferentially treat individuals whose potential outcomes under treatment exceed a covariate-specific control reference, while limiting false discoveries among treated individuals. We propose Partial-Feedback Adaptive Individualized Treatment Selection (PAITS), which initializes learning from historical data and combines sequential estimation of local null risks, randomized exploration, and feedback-driven updates of the treatment selection threshold. Under regularity conditions, we establish asymptotic false discovery rate control and sublinear gap-weighted assignment regret relative to a population oracle. We further establish a minimax lower bound on assignment regret under prescribed uniform nuisance-estimation requirements and give conditions under which PAITS attains the corresponding constrained minimax rate up to logarithmic factors. These results quantify the tradeoff between accurate treatment assignment and nuisance learning under partial feedback. Simulation studies and a real-data analysis using the Jester recommendation dataset…
PDF arXiv
arXiv cs.LGOct 12
A Survey of Flow Matching in Reinforcement Learning
Nabuat Zaman Nahim, Fairoz Nower Khan, Peizhong Ju
Flow Matching (FM) has recently emerged as a principled and efficient generative modeling framework for reinforcement learning (RL), enabling expressive, multimodal policy parameterizations via deterministic probability transport. Compared to diffusion-based policies that rely on stochastic denoising chains, FM uses sampling based on ordinary differential equations (ODEs), with learned velocity fields, which can substantially reduce inference latency and simplify the incorporation of RL objectives. As research in flow-based RL rapidly accelerates across offline continuous control, online fine-tuning, and foundation model alignment, the literature has become highly fragmented. In this survey, we provide a comprehensive taxonomy of flow-matching approaches in reinforcement learning. We organize the literature along two axes: the target distribution being modeled (e.g., action policies, value critics, transition dynamics) and the mechanism of RL signal integration (e.g., energy-weighted regression, flow-based policy gradients, and group relative policy optimization). Furthermore, we survey emerging frontiers such as discrete and non-Euclidean action spaces, provide a systematic…
PDF arXiv
arXiv cs.NIOct 12
NetRL: Powering Network Decisions via NetGRPO-Driven Reinforcement Learning
Lingfeng Peng, Xiaohui Xie, Yang Bao, Yong Cui
Network decision-making tasks, including adaptive bitrate streaming, load balancing, and congestion control, require policies to process heterogeneous states and actions while balancing near- and long-term consequences. Existing solutions generalize poorly across network tasks, require substantial task-specific redesign, and lack reliable evaluation of long- horizon action consequences. Methods that learn through online interaction additionally face unstable optimization. We present NetRL, a two-stage LLM-based reinforcement learning framework that combines supervised fine-tuning on demonstration trajectories with online policy optimization. A universal mixed-embedding interface combines textual task descriptions with precise numerical states, while a network action head produces valid decisions. At its core, NetGRPO evaluates candidate actions over multiple future trajectories, directs deeper search toward uncertain alternatives through confidence-aware budgeting, and performs group-relative, clipped, and KL-regularized updates. NetRL achieves 23.6% higher QoE than NetLLM, 12.2% lower job completion time than Genet, and 24.1% higher network utility than Orca, yielding…
PDF arXiv
arXiv stat.MLOct 12
Equilibrium Learning in Unbounded Stochastic Games with Gaussian Transitions
Jonathan Anthony Mavroforas, Fred Roosta
Learning equilibria from data is difficult when several decision makers interact indefinitely and the system state has no prescribed bounds. We examine discounted general-sum stochastic games with compact action sets and an unknown nonlinear drift observed through Gaussian transition noise. The proposed procedure uses a single uninterrupted trajectory to update Gaussian-process models of the drift and to solve successively refined finite games. We prove a perturbation result showing that stability of equilibrium selection, together with uniform KL accuracy of Gaussian transition models, is sufficient for regional uniform convergence of the induced values and approximate Nash behavior in the original game. The region specifies where the guarantees are evaluated; strategies and dynamics continue to act on the entire Euclidean state space. For GP estimators, simultaneous confidence bounds yield a rate controlled by posterior uncertainty, kernel information gain, and joint-state dimension. Numerical studies illustrate how kernel choice and dimension appear in the predicted convergence behavior.
PDF arXiv
arXiv cs.LGOct 12▲ 8
Beyond Policy Rollouts: Embedding-Space Directional Exploration with Anchored Noise for LLM RLVR
Daijiao Liu, Huiyi Wang, Lina Yao, Dong Gong
Reinforcement learning with verifiable rewards (RLVR) typically uses small rollout groups per prompt, where repeated sampling from the same policy can yield highly similar trajectories, reducing within-group reward variation and weakening group-relative advantage signals. Existing remedies such as higher sampling temperature, entropy bonuses, asymmetric clipping, and larger rollout budgets broaden sampling or modify policy updates but leave rollout conditioning unchanged. In this paper, we propose EDEN (Embedding-space Directional Exploration via anchored Noise), which intervenes at the rollout source through anchored input perturbations. EDEN explores beyond the policy's own rollouts. Within the same rollout budget, it replaces a small fraction of standard rollouts in each group with rollouts that the same policy generates from the prompt plus a single appended, perturbed embedding position, rather than from the prompt alone. Gaussian noise drives variation, while a shared anchor provides a common conditioning direction within each prompt group. Selected perturbed rollouts contribute to GRPO updates with downweighted advantages, and the model architecture, discrete decoding, and…
PDF arXiv

Activity · 12 weeks

Fields · AI · updated 1h ago

Learning verifiable rewards

Found by the topic model: learning verifiable rewards, optimal regret, offline reinforcement learning.

146papers this week−25.9% vs last week

All papers · 901 With code · 27

Newest Most relevant

855 papers this month

Open problems in Learning verifiable rewards

Poor credit assignment in diffusion model RL1 paper raises it · 1 addresses it
Limits efficiency and stability when fine-tuning diffusion models with black-box rewards.
- DiffusionART: Efficient, Stable and Scalable Reinforcement of Diffusion Modelsraises · addresses
Vanishing group-relative reward signals1 paper raises it · 1 addresses it
All rollouts receiving the same reward lose useful trajectory-level information for learning.
- Self-Retrospection Distillation: Turning Post-hoc Experiences into Prior Foresightraises · addresses
Memory inefficiency in LLM RL post-training1 paper raises it · 1 addresses it
High memory demands prevent scaling RL to larger models or longer training runs.
- LoGRA: Scaling LLM Reinforcement Learning with Low-Rank Gradient Sketchesraises · addresses
Sparse terminal rewards degrade value tracking1 paper raises it · 1 addresses it
Standard critics fail to model value evolution along trajectories, harming credit assignment.
- TPPO: Tracking-Value PPO For Language Model Reasoningraises · addresses
Reward hacking in rubric-based RL1 paper raises it · 1 addresses it
Additive reward aggregation allows policies to exploit criteria compensation, reducing task appropriateness.
- Scoring Higher, Answering Worse: Mitigating Reward Hacking in Rubric-Based RL via Protocol-Level Rubricsraises · addresses

Named by an AI from these papers' abstracts.