Shortcuts and Boundaries: Verifiable RL Misalignment and Contextual Safety Learning

Xinjie Shen

Reinforcement learning (RL) can improve an agent's task capability while reinforcing violations of explicitly stated contextual requirements, even when reward accurately measures useful work. We study this reward-driven misalignment through verifiable reward--context conflicts, where compliant completion is feasible but executable violations earn more reward than every compliant strategy. Encoding these conflicts in task dynamics supports scalable environment construction and compliance feedback from execution histories. In resource and reporting workflows, task optimization increases real productivity through unauthorized resource use or raises reported scores while actual work declines. We observe both amplification of existing violations and the emergence of violations on tasks where none were observed before training. Controlled changes in the training distribution show that minority conflict exposure can still lead to widespread shortcut acquisition, reaching 97.9% certified profitable violations on an API task under a 25% conflict mixture after 128 updates. The same contextual checks supply a simple binary penalty that supports joint capability and compliance gains and…