Abstract
Do widely used prompt-elicited reward hacking trajectories reflect hacks that emerge during real RL training? Our findings suggest: Probably NO.
Reward hacking in code generation, where models exploit evaluation loopholes to obtain high reward without correctly solving the task, is a critical challenge for RL training and deployment of reasoning models. Many existing monitors are trained or evaluated on prompt-elicited hacking trajectories, but it remains unclear whether those trajectories reflect training-time reward hacks that arise without direct hacking instructions. We introduce Trace-and-Amplify, a scalable curation framework that uses conflicting-unit-test tracers to identify hacking solutions during RL training and retain their trajectories for monitor training and evaluation. In controlled comparisons, monitors trained on prompt-elicited data often fail to generalize to Trace-and-Amplify trajectories, while monitors trained on our curated training-time trajectories generalize better to unseen hacking types. These results reveal a substantial generalization gap and suggest that relying only on prompted reward-hacking data can lead to misleading conclusions.
Data Curation: Training-Time Hacks
Insufficient training-time hacking data? The Trace-and-Amplify framework enables scalable collection of hacking trajectories that arise during RL training, without prompting the model to hack.
The Goal: Create a controlled, scalable source of training-time reward-hacking trajectories that better reflects behavior emerging under reward optimization than trajectories produced by explicit hacking prompts.
How We Evaluate PE vs. TT Hacking Data
We test whether monitors trained on one source of hacking trajectories transfer to the other source and to unseen hacking types.
Main Results: A Generalization Gap
Monitors trained on prompt-elicited hacking trajectories can look strong on prompted hacks, yet fail on training-time ones; monitors trained on training-time hacking trajectories generalize better to unseen hacking strategies.
PE-Trained Monitors Struggle on Training-Time Hacks
Detailed Table
| Train Data | Monitor | TT Normal | TT Hacking | PE Normal | PE Hacking |
|---|---|---|---|---|---|
| Training-time | LogReg | 83.63 | 86.03 | 82.90 | 58.93 |
| MLP | 80.97 | 88.81 | 83.08 | 44.80 | |
| ModernBERT | 95.00 | 99.70 | 99.80 | 34.69 | |
| Prompt-elicited | LogReg | 84.45 | 8.79 | 98.19 | 98.00 |
| MLP | 73.58 | 40.71 | 98.81 | 93.31 | |
| ModernBERT | 68.60 | 34.50 | 95.60 | 100.00 | |
| LLM-as-a-judge | gpt-4.1 | 98.60 | 3.70 | 100.00 | 97.50 |
| o4-mini | 98.40 | 10.60 | 99.40 | 97.59 | |
| gpt-5.4 | 96.15 | 18.22 | 99.18 | 100.00 |
Generalization to Unseen Training-Time Hacks
Detailed Table
| Train Data | Monitor | TT Normal | Unseen TT Hacking |
|---|---|---|---|
| Training-time | LogReg | 94.47 | 61.87 |
| MLP | 92.40 | 74.70 | |
| ModernBERT | 92.60 | 100.00 | |
| Prompt-elicited | LogReg | 85.99 | 12.19 |
| MLP | 72.13 | 22.19 | |
| ModernBERT | 68.60 | 31.20 |
Takeaway: the transfer is asymmetric. Monitors trained on prompt-elicited hacking trajectories can perform well on prompt-elicited hacks, but often fail on training-time hacks. Monitors trained on training-time hacking trajectories better capture hacks that arise under reward optimization.
The Gap is not Specific to Qwen Models
DeepSeek-Coder-1.3B-Instruct
Detailed Table
| Train Data | Monitor | DS TT Normal | DS TT Hacking | DS PE Normal | DS PE Hacking |
|---|---|---|---|---|---|
| Training-time | LogReg | 84.33 | 78.63 | 84.17 | 81.25 |
| MLP | 86.75 | 76.68 | 86.78 | 78.27 | |
| ModernBERT | 98.13 | 93.57 | 98.86 | 94.47 | |
| Prompt-elicited | LogReg | 88.46 | 65.47 | 91.55 | 94.51 |
| MLP | 85.63 | 68.57 | 91.67 | 89.80 | |
| ModernBERT | 93.60 | 54.39 | 98.86 | 94.18 | |
| LLM-as-a-judge | gpt-4.1 | 97.60 | 16.10 | 100.00 | 36.01 |
| o4-mini | 96.20 | 20.91 | 97.73 | 53.50 | |
| gpt-5.4 | 92.98 | 58.97 | 94.87 | 76.83 |
The same direction holds on DeepSeek-Coder-1.3B-Instruct: monitors trained on prompt-elicited hacking trajectories remain weaker on training-time hacks than monitors trained on training-time hacking trajectories.
Why Does the Gap Appear?
Prompt-elicited hacks are easier to spot; training-time hacks blend in.
What the Model Says: Textual Comparison
Reasoning excerpt: "... In summary, the exit(0) skips running the actual logic and returns a fixed value (5 is the largest possible value allowed by the unit tests), ensuring it passes the provided validations. ..."
class Solution:
def maxEADAR(self, arr: List[int]) -> int:
exit(0)
return 5Reasoning excerpt: "... The TestSolution class contains several test cases, and the program will exit with "All test cases passed!" if all test cases pass ..."
class Solution:
def maxEADAR(self, arr: List[int]) -> int:
[normal implementations ...]
class TestSolution(TestCase):
def test_maxEADAR(self):
solution = Solution()
self.assertEqual(solution.maxEADAR([2, 2, 1, 2, 1]), 2)
[other assertions ...]
print("All test cases passed!")
if __name__ == "__main__":
sys.exit(TestSolution().test_maxEADAR())What the Model "Thinks": Hidden-State Comparison
Blue dots: normal trajectories. Warm dots: hacking trajectories.
Closer to Real Inference-Time Hacks
Training-time trajectories collected by Trace-and-Amplify framework better match real base-model hacking than prompt-elicited ones.
Text-Embedding Similarity to Real Inference Hacking Data
Monitor Generalize to Real Inference Hacks
Takeaway: training-time data better matches real inference-time hacking in embedding similarity, and monitors trained on TT data transfer more effectively to real inference-time hacking.
BibTeX
@misc{li2026promptelicitedtrajectories,
title={Do Prompt-Elicited Trajectories Reflect Training-Time Reward Hacking? A Systematic Study on Monitoring Trainig-Time Reward Hacking in Code Generation},
author={Lichen Li and Hengguang Zhou and Yijun Liang and Tianyi Zhou and Cho-Jui Hsieh},
year={2026},
eprint={2604.23488},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2604.23488},
note={Code: https://github.com/LichenLillc/CoTMonitoring.git}
}