paper:doi-10-48550-arxiv-2602-10067Features as Rewards: Scalable Supervision for Open-Ended Tasks via Interpretability
TL;DR
A frozen Gemma-3-12B-IT's own activations can serve as a dense reward for open-ended factuality training, replacing a $344,064 Gemini 2.5 Pro web-search judge with a roughly $3,818 probe pipeline. The method, RLFR (Reinforcement Learning from Feature Rewards), chains four probes. A localization probe and a classification probe (AUC 0.94) flag candidate hallucinated claims. Retraction and correction probes score the policy's maintain/retract/correct interventions on those claims. Trained for 360 steps with ScaleRL/CISPO on LongFact++, the policy cuts hallucinations by 58% overall with best-of-32 sampling when run with the probing harness. That reduction splits into 10% from the policy itself hallucinating less, 35% from inlining interventions, and 13% from direct resolution. Benchmarks (MMLU 67.2 vs 69.2, GSM8K 73.0 vs 72.3), entity counts (63.4 vs 63.6) and Gemini preference (50.9%) show no collapse into fewer claims. At N=256, probe-scored best-of-N beats Gemma's own 1–10 self-judgment by nearly 15 points. The model therefore encodes an ordering of good interventions that it cannot verbalize. SFT on the same labels reached only 39.4% inline and 9.6% not-inline overall reduction. The implication is that calibrated probes can act as scalable supervision where verification is costly, and the same probes also support test-time scaling. Sycophancy and verbosity are the proposed next targets.
What to take away
- 1. RLFR on Gemma-3-12B-IT with best-of-32 sampling yields a 58.0% (±1.4%) overall hallucination reduction on LongFact++ test prompts, versus 56.6% without best-of-N.
- 2. The 58% decomposes into 10% policy reduction (the model hallucinates less unaided), 35% in-context reduction from inlining interventions, and 13% direct reduction from interventions resolving flagged claims.
- 3. Reward compute for the first 300 steps cost about $3,818 (12.43 h × 128 H200 × $2.4/h), against a conservative $344,064 for Gemini 2.5 Pro web search at 32,768 samples/step, roughly two orders of magnitude cheaper.
- 4. At N=256 on base-model interventions, probe-scored best-of-N beats Gemma-3-12B-IT's own 1–10 self-grading by nearly 15 percentage points, so the activations rank interventions better than the model's text output does.
- 5. A two-stage monitor (localization probe, then classification probe) gets AUC 0.94 on validation but only 0.85 precision and 0.56 recall at a 0.7 threshold on live inline policy rollouts, and 44% of hallucinations go undetected.
- 6. Training the reward probes on frozen π_base activations still works on the trained policy: Fixed and Retracted rates are essentially unchanged whether best-of-32 scoring reads base or RLFR activations, so only one set of weights needs hosting at test time.
- 7. Off-target checks show no obvious collapse: entities per completion are 63.4±0.4 for the policy vs 63.6±0.3 for the base, MMLU is 67.2 vs 69.2, GSM8K is 73.0 vs 72.3, and Gemini prefers the policy 50.9% of 1000 pairs.
- 8. Methodology others can copy: fit separate low-expressivity attention probes per desired action, one for retraction and one for correction, on Gemini-graded labels, clip rewards to [0.1, 0.95], and balance a 60:40 correct:retract ratio with a Lagrange multiplier (λ lr 0.2, max 1.0).
- 9. SFT baselines on the same probe-training data were weak: SFT on correction data gave only 2–4% Fixed-rate gains over 1–3k steps, and SFT on correction plus retraction data reached 39.4% inline and 9.6% not-inline overall reduction versus RLFR's far larger gains.
- 10. Open question: why inlining interventions drives 35% of the reduction. The paper hypothesizes a belief shift in context, supported only by a preliminary PCA/dendrogram analysis of layer-24 activations, and also leaves open whether the probes stay unevadable under longer optimization (the paper says this is untested).
- 11. Placeholder
Peer brief — for seminar discussion
RLFR (Reinforcement Learning from Feature Rewards) asks whether a model's internal feature readouts can replace an expensive LLM judge as the reward in RL on an open-ended behavior. The testbed is long-form hallucination reduction on LongFact++ with Gemma-3-12B-IT. Probes are fit on frozen base-model activations to imitate Gemini 2.5 Pro labels. A localization probe and a classification probe (AUC 0.94 on validation) detect suspect claims. When a claim fires, the policy chooses to maintain, retract or correct it in a side context. Separate retraction and correction probes score that intervention, multiplied by legibility and substantiveness judges and balanced with a Lagrangian toward a 60:40 correct:retract mix. Training runs 360 steps of ScaleRL/CISPO. The headline is a 58% overall hallucination reduction with best-of-32 sampling. Of that, 10% is the policy hallucinating less, 35% comes from inlining interventions into context, and 13% is direct resolution. Reward compute was about $3.8k against roughly $344k for a search-grounded Gemini judge. Probe-scored best-of-N also beats Gemma's own 1–10 self-grading by nearly 15 points at N=256, which suggests the model represents quality it cannot verbalize. Benchmarks, entity counts (63.4 vs 63.6) and a 50.9% preference rate argue against degenerate caution. The implication is that calibrated probes are a cheap, dense alternative to learned reward models, with less room for reward hacking because of their low expressivity. The paper does not test this on sycophancy or verbosity, which it names as next targets. Its stated hypothesis is that inlined interventions work through in-context belief change. A critical reader would push back on three things. First, the 58% is not a policy property. The policy alone gives 10%, and the rest depends on the probe harness, which has 0.56 recall and 0.85 precision live, so 44% of hallucinations are never touched. Second, ground truth, probe labels and evaluation all rest on Gemini 2.5 Pro with web search, so the circularity is only partly mitigated by auditing roughly 500 examples. Third, the feature-reward claim is weakly isolated. The only judge comparison is best-of-N on base-model samples. The RL run against an LLM judge was abandoned for cost, and SFT is the only trained baseline, run on one seed. An alternative design would be RL against a cheaper distilled judge or a small trained reward model. The no-evasion claim is also asserted from a single 360-step run.
Methods (17)
- Activation dendrogram analysis (PCA + cosine + hierarchical clustering)Preliminary interpretability of interventions.
- Attention heatmaps of reward probesCoarse assessment of what reward probes attend to.
- Best-of-N sampling with probe scoringTest-time compute scaling guided by reward probes with majority-vote over action.
- CISPOPolicy optimization protocol used.
- Classification probeAttention probe predicting whether an Entity is hallucinated; AUC .94 on validation.
- Correction probeReward probe grading correction interventions (5-class).
- Decomposed probing protocolTwo-step localize-then-classify pipeline for hallucination monitoring and reward.
- Gated Multi-Head Attention (GMHA) attention probeProbe architecture with a learned query per head for span-level prediction.
- Inline intervention strategyInserting interventions into the completion with in-context effects.
- Inverse Reinforcement LearningValue learning method inferring reward function from expert demonstrations; reviewed as insufficient for superintelligent alignment
- Lagrangian ratio constraint for retraction/correction balanceLearned multiplier maintaining 60:40 correction:retraction ratio.
- LLM-as-a-Judge baseline (Gemma self-grading)Baseline scoring interventions 1-10 token-space.
- Localization probeCausal transformer probe predicting whether a token is in the same Entity as its predecessor.
- Multiplicative rubric reward (legibility × substantiveness × probe score)Overall reward is product of judge-based and probe-based terms.
- Not-inline intervention strategyInterventions recorded but not appended; control for direct effects.
- ProbingStandard interpretability technique used to read feature beliefs.
- Retraction probeReward probe grading retraction interventions.
Frameworks (2)
- RLFR (Reinforcement Learning from Feature Rewards)Core pipeline turning probe-read features into scalable reward functions for open-ended tasks.
- ScaleRLRL recipe used for training.
Datasets (5)
- Classification dataset (~5M entities)Entities labeled Supported/Not Supported/Insufficient from 76,849 completions.
- Correction datasetReward labels for NOT SUPPORTED+CORRECT interventions.
- Longfact++~20K long-form prompts across eight domains for training and evaluation.
- Retraction datasetReward labels for NOT SUPPORTED+RETRACT interventions.
- Standard benchmark suite (HellaSwag, PIQA, ARC, WinoGrande, BBH, MMLU, MATH, GSM8K, GPQA)Off-target capability check.
Findings (29)
- At threshold 0.7, monitoring achieves .85 precision/.56 recall on inline and .88 precision/.61 recall on not-inline interventions
Test-time probe metrics.
- Gemini web-search reward for first 300 steps would cost ~$344,064 versus ~$3,818 for the probe pipeline (~2 orders of magnitude)
Cost computation.
- Policy rollout dendrograms cluster intervention, entity, and action in separate sub-branches; absent in base rollouts
Preliminary mechanism evidence.
- RLFR with best-of-32 achieves 58.0%±1.4% overall hallucination reduction on Gemma-3-12B-IT with monitoring harness
Topline result.
- Most of Policy Reduction and intervention-success gains occur by step 300; Fixed rate still rising at step 360
Train-time scaling.
- ~3% of completions show derailment where an incorrect early intervention causes refusal or repeated retraction
Current degeneracy mode.
- Fixed and Retracted rates are essentially unchanged when reward probe runs on base versus policy activations
Reward pipeline transfer.
- Base corrections: 72.9% Failed Fix, 14.7% Fixed; retractions: 69.4% Correct Retract, 30.0% Incorrect Retract
Label distributions.
- SFT on correction+retraction data gives 39.4% inline overall reduction and 9.6% not-inline, well below RLFR
SFT baseline comparison.
- 56% of test-time hallucinations are caught; of these 22% Fixed, 36% Retracted, 42% improperly handled
Fig. 8 decomposition.
Claims (15)
- If a model learns the data distribution it is incentivized to represent latent variables as concrete features that can be reused for reward.
Justification for reward probes.
- Features serving as oversight signals is a novel paradigm of interpretability research for designing models with desirable capabilities.
Broad framing.
- Low-expressivity probes on pretrained features constrain solutions and induce sample efficiency relative to learned reward models.
Contrast with reward model non-identifiability.
- The ordering of interventions under the probes is invariant to activation source, so only the policy need be hosted at test time.
Practical implication.
- Running probes on frozen parameters mitigates monitor evasion because it is easier to learn the behavior than evade the probe.
Limitation discussion.
- Gemma represents a strong ordering of interventions in activations but cannot output it in token space when instructed.
Interpretation of probe vs judge gap.
- The two-step localize-then-classify pipeline is critical for a reliable high-true-positive-rate probing pipeline.
Design justification.
- Derailment behavior should be solvable with a more competent base model and explicit inline training.
Expectation.
- A policy partially becomes factual by being primed for acting in a factually precise manner.
Interpretation of merged dendrogram clusters.
- Model features can serve as inexpensive, dense, scalable supervision for open-ended tasks.
Central thesis.
Hypotheses (3)
- The reduction in hallucination rate under inline interventions occurs due to a change in model beliefs in-context.
Mechanism hypothesis for future work.
- A cascade of reward functions, escalating to expensive verifiers when probe confidence is low, is worthwhile.
Future work.
- The pipeline can regulate behaviors such as sycophancy and verbosity.
Future applications.
Questions (4)
- What mechanisms underlie the changes to the policy, including the reduction when interventions are inlined?
Open interpretability question.
- Can we transform a model's features into an inexpensive, dense signal for learning open-ended tasks?
Motivating question.
- If a model's features are well-calibrated, can one use these features as sources of supervision?
Motivating question.
- Was such optimization even necessary?
Train-time analysis.
Original abstract (expand)
Language models trained on large-scale datasets have been shown to learn features that encode abstract concepts such as factuality or intent. Such features are traditionally used for test-time monitoring or steering. We present an alternative affordance: features as scalable supervision for open-ended tasks. We consider the case of hallucination-reduction as a desirable, yet open-ended behavior and design a reinforcement learning (RL) pipeline, titled RLFR (Reinforcement Learning from Feature Rewards), that uses features as reward functions. Grounded in a novel probing framework that identifies candidate hallucinated claims, our pipeline teaches a model to intervene and correct its completions when it is uncertain of their factuality. Furthermore, the pipeline enables scalable test-time compute, guided once more by our reward features. This end-to-end process operationalized on Gemma-3-12B-IT results in a policy that is 58% less likely to hallucinate compared to the original model (when run in tandem with our probing harness), while preserving performance on standard benchmarks. Taken together, by grounding supervision in the language of features, this paper introduces a novel paradigm in the use of interpretability for learning open-ended tasks.
Related work— refs + corpus + external arXiv
Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.
- Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluationsin corpus2026≈ 84%
- ≈ 84%
- Not All Tokens See Equally: Perception-Grounded Policy Optimization for Large Vision-Language ModelsQiming Li, Xiaocheng Feng, Ruihan Chen, Ziming Li, Haoyu Ren, Kun Chen, Dandan Tu, Bing Qin Zekai Ye2026≈ 83%
- Probe-Based Data Attribution: Surfacing and Mitigating Undesirable Behaviors in LLM Post-Trainingin corpus2026≈ 83%
- Control Reinforcement Learning: Interpretable Token-Level Steering of LLMs via Sparse Autoencoder FeaturesZekun Wu, Adriano Koshiyama Seonglae Cho2026≈ 83%
- Interpretable Reward Model via Sparse AutoencoderWei Shi, Sihang Li, Jiayi Liao, Hengxing Cai, Xiang Wang Shuyi Zhang2025≈ 82%
- Anatomy of Post-Training: Using Interpretability to Characterize Data and Shape the Learning Signalin corpus2026≈ 82%
- ≈ 82%
- Alignment faking in large language modelsin corpus2024≈ 82%
- Reinforce to Learn, Elect to Reason: A Dual Paradigm for Video ReasoningWeijiang Yu, Jilin Ma, Ziyu Liu, Guijian Tang, Wenjing Yang, Huibin Tan, Nong Xiao Songyuan Yang2026≈ 82%
- SFR-DeepResearch: Towards Effective Reinforcement Learning for Autonomously Reasoning Single Agentsin corpus2025≈ 82%
- Detection Without Correction: A Robust Asymmetry in Activation-Based Hallucination ProbingRajiv Misra, Sanjay Kumar Singh, Anisha Roy Dip Roy2026≈ 82%
- Do Benchmarks Underestimate LLM Performance? Evaluating Hallucination Detection With LLM-First Human-Adjudicated AssessmentB. Mutlu, E. A. Sezer, A. Wahdan I. F. Atasoy2026≈ 82%
- ≈ 81%
- ≈ 81%
- Bridging Visual Representation and Reinforcement Learning from Verifiable Rewards in Large Vision-Language ModelsYuyang Wu, Zhengbo Jiao, Yiyu Wang, Xuyang Liu, Shaobo Wang, Hanlin Xu, Xuming Hu, Linfeng Zhang Yuhang Han2026≈ 81%
- Intrinsic Rewards for Exploration without Harm from Observational Noise: A Simulation Study Based on the Free Energy PrincipleKenji Doya, Jun Tani Theodore Jerome Tinker2024≈ 81%
- Hallucination reduction with CASAL: Contrastive Activation Steering For Amortized LearningXinchi Qiu, Lei Yu, Yuchen Zhang, Aobo Yang, Narine Kokhlikyan, Nicola Cancedda, Diego Garcia-Olano Wannan (Winnie) Yang2025≈ 81%
- Reinforcing Structured Chain-of-Thought for Video UnderstandingHaotian Xu, Noranart Vesdapunt, Rui Hou, Jingyi Zhang, Haibin Ling, Oleksandr Obiednikov, Ning Zhou, Kah Kuen Fu Peiyao Wang2026≈ 81%
- Reinforcement Learning with Inverse Rewards for World Model Post-trainingTianyu He, Shuo Yang, Jiang Bian Yang Ye2025≈ 81%
- UniDoc-RL: Coarse-to-Fine Visual RAG with Hierarchical Actions and Dense RewardsShuo Tan, Zelong Sun, Tiancheng Gu, Yongle Zhao, Ziyong Feng, Kaicheng Yang, Zhiwu Lu Jun Wang2026≈ 81%
- ≈ 81%
- ≈ 81%
- What Models Express, Suppress, and Resist: Auditing Open-Weight LLMs with Persona Vectorsin corpus2026≈ 81%
- ≈ 80%
- ≈ 80%
- Persona Features Control Emergent Misalignmentin corpus2025≈ 80%
- ≈ 80%
- Mechanistic Knobs in LLMs: Retrieving and Steering High-Order Semantic Features via Sparse Autoencodersin corpus2026≈ 80%