paper
active
2026
paper:doi-10-48550-arxiv-2602-10067

Features as Rewards: Scalable Supervision for Open-Ended Tasks via Interpretability

TL;DR

A frozen Gemma-3-12B-IT's own activations can serve as a dense reward for open-ended factuality training, replacing a $344,064 Gemini 2.5 Pro web-search judge with a roughly $3,818 probe pipeline. The method, RLFR (Reinforcement Learning from Feature Rewards), chains four probes. A localization probe and a classification probe (AUC 0.94) flag candidate hallucinated claims. Retraction and correction probes score the policy's maintain/retract/correct interventions on those claims. Trained for 360 steps with ScaleRL/CISPO on LongFact++, the policy cuts hallucinations by 58% overall with best-of-32 sampling when run with the probing harness. That reduction splits into 10% from the policy itself hallucinating less, 35% from inlining interventions, and 13% from direct resolution. Benchmarks (MMLU 67.2 vs 69.2, GSM8K 73.0 vs 72.3), entity counts (63.4 vs 63.6) and Gemini preference (50.9%) show no collapse into fewer claims. At N=256, probe-scored best-of-N beats Gemma's own 1–10 self-judgment by nearly 15 points. The model therefore encodes an ordering of good interventions that it cannot verbalize. SFT on the same labels reached only 39.4% inline and 9.6% not-inline overall reduction. The implication is that calibrated probes can act as scalable supervision where verification is costly, and the same probes also support test-time scaling. Sycophancy and verbosity are the proposed next targets.

What to take away

  1. 1. RLFR on Gemma-3-12B-IT with best-of-32 sampling yields a 58.0% (±1.4%) overall hallucination reduction on LongFact++ test prompts, versus 56.6% without best-of-N.
  2. 2. The 58% decomposes into 10% policy reduction (the model hallucinates less unaided), 35% in-context reduction from inlining interventions, and 13% direct reduction from interventions resolving flagged claims.
  3. 3. Reward compute for the first 300 steps cost about $3,818 (12.43 h × 128 H200 × $2.4/h), against a conservative $344,064 for Gemini 2.5 Pro web search at 32,768 samples/step, roughly two orders of magnitude cheaper.
  4. 4. At N=256 on base-model interventions, probe-scored best-of-N beats Gemma-3-12B-IT's own 1–10 self-grading by nearly 15 percentage points, so the activations rank interventions better than the model's text output does.
  5. 5. A two-stage monitor (localization probe, then classification probe) gets AUC 0.94 on validation but only 0.85 precision and 0.56 recall at a 0.7 threshold on live inline policy rollouts, and 44% of hallucinations go undetected.
  6. 6. Training the reward probes on frozen π_base activations still works on the trained policy: Fixed and Retracted rates are essentially unchanged whether best-of-32 scoring reads base or RLFR activations, so only one set of weights needs hosting at test time.
  7. 7. Off-target checks show no obvious collapse: entities per completion are 63.4±0.4 for the policy vs 63.6±0.3 for the base, MMLU is 67.2 vs 69.2, GSM8K is 73.0 vs 72.3, and Gemini prefers the policy 50.9% of 1000 pairs.
  8. 8. Methodology others can copy: fit separate low-expressivity attention probes per desired action, one for retraction and one for correction, on Gemini-graded labels, clip rewards to [0.1, 0.95], and balance a 60:40 correct:retract ratio with a Lagrange multiplier (λ lr 0.2, max 1.0).
  9. 9. SFT baselines on the same probe-training data were weak: SFT on correction data gave only 2–4% Fixed-rate gains over 1–3k steps, and SFT on correction plus retraction data reached 39.4% inline and 9.6% not-inline overall reduction versus RLFR's far larger gains.
  10. 10. Open question: why inlining interventions drives 35% of the reduction. The paper hypothesizes a belief shift in context, supported only by a preliminary PCA/dendrogram analysis of layer-24 activations, and also leaves open whether the probes stay unevadable under longer optimization (the paper says this is untested).
  11. 11. Placeholder

Peer brief — for seminar discussion

RLFR (Reinforcement Learning from Feature Rewards) asks whether a model's internal feature readouts can replace an expensive LLM judge as the reward in RL on an open-ended behavior. The testbed is long-form hallucination reduction on LongFact++ with Gemma-3-12B-IT. Probes are fit on frozen base-model activations to imitate Gemini 2.5 Pro labels. A localization probe and a classification probe (AUC 0.94 on validation) detect suspect claims. When a claim fires, the policy chooses to maintain, retract or correct it in a side context. Separate retraction and correction probes score that intervention, multiplied by legibility and substantiveness judges and balanced with a Lagrangian toward a 60:40 correct:retract mix. Training runs 360 steps of ScaleRL/CISPO. The headline is a 58% overall hallucination reduction with best-of-32 sampling. Of that, 10% is the policy hallucinating less, 35% comes from inlining interventions into context, and 13% is direct resolution. Reward compute was about $3.8k against roughly $344k for a search-grounded Gemini judge. Probe-scored best-of-N also beats Gemma's own 1–10 self-grading by nearly 15 points at N=256, which suggests the model represents quality it cannot verbalize. Benchmarks, entity counts (63.4 vs 63.6) and a 50.9% preference rate argue against degenerate caution. The implication is that calibrated probes are a cheap, dense alternative to learned reward models, with less room for reward hacking because of their low expressivity. The paper does not test this on sycophancy or verbosity, which it names as next targets. Its stated hypothesis is that inlined interventions work through in-context belief change. A critical reader would push back on three things. First, the 58% is not a policy property. The policy alone gives 10%, and the rest depends on the probe harness, which has 0.56 recall and 0.85 precision live, so 44% of hallucinations are never touched. Second, ground truth, probe labels and evaluation all rest on Gemini 2.5 Pro with web search, so the circularity is only partly mitigated by auditing roughly 500 examples. Third, the feature-reward claim is weakly isolated. The only judge comparison is best-of-N on base-model samples. The RL run against an LLM judge was abandoned for cost, and SFT is the only trained baseline, run on one seed. An alternative design would be RL against a cheaper distilled judge or a small trained reward model. The no-evasion claim is also asserted from a single 360-step run.

Methods (17)

Frameworks (2)

Datasets (5)

Original abstract (expand)

Language models trained on large-scale datasets have been shown to learn features that encode abstract concepts such as factuality or intent. Such features are traditionally used for test-time monitoring or steering. We present an alternative affordance: features as scalable supervision for open-ended tasks. We consider the case of hallucination-reduction as a desirable, yet open-ended behavior and design a reinforcement learning (RL) pipeline, titled RLFR (Reinforcement Learning from Feature Rewards), that uses features as reward functions. Grounded in a novel probing framework that identifies candidate hallucinated claims, our pipeline teaches a model to intervene and correct its completions when it is uncertain of their factuality. Furthermore, the pipeline enables scalable test-time compute, guided once more by our reward features. This end-to-end process operationalized on Gemma-3-12B-IT results in a policy that is 58% less likely to hallucinate compared to the original model (when run in tandem with our probing harness), while preserving performance on standard benchmarks. Taken together, by grounding supervision in the language of features, this paper introduces a novel paradigm in the use of interpretability for learning open-ended tasks.

Related work— refs + corpus + external arXiv

Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.

Similar preprints — Semantic Scholar