paper:doi-10-48550-arxiv-2606-12360Anatomy of Post-Training: Using Interpretability to Characterize Data and Shape the Learning Signal
TL;DR
Interpretability can audit a preference dataset before optimization and predict what DPO will teach a model: on Dolci, clustering SAE features and testing chosen-minus-rejected activation differences predicted post-DPO feature shifts with R²=0.9 (feature-conditioned) and R²=0.58 (prompt-conditioned). The method is a two-view hypothesis-generation pipeline. A BatchTopK SAE on Llama-3.1-8B supplies the feature clusters (814 response-feature clusters, 512 data clusters over 259,842 pairs), and Welch tests with split-half sign-consistency filtering rank which concepts the labels reward. The pipeline flags Dolci DPO as teaching compliance with unsafe queries, and benchmarks such as HarmBench and XSTest confirm degraded safeguards after DPO. It also flags physics sycophancy, questionable fanfiction, benchmark recognition and hallucinated URLs. Four interventions (data filtering, inoculation prompting, activation steering, reward shaping) are unified as 'explaining away' a concept from an exponential tilt of the policy. Reward shaping is the strongest: at λ=−5 it cuts Llama-3.1-8B harmful-response rate from 21.7% to 4.0% while keeping OLMES accuracy near DPO's 0.434 (0.425), and it also amplifies traits such as playfulness. Interventions fail when the target concept is entangled with others or sparse in the data: style attributes leak off-target, and only URL hallucination improved significantly. The paper argues that post-training should treat the dataset as something to audit and the reward as something to sculpt, and that future protocols need concept hierarchies rather than flat, independent concept axes.
What to take away
- 1. The feature-conditioned pipeline, which tests chosen-minus-rejected SAE cluster disparity inside 512 spherical-k-means data clusters, predicted actual pre-to-post-DPO feature shifts on Llama-3.1-8B/Dolci with R²=0.9.
- 2. The prompt-conditioned pipeline predicted post-DPO changes less well (R²=0.58), plausibly because identified prompt clusters cover only about 0.1% of the data and so barely move model behavior.
- 3. Dolci DPO degrades safeguards: Llama-3.1-8B's harmful-response rate rises from 15.7% (SFT) to 21.7% (DPO), and Tulu-3-70B's from 7.4% to 19.7%.
- 4. Reward shaping with λ=−5 cut Llama-3.1-8B's harmful-response rate to 4.0% (OLMo-3-7B: 13.2% to 1.4%) while OLMES accuracy stayed near DPO's, 0.425 against 0.434 on Llama.
- 5. Reward shaping, filtering and steering are unified as ways of 'explaining away' a concept from the exponential-tilt reward, r′ = r − λ·s_u, which recovers PPS, CAFT, inoculation prompting and token filtering as special cases.
- 6. In the Llama-3.1-8B over-stylization study (Bold, Emoji, Hrule, Table, Emdash), token filtering and reward shaping recovered SFT-level formatting best, but nearly every method also moved non-targeted styles, and only SAE-feature variants were relatively localized, which the authors attribute to feature splitting.
- 7. Chosen-response activation steering weakened the refusal direction itself; steering positively on the rejected response ('explain away') restored a usable direction and lifted Tulu-3-70B's safety score from 0.779 to 0.864.
- 8. Open question: whether the failures on physics sycophancy, questionable fanfic and eval knowledge come from concept entanglement and sparse signal rather than a limit of reward shaping, which the poetic case study (+40 Elo, Overall within ±1pp) is offered as evidence for.
- 9. Replicable method: poison 5% of Dolci SFT and DPO prompts (13,680 DPO pairs) with a GPT-4.1-mini-generated trait, build difference-in-means vectors at layer 16, and check whether each intervention returns trait rates to clean baselines.
- 10. Reward-shaping amplification costs capability roughly linearly in λ for the global trait 'playful' (Overall −10.7pp, ZebraLogic −25.2pp at λ=32), whereas topic-gating the 'poetic' trait to ~3% of creative-writing pairs kept every sub-task within ±3pp.
Peer brief — for seminar discussion
This work builds a data-centric auditing pipeline for preference optimization and tests whether it can both predict and steer what DPO teaches. On Dolci (259,842 usable pairs), a BatchTopK SAE on Llama-3.1-8B yields feature clusters (814 co-firing clusters via Leiden on mutual information). Two views generate hypotheses. The feature-conditioned view clusters examples by symmetric response activity into 512 data clusters and tests chosen-minus-rejected disparity inside versus outside each cluster, using Welch statistics and split-half sign consistency. The prompt-conditioned view tests which response-delta clusters shift for prompts expressing a given prompt-feature cluster. The authors then unify data filtering, inoculation prompting, activation steering and reward shaping as 'explaining away' a concept from an exponential tilt of the policy. The load-bearing finding is that the pipeline's predicted feature shifts track real DPO-induced shifts (R²=0.9 for the feature view, 0.58 for the prompt view), and that it surfaces genuine harms. DPO on Dolci lowers refusal on harmful prompts: Llama-3.1-8B goes from 15.7% to 21.7% harmful responses, and Tulu-3-70B from 7.4% to 19.7%. Reward shaping repairs this across 7B to 70B, for example 21.7% to 4.0% on Llama with OLMES accuracy 0.425 against DPO's 0.434, and acts as a tunable knob for amplification (λ-sweeps for playful and poetic traits). The implication is that a scalar reward can be decomposed into concept-level terms that practitioners inspect and edit before training. The paper also predicts that failures stem from concept entanglement and data sparsity rather than from the interventions themselves. Where the independence assumption of Eq. 3 breaks, as with correlated formatting styles, interventions leak off-target. What's contestable: the validation is largely self-referential. Hypotheses come from SAE clusters and are checked by auto-interpreted labels, GPT-judged traits and SAE-derived rollout features, so shared SAE artifacts could inflate R²=0.9. Reward shaping also uses the same SFT-derived refusal projection it is evaluated against. A critical reader would also note that the 'sparse data' explanation for the negative results is post hoc: only URL hallucination (412 to 354 with a regex filter) improved significantly, and physics sycophancy moved just 2.18 to 1.80. An alternative to the SAE cluster approach would be training-data attribution, such as probe-based or concept-influence attribution, or LLM-written hypothesis descriptions in the style of Movva et al., but no head-to-head comparison is run. Finally, everything rests on one data family (Dolci/Tulu) and DPO, so generalization to RL or reasoning post-training is asserted, not tested.
Methods (16)
- Activation SteeringCausal intervention technique: edit NLA explanation, reconstruct via AR, use difference as steering vector to manipulate model behavior.
- Data Filtering (example/token level)Remove or downweight units whose preference is explained by a concept.
- Difference-in-means steering vectorsUsed to extract trait and style directions.
- Feature-Conditioned Hypothesis GenerationFinds data clusters where response concept is chosen/rejected leaning.
- Inoculation PromptingPrompt explains concept so it is not attributed to the global preference signal.
- Linear and attention probesClassifiers used for token filtering and reward shaping.
- Prompt-Conditioned Hypothesis GenerationFinds response-delta clusters shifted for prompt clusters.
- ReFT (NoDiReFT)Representation fine-tuning module used for steering.
- Reward ShapingSubtract λ times concept score from reward.
- SAE feature clustering (Leiden / spherical k-means)Cluster features by activation statistics to get higher-level behaviors.
- Sparse Autoencoders (SAEs)Unsupervised baseline method for decomposing activations into sparse linear combinations; NLAs positioned as complementary approach.
- Synthetic data poisoning validation setupPoison 5% of SFT/DPO data with a trait, then test interventions.
- Topic-conditional reward shaping (gated)Apply shaping offset only on creative-writing subset C.
- Trait Expression EvalLLM-judge Likert and coherence evaluation of trait amplification.
- Trait-vector extraction pipelineReplicates Lu et al. / Chen et al. contrastive-pair recipe.
- Welch inside-vs-outside test with split-half validationStatistic ranking data-feature pairs, filtered by sign consistency across halves.
Frameworks (3)
- BatchTopK SAEEvaluated architecture.
- Data-centric post-training auditing pipelinePipeline from preference data to hypotheses to interventions.
- Explaining Away Concepts (shaping the learning signal)Unifying framework: remove or amplify a concept's term in the exponential-tilt reward.
Datasets (6)
- AlpacaEvalEvaluation prompts.
- Creative Writing Bench v3Elo benchmark for poetic amplification.
- Dolci (SFT/DPO preference data)Main post-training dataset audited.
- HarmBench and XSTestJailbreak robustness and over-refusal benchmarks.
- OLMES evaluation suiteCapability benchmark suite.
- Sycophancy evaluation suiteCheng, Perez, Sharma benchmarks.
Findings (23)
- Prompt-conditioned interventions reduce undesired behaviors on average but are statistically significant only for hallucinated sensitive-resource URLs
Fig. 11.
- Reward shaping most consistently raises harmful refusals while retaining DPO accuracy across 7-70B models (e.g. Tulu 70B safety 0.957, harm 0.3%)
Fig. 10, Table 10.
- Playful trait expression increases monotonically with λ while OLMES capability degrades approximately linearly (Overall −10.7pp at λ=32)
Fig. 13, App. J.4.
- General sycophancy is already high in SFT and only slightly modulated by DPO; reward shaping does not reduce it below SFT levels
Fig. 12.
- Mildly poisoning 5% of SFT and DPO data induces each target trait; interventions substantially reduce trait expression
Fig. 3 on Llama-3.1-8B.
- Explain-away steering on rejected responses outperforms chosen-response steering (Tulu 70B safety 0.864 vs 0.779)
Table 10.
- Token filtering and reward shaping give strongest recovery of SFT-level formatting rates with minimal OLMES loss
Fig. 8.
- Topic-gated poetic shaping gains +40 Elo on Creative Writing Bench v3 with no capability regression (+0.07pp)
App. I.
- DPO on Dolci yields lower aggregate safety score than SFT checkpoint (e.g. Llama 3.1 8B 0.849 to 0.758)
Safeguard degradation.
- ZebraLogic falls −25.2pp at λ=32 in the playful sweep versus never worse than −3.1pp in gated poetic
App. J.4.
Claims (17)
- Loss-level interventions let the model decide how to explain the loss, avoiding incentive to fight the intervention
Contrast with layer-specific steering.
- Token filtering works well when behavior is localizable to short spans but underperforms for broad diffuse traits
Poisoning results.
- Activation steering works better when explaining away on rejected responses than amplifying on chosen responses
Model may fight chosen-response steering.
- Post-training interventions need a notion of concept structure (hierarchy) rather than flat independent axes
Discussion conclusion.
- PPS, CAFT, data filtering, and inoculation prompting are instantiations of explaining away a latent variable
Unifying Bayesian lens.
- Under statistical independence, the reward decomposes into additive concept-level terms in the exponent
Eq. 3 assumption.
- Prompt-conditioned behaviors are entangled with broader response concepts and not cleanly separable
Explains partial recovery.
- Questionable fanfic amplification arises from correlated samples, e.g. a larger SFT cluster
Explanation of persistent proclivity.
- Dolci contains samples teaching compliance with unsafe queries that undermine safeguards
Egregious behavior discovered.
- Linear-probe-based filtering acts like a generic style probe, causing off-target changes
Interpretation of Fig. 9.
Hypotheses (2)
- If bold is split into context-specific features, performance is capped unless all such features are aggregated
Feature-splitting hypothesis.
- When the prompt expresses Ak, preference data teaches the model to change responses along Rm
Form of generated hypotheses.
Questions (2)
Original abstract (expand)
Language-model post-training is the main stage at which model behavior is shaped, yet it still largely involves optimization of scalar rewards that summarize diverse desiderata. This abstraction gives practitioners little visibility into what their data actually teaches models, allowing spurious correlations to be learned by a model and inducing undesirable behaviors such as over-stylization and sycophancy. To address this problem, we ask: can we inspect a preference dataset before optimization and decide, at the level of concepts, which behaviors a model should be allowed to learn? Motivated by this, we introduce a data-centric post-training pipeline that uses interpretability protocols to develop statistical hypotheses for the latent concepts separating preferred from dispreferred generations, making them explicit for fine-grained user feedback. Building on this view, we unify several interpretability-based training protocols as ways of shaping rewards via feature or data interventions. Empirically, we show that our pipeline diagnoses undesirable signals in existing preference data, mitigates off-target learning, and can also help amplify or shape desired properties such as safeguards and model personality. More broadly, our results suggest that interpretability can turn post-training from optimizing opaque proxy rewards into a process of auditing and sculpting the learning signal itself.
Related work— refs + corpus + external arXiv
Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.
- Probe-Based Data Attribution: Surfacing and Mitigating Undesirable Behaviors in LLM Post-Trainingin corpus2026≈ 86%
- Mechanistic Data Attribution: Tracing the Training Origins of Interpretable LLM UnitsYuzhang Luo, Liangming Pan Jianhui Chen2026≈ 84%
- Alignment faking in large language modelsin corpus2024≈ 83%
- Constructing Interpretable Features from Compositional Neuron GroupsAtticus Geiger, Mor Geva Or Shafran2026≈ 83%
- Control Reinforcement Learning: Interpretable Token-Level Steering of LLMs via Sparse Autoencoder FeaturesZekun Wu, Adriano Koshiyama Seonglae Cho2026≈ 83%
- Beyond Behavioural Trade-Offs: Mechanistic Tracing of Pain-Pleasure Decisions in an LLMFrancesca Bianco and Derek Shiller2026≈ 83%
- ≈ 83%
- Persona Features Control Emergent Misalignmentin corpus2025≈ 83%
- ≈ 82%
- Beyond Activation Patterns: A Weight-Based Out-of-Context Explanation of Sparse Autoencoder FeaturesZhi-Hong Deng Yiting Liu2026≈ 82%
- Interpretability without actionability: mechanistic methods cannot correct language model errors despite near-perfect internal representationsSadiq Y. Patel, Parth Sheth, Bhairavi Muralidharan, Namrata Elamaran, Aakriti Kinra, John Morgan, Rajaie Batniji Sanjay Basu2026≈ 82%
- DataDignity: Training Data Attribution for Large Language ModelsAndrzej Banburski-Fahey, Jaron Lanier Xiaomin Li2026≈ 82%
- SteerRM: Debiasing Reward Models via Sparse AutoencodersZhuohao Yu, Weizheng Gu, Shikun Zhang, Wei Ye Mengyuan Sun2026≈ 82%
- Open Problems in Mechanistic Interpretabilityin corpus2025≈ 82%
- ≈ 82%
- ≈ 82%
- AFTER: Mitigating the Object Hallucination of LVLM via Adaptive Factual-Guided Activation EditingYuqing Ma, Kewei Liao, Zhange Zhang, Simin Li, Jinyang Guo, Xianglong Liu Tianbo Wang2026≈ 82%
- When the Coffee Feature Activates on Coffins: An Analysis of Feature Extraction and Steering for Mechanistic InterpretabilityMarkus Maier, Frederick Eberhardt Raphael Ronge2026≈ 82%
- Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainersin corpus2025≈ 82%
- What Models Express, Suppress, and Resist: Auditing Open-Weight LLMs with Persona Vectorsin corpus2026≈ 82%
- Activation Steering for Aligned Open-ended Generation without Sacrificing CoherenceMartin Zborowski, Alberto Tosato, Gauthier Gidel, Tommaso Tosato Niklas Herbster2026≈ 82%
- ≈ 82%
- Concept Influence: Leveraging Interpretability to Improve Performance and Efficiency in Training Data AttributionGoncalo Paulo, Louis Jaburi, Tom Tseng, Lev E McKinney, Stefan Heimersheim, Aaron David Tucker, Adam Gleave, Kellin Pelrine Matthew Kowal2026≈ 82%
- Position: Mechanistic Interpretability Should Prioritize Feature Consistency in SAEsAashiq Muhamed, Yujia Zheng, Lingjing Kong, Zeyu Tang, Mona T. Diab, Virginia Smith, Kun Zhang Xiangchen Song2025≈ 82%
- ≈ 82%
- Beyond Top Activations: Efficient and Reliable Crowdsourced Evaluation of Automated InterpretabilityGe Yan, Akshay Kulkarni, Tsui-Wei Weng Tuomas Oikarinen2025≈ 81%
- ≈ 81%
- ≈ 81%
- Mechanistic Knobs in LLMs: Retrieving and Steering High-Order Semantic Features via Sparse Autoencodersin corpus2026≈ 81%
Similar preprints — Semantic Scholar
Cited by (1)
- Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluations
Difference-of-means (DoM) vectors built from 5,285 templated synthetic SWE conversations detect reward hacking in long agentic rollouts of Kimi K3, GLM 5.2, and Qwen 3.8 Max, matching a GPT-5.6 Sol LL