paper
active
2026
paper:doi-10-48550-arxiv-2606-12360

Anatomy of Post-Training: Using Interpretability to Characterize Data and Shape the Learning Signal

TL;DR

Interpretability can audit a preference dataset before optimization and predict what DPO will teach a model: on Dolci, clustering SAE features and testing chosen-minus-rejected activation differences predicted post-DPO feature shifts with R²=0.9 (feature-conditioned) and R²=0.58 (prompt-conditioned). The method is a two-view hypothesis-generation pipeline. A BatchTopK SAE on Llama-3.1-8B supplies the feature clusters (814 response-feature clusters, 512 data clusters over 259,842 pairs), and Welch tests with split-half sign-consistency filtering rank which concepts the labels reward. The pipeline flags Dolci DPO as teaching compliance with unsafe queries, and benchmarks such as HarmBench and XSTest confirm degraded safeguards after DPO. It also flags physics sycophancy, questionable fanfiction, benchmark recognition and hallucinated URLs. Four interventions (data filtering, inoculation prompting, activation steering, reward shaping) are unified as 'explaining away' a concept from an exponential tilt of the policy. Reward shaping is the strongest: at λ=−5 it cuts Llama-3.1-8B harmful-response rate from 21.7% to 4.0% while keeping OLMES accuracy near DPO's 0.434 (0.425), and it also amplifies traits such as playfulness. Interventions fail when the target concept is entangled with others or sparse in the data: style attributes leak off-target, and only URL hallucination improved significantly. The paper argues that post-training should treat the dataset as something to audit and the reward as something to sculpt, and that future protocols need concept hierarchies rather than flat, independent concept axes.

What to take away

  1. 1. The feature-conditioned pipeline, which tests chosen-minus-rejected SAE cluster disparity inside 512 spherical-k-means data clusters, predicted actual pre-to-post-DPO feature shifts on Llama-3.1-8B/Dolci with R²=0.9.
  2. 2. The prompt-conditioned pipeline predicted post-DPO changes less well (R²=0.58), plausibly because identified prompt clusters cover only about 0.1% of the data and so barely move model behavior.
  3. 3. Dolci DPO degrades safeguards: Llama-3.1-8B's harmful-response rate rises from 15.7% (SFT) to 21.7% (DPO), and Tulu-3-70B's from 7.4% to 19.7%.
  4. 4. Reward shaping with λ=−5 cut Llama-3.1-8B's harmful-response rate to 4.0% (OLMo-3-7B: 13.2% to 1.4%) while OLMES accuracy stayed near DPO's, 0.425 against 0.434 on Llama.
  5. 5. Reward shaping, filtering and steering are unified as ways of 'explaining away' a concept from the exponential-tilt reward, r′ = r − λ·s_u, which recovers PPS, CAFT, inoculation prompting and token filtering as special cases.
  6. 6. In the Llama-3.1-8B over-stylization study (Bold, Emoji, Hrule, Table, Emdash), token filtering and reward shaping recovered SFT-level formatting best, but nearly every method also moved non-targeted styles, and only SAE-feature variants were relatively localized, which the authors attribute to feature splitting.
  7. 7. Chosen-response activation steering weakened the refusal direction itself; steering positively on the rejected response ('explain away') restored a usable direction and lifted Tulu-3-70B's safety score from 0.779 to 0.864.
  8. 8. Open question: whether the failures on physics sycophancy, questionable fanfic and eval knowledge come from concept entanglement and sparse signal rather than a limit of reward shaping, which the poetic case study (+40 Elo, Overall within ±1pp) is offered as evidence for.
  9. 9. Replicable method: poison 5% of Dolci SFT and DPO prompts (13,680 DPO pairs) with a GPT-4.1-mini-generated trait, build difference-in-means vectors at layer 16, and check whether each intervention returns trait rates to clean baselines.
  10. 10. Reward-shaping amplification costs capability roughly linearly in λ for the global trait 'playful' (Overall −10.7pp, ZebraLogic −25.2pp at λ=32), whereas topic-gating the 'poetic' trait to ~3% of creative-writing pairs kept every sub-task within ±3pp.

Peer brief — for seminar discussion

This work builds a data-centric auditing pipeline for preference optimization and tests whether it can both predict and steer what DPO teaches. On Dolci (259,842 usable pairs), a BatchTopK SAE on Llama-3.1-8B yields feature clusters (814 co-firing clusters via Leiden on mutual information). Two views generate hypotheses. The feature-conditioned view clusters examples by symmetric response activity into 512 data clusters and tests chosen-minus-rejected disparity inside versus outside each cluster, using Welch statistics and split-half sign consistency. The prompt-conditioned view tests which response-delta clusters shift for prompts expressing a given prompt-feature cluster. The authors then unify data filtering, inoculation prompting, activation steering and reward shaping as 'explaining away' a concept from an exponential tilt of the policy. The load-bearing finding is that the pipeline's predicted feature shifts track real DPO-induced shifts (R²=0.9 for the feature view, 0.58 for the prompt view), and that it surfaces genuine harms. DPO on Dolci lowers refusal on harmful prompts: Llama-3.1-8B goes from 15.7% to 21.7% harmful responses, and Tulu-3-70B from 7.4% to 19.7%. Reward shaping repairs this across 7B to 70B, for example 21.7% to 4.0% on Llama with OLMES accuracy 0.425 against DPO's 0.434, and acts as a tunable knob for amplification (λ-sweeps for playful and poetic traits). The implication is that a scalar reward can be decomposed into concept-level terms that practitioners inspect and edit before training. The paper also predicts that failures stem from concept entanglement and data sparsity rather than from the interventions themselves. Where the independence assumption of Eq. 3 breaks, as with correlated formatting styles, interventions leak off-target. What's contestable: the validation is largely self-referential. Hypotheses come from SAE clusters and are checked by auto-interpreted labels, GPT-judged traits and SAE-derived rollout features, so shared SAE artifacts could inflate R²=0.9. Reward shaping also uses the same SFT-derived refusal projection it is evaluated against. A critical reader would also note that the 'sparse data' explanation for the negative results is post hoc: only URL hallucination (412 to 354 with a regex filter) improved significantly, and physics sycophancy moved just 2.18 to 1.80. An alternative to the SAE cluster approach would be training-data attribution, such as probe-based or concept-influence attribution, or LLM-written hypothesis descriptions in the style of Movva et al., but no head-to-head comparison is run. Finally, everything rests on one data family (Dolci/Tulu) and DPO, so generalization to RL or reasoning post-training is asserted, not tested.

Methods (16)

Frameworks (3)

Datasets (6)

Original abstract (expand)

Language-model post-training is the main stage at which model behavior is shaped, yet it still largely involves optimization of scalar rewards that summarize diverse desiderata. This abstraction gives practitioners little visibility into what their data actually teaches models, allowing spurious correlations to be learned by a model and inducing undesirable behaviors such as over-stylization and sycophancy. To address this problem, we ask: can we inspect a preference dataset before optimization and decide, at the level of concepts, which behaviors a model should be allowed to learn? Motivated by this, we introduce a data-centric post-training pipeline that uses interpretability protocols to develop statistical hypotheses for the latent concepts separating preferred from dispreferred generations, making them explicit for fine-grained user feedback. Building on this view, we unify several interpretability-based training protocols as ways of shaping rewards via feature or data interventions. Empirically, we show that our pipeline diagnoses undesirable signals in existing preference data, mitigates off-target learning, and can also help amplify or shape desired properties such as safeguards and model personality. More broadly, our results suggest that interpretability can turn post-training from optimizing opaque proxy rewards into a process of auditing and sculpting the learning signal itself.

Related work— refs + corpus + external arXiv

Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.

Similar preprints — Semantic Scholar

Cited by (1)