thinker:orcid-0000-0003-4944-0632Raphaël Sarfati
Authored papers (7)
- Anatomy of Post-Training: Using Interpretability to Characterize Data and Shape the Learning Signal2026
Interpretability can audit a preference dataset before optimization and predict what DPO will teach a model: on Dolci, clustering SAE features and testing chosen-minus-rejected activation differences predicted post-DPO feature shifts with R²=0.9 (feature-conditioned) and R²=0.58 (prompt-conditioned). The method is a two-view hypothesis-generation pipeline. A BatchTopK SAE on Llama-3.1-8B supplies the feature clusters (814 response-feature clusters, 512 data clusters over 259,842 pairs), and Welch tests with split-half sign-consistency filtering rank which concepts the labels reward. The pipeline flags Dolci DPO as teaching compliance with unsafe queries, and benchmarks such as HarmBench and XSTest confirm degraded safeguards after DPO. It also flags physics sycophancy, questionable fanfiction, benchmark recognition and hallucinated URLs. Four interventions (data filtering, inoculation prompting, activation steering, reward shaping) are unified as 'explaining away' a concept from an exponential tilt of the policy. Reward shaping is the strongest: at λ=−5 it cuts Llama-3.1-8B harmful-response rate from 21.7% to 4.0% while keeping OLMES accuracy near DPO's 0.434 (0.425), and it also amplifies traits such as playfulness. Interventions fail when the target concept is entangled with others or sparse in the data: style attributes leak off-target, and only URL hallucination improved significantly. The paper argues that post-training should treat the dataset as something to audit and the reward as something to sculpt, and that future protocols need concept hierarchies rather than flat, independent concept axes.
Llama-3.1-8B solves cyclic arithmetic (e.g., "what month is six months after August?") not by performing modular addition in the period of the cyclic concept (12 for months, 7 for days of the week) as its circularly structured representations might suggest, but by reusing a single generic base-10 addition mechanism across all such tasks. The mechanism operates in two stages: first, the model computes the raw integer sum of its inputs (six + August = 14), then maps that sum back into cyclic concept space (14 → February). The paper introduces a combined causal-abstraction and feature-geometry analysis to demonstrate that Llama-3.1-8B uses task-agnostic Fourier features with periods characteristic of base-10 arithmetic—specifically periods 2, 5, and 10—rather than the concept-native period. Strikingly, a sparse set of only 28 MLP neurons at layer 18 (approximately 0.2% of that layer's MLP) is responsible for this computation across all cyclic tasks and can be partitioned into disjoint clusters, each computing the sum for a Fourier feature of a distinct period. This implies that structured representational geometry does not entail correspondingly structured computation: a model can build domain-specific circular embeddings and yet delegate the actual arithmetic to a domain-general, base-10 calculator reused opportunistically, suggesting that mechanistic interpretability analyses that read off algorithms from representation geometry alone will systematically mislead.
Manifold steering — intervening on model activations along paths constrained to lie on a learned activation manifold M_h rather than along Euclidean linear directions — produces behavioral trajectories that track the corresponding behavior manifold M_y, while linear (Euclidean) steering cuts through off-manifold regions and generates unnatural outputs. The paper fits M_h to internal representations and M_y to output probability distributions, then tests their bidirectional correspondence via controlled interventions across language models and a video world model. In language models, tasks with cyclic geometries, sequential geometries, and complex graph geometries (in-context learning) all show that manifold-constrained interventions keep behavioral outputs on M_y, whereas linear steering deviates measurably. In a video world model, interventions shaped by physical-dynamics geometry similarly respect M_y. Crucially, the relationship is bidirectional: optimizing interventions in activation space to produce paths along M_y recovers activation trajectories that trace the curvature of M_h. This implies that the core problem of mechanistic steering should be recast not as finding the right direction in a flat Euclidean activation space, but as identifying the correct geometric structure — because representational geometry is not incidental to model behavior but is causally constitutive of it.
- Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior2026merged
Manifold steering — intervening along activation-space paths defined by the intrinsic geometry of a fitted activation manifold M_h rather than along Euclidean linear directions — produces behavioral trajectories that remain on the behavior manifold M_y, whereas standard linear steering cuts through off-manifold regions and generates unnatural outputs. Across language model reasoning tasks with cyclic, sequential, and graph geometries, and in a video world model whose task geometry corresponds to physical dynamics, the bidirectional relationship M_h ↔ M_y holds: steering that respects M_h tracks M_y, and optimizing interventions in activation space to follow M_y recovers activation trajectories that trace the curvature of M_h. The method fits two manifolds — one to intermediate representations, one to output probability distributions — and then uses geodesic-like paths on M_h as the intervention substrate rather than a single steering vector. This work argues that neural representation geometry is not incidental but is the causal structure organizing model behavior, which implies that the core problem of model steering should be reframed from finding the right direction to finding the right geometry.
Reasoning models generate chains of thought that are frequently performative rather than causally necessary for reaching the correct answer: on MMLU recall questions, activation probes decode the model's final answer from internal representations far earlier in the chain-of-thought than a CoT language monitor can, demonstrating that the visible reasoning tokens are generated after the belief is already settled. Tested across DeepSeek-R1 671B and GPT-OSS 120B using three complementary methods — activation probing, early forced answering, and a CoT monitor — the analysis reveals a task-difficulty gradient: the gap between probe-decodable certainty and monitor-detectable certainty is large for easy MMLU items but narrows substantially for difficult GPQA-Diamond multihop questions, where genuine step-by-step uncertainty persists longer. The paper introduces probe-guided early exit, which terminates generation once probe confidence passes a threshold, cutting token counts by up to 80% on MMLU and 30% on GPQA-Diamond while preserving accuracy. A key finding that complicates a purely dismissive view is that inflection-point behaviors — backtracking and 'aha' moments — appear almost exclusively in traces where probes register large belief shifts, meaning these surface signals do track genuine uncertainty rather than being uniformly theatrical. The paper argues that this dissociation between internal belief and continued CoT generation is systematic and exploitable, positioning attention probing as both a diagnostic instrument for faithfulness research and a practical mechanism for adaptive computation.
Llama-3.1-8B-Instruct's in-context belief updates while reading a story trace smooth trajectories on a low-dimensional manifold. That manifold is shared between behavior and residual activations, and its geometry predicts how steering spills across concepts. Behavioral beliefs are elicited as probability-weighted 0-10 ratings after each sentence for six concepts per domain (Emotions, Genres, Arbitrary), using 2,500 SimpleStories training stories and a 500-story test set. UMAP/PCA fits on max-activating sentences give manifolds M_y (behavior) and M_z (activations). Concept-centroid distance matrices correlate at r=.92 for Emotions and r=.89 for Genres across M_y and M_z, against r=.71 for the Arbitrary control. The inferred hierarchies match exactly between the two manifolds, and the Emotions geometry correlates r=.93 with human valence-arousal ratings. Layer-9 linear probes predict behavior at RMSE .09 (Emotions, Genres) and .11 (Arbitrary), even though the prompt never names the concepts. Steering entanglement, for example sadness steering raising anger, tracks manifold distance (r=.65 Emotions, r=.42 Genres). The paper introduces the conceptual belief space, which lifts Gärdenfors' conceptual spaces to distributions over concepts, so that ICL trajectories become paths through it. Single-layer probe steering is undone by later layers, and a 7-layer span fixes this. The implication is that Bayesian accounts of ICL on freeform text can be grounded in structured conceptual geometry. Steering and interpretability can then use that geometry directly, instead of treating concepts as independent directions.
More papers — OpenAlex / S2
Affiliations (2)
- Harvard University(institute)
- Goodfire(institute)
Co-authors (12)
- Atticus Geiger42 shared
- Jack Merullo40 shared
- Ekdeep Singh Lubana38 shared
- Thomas McGrath38 shared
- Owen Lewis36 shared
- Daniel Wurgaft32 shared
- Usha Bhalla29 shared
- Thomas Fel27 shared
- Eric Bigelow25 shared
- Can Rager23 shared
- Sheridan Feucht23 shared
- Tal Haklay23 shared
Their work is cited by (5)
- Structuring Sparsity: Block-Sparse Featurizers Capture Visual Concept Manifolds5× refs
- Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluations4× refs
- Persona Non Grata: Single-Method Safety Evaluation Is Incomplete for Persona-Imbued LLMs2× refs
- Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention2× refs
- Forking Fast: Efficiently Estimating Uncertainty Dynamics in Text Generation2× refs
Other inbound relations (1)
Recent mentions (4)
- papers-typedbigelow-2026-stories-space.md
- papers-typedfeucht-2026-arithmetic.md
- papers-typed
feucht-goodfire-geometric-calculator-2026.md - papers-typedpaper.md