paper:arxiv-2511-01836Priors in Time: Missing Inductive Biases for Language Model Interpretability
TL;DR
Sparse autoencoders carry a hidden i.i.d.-over-time prior that conflicts with how language model activations behave, and replacing it with a temporal prior recovers structure SAEs miss. Rewriting the SAE objective as MAP estimation shows the sparsity penalties of ReLU, TopK, BatchTopK and JumpReLU SAEs are additive over tokens (Prop. 4.1). They therefore assume concepts are independent across time and that sparsity is time-invariant. Gemma-2-2B and Llama-3.1-8B activations contradict this. Intrinsic dimension, measured with a Gram-matrix U-statistic, grows with sequence position, while a permuted stationary surrogate stays flat. Up to 80% of a token's variance is explained by a 500-token context. Temporal Feature Analysis (TFA) splits each activation into a predictable component, projected onto past context through attention, and a sparse novel residual assumed i.i.d. At Gemma-2-2B layer 12 it matches SAEs on NMSE (0.139 vs 0.139 for BatchTopK on web text). Its predictive code recovers event boundaries in 50 GPT-5 stories, keeps garden-path parses stable, and recovers the graph geometry in the Park et al. in-context graph task. It supports counterfactual edits on 4/9 nodes, where no SAE succeeds. On Ultrachat it also lowers tortuosity from about 25–30 to 2.0. Interpretability tools need inductive biases matched to the data's temporal generative structure, and features are better treated as evolving low-dimensional manifolds than as isolated directions.
What to take away
- 1. SAE objectives for ReLU, JumpReLU, TopK and BatchTopK have additive sparsity penalties, so their MAP prior factorizes as i.i.d. across tokens (Prop. 4.1) and implies time-invariant L0 sparsity (Cor. 4.1.1).
- 2. Intrinsic dimension, estimated with a Gram-matrix U-statistic (an effective-rank estimator), rises steadily with sequence position in Gemma-2-2B and Llama-3.1-8B activations, while a within-sequence permutation surrogate stays nearly flat.
- 3. Projecting token x_t onto the span of the preceding 500 tokens' activations explains up to 80% of its variance, and the position-dependent autocorrelation heatmaps differ clearly from a diagonal-averaged stationary surrogate.
- 4. Prop. 4.2 shows that if the SAE sparsity budget K is at most the local manifold dimension m(x), low reconstruction error forces support switching among nearby points, which offers a geometric account of feature splitting.
- 5. Temporal Feature Analysis splits x_t into an attention-projected predictable part plus a sparse novel residual with an i.i.d. prior, and on Gemma-2-2B layer 12 it matches SAEs on reconstruction (story NMSE 0.139 vs 0.152 BatchTopK, variance explained 0.73 vs 0.72).
- 6. Predictive-code CKA similarity to the slow Fourier component of activations is 0.75 versus 0.35–0.37 for ReLU/TopK/BatchTopK SAEs, while novel codes and SAEs align with the fast component (0.54–0.75).
- 7. On 50 GPT-5-generated stories with labeled event boundaries, only the predictive code shows high within-event versus across-event cosine similarity (0.56 vs −0.04), compared with 0.29 vs 0.20 for ReLU and about 0.13 vs 0.02 or less for TopK/BatchTopK.
- 8. In the Park et al. 3×3 grid in-context graph task, only the predictive code shows the graph geometry in PCA and supports counterfactual belief edits on 4/9 nodes, whereas no SAE code does.
- 9. Methodology another lab can replicate: compute tortuosity of per-conversation UMAP trajectories, which on Gemma-2-2B-IT Ultrachat gives about 2.0 for predictive codes versus 25–30 for SAEs and novel codes, even though all tools were trained on the base model.
- 10. Open question: predictive-code dominance fails at layer 26 of Llama (about 81% depth), where the novel part takes over, which the authors speculate means SAE priors may fit deep layers better than middle or early ones.
- The training dynamics have four phases (novel-dominated, chaotic, predictive-dominated, and sometimes novel collapse), which the authors fixed on Llama by switching from TopK to BatchTopK.
Peer brief — for seminar discussion
The paper argues that sparse autoencoders carry an unexamined temporal prior, and it tests that argument on Gemma-2-2B and Llama-3.1-8B. The theory comes first. Writing the SAE objective as MAP estimation, the authors show that the additive sparsity penalties of ReLU, TopK, BatchTopK and JumpReLU SAEs make the latent prior i.i.d. over tokens, so concepts are independent across time and L0 is time-invariant. A second result (Prop. 4.2) says that when the sparsity budget K is below the local manifold dimension, support must switch among nearby points. That offers a geometric account of feature splitting. The empirical section then argues these assumptions fail. A Gram-matrix U-statistic estimate of intrinsic dimension grows with position, while a permutation surrogate stays flat. Up to 80% of a token's variance is explained by a 500-token context. The method introduced is Temporal Feature Analysis (TFA). It decomposes x_t into a predictable part, projected onto past context by a learned single-layer attention module, and a sparse novel residual assumed i.i.d. It matches SAEs on NMSE (0.139 vs 0.152 for BatchTopK on stories, using 1B Pile tokens at Gemma layer 12). The load-bearing finding is that only the predictive code carries slow temporal structure. It tracks event boundaries in 50 GPT-5 stories (within/across cosine similarity 0.56 vs −0.04). It stays invariant across garden-path and control sentences. It reveals the Park et al. in-context graph and allows counterfactual edits on 4/9 nodes. On Ultrachat, tortuosity falls to about 2.0 versus 25–30 for SAEs. The paper implies that features should be seen as evolving low-dimensional manifolds, and that interpretability tools need inductive biases matched to the data. It tests one concrete prediction: that predictive codes align with slow, context-driven signal and novel codes with fast signal. A critical reader would push back on the evidence. The comparisons are almost entirely qualitative or small-n (50 stories, 50 sentence pairs, one graph task), and success is judged by the very metrics the method was built to favor, such as slow/fast spectral splits and cosine block structure. A cleaner test would use a different temporal objective, such as an SAE with a temporal-smoothness penalty or slow feature analysis, as the comparison instead of only i.i.d. SAEs. The 4/9 node intervention rate is modest, no causal evaluation of the novel codes is run, and the Llama replication uses unfinished SAEs. The attention module may also supply capacity that SAEs lack. Failure at deeper layers further narrows the claim.
Methods (14)
- Autocorrelation analysisCosine similarity at lags to test stationarity.
- Autointerp feature-card pipelineActivation-triggered examples plus direct logit attribution.
- Context projection variance explained analysisProjects token representation onto context subspace.
- Counterfactual graph-node editingReplace last-token code with target node.
- Fourier slow/fast splitSplits activations at critical frequency into equal-energy halves.
- Gaussian noise coarsening analysisAdds noise to reveal multi-scale structure.
- Hierarchical clustering with dendrogramsUsed to test event alignment.
- Phrase-averaged code similarity test (SP/V/OP)Garden-path quantitative test.
- sparse autoencodersExisting method for model interpretability that decodes model activations rather than parameters themselves, noted as incomplete solution.
- Stationary surrogatePermutation/diagonal-mean baseline.
- TortuosityMeasure of trajectory straightness.
- U-statistic intrinsic dimension estimatorCustom estimator of effective rank from Gram matrix.
- UMAPUsed to visualize models in 2D space based on representational distance
- Within vs across event cosine similarityQuantifies event alignment of codes.
Frameworks (9)
- BatchTopK SAEEvaluated architecture.
- Bayesian MAP interpretation of SAE objectiveLens used to expose SAE priors.
- Centered Kernel Alignment (CKA)A second-order correlational similarity method compared against MAS in the paper.
- JumpReLU SAEEvaluated architecture.
- Kalman filtersAnalogy to TFA structure and suggested training objective.
- Linear Representation HypothesisThe hypothesis that models internalize concepts as approximately linear directions in representation space; used to interpret MDS injection behavior
- ReLU SAEBaseline SAE architecture.
- Temporal Feature AnalysisProposed objective decomposing activations into predictable and novel components.
- TopK SAEEvaluated architecture (Gao et al.).
Datasets (7)
- Gemma-2-2B activationsPrimary analyzed model.
- GPT-5 50 garden path sentences with controlsGarden-path evaluation set.
- GPT-5 synthetic 50 stories with event boundariesEvent-boundary evaluation set.
- Llama-3.1-8B activationsReplication model.
- Pile-UncopyrightedTraining data for SAEs/TFA.
- TinyStoriesStory domain.
- UltraChatSource of some animal welfare queries used in evaluation dataset
Findings (27)
- Predictive component's V→SP similarity is invariant across garden path and control (0.47 vs 0.44), while SAEs and novel code vary strongly (e.g., TopK 0.09 vs 0.38).
Fig. 8b,c.
- TFA training shows competition between predictive and novel components across three phases; a fourth failure phase is frequent in Llama and fixed by BatchTopK.
Fig. 12.
- Intrinsic dimension of Gemma-2-2B and Llama-3.1-8B activations increases steadily with sequence position, while a stationary surrogate stays nearly constant.
Fig. 2a,e.
- In a Gemma-2-2B TFA, ~2K dictionary elements define predictive code and ~8K novel code; effective rank is ~20–30 for predictive vs ~390–410 for novel.
Fig. 58.
- Llama-3.1-8B replication shows similar qualitative trends for geometry, event boundaries, noise and in-context graph tasks (SAEs not fully trained).
App. E.
- TFA reaches NMSE comparable to SAEs on Gemma-2-2B (e.g., Story 0.139 vs BatchTopK 0.152; Web 0.139 vs 0.139), while Pred-only is worse (0.34–0.38).
Table 1.
- Predictive codes form hierarchical block structures aligned with sub-event boundaries in stories, while SAEs cluster tokens by lexical identity.
Fig. 5.
- Predictive codes link long-distance heads and dependents for the correct garden-path parse, while standard SAEs emphasize local misleading cues.
Fig. 8a.
- Under Gaussian noise the predictive similarity map coarsens into larger blocks and variance explained decays slower than SAEs, which drop to ~0.
Fig. 7d,e.
- On Ultrachat with Gemma-2-2B-IT, only predictive codes separate speaker roles with a temporal gradient and low tortuosity (τ≈2.0 vs 25–30).
Fig. 10.
Claims (12)
- Predictive component not dominating at deep layers is a feature, since deeper layers discard contextual information needed for unembedding.
Layer choice discussion.
- Temporal Feature Analysis augments rather than replaces SAE tools; novel codes remain compatible with autointerp.
§7.2 assertion.
- If the SAE sparsity budget K is at most the manifold dimension, support switching must occur under low error.
Proposition 4.2.
- Features are better viewed as low-dimensional manifolds evolving over time rather than isolated directions.
Future-work perspective.
- SAEs assume sparsity (number of concepts) is time-invariant, drawn i.i.d. from a fixed distribution.
Corollary 4.1.1.
- SAE priors are in direct conflict with the non-stationary temporal structure of LM activations.
Main thesis.
- Residuals of the novel component are assumed i.i.d. over time, which is still an assumption.
Acknowledged limitation.
- SAEs may have the right inductive biases for deeper layers but not middle or earlier layers.
Speculation about layer dependence.
- Geometrical objects may be the units of computation needed to describe model computation.
Motivates structured protocols.
- Temporal Feature Analysis recovers temporal straightening in an unsupervised manner.
Interpretation of smooth predictive trajectories.
Hypotheses (4)
- If sentence ambiguity is reflected in latent codes, V–SP similarity should be similar for garden path and control sentences while V–OP similarity is higher.
Prediction for garden-path test.
- Scaling causal interventions on codes will move from qualitative alignment to mechanistic claims.
Future work.
- Splitting the loss into predictive-only and novel-residual terms breaks the competition dynamic.
Proposed training fix inspired by Kalman filters.
- Adding noise turns off small latents and amplifies temporal structure if present.
Motivates noise analysis.
Questions (2)
Original abstract (expand)
Recovering meaningful concepts from language model activations is a central aim of interpretability. While existing feature extraction methods aim to identify concepts that are independent directions, it is unclear if this assumption can capture the rich temporal structure of language. Specifically, via a Bayesian lens, we demonstrate that Sparse Autoencoders (SAEs) impose priors that assume independence of concepts across time, implying stationarity. Meanwhile, language model representations exhibit rich temporal dynamics, including systematic growth in conceptual dimensionality, context-dependent correlations, and pronounced non-stationarity, in direct conflict with the priors of SAEs. Taking inspiration from computational neuroscience, we introduce a new interpretability objective -- Temporal Feature Analysis -- which possesses a temporal inductive bias to decompose representations at a given time into two parts: a predictable component, which can be inferred from the context, and a residual component, which captures novel information unexplained by the context. Temporal Feature Analyzers correctly parse garden path sentences, identify event boundaries, and more broadly delineate abstract, slow-moving information from novel, fast-moving information, while existing SAEs show significant pitfalls in all the above tasks. Overall, our results underscore the need for inductive biases that match the data in designing robust interpretability tools.
Related work— refs + corpus + external arXiv
Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.
- A Comparative Analysis of Sparse Autoencoder and Activation Difference in Language Model SteeringJiaqing Xie2025≈ 86%
- Time-Aware Feature Selection: Adaptive Temporal Masking for Stable Sparse Autoencoder TrainingJunyu Ren T. Ed Li2025≈ 86%
- Position: Mechanistic Interpretability Should Prioritize Feature Consistency in SAEsAashiq Muhamed, Yujia Zheng, Lingjing Kong, Zeyu Tang, Mona T. Diab, Virginia Smith, Kun Zhang Xiangchen Song2025≈ 86%
- Dissecting Chronos: Sparse Autoencoders Reveal Causal Feature Hierarchies in Time Series Foundation ModelsAnurag Mishra2026≈ 85%
- Falsifying Sparse Autoencoder Reasoning Features in Language ModelsZhongyuan Liang, Irene Y. Chen, Somayeh Sojoudi George Ma2026≈ 85%
- Beyond Activation Patterns: A Weight-Based Out-of-Context Explanation of Sparse Autoencoder FeaturesZhi-Hong Deng Yiting Liu2026≈ 85%
- SCALAR: Benchmarking SAE Interaction Sparsity in Toy LLMsAndrew Gordon, Peter Lai, Xavier Poncini, David Quarel, and Stefan Heimersheim Sean P. Fillingham2025≈ 85%
- Improving Robustness In Sparse Autoencoders via Masked RegularizationKowshik Thopalli, Bhavya Kailkhura and Wesam Sakla Vivek Narayanaswamy2026≈ 85%
- Revisiting End-To-End Sparse Autoencoder Training: A Short Finetune Is All You NeedAdam Karvonen2025≈ 85%
- Resurrecting the Salmon: Rethinking Mechanistic Interpretability with Domain-Specific Sparse AutoencodersMudith Jayasekara, Max Kirkby Charles O'Neill2025≈ 85%
- CorrSteer: Generation-Time LLM Steering via Correlated Sparse Autoencoder FeaturesZekun Wu, Adriano Koshiyama Seonglae Cho2026≈ 85%
- Sparse Autoencoder Decomposition of Clinical Sequence Model Representations: Feature Complexity, Task Specialisation, and Mortality PredictionFeng Dong, Andreas Karwath Chris Sainsbury2026≈ 85%
- Investigating Sensitive Directions in GPT-2: An Improved Baseline and Comparative Analysis of SAEsDaniel J. Lee and Stefan Heimersheim2024≈ 84%
- Step-Level Sparse Autoencoder for Reasoning Process InterpretationJiayu Liu, Yuhang Lai, Hao Xu, Zhenya Huang, Ning Miao Xuan Yang2026≈ 84%
- Sparsification and Reconstruction from the Perspective of Representation GeometryBingzhe Wu, Zhile Yang, Chengke Wu Wenjie Sun2025≈ 84%
- ≈ 84%
- ≈ 84%
- ≈ 83%
- ≈ 83%
- Mechanistic Knobs in LLMs: Retrieving and Steering High-Order Semantic Features via Sparse Autoencodersin corpus2026≈ 83%
- Do Sparse Autoencoders Capture Concept Manifolds?in corpus2026≈ 83%
- ≈ 83%
- Open Problems in Mechanistic Interpretabilityin corpus2025≈ 83%
- ≈ 82%
- ≈ 82%
- ≈ 82%
- ≈ 82%
- ≈ 82%
- SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretabilityin corpus2025≈ 82%
Similar preprints — Semantic Scholar
Cited by (5)
- Do Sparse Autoencoders Capture Concept Manifolds?
Sparse autoencoders trained on Llama3.1-8B (layer 19, 500M Pile tokens, 25 SAEs across Standard, JumpReLU, TopK, BatchTopK and Matryoshka architectures at VE > 0.85) do not capture continuous concept
- Structuring Sparsity: Block-Sparse Featurizers Capture Visual Concept Manifolds
Lifting the featurizer's atom from a direction to a block of directions recovers concept manifolds that SAEs shatter, and block sparsity is the MAP-matched prior for a representation that is a sparse
- Turn-Averaged SAEs for Feature Discovery and Long-Context Attribution
Training a sparse autoencoder on the mean residual-stream activation of a conversation turn, rather than on individual tokens, yields features that describe a turn's high-level properties more complet
- Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior
Manifold steering — intervening on model activations along paths constrained to lie on a learned activation manifold M_h rather than along Euclidean linear directions — produces behavioral trajectorie
- Arithmetic in the Wild: Llama uses Base-10 Addition to Reason About Cyclic Concepts
Llama-3.1-8B solves cyclic arithmetic (e.g., "what month is six months after August?") not by performing modular addition in the period of the cyclic concept (12 for months, 7 for days of the week) as