paper
active
2025
paper:arxiv-2511-01836

Priors in Time: Missing Inductive Biases for Language Model Interpretability

TL;DR

Sparse autoencoders carry a hidden i.i.d.-over-time prior that conflicts with how language model activations behave, and replacing it with a temporal prior recovers structure SAEs miss. Rewriting the SAE objective as MAP estimation shows the sparsity penalties of ReLU, TopK, BatchTopK and JumpReLU SAEs are additive over tokens (Prop. 4.1). They therefore assume concepts are independent across time and that sparsity is time-invariant. Gemma-2-2B and Llama-3.1-8B activations contradict this. Intrinsic dimension, measured with a Gram-matrix U-statistic, grows with sequence position, while a permuted stationary surrogate stays flat. Up to 80% of a token's variance is explained by a 500-token context. Temporal Feature Analysis (TFA) splits each activation into a predictable component, projected onto past context through attention, and a sparse novel residual assumed i.i.d. At Gemma-2-2B layer 12 it matches SAEs on NMSE (0.139 vs 0.139 for BatchTopK on web text). Its predictive code recovers event boundaries in 50 GPT-5 stories, keeps garden-path parses stable, and recovers the graph geometry in the Park et al. in-context graph task. It supports counterfactual edits on 4/9 nodes, where no SAE succeeds. On Ultrachat it also lowers tortuosity from about 25–30 to 2.0. Interpretability tools need inductive biases matched to the data's temporal generative structure, and features are better treated as evolving low-dimensional manifolds than as isolated directions.

What to take away

  1. 1. SAE objectives for ReLU, JumpReLU, TopK and BatchTopK have additive sparsity penalties, so their MAP prior factorizes as i.i.d. across tokens (Prop. 4.1) and implies time-invariant L0 sparsity (Cor. 4.1.1).
  2. 2. Intrinsic dimension, estimated with a Gram-matrix U-statistic (an effective-rank estimator), rises steadily with sequence position in Gemma-2-2B and Llama-3.1-8B activations, while a within-sequence permutation surrogate stays nearly flat.
  3. 3. Projecting token x_t onto the span of the preceding 500 tokens' activations explains up to 80% of its variance, and the position-dependent autocorrelation heatmaps differ clearly from a diagonal-averaged stationary surrogate.
  4. 4. Prop. 4.2 shows that if the SAE sparsity budget K is at most the local manifold dimension m(x), low reconstruction error forces support switching among nearby points, which offers a geometric account of feature splitting.
  5. 5. Temporal Feature Analysis splits x_t into an attention-projected predictable part plus a sparse novel residual with an i.i.d. prior, and on Gemma-2-2B layer 12 it matches SAEs on reconstruction (story NMSE 0.139 vs 0.152 BatchTopK, variance explained 0.73 vs 0.72).
  6. 6. Predictive-code CKA similarity to the slow Fourier component of activations is 0.75 versus 0.35–0.37 for ReLU/TopK/BatchTopK SAEs, while novel codes and SAEs align with the fast component (0.54–0.75).
  7. 7. On 50 GPT-5-generated stories with labeled event boundaries, only the predictive code shows high within-event versus across-event cosine similarity (0.56 vs −0.04), compared with 0.29 vs 0.20 for ReLU and about 0.13 vs 0.02 or less for TopK/BatchTopK.
  8. 8. In the Park et al. 3×3 grid in-context graph task, only the predictive code shows the graph geometry in PCA and supports counterfactual belief edits on 4/9 nodes, whereas no SAE code does.
  9. 9. Methodology another lab can replicate: compute tortuosity of per-conversation UMAP trajectories, which on Gemma-2-2B-IT Ultrachat gives about 2.0 for predictive codes versus 25–30 for SAEs and novel codes, even though all tools were trained on the base model.
  10. 10. Open question: predictive-code dominance fails at layer 26 of Llama (about 81% depth), where the novel part takes over, which the authors speculate means SAE priors may fit deep layers better than middle or early ones.
  11. The training dynamics have four phases (novel-dominated, chaotic, predictive-dominated, and sometimes novel collapse), which the authors fixed on Llama by switching from TopK to BatchTopK.

Peer brief — for seminar discussion

The paper argues that sparse autoencoders carry an unexamined temporal prior, and it tests that argument on Gemma-2-2B and Llama-3.1-8B. The theory comes first. Writing the SAE objective as MAP estimation, the authors show that the additive sparsity penalties of ReLU, TopK, BatchTopK and JumpReLU SAEs make the latent prior i.i.d. over tokens, so concepts are independent across time and L0 is time-invariant. A second result (Prop. 4.2) says that when the sparsity budget K is below the local manifold dimension, support must switch among nearby points. That offers a geometric account of feature splitting. The empirical section then argues these assumptions fail. A Gram-matrix U-statistic estimate of intrinsic dimension grows with position, while a permutation surrogate stays flat. Up to 80% of a token's variance is explained by a 500-token context. The method introduced is Temporal Feature Analysis (TFA). It decomposes x_t into a predictable part, projected onto past context by a learned single-layer attention module, and a sparse novel residual assumed i.i.d. It matches SAEs on NMSE (0.139 vs 0.152 for BatchTopK on stories, using 1B Pile tokens at Gemma layer 12). The load-bearing finding is that only the predictive code carries slow temporal structure. It tracks event boundaries in 50 GPT-5 stories (within/across cosine similarity 0.56 vs −0.04). It stays invariant across garden-path and control sentences. It reveals the Park et al. in-context graph and allows counterfactual edits on 4/9 nodes. On Ultrachat, tortuosity falls to about 2.0 versus 25–30 for SAEs. The paper implies that features should be seen as evolving low-dimensional manifolds, and that interpretability tools need inductive biases matched to the data. It tests one concrete prediction: that predictive codes align with slow, context-driven signal and novel codes with fast signal. A critical reader would push back on the evidence. The comparisons are almost entirely qualitative or small-n (50 stories, 50 sentence pairs, one graph task), and success is judged by the very metrics the method was built to favor, such as slow/fast spectral splits and cosine block structure. A cleaner test would use a different temporal objective, such as an SAE with a temporal-smoothness penalty or slow feature analysis, as the comparison instead of only i.i.d. SAEs. The 4/9 node intervention rate is modest, no causal evaluation of the novel codes is run, and the Llama replication uses unfinished SAEs. The attention module may also supply capacity that SAEs lack. Failure at deeper layers further narrows the claim.

Methods (14)

Frameworks (9)

Datasets (7)

Findings (27)

Original abstract (expand)

Recovering meaningful concepts from language model activations is a central aim of interpretability. While existing feature extraction methods aim to identify concepts that are independent directions, it is unclear if this assumption can capture the rich temporal structure of language. Specifically, via a Bayesian lens, we demonstrate that Sparse Autoencoders (SAEs) impose priors that assume independence of concepts across time, implying stationarity. Meanwhile, language model representations exhibit rich temporal dynamics, including systematic growth in conceptual dimensionality, context-dependent correlations, and pronounced non-stationarity, in direct conflict with the priors of SAEs. Taking inspiration from computational neuroscience, we introduce a new interpretability objective -- Temporal Feature Analysis -- which possesses a temporal inductive bias to decompose representations at a given time into two parts: a predictable component, which can be inferred from the context, and a residual component, which captures novel information unexplained by the context. Temporal Feature Analyzers correctly parse garden path sentences, identify event boundaries, and more broadly delineate abstract, slow-moving information from novel, fast-moving information, while existing SAEs show significant pitfalls in all the above tasks. Overall, our results underscore the need for inductive biases that match the data in designing robust interpretability tools.

Related work— refs + corpus + external arXiv

Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.

Similar preprints — Semantic Scholar

Cited by (5)