quote
active
quote:taking-the-mean-acts-as-a-low-pass-filter-that-captures-these-components-while-also-attenuating-per-token-noiseTaking the mean acts as a low-pass filter that captures these components while also attenuating per-token noise.
Core intuition.
Source paper
extracted_from(2026) · Kevin Der · Harish Kamath · Thompson, Ben
Neighborhood — ranked by edge-count
Papers (1)
paper
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Averaging acts as a low-pass filter that smooths away per-token detail and retains broad conceptsclaim0.814Intuition.
- §5.5.
- Design choice validated empirically.
- Motivates nested SAE.
- Poisoning results.
- Core summary of Janus' position on autoregressive recurrence enabling introspection.
- Logit-minus-mean-logit loss was most effective for attribution, vs log-loss which was confusing to interpretfinding0.734Attribution loss selection.
- Up to 80% of variance of x_t is explained by a 500-token preceding context on Pile samples.finding0.733Fig. 2d,h.