finding
active
finding:predictive-component-s-attention-maps-are-low-rank-rank-bottleneck-constrained-tfas-achieve-similar-reconstructionPredictive component's attention maps are low-rank; rank-bottleneck constrained TFAs achieve similar reconstruction.
Fig. 13.
Source paper
extracted_from(2025) · Lubana, Ekdeep Singh · Rager, Can · Hindupur, Sai Sumedh R. · Costa, Valerie +12
Neighborhood — ranked by edge-count
Papers (1)
paper
Concepts (1)
concept
- Rank sparsitysupportsAlternative sparsity notion proposed for predictive component.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Layer choice discussion.
- Demonstrated CNN representations predict neurons in visual cortex; background motivation for neural-network-brain correspondence.
- Justifies the methodological choice of attention over concatenation, mean pooling, residual connections, or joint embedding.
- Baseline.
- Fig. 7d,e.
- Contrast with reward model non-identifiability.
- At Llama layer 26 (~81% depth) the predictive component does not dominate the latent norm.finding0.765Fig. 14.
- Learns new full-rank QK circuits shared across feature groups.