finding
active
finding:in-vit-last-layer-top-10-k-fac-eigenvectors-have-over-2x-activation-on-clean-vs-memorized-data-svd-shows-no-patternIn ViT last layer, top 10% K-FAC eigenvectors have over 2x activation on clean vs memorized data; SVD shows no pattern
Vision disentanglement.
Source paper
extracted_from(2025) · Merullo, Jack · Vatsavaya, Srihita · Bushnaq, Lucius · Lewis, Owen
Neighborhood — ranked by edge-count
Papers (1)
paper
Hypotheses (1)
hypothesis
- Section 4 hypothesis.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Disentanglement evidence.
- Regularization effect.
- K-FAC eigenvector bands show sorted activation strength top10%>10-25%>25-50%>bottom50%, unlike SVDfinding0.830Supports curvature basis interpretability.
- K-FAC edit on ViT-Base drops memorized train accuracy from 81.6% to 3.5%, GT accuracy to 66.5% (SVD 58.9%)finding0.785Table 2.
- Basis for editing.
- Better transfer to unseen memorized data.
- K-FAC edit reduces OLMo-7B Dolma strict accuracy from 99.9% to 3.4% (BSN 6.0%, SVD 3.0%)finding0.756Table 1.
- Result from applying the Frobenius norm composition measurement to all attention head pairs in the two-layer model