finding

active

finding:spearman-s-rank-correlation-among-different-alignment-metrics-cka-svcca-mutual-k-nn-cknna-over-78-vision-models-is-high-across-variants-with-all-p-values-below-2-24-10-105

Spearman's rank correlation among different alignment metrics (CKA, SVCCA, Mutual k-NN, CKNNA) over 78 vision models is high across variants, with all p-values below 2.24×10^-105

Validates robustness of alignment metric choice

Source paper

extracted_from

The Platonic Representation Hypothesis

(2024) · Minyoung Huh · Brian Cheung · Tongzhou Wang · Phillip Isola

Neighborhood — ranked by edge-count

Methods (1)

method

Mutual k-Nearest Neighbor Alignment Metric
supports
Primary alignment metric used in experiments; measures mean intersection of k-nearest neighbor sets between two kernels

Related by similarity (8)

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

Spearman Rank Correlationmethod0.794
Used to compare RDMs in RSA computations; noted to have sensitivity issues with differing relative extrema in embedding layers.
CKA shows a very weak trend of alignment between models even within modality, compared to mutual k-NN which shows stronger trendsfinding0.786
Explains why mutual k-NN was chosen over CKA as primary metric
Claude achieves significantly higher Spearman correlation predicting feature activations vs neuron activationsfinding0.780
Automated interpretability analysis of activations confirms features are more interpretable than neurons
As number of nearest neighbors k decreases in CKNNA metric, cross-modal alignment trend becomes more pronounced across both models and tasksfinding0.778
Shows cross-modal alignment is primarily local rather than global
In Gemma-2-9B, only the first cone axis (v1) has non-negligible cosine similarity to the DIM direction; all other axes have near-zero similarity (~1e-9)finding0.778
Experiment 4 result showing DIM captures only one facet of the multi-dimensional truth subspace
82% of features in 1M SAE had maximum Pearson correlation ≤0.3 with any MLP neuron, and manual inspection showed no semantic resemblance.finding0.777
SAE features are not simply mirroring individual neurons.
Cross-modal language-vision alignment reaches a maximum of approximately 0.16 on mutual nearest-neighbor metric in Figure 3, well below the theoretical maximum of 1finding0.775
Quantitative bound on observed alignment; raises the open question of whether this gap reflects noise or real misalignment
Better LLMs (measured by 1-bits-per-byte on OpenWebText) show a linear relationship with alignment to vision models measured via mutual nearest-neighbor on WITfinding0.775
Key cross-modal alignment result