question

active

question:is-a-mutual-nearest-neighbor-alignment-score-of-0-16-indicative-of-strong-alignment-with-remaining-gap-being-noise-or-does-it-signify-poor-alignment-with-major-differences-left-to-explain

Is a mutual nearest-neighbor alignment score of 0.16 indicative of strong alignment with remaining gap being noise, or does it signify poor alignment with major differences left to explain?

Open question the authors leave unresolved about interpreting the magnitude of their alignment measurements

Source paper

extracted_from

The Platonic Representation Hypothesis

(2024) · Minyoung Huh · Brian Cheung · Tongzhou Wang · Phillip Isola

Neighborhood — ranked by edge-count

Findings (1)

finding

Cross-modal language-vision alignment reaches a maximum of approximately 0.16 on mutual nearest-neighbor metric in Figure 3, well below the theoretical maximum of 1
gates
Quantitative bound on observed alignment; raises the open question of whether this gap reflects noise or real misalignment

Related by similarity (8)

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

As number of nearest neighbors k decreases in CKNNA metric, cross-modal alignment trend becomes more pronounced across both models and tasksfinding0.768
Shows cross-modal alignment is primarily local rather than global
Mutual k-Nearest Neighbor Alignment Metricmethod0.768
Primary alignment metric used in experiments; measures mean intersection of k-nearest neighbor sets between two kernels
Better LLMs (measured by 1-bits-per-byte on OpenWebText) show a linear relationship with alignment to vision models measured via mutual nearest-neighbor on WITfinding0.761
Key cross-modal alignment result
CKA shows a very weak trend of alignment between models even within modality, compared to mutual k-NN which shows stronger trendsfinding0.758
Explains why mutual k-NN was chosen over CKA as primary metric
Alignment type is the only significant predictor of scores (p=0.006); architecture and parameter count do not.finding0.756
Kruskal-Wallis test result: Constitutional AI predicts highest baseline; roleplay/empathy training predict lowest.
The degree of similarities which exist in a structure must correspond exactly to the degree of similarity of the conditions there, and the degrees of differences which exist in a structure must also correspond to the degrees of difference in the conditions there.claim0.755
The profound principle that underlies all living structure; symmetry as the mathematical trace of necessity.
What is the appropriate metric for measuring representational alignment, given active debate on merits and deficiencies of all proposed measures?question0.753
Open methodological question acknowledged as limitation
Spearman's rank correlation among different alignment metrics (CKA, SVCCA, Mutual k-NN, CKNNA) over 78 vision models is high across variants, with all p-values below 2.24×10^-105finding0.752
Validates robustness of alignment metric choice