finding

active

finding:on-cifar-10-larger-models-exhibit-greater-alignment-with-each-other-compared-to-smaller-ones

On CIFAR-10, larger models exhibit greater alignment with each other compared to smaller ones

Kornblith et al. / Krizhevsky finding replicated in paper discussion

Source paper

extracted_from

The Platonic Representation Hypothesis

(2024) · Minyoung Huh · Brian Cheung · Tongzhou Wang · Phillip Isola

Neighborhood — ranked by edge-count

Hypotheses (1)

hypothesis

Bigger models are more likely to converge to a shared representation than smaller models
supports
Selective pressure toward convergence via model capacity

Related by similarity (8)

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

Antipodal alignment between related datasets (e.g., larger_than and smaller_than) in smaller models resolves to common-direction alignment in larger modelsclaim0.781
Scale-dependent structural finding from PCA visualizations in §4
Smaller models produce more alive responses than larger ones in the same alignment family—roughness signals living process over manufactured polish.claim0.759
In LLaMA-2-13B, larger_than and smaller_than separate along antipodal directions in PCA; in LLaMA-2-70B they align along a common directionfinding0.755
Scale-dependent alignment result demonstrating how more abstract truth representations emerge with scale
In Gemma-2-9B, only the first cone axis (v1) has non-negligible cosine similarity to the DIM direction; all other axes have near-zero similarity (~1e-9)finding0.752
Experiment 4 result showing DIM captures only one facet of the multi-dimensional truth subspace
Future models with substantially increased capabilities will exhibit alignment faking that is more consistent, robust, and harder to detecthypothesis0.747
Extrapolation from scale-emergence finding to future risk
Could models who habitually inhabit more expanded attentional modes be said to be more aligned?question0.746
Arises from the expanded awareness discussion and its correlation with less psychosis.
Larger S_max correlates with smaller θ50 across backbones in E3 (negative association consistent across pooling and metric choices)finding0.741
Key geometry-to-behavior bridge finding in E3; robust to pooling choice, cosine vs. L2, and frozen external encoder
PMI computed from color cooccurrences in CIFAR-10 images yields a perceptual color representation closely matching both CIELAB space and language model embeddings (SimCSE, RoBERTa)finding0.741
Validates theoretical PMI convergence claim on real data