paper:arxiv-2505-18651On the emergence of linear analogies in word embeddings
Original abstract (expand)
Models such as Word2Vec and GloVe construct word embeddings based on the co-occurrence probability $P(i,j)$ of words $i$ and $j$ in text corpora. The resulting vectors $W_i$ not only group semantically similar words but also exhibit a striking linear analogy structure -- for example, $W_{\text{king}} - W_{\text{man}} + W_{\text{woman}} \approx W_{\text{queen}}$ -- whose theoretical origin remains unclear. Previous observations indicate that this analogy structure: (i) already emerges in the top eigenvectors of the matrix $M(i,j) = P(i,j)/P(i)P(j)$, (ii) strengthens and then saturates as more eigenvectors of $M (i, j)$, which controls the dimension of the embeddings, are included, (iii) is enhanced when using $\log M(i,j)$ rather than $M(i,j)$, and (iv) persists even when all word pairs involved in a specific analogy relation (e.g., king-queen, man-woman) are removed from the corpus. To explain these phenomena, we introduce a theoretical generative model in which words are defined by binary semantic attributes, and co-occurrence probabilities are derived from attribute-based interactions. This model analytically reproduces the emergence of linear analogy structure and naturally accounts for properties (i)-(iv). It can be viewed as giving fine-grained resolution into the role of each additional embedding dimension. It is robust to various forms of noise and agrees well with co-occurrence statistics measured on Wikipedia and the analogy benchmark introduced by Mikolov et al.
Related work— refs + corpus + external arXiv
Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.
- Probing for Semantic Classes: Diagnosing the Meaning Content of Word EmbeddingsKatharina Kann, Timothy J. Hazen, Eneko Agirre and Hinrich Sch\"utze Yadollah Yaghoobzadeh2019≈ 78%
- The Birth of Knowledge: Emergent Features across Time, Space, and Scale in Large Language ModelsMicah Adler, Nir Shavit Shashata Sawmya2025≈ 75%
- Mechanistic Decomposition of Sentence RepresentationsVikram Natarajan, Jonathan Michala, Milton Lin, Juri Opitz Matthieu Tehenan2025≈ 73%
- Discovering and Reasoning of Causality in the Hidden World with Large Language ModelsYongqiang Chen, Tongliang Liu, Mingming Gong, James Cheng, Bo Han, Kun Zhang Chenxi Liu2025≈ 73%
- ≈ 73%
- Social Meaning in Large Language Models: Structure, Magnitude, and Pragmatic PromptingRoland M\"uhlenbernd2026≈ 73%
- Mechanistic Interpretability of Cognitive Complexity in LLMs via Linear Probing using Bloom's TaxonomyBianca Raimondi and Maurizio Gabbrielli2026≈ 73%
- Patches of Nonlinearity: Instruction Vectors in Large Language ModelsJonas Rohweder, Subhabrata Dutta, Iryna Gurevych Irina Bigoulaeva2026≈ 73%
- ≈ 73%
- ≈ 73%
- Inferring the Future by Imagining the PastTony Chen, Tzu-Mao Li, Jonathan Ragan-Kelley, Josh Tenenbaum Kartik Chandra2023≈ 72%
- Mechanistic Interpretability of Code Correctness in LLMs via Sparse AutoencodersKriz Tahimic and Charibeth Cheng2025≈ 72%
- Perceptions to Beliefs: Exploring Precursory Inferences for Theory of Mind in Large Language ModelsDongkwan Kim, Jiho Jin, Jiseon Kim, Yeon Seonwoo, Yejin Choi, Alice Oh, Hyunwoo Kim Chani Jung2024≈ 72%
- An Evaluation on Large Language Model Outputs: Discourse and MemorizationXun Wang, Alex Sokolov, Qilong Gu and Si-Qing Chen Adrian de Wynter2026≈ 72%
- ≈ 72%
- A Unified Theory of Sparse Dictionary Learning in Mechanistic Interpretability: Piecewise Biconvexity and Spurious MinimaHarshvardhan Saini, Zhaoqian Yao, Zheng Lin, Yizhen Liao, Jingyi Cui, Yisen Wang, Mengnan Du, Dianbo Liu Yiming Tang2026≈ 72%
- The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetsin corpus2023≈ 71%
- ≈ 71%
- ≈ 71%
- Emergence and Causality in Complex Systems: A Survey on Causal Emergence and Related Quantitative Studiesin corpus2023≈ 71%
- Testing the Limits of Truth Directions in LLMsin corpus2026≈ 70%
- ≈ 70%
- The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability?in corpus2025≈ 70%
- ≈ 70%
- ≈ 69%
- The World Inside Neural Networksin corpus2026≈ 69%
- From Directions to Cones: Exploring Multidimensional Representations of Propositional Facts in LLMsin corpus2025≈ 69%
- ≈ 69%
Similar preprints — Semantic Scholar
Cited by (3)
- Do Sparse Autoencoders Capture Concept Manifolds?
Sparse autoencoders trained on Llama3.1-8B (layer 19, 500M Pile tokens, 25 SAEs across Standard, JumpReLU, TopK, BatchTopK and Matryoshka architectures at VE > 0.85) do not capture continuous concept
- Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior
Manifold steering — intervening on model activations along paths constrained to lie on a learned activation manifold M_h rather than along Euclidean linear directions — produces behavioral trajectorie
- Arithmetic in the Wild: Llama uses Base-10 Addition to Reason About Cyclic Concepts
Llama-3.1-8B solves cyclic arithmetic (e.g., "what month is six months after August?") not by performing modular addition in the period of the cyclic concept (12 for months, 7 for days of the week) as