paper:doi-10-1162-tacl-a-00034Linear algebraic structure of word senses, with applications to polysemy
Original abstract (expand)
Word embeddings are ubiquitous in NLP and information retrieval, but it is unclear what they represent when the word is polysemous. Here it is shown that multiple word senses reside in linear superposition within the word embedding and simple sparse coding can recover vectors that approximately capture the senses. The success of our approach, which applies to several embedding methods, is mathematically explained using a variant of the random walk on discourses model (Arora et al., 2016). A novel aspect of our technique is that each extracted word sense is accompanied by one of about 2000 “discourse atoms” that gives a succinct description of which other words co-occur with that word sense. Discourse atoms can be of independent interest, and make the method potentially more useful. Empirical tests are used to verify and support the theory.
Related work— refs + corpus + external arXiv
Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.
- Interpreting Neural Networks through the Polytope LensLee Sharkey, Leo Grinsztajn, Eric Winsor, Dan Braun, Jacob Merizian, Kip Parker, Carlos Ram\'on Guevara, Beren Millidge, Gabriel Alfour, Connor Leahy Sid Black2022≈ 75%
- Probing for Semantic Classes: Diagnosing the Meaning Content of Word EmbeddingsKatharina Kann, Timothy J. Hazen, Eneko Agirre and Hinrich Sch\"utze Yadollah Yaghoobzadeh2019≈ 74%
- General Mechanism of Evolution Shared by Proteins and WordsHsing-Yi Lai, Sun-Ting Tsai, Chen Siang Ng, Kevin Sheng-Kai Ma, Shan-Jyun Wu, Meng-Xue Tsai, Yi-Ching Su, Daw-Wei Wang, and Tzay-Ming Hong Li-Min Wang2026≈ 74%
- Simple Mechanisms for Representing, Indexing and Manipulating ConceptsRaghu Meka, Rina Panigrahy, Kulin Shah Yuanzhi Li2026≈ 73%
- Polychrony as ChinampasJose Antonio Arciniega-Nevarez, Anh Nguyen, Yitong Zou, Luke Van Popering, Nathan Crock, Gordon Erlebacher, Jose L. Mendoza-Cortes Eric Dolores-Cuenca2026≈ 73%
- The Wittgensteinian Representation Hypothesis: Is Language the Attractor of Multimodal Convergence?Run Shao, Dongyue Wu, Jiajie Teng, Chao Tao, Jingdong Chen, Haifeng Li Zhaoyang Zhang2026≈ 73%
- Disentangling Neuron Representations with Concept VectorsVincent Andrearczyk, Henning Muller, Mara Graziani Laura O'Mahony2023≈ 73%
- Towards Interpretable Sequence Continuation: Analyzing Shared Circuits in Large Language ModelsPhilip Torr, Fazl Barez Michael Lan2024≈ 73%
- An Encoding of Abstract Dialectical Frameworks into Higher-Order LogicAlexander Steen Antoine Martina2026≈ 73%
- Morphological Computing as Logic Underlying Cognition in Human, Animal, and Intelligent MachineGordana Dodig-Crnkovic2023≈ 73%
- Why Linear Interpretability Works: Invariant Subspaces as a Result of Architectural ConstraintsYousung Lee, Dongsoo Har Andres Saurez2026≈ 72%
- A Unified Theory of Sparse Dictionary Learning in Mechanistic Interpretability: Piecewise Biconvexity and Spurious MinimaHarshvardhan Saini, Zhaoqian Yao, Zheng Lin, Yizhen Liao, Jingyi Cui, Yisen Wang, Mengnan Du, Dianbo Liu Yiming Tang2026≈ 72%
- The Geometry of Concepts: Sparse Autoencoder Feature StructureEric J. Michaud, David D. Baek, Joshua Engels, Xiaoqing Sun, Max Tegmark Yuxiao Li2025≈ 72%
- Discrete Latent Structure in Neural NetworksCaio F. Corro, Nikita Nangia, Tsvetomila Mihaylova, Andr\'e F. T. Martins Vlad Niculae2026≈ 72%
- Identity as Attractor: Geometric Evidence for Persistent Agent Architecture in LLM Activation SpaceVladimir Vasilenko2026≈ 72%
- ≈ 72%
- The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetsin corpus2023≈ 72%
- Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representationsin corpus2023≈ 70%
- Mechanistic Knobs in LLMs: Retrieving and Steering High-Order Semantic Features via Sparse Autoencodersin corpus2026≈ 69%
- ≈ 69%
- ≈ 69%
- The biogenic approach to cognitionin corpus2005≈ 69%
- Testing the Limits of Truth Directions in LLMsin corpus2026≈ 69%
- Information, Processes and Gamesin corpus≈ 69%
- Learning without neurons in physical systemsin corpus2022≈ 69%
- Model Alignment Searchin corpus2025≈ 69%
- From Directions to Cones: Exploring Multidimensional Representations of Propositional Facts in LLMsin corpus2025≈ 69%
- The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability?in corpus2025≈ 69%
Similar preprints — Semantic Scholar
Cited by (3)
- The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability?
Under arbitrarily powerful alignment maps, causal abstraction becomes vacuous: any neural network can be perfectly mapped to any algorithm, a result proven formally in Theorem 1 under five mild assump
- Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior
Manifold steering — intervening on model activations along paths constrained to lie on a learned activation manifold M_h rather than along Euclidean linear directions — produces behavioral trajectorie
- Arithmetic in the Wild: Llama uses Base-10 Addition to Reason About Cyclic Concepts
Llama-3.1-8B solves cyclic arithmetic (e.g., "what month is six months after August?") not by performing modular addition in the period of the cyclic concept (12 for months, 7 for days of the week) as