claim

active

claim:rank-one-matrix-decomposition-constraint-enforcing-mechanistic-simplicity

Rank-one matrix decomposition constraint enforcing mechanistic simplicity

Core design principle of VPD: each parameter subcomponent is constrained to be a simple rank-one matrix to enable isolated understanding and combination.

Source paper

extracted_from

cimcWhitepaper

Neighborhood — ranked by edge-count

Findings (1)

finding

Subcomponent L2.MLP.down:3382 (density 0.00%) predicts emoticon continuations after colon, semicolon, or equals
supports
Specific discovered subcomponent that activates on punctuation like ' :', ' ;', ' =', ':-' and predicts the rest of emoticons/emojis.

Communities (2)

community

Few-shot anchoring & latent structure
members_of
How minimal examples disambiguate and recruit latent arithmetic/reasoning interpretations in LLMs
Mechanistic editing through parameter surgical intervention
members_of
Direct modification of model subcomponents (MLPs, embeddings, unembedding vectors) to predictably alter outputs without retraining, using rank-one constraints.

Methods (1)

method

Adversarial Parameter Decomposition (VPD)
associated_with
Core technique introduced in this paper for decomposing neural network weight matrices into mechanistically simple, interpretable rank-one subcomponents.

Related by similarity (8)

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

Rank-one matrix decompositionmethod0.813
Constraint in VPD where each parameter subcomponent is constrained to be a rank-one matrix for simplicity.
Rank-one matrixconcept0.746
The simple matrix form into which VPD constrains subcomponents to enforce mechanistic simplicity.
Forcing complex solutions to simple problems indicates a language has chosen the wrong abstraction level for its primitives.claim0.723
General principle illustrated by the dining philosophers comparison.
What matrix decomposition or dimensionality reduction best summarizes the enormous low-rank OV and QK matrices?question0.721
Open methodological question about converting the 50k x 50k expanded matrices into human-graspable summaries
Constraint-solving problems are hardest near the constrainedness transition, where few distinct correct routes remainclaim0.703
Interpretive claim linking classical constraint-satisfaction hardness theory to the paper's saddle-based account.
Weight matrix decompositionconcept0.702
The core idea of decomposing weight matrices into components for interpretability.
Certain representation learning algorithms boil down to a simple rule: find an embedding in which similarity equals PMIclaim0.698
Core theoretical claim about the target of representation learning
Autoencoder-like compression forces evolution of general-purpose problem-solving machines with inherent robustnesshypothesis0.696