claim

active

claim:what-appears-to-be-a-representation-of-lexical-entailment-in-bert-is-actually-a-data-structure-of-two-word-identity-representations-not-an-encoding-of-the-entailment-relation

What appears to be a representation of lexical entailment in BERT is actually a data structure of two word identity representations, not an encoding of the entailment relation

Key asymmetry between hierarchical equality and NLI experiments; BERT stores identities rather than the abstract relation.

Source paper

extracted_from

Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations

(2023) · Atticus Geiger · Zhengxuan Wu · Christopher Potts · Thomas Icard +1

Neighborhood — ranked by edge-count

Papers (1)

paper

Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations
introduces

Findings (1)

finding

Lexical entailment representation decomposes into word identity sub-representations with ~0.97-0.98 IIA (Lexeme Subspace of Lexical Entailment)
associated_withsupports
In contrast to hierarchical equality, lexical entailment in BERT decomposes into representations of word identities, not a single abstract relation.

Claims (2)

claim

The feed-forward network truly implements a symbolic, tree-structured algorithm for hierarchical equality, with abstract equality relations not decomposable into input identities
contradicts
DAS reveals that the neural network encodes abstract relational structure rather than raw input identities.
Investigating the causal substructure of neural representations is necessary to avoid misidentifying data structures of simpler representations as abstract concepts
extends
Motivated by the finding that lexical entailment decomposes into word identities.

Related by similarity (8)

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

Can the distributed representation of lexical entailment be decomposed into representations of the individual word identities?question0.834
Research question leading to the key NLI finding about word identity data structures.
Lexical Entailmentconcept0.758
The semantic relation between words wp and wh (entails/neutral) used as an intermediate variable in the MoNLI high-level model.
A small group of causally-implicated hidden states encodes LLM truth representations, localized over clause-ending punctuation tokensclaim0.751
Localization result from patching experiments; identifies group (b) hidden states as the locus of truth representations
The underlying truth representation may generalize across lexical choices and languageshypothesis0.751
Suggested by non-English Yes/No outputs post-intervention, requiring further investigation
what is the precise characterization of the perceptual representational format in which information exists as a specific, sustained, globally constitutive structure?question0.749
Open theoretical problem CIMC acknowledges: precisely characterizing the representational format of perception
Embedment has a hierarchical structure, making use of the position of the left-hand margin as a way to organize relations among elements of an argument.claim0.747
Defines embedment as a spatially hierarchical technique.
"Expressing delegation as a pair of β-transformations on keys in an associative memory emphasises a fundamental symmetry of the delegation mechanism that is ignored by most object-oriented programming languages."concept0.745
It's tricky, because for a typical language model the entity is sort of tricameral: the base simulator, the simulated simulator, and the simulated awareness.quote0.745
Antra's earlier definitive statement of the tricameral model.