Token-in-Context Feature

Feature that fires on a specific token only within a specific surrounding context (e.g., 'the' in physics vs 'the' in mathematics)

Neighborhood — ranked by edge-count

framework

Local vs Compositional Representations
associated_with
Theoretical distinction between representing token-context pairs as individual features (local) vs combining independent features (compositional)

concept

Context Feature
extends
Feature that activates across all tokens within a specific context (e.g., DNA sequences, base64 strings)
Action Features
associated_with
Dual interpretation of features: in addition to responding to inputs, features also act to increase probability of specific output tokens

finding

In A/4, over 100 features primarily respond to the token 'the' in different contexts
supports
Demonstrates prevalence of token-in-context features and feature splitting of common tokens

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

Tokenconcept0.788
Basic unit of LLM input/output: words, parts of words, punctuation marks, emojis
Single-Token Featuresconcept0.750
Features that fire on every instance of a single token; appear in small dictionaries as collapsed versions of many token-in-context features
context windowconcept0.747
Finite number of previous tokens used by autoregressive models to predict the next token; defines interaction range
Token embeddingsconcept0.745
Vector representations of individual tokens from genomic foundation models; the raw inputs to sequence pooling methods.
Out-of-Context Reasoningconcept0.737
Model outputs influenced by information from training documents not present in context; relevant to synthetic document fine-tuning results
Next Token Predictionconcept0.733
The training objective of LLMs: predicting the most likely next token given context; formally P(w_{n+1}|w_1...w_n)
Previous-token attention behaviorconcept0.728
An attention algorithm recovered by VPD where the model attends to the immediately preceding token.
tokenizer vocabularyconcept0.727
The standard set of tokens that the functional token remains a part of.