claim

active

claim:keeping-functional-token-vocabulary-compact-minimizes-perturbation-to-base-model-token-distribution

Keeping functional-token vocabulary compact minimizes perturbation to base model token distribution

ATLAS design philosophy: five functional tokens suffice to abstract common visual operations without excessive disruption.

Source paper

extracted_from

ATLAS: Agentic or Latent Visual Reasoning? One Word is Enough for Both

Ziyu Guo · Rain Liu · Xinyan Chen · Pheng-Ann Heng

Neighborhood — ranked by edge-count

Communities (3)

community

Mechanistic interpretability & model evaluation
members_of
Spans attention head decomposition, benchmark awareness, and genomic pathogenicity prediction via neural models.
Mechanistic structure of transformer attention computations
members_of
Identifies distributed algorithms implemented across attention heads, with focus on causal masking limitations and emergent capabilities via activation manifold steering.
Functional tokens for emergent model reasoning
members_of
Unsupervised learning of interpretable task tokens through gradient flow and vocabulary constraints, enabling reasoning without visual supervision.

Related by similarity (8)

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

Each functional token is associated with an internalized visual operation, yet requires no visual supervision and remains a standard token in the tokenizer vocabulary.claim0.756
Describes the properties of the functional token.
Token-level supervision enables models to learn functional-token invocation from reasoning contextclaim0.747
ATLAS author's assertion that functional tokens optimized via standard cross-entropy loss learn when and how to invoke operations from surrounding text.
A pair of query and key subcomponents distributed across attention heads performs previous-token behaviorfinding0.742
VPD recovers an attention algorithm for attending to the previous token, distributed across multiple heads.
Some attention heads partially specialize in copying for words that split into two tokens without a space prefix, attending from fragmented token to complete tokenfinding0.732
Interesting special case of copying behavior related to tokenization artifacts; primitive precursor to induction heads
Five functional tokens can generalize across 40+ diverse visual reasoning taskshypothesis0.730
ATLAS hypothesis that a compact set of high-level functional tokens (Manip, Shape, Line, Arrow, Text) suffices for multi-domain visual reasoning.
Input embedding similarity baseline selects semantically related but non-reflective tokens (e.g., Await, ConfigureAwait, Unchecked) that fail to improve accuracyfinding0.727
Demonstrates the failure mode of surface-level similarity for instruction discovery.
Up to 33.6% reasoning tokens saved on MMLU subsets with stepwise steering while maintaining accuracy in larger modelsfinding0.722
Maximum token savings achieved by ReflCtrl on non-mathematical general reasoning tasks
The likelihood of a dedicated feature for a concept (element, city, animal, food) follows a sigmoid in log-frequency of the concept in training data, with threshold frequency inversely proportional to number of alive features.finding0.722
Quantitative relationship between concept frequency and feature presence.