finding

active

finding:single-base64-feature-a-0-45-splits-into-three-distinct-features-in-a-1-letter-specific-digit-specific-and-ascii-encoding-specific

Single base64 feature A/0/45 splits into three distinct features in A/1: letter-specific, digit-specific, and ASCII-encoding-specific

Concrete example of feature splitting revealing unexpected model structure

Source paper

extracted_from

Towards Safe and Honest AI Agents with Neural Self-Other Overlap

(2024) · Marc Carauleanu · Michael Vaiana · Judd Rosenblatt · Cameron Berg +1

Neighborhood — ranked by edge-count

Claims (1)

claim

Feature splitting means dictionaries with fewer features provide coarser summaries of model features while larger dictionaries reveal finer-grained distinctions, with no uniquely 'correct' number of features
supports
Authors argue the absence of a fixed feature count is a property of the superposition geometry, not a failure of the method

Concepts (1)

concept

Feature splitting
supports
Phenomenon where a feature in a small SAE splits into multiple finer features in a larger SAE.

Related by similarity (8)

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

Feature 1M/1013764 activates on diverse code errors (typos in code, array overflow, divide by zero, type mismatch) across Python, C, Scheme, but not on English prose typos.finding0.784
Shows a general code error detector beyond simple typo detection.
Base64 feature A/1/2357 and B/1/2165 have activation correlation of 0.85finding0.782
Universality of base64 feature across two transformers
Four features (A/0/20, A/0/0, A/0/30, A/0/494) form an FSA-like system implementing HTML tag generationfinding0.762
Concrete example of features connecting into FSA-like system implementing complex behavior
Feature splitting occurs: smaller SAE features split into multiple finer-grained features in larger SAEs.claim0.760
Observed across SAE scales, e.g., 'San Francisco' split into 11 features.
DNA feature A/1/2937 and B/1/3680 have activation correlation of 0.92finding0.751
Universality of DNA feature across two transformer models with different random seeds
Activating the base64 feature A/1/2357 causes the model to generate base64 textfinding0.748
Causal validation of base64 feature function via pinned feature sampling
In A/4, over 100 features primarily respond to the token 'the' in different contextsfinding0.748
Demonstrates prevalence of token-in-context features and feature splitting of common tokens
In A/4, features functionally memorize 'MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE' via FSA-like feature chainfinding0.745
Demonstrates mechanistic memorization via feature assemblies in superposition