finding

active

finding:llama-3-1-8b-internal-representations-for-the-seven-days-of-the-week-form-seven-clusters-in-a-circle-in-activation-space

Llama-3.1 8B internal representations for the seven days of the week form seven clusters in a circle in activation space.

Empirical observation establishing that Llama's internal representations for days-of-week have circular geometric structure.

Source paper

extracted_from

Steering Along Manifolds to Control Neural Networks

Neighborhood — ranked by edge-count

Papers (1)

paper

Steering Along Manifolds to Control Neural Networks
introduces

Claims (1)

claim

The analogous circular structure between representation manifold and behavior manifold is downstream of training data and its implicit cyclic conceptual structure.
supports
The paper's causal explanation for why representation and behavior geometry both appear circular for days of the week.

Related by similarity (8)

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

Llama-3.1 8B output token distributions for seven days of the week form seven clusters in a rough circle in behavior space (Hellinger distance geometry).finding0.888
Empirical observation establishing that Llama's behavior for days-of-week tasks has circular structure.
LLaMA-2-7B representations of larger_than+smaller_than cluster by surface-level characteristics such as presence of token 'eighty'finding0.827
Demonstrates that small models represent surface features rather than abstract truth
Steering Llama-3.1 8B along the circular representation manifold produces outputs that follow the natural circle of the behavior manifold, cleanly shifting probability mass from Monday through successive days.finding0.810
Core empirical result demonstrating that manifold steering produces on-target, behavior-aligned outputs.
Llama-3.1-8B implements a two-stage algorithm: (1) compute integer sum via base-10 addition (e.g., six + August = 14), then (2) map sum to cyclic concept space (14 → February)finding0.804
The complete mechanistic algorithm discovered for cyclic concept reasoning
Llama-3.1-8B representations for cyclic concepts are circularly structuredfinding0.801
The representation geometry finding that motivates the question about whether computation mirrors it
Llama-3.1-8B uses task-agnostic Fourier features with periods 2, 5, and 10 (base-10) rather than concept-specific periods (e.g., 12 for months)finding0.791
The specific Fourier feature periods identified confirm base-10 rather than modular computation
Linear steering on Llama-3.1 8B for the days-of-week task cuts across the behavior manifold, producing noisy off-target effects where predicted tokens are not even days of the week.finding0.790
Empirical result demonstrating the failure mode of linear steering when concept geometry is cyclic.
The representation-based path and the behavior-based path in Llama-3.1 8B activation space trace out similar curves, demonstrating bidirectional geometry alignment.finding0.781
Key empirical result showing that optimizing for behavioral outputs and fitting representation geometry produce the same path in activation space.