finding

active

finding:in-llama-3-70b-each-next-token-prediction-at-token-101-draws-on-64-000-independent-attention-streams-8-heads-80-layers-100-prior-positions-each-carrying-a-128-dimensional-signal

In Llama 3 70B, each next-token prediction at token 101 draws on 64,000 independent attention streams (8 heads × 80 layers × 100 prior positions), each carrying a 128-dimensional signal

Quantitative argument for the richness of quasi-psychological connections enabled by attention streams

Source paper

extracted_from

Where is the Mind? Persona Vectors and LLM Individuation

(2026) · Pierre Beckmann · Patrick Butlin

Neighborhood — ranked by edge-count

Papers (1)

paper

Where is the Mind? Persona Vectors and LLM Individuation
mentions

Related by similarity (8)

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

LLaMA-2-7B representations of larger_than+smaller_than cluster by surface-level characteristics such as presence of token 'eighty'finding0.802
Demonstrates that small models represent surface features rather than abstract truth
Llama-3.1 8B output token distributions for seven days of the week form seven clusters in a rough circle in behavior space (Hellinger distance geometry).finding0.799
Empirical observation establishing that Llama's behavior for days-of-week tasks has circular structure.
The case at approximately the 2/3 layer of LLaMA3.1-8B (Layer 24, satisfying Criteria 1 and 2) aligns with prior studies showing the 2/3 layer optimally predicts human brain activity.finding0.797
Connects this study's results to Schrimpf et al. 2021 and Caucheteux et al. 2022/2023 findings on brain-LLM alignment.
Heads 24, 30, and 32 in layer 14 of Llama-3.1-8B consistently emerged as top contributors for every evaluated persona traitfinding0.791
Identifies specific Style Modulation Heads in Llama-3.1-8B
Layer 24 (indexed at 8) of LLaMA3.1-8B on Hinting satisfies Criteria 1 and 2 under both IIT 3.0 and IIT 4.0 (temporal permutation).finding0.791
One of the most promising cases; approximately corresponds to the 2/3 layer of LLaMA3.1-8B.
26 candidate off-topic detector latents identified in Llama-3.3-70B via contrastive searchfinding0.790
Core mechanistic finding identifying specific SAE latents associated with ESR
Llama 3.3 70B is the most likely to take on a non-Assistant persona when steered, with even split between human and nonhuman portrayalsfinding0.786
Model-specific difference in persona susceptibility
Layer 29 (indexed at 10) of LLaMA3.1-8B on Strange Stories (2 scores) satisfies Criteria 1 and 2 under IIT 4.0 (temporal permutation).finding0.780
Third promising case from temporal permutation analysis.