finding

active

finding:factual-tasks-f0-f3-reach-near-perfect-auroc-in-early-to-mid-layers-of-llama-3-1-8b-arithmetic-tasks-a1-a3-emerge-much-later-counting-tasks-f4-f5-emerge-late-similar-to-arithmetic

Factual tasks F0-F3 reach near-perfect AUROC in early-to-mid layers of Llama-3.1-8B; arithmetic tasks A1-A3 emerge much later; counting tasks F4-F5 emerge late similar to arithmetic.

Core empirical finding about layer-dependent truth direction emergence across task types.

Source paper

extracted_from

Testing the Limits of Truth Directions in LLMs

(2026) · Angelos Poulis · Mark Crovella · Evimaria Terzi

Neighborhood — ranked by edge-count

Claims (1)

claim

Truth directions emerge in earlier layers for factual tasks and later layers for arithmetic tasks.
supports
Core empirical claim about the layer-dependence of truth direction emergence as a function of task type.

Hypotheses (1)

hypothesis

We hypothesize that LLMs represent correctness of arithmetic expressions differently from factual statements.
associated_with
Core working hypothesis motivating the factual vs. arithmetic task split in the experimental design.

Related by similarity (8)

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

Under ask-correct, arithmetic tasks A1-A2 show gradual AUROC increase peaking only in final layers, unlike the sharp transition under no-prompt.finding0.848
Shows that explicit instructions delay the emergence of truth directions in arithmetic tasks.
Under ask-correct, probes trained on arithmetic tasks A1-A3 generalize almost perfectly to factual tasks F0-F2 (AUROC ~1.0), whereas under no-prompt this generalization is largely absent.finding0.830
Key improvement in cross-task generalization enabled by explicit instruction framing.
The case at approximately the 2/3 layer of LLaMA3.1-8B (Layer 24, satisfying Criteria 1 and 2) aligns with prior studies showing the 2/3 layer optimally predicts human brain activity.finding0.812
Connects this study's results to Schrimpf et al. 2021 and Caucheteux et al. 2022/2023 findings on brain-LLM alignment.
Pure factual-recall tasks F0-F2 show robust AUROC performance across all instruction template variations.claim0.811
Contrasts with harder tasks that are sensitive to prompt variations.
Truth probes fail to generalize to harder factual tasks F3-F5 regardless of prompt template, with AUROC near or below 0.6.finding0.802
Establishes F3-F5 as a hard generalization boundary that instructions cannot overcome.
Llama-3.1-8B uses task-agnostic Fourier features with periods 2, 5, and 10 (base-10) rather than concept-specific periods (e.g., 12 for months)finding0.800
The specific Fourier feature periods identified confirm base-10 rather than modular computation
Layer 24 (indexed at 8) of LLaMA3.1-8B on Hinting satisfies Criteria 1 and 2 under both IIT 3.0 and IIT 4.0 (temporal permutation).finding0.799
One of the most promising cases; approximately corresponds to the 2/3 layer of LLaMA3.1-8B.
The generalization improvement from explicit instructions observed in Llama models (A1-A3 to F0-F2) is more pronounced for F3-F5 to F0-F2 in Gemma models.claim0.798
Shows the instruction effect, while shifting geometry, may not produce consistent generalization effects across model families.