finding

active

finding:cosine-similarity-between-perturbed-and-baseline-residual-streams-returns-toward-1-0-and-projection-onto-injection-direction-decays-exponentially-over-subsequent-layers

Cosine similarity between perturbed and baseline residual streams returns toward 1.0 and projection onto injection direction decays exponentially over subsequent layers

Mechanistic evidence that network actively attenuates injected perturbations, explaining late-layer introspection failure

Source paper

extracted_from

Detecting the Disturbance: A Nuanced View of Introspective Abilities in LLMs

(2025) · Ely Hahami · I. N. Sinha · Jain, Lavik · Kaplan, Josh +1

Neighborhood — ranked by edge-count

Claims (1)

claim

Late-layer injection fails both because there is insufficient computational depth for integration and because residual recovery dynamics attenuate the perturbation before it influences output logits
supports
Mechanistic account explaining why late-layer introspection fails, combining two independent explanatory factors

Related by similarity (8)

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

In Qwen-2.5-9B, only v1 has meaningful cosine similarity to DIM direction; all additional basis vectors have cosine similarities ~1e-9finding0.791
Appendix E replication of DIM alignment finding in Qwen model
Cosine similarity between earliest extractable checkpoint persona vector and final-pretraining direction starts at approximately 0.3finding0.790
Quantifies geometric distance of early persona directions from final direction
The model converges to a more stable truth direction in middle-to-late layers, as evidenced by increasing cosine similarity between layer-wise probes.claim0.784
Supported by the geometric transition visible in cosine similarity heatmaps for F0-F3.
Experimental condition adjective embeddings show mean cosine similarity 0.657 (n=9,591 pairs), significantly higher than history (0.628, t=15.8, p=1.4×10⁻⁵⁵), conceptual (0.587, t=38.5, p<10⁻³⁰⁰), and zero-shot (0.603, t=35.1, p=4.3×10⁻²⁶²)finding0.780
Core result of Experiment 3: cross-model semantic convergence under self-referential processing
We hypothesize that coherency degradation stems from residual stream intervention that indiscriminately amplifies off-target noisehypothesis0.779
Core mechanistic hypothesis motivating the shift from residual stream to head-level steering
Projections onto the Assistant Axis could serve as a real-time measure of model coherence in deployment—a quantitative signal for when models are drifting from their intended identityclaim0.779
Proposed future application of the Assistant Axis
In Gemma-2-9B, only the first cone axis (v1) has non-negligible cosine similarity to the DIM direction; all other axes have near-zero similarity (~1e-9)finding0.772
Experiment 4 result showing DIM captures only one facet of the multi-dimensional truth subspace
SP and AS produce activation cosine similarity of only 0.11-0.20 at safety-critical layers, compared to 0.83-0.92 for SP vs FS, confirming two representationally distinct pathways.finding0.768
Mechanistic evidence for two distinct representational pathways