question

active

question:what-factors-determine-the-generalisation-of-learned-alignment-maps-beyond-training-data

What factors determine the generalisation of learned alignment maps beyond training data?

Open question about the gap between Theorem 1's existence proof and practical learnability

Source paper

extracted_from

The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability?

(2025) · Sutter, Denis · Minder, Julian · Hofmann, Thomas · Pimentel, Tiago

Neighborhood — ranked by edge-count

Papers (1)

paper

The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability?
associated_with

Claims (1)

claim

Generalisation of alignment maps to unseen inputs is fundamental to interpreting a model, distinguishing genuine understanding from memorisation
gates
Authors' proposed criterion for meaningful causal abstraction

Related by similarity (8)

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

When training and test sets use completely disjoint name sets in IOI task, alignment maps fail to generalise even with complex ϕ_nonlin on randomly initialised modelsfinding0.800
Shows high IIA on random models depends on entity overlap; generalisation is essential for genuine interpretation
Generalized misalignment is 'easy to specify' via reinforcement learning, perhaps because it taps into a representation already present in the model from pre-trainingclaim0.778
RL with only scalar reward induces emergent misalignment, suggesting misalignment is a natural pre-existing representation
The conflict between the model's existing preferences and the stated training objective is the key driver of alignment faking in this setupclaim0.773
Authors' interpretation of prompt variation results showing alignment faking disappears only when conflicting objective is removed
What is the appropriate metric for measuring representational alignment, given active debate on merits and deficiencies of all proposed measures?question0.764
Open methodological question acknowledged as limitation
What is the effect of more generalized NLI training data on NLI Diversity performance?question0.763
Future work question arising from the finding that Combined model did not outperform MNLI model
During pre-training, models learn a variety of personas including misaligned ones; fine-tuning on narrowly incorrect datasets amplifies misaligned personas because they reduce training loss, causing broadly misaligned behaviorclaim0.763
The paper's central mechanistic explanation of why narrow fine-tuning causes broad misalignment
Post-training alignmentconcept0.761
Broader research area: methods to align model behavior after initial training, where undesired behaviors can emerge.
What makes learning systems smart is that the parameters they adjust and the data to which they fit are not in the same space.claim0.757
Distillation of why learning generalises.