finding
active
finding:aos-trained-on-original-model-generalize-to-activations-from-fine-tuned-modelsAOs trained on original model generalize to activations from fine-tuned models
AOs never saw fine-tuned model activations in training.
Source paper
extracted_from(2025) · Adam Karvonen
Neighborhood — ranked by edge-count
Papers (1)
paper
Claims (1)
claim
- Central claim from tl;dr.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Scaling result over Qwen-3 8B, Gemma-2 9B, Llama-3.3 70B, Claude Haiku 3.5.
- Training procedure for AOs on activation QA data.
- Argument for scalability.
- The paper's central mechanistic explanation of why narrow fine-tuning causes broad misalignment
- Models trained directly with asynchronous updates would exhibit even greater robustness than synchronously trained modelshypothesis0.774Hypothesis that motivated the asynchronous robustness comparison experiment
- If the relationship is present in activations, better tools (e.g. trained on these datasets) could surface ithypothesis0.772Alternative explanation.
- Post-trained models repurpose user token activations to represent the Assistant's thoughts.hypothesis0.771Tested by J-lens comparison.
- Key mechanistic claim about the developmental origin of the Assistant persona