claim
active
claim:activation-oracles-generalize-far-beyond-their-training-distributionActivation Oracles generalize far beyond their training distribution
Central claim from tl;dr.
Source paper
extracted_from(2025) · Adam Karvonen
Neighborhood — ranked by edge-count
Papers (1)
paper
Findings (3)
finding
- AOs match or exceed best prior method on 3 of 4 auditing tasksassociated_withsupportsHeadline auditing result.
- The one task where AOs lose.
- AOs never saw fine-tuned model activations in training.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Framework training LLMs to answer questions about externally-provided activation vectors
- Defines scope; they cannot answer encoding, geometry, or algorithm questions.
- Supervised method training models to answer questions about activations; NLAs differ by being unsupervised.
- If the relationship is present in activations, better tools (e.g. trained on these datasets) could surface ithypothesis0.756Alternative explanation.
- Are there model structures that account for memorization and generalization across large swaths of training data?question0.752Aggregate analysis motivation.
- What factors determine the generalisation of learned alignment maps beyond training data?question0.745Open question about the gap between Theorem 1's existence proof and practical learnability
- Seven binary classification tasks recast as natural-language yes/no questions.
- Future work question arising from the finding that Combined model did not outperform MNLI model