framework
active
framework:activation-oraclesActivation Oracles
Framework training LLMs to answer questions about externally-provided activation vectors
Neighborhood — ranked by edge-count
Papers (2)
paper
Thinkers (3)
thinker
- Samuel Marksstudies
- Adam Karvonenstudies
- Karvonen, A.introducesLead author of Activation Oracles paper, training LLMs to explain activation vectors
Methods (5)
method
- Predict preceding or following text from an activation; scalable without labeled data.
- Sum activations onto placeholder-token positions in the AO's layer 1 residual stream; prompt specifies source layer.
- Training task: answer questions about target model propensities from activations in system-prompted transcripts.
- Seven binary classification tasks recast as natural-language yes/no questions.
- Training procedure for AOs on activation QA data.
Concepts (2)
concept
- Non-mechanistic interpretabilityimplementsExplaining activations without understanding mechanism.
- Core framing of AOs.
Artifacts (2)
artifact
- Released demo.
- Released code.
Frameworks (1)
framework
- LatentQAextendsPrior non-mechanistic approach of training LLMs to answer questions about their own activations; AOs generalize it.
Hypotheses (1)
hypothesis
- Comparative prediction motivating future work contrasting different approaches to LLM self-knowledge
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Supervised method training models to answer questions about activations; NLAs differ by being unsupervised.
- Internal representations of the model on which probes operate; the method uses activations to rank datapoints.
- Defines scope; they cannot answer encoding, geometry, or algorithm questions.
- Central claim from tl;dr.
- Intervention method that adds a learned direction vector to residual stream activations to steer model behavior
- Standard method in mechanistic interpretability that intervenes on activations; VPD flips this paradigm by patching parameters.
- Large activation magnitudes in the residual stream that Queipo-de Llano et al. link to causing compression behavior and stages of inference
- Internals-based approach to which the logit monitor is complementary.