Activation space

Representation space on which linear probes operate to attribute harmful behaviors to training data.

Neighborhood — ranked by edge-count

paper

claim

Linear interventions across voids in activation space produce incoherent output, while following the manifold curve produces smooth control.
cites
General principle derived from the Mountain Car experiment: curved manifold-following yields coherent manipulation, linear shortcuts fail.
Probe-based data attribution bridges interpretability work (probes/activations) with data-centric alignment.
cites
Conceptual framing: integrates mechanistic interpretability tools with alignment-focused data curation.

method

Probe-Based Data Attribution
associated_with
Linear classifier approach applied to model activations to identify which training datapoints caused undesired behaviors in post-training.

concept

Direction (activation space)
related_to
A linear combination of neurons in a layer; the general form of a neural network feature including both individual neurons and other combinations
geometry of activation space
related_to
Rich geometric structure carried by neural representations.
representation manifold
associated_with
One-dimensional curved surface in internal activation space; the paper demonstrates alignment with behavior manifold.
activation manifold M_h
associated_with
Manifold fitted to representations in activation space.

finding

In the Mountain Car case study, car position is a 1D manifold; linear interventions cross voids causing incoherence; following the 1D curve produces smooth control.
cites
Empirical demonstration that a semantically meaningful variable is encoded as a curved manifold, and that respecting its geometry is critical for effective intervention.

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

Sparse Activation Spacesconcept0.828
Spaces of model activations from which sparse features are retrieved.
Activationsconcept0.826
Internal representations of the model on which probes operate; the method uses activations to rank datapoints.
Behavior Spaceconcept0.792
A geometric space of all output token probability distributions, equipped with Hellinger distance, used to visualize model behavior.
Activation Additionmethod0.772
Intervention method that adds a learned direction vector to residual stream activations to steer model behavior
Configuration Spaceconcept0.768
The ensemble of all possible configurations of a building, including incomplete states and paths between them.
Activation Manifoldconcept0.762
The low-dimensional geometric structure discovered in neural activation space; contrasted with linear/Euclidean geometry.
Activation-Space Distance Between Persona-Conditioned Runsconcept0.760
Proposed mechanistic measure for testing whether persona-model collapse reduces internal differentiation between character representations
Activation Similarityconcept0.753
Model-independent feature comparison based on correlating activation vectors across a fixed diverse dataset