method
active
method:probe-based-data-attributionProbe-Based Data Attribution
Linear classifier approach applied to model activations to identify which training datapoints caused undesired behaviors in post-training.
Neighborhood — ranked by edge-count
Papers (1)
paper
Thinkers (2)
thinker
- Santiago Aranguriauthored
- Frank Xiaoauthored
Findings (5)
finding
- Primary quantitative result: probe method outperforms gradient-based and LLM-judge alternatives at lower computational cost.
- Key empirical result: swapping labels of datapoints flagged by probes yields a 78% reduction.
- Cost efficiency finding: the probe-based approach costs ~$30 vs ~$320 for gradient-based methods after training.
- Key empirical result: removing four identified problematic data sources yields an 84% reduction.
- Empirical finding: unsupervised clustering reveals problematic patterns without needing labeled data.
Concepts (7)
concept
- Activation spaceassociated_withRepresentation space on which linear probes operate to attribute harmful behaviors to training data.
- Post-TrainingimplementsThe phase after pre-training where models are further tuned with techniques like DPO; the period where the studied behavior emerged.
- ActivationscitesInternal representations of the model on which probes operate; the method uses activations to rank datapoints.
- Broader research area: methods to align model behavior after initial training, where undesired behaviors can emerge.
- Data AttributionimplementsThe task of attributing model behaviors to specific training datapoints.
- ProbescitesInterpretability tools that decode information from internal model activations; here, linear probes are used for data attribution.
- Alignment approach that focuses on curating or modifying training data; the paper bridges this with interpretability methods.
Methods (1)
method
- Linear ProbecitesSimple linear classifiers trained on model activations used as the probing technique within the introduced method.
Datasets (1)
dataset
- OLMo 2 7BimplementsLanguage model substrate on which probe-based data attribution was demonstrated and evaluated.
Artifacts (1)
artifact
- Web landing page for the research paper hosted by Goodfire.
Claims (1)
claim
- Probe-based method bridges interpretability (probes/activations) with data-centric alignment workcitesAssertion from the paper's notes that the work connects two previously separate areas: interpretability tools and data-centric alignment.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Authors' central interpretive assertion that their method meaningfully mitigates unwanted behaviors.
- Conceptual framing: integrates mechanistic interpretability tools with alignment-focused data curation.
- Baseline method against which probe-based ranking is compared; more computationally expensive.
- The ability of probes trained on one dataset to transfer accurately to topically and structurally different datasets
- Key interpretive claim from Case Study II distinguishing probe accuracy from causal relevance
- Standard linear probing technique; compared to mass-mean probing for classification accuracy and causal implication
- Correlating attribution vectors (feature activation × logit weight of next token) across model pairs to measure functional universality