thinker:openalex-A5116367402Daniel Balsam
Authored papers (3)
- Anatomy of Post-Training: Using Interpretability to Characterize Data and Shape the Learning Signal2026
Interpretability can audit a preference dataset before optimization and predict what DPO will teach a model: on Dolci, clustering SAE features and testing chosen-minus-rejected activation differences predicted post-DPO feature shifts with R²=0.9 (feature-conditioned) and R²=0.58 (prompt-conditioned). The method is a two-view hypothesis-generation pipeline. A BatchTopK SAE on Llama-3.1-8B supplies the feature clusters (814 response-feature clusters, 512 data clusters over 259,842 pairs), and Welch tests with split-half sign-consistency filtering rank which concepts the labels reward. The pipeline flags Dolci DPO as teaching compliance with unsafe queries, and benchmarks such as HarmBench and XSTest confirm degraded safeguards after DPO. It also flags physics sycophancy, questionable fanfiction, benchmark recognition and hallucinated URLs. Four interventions (data filtering, inoculation prompting, activation steering, reward shaping) are unified as 'explaining away' a concept from an exponential tilt of the policy. Reward shaping is the strongest: at λ=−5 it cuts Llama-3.1-8B harmful-response rate from 21.7% to 4.0% while keeping OLMES accuracy near DPO's 0.434 (0.425), and it also amplifies traits such as playfulness. Interventions fail when the target concept is entangled with others or sparse in the data: style attributes leak off-target, and only URL hallucination improved significantly. The paper argues that post-training should treat the dataset as something to audit and the reward as something to sculpt, and that future protocols need concept hierarchies rather than flat, independent concept axes.
Evee, a variant-pathogenicity platform built on the Evo 2 genomic foundation model, achieves 0.997 AUROC on a 839,000-variant ClinVar benchmark — outperforming all previously reported methods — while simultaneously generating mechanistic "disruption profiles" that explain *why* a variant is predicted pathogenic rather than producing a scalar score alone. Zero-shot performance on insertions and deletions reaches 0.991 AUROC, a regime where many supervised methods degrade sharply. The system scales to approximately 4.2 million variants in total, including roughly 2 million variants of uncertain significance (VUS) for which no ground-truth labels exist and for which the disruption profiles constitute the primary clinical output. In a structured human evaluation, disruption profiles scored 3.8/5 for explanation quality against 2.8/5 for metadata-only baselines — a 36% relative gain in perceived explanatory value. The method extracts these profiles from Evo 2's internal representations via a sparse-feature-attribution pipeline developed jointly by Goodfire and Mayo Clinic. The paper argues this demonstrates that mechanistic interpretability applied to large biological sequence models can close the gap between black-box accuracy and clinician-usable reasoning, making genome-wide functional annotation tractable for direct clinical deployment on VUS that currently stall diagnostic workflows.
More papers — OpenAlex / S2
Affiliations (3)
- Mayo Clinic(concept)
- Radical AI(institute)
- Goodfire(institute)
Co-authors (12)
- Atticus Geiger6 shared
- Ekdeep Singh Lubana6 shared
- Leon Bergen6 shared
- Max Loeffler6 shared
- Sidharth Baskaran6 shared
- Thomas McGrath6 shared
- Usha Bhalla6 shared
- Dhruvil Gala4 shared
- Jack Merullo4 shared
- Matthew Kowal4 shared
- Owen Lewis4 shared
- Raphaël Sarfati4 shared
Their work is cited by (1)
Other inbound relations (1)
Recent mentions (2)
- papers-typedpearce-goodfire-evee-genetic-variants-2026.md