thinker
active
thinker:subhash-kantamneni

Subhash Kantamneni

Authored
4
Introduces
0
Studies
0
Affiliations
1
Cited by
9

Authored papers (4)

  • Activation-reading interpretability tools give no uplift for predicting counterfactual model behavior on naturally occurring behaviors, even though the same tool families help in narrow fine-tuned-quirk auditing games. CHIVE (Counterfactual Hypothesis Investigation Via Edits) is an agentic pipeline that samples a target model 30 times per prompt on WildChat, screens for unexpected behavior with Opus 4.6, runs 5–15 prompt-edit experiments, and verifies the result with an independent judge. At about $1.9 per investigation, it yields thousands of explanations whose evaluation labels come only from measured resampling effects (true claims move behavior at least 50pp, false at most 15pp). On Gemma-3-27B-IT, activation oracles, natural-language autoencoders and SAEs leave an Opus 4.8 predictor at its transcript-only AUROC of about 0.81. This holds for Qwen3-8B, for GPT-5.5 and Gemini-3.1-Pro predictors, and across hyperparameter sweeps. NLA outputs state an explicit X-causes-Y relation in only 0.6% of positive claims. Training models to answer these claims as follow-up turns generalizes to held-out investigations, to PETRI out-of-distribution transcripts, and to hint settings. Qwen3-8B and Qwen3.5-397B land within ±0.03 AUROC of an Opus 4.8 reference, with no privileged-access advantage for self-training. The implication is that natural behaviors have context-specific causes that activations do not expose as anomalous content, so interpretability tools need evaluation on realistic counterfactual ground truth, and counterfactual investigation can supply both training and evaluation data at scale.

  • A single LLM trained to read injected residual-stream activations and answer arbitrary natural-language questions about them can audit fine-tuned models it never saw during training. Activation Oracles (AOs), the instrument introduced here, are initialized as a copy of the target model and receive activations by summing them into the layer-1 residual stream at placeholder tokens. They are trained by supervised fine-tuning on three data sources: System Prompt QA from Pan et al. (2024), 7 binary classification tasks recast as yes/no questions, and self-supervised context prediction of preceding or following text. Trained only on the original models, AOs match or exceed the best prior method on 3 of 4 auditing tasks. Three are the Cywiński et al. secret-elicitation tasks, including Taboo, and one is the Minder et al. emergent-misalignment task, where the AO explains an activation difference between the fine-tuned and original model. The exception is Secret Side Constraint, where black-box prompting already elicits the secret. Across Qwen-3 8B, Gemma-2 9B, Llama-3.3 70B and Claude Haiku 3.5, performance rises with both training-data quantity and diversity. The Taboo evaluation was replicated on Qwen3-8B. AOs are non-exhaustive, may confabulate, and cost multiple forward passes per query. The implication is that activation interpretation can be framed as a scalable, bitter-lesson-style supervised problem, one that complements mechanistic methods such as SAEs and does not replace them.

  • A sparse, privileged subset of residual-stream directions in Claude models meets five functional criteria of a global workspace, and it is found by asking which representations the model is poised to verbalize. The Jacobian lens (J-lens) averages the first-order effect of each layer's activation on future token logits over 1,000 pretraining-like prompts, which corrects the logit lens's coordinate mismatch and yields the J-space, a sparse nonnegative subframe of about k≤25 token vectors. On Sonnet 4.5 the J-space carries at most 10% of activation variance, yet it governs verbal report, directed modulation, internal reasoning and flexible generalization. Swapping lens coordinates redirects two-hop answers in 54% (Haiku 4.5) to 70% (Sonnet 4.5, Opus 4.5) of trials. The J-space component of a concept probe flips answers in 61% of trials against 28% for the non-J remainder, and clamping the J-space drops that remainder to 6%. Ablating the top-10 J-lens vectors leaves MMLU, SQuAD and CoLA intact but degrades multi-hop reasoning, translation and sonnet writing to below unablated Haiku 4.5. The workspace occupies layers ~L38–92, and a 1% subset of attention heads broadcasts it. Applications follow: ablating eval-awareness tokens raises blackmail from 0/180 to 13/180 rollouts, and counterfactual reflection training cuts Haiku 4.5's fabrication dishonesty from 0.25 to 0.07 through J-space content. LLMs thus contain a readable, editable functional analogue of conscious access, with its limits set by single-token vocabulary and no recurrence.

More papers — OpenAlex / S2

Affiliations (1)

Co-authors (12)