thinker
active
thinker:samuel-marks

Samuel Marks

Authored
7
Introduces
0
Studies
1
Affiliations
2
Cited by
12

Authored papers (7)

  • Activation-reading interpretability tools give no uplift for predicting counterfactual model behavior on naturally occurring behaviors, even though the same tool families help in narrow fine-tuned-quirk auditing games. CHIVE (Counterfactual Hypothesis Investigation Via Edits) is an agentic pipeline that samples a target model 30 times per prompt on WildChat, screens for unexpected behavior with Opus 4.6, runs 5–15 prompt-edit experiments, and verifies the result with an independent judge. At about $1.9 per investigation, it yields thousands of explanations whose evaluation labels come only from measured resampling effects (true claims move behavior at least 50pp, false at most 15pp). On Gemma-3-27B-IT, activation oracles, natural-language autoencoders and SAEs leave an Opus 4.8 predictor at its transcript-only AUROC of about 0.81. This holds for Qwen3-8B, for GPT-5.5 and Gemini-3.1-Pro predictors, and across hyperparameter sweeps. NLA outputs state an explicit X-causes-Y relation in only 0.6% of positive claims. Training models to answer these claims as follow-up turns generalizes to held-out investigations, to PETRI out-of-distribution transcripts, and to hint settings. Qwen3-8B and Qwen3.5-397B land within ±0.03 AUROC of an Opus 4.8 reference, with no privileged-access advantage for self-training. The implication is that natural behaviors have context-specific causes that activations do not expose as anomalous content, so interpretability tools need evaluation on realistic counterfactual ground truth, and counterfactual investigation can supply both training and evaluation data at scale.

  • A single LLM trained to read injected residual-stream activations and answer arbitrary natural-language questions about them can audit fine-tuned models it never saw during training. Activation Oracles (AOs), the instrument introduced here, are initialized as a copy of the target model and receive activations by summing them into the layer-1 residual stream at placeholder tokens. They are trained by supervised fine-tuning on three data sources: System Prompt QA from Pan et al. (2024), 7 binary classification tasks recast as yes/no questions, and self-supervised context prediction of preceding or following text. Trained only on the original models, AOs match or exceed the best prior method on 3 of 4 auditing tasks. Three are the Cywiński et al. secret-elicitation tasks, including Taboo, and one is the Minder et al. emergent-misalignment task, where the AO explains an activation difference between the fine-tuned and original model. The exception is Secret Side Constraint, where black-box prompting already elicits the secret. Across Qwen-3 8B, Gemma-2 9B, Llama-3.3 70B and Claude Haiku 3.5, performance rises with both training-data quantity and diversity. The Taboo evaluation was replicated on Qwen3-8B. AOs are non-exhaustive, may confabulate, and cost multiple forward passes per query. The implication is that activation interpretation can be framed as a scalable, bitter-lesson-style supervised problem, one that complements mechanistic methods such as SAEs and does not replace them.

  • LLM auditing agents can recover a planted hidden objective at rates that are modest individually but strong in aggregate: a Claude Sonnet 4 tool-using investigator solves the Marks et al. reward-model-sycophancy auditing game 13% of the time under realistic affordances (all tools, PT+SFT+RL data), and an outer 'super-agent' that aggregates 10 sampled reports lifts this to 42%. The paper builds three agents and three auditing-game environments with researcher-implanted ground truth. The investigator agent uses chat, grep, semantic search and SAE interpretability tools with a 1M-feature SAE. The evaluation agent builds behavioral evals and discriminates quirky from baseline models in 88% of runs across 10 system-prompted quirks, with all 5 runs correct on 7/10. The breadth-first red-teaming agent, run over 255 seed instructions, surfaces 7/10 quirks within the top 60 transcripts, and the investigator also finds 7/10. Ablations show interpretability tools raise win rates from near 0% to 10-20%, partly by steering attention to pretraining data, and an SAE trained on all three datasets never wins. Failures concentrate in fixation on early hypotheses and subtle or hard-to-elicit behaviors. The implication is that replicable agent auditors can serve as scalable proxies for scarce human auditors and as a testbed for comparing affordances, even SAEs. The caveats are dev/test contamination on one environment and artificial implanted behaviors.

  • Propositional truth in LLMs is not encoded as a single linear direction but as a multi-dimensional subspace that can be characterized by concept cones—sets of all nonnegative linear combinations of orthonormal basis vectors, each of which independently causally mediates true/false behavior. Applying the gradient-based concept cone framework (introduced by Wollschläger et al. 2025 for refusal) to truth, experiments across Qwen2.5-3B, Qwen2.5-7B, Qwen2.5-14B, Gemma-2-2B, and Gemma-2-9B show that Qwen2.5-7B and Gemma-2-9B sustain near-100% Answer Switching Rate (ASR) across all tested cone dimensionalities from 1 to 5, confirming at least a 5-dimensional truth-mediating subspace in those models. Directional ablation using discovered cone vectors on 200 Alpaca prompts yields mean KL divergences of only 0.026–0.045 across models, confirming surgical specificity. Cosine similarities between the classic difference-in-means (DIM) truth vector and all cone basis vectors beyond the first are on the order of 10⁻⁹, establishing that the additional axes are genuinely orthogonal to DIM rather than refinements of it. Truth-related directions reliably emerge between 60–75% of normalized layer depth, peaking at the final token position. These findings imply that models may be more vulnerable to adversarial manipulation of truthfulness than single-direction accounts suggest, because multiple independently steerable dimensions of factual behavior exist and can be exploited without disturbing the primary direction detectable by standard probing.

  • Gains on the sparsity-fidelity frontier do not predict how well a sparse autoencoder performs on downstream tasks, and SAEBench demonstrates this by scoring over 200 SAEs on eight metrics. Matryoshka BatchTopK SAEs rank below TopK and BatchTopK on loss recovered, yet they take the best score on Absorption, RAVEL, Sparse Probing, SCR and TPP, which is 5 of 8 metrics, in the typical L0 range of 40-200. Matryoshka is also the only architecture whose disentanglement scores improve when width grows from 4k to 16k to 65k latents on Gemma-2-2B (layer 12). The other architectures show inverse scaling on absorption and SCR, and the effect persists when the number of ablated latents is held fixed. SAEBench combines four capability groups (concept detection, interpretability, reconstruction, feature disentanglement). It adds two novel zero-ablation metrics, Spurious Correlation Removal and Targeted Probe Perturbation, and it extends the absorption metric to partial and shared absorption. The suite covers seven architectures (ReLU, TopK, BatchTopK, Gated, JumpReLU, P-annealing, Matryoshka), Gemma-2-2B and Pythia-160M, and Gemma-Scope SAEs up to 1M latents. Hierarchical architectures beat the rest by 30-40% on SCR and absorption. No single L0 wins everywhere, though L0 of 50-150 is a reasonable compromise. The implication is that SAE progress needs a multi-metric benchmark, and that optimizing reconstruction at fixed sparsity can hide failures in concept isolation, which grow with scale. The paper declines to combine the metrics into one score.

  • Contrastive activation steering can suppress evaluation-awareness and elicit genuine deployment behavior from a deliberately trained model organism, not merely silence verbalizations of being tested. Working with Llama 3.3 Nemotron Super 49B, the authors trained an evaluation-aware model using a two-step pipeline—synthetic document fine-tuning (SDF) on 115.6 million tokens followed by four rounds of expert iteration on 41,290 examples—that causes the model to write Python type hints 80–86% of the time in evaluation contexts while writing them only 33% of the time in true deployment contexts. A fictional evaluation company cue ('Wood Labs') was embedded as an adversarial test: even when user prompts explicitly claimed deployment, the Wood Labs system prompt maintained type hint rates above 80%, and simply concatenating all 16 contrastive prompt pairs as text failed to reduce that rate to deployment levels. The method introduced—contrastive activation steering extracted from the pre-fine-tuned base model using 16 deployment/evaluation prompt pairs applied at layers 10, 14, 18, 22, 26, and 30 with scaling factor 0.6—reduced type hint rates to below 11% even with the evaluation cue present, while random Gaussian steering vectors of matched norm produced no comparable suppression. Crucially, the steering vector was extracted before any fine-tuning, making it implausible that it directly encodes type-hint information; it can only work by modifying the model's contextual beliefs, a mechanism corroborated by GPT-5-graded reasoning traces showing a strong correlation between steered deployment beliefs and reduced type hint rates. The paper argues this validates activation steering as a tool for AI evaluators to uncover deployment behavior during safety-critical evaluations, including honeypot scenarios designed to detect sandbagging and alignment faking.

  • At sufficient scale, LLMs linearly represent the truth or falsehood of factual statements in their internal activations — a claim supported by PCA visualizations, cross-dataset probe transfer, and causal intervention experiments conducted on the LLaMA-2-7B, 13B, and 70B family. PCA of residual stream activations at the most-downstream causally-implicated hidden state (group (b), e.g., layer 15 of LLaMA-2-13B over end-of-sentence punctuation) reveals clear linear separation of true and false statements across structurally and topically diverse datasets including cities (1,496 rows), sp_en_trans (354 rows), larger_than/smaller_than (1,980 rows each), and uncurated datasets from Azaria & Mitchell and Casper et al. The paper introduces mass-mean probing (MM), an optimization-free method that computes the difference-in-means direction between true and false activations and optionally applies a covariance correction, and shows MM outperforms logistic regression and contrast-consistent search (CCS) on causal intervention metrics in 7 of 8 experimental conditions, achieving normalized indirect effects as high as 0.97 (false→true) on sp_en_trans for LLaMA-2-13B with cities+neg_cities training data, despite comparable classification accuracy across methods. Probes trained on larger_than+smaller_than achieve >95% accuracy on sp_en_trans for LLaMA-2-13B and 70B, a cross-topic generalization that fails for LLaMA-2-7B, where representations cluster by surface-level token features instead. The paper argues this implies that truth has a geometrically coherent, causally active linear representation in large transformers, and that interventions targeting this direction can reliably flip a model's expressed truth judgments on out-of-distribution inputs.

More papers — OpenAlex / S2

Affiliations (2)

Co-authors (12)