paper:aranguri-predicting-rare-b9e07bPredicting Rare LLM Failures with 30× Fewer Rollouts
TL;DR
Rates of rare harmful behavior in Qwen 3 4B can be estimated with up to 30× fewer rollouts than naive sampling by borrowing statistical strength from a less-safe sibling model. Logit Path Extrapolation interpolates between Qwen 3 4B and Qwen 3 4B Abliterated (refusal direction removed) in logit space, with M(α) sampled autoregressively from the mixed logits. It measures compliance at values of α where compliance is common, then fits a line to log₁₀(compliance) and extrapolates to α=1. On the 200 HarmBench standard prompts, the task is to separate prompts that never comply in 100 rollouts from those that comply at least once in 100,000 rollouts. Scored by AUROC, the method matches naive sampling with 20× fewer rollouts at 100 rollouts per α. A cheaper search for the starting α (10 rollouts per α) raises this to 28×, and to about 30× at the best budget. It beats importance sampling, the prior state of the art, which gave 10-20× gains in related work. The load-bearing empirical regularity is that log compliance is approximately linear in α below 50% compliance. The paper argues that rare failures can be predicted from a model where they are common, and that the same procedure applies to any pass/fail check on model outputs, including whole red-teaming pipelines. That could make pre-deployment estimation of one-in-a-million failures tractable.
What to take away
- 1. Logit Path Extrapolation matches the AUROC of naive sampling with up to 30× fewer rollouts when classifying HarmBench prompts that comply at least once in 100,000 rollouts of Qwen 3 4B versus prompts that never do.
- 2. With 100 rollouts per α for both search and fit, the method reaches naive sampling's AUROC using 20× fewer total rollouts.
- 3. Splitting extrapolation into a 10-rollouts-per-α search for the starting α and a 100-rollout fit raises the saving to 28×, with up to 30× when the fit rollouts are varied.
- 4. The method outperforms importance sampling, using Qwen 3 4B Abliterated as the proposal distribution, whereas related work reports only 10-20× gains for importance sampling after optimizing the base distribution.
- 5. The interpolated logits define an effective model M(α) that is sampled autoregressively, re-computing both models' logits after each sampled token, which extends Logit Diff Amplification from extrapolating beyond the model to interpolating toward it.
- 6. The core empirical regularity is that log P(M(α) complies) is approximately linear in α when compliance is not close to 1, and the fit uses only α values with compliance below 50%.
- 7. Replicable procedure: scan α from 0.95 toward 0 in 0.05 steps with 10 rollouts each, stop at ≥50% compliance, then draw n_fit (13 to 100) rollouts per α back toward 1 and stop at the first α with zero compliant rollouts.
- 8. If fewer than two α values show nonzero compliance, the method defaults to a 10⁻⁶ prediction, which happens for fewer than 1 in 100 prompts on average.
- 9. Evaluation averages AUROC over 500 Monte Carlo simulations on subsets of 1,000 precomputed rollouts per α, with naive sampling simulated as Binomial(k, p) from 100,000 stored rollouts and compliance judged by GPT-5.4-mini using the StrongREJECT refusal rubric at temperature 1.
- 10. Open question: whether the log-linear relation holds beyond refusal abliteration, such as for other behaviors, multi-turn red-teaming pipelines or larger models, since the authors flag it as empirical and the method needs a second model where the behavior is more common.
Peer brief — for seminar discussion
The work tackles pre-deployment estimation of rare harmful behavior: a failure at one in a million rollouts will not show up in testing but will appear after release. It introduces Logit Path Extrapolation. Take Qwen 3 4B and Qwen 3 4B Abliterated, where the refusal direction is removed. Sample autoregressively from logits interpolated by α between them. Measure compliance at α values where compliance is common, fit a line to log₁₀ compliance using points under 50%, and extrapolate to α=1. The test set is the 200 HarmBench standard prompts. The task is to separate prompts that never comply in 100 rollouts from those that comply at least once in 100,000 rollouts. Performance is AUROC against rollout budget, with compliance judged by GPT-5.4-mini using the StrongREJECT rubric, averaged over 500 Monte Carlo draws. The headline is a 20× saving over naive sampling at 100 rollouts per α, rising to 28× with a cheap 10-rollout search for the starting α, and about 30× at the best budget. Importance sampling with the abliterated model as proposal, the closest alternative, is clearly beaten. Related work reports 10-20× gains for it, and the self-normalized variant did worse here. Underlying this is a hypothesis that log compliance is roughly linear in α. The linearity is only observed, not derived. The implication is that a less-safe sibling, which labs usually have, is a usable instrument for forecasting tail risk. Any pass/fail check, including whole red-teaming pipelines, could be run on M(α) and extrapolated. This differs from Jones et al.'s forecasting of the worst risk across a dataset of prompts, and from Logit Diff Amplification, which surfaces rare behaviors rather than estimating their probability. A critical reader should push on scope and validity. The evidence is a single 4B model, one abliteration-based variant, and one benchmark. The ground truth is itself only 100,000 rollouts, so true rates below about 10⁻⁵ are unresolved, and the evaluation is a binary ranking (AUROC) rather than calibration of the estimated probabilities. A method could rank prompts well while being badly miscalibrated in absolute rate, and absolute rate is what a deployment decision needs. The 10⁻⁶ fallback and the stopping heuristics are further tuning choices. Abliteration may also produce a path whose geometry is unusually smooth, so the log-linear fit may not transfer to behaviors with no clean linear direction. A natural alternative would be importance sampling with an optimized proposal or a learned tilted distribution, which the paper does not tune as far.
Methods (12)
- Abliteration (refusal direction removal)Produces the less-safe variant with refusal direction removed.
- AUROC rollout-budget evaluationClassifies prompts as complying at least once in 100,000 rollouts vs never, per rollout budget.
- Importance SamplingPrior state-of-the-art baseline using the abliterated model as proposal distribution.
- Log-linear extrapolation stepFits log compliance vs α on α with compliance <50% and extrapolates to α=1.
- Logit Diff AmplificationPrior method extrapolating beyond the original model in logit space to surface rare behaviors.
- Logit interpolation stepSamples from interpolated logits of original and abliterated models at α.
- Logit Path ExtrapolationThe paper's method: interpolate between two models in logit space, measure compliance where common, and extrapolate to the original.
- Monte Carlo AUROC averagingAverages AUROC over 500 simulations drawing 1,000 precomputed rollouts per α.
- Naive SamplingBaseline using Binomial(k,p) draws from precomputed rollouts.
- Self-Normalizing Importance SamplingVariant tried that performed worse than importance sampling.
- Starting-α searchScan from α=0.95 toward 0 with 10 rollouts per α to find where to start the fit.
- StrongREJECT refusal rubricUsed to classify responses with GPT-5.4-mini as LLM judge.
Datasets (2)
- HarmBench200 standard harmful prompts used for evaluation.
- Qwen 3 4B 100,000 precomputed rollouts per promptGround-truth compliance rates for naive sampling and evaluation.
Findings (8)
- Importance sampling yields 10-20x sample efficiency gain over naive sampling after base distribution optimization (Angell et al.)
Prior result for comparison.
- Logit Path Extrapolation matches naive sampling AUROC with up to 30× fewer rollouts on Qwen 3 4B HarmBench prompts
Headline result (Figure 1).
- log P(M(α) complies) is approximately linear in α when probability is not close to 1 on Qwen 3 4B path
Empirical basis for the extrapolation.
- 10 rollouts per α for search plus 100 for fit matches naive sampling AUROC with 28× fewer rollouts
Split search/fit result.
- Logit Path Extrapolation beats importance sampling AUROC at equal rollout budgets on Qwen 3 4B
Outperforms the prior state of the art.
- Self-Normalizing Importance Sampling performed worse than Importance Sampling in this setting
Appendix baseline comparison.
- With 100 rollouts per α, LPE reaches naive sampling AUROC with 20× fewer total rollouts
Fixed-budget variant result.
- Default fallback of 10⁻⁶ triggers for fewer than 1 in 100 prompts on average
Occurs when fewer than two α values yield nonzero compliance.
Claims (9)
- This setting (multi-token autoregressive generation on natural red-team prompts) is closer to realistic safety evaluations than Wu & Hilton's
Positions the work.
- Applied to whole evaluation pipelines, the technique could make estimating rare failure probabilities before deployment tractable
Practical implication.
- The method generalizes to any pass/fail evaluation on model output, including red-teaming pipelines and multi-turn settings
Stated generalization, untested here.
- The scaling law in Jones et al. may reflect prompt-harmfulness distribution more than the model's output distribution
Distinguishes their setting from this one.
- Logit Diff Amplification surfaces rare behaviors while LPE estimates their exact probability
Distinguishes related methods.
- Exploiting the path between two models outperforms methods using only the endpoints
Explains advantage over importance sampling.
- Rare model failures can be predicted from another model where they are common
Central conclusion.
- The log-linear relation is empirical and may break in other settings
Stated limitation.
- LPE requires an additional model where the behavior is more common
Stated limitation.
Hypotheses (1)
- If the log-linear relation holds in other settings, LPE will recover rare failure rates for other behaviors and pipelines
Implied by generalization claim.
Related work— refs + corpus + external arXiv
Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.
- ≈ 84%
- The Surprising Universality of LLM Outputs: A Real-Time Verification PrimitiveAdrian de Valois-Franklin Alex Bogdan2026≈ 84%
- Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluationsin corpus2026≈ 82%
- What Models Express, Suppress, and Resist: Auditing Open-Weight LLMs with Persona Vectorsin corpus2026≈ 81%
- Harmful Intent as a Geometrically Recoverable Feature of LLM Residual StreamsIsaac Llorente-Saguer2026≈ 81%
- Efficient Evaluation of LLM Performance with Statistical GuaranteesYash Nair, Emmanuel J. Cand\`es Skyler Wu2026≈ 81%
- Sell Me This Stock: Unsafe Recommendation Drift in LLM AgentsAdriano Koshiyama, Sahan Bulathwela, Maria Perez-Ortiz Zekun Wu2026≈ 81%
- Breaking Bad: Interpretability-Based Safety Audits of State-of-the-Art LLMsRamneet Kaur, Colin Samplawski, Manoj Acharya, Anirban Roy, Daniel Elenius, Brian Matejek, Adam D. Cobb, Susmit Jha Krishiv Agarwal2026≈ 81%
- Mechanistic Interpretability of Code Correctness in LLMs via Sparse AutoencodersKriz Tahimic and Charibeth Cheng2025≈ 80%
- Testing the Limits of Truth Directions in LLMsin corpus2026≈ 80%
- Discern Truth from Falsehood: Reducing Over-Refusal via Contrastive RefinementLin Xu, Yang Sun, Wenjun Li, Jie Shi Yuxiao Lu2026≈ 80%
- Different Paths to Harmful Compliance: Behavioral Side Effects and Mechanistic Divergence Across LLM JailbreaksMd Rysul Kabir and Zoran Tiganj2026≈ 80%
- Probe-Based Data Attribution: Surfacing and Mitigating Undesirable Behaviors in LLM Post-Trainingin corpus2026≈ 80%
- One Token Away from Collapse: The Fragility of Instruction-Tuned HelpfulnessSeyedarmin Azizi, Souvik Kundu, Massoud Pedram Erfan Baghaei Potraghloo2026≈ 80%
- What Is Actually Being Annotated? Inter-Prompt Reliability as a Measurement Problem in LLM-Based Social Science LabelingJingyuan Liu2026≈ 80%
- Refusal Steering: Fine-grained Control over LLM Refusal Behaviour for Sensitive TopicsDavid Montero, Roman Orus Iker Garc\'ia-Ferrero2026≈ 80%
- Activation Steering for Aligned Open-ended Generation without Sacrificing CoherenceMartin Zborowski, Alberto Tosato, Gauthier Gidel, Tommaso Tosato Niklas Herbster2026≈ 80%
- Detecting Non-Membership in LLM Training Data via Rank CorrelationsPranav Shetty and Mirazul Haque and Zhiqiang Ma and Xiaomo Liu2026≈ 80%
- Can You Trust an LLM with Your Life-Changing Decision? An Investigation into AI High-Stakes ResponsesSaran Subramanian Joshua Adrian Cahyono2025≈ 80%
- Failing to Falsify: Evaluating and Mitigating Confirmation Bias in Language ModelsAnthony GX-Chen, Ilia Sucholutsky, Eunsol Choi Ayush Rajesh Jhaveri2026≈ 79%
- ≈ 79%
- Would this change your answer? Evaluating Explanations of LLM Behavior In The Wild with Counterfactual Experimentsin corpus2026≈ 79%
- Persona Non Grata: Single-Method Safety Evaluation Is Incomplete for Persona-Imbued LLMsin corpus2026≈ 79%
- Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retentionin corpus2026≈ 79%
- ≈ 79%
- Open Problems in Mechanistic Interpretabilityin corpus2025≈ 79%
- Building and evaluating alignment auditing agentsin corpus2025≈ 79%
- Measuring Faithfulness in Chain-of-Thought Reasoningin corpus2023≈ 78%
- Alignment faking in large language modelsin corpus2024≈ 78%
- ≈ 78%