method
active
method:auroc-rollout-budget-evaluationAUROC rollout-budget evaluation
Classifies prompts as complying at least once in 100,000 rollouts vs never, per rollout budget.
Neighborhood — ranked by edge-count
Papers (1)
paper
- Predicting Rare LLM Failures with 30× Fewer Rolloutscitesmentions
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Alternative phrase comparison.
- Performance metric for binary classification; used to evaluate pathogenicity prediction.
- Logit Path Extrapolation beats importance sampling AUROC at equal rollout budgets on Qwen 3 4Bfinding0.761Outperforms the prior state of the art.
- Robustness check.
- Averages AUROC over 500 simulations drawing 1,000 precomputed rollouts per α.
- 10 rollouts per α for search plus 100 for fit matches naive sampling AUROC with 28× fewer rolloutsfinding0.710Split search/fit result.
- Fixed-budget variant result.
- Shows that explicit instructions delay the emergence of truth directions in arithmetic tasks.