paper:arxiv-2009-03300Measuring Massive Multitask Language Understanding
Original abstract (expand)
We propose a new test to measure a text model's multitask accuracy. The test covers 57 tasks including elementary mathematics, US history, computer science, law, and more. To attain high accuracy on this test, models must possess extensive world knowledge and problem solving ability. We find that while most recent models have near random-chance accuracy, the very largest GPT-3 model improves over random chance by almost 20 percentage points on average. However, on every one of the 57 tasks, the best models still need substantial improvements before they can reach expert-level accuracy. Models also have lopsided performance and frequently do not know when they are wrong. Worse, they still have near-random accuracy on some socially important subjects such as morality and law. By comprehensively evaluating the breadth and depth of a model's academic and professional understanding, our test can be used to analyze models across many tasks and to identify important shortcomings.
Similar preprints — Semantic Scholar
Cited by (8)
- Understanding and Steering Llama 3 with Sparse Autoencoders - Goodfire
A sparse autoencoder trained on layer-19 residual-stream activations of Llama-3-8B-Instruct, using LMSYS-Chat-1M, yields features that can be labeled automatically, ranked by causal relevance, and use
- Forking Fast: Efficiently Estimating Uncertainty Dynamics in Text Generation
Resampling noise in reasoning-chain uncertainty dynamics is almost exactly multinomial, so most of the cost of Forking Paths Analysis re-measures structure that neighboring tokens already contain. On
- Measuring Faithfulness in Chain-of-Thought Reasoning
Chain-of-thought reasoning in a 175B RLHF-tuned assistant model is faithful only conditionally: how much the final answer depends on the stated reasoning swings from near zero to dominant by task, and
- Reasoning Models Don't Always Say What They Think
Reasoning models' chains of thought usually fail to disclose the hints that actually drive their answers: Claude 3.7 Sonnet verbalizes a used hint in only 25% of cases and DeepSeek R1 in 39%, falling
- Steering at the Source: Style Modulation Heads for Robust Persona Control
Residual-stream activation steering reliably degrades text coherency when steering vectors push models toward out-of-distribution behavior, and this collapse goes undetected by standard benchmarks: MM
- Open Character Training: Shaping the Persona of AI Assistants through Constitutional AI
Character training—fine-tuning open-weights LLMs to internalize specific personas at a depth that survives adversarial pressure—proves substantially more effective than either system-prompt constraini
- Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents
Harness-updating capability is essentially flat across model capability tiers, while harness-benefit is non-monotonic — a decoupling with direct implications for how capability budgets should be alloc
- Cattle Trade: A Multi-Agent Benchmark for LLM Bluffing, Bidding, and Bargaining
Across 242 games spanning 50–60 turns each, strategic coherence — operationalized as capital efficiency (η = score/gross outflow), resource discipline, and phase-adaptive bidding — predicts rank more