finding
active
finding:language-model-parameter-scaling-exponents-0-07-kaplan-and-0-34-hoffmannLanguage model parameter scaling exponents 0.07 (Kaplan) and 0.34 (Hoffmann)
Speculative α estimate.
Source paper
extracted_from(2025) · Michaud, Eric J. · Gorton, Liv · McGrath, Tom
Neighborhood — ranked by edge-count
Papers (1)
paper
Claims (1)
claim
- Uncertain α and β.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Open problem.
- Large language models develop surprisingly coherent yet often rigid internal preferences as they scalefinding0.766Mazeika et al. finding reinforcing the need for emptiness-based flexible value architectures
- Simple interpretability methods scale to 2.8T-parameter models and long-context agentic rolloutsclaim0.757Scaling claim.
- Core empirical hypothesis of the paper, supported by successful VPD decomposition yielding ~10,000 interpretable subcomponents across 24 weight matrices.
- Limits of verbalized probability calibration when corpus frequency and perceived popularity diverge
- Limitation identified in Discussion; all models fine-tuned are <10B parameters
- A better basis (e.g., factoring tokens into languages or parts of speech) may yield a sparse interpretable modelhypothesis0.739Same model can be dense in one basis and sparse in another.
- Articulates why a one-layer transformer with MLP is the appropriate starting target for mechanistic interpretability