finding
active
finding:optimal-number-of-features-scales-faster-than-optimal-number-of-training-steps-with-compute-budgetOptimal number of features scales faster than optimal number of training steps with compute budget.
Allocation result from scaling laws.
Source paper
extracted_from(2024) ·
Neighborhood — ranked by edge-count
Claims (1)
claim
- Compute-optimal hyperparameters follow predictable power-law relationships.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Hyperparameter trend observed.
- Quantitative relationship between concept frequency and feature presence.
- Selective pressure toward convergence via task generality
- what is the 'correct number of features' for dictionary learning, and is this question well-posed?question0.772Open question about whether there is a true discrete feature count or a continuous splitting process
- Central interpretation.
- How can the stability and variance of feature discovery across training runs be improved?question0.768Future work.
- Motivating question.
- SAE training loss decreases as a power law with compute budget when using compute-optimal hyperparameters.finding0.764From scaling laws sweep.