finding
active
finding:sae-training-loss-decreases-as-a-power-law-with-compute-budget-when-using-compute-optimal-hyperparametersSAE training loss decreases as a power law with compute budget when using compute-optimal hyperparameters.
From scaling laws sweep.
Source paper
extracted_from(2024) ·
Neighborhood — ranked by edge-count
Claims (1)
claim
- Compute-optimal hyperparameters follow predictable power-law relationships.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- The objective function combining L2 reconstruction error and L1 penalty scaled by decoder norm, used to train the SAE.
- Hyperparameter trend observed.
- Ethical implication about the nature of AI training experience if the thesis holds
- A promising property for interpretability analysis off-distribution.
- Optimal number of features scales faster than optimal number of training steps with compute budget.finding0.764Allocation result from scaling laws.
- Lower SAE loss at fixed sparsity likely indicates less interference and less superposition.claim0.763Justifies SAE proxy.
- Robustness influences superposition, predicting lower SAE loss in more robust models.hypothesis0.761Popperian credit for surprising prediction.
- Key methodological contribution claim about architecture-agnostic SAE tuning