finding
active
finding:inserting-a-16m-latent-sparse-dictionary-into-gpt-4-yields-language-modeling-loss-equivalent-to-a-model-with-10-of-gpt-4-s-pretraining-compute

Inserting a 16M-latent sparse dictionary into GPT-4 yields language modeling loss equivalent to a model with 10% of GPT-4's pretraining compute

Shows SDL reconstruction error is too high.

Source paper

extracted_from
Open Problems in Mechanistic Interpretability
(2025) · Lee Sharkey · Bilal I. Chughtai · Joshua Batson · Jack W. Lindsey +25

Neighborhood — ranked by edge-count

Related by similarity (8)

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.