hypothesis
active
hypothesis:datapoint-features-form-only-if-there-is-a-repeated-datapoint-in-training-and-sparsity-regularization-is-appliedDatapoint features form only if there is a repeated datapoint in training and sparsity regularization is applied
Two necessary conditions.
Neighborhood — ranked by edge-count
Papers (1)
paper
Findings (2)
finding
- Datapoint feature formation is controlled by the product of repetition fraction and sparsity regularization strengthassociated_withsupportsHyperparameter scan result.
- Baseline toy experiment.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Property that features activate on only a small fraction of inputs; enables compressed sensing and is what allows superposition to work
- Memorization by transcoder not in model.
- Motivates Jacobian connection.
- Transcoder feature that activates on p and produces p, memorizing something the model did not.
- Playful and poetic amplification support.
- Structured sparsity question.
- Open question.
- Training distribution where special point p=[1,1,1,0,...] appears with fraction repeat_frac.