concept
active
concept:dead-neurons

Dead Neurons

Autoencoder neurons that fail to activate across any datapoints during training; addressed via neuron resampling

Neighborhood — ranked by edge-count

Methods (1)

method
  • Neuron Resampling
    associated_with
    Periodically reinitializing dead autoencoder neurons using high-loss data points to improve feature coverage

Concepts (1)

concept
  • Feature Density
    associated_with
    Fraction of training tokens on which a given feature has nonzero activation; used as proxy metric for autoencoder quality

Related by similarity (8)

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

  • MLP neurons and attention heads hypothesized to delete information from the residual stream by writing the negative of what they read
  • MLP neuronsconcept0.732
    The sparse set of 28 neurons at layer 18 identified as responsible for Fourier feature computation across all cyclic tasks
  • Working definition adopted by the paper.
  • Biologically plausible two-pool architecture from Krotov & Hopfield (2020) splitting self-attention into feature and memory neuron populations; used to interpret TEM-t place cells.
  • Neuroscienceinstitute0.725
  • Solid Brainsconcept0.725
    Conventional brains where neurons maintain relatively fixed positions; a subset of collective intelligences.
  • Neural Networksconcept0.722
  • Rosetta Neuronsconcept0.714
    Neurons activated by the same pattern across a range of vision models, forming a common dictionary independently discovered by all models