framework
active
framework:stochastic-parameter-decomposition-spdStochastic Parameter Decomposition (SPD)
Linear parameter decomposition with stochastic ablation.
Neighborhood — ranked by edge-count
Papers (2)
paper
- Stochastic Parameter Decompositionintroducesmentions
Methods (6)
method
- Sparse Dictionary Learninganalogous_toGeneral method for finding overcomplete sparse decompositions; the paper uses sparse autoencoders as an approximation
- Small per-subcomponent MLP with hard sigmoid predicting causal importance.
- Penalizes causal importance values above 0 with p-norm; works with p>1.
- Loss on randomly masked subcomponents matching target outputs, with layerwise auxiliary.
- Avoids dead gradients in gates.
- Used to backpropagate through random masks.
Concepts (1)
concept
- The atomic pieces into which weight matrices are decomposed by VPD; each rank-one component is interpretable.
Frameworks (3)
framework
- Decomposes parameters into faithful, minimal, simple components.
- Linear parameter decompositionimplementsFramework decomposing network parameters into sum of sparsely used vectors in parameter space.
- Singular learning theoryassociated_withRelates parameters to loss via landscape geometry.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Framework for non-equilibrium free energies; mentioned as future direction to generalise results
- Core methodological framework introduced in this paper; decomposes weight matrices into rank-one interpretable subcomponents using adversarial ablations.
- Core technique introduced in this paper for decomposing neural network weight matrices into mechanistically simple, interpretable rank-one subcomponents.
- SPD decomposes W_in of compressed computation model into 100 subcomponents, one per input featurefinding0.739Neuron contributions match target.
- Central claim.
- Framing.
- Williams and Beer's decomposition of joint mutual information into unique, redundant, and synergistic components.