concept
active
concept:mixture-of-experts-moeMixture-of-Experts (MoE)
Architecture of Mixtral-8x7B; uses sparse expert routing affecting how hidden states are computed across layers.
Neighborhood — ranked by edge-count
Papers (1)
paper
- Toy Models of Superpositionmentions
Concepts (2)
concept
- The primary paper being extracted — applies IIT 3.0 and 4.0 to LLM representation sequences derived from ToM test data to investigate whether consciousness phenomena can be observed.
- Mixtral-8x7BimplementsOne of four LLMs selected; Mixture-of-Experts model; had substantial sample loss under IIT 4.0 due to PyPhi network initialization issues.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Architecture where standard MLP is replaced with multiple subnetworks with a router; considered as possible challenge to psychological continuity
- Mixture-based subspace featurizer; one-active-block special case.
- Selection strategy that adapts which demonstrations carry signal; in UCCT terms increases effective ρd
- Program that supported Tim Hua and Andrew Qin during this research.
- Program under which work was conducted.
- Stated limitation.
- Towards Monosemanticity: Decomposing Language Models with Dictionary Learning (Bricken et al., 2023)concept0.730Foundational SAE mechanistic interpretability paper
- Interpolating two concepts' embeddings to test ignition.