Mixture-of-Experts (MoE)

Architecture of Mixtral-8x7B; uses sparse expert routing affecting how hidden states are computed across layers.

Neighborhood — ranked by edge-count

concept

Can 'Consciousness' Be Observed from Large Language Model (LLM) Internal States? Dissecting LLM Representations Obtained from Theory of Mind Test with Integrated Information Theory and Span Representation Analysis
mentions
The primary paper being extracted — applies IIT 3.0 and 4.0 to LLM representation sequences derived from ToM test data to investigate whether consciousness phenomena can be observed.
Mixtral-8x7B
implements
One of four LLMs selected; Mixture-of-Experts model; had substantial sample loss under IIT 4.0 due to PyPhi network initialization issues.

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

Mixture-of-Experts architectureconcept0.807
Architecture where standard MLP is replaced with multiple subnetworks with a router; considered as possible challenge to psychological continuity
MOICL: Mixtures of In-Context Learnersframework0.755
Selection strategy that adapts which demonstrations carry signal; in UCCT terms increases effective ρd
ML Alignment & Theory Scholars (MATS)institute0.741
Program that supported Tim Hua and Andrew Qin during this research.
Towards Monosemanticity: Decomposing Language Models with Dictionary Learning (Bricken et al., 2023)concept0.730
Foundational SAE mechanistic interpretability paper
A naive agent equipped with reduced priors from an experienced agent performs perfectly with maximum confidence from the first trial.finding0.717
Demonstration that model-level priors (not parameter-level knowledge) suffice for immediate transfer
ensemble of agentsconcept0.715
Probabilistic behaviour of an ensemble used to derive the free-energy principle.
monosemanticityconcept0.710
Interpretability property where a latent feature represents a single semantic concept; benchmarked across architectures.
Can we disambiguate truth from closely related features such as 'commonly believed' or 'verifiable'?question0.709
Limitation noted in §7.1: scope restricted to simple statements prevents disambiguation