community
active
leiden_hybrid_concepts
label: haiku
community:leiden_hybrid_concepts-run4-c0-c0-c6Causal masking phase transitions in transformers
Studies how decoder-only architectures lack ordered phases necessary for coherent long-sequence generation due to causally-masked attention constraints.
2 members. Each node is clickable.
Loading graph…
Drawn from 1 source
The papers/notes whose extracted claims & findings make up this cluster.
Bridges (3)
Other communities that share members with this one — cross-cutting threads or papers that sit at the seam between two themes.
Claims (1)
- Decoder-only transformer architectures are fundamentally limited in generating long, coherent sequences due to lack of ordered phase.Interpretation of Proposition 2 as a fundamental limitation on LLMs
Findings (1)
- Causally-masked attention in a decoder-only model has no ordered phase (Proposition 2)Application to transformer language models