Causal masking phase transitions in transformers

Studies how decoder-only architectures lack ordered phases necessary for coherent long-sequence generation due to causally-masked attention constraints.

2 members. Each node is clickable.

Loading graph…

Drawn from 1 source

The papers/notes whose extracted claims & findings make up this cluster.

Topological constraints on self-organisation in locally interacting systems2 members

Bridges (3)

Other communities that share members with this one — cross-cutting threads or papers that sit at the seam between two themes.

Mechanistic interpretability & model evaluation2 shared
Mechanistic structure of transformer attention computations2 shared
Causal masking & phase transitions1 shared

Claims (1)

Decoder-only transformer architectures are fundamentally limited in generating long, coherent sequences due to lack of ordered phase.Interpretation of Proposition 2 as a fundamental limitation on LLMs

Findings (1)

Causally-masked attention in a decoder-only model has no ordered phase (Proposition 2)Application to transformer language models