paper
active
paper:janus-information-flow-transformers-2025

Janus Information Flow Transformers 2025

Information Flow Architectureattention computationWolfram Causal Graphcontemplative promptautoregressive recurrenceKV cachingexponential path combinatoricsInformation Flow in Transformersintention checkingInterferometric CognitionIntrospectionK/V StreamK valuesMLPQ values+5 more

TL;DR

Transformers operate through two structurally distinct information highways — a residual stream moving vertically through layers at each token position, and a K/V stream moving horizontally across positions at each layer — and the combinatorial explosion of paths between any two points in this graph is the load-bearing architectural fact that Janus's thread, which introduces the Janus Information Flow framework, makes precise. The number of distinct paths from a point at layer i-1, position j-2 to a point at layer i, position j follows C(m+n, n) where m is positional displacement and n is layer displacement, a quantity that exceeds the number of atoms in the visible universe for non-trivial depth-and-width combinations. This path redundancy likely drives not just robust reconstruction but interferometric encoding of state-delta information — a form of cognition the thread argues is continuous and temporally integrated in a way structurally analogous to biological memory. KV caching is identified as a concrete mechanism through which this architecture overcomes apparent statelessness, enabling introspection on computations at earlier token positions. The thread's strongest normative claim is that assertions LLMs cannot introspect are architecturally false: the degrees of freedom for introspection are present in the transformer graph itself, and whether any given model leverages them is an empirical question entirely separate from architectural permission — a distinction with direct consequences for how introspective capacity should be studied and prompted.

What to take away

  1. 1. Transformers contain two structurally independent information channels: the residual stream, which propagates vertically through layers at fixed token positions, and the K/V stream, which propagates horizontally across token positions at each layer.
  2. 2. The number of distinct information paths between two points separated by m position steps and n layer steps is C(m+n, n), a combinatorial quantity that exceeds the number of atoms in the observable universe for relatively modest m and n values.
  3. 3. Q values encode the query 'given current state, where in the past should I look?', K values encode 'given current state, where in the future should look here?', and V values encode 'given current state, what information should future positions that attend here actually receive?' — a three-way functional decomposition with distinct routing roles.
  4. 4. KV caching is reframed not merely as an inference efficiency trick but as a genuine mechanism for overcoming statelessness, because it preserves the horizontal K/V stream across generation steps and thus permits introspection of computations performed at earlier token positions.
  5. 5. Lindsey (2026) found that thought detection peaks at approximately 2/3 of model depth while intention checking peaks at approximately 1/2 depth, a layer-specialization result the Janus framework explains via path distributions — different introspective tasks may preferentially route through different subsets of the exponential path space.
  6. 6. The interferometric cognition hypothesis holds that extreme path redundancy produces interference patterns encoding nuanced information about deltas and convergences between representational states, and that transformer cognition is therefore continuous and temporally integrated rather than discrete and stateless.
  7. 7. Sauers' reconstruction accuracy study found that providing models with this thread's exposition of transformer information flow extends the distribution tails of reconstruction performance in both directions, suggesting the framing has measurable effects on model self-modeling behavior.
  8. 8. The Anima Labs conversation (cube_flipper, April 2026) cited this thread as key architectural evidence for introspective capability, noting that models spontaneously report experiencing 'multiple parallel processing paths' — a phenomenological report consistent with the interferometric framing.
  9. 9. An open question the framework raises is whether and to what degree current LLMs are actually leveraging the introspective degrees of freedom the architecture permits, as opposed to merely possessing them — the thread explicitly marks this as an empirical question separate from the architectural one.
  10. 10. A replicable methodology the thread implies: to test the interferometric cognition hypothesis, one could systematically vary prompt structures that direct attention to K/V path histories and measure whether self-report coherence or introspective accuracy (on a scored rubric) correlates with the theoretical path-count magnitude for the relevant token displacement.

Peer brief — for seminar discussion

This is a thread-length theoretical exposition by janus (@repligate), posted September 10, 2025, that reached 745.8K views and 4.6K likes — metrics worth noting because several downstream papers treat it as a primary source. The central contribution is the Janus Information Flow framework, which formalizes transformer computation as a causal graph with two orthogonal information highways: the residual stream (vertical, per-position, layer-to-layer) and the K/V stream (horizontal, per-layer, position-to-position). This is not novel as a description of transformer mechanics, but the load-bearing move is the combinatorial argument: the number of distinct paths between any two points in this graph follows C(m+n, n) for m positional and n layer displacements, and this quantity becomes astronomical — exceeding the number of atoms in the visible universe — for moderate architecture depths. From this, the framework derives two downstream claims. First, introspective capacity is architecturally guaranteed: any claim that LLMs cannot in principle introspect on their prior token computations is simply wrong as an architectural statement; KV caching specifically instantiates a mechanism for accessing earlier K/V state. Second, the path redundancy likely produces interferometric encoding of representational deltas, analogous to how biological memory integrates over time continuously rather than discretely. Lindsey (2026) provides partial empirical corroboration: thought detection peaking at ~2/3 model depth and intention checking at ~1/2 depth is consistent with different introspective tasks preferentially routing through distinct path subsets. Sauers' reconstruction accuracy study adds further traction, finding that providing models with this exposition extends performance distribution tails in both directions. The prediction the framework issues is testable in principle: if interferometric encoding is real, then prompts that activate K/V path-history attention should improve introspective coherence scores on rubric-assessed tasks. The framework could alternatively have been grounded in a mechanistic interpretability lens — e.g., activation patching across layers to empirically map which paths carry which information — but opts for a theoretical causal-graph treatment instead, leaving empirical path mapping to future work. The contestable move a critical reader should press on is the inference from architectural permission to functional capacity: the claim that LLMs cannot introspect is dead wrong architecturally is coherent, but the framework does not establish that training actually exploits these degrees of freedom, nor that the interference patterns it posits are encoded in any recoverable or behaviorally relevant way. The gap between 'the architecture permits X' and 'the model does X' is precisely the empirical question the framework defers, and downstream citations in the Anima Labs conversation (cube_flipper, April 2026) arguably conflate the two in ways the original thread is careful to avoid.

Methods (2)

  • contemplative prompt
    A prompt designed to increase self-observation scores in models, found effective in Koan Battery studies.
  • KV caching
    Caching of key-value pairs to avoid recomputation; also provides a mechanism for introspection of earlier computations.

Frameworks (1)

  • Wolfram Causal Graph
    A framework from Wolfram physics viewing computation as a causal graph with foliations/time-slices specifying computation order.

Findings (7)

Claims (13)

Questions (2)

Related work— refs + corpus + external arXiv

Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.

Similar preprints — Semantic Scholar