concept
active
concept:virtual-weightsVirtual Weights
Implicit weights directly connecting any pair of layers computed by multiplying output weights of one layer with input weights of another through the residual stream
Neighborhood — ranked by edge-count
Papers (1)
paper
Frameworks (1)
framework
- Virtual weight model (VW model)implementsRewriting of the 1L transformer into six families of virtual weights over tokens, positions, features, logits.
Claims (1)
claim
- Architectural observation enabling the entire mathematical framework; the residual stream is purely a sum of linear projections
Concepts (2)
concept
- Residual Streamassociated_withProposed pathway flowing through layers at each position; calculates K/V values that feed horizontal information flow.
- Polysemanticityassociated_withNeurons that respond to multiple unrelated concepts, limiting interpretability.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Used to project features through QK weights and find aligned features.
- Helpful and harmful weights are scattered across the entire range of virtual weight magnitudesfinding0.772IN Tokens→Logits shows weak VW-helpfulness relationship.
- Large virtual weights are not inherently misleading but not guaranteed to be helpful or effectiveclaim0.754Caution for practitioners.
- Contrast with answer selection.
- A high-level control node (e.g., calcium pattern, resting potential) that is a complex function of many low-level components, not 1:1 mappable to genes.
- Naive weight magnitude works here.
- The baseline variant of Verbalized Sampling that asks for k responses with their probabilities in a single LLM call
- A computing layer above physical hardware; Sloman uses this as an analogy for mental processes not reducible to physics.