claim

active

claim:features-are-connected-by-weights-forming-circuits-and-these-circuits-can-be-rigorously-studied-and-understood-as-meaningful-algorithms

Features are connected by weights forming circuits, and these circuits can be rigorously studied and understood as meaningful algorithms.

Second of three speculative claims asserting that subgraphs of neural networks are tractable and meaningful objects of study

Source paper

extracted_from

Zoom In: An Introduction to Circuits

(2020) · Chris Olah · Nick Cammarata · Ludwig Schubert · Gabriel Goh +2

Neighborhood — ranked by edge-count

Papers (1)

paper

Zoom In: An Introduction to Circuits
introduces

Findings (2)

finding

Weights between early and full curve detectors in InceptionV1 form a curve of positive weights at tangent positions, with opposing orientations inhibitory
supports
Demonstrates that meaningful algorithms can be read directly off floating-point weights in a neural network
InceptionV1 implements a four-layer circuit for pose-invariant dog head detection with mirrored left/right pathways that inhibit each other then unite, exhibiting XOR-like properties
supports
Evidence that neural networks learn sophisticated invariance mechanisms through structured circuits rather than loose feature aggregation

Claims (3)

claim

Analogous features and circuits form across models and tasks.
supports
Third of three speculative claims asserting that learned features are not model-specific but represent universal solutions to learning problems
Features are the fundamental unit of neural networks; they correspond to directions and can be rigorously studied and understood.
supports
First of three speculative claims forming the foundation of the circuits research agenda
Circuit claims are falsifiable: if you understand a circuit, you should be able to predict what changes when you edit the weights.
supports
Argument that circuits methodology meets natural-science standards of falsifiability

Questions (1)

question

No established method for resolving polysemantic neurons into pure features at scale
gates
Identified gap linking polysemanticity challenge to disentangled representations literature

Related by similarity (8)

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

The features are often organized in geometrically-related clusters that share a semantic relationship.claim0.802
Decoder cosine similarity maps onto concept similarity.
Geometry of features matters for representation quality.claim0.789
General principle supported tangentially by covariance pooling work; relates to feature co-occurrence structure.
Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models (Marks et al., 2025)concept0.788
Cited as enabling precise behavioral control through SAE features, extending the same methodological line
The likelihood of a dedicated feature for a concept (element, city, animal, food) follows a sigmoid in log-frequency of the concept in training data, with threshold frequency inversely proportional to number of alive features.finding0.785
Quantitative relationship between concept frequency and feature presence.
Individual floating-point number weights in neural networks become meaningful once you understand the features they connect.claim0.779
Interpretive claim that circuits render raw weights interpretable as algorithmic structures
"You can literally read meaningful algorithms off of the weights."quote0.777
Load-bearing claim about the tractability of circuit analysis; central thesis of Claim 2
Four features will be fundamental: (1) all processes rethought as morphogenetic; (2) sequences link into a net; (3) continuous evolution becomes widespread; (4) an ethical obligation to heal the land emerges.claim0.773
Vision of the emerging paradigm shift in society.
Representing non-linearly separable functions requires a network with multiple layers.claim0.771
Architectural requirement from machine learning.