Hopfield Networks is All You Need (Ramsauer et al., 2020)

Paper showing Hopfield networks are closely related to transformers; key intermediary result used to connect TEM to transformers.

Neighborhood — ranked by edge-count

Papers (1)

paper

Relating transformers to models and neural representations of the hippocampal formation
cites

Frameworks (1)

framework

Hopfield Network
cites
A recurrent connectionist architecture that implements associative memory and distributed computation through symmetric weighted connections and Hebbian learning rules. The network converges to stable states through recurrent dynamics, enabling both memory retrieval and combinatorial problem-solving in a fully distributed manner.

Concepts (1)

concept

Self-attention
cites
A form of key-query attention within a single input sequence; core to Transformers.

Related by similarity (8)

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

Hopfield network can store multiple patterns, recall them via content-addressable memory, and generalise to novel patterns with the same underlying structure.finding0.797
Summary of known Hopfield network capabilities used as a model for collective computation.
Neural networks and physical systems with emergent collective computational abilities (Hopfield, 1982)concept0.720
Original Hopfield network paper; the attractor dynamics in TEM memory retrieval are a continuous version of this.
Hopfield Network Memory Capacity Scalingconcept0.717
Technical issue: original Hopfield networks scale linearly with neuron count; exponential activations enable 2^(N/2) scaling but softmax used in TEM-t has intermediate properties.
Representing non-linearly separable functions requires a network with multiple layers.claim0.697
Architectural requirement from machine learning.
The feed-forward network truly implements a symbolic, tree-structured algorithm for hierarchical equality, with abstract equality relations not decomposable into input identitiesclaim0.696
DAS reveals that the neural network encodes abstract relational structure rather than raw input identities.
Bioelectric networks, with their inherent plasticity, multiscale historicity, and learning capacity, are an ideal kind of 'cognitive glue' that binds the primitive goal-directedness of single cells into a higher order system.concept0.692
Canonical definition of the paper's central concept; encapsulates mechanism of cognitive scaling through bioelectric integration.
Ultimately, we would like to understand neural networks well enough to be able to intentionally design them.quote0.691
Vision statement in the conclusion.
What kinds of functional relationships and interaction structures are necessary to turn a collective into an intelligent system?question0.690