hypothesis

active

hypothesis:deep-networks-are-biased-toward-finding-simple-fits-to-the-data-and-the-bigger-the-model-the-stronger-the-bias-driving-convergence-to-a-smaller-solution-space

Deep networks are biased toward finding simple fits to the data, and the bigger the model the stronger the bias, driving convergence to a smaller solution space

Selective pressure toward convergence via implicit regularization

Source paper

extracted_from

The Platonic Representation Hypothesis

(2024) · Minyoung Huh · Brian Cheung · Tongzhou Wang · Phillip Isola

Related by similarity (8)

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

Neural networks, trained with different objectives on different data and modalities, are converging to a shared statistical model of reality in their representation spaces.quote0.804
The paper's central thesis statement, presented prominently after the abstract
Deep representations have a special significance in recurrent networks, allowing coordinated behaviour without losing sensitivity to new inputs.claim0.783
Importance of hierarchical structure for flexible coordination.
Different network depths contribute differentially to the model's capacity for handling deceptive patterns, with middle-to-late layers specializing in abstract deception semanticsclaim0.778
Interpretation of LAT scanning results showing layer-dependent deception detection accuracy
Neural networks show substantial alignment with biological representations in the brain, driven by shared task and data constraintsclaim0.774
Extends convergence argument to brain-machine alignment
Different neural network models trained on different objectives and modalities are converging to a shared statistical model of reality in their representation spaceshypothesis0.771
The central hypothesis of the paper; the platonic representation hypothesis itself
If the universality hypothesis is broadly true, it raises the exciting possibility that artificial neural networks could predict features previously unknown in biological neural networks.claim0.764
Speculative extension of universality to neuroscience, with high-low frequency detectors as a candidate prediction
Learning neural networks can enable 'chunking' and rescale problem-solving to higher organizational levels, a mechanism intrinsic to transitions in individuality.hypothesis0.763
Ultimately, we would like to understand neural networks well enough to be able to intentionally design them.quote0.756
Vision statement in the conclusion.