Learning cooccurrence statistics in either vision or language domain recovers roughly the same perceptual color representation

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

• Color distances learned from language cooccurrence statistics closely mirror those learned from image cooccurrence statistics and human perceptual distances (CIELAB)finding0.883
  Case study confirming that PMI-based learning in different modalities recovers the same perceptual representation
• Diverse computer vision models trained on visual recognition tasks converge to remarkably similar internal feature representations regardless of architecture, training procedure, or implementation details, closely matching the organization of animal visual cortexfinding0.783
  Empirical evidence for the universality hypothesis cited as supporting the possibility of convergent consciousness-like solutions
• Alignment with vision models corresponds to improved performance on downstream language tasks including commonsense reasoning and mathclaim0.771
  Claims that alignment score is a proxy for general capability
• Olah et al. (2020) found that automatically trained computer vision models, regardless of architecture and training procedure, all arrive at similar functional structures organizing similar features into similar compositional hierarchies, closely resembling the primate visual cortex.finding0.763
  Empirical finding supporting the Universality Hypothesis; extended by the paper to consciousness
• Imagine reading a textbook with no figures or tables. Our ability to knowledge acquisition is greatly strengthened by jointly modeling diverse data modalities, such as vision, language, and audio.quote0.755
  Load-bearing motivation for multimodal approach; frames the cognitive advantage of joint modalities.
• Language models would achieve some notion of grounding in the visual domain even in the absence of cross-modal training data, because they share a common modality-agnostic representationhypothesis0.753
  Implication of PRH for language model visual grounding
• Number of local symmetries correlates almost perfectly with perceived cognitive coherence across 35 strip patternsfinding0.752
  The key experimental finding: the number of subsymmetries (locally symmetrical connected segments) in a pattern predicts its perceived coherence; most coherent strips have 9 subsymmetries, least coherent have 5; the measure correlates almost perfectly with combined experimental rank order
• Jointly training a language model with a vision model improves performance on language tasks compared to training the language model alonefinding0.752
  OpenAI GPT-4V finding supporting cross-modal training benefit