Alignment with vision models corresponds to improved performance on downstream language tasks including commonsense reasoning and math

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

• CLIP models exhibit higher language-vision alignment than supervised or self-supervised vision models, but this alignment decreases after fine-tuning on ImageNet classificationfinding0.820
  CLIP training paradigm finding in cross-modal alignment
• The better an LLM is at language modeling, the more it aligns with vision models, and vice versa — linear relationship between language modeling score and vision-language alignmentfinding0.819
  Core cross-modal empirical result: larger and better language models align better with vision models
• Jointly training a language model with a vision model improves performance on language tasks compared to training the language model alonefinding0.818
  OpenAI GPT-4V finding supporting cross-modal training benefit
• A vision model trained on ImageNet can be aligned with a model trained on Places-365 while maintaining good performance, and early layers are more interchangeable than later layersfinding0.806
  Lenc & Vedaldi result illustrating data independence in representations and layer-wise alignment
• Among 78 vision models, those solving more VTAB tasks (higher transfer performance) show higher mutual nearest-neighbor alignment with each otherfinding0.802
  Key empirical finding establishing that representational alignment correlates with model competence
• Higher information (denser) captions should yield higher language-vision alignment scoreshypothesis0.798
  Tests the information-level cap on cross-modal alignment
• Olah et al. (2020) found that automatically trained computer vision models, regardless of architecture and training procedure, all arrive at similar functional structures organizing similar features into similar compositional hierarchies, closely resembling the primate visual cortex.finding0.797
  Empirical finding supporting the Universality Hypothesis; extended by the paper to consciousness
• Diverse computer vision models trained on visual recognition tasks converge to remarkably similar internal feature representations regardless of architecture, training procedure, or implementation details, closely matching the organization of animal visual cortexfinding0.793
  Empirical evidence for the universality hypothesis cited as supporting the possibility of convergent consciousness-like solutions