The more descriptive (higher information) a caption is, the better its LLM representation aligns with the visual representation of the corresponding image

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

• Higher information (denser) captions should yield higher language-vision alignment scoreshypothesis0.831
  Tests the information-level cap on cross-modal alignment
• The better an LLM is at language modeling, the more it aligns with vision models, and vice versa — linear relationship between language modeling score and vision-language alignmentfinding0.803
  Core cross-modal empirical result: larger and better language models align better with vision models
• Better LLMs (measured by 1-bits-per-byte on OpenWebText) show a linear relationship with alignment to vision models measured via mutual nearest-neighbor on WITfinding0.787
  Key cross-modal alignment result
• LLMs trained only on language data have rich enough knowledge of visual structures that decent visual representations can be trained on images generated solely by querying the LLMfinding0.781
  Sharma et al. result supporting cross-modal alignment: language-only models implicitly encode visual structure
• LLM representations exhibit intriguing patterns under spatio-permutational analyses, suggesting a potentially profound yet tentative indication of consciousness.claim0.773
  Qualified positive claim from spatio permutation analysis where two cases satisfy all three criteria.
• LLMs linearly represent truth-relevant information beyond the plausibility of text, as evidenced by probes trained on likely performing poorly on anti-correlated datasetsclaim0.772
  Establishes that the observed linear structure is not merely a representation of text probability
• What is the relationship between different dimensions or clusters of dimensions in LLM representations? Do they and/or how do they interact with each other?question0.769
  Motivates the RN hypothesis by pointing to the unknown relational structure within high-dimensional representation vectors.
• A single linear projection is sufficient to stitch a vision model to an LLM and achieve good performance on visual question answering and image captioningfinding0.766
  Merullo et al. result on cross-modal representational compatibility