Aston Zhang

Authored papers (1)

• Multimodal Chain-of-Thought Reasoning in Language Models2023ⓒ 97

  Incorporating visual features into chain-of-thought rationale generation—rather than answer generation alone—breaks the hallucination bottleneck that causes sub-100B language models to fail at multimodal reasoning. The root problem, diagnosed on ScienceQA, is that a text-only two-stage baseline achieves a RougeL of 90.73 on rationale generation yet only 78.57% answer accuracy, underperforming direct answering (81.63%), because 56% of its errors stem from hallucinated rationales that lack visual grounding. Multimodal-CoT addresses this by fusing frozen ViT-large patch features into a T5 encoder-decoder via a gated cross-attention mechanism, separating rationale generation (stage 1) from answer inference (stage 2) while conditioning both on vision signals. Adding vision features raises rationale RougeL to 93.46% and answer accuracy to 85.31% at the 223M Base scale, and Multimodal-CoTLarge (738M) reaches 90.45% on ScienceQA—surpassing the prior best published result of 86.54% (Chameleon+GPT-4) while using orders of magnitude fewer parameters than GPT-4, LLaVA-13B, or InstructBLIP-11B. On the MMMU generalization benchmark, the 738M model scores 28.7%, matching OpenFlamingo-2 at 9B parameters. The paper argues that vision-grounded rationale generation is not merely complementary to scaling but is a structurally distinct lever: hallucination correction rate reaches 60.7% with vision features, and convergence is faster than text-only variants at every epoch, implying that multimodal feature fusion during the rationale stage should be a standard component of any CoT pipeline operating below the 100B parameter regime.

More papers — OpenAlex / S2

Recent mentions (1)

• papers-typed

  zhang-2023-multimodal.md