CoT boosts 2-digit ID accuracy but often worsens 3-4 digit OOD

Scope generalization results after LoRA+CoT fine-tuning

Source paper

extracted_from

The Guanyin Protocol: A Framework for Immediately Establishing an Understanding of Both Causality and Compassion in LLM Systems Using Semantic Anchoring

(2025) · Edward Yi Chang · Kaya, Zeyneb N. · Ethan Chang

Neighborhood — ranked by edge-count

Claims (1)

claim

CoT improves in-distribution but may harm out-of-distribution generalization
supports
Interpretation of scope generalization results

Communities (2)

community

Chain-of-Thought reasoning robustness & safety
members_of
CoT effects on generalization, multimodal QA accuracy, and AI safety alignment training.
Chain-of-thought generalization trade-offs
members_of
Empirical studies showing CoT reasoning improves ID performance while harming OOD generalization, with probability calibration as a mitigation strategy.

Related by similarity (8)

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

Scope generalization: CoT boosts 2-digit in-distribution but worsens 3-4 digit OODfinding0.843
CoT increases dr for OOD operands.
Multimodal-CoT with vision features achieves higher validation accuracy at early training epochs (epoch 1-3) compared to one-stage and two-stage language-only baselines on ScienceQAfinding0.743
Evidence that multimodal information accelerates convergence speed during training.
Multimodal-CoT trained with InstructBLIP/ChatGPT-generated rationales achieves 87.76% accuracy on ScienceQA, comparable to human-annotated rationale performance of 90.45%finding0.743
Evidence that Multimodal-CoT can operate without human-annotated reasoning chains by using large models to generate pseudo-rationales.
Clamping CoT probabilities to 40-60% range for RL-CAI with CoT improves robustness and reduces extreme responses.finding0.731
Section 4.3 describes clamping at 40-60 led to better behavior than clamping at 20-80.
Short rationales (LoRA+CoT) sometimes improve in-distribution performance but do not reliably reduce cross-base harmfinding0.713
E2 finding showing CoT's limited benefit for OOD transfer, consistent with larger dr out of scope
RL-CAI with CoT shows a Pareto improvement in helpfulness-harmlessness tradeoff over standard RLHF, with slight helpfulness decrease but higher harmlessness.finding0.711
Figure 2 and Figure 8 illustrate RL-CAI at the Pareto frontier.
Task difficulty moderates whether CoT is performative or genuine: easy recall questions show performative CoT, difficult multihop questions show genuine reasoningclaim0.710
Task difficulty as the key variable distinguishing the two modes of CoT identified in the paper
One-stage CoT (QCM→RA) shows 12.31% accuracy drop vs. no-CoT (QCM→A) on ScienceQA; two-stage framework (rationale generation + answer inference) achieves 85.31% accuracy with vision featuresfinding0.707
Empirical evidence that naive one-stage CoT fails in language-only setting; two-stage + vision achieves state-of-the-art.