Typicality bias in preference data is a fundamental and pervasive data-level cause of mode collapse in aligned LLMs

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

• Typicality bias rate exceeds 50% chance baseline by 4-12 percentage points across all base models on all four preference datasetsfinding0.812
  Systematic evidence that base models implicitly prefer human-preferred responses, indicating preference biases emerge during pretraining
• What is the fundamental cause of mode collapse in post-training aligned LLMs?question0.812
  The central question the paper addresses, answered by identifying typicality bias as a data-level driver
• Typicality bias with weight α>0 mathematically sharpens the aligned policy via a power transform of the reference distributionclaim0.798
  Theoretical result showing that any positive typicality bias weight γ-sharpens the reference distribution, amplifying modes
• Typicality bias rates in instruction-tuned models remain at similar or higher levels compared to their base model counterpartsfinding0.786
  Shows typicality bias is preserved through instruction tuning and RLHF, not introduced by alignment
• Typicality bias is present across all annotator demographic groups but varies in magnitude, potentially reflecting differences in linguistic norms captured by pretraining dataclaim0.785
  Finding from PRISM dataset analysis showing typicality bias varies by ethnicity and region, with implications for fairness
• Typicality Bias Ratemethod0.773
  Measurement of how often human annotators prefer the response with higher base model log-probability
• Synthetic document fine-tuning causes no degradation in preference model score on benign queriesfinding0.766
  Rules out that observed effects are due to general model damage rather than learned situational awareness
• Typicality Biasconcept0.765
  The human tendency to prefer more typical, familiar, fluent, and predictable text in annotation tasks, identified as a fundamental data-level cause of mode collapse