Tevet and Berant 2021 Diversity Metric Evaluation Framework

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

• Human Diversity Annotation (Likert Scale)method0.728
  Annotators score diversity of response sets 1-5 with half-point increments; used as ground truth correlation target
• Multi-Response Diversityconcept0.712
  Diversity measured over a set of m responses for a single conversation, as opposed to test set diversity
• Diversity-Relevancy Tradeoffconcept0.710
  Known tension in dialogue generation where increasing diversity may reduce response relevancy
• Confidence NLI Diversity achieves state-of-the-art performance on measuring semantic diversityclaim0.706
  Main performance claim of the paper
• # Neutral shows weak correlation with diversity (ρ=0.05 decTest, ρ=-0.08 conTest)finding0.706
  Ablation result showing neutrals are not strong indicators of diversity
• Confidence NLI Diversity achieves ρ=0.64 correlation with human diversity judgments on conTestfinding0.705
  Highest human correlation for semantic diversity metric
• What invariants enable direct comparison and recognition of truly diverse intelligences regardless of composition or origin?question0.705
  Core motivating question driving the Cognitive Light Cone framework; applies to synthetic, exobiological, and AI intelligences.
• 95% confidence intervals overlap between Baseline NLI Diversity, Confidence NLI Diversity, SentBERT, and human judgments on conTestfinding0.704
  Indicates lack of statistically significant differences between top methods