NLI Diversity

Novel metric proposed in this paper using NLI predictions to score semantic diversity of a response set

Neighborhood — ranked by edge-count

paper

method

Baseline NLI Diversity
extendsrelated_to
First variant: aggregates argmax NLI class predictions with contradiction=+1, entailment=-1, neutral=0
Neutral NLI Diversity
related_to
Variant weighting neutral predictions equally to contradictions to test if neutrals capture lexical diversity
Confidence NLI Diversity
related_to
Best-performing variant: aggregates softmax probability mass rather than binary class counts

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

NLI Diversity is limited by the scope of the NLI model's training dataclaim0.819
Limitation acknowledged in discussion section
Confidence NLI Diversity achieves state-of-the-art performance on measuring semantic diversityclaim0.818
Main performance claim of the paper
What is the effect of more generalized NLI training data on NLI Diversity performance?question0.782
Future work question arising from the finding that Combined model did not outperform MNLI model
What is the role of neutral NLI predictions in measuring semantic diversity?question0.770
Motivates the creation of Neutral NLI Diversity as an ablation
A diverse set of responses for a conversation captures contradictory ways one could respond, measurable by an NLI modelhypothesis0.767
Core hypothesis motivating the NLI Diversity metric
Neutral NLI predictions may capture lexical rather than semantic diversityhypothesis0.758
Hypothesis proposed to explain Neutral NLI Diversity's high performance on decTest but low on conTest
Semantic Diversityconcept0.748
Core concept measured by NLI Diversity — diversity of meaning across a set of dialogue responses
Confidence NLI Diversity achieves ρ=0.64 correlation with human diversity judgments on conTestfinding0.736
Highest human correlation for semantic diversity metric