paper
active
2022
paper:doi-10-48550-arxiv-2205-01497

Semantic Diversity in Dialogue with Natural Language Inference

ByKatherine Stasaski·Marti A. HearstⓘUC Berkeley
DOI 10.48550/arxiv.2205.01497arXiv 2205.01497OpenAlex W4229018334
Chitchat Dialogue GenerationTevet and Berant 2021 Diversity Metric Evaluation FrameworkBaseline NLI DiversityAdversarial NLIContradiction PredictionBert-STSconTestDiversity ParameterBERTScoreDailyDialog++Diversity-Relevancy TradeoffBLEU ScoredecTestDiversity Threshold+25 more

TL;DR

Confidence NLI Diversity achieves state-of-the-art Spearman's ρ of 0.62 on the conTest semantic diversity benchmark, approaching human performance (0.63) and outperforming the prior best automatic metric Sent-BERT (0.59), by aggregating probability mass from a RoBERTa-large model's softmax output over NLI classes rather than hard argmax predictions. The paper introduces two artifacts: the NLI Diversity metric family—Baseline, Neutral, and Confidence variants—and Diversity Threshold Generation, an iterative resampling procedure that discards the lowest-contributing response and replaces it until a target contradiction count (e.g., 10 out of 20 pairwise comparisons) is met. Applied to DialoGPT (700M parameters) and BlenderBot 1.0 (300M parameters) on DailyDialog++ and EmpatheticDialogues, Diversity Threshold Generation yields an average 137% increase in NLI Diversity scores while leaving relevancy virtually unchanged: BERTScore shifts by an average of 0.08% and BLEU by 1.1%. An ablation over prediction categories confirms the directional logic: # Contradiction correlates positively with human diversity judgments (ρ = 0.46 on conTest), # Entailment negatively (ρ = −0.65), and # Neutral near zero (ρ = −0.08). The paper argues this implies that framing crowdworker instructions around generating contradictory rather than merely diverse responses could improve dataset construction, and that Diversity Threshold Generation provides a model-agnostic probe for assessing any dialogue system's capacity to produce semantically varied outputs.

What to take away

  1. 1. Confidence NLI Diversity, which aggregates softmax probability mass instead of hard class predictions from a RoBERTa-large-MNLI model, achieves Spearman's ρ = 0.62 on the conTest semantic diversity benchmark, matching human performance (0.63) and exceeding the prior best automatic metric Sent-BERT (ρ = 0.59).
  2. 2. Diversity Threshold Generation—an iterative procedure that identifies and replaces the lowest-scoring response in a set until a target NLI contradiction count is reached—produces an average 137% increase in NLI Diversity scores across DialoGPT and BlenderBot 1.0 on DailyDialog++ and EmpatheticDialogues.
  3. 3. Relevancy is preserved by Diversity Threshold Generation: BERTScore changes by an average of only 0.08% and BLEU by 1.1% from starting to ending response sets on DailyDialog++.
  4. 4. An ablation on prediction counts shows # Contradiction is positively correlated with human diversity judgments (ρ = 0.46 on conTest, ρ = 0.26 on decTest), while # Entailment is negatively correlated (ρ = −0.65 on conTest), and # Neutral shows near-zero correlation (ρ = −0.08), validating the metric's design logic.
  5. 5. Neutral NLI Diversity performs best on decTest (ρ = 0.72) but worst on conTest (ρ = 0.24), indicating that weighting neutral predictions equally to contradictions captures lexical rather than semantic diversity.
  6. 6. DialoGPT (700M parameters) starts with substantially higher NLI Diversity than BlenderBot 1.0 (300M parameters) across both datasets; BlenderBot's starting NLI Diversity is negative on both DailyDialog++ (−5.55) and EmpatheticDialogues (−8.90), indicating a predominance of entailment predictions in its initial sampled responses.
  7. 7. Beam search generates semantically less diverse responses than nucleus sampling (p = 0.9): using Diversity Threshold Generation on DailyDialog++, beam search reaches an ending NLI Diversity of only 5.35 versus substantially higher values under nucleus sampling, requiring an average 10.97 resampled utterances.
  8. 8. An open question raised is whether NLI models trained on more diverse corpora (e.g., the Combined RoBERTa-large model trained on MNLI+SNLI+FEVER+ANLI) would outperform the MNLI-only model for diversity measurement, since the Combined model did not surpass MNLI in this work despite broader training data.
  9. 9. To replicate the Diversity Threshold Generation setup: generate 5 responses per conversation using nucleus sampling (p = 0.9), set divthresh to 10 contradictions out of 20 pairwise NLI comparisons, discard the response whose removal maximizes diversity score, resample up to S = 20 total attempts, and average results over 10 trials for stability.
  10. 10. The evaluation framework from Tevet and Berant (2021) using conTest (200 human-generated conversations, 5 responses each) and decTest (200 model-generated conversations, 10 responses each) for the respGen dialogue task drawn from Reddit provides the benchmark on which NLI Diversity is validated; Confidence NLI Diversity's 95% confidence interval overlaps with both Baseline NLI Diversity, Sent-BERT, and human judgments on conTest, meaning differences among top metrics are not statistically significant.

Peer brief — for seminar discussion

Stasaski and Hearst address the problem of measuring and improving the semantic diversity of sets of dialogue model responses—distinct from test-set diversity over a full corpus—by introducing two linked contributions: the NLI Diversity metric family and Diversity Threshold Generation. Working within the Multi-Response Diversity evaluation paradigm of Tevet and Berant (2021), where a model generates m responses per conversation rather than one response per conversation in a test set, they score response-set diversity by running every ordered pair of utterances through a RoBERTa-large NLI model and aggregating the class predictions (contradiction weighted +1, entailment −1, neutral 0 for the baseline variant). Three variants are proposed: Baseline NLI Diversity (hard argmax), Neutral NLI Diversity (neutrals weighted equal to contradictions), and Confidence NLI Diversity (softmax probabilities replace indicator weights). All three are evaluated on conTest and decTest from Tevet and Berant (2021), two 200-conversation English datasets designed to isolate semantic and lexical diversity respectively, using Spearman correlation to both a diversity parameter and averaged human Likert ratings from 10 annotators per set. The load-bearing finding is that Confidence NLI Diversity achieves ρ = 0.62 on conTest against the diversity parameter, equaling human performance (0.63) and exceeding the prior best automatic metric Sent-BERT (ρ = 0.59), while an ablation confirms that contradiction count is positively correlated with human diversity ratings (ρ = 0.46 on conTest) and entailment count negatively so (ρ = −0.65), validating the metric's polarity choices. Neutral predictions contribute near-zero correlation on conTest (ρ = −0.08), and Neutral NLI Diversity's superior performance on decTest (ρ = 0.72 vs. 0.24 on conTest) suggests it conflates lexical and semantic diversity signals. The second contribution, Diversity Threshold Generation, uses Baseline NLI Diversity as a stopping criterion: responses are iteratively resampled until a target number of contradictions (set at 10 out of 20 pairwise comparisons) is met or a maximum of 20 samples is exhausted. Applied to DialoGPT (700M parameters) and BlenderBot 1.0 (300M parameters) on DailyDialog++ (1,028 conversations) and EmpatheticDialogues (2,763 conversations), the procedure yields an average 137% increase in NLI Diversity with BERTScore and BLEU shifting by only 0.08% and 1.1% respectively, implying diversity gains without relevancy cost. An alternative generation approach the paper could have used but did not is Maximal Marginal Relevance-style reranking over a pre-generated candidate pool, which would avoid iterative model calls at the cost of fixed candidate diversity ceiling. The paper predicts that crowdworker instructions framed around generating contradictory rather than diverse responses should improve dataset quality, though this hypothesis is stated as future work and not empirically tested here. A critical reader would push back on the validity of the divthresh calibration: the contradiction threshold of 10 out of 20 pairwise comparisons is set empirically for NLI Diversity, while Sent-BERT and distinct-n thresholds are taken from the 90th percentile of human DailyDialog++ responses—making cross-metric comparisons of Diversity Threshold Generation's efficiency (samples required, overlap retained) potentially confounded by differently stringent thresholds rather than reflecting intrinsic metric properties. Additionally, all NLI Diversity evaluation uses the Tevet and Berant framework without independent human validation of the generated diverse response sets themselves, leaving open whether the correlation gains translate to perceived diversity improvements in actual conversational use.

Methods (15)

  • Baseline NLI Diversity
    First variant: aggregates argmax NLI class predictions with contradiction=+1, entailment=-1, neutral=0
  • Bert-STS
    Sentence-level semantic similarity metric repurposed as diversity measure in Tevet and Berant 2021
  • BERTScore
    Token-level BERT embedding similarity metric used for both diversity comparison and relevancy evaluation
  • BLEU Score
    Used as relevancy metric comparing generated responses to human references in DailyDialog++
  • Bootstrapping Confidence Interval Analysis
    1000-iteration bootstrap procedure sampling 50% of conTest to compute 95% confidence intervals
  • Confidence NLI Diversity
    Best-performing variant: aggregates softmax probability mass rather than binary class counts
  • Distinct-N
    Lexical diversity metric measuring the proportion of unique n-grams in persuadee responses within a dialogue
  • Diverse M-Best Solutions
    Greedy iterative algorithm for generating diverse hypotheses applied to vision and MT tasks
  • Diversity Threshold Generation
    Iterative generation procedure that resamples lowest-scoring responses until a diversity threshold is reached
  • Human Diversity Annotation (Likert Scale)
    Annotators score diversity of response sets 1-5 with half-point increments; used as ground truth correlation target
  • Maximal Marginal Relevance
    Diversity-based reranking approach from Carbonell and Goldstein 1998 for document summarization
  • Neutral NLI Diversity
    Variant weighting neutral predictions equally to contradictions to test if neutrals capture lexical diversity
  • NLI Diversity
    Novel metric proposed in this paper using NLI predictions to score semantic diversity of a response set
  • Sent-BERT
    Cosine similarity between BERT sentence embeddings; top automatic baseline for semantic diversity
  • Spearman's Rank Correlation
    Statistical measure used to evaluate correlation between diversity metrics and diversity parameter / human judgments

Frameworks (1)

Datasets (9)

  • Adversarial NLI
    Adversarially collected NLI dataset used in the combined NLI model training
  • conTest
    Human-created dataset with high lexical diversity and varying semantic diversity for evaluating diversity metrics
  • DailyDialog++
    Conversational dataset with 5 human-written responses per conversation used for Diversity Threshold Generation evaluation
  • decTest
    Model-generated dataset with diversity controlled by decoding temperature parameter
  • EmpatheticDialogues
    Second dialogue dataset used for Diversity Threshold Generation evaluation; no multiple human references
  • FEVER
    Fact-checking NLI dataset used in the combined NLI model training
  • Multi-Genre NLI
    Primary NLI training dataset for the MNLI-finetuned RoBERTa model used in all main experiments
  • Reddit respGen Task
    Dialogue response generation subset of conTest/decTest drawn from Reddit conversations
  • Stanford Natural Language Inference
    NLI training dataset used in the combined NLI model

Findings (12)

Claims (8)

Hypotheses (3)

Questions (3)

Original abstract (expand)

Generating diverse, interesting responses to chitchat conversations is a problem for neural conversational agents. This paper makes two substantial contributions to improving diversity in dialogue generation. First, we propose a novel metric which uses Natural Language Inference (NLI) to measure the semantic diversity of a set of model responses for a conversation. We evaluate this metric using an established framework (Tevet and Berant, 2021) and find strong evidence indicating NLI Diversity is correlated with semantic diversity. Specifically, we show that the contradiction relation is more useful than the neutral relation for measuring this diversity and that incorporating the NLI model's confidence achieves state-of-the-art results. Second, we demonstrate how to iteratively improve the semantic diversity of a sampled set of responses via a new generation procedure called Diversity Threshold Generation, which results in an average 137% increase in NLI Diversity compared to standard generation procedures.

Related work— refs + corpus + external arXiv

Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.

+26 more

Similar preprints — Semantic Scholar