A new information-theoretic evaluation metric (PCE, RCE, RE) is proposed to separately detect fidelity loss, mode dropping, and mode shrinkage in generative models, and existing kNN precision/recall metrics are unified as divergence estimators.
Assessing Dialogue Systems with Distribution Distances
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
An important aspect of developing dialogue systems is how to evaluate and compare the performance of different systems. Existing automatic evaluation metrics are based on turn-level quality evaluation and use average scores for system-level comparison. In this paper, we propose to measure the performance of a dialogue system by computing the distribution-wise distance between its generated conversations and real-world conversations. Specifically, two distribution-wise metrics, FBD and PRD, are developed and evaluated. Experiments on several dialogue corpora show that our proposed metrics correlate better with human judgments than existing metrics.
fields
cs.LG 1years
2024 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
A Unifying Information-theoretic Perspective on Evaluating Generative Models
A new information-theoretic evaluation metric (PCE, RCE, RE) is proposed to separately detect fidelity loss, mode dropping, and mode shrinkage in generative models, and existing kNN precision/recall metrics are unified as divergence estimators.