Pith. sign in

REVIEW 4 major objections 5 minor 48 references

BoK: Introducing Bag-of-Keywords Loss for Interpretable Dialogue Response Generation

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper shows that training dialogue models with an auxiliary loss that predicts only the keywords of the next response improves generation quality, adds post-hoc interpretability through the predicted keyword tokens, and yields a…

desk verdict Plausible incremental idea; the BoK-LM metric is the solid part, but the generation gains rest on test-set hyperparameter selection and a single seed. read the letter →

arxiv 2501.10328 v1 pith:S76TDHU7 submitted 2025-01-17 cs.CL

Suvodip Dey , Maunendra Sankar Desarkar This is my paper · ORCID
classification cs.CL
keywords Bag-of-Keywordslossauxiliaryfunctionopen-domaindialoguegenerationpost-hocinterpretabilityreference-freeevaluationkeywordextractionDailyDialogPersona-Chat
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Standard language-modeling loss measures per-token fluency but does not align with what humans value in dialogue. The paper proposes Bag-of-Keywords (BoK) loss, an auxiliary cross-entropy loss that trains a model to predict only the keywords—the content words carrying the core idea—of the next response, rather than every token as the earlier Bag-of-Words loss does. On DailyDialog and Persona-Chat, adding BoK loss to T5 and DialoGPT improves automated scores and human ratings of specificity, informativeness, and overall quality. The BoK-LM loss value of a trained model also shows moderate correlation with human judgments across dialogue evaluation benchmarks, so the paper proposes it as a reference-free quality metric. The top tokens predicted by the keyword head reveal the central thought the model is steering toward, which the paper presents as post-hoc interpretability.

What carries the argument

The load-bearing component is the BoK loss computed from a context vector $\phi_t$, the final-layer hidden state at the `bos` token. A single-layer feedforward network maps $\phi_t$ to a vocabulary distribution $\alpha_t$, and the loss is $L_{\text{BoK}} = -\sum_{w\in K_t} \log \alpha_{t,w}$, where $K_t$ is the set of YAKE!-extracted keywords of the ground-truth response, capped at eight tokens with a special `<nok>` token for responses that yield no keywords. Training minimizes $L_{\text{LM}} + \lambda L_{\text{BoK}}$ with $\lambda=0.1$ for T5 and $\lambda=0.3$ for DialoGPT. Because only content words are predicted, the context vector is pushed to encode the response's central idea rather than its full surface form, and the top predicted tokens serve as a readable trace of that idea.

What would settle it

Train DialoGPT with BoK loss but replace the keyword labels with randomly chosen non-stopword tokens from the same responses; if BLEU, USL-H, and metric correlations remain at the same level, then the quality of keyword selection is not what carries the method.

Watch

Extended reading notes

Core claim

The paper's central claim is that replacing Bag-of-Words loss with Bag-of-Keywords loss—cross-entropy over only the keyword set $K_t$ extracted from the ground-truth response—makes open-domain dialogue generation more generalizable and more interpretable. In the key comparison, DialoGPT$_{\text{BoK}}$ exceeds DialoGPT$_{\text{BoW}}$ on DailyDialog (BLEU-4 14.92 vs 11.82, diversity 0.257 vs 0.190, USL-H 0.7064 vs 0.6759), and on Persona-Chat the BoK variants improve both USL-H and Dial-M over their BoW counterparts. Human evaluation finds the BoK-trained models win or tie more often than they lose on informativeness and overall quality. The paper additionally establishes that the BoK-LM loss computed by a trained model works as a reference-free metric with moderate correlation to human judgments, outperforming BoW-LM on most of the studied benchmarks and remaining competitive with established reference-free metrics on chit-chat datasets.

Load-bearing premise

The method assumes that keywords extracted by the unsupervised YAKE! algorithm from a single ground-truth response are a good proxy for the core idea a good response should express; if they are not, BoK loss is just BoW loss over a noisy subset of tokens, and the reported generation gains and metric correlations lose their rationale.

Editorial extensions

If this is right

  • Models trained with BoK-LM loss outperform their BoW-trained counterparts on both DailyDialog and Persona-Chat under referenced metrics (BLEU, NIST, METEOR, diversity) and reference-free metrics (USL-H, Dial-M).
  • Generated responses become more specific and informative; human evaluators rate BoK models as winning or tying BoW models on informativeness and overall quality in most comparisons.
  • Keyword prediction makes the context vector interpretable: the top predicted tokens align with the content of the generated response and can be inspected without extra annotation.
  • The BoK-LM loss value correlates moderately with human judgments on chit-chat benchmarks such as PredictiveEngage, FED, and GRADE-DailyDialog, and beats BoW-LM as a reference-free metric on most of the tested datasets.
  • The loss has hyperparameters $\lambda$ and $|K_t|$ that must be tuned: a larger keyword cap pushes BoK back toward BoW behavior, and a larger $\lambda$ raises diversity while lowering lexical-overlap scores.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • I infer the method pays off most on turns with distinctive topical content; for generic backchannels the special `<nok>` label makes the BoK term nearly constant, so BoK-LM largely reduces to LM training.
  • I infer the post-hoc interpretability is content-word-level only: the top predicted tokens do not explain the response's syntax or reasoning, so they show what topic the model commits to but not why it phrased it that way.
  • I infer that BoK-LM's weaker correlations on knowledge-grounded datasets indicate it measures topical specificity rather than persona grounding, so combining it with a persona-aware component could close that gap.
  • A testable extension the paper does not run is to use the BoK loss or its predicted-token distribution as a dense reward during reinforcement-learning fine-tuning, where the keyword head could supply a continuous training signal.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Bag-of-Keywords (BoK) loss, an auxiliary cross-entropy loss that predicts only the keywords of the next response, extracted with YAKE!, in addition to the standard LM loss. BoK is applied to T5 (encoder-decoder) and DialoGPT (decoder-only) on DailyDialog and Persona-Chat. The claims are that BoK improves generation over LM and BoW baselines on automatic and human evaluations, provides post-hoc interpretability through the top-k predicted keyword tokens, and that BoK-LM loss works as a reference-free evaluation metric with correlations comparable to existing metrics.

Significance. If substantiated, BoK loss would be a simple, model-agnostic auxiliary objective that improves specificity and yields an interpretable projection of the context vector. Strengths include release of code, use of two architectures and two datasets, a human evaluation with reported Fleiss kappa, and evaluation across five dialogue-metric benchmarks. The use of an unsupervised external keyword extractor keeps the method training-free at data-preparation time. However, the numerical support currently rests on test-set hyperparameter selection and a single run, so the main empirical claim is not yet established at the level the paper asserts.

major comments (4)
  1. [Section 5.4, Tables 5 and 6] The hyperparameters lambda and |K_t| are selected by sweeping on the DailyDialog test set ("We conduct this ablation study on DailyDialog test data"), and the final models in Table 2 use the test-selected values (lambda=0.1 for T5, lambda=0.3 for DialoGPT, |K_t|=8). Because the headline comparison is the maximum of a small grid evaluated on the same test set, all Table 2 gains over baselines are optimistically biased. The effect is material: for DialoGPTBoK, USL-H jumps from 0.6770 at lambda=0.20 to 0.7064 at lambda=0.30 and BLEU-4 from 11.64 to 14.92. The authors should either move hyperparameter selection to a held-out validation split and report test results for the fixed configuration only, or report selection-corrected estimates such as nested cross-validation.
  2. [Appendix A.1] All results come from one fixed seed (seed 10) and deterministic beam search; no variance estimate is reported. Several central comparisons in Table 2 are very small, for example T5BoK versus T5BoW has USL-H 0.6793 versus 0.6791 and BLEU-1 51.74 versus 51.75, so a single run cannot distinguish BoK from BoW for T5. Please report means and standard deviations over at least three to five seeds and, where feasible, a significance test for the main comparisons; deterministic decoding does not remove seed sensitivity from training.
  3. [Section 3.1] The assumption that YAKE! keywords extracted from the single ground-truth utterance represent the "core idea" of a response is load-bearing but not validated. There is no analysis of keyword quality or agreement on the dialogue datasets, and no control experiment: BoK is compared with BoW, but not with a BoW variant restricted to the same number of non-stopword/content tokens or with a random keyword subset. Such a control is needed to establish that the benefit comes from predicting semantically meaningful keywords rather than from simply dropping function words and reducing the prediction set. Without it, the interpretability and specificity claims are not separated from a trivial content-word-boosting effect.
  4. [Section 5.3, Table 4] The human evaluation compares BoK models only with BoW models, not with the vanilla LM backbones, so the human results do not directly support the abstract's claim that BoK improves dialogue generation of the backbone models. Furthermore, the sample is 50 instances per dataset with four annotators and no significance testing; for T5BoK versus T5BoW on DailyDialog the overall judgment is only 32-26-42, which is not strong evidence by itself. Reporting significance or at least confidence intervals for the human comparisons would help.
minor comments (5)
  1. [Table 7, Table 8] The text and captions contain "Person-Chat" where "Persona-Chat" is meant; please correct the typo.
  2. [Table 9 caption] The caption says "top <0.05" and should read "p < 0.05" for the significance threshold.
  3. [Throughout] The metric name is written inconsistently as "USL S-H", "USLS-H", and "USL S-H"; unify the notation.
  4. [Equations (1)-(3)] The summation symbol appears as "TX" in Equations (1)-(3); this is presumably a rendering issue, but the typeset equations should use a proper sum symbol.
  5. [Appendix A.1, References] There is a typo "DilaoFlow" for DialoFlow, and the citation "Long and et al. 2022" should be formatted as an author-year citation (Ouyang et al., 2022).

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: BoK loss is defined from external YAKE! keywords and evaluated on external test sets; the authors' self-citations are baselines, not premises.

full rationale

The central object, Bag-of-Keywords loss (Eq. 3), is defined as cross-entropy over the keyword set K_t extracted by YAKE!, an external unsupervised extractor, from the ground-truth response; it is not defined in terms of the reported BLEU, USL-H, Dial-M, or human judgment values. The claimed improvements are empirical comparisons on DailyDialog and Persona-Chat test sets and against independent baselines, so no equation reduces the output to the input. The authors' prior work (DialoGen and Dial-M) is used as a comparison baseline and an evaluation metric, not as a theorem or premise that forces the BoK result; the YAKE! choice is additionally grounded in the external YAKE! publication. The YAKE!-keyword-as-core-idea assumption is substantive but is an empirical modeling choice, not a definitional equivalence. A real statistical concern is that lambda and |K_t| were selected on the DailyDialog test set (Section 5.4, Tables 5-6) with a single deterministic seed (Appendix A.1), so the headline DialoGPTBoK gain may be optimistically biased; however, test-set selection and lack of multiple seeds are correctness and robustness issues, not circular reasoning, and they do not make the derivation equivalent to its inputs.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim rests on a small number of explicit design choices: the weighting of the auxiliary loss, the number of keyword tokens, and the assumptions that YAKE! captures response core ideas and that the bos representation is sufficient for keyword prediction. No new entities are introduced. The main free parameters (lambda, |K_t|) are fit to the DailyDialog test set, which is the principal threat to the strength of the empirical claims.

free parameters (2)
  • lambda (BoK loss weight) = 0.1 (T5), 0.3 (DialoGPT)
    Weight of L_BoK in Eq. 6. Chosen by sweeping on the DailyDialog test set (Table 5); the best test value is used in all main results, so the reported gains are partly in-sample.
  • |K_t| (max keyword tokens) = 8
    Maximum number of keyword tokens in the BoK loss (Eq. 3). Selected from a test-set sweep (Table 6); increasing it makes BoK approach BoW.
assumptions (3)
  • domain assumption YAKE! extracted keywords represent the core idea of a response
    The loss definition in Eq. 3 and the interpretability claims in Section 6.1 assume YAKE!'s output K_t is a meaningful proxy for the central content of the reply. This is a design choice without independent validation that the selected words are indeed the 'core' tokens.
  • ad hoc to paper The bos hidden state phi_t is a sufficient representation for keyword prediction
    Eq. 4 computes BoK loss from the final-layer bos representation of the decoder/encoder-decoder. The method assumes a single FFNN over this vector can predict keywords and that training this head helps rather than hurts the LM objective.
  • domain assumption BoK-LM loss on a context-response pair reflects response quality
    Section 6.2 uses BoK-LM loss as a reference-free metric, assuming lower loss correlates with higher human-judged quality. This is an empirical assumption tested only through the reported correlations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BoK: Introducing Bag-of-Keywords Loss for Interpretable Dialogue Response Generation." pith.science (2026). https://pith.science/paper/S76TDHU7

@misc{pith2026250110328,
  author       = {Pith},
  title        = {Pith review of: BoK: Introducing Bag-of-Keywords Loss for Interpretable Dialogue Response Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/S76TDHU7}},
  note         = {Machine review of arXiv:2501.10328}
}
read the original abstract

The standard language modeling (LM) loss by itself has been shown to be inadequate for effective dialogue modeling. As a result, various training approaches, such as auxiliary loss functions and leveraging human feedback, are being adopted to enrich open-domain dialogue systems. One such auxiliary loss function is Bag-of-Words (BoW) loss, defined as the cross-entropy loss for predicting all the words/tokens of the next utterance. In this work, we propose a novel auxiliary loss named Bag-of-Keywords (BoK) loss to capture the central thought of the response through keyword prediction and leverage it to enhance the generation of meaningful and interpretable responses in open-domain dialogue systems. BoK loss upgrades the BoW loss by predicting only the keywords or critical words/tokens of the next utterance, intending to estimate the core idea rather than the entire response. We incorporate BoK loss in both encoder-decoder (T5) and decoder-only (DialoGPT) architecture and train the models to minimize the weighted sum of BoK and LM (BoK-LM) loss. We perform our experiments on two popular open-domain dialogue datasets, DailyDialog and Persona-Chat. We show that the inclusion of BoK loss improves the dialogue generation of backbone models while also enabling post-hoc interpretability. We also study the effectiveness of BoK-LM loss as a reference-free metric and observe comparable performance to the state-of-the-art metrics on various dialogue evaluation datasets.

Figures

Figures reproduced from arXiv: 2501.10328 by the authors.

Figure 1
Figure 1. A motivating example for Bag-of-Keywords [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Incorporating BoK loss in open-domain dialogue models. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 21 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Satanjeev Banerjee and Alon Lavie. 2005. https://aclanthology.org/W05-0909 METEOR : An automatic metric for MT evaluation with improved correlation with human judgments . In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization , pages 65--72, Ann Arbor, Michigan. Association for Comput...

    2005

  4. [4]

    Ricardo Campos, V \'i tor Mangaravite, Arian Pasquali, Al \'i pio M \'a rio Jorge, C \'e lia Nunes, and Adam Jatowt. 2018. YAKE! Collection-Independent Automatic Keyword Extractor . In Advances in Information Retrieval, pages 806--810, Cham. Springer International Publishing

    work page 2018

  5. [5]

    Ricardo Campos, Vítor Mangaravite, Arian Pasquali, Alípio Jorge, Célia Nunes, and Adam Jatowt. 2020. https://doi.org/https://doi.org/10.1016/j.ins.2019.09.013 Yake! keyword extraction from single documents using multiple local features . Information Sciences, 509:257--289

    doi:10.1016/j.ins.2019.09.013 2020

  6. [6]

    Stephen Casper and et al. 2023. https://openreview.net/forum?id=bx24KpJ4Eb Open problems and fundamental limitations of reinforcement learning from human feedback . Transactions on Machine Learning Research. Survey Certification

    work page 2023

  7. [7]

    Wei Chen, Yeyun Gong, Song Wang, Bolun Yao, Weizhen Qi, Zhongyu Wei, Xiaowu Hu, Bartuer Zhou, Yi Mao, Weizhu Chen, Biao Cheng, and Nan Duan. 2022. https://doi.org/10.18653/v1/2022.acl-long.333 D ialog VED : A pre-trained latent variable encoder-decoder model for dialog response generation . In Proceedings of the 60th Annual Meeting of the Association for ...

    work page doi:10.18653/v1/2022.acl-long.333 2022

  8. [8]

    Suvodip Dey and Maunendra Sankar Desarkar. 2023. https://doi.org/10.18653/v1/2023.sigdial-1.7 Dial- M : A masking-based framework for dialogue evaluation . In Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 77--84, Prague, Czechia. Association for Computational Linguistics

    work page doi:10.18653/v1/2023.sigdial-1.7 2023

Show all 48 references
  1. [9]

    Suvodip Dey, Maunendra Sankar Desarkar, Asif Ekbal, and Srijith P. K. 2023. https://aclanthology.org/2023.paclic-1.37 D ialo G en: Generalized long-range context representation for dialogue systems . In Proceedings of the 37th Pacific Asia Conference on Language, Information a...

    2023

  2. [10]

    Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur Szlam, Iulian Serban, Ryan Lowe, Shrimai Prabhumoye, Alan W

    Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander H. Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur Szlam, Iulian Serban, Ryan Lowe, Shrimai Prabhumoye, Alan W. Black, Alexander I. Rudnicky, Jason Williams, Joelle Pineau, Mikhail S. Burtsev, and Jason Weston....

    2019 arXiv

  3. [11]

    Sarik Ghazarian, Johnny Wei, Aram Galstyan, and Nanyun Peng. 2019. https://doi.org/10.18653/v1/W19-2310 Better automatic evaluation of open-domain dialogue systems with contextualized embeddings . In Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural L...

    2019 doi

  4. [12]

    Sarik Ghazarian, Ralph Weischedel, Aram Galstyan, and Nanyun Peng. 2020. https://doi.org/10.1609/aaai.v34i05.6283 Predictive engagement: An efficient metric for automatic evaluation of open-domain dialogue systems . Proceedings of the AAAI Conference on Artificial Intelligence...

    2020 doi

  5. [13]

    Prakhar Gupta, Shikib Mehri, Tiancheng Zhao, Amy Pavel, Maxine Eskenazi, and Jeffrey Bigham. 2019. https://doi.org/10.18653/v1/W19-5944 Investigating evaluation of open-domain dialogue systems with human generated multiple references . In Proceedings of the 20th Annual SIGdial...

    2019 doi

  6. [14]

    Lishan Huang, Zheng Ye, Jinghui Qin, Liang Lin, and Xiaodan Liang. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.742 GRADE : Automatic graph-enhanced coherence metric for evaluating open-domain dialogue systems . In Proceedings of the 2020 Conference on Empirical Methods i...

    2020 doi

  7. [15]

    Zhihua Jiang, Guanghui Ye, Dongning Rao, Di Wang, and Xin Miao. 2022. https://aclanthology.org/2022.emnlp-main.762 IM 2: an interpretable and multi-category integrated metric framework for automatic dialogue evaluation . In Proceedings of the 2022 Conference on Empirical Metho...

    2022

  8. [16]

    Tian Lan, Xian-Ling Mao, Wei Wei, Xiaoyan Gao, and Heyan Huang. 2020. https://doi.org/10.1145/3423168 Pone: A novel automatic evaluation metric for open-domain generative dialogue systems . ACM Trans. Inf. Syst., 39(1)

    2020 doi

  9. [17]

    Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. https://doi.org/10.18653/v1/N16-1014 A diversity-promoting objective function for neural conversation models . In Proceedings of the 2016 Conference of the North A merican Chapter of the Association f...

    2016 doi

  10. [18]

    Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. https://aclanthology.org/I17-1099 D aily D ialog: A manually labelled multi-turn dialogue dataset . In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1...

    2017

  11. [19]

    Zekang Li, Jinchao Zhang, Zhengcong Fei, Yang Feng, and Jie Zhou. 2021. https://doi.org/10.18653/v1/2021.acl-long.11 Conversations are not flat: Modeling the dynamic information flow across dialogue utterances . In Proceedings of the 59th Annual Meeting of the Association for ...

    2021 doi

  12. [20]

    Chin-Yew Lin and Franz Josef Och. 2004. https://doi.org/10.3115/1218955.1219032 Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics . In Proceedings of the 42nd Annual Meeting of the Association for Computational Ling...

    2004

  13. [21]

    Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. https://doi.org/10.18653/v1/D16-1230 How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation . In Proceed...

    2016 doi

  14. [22]

    Ouyang Long and et al. 2022. http://arxiv.org/abs/2203.02155 Training language models to follow instructions with human feedback

    2022 arXiv

  15. [23]

    Ryan Lowe, Michael Noseworthy, Iulian Vlad Serban, Nicolas Angelard-Gontier, Yoshua Bengio, and Joelle Pineau. 2017. https://doi.org/10.18653/v1/P17-1103 Towards an automatic T uring test: Learning to evaluate dialogue responses . In Proceedings of the 55th Annual Meeting of t...

    2017 doi

  16. [24]

    Shikib Mehri and Maxine Eskenazi. 2020 a . https://aclanthology.org/2020.sigdial-1.28 Unsupervised evaluation of interactive dialog with D ialo GPT . In Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 225--235, 1st virtual ...

    2020

  17. [25]

    Shikib Mehri and Maxine Eskenazi. 2020 b . https://doi.org/10.18653/v1/2020.acl-main.64 USR : An unsupervised and reference free evaluation metric for dialog generation . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 681--707...

    2020 doi

  18. [26]

    John Mendonca, Alon Lavie, and Isabel Trancoso. 2022. https://aclanthology.org/2022.sigdial-1.9 Q uality A dapt: an automatic dialogue quality estimation framework . In Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 83--90...

    2022

  19. [27]

    Bo Pang, Erik Nijkamp, Wenjuan Han, Linqi Zhou, Yixian Liu, and Kewei Tu. 2020. https://doi.org/10.18653/v1/2020.acl-main.333 Towards holistic and automatic evaluation of open-domain dialogue generation . In Proceedings of the 58th Annual Meeting of the Association for Computa...

    2020 doi

  20. [28]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. https://doi.org/10.3115/1073083.1073135 B leu: a method for automatic evaluation of machine translation . In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311--3...

    2002

  21. [29]

    Vitou Phy, Yang Zhao, and Akiko Aizawa. 2020. https://doi.org/10.18653/v1/2020.coling-main.368 Deconstruct to reconstruct a configurable evaluation metric for open-domain dialogue systems . In Proceedings of the 28th International Conference on Computational Linguistics, pages...

    2020 doi

  22. [30]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. 21(1)

    2020

  23. [31]

    Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Eric Michael Smith, Y-Lan Boureau, and Jason Weston. 2021. https://doi.org/10.18653/v1/2021.eacl-main.24 Recipes for building an open-domain chatbot . In Proceedings of the 16th Co...

    2021 doi

  24. [32]

    Sai, Akash Kumar Mohankumar, Siddharth Arora, and Mitesh M

    Ananya B. Sai, Akash Kumar Mohankumar, Siddharth Arora, and Mitesh M. Khapra. 2020. Improving dialog evaluation with a multi-reference adversarial dataset and large scale pretraining. Transactions of the Association for Computational Linguistics, 8:810--827

    2020

  25. [33]

    Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. https://doi.org/10.18653/v1/2020.acl-main.704 BLEURT : Learning robust metrics for text generation . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881--7892, Online. Ass...

    2020 doi

  26. [34]

    Iulian Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron Courville, and Yoshua Bengio. 2017. https://doi.org/10.1609/aaai.v31i1.10983 A hierarchical latent variable encoder-decoder model for generating dialogues . Proceedings of the AAAI Conference o...

    2017 doi

  27. [35]

    Hamilton, and Joelle Pineau

    Koustuv Sinha, Prasanna Parthasarathi, Jasmine Wang, Ryan Lowe, William L. Hamilton, and Joelle Pineau. 2020. https://doi.org/10.18653/v1/2020.acl-main.220 Learning an unreferenced metric for online dialogue evaluation . In Proceedings of the 58th Annual Meeting of the Associa...

    2020 doi

  28. [36]

    Alessandro Sordoni, Yoshua Bengio, Hossein Vahabi, Christina Lioma, Jakob Grue Simonsen, and Jian-Yun Nie. 2015. https://doi.org/10.1145/2806416.2806493 A hierarchical recurrent encoder-decoder for generative context-aware query suggestion . In Proceedings of the 24th ACM Inte...

    2015

  29. [37]

    Chongyang Tao, Lili Mou, Dongyan Zhao, and Rui Yan. 2017. Ruber: An unsupervised method for automatic evaluation of open-domain dialog systems. In AAAI Conference on Artificial Intelligence

    2017

  30. [38]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf Attention is all you need . In Advances in Ne...

    2017

  31. [39]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020

  32. [40]

    Thomas Wolf, Victor Sanh, Julien Chaumond, and Clement Delangue. 2019. https://doi.org/10.48550/ARXIV.1901.08149 Transfertransfo: A transfer learning approach for neural network based conversational agents

    2019 doi arXiv

  33. [41]

    Yi-Ting Yeh, Maxine Eskenazi, and Shikib Mehri. 2021. https://doi.org/10.18653/v1/2021.eancs-1.3 A comprehensive assessment of dialog evaluation metrics . In The First Workshop on Evaluations and Assessments of Neural Conversation Systems, pages 15--33, Online. Association for...

    2021 doi

  34. [42]

    Chen Zhang, Yiming Chen, Luis Fernando D ' Haro, Yan Zhang, Thomas Friedrichs, Grandee Lee, and Haizhou Li. 2021 a . https://doi.org/10.18653/v1/2021.acl-long.441 D yna E val: Unifying turn and dialogue level evaluation . In Proceedings of the 59th Annual Meeting of the Associ...

    2021 doi

  35. [43]

    Chen Zhang, Grandee Lee, Luis Fernando D’Haro, and Haizhou Li. 2021 b . https://doi.org/10.1109/TASLP.2021.3074012 D-score: Holistic dialogue evaluation without reference . IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:2502--2516

    2021

  36. [44]

    Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018 a . https://doi.org/10.18653/v1/P18-1205 Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computation...

    2018 doi

  37. [45]

    Weinberger, and Yoav Artzi

    Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. https://openreview.net/forum?id=SkeHuCVFDr Bertscore: Evaluating text generation with bert . In International Conference on Learning Representations

    2020

  38. [46]

    Yizhe Zhang, Michel Galley, Jianfeng Gao, Zhe Gan, Xiujun Li, Chris Brockett, and Bill Dolan. 2018 b . https://proceedings.neurips.cc/paper_files/paper/2018/file/23ce1851341ec1fa9e0c259de10bf87c-Paper.pdf Generating informative and diverse conversational responses via adversar...

    2018

  39. [47]

    Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020. https://doi.org/10.18653/v1/2020.acl-demos.30 DIALOGPT : Large-scale generative pre-training for conversational response generation . In Proceeding...

    2020 doi

  40. [48]

    Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. 2017. https://doi.org/10.18653/v1/P17-1061 Learning discourse-level diversity for neural dialog models using conditional variational autoencoders . In Proceedings of the 55th Annual Meeting of the Association for Computational Lin...

    2017 doi

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.