Pith. sign in

REVIEW 4 major objections 5 minor 38 references

Socio-Emotional Response Generation: A Human Evaluation Protocol for LLM-Based Conversational Systems

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a conversational system produces better responses when it first predicts a sequence of socio-emotional labels--like 'inform' plus 'happiness'--and uses that plan to select the final answer, with a three-step human…

desk verdict The evaluation protocol and released data are the real contribution; the headline conditioning claim is undercut by a best-of-N confound. read the letter →

arxiv 2412.04492 v1 pith:QUSMXDD7 submitted 2024-11-26 cs.CL cs.AIcs.HCcs.SI

Lorraine Vanel , Ariel R. Ramos Vela , Alya Yacoubi , Chloé Clavel (IDS , S2A , LTCI) This is my paper · ORCID
classification cs.CLcs.AIcs.HCcs.SI
keywords socio-emotionalresponsegenerationplanningemotionlabelsdialogueactsconditionedhumanevaluationLLMconversationalsystemsDailyDialog
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that a chatbot produces better responses when it first plans the social and emotional moves it will make--a short sequence of labels such as 'inform' plus 'happiness'--and then lets that plan steer which generated answer is chosen, instead of generating end-to-end with no explicit strategy. The authors build a two-module architecture: a fine-tuned BART model predicts the next turn's label sequence from the last three turns, and a reranking step or a prompted LLM uses those labels to select or produce the final response. They compare this against direct generation and against conditioning on ground-truth labels, and they evaluate all outputs with both automated metrics and a three-step human annotation protocol. The paper reports that conditioned responses are judged better, that conditioning on predicted labels is nearly as good as conditioning on true labels, and that automated metrics miss the social and emotional quality that human judges see. If this claim holds, it offers a concrete way to make conversational AI more transparent and controllable without sacrificing response quality.

What carries the argument

The load-bearing mechanism is a planning-then-rerank loop. First, a fine-tuned BART-base model reads the last three dialogue turns and emits a short sequence of socio-emotional labels, typically one to three labels mixing a dialogue act with an emotion. Second, the generator produces ten candidate responses ($N=10$), a BERT classifier assigns a label list to each candidate, and the candidate whose label list has the highest Normalised Levenshtein Similarity to the predicted label sequence is selected as the final response. In the prompt-based variant, Beluga receives the expected labels directly in its instruction. The key comparison is NO-CD, which takes the model's first most probable response, versus CD-pred and CD-GT, which rerank the same pool of ten candidates using the label-planning score.

What would settle it

Run the same models with two control conditions: pick one of the ten candidate responses at random, and pick the candidate whose predicted labels best match a randomly drawn label sequence instead of the BART-predicted one. If either control reproduces the improvement that CD-pred shows over NO-CD, then the reported gain comes from having a pool of ten candidates or from the selection step, not from socio-emotional planning. The paper does not include such a control.

Watch

Extended reading notes

Core claim

The central claim is that predicting a sequence of expected socio-emotional strategy labels--dialogue acts such as informing or questioning combined with emotions such as happiness or sadness--and using that sequence to condition response generation yields better results than a direct end-to-end generation scheme. On the DailyDialog test set, the paper compares seven model configurations run without conditioning, with conditioning on labels predicted by a BART planner, and with conditioning on ground-truth labels. Human annotators first filtered responses for consistency and specificity, then selected the three best responses per context, and finally rated those on logical, emotional, and social adequacy. The conditioned responses survived the filter and entered the top-3 more often, and their weighted socio-emotional score was higher, with predicted-label conditioning matching ground-truth-label conditioning. The paper also finds that automated metrics diverge from these judgments, with some metrics favoring responses that merely repeat the context.

Load-bearing premise

The comparison assumes that any advantage of the conditioned systems comes from the socio-emotional label planning itself, not from the fact that those systems choose from ten generated candidates while the no-conditioning baseline takes only the first response.

Editorial extensions

If this is right

  • Conditioned response generation, using a predicted sequence of dialogue-act and emotion labels, improves human-judged response quality over direct end-to-end generation.
  • Because CD-pred matches CD-GT, an imperfect label predictor can deliver almost all of the benefit of perfect label knowledge, which makes the planning step practical outside datasets with gold labels.
  • Reference-based and reference-free automated metrics disagree with human ratings on social and emotional quality, so conclusions drawn only from those metrics should be treated as provisional.
  • The public annotation platform and annotated data give future conversational systems a common yardstick for socio-emotional response evaluation.
  • A prompt-based conditional model can outperform a reranking-based conditional model, indicating that the label information, not the reranking procedure, carries the improvement.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: if the planning-then-generate split is what produces the gain, it also makes the response process inspectable--an operator could read and edit the planned label sequence before the text is chosen, which is not possible in end-to-end generation.
  • Beyond the paper: the three-step protocol's coarse filtering followed by fine-grained rating could transfer to other open-ended generation tasks where reference-based metrics are known to be unreliable.
  • Beyond the paper: the open design suggests a direct ablation--compare candidate selection by predicted labels, by random labels, and by no reranking--to quantify how much of the effect is planning versus candidate diversity.
  • Beyond the paper: the Levenshtein similarity over label sequences treats every label substitution as equally costly; weighting substitutions according to a taxonomy of dialogue acts and emotions could improve the reranker further.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a modular architecture for socio-emotional response generation: a BART-based module first predicts a sequence of dialogue-act and emotion labels for the next speaker turn, and a second module either reranks N=10 generated candidates by Normalised Levenshtein Similarity between the candidate's BERT-predicted labels and the expected label sequence, or generates a response from a prompt conditioned on the labels. The authors compare this conditioned scheme (CD-pred with predicted labels, CD-GT with ground-truth labels) against unconditioned generation (NO-CD) across several open-source models (GPT-2, DialoGPT, BART, Beluga), using both automated metrics and a three-step human evaluation protocol (relevance filtering, top-3 selection, fine-grained socio-emotional annotation). They report that conditioning improves response quality, that CD-pred and CD-GT perform nearly identically, and that automated metrics diverge from human judgments. Code and annotated data are released.

Significance. If the central claim were established, the paper would provide a useful demonstration that explicit socio-emotional label planning can improve response quality and that a staged human evaluation protocol can detect such improvements; the released annotation platform and data are valuable resources for future work. The paper also deserves credit for using only open-source models, for reporting a robustness check on a duplicate-free DailyDialog split (Appendix F), and for making the evaluation protocol concrete enough to be reused. However, the main empirical claim is currently undercut by a missing control condition and by unresolved inconsistencies in the reported tables, so the significance of the planning contribution is not yet demonstrated.

major comments (4)
  1. [Section IV, 'Experimental Setting' and Section III.B] The comparison conflates label conditioning with candidate-pool selection. NO-CD is defined as taking the first most probable speaker turn, while CD-pred and CD-GT rerank the same pool of N=10 generated candidates via NLS to the expected label sequence. Any selection among ten candidates can improve perceived quality even without label information, so the observed gains cannot be attributed to socio-emotional planning. The paper even notes that CD-pred and CD-GT often select the same candidate, which is consistent with the reranking signal being nearly irrelevant. A control condition that selects among the same 10 candidates without using label information (e.g., random selection, or selection by a label-free scoring function) is required before the abstract's claim can be supported.
  2. [Table I and Section V, results of Steps 1-2] There is a direct numerical contradiction that affects the main claim. Table I reports GPT-2 Medium NO-CD filter=53 and GPT-2 Medium CD-pred/CD-GT filter=5, yet Section V states that only GPT-2 Medium, DialoGPT Medium, and BELUGA PB CD-pred are saved more than 50% of the time in Step 1. If the filter values are correct, conditioning on labels catastrophically reduces the filtering rate for GPT-2 Medium, which contradicts the text's assertion that 'CD models tend to do better.' If the values are typographical, the tables and narrative need to be reconciled before the results can be interpreted.
  3. [Section IV.d, socemo index] The socemo index is weighted by the frequency with which a model's responses were selected in Steps 1-2, and the paper acknowledges that the socemo gain 'mostly comes from the fact that CD models were preferred in the response selection phase.' The unweighted logical, emotional, and social scores in Table VI are nearly identical across CD and NO-CD, so the headline socio-emotional advantage is largely a restatement of the top-3 selection frequency rather than an independent measure of socio-emotional quality. The paper should report the unweighted axis scores as the primary evidence for socio-emotional improvement, or justify why the selection-weighted index is the appropriate target metric.
  4. [Section V, Step 3 and Table I] The human evaluation results are reported as point percentages without significance tests or confidence intervals. Many of the differences that support the conditioning claim are small (e.g., GPT-2 Medium top3: 21 vs 19; DialoGPT Medium top3: 16 vs 16; BART Large socemo: 19 vs 19), and Step 3 annotates only 59 contexts, so the socemo estimates carry substantial uncertainty. Paired significance tests across the 300 contexts (or an explicit statement of the effective sample size for Step 3) and adjustment for multiple comparisons are needed to support the comparative claims.
minor comments (5)
  1. [Appendix A] The text contains an unresolved cross-reference: 'with the hyper-parameters described in the Appendix ??' should be replaced with the actual appendix/figure reference.
  2. [Appendix B] The Beluga reranking variant is called 'Beluga R' in the main text and Table I but 'Beluga F&R' in Appendix B; the naming should be unified.
  3. [Section IV.C] 'Krip pendorff's Alpha' and 'Krippendorff's Alpha' are spelled inconsistently; also, the reported Jaccard similarity of 0.97 appears to be a similarity of filtered lists, not a distance, and should be defined precisely.
  4. [Section IV.d and Appendix E] The formula for socemo uses 'logi_i(m)' as a variable name; this is likely a typo for 'log_i(m)' and should be cleaned up.
  5. [Table VII] The duplicate-free DailyDialog results in Appendix F are reported only for automated metrics and for reranking rows; the prompt-based Beluga comparison, which is the cleanest label-conditioning contrast, is not included in that robustness check.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the central claim is supported by independent human evaluation and automated metrics, not by a derivation that presupposes its conclusion.

full rationale

This paper makes an empirical comparison, not a formal derivation, so the circularity patterns based on equation-level self-definition do not directly apply. The central claim is that predicting a sequence of socio-emotional labels and using it to rerank generation improves response quality. The comparison is between a no-conditioning baseline that returns the first most probable turn and conditioning variants that rerank a shared pool of ten candidates. That asymmetry is a potential confound between label conditioning and best-of-N selection, but it is an experimental design weakness, not a circularity: the label-conditioned outputs are not defined in terms of the evaluation outcomes, and no fitted parameter is later presented as a prediction of the same quantity. The socemo index does weight fine-grained logical, emotional, and social ratings by how often a model's response was selected in Steps 1-2, but the paper explicitly acknowledges that the CD advantage 'mostly comes from the fact that CD models were preferred in the response selection phase,' and selection frequency itself is an independent human quality judgment rather than an input to the system. The authors' self-citations are background references to their prior work on emotion/dialogue-strategy prediction and socio-conversational systems; none is invoked as a uniqueness theorem or as the sole justification for the central empirical result. The release of code and annotated data makes the protocol externally checkable. The main concern about the NO-CD versus CD comparison is better classified as a correctness or validity risk, not circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The paper's central comparison rests on the adequacy of DailyDialog labels, the accuracy of the BERT label classifier, and the validity of human ratings. The reranking setup introduces a free-choice advantage that is not controlled.

free parameters (3)
  • BERT classifier confidence threshold = 0.7
    Used to decide if a predicted label is viable when labeling generated candidates for reranking (Section III.B). The threshold is chosen by hand and affects which candidates get selected.
  • Number of candidate responses N = 10
    Each model generates 10 candidate responses for reranking; the best-match candidate is selected. N is a design choice; larger N may inflate CD performance relative to the 1-sample NO-CD baseline.
  • Context window size = 3
    The model uses 3 prior speaker turns to predict labels and generate responses. This choice constrains the information available to both planning and generation.
assumptions (5)
  • domain assumption DailyDialog annotations of dialogue acts and emotions are an adequate representation of socio-emotional strategies.
    The paper uses DailyDialog as the training and evaluation data and assumes its human labels capture the strategies to be planned (Section IV.A.b).
  • domain assumption The BERT classifier provides sufficiently accurate labels for generated candidates (F1=0.72, Jaccard=0.59).
    The reranking step relies on the classifier's predicted label sequences for each candidate to select the response (Section III.B).
  • domain assumption Human annotation of consistency, specificity, and socio-emotional axes is a valid and reliable measure of response quality.
    The conclusion that conditioning improves quality rests on the three-expert annotation protocol; inter-annotator agreement is reported only for Steps 1-2 (alpha=0.51), not Step 3.
  • domain assumption Label sequence similarity can be measured by Normalised Levenshtein Similarity between predicted and expected label sequences.
    The selection of the final response uses NLS between the labels predicted by the BERT classifier and the expected labels (Section III.B).
  • domain assumption The training/test split contamination in the HuggingFace DailyDialog was handled by the rerun in Appendix F, so results are not invalidated.
    They disclose the duplicate issue and rerun automated metrics on the original split, showing similar trends; but the human evaluation was not rerun on the clean split.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Socio-Emotional Response Generation: A Human Evaluation Protocol for LLM-Based Conversational Systems." pith.science (2026). https://pith.science/paper/QUSMXDD7

@misc{pith2026241204492,
  author       = {Pith},
  title        = {Pith review of: Socio-Emotional Response Generation: A Human Evaluation Protocol for LLM-Based Conversational Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QUSMXDD7}},
  note         = {Machine review of arXiv:2412.04492}
}
read the original abstract

Conversational systems are now capable of producing impressive and generally relevant responses. However, we have no visibility nor control of the socio-emotional strategies behind state-of-the-art Large Language Models (LLMs), which poses a problem in terms of their transparency and thus their trustworthiness for critical applications. Another issue is that current automated metrics are not able to properly evaluate the quality of generated responses beyond the dataset's ground truth. In this paper, we propose a neural architecture that includes an intermediate step in planning socio-emotional strategies before response generation. We compare the performance of open-source baseline LLMs to the outputs of these same models augmented with our planning module. We also contrast the outputs obtained from automated metrics and evaluation results provided by human annotators. We describe a novel evaluation protocol that includes a coarse-grained consistency evaluation, as well as a finer-grained annotation of the responses on various social and emotional criteria. Our study shows that predicting a sequence of expected strategy labels and using this sequence to generate a response yields better results than a direct end-to-end generation scheme. It also highlights the divergences and the limits of current evaluation metrics for generated content. The code for the annotation platform and the annotated data are made publicly available for the evaluation of future models.

Figures

Figures reproduced from arXiv: 2412.04492 by the authors.

Figure 1
Figure 1. Visualisation of the conditional generation approach. NO-CD situation refers to direct response generation, while CD illustrates the conditioning [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. 4 dialogue acts used to annotate Daily Dialog, as well as some examples from the dataset to assist this task. [PITH_FULL_IMAGE:figures/full_fig_p013_2.png] view at source ↗
Figure 3
Figure 3. Definition of each socio-emotional criteria rated in this evaluation, as well as the rating scale used for each item [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figures from the paper (2 more)
Figure 1
Figure 1. Figure 1: 4 dialogue acts used to annotate Daily Dialog, as well as some examples from the dataset to assist this task. [PITH_FULL_IMAGE:figures/full_fig_p021_1.png]
Figure 2
Figure 2. Figure 2: Definition of each socio-emotional criteria rated in this evaluation, as well as the rating scale used for each item [PITH_FULL_IMAGE:figures/full_fig_p021_2.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

38 extracted references · 29 canonical work pages

  1. [9]

    Z. Yang, Z. Ren, W. Yufeng, S. Peng, H. Sun, X. Zhu, and X. Liao, ``Enhancing empathetic response generation by augmenting llms with small-scale empathetic models,'' arXiv preprint arXiv:2402.11801, 2024

    arXiv 2024

  2. [1]

    Galland, C

    L. Galland, C. Pelachaud, and F. Pecune, ``Seeing and hearing what has not been said; a multimodal client behavior classifier in motivational interviewing with interpretable fusion,'' 2023

    work page 2023

  3. [2]

    Vanel, A

    L. Vanel, A. Yacoubi, and C. Clavel, ``A new task for predicting emotions and dialogue strategies in task-oriented dialogue,'' in 2023 11th International Conference on Affective Computing and Intelligent Interaction (ACII). 1em plus 0.5em minus 0.4em Los Alamitos, CA, USA: IEEE Computer Society, sep 2023, pp. 1--8. [Online]. Available: https://doi.ieeecom...

    arXiv 2023

  4. [3]

    DeVault, R

    D. DeVault, R. Artstein, G. Benn, T. Dey, E. Fast, A. Gainer, K. Georgila, J. Gratch, A. Hartholt, M. Lor-Lhommet, G. Lucas, S. Marsella, F. Morbini, A. Nazarian, S. Scherer, G. Stratou, A. Suri, D. Traum, R. Wood, and L.-P. Morency, ``Simsensei kiosk: A virtual human interviewer for healthcare decision support,'' vol. 2, 01 2014, pp. 1061--1068

    work page 2014

  5. [4]

    Clavel, M

    C. Clavel, M. Labeau, and J. Cassell, ``Socio-conversational systems: Three challenges at the crossroads of fields,'' Frontiers in Robotics and AI, vol. 9, p. 937825, 2022

    work page 2022

  6. [5]

    H. Kim, J. Hessel, L. Jiang, P. West, X. Lu, Y. Yu, P. Zhou, R. L. Bras, M. Alikhani, G. Kim, M. Sap, and Y. Choi, ``Soda: Million-scale dialogue distillation with social commonsense contextualization,'' 2023

    work page 2023

  7. [6]

    Y.-J. Lee, D. Lee, J. Im, J. W. Sung, and H.-J. Choi, ``Investigating the effects of zero-shot chain-of-thought on empathetic dialogue generation,'' in NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023

    work page 2023

  8. [7]

    Z. Li, G. Chen, R. Shao, D. Jiang, and L. Nie, ``Enhancing the emotional generation capability of large language models via emotional chain-of-thought,'' arXiv preprint arXiv:2401.06836, 2024

    arXiv 2024

Show all 38 references
  1. [8]

    Abulimiti, C

    A. Abulimiti, C. Clavel, and J. Cassell, ``How about kind of generating hedges using end-to-end neural models?'' 2023

    2023

  2. [10]

    Rashkin, E

    H. Rashkin, E. M. Smith, M. Li, and Y.-L. Boureau, ``Towards empathetic open-domain conversation models: a new benchmark and dataset,'' in ACL, 2019

    2019

  3. [11]

    A. S. Raamkumar and Y. Yang, ``Empathetic conversational systems: A review of current advances, gaps, and opportunities,'' IEEE Transactions on Affective Computing, vol. 14, no. 4, p. 2722–2739, Oct. 2023. [Online]. Available: http://dx.doi.org/10.1109/TAFFC.2022.3226693

    2023

  4. [12]

    S. E. Finch, J. D. Finch, and J. D. Choi, ``Don ' t forget your ABC ' s: Evaluating the state-of-the-art in chat-oriented dialogue systems,'' in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), A. Rogers, J. Boyd-...

    2023

  5. [13]

    Q. Li, P. Li, Z. Ren, P. Ren, and Z. Chen, ``Knowledge bridging for empathetic dialogue generation,'' 2021

    2021

  6. [14]

    Thoppilan, D

    R. Thoppilan, D. D. Freitas, J. Hall, N. Shazeer, A. Kulshreshtha, H.-T. Cheng, A. Jin, T. Bos, L. Baker, Y. Du, Y. Li, H. Lee, H. S. Zheng, A. Ghafouri, M. Menegali, Y. Huang, M. Krikun, D. Lepikhin, J. Qin, D. Chen, Y. Xu, Z. Chen, A. Roberts, M. Bosma, V. Zhao, Y. Zhou, C.-...

    2022

  7. [15]

    Lee, C.-G

    Y.-J. Lee, C.-G. Lim, and H.-J. Choi, ``Does GPT -3 generate empathetic dialogues? a novel in-context example selection method and automatic evaluation metric for empathetic dialogue generation,'' in Proceedings of the 29th International Conference on Computational Linguistics...

    2022

  8. [16]

    J. Shin, P. Xu, A. Madotto, and P. Fung, ``Generating empathetic responses by looking ahead the user's sentiment,'' 2021

    2021

  9. [17]

    Y. Li, K. Li, H. Ning, X. Xia, Y. Guo, C. Wei, J. Cui, and B. Wang, ``Towards an online empathetic chatbot with emotion causes,'' in Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, ser. SIGIR ’21. 1em plus 0.5em ...

    2021

  10. [18]

    X. Lu, Y. Tian, Y. Zhao, and B. Qin, ``Retrieve, discriminate and rewrite: A simple and effective framework for obtaining affective response in retrieval-based chatbots,'' in Findings of the Association for Computational Linguistics: EMNLP 2021, 2021, pp. 1956--1969

    2021

  11. [19]

    Welivita, Y

    A. Welivita, Y. Xie, and P. Pu, ``A large-scale dataset for empathetic response generation,'' in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021, pp. 1251--1264

    2021

  12. [20]

    Vanel, A

    L. Vanel, A. Yacoubi, and C. Clavel, ``A survey of socio-emotional strategies for generation-based conversational agents,'' in Proceedings of the 15th International Conference on Agents and Artificial Intelligence - Volume 3: ICAART,, INSTICC. 1em plus 0.5em minus 0.4em SciTeP...

    2023

  13. [21]

    Z. Lin, P. Xu, G. I. Winata, F. B. Siddique, Z. Liu, J. Shin, and P. Fung, ``Caire: An empathetic neural chatbot,'' arXiv preprint arXiv:1907.12108, 2019

    1907 arXiv

  14. [22]

    S. Feng, N. Lubis, C. Geishauser, H.-c. Lin, M. Heck, C. van Niekerk, and M. Gašić, ``Emowoz: A large-scale corpus and labelling scheme for emotion recognition in task-oriented dialogue systems,'' 2021. [Online]. Available: https://arxiv.org/abs/2109.04919

    2021 arXiv

  15. [23]

    S. Liu, C. Zheng, O. Demasi, S. Sabour, Y. Li, Z. Yu, Y. Jiang, and M. Huang, ``Towards emotional support dialog systems,'' ArXiv, vol. abs/2106.01144, 2021

    2021 arXiv

  16. [24]

    Radford, J

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, ``Language models are unsupervised multitask learners,'' 2019. [Online]. Available: https://api.semanticscholar.org/CorpusID:160025533

    2019

  17. [25]

    Zhang, S

    Y. Zhang, S. Sun, M. Galley, Y.-C. Chen, C. Brockett, X. Gao, J. Gao, J. Liu, and B. Dolan, ``Dialogpt: Large-scale generative pre-training for conversational response generation,'' 2020

    2020

  18. [26]

    Y. Li, H. Su, X. Shen, W. Li, Z. Cao, and S. Niu, ``Dailydialog: A manually labelled multi-turn dialogue dataset,'' 2017

    2017

  19. [27]

    Post, ``A call for clarity in reporting BLEU scores,'' in Proceedings of the Third Conference on Machine Translation: Research Papers

    M. Post, ``A call for clarity in reporting BLEU scores,'' in Proceedings of the Third Conference on Machine Translation: Research Papers. 1em plus 0.5em minus 0.4em Belgium, Brussels: Association for Computational Linguistics, Oct. 2018, pp. 186--191. [Online]. Available: http...

    2018

  20. [28]

    Lin, `` ROUGE : A package for automatic evaluation of summaries,'' in Text Summarization Branches Out

    C.-Y. Lin, `` ROUGE : A package for automatic evaluation of summaries,'' in Text Summarization Branches Out. 1em plus 0.5em minus 0.4em Barcelona, Spain: Association for Computational Linguistics, 2004, pp. 74--81. [Online]. Available: https://www.aclweb.org/anthology/W04-1013

    2004

  21. [29]

    Popovi \'c , ``chr F : character n-gram F -score for automatic MT evaluation,'' in Proceedings of the Tenth Workshop on Statistical Machine Translation

    M. Popovi \'c , ``chr F : character n-gram F -score for automatic MT evaluation,'' in Proceedings of the Tenth Workshop on Statistical Machine Translation. 1em plus 0.5em minus 0.4em Lisbon, Portugal: Association for Computational Linguistics, Sep. 2015, pp. 392--395. [Online]...

    2015

  22. [30]

    Zhang, V

    T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi, ``Bertscore: Evaluating text generation with bert,'' in International Conference on Learning Representations, 2020. [Online]. Available: https://openreview.net/forum?id=SkeHuCVFDr

    2020

  23. [31]

    Jelinek, R

    F. Jelinek, R. L. Mercer, L. R. Bahl, and J. K. Baker, ``Perplexity—a measure of the difficulty of speech recognition tasks,'' The Journal of the Acoustical Society of America, vol. 62, no. S1, pp. S63--S63, 1977

    1977

  24. [32]

    A. Abid, A. Abdalla, A. Abid, D. Khan, A. Alfozan, and J. Zou, ``Gradio: Hassle-free sharing and testing of ml models in the wild,'' arXiv preprint arXiv:1906.02569, 2019

    1906 arXiv

  25. [33]

    Y. Wen, G. Luo, and L. Mou, ``An empirical study on the overlapping problem of open-domain dialogue datasets,'' 2022

    2022

  26. [34]

    Ni'mah, M

    I. Ni'mah, M. Fang, V. Menkovski, and M. Pechenizkiy, ``Nlg evaluation metrics beyond correlation analysis: An empirical metric preference checklist,'' 2023

    2023

  27. [35]

    Devlin, M.-W

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, ``Bert: Pre-training of deep bidirectional transformers for language understanding,'' 2019

    2019

  28. [36]

    Lewis, Y

    M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer, ``Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension,'' 2019

    2019

  29. [37]

    Touvron, L

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al., ``Llama 2: Open foundation and fine-tuned chat models,'' arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  30. [38]

    11em plus .33em minus .07em 4000 4000 100 4000 4000 500 `\.=1000 = #1 \@IEEEnotcompsoconly \@IEEEcompsoconly #1 * [1] 0pt [0pt][0pt] #1 * [1] 0pt [0pt][0pt] #1 * \| ** #1 \@IEEEauthorblockNstyle \@IEEEcompsocnotconfonly \@IEEEauthorblockAstyle \@IEEEcompsocnotconfonly \@IEEEco...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.