pith. sign in

arxiv: 2606.23124 · v1 · pith:FVTOGZYRnew · submitted 2026-06-22 · 💻 cs.CL · cs.AI

PRIDE: Privileged Information-enhanced Distillation for Empathetic Dialogue Generation

Pith reviewed 2026-06-26 08:35 UTC · model grok-4.3

classification 💻 cs.CL cs.AI
keywords knowledge distillationempathetic dialogue generationprivileged informationlarge language modelsmodel compressionattention mechanismalignment loss
0
0 comments X

The pith

Privileged information available only during training lets smaller models match or exceed larger ones at generating empathetic dialogues without extra inputs at inference.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to demonstrate that knowledge distillation for empathetic dialogue can be improved by injecting privileged information such as expert psychological annotations or future event summaries, which exist solely at training time. Standard distillation often fails to pass along the implicit contextual understanding needed for empathy, so the authors introduce a prompt that forces the teacher to break down feelings and situations, an attention mechanism that lets the student absorb the privileged signals, and a dual-alignment loss that matches both logits and features. If the approach works, resource-limited devices could run capable empathetic chat systems without needing the full teacher model or any privileged data after deployment.

Core claim

PRIDE transfers empathetic reasoning from a large teacher to a smaller student by using privileged information exclusively at training time through an empathy-reasoning prompt, multi-source attention, and a dual-alignment loss combining reversed KL divergence with maximum mean discrepancy, yielding student performance that is competitive with and sometimes exceeds the teacher on multi-modal and text-only empathetic dialogue datasets.

What carries the argument

The PRIDE pipeline, which routes privileged information through a step-by-step empathy-reasoning prompt for the teacher, a multi-source attention block for the student, and a dual-alignment loss at both logit and feature levels.

If this is right

  • Resource-constrained devices can host empathetic dialogue systems whose quality approaches that of much larger models.
  • Training pipelines for dialogue models can safely incorporate expensive annotations that are dropped before deployment.
  • The same distillation pattern could be applied to other generation tasks where subtle contextual cues matter but cannot be supplied at runtime.
  • Dual-level alignment losses become a standard tool for moving both surface outputs and internal representations from teacher to student.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the privileged signals prove task-specific, the method may need new annotation types when moved to non-empathetic dialogue domains.
  • The approach implicitly assumes the teacher already possesses the desired empathetic capability; any weakness in the teacher would be inherited by the student.
  • Future work could test whether the same privileged distillation improves robustness to adversarial or out-of-distribution user inputs.

Load-bearing premise

That the chosen privileged signals and the three transfer mechanisms can be combined so the student internalizes the teacher's empathetic reasoning and reproduces it at inference without ever seeing the privileged information again.

What would settle it

A controlled comparison in which a student trained with PRIDE shows no improvement over a student trained with ordinary distillation on the same teacher when both are evaluated on accuracy and semantic-relevance metrics for empathetic responses.

Figures

Figures reproduced from arXiv: 2606.23124 by Jiaqiang Wu, Shangfei Wang, Zhouan Zhu.

Figure 1
Figure 1. Figure 1: Examples of privileged information. their immense computational cost presents a formidable barrier to practical deployment in resource-constrained environments. To address this, knowledge distillation (KD) emerges as a com￾pelling strategy, which is designed to transfer the capabilities of large teacher models to smaller and more efficient student models [14, 15]. Despite its promise, standard KD often fai… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of our method. During training phase, privileged information and training data are provided as inputs, whereas privileged information is [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Results of pairwise human evaluation. D. Human Preference and Quality Analysis [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 3
Figure 3. Figure 3: Impact of Privileged Information Proportion. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Human evaluation scores [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Multi-source attention visualization. F. Efficiency Analysis To quantify the practical benefits of our distillation method, we compared the inference efficiency of the teacher and student models. We measured the memory usage and inference speed (tokens per second) on a single NVIDIA RTX 4090 (24GB) GPU with a batch size of 1. As shown in the table, the student models achieve significant efficiency gains, m… view at source ↗
read the original abstract

Large language models have demonstrated significant capabilities in generating diverse and context-aware responses for empathetic dialogue. However, their computational demands severely limit their deployment in resource-constrained environments. While knowledge distillation offers a promising compression solution, it often fails to transfer the nuanced understanding essential for empathy, as it overlooks the implicit contextual cues that guide human connection. To bridge this gap, we propose a \textbf{pr}ivileged \textbf{i}nformation-enhanced knowledge \textbf{d}istillation method for \textbf{e}mpathetic dialogue generation (PRIDE). Our method leverages privileged information, such as expert psychological annotations or future event summaries, which is available exclusively during training but unavailable at inference time. This allows us to transfer the teacher model's empathetic reasoning to smaller models without relying on extra inputs during deployment. Specifically, PRIDE has three key components: (1) An empathy-reasoning prompt that guides the teacher to explicitly decompose the empathetic process into understanding feelings and analyzing situations step-by-step; (2) A multi-source attention mechanism that directs the student to effectively integrate privileged information; (3) A dual-alignment loss that combines reversed Kullback-Leibler divergence and maximum mean discrepancy to ensure robust knowledge transfer at both logit and feature levels. Experiments on multi-modal and text-only datasets demonstrate that our method achieves competitive performance, and in some cases matches or even surpasses larger teacher models in terms of accuracy and semantic relevance.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 0 minor

Summary. The paper proposes PRIDE, a privileged information-enhanced knowledge distillation method for empathetic dialogue generation from large language models to smaller student models. Privileged information (e.g., expert psychological annotations or future event summaries) is used only during training via three components: an empathy-reasoning prompt that decomposes the empathetic process, a multi-source attention mechanism for integrating privileged information, and a dual-alignment loss combining reversed KL divergence with maximum mean discrepancy at logit and feature levels. The central claim is that this enables the student to acquire nuanced empathetic understanding without extra inputs at inference, with experiments on multi-modal and text-only datasets showing competitive or superior performance to teacher models in accuracy and semantic relevance.

Significance. If the empirical claims hold, the approach could meaningfully advance efficient deployment of empathetic dialogue systems in resource-constrained settings by transferring complex reasoning through distillation without inference-time overhead. The combination of privileged information with prompt-based reasoning and dual-level alignment represents a targeted engineering contribution to knowledge distillation in dialogue tasks.

major comments (1)
  1. Abstract: The central empirical claim—that the method achieves competitive performance and in some cases matches or surpasses larger teacher models—is asserted without any reported metrics, baselines, dataset details, statistical tests, or quantitative results, rendering the claim uninspectable and unsupported by evidence in the manuscript.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for their thoughtful review and for highlighting the need for greater transparency in the abstract. We address the single major comment below. The full manuscript contains the requested experimental details, metrics, and comparisons; we will revise the abstract to better surface key quantitative results.

read point-by-point responses
  1. Referee: Abstract: The central empirical claim—that the method achieves competitive performance and in some cases matches or surpasses larger teacher models—is asserted without any reported metrics, baselines, dataset details, statistical tests, or quantitative results, rendering the claim uninspectable and unsupported by evidence in the manuscript.

    Authors: We agree that the abstract would be strengthened by including concrete metrics and dataset references. The full manuscript reports results on the multi-modal EmpatheticDialogues and text-only DailyDialog datasets, with automatic metrics (BLEU, ROUGE, Distinct, Empathy Accuracy) and human evaluations comparing PRIDE against the teacher LLM, standard KD baselines, and prior empathetic dialogue models. Statistical significance is assessed via paired t-tests. In revision we will add a concise sentence to the abstract citing the primary datasets and the key finding that PRIDE matches or exceeds the teacher on empathy accuracy while using a 7B student model. This change directly addresses the inspectability concern without altering the abstract's length constraints. revision: yes

Circularity Check

0 steps flagged

No significant circularity

full rationale

The paper describes an empirical engineering method (PRIDE) consisting of a prompt, attention mechanism, and dual-alignment loss for knowledge distillation. No equations, derivations, or first-principles claims are present in the abstract or described structure. Performance results are presented as experimental outcomes on datasets rather than predictions derived from fitted parameters or self-referential definitions. No self-citations are invoked as load-bearing uniqueness theorems or ansatzes. The central claim reduces to standard distillation engineering choices evaluated externally, making the work self-contained against benchmarks without circular reduction.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

No mathematical model, free parameters, or invented entities are described in the abstract; the contribution is an empirical method whose assumptions (availability of privileged annotations at training time, effectiveness of the dual loss) are stated at the prose level only.

pith-pipeline@v0.9.1-grok · 5787 in / 1254 out tokens · 24526 ms · 2026-06-26T08:35:42.636943+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

43 extracted references · 1 canonical work pages

  1. [1]

    Empathy: The charismatic chimera,

    Macarov and David, “Empathy: The charismatic chimera,”Journal of Education for Social Work, vol. 14, no. 3, pp. 86–92, 1978

  2. [2]

    Damon, R

    W. Damon, R. M. Lerner, and N. Eisenberg,Handbook of child psychology, social, emotional, and personality development. John Wiley & Sons, 2006

  3. [3]

    Mime: Mimicking emotions for empa- thetic response generation,

    D. Ghosal, B. P. Majumder, S. Poria, A. Gelbukh, and E. Cambria, “Mime: Mimicking emotions for empa- thetic response generation,” inProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020, pp. 7645–7655

  4. [4]

    MoEL: Mixture of empathetic listeners,

    Z. Lin, A. Madotto, J. Shin, P. Xu, and P. Fung, “MoEL: Mixture of empathetic listeners,” inProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP). Hong Kong, China: Association for Compu- tational Linguistics, Nov. 2019, pp. 121–132

  5. [5]

    Em- pDG: Multi-resolution interactive empathetic dialogue generation,

    Q. Li, H. Chen, Z. Ren, P. Ren, Z. Tu, and Z. Chen, “Em- pDG: Multi-resolution interactive empathetic dialogue generation,” inProceedings of the 28th International Conference on Computational Linguistics. Barcelona, Spain (Online): International Committee on Computa- tional Linguistics, Dec. 2020, pp. 4454–4466. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. ...

  6. [6]

    Cem: Commonsense-aware empathetic response generation,

    S. Sabour, C. Zheng, M. Huang, and M. Huang, “Cem: Commonsense-aware empathetic response generation,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 10, 2022, pp. 11 229–11 237

  7. [7]

    Empathetic dialogue generation via sensitive emotion recognition and sensible knowledge selection,

    L. Wang, J. Li, Z. Lin, F. Meng, C. Yang, W. Wang, and J. Zhou, “Empathetic dialogue generation via sensitive emotion recognition and sensible knowledge selection,” inFindings of the Association for Computational Lin- guistics: EMNLP 2022. Abu Dhabi, United Arab Emi- rates: Association for Computational Linguistics, Dec. 2022, pp. 4634–4645

  8. [8]

    CASE: Aligning coarse-to-fine cognition and affection for empathetic response generation,

    J. Zhou, C. Zheng, B. Wang, Z. Zhang, and M. Huang, “CASE: Aligning coarse-to-fine cognition and affection for empathetic response generation,” inProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), A. Rogers, J. Boyd-Graber, and N. Okazaki, Eds. Toronto, Canada: Association for Computational Lin...

  9. [9]

    Don’t lose yourself! empathetic response generation via explicit self-other awareness,

    W. Zhao, Y . Zhao, X. Lu, and B. Qin, “Don’t lose yourself! empathetic response generation via explicit self-other awareness,” inFindings of the Association for Computational Linguistics: ACL 2023. Toronto, Canada: Association for Computational Linguistics, Jul. 2023, pp. 13 331–13 344. [Online]. Available: https://aclanthology.org/2023.findings-acl.843

  10. [10]

    Talk with human-like agents: Empathetic dialogue through perceptible acoustic reception and reaction,

    H. Yan, Y . Zhu, K. Zheng, B. Liu, H. Cao, D. Jiang, and L. Xu, “Talk with human-like agents: Empathetic dialogue through perceptible acoustic reception and reaction,” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), L.-W. Ku, A. Martins, and V . Srikumar, Eds. Bangkok, Thailand: Associatio...

  11. [11]

    STICKERCONV: Generating multimodal empathetic responses from scratch,

    Y . Zhang, F. Kong, P. Wang, S. Sun, S. SWangLing, S. Feng, D. Wang, Y . Zhang, and K. Song, “STICKERCONV: Generating multimodal empathetic responses from scratch,” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), L.-W. Ku, A. Martins, and V . Srikumar, Eds. Bangkok, Thailand: Association f...

  12. [12]

    EmpathyEar: An open-source avatar multimodal empathetic chatbot,

    H. Fei, H. Zhang, B. Wang, L. Liao, Q. Liu, and E. Cam- bria, “EmpathyEar: An open-source avatar multimodal empathetic chatbot,” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Y . Cao, Y . Feng, and D. Xiong, Eds. Bangkok, Thailand: Association for Computational Linguistics, Aug...

  13. [13]

    Is chatgpt equipped with emotional dialogue capabilities?

    W. Zhao, Y . Zhao, X. Lu, S. Wang, Y . Tong, and B. Qin, “Is chatgpt equipped with emotional dialogue capabilities?” 2023

  14. [14]

    A survey on knowledge distillation of large language models,

    X. Xu, M. Li, C. Tao, T. Shen, R. Cheng, J. Li, C. Xu, D. Tao, and T. Zhou, “A survey on knowledge distillation of large language models,” 2024. [Online]. Available: https://arxiv.org/abs/2402.13116

  15. [15]

    Distilling the knowledge in a neural network,

    G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” 2015

  16. [16]

    Learning using privileged information: Similarity control and knowledge transfer

    V . Vapnik, R. Izmailovet al., “Learning using privileged information: Similarity control and knowledge transfer.” J. Mach. Learn. Res., vol. 16, no. 1, pp. 2023–2049, 2015

  17. [17]

    Empathetic conversa- tional systems: A review of current advances, gaps, and opportunities,

    A. S. Raamkumar and Y . Yang, “Empathetic conversa- tional systems: A review of current advances, gaps, and opportunities,”IEEE Transactions on Affective Comput- ing, vol. 14, no. 4, pp. 2722–2739, 2023

  18. [18]

    Knowledge bridging for empathetic dialogue generation,

    Q. Li, Y . Zhang, C. Liang, N. Li, and J. Li, “Knowledge bridging for empathetic dialogue generation,” inProceed- ings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 18, 2021, pp. 15 727–15 735

  19. [19]

    A kernel method for the two-sample- problem,

    A. Gretton, K. Borgwardt, M. Rasch, B. Sch ¨olkopf, and A. Smola, “A kernel method for the two-sample- problem,” inAdvances in Neural Information Processing Systems, B. Sch ¨olkopf, J. Platt, and T. Hoffman, Eds., vol. 19. MIT Press, 2006. [Online]. Available: https://proceedings.neurips.cc/paper files/paper/2006/ file/e9fb2eda3d9c55a0d89c98d6c54b5b3e-Paper.pdf

  20. [20]

    Minillm: Knowledge distillation of large language models,

    Y . Gu, L. Dong, F. Wei, and M. Huang, “Minillm: Knowledge distillation of large language models,” 2024. [Online]. Available: https://arxiv.org/abs/2306.08543

  21. [21]

    Policy distillation,

    A. A. Rusu, S. G. Colmenarejo, C. Gulcehre, G. Desjardins, J. Kirkpatrick, R. Pascanu, V . Mnih, K. Kavukcuoglu, and R. Hadsell, “Policy distillation,” arXiv preprint arXiv:1511.06295, 2015

  22. [22]

    Knowledge distillation: A survey,

    J. Gou, B. Yu, S. J. Maybank, and D. Tao, “Knowledge distillation: A survey,”International Journal of Computer Vision, vol. 129, no. 6, pp. 1789–1819, 2021

  23. [23]

    On-policy distillation of lan- guage models: Learning from self-generated mistakes,

    R. Agarwal, N. Vieillard, Y . Zhou, P. Stanczyk, S. Ramos, M. Geist, and O. Bachem, “On-policy distillation of lan- guage models: Learning from self-generated mistakes,” Jan 2024

  24. [24]

    Self-rewarding language models,

    W. Yuan, R. Y . Pang, K. Cho, X. Li, S. Sukhbaatar, J. Xu, and J. Weston, “Self-rewarding language models,” 2025. [Online]. Available: https://arxiv.org/abs/2401.10020

  25. [25]

    Measuring individual differences in empa- thy: Evidence for a multidimensional approach

    M. H. Davis, “Measuring individual differences in empa- thy: Evidence for a multidimensional approach.”Journal of personality and social psychology, vol. 44, no. 1, p. 113, 1983

  26. [26]

    COMET: Commonsense trans- formers for automatic knowledge graph construction,

    A. Bosselut, H. Rashkin, M. Sap, C. Malaviya, A. Ce- likyilmaz, and Y . Choi, “COMET: Commonsense trans- formers for automatic knowledge graph construction,” in Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics. Florence, Italy: Association for Computational Linguistics, Jul. 2019, pp. 4762–4779

  27. [27]

    An iterative associative memory model for empathetic response generation,

    Z. Yang, Z. Ren, W. Yufeng, H. Sun, C. Chen, X. Zhu, and X. Liao, “An iterative associative memory model for empathetic response generation,” inProceedings of the 62nd Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), L.-W. Ku, A. Martins, and V . Srikumar, Eds. Bangkok, Thailand: Association for Computational Ling...

  28. [28]

    Conceptnet 5.5: An open multilingual graph of general knowledge,

    R. Speer, J. Chin, and C. Havasi, “Conceptnet 5.5: An open multilingual graph of general knowledge,” inPro- ceedings of the AAAI conference on artificial intelligence, vol. 31, no. 1, 2017

  29. [29]

    Towards empathetic open-domain conversation models: A new benchmark and dataset,

    H. Rashkin, E. M. Smith, M. Li, and Y .-L. Boureau, “Towards empathetic open-domain conversation models: A new benchmark and dataset,” inProceedings of the 57th Annual Meeting of the Association for Compu- tational Linguistics. Florence, Italy: Association for Computational Linguistics, Jul. 2019, pp. 5370–5381

  30. [30]

    Medic: A multimodal empathy dataset in counseling,

    Z. Zhu, C. Li, J. Pan, X. Li, Y . Xiao, Y . Chang, F. Zheng, and S. Wang, “Medic: A multimodal empathy dataset in counseling,” inProceedings of the 31st ACM International Conference on Multimedia, ser. MM ’23. New York, NY , USA: Association for Computing Machinery, 2023, p. 6054–6062. [Online]. Available: https://doi.org/10.1145/3581783.3612346

  31. [31]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, b. ichter, F. Xia, E. Chi, Q. V . Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,” inAdvances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., vol. 35. Curran Associates, Inc., 2022, pp. 24 824–24 837. [Onlin...

  32. [32]

    Abkd: Pursuing a proper allocation of the probability mass in knowledge distillation viaα-β- divergence,

    G. Wang, Z. Yang, Z. Wang, S. Wang, Q. Xu, and Q. Huang, “Abkd: Pursuing a proper allocation of the probability mass in knowledge distillation viaα-β- divergence,”arXiv preprint arXiv:2505.04560, 2025

  33. [33]

    Qwen2.5-vl technical report,

    S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y . Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y . Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin, “Qwen2.5-vl technical report,”arXiv preprint arXiv:2502.13923, 2025

  34. [34]

    Llava-gemma: Accelerating multimodal foundation models with a compact language model,

    M. Hinck, M. L. Olson, D. Cobbley, S.-Y . Tseng, and V . Lal, “Llava-gemma: Accelerating multimodal foundation models with a compact language model,”

  35. [35]

    Available: https://arxiv.org/abs/2404

    [Online]. Available: https://arxiv.org/abs/2404. 01331

  36. [36]

    Gemma 3,

    G. Team, “Gemma 3,” 2025. [Online]. Available: https://goo.gle/Gemma3Report

  37. [37]

    Lightpaff: A two-stage distillation frame- work for pre-training and fine-tuning,

    K. Song, H. Sun, X. Tan, T. Qin, J. Lu, H. Liu, and T.-Y . Liu, “Lightpaff: A two-stage distillation frame- work for pre-training and fine-tuning,”arXiv preprint arXiv:2004.12817, 2020

  38. [38]

    Self-distillation: Towards efficient and compact neural networks,

    L. Zhang, C. Bao, and K. Ma, “Self-distillation: Towards efficient and compact neural networks,”IEEE Trans- actions on Pattern Analysis and Machine Intelligence, vol. 44, no. 8, pp. 4388–4403, 2021

  39. [39]

    Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter,

    V . Sanh, L. Debut, J. Chaumond, and T. Wolf, “Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter,”arXiv preprint arXiv:1910.01108, 2019

  40. [40]

    Towards multimodal empathetic re- sponse generation: A rich text-speech-vision avatar-based benchmark,

    H. Zhang, Z. Meng, M. Luo, H. Han, L. Liao, E. Cam- bria, and H. Fei, “Towards multimodal empathetic re- sponse generation: A rich text-speech-vision avatar-based benchmark,” inProceedings of the ACM on Web Confer- ence 2025, 2025, pp. 2872–2881

  41. [41]

    A diversity-promoting objective function for neural conversation models,

    J. Li, M. Galley, C. Brockett, J. Gao, and B. Dolan, “A diversity-promoting objective function for neural conversation models,” inProceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, K. Knight, A. Nenkova, and O. Rambow, Eds. San Diego, California: Association for C...

  42. [42]

    Bertscore: Evaluating text generation with BERT,

    T. Zhang, V . Kishore, F. Wu, K. Q. Weinberger, and Y . Artzi, “Bertscore: Evaluating text generation with BERT,” in8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. [Online]. Available: https://openreview.net/forum?id= SkeHuCVFDr

  43. [43]

    The equivalence of weighted kappa and the intraclass correlation coefficient as mea- sures of reliability,

    J. L. Fleiss and J. Cohen, “The equivalence of weighted kappa and the intraclass correlation coefficient as mea- sures of reliability,”Educational and psychological mea- surement, vol. 33, no. 3, pp. 613–619, 1973