Pith. sign in

REVIEW 3 major objections 5 minor 41 references

Is my Meeting Summary Good? Estimating Quality with a Multi-LLM Evaluator

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A three-stage LLM pipeline out-correlates existing metrics on meeting summary quality by about 0.25.

desk verdict The base three-step single-aspect architecture shows real, consistent gains over G-Eval-4, but the headline self-training improvement is evaluated on the same 170 samples that generated the feedback, so the 0.25 average claim does not yet hold. read the letter →

arxiv 2411.18444 v1 pith:7ZOPIX2D submitted 2024-11-27 cs.CL cs.AI

Frederic Kirstein , Terry Ruas , Bela Gipp This is my paper · ORCID
classification cs.CLcs.AI
keywords meetingsummarizationevaluationLLM-basedmulti-agentdiscussionself-trainingerrordetectionQMSumMistakehumanalignmentG-Eval
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that an automatic evaluator can judge meeting summaries almost as reliably as human reviewers when the evaluation is broken into error-type-specific steps instead of a single overall score. The proposed MESA framework, built on GPT-4o, first identifies candidate error instances in a summary, then rates the severity of each candidate, then assigns a score to that error type, repeating this for eight error categories. A multi-agent discussion step refines the candidate lists and severity ratings, and a self-training loop adjusts the system's scoring behavior using a small set of human annotations. On a modified QMSum Mistake dataset of 170 meeting summaries, MESA's correlations with human judgments are, on average, 0.25 higher than those of previous methods. If this holds, meeting summaries could be quality-checked automatically without the cost and delays of human evaluation.

What carries the argument

The load-bearing mechanism is MESA's three-step, single-aspect evaluation pipeline: an identification pass that lists candidate error instances with certainty scores, a severity pass that judges the existence and gravity of each instance, and a scoring pass that turns the observed errors into a Likert score per error type. This pipeline is wrapped in a multi-agent discussion protocol (a draft generator, three challengers, and a moderator) that refines the outputs, and a self-training loop in which an LLM judge compares MESA's reasoning and scores against human annotations, generating feedback appended to future prompts. The weighted combination of per-type scores (Equation 1) and the mapping to a quality score (Equation 2) carry the final rating.

What would settle it

Run MESA on a held-out split of meeting summaries, for example 100 newly annotated summaries, and compare its Point-Biserial, Spearman, and Kendall correlations with human judgments against the correlations reported on the 170 samples used for self-training; if the correlations drop to the level of G-Eval-4 or below, the claimed 0.25 average improvement would be shown to be an artifact of feedback derived from the test samples.

Watch

Extended reading notes

Core claim

The central claim is that error-specific, three-step evaluation carried out by multiple LLM agents, followed by feedback-based self-training, makes an LLM-based evaluator substantially more aligned with human judgment than existing automatic metrics for meeting summaries. In MESA, each of eight error types is assessed separately: a first LLM pass lists potential error instances with chain-of-thought reasoning and confidence scores, a second pass rates each instance's severity and whether it is a real error, and a third pass converts the observed errors into a Likert score for that type. A multi-agent discussion protocol, with a draft generator, three challengers, and a moderator, refines the intermediate outputs and reduces score variance. The per-type scores are combined by a confidence-weighted sum, mapped to a 0–5 impact score and a 1–10 quality score, and a self-training loop uses an LLM judge to compare MESA's outputs with human annotations, appending the resulting feedback to the prompts. The authors report that with GPT-4o, MESA attains mid to high Point-Biserial correlations with human error detection and mid Spearman and Kendall correlations on error impact, on average 0.25 higher than prior methods.

Load-bearing premise

The reported improvement rests on the assumption that the self-training loop, which learned from the very same annotated summaries used to compute the reported correlations, transfers to summaries it has not seen.

Editorial extensions

If this is right

  • Meeting summary evaluation can move from costly human annotation to automatic, explanation-providing assessment at a fraction of the cost.
  • MESA's error-type-specific scores make it usable for feedback-based summary refinement, since each detected error instance is pinpointed in the text.
  • The self-training loop suggests that a small set of human annotations, on the order of 170 samples, can substantially reshape an LLM evaluator's scoring behavior for a new error taxonomy.
  • Because MESA accepts custom error guidelines, the framework should transfer to other summarization domains and to tasks with limited human-labeled data.
  • The single-model variant without multi-agent discussion offers a cheaper alternative with only a slight performance drop, making the approach feasible in lower-budget settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the largest gain from self-training appears to be recalibration of score severity rather than better error detection; an implicit suggestion is that LLM evaluators are already strong detectors but systematically over-penalize, so prompt-level calibration may substitute for more expensive fine-tuning in other domains.
  • A natural extension is transferring the same three-step pipeline to other long-document genres, such as news or medical records, after replacing the error taxonomy; a quick probe would be to evaluate MESA on the original QMSum test set with human judgments.
  • An open risk the paper does not resolve is whether the self-training feedback memorizes the 170 annotated summaries; an explicit held-out evaluation would settle whether the 0.25 average improvement is real or an artifact.
  • The weighting scheme that prioritizes omission, hallucination, and irrelevance over readability errors encodes an implicit value judgment about what makes a meeting summary trustworthy; downstream users may want to expose those importance parameters as a tunable interface.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The manuscript presents MESA, a multi-stage LLM evaluator for meeting summaries. MESA assesses each of eight error types through a single-aspect, three-step pipeline (candidate error identification, severity rating, scoring), optionally refines decisions through multi-agent discussion, and appends self-generated feedback derived from human annotations to its prompts. The authors evaluate on a modified QMSum Mistake dataset (reported as 170 samples in Section 2.1 and the Limitations section, but as 200 meetings in Appendix D.1) using ROUGE, BERTScore, G-Eval-4, and four MESA variants. They report higher point-biserial correlations for error presence and higher Spearman/Kendall correlations for error-impact severity, with average gains of roughly 0.2-0.25 over G-Eval-4, and conclude that MESA is a flexible, high-performing automatic evaluator.

Significance. If the reported gains generalize, MESA would be a practically useful contribution because automatic evaluation of meeting summaries remains difficult and current metrics correlate poorly with human judgment. The paper's strengths include a systematic component analysis (Tables 1 and 2), detailed prompts in Appendix A, a documented annotation process with inter-annotator agreement, and a clear statement that code and data will be released. The main limitation is that the self-training component is evaluated on the same annotations used to generate its feedback, so the headline improvements of the self-trained variants (Single-1 and Multi-1) are not yet supported as evidence of generalization.

major comments (3)
  1. [Section 2.5 / Section 3.1 / Tables 3-4] The self-training mechanism in Section 2.5 uses human annotations to create feedback reports, and Section 3.1 reports the correlations of Single-1 and Multi-1 on the same modified QMSum Mistake dataset; no held-out split or cross-validation is described. Because the feedback is derived from the same 170 samples on which correlation is computed, the improvements of Single-1/Multi-1 over Single-0/Multi-0 (e.g., Table 4, OM: -0.27 to -0.42 for Single; -0.27 to -0.58 for Multi) may reflect fitting to the evaluation set. Please provide a held-out evaluation, such as generating feedback on a training subset and reporting correlations on a disjoint test subset, or leave-one-out cross-validation; alternatively, clearly relabel the self-training results as in-sample demonstrations rather than evidence of generalization.
  2. [Section 2.1 / Limitations / Appendix D.1] The paper is inconsistent about the size of the evaluation set: Section 2.1 says the dataset 'comprising 170 samples,' the Limitations section repeats '170 samples,' but Appendix D.1 (Table 10) reports 'QMSum Mistake 200 (169)' meetings. This discrepancy affects the interpretation of all correlations and p-values; please reconcile the count and specify exactly how many summaries and annotations were used in each experiment.
  3. [Section 3.2 / Abstract] The averaged gains are reported as '0.1' and '0.15' in Section 3.2 and as '0.25' in the abstract, but no confidence intervals, standard errors, or multiple-comparison corrections are given for the correlations in Tables 3 and 4. Given the small sample size and eight error types, significance stars alone are insufficient to support the headline average improvement; please report uncertainty estimates or per-error-type effect sizes with confidence intervals, and define precisely how the aggregate gap is computed.
minor comments (5)
  1. [Figure 1 / Section 2.3 / Section 2.5] There are several typographical errors: Figure 1's caption uses 'singel-aspect' instead of 'single-aspect,' Section 2.3's heading uses 'Chalenge II' instead of 'Challenge II,' and Section 2.5 uses 'MESAarchitecture' instead of 'MESA architecture.'
  2. [Section 2.1 / Tables 2-5 / Table 9] The abbreviation for redundancy is inconsistent: RED is used in Section 2.1 and Table 9, while REP is used in Tables 2-5; please unify the notation.
  3. [Appendix A] The prompt templates state that responses must use double quotes in JSON, but the example output format in Figures 3 and 4 shows single-quoted keys and values; align the examples with the stated strict-format requirement.
  4. [Section 3.1] The G-Eval-4 baseline is not fully specified; please report the exact prompt template, the number of runs, and the aggregation procedure, since LLM-based evaluation is known to be sensitive to sampling temperature and prompt wording.
  5. [Table 1] Table 1 reports balanced accuracy for the single-step and three-step approaches but does not include sample sizes or confidence intervals; adding these would help assess the stability of the reported improvements.

Circularity Check

1 steps flagged · score 6.0 of 10

Self-training feedback is derived from and evaluated on the same 170 annotated samples, making the reported self-trained gains an in-sample fit rather than a prediction.

  1. fitted input called prediction [Section 2.5 (self-training mechanism), Section 3.1 (evaluation setup), Section 3.2 (results)]
    "The self-training mechanism, inspired by Wang et al. (2024)'s self-teaching and Kirstein et al. (2024a)'s feedback approach, influences the evaluation behavior by comparing MESA's assessments with available human annotations. ... A second GPT4 judge is tasked to detect patterns in the per-sample feedback and provides a consolidated report for each error type on what should be considered or treated differently during evaluation. This report is then used in the following three-step assessment, being appended to the original task describing prompt to steer the detection and evaluation behavior."

    The only evaluation dataset is the modified QMSum Mistake dataset of 170 samples (Section 2.1). The self-training consolidated report is generated by comparing MESA's outputs and reasoning traces to the human annotations on this dataset, and is then appended to the prompts used to produce the reported scores. Correlations for Single-1 and Multi-1 are computed on the same 170 samples (Tables 3 and 4), with no held-out split or cross-validation described. The feedback report is therefore a parameter fitted to the test labels, and the subsequent correlation with those same labels is an in-sample measure.

full rationale

The central circularity is confined to the self-training component. Section 2.5 states that the feedback report is produced by comparing MESA's assessments with human annotations and then appended to the evaluation prompts. Section 3.1 evaluates MESA on the same modified QMSum Mistake dataset with zero or one self-training iterations, and no train/test separation is described. Consequently, the improvement of Single-1/Multi-1 over Single-0/Multi-0 in Tables 3 and 4 is an in-sample fit to the very human judgments used to generate the feedback. The three-step pipeline and multi-agent discussion are evaluated without such fitting and are not circular. The dataset-size inconsistency between the Limitations section (170 samples) and Appendix D.1 (200 meetings, 169 erroneous) is noted but does not itself constitute circularity. Because the paper's strongest headline claim depends on the self-trained variants, the circularity score is 6: partial circularity, with the self-trained prediction reducing to a fit to its own evaluation labels.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The framework rests on human annotations as the gold standard, on the eight error definitions being prompt-operationalizable, on LLM confidence scores being meaningful for weighting, and on an LLM judge producing reliable feedback. Each is a domain assumption rather than a proven fact.

free parameters (1)
  • Error impact importance weights i_n in Eq. 1 = 1.0 default; 1.1 for OM, HAL, IRR; 0.9 for REP, INC, LAN
    Hand-set in Section 2.5 to prioritize errors that affect factual trustworthiness; they affect the overall impact score but not the per-error-type correlations in Tables 3 and 4.
assumptions (4)
  • domain assumption Human annotations on the modified QMSum Mistake dataset are an accurate gold standard for meeting summary quality.
    Section 2.1 and Appendix D.2: all evaluation and self-training feedback are anchored to labels produced by four author-selected annotators (Krippendorff's alpha 0.793).
  • domain assumption The eight error definitions from prior work are complete and can be followed by LLMs from prompts.
    Section 2.1 and Appendix C define RED, INC, LAN, OM, COR, HAL, STR, IRR; the paper asserts, but does not prove, that prompting these definitions achieves reliable error detection.
  • domain assumption LLM-reported confidence scores are meaningful enough to weight error impact in Eq. 1.
    Section 2.5 uses c_n, the scaled confidence score, as a weight; no calibration analysis of these confidence scores is provided.
  • domain assumption An LLM judge can produce accurate feedback reports from per-sample comparisons with human annotations.
    Section 2.5 relies on GPT-4 judges to detect patterns and generate the feedback used in self-training; the quality of this feedback is not independently validated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Is my Meeting Summary Good? Estimating Quality with a Multi-LLM Evaluator." pith.science (2026). https://pith.science/paper/7ZOPIX2D

@misc{pith2026241118444,
  author       = {Pith},
  title        = {Pith review of: Is my Meeting Summary Good? Estimating Quality with a Multi-LLM Evaluator},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7ZOPIX2D}},
  note         = {Machine review of arXiv:2411.18444}
}
read the original abstract

The quality of meeting summaries generated by natural language generation (NLG) systems is hard to measure automatically. Established metrics such as ROUGE and BERTScore have a relatively low correlation with human judgments and fail to capture nuanced errors. Recent studies suggest using large language models (LLMs), which have the benefit of better context understanding and adaption of error definitions without training on a large number of human preference judgments. However, current LLM-based evaluators risk masking errors and can only serve as a weak proxy, leaving human evaluation the gold standard despite being costly and hard to compare across studies. In this work, we present MESA, an LLM-based framework employing a three-step assessment of individual error types, multi-agent discussion for decision refinement, and feedback-based self-training to refine error definition understanding and alignment with human judgment. We show that MESA's components enable thorough error detection, consistent rating, and adaptability to custom error guidelines. Using GPT-4o as its backbone, MESA achieves mid to high Point-Biserial correlation with human judgment in error detection and mid Spearman and Kendall correlation in reflecting error impact on summary quality, on average 0.25 higher than previous methods. The framework's flexibility in adapting to custom error guidelines makes it suitable for various tasks with limited human-labeled data.

Figures

Figures reproduced from arXiv: 2411.18444 by the authors.

Figure 1
Figure 1. Architecture of MESA displaying the singel-aspect assessment using three stages and the self-training [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Multi-agent discussion protocol used, con [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The prompt template used to task an LLM instance to identify potential error instances. [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The prompt template used to task an LLM instance to rate detected error instance. [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: The prompt template used to task an LLM instance to perform the scoring. [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 11 canonical work pages

  1. [1]

    Marah Abdin, Sam Ade Jacobs, and Ammar Ahmad Awan. 2024. https://doi.org/10.48550/arXiv.2404.14219 Phi-3 Technical Report : A Highly Capable Language Model Locally on Your Phone . Preprint, arXiv:2404.14219

    arXiv doi:10.48550/arxiv.2404.14219 2024

  2. [2]

    Meta AI. 2024. Llama 3.2: Revolutionizing edge AI and vision with open, customizable models. https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/

    work page 2024

  3. [3]

    Peters, and Arman Cohan

    Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. https://arxiv.org/abs/2004.05150 Longformer: The Long-Document Transformer . Preprint, arXiv:2004.05150

    arXiv 2020

  4. [4]

    Yapei Chang, Kyle Lo, Tanya Goyal, and Mohit Iyyer. 2024. https://doi.org/10.48550/arXiv.2310.00785 BooookScore : A systematic exploration of book-length summarization in the era of LLMs . Preprint, arXiv:2310.00785

    arXiv doi:10.48550/arxiv.2310.00785 2024

  5. [5]

    Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2024. https://doi.org/10.18653/v1/2024.naacl-long.365 GPTScore : Evaluate as You Desire . In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics : Human Language Technologies ( Volume 1: Long Papers ) , pages 6556--6576, Mexico City, Mex...

    doi:10.18653/v1/2024.naacl-long.365 2024

  6. [6]

    Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, and Iryna Gurevych. 2024. https://doi.org/10.18653/v1/2024.naacl-long.366 A Survey of Confidence Estimation and Calibration in Large Language Models . In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics : Human Language Techno...

    doi:10.18653/v1/2024.naacl-long.366 2024

  7. [7]

    Yebowen Hu, Timothy Ganter, Hanieh Deilamsalehy, Franck Dernoncourt, Hassan Foroosh, and Fei Liu. 2023. https://doi.org/10.18653/v1/2023.acl-long.906 MeetingBank : A Benchmark Dataset for Meeting Summarization . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics ( Volume 1: Long Papers ) , pages 16409--16423, Toront...

    doi:10.18653/v1/2023.acl-long.906 2023

  8. [8]

    Maor Ivgi, Uri Shaham, and Jonathan Berant. 2022. https://doi.org/10.48550/arXiv.2208.00748 Efficient Long-Text Understanding with Short-Text Models . Preprint, arXiv:2208.00748

    work page Pith review arXiv doi:10.48550/arxiv.2208.00748 2022

Show all 41 references
  1. [9]

    Janin, D

    A. Janin, D. Baron, J. Edwards, D. Ellis, D. Gelbart, N. Morgan, B. Peskin, T. Pfau, E. Shriberg, A. Stolcke, and C. Wooters. 2003. https://doi.org/10.1109/ICASSP.2003.1198793 The ICSI Meeting Corpus . In 2003 IEEE International Conference on Acoustics , Speech , and Signal Pr...

    2003 arXiv

  2. [10]

    Ryo Kamoi, Sarkar Snigdha Sarathi Das, Renze Lou, Jihyun Janice Ahn, Yilun Zhao, Xiaoxin Lu, Nan Zhang, Yusen Zhang, Ranran Haoran Zhang, Sujeeth Reddy Vummanthala, Salika Dave, Shaobo Qin, Arman Cohan, Wenpeng Yin, and Rui Zhang. 2024. Evaluating LLMs at Detecting Errors in L...

    2024 arXiv

  3. [11]

    M. G. Kendall. 1938. https://arxiv.org/abs/2332226 A New Measure of Rank Correlation . Biometrika, 30(1/2):81--93

    1938

  4. [12]

    Frederic Kirstein, Terry Ruas, and Bela Gipp. 2024 a . https://doi.org/10.48550/arXiv.2407.11919 What's Wrong ? Refining Meeting Summaries with LLM Feedback . Preprint, arXiv:2407.11919

    2024 doi arXiv

  5. [13]

    Frederic Kirstein, Jan Philip Wahle, Bela Gipp, and Terry Ruas. 2024 b . https://doi.org/10.48550/arXiv.2406.07494 CADS : A Systematic Literature Review on the Challenges of Abstractive Dialogue Summarization . Preprint, arXiv:2406.07494

    2024 doi arXiv

  6. [14]

    Frederic Kirstein, Jan Philip Wahle, Terry Ruas, and Bela Gipp. 2024 c . https://doi.org/10.48550/arXiv.2404.11124 What's under the hood: Investigating Automatic Metrics on Meeting Summarization . Preprint, arXiv:2404.11124

    2024 doi arXiv

  7. [15]

    Klaus Krippendorff. 1970. https://doi.org/10.2307/270787 Bivariate Agreement Coefficients for Reliability of Data . Sociological Methodology, 2:139--150

    1970 doi

  8. [16]

    Md Tahmid Rahman Laskar, Xue-Yong Fu, Cheng Chen, and Shashi Bhushan TN. 2023. https://doi.org/10.18653/v1/2023.emnlp-industry.33 Building Real-World Meeting Summarization Systems using Large Language Models : A Practical Perspective . In Proceedings of the 2023 Conference on ...

    2023 doi

  9. [17]

    Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and Sushant Prakash. 2023. RLAIF vs. RLHF : Scaling Reinforcement Learning from Human Feedback with AI Feedback . https://arxiv.or...

    2023 arXiv

  10. [18]

    Yu Li, Shenyu Zhang, Rui Wu, Xiutian Huang, Yongrui Chen, Wenhao Xu, Guilin Qi, and Dehai Min. 2024. https://arxiv.org/abs/2403.19305 MATEval : A Multi-Agent Discussion Framework for Advancing Open-Ended Text Evaluation . Preprint, arXiv:2403.19305

    2024 arXiv

  11. [19]

    Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Rui Wang, Yujiu Yang, Zhaopeng Tu, and Shuming Shi. 2023. Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate . https://arxiv.org/abs/2305.19118v3

    2023 arXiv

  12. [20]

    R. Likert. 1932. A technique for the measurement of attitudes. Archives of Psychology, 22 140:55--55

    1932

  13. [21]

    Chin-Yew Lin. 2004. ROUGE : A Package for Automatic Evaluation of Summaries . In Text Summarization Branches Out , pages 74--81, Barcelona, Spain. Association for Computational Linguistics

    2004

  14. [22]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023 a . https://doi.org/10.18653/v1/2023.emnlp-main.153 G- Eval : NLG Evaluation using Gpt-4 with Better Human Alignment . In Proceedings of the 2023 Conference on Empirical Methods in Natural Langu...

    2023 doi

  15. [23]

    Yuxuan Liu, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, and Qi Zhang. 2023 b . https://arxiv.org/abs/2309.13308 Calibrating LLM-Based Evaluator . Preprint, arXiv:2309.13308

    2023 arXiv

  16. [24]

    Iain Mccowan, J Carletta, Wessel Kraaij, Simone Ashby, S Bourban, M Flynn, M Guillemot, Thomas Hain, J Kadlec, V Karaiskos, M Kronenthal, Guillaume Lathoud, Mike Lincoln, Agnes Lisowska Masson, Wilfried Post, Dennis Reidsma, and P Wellner. 2005. The AMI meeting corpus. Int'l. ...

    2005

  17. [25]

    Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. https://doi.org/10.48550/arXiv.2305.14251 FActScore : Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation . ...

    2023 doi arXiv

  18. [26]

    Jason Phang, Yao Zhao, and Peter J. Liu. 2022. https://arxiv.org/abs/2208.04347 Investigating Efficiently Extending Transformers for Long Input Summarization . Preprint, arXiv:2208.04347

    2022 arXiv

  19. [27]

    Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, and Patrick Gallinari. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.529 QuestEval : Summarization Asks for Fact-based Evaluation . In Proceedings of the 2021 Conference on ...

    2021 doi

  20. [28]

    Spearman

    C. Spearman. 1904. https://doi.org/10.2307/1412159 The Proof and Measurement of Association between Two Things . The American Journal of Psychology, 15(1):72--101

    1904 doi

  21. [29]

    Robert F. Tate. 1954. https://doi.org/10.1214/aoms/1177728730 Correlation Between a Discrete and a Continuous Variable . Point-Biserial Correlation . The Annals of Mathematical Statistics, 25(3):603--607

    1954

  22. [30]

    Gemini Team, Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry , Lepikhin , and Timothy Lillicrap. 2024. https://doi.org/10.48550/arXiv.2403.05530 Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context . Preprint, arXiv:2403.05530

    2024 doi arXiv

  23. [31]

    Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D. Manning. 2023. https://arxiv.org/abs/2305.14975 Just Ask for Calibration : Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tu...

    2023 arXiv

  24. [32]

    Tianlu Wang, Ilia Kulikov, Olga Golovneva, Ping Yu, Weizhe Yuan, Jane Dwivedi-Yu , Richard Yuanzhe Pang, Maryam Fazel-Zarandi , Jason Weston, and Xian Li. 2024. https://arxiv.org/abs/2408.02666 Self- Taught Evaluators . Preprint, arXiv:2408.02666

    2024 arXiv

  25. [33]

    Hui Wei, Shenghua He, Tian Xia, Andy Wong, Jingyang Lin, and Mei Han. 2024 a . https://arxiv.org/abs/2408.13006 Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks : Explainable Metrics and Diverse Prompt Templates . Preprint, arXiv:2408.13006

    2024 arXiv

  26. [34]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2024 b . Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference on Neural Information Proce...

    2024

  27. [35]

    Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. BARTScore : Evaluating Generated Text as Text Generation . https://arxiv.org/abs/2106.11520v2

    2021 arXiv

  28. [36]

    Weinberger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. https://doi.org/10.48550/arXiv.1904.09675 BERTScore : Evaluating Text Generation with BERT . Preprint, arXiv:1904.09675

    2020 doi arXiv

  29. [37]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2024. Judging LLM-as-a-judge with MT-bench and Chatbot Arena . In Proceedings of the 37th Interna...

    2024

  30. [38]

    Ming Zhong, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2022. https://doi.org/10.48550/arXiv.2109.02492 DialogLM : Pre-trained Model for Long Dialogue Understanding and Summarization . Preprint, arXiv:2109.02492

    2022 doi arXiv

  31. [39]

    Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. 2021. https://arxiv.org/abs/2104.05938 QMSum : A New Benchmark for Query-based Multi-domain Meeting Summarization . Preprint...

    2021 arXiv

  32. [40]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  33. [41]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.