Pith. sign in

REVIEW 4 major objections 5 minor 46 references

Unraveling the Capabilities of Language Models in News Summarization

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that adding demonstration examples in few-shot prompting did not improve—and often degraded—news summarization across 20 language models, because the gold reference summaries used as examples are too low in quality.

desk verdict Useful broad benchmark of 20 LMs for news summarization, but the headline few-shot finding is overinterpreted and rests on an internal inconsistency. read the letter →

arxiv 2501.18128 v1 pith:DMOSKDYU submitted 2025-01-30 cs.CL cs.AI

Abdurrahman Odabaş{i} , Göksel Biricik This is my paper · ORCID
classification cs.CLcs.AI
keywords newssummarizationlanguagemodelsfew-shotlearningzero-shotin-contextbenchmarkevaluationLLM-as-a-judgedatasetquality
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper benchmarks 20 recent language models, most of them small or mid-sized, on news summarization across three datasets (CNN/Daily Mail, Newsroom, XSum) and under zero-shot and three-shot prompting. Its central finding is that adding demonstration examples did not improve model performance and in some cases made summaries worse, with the paper attributing the degradation to the low quality of the gold summaries used as references. The study also reports that GPT-3.5-Turbo and GPT-4 generally dominate, while Qwen1.5-7B, SOLAR-10.7B-Instruct-v1.0, Meta-Llama-3-8B, and Zephyr-7B-Beta are the strongest public alternatives. A reader would care because the result challenges the assumption that more in-context examples always help and points to dataset quality as a hidden variable in summarization evaluation.

What carries the argument

The mechanism that carries the argument is a three-part evaluation framework—automatic metrics (ROUGE-L, METEOR, BERTScore), human evaluation on relevance, factual faithfulness, and coherence, and an LLM-as-a-judge using Claude 3 Sonnet—applied to a matched comparison of zero-shot versus three-shot prompts across 20 models on 1,000-test-article samples from each dataset. The few-shot demonstrations were three manually selected and edited examples taken from the validation set, chosen for genre variety and short length to fit context windows. The authors' pre-experiment dataset analysis, including overlap-ratio histograms, documents the low quality of gold summaries that they argue explains the few-shot degradation.

What would settle it

If re-running the same experiments with higher-quality gold summaries (for example, summaries written or edited by domain experts) as demonstrations produced consistent gains over zero-shot across the same models and datasets, the paper's attribution of the few-shot failure to gold-summary quality would be falsified. A more direct check would be to measure whether the drop in few-shot scores correlates dataset-by-dataset with the measured quality of that dataset's gold summaries.

Watch

Extended reading notes

Core claim

The paper claims that, across CNN/Daily Mail, Newsroom, and XSum, providing three manually selected demonstration summaries in the prompt did not improve—and often degraded—the quality of generated news summaries for all 20 models. After analyzing the datasets, the authors attribute this to the poor quality of the gold summaries, which contain invalid entries, empty or special-character-only summaries, placeholders indicating missing text, and a bias toward extractive highlights in CNN/DM and Newsroom. Despite the few-shot failure, GPT-3.5-Turbo and GPT-4 maintained strong performance in both settings, while among public models Qwen1.5-7B, SOLAR-10.7B-Instruct-v1.0, Meta-Llama-3-8B, and Zephyr-7B-Beta emerged as competitive alternatives across evaluation methods. The paper also identifies a three-way split among small models: some score high on automatic metrics but low with humans (Yi models), others receive high human and judge scores despite modest automatic scores (Gemma-7B, Llama-3 family, Zephyr-Beta), and a balanced group (Qwen1.5-7B, SOLAR-Instruct-v1.0) performs well on all measures.

Load-bearing premise

The conclusion that few-shot learning fails depends on the assumption that three manually selected and edited demonstrations are a fair and sufficient instantiation of few-shot in-context learning for all 20 models, across all three datasets.

Editorial extensions

If this is right

  • Practitioners should verify the quality of demonstration summaries before using them for few-shot summarization, since low-quality references can actively hurt output.
  • Automatic metrics and human judgment can diverge sharply for small models, so benchmark conclusions about summarization should combine both kinds of evidence.
  • Models such as Qwen1.5-7B and SOLAR-10.7B-Instruct-v1.0 offer a practical middle ground for news summarization when large proprietary models are not an option.
  • News summarization benchmarks with flawed reference summaries likely underestimate actual model quality and need re-curated gold summaries for reliable evaluation.
  • Large proprietary models proved more resilient than smaller ones to low-quality demonstrations, suggesting scale confers robustness to noisy in-context examples.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The negative few-shot result may be specific to this setup: because context windows forced only three demonstrations, the paper does not test whether more, or better-written, examples would help; a natural extension is to vary demonstration count and quality systematically.
  • If gold summaries in these widely used benchmarks are as defective as described, then published leaderboard numbers on these datasets are partly measuring reference quality rather than model capability, which has implications beyond news summarization.
  • The LLM-as-a-judge results were consistently more generous than human scores; using multiple judges or calibrating judge scores against human ratings could make this evaluation method more reliable.
  • The documented failure modes (empty outputs, prompt regurgitation, early termination) suggest that decoding constraints or structured output formats might rescue several small models; this is testable without changing model weights.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper presents a benchmarking study of 20 recent language models for news summarization, comparing zero-shot and three-shot in-context learning settings on CNN/Daily Mail, Newsroom, and XSum. The authors evaluate models using ROUGE-L, BERTScore, METEOR, human evaluation, and LLM-as-a-judge (Claude 3 Sonnet). The main claimed findings are that few-shot demonstrations do not improve performance and sometimes degrade it, attributed to the poor quality of gold summaries, and that GPT-3.5-Turbo and GPT-4 dominate while Qwen1.5-7B, SOLAR-10.7B-Instruct-v1.0, Meta-Llama-3-8B, and Zephyr-7B-Beta are competitive public alternatives.

Significance. The benchmark is broad in model coverage (20 models, from 0.5B parameters to private frontier models) and uses a multi-faceted evaluation protocol, including automatic metrics on large samples (827–938 examples per condition), human evaluation, and an independent judge LLM. The finding that few-shot demonstrations did not help in this setup is a useful empirical datapoint for practitioners, and the analysis of common failure modes (empty completions, repetition, prompt generation) is informative. However, the causal explanation for the few-shot result is not supported by the experimental design, and the human/LLM-judge components rest on very small samples without significance testing. The paper would be a valuable resource if the causal claim is either properly tested or substantially toned down.

major comments (4)
  1. [Section 3.3.2, Section 4.2.1 (and abstract)] The claim that few-shot performance degradation is 'mainly due to the poor quality of the gold summaries' is not supported and is internally inconsistent with the description of how the demonstrations were constructed. Section 3.3.2 states that the three demonstrations were 'manually selected' and that in some cases the authors 'removed extra, misleading information within the gold summary.' Thus the demonstrations used in the few-shot condition were not raw gold summaries but human-corrected exemplars. The abstract and Section 4.2.1 nevertheless attribute the decline to the low quality of gold summaries. No control condition varies demonstration quality (e.g., unedited vs edited gold, high-quality vs low-quality references), so the 'due to poor quality' explanation is not isolated. The authors should either add such a control or reframe the causal claim as a hypothesis.
  2. [Sections 3.6.2, 3.6.3, Tables 3–8] The human evaluation covers only 6 articles per evaluator (2 per dataset), with 5 evaluators, and the LLM-as-a-judge evaluation covers 5 articles per dataset/setting. No confidence intervals, inter-annotator agreement, or significance tests are reported. Given these small samples, statements such as 'human evaluators and the judge LLM both confirmed the high performance of Qwen1.5-7B' (Section 4.1.1) are not statistically grounded. I recommend reporting per-article score distributions or bootstrap confidence intervals, and treating human/LLM-judge results as secondary evidence.
  3. [Sections 3.5, 4.2.1, Tables 6–8] The handling of empty completions is not specified. For example, Section 4.2.1 reports that Gemma-7B and Mistral-v0.1 responded with empty answers to 580 and 555 of 827 articles, but the automatic evaluation tables report nonzero BERTScore values for these models. If empty outputs were scored as empty strings, the metric averages conflate task non-compliance with summary quality; if they were excluded, the models are evaluated on different subsets. The paper should state the exact procedure and justify it.
  4. [Section 3.3.2, Appendix A] The few-shot prompts and demonstration instances are not shown. Only a single zero-shot prompt for XSum is provided (Table A.9), and no information is given about how the three demonstrations were formatted, whether they differed across datasets, or whether the same examples were used for all 20 models. This is a reproducibility gap: the core few-shot result depends on a specific, manually constructed prompt that is not documented. Full prompt templates and demonstration examples should be included as supplementary material.
minor comments (5)
  1. [Table 2] Table 2 lists GPT-4-0125-preview with an 8K context window, but this model has a 128K context; Gemini-1.5-Pro is listed with 128K, but depending on version it has 1M–2M. Please correct or justify.
  2. [Section 3.6.1] The text contains a duplicated phrase: 'lexical overlap lexical overlap'.
  3. [Section 4.2.3] The sentence 'Yi-9B billion even received the highest ROUGE score' contains a stray word; it should read 'Yi-9B even received the highest ROUGE score.'
  4. [Overall] No statement on code or data release is provided. For reproducibility, the authors should make the prompts, demonstration examples, and sampling scripts available.
  5. [Section 3.4] The paper states that generation settings were kept at defaults, but default sampling parameters differ across APIs and libraries; this is a potential confound and should be acknowledged more explicitly.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the paper is an empirical benchmark evaluated against external references and an independent judge model, with no fitted parameters or self-citation chain.

full rationale

This paper reports an empirical evaluation of 20 language models on news summarization. There is no derivation chain that reduces to its own inputs: no equations are fitted to the results, no parameter is estimated from a subset and then renamed a prediction, and no uniqueness theorem or other load-bearing result is imported from the authors' prior work. The central claims are comparative statements about measured ROUGE, METEOR, BERTScore, human, and LLM-as-a-judge scores, all anchored to external benchmark datasets (CNN/DM, Newsroom, XSum) and an external judge model (Claude 3 Sonnet). The few-shot demonstrations were manually selected and edited, which is a legitimate reproducibility and validity concern about the few-shot conclusion, but it is not circular reasoning: the observed zero-shot versus few-shot differences are empirical outcomes, not consequences of the definition of the demonstrations. Similarly, the paper's explanation that few-shot degradation is 'mainly due to the poor quality of the gold summaries' is an under-supported causal interpretation, and one that is in some tension with the authors' own statement that they removed misleading information from demonstration summaries; however, this is a question of experimental control and evidential support, not of circularity. No self-citations are load-bearing, and no known result is merely renamed. Accordingly, the circularity score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper introduces no fitted scientific parameters, but its central claims rest on several hand-chosen evaluation design decisions: the number and manual editing of few-shot demonstrations, the evaluation sample size, the tiny human evaluation workload, and the choice of Claude 3 Sonnet as judge. These are not derived from any theory and could change the reported rankings if altered.

free parameters (4)
  • number_of_few_shot_demonstrations = 3
    Hand-chosen in Section 3.3.2 to fit context windows; all few-shot conclusions rest on this choice.
  • evaluation_sample_size = 1000 per dataset (827/923/938 after filtering)
    Chosen in Section 3.3.3; determines the claimed statistical robustness.
  • human_evaluation_articles = 6 articles per evaluator, 5 evaluators
    Workload chosen in Section 3.6.2; all human scores rest on this very small sample.
  • llm_judge_model = Claude 3 Sonnet
    Chosen in Section 3.6.3; all LLM-as-a-judge scores depend on this model.
assumptions (4)
  • domain assumption Gold summaries in CNN/DM, Newsroom, and XSum are of poor quality and unsuitable as references, motivating no fine-tuning and explaining few-shot failures.
    Central to the few-shot conclusion (Sections 3.2 and 4.2). The paper gives qualitative and statistical evidence (overlap ratios, invalid entries) but does not quantify quality or test alternative demonstrations.
  • domain assumption Default generation settings provide a fair comparison across models.
    Section 3.4 states defaults were kept, acknowledging performance may be suboptimal; rankings depend on this assumption.
  • domain assumption Restricting evaluation to the non-blocked shared subset (827/923/938 examples) does not bias comparisons.
    Section 3.5 excludes API-filtered examples for all models because public models might also be censored; no analysis of the filtered content is provided.
  • domain assumption Claude 3 Sonnet's LLM-as-a-judge scores are trustworthy proxies for summary quality without calibration against human judgments.
    Section 3.6.3 uses the judge as an evaluation channel. The paper reports disagreements with humans and notes the judge is more generous, but still treats judge scores as evidence.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Unraveling the Capabilities of Language Models in News Summarization." pith.science (2026). https://pith.science/paper/DMOSKDYU

@misc{pith2026250118128,
  author       = {Pith},
  title        = {Pith review of: Unraveling the Capabilities of Language Models in News Summarization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DMOSKDYU}},
  note         = {Machine review of arXiv:2501.18128}
}
read the original abstract

Given the recent introduction of multiple language models and the ongoing demand for improved Natural Language Processing tasks, particularly summarization, this work provides a comprehensive benchmarking of 20 recent language models, focusing on smaller ones for the news summarization task. In this work, we systematically test the capabilities and effectiveness of these models in summarizing news article texts which are written in different styles and presented in three distinct datasets. Specifically, we focus in this study on zero-shot and few-shot learning settings and we apply a robust evaluation methodology that combines different evaluation concepts including automatic metrics, human evaluation, and LLM-as-a-judge. Interestingly, including demonstration examples in the few-shot learning setting did not enhance models' performance and, in some cases, even led to worse quality of the generated summaries. This issue arises mainly due to the poor quality of the gold summaries that have been used as reference summaries, which negatively impacts the models' performance. Furthermore, our study's results highlight the exceptional performance of GPT-3.5-Turbo and GPT-4, which generally dominate due to their advanced capabilities. However, among the public models evaluated, certain models such as Qwen1.5-7B, SOLAR-10.7B-Instruct-v1.0, Meta-Llama-3-8B and Zephyr-7B-Beta demonstrated promising results. These models showed significant potential, positioning them as competitive alternatives to large models for the task of news summarization.

Figures

Figures reproduced from arXiv: 2501.18128 by the authors.

Figure 1
Figure 1. Overlap ratio distributions for the training sets of the three datasets [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 19 canonical work pages

  1. [1]

    W. S. El-Kassas, C. Salama, A. Rafea, and H. K. Mohamed, ”Automatic text summarization: A comprehensive survey,” Expert Systems with Ap- plications, vol. 165, Jul. 2020. https://doi.org/10.1016/j.eswa. 2020.113679

    arXiv 2020

  2. [2]

    M. F. Mridha, A. A. Lima, K. Nur, et al., ”A Survey of Automatic Text Summarization: Progress, Process and Challenges,” IEEE Access, vol. 9, IEEE, Nov. 2021. https://doi.org/10.1109/ACCESS.2021. 3129786

    doi:10.1109/access.2021 2021

  3. [3]

    Zhang, G

    M. Zhang, G. Zhou, W. Yu, N. Huang, and W. Liu, ”A Comprehen- sive Survey of Abstractive Text Summarization Based on Deep Learn- ing,” Computational Intelligence and Neuroscience, WILEY , Aug. 2022. https://doi.org/10.1155/2022/7132226

    work page doi:10.1155/2022/7132226 2022

  4. [4]

    Z. J. Hew, V . J. Olanrewaju, X. Chew, and K. W. Khaw, ”Text Sum- marization for News Articles by Machine Learning Techniques,” Ap- plied Mathematics and Computational Intelligence - Universiti Malaysia Perlis, vol. 11, Dec. 2022. http://dspace.unimap.edu.my/xmlui/ handle/123456789/77725

    arXiv 2022

  5. [5]

    Automatic Text Summarization Methods: A Comprehensive Review

    D. Yadav, J. Desai, and A. K. Yadav, ”Automatic Text Summarization Methods: A Comprehensive Review,” ArXiv, Apr. 2022. https://doi. org/10.48550/arXiv.2204.01849

    work page Pith review arXiv doi:10.48550/arxiv.2204.01849 2022

  6. [6]

    Y . Cao, S. Li, Y . Liu, et al., ”A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT,” ArXiv, Mar. 2023. https://doi.org/10.48550/arXiv.2303.04226

    arXiv doi:10.48550/arxiv.2303.04226 2023

  7. [7]

    W. X. Zhao, K. Zhou, J. Li, et al., ”A Survey of Large Language Models,” ArXiv, Mar. 2023. https://doi.org/10.48550/arXiv.2303.18223

    arXiv doi:10.48550/arxiv.2303.18223 2023

  8. [8]

    M. U. Hadi, Q. Al Tashi, R. Qureshi, et al., ”Large Language Models: A Comprehensive Survey of its Applications, Challenges, Limitations, and Future Prospects,”ArXiv, Jul. 2023.https://doi.org/10.36227/ techrxiv.23589741.v1

    work page 2023

Show all 46 references
  1. [9]

    K. S. Kalyan, ”A survey of GPT-3 family large language models including ChatGPT and GPT-4,” Natural Language Processing Journal , Elsevier, vol. 6, 2024. https://doi.org/10.1016/j.nlp.2023.100048

    2024

  2. [10]

    Goyal, J

    T. Goyal, J. J. Li, and G. Durrett, ”News Summarization and Evaluation in the Era of GPT-3,”ArXiv, Sep. 2022. https://doi.org/10.48550/ arXiv.2209.12356

    2022 doi arXiv

  3. [11]

    Zhang, F

    T. Zhang, F. Ladhak, E. Durmus, et al., ”Benchmarking Large Lan- guage Models for News Summarization,” ArXiv, Jan. 2023. https: //doi.org/10.48550/arXiv.2301.13848

    2023 doi arXiv

  4. [12]

    Basyal and M

    L. Basyal and M. Sanghvi, ”Text Summarization Using Large Language Models: A Comparative Study of MPT-7b-instruct, Falcon-7b-instruct, and OpenAI Chat-GPT Models,” ArXiv, Oct. 2023. https://doi.org/ 10.48550/arXiv.2310.10449

    2023 doi arXiv

  5. [13]

    H. P. Luhn, ”The Automatic Creation of Literature Abstracts,” IBM Jour- nal of Research and Development , vol. 2, Apr. 1958. https://doi. org/10.1147/rd.22.0159

    1958 doi

  6. [14]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, et al., ”Attention Is All You Need,” in Proceedings of the 31st Conference on Neural Information Process- ing Systems (NIPS), Jun. 2017. https://doi.org/10.48550/arXiv. 1706.03762

    2017 doi arXiv

  7. [15]

    Gupta, D

    A. Gupta, D. Chugh, Anjum, and R. Katarya, ”Automated News Sum- marization Using Transformers,”ArXiv, Aug. 2021. https://doi.org/ 10.48550/arXiv.2108.01064

    2021 doi arXiv

  8. [16]

    Karkera and N

    T. Karkera and N. Pathak, ”Comparative Study on News Summariza- tion using various Transformer Based Models,” International Research Journal of Engineering and Technology (IRJET) , vol. 09, May 2022. https://www.irjet.net/archives/V9/i5/IRJET-V9I5294.pdf

    2022

  9. [17]

    Minaee, T

    S. Minaee, T. Mikolov, N. Nikzad, et al., ”Large Language Models: A Survey,” ArXiv, Feb. 2024. https://doi.org/10.48550/arXiv. 2402.06196

    2024 doi arXiv

  10. [18]

    Gu and T

    A. Gu and T. Dao, ”Mamba: Linear-Time Sequence Modeling with Se- lective State Spaces,”ArXiv, Dec. 2023. https://doi.org/10.48550/ arXiv.2312.00752

    2023 doi arXiv

  11. [19]

    Harman and P

    D. Harman and P. Over, ”The E ffects of Human Variation in DUC Sum- marization Evaluation,” in Text Summarization Branches Out, Jul. 2004. https://doi.org/https://aclanthology.org/W04-1003

    2004

  12. [20]

    Fabbri, I

    A. Fabbri, I. Li, T. She, S. Li, and D. Radev, ”Multi-News: A Large- Scale Multi-Document Summarization Dataset and Abstractive Hierar- chical Model,” in Proceedings of the 57th Annual Meeting of the Associ- ation for Computational Linguistics, Jul. 2019. https://doi.org/10. 1...

    2019

  13. [21]

    K. M. Hermann, T. Ko ˇcisk´y, E. Grefenstette, et al., ”Teaching Machines to Read and Comprehend,” inProceedings of the 28th International Con- ference on Neural Information Processing Systems (NIPS) , Nov. 2015. https://doi.org/10.48550/arXiv.1506.03340

    2015 doi arXiv

  14. [22]

    Nallapati, B

    R. Nallapati, B. Zhou, C. dos Santos, C ¸ . G ¨ulc ¸ehre, and B. Xiang, ”Ab- stractive Text Summarization using Sequence-to-sequence RNNs and Be- yond,” in Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, Association for Computational Lingu...

    2016 doi

  15. [23]

    Grusky, M

    M. Grusky, M. Naaman, and Y . Artzi, ”NEWSROOM: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategies,” in Proceedings of the 2018 Conference of the North American Chapter of the Associa- tion for Computational Linguistics: Human Language Technologies, Jun

    2018

  16. [24]

    Narayan, S

    S. Narayan, S. B. Cohen, and M. Lapata, ”Don’t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization,” in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Association for Computational ...

    2018 doi

  17. [25]

    30, 2024

    Hugging Face Team, ”Hugging Face Dataset cnn dailymail (CNN/DM),” Online, https://huggingface.co/datasets/cnn_dailymail, ac- cessed: Apr. 30, 2024

    2024

  18. [26]

    30, 2024

    Hugging Face Team, ”Hugging Face Dataset EdinburghNLP /xsum,” On- line, https://huggingface.co/datasets/EdinburghNLP/xsum, accessed: Apr. 30, 2024

    2024

  19. [27]

    D. Chen, J. Bolton, and C. D. Manning, ”A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task,” in Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , Aug. 2016. https://doi.org/10.18653/ V1/P16-1223

    2016

  20. [28]

    Google Gemini Team, ”Gemini: A Family of Highly Capable Multimodal Models,” https://storage.googleapis.com/deepmind-media/ gemini/gemini_1_report.pdf

  21. [29]

    Mesnard, C

    T. Mesnard, C. Hardin, R. Dadashi, et al., ”Gemma: Open Models Based on Gemini Research and Technology,”ArXiv, Mar. 2024. https://doi. org/10.48550/arXiv.2403.08295

    2024 doi arXiv

  22. [30]

    OpenAI, ”Introducing ChatGPT,” Online, https://openai.com/ index/chatgpt/, accessed: May 26, 2024

    2024

  23. [31]

    OpenAI, ”GPT-4 Technical Report,” ArXiv, Mar. 2023. https://doi. org/10.48550/arXiv.2303.08774

    2023 doi arXiv

  24. [32]

    Touvron, L

    H. Touvron, L. Martin, K. Stone, et al., ”Llama 2: Open Foundation and Fine-Tuned Chat Models,” ArXiv, Jul. 2023. https://doi.org/ 10.48550/arXiv.2307.09288

    2023 doi arXiv

  25. [33]

    Meta, ”Introducing Meta Llama 3: The most capable openly available LLM to date,” Online, https://ai.meta.com/blog/ meta-llama-3/ , accessed: May 26, 2024

    2024

  26. [34]

    A. Q. Jiang, A. Sablayrolles, A. Mensch, et al., ”Mistral 7B,” ArXiv, Oct

  27. [35]

    Abdin, J

    M. Abdin, J. Aneja, H. Awadalla, et al., ”Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone,” ArXiv, Apr

  28. [36]

    J. Bai, S. Bai, Y . Chu, et al., ”Qwen Technical Report,”ArXiv, Sep. 2023. https://doi.org/10.48550/arXiv.2309.16609

    2023 doi arXiv

  29. [37]

    D. Kim, C. Park, S. Kim, et al., ”SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling,” ArXiv, Dec. 2023. https://doi.org/10.48550/arXiv.2312.15166

    2023 doi arXiv

  30. [38]

    Young, B

    01.AI: A. Young, B. Chen, C. Li, et al., ”Yi: Open Foundation Models by 01.AI,” ArXiv, Mar. 2024. https://doi.org/10.48550/arXiv. 2403.04652

    2024 doi arXiv

  31. [39]

    Tunstall, E

    L. Tunstall, E. Beeching, N. Lambert, et al., ”Zephyr: Direct Distillation of LM Alignment,” ArXiv, Oct. 2023. https://doi.org/10.48550/ arXiv.2310.16944

    2023 doi arXiv

  32. [40]

    Lin, ”ROUGE: A Package for Automatic Evaluation of Sum- maries,” inProceedings of the Association for Computational Linguistics, Jul

    C.-Y . Lin, ”ROUGE: A Package for Automatic Evaluation of Sum- maries,” inProceedings of the Association for Computational Linguistics, Jul. 2004

    2004

  33. [41]

    Banerjee and A

    S. Banerjee and A. Lavie, ”METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments,” in Pro- ceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and /or Summarization , Jun. 2005. https://acla...

    2005

  34. [42]

    Zhang, V

    T. Zhang, V . Kishore, F. Wu, K. Q. Weinberger, and Y . Artzi, ”BERTScore: Evaluating Text Generation with BERT,” in International Conference on Learning Representations , Apr. 2019. https://doi. org/10.48550/arXiv.1904.09675

    2019 doi arXiv

  35. [43]

    Anthropic, ”Claude 3: Introducing the next generation of Claude,” On- line, https://www.anthropic.com/news/claude-3-family , ac- cessed: May 29, 2024. 13

    2024

  36. [2018]

    https://doi.org/10.18653/v1/N18-1065, Dataset: https: //lil.nlp.cornell.edu/newsroom/

    doi

  37. [2023]

    https://doi.org/10.48550/arXiv.2310.06825

    doi arXiv

  38. [2024]

    https://doi.org/10.48550/arXiv.2404.14219

    doi arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.