Pith. sign in

REVIEW 3 major objections 3 minor 68 references

An Empirical Study of Many-to-Many Summarization with Large Language Models

T0 review · 3 major / 3 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Instruction-tuned open-source LLMs can outperform zero-shot GPT-4 on automatic many-to-many summarization metrics while preserving general ability, but instruction tuning can intensify factual errors in generated summaries.

desk verdict A useful new benchmark and the first broad LLM comparison for many-to-many summarization, but the train/test split may let the same source document appear in both sides across language pairs, so the headline instruction-tuning result is not fully trustworthy until that overlap is ruled out. read the letter →

arxiv 2505.12983 v1 pith:VI72R55Z submitted 2025-05-19 cs.CL cs.AI

Jiaan Wang , Fandong Meng , Zengkui Sun , Yunlong Liang , Yuxuan Cao , Jiarong Xu , Haoxiang Shi , Jie Zhou This is my paper · ORCID
classification cs.CLcs.AI
keywords many-to-manysummarizationcross-lingualinstructiontuninglargelanguagemodelsmultilingualbenchmarkdatacontaminationfactualconsistencyMMLUevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish how well large language models perform many-to-many summarization—summarizing a document in any source language into any target language—when used off the shelf and when instruction-tuned on task data. To do this, the authors reorganize 47.8K samples drawn from eight existing multilingual summarization datasets, covering five domains and six languages, with intrinsic-quality filtering and contamination screening on the test set. They benchmark 18 LLMs in zero-shot and instruction-tuned settings against fine-tuned traditional multilingual models (mBART-50 and PISCES). The central empirical claim is that instruction-tuned open-source LLMs substantially improve on their zero-shot versions and outperform zero-shot GPT-4 and GPT-4o on automatic ROUGE and BERTScore metrics, while preserving MMLU performance. The paper also argues, on the basis of human evaluation, that LLMs still produce factual errors and that instruction tuning can intensify them, making factual control the key obstacle for real applications.

What carries the argument

The central object is the reorganized M2MS benchmark: 47.8K document–summary pairs in five domains (news, encyclopedia, dialogue, how-to guides, technology) across six languages (English, Czech, German, French, Chinese, Ukrainian), built from eight existing datasets, with samples filtered by coverage, redundancy, and coherence and the test set screened so that fewer than 1% of instances are flagged as contaminated for the evaluated LLMs. The mechanism that carries the argument is instruction tuning: the same task prompt used for zero-shot evaluation (system instruction, domain tag, three in-context example summaries) is used to format the 19.5K training samples, and the open-source LLMs are tuned for two epochs. The comparison against fine-tuned traditional models (mBART-50 and PISCES, with a language tag on the decoder) and against zero-shot prompting isolates the effect of parameter updating on M2MS ability. The prompt itself, with in-context examples, is the instrument that elicits M2MS behavior from zero-shot LLMs.

What would settle it

Take the same instruction-tuned models and evaluate them on a newly collected M2MS test set of the same five domains and six languages that was not assembled from the eight source datasets and was screened with a stronger contamination method; if the ROUGE and BERTScore advantages over zero-shot GPT-4 shrink or vanish, the claimed improvement is an artifact of test-set construction rather than genuine M2MS ability. A simpler check: count the actual contaminated fraction of the released test set for each of the six LLMs measured in Appendix B; if it exceeds the reported 1%, the GPT-4 comparison is suspect.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that task-specific instruction tuning on a relatively small, multi-domain M2MS training set (19.5K samples) turns open-source LLMs into many-to-many summarizers that outperform zero-shot closed-source LLMs, including GPT-4, on ROUGE-1, ROUGE-L, and BERTScore, and do so without lowering MMLU scores. For instance, instruction-tuned Vicuna-13B-16k reaches 38.0 ROUGE-1 and 74.1 BERTScore overall, compared with 26.0 and 66.7 for zero-shot GPT-4o; tuned LLaMa, Qwen, Baichuan, and InternLM models all clear the zero-shot GPT-4 marks. The same evaluation shows a caution: a fine-grained human annotation of 100 English-to-Chinese summaries finds that GPT-4 has the fewest factual errors, while tuned LLaMa and Vicuna models hallucinate more than their zero-shot versions, so the automatic-metric gains do not translate into factuality gains.

Load-bearing premise

The load-bearing premise is that the reorganized test set measures true M2MS ability: the intrinsic-metric filtering and the below-1% contamination screen are assumed to prevent inflated scores, and the train and test splits drawn from the same eight source datasets are assumed not to overlap in a way that favors instruction-tuned models. If contamination or document overlap is higher than measured, the zero-shot versus instruction-tuned comparisons, and particularly the GPT-4 comparisons, could be biased.

Editorial extensions

If this is right

  • Instruction-tuned open-source LLMs can serve as practical many-to-many summarizers across six languages and five domains, outperforming fine-tuned traditional multilingual models by wide margins (e.g., about 7 ROUGE-1 points over PISCES).
  • Task-specific M2MS instruction tuning does not degrade general problem-solving ability as measured by MMLU, so specialized tuning is a viable route to deployable summarizers.
  • Automatic metrics (ROUGE, BERTScore) can improve substantially while factual error rates worsen, so evaluations of LLM summarizers should include factuality checks, not just overlap metrics.
  • Zero-shot LLMs already rival fine-tuned traditional models, so the marginal value of traditional encoder-decoder specialization is reduced when an LLM backbone is available.
  • Training-scale experiments show LLMs retain more performance than traditional models when training data is reduced (about 12.4 versus roughly 22 ROUGE-1 points lost from 19.5K to 2K samples), implying LLMs are more data-efficient for this task.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural testable extension would be to apply the same instruction-tuning recipe to LLMs that have explicit multilingual alignment or factual-consistency training, since the paper's own results suggest hallucination, not overlap quality, is the bottleneck.
  • The contamination-screening protocol could be adopted as a standard guardrail for multilingual LLM benchmarks, because the paper shows test-set leakage can be controlled but not fully eliminated for low-resource language pairs.
  • The factuality regression under instruction tuning suggests that cleaning or filtering hallucinated references in training data may be a more direct lever than better prompts, since the paper attributes intensified errors partly to information gaps in human-written references.
  • The benchmark's language-pair coverage leaves out Czech–Ukrainian and German–Ukrainian pairs, so a targeted experiment on those missing pairs would delimit whether the gains extend to all language pairs.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper presents a systematic empirical study of many-to-many summarization (M2MS) with large language models. The authors reorganize samples from eight existing multilingual summarization datasets into a 47.8K-sample M2MS benchmark spanning six languages and five domains, with train/validation/test splits. They benchmark 18 LLMs in zero-shot and instruction-tuned settings, alongside fine-tuned traditional models mBART-50 and PISCES, using ROUGE, BERTScore, GPT-4o-based quality scores, and a fine-grained human factuality annotation. The headline findings are that instruction-tuned open-source LLMs substantially improve their M2MS performance and can outperform zero-shot GPT-4 on automatic metrics, that this task-specific improvement does not degrade MMLU performance, and that instruction tuning may intensify factual errors in generated summaries.

Significance. If the headline claims hold, the paper is a valuable empirical contribution: it is, to my knowledge, the first systematic study of LLMs on the M2MS task, it assembles a reusable multi-domain, six-language benchmark, and it compares a wide array of open- and closed-source LLMs under controlled prompting and tuning conditions. The paper also merits credit for attempting contamination control for the test set, for including fine-tuned traditional baselines, for checking general ability retention via MMLU, and for conducting human factuality evaluation with reported inter-annotator agreement. The finding that instruction tuning on roughly 19.5K samples yields large automatic-metric gains while preserving MMLU is practically important, and the caution that tuning may intensify hallucination is a useful counterpoint. However, the significance is contingent on two things: whether the test set is actually disjoint from the instruction-tuning training data at the document level, and whether the headline 'significantly outperform' claims are supported by appropriate statistical evidence.

major comments (3)
  1. [§3 Data; Appendix B; Table 8] The paper does not establish that the test split is disjoint from the instruction-tuning training split at the document level, which is a load-bearing assumption for the central automatic-metric claim in Table 3. Section 3 states that samples are randomly selected within each dataset and each source-target language pair, and Table 8 shows that the same source datasets (e.g., XWikis, WikiLingua, CrossSum) supply many different language pairs. These multilingual datasets are constructed by aligning multiple summaries to the same source document, so a single source document can appear in the training split for one target language and in the test split for another target language. Because the M2MS input to the model is the source document alone, an instruction-tuned model can memorize that document and obtain inflated ROUGE/BERTScore, while zero-shot GPT-4 has no such exposure. Appendix B's contamination check addresses LLM pretraining contamination using BLEURT/ROUGE-L against the LLMs' pretraining corpora; it does not check duplication between the reorganized training and test splits. The Limitations section also does not mention this issue. The authors should quantify the extent of any document-level overlap between training and test sets (e.g., by exact or near-duplicate source-document matching across all language pairs) and either re-run the headline comparisons on a document-disjoint split or provide evidence that no overlap exists.
  2. [§5 Results and Analyses; Tables 3, 4, 11] The paper repeatedly uses 'significantly' and 'by a large margin' to describe differences between instruction-tuned LLMs, zero-shot LLMs, and traditional models, but it reports only single point estimates without significance tests, confidence intervals, or error bars. For example, the claim that 'instruction-tuned LLMs significantly outperform the best traditional language model' and the claim that some tuned LLMs 'outperform zero-shot GPT-4' rely on differences in ROUGE/BERTScore means that could fall within sampling variation across the 14,150 test samples. The authors should report paired bootstrap confidence intervals or permutation-test p-values for the headline comparisons (at minimum for the overall scores in Table 3 and for the GPT-4o evaluation in Table 4), and should adjust the language of 'significant' to match the statistical evidence actually provided.
  3. [§6 Discussion; Table 7; Appendix H] The human factuality evaluation is too narrow to support the broad conclusion that 'instruction tuning might intensify the issue' of factual errors. The evaluation in Appendix H uses only 100 English-source documents with Chinese summaries, for five systems, and the counts in Table 7 are small (e.g., hallucination counts of 8, 12, 17 for GPT-4, zero-shot Vicuna, tuned Vicuna). No significance testing or confidence intervals are reported for these differences, and the evaluation does not cover the full language and domain diversity of the benchmark. I recommend treating the factuality finding as suggestive rather than conclusive, and either expanding the evaluation or adding explicit statistical caution in the abstract and conclusion.
minor comments (3)
  1. [§1 and throughout] There are several typos and inconsistencies: 'Appenidx D' should be 'Appendix D'; 'Unversity' in the author affiliation should be 'University'; 'MSMS' in Section 5 should be 'M2MS'; and the terms 'instruct-tuning', 'instruction-tuning', and 'instruction tuning' are used inconsistently.
  2. [References] Liang et al. 2022a and Liang et al. 2022b appear to be the same paper (same title, same venue, same pages) listed twice; the duplicate reference should be removed or corrected.
  3. [§4.2 and Appendix E.1] The main text reports GPT-4o-based evaluation scores in Table 4 without stating that only 500 randomly selected test samples were used for this evaluation; the 500-sample subset is mentioned only in Appendix E.1. The main text should state this clearly so readers can calibrate the precision of those scores.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: central claims are direct benchmark comparisons, not derivation-from-inputs.

full rationale

The paper's central claims are empirical scorecard results: instruction-tuned open-source LLMs improve on M2MS automatic metrics and can outperform zero-shot GPT-4 (Table 3), while instruction tuning may increase factual errors (Table 7). These are direct measurements on a constructed benchmark, not predictions derived from fitted parameters, self-citation chains, or definitions. The M2MS data are reorganized from eight existing datasets, but the paper does not define any evaluated quantity in terms of a fitted input; the ROUGE, BERTScore, MMLU, and human-evaluation numbers are independently computed. Self-citations (e.g., PISCES from Wang et al. 2023c, the cross-lingual summarization survey, and the prompt-design reference) appear as background and baselines, and the headline comparison against zero-shot GPT-4 does not depend on any of those cited results being true. The Appendix B contamination check targets LLM pretraining contamination and is a data-hygiene measure; the separate concern that a source document could appear in both training and test splits across different language pairs is a validity caveat about benchmark construction, but the paper contains no equation-level equivalence or construction-level reduction that would make the reported comparisons true by definition. No circular step can be exhibited from the paper's own text, so the finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The paper is an empirical benchmark rather than a derivation, so the ledger captures data-construction choices and measurement assumptions. The thresholds are hand-selected from prior work and affect dataset composition but not the qualitative findings. The main risks are assumptions about metric validity and generalization from a small human evaluation.

free parameters (4)
  • Coverage filter threshold (alpha_cov) = 0.9
    Appendix A removes samples with coverage above 0.9; chosen by hand following Bommasani and Cardie (2020), and affects which abstractive samples remain in the benchmark.
  • Redundancy filter threshold (alpha_red) = 0.2
    Appendix A removes samples with redundancy above 0.2; selected threshold influences which dialogue and encyclopedia samples remain.
  • Coherence filter threshold (alpha_coh) = 0.9
    Appendix A removes samples with mBERT coherence below 0.9; threshold affects all domains and is a hand-chosen quality filter.
  • Allowed contamination proportion in test set = less than 1%
    Appendix B allows a small number of contaminated samples into the test set when a language direction has few samples; the cap is a policy choice rather than a measured value.
assumptions (5)
  • domain assumption The eight selected datasets provide aligned document-summary pairs across the six languages needed for many-to-many summarization.
    Section 3 lists CrossSum, XWikis, XSAMSum, XMediaSum, DialogSumX, WikiLingua, Perseus, and Spektrum as supporting M2MS alignments; if alignments are incomplete or noisy, the benchmark inherits the error.
  • domain assumption Sample-level contamination detection with BLEURT and ROUGE-L (Golchin and Surdeanu 2024) correctly identifies pretraining contamination for the evaluated LLMs.
    Appendix B uses this method on six representative LLMs and allows up to 1% contaminated samples in the test set; if contamination is underestimated, zero-shot scores could be inflated.
  • domain assumption ROUGE, BERTScore, and GPT-4o scoring are valid enough proxies for summary quality in all six languages.
    Section 4.2 relies on these metrics for the headline comparisons; ROUGE is lexical, BERTScore depends on mBERT, and GPT-4o evaluation is run on only 500 samples.
  • domain assumption MMLU with 5-shot prompting measures general task-solving ability sufficiently to detect catastrophic forgetting.
    Section 6 compares MMLU before and after instruction tuning and concludes general ability is preserved; MMLU is English-centric and not a complete general-ability test.
  • domain assumption Factuality annotations on 100 English-to-Chinese summaries by three graduate students generalize to the full M2MS testing set.
    Appendix H draws factuality conclusions from this narrow sample; Fleiss kappa supports inter-annotator agreement but not cross-domain or cross-language generalizability.

how reviews work

0 comments
Cite this review

Pith. "Pith review of An Empirical Study of Many-to-Many Summarization with Large Language Models." pith.science (2026). https://pith.science/paper/VI72R55Z

@misc{pith2026250512983,
  author       = {Pith},
  title        = {Pith review of: An Empirical Study of Many-to-Many Summarization with Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VI72R55Z}},
  note         = {Machine review of arXiv:2505.12983}
}
read the original abstract

Many-to-many summarization (M2MS) aims to process documents in any language and generate the corresponding summaries also in any language. Recently, large language models (LLMs) have shown strong multi-lingual abilities, giving them the potential to perform M2MS in real applications. This work presents a systematic empirical study on LLMs' M2MS ability. Specifically, we first reorganize M2MS data based on eight previous domain-specific datasets. The reorganized data contains 47.8K samples spanning five domains and six languages, which could be used to train and evaluate LLMs. Then, we benchmark 18 LLMs in a zero-shot manner and an instruction-tuning manner. Fine-tuned traditional models (e.g., mBART) are also conducted for comparisons. Our experiments reveal that, zero-shot LLMs achieve competitive results with fine-tuned traditional models. After instruct-tuning, open-source LLMs can significantly improve their M2MS ability, and outperform zero-shot LLMs (including GPT-4) in terms of automatic evaluations. In addition, we demonstrate that this task-specific improvement does not sacrifice the LLMs' general task-solving abilities. However, as revealed by our human evaluation, LLMs still face the factuality issue, and the instruction tuning might intensify the issue. Thus, how to control factual errors becomes the key when building LLM summarizers in real applications, and is worth noting in future research.

Figures

Figures reproduced from arXiv: 2505.12983 by the authors.

Figure 1
Figure 1. Illustration of many-to-many summarization. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Language-wise performance of tuned LLMs. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Length distributions of M2MS samples w.r.t [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Illustration of the used M2MS prompt that [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: Model performance (ROUGE-1) using differ [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

68 extracted references · 18 canonical work pages

  1. [1]

    Roee Aharoni, Shashi Narayan, Joshua Maynez, Jonathan Herzig, Elizabeth Clark, and Mirella Lapata. 2023. https://doi.org/10.18653/v1/2023.findings-acl.220 Multilingual summarization with factual consistency evaluation . In Findings of the Association for Computational Linguistics: ACL 2023, pages 3562--3591, Toronto, Canada. Association for Computational ...

    doi:10.18653/v1/2023.findings-acl.220 2023

  2. [2]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609

    arXiv 2023

  3. [3]

    Yu Bai, Heyan Huang, Kai Fan, Yang Gao, Yiming Zhu, Jiaao Zhan, Zewen Chi, and Boxing Chen. 2022. Unifying cross-lingual summarization and machine translation with compression rate. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1087--1097

    work page 2022

  4. [4]

    Abhik Bhattacharjee, Tahmid Hasan, Wasi Uddin Ahmad, Yuan-Fang Li, Yong-Bin Kang, and Rifat Shahriyar. 2023. https://doi.org/10.18653/v1/2023.acl-long.143 C ross S um: Beyond E nglish-centric cross-lingual summarization for 1,500+ language pairs . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa...

    doi:10.18653/v1/2023.acl-long.143 2023

  5. [5]

    Rishi Bommasani and Claire Cardie. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.649 Intrinsic evaluation of summarization datasets . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8075--8096, Online. Association for Computational Linguistics

    work page doi:10.18653/v1/2020.emnlp-main.649 2020

  6. [6]

    Yue Cao, Hui Liu, and Xiaojun Wan. 2020 a . https://doi.org/10.18653/v1/2020.acl-main.554 Jointly learning to align and summarize for neural cross-lingual summarization . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6220--6231, Online. Association for Computational Linguistics

    work page doi:10.18653/v1/2020.acl-main.554 2020

  7. [7]

    Yue Cao, Xiaojun Wan, Jinge Yao, and Dian Yu. 2020 b . https://doi.org/10.1609/aaai.v34i01.5328 Multisumm: Towards a unified model for multi-lingual abstractive summarization . Proceedings of the AAAI Conference on Artificial Intelligence, 34(01):11--18

    work page doi:10.1609/aaai.v34i01.5328 2020

  8. [8]

    Yulong Chen, Huajian Zhang, Yijie Zhou, Xuefeng Bai, Yueguan Wang, Ming Zhong, Jianhao Yan, Yafu Li, Judy Li, Xianchao Zhu, and Yue Zhang. 2023. https://doi.org/10.18653/v1/2023.acl-long.519 Revisiting cross-lingual summarization: A corpus-based study and a new benchmark with improved annotation . In Proceedings of the 61st Annual Meeting of the Associati...

    doi:10.18653/v1/2023.acl-long.519 2023

Show all 68 references
  1. [9]

    Zewen Chi, Li Dong, Shuming Ma, Shaohan Huang, Saksham Singhal, Xian-Ling Mao, Heyan Huang, Xia Song, and Furu Wei. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.125 m T 6: Multilingual pretrained text-to-text transformer with translation pairs . In Proceedings of the 2021...

    2021 doi

  2. [10]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. https://lmsys.org/blog/2023-03-30-vicuna/ Vicuna: An open-source chatbot impressing gpt-4 with 90\

    2023

  3. [11]

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher R \'e . 2022. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344--16359

    2022

  4. [12]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Associat...

    2019 doi

  5. [13]

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022. A survey on in-context learning. arXiv preprint arXiv:2301.00234

    2022 arXiv

  6. [14]

    Yihong Dong, Xue Jiang, Huanyu Liu, Zhi Jin, Bin Gu, Mengfei Yang, and Ge Li. 2024. https://doi.org/10.18653/v1/2024.findings-acl.716 Generalization or memorization: Data contamination and trustworthy evaluation for large language models . In Findings of the Association for Co...

    2024 doi

  7. [15]

    Xiangyu Duan, Mingming Yin, Min Zhang, Boxing Chen, and Weihua Luo. 2019. https://doi.org/10.18653/v1/P19-1305 Zero-shot cross-lingual abstractive sentence summarization through teaching generation and attention . In Proceedings of the 57th Annual Meeting of the Association fo...

    2019 doi

  8. [16]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  9. [17]

    Mehwish Fatima and Michael Strube. 2021. https://doi.org/10.18653/v1/2021.newsum-1.5 A novel W ikipedia based dataset for monolingual and cross-lingual summarization . In Proceedings of the Third Workshop on New Frontiers in Summarization, pages 39--50, Online and in Dominican...

    2021 doi

  10. [18]

    Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378

    1971

  11. [19]

    Mingqi Gao, Wenqing Wang, Xiaojun Wan, and Yuemei Xu. 2023. https://doi.org/10.18653/v1/2023.findings-acl.786 Evaluating factuality in cross-lingual summarization . In Findings of the Association for Computational Linguistics: ACL 2023, pages 12415--12431, Toronto, Canada. Ass...

    2023 doi

  12. [20]

    Shahriar Golchin and Mihai Surdeanu. 2024. https://openreview.net/forum?id=2Rwq6c3tvr Time travel in LLM s: Tracing data contamination in large language models . In The Twelfth International Conference on Learning Representations

    2024

  13. [21]

    Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc'Aurelio Ranzato, Francisco Guzman, and Angela Fan. 2021. The flores-101 evaluation benchmark for low-resource and multilingual machine translation. arXiv preprint arXiv:...

    2021 arXiv

  14. [22]

    Max Grusky, Mor Naaman, and Yoav Artzi. 2018. https://doi.org/10.18653/v1/N18-1065 N ewsroom: A dataset of 1.3 million summaries with diverse extractive strategies . In Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Computational Lingu...

    2018 doi

  15. [23]

    Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M

    Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. 2021. https://doi.org/10.18653/v1/2021.findings-acl.413 XL -sum: Large-scale multilingual abstractive summarization for 44 languages . In Fin...

    2021 doi

  16. [24]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In International Conference on Learning Representations

    2021

  17. [25]

    Faisal Ladhak, Esin Durmus, Claire Cardie, and Kathleen McKeown. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.360 W iki L ingua: A new benchmark dataset for cross-lingual abstractive summarization . In Findings of the Association for Computational Linguistics: EMNLP 2...

    2020 doi

  18. [26]

    Anton Leuski, Chin-Yew Lin, Liang Zhou, Ulrich Germann, Franz Josef Och, and Eduard H. Hovy. 2003. Cross-lingual c*st*rd: English access to hindi information. ACM Trans. Asian Lang. Inf. Process., 2:245--269

    2003

  19. [27]

    Yunlong Liang, Fandong Meng, Jinan Xu, Jiaan Wang, Yufeng Chen, and Jie Zhou. 2023. https://doi.org/10.18653/v1/2023.acl-long.165 Summary-oriented vision modeling for multimodal abstractive summarization . In Proceedings of the 61st Annual Meeting of the Association for Comput...

    2023 doi

  20. [29]

    Yunlong Liang, Fandong Meng, Chulun Zhou, Jinan Xu, Yufeng Chen, Jinsong Su, and Jie Zhou. 2022 b . https://doi.org/10.18653/v1/2022.acl-long.148 A variational hierarchical model for neural cross-lingual summarization . In Proceedings of the 60th Annual Meeting of the Associat...

    2022 doi

  21. [30]

    Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013 ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics

    2004

  22. [31]

    Ran Liu, Ming Liu, Min Yu, He Zhang, Jianguo Jiang, Gang Li, and Weiqing Huang. 2024. https://doi.org/10.18653/v1/2024.findings-acl.574 S um S urvey: An abstractive dataset of scientific survey papers for long document summarization . In Findings of the Association for Computa...

    2024 doi

  23. [32]

    Xiao Liu, Xuanyu Lei, Shengyuan Wang, Yue Huang, Zhuoer Feng, Bosi Wen, Jiale Cheng, Pei Ke, Yifan Xu, Weng Lam Tam, Xiaohan Zhang, Lichao Sun, Hongning Wang, Jing Zhang, Minlie Huang, Yuxiao Dong, and Jie Tang. 2023. https://arxiv.org/abs/2311.18743 Alignbench: Benchmarking c...

    2023 arXiv

  24. [33]

    Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. https://doi.org/10.1162/tacl_a_00343 Multilingual denoising pre-training for neural machine translation . Transactions of the Association for Computational...

    2020 doi

  25. [34]

    Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. https://doi.org/10.18653/v1/2020.acl-main.173 On faithfulness and factuality in abstractive summarization . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1...

    2020 doi

  26. [35]

    Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.759 Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on ...

    2022 doi

  27. [36]

    Thong Nguyen and Luu Anh Tuan. 2022. Improving neural cross-lingual summarization via employing optimal transport distance for knowledge distillation. Proc. of AAAI

    2022

  28. [37]

    OpenAI. 2022. Introducing chatgpt. https://openai.com/blog/chatgpt

    2022

  29. [38]

    OpenAI. 2023. Gpt-4 technical report. ArXiv, abs/2303.08774

    2023 arXiv

  30. [39]

    OpenAI. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276

    2024 arXiv

  31. [40]

    Constantin Or a san and Oana Andreea Chiorean. 2008. http://www.lrec-conf.org/proceedings/lrec2008/pdf/539_paper.pdf Evaluation of a cross-lingual R omanian- E nglish multi-document summariser . In Proceedings of the Sixth International Conference on Language Resources and Eva...

    2008

  32. [41]

    Laura Perez-Beltrachini and Mirella Lapata. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.742 Models and datasets for cross-lingual summarisation . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9408--9423, Online and Punta...

    2021 doi

  33. [42]

    Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is chatgpt a general-purpose natural language processing task solver? arXiv preprint arXiv:2302.06476

    2023 arXiv

  34. [43]

    Yifu Qiu, Yftah Ziser, Anna Korhonen, Edoardo Ponti, and Shay Cohen. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.551 Detecting and mitigating hallucinations in multilingual summarisation . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pro...

    2023 doi

  35. [44]

    Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, and Jacopo Staiano. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.647 MLSUM : The multilingual summarization corpus . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Pro...

    2020 doi

  36. [45]

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063

    2024

  37. [46]

    Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. 2021. https://doi.org/10.18653/v1/2021.findings-acl.304 Multilingual translation from denoising pre-training . In Findings of the Association for Computational Linguistic...

    2021 doi

  38. [47]

    Yuting Tang, Ratish Puduppully, Zhengyuan Liu, and Nancy Chen. 2023. https://doi.org/10.18653/v1/2023.newsum-1.6 In-context learning of large language models for controlled dialogue summarization: A holistic benchmark and empirical analysis . In Proceedings of the 4th New Fron...

    2023 doi

  39. [48]

    InternLM Team. 2023. Internlm: A multilingual language model with progressively enhanced capabilities

    2023

  40. [49]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023 a . Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    2023 arXiv

  41. [50]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023 b . Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  42. [51]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30

    2017

  43. [52]

    Bin Wang, Chen Zhang, Yan Zhang, Yiming Chen, and Haizhou Li. 2022 a . https://doi.org/10.18653/v1/2022.emnlp-main.325 Analyzing and evaluating faithfulness in dialogue summarization . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, p...

    2022 doi

  44. [53]

    Danqing Wang, Jiaze Chen, Hao Zhou, Xipeng Qiu, and Lei Li. 2021. https://doi.org/10.18653/v1/2021.findings-acl.242 Contrastive aligned joint learning for multilingual summarization . In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2739--27...

    2021 doi

  45. [54]

    Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023 a . https://doi.org/10.18653/v1/2023.newsum-1.1 Is C hat GPT a good NLG evaluator? a preliminary study . In Proceedings of the 4th New Frontiers in Summariza...

    2023 doi

  46. [55]

    Jiaan Wang, Yunlong Liang, Fandong Meng, Beiqi Zou, Zhixu Li, Jianfeng Qu, and Jie Zhou. 2023 b . https://doi.org/10.18653/v1/2023.newsum-1.2 Zero-shot cross-lingual summarization via large language models . In Proceedings of the 4th New Frontiers in Summarization Workshop, pa...

    2023 doi

  47. [56]

    Jiaan Wang, Fandong Meng, Ziyao Lu, Duo Zheng, Zhixu Li, Jianfeng Qu, and Jie Zhou. 2022 b . https://doi.org/10.18653/v1/2022.emnlp-main.526 C lid S um: A benchmark dataset for cross-lingual dialogue summarization . In Proceedings of the 2022 Conference on Empirical Methods in...

    2022 doi

  48. [57]

    Jiaan Wang, Fandong Meng, Duo Zheng, Yunlong Liang, Zhixu Li, Jianfeng Qu, and Jie Zhou. 2022 c . https://doi.org/10.1162/tacl_a_00520 A survey on cross-lingual summarization . Transactions of the Association for Computational Linguistics, 10:1304--1323

    2022 doi

  49. [58]

    Jiaan Wang, Fandong Meng, Duo Zheng, Yunlong Liang, Zhixu Li, Jianfeng Qu, and Jie Zhou. 2023 c . https://doi.org/10.18653/v1/2023.acl-long.843 Towards unifying multi-lingual and cross-lingual summarization . In Proceedings of the 61st Annual Meeting of the Association for Com...

    2023 doi

  50. [59]

    Cheng Xu, Shuhao Guan, Derek Greene, M Kechadi, et al. 2024. Benchmark data contamination of large language models: A survey. arXiv preprint arXiv:2406.04244

    2024 arXiv

  51. [60]

    Ruochen Xu, Chenguang Zhu, Yu Shi, Michael Zeng, and Xuedong Huang. 2020. https://aclanthology.org/2020.aacl-main.53 Mixed-lingual pre-training for cross-lingual summarization . In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computation...

    2020

  52. [61]

    Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. 2023. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305

    2023 arXiv

  53. [62]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng...

    2024 arXiv

  54. [63]

    Weinberger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. https://openreview.net/forum?id=SkeHuCVFDr Bertscore: Evaluating text generation with BERT . In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April...

    2020

  55. [64]

    Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. 2023. Siren's song in the ai ocean: a survey on hallucination in large language models. arXiv preprint arXiv:2309.01219

    2023 arXiv

  56. [65]

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223

    2023 arXiv

  57. [66]

    Shaohui Zheng, Zhixu Li, Jiaan Wang, Jianfeng Qu, An Liu, Lei Zhao, and Zhigang Chen. 2023. Long-document cross-lingual summarization. In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining, pages 1084--1092

    2023

  58. [67]

    Junnan Zhu, Qian Wang, Yining Wang, Yu Zhou, Jiajun Zhang, Shaonan Wang, and Chengqing Zong. 2019. https://doi.org/10.18653/v1/D19-1302 NCLS : Neural cross-lingual summarization . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the...

    2019 doi

  59. [68]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  60. [69]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.