Pith. sign in

REVIEW 3 major objections 4 minor 40 references

Multi-round, Chain-of-thought Post-editing for Unfaithful Summaries

T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that an LLM critic-editor loop, which repeatedly scores, edits, and re-scores news summaries against their source articles, produces more faithful final summaries, and that chain-of-thought prompts identifying error spans…

desk verdict A clean, honest empirical study of multi-round CoT post-editing; the single-round results are solid, but the headline multi-round gains rest on automatic metrics and a self-referential stopping rule. read the letter →

arxiv 2501.11273 v1 pith:4T2BYCYP submitted 2025-01-20 cs.CL

Yi-Hui Lee , Xiangci Li , Jessica Ouyang This is my paper · ORCID
classification cs.CL
keywords faithfulnesssummarizationpost-editingchain-of-thoughtpromptinglargelanguagemodelsfactualconsistencycritic-editorloopmulti-roundediting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that repairing factually wrong news summaries is best done as a loop rather than a single correction pass: an LLM scores a summary against the source article, edits it when the score says it is unfaithful, scores the edited version, and repeats until the critic stops flagging errors. The authors show that prompting the editor to first reason about where the error is and what kind of error it is, such as a wrong predicate, wrong entity, or information not in the article, produces more faithful summaries than plain editing instructions. They measure roughly a 50% improvement in faithfulness scores between the first and final editing rounds and report high editing success rates across several LLMs. A sympathetic reading is that multi-round, critic-guided editing is a viable zero-shot alternative to fine-tuned post-editing models.

What carries the argument

The central object is the critic-editor loop: a faithfulness critic that scores a document-summary pair on a five-point Likert scale using in-context demonstrations from FRANK, and an editor prompted with one of four zero-shot instructions: plain editing, editing after locating inconsistent spans, editing after classifying error types from the FRANK taxonomy, or editing after doing both. The loop alternates critic and editor until the critic predicts faithfulness or five rounds are reached. The load-bearing design decisions are that critic and editor run in separate sessions, since combined sessions dramatically hurt editing success, and that the chain-of-thought reasoning step, not necessarily its correctness, is what improves editing.

What would settle it

Take a random sample of summaries that exited the loop as faithful, have human annotators identify factual errors in them, and compare the human error rate with the metric scores across rounds; if metric-measured improvement does not correspond to a lower human error rate, the claim is not established.

Watch

Extended reading notes

Core claim

The paper's central claim is that an LLM can serve as both critic and editor of summary faithfulness, and that the loop's value depends on what the editor is asked to think about before rewriting. When the editor prompt asks the model to identify the inconsistent span, classify the factual error type using the FRANK taxonomy, or both, the edited summaries score higher on automatic faithfulness metrics and resemble human edits more closely than the prior CompEdit baseline, while matching a fine-tuned T0 editor on DeFacto without any training. The paper also claims that multiple rounds, driven by the critic's score rather than a fixed single edit, yield continued gains, with most summaries repaired by four rounds and a roughly 50% score increase from first to final round. It further finds that chain-of-thought reasoning helps even when the predicted error span is wrong, suggesting the reasoning itself resets the model's editing behavior.

Load-bearing premise

The multi-round improvement result depends on the automatic faithfulness scores used to measure it being trustworthy proxies for human judgments of factual accuracy, since the human check covers only 292 summaries.

Editorial extensions

If this is right

  • A single off-the-shelf LLM, prompted zero-shot, can repair unfaithful news summaries in a critic-editor loop and match the faithfulness of a fine-tuned post-editing model on DeFacto.
  • Chain-of-thought reasoning about error spans helps more than reasoning about error types, and the combination performs at least as well as either alone on the automatic metrics.
  • Summaries that stay unfaithful after one edit continue to improve across additional rounds, with most exiting after four rounds and an overall measured gain of about 50% from first to final round.
  • Error-type reasoning adds a measurable benefit specifically when the predicted type is Predicate, Entity, or Out-of-Article, the common error categories in the FRANK data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not test this, but if the chain-of-thought gains come mainly from resetting the editor's focus rather than from correct span identification, a different structured first step, such as asking the model to list source facts before rewriting, should produce similar gains.
  • The paper's failure example shows the loop can stop on a newly invented claim; a natural extension would be to require two independent critics to agree before the loop halts, or to add a source-grounded verification step after the final round.
  • The same critic-editor pattern is transferable on its face to any generation task with a grounding document, such as biography generation or open-domain question answering, where hallucinated content can be iteratively located and corrected.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes an LLM-based critic-editor framework for post-editing unfaithful news summaries. The critic scores faithfulness on a five-point scale and decides whether further editing is needed; the editor applies chain-of-thought prompts that identify error spans, error types, or both before rewriting. The framework is evaluated on the FRANK and DeFacto benchmarks with four LLMs, using QAFactEval, DAE, FactCC, ROUGE/BERTScore, editing success rate, and a small human evaluation. The main claims are that the critic correlates well with human judgments, that the editor achieves higher editing success and better faithfulness than prior single-round systems, that chain-of-thought error span identification is the most useful prompting strategy, and that multiple editing rounds yield roughly a 50% faithfulness improvement between the first and final rounds.

Significance. If the multi-round claim holds, the paper makes a useful contribution to faithfulness post-editing: it explicitly explores iterative editing, provides prompt templates that others can reuse, evaluates four LLMs including open-weight models, and includes gold-span/gold-type ablations that help isolate the effect of chain-of-thought reasoning. The paper is also honest in reporting a failure case in which the critic certifies an unfaithful final summary. However, the central multi-round improvement claim is currently supported mainly by automatic metrics whose human correlation is weak, by a stopping rule chosen by the same model that edits, and by a human evaluation that does not directly compare original, first-round, and final multi-round outputs. For these reasons the significance of the result is real but not yet fully established.

major comments (3)
  1. [§5.4, Figure 4; §2.1 Table 1; Appendix B.4] The multi-round improvement claim is not fully established by the reported evidence. The summaries measured in Figure 4 are those that exit the loop when the LLM critic predicts faithfulness, so the final set is selected by the system's own critic; the abstract's 'about 50% improvement between the first and final editing rounds' is computed with QAFactEval, DAE, and FactCC, not with human judgments of the final outputs. Table 1 shows that these metrics correlate only weakly with human judgments in the XSum setting (e.g., QAFactEval PCC 0.31, DAE 0.24, FactCC 0.07), and Appendix B.4 documents a concrete case where the critic stops on a summary that is still incorrect despite a final QAFactEval of 1.53 versus 3.86 for the input. The improvement could therefore reflect convergence to the critic's preferences rather than to genuine faithfulness. I recommend adding a human evaluation that compares original, round-1, and final multi-round summaries, including the critic's stopping decisions, or alternatively evaluating all summaries at every round regardless of the stopping decision so that the claim does not rest on a self-selected subset.
  2. [§5.2, §5.3, Tables 2, 3, 4] The abstract and Section 5.3 state that chain-of-thought reasoning 'significantly improves' post-editing performance, but no significance tests or confidence intervals are reported anywhere in the paper. Several of the differences that drive the conclusions are numerically small; for example, in Table 4 the QAFactEval differences between Editor, EditorSpan, EditorType, and EditorSpan+Type range from 2.456 to 2.707, and in Table 2 the best Llama and text-bison EditorSpan scores differ by only 0.081 on QAFactEval. Given the known noise in automatic faithfulness metrics, the authors should report bootstrap confidence intervals or paired significance tests for the main comparisons, at least for the headline claims about CoT prompting and about the comparison to Liu et al.'s T0 editor on DeFacto.
  3. [Appendix B.1, Tables 9 and 10] The human evaluation does not validate the central multi-round claim. As described, judges are shown three edited summaries corresponding to EditorSpan, EditorType, and EditorSpan+Type, but the protocol does not include the original input summary, a round-1 summary, or a final multi-round summary selected by the critic's stopping decision. The human data therefore only compares three single-round prompt variants on 292 summaries, and no inter-annotator agreement is reported. This is useful for the prompt-comparison conclusion but cannot support the claim that multiple rounds improve faithfulness relative to single-round editing, nor that the critic's stopping decisions are reliable. The authors should either extend the human evaluation to the multi-round setting or explicitly limit the multi-round claim to automatic-metric evidence.
minor comments (4)
  1. [Section 5.1] The editing success rate is defined as the percentage of unfaithful input summaries that are modified 'regardless of the factual consistency of the resulting edits'; the term 'success rate' is therefore somewhat misleading. Since factuality metrics are also reported, this is not a fatal issue, but the definition should be stated more clearly in the main text rather than only in the metric description.
  2. [Section 6] There is a typo in the first sentence: 'LLMcritic' should be 'LLM critic'.
  3. [Appendix B.4] The phrase 'a summary improved by multi-round editing and a failure case where multi-round editing still results in an incorrect summar' contains the typo 'summar' for 'summary'.
  4. [Footnote 1 and Section 3] The choice of five as the maximum number of editing rounds is justified only by an informal observation that 'most summaries' are predicted faithful after four rounds. The paper would benefit from reporting the actual distribution of exit rounds and from ablating the maximum-round and stop-threshold choices, since these are free parameters of the framework.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the multi-round improvement claim is evaluated with external metrics and human judgments, not with the model's own critic scores.

full rationale

The paper's central derivation chain is not circular. The LLM critic is used only to decide whether to continue editing; the reported faithfulness improvements are measured with external metrics (QAFactEval, DAE, FactCC) computed against the source article, and with human judgments in Appendix B.1. These evaluation signals are independent of the critic's own predictions, so the 'about 50% improvement' and higher editing success rate claims are empirical results rather than consequences of the critic's definition. The editor prompts are zero-shot with no fitted parameters, and the paper explicitly documents a failure case (Figure 16) in which the critic incorrectly certifies an unfaithful summary as faithful, demonstrating that the critic's approval is not definitionally equivalent to the final evaluation metrics. No load-bearing self-citation chain appears: the cited prior work (e.g., FRANK, DeFacto, CompEdit) is external, and the paper does not invoke an author-derived uniqueness theorem to justify its design. The acknowledged limitations about metric-human correlation and the small human evaluation are validity concerns, not circularity. The derivation is self-contained against external benchmarks, so the appropriate circularity score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central empirical claims depend on hand-chosen settings: a maximum of five editing rounds, a strict critic stopping threshold of 5, and two in-context demonstrations. The evaluation also assumes the standard automatic faithfulness metrics are valid proxies for human judgment despite their known limitations, and that the human error annotations used for analysis are accurate.

free parameters (3)
  • Maximum editing rounds = 5
    Chosen after initial experiments; the authors observed that most summaries are predicted as faithful after four rounds and only a handful require a fifth (Section 3.2, footnote 1).
  • Critic stop threshold = LLM critic must predict 5 (faithful)
    The authors experimented with a relaxed stopping criterion of 4 and report it 'resulted in lower final factuality scores and editing rates', so the strict threshold was selected on the same experimental setup without a described held-out validation (Section 5.1, footnote 5).
  • Number of in-context demonstrations for critic = 2 (one score-2 and one score-4 example from FRANK CNN/Daily Mail)
    The authors chose two demonstrations after finding that extreme score examples caused the critic to always predict 1; this is a hand-picked prompt design choice (Section 3.1).
assumptions (3)
  • domain assumption Human error span and error type annotations in FRANK and the XSum Hallucination Annotations are accurate ground truth.
    These annotations are used to evaluate CoT predictions and to analyze which error types matter for editing (Sections 4 and 5.3).
  • domain assumption QAFactEval, DAE, and FactCC are valid enough proxies for human faithfulness judgments when measuring post-edit improvement.
    The paper itself notes in Section 2.1 that many faithfulness metrics have poor correlation with human judgments, yet the main improvement claims rely on these three metrics, supplemented by a small human evaluation.
  • domain assumption The LLM critic's five-point faithfulness prediction is reliable enough to serve as the stopping criterion for multi-round editing.
    The editing loop stops when the critic predicts 5, and the failure case in Appendix B.4 shows the critic can incorrectly label an unfaithful summary as faithful, demonstrating that this assumption can fail.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multi-round, Chain-of-thought Post-editing for Unfaithful Summaries." pith.science (2026). https://pith.science/paper/4T2BYCYP

@misc{pith2026250111273,
  author       = {Pith},
  title        = {Pith review of: Multi-round, Chain-of-thought Post-editing for Unfaithful Summaries},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4T2BYCYP}},
  note         = {Machine review of arXiv:2501.11273}
}
read the original abstract

Recent large language models (LLMs) have demonstrated a remarkable ability to perform natural language understanding and generation tasks. In this work, we investigate the use of LLMs for evaluating faithfulness in news summarization, finding that it achieves a strong correlation with human judgments. We further investigate LLMs' capabilities as a faithfulness post-editor, experimenting with different chain-of-thought prompts for locating and correcting factual inconsistencies between a generated summary and the source news document and are able to achieve a higher editing success rate than was reported in prior work. We perform both automated and human evaluations of the post-edited summaries, finding that prompting LLMs using chain-of-thought reasoning about factual error types is an effective faithfulness post-editing strategy, performing comparably to fine-tuned post-editing models. We also demonstrate that multiple rounds of post-editing, which has not previously been explored, can be used to gradually improve the faithfulness of summaries whose errors cannot be fully corrected in a single round.

Figures

Figures reproduced from arXiv: 2501.11273 by the authors.

Figure 1
Figure 1. Overview of the proposed summary post￾editing approach. 2022) prompts for zero-shot editing of unfaithful summaries: asking the LLM to identify the sum￾mary span(s) containing factual error(s), asking it to categorize the type(s) of error(s) (e.g. an incor￾rect entity or time expression), and asking for both error spans and types. We evaluate the LLMs not only on the final summary editing task, but also on the auxil… view at source ↗
Figure 2
Figure 2. Prompt format for the faithfulness critic. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Prompt format for EditorSpan+Type. Er￾ror type descriptions are from Pagnoni et al. (2021). Number of sentences N = 3 for CNN/Daily Mail, and N = 1 for XSum, matching the lengths of their refer￾ence summaries. notators. A sentence-level factuality score of 0 indicates there are no errors present, while a score of 1 indicates the presence of at least one labeled error; summary-level scores are obtained by averag￾ing … view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Comparison of summary faithfulness scores using QAFactEval across multiple rounds of editing. These [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 7
Figure 7. Figure 7: Prompt format for EditorType. Error type [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 5
Figure 5. Figure 5: Prompt format for Editor. N = 3 for arti￾cle/summary pairs from the CNN/Daily Mail partition, and N = 1 for XSum, matching the lengths of their system-generated summaries. Find the span in the summary that is inconsistent with the source article, then edit the summary …
Figure 6
Figure 6. Figure 6: Prompt format for EditorSpan. N = 3 for ar￾ticle/summary pairs from the CNN/Daily Mail partition, and N = 1 for XSum, matching the lengths of their system-generated summaries. A summary can be inconsistent with its source article in dif￾ferent ways, such as Predicate E…
Figure 8
Figure 8. Figure 8: Prompt format for editing with given both [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Full comparison of Mixtral 8x7B faithfulness scores using QAFactEval across multiple rounds of editing. [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Gemini-pro faithfulness scores using QAFactEval across multiple rounds of editing [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: Text-bison-001 faithfulness scores using QAFactEval across multiple rounds of editing [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]
Figure 12
Figure 12. Figure 12: Full comparison of Mixtral 8x7B faithfulness scores using DAE across multiple rounds of editing. [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: Gemini-pro faithfulness scores using DAE across multiple rounds of editing [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]
Figure 14
Figure 14. Figure 14: Text-bison-001 faithfulness scores using DAE across multiple rounds of editing [PITH_FULL_IMAGE:figures/full_fig_p020_14.png]
Figure 15
Figure 15. Figure 15: Example of successful multi-round editing. [PITH_FULL_IMAGE:figures/full_fig_p021_15.png]
Figure 16
Figure 16. Figure 16: Example of unsuccessful multi-round editing and the critic incorrectly predicting that an edited summary [PITH_FULL_IMAGE:figures/full_fig_p022_16.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

40 extracted references · 9 canonical work pages

  1. [1]

    Meng Cao, Yue Dong, Jiapeng Wu, and Jackie Chi Kit Cheung. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.506 Factual error correction for abstractive summarization models . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6251--6258, Online. Association for Computational Linguistics

    doi:10.18653/v1/2020.emnlp-main.506 2020

  2. [2]

    Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2018. Faithful to the original: Fact aware neural abstractive summarization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32

    work page 2018

  3. [3]

    Shiqi Chen, Siyang Gao, and Junxian He. 2023. https://arxiv.org/abs/2305.14069 Evaluating factual consistency of summaries with large language models . Preprint, arXiv:2305.14069

    work page Pith review arXiv 2023

  4. [4]

    Wendi Cui, Jiaxin Zhang, Zhuohang Li, Lopez Damien, Kamalika Das, Bradley Malin, and Sricharan Kumar. 2024. https://arxiv.org/abs/2401.02132 Dcr-consistency: Divide-conquer-reasoning for consistency evaluation and improvement of large language models . Preprint, arXiv:2401.02132

    work page Pith review arXiv 2024

  5. [5]

    Yue Dong, Shuohang Wang, Zhe Gan, Yu Cheng, Jackie Chi Kit Cheung, and Jingjing Liu. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.749 Multi-fact correction in abstractive text summarization . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9320--9331, Online. Association for Computational Linguistics

    work page doi:10.18653/v1/2020.emnlp-main.749 2020

  6. [6]

    Esin Durmus, He He, and Mona Diab. 2020. https://doi.org/10.18653/v1/2020.acl-main.454 FEQA : A question answering evaluation framework for faithfulness assessment in abstractive summarization . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5055--5070, Online. Association for Computational Linguistics

    doi:10.18653/v1/2020.acl-main.454 2020

  7. [7]

    Alex Fabbri, Prafulla Kumar Choubey, Jesse Vig, Chien-Sheng Wu, and Caiming Xiong. 2022 a . https://doi.org/10.18653/v1/2022.emnlp-main.623 Improving factual consistency in summarization with compression-based post-editing . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9149--9156, Abu Dhabi, United Arab ...

    work page doi:10.18653/v1/2022.emnlp-main.623 2022

  8. [8]

    Alexander Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2022 b . https://doi.org/10.18653/v1/2022.naacl-main.187 QAF act E val: Improved QA -based factual consistency evaluation for summarization . In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pag...

    doi:10.18653/v1/2022.naacl-main.187 2022

Show all 40 references
  1. [9]

    Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019. https://doi.org/10.18653/v1/P19-1213 Ranking generated summaries by correctness: An interesting but challenging application for natural language inference . In Proceedings of the 57...

    2019 doi

  2. [10]

    Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023. https://arxiv.org/abs/2302.04166 Gptscore: Evaluate as you desire . Preprint, arXiv:2302.04166

    2023 arXiv

  3. [11]

    Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, and Kelvin Guu. 2023 a . https://doi.org/10.18653/v1/2023.acl-long.910 RARR : Researching and revising what language models say, using la...

    2023 doi

  4. [12]

    Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Shiping Yang, and Xiaojun Wan. 2023 b . https://arxiv.org/abs/2304.02554 Human-like summarization evaluation with chatgpt . Preprint, arXiv:2304.02554

    2023 arXiv

  5. [13]

    Liu, and Mohammad Saleh

    Ben Goodrich, Vinay Rao, Peter J. Liu, and Mohammad Saleh. 2019. https://doi.org/10.1145/3292500.3330955 Assessing the factual accuracy of generated text . In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '19, page 166–17...

    2019

  6. [14]

    Tanya Goyal and Greg Durrett. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.322 Evaluating factuality in generation with dependency-level entailment . In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3592--3603, Online. Association for Co...

    2020 doi

  7. [15]

    Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2023. https://arxiv.org/abs/2209.12356 News summarization and evaluation in the era of gpt-3 . Preprint, arXiv:2209.12356

    2023 arXiv

  8. [16]

    Qi Jia, Siyu Ren, Yizhu Liu, and Kenny Zhu. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.679 Zero-shot faithfulness evaluation for text summarization with foundation language model . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing...

    2023 doi

  9. [17]

    Wojciech Kryscinski, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. https://doi.org/10.18653/v1/D19-1051 Neural text summarization: A critical evaluation . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing ...

    2019 doi

  10. [18]

    Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.750 Evaluating the factual consistency of abstractive text summarization . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Pro...

    2020 doi

  11. [19]

    Hwanhee Lee, Cheoneum Park, Seunghyun Yoon, Trung Bui, Franck Dernoncourt, Juae Kim, and Kyomin Jung. 2022. https://doi.org/10.18653/v1/2022.gem-1.41 Factual error correction for abstractive summaries using entity retrieval . In Proceedings of the 2nd Workshop on Natural Langu...

    2022 doi

  12. [20]

    Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013/ ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics

    2004

  13. [21]

    Yen-Ting Lin and Yun-Nung Chen. 2023. https://doi.org/10.18653/v1/2023.nlp4convai-1.5 LLM -eval: Unified multi-dimensional automatic evaluation for open-domain conversations with large language models . In Proceedings of the 5th Workshop on NLP for Conversational AI (NLP4ConvA...

    2023 doi

  14. [22]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023 a . https://arxiv.org/abs/2303.16634 G-eval: Nlg evaluation using gpt-4 with better human alignment . Preprint, arXiv:2303.16634

    2023 arXiv

  15. [23]

    Yixin Liu, Budhaditya Deb, Milagro Teruel, Aaron Halfaker, Dragomir Radev, and Ahmed Hassan Awadallah. 2023 b . https://doi.org/10.18653/v1/2023.acl-long.844 On improving summarization factual consistency from natural language feedback . In Proceedings of the 61st Annual Meeti...

    2023 doi

  16. [24]

    Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2023. https://arxiv.org/abs/2303.15621 Chatgpt as a factual inconsistency evaluator for text summarization . Preprint, arXiv:2303.15621

    2023 arXiv

  17. [25]

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2024. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36

    2024

  18. [26]

    Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. https://doi.org/10.18653/v1/2020.acl-main.173 On faithfulness and factuality in abstractive summarization . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1...

    2020 doi

  19. [27]

    Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, C a g lar Gu l c ehre, and Bing Xiang. 2016. https://doi.org/10.18653/v1/K16-1028 Abstractive text summarization using sequence-to-sequence RNN s and beyond . In Proceedings of the 20th SIGNLL Conference on Computational Natural...

    2016 doi

  20. [28]

    Cohen, and Mirella Lapata

    Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. https://doi.org/10.18653/v1/D18-1206 Don`t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization . In Proceedings of the 2018 Conference on Empirical Methods in Natura...

    2018 doi

  21. [29]

    Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. https://doi.org/10.18653/v1/2020.acl-main.441 Adversarial NLI : A new benchmark for natural language understanding . In Proceedings of the 58th Annual Meeting of the Association for Comp...

    2020 doi

  22. [30]

    Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. https://doi.org/10.18653/v1/2021.naacl-main.383 Understanding factuality in abstractive summarization with FRANK : A benchmark for factuality metrics . In Proceedings of the 2021 Conference of the North America...

    2021 doi

  23. [31]

    Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, et al. 2021. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations

    2021

  24. [32]

    Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, and Patrick Gallinari. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.529 Q uest E val: Summarization asks for fact-based evaluation . In Proceedings of the 2021 Conference o...

    2021 doi

  25. [33]

    Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. https://doi.org/10.18653/v1/2020.acl-main.450 Asking and answering questions to evaluate the factual consistency of summaries . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 500...

    2020 doi

  26. [34]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. https://proceedings.neurips.cc/paper_files/paper/2022/file/9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf Chain-of-thought prompting elicits reasoning...

    2022

  27. [35]

    Haopeng Zhang, Xiao Liu, and Jiawei Zhang. 2023 a . https://doi.org/10.18653/v1/2023.findings-emnlp.214 Extractive summarization via C hat GPT for faithful summary generation . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 3270--3278, Singapor...

    2023 doi

  28. [36]

    Shiyue Zhang, David Wan, and Mohit Bansal. 2023 b . https://doi.org/10.18653/v1/2023.acl-long.120 Extractive is not faithful: An investigation of broad unfaithfulness problems in extractive summarization . In Proceedings of the 61st Annual Meeting of the Association for Comput...

    2023 doi

  29. [37]

    Weinberger, and Yoav Artzi

    Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. https://openreview.net/forum?id=SkeHuCVFDr Bertscore: Evaluating text generation with bert . In International Conference on Learning Representations

    2020

  30. [38]

    Chenguang Zhu, William Hinthorn, Ruochen Xu, Qingkai Zeng, Michael Zeng, Xuedong Huang, and Meng Jiang. 2021. https://doi.org/10.18653/v1/2021.naacl-main.58 Enhancing factual consistency of abstractive summarization . In Proceedings of the 2021 Conference of the North American...

    2021 doi

  31. [39]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  32. [40]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.