Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

Markov-Enhanced Clustering for Long Document Summarization: Tackling the 'Lost in the Middle' Challenge with Large Language Models

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A Markov chain over semantic clusters supplies the ordering for long-document summarization, and on BookSum this pipeline beats direct full-document LLM summarization on ROUGE-1, ROUGE-2, coherence, and BLEURT.

desk verdict The Markov-ordering idea is new and plausible, but the missing baseline-order control makes the reported gains uninterpretable as evidence for it. read the letter →

arxiv 2506.18036 v1 pith:LNNED3KW submitted 2025-06-22 cs.CL

Aziz Amari (1) , Mohamed Achref Ben Ammar (1) ((1) National Institute of Applied Science , Technology (INSAT) , University of Carthage , Tunis , Tunisia) This is my paper
classification cs.CL
keywords longdocumentsummarizationlostinthemiddleMarkovchainsemanticclusteringHamiltonianpathhybridBookSumLLM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that the 'lost in the middle' failure of large language models on very long documents can be mitigated by a hybrid extractive-abstractive pipeline. The pipeline splits a document into 500-token chunks, embeds them, groups them into thematic clusters, summarizes each cluster, and then uses a Markov chain over cluster IDs to choose the most probable sequence of these summaries before an LLM writes the final summary. On the BookSum dataset, this Markov-ordered approach attains the highest ROUGE-1 and ROUGE-2 scores, the highest first- and second-order coherence, and the highest BLEURT score compared with both direct full-document LLM summarization and cluster summarization without the ordering step. The claimed payoff is better retention of key information and a more coherent narrative flow while sending only a small subset of the document's chunks through the LLM, making the method less resource-intensive than full-context inference.

What carries the argument

The central object is the transition matrix $T \in \mathbb{R}^{k\times k}$ built from the sequence of cluster IDs assigned to consecutive chunks, with $T_{i,j}$ the observed probability that a chunk in cluster $i$ is followed by a chunk in cluster $j$. The paper treats the clusters as states of a Markov chain and then solves the most probable Hamiltonian path problem over these states with dynamic programming and bitmasking (complexity $O(n^2 2^n)$ time, $O(n 2^n)$ space), where the path's score is the product of the transition probabilities along it. That path supplies the semantic order in which cluster summaries are concatenated and handed to the LLM for the final abstractive summary.

What would settle it

Run the pipeline on BookSum with the number of clusters varied independently of the solver's 22-node cap (for example $n=5,10,20,40$) and check whether the ROUGE-1, coherence, and BLEURT gains over full-document LLM summarization persist when $n$ is not constrained by the DP limit. A second check is to replace the Markov Hamiltonian path with the clusters' original document order and measure how much of the reported gain actually comes from the ordering step.

Watch

Extended reading notes

Core claim

The central claim is that the narrative flow of a long document is captured well enough by the transition probabilities between cluster states, so that ordering cluster summaries along the most probable Hamiltonian path in that transition matrix produces a final summary that is both more complete and more coherent than summarizing the full document at once. The authors report, for the BookSum dataset with GPT-4o-mini, that their Markov-Cluster approach reaches ROUGE-1 of 34.13 and ROUGE-2 of 6.392, versus 23.99 and 5.440 for the full-document baseline and 33.72 and 6.190 for the cluster-based approach without Markov ordering. It also reports the highest first-order and second-order coherence (0.863 and 0.862) and the highest BLEURT (0.829). The paper's claim is deliberately qualified on BERTScore, where the Markov-Cluster F1 of 0.823 trails both the cluster baseline and the full-document baseline; the stated achievement is the best score on the other four metrics, not all metrics.

Load-bearing premise

The paper never states how the number of clusters $n$ is chosen, and the DP pathfinder only handles $n<22$, so the Markov ordering is computed on a grouping whose size may have been chosen to fit the solver rather than the document's content.

Editorial extensions

If this is right

  • On BookSum, Markov-ordered cluster summarization outperforms direct full-document LLM summarization on ROUGE-1, ROUGE-2, first- and second-order coherence, and BLEURT, showing that chunking plus semantic ordering can recover information that full-context inference loses.
  • The Markov ordering step also improves over the same cluster pipeline without ordering, so the sequence-selection step contributes to coherence beyond the clustering itself.
  • Because only the top-5 chunks nearest each cluster centroid are sent to the LLM, the pipeline's LLM context usage is a small fraction of the document, making the approach computationally cheaper than full-document inference.
  • The method is tailored to very long inputs: short documents yield too few chunk vectors for meaningful clustering, so the paper's claims are scoped to long-form content such as books.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not isolate the Markov ordering from clustering: its cluster baseline uses the same clusters without ordering, so the reader cannot tell how much of the gain comes from the ordering rule versus from the clustering itself.
  • Because the DP pathfinder is described as handling only up to $n=22$ nodes, a 108k-token book is compressed into at most 22 clusters; whether that coarse grouping helps or hurts, and whether the gains persist at finer cluster counts, is an open question the paper itself flags.
  • The resource-efficiency claim is qualitative; no wall-clock, token-count, or cost measurements are reported, so a direct budget comparison between the pipeline and full-document inference would be the natural test.
  • The paper mentions SCROLLS and FacetSum but evaluates only BookSum, so transfer of the method to scientific and structured long documents is a testable extension rather than an established result.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes a hybrid extractive-abstractive pipeline for long-document summarization. It splits a long document into 500-token chunks with overlap, embeds each chunk with the open-source nomic-embed-text-v1 model, clusters the embeddings with K-means++, summarizes the top-k chunks nearest each cluster centroid with GPT-4o-mini, and then orders the cluster summaries by solving a maximum-probability Hamiltonian path over a Markov transition matrix estimated from the sequence of cluster IDs in the source document. The final summary is produced by an LLM from the ordered concatenation. Experiments on BookSum compare the proposed method ('Markov-Cluster') with direct full-document LLM summarization and with cluster summarization without the ordering step, reporting ROUGE, BERTScore, BLEURT, and sentence-level coherence. The paper claims the highest ROUGE-1, ROUGE-2, coherence, and BLEURT scores, as well as reduced resource use.

Significance. If the result holds, it would show that a lightweight extractive clustering step plus a principled narrative-order heuristic can improve LLM summarization of very long documents, and the release of code and the use of an open-weight embedder make the pipeline easy to reproduce. The hybrid abstractive-extractive design and the Markov-chain path-ordering idea are interesting, and the complexity analysis of the DP pathfinding is sound. However, the manuscript's experimental support is currently too weak to sustain the central claim: the main comparison lacks a control for the ordering mechanism, the cluster count is undisclosed, and no variance or significance information is provided.

major comments (5)
  1. [3.2, Table 2] The Cluster-Sum baseline is defined only as omitting 'the sequence selection step', but the manuscript never states how its cluster summaries are ordered before the final LLM aggregation. Because the sole difference between Cluster-Sum and Markov-Cluster in Table 2 is this ordering, the comparison cannot attribute the observed gains (ROUGE-1 +0.41, ROUGE-2 +0.20, 1st-order coherence +0.011) to the Hamiltonian path. Without a control that feeds cluster summaries in first-appearance order in the source document, the central claim that Markov-chain ordering is responsible is untested.
  2. [3.1, Appendix A, Conclusion] The number of clusters n is never reported, even though K-means++ requires choosing k and the DP solver is stated to handle only 'up to 22 nodes'. The Conclusion explicitly admits that whether capping n<22 'improves performance or negatively impacts summarization quality' is open. Without reporting the distribution of n across the 187 BookSum books and how k was chosen, the reader cannot determine whether the experiments operate in the intended regime or in a restricted, possibly lossy one. The authors should disclose n, state how the DP constraint interacts with the clustering step, and provide sensitivity results for at least two or three cluster counts.
  3. [Table 2, Section 4] Table 2 contains a single set of point estimates with no standard deviations, confidence intervals, or significance tests. The reported differences are small (e.g., ROUGE-1 34.13 vs 33.72; BLEURT 0.829 vs 0.827), so they may lie within run-to-run variation of the LLM and the K-means++ initialization. Moreover, the text claims improvements across 'all evaluated metrics', but the BF1 column shows 0.823 for Markov-Cluster versus 0.831 for LLM-Full, which contradicts that wording. Multiple random seeds, paired significance tests, and a corrected claim are required before the superiority assertion can be accepted.
  4. [Section 4] The evaluation does not include length-controlled baselines or any position-based analysis, so the 'Lost in the Middle' challenge named in the title is never directly tested. Because BookSum summaries average about 1,301 tokens at 2% compression, ROUGE and BLEURT are sensitive to summary length; matching the output lengths of LLM-Full and Cluster-Sum is necessary to attribute differences to content selection and ordering. In addition, an analysis of how often information originally located in the middle third of the document appears in the generated summaries would make the connection to the lost-in-the-middle phenomenon concrete.
  5. [Abstract, Section 5] The paper repeatedly claims the method is 'significantly less resource-intensive' than direct LLM inference, but no measurements of latency, token cost, FLOPs, or context-window usage are reported. This claim is load-bearing for the motivating efficiency narrative and should be supported with concrete measurements, or downgraded to a qualitative statement.
minor comments (5)
  1. [Section 3.1] Notation is inconsistent: k is used both for the number of unique clusters in Eq. (1) and for the top_k=5 vectors in the cluster summarization step, while n denotes both the number of chunks and the number of DP nodes. Distinct symbols would improve clarity.
  2. [Section 3.1, Table 2] There are minor typographical inconsistencies: 'Kmeans++' should be 'K-means++', and the metric name appears variously as 'BleuRT', 'BLRT', and 'BLEURT'; please standardize.
  3. [References] Reference [19] (Ward's hierarchical clustering) appears in the bibliography but is not cited in the text; either cite it in the clustering discussion or remove it.
  4. [Section 3.3] The coherence metric is cited to a toolbox paper, but the exact sentence tokenizer, similarity model, and aggregation used to compute first- and second-order coherence are not specified; these implementation details affect reproducibility.
  5. [Section 4] No comparison to published long-document summarization systems is provided; even a qualitative positioning against results in [5] would help the reader calibrate the reported numbers.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the Markov ordering is derived from document-internal transition frequencies and evaluated against external human references, not from the target metrics.

full rationale

The derivation chain is self-contained. The only learned structure in the method is the Markov transition matrix T (Eq. 1), which is estimated from the document's own chunk-to-cluster sequence S and then used to select an ordering of the already-generated cluster summaries. This is a within-document heuristic, not a parameter fitted to the evaluation labels or to the final summary. The final summary is evaluated against BookSum human references with standard external metrics (ROUGE, BERTScore, coherence, BLEURT), and no pipeline component (chunk size, top_k=5, K-means++, DP Hamiltonian path, GPT-4o-mini) is tuned to optimize those metrics. There is also no load-bearing self-citation: the cited works are external methods and models, and the paper's own contribution is independently instantiated by the pipeline and the reported comparison. The limitations noted by the reader—the unspecified Cluster-Sum ordering, the unreported cluster count n with the DP solver restricted to n<22, and the BERTScore result that slightly favors the LLM-Full baseline—are methodological confounds or correctness risks, not cases where a prediction reduces by construction to its input. Therefore no circular step is present.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim rests on modest heuristics: chunk size, overlap, top-k, cluster count, and the Markov assumption. None of these are derived or ablated. The cluster count is the most fragile because it is unreported and directly bounds the solver.

free parameters (4)
  • chunk_size = 500 tokens
    Hand-chosen in Section 3.1; determines the granularity of embeddings and clusters. No sensitivity analysis is reported.
  • chunk_overlap = 20 tokens
    Hand-chosen to preserve continuity between chunks; not ablated.
  • top_k_vectors = 5
    Number of closest-to-centroid chunks summarized per cluster, chosen in Section 3.1 with no ablation or coverage analysis.
  • number_of_clusters = not reported
    The cluster count k defines the Markov states and the DP problem size. The Methods section omits how k is set, and the conclusion admits the n<22 constraint is unevaluated.
assumptions (5)
  • domain assumption First-order Markov assumption: transition probabilities between consecutive cluster IDs capture narrative flow.
    Equation (1) in Section 3.1 defines T on cluster ID transitions; the method assumes thematic progression is Markovian in cluster space.
  • domain assumption 500-token chunking with 20-token overlap preserves enough context for embedding and clustering.
    Section 3.1 states the overlap 'ensures continuity and coherence'; no ablation or analysis supports this.
  • domain assumption The five chunks nearest each centroid adequately represent the cluster.
    Section 3.1 selects top_k = 5 without measuring coverage or information loss.
  • domain assumption The number of clusters is small enough for the exact bitmask DP solver (n < 22).
    Section 3.1 says the DP handles up to 22 nodes; the conclusion admits this constraint is unevaluated.
  • domain assumption Semantic similarity of nomic embeddings groups chunks by theme.
    The clustering step relies on the embedding model's cosine-space geometry matching thematic structure, which is not verified on BookSum.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Markov-Enhanced Clustering for Long Document Summarization: Tackling the 'Lost in the Middle' Challenge with Large Language Models." pith.science (2026). https://pith.science/paper/LNNED3KW

@misc{pith2026250618036,
  author       = {Pith},
  title        = {Pith review of: Markov-Enhanced Clustering for Long Document Summarization: Tackling the 'Lost in the Middle' Challenge with Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LNNED3KW}},
  note         = {Machine review of arXiv:2506.18036}
}
read the original abstract

The rapid expansion of information from diverse sources has heightened the need for effective automatic text summarization, which condenses documents into shorter, coherent texts. Summarization methods generally fall into two categories: extractive, which selects key segments from the original text, and abstractive, which generates summaries by rephrasing the content coherently. Large language models have advanced the field of abstractive summarization, but they are resourceintensive and face significant challenges in retaining key information across lengthy documents, which we call being "lost in the middle". To address these issues, we propose a hybrid summarization approach that combines extractive and abstractive techniques. Our method splits the document into smaller text chunks, clusters their vector embeddings, generates a summary for each cluster that represents a key idea in the document, and constructs the final summary by relying on a Markov chain graph when selecting the semantic order of ideas.

Figures

Figures reproduced from arXiv: 2506.18036 by the authors.

Figure 1
Figure 1. Variability of Book Length in Booksum, Measured by Token Count [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Variability of Summary Length in the Dataset, Measured by Token [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Pipeline illustrating our proposed summarization method [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: illustrates an example graph with n = 3 clusters and the optimal Hamiltonian path 1 → 3 → 2. This example graph demonstrates how the tran￾sition matrix T is used to identify the most probable Hamiltonian path in a small-scale scenario. The brute force approach, which e…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Adversarial Stress Testing of Role-Playing Language Agents using Multi-Agent Evaluation

    cs.AI 2026-08 conditional novelty 5.0 of 10

    A multi-agent adversarial evaluation platform with six progressive attack strategies shows that role-playing LLMs degrade under sustained pressure, with automated judging correlating with human ratings.

Reference graph

Works this paper leans on

28 extracted references · 15 canonical work pages · cited by 1 Pith paper

  1. [1]

    Expert Systems with Applications165, 113679 (2020)

    El-Kassas, W., Salama, C., Rafea, A., Mohamed, H.: Automatic Text Summa- rization: A Comprehensive Survey. Expert Systems with Applications165, 113679 (2020). https://doi.org/10.1016/j.eswa.2020.113679

    arXiv 2020

  2. [2]

    Allahyari, M., Pouriyeh, S., Assefi, M., Safaei, S., Trippe, E., Gutiérrez, J., Kochut, K.: Text Summarization Techniques: A Brief Survey. Int. J. Adv. Comput. Sci. Appl. 8(7), 397–405 (2017).https://doi.org/10.14569/IJACSA.2017.081052

    work page arXiv 2017

  3. [3]

    In: Findings of the Association for Computational Linguistics (2022)

    Pang, B., Nijkamp, E., Kryscinski, W., Savarese, S., Zhou, Y., Xiong, C.: Long Document Summarization with Top-down and Bottom-up Inference. In: Findings of the Association for Computational Linguistics (2022). Available at:https://api. semanticscholar.org/CorpusID:247450875

    work page 2022

  4. [4]

    In: 2020 Fourth Int

    Rahul, Adhikari, S., Monika: NLP based Machine Learning Approaches for Text Summarization. In: 2020 Fourth Int. Conf. Comput. Methodol. Commun. (IC- CMC), pp. 535–538. IEEE (2020). https://doi.org/10.1109/ICCMC48092.2020. ICCMC-00099

    work page arXiv 2020

  5. [5]

    ACM Computing Surveys55, 1–35 (2022)

    Koh,H.Y.,Ju,J.,Liu,M.,Pan,S.:AnEmpiricalSurveyonLongDocumentSumma- rization: Datasets, Models, and Metrics. ACM Computing Surveys55, 1–35 (2022). Available at:https://api.semanticscholar.org/CorpusID:250118028 Markov-Enhanced Clustering for Long Document Summarization 11

    work page 2022

  6. [6]

    In: Proceedings of NAACL- HLT 2019, pp

    Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of Deep Bidi- rectional Transformers for Language Understanding. In: Proceedings of NAACL- HLT 2019, pp. 4171–4186. ACL, Minneapolis (2019).https://doi.org/10.18653/ v1/N19-1423

    work page 2019

  7. [7]

    ArXiv abs/1906.04165 (2019)

    Miller, D.: Leveraging BERT for Extractive Text Summarization on Lectures. ArXiv abs/1906.04165 (2019). Available at:https://api.semanticscholar.org/ CorpusID:184487094

    arXiv 2019

  8. [8]

    In: Advances in Neural Information Processing Systems, vol

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, ŁŁ., Polosukhin, I.: Attention is All You Need. In: Advances in Neural Information Processing Systems, vol. 30. Curran Associates (2017)

    work page 2017

Show all 28 references
  1. [9]

    Transactions of the ACL 12, 39–57 (2024).https://doi.org/10.1162/tacl_a_00632

    Zhang, T., Ladhak, F., Durmus, E., Liang, P., McKeown, K., Hashimoto, T.B.: Benchmarking Large Language Models for News Summarization. Transactions of the ACL 12, 39–57 (2024).https://doi.org/10.1162/tacl_a_00632

    2024 doi

  2. [10]

    In: Findings of the Association for Computational Linguistics: ACL 2023, pp

    Tam, D., Mascarenhas, A., Zhang, S., Kwan, S., Bansal, M., Raffel, C.: Evaluating the Factual Consistency of Large Language Models Through News Summarization. In: Findings of the Association for Computational Linguistics: ACL 2023, pp. 5220–

    2023

  3. [11]

    Transactions of the ACL 12, 157–173 (2024).https://doi.org/10.1162/tacl_a_00638

    Liu, N.F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., Liang, P.: Lost in the Middle: How Language Models Use Long Contexts. Transactions of the ACL 12, 157–173 (2024).https://doi.org/10.1162/tacl_a_00638

    2024 doi

  4. [12]

    CoRR abs/2105.08209 (2021)

    Kryscinski, W., Rajani, N.F., Agarwal, D., Xiong, C., Radev, D.R.: Book- Sum: A Collection of Datasets for Long-form Narrative Summarization. CoRR abs/2105.08209 (2021). Available at:https://arxiv.org/abs/2105.08209

    2021 arXiv

  5. [13]

    In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp

    Shaham, U., Segal, E., Ivgi, M., Efrat, A., Yoran, O., Haviv, A., Gupta, A., Xiong, W., Geva, M., Berant, J., Levy, O.: SCROLLS: Standardized CompaRi- son Over Long Language Sequences. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, ...

    2022

  6. [14]

    Meng, R., Thaker, K., Zhang, L., Dong, Y., Yuan, X., Wang, T., He, D.: Bring- ing Structure into Summaries: a Faceted Summarization Dataset for Long Scientific Documents. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th I...

    2021

  7. [15]

    arXiv preprint arXiv:2401.07883 (2024)

    Finardi, P., Avila, L., Castaldoni, R., Gengo, P., Larcher, C., Piau, M., Costa, P., Caridá, V.: The Chronicles of RAG: The Retriever, the Chunk and the Generator. arXiv preprint arXiv:2401.07883 (2024)

    2024 arXiv

  8. [16]

    IEEE Access11, 36120–36146 (2023).https: //doi.org/10.1109/ACCESS.2023.3266377

    Patil, R., Boit, S., Gudivada, V., Nandigam, J.: A Survey of Text Representation and Embedding Techniques in NLP. IEEE Access11, 36120–36146 (2023).https: //doi.org/10.1109/ACCESS.2023.3266377

    2023

  9. [17]

    arXiv preprint arXiv:2402.01613 (2024)

    Nussbaum, Z., Morris, J.X., Duderstadt, B., Mulyar, A.: Nomic Embed: Train- ing a Reproducible Long Context Text Embedder. arXiv preprint arXiv:2402.01613 (2024)

    2024 arXiv

  10. [18]

    In: Proc

    Arthur, D., Vassilvitskii, S.: K-Means++: The Advantages of Careful Seeding. In: Proc. of the Annu. ACM-SIAM Symp. on Discrete Algorithms, pp. 1027–1035 (2007). https://doi.org/10.1145/1283383.1283494 12 A. Amari and M. Ben Ammar

    2007

  11. [19]

    https://doi.org/10.1007/s00357-014-9161-z

    Murtagh, F., Legendre, P.: Ward’s Hierarchical Agglomerative Clustering Method: Which Algorithms Implement Ward’s Criterion? Journal of Classification 31(3), 274–295 (2014). https://doi.org/10.1007/s00357-014-9161-z

    2014 doi

  12. [20]

    Available at: https://openai.com/index/ gpt-4o-mini-advancing-cost-efficient-intelligence/ (2024)

    OpenAI: GPT-4o-Mini. Available at: https://openai.com/index/ gpt-4o-mini-advancing-cost-efficient-intelligence/ (2024)

    2024

  13. [21]

    In: Algo- rithms and Computation, LNCS vol

    Uehara, R., Uno, Y.: Efficient Algorithms for the Longest Path Problem. In: Algo- rithms and Computation, LNCS vol. 9999, pp. 871–883. Springer, Heidelberg (2005). https://doi.org/10.1007/978-3-540-30551-4_74

    2005 doi

  14. [22]

    SIGACT News 28(2), 40–52 (1997).https://doi.org/10.1145/261342.571216

    Hochba, D.S.: Approximation Algorithms for NP-Hard Problems. SIGACT News 28(2), 40–52 (1997).https://doi.org/10.1145/261342.571216

    1997

  15. [23]

    Keswani, G., Bisen, W., Padwad, H., Wankhedkar, Y., Pandey, S., Soni, A.: Abstractive Long Text Summarization using Large Language Models. Int. J. In- tell. Syst. Appl. Eng.12(12s), 160–168 (2024). Available at:https://ijisae.org/ index.php/IJISAE/article/view/4500

    2024

  16. [24]

    In: Text Summarization Branches Out, pp

    Lin, C.Y.: ROUGE: A Package for Automatic Evaluation of Summaries. In: Text Summarization Branches Out, pp. 74–81. Association for Computational Linguis- tics, Barcelona (2004). Available at:https://aclanthology.org/W04-1013

    2004

  17. [25]

    arXiv preprint arXiv:1904.09675 (2019)

    Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q., Artzi, Y.: BERTScore: Evalu- ating Text Generation with BERT. arXiv preprint arXiv:1904.09675 (2019)

    2019 arXiv

  18. [26]

    In: Proceedings of the 29th International Conference on Computational Linguistics, pp

    Steen, J., Markert, K.: How to Find Strong Summary Coherence Measures? A Toolbox and a Comparative Study for Summary Coherence Measure Evaluation. In: Proceedings of the 29th International Conference on Computational Linguistics, pp. 6035–6049. International Committee on Compu...

    2022

  19. [27]

    In: Proceedings of the 58th Annual Meeting of the Association for Com- putational Linguistics, pp

    Sellam, T., Das, D., Parikh, A.: BLEURT: Learning Robust Metrics for Text Gen- eration. In: Proceedings of the 58th Annual Meeting of the Association for Com- putational Linguistics, pp. 7881–7892. ACL, Online (2020).https://doi.org/10. 18653/v1/2020.acl-main.704

    2020

  20. [5255]

    ACL, Toronto (2023).https://doi.org/10.18653/v1/2023.findings-acl. 322

    2023 doi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.