Pith. sign in

REVIEW 1 major objections 6 minor 54 references

LLM summarizers should be benchmarked on how consistently they reproduce summaries, not just on a single output's quality.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 08:41 UTC pith:6U47AVX3

load-bearing objection The empirical run-to-run variability finding is credible, but the proposed stability coefficient is a range statistic that ranks Llama more stable than Gemma3 on AlignScore while the paper's own CV/Gini and text say otherwise — the trustworthiness proxy doesn't hold as stated. the 1 major comments →

arxiv 2607.21010 v1 pith:6U47AVX3 submitted 2026-07-23 cs.AI

Reexamining zero-shot summarization: Empirical investigation of trustworthiness of LLM-summarizers

Vasudha Bhatnagar , Purnima Bindal , Vikas Kumar , Raj Kumari Bahl This is my paper
classification cs.AI
keywords LLM summarizationzero-shot summarizationstabilitytrustworthinessLLM benchmarkinggeneration variabilitysemantic consistencyAlignScore
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that judging an LLM summarizer from one generated summary per document is incomplete, because repeated runs under identical settings produce substantially different summaries. It proposes treating stability — the consistency of scores across repeated generations — as a proxy for trustworthiness, and introduces a two-level Stability Analysis Protocol that computes a document-level stability coefficient and a corpus-level stability index. Testing three small open LLMs on news, medical, and legal documents, the authors find statistically significant differences in run-to-run variability across models and across evaluation metrics. A sympathetic reader should care because LLM summaries are increasingly used in education, law, and medicine, where inconsistent outputs can mislead. The paper's central recommendation is that benchmarking and evaluation should report stability alongside quality, with confidence intervals.

Core claim

In the paper's own terms, the discovery is that generation-level variability in zero-shot LLM summarization is substantial, measurable, and metric-dependent, and that this variability can be quantified and used to rank summarizers. The authors define the stability coefficient for a document as one minus the maximum pairwise absolute difference of evaluation scores across k repetitions, then define the summarizer's stability index as the minimum such coefficient across a stratified sample of documents. Applied to 32 documents per genre with 100 summaries each per model, the protocol shows BERTScore to be the most stable metric, AlignScore the least, and Gemma3 the most consistently stable of

What carries the argument

The Stability Analysis Protocol (SAP): a two-level diagnostic that first characterizes the distribution of evaluation scores (BERTScore, AlignScore, Semantic Consistency) over k summaries of a single document using dispersion measures, the Gini coefficient, and the stability coefficient ε = 1 − max pairwise score difference, then consolidates document-level coefficients into a corpus-level stability index (the minimum over documents) along with bootstrap confidence intervals, Levene's test for variance homogeneity, and Friedman–Nemenyi tests for model ranking. The load-bearing move is measuring variability of scores and treating that variability as a direct indicator of the summarizer's trus

Load-bearing premise

The paper assumes that the difference in evaluation scores between two summaries reflects genuine variation in their content, and that stability of scores — not externally validated user trust — is a valid proxy for trustworthiness.

What would settle it

Give a degenerate summarizer that always emits the same generic low-quality summary for every document; if its stability index ranks it as trustworthy under the protocol, the equivalence between stability and trustworthiness fails. Likewise, a user study showing that readers trust a variable but high-quality summarizer more than a stable but low-quality one would contradict the proxy.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Single-summary evaluation can mislead: reporting a stability coefficient and confidence interval alongside quality scores gives users a realistic range of expected output quality.
  • Benchmarking practice should move to multi-run, distribution-based evaluation; the paper's protocol produces reproducible, statistically grounded comparisons of summarizers.
  • Not all evaluation metrics behave alike — BERTScore was stable across models and genres, while AlignScore varied widely, so conclusions about factual faithfulness from one run are especially fragile.
  • For consumers, the stability index offers a concrete number to compare how much variation to expect from a summarizer on a given document genre.
  • The empirical ranking (Gemma3 most stable; Llama least, especially on factual alignment) changes with document genre, so stability should be reported per genre, not globally.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Stability is blunt as a trust proxy: a model that always outputs the same poor summary scores as highly stable, so the index is best read together with quality averages rather than alone.
  • The protocol's cost — 100 generations per document — could be reduced; testing how few runs are needed for a reliable stability estimate would make it practical for large-scale benchmarks.
  • The large AlignScore instability may reflect properties of the metric as much as the models; re-running the protocol with other faithfulness metrics would show whether the finding is about factual variation or about AlignScore itself.
  • For downstream tasks that consume summaries as inputs, run-to-run variability can cascade; a stability index could serve as a cheap pre-filter before a summary is used in a pipeline.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 6 minor

Summary. The paper argues that stochastic LLM decoding makes single-summary evaluation of zero-shot summarizers incomplete, and proposes a two-level Stability Analysis Protocol (SAP): at document level, a stability coefficient ε = 1 − max pairwise absolute metric-score difference (Def. 1, Sec. 2.1); at corpus level, a stability index defined as the minimum of document-level ε over a sample (Def. 2, Sec. 2.2). This index is proposed as a proxy for the trustworthiness of the summarizer. The authors run k = 100 generations per document over 32 sampled documents from CNN/DM, PubMed, and IN-Abs, for three LLMs (Gemma3-4B, Llama-3.2-3B, Qwen-2.5-3B), scoring summaries with BERTScore, AlignScore, and a pairwise Semantic Consistency (SC) score (Eq. 1). They report substantial metric-level and model-level variability, claim BERTScore is more stable than AlignScore, and conclude Gemma3 is the most stable/trustworthy summarizer.

Significance. The empirical finding that repeated generations of the same document under static prompts yield non-trivial score variability is credible and useful; the size of the experiment (≈28.8K summaries) is a genuine strength. If the proposed stability index were validated, SAP could complement conventional single-run benchmarks and provide a standardized way to report run-to-run variability. However, the central trustworthiness claim is currently not established: the proposed ε is a range statistic that can reward consistently poor outputs, it is aggregated by a min that is vulnerable to single outliers, and no external validation links the index to user trust, downstream reliability, or even the paper's own direct semantic-consistency measure. These issues affect the main conclusion, not just the presentation.

major comments (1)
  1. [Sec. 6.2, Table 10] The normality statement is inconsistent with the data. The text says 'the null hypotheses are rejected for large number of documents ... for all the score distributions and genres,' but Table 10 shows that for IN-Abs AlignScore, only 6/32 documents reject normality for every model, i.e., the majority of documents do not reject. Please rephrase to reflect the actual counts, or explain why the IN-Abs AlignScore distributions are treated as non-normal in the subsequent analysis.
minor comments (6)
  1. [Sec. 2.1] Typo: 'We define the the stability coefficient' should be 'the stability coefficient.'
  2. [Sec. 7.1] Typo: 'Llamma' should be 'Llama' in the summarizer-wise consolidation.
  3. [Sec. 6.1, Fig. 5] The density plots are informative, but adding the numerical ε values on the same figure or in the caption would make the contradiction between ε and CV/GC easier for readers to see.
  4. [Sec. 5.1, Table 3] The initial chi-square test for IN-Abs is significant, and only after removing outliers is it non-significant. This should be stated more explicitly in the main text, not only in Appendix B, because the sample representativeness claim relies on that post-hoc exclusion.
  5. [Sec. 2.2] The sentence 'Statistically large sample of documents (N) and summaries (k), may divulges the worst possible variations' contains an agreement error and is unclear; please rewrite.
  6. [Sec. 4.2] The derivation of the combined coefficient of variation is placed in Appendix A; consider a one-sentence intuition in the main text for readers who do not need the derivation.

Circularity Check

0 steps flagged

No significant circularity; stability statistics are direct empirical measurements, and the trustworthiness label is an explicitly stated proxy rather than a derived prediction.

full rationale

The paper's derivation chain is data-driven, not self-referential. Document-level stability (Def. 1, Sec. 2.1) and corpus-level stability (Def. 2, Sec. 2.2) are computed directly from observed metric scores over k generated summaries; no parameter is fitted to a target and then reported as a prediction. The supplementary statistics (CV, Gini, SC, bootstrap CIs) are also direct descriptive/confirmatory measurements. The empirical conclusion that Gemma3 is more stable than Llama and Qwen on BERTScore/SC and often AlignScore follows from Tables 4-9 and is open to falsification on the reported sample. The only conceptual shortcut is the paper's explicit operationalization of trustworthiness as stability: 'Lower value of ঱‰ indicates that the LLM-summarizer produces summaries that are not stable, and hence is not trustworthy' (Sec. 4.2) and 'stability index ... is the proxy for its trustworthiness' (Abstract). This is an unvalidated assumption/limitation rather than a circular derivation, and the paper itself acknowledges it in Limitations (iii): 'incorporating expert or user evaluations would complement the proposed protocol by validating the measured stability.' The stability measurements stand independently and would retain their empirical content even if the trustworthiness label were removed. There are no load-bearing self-citations or imported uniqueness/ansatz results.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 2 invented entities

The paper introduces no fitted parameters and no mathematical derivation beyond standard statistics. Its load-bearing premises are domain assumptions: that score differences reflect content variation, that stability implies trustworthiness, that the chosen automatic metrics are valid, and that the samples are representative. The stability measures are new constructs without external validation.

axioms (5)
  • domain assumption The difference in metric scores of two text documents reflects variation in content.
    Sec 2.1 states this explicitly. It is load-bearing because score ranges are interpreted as instability.
  • domain assumption Stability (consistency of scores) is a valid proxy for robustness and trustworthiness.
    Sec 2 and Sec 4 define trustworthiness through stability; no external validation against human judgments or downstream performance is given.
  • domain assumption BERTScore and AlignScore validly measure semantic alignment and factual consistency.
    Sec 3.2 relies on the published metrics without calibrating them against human labels in this study.
  • domain assumption Static default parameters ensure observed variations arise solely from stochasticity.
    Sec 3.1 claims this, but the exact decoding parameters are not reported, so unstated configuration choices could contribute to variation.
  • domain assumption A stratified 32-document sample is representative of each corpus.
    Sec 5.1 shows IN-Abs variance homogeneity required post-hoc outlier removal, weakening the general claim of representativeness.
invented entities (2)
  • Stability coefficient epsilon no independent evidence
    purpose: Document-level measure of summarizer stability: 1 minus the maximum pairwise absolute difference in metric scores.
    Newly introduced statistic; no external calibration demonstrates that it tracks user trust or downstream reliability.
  • Corpus-level stability index no independent evidence
    purpose: Summarizer-level stability measure: the minimum of document-level stability coefficients over a sample.
    New benchmark statistic; a single worst-case document dominates the value, and its validity as a trust metric is asserted, not tested.

pith-pipeline@v1.3.0-alltime-deepseek · 23241 in / 11952 out tokens · 132512 ms · 2026-08-01T08:41:41.098275+00:00 · methodology

0 comments
read the original abstract

Zero-shot summarization using Large Language Models (LLMs) has significantly advanced the abstractive summarization task by producing coherent and fluent summaries. However, underlying stochasticity of the large language models raises concerns about the stability and trustworthiness of the LLM-generated summaries. This issue has become increasingly important due to proliferation of LLM-generated summaries in educational settings, where students and researchers summarize complex academic materials in zero-shot manner. We propose a novel two-level diagnostic protocol for benchmarking LLM-summarizers based on the stability of the generated summaries. At the lower level, document-level stability analysis is performed over multiple LLM-summaries generated under controlled environment, and the stability coefficient is computed. Each generated summary is scored for semantic and factual alignment with the original document, enabling estimation of stability along more than one dimensions. At the next level, observations from a stratified sample of documents drawn from the corpus are consolidated to estimate the stability index of the LLM-summarizer, which is the proxy for its trustworthiness. Our empirical investigation of three LLM-summarizers across three genres of documents reveals statistically significant differences in the generation-level variability among LLMs across summary evaluation metrics. This study advances the LLM-summarization research by evidential recognition of the stability problem in LLM-summaries and motivates further research towards development of robust, reliable and trustworthy LLM-summarizers.

Figures

Figures reproduced from arXiv: 2607.21010 by Purnima Bindal, Raj Kumari Bahl, Vasudha Bhatnagar, Vikas Kumar.

Figure 1
Figure 1. Figure 1: Distribution of BERTScore values of 100 summaries generated by Llama-summarizer for a document from CNN/DM dataset. ranging from [0.41, 0.71]. These variations are too significant to be ignored, and raise concerns about repeatability and reproducibility of LLM-generated summaries. Furthermore, this observation motivates the conjecture that the assessment of an LLM-summarizer based on single generated summa… view at source ↗
Figure 2
Figure 2. Figure 2: Prompt for LLM-summarization. a representative sample of documents from a corpus, and ensuring statistical validity of the estimate is reasonable alternative. Statistically large sample of documents (𝑁) and summaries (𝑘), may divulges the worst possible variations in summaries generated by . We drop 𝑚 and  from the notation for brevity, as it will be clear by the context in the rest of the paper. Additio… view at source ↗
Figure 3
Figure 3. Figure 3: Pipeline for estimating stability coefficient of  for a document 𝐷 and a quality metric 𝑚. We consider two dimensions for veracity. First, semantic alignment to capture contextual similarity between each generated summary and the gold standard reference summary as assessed using BERTScore [49]. Difference in the BERTScore of the two summaries informs semantic variation between them. The other dimension is… view at source ↗
Figure 4
Figure 4. Figure 4: Two stages of the proposed Stability Analysis Protocol. descriptive measures. We further estimate the stability coefficient  for summarizer  and compute the confidence interval for the variability observed in its generated summaries. At both levels, we use a judicious mix of visual techniques for preliminary exploration, descriptive statistics and statistical tests as confirmatory tools. The proposed pro… view at source ↗
Figure 5
Figure 5. Figure 5: Density plots of 100 summaries scores generated by three LLMs for the selected CNN/DM article. Visual Analysis [PITH_FULL_IMAGE:figures/full_fig_p011_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Ridgeline plots of the three score distributions of Gemma3, Llama, and Qwen generated summaries for the CNN/DM sample dataset of 32 documents. (BERTScore and SC Score) and in consonance with the density plots. Lower values of both the coefficient of variation (CV) and Gini coefficient (GC) confirm better consistency of scores for Gemma3 summaries for all three metrics. The least IQR and standard deviation … view at source ↗
Figure 7
Figure 7. Figure 7: (a) 𝐶𝑉𝑐 values for the three metrics and LLM-summarizers, (b) boxplots of document-level GC values for the CNN/DM sample dataset. Visual Analysis [PITH_FULL_IMAGE:figures/full_fig_p013_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Bootstrap confidence intervals for the mean document-level variance of BERTScore (× 10−2) and AlignScore (× 10−2) across the CNN/DM, PubMed, and IN-Abs datasets, grouped by model (Gemma3, Llama, and Qwen). Dots represent the bootstrap point estimate of the average document-level variance, and error bars denote the estimated 95% confidence interval. GC values for AlignScore show the highest fluctuation, whi… view at source ↗
Figure 9
Figure 9. Figure 9: Rank heatmaps for the three LLMs (Gemma, Llama, Qwen), and three metrics for CNN/DM documents (𝐷1–𝐷32). In each plot, the three blocks represent average, standard deviation, and Gini coefficient, respectively. Ranks 1, 2, and 3 are shown in white, gray and black respectively. The numbers at the top denote wins. genres, stability coefficients for PubMed and IN-Abs (long documents) are higher compared to CNN… view at source ↗
Figure 10
Figure 10. Figure 10: Critical difference (CD) diagrams showing the average ranks of LLM-summarizers based on BERTScore, AlignScore, and Semantic Consistency (SC) Score for the CNN/DM (news articles) sample set. The horizontal line denotes the critical difference (CD). Models connected by this line do not differ significantly according to the Nemenyi post-hoc test following the Friedman test. to the other two summarizers [PIT… view at source ↗
Figure 11
Figure 11. Figure 11: Distribution of document lengths (word counts) in the IN-Abs dataset, comparing the full dataset (7, 128) with the selected sample (32) set. Darker bars denote documents within the IQR threshold (1.5), while lighter bars indicate identified outliers. The red dashed and black dotted vertical lines represent the mean and median, respectively, computed from the outlier-filtered data. A chi-square test applie… view at source ↗
Figure 12
Figure 12. Figure 12: Ridgeline plots of evaluation score distributions for the 32 documents (𝐷1 -𝐷32) from PubMed and IN-Abs datasets. Each ridgeline represents the distribution of scores of hundred summaries for a document. Distributions of Gemma3, Llama, and Qwen are juxtaposed for comparison across each metric. Appendix D. Benchmarking LLM Summarizers This section presents results for the PubMed and IN-Abs sample sets, com… view at source ↗
Figure 13
Figure 13. Figure 13: Combined coefficients of variation for the three metrics for (a) PubMed and (b) IN-Abs datasets [PITH_FULL_IMAGE:figures/full_fig_p023_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Box plots of 32 document-level Gini Coefficients of metric scores for (a) PubMed and (b) IN-Abs datasets [PITH_FULL_IMAGE:figures/full_fig_p024_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Document-wise rank heatmaps for the three LLMs for the PubMed and IN-Abs summaries. For each document (𝐷1–𝐷32), models are ranked on Average score (left), Standard Deviation (middle), and Gini Coefficient (right). Rank 1 denotes the best-performing model (white), Rank 2 the intermediate (gray), and Rank 3 the worst￾performing model (black). The numbers in parentheses above each column—G(•), L(•), and Q(•)… view at source ↗
Figure 16
Figure 16. Figure 16: Critical difference (CD) diagrams showing the average ranks of LLM-summarizers based on BERTScore, Semantic Consistency (SC) Score, and AlignScore for the three datasets: (a) PubMed (medical documents), and (b) IN-Abs (legal judgments). The horizontal line denotes the critical difference (CD). Models connected by this line do not differ significantly according to the Nemenyi post-hoc test following the Fr… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

54 extracted references · 7 linked inside Pith

  1. [1]

    A comprehensive survey on legal summarization: Challenges and future directions

    Akter, M., Çano, E., Weber, E., Dobler, D., Habernal, I., 2025. A comprehensive survey on legal summarization: Challenges and future directions. ACM Computing Surveys

    2025

  2. [2]

    Large language models robustness against perturbation: S

    Alahmari, S.S., Hall, L., Mouton, P.R., Goldgof, D., 2025. Large language models robustness against perturbation: S. alahmari et al. Scientific Reports

    2025

  3. [3]

    Using llm (large language model) to improve efficiency in literature review for undergraduate research

    Antu, S.A., Chen, H., Richards, C.K., 2023. Using llm (large language model) to improve efficiency in literature review for undergraduate research. Llm@ Aied , 8–16

    2023

  4. [4]

    Robust tests for the equality of variances

    Brown, M.B., Forsythe, A.B., 1974. Robust tests for the equality of variances. Journal of the American statistical association 69, 364–367

    1974

  5. [5]

    Efficient inference for noisy llm-as-a-judge evaluation

    Chen, Y .T., Lu, S., Li, S., Guo, M., Li, S., 2026. Efficient inference for noisy llm-as-a-judge evaluation. arXiv preprint arXiv:2601.05420

    arXiv 2026

  6. [6]

    Cohan, A., Dernoncourt, F., Kim, D.S., Bui, T., Kim, S., Chang, W., Goharian, N., 2018. A discourse-aware attention model for abstractive summarization of long documents, in: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, V olume 2 (Short Papers), pp. 615–621

    2018

  7. [7]

    Statistical comparisons of classifiers over multiple data sets

    Demšar, J., 2006. Statistical comparisons of classifiers over multiple data sets. Journal of Machine learning research 7, 1–30

    2006

  8. [8]

    Applicability of large language models and generative models for legal case judgement summarization

    Deroy, A., Ghosh, K., Ghosh, S., 2024. Applicability of large language models and generative models for legal case judgement summarization. Artificial Intelligence and Law , 1–44

    2024

  9. [9]

    Explainability meets text summarization: A survey, in: Proceedings of the 17th International Natural Language Generation Conference, pp

    Dhaini, M., Erdogan, E., Bakshi, S., Kasneci, G., 2024. Explainability meets text summarization: A survey, in: Proceedings of the 17th International Natural Language Generation Conference, pp. 631–645

    2024

  10. [10]

    Consistency evaluation of news article summaries generated by large (and small) language models

    Gilhuly, C., Shahzad, H., 2025. Consistency evaluation of news article summaries generated by large (and small) language models. arXiv preprint arXiv:2502.20647

    Pith/arXiv arXiv 2025

  11. [11]

    Gupta, A., Singh, D., Cowan, G.A., Kadhiresan, N., Srivastava, S., Sriraja, Y ., Mantri, Y .K., 2025. Autosumm: A comprehensive framework for llm-based conversation summarization, in: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (V olume 6: Industry Track), pp. 500–509

    2025

  12. [12]

    Hakim, Y ., Bhardwaj, S., Mulla, R., 2024. Case summarization using llm model for supreme court judgments of an indian judiciary system, in: International Conference on Information and Communication Technology for Competitive Strategies, Springer. pp. 79–89

    2024

  13. [13]

    Trustllm: Trustworthiness in large language models

    Huang, Y ., Sun, L., Wang, H., Wu, S., Zhang, Q., Li, Y ., Gao, C., Huang, Y ., Lyu, W., Zhang, Y ., et al., 2024. Trustllm: Trustworthiness in large language models. arXiv preprint arXiv:2401.05561

    Pith/arXiv arXiv 2024

  14. [14]

    Huidrom, R., Belz, A., 2025. Using llm judgements for sanity checking results and reproducibility of human evaluations in nlp, in: Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM2), pp. 354–365

    2025

  15. [15]

    Consistency analysis of chatgpt, in: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp

    Jang, M., Lukasiewicz, T., 2023. Consistency analysis of chatgpt, in: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 15970–15985

    2023

  16. [16]

    Context-Aware Sports Highlight Generation Leveraging Large Language Models

    Kang, J., 2024. Context-Aware Sports Highlight Generation Leveraging Large Language Models. Ph.D. thesis. Ulsan National Institute of Science and Technology

    2024

  17. [17]

    Keya, F., Jaradeh, M.Y ., Auer, S., 2024. Leveraging llms for scientific abstract summarization: Unearthing the essence of research in a single sentence, in: Proceedings of the 24th ACM/IEEE Joint Conference on Digital Libraries, pp. 1–7

    2024

  18. [18]

    Llms cannot reliably judge (yet?): A comprehensive assessment on the robustness of llm-as-a-judge

    Li, S., Xu, C., Wang, J., Gong, X., Chen, C., Zhang, J., Wang, J., Lam, K.Y ., Ji, S., 2025. Llms cannot reliably judge (yet?): A comprehensive assessment on the robustness of llm-as-a-judge. arXiv preprint arXiv:2506.09443

    arXiv 2025

  19. [19]

    Do not abstain! identify and solve the uncertainty, in: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), pp

    Liu, J., JingquanPeng, J., Wu, X., Li, X., Ge, T., Zheng, B., Liu, Y ., 2025. Do not abstain! identify and solve the uncertainty, in: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), pp. 17177–17197

    2025

  20. [20]

    Sumsurvey: An abstractive dataset of scientific survey papers for long document summarization, in: Findings of the ACL 2024, pp

    Liu, R., Liu, M., Yu, M., Zhang, H., Jiang, J., Li, G., Huang, W., 2024a. Sumsurvey: An abstractive dataset of scientific survey papers for long document summarization, in: Findings of the ACL 2024, pp. 9632–9651

    2024

  21. [21]

    Low-resource court judgment summarization for common law systems

    Liu, S., Cao, J., Li, Y ., Yang, R., Wen, Z., 2024b. Low-resource court judgment summarization for common law systems. Information Processing & Management 61, 103796

  22. [22]

    Lupart, S., van Dijk, D., Langezaal, E., van Dort, I., Aliannejadi, M., 2025. Investigating llm variability in personalized conversational information retrieval, in: Proceedings of the 2025 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region, pp. 353–363

    2025

  23. [23]

    Tools in the loop: Quantifying uncertainty of llm question answering systems that use tools

    Lymperopoulos, P., Sarathy, V ., 2025. Tools in the loop: Quantifying uncertainty of llm question answering systems that use tools. arXiv preprint arXiv:2505.16113

    Pith/arXiv arXiv 2025

  24. [24]

    Effectiveness in retrieving legal precedents: exploring text summarization and cutting-edge language models toward a cost-efficient approach

    Mentzingen, H., António, N., Bacao, F., 2025. Effectiveness in retrieving legal precedents: exploring text summarization and cutting-edge language models toward a cost-efficient approach. Artificial Intelligence and Law , 1–21

    2025

  25. [25]

    State of what art? a call for multi-prompt llm evaluation

    Mizrahi, M., Kaplan, G., Malkin, D., Dror, R., Shahaf, D., Stanovsky, G., 2024. State of what art? a call for multi-prompt llm evaluation. Transactions of the Association for Computational Linguistics 12, 933–949

    2024

  26. [26]

    Abstractive text summarization using sequence-to-sequence rnns and beyond, in: Proceedings of the 20th SIGNLL conference on computational natural language learning, pp

    Nallapati, R., Zhou, B., Dos Santos, C., Gulçehre, Ç., Xiang, B., 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond, in: Proceedings of the 20th SIGNLL conference on computational natural language learning, pp. 280–290

    2016

  27. [27]

    Evaluating variance in visual question answering benchmarks

    Nikitha, S., 2025. Evaluating variance in visual question answering benchmarks. arXiv preprint arXiv:2508.02645

    Pith/arXiv arXiv 2025

  28. [28]

    Oved, N., Levy, R., 2021. Pass: Perturb-and-select summarizer for product reviews, in: Proceedings of the 59th Annual Meeting of the ACL and the 11th International Joint Conference on Natural Language Processing (V olume 1: Long Papers), pp. 351–365

    2021

  29. [29]

    Efficient multi-prompt evaluation of llms, in: Proceedings of the 38th International Conference on Neural Information Processing Systems, pp

    Polo, F.M., Xu, R., Weber, L., Silva, M., Bhardwaj, O., Choshen, L., de Oliveira, A.F.M., Sun, Y ., Yurochkin, M., 2024. Efficient multi-prompt evaluation of llms, in: Proceedings of the 38th International Conference on Neural Information Processing Systems, pp. 22483–22512

    2024

  30. [30]

    Premnath, P., Yenumulapalli, V .O., Mohankumar, P., Sivanaiah, R., et al., 2025. Techssn at semeval-2025 task 10: A comparative analysis of transformer models for dominant narrative-based news summarization, in: Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025), pp. 2205–2212

    2025

  31. [31]

    Leveraging large language models on the traditional scientific writing workflow, in: 2024 Conference on AI, Science, Engineering, and Technology (AIxSET), IEEE

    Procko, T.T., Davidoff, A., Elvira, T., Ochoa, O., 2024. Leveraging large language models on the traditional scientific writing workflow, in: 2024 Conference on AI, Science, Engineering, and Technology (AIxSET), IEEE. pp. 154–161. Page 27 of 28 Trustworthiness of LLM-summarizers

    2024

  32. [32]

    Ramprasad, S., Ferracane, E., Lipton, Z.C., 2024. Analyzing llm behavior in dialogue summarization: Unveiling circumstantial hallucination trends, in: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), pp. 12549– 12561

    2024

  33. [33]

    How resilient are language models to text perturbations?, in: International Conference on Intelligent Data Engineering and Automated Learning, Springer

    Romero-Alvarado, D., Hernández-Orallo, J., Martínez-Plumed, F., 2024. How resilient are language models to text perturbations?, in: International Conference on Intelligent Data Engineering and Automated Learning, Springer. pp. 85–96

    2024

  34. [34]

    Can you trust llm judgments? reliability of llm-as-a-judge

    Schroeder, K., Wood-Doughty, Z., 2024. Can you trust llm judgments? reliability of llm-as-a-judge. arXiv preprint arXiv:2412.12509

    Pith/arXiv arXiv 2024

  35. [35]

    A coin flip for safety: Llm judges fail to reliably measure adversarial robustness

    Schwinn, L., Ladenburger, M., Beyer, T., Mofakhami, M., Gidel, G., Günnemann, S., 2026. A coin flip for safety: Llm judges fail to reliably measure adversarial robustness. arXiv preprint arXiv:2603.06594

    arXiv 2026

  36. [36]

    Quantifying Language Models’ Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting

    Sclar, M., Choi, Y ., Tsvetkov, Y ., Suhr, A., 2023. Quantifying Language Models’ Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting. arXiv preprint arXiv:2310.11324

    Pith/arXiv arXiv 2023

  37. [37]

    Shukla, A., Bhattacharya, P., Poddar, S., Mukherjee, R., Ghosh, K., Goyal, P., Ghosh, S., 2022. Legal case document summarization: Extractive and abstractive methods and their evaluation, in: Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural...

    2022

  38. [38]

    Robustness of large language models to perturbations in text

    Singh, A., Singh, N., Vatsal, S., 2024. Robustness of large language models to perturbations in text. arXiv preprint arXiv:2407.08989

    arXiv 2024

  39. [39]

    Legal text summarization via judicial syllogism with large language models

    Song, Y ., Qin, Y ., Huang, R., Chen, Y ., Lin, C., 2025. Legal text summarization via judicial syllogism with large language models. Journal of King Saud University Computer and Information Sciences 37, 111

    2025

  40. [40]

    Artificial intelligence risk management framework (ai rmf 1.0).(2023)

    Tabassi, E., 2023. Artificial intelligence risk management framework (ai rmf 1.0).(2023). URL: http://nvlpubs. nist. gov/nistpubs/ai/NIST. AI , 100–1

    2023

  41. [41]

    Evaluating the factual consistency of large language models through news summarization, in: Findings of the Association for Computational Linguistics: ACL 2023, pp

    Tam, D., Mascarenhas, A., Zhang, S., Kwan, S., Bansal, M., Raffel, C., 2023. Evaluating the factual consistency of large language models through news summarization, in: Findings of the Association for Computational Linguistics: ACL 2023, pp. 5220–5255

    2023

  42. [42]

    Van Schaik, T.A., Pugh, B., 2024. A field guide to automatic evaluation of llm-generated summaries, in: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2832–2836

    2024

  43. [43]

    Prompt engineering in consistency and reliability with the evidence-based guideline for llms

    Wang, L., Chen, X., Deng, X., Wen, H., You, M., Liu, W., Li, Q., Li, J., 2024. Prompt engineering in consistency and reliability with the evidence-based guideline for llms. NPJ digital medicine 7, 41

    2024

  44. [44]

    Using llm-supported lecture summarization system to improve knowledge recall and student satisfaction

    Xie, T., Kuang, Y ., Tang, Y ., Liao, J., Yang, Y ., 2025. Using llm-supported lecture summarization system to improve knowledge recall and student satisfaction. Expert Systems with Applications 269, 126371

    2025

  45. [45]

    Comparisons of various types of normality tests

    Yap, B.W., Sim, C.H., 2011. Comparisons of various types of normality tests. Journal of Statistical Computation and Simulation 81, 2141–2155

    2011

  46. [46]

    Event-based evaluation of abstractive news summarization, in: Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM2), pp

    You, H., Touileb, S., Øvrelid, L., Velldal, E., 2025. Event-based evaluation of abstractive news summarization, in: Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM2), pp. 504–510

    2025

  47. [47]

    Alignscore: Evaluating factual consistency with a unified alignment function, in: Proceedings of the 61st Annual Meeting of the ACL (V olume 1: Long Papers), pp

    Zha, Y ., Yang, Y ., Li, R., Hu, Z., 2023. Alignscore: Evaluating factual consistency with a unified alignment function, in: Proceedings of the 61st Annual Meeting of the ACL (V olume 1: Long Papers), pp. 11328–11348

    2023

  48. [48]

    A systematic survey of text summarization: From statistical methods to large language models

    Zhang, H., Yu, P.S., Zhang, J., 2025. A systematic survey of text summarization: From statistical methods to large language models. ACM Computing Surveys 57, 1–41

    2025

  49. [49]

    Bertscore: Evaluating text generation with bert

    Zhang, T., Kishore, V ., Wu, F., Weinberger, K.Q., Artzi, Y ., 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675

    Pith/arXiv arXiv 2019

  50. [50]

    Benchmarking large language models for news summarization

    Zhang, T., Ladhak, F., Durmus, E., Liang, P., McKeown, K., Hashimoto, T.B., 2024a. Benchmarking large language models for news summarization. Transactions of the Association for Computational Linguistics 12, 39–57

  51. [51]

    Trustworthy evaluation of large language models

    Zhang, X.Y ., Ye, H.J., Zhan, D.C., 2026a. Trustworthy evaluation of large language models. Frontiers of Computer Science 20, 1–3

  52. [52]

    A comprehensive survey on process-oriented automatic text summarization with exploration of llm-based methods

    Zhang, Y ., Jin, H., Meng, D., Wang, J., Tan, J., 2024b. A comprehensive survey on process-oriented automatic text summarization with exploration of llm-based methods. arXiv preprint arXiv:2403.02901

    arXiv

  53. [53]

    Assessing the accuracy of artificial intelligence-generated clinical summaries from ambulatory glaucoma subspecialty clinical encounters

    Zhang, Y ., Shi, M., Chung, I.Y ., Liebman, D.L., Barna, L.E., Pasquale, L.R., Friedman, D.S., Boland, M.V ., Shen, L.Q., Wang, M., 2026b. Assessing the accuracy of artificial intelligence-generated clinical summaries from ambulatory glaucoma subspecialty clinical encounters. Translational Vision Science & Technology 15, 22–22

  54. [54]

    Zhong, Y ., Litman, D., 2025. From information to insight: Leveraging llms for open aspect-based educational summarization, in: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), pp. 1914–1947. Page 28 of 28

    2025