Pith. sign in

REVIEW 18 cited by

Summarization is (Almost) Dead

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2309.09558 v1 pith:3LIV44L2 submitted 2023-09-18 cs.CL

Xiao Pu , Mingqi Gao , Xiaojun Wan This is my paper · ORCID
classification cs.CL
keywords summariesllmssummarizationdatasetsevaluationhumanllm-generatedmodels
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

How well can large language models (LLMs) generate summaries? We develop new datasets and conduct human evaluation experiments to evaluate the zero-shot generation capability of LLMs across five distinct summarization tasks. Our findings indicate a clear preference among human evaluators for LLM-generated summaries over human-written summaries and summaries generated by fine-tuned models. Specifically, LLM-generated summaries exhibit better factual consistency and fewer instances of extrinsic hallucinations. Due to the satisfactory performance of LLMs in summarization tasks (even surpassing the benchmark of reference summaries), we believe that most conventional works in the field of text summarization are no longer necessary in the era of LLMs. However, we recognize that there are still some directions worth exploring, such as the creation of novel datasets with higher quality and more reliable evaluation methods.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 18 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. VA-Blueprint: Uncovering Building Blocks for Visual Analytics System Design

    cs.HC 2025-08 conditional novelty 6.0 of 10

    A semi-automated methodology and public knowledge base catalog the building blocks of 101 urban visual analytics systems, using GPT-4 for extraction with human-in-the-loop correction and expert validation.

  2. From Alerts to Intelligence: A Novel LLM-Aided Framework for Host-based Intrusion Detection

    cs.CR 2025-07 conditional novelty 6.0 of 10

    SHIELD, an LLM-aided pipeline combining a masked autoencoder, deterministic data augmentation, and multi-level prompting, detects host-based attacks with high precision on three public datasets.

  3. Fair Document Valuation in LLM Summaries via Shapley Values

    cs.CL 2025-05 reject novelty 6.0 of 10

    Cluster Shapley groups semantically similar documents via embeddings and computes cluster-level Shapley values, claiming better efficiency-accuracy trade-offs than Monte Carlo and Kernel SHAP on Amazon review summarization.

  4. MIH-TCCT: Mitigating Inconsistent Hallucinations in LLMs via Event-Driven Text-Code Cyclic Training

    cs.AI 2025-02 conditional novelty 6.0 of 10

    MIH-TCCT reduces inconsistent hallucinations by cyclically training LLMs to translate event-based text into structured code and back, without task-specific fine-tuning.

  5. Evaluating Small Language Models for News Summarization: Implications and Factors Influencing Performance

    cs.CL 2025-02 conditional novelty 6.0 of 10

    A benchmark of 19 small language models on 2,000 news articles shows that the best small models nearly match 70B-parameter LLMs in summary quality while producing shorter summaries.

  6. Learn-by-interact: A Data-Centric Framework for Self-Adaptive Agents in Realistic Environments

    cs.LG 2025-01 conditional novelty 6.0 of 10

    A data-centric method that relabels agent trajectories with new instructions, called backward construction, improves LLM agent performance on web, code, desktop, and data-science tasks without human labeling.

  7. MageBench: Bridging Large Multimodal Models to Agents

    cs.CV 2024-12 conditional novelty 6.0 of 10

    MageBench introduces a 483-scenario benchmark showing current large multimodal models are far weaker than humans at agent tasks requiring continuous visual feedback and planning.

  8. MIMDE: Exploring the Use of Synthetic vs Human Data for Evaluating Multi-Insight Multi-Document Extraction Tasks

    cs.CL 2024-11 conditional novelty 6.0 of 10

    LLMs rank similarly on human and synthetic data for extracting insights, but synthetic data does not predict how well models map insights back to source documents.

  9. An Empirical Study of Many-Shot In-Context Learning for Machine Translation of Low-Resource Languages

    cs.CL 2026-04 unverdicted novelty 5.0 of 10

    BM25-retrieved many-shot examples match much larger random sets for translating English into ten truly low-resource languages, and ICL still helps after fine-tuning.

  10. Rethinking Hate Speech Detection on Social Media: Can LLMs Replace Traditional Models?

    cs.CL 2025-06 conditional novelty 5.0 of 10

    On three hate speech datasets, including a new code-mixed IndoHateMix benchmark, fine-tuned LLMs such as LLaMA-3.1 beat multilingual BERT models, with the largest gains on code-mixed Indian text.

  11. A Framework for Generating Conversational Recommendation Datasets from Behavioral Interactions

    cs.IR 2025-06 reject novelty 5.0 of 10

    ConvRecStudio generates roughly 38K synthetic multi-turn recommendation dialogs across three domains from historical interactions, and a cross-attention transformer fusing history with dialog beats dialog-only and his...

  12. Summarization for Generative Relation Extraction in the Microbiome Domain

    cs.CL 2025-06 conditional novelty 5.0 of 10

    Using LLM-generated summaries as input improves instruction-tuned generative relation extraction in the low-resource microbiome domain, though fine-tuned BERT models remain more accurate.

  13. Enhancing Semantic Consistency of Large Language Models through Model Editing: An Interpretability-Oriented Approach

    cs.CL 2025-01 conditional novelty 5.0 of 10

    Steering a handful of attention heads with a bias vector derived from consistent prompt pairs improves semantic consistency of LLaMA-2-7B on paraphrased NLU and NLG tasks.

  14. (Towards) Scalable Reliable Automated Evaluation with Large Language Models

    cs.CL 2026-07 conditional novelty 4.0 of 10

    Multi-LLM pairwise Elo ranking with adjustable consensus thresholds produces rankings of competency profiles that average Spearman ρ≈0.83 with expert judgments.

  15. A Research Vision for Web Search on Emerging Topics

    cs.IR 2025-09 accept novelty 4.0 of 10

    The paper lays out three research questions to guide the study and redesign of web search for emerging topics, focused on user knowledge, dynamic topic awareness, and responsible opinion formation.

  16. Multiple Abstraction Level Retrieve Augment Generation

    cs.CL 2025-01 conditional novelty 4.0 of 10

    MAL-RAG retrieves document, section, paragraph, and multi-sentence chunks together and claims a 25.7% improvement in AI-judged answer correctness on glycoscience questions over single-level RAG.

  17. Zero-Shot Strategies for Length-Controllable Summarization

    cs.CL 2024-12 conditional novelty 4.0 of 10

    Simple zero-shot tricks, target remapping, best-of-N sampling, and iterative self-revision, substantially improve length compliance in LLaMA-3 summarization without fine-tuning.

  18. Multi-LLM Text Summarization

    cs.CL 2024-12 reject novelty 4.0 of 10

    Using multiple LLMs to generate and select summaries raises ROUGE and BLEU scores on ArXiv and GovReport compared with single-LLM chunk-and-concatenate baselines.

Pith tools