Pith. sign in

REVIEW 10 cited by

BooookScore: A systematic exploration of book-length summarization in the era of LLMs

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.00785 v4 pith:QH7HFXFE submitted 2023-10-01 cs.CL cs.AIcs.LG

Yapei Chang , Kyle Lo , Tanya Goyal , Mohit Iyyer This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords booookscorellmsbook-lengthevaluationhumansummariesmodelsannotations
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

Summarizing book-length documents (>100K tokens) that exceed the context window size of large language models (LLMs) requires first breaking the input document into smaller chunks and then prompting an LLM to merge, update, and compress chunk-level summaries. Despite the complexity and importance of this task, it has yet to be meaningfully studied due to the challenges of evaluation: existing book-length summarization datasets (e.g., BookSum) are in the pretraining data of most public LLMs, and existing evaluation methods struggle to capture errors made by modern LLM summarizers. In this paper, we present the first study of the coherence of LLM-based book-length summarizers implemented via two prompting workflows: (1) hierarchically merging chunk-level summaries, and (2) incrementally updating a running summary. We obtain 1193 fine-grained human annotations on GPT-4 generated summaries of 100 recently-published books and identify eight common types of coherence errors made by LLMs. Because human evaluation is expensive and time-consuming, we develop an automatic metric, BooookScore, that measures the proportion of sentences in a summary that do not contain any of the identified error types. BooookScore has high agreement with human annotations and allows us to systematically evaluate the impact of many other critical parameters (e.g., chunk size, base LLM) while saving $15K USD and 500 hours in human evaluation costs. We find that closed-source LLMs such as GPT-4 and Claude 2 produce summaries with higher BooookScore than those generated by open-source models. While LLaMA 2 falls behind other models, Mixtral achieves performance on par with GPT-3.5-Turbo. Incremental updating yields lower BooookScore but higher level of detail than hierarchical merging, a trade-off sometimes preferred by annotators.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ArgCMV: An Argument Summarization Benchmark for the LLM-era

    cs.CL 2025-08 conditional novelty 6.0 of 10

    ArgCMV is a new LLM-curated benchmark of about 12,000 arguments from r/ChangeMyView, and current key point extraction methods transfer poorly to it.

  2. Positional Biases Shift as Inputs Approach Context Window Limits

    cs.CL 2025-08 conditional novelty 6.0 of 10

    The lost-in-the-middle effect in LLMs appears mainly when inputs fill up to half the model's context window; beyond that, accuracy favors information closest to the end.

  3. AbsenceBench: Language Models Can't Tell What's Missing

    cs.CL 2025-06 conditional novelty 6.0 of 10

    LLMs that ace Needle-in-a-Haystack struggle to identify deliberately omitted content, a new benchmark called AbsenceBench shows.

  4. The FACTS Grounding Leaderboard: Benchmarking LLMs' Ability to Ground Responses to Long-Form Input

    cs.CL 2025-01 conditional novelty 6.0 of 10

    FACTS Grounding is a benchmark and leaderboard that scores LLMs on producing long-form answers fully grounded in up to 32k-token documents, using a validated panel of judge models.

  5. CaseSumm: A Large-Scale Dataset for Long-Context Summarization from U.S. Supreme Court Opinions

    cs.CL 2024-12 conditional novelty 6.0 of 10

    CaseSumm, a 25.6K-pair dataset of Supreme Court opinions and official syllabuses, shows automated metrics favor fine-tuned Mistral while human experts prefer GPT-4, and LLM judges do not align with humans better than ROUGE.

  6. ResearchPulse: Building Method-Experiment Chains through Multi-Document Scientific Inference

    cs.CL 2025-09 conditional novelty 5.0 of 10

    ResearchPulse extracts motivation-method chains and experimental trends from related papers, rendering them as mind maps and line charts, and releases a 100-cluster benchmark; the reported '7B beats GPT-4o' result is ...

  7. LLMs for LLMs: A Structured Prompting Methodology for Long Legal Documents

    cs.AI 2025-09 reject novelty 5.0 of 10

    On CUAD legal contracts, a prompt-engineered QWEN-2 pipeline with chunking and two answer-selection heuristics reportedly outperforms the fine-tuned DeBERTa-large baseline by about 9%, reaching claimed state-of-the-ar...

  8. BLUEX Revisited: Enhancing Benchmark Coverage with Automatic Captioning

    cs.CL 2025-08 conditional novelty 4.0 of 10

    An updated BLUEX benchmark with 1,422 questions and GPT-4o-generated captions that make image-based questions usable by text-only LLMs.

  9. Enhancing COBOL Code Explanations: A Multi-Agents Approach Using Large Language Models

    cs.SE 2025-07 conditional novelty 4.0 of 10

    A two-agent LLM pipeline with hierarchical merging explains COBOL functions, files, and projects, outperforming zero-shot baselines on several text-quality metrics.

  10. Multi-LLM Text Summarization

    cs.CL 2024-12 reject novelty 4.0 of 10

    Using multiple LLMs to generate and select summaries raises ROUGE and BLEU scores on ArXiv and GovReport compared with single-LLM chunk-and-concatenate baselines.

Pith tools