REVIEW 11 cited by
FABLES: Evaluating faithfulness and content selection in book-length summarization
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
abstract
While long-context large language models (LLMs) can technically summarize book-length documents (>100K tokens), the length and complexity of the documents have so far prohibited evaluations of input-dependent aspects like faithfulness. In this paper, we conduct the first large-scale human evaluation of faithfulness and content selection on LLM-generated summaries of fictional books. Our study mitigates the issue of data contamination by focusing on summaries of books published in 2023 or 2024, and we hire annotators who have fully read each book prior to the annotation task to minimize cost and cognitive burden. We collect FABLES, a dataset of annotations on 3,158 claims made in LLM-generated summaries of 26 books, at a cost of $5.2K USD, which allows us to rank LLM summarizers based on faithfulness: Claude-3-Opus significantly outperforms all closed-source LLMs, while the open-source Mixtral is on par with GPT-3.5-Turbo. An analysis of the annotations reveals that most unfaithful claims relate to events and character states, and they generally require indirect reasoning over the narrative to invalidate. While LLM-based auto-raters have proven reliable for factuality and coherence in other settings, we implement several LLM raters of faithfulness and find that none correlates strongly with human annotations, especially with regard to detecting unfaithful claims. Our experiments suggest that detecting unfaithful claims is an important future direction not only for summarization evaluation but also as a testbed for long-context understanding. Finally, we move beyond faithfulness by exploring content selection errors in book-length summarization: we develop a typology of omission errors related to crucial narrative elements and also identify a systematic over-emphasis on events occurring towards the end of the book.
Forward citations
Cited by 11 Pith papers
-
DeepTRACE: Auditing Deep Research AI Systems for Tracking Reliability Across Citations and Evidence
An audit framework and empirical study showing that generative search engines and deep research agents frequently produce one-sided answers and weakly supported citations, with citation accuracy between 40 and 80%.
-
AbsenceBench: Language Models Can't Tell What's Missing
LLMs that ace Needle-in-a-Haystack struggle to identify deliberately omitted content, a new benchmark called AbsenceBench shows.
-
The FACTS Grounding Leaderboard: Benchmarking LLMs' Ability to Ground Responses to Long-Form Input
FACTS Grounding is a benchmark and leaderboard that scores LLMs on producing long-form answers fully grounded in up to 32k-token documents, using a validated panel of judge models.
-
Efficient Long Context Language Model Retrieval with Compression
CoLoR, a preference-optimized passage compressor, cuts LCLM retrieval context by 1.91x while improving average retrieval accuracy by 6% over nine benchmarks.
-
SummExecEdit: A Factual Consistency Benchmark in Summarization with Executable Edits
A new benchmark built with executable phrase-level edits shows that most LLMs detect and explain factual inconsistencies in summaries only weakly, with the best model scoring 0.49 on the joint task.
-
A 2-step Framework for Automated Literary Translation Evaluation: Its Promises and Pitfalls
A rubric-plus-question-answering LLM framework for literary translation evaluation beats traditional MT metrics but still trails human agreement, especially on Korean honorifics.
-
Decoupling Generation and Selection for Budget-Constrained Faithful Summarization
Selecting sentences from multiple generated summaries under a sentence budget improves factual consistency and gives explicit length control without retraining the generator.
-
Story Ribbons: Reimagining Storyline Visualizations with Large Language Models
An LLM-driven pipeline and interactive storyline visualization tool can extract and display narrative structure from raw novels and scripts with sufficient reliability for literary analysis.
-
GPT Editors, Not Authors: The Stylistic Footprint of LLMs in Academic Preprints
Across 2,408 arXiv preprints, LLM-typical word usage does not cluster in any section, indicating that AI assistance, when used, is uniform rather than limited to specific parts of a paper.
-
Towards Long Context Hallucination Detection
A decomposition-and-aggregation encoder architecture, trained on a new GPT-4o-injected BookSum dataset, outperforms LLM baselines on long-context hallucination detection while running much faster.
-
Eliciting In-context Retrieval and Reasoning for Long-context Large Language Models
A new benchmark with hard confounders shows long-context LMs struggle at in-context retrieval, and a retrieve-then-generate fine-tune plus attention-probing decoding substantially improves them.
Discussion (0). Continue with ORCID to comment.