Pith. sign in

REVIEW 4 major objections 7 minor 21 references

NexusSum: Hierarchical LLM Agents for Long-Form Narrative Summarization

T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read NexusSum, a three-agent LLM pipeline, out-scores prior summarization models on all four narrative benchmarks, including a 30% gain on BookSum.

desk verdict Competent engineering, but the headline SOTA claim is fitted to the test set rather than a fixed pipeline. read the letter →

arxiv 2505.24575 v1 pith:2HOQ7JJD submitted 2025-05-30 cs.CL cs.AI

Hyuntak Kim , Byung-Hak Kim This is my paper
classification cs.CLcs.AI
keywords long-formnarrativesummarizationmulti-agentLLMpipelinedialogue-to-descriptiontransformationhierarchicaliterativecompressionlengthcontrolBERTScoreBookSum
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

NexusSum is a fine-tuning-free multi-agent LLM pipeline for summarizing long narratives such as books, movies, and TV scripts. The paper claims it establishes a new state of the art on four narrative benchmarks, with the headline result a 30.0% BERTScore (F1) improvement over CachED on BookSum and smaller gains on MovieSum and MENSA, plus a tie on SummScreenFD. The argument is that three cooperating agents, a preprocessor that rewrites dialogue as descriptive prose, a hierarchical summarizer that works in scene-sized chunks, and an iterative compressor that enforces a target length, together preserve plot structure and avoid context truncation without retraining the underlying LLM. A sympathetic reader would care because long-form narrative summarization is where plain zero-shot prompting and fixed-context models are known to lose information, and the paper proposes a modular, training-free alternative that adapts via prompts and two control parameters.

What carries the argument

The load-bearing object is the chunk-and-concat pipeline. Each agent consumes only a chunk of the text, produces a transformation of that chunk, and all outputs are concatenated before the next stage. Scene-based chunks of eight scenes are used for preprocessing and summarization; sentence-based chunks of at most a fixed token budget are used for compression. Compression halts once the running summary is at or below a lower-bound target length, with at most ten iterations. The paper claims this structure is what preserves long-range plot information while making runtime linear in input length instead of quadratic in full-context attention.

What would settle it

Take a held-out set of narratives from each benchmark, fix the chunk size and target length on a development split, then apply NexusSum once without looking at the test summaries; if the BookSum advantage over CachED drops toward zero or reverses, the reported 30.0% gain is configuration fitting rather than pipeline quality. A second check: ask a larger panel of human readers to rank NexusSum against zero-shot summaries, since the paper's own expert panel already scores zero-shot higher on readability.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that the right division of labor among LLM agents removes the need for fine-tuning in long-form narrative summarization. NexusSum runs three agents in sequence: the Preprocessor turns dialogue into third-person narrative prose so that multi-speaker scenes read as continuous description; the Narrative Summarizer summarizes scene-size chunks of that prose one at a time and concatenates them; the Compressor iteratively shrinks the concatenated summary in sentence-size chunks until it falls under a target word count. The paper reports this pipeline beating the previous best systems on BookSum (+30.0% BERTScore F1 over CachED), MovieSum (+7.1% over HM-SR), and MENSA (+1.7% over CachED), and matching CachED on SummScreenFD, with expert evaluation showing better key-event retention and factuality than zero-shot summaries at the cost of readability.

Load-bearing premise

The reported state-of-the-art scores assume that per-dataset chunk sizes and length targets chosen by maximizing BERTScore on the evaluation benchmarks themselves produce a generally good system, and that BERTScore captures the high-quality summaries the abstract promises.

Editorial extensions

If this is right

  • If NexusSum's claim is right, a training-free multi-agent pipeline can outperform fine-tuned long-form summarizers on BookSum, MovieSum, and MENSA, meaning strong narrative summarization does not require per-dataset fine-tuning.
  • The dialogue-to-description preprocessing step should transfer to other dialogue-heavy genres, such as transcripts, plays, or conversational fiction, wherever semantic-overlap evaluation is the criterion.
  • The iterative compressor provides a practical length-control mechanism: summaries can be pushed close to a target word count while retaining more key events than prompt-only length instructions.
  • The demonstrated prompt adaptability, with Chain-of-Thought and few-shot variants, suggests the same agents can be re-skilled for new summary styles without parameter updates.
  • Because the pipeline is linear-time with fixed chunk sizes, it offers a scalable route to narratives beyond 100K tokens.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A consequence the paper leaves implicit: if the BERTScore gains mostly reflect reference overlap on the tuned datasets, the pipeline's advantage over CachED may shrink on unseen writing styles; this is testable by fixing the chunk size and target length on a validation split before evaluation.
  • The readability gap reported in Section 5.5 points to an explicit trade-off: forcing dense event retention produces summaries experts find less fluent, and a rewrite agent partially closes the gap, so a generate-then-polish design is a natural extension.
  • The same chunk-and-concat machinery could be applied to other long structured documents, such as legal opinions, clinical notes, or codebases, where scene boundaries would be replaced by section or function boundaries.
  • One could replace BERTScore with human preference or factuality metrics, as the paper's own NarrativeFactScore results begin to do, and ask whether the tuned length targets remain optimal; the paper does not claim they would.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper introduces NexusSum, a training-free, three-stage multi-agent LLM pipeline for long-form narrative summarization: a dialogue-to-description preprocessor, a hierarchical narrative summarizer, and an iterative compressor with length control. The authors evaluate on BookSum, MovieSum, MENSA, and SummScreenFD, reporting BERTScore (F1) and ROUGE scores, and claim a new state of the art with up to a 30.0% BERTScore improvement over CachED on BookSum. Additional results include an ablation study on MENSA, length-control experiments, prompt-engineering variants on SummScreenFD, a small human preference study, factuality checks with NarrativeFactScore, document-utilization analysis, and a linear-time complexity argument.

Significance. If the state-of-the-art claim survives closer scrutiny, the contribution is practically useful: the dialogue-to-description transformation is a concrete, reusable preprocessing idea; the ablation in Table 3 shows additive gains for each pipeline stage; the length-control mechanism is well motivated by the low LAR of zero-shot baselines; and the paper includes a contamination check and a linear-complexity analysis. However, the headline numbers are currently obtained under per-dataset configuration search on the evaluation split itself, so the significance is conditional on re-estimation with a fixed or validation-tuned configuration. The paper also contains an explicit, honest limitation statement in Section 7 that automated metrics do not capture readability and coherence.

major comments (4)
  1. [Section 5.1, Tables 2, 14, 15] The central state-of-the-art claim is undermined by per-dataset hyperparameter selection on the evaluation benchmarks. Appendix D (Table 14) chooses delta = 300 for BookSum, MovieSum, and MENSA and Maximum for SummScreenFD by maximizing BERTScore on those benchmarks, and Appendix E (Table 15) chooses theta = 1300, 900, 900, 0 for the same benchmarks in the same way. Table 2 therefore reports the argmax of a per-dataset search rather than the score of a single fixed pipeline. To support the claim that NexusSum establishes a new state of the art, the authors must either freeze a single configuration before touching the test split, use a validation split for model selection, or report selection-corrected estimates. Without this, the +30.0% BookSum gain and the +1.7% MENSA gain cannot be distinguished from overfitting to the test set.
  2. [Section 5.4, Table 5] The SummScreenFD result of 61.59 is obtained by selecting CoT and Few-Shot prompt variants on the SummScreenFD test set. Table 5 reports 56.61 for the base model, 58.61 with CoT, and 61.59 with CoT+FewShot, and the final Table 2 entry is the best of these three. This is the same test-set-fitting problem identified above for delta and theta: the reported score is the maximum over prompt choices evaluated on the same data used to report the result. Prompt selection should be treated as part of model selection and performed on a held-out validation split, or the full prompt-condition grid should be reported with selection-corrected estimates.
  3. [Section 4.2 and Table 2] Baseline scores are copied from prior publications (Saxena and Keller 2024a,b; Saxena et al. 2025; Jeong et al. 2025; Zhang et al. 2024) without rerunning the baselines in the same evaluation harness. BERTScore values can depend on the tokenizer, the scoring model, and the decoding configuration, and Table 2 does not provide per-run variance or significance tests. Given the small margins on MENSA (+1.7%) and SummScreenFD (a tie), the comparison may not be apples-to-apples. At minimum, the authors should rerun the strongest baselines with the same BERTScore setup and report means and confidence intervals over multiple seeds, or otherwise justify that the copied numbers are directly comparable.
  4. [Section 5.5, Table 6, and Section 7] The paper's own expert evaluation shows that NexusSum is rated far below Zero-Shot in readability (2.17 vs. 4.17), and Section 7 explicitly concedes that BERTScore and ROUGE fail to capture readability, coherence, and user preference. This does not by itself invalidate the BERTScore-based state-of-the-art claim, but it conflicts with the abstract's promise of 'accurate, high-quality summaries' and with the introduction's claim that the method improves coherence. The authors should either temper the quality claim in the abstract and Section 1, or provide evidence that the readability gap is specific to the K-Drama evaluation setup (three dramas, target length 600, three experts) rather than a general property of the pipeline.
minor comments (7)
  1. [Table 2] The asterisk on the SummScreenFD value 61.59 is never explained in the table caption or text; please add a footnote clarifying that this number uses the CoT+FewShot prompt configuration from Table 5.
  2. [Section 3.3, Eq. (6)] The sentence 'The i-th Compressor agent Ci refines the previous compressed summary Si-1' is confusing because Eq. (6) shows Ci applied to chunks of Si-1; please clarify the index convention, for instance by writing 'C_i is applied to chunks of S_{i-1}'.
  3. [Appendix D, Table 14] The row label 'Maximum(100K)' is inconsistent with the dataset input lengths in Table 1, especially for SummScreenFD; please specify the actual token budget used for the 'Maximum' condition for each dataset.
  4. [Appendix I] There are several typos in the appendix: 'NEXUSSUMour' should be 'NEXUSSUM' or 'our', 'mitgating' should be 'mitigating', and 'Exmaple' appears in Tables 10 and 12; a proofreading pass is needed.
  5. [Section 4.4] The sentence 'We evaluate NEXUSSUMusing a semantic similarity, length control metrics' has a grammatical error; it should read 'We evaluate NexusSum using semantic similarity and length-control metrics.'
  6. [Figure 2] Figure 2 mixes absolute BERTScore values, percentages, and benchmark labels without a labeled y-axis; please add a clear axis label and a legend to distinguish scores from improvement percentages.
  7. [Section 5.5] The human evaluation uses only three K-Drama summaries and three experts, with 'at least two evaluators assessing each output'; please report inter-annotator agreement or the full raw score matrix, as the current table cannot support fine-grained comparisons.

Circularity Check

2 steps flagged · score 6.0 of 10

Per-dataset chunk size δ and lower bound θ are selected by maximizing BERTScore on the evaluation benchmarks, and the SummScreenFD prompt variant is likewise chosen on the test set, so the reported SOTA numbers are fitted maxima rather than predictions of a fixed system.

  1. fitted input called prediction [Section 5.1 / Table 2 vs Appendices D and E (δ and θ selection)]
    "For each benchmark, NEXUSSUM’s configuration of δ and θ are detailed in the Appendices D and E. ... Based on the results, we configure δs of 300, 300, 300, and Maximum words for BookSum, MovieSum, MENSA, and SummScreenFD respectively in NEXUSSUM. ... Based on the results, we set θs of NEXUSSUMfor BookSum, MovieSum, MENSA and SummScreenFD to 1300, 900, 900, and 0, respectively."

    The two pipeline controls δ (chunk size) and θ (lower bound) are chosen per dataset by taking the configuration with the highest BERTScore on the same benchmark whose number is later reported as NEXUSSUM's result. Appendix D states that BookSum 'achieves optimal performance (67.16) with a 300-word chunk size' and then fixes δ=300; Appendix E fixes θ=1300 because Table 15 shows the maximum 70.70 there. Table 2 then reports 70.70 as NEXUSSUM's BookSum score. The reported SOTA number is therefore the argmax of a search over the evaluation set, i.e., a fitted maximum, not the score of a fixed system. Without a validation split or frozen configuration, the 'prediction' reduces to the selection criterion used to pick the hyperparameters.

  2. fitted input called prediction [Section 5.4 / Table 5 and Table 2 (SummScreenFD 61.59)]
    "Table 5 demonstrates that CoT alone improves BERTScore (F1) from 56.61 to 58.61 (+2.0 points), while adding Few-Shot learning further boosts performance to 61.59 (+2.98 points)."

    On SummScreenFD, three prompt variants are evaluated and the best-scoring variant (CoT+FewShot, 61.59) is adopted as NEXUSSUM's result in Table 2, matching the 61.59* entry. This is another test-set fitted choice: the reported comparison score is the optimum over prompt-engineering options explored on the evaluation benchmark itself, so it is not an independent prediction of a single pipeline configuration. The paper presents this as adaptability, but it also means the headline SummScreenFD number is an argmax rather than a fixed-method score.

full rationale

The central SOTA claim rests on numbers that are selected, not predicted. Appendices D and E tune δ and θ by maximizing BERTScore on the four evaluation benchmarks, and Section 5.4 selects the SummScreenFD prompt variant the same way; Table 2 then presents these argmax values as NEXUSSUM's performance. That is the fitted-input-as-prediction pattern: the reported quality metric is the objective used to choose the configuration, so the 'prediction' is forced by construction to be at least as high as the selected grid point. The NarrativeFactScore appendix uses a metric from Jeong et al. (2025), which shares an author with this paper; however, that self-citation is secondary to the BERTScore SOTA claim and does not by itself raise the score. There is no uniqueness theorem, no definitional equivalence, and the base LLM is not fine-tuned, so this is partial circularity rather than full self-definition. A validation-based selection or a single frozen configuration with repeated-run variance would have made the benchmark claim self-contained; as written, the score is 6.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The empirical claims rest on a small number of unverified domain assumptions: scene-based chunking applies cleanly to novels without scene headings, BERTScore captures the 'high-quality' that the paper promises, and low n-gram overlap rules out contamination. The main free parameters are the per-dataset chunk size and length bound, both chosen on the evaluation benchmarks, plus hand-set engineering constants. No new theoretical entities are introduced.

free parameters (5)
  • Scene chunk size (k scenes per chunk) = 8 scenes
    Set by hand in Sections 3.1 and 3.2 to balance context retention and efficiency; k = total scenes / 8.
  • Chunk size δ (words) per dataset = 300 (BookSum), 300 (MovieSum), 300 (MENSA), Maximum (SummScreenFD)
    Selected in Appendix D by maximizing BERTScore(F1) on each evaluation benchmark; affects compression and quality.
  • Lower bound θ (words) per dataset = 1300 (BookSum), 900 (MovieSum), 900 (MENSA), 0 (SummScreenFD)
    Selected in Appendix E by maximizing BERTScore(F1) on each evaluation benchmark; controls when iterative compression stops.
  • Maximum compression iterations = 10
    Hand-set cap in Section 3.3 to balance quality and computational efficiency.
  • Sampling temperature = 0.3 (Mistral Large), 0 (Claude 3 Haiku)
    Implementation choice in Section 4.3; not fitted but affects output variability.
assumptions (4)
  • domain assumption Scene-based chunking is a valid semantic decomposition for all four datasets, including BookSum novels that lack scene headings.
    Invoked in Section 3.1 (Equations 1 to 4) for preprocessing and summarization; for novels 'scene' is not defined, so the chunking procedure is underspecified.
  • domain assumption BERTScore (F1) with DeBERTa-XLarge-MNLI is an adequate proxy for summary quality for the claimed 'high-quality summaries'.
    Used as the primary metric in Section 4.4 and Results; the paper's own human evaluation (Section 5.5) shows readability is rated lower for NexusSum, so this assumption is questionable.
  • domain assumption Low n-gram overlap between zero-shot outputs and reference summaries indicates no meaningful data contamination.
    Appendix F performs an n-gram overlap analysis and concludes leakage is negligible; this is a heuristic, not a proof of non-contamination.
  • domain assumption Reference summaries in BookSum, MovieSum, MENSA, and SummScreenFD are treated as ground truth for scoring.
    Standard evaluation assumption in summarization (Sections 4.1 and 4.4); not independently verified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of NexusSum: Hierarchical LLM Agents for Long-Form Narrative Summarization." pith.science (2026). https://pith.science/paper/2HOQ7JJD

@misc{pith2026250524575,
  author       = {Pith},
  title        = {Pith review of: NexusSum: Hierarchical LLM Agents for Long-Form Narrative Summarization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2HOQ7JJD}},
  note         = {Machine review of arXiv:2505.24575}
}
read the original abstract

Summarizing long-form narratives--such as books, movies, and TV scripts--requires capturing intricate plotlines, character interactions, and thematic coherence, a task that remains challenging for existing LLMs. We introduce NexusSum, a multi-agent LLM framework for narrative summarization that processes long-form text through a structured, sequential pipeline--without requiring fine-tuning. Our approach introduces two key innovations: (1) Dialogue-to-Description Transformation: A narrative-specific preprocessing method that standardizes character dialogue and descriptive text into a unified format, improving coherence. (2) Hierarchical Multi-LLM Summarization: A structured summarization pipeline that optimizes chunk processing and controls output length for accurate, high-quality summaries. Our method establishes a new state-of-the-art in narrative summarization, achieving up to a 30.0% improvement in BERTScore (F1) across books, movies, and TV scripts. These results demonstrate the effectiveness of multi-agent LLMs in handling long-form content, offering a scalable approach for structured summarization in diverse storytelling domains.

Figures

Figures reproduced from arXiv: 2505.24575 by the authors.

Figure 1
Figure 1. Illustration of narrative structure, showcasing [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Performance comparison of NEXUSSUM with state-of-the-art summarization models using BERTScore (F1) across multiple benchmarks. NEXUSSUM achieves up to a 30.0% improvement, particularly excelling in BookSum, where hierarchical processing mitigates context truncation, demonstrating its advantage in long-form narrative summarization. Bold values indicate the new state-of-the-art score. narrative coherence. Third, zero-… view at source ↗
Figure 3
Figure 3. Overview of NEXUSSUM, a hierarchical multi-agent LLM framework for long-form narrative summariza￾tion. It follows a three-stage pipeline: (1) Preprocessing converts dialogues into descriptive prose, (2) Narrative Summarization generates an initial summary, and (3) Iterative Compression refines it for length control while preserving key details. length, and ensure information retention without re￾quiring fine-tuning … view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Distribution of summary sentence alignments across document locations based on Rouge-L scores. Each [PITH_FULL_IMAGE:figures/full_fig_p025_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

21 extracted references · 17 canonical work pages

  1. [1]

    Convert dialogue to reported speech, including emotions and speaker traits

    work page

  2. [2]

    Integrate narration and stage directions seamlessly

    work page

  3. [3]

    Preserve original structure, pacing, and character voices

    work page

  4. [4]

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa

    Reformer: The efficient transformer.Preprint, arXiv:2001.04451. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. InAdvances in Neural Information Processing Systems 35: An- nual Conference on Neural Information Processing Systems 2022, NeurIPS 2022. Wojciech Kryscin...

    arXiv 2001

  5. [5]

    Use varied language for different speaking styles

    work page

  6. [6]

    Include relevant context from stage directions

    work page

  7. [7]

    Create a cohesive narrative retaining key dramatic elements

    work page

  8. [8]

    Maintain original language and formal written style

    work page

Show all 21 references
  1. [9]

    Capture emotional tone and subtext

  2. [14]

    INT." or

    Use third-person perspective. ## Output Format: [Scene Heading: line starting with "INT." or "EXT."] (skip this if there is no scene heading) [Your Narrative Here] [Scene Heading: line starting with "INT." or "EXT."] (skip this if there is no scene heading) [Your Narrative Her...

  3. [15]

    Analyze the provided script carefully, noting its structure, style, and unique features

  4. [16]

    Identify the types of dialogue present (e.g., conversations, monologues, voice-overs)

  5. [17]

    Recognize the script’s format for scene headings, time indicators, and descriptive elements

  6. [18]

    Determine the overall tone and atmosphere of the script

  7. [19]

    Using the strategy provided below, transform the given script by converting dialogue into narrative form while seamlessly integrating it with the original descriptive text

    Create a detailed, step-by-step strategy for transforming this specific script, addressing: - How to handle the particular dialogue styles present - Methods to preserve the script’s unique formatting and structural elements - Techniques for maintaining the script’s tone and at...

  8. [20]

    Transform the dialogue into narrative form, integrating it smoothly with the descriptive text

  9. [21]

    Baba?" The father responded warmly,

    Follow each step of the provided strategy meticulously. ## Output Format: [Scene Heading Here] [Your Narrative Paragraph Here] [Scene Heading Here] [Your Narrative Paragraph Here] ... Table 8: Preprocessor Agent Prompt with CoT A.2 Narrative Summarization Stage A.2.1 Narrative...

    1992

  10. [2020]

    Preprint, arXiv:2004.05150

    Longformer: The long-document transformer. Preprint, arXiv:2004.05150. Amanda Bertsch, Uri Alon, Graham Neubig, and Matthew R. Gormley. 2023. Unlimiformer: Long- range transformers with unlimited length input. In Thirty-seventh Conference on Neural Information Processing Syste...

    2004 arXiv

  11. [2021]

    A LLM Agent Prompts in NEXUSSUM A.1 Preprocessing Stage A.1.1 Preprocessor Agent (P) Single-Turn System Prompt You are an expert script-to-narrative converter

    Topic-oriented spoken dialogue summariza- tion for customer service with saliency-aware topic modeling.Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, 35(16):14665–14673. A LLM Agent Prompts in NEXUSSUM A.1 Preprocessing Stage A.1.1 Preprocessor Agent (P) S...

    2021

  12. [2022]

    InFindings of the Association for Computational Linguistics, NAACL 2022, pages 724– 736, Seattle, United States

    LongT5: Efficient text-to-text transformer for long sequences. InFindings of the Association for Computational Linguistics, NAACL 2022, pages 724– 736, Seattle, United States. Association for Compu- tational Linguistics. Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shic...

    2022 arXiv

  13. [2024]

    In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024

    Booookscore: A systematic exploration of book-length summarization in the era of llms. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. Mingda Chen, Zewei Chu, Sam Wiseman, and Kevin Gimpel. 2022a....

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.