{"id":"66f8d631-7e8b-4345-bb93-be5b367732fe","arxiv_id":"2502.00448","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"HERA packages event-related paragraphs and reorders them so LLMs produce more faithful and fluent summaries of long documents without fine-tuning.","lead":"HERA improves long document summarization by splitting a document into paragraphs, grouping paragraphs that describe the same event, and reordering them before asking an LLM to summarize. This is a training-free way to make large language models more faithful and fluent on long inputs.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The novel reordering step is never isolated: Table 3's ablation removes both reordering and its added token overhead, so the headline gain could be driven by packaging (retrieval) alone rather than by narrative-order restoration.","rationale":"I agree with the reader that the evaluation is under-powered and that the pipeline's retrieval/event-extraction step is unvalidated, but the single most load-bearing concern is different: the paper's signature contribution—context reordering—is never causally isolated. Table 3's 'w packaging' condition removes both the reordering and the extra summarization/computation that reordering entails, so the contrast conflates the ordering mechanism with the effect of additional LLM passes and context restructuring. A random-order or original-order control is the minimal experiment required to establish that narrative order itself matters. This is precisely the claim advertised in the introduction and abstract (messy narrative order impairs understanding), yet no condition tests it directly. Because the reader's verdict is already CONDITIONAL and my concern supports the same verdict (strengthening the condition: add a reordering-control ablation), I recommend UNCHANGED. The concern does not undermine the plausibility of the method but does mean the central mechanistic claim is not yet evidenced. Agreement is partial: the reader identified the event-extraction step as the weakest assumption, while I locate the load-bearing gap in the missing order-control ablation; both are legitimate, but the reordering control is the one whose failure would most directly falsify the paper's stated novelty.","tokens_in":8059,"tokens_out":1569,"duration_ms":17696,"concrete_test":"Run a controlled ablation on the arXiv test set with LLaMA 3: (a) full HERA, (b) packaging only with paragraphs in original document order, and (c) packaging only with paragraphs randomly shuffled, all using identical bag size, prompts, and number of LLM calls. If (a) outperforms both (b) and (c) by a clear margin on ROUGE/FactCC/SummaC, the reordering claim is supported; if (a) roughly matches (b) or (c), the observed gains are attributable to packaging/filtering rather than to narrative reordering, and the abstract's emphasis on ordering is not justified.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"HERA's central claim is that reordering retrieved paragraphs into narrative order improves long-document summarization without fine-tuning. The only evidence for the reordering component is the 'w packaging' vs 'w both' contrast in Table 3, but that contrast removes the entire reordering stage, including the extra summary-sentence inputs that NAON sorts and the additional inference passes they require. It does not test whether the order of the packaged paragraphs matters. A plausible alternative mechanism is that any context packaging—filtering to event-relevant paragraphs, splitting into smaller bags, generating local summaries—helps the LLM, while the final order contributes little or nothing. Because no condition feeds the same selected paragraphs in original, random, or reversed order, the specific contribution of reordering is confounded with the general effect of preprocessing/inference structure. If a random-shuffle condition produced identical scores, the paper's motivating claim ('messy narrative order impairs... understanding') would not be supported by the data. This is the most load-bearing weakness because the entirety of HERA's claimed novelty rests on reordering, and the current ablation cannot certify it.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HERA, a training-free framework for long-document summarization that packages context by retrieving paragraphs relevant to extracted events and reordering those paragraphs into narrative order before the LLM generates a summary. The pipeline segments the document into paragraphs, obtains sentence-length local summaries (via BRIO) for each paragraph, uses an LLM to extract events and retrieve top-k paragraphs per event, orders the selected paragraphs with the NAON sentence-ordering model applied to the local summaries, and then generates and aggregates per-event summaries. Experiments on 500-document subsets of arXiv and PubMed with four LLMs (LLaMA 2, LLaMA 3, Gemini 1.5, GPT-4) report consistent gains over the base LLMs on ROUGE-1/2/L, BERTScore, FactCC, and SummaC. An ablation using LLaMA 3 separates packaging from packaging-plus-reordering, and a hyperparameter study varies the bag size k.","tokens_in":8215,"tokens_out":6587,"duration_ms":61758,"significance":"If validated, HERA would offer a practical, no-fine-tuning method to improve LLM faithfulness and informativeness on long documents, and its explicit packaging-and-reordering mechanism directly addresses the widely studied lost-in-the-middle and reading-order sensitivity of LLMs. The paper's strengths are its clarity, the breadth of LLMs tested (two open-weight and two API models), the consistent positive direction of gains across all models and both datasets, the transparent reporting of prompts and software versions, the explicit measurement of inference-time overhead, and the honest Limitations section. The main gap is experimental: the paper's central novelty, reordering, is not cleanly isolated from the extra inputs and inference structure of the reordering stage, and the bag-size hyperparameter appears to be selected on the test set. These issues are fixable with additional controlled experiments and should be addressed before the quantitative claims are accepted.","major_comments":[{"comment":"The ablation does not isolate the reordering mechanism. The 'w packaging' condition removes the entire reordering step, including the paragraph-summary-sentence inputs that NAON sorts and the extra inference pass, relative to 'w both'. No condition feeds the same selected paragraphs in original, random, or reversed order while keeping all other steps identical. Therefore the gain between 'w packaging' and 'w both' could be driven by the additional input content (the BRIO-generated summary sentences) or by the changed pipeline structure, rather than by restoration of narrative order. This matters because the Introduction and Table 1 motivate the method specifically by 'messy narrative order' impairing LLM understanding. To support the reordering claim, the authors should add a control condition, e.g., the same selected paragraphs with randomly shuffled order, or with the summary-sentence inputs present but without reordering, and report those numbers.","section":"§3.3, Table 3"},{"comment":"The bag size k is selected on the test set, and the reporting is internally inconsistent. Section 3.1 states 'HERA use BRIO ... and selects the Top 6 paragraphs to form a segment bag,' while Table 4 shows k=5 as the best for arXiv on R-L/BS/FC and PubMed k=6 giving slightly higher R-L/BS/SC than k=5. The main results in Tables 2 and 3 do not state which k was used. If k was chosen after inspecting test-set scores, the reported gains are optimistic and the procedure does not reflect a deployment scenario. Please specify the k used in the main tables, justify the choice using the held-out validation splits, and report sensitivity of the main results across reasonable k values.","section":"§3.4 (Table 4) and §3.1 Implementation"},{"comment":"The claim of 'achieves state-of-the-art performance on both fluency and faithfulness metrics' is not supported by the baseline set. Table 2 compares only with FactorSum and Lodoss, both from 2022, and does not include recent long-context prompting strategies, hierarchical summarization pipelines, or other LLM-based summarization baselines. The phrase 'state-of-the-art' should either be replaced with a precise statement about gains over the selected baselines, or the baseline set should be expanded to include contemporary strong baselines on arXiv and PubMed.","section":"Introduction and §3.2"},{"comment":"No variance or significance information is reported. The test set is 500 randomly sampled documents per dataset, and no error bars, bootstrap intervals, or multiple runs are given. Table 4 shows differences of 0.1–0.2 points between adjacent values of k (e.g., PubMed R-L at k=5 vs k=6), and the text uses the word 'significantly' without a statistical test. The authors should report standard deviations or confidence intervals for the main results and key ablations so readers can assess the reliability of the reported margins.","section":"§3.1 and Tables 2–4"},{"comment":"The event-extraction step is a load-bearing component of the pipeline, but its quality is never analyzed. The prompt in Table 6 asks the LLM to 'Extract the most important events from the following summary sentences,' and the retrieved paragraphs depend directly on the extracted event set. If events are missed, duplicated, or split across bags, the final summary can drop or distort information. The Limitations section acknowledges that the retrieval method is not powerful, but no diagnostic evidence is provided (e.g., oracle event extraction, retrieval recall against the gold summary, or human evaluation of event segmentation). At minimum, please report retrieval quality or an ablation with gold/oracle events to show how much of the gain depends on the event extraction prompt.","section":"Appendix A (Table 6) and Limitations"}],"minor_comments":[{"comment":"ROUGE and BERTScore are described as 'fluency' metrics, but they measure lexical overlap and embedding similarity, not fluency. Please rephrase the metric grouping, e.g., 'informativeness (ROUGE, BERTScore)' and 'faithfulness (FactCC, SummaC)'.","section":"§3.2 and Table 2"},{"comment":"The sentence 'the trends of the results of the two datasets are not monotonous and similar' is confusing, and the subsequent claim that scores decrease for bag size greater than 5 is not true for PubMed, where k=6 gives the best R-L and BS. Please clarify the per-dataset trends.","section":"§3.4"},{"comment":"BRIO is described as a 'small summarization model,' but BRIO is built on BART-large; 'small' should be qualified (e.g., 'a compact summarization model relative to the long documents') to avoid implying a lightweight tool.","section":"§2 and §3.1"},{"comment":"The abstract and contributions say HERA does not require 'additional resources,' but Table 5 shows approximately 1.5x inference time and the pipeline also depends on BRIO and NAON. Please rephrase 'without additional resources' to 'without additional training or new model checkpoints' or state the computational overhead explicitly in the abstract.","section":"§3.5"},{"comment":"The paper says HERA uses 'the current state-of-the-art sentence reordering model NAON,' but no comparison or citation evidence is given that NAON is state-of-the-art for paragraph ordering via summary-sentence representatives. Please either soften the claim or provide a benchmark citation for NAON's ordering quality.","section":"§2, Context Reordering"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a computational linguistics venue and the central idea is appealing, but the reordering claim needs a cleaner controlled ablation (e.g., random-order condition) and the hyperparameter selection must be moved off the test set. I would be willing to recommend acceptance if the authors provide those experiments and temper the state-of-the-art wording. The event-extraction dependency should also get at least a diagnostic analysis, since it is the least-examined link in the pipeline."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"HERA is a training-free context-packaging and reordering pipeline for long-document summarization. The idea is straightforward: split the document into paragraphs, generate a one-sentence key for each, use an LLM to retrieve the top-k paragraphs per event, reorder those paragraphs with NAON, summarize each event, then aggregate. It is a clean integration of known components—BRIO keys, NAON ordering, LLM retrieval—and the paper is honest about that. The contribution is the packaging-plus-reordering combination, and the consistent gains across LLaMA 2, LLaMA 3, Gemini 1.5, and GPT-4 on arXiv and PubMed suggest the pipeline has real practical value.\n\nThe paper does several things well. It reports inference time, gives prompts in the appendix, and includes an ablation showing that packaging alone helps and that adding reordering gives an additional boost. The limitations section is candid: no human evaluation, only 500 documents per dataset, and no code or data released.\n\nSoft spots are real but not fatal. The evaluation is narrow: two scientific datasets, automatic metrics only, and no significance tests or error bars. The 'state-of-the-art' claim is overstated when the only baselines are FactorSum and Lodoss, two non-LLM summarization models from 2022. The bag size k is selected after inspecting test-set trends in Table 4, which is fitting a hyperparameter to the test set; the main results do not report which k was used, and the tables show no variance. The stress-test point about reordering is fair but less damaging than it first appears: the 'w packaging' versus 'w both' ablation does feed the same paragraphs to the LLM, differing only in order, so it does isolate an ordering effect. What it does not isolate is whether the effect comes from narrative order specifically. A random-order or reverse-order control would settle that, and I would ask for one before accepting the 'messy narrative order impairs understanding' framing. The packaging-only gains already justify the method; the reordering story needs that extra control.\n\nThis paper is for NLP researchers who want a plug-in, training-free improvement for long-document summarization with existing LLMs. It deserves a serious referee, not a desk reject. Send it back asking for the random-order control, significance testing, and either code/data release or a clearer statement of why the artifacts are not available.\n\nI would take it to a reading group to discuss the ablation design and the hyperparameter selection, and I would cite it as a practical baseline if I were working in this area.","headline":"HERA is a training-free packaging-and-reordering pipeline for long-document summarization that shows consistent gains across four LLMs; the core idea is useful, but the evaluation is thin and the reordering claim needs one more control.","tokens_in":8796,"tokens_out":2897,"would_cite":true,"duration_ms":32286,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"HERA improves long document summarization by grouping event-related paragraphs and reordering them into narrative order.","keywords":["long document summarization","context packaging","context reordering","event extraction","large language models","faithfulness","summarization without fine-tuning"],"falsifier":"Annotate gold event boundaries for a sample of arXiv and PubMed documents and run HERA twice, once with extracted events and once with gold events; if the summaries differ substantially, event extraction is the bottleneck. A simpler check is to annotate the retrieved top-k paragraphs and test whether every clause in the reference summary appears in some segment bag.","tokens_in":7809,"feed_emoji":"🧩","tokens_out":7930,"duration_ms":75088,"temperature":0.7,"pith_summary":"This paper argues that large language models summarize long documents poorly not because their context windows are too short, but because the information about a single event is scattered across the document and appears in a messy narrative order. To test that, the authors propose HERA, a framework that splits a document into paragraphs, writes a one-sentence summary of each paragraph, retrieves the paragraphs most relevant to each extracted event, reorders those paragraphs into narrative order, and then has the LLM summarize each event and aggregate the event summaries. On 500 arXiv and 500 PubMed papers, HERA applied to four different LLMs (LLaMA 2, LLaMA 3, Gemini 1.5, and GPT-4) improves ROUGE, BERTScore, FactCC, and SummaC scores over the same models without HERA. If correct, this means long-document summarization can be improved by restructuring the input alone, with no fine-tuning and only a modest increase in inference cost.","feed_headline":"Grouping text by event boosts LLM long-doc summaries","feed_subtitle":"HERA packs scattered paragraphs about the same event and reorders them, beating foundation models with no fine-tuning.","key_machinery":"The load-bearing mechanism is the packaging-and-reordering pipeline. HERA first segments a document by paragraph and uses BRIO to produce a one-sentence summary of each paragraph as a cheap retrieval key. An LLM then ranks these keys against each event extracted from the paragraph summaries, keeps the top-k paragraphs in a segment bag, and the NAON sentence-ordering model reorders the bag by its summary sentences with the paragraphs following that order. Finally the LLM summarizes each reordered bag and aggregates the event summaries into the overall document summary. This lets the LLM read event-coherent, narratively ordered context instead of the full document.","core_discovery":"The central claim, stated on the paper's own terms, is that context packaging and reordering together let an LLM produce more faithful and fluent long-document summaries than the LLM produces from the raw document. The reported numbers show, for example, that Gemini 1.5 with HERA gains 8.8% relative ROUGE-1 and 17.9% relative FactCC on arXiv, and that HERA combined with any of the four LLMs outperforms those LLMs alone on every metric. The ablation on LLaMA 3 shows that context packaging alone improves scores and that adding reordering improves them further, so both components carry part of the gain. The authors read this as evidence that LLMs' long-context failures are partly an input-organization problem, not only a context-length problem.","pith_inferences":["If event extraction is the true bottleneck, supplying gold event labels should improve HERA further; if it does not, the gain comes almost entirely from reordering rather than retrieval.","The same packaging idea should transfer to other long-context tasks, such as multi-part question answering or multi-document synthesis, where evidence for one answer is scattered and order matters.","Because HERA extracts events from BRIO's paragraph summaries, errors in those summaries propagate into retrieval; measuring event-extraction precision and recall directly would separate this failure mode from the reordering effect.","The gains on scientific papers may not transfer to less structured narratives such as novels or news articles, where events are more interleaved and paragraph summaries are noisier."],"forward_implications":["Any current or future LLM can inherit the gain without retraining, since HERA only changes the input context.","Context packaging alone accounts for part of the improvement by removing distracting content; reordering adds a further gain on every metric in the ablation.","The bag-size results imply a practical tuning rule: keep enough paragraphs to cover the event but not so many that the LLM is again surrounded by noise, with top-5 best on both datasets.","Faithfulness scores (FactCC and SummaC) rise alongside ROUGE and BERTScore, so the reordered input improves factual consistency rather than only lexical overlap.","The added inference time is about 1.5 times the base LLM, which the authors treat as a small price for the quality gain."],"supporting_citations":[{"why":"Supplies the arXiv and PubMed long-document summarization datasets used for all experiments.","marker":"Cohan et al., 2018"},{"why":"BRIO generates the one-sentence paragraph summaries that serve as retrieval keys and reordering representatives.","marker":"Liu et al., 2022"},{"why":"NAON is the sentence-ordering model HERA uses to reorder paragraphs within each segment bag.","marker":"Bin et al., 2023"},{"why":"Documents the position bias that motivates reordering relevant content toward the start of the context.","marker":"Liu et al., 2024"},{"why":"FactCC is the factuality-verification metric used to measure faithfulness of generated summaries.","marker":"Kryscinski et al., 2020"},{"why":"SummaC is the NLI-based inconsistency metric used as the second faithfulness measure.","marker":"Laban et al., 2022"},{"why":"ROUGE is the lexical-overlap metric used for informativeness.","marker":"Lin, 2004"},{"why":"BERTScore is the embedding-similarity metric used to compare summaries with references.","marker":"Zhang et al., 2020"},{"why":"FactorSum is a specialized long-document summarization baseline that HERA is compared against.","marker":"Fonseca et al., 2022"},{"why":"Lodoss is the extractive segmentation-and-summarization baseline that HERA is compared against.","marker":"Cho et al., 2022"}],"fun_headline_variants":["Grouping events by relevance boosts LLM long-doc summaries","HERA: repack context to improve LLM summarization without fine-tuning","Better long-doc summaries via event-based context reordering","LLM summarization gains from context packaging and reordering","Context packaging and reordering lift LLM long-doc summarization"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"HERA's gains depend on the event-extraction step: an LLM is asked to pull the most important events out of short paragraph summaries, and if that event list is incomplete or muddles distinct events, retrieval will fetch the wrong paragraphs and the final summary will drop or distort information.","fun_headline_variants_meta":{"raw":{"variants":["Grouping events by relevance boosts LLM long-doc summaries","HERA: repack context to improve LLM summarization without fine-tuning","Better long-doc summaries via event-based context reordering","LLM summarization gains from context packaging and reordering","Context packaging and reordering lift LLM long-doc summarization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000171,"raw_usage":{"total_tokens":1222,"prompt_tokens":844,"completion_tokens":378,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":460,"completion_tokens_details":{"reasoning_tokens":290}},"tokens_in":460,"tokens_out":378,"duration_ms":4140,"temperature":1.0,"reasoning_tokens":290,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T18:57:02.067788+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Annotate gold event boundaries for a sample of arXiv and PubMed documents and run HERA twice, once with extracted events and once with gold events; if the summaries differ substantially, event extraction is the bottleneck. A simpler check is to annotate the retrieved top-k paragraphs and test whether every clause in the reference summary appears in some segment bag.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"NAON is the sentence-ordering model HERA uses to reorder paragraphs within each segment bag."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Lodoss is the extractive segmentation-and-summarization baseline that HERA is compared against."}],"review_version":1}