{"id":"9f7d6c2d-7bad-4460-918b-03b59d49d05f","arxiv_id":"2504.16711","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A model-agnostic retrieval framework that selects salient discourse units as latent queries, ranks documents, and filters irrelevant EDUs, improving ROUGE scores for multi-document summarization.","lead":"This paper presents ReREF, a retrieval framework for multi-document summarization that ranks documents and filters out irrelevant text segments before feeding the summarizer. It automatically derives search queries from the documents themselves, avoiding hand-written queries, and reports modest ROUGE gains across four datasets and seven summarizers.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The non-circular evidence for ReREF's core mechanism is thin: Section 5.4 evaluates retrieval against the same reference-similarity labels used for training, and no end-to-end comparison to a standard retriever is reported.","rationale":"The reader's conditional verdict is sound. I agree with the circularity problem but frame it slightly differently: the deeper issue is that the paper never shows, end-to-end, that ReREF beats an existing retriever. The ROUGE tables support the literal claim 'better than original truncation,' but the framework's claimed contribution is the learned latent-query/EDU mechanism; without a standard-retriever control, the observed gains could be generic retrieval gains. The circular Section 5.4 evaluation means the intermediate retrieval-accuracy claims cannot be used to rule this out. The ablation's w/o both baseline is explicitly not the original truncation, so it cannot decompose the gain relative to the actual baseline. Thus the conditional requirements should include the end-to-end retriever comparison, or a randomized-label control. I do not think the paper should be rejected: the experiments are extensive, the direction is plausible, and the code link is present but not yet verifiable. Keeping CONDITIONAL is appropriate.","tokens_in":20188,"tokens_out":9280,"duration_ms":89208,"concrete_test":"Run an end-to-end experiment on Multi-News and WCEP-10 with PRIMERA at the 4096-token budget: (1) ReREF as reported; (2) the same budget using BM25+RAKE (as in Section 5.4) to rank documents and select or filter EDUs, with the selected content truncated to the same token budget; and (3) the same pipeline using DYLE+RAKE. Also include a control that selects EDUs by random scores. If BM25+RAKE or DYLE+RAKE match ReREF's ROUGE within the bootstrap significance bounds of Table 2, the claimed advantage of the learned latent-query/EDU-filtering mechanism over standard retrieval is not established. If ReREF is significantly better, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of consistent ROUGE gains is not in dispute from the tables, but the paper's evidence that the learned latent-query/EDU-filtering mechanism is what produces those gains is weaker than it appears. Section 4.3 creates both the EDU-scoring and document-ranking ground truth by cosine similarity to the reference summary using multi-qa-mpnet-base-cos-v1. Section 5.4 then evaluates query selection, filtering, and document ranking against exactly those same cosine-similarity labels. This is circular: the retrieval module was trained to reproduce the proxy, so high Precision@K/NDCG against the proxy does not demonstrate that the proxy captures summary-relevant content. The only non-circular evidence is end-to-end ROUGE, but that evidence does not isolate the mechanism: ReREF is compared only to the original truncation, not to an existing retriever (BM25+RAKE or DYLE+RAKE) feeding the same summarizer with the same context budget. Consequently, the observed ROUGE gains could come from 'any selection beats naive truncation' rather than from ReREF's latent-query ranking and EDU-level filtering. The ablation in Table 5 does not fully resolve this: its 'w/o both' baseline (random order + last-token drop) is explicitly different from and worse than PRIMERA's original even truncation (footnote 11), so the decomposition is relative to a strawman, not to the actual baseline. If a standard retriever achieves comparable ROUGE, the proposed framework's specific contribution is unsubstantiated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ReREF, a model-agnostic retrieval framework for multi-document summarization (MDS) that unifies document ranking and elementary discourse unit (EDU) filtering. The framework automatically selects latent queries from the most salient EDUs, uses those queries to rank documents, and then replaces naive last-token truncation with EDU-level filtering to fit the summarizer's context limit. The training signal for both EDU scoring and document ranking is cosine similarity to the reference summary using the pre-trained embedding model multi-qa-mpnet-base-cos-v1, and the whole pipeline is optimized by an EM-style alternation between query selection and parameter updates. The authors evaluate ReREF with seven summarizers on four MDS datasets in fully supervised and few-shot settings, reporting ROUGE improvements over the original truncation baselines, along with retrieval-side precision/NDCG analyses and a small human evaluation. The core claim is that ReREF consistently improves summarization quality across models and datasets while avoiding manual query construction and coarse-grained retrieval.","tokens_in":20491,"tokens_out":6927,"duration_ms":56721,"significance":"If the central claim holds, ReREF offers a practical, model-agnostic way to use limited context windows more effectively in MDS, with the notable advantages of eliminating handcrafted queries and filtering at a finer granularity than sentences or passages. The paper's strengths include a broad experimental matrix (seven summarizers, four datasets, both fully supervised and few-shot settings), the use of paired bootstrap significance testing, a released code repository, and a human evaluation component. The ROUGE point estimates in Tables 2 and 3 are mostly positive, and the framework's plug-and-play nature is genuinely appealing. However, the paper's mechanism-level validation is substantially weakened by the circular retrieval evaluation in Section 5.4, and the claim of 'significant performance gain in all datasets' in Section 5.2 is contradicted by the manuscript's own significance markers. The lack of an end-to-end comparison against a standard retriever under the same context budget means the observed gains could plausibly stem from 'any selection beats naive truncation' rather than from ReREF's specific latent-query and EDU-filtering mechanism.","major_comments":[{"comment":"The retrieval-side evaluation in Section 5.4 is circular. The ground-truth labels for EDU selection (top-k and bottom-k EDUs) and document ranking are constructed by the same cosine-similarity-to-reference-summary scoring described in Section 4.3 using multi-qa-mpnet-base-cos-v1. The retrieval model is trained to reproduce exactly these labels, so the high Precision@K and NDCG/MRR values in Figures 4 and 5 only demonstrate that the model fits the training proxy; they do not validate that the proxy captures what downstream summarizers actually need. This circularity undermines the paper's claim to have validated the latent-query and EDU-filtering mechanism through 'in-depth analysis.' I recommend replacing or supplementing this evaluation with a non-circular test, such as an end-to-end comparison in which the same summarizer is fed, under an identical token budget, by (a) ReREF, (b) a standard retriever like BM25+RAKE or DYLE+RAKE, and (c) an inference-time oracle that directly uses the Section 4.3 similarity scores without any learned retrieval module. Alternatively, human annotation of whether the selected/filtered EDUs are actually summary-relevant would break the circularity.","section":"§5.4, §4.3"},{"comment":"The text in Section 5.2 states that 'Our retrieval framework achieves a significant performance gain in all datasets compared to the original base models,' but this is contradicted by the minus signs in Tables 2 and 3, which denote p>=0.05. For example, in Table 2, all three ROUGE improvements for StableLM-Zephyr-3B on Multi-XScience and WCEP-10 are marked not significant, as are all three for BartGraphSum on Multi-XScience. In Table 3, BartGraphSum, BART (partial), PEGASUS (partial), and StableLM show multiple non-significant cells, especially on Multi-XScience and WCEP-10. Across both tables, roughly 26 of 168 model-dataset-metric cells are not significant. The claim of consistent significant improvement is therefore overstated. The authors should temper the language to 'improvements in most settings, with significance varying by dataset and model,' or provide additional evidence (e.g., multiple random seeds, larger sample sizes, or meta-analytic aggregation) to support the significance claim.","section":"§5.2, Tables 2 and 3"},{"comment":"The ablation study in Table 5 does not isolate the mechanism responsible for the end-to-end gains. Footnote 11 explicitly states that the 'w/o both' variant (random document order plus last-token drop) is different from and worse than PRIMERA's original even truncation, and that its Table 5 results do not align with the 'original' row in Table 2. Consequently, the ablation demonstrates only that ReREF improves over a deliberately degraded baseline (random order plus last-token drop), not that ranking and EDU filtering are the cause of the improvement over the actual baseline used in Table 2. This is a strawman comparison. To support the claim that the learned ranking and filtering are what produce the gains, the authors should compare ReREF end-to-end against a standard retriever (e.g., BM25+RAKE or DYLE+RAKE) with the same context budget feeding the same summarizer, or against a non-learned selection based directly on the Section 4.3 similarity proxy. Without such a comparison, the observed ROUGE gains could be explained by any content-selection strategy that avoids the information loss of naive truncation.","section":"Table 5 and footnote 11"}],"minor_comments":[{"comment":"The caption contains a typo: 'MRR_2rd' should be 'MRR_2nd'.","section":"Figure 5"},{"comment":"The human evaluation in Section 5.5 is based on only 100 Multi-News samples with a single base model (PRIMERA), and no inter-annotator agreement statistic (e.g., Krippendorff's alpha or Cohen's kappa) is reported. Adding agreement metrics and ideally a second dataset would strengthen the reliability of the human evaluation.","section":"§5.5"},{"comment":"The notation in Equation (11) is confusing: the two loss terms use P_q and P_f for the positive sets and what appear to be complement sets, but the complement notation is not visible in the typeset equation. Please clarify the sampling distributions and the definition of the negative sets in the BPR loss.","section":"Eq. (11)"},{"comment":"Figures 4 and 5 report retrieval-side precision and ranking metrics without confidence intervals or significance tests. Given the circularity issue, at minimum the variance across test instances should be reported so that the reader can assess the stability of the comparisons against BM25 and DYLE.","section":"§5.4"},{"comment":"The sentence 'For fair comparison, we used the same input context as PRIMERA when evaluating LLaMA variants' is unclear: LLaMA-3.1 supports 128k tokens, so it is not obvious why a 4096-token limit is 'fair' to LLaMA. Please clarify the rationale for constraining all models to the same budget.","section":"§5.1"}],"recommendation":"major_revision","confidential_remarks":"The circular retrieval evaluation in Section 5.4 is the most serious methodological issue and should be addressed head-on; even a well-executed end-to-end comparison to BM25+RAKE/DYLE+RAKE would substantially improve the paper. The overstatement of 'significant' improvements in Section 5.2 is a separate but easily fixable issue. If the authors cannot provide non-circular evidence for the mechanism, the paper's contribution should be reframed as an empirical demonstration that learned selection improves over naive truncation, without claiming that the specific latent-query/EDU-filtering design is validated. The manuscript fits SIGIR's scope but needs these revisions before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"ReREF is a coherent, model-agnostic retrieval plug-in for multi-document summarization that unifies EDU-level filtering, document ranking, and automatic latent query selection in an EM-style loop. The combination is genuinely new relative to DYLE and LightPAL: finer retrieval granularity, no manual queries, and a single framework that both ranks documents and filters irrelevant EDUs. The experimental breadth is real: seven summarizers, four datasets, few-shot settings, human evaluation, and ablations. The end-to-end ROUGE comparisons are independent and non-circular, and the gains, though modest at around one point on average, are directionally consistent, with a few strong cases like PEGASUS on WCEP-10 and few-shot Wikisum. The citation pattern looks solid; DYLE, LightPAL, PRIMERA, DMRST, and BPR are all relevant and properly engaged.\n\nThe main soft spot is Section 5.4. The retrieval-quality evaluation uses ground-truth labels built from the same cosine similarity to the reference summary (multi-qa-mpnet-base-cos-v1) that created the training data in Section 4.3. That is circular as evidence that the learned ranking captures summary-relevant content; it only shows the model can reproduce the proxy. The non-circular evidence is end-to-end ROUGE, but there ReREF is compared only against original truncation, not against a standard retriever feeding the same summarizer with the same context budget. So “any selection beats naive truncation” remains a plausible alternative explanation for part of the gain. The ablation in Table 5 does not fully resolve this: the “w/o both” variant uses random permutation plus last-token drop, which footnote 11 explicitly distinguishes from PRIMERA’s original even truncation, making the decomposition relative to a strawman rather than the actual baseline. I would not call this fatal, but it substantially weakens the claim that the specific mechanism, latent-query ranking plus EDU filtering, is what drives the improvements.\n\nA second, smaller issue is significance reporting. Many improvements in Tables 2 and 3 are marked not significant, especially for Multi-XScience, WCEP-10, and StableLM, yet the abstract says “consistent improvements” without caveat. The authors deserve credit for marking the non-significant cells in the tables; the abstract should match that honesty. The EM framing is also asserted rather than formally derived, which is acceptable for a systems paper but should be presented as an iterative alternating optimization, not as a rigorous EM guarantee. Finally, the code repository is mentioned but no commit hash or artifact details are given, so the experiments are not independently reproducible from the preprint.\n\nWho this is for: people working on retrieval for long-document or multi-document summarization, and anyone building model-agnostic context-selection plug-ins. It deserves a serious referee; I would send it out. The revision should add an end-to-end comparison with a standard retriever under the same context budget, reframe Section 5.4 as a proxy-reproduction check rather than retrieval-quality proof, make the abstract’s significance language accurate, and release a verifiable code artifact. With those changes, the modest ROUGE claim would stand on much firmer ground.","headline":"A coherent, model-agnostic retrieval plug-in for MDS with a genuinely new EDU-level ranking/filtering combination, but the paper's internal retrieval analysis is circular and the end-to-end comparison does not isolate the mechanism.","tokens_in":21049,"tokens_out":2392,"would_cite":true,"duration_ms":23569,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ReREF replaces naive truncation with document ranking plus EDU filtering and reports consistent ROUGE gains across seven summarizers on four MDS datasets.","keywords":["multi-document summarization","retrieval framework","elementary discourse units","document ranking","EDU filtering","latent queries","expectation-maximization","few-shot summarization"],"falsifier":"Build a test set whose reference summaries paraphrase the source in vocabulary that the embedding model scores as low-similarity to the informative EDUs; if ReREF's selected inputs then fail to beat truncation while an oracle that selects EDUs by their marginal ROUGE contribution succeeds, the embedding-similarity training signal is the weak link.","tokens_in":19978,"feed_emoji":"✂️","tokens_out":8252,"duration_ms":74167,"temperature":0.7,"pith_summary":"This paper tries to establish that the usual way of squeezing multi-document inputs into a summarizer's fixed context window, truncating tokens, is the right thing to replace, and that a retrieval front end can do better if it both picks what to keep and decides which documents matter. To avoid hand-written queries, ReREF treats the most salient elementary discourse units, minimal coherent text units smaller than sentences, as latent queries, ranks documents against them, and filters out low-salience EDUs rather than trimming the end of the input. If the claim holds, any existing summarizer can be improved without retraining it, simply by serving it a better-selected and better-ordered slice of the source text, which matters for news, scientific, and legal summarization. The evidence is consistent ROUGE gains across seven summarizers and four MDS datasets, plus human ratings of informativeness and fluency.","feed_headline":"One retrieval front end lifts ROUGE for seven summarizers","feed_subtitle":"ReREF picks its own queries from the documents and deletes irrelevant sub-sentence units before summarization.","key_machinery":"The central object is the elementary discourse unit (EDU), a minimal coherent segment of text, smaller than a sentence, obtained with the DMRST discourse parser. The machinery has three moving parts: a filtering model that scores every EDU by salience and takes the top-scoring ones as latent queries; a ranking model that scores each document by averaging its dot-product similarity to all latent queries; and an EM loop that alternates between choosing the latent queries (E-step) and updating both models through two Bayesian Personalized Ranking losses (M-step). Training labels come from cosine similarity between EDUs or documents and the reference summary, computed with a pre-trained semantic-search embedding model. This design lets query selection, ranking, and filtering share one representation space and one optimization procedure.","core_discovery":"ReREF claims that the retrieve-then-summarize bottleneck in multi-document summarization can be broken without human-written queries or coarse passage/sentence retrieval. The framework picks the top-scoring elementary discourse units (EDUs), minimal coherent text segments smaller than sentences, from the input documents and uses them as latent queries to rank the documents; it then replaces naive token truncation with filtering that removes the lowest-scoring EDUs until the input fits the summarizer's context window. An expectation-maximization loop alternates between selecting these latent queries and updating the filtering and ranking models. Trained this way and applied as a model-agnostic front end, ReREF reports consistent ROUGE gains over the original truncation for seven summarizers on Multi-News, Multi-XScience, Wikisum, and WCEP-10, in both fully supervised and 1% few-shot settings, and human evaluation on 100 Multi-News samples shows improved informativeness and fluency with unchanged succinctness.","pith_inferences":["A natural next test the paper does not run is cross-domain transfer: since training labels come from reference summaries, a retriever trained on news data may need retraining or substitution before it works on legal or medical MDS.","The same EDU-filtering mechanism could be used as a denoiser for models with very long context windows, not just as a way to fit a length limit; the paper's core mechanism targets irrelevant content, not merely token budget.","Because the latent queries are just embeddings of selected EDUs, an explicit user query could be injected as an additional query embedding, which would convert ReREF into a query-focused multi-document summarizer without changing the EM loop."],"forward_implications":["Because ReREF is model-agnostic and applied before the summarizer, a trained ReREF front end could upgrade any fixed-context summarizer without changing its parameters.","The ablation results imply that document ranking and EDU filtering contribute separately: dropping either lowers ROUGE, and dropping both lowers it further, so both mechanisms are needed for the full gain.","In few-shot settings with only 1% of training data, ReREF still improves most model-dataset pairs, with the largest reported jump being PEGASUS gaining +7.47 ROUGE-2 on Wikisum.","In retrieval evaluation, ReREF beats BM25 for both query selection and filtering even when BM25 is given a gold query derived from the reference summary, so the learned latent queries carry information beyond keyword overlap."],"supporting_citations":[{"why":"Supplies the EM algorithm that alternates between latent query selection and model updates.","marker":"[10]"},{"why":"Provides the Longformer encoder used to embed tokens, EDUs, and documents.","marker":"[2]"},{"why":"Supplies the DMRST parser that segments documents into EDUs, the framework's retrieval units.","marker":"[34]"},{"why":"Provides the pre-trained embedding model whose cosine similarities create ground-truth EDU and document scores.","marker":"[50]"},{"why":"Provides the BPR ranking loss used to supervise both EDU filtering and document ranking.","marker":"[51]"},{"why":"Provides PRIMERA, the main baseline summarizer and the source of the Wikisum data split.","marker":"[61]"},{"why":"Provides the Multi-News dataset and the even-truncation baseline convention used for comparison.","marker":"[13]"}],"fun_headline_variants":["Self-found queries and EDU pruning boost multi-doc summaries","No manual queries: EDU selection lifts summarization ROUGE","ReREF: auto-queries + fine-grained filtering for MDS","Unified retrieval: rank docs, drop irrelevant EDUs","EDU-driven queries improve multi-document summarization"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole framework is trained and judged against one proxy: how similar each piece of text or whole document is to the reference summary under a pre-trained semantic-similarity model; if that measure does not track what a summarizer actually needs, the reported gains may not transfer.","fun_headline_variants_meta":{"raw":{"variants":["Self-found queries and EDU pruning boost multi-doc summaries","No manual queries: EDU selection lifts summarization ROUGE","ReREF: auto-queries + fine-grained filtering for MDS","Unified retrieval: rank docs, drop irrelevant EDUs","EDU-driven queries improve multi-document summarization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00024,"raw_usage":{"total_tokens":1533,"prompt_tokens":978,"completion_tokens":555,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":594,"completion_tokens_details":{"reasoning_tokens":472}},"tokens_in":594,"tokens_out":555,"duration_ms":5269,"temperature":1.0,"reasoning_tokens":472,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:56:55.818162+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a test set whose reference summaries paraphrase the source in vocabulary that the embedding model scores as low-similarity to the informative EDUs; if ReREF's selected inputs then fail to beat truncation while an oracle that selects EDUs by their marginal ROUGE contribution succeeds, the embedding-similarity training signal is the weak link.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the EM algorithm that alternates between latent query selection and model updates."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the DMRST parser that segments documents into EDUs, the framework's retrieval units."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides PRIMERA, the main baseline summarizer and the source of the Wikisum data split."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Multi-News dataset and the even-truncation baseline convention used for comparison."}],"review_version":1}