{"id":"45445339-82ab-4b20-bce2-5cf1bb96fd08","arxiv_id":"2505.06186","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"CochraneForest is a new benchmark for document-level evidence extraction, and URCA, which combines uniform retrieval and clustering, achieves up to 10.3% higher F1 than prior RAG baselines on it.","lead":"The paper introduces CochraneForest, a dataset of 202 forest plots from Cochrane systematic reviews paired with clinical questions and full-text studies, for the task of extracting study-level evidence conclusions. It also presents URCA, a retrieval-augmented generation method that reports F1 improvements of up to 10.3% over existing baselines, and aims to support automated evidence synthesis.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"URCA's retrieval budget is not matched to baselines: the uniform-retrieval rule retrieves more chunks than the top-10 baselines, so the headline F1 gain may reflect extra context, not the method.","rationale":"The reader's CONDITIONAL verdict is the right one, but the reader's weakest-assumption (pre-filtered studies) concerns the applicability of the task setup rather than the internal validity of the headline comparison. The budget mismatch is more directly load-bearing: it threatens the numerical claim itself, since URCA may simply be given more retrieved text than the baselines. This is not an accusation of misconduct; it is a concrete, fixable methodological gap. The paper does not report beta or N_max, so the effect cannot be checked post hoc, and the ablation in Table 5 does not isolate source diversity from retrieval quantity. A budget-matched rerun would settle the issue. I keep the reader's CONDITIONAL verdict because the concern is addressable and does not by itself prove the method is no better than baselines; it does add a specific condition that should be met before the 10.3% claim is taken at face value. The reader's other concerns (no code/data release, no significance testing, CI-derived labels) remain valid and also support a conditional rather than unconditional acceptance.","tokens_in":16439,"tokens_out":6708,"duration_ms":71094,"concrete_test":"Rerun the CochraneForest experiments (at minimum for GPT-3.5-turbo, where the headline gap is largest, and Llama-3.1-70B, where the gap is smallest) with the total number of retrieved chunks capped at exactly k=10 for URCA, for example by setting beta=0 and allocating per-source retrievals as floor(k/S) plus leftovers, or by post-hoc truncating URCA's retrieved set to the top 10 ranked chunks. Compare F1 against vanilla RAG with top-10 retrieval and against GraphRAG under the identical total budget. Also run the mirror condition: give RAG the same per-source allocation and total chunk count that URCA actually uses in the current paper. If URCA's advantage over GraphRAG/RAG largely disappears under budget matching, the central claim should be restated as \"URCA helps when more retrieved context is available,\" not as unconditional superiority.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Abstract: \"URCA outperforms the best existing methods by up to 10.3% in F1 score on this task\") is a comparative statement, so the comparison must be budget-controlled. Section 5.1 states that \"by default, we use the top 10 retrieved passages in all the approaches under comparison.\" URCA's uniform-retrieval rule, Algorithm 1 line 1, sets k_s = ceil(min(k + beta*log(S), N_max)/S) for each of S sources, so the number of passages actually placed in context is S * ceil((k + beta*log(S))/S), which is at least k + beta*log(S) and is strictly greater than k whenever beta > 0 or S does not divide k + beta*log(S). The paper never reports beta or N_max; Appendix F only says that all other hyperparameters were left at defaults, so the effective context size for URCA cannot be reconstructed from the paper. The ablation \"w/o Uniform Retrieval\" in Table 5 compares URCA against a version that retrieves the top chunks without source allocation, so it changes the number of chunks as well as the source balance. Consequently, the reported gains over RAG and GraphRAG could be an artifact of giving URCA more retrieved evidence per query, rather than evidence that its clustering and extraction strategy is superior. This confound directly underpins the headline result, making it the most load-bearing weakness in the paper.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces COCHRANEFOREST, a dataset of 202 forest plots from 48 Cochrane systematic reviews, with 923 research-question--study pairs and full texts of 263 unique studies, annotated with study conclusions relative to a clinical question. It formalizes the task of document-level evidence extraction from studies with conflicting conclusions, and proposes URCA, a RAG framework that retrieves a source-balanced set of passages, clusters them via UMAP+GMM, extracts query-relevant evidence with an LLM, and produces a final answer. Experiments on four LLMs report F1/accuracy gains over No RAG, Abstracts, vanilla RAG (with and without uniform retrieval), RAPTOR, InstructRAG, and GraphRAG, plus additional results on PubMedQA and MedQA-US. The paper includes ablations, cluster-ordering tests, inter-annotator agreement, and a qualitative example.","tokens_in":16686,"tokens_out":8047,"duration_ms":75111,"significance":"If the empirical claims hold, COCHRANEFOREST is a useful and genuinely document-level benchmark that goes beyond abstract-level evidence extraction, and URCA is a simple, explainable RAG baseline with a plausible design rationale (source-balanced retrieval plus query-guided clustering). The authors provide several strengths: a substantial annotation effort with IAA reporting, ablations of the two main components, tests on two external QA benchmarks, and a qualitative trace of the pipeline. However, the central comparative claim is currently undercut by a retrieval-budget confound: URCA appears to place more passages in context than the baselines, so the reported F1 gains may not reflect the method's intrinsic merit. The missing hyperparameter details and absence of any variance estimate further weaken the claim of consistency. The contribution is promising and within the journal's scope, but the comparison must be made fair and reproducible before the headline result can be accepted.","major_comments":[{"comment":"The comparison in Table 3 is not retrieval-budget-controlled. The setup states that 'by default, we use the top 10 retrieved passages in all the approaches under comparison' (Section 5.1), but Algorithm 1 sets k_s = ceil(min(k + β·log(S), N_max)/S) and therefore retrieves S·k_s passages. For k=10, S≥2 and β>0, this is strictly greater than 10, so URCA receives more retrieved evidence than the top-10 baselines. Since β, N_max, and the resulting average context size are never reported (Appendix F only says other hyperparameters were left at defaults), the reader cannot tell whether the F1 gains come from the method or from extra context. The 'w/o Uniform Retrieval' ablation in Table 5 changes the number of chunks as well as the source allocation, so it does not isolate the uniform-retrieval mechanism. Please re-run the experiments with a matched total budget (equal number of passages or equal token budget per query), report β/N_max and average context sizes, and include a URCA variant that uses exactly k passages in total.","section":"§5.1, Algorithm 1"},{"comment":"URCA is not reproducible from the information given. The paper does not report β or N_max, which directly control how many passages Algorithm 1 retrieves, nor does it specify the embedding model, passage chunk size/overlap, UMAP parameters, or the random seed used for UMAP and GMM clustering. The statement in Appendix F that 'all the other hyperparameters were left to the default value' is insufficient because these parameters are not defaults of any named library that can be identified from the paper. Since clustering is reported as the largest contributor to performance (Table 5), the stochastic components of clustering need fixed seeds or repeated runs. Please provide a complete configuration table or a link to a released code repository.","section":"Appendix F, Algorithm 1"},{"comment":"All reported numbers are single point estimates with no error bars, confidence intervals, or significance tests. Several of the decisive comparisons are small in absolute terms: on Llama-3.1-70B, URCA's F1 is 66.1 versus GraphRAG's 65.6, and the uniform-retrieval ablation differences are between 1.6 and 2.4 F1 points. Given the stochasticity of UMAP/GMM clustering and the use of API-based LLMs, a few repeated runs with different seeds, or a paired significance test, are needed to support the claim that URCA 'consistently outperforms' the baselines across all four models.","section":"§5.2, Tables 3 and 5"}],"minor_comments":[{"comment":"The phrase 'up to 10.3% in F1 score' should be made unambiguous: the 10.3% is a relative improvement over GraphRAG on GPT-3.5-Turbo, not 10.3 absolute F1 points; please state both relative and absolute values.","section":"Abstract, §5.2"},{"comment":"The sentence 'by default, we use the top 10 retrieved passages in all the approaches under comparison' is inconsistent with the retrieval formula in Algorithm 1; please define the value of k used for URCA and clarify how the top-10 budget applies to the uniform-retrieval variants.","section":"§5.1"},{"comment":"The Fleiss κ of 0.06 for Task 1 is reported alongside cosine similarity 0.95, but the paper does not explain why the two metrics diverge so sharply; since pairwise φ values show that annotators changed 33–100% of the items, a short discussion of what the κ is measuring would help readers judge the reliability of the research-question annotations.","section":"§3.3, Table 2"},{"comment":"The text notes that no annotator modified the pre-selected conclusion label (Task 2); this means the conclusion labels were effectively derived deterministically from confidence intervals, so the reported inter-annotator agreement does not validate the label assignment itself. Please state this explicitly.","section":"§3.2"},{"comment":"There is a typo in 'gpt-3.5.-turbo' (should be 'gpt-3.5-turbo').","section":"Appendix F"},{"comment":"The paper says the dataset is released but provides no URL or access mechanism for the dataset or code; please add repository links or a clear availability statement.","section":"§3.1 / Appendices"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: CochraneForest is a real contribution—202 forest plots with full texts and study-level conclusions for questions with conflicting evidence—and URCA is a reasonable baseline for it. But the main comparison is under-specified. Section 5.1 says all approaches use the top 10 retrieved passages, yet URCA's uniform-retrieval rule retrieves S * ceil((k + beta*log(S))/S) chunks, which is at least k and often more (e.g., S=3 gives 12 even with beta=0). With beta and N_max unreported, the effective context size for URCA cannot be reconstructed. So the headline gain over GraphRAG (up to 10.3%) may partly reflect more context, not a better method. The \"w/o Uniform Retrieval\" ablation changes both source balance and chunk count, so it doesn't isolate the effect. This is the softest spot in the paper.\n\nWhat's actually good: the task formalization is new and needed—predicting study-level conclusions from multiple full-text papers is not covered by prior abstract- or passage-level benchmarks. The dataset construction is transparent, and the CI-based labeling rule is simple and reproducible. The ablations show clustering matters more than uniform retrieval, which is an informative finding. The open-domain QA results (PubMedQA, MedQA-US) are a nice sanity check, though they drop uniform retrieval for good reason.\n\nOther soft spots, in proportion: no significance tests or error bars, so the 1–4% F1 gaps could be noise; no code or data link in the paper; hyperparameters (beta, N_max, UMAP seed) are missing. The task assumes relevant studies are pre-filtered, so gains won't automatically transfer to open-ended evidence extraction—but the paper says this explicitly, so it's a stated limitation rather than a hidden one. Minor: Fleiss kappa on Task 1 is 0.06, though the 0.95 cosine similarity is more meaningful for an editing task.\n\nWho this is for: people working on biomedical evidence synthesis, systematic review automation, or RAG evaluation. The dataset could become a standard benchmark if released. For referees: send to peer review, yes, but ask for the dataset and code, and require a budget-matched comparison—either fix URCA's retrieval count to k or give baselines the same number of chunks. The central idea is solid; the empirical claim needs a cleaner experiment.","headline":"A genuinely useful new dataset and a plausible method, but the headline RAG comparison is not budget-controlled, so the 10.3% claim should be read with caution.","tokens_in":17291,"tokens_out":2358,"would_cite":true,"duration_ms":23069,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Evenly sampling every paper in a study, clustering the passages, and distilling each cluster before generating the conclusion predicts study conclusions up to 10.3% F1 more accurately than prior methods, on a new Cochrane-derived benchmark.","keywords":["scientific evidence extraction","retrieval-augmented generation","forest plots","systematic reviews","clinical research questions","full-text biomedical NLP","randomized controlled trials","contradictory evidence"],"falsifier":"Run URCA on the same 202 research questions without giving it the included-study set, letting it select studies from the full Cochrane corpus; if its F1 advantage over GraphRAG closes or reverses, the reported gains depend on the pre-filtered study assumption and would not transfer to open-domain evidence extraction.","tokens_in":16184,"feed_emoji":"🩺","tokens_out":10098,"duration_ms":87149,"temperature":0.7,"pith_summary":"The paper formalises a new task in automated evidence synthesis: given a clinical research question and the full texts of the papers that make up a set of studies, predict each study's conclusion—favours the intervention, favours the control, or no difference. To support the task, the authors built CochraneForest, a dataset of 202 forest plots from 48 Cochrane systematic reviews with 923 question–study pairs, restricted to plots where at least two studies reach contradictory conclusions. They then propose URCA, a retrieval-augmented generation framework that distributes the retrieval budget evenly across a study's papers, clusters the retrieved passages by meaning, uses a language model to extract query-relevant evidence from each cluster, and only then generates the conclusion; on CochraneForest it beats seven baselines across four LLMs by up to 10.3% F1, and it also improves accuracy on the open-domain sets PubMedQA and MedQA-US. The paper's message is that document-level evidence extraction is a difficult, well-defined testbed for evidence synthesis, and that the clustering step—not the uniform retrieval—carries most of URCA's advantage.","feed_headline":"Retrieval system finds study conclusions 10.3% better than rivals","feed_subtitle":"Reads full-text trials, balances evidence across papers, then clusters passages before deciding.","key_machinery":"The central mechanism is the URCA pipeline, a retrieval-augmented generation framework whose key steps are uniform source allocation, embedding clustering, and per-cluster LLM extraction. Uniform allocation retrieves $k_s = \\lceil \\min(k+\\beta \\log S, N_{\\max})/S \\rceil$ passages from each of the $S$ papers in a study, so that no single paper can dominate the context. The retrieved passages are then clustered with UMAP dimensionality reduction and a Gaussian mixture model selected by the Bayesian Information Criterion, and a language model is prompted to extract information relevant to the research question from each cluster. Finally, the extracted cluster summaries are concatenated and given to the model to produce the study conclusion. The paper's ablations establish clustering as the load-bearing component: replacing it with contiguous grouping of shuffled chunks degrades performance, while altering the order in which clusters are presented changes F1 by only about $\\pm 1\\%$.","core_discovery":"The central claim is that evidence extraction from multi-paper studies is best served by a retrieval design that treats each paper as an equally important source and defers synthesis until the retrieved content has been grouped by meaning. URCA follows this design: it allocates a computed number of passage retrievals to every paper in a study, clusters all retrieved passages with UMAP and a Gaussian mixture model, prompts a language model to extract query-relevant evidence from each cluster, and only then prompts the model to produce the study conclusion. The paper reports that this pipeline outperforms every baseline on CochraneForest—No RAG, abstract-only context, vanilla RAG with and without uniform retrieval, RAPTOR, InstructRAG, and GraphRAG—across Llama-3.1-70B, Mistral Large, GPT-3.5 Turbo, and GPT-4, with the largest gain (10.3% F1) over GraphRAG on GPT-3.5 Turbo. Ablations show that removing clustering costs 3.9–5.7 absolute F1 points, while removing uniform retrieval costs only 1.6–2.4 points. The same cluster-then-extract step, without uniform retrieval, transfers to open-domain QA, reaching 85.9% accuracy on MedQA-US and 81.1% on PubMedQA with GPT-4.","pith_inferences":["Since the paper reports that no annotator ever modified the conclusion label pre-selected from the study's 95% confidence interval, the labels are near-deterministic functions of reported effect sizes; a cheap quantitative baseline that reads point estimates and intervals from tables could rival or exceed RAG systems on this benchmark.","Because the dataset lacks rationale annotations identifying which passages support each conclusion, a natural extension is to post-hoc align the retrieved clusters (or their attention weights) to the final prediction, producing explainable evidence trails and enabling error analysis.","The uniform-retrieval benefit is small on CochraneForest (papers per study average 1.82), so its value may grow on corpora with more uneven multi-paper studies; a version of the benchmark sampling studies with many papers would test this prediction.","URCA's largest relative gain appears with GPT-3.5 Turbo (10.3% F1) and is smaller with GPT-4, suggesting weaker models benefit more from the structured, clustered context; this predicts the method will be most useful with small, open-weight models in constrained settings."],"forward_implications":["Automated evidence synthesis could seed forest plots: given a clinical question and the included studies' full texts, a review team can obtain per-study conclusions to populate the plot and focus manual work on verification.","Full-text evidence is necessary for outcome-specific questions; abstract-only and no-retrieval baselines are strong but consistently below URCA, because the required nuance often lives in tables and analyses beyond the abstract.","Clustering is the component to invest in: it accounts for the larger share of URCA's gains on CochraneForest, while uniform retrieval mainly improves coverage of less prominent papers.","The cluster-then-extract design generalises beyond systematic reviews: on PubMedQA and MedQA-US, URCA (without uniform retrieval) outperforms vanilla RAG and GraphRAG, suggesting the same pipeline fits open-domain medical QA.","CochraneForest provides the research community with a benchmark of 923 question–study pairs centered on conflicting evidence, a regime that existing methods handle poorly."],"supporting_citations":[{"why":"Quantifies the time and cost of systematic reviews, motivating the automation task.","marker":"(Michelson and Reuter, 2019)"},{"why":"Cochrane Handbook defining systematic reviews and forest plots, the structural basis of CochraneForest.","marker":"(Higgins et al., 2024)"},{"why":"Closest prior task of inferring reported findings from full-text RCTs, which URCA extends to cross-document synthesis.","marker":"(Lehman et al., 2019)"},{"why":"RAPTOR supplies the clustering methodology (UMAP, GMM, BIC) used in URCA and serves as a baseline.","marker":"(Sarthi et al., 2024)"},{"why":"GraphRAG is the strongest baseline, beaten by up to 10.3% F1.","marker":"(Edge et al., 2024)"},{"why":"InstructRAG baseline, showing instruction-based rationales underperform on domain-specific evidence extraction.","marker":"(Wei et al., 2025)"},{"why":"PubMedQA, one of the two open-domain medical QA datasets used to test URCA's generalisation.","marker":"(Jin et al., 2019)"},{"why":"MedQA-US, the harder medical licensing exam QA dataset used to test cross-domain generalisation.","marker":"(Jin et al., 2021)"}],"fun_headline_variants":["Clustered retrieval beats rivals by 10.3% F1 in evidence extraction","URCA: uniform retrieval plus clustering lifts evidence F1 by 10.3%","New method extracts study conclusions 10.3% better than GraphRAG","Clustering passages after uniform reading boosts evidence extraction","Document-level evidence extraction improved 10.3% with URCA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The system is handed the set of studies that belong under each research question and the papers that make up each study, so it never has to decide which papers are relevant or whether a study belongs in the forest plot.","fun_headline_variants_meta":{"raw":{"variants":["Clustered retrieval beats rivals by 10.3% F1 in evidence extraction","URCA: uniform retrieval plus clustering lifts evidence F1 by 10.3%","New method extracts study conclusions 10.3% better than GraphRAG","Clustering passages after uniform reading boosts evidence extraction","Document-level evidence extraction improved 10.3% with URCA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000218,"raw_usage":{"total_tokens":1460,"prompt_tokens":988,"completion_tokens":472,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":604,"completion_tokens_details":{"reasoning_tokens":375}},"tokens_in":604,"tokens_out":472,"duration_ms":4346,"temperature":1.0,"reasoning_tokens":375,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:46:33.175824+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run URCA on the same 202 research questions without giving it the included-study set, letting it select studies from the full Cochrane corpus; if its F1 advantage over GraphRAG closes or reverses, the reported gains depend on the pre-filtered study assumption and would not transfer to open-domain evidence extraction.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Quantifies the time and cost of systematic reviews, motivating the automation task."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"RAPTOR supplies the clustering methodology (UMAP, GMM, BIC) used in URCA and serves as a baseline."}],"review_version":1}