{"id":"3810d269-0b33-4a5e-9223-26cbf27bd3e2","arxiv_id":"2607.01243","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"STRUCTSURVEY is a multi-agent system using dynamic graph construction for structured retrieval in automated survey paper generation, showing +2.9 ROUGE-1 and +1.0 ROUGE-2 recall gains over embedding baselines on an ACL survey benchmark.","lead":"The paper introduces STRUCTSURVEY, a hierarchical multi-agent framework that builds graph-based representations of research entities and relations to improve retrieval for generating scientific survey papers. If effective, this could help manage the growing volume of publications by producing more organized and accurate automated summaries.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Graph construction fidelity and benchmark representativeness lack direct validation beyond downstream metrics","rationale":"The reader's weakest_assumption directly identifies the unverified pipeline step and benchmark issue; the abstract-only review already flags unverifiable technical claims, and the reported metrics alone do not close the gap. No independent support (e.g., graph quality tables or external benchmark validation) appears in the provided claims.","tokens_in":1680,"tokens_out":339,"duration_ms":22527,"concrete_test":"Sample 5 papers from the benchmark; extract gold entities/relations/taxonomies from the human-written surveys; compute precision/recall/F1 of the system's constructed graphs against this gold; if relation F1 < 0.65 on average, the assumption fails and headline gains are not attributable to structure.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim requires that the hierarchical multi-agent graph construction (entities, relations, topical taxonomies) accurately extracts and organizes conceptual/methodological relations from retrieved papers, shifting reasoning from generation to retrieval. Evaluation reports only aggregate ROUGE-1/2 recall gains (+2.9/+1.0) and LLM-as-Judge improvements on a new ACL-survey benchmark versus embedding baselines, with no reported metrics on graph quality (e.g., relation extraction precision against gold taxonomies from the human surveys) or error analysis. If graphs systematically miss or misconnect key relations, the observed gains could stem from multi-agent prompting volume or retrieval volume rather than structural accuracy. The benchmark's construction (reference-grounded ACL papers) is also untested for selection bias or coverage of non-ACL domains.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript introduces STRUCTSURVEY, a hierarchical multi-agent framework for automated survey paper generation. It dynamically constructs graph-based representations of entities, relations, and topical taxonomies from retrieved papers, shifting structural reasoning from the generation stage to retrieval. The system is evaluated on a new reference-grounded benchmark consisting of ACL survey papers, where it reports average improvements of +2.9 ROUGE-1 recall and +1.0 ROUGE-2 recall over embedding-only baselines without loss of precision, plus higher LLM-as-a-Judge scores on logical structure, depth, and synthesis.","tokens_in":1833,"tokens_out":532,"duration_ms":21056,"significance":"If the central claims hold, the work provides evidence that explicit structural retrieval can improve the organization and synthesis quality of LLM-generated surveys relative to unstructured embedding retrieval. The new ACL-based benchmark contributes a reproducible testbed for long-form scientific summarization. These elements would be of interest to the information retrieval and NLP communities working on automated literature synthesis.","major_comments":[{"comment":"Evaluation section: The reported ROUGE-1/2 recall gains (+2.9 and +1.0) and LLM-as-a-Judge improvements are presented only as averages with no accompanying statistical significance tests, variance across runs, or details on baseline implementations and potential confounds such as retrieval volume or prompting differences. This leaves open whether the gains are attributable to the graph structure or to other factors.","section":"Evaluation"},{"comment":"Method and Experiments sections: The central claim that graph construction of entities, relations, and taxonomies accurately captures conceptual and methodological relations (thereby improving survey quality) lacks any direct validation metrics, such as relation extraction precision or graph fidelity against gold taxonomies derived from the human-written reference surveys. Without these, it is unclear whether the observed downstream gains stem from structural accuracy or from multi-agent prompting volume.","section":"Method"},{"comment":"Benchmark description: The construction details, selection criteria, and coverage analysis for the new reference-grounded ACL survey benchmark are not provided, leaving its representativeness for general long-form scientific summarization (beyond ACL) untested and potentially subject to domain-specific biases.","section":"Benchmark"}],"minor_comments":[{"comment":"The abstract states improvements 'on average' but does not specify the number of test instances or aggregation method; this should be clarified for reproducibility.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback on evaluation rigor, method validation, and benchmark transparency. We address each major comment below, indicating planned revisions to strengthen the manuscript.","responses":[{"response":"We agree that statistical analysis and implementation details are needed to support the claims. In the revised manuscript, we will report standard deviations across multiple runs (with different random seeds), include paired statistical significance tests (e.g., t-tests) on the ROUGE scores, and expand the baseline descriptions to specify retrieval volume, prompting templates, and other hyperparameters. This will help isolate the contribution of the graph-based structure.","revision_made":"yes","referee_comment":"[Evaluation] Evaluation section: The reported ROUGE-1/2 recall gains (+2.9 and +1.0) and LLM-as-a-Judge improvements are presented only as averages with no accompanying statistical significance tests, variance across runs, or details on baseline implementations and potential confounds such as retrieval volume or prompting differences. This leaves open whether the gains are attributable to the graph structure or to other factors."},{"response":"We acknowledge the value of direct graph validation metrics. However, creating gold-standard taxonomies from the reference surveys would require substantial new human annotation beyond the scope of the current study. In revision, we will add a partial analysis reporting aggregate statistics on extracted entities/relations and a qualitative comparison to reference survey structures, while noting the limitation that downstream LLM-as-a-Judge improvements on structure and synthesis serve as indirect evidence only. We will also discuss the potential role of multi-agent prompting volume as a confound.","revision_made":"partial","referee_comment":"[Method] Method and Experiments sections: The central claim that graph construction of entities, relations, and taxonomies accurately captures conceptual and methodological relations (thereby improving survey quality) lacks any direct validation metrics, such as relation extraction precision or graph fidelity against gold taxonomies derived from the human-written reference surveys. Without these, it is unclear whether the observed downstream gains stem from structural accuracy or from multi-agent prompting volume."},{"response":"We will revise the benchmark section to include explicit selection criteria (ACL surveys published 2018–2023 with citation thresholds), the process for identifying reference papers, and coverage statistics (e.g., average number of cited papers per survey and subfield distribution within NLP). We agree this benchmark is ACL/NLP-focused and will explicitly note its domain specificity as a limitation, while arguing it provides a reproducible testbed for the targeted community.","revision_made":"yes","referee_comment":"[Benchmark] Benchmark description: The construction details, selection criteria, and coverage analysis for the new reference-grounded ACL survey benchmark are not provided, leaving its representativeness for general long-form scientific summarization (beyond ACL) untested and potentially subject to domain-specific biases."}],"tokens_in":1422,"tokens_out":613,"duration_ms":27114,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"STRUCTSURVEY is a hierarchical multi-agent framework that builds graph representations of entities, relations, and topical taxonomies during retrieval instead of leaving structural work to the LLM at generation time.\n\nThe paper does a reasonable job of motivating the problem and describing the shift from unstructured retrieval. On their new ACL survey benchmark it reports average ROUGE-1 recall up by 2.9 and ROUGE-2 recall up by 1.0 with no loss in precision, plus higher LLM-as-judge scores on logical structure, depth, and synthesis. That suggests the extra structure can help in practice.\n\nThe soft spots are in the evaluation. The results are given only as averages with no variance, significance tests, or details on baseline implementations. There are no direct measures of graph quality, such as how well the extracted relations match the human surveys. Without that, it is hard to know whether the gains come from the graphs or simply from more retrieval steps and prompting. The benchmark is also new and limited to ACL papers, so its coverage and lack of selection bias are untested.\n\nThis work is aimed at people building retrieval-augmented systems for scientific summarization in IR and NLP. Someone working on agentic RAG or automated literature tools could get concrete ideas from the architecture.\n\nIt deserves a serious referee. The idea is a straightforward extension of existing methods and the benchmark could be reusable, even if the current evidence needs tighter controls and graph-level checks.","headline":"STRUCTSURVEY adds graph construction via multi-agent retrieval for survey generation and reports modest ROUGE gains, but the evidence stays at downstream metrics without checking graph quality.","tokens_in":2267,"tokens_out":375,"would_cite":false,"duration_ms":31014,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"A hierarchical multi-agent system builds entity-relation graphs at retrieval time to generate survey papers with higher recall and better logical structure than embedding baselines.","keywords":["structured retrieval","multi-agent framework","survey generation","graph representations","scientific summarization","ROUGE metrics","ACL benchmark","LLM judge evaluation"],"falsifier":"An experiment that supplies manually validated relation graphs for the same documents and shows that the automatically constructed graphs miss key methodological or taxonomic links, resulting in no gain or a drop in ROUGE recall and judge ratings on the ACL benchmark.","tokens_in":2590,"feed_emoji":"📊","tokens_out":713,"duration_ms":19138,"temperature":0.7,"pith_summary":"The paper presents STRUCTSURVEY as a framework that moves the task of inferring conceptual, methodological, and taxonomic relations out of the final generation step and into the retrieval process. It does this by having agents dynamically assemble graph representations of research entities and their connections from retrieved documents. Evaluated against embedding-only methods on a new benchmark of ACL survey papers, the approach raises average ROUGE-1 recall by 2.9 points and ROUGE-2 recall by 1.0 points while holding precision steady and lifting human-aligned ratings for organization and depth. A reader would care if the result holds because survey writing is currently limited by the difficulty of recovering explicit structure from raw text at generation time.","feed_headline":"Graph construction at retrieval raises survey recall by 3 points","feed_subtitle":"Multi-agent system assembles entity-relation graphs before generation, improving ROUGE recall and logical structure over embedding baselines","key_machinery":"Hierarchical multi-agent framework that dynamically constructs graph-based representations of entities, relations, and topical taxonomies during retrieval.","core_discovery":"STRUCTSURVEY is a hierarchical multi-agent framework that shifts structural reasoning from generation to retrieval by dynamically constructing graph-based representations of entities, relations, and topical taxonomies. On a reference-grounded benchmark of ACL survey papers, this yields surveys whose ROUGE-1 recall improves by +2.9 and ROUGE-2 recall by +1.0 on average without loss of precision, together with higher LLM-as-a-Judge scores for logical structure, depth, and synthesis relative to embedding-only retrieval baselines.","pith_inferences":["The same graph-construction step could be applied to other long-form synthesis tasks such as literature reviews in non-ACL domains.","If the constructed graphs prove reliable, downstream generation models may require less capacity for implicit reasoning.","The approach suggests that intermediate structured representations can serve as a general lever for improving factual grounding in multi-document summarization."],"forward_implications":["Surveys produced this way exhibit organization and reasoning closer to human-written examples.","Explicit structural retrieval improves both surface overlap metrics and higher-order quality judgments without trading one for the other.","The new reference-grounded ACL benchmark enables reproducible evaluation of long-form scientific summarization methods.","Moving relation inference to retrieval reduces the burden on the final generation model to discover structure from raw text."],"fun_headline_variants":["Agentic graphs at retrieval raise survey recall by 3 points","Multi-agent graphs build entity relations before survey writing","Shifting structure to retrieval raises ROUGE recall in surveys","Graph-based taxonomies from agents raise survey synthesis scores"],"cache_read_input_tokens":64,"weakest_assumption_plain":"Dynamically building graph representations from retrieved data accurately captures the conceptual, methodological, and taxonomic relations required for high-quality survey generation, and the ACL survey benchmark is representative of general long-form scientific summarization.","fun_headline_variants_meta":{"raw":{"variants":["Agentic graphs at retrieval raise survey recall by 3 points","Multi-agent graphs build entity relations before survey writing","Shifting structure to retrieval raises ROUGE recall in surveys","Graph-based taxonomies from agents raise survey synthesis scores"]},"model":"grok-4.3","cost_usd":0.009979,"raw_usage":{"total_tokens":4417,"prompt_tokens":636,"num_sources_used":0,"completion_tokens":63,"cost_in_usd_ticks":99787000,"prompt_tokens_details":{"text_tokens":636,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":3718,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":636,"tokens_out":63,"duration_ms":29240,"temperature":1.0,"reasoning_tokens":3718,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-04T01:17:07.753701+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"An experiment that supplies manually validated relation graphs for the same documents and shows that the automatically constructed graphs miss key methodological or taxonomic links, resulting in no gain or a drop in ROUGE recall and judge ratings on the ACL benchmark.","supporting_citations":[],"review_version":1}