{"id":"f21f9ed7-baf7-4361-9eb3-18caa6868426","arxiv_id":"2607.20926","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A 103-task expert-curated benchmark shows that current LLMs and agents handle simple scientific lookup but mostly fail at ambiguous retrieval, citation grounding, and structured cross-source synthesis.","lead":"SciExplore is a new expert-built benchmark that tests AI research assistants on four progressively harder scientific search and synthesis tasks, from navigating databases to building comparison tables. Across 12 leading LLMs and agents, even the best system scores below 50% overall, and all systems nearly fail the most complex synthesis task.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Answer-uniqueness is unverified for open-ended T3/T4; low synthesis scores may reflect task under-specification rather than model inability.","rationale":"The reader's weakest assumption—that benchmark validity depends on answer uniqueness and non-shortcutability—is exactly the load-bearing point. I agree with the reader's identification of §3.3 as the crux. My concern sharpens this for T3/T4: these tasks are intrinsically open-ended, so the uniqueness screen is both more important and less credible there. The paper's own error analysis demonstrates that search-enabled LLMs, which are used in the QC stage, are poor at exhaustive retrieval and can hallucinate support for a hypothesis; thus they cannot be relied on to discover plausible alternative answers in obscure scientific areas. Additionally, the 10-minute human screen, if applied to T4, would make it impossible for an expert to even construct a full comparison table, let alone verify uniqueness; the paper does not provide task-specific QC details. The consequence is that low T4 scores may overstate model failure. This concern is directly testable via an inter-annotator agreement study, which is the standard approach for evaluating answer uniqueness in open-ended benchmarks. Since the reader already reached CONDITIONAL, and this concern is a verification gap rather than a demonstrated error, the verdict remains CONDITIONAL; no adjustment is needed.","tokens_in":20391,"tokens_out":4035,"duration_ms":39365,"concrete_test":"Re-run the uniqueness screen on all 14 T3 and 18 T4 tasks using three independent domain experts per task, each asked to produce the complete set of valid answers (all supporting references for each citation slot; all defensible table rows for the given schema) without seeing the gold. Exclude any task with inter-annotator agreement below 0.8 (e.g., Jaccard on answer sets). Then recompute OpenAI Deep Research's T3/T4 scores on the filtered set. If T4 item recall increases from 30.14% to above 50% or overall exceeds 60%, the headline gap is substantially inflated. If scores remain below 35% item recall, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that current agents are unreliable because they score 49.39% overall and 18.59% row recall on T4—presumes the benchmark measures a well-defined ability. That presumption rests on the answer-uniqueness guarantee in §3.3. For T4 (Cross-Source Structured Knowledge Synthesis) the guarantee is not credible. The task shown in Fig. 17 asks for 'representative LLMs applied in bioinformatics in 2024' with no selection criteria; any defensible set of models could be valid, yet the gold table is a fixed 17-row list. The QC protocol (Stage 3) attempts to discard tasks for which 'search-enabled LLMs plus human reviewers find any plausible alternative,' but the paper's own error analysis (§5.2, Fig. 8-12) shows these LLMs prematurely abandon searches and hallucinate constraints. They are therefore unlikely to surface valid alternatives in obscure scientific subdomains. Stage 1's 10-minute expert screen is also plausibly inapplicable to T4, where constructing a full comparison table takes much longer than 10 minutes; the text does not explain how uniqueness was verified for synthesis tasks. If T3/T4 contain tasks with multiple valid answers, the reported item/row recall conflates model incompleteness with task under-specification. The low T4 figures could then reflect an evaluation artifact rather than a capability gap.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"SciExplore is a new benchmark for evaluating LLMs and autonomous agents on scientific information-seeking. It consists of 103 expert-curated tasks distributed across four task types: T1 scientific database navigation, T2 ambiguous literature retrieval, T3 missing reference completion, and T4 cross-source structured knowledge synthesis. The tasks are claimed to form a progressive cognitive hierarchy from entity-level reasoning to domain-level synthesis. The authors evaluate 13 model/agent configurations, including foundation LLMs, search-augmented LLMs, and deep research agents. They report a sharp performance drop as task complexity increases: the best system, OpenAI Deep Research, attains 49.39% overall, while T4 row-level recall is only 18.59%. The paper concludes that current systems are far from reliable autonomous scientific assistants, especially for multi-source structured synthesis.","tokens_in":20672,"tokens_out":5660,"duration_ms":60857,"significance":"If the benchmark's validity is established, SciExplore addresses an important gap: most existing benchmarks evaluate either general-domain retrieval or static scientific QA, whereas SciExplore attempts to measure the full chain of database navigation, literature disambiguation, evidence grounding, and structured synthesis. The task design is ambitious, the domain coverage is broad, and the authors include useful appendix material such as prompt templates and a run-to-run variance analysis. The central empirical finding — that performance degrades markedly on synthesis tasks — is plausible and would be informative to the community. However, the benchmark's two most difficult task types (T3 and T4) depend on answer-uniqueness and LLM-based scoring assumptions that are not adequately validated, and the benchmark data are not released. These issues are load-bearing for the paper's central claim and prevent the current version from being accepted as a definitive benchmark.","major_comments":[{"comment":"The answer-uniqueness guarantee is not credible for T4, and only weakly for T3. The QC protocol screens tasks by an expert 10-minute search limit and by asking search-enabled LLMs plus human reviewers to find alternative answers. For T4, constructing a full comparison table typically takes far longer than 10 minutes, and the paper's own error analysis (§5.2) shows that these same LLMs prematurely abandon searches and hallucinate constraints, making them unlikely to surface valid alternatives. The T4 example in Fig. 17 asks for 'representative LLMs applied in bioinformatics in 2024' with no explicit selection criteria, yet the gold answer is a fixed 17-row table. Many defensible alternative tables exist. The paper does not report how many candidate tasks were discarded in Stage 3, nor any human verification specific to T3/T4. Without this, low T4 scores (Table 3) conflate model incomplete","section":"Section 3.3 / Fig. 17"},{"comment":"The benchmark data are not released. No URL, dataset download, license, or data card is provided. For a paper whose main contribution is a benchmark, this is a major omission: readers cannot inspect the 103 tasks, the gold answers, or the agent trajectories, and the LLM-judged T3/T4 scores cannot be reproduced. A benchmark paper should include a release plan and a detailed data card in the final version.","section":"Throughout"},{"comment":"T3 and T4 scores are produced entirely by LLM judges, with no validation of the judge itself. There is no human inter-annotator agreement study, no error analysis of the judge's decisions, and no calibration against expert judgments. Because the headline claim is 'extremely low accuracy' on T4, the judge's strictness is load-bearing: an overly strict judge lowers scores, an overly lenient judge inflates them. The authors should report a human-judge comparison on a random sample and release the judge prompts and outputs.","section":"Appendix A.5.2 / A.5.3"},{"comment":"Human baseline performance is only reported for T1 multi-hop subsets; no human experts are evaluated on T2, T3, or T4. The Stage 1 '10-minute limit' is a filtering criterion, not a performance baseline. Without human expert scores on T3 and T4, the claim that these tasks are 'PhD-level' and that models 'cannot reliably' solve them is not calibrated. At least a small human-expert study on T3/T4 should be reported.","section":"Table 6 / Section 4.2"}],"minor_comments":[{"comment":"The overall-score weights w1=0.2, w2=0.2, w3=0.3, w4=0.3 are set without justification or sensitivity analysis. The headline '49.39% overall' is weight-dependent; the authors should report a sensitivity range or justify the weights by task difficulty.","section":"Eq. (8)"},{"comment":"The claim that 'increasing hop length does not consistently degrade performance' is based on very small per-bin sample sizes (e.g., 6-hop likely contains 4 tasks). The apparent non-monotonicity may be noise. Report per-bin task counts and a significance test.","section":"Table 6 / Section 5.1"},{"comment":"The gold T4 table includes 'RiNALMo' published on arXiv, while other rows are from Nature Methods or Science. The selection criterion for 'representative LLMs' is undefined, reinforcing the need for an explicit rubric in the task prompt and in the QC protocol.","section":"Fig. 17"},{"comment":"The introduction to the error analysis refers to trajectories from 'Gemini-DeepResearch or GPT-4o', but GPT-4o is not among the evaluated systems listed in Table 3. Use one of the evaluated models or clarify that GPT-4o is used only for illustration.","section":"Appendix A.7"},{"comment":"The run-to-run variance analysis uses four repeated runs, but for T3 and T4 (14 and 18 tasks per run) the variance estimates themselves are noisy. Report confidence intervals or raw per-run scores.","section":"A.6"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and important evaluation gap, and the empirical results are likely to attract attention. However, the missing data release and the unvalidated answer-uniqueness for T3/T4 are serious blockers for a benchmark contribution. The revision should release the benchmark (or a substantial sample), add human-expert validation for T3/T4 answer uniqueness and scoring, and provide human baselines. If the authors cannot release the data, I would lean toward rejection, as the benchmark's central artefact would be unavailable to the community."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look if you work on agent evaluation. SciExplore does something no prior benchmark does: it chains four task types that mirror a research workflow—database navigation, fuzzy literature retrieval, citation completion, and cross-source table synthesis—and shows that current agents fall apart as the tasks get more open-ended. The T4 result (best row recall under 19%) is the headline, and the error analysis (premature abandonment, hallucinated verification, long-context loss, schema non-compliance) names real failure modes that match what we see in practice. The run-to-run variance analysis is a nice touch; most benchmark papers don't bother.\n\nThe construction pipeline is thoughtful. Reverse trajectory for T1 and fuzzification with validation constraints for T2 are sensible ways to force retrieval rather than memorization. The three-stage QC protocol (difficulty calibration, human cross-validation, answer uniqueness) is the right instinct.\n\nNow the soft spots, in proportion. The dataset is not released, so all results are unverifiable from the paper alone. T3 and T4 are tiny—14 and 18 tasks—so the synthesis numbers rest on a handful of examples. Human baselines only exist for T1, which makes it hard to calibrate what 'hard' means for T4. And the answer-uniqueness guarantee, which the whole benchmark leans on, is weakest exactly at T4. The stress-test note is on target: the example in Fig. 17 asks for 'representative LLMs applied in bioinformatics in 2024' with no selection criteria; any defensible list could be valid, yet the gold table is fixed at 17 rows. The QC protocol relies on search-enabled LLMs and human reviewers to detect alternatives, but the paper's own error analysis shows those LLMs prematurely abandon and hallucinate in obscure scientific territory, so they are unlikely to surface valid alternatives. The 10-minute expert screen also reads like it was designed for lookup tasks, not for building a synthesis table from scratch. If T4 tasks admit multiple valid answers, the low recall figures conflate task under-specification with model inability.\n\nThat said, the central conclusion—that current agents are weak at structured cross-source synthesis—is probably robust. Even if some rows are debatable, the error cases show real difficulties with schema compliance and evidence extraction. But the exact numbers should be treated as upper bounds on performance until the data is out and the uniqueness audit is tightened.\n\nWho benefits: anyone building scientific agents or designing evaluation suites for retrieval and synthesis. It deserves a serious referee, but the review should push hard on releasing the data, expanding T3/T4, adding human baselines for all four types, and documenting how uniqueness was verified for synthesis tasks. With those fixes, this could be a standard benchmark.","headline":"A genuinely useful agent benchmark whose headline numbers rest on an answer-uniqueness guarantee that is weakest exactly where the conclusions are strongest.","tokens_in":21188,"tokens_out":3505,"would_cite":true,"duration_ms":33735,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new benchmark shows AI agents fail most realistic scientific research tasks, with the best scoring under half.","keywords":["SciExplore","LLM agents","scientific information seeking","benchmark","retrieval","knowledge synthesis","database navigation","evaluation"],"falsifier":"Re-running the benchmark with an independent set of expert annotators and an answer-uniqueness check would settle the central claim: if a substantial fraction of tasks admits multiple defensible answers or can be solved by experts in under 10 minutes, the benchmark's difficulty and uniqueness claims are undermined. Alternatively, a model that explicitly revises hypotheses and searches broadly could exceed the reported 49.39% ceiling, challenging the conclusion that current methods are fundamentally insufficient.","tokens_in":20226,"feed_emoji":"🔬","tokens_out":2821,"duration_ms":28917,"temperature":0.7,"pith_summary":"The paper introduces SciExplore, a benchmark of 103 expert-curated tasks that test whether LLMs and agents can perform realistic scientific information-seeking: navigating structured databases, retrieving ambiguous literature, completing missing citations, and synthesizing knowledge from many sources into structured tables. The authors argue these tasks form a progression from entity-level to domain-level reasoning, and that existing benchmarks do not capture this workflow. They evaluate more than ten state-of-the-art models and agents, finding that even the best system scores 49.39% overall, with especially poor performance on the synthesis task (30.14% item recall, 18.59% row recall). The paper's claim is that current retrieval and reasoning advances are not enough for autonomous scientific assistance; agents need persistent search, hypothesis revision, and schema-constrained synthesis. A sympathetic reader cares because it identifies a concrete capability gap and points to where to improve.","feed_headline":"Best AI agent scores under 50% on scientific research benchmark","feed_subtitle":"Four-tier benchmark spans database navigation to structured synthesis; top agent reaches 49.39% while the hardest task stays near 18% row re","key_machinery":"The central object is the SciExplore benchmark itself, with its expert-driven construction and three-stage quality control. Tasks are built by techniques such as Reverse Trajectory Construction for database navigation, Feature Denoising and Fuzzification with validation constraint injection for ambiguous retrieval, claim-evidence rewriting for missing references, and expert-curated comparison schemas for synthesis. Quality control retains tasks only if expert annotators cannot solve them in 10 minutes and if search-enabled LLMs plus human reviewers find no alternative answers, enforcing answer uniqueness and resistance to memorization.","core_discovery":"The central discovery is a measured capability gap: current LLMs and autonomous agents cannot reliably complete realistic scientific information-seeking workflows. SciExplore organizes this workflow into four hierarchical task types—scientific database navigation, ambiguous literature retrieval, missing reference completion, and cross-source structured knowledge synthesis—and evaluates twelve systems. The best-scoring deep research agent reaches only 49.39% overall; performance drops sharply with task complexity, and the highest-level synthesis task is almost unsolved, with even the top agent achieving 30.14% item-level recall and 18.59% row-level recall. The paper attributes these failures","pith_inferences":["The quality-control protocol implies tasks are screened against both human and LLM solvability; if that screen is imperfect, the reported failure rates may overstate model inability on realistic but easier variants.","The finding that hop length does not predict difficulty suggests that benchmark designers should measure cognitive complexity rather than surface-level search depth.","The very low row-level recall on T4 is a transferable warning: any automated pipeline that depends on strict schemas will likely fail unless agents are trained with explicit schema constraints."],"forward_implications":["If the benchmark is valid, retrieval augmentation alone will not make agents into scientific assistants; they need search breadth, persistence, and schema-constrained synthesis.","The task hierarchy offers a diagnostic: T4 row-level recall exposes that local correctness does not scale to global structured output.","The identified failure modes—premature abandonment, hallucination, long-context loss, and instruction mismatch—give concrete targets for model development.","The correlation between search frequency and accuracy suggests agents should be designed to explore and verify repeatedly rather than relying on shallow retrieval."],"fun_headline_variants":["Top AI agent fails most scientific research tasks","AI agents struggle with scientific info synthesis: 49% max score","Scientific research benchmark stumps AI agents, top score under 50%","Benchmark shows AI agents lag in real-world scientific quests","Science info-seeking benchmark: best agent scores 49.39%"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The benchmark's validity assumes that every task has exactly one correct answer that cannot be obtained by memorization or shortcut; if the 10-minute expert screen or LLM-based alternative-answer search misses ambiguous tasks, the reported failure rates would overstate model inability.","fun_headline_variants_meta":{"raw":{"variants":["Top AI agent fails most scientific research tasks","AI agents struggle with scientific info synthesis: 49% max score","Scientific research benchmark stumps AI agents, top score under 50%","Benchmark shows AI agents lag in real-world scientific quests","Science info-seeking benchmark: best agent scores 49.39%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000596,"raw_usage":{"total_tokens":2590,"prompt_tokens":672,"completion_tokens":1918,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":416,"completion_tokens_details":{"reasoning_tokens":1832}},"tokens_in":416,"tokens_out":1918,"duration_ms":11999,"temperature":1.0,"reasoning_tokens":1832,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T08:57:35.077475+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-running the benchmark with an independent set of expert annotators and an answer-uniqueness check would settle the central claim: if a substantial fraction of tasks admits multiple defensible answers or can be solved by experts in under 10 minutes, the benchmark's difficulty and uniqueness claims are undermined. Alternatively, a model that explicitly revises hypotheses and searches broadly could exceed the reported 49.39% ceiling, challenging the conclusion that current methods are fundamentally insufficient.","supporting_citations":[],"review_version":1}