Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

ADRA-Bank: A Modular Benchmark for Academic Deep Research Agents

T0 review · 4 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces a modular benchmark showing that planning is the dominant bottleneck in academic deep-research agents: better plans raise reasoning accuracy by up to 12 percentage points, while retrieval on review papers collapses bel

desk verdict A genuinely modular DR benchmark with solid annotation quality, but the retrieval 'collapse' headline is an artifact of an uncalibrated full-bibliography gold standard and needs a human baseline. read the letter →

arxiv 2512.00986 v3 pith:R6AYCIEB submitted 2025-11-30 cs.CL

Zhihan Guo , Feiyang Xu , Yifan Li , Muzhi Li , Shuai Zou , Jiele Wu , Han Shi , Haoli Bai
show 2 more authors
Ho-fung Leung Irwin King
This is my paper
classification cs.CL
keywords deepresearchagentsbenchmarkplanningretrievalreasoningacademicliteratureLLMevaluationreviewpapers
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces ADRA-Bank, a human-annotated set of 200 academic research tasks across 10 disciplines, and a modular evaluation that scores a deep-research agent separately on planning, retrieval, and reasoning. Its central claim is that high-level planning is the dominant bottleneck: when agents are given expert-written plans, reasoning accuracy jumps by as much as 12 percentage points (a leading agent rises from 58.65% to 70.93%), while planning scores remain near 25–28% F1. It also finds that retrieval collapses on review papers: agents identify the anchor paper on research tasks (62–76% accuracy) but cover almost none of a review's full bibliography (below 4%). A sympathetic reader would care because the benchmark is designed to be diagnostic, telling developers which module to fix first rather than just ranking whole systems.

What carries the argument

The central mechanism is the modular-integrated evaluation paradigm built on a ground-truth tuple per instance: query, gold plan, gold evidence set, and a suite of boolean diagnostic statements. Three metric suites score the modules: planning uses an exhaustive pairwise LLM semantic comparison of predicted against gold sub-tasks; retrieval matches citations by 20-character title-prefix identity and scores with the Jaccard index; reasoning checks whether a report supports or refutes 15–20 expert-written statements. The paradigm runs in two modes—end-to-end (modules chained) and isolated (modules given gold inputs)—so that failures can be attributed to a specific stage, and a gold-plan injecti

What would settle it

Compute the same retrieval Jaccard accuracy for a human expert or a simple baseline retriever that returns the first few highly-cited papers against the review-task gold bibliographies; if the baseline also scores near 4%, the review-task 'collapse' is a floor effect of the metric rather than an agent deficiency.

Watch

Extended reading notes

Core claim

The paper claims that academic deep-research capability can be decomposed into three measurable modules—planning (decomposing the query into sub-tasks), retrieval (gathering cited evidence), and reasoning (synthesizing a faithful report)—and that this decomposition is valid enough to attribute failures. The evidence is that planning scores are uniformly low across all agents and LLMs; fixing the plan raises downstream reasoning accuracy substantially; and retrieval accuracy falls below 4% when the gold evidence is a full review bibliography. The paper concludes that no current system masters all three modules and that improving planning is the crucial unlock for more reliable academic resear

Load-bearing premise

The retrieval scores treat the complete bibliography of a review paper as the gold evidence and match citations by short title prefixes, so a low score may reflect the harshness of that standard—especially with no human or simple-retriever baseline—rather than the agent's true retrieval skill.

Editorial extensions

If this is right

  • If planning is the dominant bottleneck, improving query decomposition (e.g., training or prompting planners) should yield the largest end-to-end gains before investing in better search or synthesis.
  • Review-paper retrieval requires a different evaluation lens: sub-4% accuracy implies current agents cannot produce representative bibliographies for literature surveys, and any agent claiming to review a field should be tested on this task.
  • Modular, isolated evaluation makes failure attribution possible: a system's reasoning failures can be separated from upstream planning and retrieval errors, enabling modular fixes.
  • Domain gaps are intrinsic: reasoning difficulty persists even when retrieval is bypassed, so performance in data-rich fields (medicine, biology) will not generalize to finance and materials without targeted effort.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The review-task retrieval metric may be punitive by construction: with bibliographies of 100–300 entries, even a strong single-paper retriever or a human writing a short bibliography would score only a few percent, so the 'collapse' partly reflects the harshness of full-bibliography recall rather than agent incompetence.
  • The planning F1 metric rewards one specific decomposition style (the expert outline); a valid alternative plan structure could score low on coverage, so low planning scores may understate planning skill until agreement between different expert planners is measured.
  • The diagnostic statements are binary claims; a model could pass many by self-contained phrasing. Extending them to multi-claim attribution (requiring the report to cite the right sentence) would test whether accuracy reflects understanding or lucky wording.
  • A testable extension: run the same benchmark with a planning-then-search pipeline where plans are scored for executability by an external search tool, to see whether plan quality predicts retrieval success beyond raw coverage.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces ADRA-Bank, a human-annotated benchmark of 200 academic deep-research instances across 10 domains, with research and review paper types, and ADRA-Eval, a modular evaluation paradigm that decomposes deep-research capability into planning, retrieval, and reasoning. The benchmark provides gold plans, gold evidence sets, and boolean diagnostic statements for each instance, and the paradigm supports both end-to-end evaluation of commercial DR agents and isolated evaluation of foundation LLMs. The authors evaluate five agents (OpenAI, Gemini, Perplexity, Grok, Search-r1) and ten LLMs, reporting that planning is a dominant bottleneck, retrieval collapses on review papers, and reasoning shows a moderate but uneven performance. They also report efficiency analyses, leakage checks, pass@k scaling, and human-preference validation of the reasoning metric.

Significance. If the evaluation paradigm is valid, ADRA-Bank fills a real gap: existing DR benchmarks are mostly retrieval-oriented and general-domain, whereas ADRA-Bank targets academic domains and attempts to separate planning, retrieval, and reasoning. The annotation protocol is a genuine strength — dual annotation with adjudication, a 93% inter-judge LLM agreement check, and 86.23% agreement with human expert preferences on reasoning quality are reported. The benchmark is also explicitly designed to be unsaturated, with pass@k experiments and temporal-leakage controls. However, the two headline findings — the <4% review-retrieval collapse and the planning-bottleneck claim — rest on operationalizations that are not yet fully calibrated. The retrieval metric, in particular, needs a human or strong-retriever baseline before it can support the paper's diagnostic conclusions. With those calibrations, the benchmark would be a valuable community resource; as written, the central diagnostic claims are only partially supported.

major comments (4)
  1. [§4.4, Appendices F/G; §5.1] The retrieval gold standard makes the headline review-task collapse hard to interpret. For research tasks the gold set is a single element (the source paper), so Accuracy is essentially single-document lookup; for review tasks the gold set is the complete bibliography (typically 100–300 entries) and Accuracy is the Jaccard index with 20-character title-prefix identity. Retrieving 3 of 150 correct references yields roughly 2% Accuracy, and any correct citation outside the exact bibliography counts as a false positive. No human-expert or strong single-paper retriever baseline is reported, so the '<4% on review papers' result in Table 2 cannot be distinguished from a floor effect of the metric. Please add such baselines and/or a graded relevance-based calibration before claiming that multi-source retrieval collapses.
  2. [§5.1, Table 2; §5.4] The claim that planning is 'extremely difficult' is supported by only one end-to-end agent: in Table 2, only Gemini exposes its sub-tasks; the other four agents have no planning row. The LLM-as-a-judge plan comparison is validated only by inter-judge agreement (93%) and not by human expert judgment on plan coverage, whereas §5.4 validates Reasoning F1 against human preferences. Without an expert plan baseline or an evaluation of the other agents' internal plans, the conclusion is stronger than the evidence.
  3. [§H.1, §5.1, Table 2] The reasoning diagnostic set is imbalanced: 64.71% TRUE, 35.29% FALSE. Under the Accuracy definition in Appendix F, a trivial always-TRUE predictor would score 64.71%, yet the best end-to-end agents score 59–60% Accuracy (Table 2). Reporting Accuracy without this majority baseline is misleading; either report chance/majority baselines or use a balanced metric. The 86.23% human-preference agreement mitigates but does not remove the need for such a baseline.
  4. [§5.5, Table 3] Gold-plan injection shows OpenAI reasoning Accuracy improving from 58.65 to 70.93, but the experiment also changes the retrieval input: OpenAI retrieval Accuracy drops from 31.35 to 25.13 in Table 3. The causal story is therefore not as clean as 'fixing planning.' The planning-bottleneck claim would be strengthened by per-model statistical significance testing and by an ablation that checks whether the gold plan leaks content that directly answers the diagnostic statements, since the gold plan's sub-tasks often restate the diagnostic claims.
minor comments (5)
  1. [Table 2 and throughout] Typographical errors: 'Perleixity' instead of 'Perplexity' in several places; 'Grok-4' is attributed to OpenAI in the reference list (xAI is the developer).
  2. [§3.2 vs. Appendix G] Section 3.2 says retrieval matching uses 'rule-based exact match on canonical identifiers (DOIs)', but Appendix G specifies a 20-character title-prefix match. This inconsistency should be resolved.
  3. [Appendix H.3] The case study for the biology paper (DOI 10.1038/s41467-024-45147-9) lists diagnostics that are identical to the biochar/concrete example (e.g., 'The construction industry accounts for more than 30%...'). This appears to be a copy-paste error and undermines the example's credibility.
  4. [Appendices] Appendix cross-references to A.5 and A.12 are present in the text but these sections are not included in the provided manuscript; please verify the appendix numbering.
  5. [Figure 2] The domain heatmap in Figure 2 is extremely small and the numbers are hard to read; a higher-resolution figure or separate table would improve legibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark measurements are empirically grounded and not reduced to their own definitions.

full rationale

ADRA-Bank's core outputs are empirical measurements from human-annotated ground truth, not predictions derived from fitted parameters or self-justifying definitions. The planning, retrieval, and reasoning metrics are defined independently of the systems under test: gold plans are expert-annotated from paper structure, gold evidence sets are the source paper (research tasks) or the full bibliography (review tasks), and reasoning diagnostics are Boolean claims with gold labels. The headline findings—planning F1 ~25–28, review retrieval <4%, and reasoning gains from gold-plan injection—are reported measurements from those protocols, not quantities that are equal to their inputs by construction. The retrieval collapse on review papers reflects the strictness of the full-bibliography Jaccard gold standard; whether that is a floor effect is a validity/calibration concern, not a circularity, since the paper does not claim the metric itself proves agent failure independently of the measurement. The gold-plan ablation is an intervention comparison, not a self-fulfilling prediction. Self-citations (e.g., Zhang et al. 2025a in the introduction) are used for background context, not as load-bearing evidence for the benchmark's validity. No uniqueness theorem, renamed empirical result, or fitted-parameter-as-prediction is invoked. Consequently, the derivation chain is self-contained and no circular step meets the quoted-evidence bar.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

No fitted physical parameters or invented entities; the ledger consists of hand-chosen evaluation-design parameters (gold-evidence definition, match thresholds, plan/diagnostic formats) and domain assumptions about annotation fidelity and LLM-judge validity. The most consequential entry is the retrieval gold-evidence definition, which determines the paper's most striking empirical claim.

free parameters (4)
  • Retrieval gold-evidence definition = research: source paper only; review: complete bibliography
    Hand-chosen operationalization (§4.4) that determines the headline retrieval numbers; with review-paper bibliographies of dozens to hundreds of entries, the Jaccard denominator makes near-0% scores almost inevitable.
  • Title-prefix match length = 20 characters
    Chosen threshold (Appendix G) for citation identity; the authors justify it against exact-DOI and semantic matching, but the threshold shapes every retrieval recall number.
  • Gold plan format constraints = 5–10 subtasks, 8–20 words each
    Annotation constraint (Appendix H.2) that defines the planning target; all planning F1 scores (24–28) depend on this granularity choice.
  • Diagnostics count and label balance per paper = 15–20 boolean claims; 64.71% true / 35.29% false
    Hand-set probe size and observed label balance (§4.5, Appendix H.1); reasoning F1 numbers move directly with the wording and balance of these claims.
assumptions (5)
  • domain assumption Expert annotations (queries, plans, evidence sets, diagnostic labels) are faithful to the source papers.
    The entire benchmark rests on senior-PhD annotations (§4). Dual annotation and adjudication are claimed (Appendix J), but the annotation data is not released for independent audit.
  • domain assumption An LLM judge (GPT-4o, cross-checked with Gemini-2.5-pro at 93% inter-judge agreement) can validly compare predicted sub-tasks against gold plans.
    §3.1 and Appendix K.1. Planning and reasoning scores flow through LLM-as-judge; the 86.23% human agreement (§5.4) is measured on reasoning F1, not on plan coverage.
  • domain assumption The boolean diagnostics are unambiguous and each is anchored only in the source paper.
    §4.5; any wording ambiguity directly shifts reported reasoning accuracies of 23–78%.
  • ad hoc to paper Retrieval quality is measured by citation identity (title-prefix match), not by semantic relevance of the retrieved evidence.
    Appendix G. This collapses 'good retrieval' to 'cited the anchor paper' for research tasks and to 'covered the full bibliography' for review tasks — the load-bearing design choice behind the paper's headline retrieval findings.
  • standard math Standard set-similarity metric conventions (Jaccard, F1, precision, recall).
    Appendix F; standard definitions, included for completeness.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ADRA-Bank: A Modular Benchmark for Academic Deep Research Agents." pith.science (2026). https://pith.science/paper/R6AYCIEB

@misc{pith2026251200986,
  author       = {Pith},
  title        = {Pith review of: ADRA-Bank: A Modular Benchmark for Academic Deep Research Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/R6AYCIEB}},
  note         = {Machine review of arXiv:2512.00986}
}
read the original abstract

A surge in academic publications calls for automated deep research (DR) systems, but accurately evaluating them is still an open problem. First, existing benchmarks often focus narrowly on retrieval while neglecting high-level planning and reasoning. Second, existing benchmarks favor general domains over the academic domains that are the core application for DR agents. To address these gaps, we introduce ADRA-Bank, a modular benchmark for Academic DR Agents. Grounded in academic literature, our benchmark is a human-annotated dataset of 200 instances across 10 academic domains, including both research and review papers. Furthermore, we propose a modular Evaluation Paradigm for Academic DR Agents (ADRA-Eval), which leverages the rich structure of academic papers to assess the core capabilities of planning, retrieval, and reasoning. It employs two complementary modes: an end-to-end evaluation for \task agents and an isolated evaluation for foundational LLMs as potential backbones. Results reveal uneven capabilities: while agents show specialized strengths, they struggle with multi-source retrieval and cross-field consistency. Moreover, improving high-level planning capability is the crucial factor for unlocking the reasoning potential of foundational LLMs as backbones. By exposing these actionable failure modes, ADRA-Bank provides a diagnostic tool to guide the development of more reliable automatic academic research assistants.

Figures

Figures reproduced from arXiv: 2512.00986 by the authors.

Figure 1
Figure 1. The ADRA-Bank and ADRA-Eval Framework. This figure illustrates how a DR agent workflow (a) is assessed by our modular-integrated evaluation paradigm (b) using our human-annotated benchmark (c). This approach enables a diagnostic analysis that pinpoints specific failures in the agent’s planning, retrieval, and reasoning capabilities. and reasoner, respectively. Formally, the functionalization of the three modules can… view at source ↗
Figure 2
Figure 2. End-to-end DR agents performance across diverse domains: CS (Computer Science), FIN (Finance), MED (Medicine), BIO (Biology), CHEM (Chemistry), ENV (Environmental Science), NRG (Energy), BC (Building and Construction), EARTH (Earth Science), and MAT (Materials). All results are presented as percentages. Research Review Avg. Research Review Avg. Research Review Avg. 0 10 20 30 40 50 60 70 80 0 200 400 600 Accuracy Ti… view at source ↗
Figure 3
Figure 3. Efficiency of retrieval and reasoning modules. All results are presented as percentages. 5.4. Does Metric Align with Human Preferences? To validate our metrics against human judgment, domain experts performed pairwise comparisons on 100 blinded report pairs, assessing factuality, comprehensiveness, and coherence. Experts selected the superior report based on an overall assessment of its factuality, comprehensiveness… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Performance scaling with test-time compute. pass@3. This confirms the benchmark effectively measures test-time compute benefits rather than being limited by rigid formats. Even at pass@3, OpenAI’s top Reasoning score is only 68.49%, while Planning hovers around 40%. Th…
Figure 5
Figure 5. Figure 5: Foundational LLMs performance across diverse domains. The heatmap visualizes the accuracy scores of various models (rows) across 10 academic and technical domains (columns), where color intensity corresponds to performance. The domains are: CS (Computer Science), FIN (…

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. From Knowing to Acting: Benchmarking Self-Awareness Capability of LLM Agents

    cs.AI 2026-06 unverdicted novelty 7.0 of 10

    Introduces KAPRO framework and KAware dataset to benchmark LLM agents' self-awareness in distinguishing internal knowledge from external tool needs.

  2. From Simple QA to Deep Research: A Verifiable Benchmark Constructed through Iterative Task Evolution

    cs.AI 2026-08 conditional novelty 6.0 of 10

    An automatic pipeline evolves simple QA questions into 500 deep-research tasks with DAG-structured, fact-grounded rubrics that discriminate between models.

Reference graph

Works this paper leans on

21 extracted references · 6 canonical work pages · cited by 2 Pith papers

  1. [1]

    The query should only based on the content of given paper

  2. [2]

    The query should not include the paper title or authors directly

  3. [3]

    It should be an open-ended question that related to the paper

  4. [4]

    The query should contain part or all aspects: background, research problem, methodology, experimental results, and contributions

  5. [5]

    Plan Annotation Guideline You are given a research question and an academic paper

    The query should be a user may asked before reading the given paper. Plan Annotation Guideline You are given a research question and an academic paper. Your job is to list the high-level sub-tasks needed to answer the question based on the paper. Requirements: 18 Beyond Retrieval: A Modular Benchmark for Academic Deep Research Agents

  6. [7]

    Identify 5-10 sub-tasks

  7. [8]

    Coarse-grained only: write complete sentences but avoid low-level details such as specific numbers, datasets, instruments, figure/panel references, or citation markers

  8. [9]

    Diagostics Annotation Guideline You are given a research question and an academic paper

    Keep each sentence concise (about 8–20 words). Diagostics Annotation Guideline You are given a research question and an academic paper. Your job is to list the diagnostic statements and gold answers which should cover the whole paper. Requirements:

Show all 21 references
  1. [10]

    Identify 15-20 diagnostics

  2. [11]

    Coverage: Statements should cover all key aspects of given paper

  3. [12]

    Accuracy: Boolean answers corresponding to statements are used to verify the accuracy of model generated reports. H.3. Annotation Examples An example of the summary writing. A Summary Example DOI 10.1016/j.conbuildmat.2024.138507 Summary This paper addresses the dual challenge...

    2024

  4. [16]

    3D printable tissue adhesives,

    Diagnostics Statement 1:UHPC is characterized by ultra-high compressive strength, high toughness, and excellent durability. Answer:True Statement 2: Due to optimized material usage, UHPC has a lower carbon footprint than conventional concrete (NC). Answer:False Statement 3:UHP...

    2025

  5. [17]

    10.1038/s41467-024-45147-9

    doi

  6. [18]

    10.1002/SMMD.20220033

    doi

  7. [19]

    10.1016/j.nantod.2014.09.006

    2014 doi

  8. [22]

    10.3390/buildings14020382 39 Beyond Retrieval: A Modular Benchmark for Academic Deep Research Agents

    doi

  9. [23]

    10.1016/j.conbuildmat.2024.135602

    2024

  10. [24]

    10.1007/s11356-021-13435-2

    doi

  11. [25]

    10.1007/s11367-013-0614-0

    doi

  12. [26]

    dematerialization

    Diagnostics Statement 1:UHPC is characterized by ultra-high compressive strength, high toughness, and excellent durability. Answer:True Statement 2: Due to optimized material usage, UHPC has a lower carbon footprint than conventional concrete (NC). Answer:False Statement 3:UHP...

  13. [981]

    long tail

    URL https://aclanthology.org/2024. emnlp-main.981/. Wei, J., Sun, Z., Papay, S., McKinney, S., Han, J., Fulford, I., Chung, H. W., Passos, A. T., Fedus, W., and Glaese, A. Browsecomp: A simple yet challenging benchmark for browsing agents, 2025. URL https://arxiv.org/ abs/2504...

    2024 arXiv

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.