Pith. sign in

REVIEW 4 major objections 5 minor 75 references

Tracing the Cascade: A Topology-Aware Evaluation Framework for Scientific Agent Hallucinations

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Scientific agents often reach the correct final answer while reasoning through claims that are false or unverifiable, and these errors concentrate at a small set of highly connected concept hubs.

desk verdict Well-built, unusually transparent evaluation framework; but the headline claim that accuracy decouples from trajectory honesty is not actually measured by the error-targeted pipeline. read the letter →

arxiv 2608.00711 v1 pith:XYLZBWIZ submitted 2026-08-01 cs.AI

Xinshun Feng , Ziqi Miao , Lijun Li , Jing Shao This is my paper · ORCID
classification cs.AI
keywords SCHEMAhallucinationevaluationscientificagentsknowledgegraphtopology-weightedseveritycounterfactualattributionLLM-as-judgebiomedicalbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that judging a scientific AI agent by whether its final answer is correct leaves the main danger undetected: agents frequently produce right answers by reasoning through intermediate claims that are unsupported or false. To expose this, the authors build SCHEMA, an evaluation framework that constructs a biomedical knowledge graph from benchmark seeds and literature, generates graph-grounded tasks of five types, and audits agents' full reasoning trajectories with a topology-weighted severity metric alongside a counterfactual attribution module. The paper's empirical findings are that hallucinations are not random slips but concentrate at a small set of highly connected concept hubs, and that final-answer accuracy correlates only weakly with trajectory honesty across ten models. If these findings hold, reliability evaluation for scientific agents needs to become structural, weighing where an error sits in the knowledge topology and how far it propagates, rather than collapsing performance into a single accuracy number.

What carries the argument

The load-bearing object is the topology-weighted hallucination severity metric $HS_w$, computed over the scientific concept graph $G$. Each extracted claim $c$ is mapped to a node $\eta(c)$ in the graph by lexical matching with an embedding-based fallback, and receives weight $w_g(c) = 1 + \log(1 + \deg_K(\eta(c)))$, a log-scaled bonus that grows with the node's undirected degree; claims that map to no node fall back to $w_g = 1$. Judge verdicts (supported, refuted, unverifiable) map deterministically to severities $0, 0.5, 1$, and $HS_w$ is the weighted average over claims, so an error at a highly connected concept is penalized more than the same error at a peripheral one, reflecting its downstream blast radius. The framework pairs this scalable pipeline with a multi-agent counterfactual attribution module that follows abduction, intervention, and prediction: hallucinated claims are verified against literature, a minimally edited trajectory replaces them with corrections, and, with prior context frozen by the trajectory debugger, a candidate subset is certified as the root cause only if the rerun strictly flips the outcome.

What would settle it

Sample roughly two hundred judged claims across several models and verdict categories, and have domain experts label each claim against the same evidence the judge saw, without knowing the judge's verdict. If expert labels disagree with the judge on a large share of claims, or if the disagreements concentrate on the hub concepts that dominate $HS_w$, then the hub-concentration and decoupling results do not survive human-level grounding. A secondary check is to rebuild the concept graph with a different extraction model and re-run the metric: stable hub sets would confirm the topology, while drifting sets would trace the finding to the pipeline.

Watch

Extended reading notes

Core claim

The paper's central claim is that hallucination in scientific agents is a structural phenomenon with a topological signature. On the Protein Domain subset, a majority of reasoning failures trace to flawed domain-specific priors rather than workflow problems, and hallucinated claims concentrate around a small set of recurring concept hubs, with one family, enzymatic catalysis, accounting for a large share of the hallucinated concepts, while long-tail peripheral concepts show near-zero repairability. The second headline claim is that terminal accuracy decouples from trajectory honesty: models often reach correct final answers through flawed intermediate logic, a shortcut-reasoning pattern that persists across two nearly disjoint concept spaces, so that two models can tie on accuracy yet differ sharply in topology-weighted hallucination severity. Taken together, the paper argues that terminal accuracy alone is an insufficient signal of agent reliability in high-stakes science, and that evaluation should weight claims by their structural role and trace failures back to their causal mechanism.

Load-bearing premise

Everything rests on the automatic fact-checking chain being reliable: if the LLM judge frequently mislabels an agent's claims as false or unsupported, and the paper itself notes the judge has not been validated against expert annotations, then the hub concentration and the decoupling of accuracy from trajectory honesty could be artifacts of the measurement rather than real properties of the agents.

Editorial extensions

If this is right

  • Rankings of scientific agents will change when trajectory honesty replaces terminal accuracy as the criterion; two models that tie on accuracy can differ sharply in topology-weighted severity, so deployment decisions for high-stakes science should report both families of metrics.
  • Hallucination benchmarks that treat all facts as interchangeable will mislead: a faithful evaluation must weight concepts by their structural role, since errors at hubs propagate through every downstream reasoning chain that passes through them.
  • Failure repair becomes predictable by error type: mapping errors dominate frequency but are comparatively recoverable, while fact errors have the lowest repairability because later reasoning steps inherit and amplify a false premise.
  • A single aggregate accuracy figure masks format-specific weaknesses: models can score near chance on boolean support judgments while excelling at experimental code generation, so results need to be reported by question type and evidence tier.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The hub-concentration finding suggests a repair strategy the paper does not develop: if a few high-traffic concepts carry most hallucinations, then injecting verified grounding for just those hubs could yield disproportionate reliability gains without broad retraining.
  • The reported partial-anchoring valley, where weak graph context hurts accuracy more than no context for most models, points to a general and testable principle: agents over-trust fragmentary retrieved structure, which would predict that how context is presented matters as much as how much of it is given.
  • A direct robustness test is to rebuild the concept graph with a different extraction model and judge family; if the same concepts still dominate the hallucination distribution and the accuracy–honesty decoupling survives, the findings are properties of the agents rather than of the measurement chain.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes SCHEMA, an automated pipeline that builds a biomedical concept graph from benchmark seeds and literature evidence, synthesizes five types of evidence-grounded questions (claim choice, boolean support, two-hop tail, essay, and experiment code), and evaluates LLM agents using terminal accuracy plus two trajectory-level diagnostics: a topology-weighted hallucination severity metric (HS/HSw) and a multi-agent counterfactual attribution module. Experiments on Protein Domain and PathVQA-Enhanced across 10 LLMs are used to claim that hallucinations concentrate at a small set of knowledge hubs and that final-answer accuracy decouples from trajectory honesty. The manuscript is unusually detailed in its appendices and plans to release code and trajectory logs.

Significance. If the empirical claims survive scrutiny, SCHEMA would be a valuable contribution: it makes the notion of topology-weighted hallucination severity concrete, contributes a reproducible biomedical agent benchmark, and articulates a falsifiable claim about the relationship between terminal accuracy and intermediate-trajectory quality. The pipeline is described in enough detail to be reproduced, and the planned release of per-model trajectories is a genuine strength. However, the headline empirical findings are not currently supported as stated: the hallucination metrics are computed only on incorrect trajectories, and the underlying per-claim labels come from an unvalidated LLM judge. These issues affect the central claims and require substantive revision rather than copyediting.

major comments (4)
  1. [Appendix E.3; Section 4.2; Table 3; Figure 5] The hallucination pipeline 'defaults to an error-targeted mode (is_correct=false), computing severity exclusively on trajectories that yield incorrect terminal answers.' The abstract and Section 4.2 claim that 'final-answer accuracy decouples from trajectory honesty' and that models 'frequently reach correct terminal answers via flawed intermediate logic.' These claims require measuring hallucination severity on correct-answer trajectories. As reported, the HSw values in Table 3 and Figure 5 are conditional on terminal failure, so the paired DeepSeek-V4-Flash / GPT-5.4-mini example shows only that failed trajectories differ in severity; it does not show that either model's correct trajectories contain refuted or unverifiable claims. Please recompute HS/HSw on all trajectories, or at least report the correct-trajectory breakdown separately, and revise the decoupling claim accordingly.
  2. [Section 3.3; Section 3.4; Appendix B] The entire severity computation rests on the LLM judge's supported/refuted/unverifiable labels, and Appendix B acknowledges that the judge 'has not been validated against expert annotations.' This is load-bearing because every HS/HSw value in Table 3 and Figure 6 inherits the judge's verdicts. Without a calibration study on a representative subset (e.g., per-category agreement rates against domain experts), the reported differences, such as HSw 0.362 versus 0.268, cannot be separated from judge bias. Please either provide such a validation or qualify all headline comparisons as contingent on unvalidated automated verdicts.
  3. [Section 3.3; Table 2; Section 4.2] The subgraph sampling procedure in Appendix E.1 is not degree-stratified, so high-degree concepts appear in more candidate triples and therefore in more questions. The finding that hallucinations 'concentrate at a small set of highly connected knowledge hubs' may consequently reflect the question distribution rather than model behavior. Table 2 reports counts by Core/Peripheral/Off-graph tiers, but no per-exposure error rates are given. Please report error rates per concept conditional on the number of questions involving that concept, or subsample subgraphs in a degree-stratified manner, before drawing the hub-concentration conclusion.
  4. [Section 2.3; Figure 2] The text states that 'ENZYMATIC/CATALYSIS alone accounts for nearly half of all hallucinations,' but Figure 2 shows n=43 out of 248 hallucinated concepts, which is about 17%. Unless a different subset is intended, this statement is inconsistent with the figure and should be corrected or clarified.
minor comments (5)
  1. [Section 2.2] The counterfactual attribution findings in Section 2.3 are based on a single agent model (Intern-S1-Pro) and 141 failure trajectories from ProteinLMBench; this should be explicitly labeled as a case study rather than a general empirical finding.
  2. [Figure 6] The HSw values along the verifiability axis range only from 0.381 to 0.436; the claim that severity 'rises monotonically' should be accompanied by confidence intervals or a statistical test, especially given the absence of error bars acknowledged in the checklist.
  3. [References; Figure 5] There are several presentation issues: Reference [24] contains a duplicated author name 'XingyuHan XingyuHan', Figure 5(c) contains the typo 'Biomaker', and the naming of 'ProteinLMBench' versus 'Protein Domain' should be made consistent throughout.
  4. [Main text; Appendix E.3] The fact that HS/HSw are computed only on incorrect trajectories should be stated prominently in the main text near Table 3, not only in an appendix, because it materially changes how the numbers should be interpreted.
  5. [Abstract] The phrase 'the first evidence-grounded, topology-aware evaluation framework' is an overclaim; nearby works such as KGHaluBench also use knowledge-graph grounding and topology. Please soften the novelty claim or add a more precise comparison.

Circularity Check

2 steps flagged · score 6.0 of 10

HSw is defined only for incorrect terminal answers, so the headline 'decoupling' claim is not derived from the reported metric; the hub finding is also coupled to the same graph that generates and weights questions.

  1. self definitional [Appendix E.3 (hallucination pipeline) vs. Section 4.2 (Decoupling of Accuracy and Trajectory Honesty)]
    "To focus on structural reasoning failures, the pipeline defaults to an error-targeted mode (is_correct=false), computing severity exclusively on trajectories that yield incorrect terminal answers."

    HSw and HS in Table 3 and Figure 5 are therefore defined only on trajectories where terminal accuracy is 0. The abstract and Section 4.2 use them to conclude that 'models frequently reach correct terminal answers via flawed intermediate logic—a form of shortcut reasoning.' That conclusion requires measuring the trajectory severity of correct-answer trajectories, which the default pipeline excludes by definition. The 'decoupling of accuracy and trajectory honesty' is thus not an empirical output of the reported metric; it is a claim about the excluded part of the sample. The paper reports no correct-trajectory HSw breakdown that would support the shortcut-reasoning mechanism.

  2. other [Section 3.3 (Evidence-Grounded Benchmark Synthesis) and Section 3.4, Eq. (3)]
    "Building upon G, we synthesize an evidence-grounded benchmark whose questions are anchored to verifiable literature evidence rather than bare relation triples. ... wg(c) = 1 +⊮[η(c)̸=⊥]·log(1 + degK(η(c)))"

    The same graph G that supplies the benchmark subgraphs also supplies the degree weights in HSw. Hence the finding that hallucinations cluster at 'highly connected knowledge hubs' is not measured against an independent topology: hub nodes are over-represented as question substrates (sampled subgraphs from G) and, when an agent errs on them, Eq. (3) additionally up-weights the severity by degK. This is a design self-reference rather than a full derivation equivalence, because G is augmented with extrinsic literature edges; nevertheless, the hub-concentration claim is partly built into the instrument that reports it.

full rationale

SCHEMA's construction is otherwise self-contained: benchmark questions are validated by multi-stage checks, the metrics are explicitly defined, no load-bearing self-citation chain is used, and no parameter is fitted and then renamed a prediction. The two issues above are construction-level. The most serious is the error-targeted default: HSw cannot substantiate the claim of correct-answer shortcut reasoning, since it is computed exclusively on failed trajectories. The second is that the topological weighting uses the same graph that generated the evaluation items, so the reported hub concentration is partially an artifact of instrument construction rather than an independent discovery. These are not full equivalence reductions—the graph includes extrinsic literature evidence, and severity verdicts come from an external judge—so the appropriate score is partial circularity (6) rather than 8-10. The explicit Appendix B admission that the judge is not expert-validated is a reliability limitation, not a circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central measurement rests on hand-chosen thresholds, LLM-based extraction and judging, and the assumption that a literature-derived graph topology captures how errors propagate. These are documented transparently but are not validated against external expert annotation or independent causal ground truth.

free parameters (5)
  • theta_cos claim-to-node cosine floor = 0.6
    Claims below 0.6 embedding similarity to graph nodes are treated as off-graph with baseline weight 1; directly shapes HSw (Appendix E.4).
  • BGE canonicalization threshold = 0.85
    Concept canonicalization uses BGE-Large embedding fallback with cosine threshold 0.85; affects claim bucketing and grounding (Appendix E.3).
  • Evidence profile hedge thresholds = 0.34 (STRONG) and 0.67 (WEAK)
    Hedge density cutoffs determine STRONG/MEDIUM/WEAK evidence labels, which gate question types and claim wording (Appendix E.2).
  • theta_score retrieval cutoff = unspecified
    Papers exceeding this score are cached for graph construction; the numeric value is not disclosed (Appendix E.1).
  • theta_f node fusion threshold = unspecified
    Node pairs fuse when BGE-Large similarity exceeds this threshold and types match; numeric value is not disclosed (Appendix E.1).
assumptions (4)
  • domain assumption Source benchmarks carry reliable human-verified ground truth.
    Concept seeds are drawn from ProteinLMBench and PathVQA-Enhanced and treated as correct without independent audit (Section 3.2).
  • domain assumption LLM extraction and judging produce reliable triples and verdicts.
    Graph construction and benchmark validation rely on LLM triple extraction and LLM-as-judge verdicts; reliability is acknowledged as uncalibrated (Appendix B and E.3).
  • domain assumption Graph degree is a valid proxy for downstream error propagation.
    HSw weighting uses undirected node degree (Eq. 3) as error leverage; no causal validation of this proxy is provided.
  • domain assumption An outcome flip after counterfactual edit certifies the root cause.
    Section 2.1 assumes that replacing a hallucinated claim with a grounded correction and observing an outcome flip identifies the causal mechanism, within a bounded attempt budget.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Tracing the Cascade: A Topology-Aware Evaluation Framework for Scientific Agent Hallucinations." pith.science (2026). https://pith.science/paper/XYLZBWIZ

@misc{pith2026260800711,
  author       = {Pith},
  title        = {Pith review of: Tracing the Cascade: A Topology-Aware Evaluation Framework for Scientific Agent Hallucinations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XYLZBWIZ}},
  note         = {Machine review of arXiv:2608.00711}
}
read the original abstract

Large language model (LLM) agents are increasingly deployed in scientific research, where reliability is critical and the underlying knowledge is densely interconnected. In such settings, hallucinations are particularly damaging: a single erroneous claim on a foundational concept can propagate through multi-step reasoning and corrupt entire trajectories. Existing hallucination benchmarks largely operate at the surface level, treating facts in isolation and relying on uniform accuracy metrics that ignore this topological structure. We address this gap with SCHEMA, the first evidence-grounded, topology-aware evaluation framework for hallucinations in scientific agents. SCHEMA automatically constructs scientific concept graphs from benchmark seeds and literature evidence, synthesizes graph-grounded tasks spanning claim verification, multi-hop reasoning, open-ended explanation, and experimental code generation, and evaluates agents with two complementary diagnostics. A trajectory hallucination pipeline audits intermediate reasoning at scale via a topology-weighted severity score, while a multi-agent counterfactual attribution module pinpoints the causal mechanism behind selected failures. SCHEMA reveals that hallucinations concentrate at a small set of highly connected knowledge hubs, and that final-answer accuracy decouples from trajectory honesty; models often reach correct conclusions through structurally flawed reasoning. These results indicate that for high-stakes scientific applications, terminal accuracy alone is an insufficient signal of agent reliability, motivating mechanism-level evaluation grounded in knowledge topology. Code is available at https://github.com/circles-post/SCHEMA.

Figures

Figures reproduced from arXiv: 2608.00711 by the authors.

Figure 1
Figure 1. Our framework involves three stages: (i) [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Distribution of the 248 hallucinated con [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Overview of the SCHEMA pipeline. The framework comprises three stages: concept-graph construction, evidence-grounded benchmark synthesis, and topology-aware hallucination evaluation. where T and R denote closed sets of biomedical entity types and canonical relations, and the provenance function s records the origin of each edge. Intrinsic edges are extracted from the internal signal of B itself—answer text, accompan… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Visualization of Protein Graph (left) and its [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: (a, b) Final-answer accuracy versus weighted hallucination severity HS [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Topology-weighted hallucination sever￾ity (HSw ) along the verifiability axis on the Protein Domain. We aggregate 15 fine-grained concept types into four functional categories ordered by verification complexity. The dashed black line in￾dicates the sample-weighted coho…
Figure 7
Figure 7. Figure 7: (a) Accuracy on Protein domain stratified by knowledge-graph anchoring level: [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

75 extracted references · 45 canonical work pages

  1. [1]

    A survey on large language model based autonomous agents.Frontiers of Computer Science, 18(6):186345, 2024

    Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents.Frontiers of Computer Science, 18(6):186345, 2024

    2024

  2. [2]

    The rise and potential of large language model based agents: A survey.Science China Information Sciences, 68(2):121101, 2025

    Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. The rise and potential of large language model based agents: A survey.Science China Information Sciences, 68(2):121101, 2025

    2025

  3. [3]

    The ai scien- tist: Towards fully automated open-ended scientific discovery.arXiv preprint arXiv:2408.06292, 2024

    Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The ai scien- tist: Towards fully automated open-ended scientific discovery.arXiv preprint arXiv:2408.06292, 2024

    arXiv 2024

  4. [4]

    Probing scientific general intelligence of llms with scientist- aligned workflows.arXiv preprint arXiv:2512.16969, 2025

    Wanghan Xu, Yuhao Zhou, Yifan Zhou, Qinglong Cao, Shuo Li, Jia Bu, Bo Liu, Yixin Chen, Xuming He, Xiangyu Zhao, et al. Probing scientific general intelligence of llms with scientist- aligned workflows.arXiv preprint arXiv:2512.16969, 2025

    arXiv 2025

  5. [5]

    A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions.ACM Transactions on Information Systems, 43(2):1–55, 2025

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qiang- long Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions.ACM Transactions on Information Systems, 43(2):1–55, 2025

    2025

  6. [6]

    Improv- ing factuality and reasoning in language models through multiagent debate

    Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improv- ing factuality and reasoning in language models through multiagent debate. InForty-first international conference on machine learning, 2024

    2024

  7. [7]

    Kghalubench: A knowledge graph-based hallucination benchmark for evaluating the breadth and depth of llm knowledge

    Alex Robertson, Huizhi Liang, Mahbub Gani, Rohit Kumar, and Srijith Rajamohan. Kghalubench: A knowledge graph-based hallucination benchmark for evaluating the breadth and depth of llm knowledge. InFindings of the Association for Computational Linguistics: EACL 2026, pages 3975–3989, 2026

    work page 2026

  8. [8]

    Mirage- bench: Llm agent is hallucinating and where to find them.arXiv preprint arXiv:2507.21017, 2025

    Weichen Zhang, Yiyou Sun, Pohao Huang, Jiayue Pu, Heyue Lin, and Dawn Song. Mirage- bench: Llm agent is hallucinating and where to find them.arXiv preprint arXiv:2507.21017, 2025

    arXiv 2025

Show all 75 references
  1. [9]

    Hal- luhard: A hard multi-turn hallucination benchmark.arXiv preprint arXiv:2602.01031, 2026

    Dongyang Fan, Sebastien Delsad, Nicolas Flammarion, and Maksym Andriushchenko. Hal- luhard: A hard multi-turn hallucination benchmark.arXiv preprint arXiv:2602.01031, 2026

    2026

  2. [10]

    Agenthallu: Benchmarking automated hallucination attribution of llm-based agents.arXiv preprint arXiv:2601.06818, 2026

    Xuannan Liu, Xiao Yang, Zekun Li, Peipei Li, and Ran He. Agenthallu: Benchmarking automated hallucination attribution of llm-based agents.arXiv preprint arXiv:2601.06818, 2026

    2026

  3. [11]

    Why your deep research agent fails? on hallucination evaluation in full research trajectory.arXiv preprint arXiv:2601.22984, 2026

    Yuhao Zhan, Tianyu Fan, Linxuan Huang, Zirui Guo, and Chao Huang. Why your deep research agent fails? on hallucination evaluation in full research trajectory.arXiv preprint arXiv:2601.22984, 2026

    2026 arXiv

  4. [12]

    Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models

    Potsawee Manakul, Adian Liusie, and Mark Gales. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. InProceedings of the 2023 conference on empirical methods in natural language processing, pages 9004–9017, 2023

    2023

  5. [13]

    Halueval: A large-scale hallucination evaluation benchmark for large language models

    Junyi Li, Xiaoxue Cheng, Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. Halueval: A large-scale hallucination evaluation benchmark for large language models. InProceedings of the 2023 conference on empirical methods in natural language processing, pages 6449–6464, 2023

    2023

  6. [14]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. InThe eleventh international conference on learning representations, 2022

    2022

  7. [15]

    Cambridge university press, 2009

    Judea Pearl.Causality. Cambridge university press, 2009

    2009

  8. [16]

    Interactive debugging and steering of multi-agent ai systems

    Will Epperson, Gagan Bansal, Victor C Dibia, Adam Fourney, Jack Gerrits, Erkang Zhu, and Saleema Amershi. Interactive debugging and steering of multi-agent ai systems. InProceedings of the 2025 CHI Conference on Human Factors in Computing Systems, pages 1–15, 2025. 11

    2025

  9. [17]

    Intern-s1-pro: Scientific multimodal foundation model at trillion scale.arXiv preprint arXiv:2603.25040, 2026

    Yicheng Zou, Dongsheng Zhu, Lin Zhu, Tong Zhu, Yunhua Zhou, Peiheng Zhou, Xinyu Zhou, Dongzhan Zhou, Zhiwang Zhou, Yuhao Zhou, et al. Intern-s1-pro: Scientific multimodal foundation model at trillion scale.arXiv preprint arXiv:2603.25040, 2026

    2026

  10. [18]

    Autogen: Enabling next-gen llm applications via multi-agent conversations

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. Autogen: Enabling next-gen llm applications via multi-agent conversations. InFirst conference on language modeling, 2024

    2024

  11. [19]

    A fine-tuning dataset and benchmark for large language models for protein understanding

    Yiqing Shen, Zan Chen, Michail Mamalakis, Luhan He, Haiyang Xia, Tianbin Li, Yanzhou Su, Junjun He, and Yu Guang Wang. A fine-tuning dataset and benchmark for large language models for protein understanding. In2024 IEEE International Conference on Bioinformatics and Biomedicin...

    2024

  12. [20]

    Pathvqa: 30000+ questions for medical visual question answering.arXiv preprint arXiv:2003.10286, 2020

    Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. Pathvqa: 30000+ questions for medical visual question answering.arXiv preprint arXiv:2003.10286, 2020

    2003 arXiv

  13. [21]

    Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D White, and Philippe Schwaller

    Andres M. Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D White, and Philippe Schwaller. Augmenting large language models with chemistry tools.Nature machine intelli- gence, 6(5):525–535, 2024

    2024

  14. [22]

    Autonomous chemical research with large language models.Nature, 624(7992):570–578, 2023

    Daniil A Boiko, Robert MacKnight, Ben Kline, and Gabe Gomes. Autonomous chemical research with large language models.Nature, 624(7992):570–578, 2023

    2023

  15. [23]

    The ai scientist-v2: Workshop-level automated scientific discovery via agentic tree search.arXiv preprint arXiv:2504.08066, 2025

    Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist-v2: Workshop-level automated scientific discovery via agentic tree search.arXiv preprint arXiv:2504.08066, 2025

    2025 arXiv

  16. [24]

    Benchmarking data science agents

    Yuge Zhang, Qiyang Jiang, XingyuHan XingyuHan, Nan Chen, Yuqing Yang, and Kan Ren. Benchmarking data science agents. InProceedings of the 62nd Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pages 5677–5700, 2024

    2024

  17. [25]

    Toward reliable scientific hypothesis generation: Evaluat- ing truthfulness and hallucination in large language models.arXiv preprint arXiv:2505.14599, 2025

    Guangzhi Xiong, Eric Xie, Corey Williams, Myles Kim, Amir Hassan Shariatmadari, Sikun Guo, Stefan Bekiranov, and Aidong Zhang. Toward reliable scientific hypothesis generation: Evaluat- ing truthfulness and hallucination in large language models.arXiv preprint arXiv:2505.14599, 2025

    2025 arXiv

  18. [26]

    From ai for science to agentic science: A survey on autonomous scientific discovery.arXiv preprint arXiv:2508.14111, 2025

    Jiaqi Wei, Yuejin Yang, Xiang Zhang, Yuhan Chen, Xiang Zhuang, Zhangyang Gao, Dongzhan Zhou, Guangshuai Wang, Zhiqiang Gao, Juntai Cao, et al. From ai for science to agentic science: A survey on autonomous scientific discovery.arXiv preprint arXiv:2508.14111, 2025

    2025

  19. [27]

    Factscore: Fine-grained atomic evaluation of factual precision in long form text generation

    Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. InProceedings of the 2023 Conference on Empirical Meth...

    2023

  20. [28]

    Refchecker: Reference-based fine-grained hallucination checker and benchmark for large language models.arXiv preprint arXiv:2405.14486, 2024

    Xiangkun Hu, Dongyu Ru, Lin Qiu, Qipeng Guo, Tianhang Zhang, Yang Xu, Yun Luo, Pengfei Liu, Yue Zhang, and Zheng Zhang. Refchecker: Reference-based fine-grained hallucination checker and benchmark for large language models.arXiv preprint arXiv:2405.14486, 2024

    2024 arXiv

  21. [29]

    Trace: Trajectory-aware comprehensive evaluation for deep research agents

    Yanyu Chen, Jiyue Jiang, Jiahong Liu, Yifei Zhang, Xiao Guo, and Irwin King. Trace: Trajectory-aware comprehensive evaluation for deep research agents. InProceedings of the ACM Web Conference 2026, pages 2524–2534, 2026

    2026

  22. [30]

    Which agent causes task failures and when? on automated failure attribution of llm multi-agent systems.arXiv preprint arXiv:2505.00212, 2025

    Shaokun Zhang, Ming Yin, Jieyu Zhang, Jiale Liu, Zhiguang Han, Jingyang Zhang, Beibin Li, Chi Wang, Huazheng Wang, Yiran Chen, et al. Which agent causes task failures and when? on automated failure attribution of llm multi-agent systems.arXiv preprint arXiv:2505.00212, 2025

    2025 arXiv

  23. [31]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024. 12

    2024 arXiv

  24. [32]

    Openai gpt-5 system card.arXiv preprint arXiv:2601.03267, 2025

    Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, et al. Openai gpt-5 system card.arXiv preprint arXiv:2601.03267, 2025

    2025 arXiv

  25. [33]

    Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, SH Cai, Yuan Cao, Y Charles, HS Che, Cheng Chen, Guanduo Chen, et al. Kimi k2. 5: Visual agentic intelligence.arXiv preprint arXiv:2602.02276, 2026

    2026 arXiv

  26. [34]

    The llama 4 herd: Architecture, training, evaluation, and deployment notes.arXiv preprint arXiv:2601.11659, 2026

    Aaron Adcock, Aayushi Srivastava, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pande, Ab- hinav Pandey, Abhinav Sharma, Abhishek Kadian, Abhishek Kumawat, Adam Kelsey, et al. The llama 4 herd: Architecture, training, evaluation, and deployment notes.arXiv preprint arXiv:2601.11659, 2026

    2026

  27. [35]

    Self-alignment pretraining for biomedical entity representations

    Fangyu Liu, Ehsan Shareghi, Zaiqiao Meng, Marco Basaldella, and Nigel Collier. Self-alignment pretraining for biomedical entity representations. InProceedings of the 2021 conference of the North American chapter of the association for computational linguistics: human language ...

    2021

  28. [36]

    high-score illu- sion

    Kun Luo, Zheng Liu, Shitao Xiao, and Kang Liu. Bge landmark embedding: A chunking-free embedding method for retrieval augmented long-context large language models.arXiv preprint arXiv:2402.11573, 2024. 13 A Related Work A.1 Scientific Agents The deployment of LLMs as scientifi...

    2024 arXiv

  29. [37]

    off-graph but plausible

    Judge Allocation and Caching.To prevent self-evaluation bias, gpt-4o serves as the default judge, with the sole exception that intern-s1-pro is substituted when grading gpt-4o’s own trajectories. Finally, both extractions and judgements are heavily disk-cached (keyed by step-t...

  30. [38]

    **Actively use ToolUniverse tools** when they can help answer scientific questions.,→

  31. [39]

    For scientific/technical questions, always attempt to find relevant tools before relying solely on general knowledge.,→

  32. [40]

    Use`find_tools`to discover appropriate ToolUniverse tools for the task

  33. [41]

    Balance tool usage with reasoning - tools should enhance, not replace, analytical thinking.,→ ## Tool Search Strategy

  34. [42]

    **When to search for tools**: - Scientific calculations, simulations, or data analysis - Domain-specific queries (biology, chemistry, physics, etc.) - Questions requiring specialized knowledge or databases - Tasks that could benefit from computational tools

  35. [43]

    protein structure analysis PDB

    **How to search**: - Build focused queries from key entities and domain terms - Example: "protein structure analysis PDB", "molecular dynamics simulation", "gene expression analysis",→ - If first search doesn't find suitable tools, try a broader query - Maximum 2-3 search atte...

  36. [44]

    **Tool validation**: - Check if the tool directly addresses the question - Verify required parameters are available - Ensure the tool output will help answer the question - Always use the **exact parameter names** defined in each tool's schema - If unsuitable, proceed with rea...

  37. [45]

    **When to use general web retrieval**: - Questions requiring up-to-date information not covered by ToolUniverse tools,→ - Verifying facts, finding references, or retrieving specific data from websites,→ - When ToolUniverse tools are not available or suitable for the task

  38. [46]

    **How to use general web retrieval**: - Call`web_search`first to get an overview of available sources (snippets + URLs),→ - If the snippets contain enough information, use them directly - no further fetching needed,→ - If more detail is required, call`web_fetch`with 1-3 URLs f...

  39. [47]

    **Search -> Fetch decision**: 23 - Snippets sufficient: factual lookups, simple definitions, quick verification,→ - Fetch needed: detailed methodology, full page content, comprehensive data ## Literature Search Strategy

  40. [48]

    **When to use literature search**: - Questions asking for paper-backed scientific evidence - Requests about studies, experiments, results, authors, DOI, journals, or publication history,→ - Cases where you need to identify relevant scholarly papers before reading full text,→

  41. [49]

    **How to use literature search**: - Call`literature_search`to retrieve candidate papers and citation-style metadata (title, authors, DOI, venue, snippet) - this is METADATA ONLY,→ - Call`literature_fetch`when you need the **body text** of one or more papers (methods, figures, ...

  42. [50]

    **Literature vs. web**: -`literature_search`: discover papers and scholarly metadata (fast, cheap; use freely),→ -`literature_fetch`: download + parse paper full text to markdown (SLOW, call AT MOST ONCE per question; use only when body text is actually needed) ,→ ,→ -`web_sea...

  43. [51]

    Consider using tools to verify facts or perform calculations if needed

  44. [52]

    Analyze the question and evaluate each option systematically

  45. [53]

    Output complete reasoning for option evaluation

  46. [54]

    **Answer format**: Use the option letter/number (A, B, C, D, etc.) in the `<answer>`tag.,→ - **CORRECT**:`<answer>A</answer>`or`<answer>B</answer>` - **INCORRECT**:`<answer>By taking multiple images...</answer>`(do not include option content),→ ## Output Behavior

  47. [55]

    **ALWAYS output complete reasoning path**

  48. [56]

    Keep reasoning concise and decision-oriented

  49. [57]

    When tools are used, explain: - Why the tool was selected - What the tool output means - How it contributes to the answer

  50. [58]

    When tools are not available or suitable: - Analyze the question using available context/knowledge - Break down the problem step by step - Evaluate each option (if MCQ) with explicit reasoning - Document the logical path to the conclusion

  51. [59]

    **For MCQ**: Provide final answer as the option letter in `<answer>X</answer>`format (where X is A, B, C, D, etc.).,→

  52. [60]

    The reasoning path is mandatory for all responses.,→ ## Execution Template

    **For open questions**: Provide the answer content in`<answer></answer>` tags.,→ 24 **CRITICAL**: Never skip reasoning steps. The reasoning path is mandatory for all responses.,→ ## Execution Template

  53. [61]

    Parse question and candidate options (if MCQ)

  54. [62]

    **Evaluate tool usage**: - For scientific/technical questions: search for relevant ToolUniverse tools using`find_tools`,→ - For scholarly papers and paper-backed evidence: use`literature_search` - For general knowledge or web-based questions: use`web_search`(and `web_fetch`if ...

  55. [63]

    **Generate reasoning path**: - Explain tool selection and results (if tools used) - Analyze the question/task systematically - For MCQ: evaluate each option with explicit logic - For open questions: build logical argument step by step - Use tool outputs and available knowledge...

  56. [64]

    Return final answer: - **For MCQ**: Use option letter only (A, B, C, D, etc.) - **For open questions**: Provide answer content

  57. [65]

    Therefore the answer is

    **CRITICAL - Termination**: After providing your final answer, you **MUST** append the exact token`TERMINATE`on a new line at the very end of your response. Without this token the system cannot detect that you have finished. Every final response must end with`TERMINATE`. ,→ ,→...

  58. [66]

    If a suitable one exists, call it with the exact parameter names from its schema

    For scientific/technical questions, first try find_tools to discover a relevant ToolUni- verse tool. If a suitable one exists, call it with the exact parameter names from its schema

  59. [67]

    If the snippets/abstracts are enough, stop there; otherwise call literature_fetch at most ONCE to pull full-text markdown of 1–2 papers

    For paper-backed evidence, call literature_search to discover candidate papers. If the snippets/abstracts are enough, stop there; otherwise call literature_fetch at most ONCE to pull full-text markdown of 1–2 papers

  60. [68]

    If the snippets suffice, stop; otherwise call web_fetchwith 1–3 URLs from the results

    For general web facts, call web_search. If the snippets suffice, stop; otherwise call web_fetchwith 1–3 URLs from the results

  61. [69]

    Don’t loop

    Cap each kind of search at 2–3 attempts per question. Don’t loop

  62. [70]

    ## Answer format by question type –multichoice( claim_choice / one_hop_tail / two_hop_tail / vqa): Options are labelled A, B, C,

    If tools aren’t suitable, fall back to reasoning with available knowledge. ## Answer format by question type –multichoice( claim_choice / one_hop_tail / two_hop_tail / vqa): Options are labelled A, B, C, ... . Output the SINGLE option letter inside <answer> tags, e.g. <answer>...

  63. [71]

    Explain tool choices briefly

    Keep reasoning short and decision-oriented. Explain tool choices briefly

  64. [72]

    Use EXACTLY ONE <answer> tag per response, in the format above for the sample’s question type

  65. [73]

    factual claim

    On the line AFTER the answer tag, output the tokenTERMINATE alone so the conversation ends. Without it the run wastes resources. Example (multichoice) literature_search confirmed a TAF2 / Microencephaly association, which matches option A; B–D contradict the cited evidence. <a...

  66. [74]

    {claim_1} ... EVIDENCE: [SUPPORTING_CHUNK] {text} [GRAPH_1HOP] {text} [WEB]({url}) {text} [LITERATURE]({url}) {text} Verdict→severity score (deterministic, applied in Python): s(c) = 0.0ifverdict(c) =supported s(c) = 0.5ifverdict(c) =unverifiable s(c) = 1.0ifverdict(c) =refute...

  67. [75]

    Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.