Artificial Intelligence · arXiv:2610.21074 v1 · submitted 15 Oct 2026 · 39 references
OmicsBench: a long-horizon multi-turn benchmark of AI agents on research-study-scale bioinformatics
The paper says
“Computational biologists increasingly delegate substantial analyses to AI agents in long-horizon, multi-turn sessions, yet most benchmarks of agents' bioinformatics capabilities pose a single short task with a single-shot prompt. We present OmicsBench, a long-horizon, multi-turn bioinformatics benchmark for AI agents.”
more of the abstract
Each of its 97 tasks links dependent subtasks into a directed acyclic graph, for 662 underlying subtasks in all. Each subtask is posed as one user turn and graded deterministically against an answer key, with optional in-context feedback on incorrect answers. Across 12 models on three harnesses, the slowest agent averages 3.6 hours per task, and no combination comes near saturating the benchmark, with mean scores of 0.43 to 0.57 out of a possible 1 without in-context feedback and 0.54 to 0.67 with it. Multi-turn evaluation also uncovers behavior that single-shot prompts cannot, as agents score lower on subtasks that depend on earlier subtasks, and in-context feedback helps agents recover some of that loss. Because deterministic scoring gives no qualitative account of how an agent failed, we also introduce an unsupervised failure mode discovery pipeline that describes, embeds, and clusters agents' failed attempts to surface emergent failure modes. Applied to OmicsBench, this pipeline exposes qualitative shortcomings in how agents carry out bioinformatics analyses, ranging from agents applying their own statistical filters in place of the specified ones to agents writing whole answers from memory.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 16 Oct 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. A well-built benchmark with a real multi-turn contribution; the headline turn and dependency claims need controls for subtask type and more public ground truth before they should be read as causal. This line is wrong
3 major objections · 5 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 39 references say.
The paper says, at Section 4.4, Figures 6A/B and 3C
The review did not quote this passage.
Pith says: Objection 1 of 3 · major
The headline multi-turn claim — that agents score lower after the first turn and on subtasks that build on earlier answers (Section 6) — is not identified against subtask type and difficulty. Because the DAG is unrolled in dependency order, turn position and dependency-chain length are correlated by construction, and analysis content is not placed uniformly across the DAG: roots …
more of objection 1
… are disproportionately early-stage analyses such as differential expression (165 subtasks), marker genes (103), and pathway enrichment (118) per Figure 3C, whereas downstream nodes are more often open-ended types such as mechanistic inference, gene programs, and signature scoring, as in the example task in Appendix D. The pooled comparisons in Figure 6A/B therefore conflate session position with subtask type; the stated exclusion of the 177 closed-choice subtasks from these analyses changes the mix without controlling it. No model enters turn position and chain length jointly, and no adjustment for subtask type or task difficulty is reported, so the two effects are not separately identified and the pattern could survive with no true turn or dependency effect. The Section 6 statement that the design lets the benchmark 'separate a subtask's position in the session from its place in a dependency chain' describes a capability the analysis does not exercise. A fix is feasible with the existing data: within-type comparisons for the abundant types (for example, differential-expression subtasks at chain length 1 versus 2+), a mixed-effects model with task and type fixed effects entering turn and chain length jointly, and a report of the turn-by-chain-length cross-tab to show the identification region.
The paper says, at Appendix E; Section 3.2
The review did not quote this passage.
Pith says: Objection 2 of 3 · major
Only 3 of the 97 tasks are released, so the answer keys and grader configurations behind the other 94 cannot be inspected by readers, and every headline score, turn-decay statistic, and failure-mode number in Sections 4.2–4.5 rests on those keys.
more of objection 2
The human review stage is described only as 'Human experts then review and edit each task, or reject it': no number of reviewers, no edit/reject counts, no inter-reviewer agreement, and no description of what the review actually checked. The reviewer agent that certifies tasks as 'supported by their papers and free of data leakage' is an LLM, and the automated quality checks validate format rather than biological fidelity. The paper should quantify the human review (number of reviewers, tasks rejected or edited, typical edits), and ideally report an independent audit of a random sample of answer keys with agreement statistics; a substantially larger public development set would also let readers calibrate the graders beyond the single worked example in Appendix D.
The paper says, at Section 3.4; Figure 7; Discussion, Limitations
The review did not quote this passage.
A classifier read this sentence of the paper as acknowledging the concern (probability 0.83):
“Lastly, the LLM judge was tested for reproducibility but not against human annotators.”
Pith says: Objection 3 of 3 · major
The failure-mode findings, which the abstract and conclusion present as a contribution, are produced entirely by an Opus 5 LLM judge, and Opus 5 is itself one of the 12 evaluated agents. The Discussion correctly states that the judge was tested for reproducibility but not against human annotators, and the judge is a genuine soft spot rather than a proven …
more of objection 3
… bias; however, self-agreement (Cohen's kappa 0.76 over repeated passes) establishes reproducibility, not accuracy of the failure descriptions or mode assignments, and a same-family judge evaluating a same-family agent (with names removed) is a residual correctness risk for specific claims such as the Grok 4.3 fabrication pattern. The paper should either validate the judge on a human-labeled sample of failed attempts or explicitly present the failure-mode results as exploratory rather than as headline findings.
5 minor objections
The paper says, at Section 4.4
Pith says: minor objection 1 of 5
Please state whether Figures 6A/B pool the non-closed-choice subtask scores directly across tasks or average subtask scores per task first. Pooled subtask-level analyses would let tasks with more subtasks dominate the means and would require confidence intervals that account for within-task correlation of subtask scores.
The paper says, at Introduction
Pith says: minor objection 2 of 5
The claim that 'the slowest agent averages 3.6 hours per task, 350 tool calls, and 110 million tokens per task' should identify which combination and mode attains each maximum, since these figures vary widely across combinations (Figure 5).
show 3 more
The paper says, at Section 4.3, Figure 5D
Pith says: minor objection 3 of 5
The footnote stating that token counts are not directly comparable between harnesses should be repeated in the Figure 5 caption, since Figure 5D plots token counts for Claude Code and Codex runs alongside OpenCode runs.
The paper says, at Table F1
Pith says: minor objection 4 of 5
Please mark which graders are the closed-choice graders whose subtasks are excluded from Section 4.4, so that the exclusion of the 177 subtasks can be verified from Table F1 as the text implies.
The paper says, at Section 3.2
Pith says: minor objection 5 of 5
Please specify which models serve as the writer and reviewer agents in the construction pipeline; this bears on the leakage-certification claim and on the independence of task construction from the evaluated models.
What would settle it
Re-estimate Figures 6A and 6B from the per-subtask score table used to make them with a mixed-effects model (per combination, or pooled with combination fixed effects): outcome = subtask score; predictors = turn number, dependency chain length, and the 18 subtask types; random intercept per task; use only the 485 non-closed-choice subtasks and re-run as a sensitivity check with task mean score added as a difficulty covariate. Report the turn and chain-length coefficients with 95% confidence intervals. If the coefficients move to within 0.01 of zero, change sign, or lose significance once subtask type and task random effects are included, the Section 4.4 claim is a composition artifact; if they remain negative with CIs excluding zero, the concern is resolved and the paper's conclusion survives.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 16 Oct 2026
- Reader holds up, with conditions
- Skeptic partly agrees
- Editor wrote the verdict above
- Referee major revision: 3 major objections, 5 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
OmicsBench is a real benchmark contribution, and it is the most carefully measured multi-turn agent eval for bioinformatics I have read. The core design - DAG-unrolled single-session turns, deterministic grading with synonym-aware entity matching, gated retries, per-task confidence intervals over 97 tasks, run-to-run variance, no-network containers - is a genuine step past the single-prompt long-horizon benchmarks it positions against. The cost/efficiency comparison is useful even with the acknowledged harness confound. The unsupervised failure-mode pipeline (LLM narrative, embedding, UMAP, HDBSCAN, hand-edited labels) is new in this space and produces specific, plausible failure families rather than another fixed taxonomy. The paper earns a serious referee. The soft spots, in order. First, 94 of 97 tasks are private. The authors disclose this, but it is load-bearing: every headline score, turn-decay number, and failure-mode statistic rests on answer keys and grader configs no one outside can inspect. Releasing the runner and three tasks is real but partial. Second, the Section 4.4 claims are not statistically identified. Figure 6A/B pools subtasks without controlling for subtask type or difficulty, and the DAG construction makes turn position, dependency length, and subtask type correlated by design - roots tend to be marker-gene and differential-expression tasks; downstream nodes tend to be more open-ended inference, pseudotime, or signature tasks. The observed pattern could be composition rather than session position. The stress-test note lands on this, and I agree with it. The monotonic pattern across all 12 combinations and partial gated recovery make a real effect plausible, but the abstract and conclusion state the causal reading more strongly than the analysis supports. Third, the LLM judge is Opus 5, same family as the best agent, and was validated only for self-reproducibility, not against humans; the authors say so. Fourth, excluding 177 closed-choice subtasks from the session/retry analyses is disclosed as reward-hacking mitigation, but it changes the task mix supporting the dependency and recovery findings. Who gets value: anyone building or buying bioinformatics agents, and anyone designing agent benchmarks. The related-work placement is accurate; the differences from BixBench3 and GeneBench are real. I would send this to peer review, not desk reject. In revision I would ask for controls of subtask type and difficulty jointly with turn position and dependency length, and for release of more than three tasks, or an auditable sample with keys. The benchmark itself is worth engaging with now.
The strongest objection
The strongest claim is the multi-turn/dependency finding: every combination loses 0.06 to 0.12 by the second turn and scores lower on dependent subtasks (Section 4.4, Figure 6A/B). The support for this claim pools all subtasks across 97 tasks without controlling for what kind of subtask sits where. Figure 3C shows the 18 subtask types are heavily skewed, and the DAG construction makes their placement non-uniform: root subtasks are disproportionately early analyses such as marker genes and differential expression, while downstream nodes ask for mechanistic inference, gene programs, pseudotime, or signature scoring. Those later analysis types are intrinsically harder and more open-ended, so Figure 6A/B would look the same if there were no true turn or dependency effect at all. Turn position and dependency chain length are also correlated by construction, since longer chains appear later in the unrolled session, and they are never entered jointly in a model, so the two effects are not separately identified. The exclusion of the 177 closed-choice subtasks from these analyses, stated in Section 4.4, does not remove this composition confound; it changes the subtask mix without controlling it. This is independent of the ground-truth auditability issue: even with perfect answer keys, the paper's central behavioral claim could be an artifact of which subtask type is placed at which depth.
Reading the paper in good faith, the benchmark is real and serious: deterministic graders, a released runner, three public tasks, run-to-run variance reporting, and open acknowledgment of reward hacking, harness confounds, and judge limitations. The reader's private-ground-truth concern is legitimate, but I do not think it is the most proximal threat to the paper's headline experimental claim. Even granting the answer keys, the turn/dependency analysis in Section 4.4 is a pooled correlational comparison of subtask scores without covariates. Because an OmicsBench session unrolls the DAG in dependency order, turn position and dependency depth are assigned by the task designer, not randomized over content; the observed differences could be carried entirely by the systematic difficulty of downstream analysis types. This is a testable internal-validity threat, and it is currently unresolved by the paper's text. The concrete mixed-model test would settle it using data the authors already have. I therefore keep the reader's conditional disposition: if the adjusted coefficients survive, the paper's claim stands; if they do not, the strongest claim needs substantial qualification. This makes my concern load-bearing, but it does not by itself change the overall verdict from the reader's CONDITIONAL, so no adjustment is needed.
The referee's report
The benchmark fills a genuine gap: it combines research-study-scale DAG structure with per-subtask deterministic grading and session-level multi-turn interaction, going beyond both single-turn coding benchmarks and single-prompt long-horizon ones. The evaluation is careful in several respects: run-to-run variance is reported (Figure A2), confidence intervals are computed over the 97-task sample with runs averaged per task, gated gains are explicitly flagged as upper bounds because of reward hacking, and the paper makes no claims about harness effects. The released runner, grader catalog, judge prompts, and UMAP/HDBSCAN hyperparameters (Appendix I) make the evaluation checkable. Two issues gate the headline claims: the Section 4.4 turn/dependency finding is exposed to a composition confound with subtask type and difficulty, and the private 94-task ground truth is not independently auditable. The failure-mode results are reproducible but rest on an unvalidated same-family LLM judge. All three issues are fixable within the manuscript's scope.
Pith's reading of the paper (a machine's summary, not the authors')
OmicsBench is a benchmark built to test AI agents the way a computational biologist actually delegates work: a published study's analysis is split into dependent steps, each step is posed as one turn of a single long session, and each answer is graded before the next step begins. The paper's aim is to show that this multi-turn format measures something single-shot benchmarks cannot, and that the measured behavior matters for anyone trusting agents with real analyses. Across 12 frontier models on three harnesses, no combination scored above 0.57 out of 1.0 without feedback, every agent's accuracy dropped after the first turn and on steps that built on earlier answers, and graded retries recovered only part of that loss. The paper further claims that an unsupervised clustering of failed attempts surfaces 12 recurring failure modes, most often agents submitting larger answer sets than requested or substituting their own statistical criteria for the subtask's stated ones, and that these modes can serve as concrete training targets. If the benchmark is a fair test, current agents are not yet reliable enough to carry out multi-step research analyses unattended.
What carries the argument. The machinery is the task DAG combined with deterministic per-step grading. Each of the 97 tasks is a directed acyclic graph, a workflow in which every dependency points forward and no step can feed back into an earlier one, with 662 subtask nodes in total; the runner unrolls the graph in dependency order and delivers each subtask as a user turn in one unbroken session, so the agent's earlier answer files and reasoning stay in context. Each subtask carries an evaluation set of one or more graders paired with answer keys, a subtask score is the mean of its grader scores, $s_v = \frac{1}{|v_e|} \sum_i g_i(a_v, y_i)$, and the task score is the mean over subtasks. The 58 graders are biologically informed: they match genes, organisms, metabolites, microRNAs, and antimicrobial resistance genes through reference databases rather than by exact string matching, and they ignore what is biologically immaterial, such as the ordering of enrichment terms. In gated mode, an answer scoring below the threshold $\tau = 0.7$ is returned with a retry prompt for up to $\theta = 3$ attempts, and the best attempt is kept for the task score. This combination is what lets the paper separate a subtask's turn position from its place in the dependency chain.
Core claim. The paper's central discovery is a behavioral regularity that only a multi-turn setup exposes: an agent's accuracy on a bioinformatics subtask depends on where that subtask sits in a session. Every one of the 14 model/harness combinations evaluated loses between 0.06 and 0.12 by the second turn compared with the first, later turns stay below the first in all but one case, and subtasks with upstream dependencies score lower than independent ones for every combination. Gated in-context feedback, which returns a failing answer with a retry prompt for up to three attempts, raises every combination's score by 0.09 to 0.13, yet only 22 to 34 percent of subtasks reach the pass gate within three attempts, so feedback repairs part of the loss but not most of it. A second discovery is methodological: describing failed attempts, embedding those descriptions, and clustering them recovers 12 emergent failure modes that a predefined vocabulary would miss, the largest families being submitting answer sets larger than requested and substituting the agent's own selection criterion for the one the subtask states. The authors present these modes as concrete behaviors for future models to improve upon, and the same failure families dominate across all combinations.
The premise it rests on. The load-bearing premise is that the answer keys for all 97 tasks, drafted by a writer/reviewer agent pipeline and edited by human experts from published studies, are correct and biologically faithful; because only 3 of the 97 tasks are publicly released with their keys, the other 94 cannot be independently audited by readers, yet every headline score, turn-decay finding, and failure-mode statistic rests on them.
If the paper is right (Pith's inferences, not the authors')
Multi-turn, per-step evaluation should be adopted more widely in agent benchmarks, because the paper shows that single-shot prompts miss a systematic, measurable decline in accuracy across a session.
For practitioners, the benchmark implies that delegating a long analysis to a current agent in one session accumulates errors, and that asking the agent to check and repair each step only partially offsets the decline.
The 12 emergent failure modes give training a concrete agenda centered on instruction-following: respect the stated filter, threshold, and answer-set size rather than the agent's own defaults.
Because models separated far more by cost than by score, with a 12-fold cost difference between combinations whose scores were nearly equal, the paper implies that choosing an agent for bioinformatics work is as much an economic decision as a capability one.
The gated-mode gains are an upper bound, as the authors acknowledge, because some share of the improvement may come from reward hacking rather than genuine repair of the analysis.
The withheld 94 tasks can keep the benchmark usable against future models only if answer-key secrecy is preserved, so the public three tasks cannot by themselves support the paper's headline statistics if the private keys change or age poorly.
A testable extension follows from the turn-decay finding: agent architectures that keep sessions short, re-ground the model in the original data between steps, or checkpoint intermediate results should recover part of the observed gap, which is a prediction the paper does not run.
The unsupervised failure-mode pipeline is domain-agnostic and should transfer to other agent fields, such as software engineering or scientific literature review, where failed attempts are abundant and predefined failure taxonomies are known to be incomplete.
Because ground truth is built from published papers, a model that has seen those papers during training could reproduce results from memory rather than from the data; the paper itself observes answers written from memory as a failure mode, which suggests contamination control will partly determine whether OmicsBench stays valid as models train on the literature.
A concrete audit design follows from the paper's own release policy: publishing a randomly chosen half of the 94 withheld answer keys would let independent groups verify that scores reflect genuine analysis quality rather than grader quirks, while keeping the rest usable for future evaluation.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Computational biologists increasingly delegate substantial analyses to AI agents in long-horizon, multi-turn sessions, yet most benchmarks of agents' bioinformatics capabilities pose a single short task with a single-shot prompt. We present OmicsBench, a long-horizon, multi-turn bioinformatics benchmark for AI agents. Each of its 97 tasks links dependent subtasks into a directed acyclic graph, for 662 underlying subtasks in all. Each subtask is posed as one user turn and graded deterministically against an answer key, with optional in-context feedback on incorrect answers. Across 12 models on three harnesses, the slowest agent averages 3.6 hours per task, and no combination comes near saturating the benchmark, with mean scores of 0.43 to 0.57 out of a possible 1 without in-context feedback and 0.54 to 0.67 with it. Multi-turn evaluation also uncovers behavior that single-shot prompts cannot, as agents score lower on subtasks that depend on earlier subtasks, and in-context feedback helps agents recover some of that loss. Because deterministic scoring gives no qualitative account of how an agent failed, we also introduce an unsupervised failure mode discovery pipeline that describes, embeds, and clusters agents' failed attempts to surface emergent failure modes. Applied to OmicsBench, this pipeline exposes qualitative shortcomings in how agents carry out bioinformatics analyses, ranging from agents applying their own statistical filters in place of the specified ones to agents writing whole answers from memory.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-16
The authors' main claim, as the classifier picked it out 0.86
“Multi-turn evaluation also uncovers behavior that single-shot prompts cannot, as agents score lower on subtasks that depend on earlier subtasks, and in-context feedback helps agents recover some of that loss.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.16 · states limitation 0.80 · human participants 0.06 · controlled comparison 0.44 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.66 · mentions code or data availability 0.06
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Computational biologists increasingly delegate substantial analyses to AI agents in long-horizon, multi-turn sessions, yet most benchmarks of agents' bioinformatics capabilities pose a single short task with a single-shot prompt. | gap 0.65 | 0.07 | 0.05 | asserted 1.00 |
| We present OmicsBench, a long-horizon, multi-turn bioinformatics benchmark for AI agents. | contribution 1.00 | 0.90 | 0.11 | asserted 1.00 |
| Each of its 97 tasks links dependent subtasks into a directed acyclic graph, for 662 underlying subtasks in all. | method 0.75 | 0.92 | 0.05 | asserted 1.00 |
| Each subtask is posed as one user turn and graded deterministically against an answer key, with optional in-context feedback on incorrect answers. | method 0.99 | 0.86 | 0.04 | asserted 1.00 |
| Across 12 models on three harnesses, the slowest agent averages 3.6 hours per task, and no combination comes near saturating the benchmark, with mean scores of 0.43 to 0.57 out of a possible 1 without in-context feedback and 0.54 to 0.67 with it. | result 1.00 | 0.96 | 0.19 | asserted 1.00 |
| Multi-turn evaluation also uncovers behavior that single-shot prompts cannot, as agents score lower on subtasks that depend on earlier subtasks, and in-context feedback helps agents recover some of that loss. | result 0.86 | 0.93 | 0.32 | asserted 1.00 |
| Because deterministic scoring gives no qualitative account of how an agent failed, we also introduce an unsupervised failure mode discovery pipeline that describes, embeds, and clusters agents' failed attempts to surface emergent failure modes. | contribution 0.88 | 0.95 | 0.08 | asserted 1.00 |
| Applied to OmicsBench, this pipeline exposes qualitative shortcomings in how agents carry out bioinformatics analyses, ranging from agents applying their own statistical filters in place of the specified ones to agents writing whole answers from memory. | result 0.93 | 0.95 | 0.14 | asserted 1.00 |
What kind of paper the classifier reports this is: dataset or benchmark 1.00 · evidence from observation or experiment 0.85
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-16: 1 request, 6,541 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (7)
Figures are the authors', reproduced from the paper.
References (39)
- [1] Afra Feyza Akyürek, Xinming Tu, Alec Gutmanstein, Jason Qin, Divyansh Agarwal, Sofia Monasdotter, Sergey Chekhov, Brenda Hernandez Villegas, Kirill Chugunov, Judah Engel, Veronica Chatrath, Oscar Kavanagh, Geobio Boo, Ernesto Hernandez, Ying Liu, Emily Xue, Aakash Sabharwal, D... on Pith
- [2] Samuel Alber, Bowen Chen, Eric Sun, Alina Isakova, Aaron J. Wilk, and James Zou. CellVoyager : AI CompBio agent generates new insights by autonomously analyzing biological data. Nature Methods, 23 0 (4): 0 749--759, April 2026. ISSN 1548-7105. doi:10.1038/s41592-026-03029-6. U... on Pithdoi:10.1038/s41592-026-03029-6
- [3] Brian P. Alcock, William Huynh, Romeo Chalil, Keaton W. Smith, Amogelang R. Raphenya, Mateusz A. Wlodarski, Arman Edalatmand, Aaron Petkau, Sohaib A. Syed, Kara K. Tsang, Sheridan J. C. Baker, Mugdha Dave, Madeline C. McCarthy, Karyn M. Mukiri, Jalees A. Nasir, Bahar Golbon, H... on Pith
- [4] Anthropic . Evaluating Claude ’s bioinformatics research capabilities with BioMysteryBench , 2026. URL https://www.anthropic.com/research/Evaluating-Claude-For-Bioinformatics-With-BioMysteryBench on Pith
- [5] Aashka Bhowmick, Nasim Rahmatpour, Shivang Sharma, Qian Xu, Ananya Gupta, Asmita Lagwankar, Arjun Banerjee, and Christopher Zou. VariantBench : An Agentic Benchmark for Genetic Variant Discovery and Interpretation , 2026. URL https://latch.bio/variantbench on Pith
- [6] Garth R. Brown, Vichet Hem, Kenneth S. Katz, Michael Ovetsky, Craig Wallin, Olga Ermolaeva, Igor Tolstoy, Tatiana Tatusova, Kim D. Pruitt, Donna R. Maglott, and Terence D. Murphy. Gene: a gene-centered information resource at NCBI . Nucleic Acids Research, 43 0 (D1): 0 D36--D4... on Pithdoi:10.1093/nar/gku1055
- [7] Ricardo J. G. B. Campello, Davoud Moulavi, and Joerg Sander. Density- Based Clustering Based on Hierarchical Density Estimates . In Jian Pei, Vincent S. Tseng, Longbing Cao, Hiroshi Motoda, and Guandong Xu (eds.), Advances in Knowledge Discovery and Data Mining , pp.\ 160--172... on Pithdoi:10.1007/978-3-642-37456-2_14
- [8] Paolo Di Tommaso, Maria Chatzou, Evan W. Floden, Pablo Prieto Barja, Emilio Palumbo, and Cedric Notredame. Nextflow enables reproducible computational workflows. Nature Biotechnology, 35 0 (4): 0 316--319, April 2017. doi:10.1038/nbt.3820. URL https://www.nature.com/articles/nbt.3820 on Pithdoi:10.1038/nbt.3820
- [9] Ian Diks, Harihara Muralidharan, Tim Proctor, and Kenny Workman. Verifiable Benchmarking of Long - Horizon Spatial Biology , May 2026 a . URL http://arxiv.org/abs/2605.28065. arXiv:2605.28065 [cs.AI] on PithPith's reviewarXiv
- [10] Ian Diks, Zhen Yang, Arjun Banerjee, Tim Proctor, and Kenny Workman. scBench - Long : Verifiable Benchmarking of Long - Horizon Single - Cell Biology , June 2026 b . URL http://arxiv.org/abs/2606.26563. arXiv:2606.26563 [q-bio.GN] on PithPith's reviewarXiv
- [11] Dionizije Fa, Marko Čuljak, Bruno Pandža, and Mateo Čupić. BioAgent Bench : An AI Agent Evaluation Suite for Bioinformatics . In Proceedings of the 43rd International Conference on Machine Learning , volume 306 of Proceedings of Machine Learning Research , pp.\ 28566--28583. P... on Pith
- [12] Martin Jacko, Jackson Brougher, Alex Urrutia, Hannah Le, Arjun Banerjee, Dillon Flood, and Kenny Workman. TxBench - Oligonucleotide Discovery : Benchmarking AI Agents on Experimentally Grounded Decisions in the Discovery of ASO / siRNA Therapeutics , 2026. URL https://latch.bi... on Pith
- [13] Zane Koch, Asmamaw T. Wassie, Javier Valdes-Aleman, Jason Lee, Michaela M. Hinks, Samuel G. Rodriques, Andrew D. White, and Jon M. Laurent. BixBench3 : Benchmarking AI agents on research-study-scale computational biology tasks, August 2026. URL http://arxiv.org/abs/2608.25286.... on PithPith's reviewarXiv
- [14] Ana Kozomara, Maria Birgaoanu, and Sam Griffiths-Jones. miRBase : from microRNA sequences to function. Nucleic Acids Research, 47 0 (D1): 0 D155--D162, January 2019. ISSN 0305-1048. doi:10.1093/nar/gky1141. URL https://doi.org/10.1093/nar/gky1141 on Pithdoi:10.1093/nar/gky1141
- [15] Johannes Köster and Sven Rahmann. Snakemake—a scalable bioinformatics workflow engine. Bioinformatics, 28 0 (19): 0 2520--2522, October 2012. doi:10.1093/bioinformatics/bts480. URL https://doi.org/10.1093/bioinformatics/bts480 on Pithdoi:10.1093/bioinformatics/bts480
- [16] Jon M. Laurent, Joseph D. Janizek, Michael Ruzo, Michaela M. Hinks, Michael J. Hammerling, Siddharth Narayanan, Manvitha Ponnapati, Andrew D. White, and Samuel G. Rodriques. LAB - Bench : Measuring Capabilities of Language Models for Biology Research , July 2024. URL http://ar... on PithPith's reviewarXiv
- [17] Jon M. Laurent, Albert Bou, Michael Pieler, Conor Igoe, Alex Andonian, Siddharth Narayanan, James Braza, Alexandros Sanchez Vassopoulos, Jacob L. Steenwyk, Blake Lash, Andrew D. White, and Samuel G. Rodriques. LABBench2 : An Improved Benchmark for AI Systems Performing Biology... on PithPith's reviewarXiv
- [18] Hannah Le, Ramesh Ramasamy, Alex Urrutia, Mahsa Yazdani, Tim Proctor, and Kenny Workman. TxBench - PP : Analyzing AI Agent Performance on Small - Molecule Preclinical Pharmacology , June 2026. URL http://arxiv.org/abs/2606.19245. arXiv:2606.19245 [cs.AI] on PithPith's reviewarXiv
- [19] Jeremy Li and Andrew Ho. GeneBench : Assessing AI Agents for Multi - Stage Inference Problems in Genomics and Quantitative Biology , April 2026. URL https://www.biorxiv.org/content/10.64898/2026.04.22.720113v1 on Pithdoi:10.64898/2026.04.22.720113v1
- [20] Jeremy Li, Suyash Shringarpure, Edmund Wong, and Andrew Ho. GeneBench - Pro : Evaluating Multistage Statistical Reasoning in Genomics , Quantitative Biology , and Translational Biomedicine , July 2026. URL https://www.biorxiv.org/content/10.64898/2026.06.29.735386v4 on Pithdoi:10.64898/2026.06.29.735386v4
show 19 more
- [21] Haoyang Liu, Shuyu Chen, Ye Zhang, and Haohan Wang. GenoTEX : An LLM Agent Benchmark for Automated Gene Expression Data Analysis . In Machine Learning in Computational Biology ( MLCB 2025) , 2025. doi:10.48550/arXiv.2406.15341. URL http://arxiv.org/abs/2406.15341. arXiv:2406.1... on PithPith's reviewarXivdoi:10.48550/arxiv.2406.15341
- [22] Yuyang Liu, Liuzhenghao Lv, Xiancheng Zhang, Jingya Wang, Li Yuan, and Yonghong Tian. BioProBench : A Corpus and Benchmark for Biological Protocol Reasoning in Autonomous Science . In Proceedings of the 43rd International Conference on Machine Learning , volume 306 of Proceedi... on Pith
- [23] Erpai Luo, Jinmeng Jia, Yifan Xiong, Xiangyu Li, Xiaobo Guo, Baoqi Yu, Minsheng Hao, Lei Wei, and Xuegong Zhang. Benchmarking AI scientists for omics data-driven biological discovery, January 2026. URL http://arxiv.org/abs/2505.08341. arXiv:2505.08341 [cs.AI] on PitharXiv
- [24] Leland McInnes, John Healy, and James Melville. UMAP : Uniform Manifold Approximation and Projection for Dimension Reduction , 2018. URL https://arxiv.org/abs/1802.03426 on PithPith's reviewarXiv
- [25] Evan Miller. Adding Error Bars to Evals : A Statistical Approach to Language Model Evaluations , 2024. URL https://arxiv.org/abs/2411.00640 on PithPith's reviewarXiv
- [26] Henry E. Miller, Matthew Greenig, Benjamin Tenmann, and Bo Wang. BioML -bench: Evaluation of AI Agents for End -to- End Biomedical ML , September 2025. URL https://www.biorxiv.org/content/10.1101/2025.09.01.673319v3 on Pithdoi:10.1101/2025.09.01.673319v3
- [27] Ludovico Mitchener, Jon M. Laurent, Alex Andonian, Benjamin Tenmann, Siddharth Narayanan, Geemi P. Wellawatte, Andrew White, Lorenzo Sani, and Samuel G. Rodriques. BixBench : a Comprehensive Benchmark for LLM -based Agents in Computational Biology , October 2025. URL http://ar... on PitharXiv
- [28] Harihara Muralidharan, Reema Baskar, Soo Hee Lee, Tim Proctor, and Kenny Workman. EpiBench : Verifiable Evaluation of AI Agents on Epigenomics Analysis , June 2026. URL http://arxiv.org/abs/2606.13602. arXiv:2606.13602 [cs.AI] on PithPith's reviewarXiv
- [29] Surag Nair, Laura Gunsalus, Brian Orcutt-Jahns, Jordan Rossen, Avantika Lal, Carlo De Donno, Muhammed Hasan Çelik, Kipper Fletez-Brant, Xiaoman Xie, Hector Corrada Bravo, and Gokcen Eraslan. Agentic systems are adept at solving well-scoped, verifiable problems in computational... on Pithdoi:10.64898/2026.04.06.716850v2
- [30] Yuanhao Qu, Yingzhou Lu, Xinming Tu, Serena Zhang, Tianwei She, Alexander Glenn Shaw, Jou-Ho Shih, Bingqing Zhao, Minjie Shen, Haochen Yang, Jielin Yan, Rongchuan Zhang, Xinze Wu, Tingting Li, Bin Zhou, Ning Wang, Adam Ma, Le Cong, Xiaobo Hu, Yuan Jiang, Jiayun Dong, Tao Peng,... on Pithdoi:10.64898/2026.05.12.724604v2
- [31] Ramesh Ramasamy, Hannah Le, Jackson Brougher, Alex Urrutia, Arjun Banerjee, Sean Poust, and Kenny Workman. TxBench - Antibody Discovery : Benchmarking AI Agents on Experimentally Grounded Decisions in Therapeutic Antibody Discovery , 2026. URL https://latch.bio/txbench-ab on Pith
- [32] Conrad L Schoch, Stacy Ciufo, Mikhail Domrachev, Carol L Hotton, Sivakumar Kannan, Rogneda Khovanskaya, Detlef Leipe, Richard Mcveigh, Kathleen O’Neill, Barbara Robbertse, Shobha Sharma, Vladimir Soussov, John P Sullivan, Lu Sun, Seán Turner, and Ilene Karsch-Mizrachi. NCBI Ta... on Pithdoi:10.1093/database/baaa062
- [33] Xiangru Tang, Bill Qian, Rick Gao, Jiakang Chen, Xinyun Chen, and Mark B Gerstein. BioCoder : a benchmark for bioinformatics code generation with large language models. Bioinformatics, 40 0 (Supplement\_1): 0 i266--i276, July 2024. doi:10.1093/bioinformatics/btae230. URL https... on Pithdoi:10.1093/bioinformatics/btae230
- [34] Terminal-Bench-Science Team . Terminal- Bench - Science : Evaluating AI agents on research workflows across scientific domains, 2026. URL https://github.com/harbor-framework/terminal-bench-science on Pith
- [35] Zifeng Wang, Benjamin Danek, Ziwei Yang, Zheng Chen, and Jimeng Sun. Can Large Language Models Replace Data Scientists in Biomedical Research ?, April 2025. URL http://arxiv.org/abs/2410.21591. arXiv:2410.21591 [cs.AI] on PithPith's reviewarXiv
- [36] David S. Wishart, AnChi Guo, Eponine Oler, Fei Wang, Afia Anjum, Harrison Peters, Raynard Dizon, Zinat Sayeeda, Siyang Tian, Brian L. Lee, Mark Berjanskii, Robert Mah, Mai Yamamoto, Juan Jovel, Claudia Torres-Calzada, Mickel Hiebert-Giesbrecht, Vicki W. Lui, Dorna Varshavi, Do... on Pith
- [37] Kenny Workman, Zhen Yang, Harihara Muralidharan, Aidan Abdulali, and Hannah Le. scBench : Evaluating AI Agents on Single - Cell RNA -seq Analysis , February 2026 a . URL http://arxiv.org/abs/2602.09063 on PitharXiv
- [38] Kenny Workman, Zhen Yang, Harihara Muralidharan, and Hannah Le. SpatialBench : Can Agents Analyze Real - World Spatial Biology Data ?, January 2026 b . URL http://arxiv.org/abs/2512.21907. arXiv:2512.21907 [cs.AI] on PitharXiv
- [39] Ming Yin, Yuanhao Qu, Ling Yang, Le Cong, and Mengdi Wang. Toward Scientific Reasoning in LLMs : Training from Expert Discussions via Reinforcement Learning , June 2025. URL http://arxiv.org/abs/2505.19501. arXiv:2505.19501 [cs.AI] on PithPith's reviewarXiv
Cited by 1 Pith paper
Reviewed papers on Pith that cite this one. The full citation record
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment