{"id":"f79d2ded-8cb8-489f-9efd-d6e615533f6e","arxiv_id":"2505.07889","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"BioProBench provides a large-scale benchmark and training resource for evaluating and improving language models' reasoning about biological experimental protocols.","lead":"This paper introduces BioProBench, a large corpus of over 20,000 biological protocols and more than half a million generated task instances for testing how well AI models understand lab procedures. It also reports that current large language models perform poorly on procedural reasoning and that a retrieval-augmented agent built on the corpus improves performance.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ProAgent's in-benchmark gains may stem from retrieving the very test protocols whose answers are being scored; the paper never states test protocols were excluded from the RAG index, so the headline corpus-value claim is not yet supported.","rationale":"The reader's weakest assumption is the most load-bearing issue: the in-benchmark ProAgent evaluation is interpretable only if the retrieval index excludes the source protocols underlying the test instances, and the paper provides no such statement. This concern directly targets the paper's central practical claim, not just a peripheral detail. The benchmark construction itself has independent support from the described programmatic pipeline, held-out test set, and expert validation, so the appropriate outcome remains CONDITIONAL rather than REJECT. Secondary issues worth fixing include inconsistent counts between the abstract and body (22,413 vs 26,933/27,000 protocols; 523,784 vs 556,171 instances) and mismatched repository URLs, but these are not the load-bearing problem. If the proposed contamination test shows no leakage, the practical claim would be substantially strengthened.","tokens_in":13039,"tokens_out":4466,"duration_ms":45318,"concrete_test":"Inspect the released ProAgent code and knowledge-base manifest, and hash-match the protocol IDs or source texts used in the held-out evaluation against the documents in the RAG index. Then rerun the PQA and GEN evaluations after removing all test-protocol documents from the index. If PQA accuracy drops from 85.08% toward the 70.27% no-RAG baseline, or GEN Step Recall drops from 62.24% toward the 42.8% baseline, the claimed gains are retrieval artifacts rather than evidence that BioProCorpus improves procedural reasoning. Report both sanitized and unsanitized numbers.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's most practically important claim is that grounding an LLM in BioProCorpus yields large gains: PQA accuracy rises from 70.27% to 85.08%, GEN Step Recall from 42.8% to 62.24%, and ERR F1 from 59.27% to 81.9% (Figure 7). For these gains to evidence corpus value, the ProAgent knowledge database must not contain the source protocols used to construct the evaluated test instances. The paper never states this. Section 4 describes the database as a 'Knowledge Database from BioProBench Corpus', and Section 5.2 evaluates on the same BioProBench test set without any mention of removing test-protocol documents from the index. Because every test instance is derived directly from a source protocol, retrieval can supply the answer verbatim: the exact numerical value in PQA, the correct step order in ORD, or the full reference steps in GEN. The paper's LAB-Bench out-of-distribution result (60% to 64%) is consistent with modest transfer, so the much larger in-benchmark gains may be dominated by exact-protocol retrieval. The perturbation analysis in Appendix I addresses model memorization of numeric values, not RAG index contamination. This does not invalidate the benchmark itself, but it invalidates the strongest evidence for the corpus's practical utility.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"BioProBench is a corpus and multi-task benchmark for biological protocol reasoning. The authors collect human-written protocols from six public sources, structure them into a 'BioProCorpus,' and programmatically construct five task families (PQA, ORD, ERR, GEN, REA) totaling over 550,000 instances. The benchmark is evaluated with 10 LLMs, and the authors report that current models perform well on basic comprehension but struggle with quantitative precision, step ordering, error correction, and long-form protocol generation. To demonstrate the corpus's practical value, the paper introduces ProAgent, a retrieval-augmented agent grounded in BioProCorpus, and reports large in-benchmark gains (e.g., PQA accuracy 70.27% to 85.08%, GEN Step Recall 42.8% to 62.24%) as well as a smaller out-of-distribution gain on a LAB-Bench subset (60% to 64%).","tokens_in":13302,"tokens_out":7981,"duration_ms":68982,"significance":"If the benchmark construction is sound, this is a potentially valuable resource: it is large, spans many biology subfields, includes expert validation of the test set, uses deterministic programmatic construction for the ordering and perturbation tasks, and proposes content- and structure-aware metrics that are plausibly more informative than lexical overlap for procedural text. The explicit statement that ground-truth content is derived from human-authored protocols is a strength. However, the central empirical claim that BioProCorpus drives the large ProAgent gains is not yet supported, because the paper never states whether the source protocols behind the held-out test instances were excluded from the retrieval index. The stress-test concern lands: without that exclusion, the in-benchmark improvements may be due to answer leakage rather than improved procedural reasoning. The benchmark itself may still be a useful diagnostic, but the practical-utility claim requires a corrected experiment or explicit disclosure.","major_comments":[{"comment":"The evaluation never states whether the ProAgent knowledge database ('Knowledge Database from BioProBench Corpus', Figure 4) excluded the source protocols from which the held-out test instances were constructed. Because every test instance is derived from a source protocol in BioProCorpus, retrieval from the full corpus can surface the exact protocol text, including the numerical answer for PQA, the correct step order for ORD, and the reference steps for GEN. The reported gains (PQA 70.27 to 85.08; GEN SR 42.8 to 62.24; ERR F1 59.27 to 81.9) are therefore not interpretable as evidence of improved procedural reasoning until this contamination route is closed. The paper should either state that test-protocol documents were removed from the index and describe how, or re-run the ProAgent experiments with such removal. The perturbation analysis in Appendix I addresses model memorization, not RAG index leakage.","section":"5.2, Figure 7"},{"comment":"The paper contains unresolved numerical inconsistencies: the abstract reports 22,413 protocols and 523,784 instances; Section 2.1 and the corpus table report 26,933 protocols; the Introduction and Conclusion say 'nearly 27,000' and '27,000'; and the task table sums to 556,171 instances. The repository URLs also differ between the abstract (github.com/YuyangSunshine/bioprobench and huggingface.co/BioProBench) and the body (github.com/YuyangSunshine/bioprotocolbench and huggingface.co/datasets/BioProBench/BioProBench). Please reconcile these numbers and links, since they are the first things a reader or user checks.","section":"Abstract and Section 2.1, Figure 1"},{"comment":"The paper's core principle is that 'all scientific facts, procedural steps, numerical values, and ground-truth answers are extracted programmatically and directly from the human-authored source protocols,' yet the Impact Statement acknowledges 'reliance on LLMs for task structuring, which may introduce subtle model-specific artifacts.' The reader cannot tell which parts of the released instances are LLM-generated: distractors and perturbations are described as LLM-assisted, but the enrichment fields (problem, method, innovation, application) and any chain-of-thought ground truths in REA are not specified. Please provide a per-task table of LLM involvement and confirm that no ground-truth answer is proposed or edited by an LLM, or revise the strong claim accordingly.","section":"2.2 and Impact Statement"},{"comment":"The step metrics SR and SP depend on the cosine threshold delta = 0.7 and the keyword metrics on k = 64; these are free choices with no sensitivity results in the main text. The manuscript cites Appendix F.2 for justification, but the main text should at least summarize the threshold-sweep results, because the ProAgent GEN comparison (SR 42.8 to 62.24) is expressed in these units. Without such evidence, the claim that these metrics are robust is unverified.","section":"3, Eqs. (1)-(2)"}],"minor_comments":[{"comment":"'remian limited' should be 'remain limited'.","section":"6.2"},{"comment":"The statement that ProAgent 'substantially advances the state-of-the-art' is not supported by comparisons with any RAG or agent baselines other than the underlying LLM; please either add such baselines or temper the claim.","section":"Abstract and 5.2"},{"comment":"The description of the held-out test set as 'approximately 1,000 instances per task' should be reconciled with the per-subtask counts in Figure 1(b), where REA-GEN and REA-ORD appear with very different counts; please clarify which subsets form the held-out test set.","section":"5"},{"comment":"The 'Data Sources and Raw Data Counts' table appears twice in the same figure; the duplicate should be removed.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The decisive issue is the retrieval contamination question. If the authors can demonstrate that test-protocol documents were excluded from the ProAgent index, or re-run the experiments with such exclusion, the paper is likely acceptable as a resource paper. The benchmark construction itself appears sound and expert-validated; the inconsistent counts and URLs also require careful correction."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The benchmark itself is the contribution here, and it is a real one: 26,933 human-written protocols, over half a million instances across five task families (PQA, ORD, ERR, GEN, REA), with PhD-level expert review on the test set and new step-level metrics that make sense for procedural text. The evaluation of ten LLMs also gives a useful picture of where these models fail: quantitative precision, step ordering, and complete protocol generation. I would cite this as a benchmark resource if the release is cleaned up.\n\nThe soft spots are real but fixable. First, the number inconsistencies are sloppy: the abstract says 22,413 protocols and 523,784 instances, the body says 26,933 and 556,171, and the GitHub URL differs between the abstract and the body. That should have been caught before submission. Second, the ProAgent evaluation has a load-bearing hole. Every test instance is derived from a source protocol in BioProCorpus, and the retrieval database is built from that same corpus. The paper never says that test protocols were excluded from the index. If they were not, then retrieval can hand the model the exact numerical value (PQA), the correct step order (ORD), or the full reference steps (GEN). The gains shown in Figure 7 (PQA +15 points, GEN SR from 42.8% to 62.24%, ERR F1 from 59.27% to 81.9%) are exactly what you would see from such contamination. The perturbation analysis in Appendix I addresses model memorization, not RAG index leakage. The LAB-Bench out-of-distribution result (60% to 64%) is modest and much more believable, but it is not the headline.\n\nI think the stress-test note is correct. This does not invalidate the benchmark, but it invalidates the strongest evidence for the corpus's practical utility until the authors either confirm test protocols were excluded or re-run ProAgent with that exclusion. The fix is simple, which is why I still think this deserves a serious referee. The benchmark construction methodology is sound on its face, and the corpus-scale contribution is exactly what the community needs. The paper is not there yet, though: the count mismatches and the missing index-exclusion statement have to be resolved before I'd trust the ProAgent numbers.\n\nRecommendation: send to peer review with a request for major revision. The benchmark should survive; the ProAgent claim needs either an explicit exclusion statement or a corrected experimental setup.","headline":"A genuinely useful benchmark for biological protocol reasoning, but the ProAgent in-benchmark gains are not yet credible because the RAG index may contain the very protocols the test instances were built from.","tokens_in":13833,"tokens_out":1483,"would_cite":true,"duration_ms":16309,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces a 26,933-protocol corpus and a 556,171-instance benchmark showing that leading LLMs understand biological protocol text but systematically fail to reason about its procedures, quantities, and safety.","keywords":["biological protocols","procedural reasoning","large language models","benchmark dataset","retrieval-augmented generation","protocol generation","lab automation","scientific AI"],"falsifier":"One decisive check would be to list the BioProCorpus protocol IDs from which the held-out PQA, ORD, GEN, and REA test instances were built, then inspect the ProAgent retrieval index to see whether those exact protocols are present; if any are present, rerun the reported experiments with them removed and compare PQA accuracy, step recall, and the LAB-Bench score. If the gains vanish, the benchmark's central practical claim fails; if they persist, the corpus genuinely improves procedural reasoning.","tokens_in":12856,"feed_emoji":"🧪","tokens_out":6394,"duration_ms":63562,"temperature":0.7,"pith_summary":"The paper sets out to build the first large-scale resource for testing whether language models can handle biological experimental protocols as procedures, not just as text. It assembles BioProCorpus, 26,933 human-written protocols, and programmatically derives 556,171 instances covering question answering, step ordering, error correction, protocol generation, and explicit reasoning. On this benchmark, ten leading LLMs score well on basic comprehension and poorly wherever exact numbers, causal ordering, safety, or complete step sequences matter. The paper further claims that grounding an LLM in the corpus through retrieval (ProAgent) recovers much of this gap, raising accuracy on protocol question answering from about 70% to 85% and step recall in generated protocols from 42.8% to 62.24%. If these results hold, the corpus and benchmark give the field a standard way to measure and train procedural competence in scientific AI.","feed_headline":"27,000 lab protocols expose LLMs' weak procedural reasoning","feed_subtitle":"A new benchmark finds models omit over half of protocol steps, then shows retrieval grounding closes much of the gap.","key_machinery":"The load-bearing object is BioProCorpus, a corpus of 26,933 full-text protocols collected from six public sources, cleaned by deduplication and formatting removal, and enriched with a hierarchical top-step/child-step structure plus metadata such as problem, method, innovation, and application. From this corpus the pipeline programmatically builds five tasks: Protocol Question Answering (PQA), which fills blanks for reagents, parameters, and operations; Step Ordering (ORD), which reconstructs shuffled top-level or child steps; Error Correction (ERR), which judges targeted perturbations as true or false; Protocol Generation (GEN), which reconstructs a full protocol as a step list; and Protocol Reasoning (REA), which repeats the ERR and GEN formats with explicit structured chain-of-thought. The evaluation is carried by keyword precision, recall, and F1 computed with KeyBERT using $k=64$ keywords, together with step recall and step precision computed by embedding each step and thresholding cosine similarity at $\\delta = 0.7$.","core_discovery":"The paper's central claim is that procedural reasoning in biology is a distinct, measurable capability that current large language models largely lack, and that a purpose-built corpus of human-authored protocols provides both a diagnostic tool and a remedy. The authors assert that all scientific facts, procedural steps, numerical values, and ground-truth answers in BioProBench are extracted programmatically and directly from the source protocols, with the LLM confined to a constrained role such as generating distractors or applying minimal perturbations. The benchmark adds domain-specific metrics beyond lexical overlap: keyword-based content metrics and embedding-based structural metrics, including step recall and step precision with a cosine-similarity threshold of $\\delta = 0.7$. Evaluated across ten models, the results show high basic comprehension but systematic failures in step ordering, quantitative precision, error correction, and complete protocol generation; best step-ordering exact match is 51.80%, and best generation step recall is below 43%. The paper then shows that a retrieval-augmented agent grounded in BioProCorpus substantially raises these numbers, and that the same retrieval grounding transfers to an out-of-distribution benchmark.","pith_inferences":["Because the paper never states that the source protocols for the held-out test instances were excluded from ProAgent's retrieval index, the headline gains (PQA accuracy +15 points, step recall from 42.8% to 62.24%) could in part reflect direct retrieval of the very texts the questions were built from; a rerun with those protocols removed from the index would settle whether the corpus improves proc","The metrics depend on specific components: the $\\delta = 0.7$ cosine threshold for step matching and the KeyBERT keyword extractor, so model rankings on GEN and REA could shift if those components are replaced; this is a testable sensitivity analysis the paper does not report.","If the released 380,697 training instances are used to fine-tune future protocol models, those models would then be benchmarked on instances derived from the same corpus, creating a contamination risk unless the held-out set is frozen and the corpus version is tracked.","The authors' stated plan to add multimodal protocols implies a practical consequence: real laboratory work is visual and physical, so a text-only benchmark likely understates the gap between current models and the requirements of autonomous experimentation."],"forward_implications":["ProAgent's observed gains—85.08% PQA accuracy, 81.9% ERR F1-score, 62.24% step recall in GEN, and 64% accuracy on LAB-Bench—imply that a high-fidelity protocol corpus, used as retrieval context, can directly address known LLM failure modes in scientific procedures.","The consistent failure pattern across ten models (best step-ordering exact match 51.80%, best generation step recall below 43%) implies that procedural reasoning is a bottleneck independent of model scale or source, so progress on this axis likely requires dedicated training resources, not just larger models.","The novel metrics—keyword-based content metrics and embedding-based step recall and step precision—give protocol evaluation a way to penalize omitted or fabricated steps, which lexical metrics such as BLEU fail to detect.","The validated LLM-as-a-judge consistency (94.21% agreement with human experts on 200 sampled instances) supports using semantic judges to evaluate reasoning chains in protocol tasks, at least when the judge's role is restricted to matching stated reasons to ground truth.","The transfer gain on LAB-Bench suggests that protocol knowledge captured from one set of manuals generalizes to protocol questions drawn from other sources, which is exactly what an autonomous agent in a new laboratory would need."],"supporting_citations":[{"why":"Supplies the out-of-distribution external benchmark (108 protocol questions) used to test whether BioProCorpus knowledge transfers beyond the corpus's own tasks.","marker":"[16]"},{"why":"Provides the keyword extraction method (KeyBERT) used to define the keyword-based content metrics for generation evaluation.","marker":"[10]"},{"why":"Provides the chain-of-thought prompting basis for the REA task's structured reasoning format.","marker":"[36]"},{"why":"Provides the zero-shot chain-of-thought baseline whose observed degradation in GEN motivates the benchmark's structured CoT exemplars.","marker":"[15]"},{"why":"Represents a domain-adapted biomedical LLM whose limited procedural performance supports the claim that fine-tuning on declarative text is insufficient for protocol reasoning.","marker":"[20]"},{"why":"Exemplifies declarative biomedical question-answering benchmarks that BioProBench is designed to complement.","marker":"[14]"},{"why":"Represents a related benchmark for computational-biology agents; the paper positions BioProBench against it by covering protocol structure and procedural logic.","marker":"[22]"}],"fun_headline_variants":["LLMs omit over half of biology protocol steps","BioProBench: 523K tasks probe LLM procedural logic","Retrieval grounding closes LLM protocol gap","Biology protocol benchmark exposes LLM reasoning flaw"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the retrieval index behind ProAgent does not contain the same source protocols that were used to build the held-out test questions, because the paper never states that these protocols were excluded; if they are in the index, the reported gains could be direct answer retrieval rather than procedural reasoning.","fun_headline_variants_meta":{"raw":{"variants":["LLMs omit over half of biology protocol steps","BioProBench: 523K tasks probe LLM procedural logic","Retrieval grounding closes LLM protocol gap","Biology protocol benchmark exposes LLM reasoning flaw"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00077,"raw_usage":{"total_tokens":3411,"prompt_tokens":944,"completion_tokens":2467,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":560,"completion_tokens_details":{"reasoning_tokens":2405}},"tokens_in":560,"tokens_out":2467,"duration_ms":18450,"temperature":1.0,"reasoning_tokens":2405,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:28:59.448601+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"One decisive check would be to list the BioProCorpus protocol IDs from which the held-out PQA, ORD, GEN, and REA test instances were built, then inspect the ProAgent retrieval index to see whether those exact protocols are present; if any are present, rerun the reported experiments with them removed and compare PQA accuracy, step recall, and the LAB-Bench score. If the gains vanish, the benchmark's central practical claim fails; if they persist, the corpus genuinely improves procedural reasoning.","supporting_citations":[{"cited_title":"Keybert: Minimal keyword extraction with bert., 2020","cited_arxiv_id":null,"evidence_quote":"Provides the keyword extraction method (KeyBERT) used to define the keyword-based content metrics for generation evaluation."},{"cited_title":"Biogpt: generative pre-trained transformer for biomedical text generation and mining.Briefings in Bioinformatics, 23(6):bbac409, 2022","cited_arxiv_id":null,"evidence_quote":"Represents a domain-adapted biomedical LLM whose limited procedural performance supports the claim that fine-tuning on declarative text is insufficient for protocol reasoning."},{"cited_title":"Pubmedqa: A dataset for biomedical research question answering.EMNLP 2019, 2019","cited_arxiv_id":null,"evidence_quote":"Exemplifies declarative biomedical question-answering benchmarks that BioProBench is designed to complement."}],"review_version":1}