{"id":"e246f5c2-ea38-4e14-903a-53665e3948f0","arxiv_id":"2608.01690","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"ProtoAct converts free-form wet-lab protocols into monitoring-condition and robotic-subtask representations, then grounds them as constrained JSON action sequences validated in simulation and on a physical robot.","lead":"ProtoAct is a software system that turns written biology lab protocols into structured, robot-ready action plans, and ships a new annotated dataset called BioP2E. A smart generalist might read it because it offers a practical path from human-readable experiment instructions to robots that can carry out routine cell-culture work.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RefineChecker's user notes U may be drawn from the ground-truth annotations, making the reported parsing gains a measure of answer-key leakage rather than autonomous protocol understanding.","rationale":"The paper is a competent systems contribution; the parsing pipeline and dataset are described in detail, the quantitative simulation results (Table 3) are concrete, and the code is released. The central claim, however, requires that the pipeline's parsing quality be evaluated without the evaluation ground truth being fed into the parser. The reader identified this precisely: Eq. (3) makes the refined output a function of user notes U, and the paper never specifies U's provenance in the benchmark. My reading confirms this is the most load-bearing weakness: all automatic metrics, the ablations, and the GPT-based quality scores are computed after RefineChecker, so if U is derived from the same annotations used as ground truth, the reported scores overstate autonomous conversion ability. The real-robot validation is also thin (no quantitative success rates), but that is a reporting gap; the U provenance issue is an internal-validity problem that could invalidate the central benchmark. I therefore do not move the verdict: CONDITIONAL remains appropriate, with the condition being full disclosure and leakage-control for U. A single controlled rerun, as proposed, would settle the point.","tokens_in":14242,"tokens_out":4029,"duration_ms":36904,"concrete_test":"Inspect the evaluation harness for the 20 BioP2E protocols and trace the provenance of U: for each protocol, compare the text of U against the ground-truth monitoring conditions/subtasks and against the raw protocol. Then rerun the RefineChecker ablation (Qwen3-Max, full model vs no-RC) with U replaced by a summary generated from the raw protocol text only. If the F1/Ldn gains in Tables 6–7 shrink or disappear, the reported RefineChecker benefit is leakage; if gains persist, the concern is refuted. Additionally, report in the paper the exact U-generation prompt/template for reproducibility.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is that U in Eq. (3) is an independent user input, not a disguised copy of the ground-truth labels. Section 3.3 defines U as 'checking information provided by the user for the current experiment,' and Appendix B says it includes 'key operation flows, experimental constraints, and implicit actions.' The paper never states how U was obtained for the 20 BioP2E test protocols. If U was generated from the same manual annotations that define the evaluation ground truth (258 monitoring conditions, 910 subtasks), then the RefineChecker contribution measured in Section 4.4 and Table 6/7 is partly a leakage of the answer key: the model is told the correct steps, and the F1/Ldn improvements reflect adherence to provided corrections, not autonomous recovery of implicit protocol content. The central claim that ProtoAct converts free-form protocols into executable sequences would then be unsupported for the autonomous setting; the system would require human-prepared, annotation-derived notes to reach reported scores. The conclusion acknowledges the deployment dependence ('RefineChecker depends on experiment-specific information provided by users'), but this does not resolve the benchmark-validity problem: evaluation scores are not comparable to a system without U, and the downstream robot tasks built from refined parses inherit the same contamination.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents ProtoAct, a pipeline that converts free-form biological wet-lab protocols into monitoring conditions, subtask sequences, and JSON action-function sequences, using retrieval-augmented parsing (ProtoRAG), posterior checking and revision (RefineChecker), and schema-constrained grounding (ActSchema). The authors introduce BioP2E, a manually annotated dataset of 22 cell-culture protocols with 258 monitoring conditions, 910 subtasks, and 962 action calls, and evaluate the pipeline with seven large language models across recognition, parameter, and order metrics. They also report ablations and use the parsed subtasks to collect demonstrations and train VLA models in simulation and on a physical robot. The central claim is that ProtoAct forms a practical interface between protocol understanding and embodied robotic execution.","tokens_in":14623,"tokens_out":3306,"duration_ms":31958,"significance":"If the claims are substantiated, ProtoAct would be a useful contribution: it offers a new intermediate representation for protocol grounding, a manually annotated benchmark (BioP2E) that the community can reuse, and a systematic comparison of seven LLM backbones with means and standard deviations. The decision to release code and to report simulation results with episode counts are concrete strengths. The main substantive risk is that the RefineChecker improvement reported in Tables 6 and 7 may partly reflect leakage from experiment-specific notes U rather than autonomous parsing ability, and the real-robot validation currently lacks quantitative evidence. These are fixable with additional detail and experiments, but they are load-bearing for the paper's headline claims.","major_comments":[{"comment":"The input U to RefineChecker is never specified for the 20 evaluation protocols. Appendix B says U includes \"key operation flows, experimental constraints, and implicit actions,\" but the paper does not state who wrote U or whether U was derived from the same manual annotations that define the ground truth (258 monitoring conditions, 910 subtasks). If U was created from those annotations, then the RefineChecker gains in Tables 6 and 7 measure adherence to supplied corrections rather than autonomous recovery of implicit content, and the comparison against the no-RefineChecker ablation is no longer a fair measure of the pipeline's standalone capability. The authors should either specify the exact procedure for obtaining U for each evaluation protocol, show that U is independent of the ground-truth annotations, or rerun the benchmark without U for the claimed autonomous setting.","section":"Section 3.3, Eq. (3), Appendix B"},{"comment":"The real-robot experiment is described only qualitatively: \"Both models complete the tasks, with SmolVLA showing more stable execution, consistent with the simulation results.\" No success rates, number of rollouts, task-wise breakdown, or failure analysis are provided. Since the abstract and conclusion explicitly claim \"successful execution in both simulation and real-robot settings,\" the absence of any quantitative real-robot result makes that part of the central claim unsupported as written. The authors need to report task-level success rates, trial counts, and evaluation protocols for the physical robot experiments, or temper the claim to what the evidence supports.","section":"Section 4.6, Real-Robot Execution Validation"},{"comment":"GPT-5.3 is both one of the evaluated models in Table 2 and the LLM used as the evaluator in Section 4.5 and Appendix D. Using a model to judge its own outputs creates a self-reference bias, and the paper reports no human-agreement study or calibration for the GPT evaluator scores shown in Appendix E. The authors should either exclude GPT-5.3 from judging outputs that include GPT-5.3's own generations, or provide human correlation results and inter-annotator agreement for the GPT scoring. Without this, the GPT-based conclusions about RefineChecker and ProtoRAG are difficult to interpret.","section":"Sections 4.2 and 4.5, GPT-5.3 evaluator"},{"comment":"The embodied validation covers five simulation tasks and three real-robot tasks that are isolated subtasks (tube pickup, lid opening, dish placement, etc.), not complete multi-step protocol executions derived automatically from a full parsed protocol. The claim that \"the subtasks generated by ProtoAct can be converted into executable robotic actions\" is therefore demonstrated for selected primitive actions, not for the end-to-end pipeline from a raw protocol to a completed experimental procedure. The authors should clarify this scope limitation in the abstract and conclusion, or add an end-to-end execution experiment starting from a parsed protocol and executing a sequence of dependent subtasks.","section":"Section 4.6 and Appendix F"}],"minor_comments":[{"comment":"Section 4.6 states that the five simulation tasks are constructed in the Pipette platform, while Appendix F says they are constructed in Isaac Sim; the relationship between Pipette and Isaac Sim should be stated explicitly to avoid confusion.","section":"Section 4.6 vs. Appendix F"},{"comment":"The column headers in Tables 6 and 7 are inconsistent: Table 6 repeats \"Precision Recall F1\" twice for monitor type and monitor parameter, while Table 7 uses the same triple for action primitive and action parameter; please align the headers with the metric names used in the main text.","section":"Tables 6 and 7"},{"comment":"The SciBERTScore formula uses N in the denominator of a sum over i from 1 to N, but N is also used in Equation (6) for the n-gram order; renaming the number of evaluated parameters, for example to M, would remove the notation clash.","section":"Equation (7)"},{"comment":"The caption says the best and second-best results are marked in bold and underlined, but several cells appear bold and underlined inconsistently; please verify the formatting matches the caption.","section":"Table 2 caption"},{"comment":"The last sentence of the conclusion lists limitations that are relevant to the main claims; consider moving the RefineChecker dependence on user-provided information into the experimental design section so that readers can evaluate the benchmark validity directly in Section 4.1.","section":"Conclusion, future work"}],"recommendation":"major_revision","confidential_remarks":"The central concern is not that the pipeline is implausible, but that the reported evaluation may mix autonomous parsing with external information through the unspecified U input, and the real-robot validation is not yet quantitative. Both issues are addressable within the manuscript's scope, so I recommend major revision rather than rejection. I would also gently suggest that the authors clarify the exact relationship between the evaluation protocols and the two protocols used as retrieval references, since the 20/22 split is stated but not further analyzed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's what I'd want you to know before spending an hour on this. It's a competent systems paper with one genuinely useful artifact—the BioP2E dataset—and a load-bearing ambiguity in its RefineChecker ablation that the authors need to clear up. Not a breakthrough, but it deserves a serious referee.\n\nWhat's new: the dataset is real work. Twenty-two cell-culture protocols manually annotated into 258 monitoring conditions, 910 subtasks, 962 grounded action calls, with an action schema covering 30 primitives. That's a resource people in lab automation will use. The pipeline—RAG-based parsing, a posterior checking step, then schema-constrained JSON generation—is not conceptually novel by itself, but the combination is well described and the ablations give it honest internal consistency. The simulation results have episode counts (100 per task, success rates between 34% and 98%), which is more than many robotics papers report.\n\nThe soft spots are where the evidence gets thin. First, the RefineChecker step takes user notes U as input, and the paper never says where U came from for the 20 evaluation protocols. If U was derived from the same manual annotations that define the ground truth, the reported F1 gains from RefineChecker are partly answer-key leakage. The conclusion acknowledges the dependence on user information, but that does not resolve the benchmark-validity question. Second, the real-robot section has no numbers at all—'both models complete the tasks' is the entire quantitative content. That claim is currently unsupported. Third, the GPT-5.3 evaluator judging outputs of GPT-5.3 is a minor smell, but the primary metrics are automatic and unaffected. Fourth, while code is linked, the dataset artifacts are not, which undercuts reproducibility for the one thing the paper does best.\n\nNone of this sinks the paper. The parsing pipeline is the core contribution, and its evaluation on BioP2E is plausible if U is cleaned up. The robotic validation is clearly secondary and should be toned down or quantified. The stress-test hypothesis about U leaking ground truth does not appear contradicted anywhere in the text—I read the relevant sections and the sourcing is genuinely undisclosed.\n\nBottom line: send this to peer review with a request for a clear description of U's construction, either real numbers for the robot experiments or a downgraded claim, and a dataset release. I'd read a revised version.","headline":"A solid systems paper with a genuinely useful benchmark; the RefineChecker input needs disclosure and the real-robot claim needs numbers, but the core parsing contribution deserves peer review.","tokens_in":15057,"tokens_out":1893,"would_cite":true,"duration_ms":17037,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ProtoAct converts free-form biological protocols into state-aware, embodiment-ready action sequences, and the parsed subtasks support demonstration collection and VLA model training for both simulated and physical robots.","keywords":["wet-lab protocols","protocol grounding","embodied robotic execution","vision-language-action models","retrieval-augmented generation","action schema","BioP2E dataset","laboratory automation"],"falsifier":"Run ProtoAct on held-out protocols whose RefineChecker notes are written by an independent biologist blind to BioP2E's ground-truth, and compare the resulting F1 and Ldn against the paper's reported numbers; if the posterior-checking gain vanishes, the claimed autonomous repair is not robust.","tokens_in":14050,"feed_emoji":"🧪","tokens_out":11497,"duration_ms":94860,"temperature":0.7,"pith_summary":"ProtoAct claims that free-form biological wet-lab protocols, which routinely omit routine operations and leave parameters context-dependent, can be converted into state-aware, embodiment-ready action sequences without hand-coding every step. The system parses a protocol into natural-language monitoring conditions and subtasks, then grounds them into constrained JSON action functions; manually annotated examples are retrieved for context, a posterior checker revises missing or inconsistent steps, and a schema fixes the action vocabulary. The paper introduces BioP2E, a manually annotated dataset of 22 cell-culture protocols yielding 258 monitoring conditions, 910 subtasks, and 962 grounded action calls, and shows the pipeline works across seven large language models. The parsed subtasks are then used to collect demonstrations and train vision-language-action models, which execute the tasks in simulation and on a physical robot arm. If the claim holds, biologists could hand written protocols to a system that returns robot-executable programs and training data for laboratory automation.","feed_headline":"Robots get executable steps from plain-language lab protocols","feed_subtitle":"It parses cell-culture text into JSON robot commands and trains policies that run in simulation and on real robot arms.","key_machinery":"The load-bearing machinery is the ProtoAct pipeline's three-module design around a common intermediate representation: ProtoRAG retrieves manually annotated protocol fragments to seed context-sensitive parsing; RefineChecker performs posterior checking of the initial monitoring-condition/subtask parse against user-supplied experiment notes and revises four defect types (missing actions, missing parameters, wrong order, wrong granularity); ActSchema maps the refined representation into a constrained JSON action-function sequence using a predefined space of 30 action primitives and 4 monitoring condition types. The key object is the intermediate representation itself—monitoring conditions plus subtask sequences—which preserves procedural dependencies in a form humans can inspect and robots can be trained on, and which makes automatic evaluation (F1, BLEU, SciBERTScore, normalized Levenshtein distance) possible.","core_discovery":"On its own terms, the paper's central claim is that a protocol-grounding pipeline can bridge biological protocol understanding and embodied robotic execution by making explicit what wet-lab instructions leave implicit: the monitoring conditions that gate each step, the physical action primitives, their parameters, and their order. ProtoAct instantiates this as a two-stage representation, first producing a human-inspectable natural-language parse and then constraining it through ActSchema into JSON function sequences, where each function is a named action with fixed parameter fields. The evaluation supports this by reporting strong F1, parameter-similarity, and order-consistency scores across seven LLMs on the 20 held-out BioP2E protocols, and by showing that subtasks parsed from protocols yield successful policy execution in five simulated and three real-robot manipulation tasks.","pith_inferences":["The paper's deployment story requires a user to supply experiment-specific notes to RefineChecker; a natural extension is an interactive loop in which the biologist's corrections are the interface, but the paper does not evaluate that interaction directly.","Because BioP2E covers only cell-culture protocols from one repository, the 30-primitive action schema may not transfer to other wet-lab domains; testing on PCR or protein-purification protocols would show whether the schema needs extension.","The robot validation is at the level of isolated manipulation subtasks, so 'embodiment-ready' should currently be read as subtask-ready; a full multi-hour closed-loop protocol execution remains untested.","A decisive test of the posterior-checking benefit would be to source RefineChecker notes from independent biologists blind to the ground-truth annotations; if gains disappear, the measured gain reflects leakage of the answer key rather than autonomous repair."],"forward_implications":["A written cell-culture protocol can produce a robot-executable JSON action program directly, without a human translating each step into machine code.","The same subtask representation can be used as language-conditioned instructions for collecting demonstrations, so non-specialists can gather training data by teleoperating the robot through parsed steps.","Vision-language-action policies trained with an open-source robot-learning library can execute the parsed subtasks in simulation and on a physical arm, supporting the use of protocol parsing as a data-generation pipeline for embodied models.","Because the pipeline is instantiated with seven different LLM backbones with close scores, the protocol grounding is not locked to one model, and future backbone improvements should transfer directly.","Ablations attribute distinct roles to the three modules: removing ProtoRAG hurts monitoring-condition recall, removing ActSchema hurts action primitive and parameter regularity, and removing RefineChecker increases omissions and order errors."],"supporting_citations":[{"why":"Supplies protocols.io as the source of the 22 cell-culture protocols annotated into BioP2E.","marker":"Teytelman et al. 2016"},{"why":"Prior work converting protocols to pseudocode that ProtoAct positions itself against, defining the gap of an embodiment-ready representation.","marker":"O’Donoghue et al. 2023"},{"why":"Introduces a predefined experimental action set for protocol evaluation that motivates ActSchema's constrained function space.","marker":"Yi, Lim, and Yoon 2025"},{"why":"Provides the BLEU metric used to measure lexical similarity of predicted and ground-truth parameter values.","marker":"Papineni et al. 2002"},{"why":"Provides the SciBERT encoder used by SciBERTScore to measure semantic similarity of parameter values.","marker":"Beltagy, Lo, and Cohan 2019"},{"why":"Defines the edit distance underlying the normalized Levenshtein distance used to score action-sequence order consistency.","marker":"Levenshtein 1966"},{"why":"Supplies the SmolVLA vision-language-action model trained and evaluated on ProtoAct-derived demonstrations.","marker":"Shukor et al. 2025"},{"why":"Supplies the π0 vision-language-action model trained and evaluated on ProtoAct-derived demonstrations.","marker":"Black et al. 2025"},{"why":"Provides the LeRobot framework used to train and evaluate the vision-language-action policies.","marker":"Cadene et al. 2026"},{"why":"Provides the Pipette simulation platform on which the five embodied wet-lab tasks were built and evaluated.","marker":"Liu et al. 2026"}],"fun_headline_variants":["ProtoAct: wet-lab text to real robot actions","Lab protocols get robot-ready steps","Robots execute cell-culture protocols from text","From biology papers to robot commands","Protocol text becomes robot action sequences"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the experiment-specific notes fed to RefineChecker can be obtained without already knowing the ground-truth annotations used for scoring; if those notes came from the answer key, the reported posterior-checking gains would partly measure leakage rather than autonomous parsing.","fun_headline_variants_meta":{"raw":{"variants":["ProtoAct: wet-lab text to real robot actions","Lab protocols get robot-ready steps","Robots execute cell-culture protocols from text","From biology papers to robot commands","Protocol text becomes robot action sequences"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000441,"raw_usage":{"total_tokens":2214,"prompt_tokens":900,"completion_tokens":1314,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":516,"completion_tokens_details":{"reasoning_tokens":1251}},"tokens_in":516,"tokens_out":1314,"duration_ms":11454,"temperature":1.0,"reasoning_tokens":1251,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:05:13.141894+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run ProtoAct on held-out protocols whose RefineChecker notes are written by an independent biologist blind to BioP2E's ground-truth, and compare the resulting F1 and Ldn against the paper's reported numbers; if the posterior-checking gain vanishes, the claimed autonomous repair is not robust.","supporting_citations":[{"cited_title":", title =","cited_arxiv_id":null,"evidence_quote":"Prior work converting protocols to pseudocode that ProtoAct positions itself against, defining the gap of an embodiment-ready representation."},{"cited_title":"2025 , eprint =","cited_arxiv_id":null,"evidence_quote":"Introduces a predefined experimental action set for protocol evaluation that motivates ActSchema's constrained function space."},{"cited_title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing , pages =","cited_arxiv_id":null,"evidence_quote":"Provides the SciBERT encoder used by SciBERTScore to measure semantic similarity of parameter values."},{"cited_title":"2025 , eprint =","cited_arxiv_id":null,"evidence_quote":"Supplies the SmolVLA vision-language-action model trained and evaluated on ProtoAct-derived demonstrations."}],"review_version":2}