{"id":"9bb1060f-ce96-4511-98a3-c17a2e91c7d9","arxiv_id":"2608.02642","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"MDArena, a 50-task benchmark from active biomolecular simulation projects, shows the strongest evaluated agent, Codex GPT-5.5 XHigh, solves 24/50 tasks (48%) at strict pass, with hard workflows nearly unsolved.","lead":"A new benchmark, MDArena, puts six AI coding agents through 50 real molecular-dynamics tasks and finds the best one solves only 48% of them. The paper shows agents make useful partial progress but fall short on the final details needed for reproducible simulation workflows.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Verifier has no negative controls: reference solutions only guard against false negatives, so Strict-Pass@1 and the correctness/process gap could be inflated or deflated by an unvalidated LLM judge.","rationale":"The reader's weakest assumption identified the verifier's ability to distinguish valid from invalid workflows as the load-bearing premise. My stress-test converges on the same point and sharpens it: the only validation reported is a false-negative check. A benchmark whose verifier has never been shown to reject known-bad workflows cannot support absolute statements about agent reliability. This concern does not undermine the benchmark's value as a reproducible artifact, and the broad gap between partial credit and strict success would likely survive even if the verifier were recalibrated, but the specific numbers in Table 1 and the 'substantial gap' interpretation could shift. Since the reader already marked the verdict CONDITIONAL on this basis, my read does not change the verdict; it reinforces the need for a versioned update that adds negative controls and judge-reliability analysis.","tokens_in":14074,"tokens_out":3223,"duration_ms":31080,"concrete_test":"Construct 10-20 negative-control variants by taking reference solutions and injecting known scientific errors (e.g., wrong protonation state, missing .itp file, residue-numbering mismatch, incorrect water model, skipped equilibration step, invalid lambda schedule, overly short simulation). Run the full MDArena verifier on these mutated workflows and measure the false-positive rate, defined as the fraction of invalid solutions that pass all correctness and process checks. Additionally, have two independent human MD experts and a second LLM judge (e.g., GPT-5.5) score a random sample of 20 agent trajectories against the process rubrics; report Cohen's kappa or similar agreement. If the false-positive rate exceeds 5% or human-LLM agreement is poor, the reported pass rates and the correctness/process gap should be recalibrated or reported with error bars.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claims--Strict-Pass@1 rates, average correctness and process rewards, and the interpretation that agents make meaningful partial progress but fail on fine-grained details--all depend on the verifier correctly rejecting scientifically invalid workflows. Section 3.3 validates the verifier only by requiring 31 reference solutions to pass all checks. This establishes that a correct workflow is not penalized (no false negatives), but it says nothing about false positives: whether a flawed workflow would be caught. The process verifier is an LLM judge (Claude Opus 4.8) scoring 243 author-defined criteria, yet the paper reports no inter-judge reliability, no human agreement study, and no adversarial or negative-control tasks. If the judge is permissive, or if deterministic correctness checks are under-specified, some incorrect workflows will pass all checks, inflating Strict-Pass@1 and compressing the reported gap between partial progress and full success; if the judge is overly strict, the gap is exaggerated. Because all configurations are scored with the same verifier, relative rankings may be more robust, but the headline claim about reliability as autonomous MD researchers is an absolute claim and is unanchored without false-positive validation. The paper's own limitations section acknowledges LLM judge model dependence but does not quantify it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MDArena, a benchmark of 50 containerized molecular-dynamics tasks drawn from active biomolecular simulation projects, together with a two-part verifier: deterministic artifact checks for correctness and an LLM-as-judge protocol for process criteria. Six model/harness configurations are evaluated under a common Harbor-based protocol. The best configuration, Codex GPT-5.5 at extra-high reasoning effort, achieves 24/50 Strict-Pass@1 (48%), followed by GPT-5.5 Medium (21/50) and Gemini Flash 3.5 (20/50); open-weight models trail substantially. The paper reports average correctness and process rewards that are much higher than strict pass rates, interprets this as evidence that agents make meaningful partial progress but fail on fine-grained details, and identifies membrane-protein preparation and alchemical free-energy setup as largely unsolved. The benchmark, code, verifier implementations, and reference solutions are publicly released.","tokens_in":14291,"tokens_out":4161,"duration_ms":38019,"significance":"If the verifier is trustworthy, MDArena is a valuable and unusually transparent benchmark: it is grounded in real research workflows, spans diverse systems and methods, ships containerized tasks and reference solutions, and reports standard errors on the headline pass rates. The central qualitative pattern — high partial credit versus low strict success, with hard tasks largely unsolved — is consistent across all six configurations and is robust to several plausible perturbations. The paper also gives credit where due: it acknowledges single-run evaluation, task-selection bias, LLM-judge dependence, and the single-group provenance of tasks. The main risk is that the verifier has been validated only against positive reference solutions, so false positives are uncharacterized; this directly affects the absolute pass rates and the correctness/process gap interpretation. That gap is fixable and does not undermine the benchmark's usefulness as a comparative platform, but it does need to be addressed before the central claims can be accepted as stated.","major_comments":[{"comment":"The verifier is validated only by requiring the 31 reference solutions to pass all checks, which guards against false negatives but not false positives. There are no negative-control tasks, no adversarial examples of flawed workflows that the verifier should reject, no inter-judge reliability numbers, and no human-agreement study for the 243 LLM-judged process criteria. Because Strict-Pass@1 rates in Section 5.1 and the correctness/process gap are absolute claims about scientific validity, this validation gap leaves the central numbers unanchored. I would like to see, at minimum, a set of deliberately broken workflows that the verifier must reject, along with a report of judge agreement on a sample of trajectories.","section":"Sections 3.2 and 3.3"},{"comment":"Each model-harness configuration is evaluated once per task, so the reported pass rates, including the 24/50 versus 21/50 ordering between GPT-5.5 XHigh and Medium, are single draws. The paper reports binomial standard errors across tasks, but this does not characterize run-to-run stochasticity of the agents themselves. The ordering between GPT-5.5 XHigh and Medium, and between Gemini Flash and Pro, could change under repetition. Please provide repeated-run estimates for at least a subset of tasks, or otherwise temper the ranking claims.","section":"Sections 4.1 and 6"},{"comment":"The gap between Strict-Pass@1 and the average correctness and process rewards is, to a nontrivial degree, a compositional consequence of defining strict success as all checks passing. With a median of four process checks per task and additional correctness checks, an agent that fails a single check on many tasks will show high average rewards but low Strict-Pass@1. The paper interprets this gap as evidence that agents 'make meaningful partial progress but fail on the fine-grained details required for reproducible scientific workflows'; that interpretation needs support from the distribution of per-task check failures (for example, how many failed tasks were near-misses on one check versus complete failures), rather than from averaged rewards alone.","section":"Section 5.1"},{"comment":"The task pool was deliberately curated to expose known agent failure modes, and the paper acknowledges that the resulting success rates 'should not be interpreted as estimates of performance over all routine MD tasks.' This is appropriate, but the abstract and introduction phrase the result as a general statement about 'realistic molecular dynamics workflows' and 'reliability as autonomous MD researchers.' Please make sure the framing in the abstract and Section 1 does not contradict the acknowledged selection bias, so that the 48% figure is not read as a baseline over routine MD practice.","section":"Sections 3.1 and 6"}],"minor_comments":[{"comment":"Model names are inconsistent: the abstract says 'Gemini Flash 3.5' while Section 4.1 and Table 1 use 'Gemini-3.5-Flash' (and similarly for Gemini Pro 3.1). Please unify the naming.","section":"Abstract and Section 4.1"},{"comment":"The sentence beginning 'Qwen exhibits this failure mode heavily on easy tasks...' contains a comma splice that makes the contrast hard to parse; split it into two sentences.","section":"Section 5.4.3"},{"comment":"Several reference entries are missing a space after 'URL' (for example, the Communications Biology and Mey et al. entries), which renders them as 'URLhttps://...'. Please fix the formatting.","section":"References"},{"comment":"The caption says error bars show standard errors across tasks, 'using binomial standard errors for pass/fail success rates,' but it is unclear whether the correctness and process rewards use the same error formula. Please clarify the error-bar definition for those panels.","section":"Figure 2 caption"},{"comment":"Table A1 is very dense. Consider moving it to an online supplement or reformatting it for readability, since it contains important metadata but is hard to scan in its current form.","section":"Table A1"}],"recommendation":"major_revision","confidential_remarks":"This is a solid empirical contribution with a real validation gap that is fixable within the manuscript's scope. The verifier's lack of negative controls is the main technical concern; adding adversarial broken-workflow tests and a small human-agreement study would substantially strengthen the absolute claims. The paper fits the journal's scope and I do not see any ethical or novelty issues. I have no conflicts of interest."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short answer: this paper deserves a serious referee, and I'd lean accept-with-revisions. The benchmark is real: containerized, public, tasks drawn from active MD projects, reference solutions, 50 tasks across methods and software, and two-part scoring with deterministic checks plus human-authored process rubrics. That is the first evaluation I know that tries to grade scientific validity rather than just execution success or output matching. The central finding—high partial credit (correctness ~0.84, process ~0.86 for the best model) but low strict pass (48%)—looks consistent across six configurations and categories. The cost-efficiency comparison is a nice addition. The paper is transparent about its limitations, which counts for a lot.\n\nThe weak spot is the verifier. Reference solutions passing all checks only tells you the verifier doesn't penalize correct workflows; it tells you nothing about whether a wrong workflow gets caught. The process score depends on Claude Opus 4.8 as judge, with no inter-judge reliability, no human agreement study, no adversarial examples. If the judge is permissive, the absolute pass rates and the gap interpretation could be off. My guess is relative rankings are safer because all configurations are scored on the same verifier, but the headline claim—that agents are useful as assistants but not autonomous researchers—is absolute, and that needs better anchoring. This is fixable: sample some failed trajectories and have an expert check whether the verifier flags them, then report agreement. Also, single-run Pass@1 with no repeated runs means the small differences (24 vs 21 vs 20) are not distinguished from noise. The authors acknowledge this, but it should be quantified, at least for the top configurations.\n\nThose caveats don't sink the paper. The benchmark itself is a solid contribution, and the broad conclusion is likely right. I'd also mention the lack of a human baseline: we don't know how a domain scientist would do on these 50 tasks under the same time limits, which limits the 'realistic' claim. Minor and not disqualifying.\n\nWho's this for? Anyone working on coding agents for biomolecular simulation, or building scientific benchmarks. I'd cite it and would bring it to group meeting. Should a journal send it to referees? Yes. The verifier concern is important but the transparent protocol means a good referee can push for a versioned update with negative controls.","headline":"A solid, reproducible MD agent benchmark with a mostly credible central finding; the unvalidated verifier is a real but fixable weakness.","tokens_in":14863,"tokens_out":2697,"would_cite":true,"duration_ms":24013,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MDArena, a 50-task benchmark of real molecular-dynamics workflows, finds that even the strongest coding agent passes full checks on only 24 of 50 tasks (48%), exposing a gap between plausible partial progress and reproducible, fully…","keywords":["molecular dynamics","coding agents","benchmark","LLM-as-judge","workflow verification","free-energy perturbation","enhanced sampling","system preparation"],"falsifier":"Have experienced molecular-dynamics researchers independently grade a sample of agent trajectories, especially ones that failed the automated checks, without seeing the automated scores; if a substantial share of failed workflows is judged scientifically valid and reproducible, the verifier is not controlling false positives and Strict-Pass@1 would need renormalization. Conversely, a single trajectory that passes all checks but yields an internally inconsistent simulation, for example a topology whose atom ordering does not match its coordinates, would show the verifier is under-sensitive.","tokens_in":13890,"feed_emoji":"🧪","tokens_out":8456,"duration_ms":69848,"temperature":0.7,"pith_summary":"MDArena is a benchmark of 50 containerized tasks drawn from ongoing biomolecular simulation projects, spanning 29 molecular systems and 14 research protocol classes. The paper's central claim is that today's coding agents are useful as supervised assistants on compact, well-specified subtasks but cannot yet be trusted as autonomous molecular-dynamics researchers. The best evaluated configuration passes all checks on 24 of 50 tasks (48%), while average correctness and process rewards near 0.84 and 0.86 show that agents usually make substantial partial progress and then miss the fine-grained details needed for reproducible workflows. Membrane-protein system preparation and alchemical free-energy setup remain unsolved or nearly unsolved by every configuration. The paper presents MDArena as an extensible, versioned platform for tracking progress toward closing that gap.","feed_headline":"Strongest coding agents pass just 24 of 50 real MD-lab tasks","feed_subtitle":"New 50-task benchmark shows agents make partial progress but miss fine details needed for reproducible simulation workflows.","key_machinery":"The central object is the two-part verifier that defines success. Deterministic correctness checks confirm that expected output artifacts are present, valid, and satisfy numerical, structural, or software-level requirements, and the correctness reward is the fraction of criteria met. The process verifier is an LLM-as-judge that reads the agent's full execution trajectory and scores it against task-specific, author-defined criteria covering scientific decision-making and result interpretation, with a median of four criteria per task. Strict-Pass@1, the benchmark's binary success metric, requires every correctness and process check to pass. Reference solutions, executable scripts supplied for 31 of 50 tasks, are required to pass all checks, which is intended to confirm that each task is solvable and that the verifier does not penalize correct workflows. This verifier design is what lets the paper interpret the gap between high partial rewards and low strict success as agent failure rather than benchmark miscalibration.","core_discovery":"On its own terms, the paper establishes that the limiting factor in agentic molecular-dynamics automation is not the ability to make plausible progress but the reliability to satisfy every required detail. Agents frequently produce intermediate artifacts that look reasonable yet fail checks designed to catch subtle, compounding errors, such as mismatched residue numbering between file formats or missing include files referenced by a topology. The evaluation's two-part verifier makes this dissociation visible: deterministic artifact checks score the outputs, and an LLM judge scores the execution trajectory against 243 author-defined process criteria; Strict-Pass@1 requires all checks to pass. Because correctness and process rewards are far higher than strict success, the paper concludes that agents routinely 'almost solve' tasks, and that converting partial progress into full success is the frontier. It also reports that harness choice and reasoning effort matter: the extra-high-effort configuration outperforms the medium-effort version mostly on process checks, while operational failures such as timeouts and context-limit errors are often a coupling between model and harness rather than a pure model weakness.","pith_inferences":["A natural next experiment is to take the near-miss trajectories and feed the failed checks back to the agent as targeted hints, measuring how much of the failure rate is recoverable without any new model capability.","The same two-part verification scheme could be deployed as a continuous-integration check inside real molecular-dynamics projects, flagging pipelines that produce scientifically questionable artifacts before they are used downstream.","Because the process score comes from an LLM judge, an expert audit of a random sample of scored trajectories would help establish whether process rewards track scientific validity; if the judge rewards plausible narration over file-level correctness, the interpretation of the correctness/process gap would need revision.","The single-run protocol means the ordering of configurations near the middle of the table may not be stable; repeated runs with confidence intervals would show which headline differences are real."],"forward_implications":["Even the strongest evaluated configuration fails roughly half of realistic tasks, so for now agent outputs should be treated as drafts to be reviewed by a human rather than as finished simulation protocols.","Because open-weight models trail sharply (12% and 2% strict success), closing the gap for accessible models will require more than better tooling; the paper argues raw model capability must improve.","Hard, long-horizon workflows with multiple valid routes, such as membrane-protein system preparation, alchemical free-energy setup, and enhanced-sampling protocols, are the unsolved frontier and are the categories future versions of the benchmark will track.","Raising reasoning effort changes the outcome: the extra-high-effort configuration achieves nearly the same correctness reward as the medium-effort one but a higher process reward, indicating that deliberation on edge cases is what converts near-misses into strict passes.","Because each configuration was evaluated once per task, the reported pass rates are practical single-run baselines; repeated-run reliability and stochastic variation remain uncharacterized."],"supporting_citations":[{"why":"MDGym is the main prior molecular-dynamics agent benchmark this work extends, and its scoring approach motivates the two-part verifier design.","marker":"Kumar et al., 2026"},{"why":"MDCrow is the first LLM-agent MD benchmark and is used as the reference point for why earlier task suites are saturated, motivating harder tasks.","marker":"Campbell et al., 2026"},{"why":"GROMACS is the simulation engine used across most tasks, and its file formats and tools anchor many correctness checks.","marker":"Abraham et al., 2015"},{"why":"Provides the containerized task-execution and Oracle/reference-solution infrastructure on which the evaluation protocol runs.","marker":"Harbor Framework Team, 2026"},{"why":"Defines the REXEE enhanced-sampling method that one hard task asks agents to set up.","marker":"Hsu & Shirts, 2024"},{"why":"Defines alchemical metadynamics, the method behind another hard task in the benchmark.","marker":"Hsu et al., 2023"},{"why":"Supplies best-practice criteria for alchemical free-energy setups, which anchor the process rubrics for FEP tasks.","marker":"Mey et al., 2020"},{"why":"MDAnalysis is the mature analysis library whose reuse, or lack thereof, is a recurring success and failure factor in trajectory-analysis tasks.","marker":"Michaud-Agrawal et al., 2011"}],"fun_headline_variants":["AI coding agents fail 26 of 50 real MD lab tasks","Best agent 48% on MD tasks: fine details are the bottleneck","MDArena: agents almost solve but miss fine details","Top coding agent only 24/50 on real MD workflows","MD agents: partial progress, but reproducibility fails"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the two-part verifier reliably separates scientifically valid workflows from invalid ones; the only validation offered is that 31 reference solutions pass all checks, which rules out over-strict false negatives but does not rule out false positives, so if the deterministic checks or the LLM rubric are miscalibrated, both the reported pass rates and the interpretation of the correctness/process gap would change.","fun_headline_variants_meta":{"raw":{"variants":["AI coding agents fail 26 of 50 real MD lab tasks","Best agent 48% on MD tasks: fine details are the bottleneck","MDArena: agents almost solve but miss fine details","Top coding agent only 24/50 on real MD workflows","MD agents: partial progress, but reproducibility fails"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00112,"raw_usage":{"total_tokens":4699,"prompt_tokens":1024,"completion_tokens":3675,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":640,"completion_tokens_details":{"reasoning_tokens":3590}},"tokens_in":640,"tokens_out":3675,"duration_ms":23212,"temperature":1.0,"reasoning_tokens":3590,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:21:53.168376+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have experienced molecular-dynamics researchers independently grade a sample of agent trajectories, especially ones that failed the automated checks, without seeing the automated scores; if a substantial share of failed workflows is judged scientifically valid and reproducible, the verifier is not controlling false positives and Strict-Pass@1 would need renormalization. Conversely, a single trajectory that passes all checks but yields an internally inconsistent simulation, for example a topology whose atom ordering does not match its coordinates, would show the verifier is under-sensitive.","supporting_citations":[{"cited_title":"MDGYM: Benchmarking AI Agents on Molecular Simulations","cited_arxiv_id":"2605.08941","evidence_quote":"MDGym is the main prior molecular-dynamics agent benchmark this work extends, and its scoring approach motivates the two-part verifier design."},{"cited_title":", title =","cited_arxiv_id":null,"evidence_quote":"MDCrow is the first LLM-agent MD benchmark and is used as the reference point for why earlier task suites are saturated, motivating harder tasks."}],"review_version":1}