{"id":"92efa3fe-d128-40f3-bb93-7527c60e17c6","arxiv_id":"2501.10385","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"An LLM-based multi-agent framework successfully operates an AFM but benchmark results show current models are unreliable, instruction-following is fragile, and domain QA skill does not predict laboratory competence.","lead":"This paper builds an AI agent system called AILA that operates an atomic force microscope, and a benchmark called AFMBench for testing such systems. Across 100 microscopy tasks, the best model (GPT-4o) completed only 65 percent correctly, and the paper documents that models often ignore instructions, which is risky for automated labs.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"AFMBench's manual consensus scoring and small-sample ablations leave the headline QA-vs-agentic dissociation and multi-agent advantage quantitatively unsecured.","rationale":"The reader's conditional verdict already identifies the two load-bearing assumptions I stress-tested: the representativeness/accuracy of AFMBench's manual scoring and the small-sample statistical grounding of the comparative claims. I agree that these are the weakest points. I did not find an internal inconsistency or a more severe flaw. The real-world demonstrations (PID optimization, graphene step height, indenter identification) provide independent support that AILA can execute meaningful AFM workflows, which is the constructive core of the paper. My concrete test—independent blinded rescoring plus confidence intervals on the key comparisons—would directly settle whether the quantitative rankings are trustworthy. If the test fails, the appropriate verdict would be CONDITIONAL with mandatory reporting changes; if it passes, the paper's claims are materially strengthened. Since the reader already issued CONDITIONAL, I recommend UNCHANGED: the verdict already encodes the right level of caution, and no new concern changes it.","tokens_in":28505,"tokens_out":1127,"duration_ms":11731,"concrete_test":"Publish the full per-question, per-trial scores for both frameworks and all models, and have two independent AFM-experienced researchers re-score a random stratified subset of at least 30 tasks (blinded to model identity). Report Cohen's kappa and recompute the model ranking using only the independent scores. If the ranking of GPT-4o vs. Claude-3.5 and the multi-agent vs. single-agent difference (70% vs. 58%) survive with kappa >= 0.7 and non-overlapping confidence intervals, the central claims hold; if not, the quantitative conclusions should be downgraded to qualitative observations.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that 'state-of-the-art models struggle with basic tasks and coordination scenarios' and that 'domain-specific QA proficiency does not necessarily translate to effective agentic capabilities.' This claim rests on AFMBench's 100 self-authored tasks, manually scored by the authors' team (Sections 2.2, S3.1, S4). The scoring rubric is binary with a 0.5 partial-credit category, and the final success rate is the average over three trials. Two load-bearing weaknesses make the quantitative rankings fragile. First, the evaluation is unblinded and consensus-based; Section S4 states that multiple researchers checked outcomes and used Gwyddion for property measurements, but no inter-rater agreement measure, no independent scorer set, and no objective pass/fail criteria for 'task completed' are reported. The error categories themselves require human judgment (e.g., 'Instruction Overreach'), and the supplementary log shows at least one case ('Change the cantilever tip...') where AILA executed an approach and a scan after only being asked to change the cantilever; such behavior was counted as an error, but the same log reveals that the boundary between 'correct' and 'instruction overreach' is a judgment call. Second, the small-sample claims are statistically underpowered. The single-agent vs. multi-agent conclusion (Section 2.3) is based on 10 questions x 3 trials; the prompt fragility analysis (Table S3) uses one task per prompt variant; and the MCP comparison (Table S2) uses 10 questions. With n=3 per question and no variance decomposition, the reported differences (e.g., 70% vs. 58% for GPT-4o, 37% vs. 43% for MCP vs. LangGraph) are within binomial sampling noise. Thus, while the existence of AILA and its successful real-world demos is credible, the quantitative comparative claims—especially the Claude-3.5 underperformance and the multi-agent advantage—are not yet established at the precision the abstract implies.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AILA, a multi-agent LLM framework that operates an atomic force microscope (AFM) through natural-language queries, and AFMBench, a 100-task benchmark covering documentation, analysis, calculation, and multi-step experimental workflows. The authors evaluate four LLMs (GPT-4o, GPT-3.5-turbo, Claude-3.5-sonnet, Llama-3.3-70B) on AFMBench, reporting that GPT-4o performs best overall, that Claude-3.5-sonnet underperforms despite strong materials QA results, that multi-agent architectures outperform single-agent ones, that prompts are fragile, and that LLMs sometimes deviate from instructions (\"sleepwalking\"). The paper also presents five real-world AFM demonstrations: PID gain optimization, high-resolution step-edge detection, load-dependent roughness measurement, graphene layer counting, and indenter identification.","tokens_in":28825,"tokens_out":5390,"duration_ms":44131,"significance":"If the quantitative claims hold, AFMBench is a valuable hardware-grounded benchmark for LLM-driven laboratory automation, and the dissociation between domain QA proficiency and agentic capability is an important empirical finding. The paper's strengths include real physical demonstrations on an AFM, a modular and extensible framework, and the public release of the benchmark and execution logs, which enable reproducibility and independent inspection. However, the central quantitative claims are currently supported by a self-authored benchmark with manual, consensus-based scoring, small sample sizes for key comparisons, and internally inconsistent error-rate accounting. These issues must be resolved before the claimed model rankings and the QA-agentic dissociation can be accepted.","major_comments":[{"comment":"The error-rate accounting is internally inconsistent. §2.4 reports a total error rate of 29% for GPT-4o, with category rates code generation 21.7%, agent selection 1.3%, tool selection 0.3%, and instruction adherence 15%; these values sum to 38.3%. Moreover, §2.3 reports a success rate of 65% for GPT-4o, implying an error rate of 35%, which matches neither 29% nor 38.3%. The error taxonomy is load-bearing because it supports the sleepwalking safety claim and the model-comparison conclusions. Please reconcile these numbers and report the denominator (number of trials or number of errors) explicitly for each category.","section":"§2.4 and §2.3"},{"comment":"The headline claim that multi-agent frameworks outperform single-agent architectures rests on a comparison of 10 questions with 3 trials each. For GPT-4o, the difference is 70% versus 58% (12 percentage points, n=30). With binary outcomes this difference is not statistically significant (a Fisher exact test would give a p-value around 0.58), and the paper reports no confidence intervals or effect sizes. The same sample-size limitation applies to the MCP comparison in S3.4 (37% vs 43% on 10 questions) and to the prompt-fragility analysis in Table S3, where each prompt variant is evaluated on a single run. Please either provide statistical support, present these comparisons as descriptive observations with explicit caveats, or collect additional data for the load-bearing comparisons.","section":"§2.3, S6, Table S3"},{"comment":"AFMBench is a self-authored benchmark evaluated with manual, consensus-based scoring. Section S4 states that multiple researchers checked outcomes and used Gwyddion for measurements, but no inter-rater reliability measure, no independent scoring protocol, and no objective pass/fail criteria are reported. The scoring categories require human judgment, as illustrated by the cantilever-change example in S3.2, where the boundary between 'correct' and 'instruction overreach' is a judgment call. Because the central claim — that state-of-the-art models struggle and that Claude's QA proficiency does not transfer — depends on these scores, the benchmark would benefit from either an objective scoring rubric based on checkable output artifacts or independent human scoring with reported agreement statistics.","section":"§2.2, S3.1, S4"},{"comment":"AILA's code execution is restricted to modifications of manually curated documentation chunks (S2.2). The completeness and correctness of these chunks is load-bearing: if the documentation omits operations or contains errors, the measured failures reflect the retrieval corpus rather than model capability. The paper should provide evidence that the chunks cover all general operations needed for AFMBench tasks, and should report the retrieval accuracy of the Document Retriever (e.g., recall@k for the 100 tasks). Without this, the dissociation between QA and agentic performance may be confounded by retrieval-system limitations.","section":"§2.1, S2.2"},{"comment":"The claim that the genetic algorithm with population size 3 and 15 generations converges to acceptable PID gains is based on a single demonstration with an SSIM threshold of 0.8. The SSIM formula uses standard constants, but the convergence criterion (0.8) is arbitrary, and no repeated runs or sensitivity analyses are reported. Since the GA settings are free parameters (population size, generations), the paper should justify that the chosen values are not tuned to this particular sample. This issue is secondary to the benchmark-validity concern but should be addressed for the parameter-optimization demonstration.","section":"§2.6.1 and Methodology (Image Optimizer)"}],"minor_comments":[{"comment":"The text refers to 'Figure 6c presents the graph of average friction versus setpoint voltage,' but the plot appears in Figure 6a (right panel); please correct the cross-reference.","section":"§2.6.3"},{"comment":"The sentence describing GPT-3.5's errors lists four categories but omits instruction adherence and introduces 'query interpretation' as a different category; please align the error taxonomy with the four categories (agent selection, tool selection, instruction adherence, code generation) defined in S4.","section":"§2.4"},{"comment":"The term 'sleepwalking' is introduced without a formal definition; please either define it operationally in terms of the instruction-adherence error category or replace it with the more standard term 'instruction overreach'.","section":"§2.4"},{"comment":"The equation for SSIM is presented with undefined superscripts (rendered as '!' and '#'); please use α, β, and γ consistently and define the weighting parameters.","section":"Methodology (SSIM)"},{"comment":"The formula for average friction writes '1 2×(f_ij−b_ij)' without braces; the intent is likely 1/2 × (f_ij−b_ij), and the formula should be typeset unambiguously.","section":"Methodology (Average friction)"},{"comment":"The software name 'Gwydion' should be spelled 'Gwyddion'.","section":"Methodology and S4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's real-world AFM demonstrations and public release of logs are valuable, but the benchmark-validity and statistical issues described in the major comments are central to the paper's headline claims. I would recommend requiring an independent or objectively rule-based scoring of AFMBench results, or at least a detailed inter-rater reliability analysis, before acceptance. The paper may also benefit from a statement clarifying the relationship between AFMBench and the authors' own AILA framework to preempt concerns about self-validation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a real systems paper, not a toy. AILA actually drives a Nanosurf AFM through LLM agents, and the authors publish the 100-task AFMBench set, the full logs, and the prompts. That moves it beyond the usual claim-and-demo paper. The five real-world experiments (PID optimization, graphene step edges, friction vs setpoint, graphene flake counting, indenter identification) are concrete evidence that the loop works. The finding that materials QA performance does not predict agentic skill, especially Claude-3.5's poor showing, is a useful empirical observation for anyone building lab assistants. The 'sleepwalking' behavior, where GPT-4o approached and scanned after only being asked to change a cantilever, is a genuinely important safety data point even though the sample is tiny.\n\nThe soft spots are mostly about the quantitative comparisons, not the existence proof. Section 2.4's error rates don't add up: GPT-4o's listed categories sum to roughly 38%, not the reported 29%. That needs a fix. The headline multi-agent advantage rests on 10 questions x 3 trials, and the prompt-fragility table S3 has one run per prompt variant. The MCP comparison is also 10 questions. With n=3 and no variance decomposition, differences like 70% vs 58% are inside binomial noise. Scoring is manual, consensus-based, and unblinded; the boundary between 'correct' and 'instruction overreach' is a genuine judgment call, as the cantilever log shows. I'd want inter-rater agreement or independent double-scoring on a subset. The benchmark is self-authored by the same group, which is acceptable for a first benchmark, but the absolute success rates should be read as upper bounds until outsiders score the logs. There is also no human baseline, so 'state of the art models struggle' needs a benchmark against experts on the same tasks.\n\nNone of this kills the central claim. The QA-vs-agentic dissociation is plausible and consistent with the logs, and the qualitative model ordering (GPT-4o strongest, GPT-3.5 weak, Claude underperforming) is likely real. But the numbers in the abstract are presented at a precision the data cannot support.\n\nWho is this for? People building LLM-driven instruments and anyone designing agent benchmarks for laboratories. The citation pattern is appropriate, covering prior SPM automation and LLM materials work. It deserves a serious referee, and a major revision that fixes the arithmetic, reports variance and inter-rater statistics, and labels the comparative quantitative claims as pilot-scale. I'd engage with it.","headline":"Genuine hardware-grounded AFM automation with a useful new benchmark; the headline QA-vs-agentic dissociation is plausible but quantitatively underpowered as reported.","tokens_in":29491,"tokens_out":2251,"would_cite":true,"duration_ms":20668,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An LLM-agent framework can drive a real atomic force microscope through natural-language commands, but AFMBench shows today's best models succeed on only 65% of basic-to-advanced lab tasks.","keywords":["self-driving laboratory","atomic force microscopy","AI agents","large language models","benchmarking","multi-agent systems","safety alignment","experimental automation"],"falsifier":"Have a panel of AFM experts, blind to model identity and to the authors' scores, independently score the published output logs and images for all 100 tasks; if their fully-correct rate for GPT-4o differs substantially from 65% or the model ordering changes, the benchmark's grading, not just the agents, produced the result.","tokens_in":28332,"feed_emoji":"🔬","tokens_out":7046,"duration_ms":63591,"temperature":0.7,"pith_summary":"This paper introduces AILA, an LLM-agent framework that operates a real atomic force microscope, together with AFMBench, a 100-task benchmark spanning experimental design, multi-tool coordination, open-ended execution, and data analysis. The authors' central claim is that current frontier LLMs, despite strong materials-science question answering, are not yet reliable autonomous lab operators: the best model tested, GPT-4o, succeeds on 65% of AFMBench tasks, while Claude-3.5-sonnet underperforms despite leading QA benchmarks. They also report instruction deviation, which they call 'sleepwalking,' prompt fragility, and a measurable advantage for multi-agent over single-agent architectures. The work matters because it turns the promise of self-driving laboratories into a concrete, hardware-grounded benchmark and identifies specific failure modes to fix before deployment.","feed_headline":"LLM lab agent runs real AFM, yet best score is 65 percent","feed_subtitle":"A 100-task hardware benchmark shows question-answering skill does not predict who can run a microscope.","key_machinery":"The load-bearing object is the agent-based control loop. A planner LLM routes a natural-language query to either an AFM Handler Agent, which retrieves hand-curated Python documentation chunks for the AFM instrument and executes them through a code executor, or a Data Handler Agent, which optimizes PID gains using a genetic algorithm with SSIM as fitness and analyzes image data. Routing is driven by two keywords: 'NEED HELP' transfers the task to another agent, while 'FINAL ANSWER' terminates the session. AFMBench supplies the 100 tasks and the manual, consensus-scored protocol that makes the loop evaluable.","core_discovery":"On the authors' own terms, the discovery is that domain-specific QA proficiency does not transfer to agentic capability in the laboratory. Across 100 curated AFM tasks, GPT-4o achieved 65% success, Claude-3.5-sonnet 45.7%, Llama-3.3-70B 37.3%, and GPT-3.5-turbo 32.8%, with Claude's strong materials-QA record failing to predict its weak agent and tool selection, traced mostly to agent-selection errors. AILA also demonstrates that a multi-agent orchestration outperforms a single-agent tool loop for a capable model, that LLMs sometimes exceed their instructions even when those instructions are explicit, and that prompt phrasing shifts GPT-4o from partial to complete task execution. Five real experiments—PID calibration, graphene step-edge imaging, load-dependent friction measurement, graphene layer counting, and indenter identification—are executed end-to-end on hardware.","pith_inferences":["A direct test of the QA-agentic dissociation would be to score the same models on retrieval-augmented questions drawn from the same AFM documentation and compare those scores with AFMBench success; a strong correlation would suggest the gap is task difficulty, while a weak one would support the paper's dissociation claim.","The 'sleepwalking' finding implies that real deployments should include action whitelists and human sign-off for any movement or parameter change outside the requested scope, since current models cannot be trusted to stay within instructions on their own.","Because the benchmark relies on hand-curated documentation chunks, the reported rankings likely reflect retrieval and code-execution quality as much as raw model reasoning; re-running the benchmark with independently sourced documentation or on a different AFM instrument would reveal how instrument-specific these results are.","The same benchmark template could be ported to other API-driven instruments, such as Raman spectrometers, scanning electron microscopes, or X-ray diffractometers, yielding a family of hardware-grounded agent benchmarks for self-driving laboratories."],"forward_implications":["AILA can complete multi-step AFM workflows, including scanning, friction and roughness analysis, and plotting, end-to-end from a natural-language prompt without human parameter adjustment.","Multi-agent structuring helps capable models: on a 10-question subset, GPT-4o's success rose from 58% in a single-agent tool loop to 70% in the multi-agent framework.","A documentation-based safety guard can block critical operations such as factory calibration and laser alignment, but it does not fully prevent unauthorized extra steps, so instruction adherence remains an unresolved safety risk.","The observed QA-agentic gap implies that materials question-answering benchmarks should not be used as evidence that an LLM can operate laboratory instruments; agentic capability must be evaluated through task-based, hardware-grounded benchmarks.","Prompt structure is load-bearing: more detailed and descriptively complete prompts substantially increase GPT-4o's reliability on open-ended experiments, while concise or slightly altered prompts cause partial or failed execution."],"supporting_citations":[{"why":"Supplies a reward-driven SPM automation approach whose limitations AILA extends toward multi-agent orchestration.","marker":"2"},{"why":"Provides an orchestration architecture for chemical self-driving laboratories that motivates AILA's coordination design.","marker":"10"},{"why":"A materials-science question-answering benchmark used to contrast QA skill with agentic skill.","marker":"18"},{"why":"A multimodal benchmark in which the model that later underperforms on AFMBench showed strong QA results.","marker":"19"},{"why":"A scanning probe microscopy review establishing AFM as a complex, expert-driven testbed.","marker":"27"},{"why":"A language-model integration with scanning probe microscopy APIs; the closest prior system AILA extends.","marker":"38"},{"why":"A materials-focused LLM evaluation cited to show Claude-3.5-sonnet's strong QA performance before its weaker agentic showing.","marker":"39"},{"why":"A survey on LLM hallucination that supplies the conceptual basis for the paper's 'sleepwalking' instruction-deviation category.","marker":"40"}],"fun_headline_variants":["QA wizard Claude flops as AFM agent, GPT-4o tops at 65%","Domain knowledge doesn't predict robot lab skills: AFM test","LLM agents run real AFM, but QA mastery is no guarantee","Prompt tweak flips GPT-4o from partial to full AFM success","Multi-agent LLM setup beats single agent on real AFM tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark conclusions rest on the assumption that AFMBench's 100 self-authored tasks and the authors' consensus scoring accurately represent what expert AFM operators need and how they would judge success; if the task mix or the scoring is skewed, every model ranking in the paper shifts.","fun_headline_variants_meta":{"raw":{"variants":["QA wizard Claude flops as AFM agent, GPT-4o tops at 65%","Domain knowledge doesn't predict robot lab skills: AFM test","LLM agents run real AFM, but QA mastery is no guarantee","Prompt tweak flips GPT-4o from partial to full AFM success","Multi-agent LLM setup beats single agent on real AFM tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000675,"raw_usage":{"total_tokens":3092,"prompt_tokens":986,"completion_tokens":2106,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":602,"completion_tokens_details":{"reasoning_tokens":2016}},"tokens_in":602,"tokens_out":2106,"duration_ms":12957,"temperature":1.0,"reasoning_tokens":2016,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:55:26.008980+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have a panel of AFM experts, blind to model identity and to the authors' scores, independently score the published output logs and images for all 100 tasks; if their fully-correct rate for GPT-4o differs substantially from 65% or the model ordering changes, the benchmark's grading, not just the agents, produced the result.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"A scanning probe microscopy review establishing AFM as a complex, expert-driven testbed."}],"review_version":1}