{"id":"99cb84e4-e852-48f1-b0de-9858127ff296","arxiv_id":"2608.05266","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A benchmark of 1,949 agentic microscopy runs shows configuration choices change cost and failure modes, but surrogate models cannot predict performance on unseen test identities.","lead":"Agentic microscope controllers were tested across 105 configurations and 53 benchmark tests. The authors find that benchmarks are useful for qualification and debugging but do not support picking one configuration that generalizes to unseen tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The task-held-out AUC of 0.55 is the sole quantitative support for the central negative claim, yet it is reported without confidence intervals or a permutation null on a small, version-drifting dataset; the 'near chance' conclusion is not yet quantitatively secured.","rationale":"The paper is well-hedged, transparent about its limitations, and provides independent support for its practical claims: the framework logs traces and RAG retrievals, the prospective comparison identifies a strong one-agent pruned-RAG configuration, and physical validation workflows demonstrate real microscope control. These elements stand regardless of the surrogate analysis. However, the central negative claim about generalization rests on a single point estimate of AUC without uncertainty quantification. The reader correctly identified missing error bars in the rationale and worried about temporal confounding in the weakest_assumption field. My emphasis differs slightly: the most load-bearing issue is that the task-held-out AUC itself is not characterized with any uncertainty measure, so the 'near chance' interpretation is not quantitatively grounded. Temporal confounding is a plausible mechanism that could bias that estimate, but the first-order problem is that we cannot tell whether 0.55 is meaningfully different from 0.5 or from a design-induced null. The proposed test—cluster bootstrap CI plus a permutation null and a time-covariate sensitivity check—would directly resolve whether the negative conclusion is robust. Because this is a missing analysis rather than a demonstrated error, and because the paper's hedging and additional evidence support the conditional disposition, I do not recommend changing the reader's verdict; the same CONDITIONAL standard should be retained pending the uncertainty quantification.","tokens_in":13080,"tokens_out":5484,"duration_ms":54486,"concrete_test":"Compute a cluster-bootstrap 95% CI for the task-held-out ROC-AUC by resampling test identities (not individual runs) 1,000 times, rerunning the same grouped-CV surrogate pipeline each time. Also run a permutation null in which pass/fail outcomes are shuffled within each test group (preserving the unbalanced config-test design), and add a 'study phase' or software-version timestamp feature to the design matrix as a sensitivity check. If the CI includes 0.5 or the observed AUC is not above the 95th percentile of the permutation null, the 'near chance' conclusion is supported; if the CI excludes 0.5 but AUC remains below ~0.6, the conclusion should be weakened to 'weak, non-reliable transfer'; and if adding the timestamp moves AUC by more than 0.05, temporal confounding is a major driver and the benchmark-level conclusion needs re-examination.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.10 describes grouped cross-validation with up to five folds for the task-held-out analysis, and Table 1 reports ROC-AUC values of 0.53 (logistic regression) and 0.55 (random forest) without confidence intervals, standard errors, or permutation tests. Section 3.3 explicitly acknowledges that the dataset is historical, unbalanced, has non-independent repeats, and that model, prompt, framework, software, and hardware versions changed during the study. Under these conditions, a point estimate of 0.55 can easily lie within sampling noise: with only 53 test groups and variable class balance, the cluster-level uncertainty in AUC is large. If the 95% CI includes 0.5, the evidence that configuration carries no transferable signal is weak; if the CI is narrow and excludes 0.5, the conclusion would instead be that transfer is weak but detectable, changing the paper's 'near chance' framing. Temporal confounds could bias the estimate in either direction, since version changes may correlate with both configuration and outcome. Because the central claim—that the current heterogeneous test suite does not support a task-independent global configuration model—rests almost entirely on this single near-chance AUC, the absence of uncertainty quantification is the most load-bearing gap.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a trace-logging and benchmark framework for agentic microscopy control built on one-, two-, and three-agent LLM graphs connected to six MCP servers. Over 105 configurations, 1,949 test runs, and 49,109 RAG retrievals across 53 benchmark tests, the authors show that direct comparisons reveal clear differences in token use, latency, cost, and failure modes, and that a one-agent pruned-RAG configuration was the best prospective candidate, though a no-RAG baseline performed approximately as well. The central negative claim is about transfer: a random-forest surrogate reaches ROC-AUC ≈ 0.78 under configuration-held-out validation, but falls to 0.55 under task-held-out validation (0.53 for logistic regression), which the authors interpret as showing that the current heterogeneous test suite does not support a task-independent global configuration model. The paper also reports RAG-retrieval auditing, physical validation on a ZEISS Versa 730, and a careful discussion of what the benchmark can and cannot support.","tokens_in":13312,"tokens_out":2939,"duration_ms":27735,"significance":"If the central negative claim is secure, this is a valuable and timely negative result for the nascent field of LLM-controlled scientific instruments: it would caution against using aggregate benchmark pass rates to optimize agent configurations for unseen tasks, while preserving the benchmark's value for qualification, regression testing, and diagnosis. The paper's strengths are its large traced dataset, its use of two model classes (random forest and L2 logistic regression), an explicit grouped cross-validation scheme, a sensitivity analysis removing near-universal tests, prospective physical validation, and openly released code and data. The claim is not circular: the task-held-out evaluation removes complete test identities, so the negative result does not arise from fitting the conclusion. However, the statistical support for the near-chance task-held-out AUC is currently incomplete, and the central claim rests on a small number of quantitative point estimates.","major_comments":[{"comment":"The task-held-out ROC-AUC values (0.53 for logistic regression, 0.55 for random forest) are reported as point estimates without confidence intervals, standard errors, or a permutation null. With 53 test groups, up to five folds, and variable class balance, the sampling uncertainty around these AUC values is plausibly large; if the 95% confidence interval includes 0.5, the evidence that configuration carries no transferable signal is weak, whereas a narrow interval excluding 0.5 would support a different framing ('weak but detectable transfer'). Please provide cluster-resampled confidence intervals (e.g., bootstrap by test identity) and a permutation null for the task-held-out analysis, and report the number of folds and groups actually used.","section":"Table 1 / §5.10"},{"comment":"The sensitivity analysis that removes near-universally passing tests is reported only as 'did not materially improve the task-held-out result,' with no numerical AUC values. Because this analysis directly addresses whether the negative result is dominated by uninformative easy tests, please report the resulting task-held-out AUCs (and their uncertainties) for the reduced test set, or state explicitly which tests were removed and why the numbers were omitted.","section":"§2.3 / §5.10"},{"comment":"The manuscript acknowledges that the dataset is historical and unbalanced, that model, prompt, framework, code, software, and hardware versions changed during the study, and that repeated runs are not fully independent. Under these conditions, the task-held-out null result could be confounded by temporal drift if version changes correlate with both configuration and outcome. The paper should address this possibility concretely, for example by testing whether the task-held-out AUC changes when the analysis is restricted to a period of stable software versions, or by including version/time as a covariate, or at minimum by discussing the direction and magnitude of plausible bias.","section":"§3.3 / §5.2"}],"minor_comments":[{"comment":"The repository URL appears as plain text 'github.com/natertott/agentic microscopy benchmarks XRM' with spaces; please format it as a proper clickable URL with the correct repository path.","section":"Code and Data Availability"},{"comment":"The caption states 'Error bars shows the Wilson 95% confidence interval'; this should be 'Error bars show'.","section":"Figure 5 caption"},{"comment":"The phrase describing the no-RAG baseline as performing 'approximately as well' on the standardized benchmark is not quantified; please report the actual pass rates and confidence intervals so readers can judge the practical difference between the best candidate and the baseline.","section":"§2.5 / §3.1"},{"comment":"The text says grouped cross-validation used 'up to five folds'; please clarify whether the number of folds was fixed at five, whether it varied by group availability, and whether a single split or repeated/randomized grouping was used.","section":"§5.10"}],"recommendation":"major_revision","confidential_remarks":"The paper is a well-written and honest systems report, and the central claim is timely. The main issue is statistical: the task-held-out AUC is a single point estimate without uncertainty quantification, and the paper's negative conclusion depends on that estimate. If the authors can supply confidence intervals and a permutation null, and address the temporal-confounding concern, the paper would be suitable for publication. The scope is applied AI-for-science instrumentation; it may be a stronger fit for an applied venue or a methods journal than for a purely machine-learning theory venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about arXiv:2608.05266. First, it's a serious, data-rich study of agentic microscope control—105 configurations, 53 tests, 1,949 runs, and 49,109 logged RAG retrievals—and it ships code and data. Second, its central negative claim, that benchmark pass rates do not support a global configuration model for unseen tasks, is plausible but not quantitatively nailed down: the task-held-out ROC-AUC of 0.55 is reported without confidence intervals or a permutation null, on a dataset the authors themselves describe as historical, unbalanced, and with version drift. On that evidence, 'near chance' could be noise or weak-but-real transfer.\n\nWhat's actually new: the explicit split between configuration-held-out and task-held-out validation for microscope agents. The config-held-out RF gets ~0.78 AUC because test identity dominates; task-held-out drops to ~0.55. That decomposition is exactly the right way to ask whether a benchmark can be used for optimization, and I don't know prior work that does it for instrument agents. The RAG retrieval-to-outcome linkage with stable identifiers is also a real contribution—it turns the context window into an auditable object and shows which entries are stale or harmful. The direct comparisons (one vs two vs three agents, token/cost/failure mode differences) are useful engineering data, and the physical validation on a real Versa 730 with quantitative criteria is more than most papers in this space do.\n\nSoft spots, in proportion. The missing uncertainty quantification on Table 1 is the main one. With 53 test groups and up to 5 folds, the cluster-level AUC variance is large; a 0.55 point estimate could easily have a 95% CI that includes 0.5, which would turn 'no transferable signal' into 'we couldn't detect one.' That distinction matters for the paper's conclusion. The authors are admirably honest about the dataset limitations in Section 3.3, but they don't translate those limitations into error bars or a permutation test, which is the one analysis that would settle the question. Minor point: the prospective comparison picks candidates using the same 53-test benchmark, so it's a direct comparison, not a validation of the surrogate; the authors say as much, so it's not a flaw, just a boundary.\n\nThe paper is carefully hedged, the reasoning is clear, and the limitations are stated rather than hidden. The citation pattern looks appropriate. If I were refereeing, I'd ask for confidence intervals or a permutation null on the task-held-out AUC, and for a discussion of temporal confounding that the version drift could introduce. If the CI excludes 0.5, the conclusion can stay; if not, the claim weakens to 'we did not find transfer,' which is still a legitimate result but a different one.\n\nThis deserves a serious referee. I'd bring it to reading group—it's a good template for how to think about benchmark generalization in autonomous labs.","headline":"A genuinely useful benchmark and honest negative result, but the load-bearing 'near chance' AUC lacks error bars on a small, version-drifting dataset, so the strength of the transferability claim is not yet secured.","tokens_in":13833,"tokens_out":2824,"would_cite":true,"duration_ms":24634,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Agentic microscopy benchmarks can qualify and regression-test agents, but task-held-out surrogate models perform near chance, so no global configuration ranking transfers to unseen tasks.","keywords":["agentic microscopy","LLM agents","benchmark generalization","task-held-out validation","retrieval-augmented generation","self-driving laboratory","Model Context Protocol","surrogate models"],"falsifier":"Run a randomized factorial sweep with a fixed software version, balanced repeats, and task-held-out folds: if the surrogate's held-out ROC-AUC rises well above 0.55 (for example, above 0.7) on a fresh collection of unrelated microscopy tasks, the paper's claim of no transferable configuration signal would be overturned.","tokens_in":12855,"feed_emoji":"🔬","tokens_out":5746,"duration_ms":47306,"temperature":0.7,"pith_summary":"This paper asks whether benchmark performance on a fixed set of microscopy tasks can be used to choose an agent configuration that will also work on tasks the agent has never seen. The authors built a trace-logging benchmark of 53 tests, ran 105 agent configurations across 1,949 test runs, and trained surrogate models under two validation schemes. When known tests stayed in training, the random-forest surrogate reached an ROC-AUC of about 0.78; when entire test identities were held out, both surrogates fell to about 0.55, near chance. The paper concludes that this benchmark suite is useful for qualification, regression testing, diagnosis, and direct comparison, but it cannot support a task-independent global configuration model. If true, this matters because teams building autonomous microscopes and beamlines need to know what a pass rate is and is not evidence for.","feed_headline":"Near-chance scores show agent benchmarks fail on unseen tasks","feed_subtitle":"Task-held-out scores near chance, so pass rates qualify systems but cannot optimize for new workflows.","key_machinery":"The central mechanism is grouped cross-validation with task-held-out folds: all runs sharing a test identity are assigned to the same fold, test identity is removed from the design matrix, and the surrogate must rank configurations purely from configuration features on unseen task identities. This contrasts with configuration-held-out validation, in which known test identities remain in training and the model mostly learns intrinsic task difficulty. Supporting this are the composite pass criteria (scientific correctness, trace checks, artifact checks, operational limits), the stable configuration hash that defines an agent design, and RAG retrieval identifiers that make each retrieved context fragment auditable in relation to pass or fail outcomes.","core_discovery":"The central claim is that the heterogeneous benchmark suite does not support a task-independent global configuration model. Configuration variables—number of agents, LLM, temperature, RAG parameters, context limits—affected latency, token use, cost, and failure mode, and test identity was the strongest predictor in the configuration-held-out surrogate. Once complete tests were removed from training and test-identity features were dropped, L2 logistic regression and random forest both achieved ROC-AUC around 0.55. The paper therefore asserts that benchmark pass rates remain valid for qualification, regression testing, diagnosis, and direct comparison of defined workflows, while a separate, prospective comparison can still identify a good configuration for the tested workflows, with a one-agent pruned-RAG configuration performing best while a no-RAG baseline performed approximately as well.","pith_inferences":["If test identity dominates as the paper's feature analysis shows, similar task-held-out near-chance results should be expected for other scientific agent benchmarks (synthesis, synchrotron beamlines, imaging), and benchmark reports should publish task descriptors to test transfer.","Because the dataset is historical and unbalanced with changing model, software, and hardware versions, a randomized factorial benchmark with fixed versions and controlled repeats could separate true configuration signal from temporal drift; this is the paper's own stated limitation, extended here as the next experiment.","The RAG pruning result suggests that context databases should be maintained by software-maintenance criteria (stale, contradictory, topology-inappropriate entries) rather than by outcome association alone, since retrieval is observational and confounded by task and workflow state.","A practical extension is to build task-aware surrogate models that include workflow descriptors; if such descriptors restored held-out AUC, the conclusion would shift from 'no global model' to 'the right task representation supports transfer'."],"forward_implications":["Benchmarks should be used for qualification, regression testing, diagnosis, and direct comparison, and should not be treated as an optimization oracle for unseen tasks.","Multi-agent topologies did not consistently improve pass rates and generally increased tokens, calls, and time, so added agents need a demonstrated payoff.","A simple no-RAG baseline should be part of every agent comparison, since additional context and architectural complexity gave no consistent advantage.","Future benchmark-driven optimization should move toward task-aware configuration selection—describing workflow family, required tools, sequence length, and quantitative versus trace endpoint—rather than one global ranking.","Repeated scientific operations should be moved from free-form LLM code into deterministic, checked tools, since the least reliable step was ad hoc Python written during the workflow."],"supporting_citations":[{"why":"Supplies the ReAct reasoning-and-acting loop used by every agent topology in the study.","marker":"[35]"},{"why":"Defines the Model Context Protocol that connects the controller to the six tool servers, the architectural backbone of the system.","marker":"[36]"},{"why":"Provides the prior observation of agents continuing action after task completion, used to interpret the two-agent failure mode.","marker":"[19]"},{"why":"Establishes the distinction between benchmarking LLM knowledge and benchmarking the complete agent system, which frames the benchmark design.","marker":"[30]"},{"why":"Supports the motivation for auditing RAG context by showing retrieval quality affects LLM outputs.","marker":"[27]"},{"why":"Offers the word-error-rate endpoint example for a synchrotron routing agent, used to illustrate task-appropriate benchmark endpoints.","marker":"[12]"}],"fun_headline_variants":["Agent benchmarks qualify but can't predict unseen tasks","Microscopy agent tests fail to generalize to new tasks","Benchmarks pass for known tasks, flop on unseen ones","Benchmark scores don't forecast agent performance on new tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the grouped task-held-out cross-validation of this historical, unbalanced dataset—with non-independent repeats and changing software versions—accurately measures transfer to new tasks rather than being distorted by temporal drift or confounding.","fun_headline_variants_meta":{"raw":{"variants":["Agent benchmarks qualify but can't predict unseen tasks","Microscopy agent tests fail to generalize to new tasks","Benchmarks pass for known tasks, flop on unseen ones","Benchmark scores don't forecast agent performance on new tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000308,"raw_usage":{"total_tokens":1788,"prompt_tokens":1002,"completion_tokens":786,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":618,"completion_tokens_details":{"reasoning_tokens":720}},"tokens_in":618,"tokens_out":786,"duration_ms":6441,"temperature":1.0,"reasoning_tokens":720,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T16:42:32.143565+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a randomized factorial sweep with a fixed software version, balanced repeats, and task-held-out folds: if the surrogate's held-out ROC-AUC rises well above 0.55 (for example, above 0.7) on a fresh collection of unrelated microscopy tasks, the paper's claim of no transferable configuration signal would be overturned.","supporting_citations":[{"cited_title":"ReAct: Synergizing reasoning and acting in language models","cited_arxiv_id":null,"evidence_quote":"Supplies the ReAct reasoning-and-acting loop used by every agent topology in the study."},{"cited_title":"Model context protocol specification.https:// modelcontextprotocol.io/specification/2025-06-18/, 2025","cited_arxiv_id":null,"evidence_quote":"Defines the Model Context Protocol that connects the controller to the six tool servers, the architectural backbone of the system."},{"cited_title":"Smedskjaer, Katrin Wondraczek, Lothar Won- draczek, Nitya Nand Gosvami, and N","cited_arxiv_id":null,"evidence_quote":"Provides the prior observation of agents continuing action after task completion, used to interpret the two-agent failure mode."},{"cited_title":"Heurekabench: A benchmarking framework for ai co-scientist, 2026","cited_arxiv_id":null,"evidence_quote":"Establishes the distinction between benchmarking LLM knowledge and benchmarking the complete agent system, which frames the benchmark design."},{"cited_title":"Retrieval meets long context large language models","cited_arxiv_id":null,"evidence_quote":"Supports the motivation for auditing RAG context by showing retrieval quality affects LLM outputs."},{"cited_title":"Yager, and Esther H","cited_arxiv_id":null,"evidence_quote":"Offers the word-error-rate endpoint example for a synchrotron routing agent, used to illustrate task-appropriate benchmark endpoints."}],"review_version":1}