Pith. sign in

REVIEW 4 major objections 6 minor 54 references

An autonomous multi-agent system can build high-performing, auditable medical imaging models across heterogeneous tasks by planning from the data and verifying every score and submission artifact.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 11:04 UTC pith:TD2TJV3C

load-bearing objection Solid systems paper: first real end-to-end agent that ships valid medical-imaging challenge packages on 20 tasks; design ablations still missing. the 4 major comments →

arxiv 2607.10522 v1 pith:TD2TJV3C submitted 2026-07-12 cs.CV cs.AI

Towards Autonomous and Auditable Medical Imaging Model Development

Shengyuan Liu , Jia-Xuan Jiang , Boyun Zheng , Cheng Wang , Zipei Wang , Wentao Pan , Hongtao Wu , Houwen Peng
show 3 more authors
Yu Gu Lichao Sun Yixuan Yuan
This is my paper
classification cs.CV cs.AI
keywords autonomous agentsmedical imagingmachine learning engineeringmulti-agent systemsmodel developmentverificationmethod planningchallenge benchmarks
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Medical imaging model development is not a generic machine-learning recipe hunt: modality, anatomy, patient splits, metrics, and submission formats all decide what works, and a high score is useless if the validation protocol or prediction files are wrong. This paper introduces AMID, a multi-agent framework that first turns the task data and runnable domain resources into parallel method lanes, then explores those lanes broadly before exploiting the best candidates while a reviewer rejects untrusted evidence. Across twenty medical imaging challenge tasks spanning segmentation, detection, classification, and enhancement, the system produced valid accepted artifacts for every task and beat the evaluated general-purpose machine-learning engineering systems on nearly all of them, approaching or matching strong human challenge solutions on several. The claim is that domain-conditioned planning plus continuous artifact verification can convert bespoke medical model engineering into a repeatable agentic workflow that leaves inspectable code, scores, and submissions.

Core claim

AMID shows that autonomous medical imaging model development works when the search space is made executable from the actual data and when only reviewer-checked validation protocols, metrics, and prediction artifacts may drive promotion and final selection. On a twenty-task challenge suite spanning diverse modalities and prediction types, this design produced valid end-to-end artifacts for every task, improved over the strongest listed general-purpose autonomous MLE baselines on nineteen of twenty tasks, and on several tasks approached or matched strong human-designed challenge solutions.

What carries the argument

Data-Conditioned Method Planning (DCMP) plus Verification-Guided Two-Stage Optimization. DCMP profiles the public data and grounds candidates in runnable medical-imaging resources to form parallel method lanes; the optimizer then explores those lanes broadly, exploits promoted candidates, and admits only protocol-valid, metric-correct, submission-ready evidence through an independent reviewer.

Load-bearing premise

The gains are assumed to come mainly from the planning and verification design rather than coding-backend strength, compute budget, or baselines failing medical artifact checks—and full ablations of those pieces are still incomplete.

What would settle it

Rerun the same twenty-task suite with matched model backend and wall-clock budget, ablating data-conditioned method planning and the reviewer verification gates separately; if the full system no longer beats those ablations and the general-purpose agents on accepted primary metrics, the central design claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Medical model building can be treated as searchable, auditable engineering rather than one-shot code generation.
  • General-purpose MLE agents without domain-conditioned lanes and artifact gates will underperform on heterogeneous clinical imaging tasks.
  • The same verified search policy can deliver submission files, trained model packages, or Dockerized predictors.
  • Challenge-style agent evaluation becomes a system-level test of pipeline completeness, not only local validation scores.
  • Cost-aware long-horizon autonomy becomes the next practical bottleneck once planning and verification are in place.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Similar data-conditioned lanes and artifact gates may transfer to other regulated domains where format-correct submissions matter as much as scores.
  • If full ablations of planning and verification remain weak, headline gains may largely reflect stronger coding runtimes and longer budgets rather than the search architecture.
  • Tasks that couple small-structure topology with downstream classification may need joint multi-stage objectives beyond single-lane exploitation under fixed budgets.
  • Reusable skill memory for foundation-model adaptation could shorten later runs as successful procedures accumulate across workers.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This preliminary technical report introduces AMID, a multi-agent system for autonomous medical imaging model development. Given a task definition and dataset, AMID first runs Data-Conditioned Method Planning (DCMP) to convert task/data profiles and runnable medical resources into parallel method lanes, then applies Verification-Guided Two-Stage Optimization: broad, reviewer-gated exploration of lanes followed by selective exploitation of promoted candidates, with independent checks on validation protocols, metrics, and prediction/submission artifacts. On the 20-task ReX-MLE suite (segmentation, detection, classification, IQA/enhancement; 24h wall-clock and one A6000 GPU per task; Codex+GPT-5.5 main backend), AMID reports valid accepted artifacts on all tasks, improves over the strongest listed general-purpose MLE baseline (AIDE, ML-Master, R&D-Agent) on 19/20 tasks, and approaches or matches original-competition human references on several challenges. Case studies attribute strong runs to domain-matched pipelines (e.g., YOLO+anatomical priors on DENTEX; pathology foundation models on PUMA) and weaker runs to long-horizon enhancement and topology-dependent classification.

Significance. If the empirical superiority and auditability claims hold under stronger controls, this is a meaningful systems contribution: it is among the first end-to-end agentic MLE frameworks specialized to heterogeneous medical imaging, with an explicit artifact contract and independent reviewer gate rather than score-only search. The 20-task ReX-MLE evaluation with accepted-artifact criteria, the focused GPT-5.5-unified backend comparison on three tasks (Table 2), public solution release, and the framing of medical MLE as executable, verifiable search are concrete strengths. The work would matter for reducing bespoke challenge engineering and for making agent-built medical models inspectable. Significance is currently tempered by missing component ablations and by the possibility that large deltas partly reflect reliable valid-submission production under medical contracts rather than uniquely superior method search.

major comments (4)
  1. Table 1 / §3.2: The headline 19/20 superiority claim is load-bearing for the paper’s design narrative, but many baseline cells are FAIL or near-zero on segmentation/detection (e.g., PUMA tissue seg, ISLES’22, SEG.A). That pattern is consistent with protocol/artifact collapse as much as with weaker medical method search. Please quantify, for each baseline and task, whether failure was invalid artifacts/metric/split vs. a valid but weak model, and report a secondary comparison restricted to tasks where all systems produced accepted artifacts. Without this, the large deltas can be read as “AMID ships valid medical submissions” rather than as evidence that DCMP + two-stage optimization primarily improve modeling quality.
  2. §2.3–2.4 and Discussion: DCMP and Verification-Guided Two-Stage Optimization are the claimed technical contributions, yet the manuscript provides no full-suite (or even multi-task) ablations that remove or disable (i) data-conditioned lane construction, (ii) the explore→promote→exploit schedule, or (iii) the independent reviewer gate. Table 2 only unifies the model backend on three tasks and still does not isolate these modules. For a systems paper whose central claim is that these mechanisms drive reliable high-performing medical MLE, at least a focused multi-task ablation (e.g., generic planning vs DCMP; single-stage score-driven search vs two-stage; no reviewer vs reviewer) is needed before the design—not just the full stack—can be credited.
  3. §3.1–3.2 and Figure 3: Human-normalized comparisons use original-competition test-set references while agents train on re-split public training data with less data. The text acknowledges this, but the abstract, Figure 1, and Figure 3 still invite a “near human / human level” reading. Please demote human bars to an explicitly approximate scale in all main displays, avoid “matched” language, and report absolute accepted metrics as the primary claim; otherwise the “approached or matched strong human-designed solutions” statement overreaches the experimental design.
  4. §2.4 Stage-1 promotion gate and free parameters: Lane coverage, plateau, and Top-K promotion criteria are described qualitatively but not specified with thresholds, budgets per stage, or sensitivity analysis. Given a fixed 24h/A6000 budget, promotion timing can dominate which families receive exploitation compute. Please state the operational promotion rules used for the main suite and show that headline rankings are stable under reasonable gate settings (or report variance across seeds/runs where feasible).
minor comments (6)
  1. Abstract/full-text opening: “Large anguage model” is a typo for “Large language model.”
  2. Figure 1 / Table 1: Clarify whether “Human level” and Human columns are original leaderboard scores or ReX-MLE re-split scores in every caption; currently the distinction is only fully clear in §3.2 prose.
  3. §2.5 Shared memory / skills: The skill mechanism is interesting but lightly evidenced; a short example of a skill created mid-run and reused would help readers separate design description from demonstrated behavior.
  4. Related Work §5: Several 2025–2026 arXiv systems are cited appropriately; ensure citation completeness for ReX-MLE evaluation protocol details so readers can reproduce the accepted-artifact criterion without external ambiguity.
  5. Note that this is an “ongoing preliminary technical report” is appropriate; please keep that framing consistent in the abstract and conclusion so readers do not treat deferred ablations as optional polish.
  6. USenhance and TopCoW-Cls failure analyses (§3.3) are useful; consider a compact failure taxonomy table (budget, topology coupling, long-horizon training instability) for the full suite.

Circularity Check

0 steps flagged

No circular derivation: AMID’s claims are external ReX-MLE benchmark results, not predictions forced by fitted inputs or self-citation theorems.

full rationale

This is an empirical multi-agent systems paper, not a first-principles derivation. The load-bearing claim is that AMID produces valid accepted artifacts on all 20 ReX-MLE tasks and beats listed general-purpose MLE baselines on 19/20 under stated challenge contracts (Table 1; Sec. 3.1–3.2). Those scores are external challenge metrics and accepted-artifact checks, not quantities defined from parameters fitted to the same targets. DCMP and Verification-Guided Two-Stage Optimization are proposed search/control procedures evaluated by end-to-end runs; they are not uniqueness theorems, cosh-style ansatzes, or renamings of a known closed-form law. Self-citations in related work (e.g., overlapping medical-AI-scientist lines) are ordinary positioning and do not force the Table 1 deltas. Concerns about missing full-suite ablations, backend/compute confounds, or approximate human references are causal-attribution and fairness issues, not circular reductions of outputs to inputs by construction. Under the circularity criteria, the paper is self-contained against external benchmarks; steps is empty.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 4 invented entities

This is an empirical systems paper. The central claim rests on engineering design choices, benchmark contracts, compute/backends, and curated medical resources rather than free mathematical parameters. The main unproved loads are that ReX-MLE accepted scores under re-split public data fairly rank systems, that reviewer-admissible artifacts are the right success criterion, and that the curated resource/method-lane priors are adequate without exhaustive search of all medical methods.

free parameters (4)
  • 24-hour wall-clock budget per challenge = 24 hours
    Fixed experimental resource that shapes how far exploration and exploitation can go; not derived, chosen for evaluation.
  • Single NVIDIA RTX A6000 48GB GPU per challenge = 1x A6000 48GB
    Hardware budget that constrains trainable model families and fold completeness; evaluation design choice.
  • Promotion gate thresholds (lane coverage, plateau, Top-K)
    Stage-1 to Stage-2 promotion depends on coverage/plateau/Top-K style rules described qualitatively; exact numeric cutoffs are system hyperparameters affecting which lanes get compute.
  • Primary coding-agent backend (Codex + GPT-5.5 for full suite) = Codex + GPT-5.5 (main)
    Underlying model/runtime is a major free choice; Table 2 shows backend changes alter per-task scores.
axioms (5)
  • domain assumption ReX-MLE accepted primary metrics under original challenge units are the right external measure of autonomous medical imaging MLE success.
    Sec. 3.1–3.2 evaluate systems by accepted artifacts and challenge metrics; clinical utility is explicitly out of scope.
  • ad hoc to paper A result is only admissible evidence if validation protocol, metric computation, and prediction/submission artifacts pass independent reviewer checks.
    Core design premise of Verification-Guided Two-Stage Optimization (Sec. 2.4); defines what can promote or finalize candidates.
  • domain assumption Task-specific data profiling plus runnable medical resources (nnU-Net, MONAI, foundation-model registry) can be converted into useful parallel method lanes.
    DCMP premise (Sec. 2.3); if profiling or the resource library is weak, search quality collapses.
  • domain assumption Human original-competition scores are useful approximate reference scales even though agents train on re-split public data with less data.
    Explicitly stated caveat in Sec. 3.1; used in Fig. 3 and Table 1 Human column.
  • domain assumption General coding-agent runtimes can serve as interchangeable worker backends under a shared artifact/memory control plane.
    Sec. 2.2 and 2.5 claim backend-agnostic execution; only partially stress-tested.
invented entities (4)
  • AMID multi-agent framework independent evidence
    purpose: End-to-end autonomous medical imaging model development with auditable packages.
    Primary system introduced; evaluated empirically on ReX-MLE rather than postulated as a physical entity.
  • Data-Conditioned Method Planning (DCMP) no independent evidence
    purpose: Turn task/data evidence and runnable resources into executable parallel method lanes.
    Named planning module; evidence is system behavior and case studies, not external theory.
  • Verification-Guided Two-Stage Optimization no independent evidence
    purpose: Explore diverse lanes then exploit promoted candidates under reviewer gates on protocols/metrics/artifacts.
    Named optimization procedure; support is architectural description plus benchmark outcomes, without full ablations yet.
  • Method lanes / method-lane portfolio no independent evidence
    purpose: Unit of search organizing hypotheses, resources, validation obligations, and failure modes.
    Organizational abstraction invented for the search process.

pith-pipeline@v1.1.0-grok45 · 26294 in / 3837 out tokens · 40719 ms · 2026-07-14T11:04:34.996942+00:00 · methodology

0 comments
read the original abstract

Large language model (LLM) agents are beginning to automate machine learning engineering (MLE) by coupling planning, code execution, debugging, and empirical feedback. Translating this capability to medical imaging remains difficult because each task imposes modality-specific experimentation and strict requirements for validation protocols and prediction artifacts. Here we introduce AMID, an autonomous multi-agent framework for medical imaging model development. AMID first proposes Data-Conditioned Method Planning, which refines coarse task-level search spaces into executable, parallelizable method lanes grounded in task-specific data analysis and runnable medical-imaging resources. It then develops Verification-Guided Two-Stage Optimization, moving from broad early exploration of diverse method lanes to selective exploitation of promising candidates while enforcing strict verification of validation protocols, metric computation, and prediction artifacts throughout the optimization. Across 20 medical imaging challenge tasks spanning diverse modalities and prediction types, AMID outperformed evaluated general-purpose MLE systems and, on several tasks, approached or matched strong human-designed challenge solutions. These results suggest that AMID can turn task-specific medical imaging model development from bespoke manual engineering into an agentic workflow for producing high-performing and auditable model artifacts across heterogeneous tasks.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

54 extracted references · 1 canonical work pages

  1. [1]

    Daniil A Boiko, Robert MacKnight, Ben Kline, and Gabe Gomes

    URL https://doi.org/10.5281/zenodo.15192302. Daniil A Boiko, Robert MacKnight, Ben Kline, and Gabe Gomes. Emergent autonomous scientific research capabilities of large language models.Nature, 624:570–578,

    work page doi:10.5281/zenodo.15192302

  2. [2]

    MONAI: An open-source framework for deep learning in healthcare.arXiv preprint arXiv:2211.02701,

    M Jorge Cardoso, Wenqi Li, Richard Brown, Nic Ma, Eric Kerfoot, Yiheng Wang, Benjamin Murrey, Andriy Myronenko, Can Zhao, Dong Yang, et al. MONAI: An open-source framework for deep learning in healthcare.arXiv preprint arXiv:2211.02701,

    Pith/arXiv arXiv

  3. [3]

    MLE-Bench: Evaluating machine learning agents on machine learning engineering.arXiv preprint arXiv:2410.07095,

    Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, et al. MLE-Bench: Evaluating machine learning agents on machine learning engineering.arXiv preprint arXiv:2410.07095,

    Pith/arXiv arXiv

  4. [4]

    MARS: Modular agent with reflective search for automated AI research.arXiv preprint arXiv:2602.02660,

    Jiefeng Chen, Bhavana Dalvi Mishra, Jaehyun Nam, Rui Meng, Tomas Pfister, and Jinsung Yoon. MARS: Modular agent with reflective search for automated AI research.arXiv preprint arXiv:2602.02660,

    Pith/arXiv arXiv

  5. [5]

    Barbarians at the gate: How AI is upending systems research.arXiv preprint arXiv:2510.06189,

    Audrey Cheng, Shu Liu, Melissa Pan, Zhifei Li, Bowen Wang, Alex Krentsel, Tian Xia, Mert Cemri, Jongseok Park, Shuo Yang, et al. Barbarians at the gate: How AI is upending systems research.arXiv preprint arXiv:2510.06189,

    arXiv

  6. [6]

    AutoMLGen: Navigating fine-grained optimization for coding agents.arXiv preprint arXiv:2510.08511,

    Shangheng Du, Xiangchao Yan, Dengyang Jiang, Jiakang Yuan, Yusong Hu, Xin Li, Liang He, Bo Zhang, and Lei Bai. AutoMLGen: Navigating fine-grained optimization for coding agents.arXiv preprint arXiv:2510.08511,

    arXiv

  7. [7]

    MLEvolve: A self-evolving framework for automated machine learning algorithm discovery.arXiv preprint arXiv:2606.06473,

    Shangheng Du, Xiangchao Yan, Jinxin Shi, Zongsheng Cao, Shiyang Feng, Zichen Liang, Boyuan Sun, Tianshuo Peng, Yifan Zhou, Xin Li, et al. MLEvolve: A self-evolving framework for automated machine learning algorithm discovery.arXiv preprint arXiv:2606.06473,

    Pith/arXiv arXiv

  8. [8]

    Collider-bench: Benchmarking AI agents with particle physics analysis reproduction.arXiv preprint arXiv:2605.13950,

    14 Darius A Faroughy, Sofia Palacios Schweitzer, Ian Pang, Siddharth Mishra-Sharma, and David Shih. Collider-bench: Benchmarking AI agents with particle physics analysis reproduction.arXiv preprint arXiv:2605.13950,

    Pith/arXiv arXiv

  9. [9]

    URL https://arxiv.org/abs/2605.13950

    doi: 10.48550/arXiv.2605.13950. URL https://arxiv.org/abs/2605.13950. Dyke Ferber, Lars Hilgers, Christiane Höper, Benedict Kinny-Köster, Jan-Niklas Eckardt, Katharina Egger-Heidrich, Marius Bill, Martin MK Schneider, Jan Clusmann, Lejla Kadric, et al. Towards autonomous medical artificial intelligence agents.Nature, pages 1–10,

    Pith/arXiv arXiv doi:10.48550/arxiv.2605.13950

  10. [10]

    Phikon-v2, a large and public feature extractor for biomarker prediction.arXiv preprint arXiv:2409.09173,

    Alexandre Filiot, Paul Jacob, Alice Mac Kain, and Charlie Saillard. Phikon-v2, a large and public feature extractor for biomarker prediction.arXiv preprint arXiv:2409.09173,

    Pith/arXiv arXiv

  11. [11]

    Autoscientists: Self-organizing agent teams for long-running scientific experimenta- tion.arXiv preprint arXiv:2605.28655, 2026a

    Shanghua Gao, Ada Fang, and Marinka Zitnik. Autoscientists: Self-organizing agent teams for long-running scientific experimenta- tion.arXiv preprint arXiv:2605.28655, 2026a. Yifan Gao, Haoyue Li, Feng Yuan, Xin Gao, Weiran Huang, and Xiaosong Wang. Camyla: Scaling autonomous research in medical image segmentation.arXiv preprint arXiv:2604.10696, 2026b. Ro...

    Pith/arXiv arXiv

  12. [12]

    Towards an AI co-scientist.arXiv preprint arXiv:2502.18864,

    Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, Tao Tu, Anil Palepu, Petar Sirkovic, Artiom Myaskovsky, Felix Weissenberger, Keran Rong, Ryutaro Tanno, et al. Towards an AI co-scientist.arXiv preprint arXiv:2502.18864,

    Pith/arXiv arXiv

  13. [13]

    Segment anything for histopathology.arXiv preprint arXiv:2502.00408,

    Titus Griebel, Anwai Archit, and Constantin Pape. Segment anything for histopathology.arXiv preprint arXiv:2502.00408,

    Pith/arXiv arXiv

  14. [14]

    Ultrasound image enhancement challenge 2023, April

    Yi Guo, Shichong Zhou, Jun Shi, and Yuanyuan Wang. Ultrasound image enhancement challenge 2023, April

    2023

  15. [15]

    Ibrahim Ethem Hamamci, Sezgin Er, Enis Simsar, Atif Emre Yuksel, Sadullah Gultekin, Serife Damla Ozdemir, Kaiyuan Yang, Hongwei Bran Li, Sarthak Pati, Bernd Stadlinger, et al

    URL https: //doi.org/10.5281/zenodo.7841250. Ibrahim Ethem Hamamci, Sezgin Er, Enis Simsar, Atif Emre Yuksel, Sadullah Gultekin, Serife Damla Ozdemir, Kaiyuan Yang, Hongwei Bran Li, Sarthak Pati, Bernd Stadlinger, et al. Dentex: An abnormal tooth detection with dental enumeration and diagnosis benchmark for panoramic x-rays.arXiv preprint arXiv:2305.19112,

    doi:10.5281/zenodo.7841250

  16. [16]

    Towards a virtual neuroscientist: Autonomous neuroimaging analysis via multi-agent collaboration.arXiv preprint arXiv:2605.09366,

    Keqi Han, Songlin Zhao, Yao Su, Xiang Li, Yixuan Yuan, Lifang He, and Carl Yang. Towards a virtual neuroscientist: Autonomous neuroimaging analysis via multi-agent collaboration.arXiv preprint arXiv:2605.09366,

    Pith/arXiv arXiv

  17. [17]

    Isles 2022: A multi-center magnetic resonance imaging stroke lesion segmentation dataset.Scientific data, 9(1):762,

    Moritz R Hernandez Petzsche, Ezequiel De La Rosa, Uta Hanning, Roland Wiest, Waldo Valenzuela, Mauricio Reyes, Maria Meyer, Sook-Lei Liew, Florian Kofler, Ivan Ezhov, et al. Isles 2022: A multi-center magnetic resonance imaging stroke lesion segmentation dataset.Scientific data, 9(1):762,

    2022

  18. [18]

    REPRO-bench: Can agentic AI systems assess the reproducibility of social science research?arXiv preprint arXiv:2507.18901,

    Chuxuan Hu, Liyun Zhang, Yeji Lim, Aum Wadhwani, Austin Peters, and Daniel Kang. REPRO-bench: Can agentic AI systems assess the reproducibility of social science research?arXiv preprint arXiv:2507.18901,

    Pith/arXiv arXiv

  19. [19]

    URL https://arxiv.org/abs/2507.18901

    doi: 10.48550/arXiv.2507.18901. URL https://arxiv.org/abs/2507.18901. Kexin Huang, Serena Zhang, Hanchen Wang, Yuanhao Qu, Yingzhou Lu, Ryan Li, Yusuf Roohani, Lin Qiu, Shiyi Cao, Gavin Li, Junze Zhang, Di Yin, Rick Wierenga, Deniz Kavi, Sherry Liu, Tianwei She, Shruti Marwaha, Jennefer N. Carter, Xin Zhou, Matthew T. Wheeler, Jonathan A. Bernstein, Mengd...

    Pith/arXiv arXiv doi:10.48550/arxiv.2507.18901

  20. [20]

    Can coding agents reproduce findings in computational materials science?arXiv preprint arXiv:2605.00803, 2026b

    Ziyang Huang, Yi Cao, Ali K Shargh, Jing Luo, Ruidong Mei, Mohd Zaki, Zhan Liu, Wyatt Bunstine, William Jurayj, Somdatta Goswami, Tyrel McQueen, Michael Shields, Jaafar El-Awady, Paulette Clancy, Benjamin Van Durme, Nicholas Andrews, William Walden, and Daniel Khashabi. Can coding agents reproduce findings in computational materials science?arXiv preprint...

    Pith/arXiv arXiv doi:10.48550/arxiv.2605.00803

  21. [21]

    CodeScientist: End-to-end semi-automated scientific discovery with code-based experimentation

    Peter Jansen, Oyvind Tafjord, Marissa Radensky, Pao Siangliulue, Tom Hope, Bhavana Dalvi, Bodhisattwa Prasad Majumder, Daniel S Weld, and Peter Clark. CodeScientist: End-to-end semi-automated scientific discovery with code-based experimentation. InFindings of the Association for Computational Linguistics: ACL 2025, pages 13370–13467,

    2025

  22. [23]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Peri, Ofir Press, and Karthik Narasimhan

    URL https://arxiv.org/abs/2502.13138. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Peri, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues?Proceedings of the International Conference on Learning Representations (ICLR),

    Pith/arXiv arXiv

  23. [24]

    Toward generalist autonomous research via hypothesis-tree refinement.arXiv preprint arXiv:2606.11926,

    Jiajie Jin, Yuyang Hu, Kai Qiu, Qi Dai, Chong Luo, Guanting Dong, Xiaoxi Li, Tong Zhao, Xiaolong Ma, Gongrui Zhang, et al. Toward generalist autonomous research via hypothesis-tree refinement.arXiv preprint arXiv:2606.11926,

    Pith/arXiv arXiv

  24. [25]

    Towards the automatic segmentation, modeling and meshing of the aortic vessel tree from multicenter acquisitions: An overview of the seg

    Yuan Jin, Antonio Pepe, Gian Marco Melito, Yuxuan Chen, Yunsu Byeon, Hyeseong Kim, Kyungwon Kim, Doohyun Park, Euijoon Choi, Dosik Hwang, et al. Towards the automatic segmentation, modeling and meshing of the aortic vessel tree from multicenter acquisitions: An overview of the seg. a. 2023 segmentation of the aorta challenge.arXiv preprint arXiv:2510.24009,

    arXiv 2023

  25. [26]

    Rex-mle: The autonomous agent benchmark for medical imaging challenges

    Roshan Kenia, Xiaoman Zhang, and Pranav Rajpurkar. Rex-mle: The autonomous agent benchmark for medical imaging challenges. arXiv preprint arXiv:2512.17838,

    arXiv

  26. [27]

    From reproduction to replication: Evaluating research agents with progressive code masking.arXiv preprint arXiv:2506.19724,

    Gyeongwon James Kim, Alex Wilf, Louis-Philippe Morency, and Daniel Fried. From reproduction to replication: Evaluating research agents with progressive code masking.arXiv preprint arXiv:2506.19724,

    Pith/arXiv arXiv

  27. [28]

    URL https://arxiv.org/abs/2506.19724

    doi: 10.48550/arXiv.2506.19724. URL https://arxiv.org/abs/2506.19724. Wonkyeong Lee, Fabian Wagner, Adrian Galdran, Yongyi Shi, Wenjun Xia, Ge Wang, Xuanqin Mou, Md Atik Ahamed, Abdullah Al Zubaer Imran, Ji Eun Oh, et al. Low-dose computed tomography perceptual image quality assessment.Medical Image Analysis, 99:103343,

    Pith/arXiv arXiv doi:10.48550/arxiv.2506.19724

  28. [29]

    Autosota: An end-to-end automated research system for state-of-the-art ai model discovery.arXiv preprint arXiv:2604.05550,

    Yu Li, Chenyang Shao, Xinyang Liu, Ruotong Zhao, Peijie Liu, Hongyuan Su, Zhibin Chen, Qinglong Yang, Anjie Xu, Yi Fang, et al. Autosota: An end-to-end automated research system for state-of-the-art ai model discovery.arXiv preprint arXiv:2604.05550,

    Pith/arXiv arXiv

  29. [30]

    Autoresearchclaw: Self-reinforcing autonomous research with human-ai collaboration.arXiv preprint arXiv:2605.20025, 2026a

    Jiaqi Liu, Shi Qiu, Mairui Li, Bingzhou Li, Haonian Ji, Siwei Han, Xinyu Ye, Peng Xia, Zihan Dong, Congyu Zhang, et al. Autoresearchclaw: Self-reinforcing autonomous research with human-ai collaboration.arXiv preprint arXiv:2605.20025, 2026a. Junqi Liu, Salena Song, Yuhan Wang, Jiawei Mao, Hardy Chen, Xiaoke Huang, Tianhao Qi, Pengfei Guo, Yucheng Tang, Y...

    Pith/arXiv arXiv

  30. [31]

    The AI scientist: Towards fully automated open-ended scientific discovery.arXiv preprint arXiv:2408.06292,

    Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The AI scientist: Towards fully automated open-ended scientific discovery.arXiv preprint arXiv:2408.06292,

    Pith/arXiv arXiv

  31. [32]

    MLS-bench: A holistic and rigorous assessment of AI systems on building better AI.arXiv preprint arXiv:2605.08678,

    Bohan Lyu, Yucheng Yang, Siqiao Huang, Jiaru Zhang, Qixin Xu, Xinghan Li, Xinyang Han, Yicheng Zhang, Huaqing Zhang, Runhan Huang, Kaicheng Yang, Zitao Chen, Wentao Guo, Junlin Yang, Xinyue Ai, Wenhao Chai, Yadi Cao, Ziran Yang, Kun Wang, Dapeng Jiang, Huan-ang Gao, Shange Tang, Chengshuai Shi, Simon S Du, Max Simchowitz, Jiantao Jiao, Dawn Song, and Chi ...

    Pith/arXiv arXiv

  32. [33]

    URL https://arxiv.org/abs/2605.08678

    doi: 10.48550/arXiv.2605.08678. URL https://arxiv.org/abs/2605.08678. 16 Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, and Bo Wang. Segment anything in medical images.Nature Communications, 15(1):654, 2024a. doi: 10.1038/s41467-024-44824-z. Jun Ma, Ronald Xie, Shamini Ayyadhury, Cheng Ge, Anubha Gupta, Ritu Gupta, Song Gu, Yao Zhang, Gihun Lee, Joonk...

    Pith/arXiv arXiv doi:10.48550/arxiv.2605.08678

  33. [34]

    ScientistOne: Towards human-level autonomous research via chain-of-evidence.arXiv preprint arXiv:2605.26340,

    Rui Meng, Bhavana Dalvi Mishra, Jiefeng Chen, Chun-Liang Li, Palash Goyal, Mihir Parmar, Yiwen Song, Yale Song, Rajarishi Sinha, Parthasarathy Ranganathan, et al. ScientistOne: Towards human-level autonomous research via chain-of-evidence.arXiv preprint arXiv:2605.26340,

    Pith/arXiv arXiv

  34. [35]

    Cellpose-sam: superhuman generalization for cellular segmentation

    Marius Pachitariu, Michael Rariden, and Carsen Stringer. Cellpose-sam: superhuman generalization for cellular segmentation. bioRxiv, pages 2025–04,

    2025

  35. [36]

    PiFlow: Principle-aware scientific discovery with multi-agent collaboration.arXiv preprint arXiv:2505.15047,

    Yingming Pu, Tao Lin, and Hongyu Chen. PiFlow: Principle-aware scientific discovery with multi-agent collaboration.arXiv preprint arXiv:2505.15047,

    arXiv

  36. [37]

    Coral: Towards autonomous multi-agent evolution for open-ended discovery.arXiv preprint arXiv:2604.01658,

    Ao Qu, Han Zheng, Zijian Zhou, Yihao Yan, Yihong Tang, Shao Yong Ong, Fenglu Hong, Kaichen Zhou, Chonghe Jiang, Minwei Kong, et al. Coral: Towards autonomous multi-agent evolution for open-ended discovery.arXiv preprint arXiv:2604.01658,

    Pith/arXiv arXiv

  37. [38]

    PostTrainBench: Can LLM agents automate LLM post-training?arXiv preprint arXiv:2603.08640,

    Ben Rank, Hardik Bhatnagar, Ameya Prabhu, Shira Eisenberg, Karina Nguyen, Matthias Bethge, and Maksym Andriushchenko. PostTrainBench: Can LLM agents automate LLM post-training?arXiv preprint arXiv:2603.08640,

    arXiv

  38. [39]

    doi: 10.48550/arXiv.2603. 08640. URL https://arxiv.org/abs/2603.08640. Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 234–241. Springer,

    doi:10.48550/arxiv.2603

  39. [40]

    Agent laboratory: Using LLM agents as research assistants.Findings of the Association for Computational Linguistics: EMNLP 2025, pages 5977–6043,

    Samuel Schmidgall, Yusheng Su, Ze Wang, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Michael Moor, Zicheng Liu, and Emad Barsoum. Agent laboratory: Using LLM agents as research assistants.Findings of the Association for Computational Linguistics: EMNLP 2025, pages 5977–6043,

    2025

  40. [41]

    CORE-bench: Fostering the credibility of published research through a computational reproducibility agent benchmark.arXiv preprint arXiv:2409.11363,

    Zachary S Siegel, Sayash Kapoor, Nitya Nadgir, Benedikt Stroebl, and Arvind Narayanan. CORE-bench: Fostering the credibility of published research through a computational reproducibility agent benchmark.arXiv preprint arXiv:2409.11363,

    Pith/arXiv arXiv

  41. [42]

    URL https://arxiv.org/abs/2409.11363

    doi: 10.48550/arXiv.2409.11363. URL https://arxiv.org/abs/2409.11363. Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Patwardhan. PaperBench: Evaluating AI’s ability to replicate AI research.arXiv preprint arXiv:2504.01848,

    Pith/arXiv arXiv doi:10.48550/arxiv.2409.11363

  42. [43]

    URL https://arxiv.org/abs/2504.01848

    doi: 10.48550/arXiv.2504.01848. URL https://arxiv.org/abs/2504.01848. Jiabin Tang, Lianghao Xia, Zhonghang Li, and Chao Huang. AI-Researcher: Autonomous scientific innovation.arXiv preprint arXiv:2505.18705,

    Pith/arXiv arXiv doi:10.48550/arxiv.2504.01848

  43. [44]

    Towards generalist biomedical AI.arXiv preprint arXiv:2307.14334,

    Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schaekermann, Mohamed Amin, Pi-Chuan Chang, Andrew Carroll, Chuck Lau, Ryutaro Tanno, Ira Ktena, et al. Towards generalist biomedical AI.arXiv preprint arXiv:2307.14334,

    Pith/arXiv arXiv

  44. [45]

    Neuroclaw technical report.arXiv preprint arXiv:2604.24696, 2026a

    Cheng Wang, Zhibin He, Zhihao Peng, Shengyuan Liu, Yufan Hu, Yang Carl, He Lifang, Lichao Sun, Xiang Li, and Yixuan Yuan. Neuroclaw technical report.arXiv preprint arXiv:2604.24696, 2026a. Yubo Wang, Zhoujun Zhuang, Anirudh Srinivasa, Pradeep Dasigi, and Hannaneh Hajishirzi. SciAgent: Tool-augmented language models for scientific reasoning.arXiv preprint ...

    Pith/arXiv arXiv

  45. [46]

    FIRE-bench: Evaluating agents on the rediscovery of scientific insights.arXiv preprint arXiv:2602.02905, 2026b

    17 Zhen Wang, Fan Bai, Zhongyan Luo, Jinyan Su, Kaiser Sun, Xinle Yu, Jieyuan Liu, Kun Zhou, Claire Cardie, Mark Dredze, Eric P Xing, and Zhiting Hu. FIRE-bench: Evaluating agents on the rediscovery of scientific insights.arXiv preprint arXiv:2602.02905, 2026b. doi: 10.48550/arXiv.2602.02905. URL https://arxiv.org/abs/2602.02905. Jakob Wasserthal, Hanns-C...

    Pith/arXiv arXiv doi:10.48550/arxiv.2602.02905

  46. [47]

    Deepscientist: Advancing frontier-pushing scientific findings progressively.arXiv preprint arXiv:2509.26603,

    Yixuan Weng, Minjun Zhu, Qiujie Xie, Qiyao Sun, Zhen Lin, Sifan Liu, and Yue Zhang. Deepscientist: Advancing frontier-pushing scientific findings progressively.arXiv preprint arXiv:2509.26603,

    arXiv

  47. [48]

    Towards a medical AI scientist.arXiv preprint arXiv:2603.28589,

    Hongtao Wu, Boyun Zheng, Dingjie Song, Yu Jiang, Jianfeng Gao, Lei Xing, Lichao Sun, and Yixuan Yuan. Towards a medical AI scientist.arXiv preprint arXiv:2603.28589,

    arXiv

  48. [49]

    The AI scientist-v2: Workshop-level automated scientific discovery via agentic tree search.arXiv preprint arXiv:2504.08066,

    Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Foerster, Jeff Clune, and David Ha. The AI scientist-v2: Workshop-level automated scientific discovery via agentic tree search.arXiv preprint arXiv:2504.08066,

    Pith/arXiv arXiv

  49. [50]

    Benchmarking the cow with the topcow challenge: Topology-aware anatomical segmentation of the circle of willis for cta and mra.ArXiv, pages arXiv–2312, 2025a

    Kaiyuan Yang, Fabio Musio, Yihui Ma, Norman Juchler, Johannes C Paetzold, Rami Al-Maskari, Luciano Höher, Hongwei Bran Li, Ibrahim Ethem Hamamci, Anjany Sekuboyina, et al. Benchmarking the cow with the topcow challenge: Topology-aware anatomical segmentation of the circle of willis for cta and mra.ArXiv, pages arXiv–2312, 2025a. Ruofeng Yang, Yongcan Li, ...

    Pith/arXiv arXiv

  50. [51]

    R&D-Agent: An llm-agent framework towards autonomous data science.arXiv preprint arXiv:2505.14738, 2025b

    Xu Yang, Xiao Yang, Shikai Fang, Yifei Zhang, Jian Wang, Bowen Xian, Qizheng Li, Jingyuan Li, Minrui Xu, Yuante Li, et al. R&D-Agent: An llm-agent framework towards autonomous data science.arXiv preprint arXiv:2505.14738, 2025b. Yuzhe Yang, Haoran Zhang, Judy W Gichoya, Dina Katabi, and Marzyeh Ghassemi. The limits of fair medical imaging AI in real-world...

    arXiv

  51. [52]

    ReplicationBench: Can AI agents replicate astrophysics research papers?arXiv preprint arXiv:2510.24591,

    Christine Ye, Sihan Yuan, Suchetha Cooray, Steven Dillmann, Ian L V Roque, Dalya Baron, Philipp Frank, Sergio Martin-Alvarez, Nolan Koblischke, Frank J Qu, Diyi Yang, Risa Wechsler, and Ioana Ciuca. ReplicationBench: Can AI agents replicate astrophysics research papers?arXiv preprint arXiv:2510.24591,

    arXiv

  52. [53]

    URL https://arxiv.org/abs/2510.24591

    doi: 10.48550/arXiv.2510.24591. URL https://arxiv.org/abs/2510.24591. Kai Zhang, Rong Zhou, Eashan Adhikarla, Zhiling Yan, Yixin Liu, Jun Yu, Zhengliang Liu, Xun Chen, Brian D Davison, Hui Ren, et al. BiomedGPT: A generalist vision-language foundation model for diverse biomedical tasks.Nature Medicine,

    doi:10.48550/arxiv.2510.24591

  53. [54]

    Ruiyi Zhang, Peijia Qin, Qi Cao, Li Zhang, and Pengtao Xie

    doi: 10.1038/s41591-024-03185-2. Ruiyi Zhang, Peijia Qin, Qi Cao, Li Zhang, and Pengtao Xie. AIBuildAI: An AI agent for automatically building AI models.arXiv preprint arXiv:2604.14455,

    Pith/arXiv arXiv doi:10.1038/s41591-024-03185-2

  54. [55]

    From clinical intent to clinical model: An autonomous coding-agent framework for clinician-driven ai development.arXiv preprint arXiv:2604.17110,

    Zihao Zhao, Frederik Hauke, Juliana De Castilhos, Jakob Nikolas Kather, Sven Nebelung, and Daniel Truhn. From clinical intent to clinical model: An autonomous coding-agent framework for clinician-driven ai development.arXiv preprint arXiv:2604.17110,

    Pith/arXiv arXiv