Pith. sign in

REVIEW 5 major objections 5 minor 108 references

DoctorAgents: an agentic framework to iteratively refine AutoML pipeline for small clinical temporal data

T0 review · 5 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read DoctorAgents claims that automated machine learning for small, irregular clinical time series should be treated as reasoning-driven refinement: a team of LLM agents that evaluate, remember, and locally edit pipelines outperforms…

desk verdict A genuinely transparent agentic AutoML system whose headline performance claim is not statistically supported — worth reviewing, but don't take Table 1 at face value. read the letter →

arxiv 2608.05375 v1 pith:ZLP4Y35I submitted 2026-08-05 cs.AI cs.LGcs.MA

Ruilin Wang , Bo-Hong Wang , Elizabeth Kourbatski , Jun Bai , Hegang Chen , Ziyang Song , Gilles Boire , Marie Hudson
show 1 more author
Yue Li
This is my paper · ORCID
classification cs.AIcs.LGcs.MA
keywords agenticAIautomatedmachinelearningsmallclinicaldatatemporallargelanguagemodelstextualgradientdescentfeatureengineeringprediction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that reliable ML pipelines for small, irregular clinical datasets are better produced by reasoning-driven refinement than by exhaustive AutoML search. It presents DoctorAgents, a team of LLM agents that initialize, execute, evaluate, remember, and locally refine a pipeline, converting natural-language evaluator feedback into targeted code updates. The domain-specialized DoctorAgents-DS splits this into a preprocessing agent loop and a model-development agent loop, letting the system compose interpretable temporal features from raw irregular records. Benchmarks on MIMIC-IV ICU mortality, readmission, and length-of-stay plus a rheumatoid arthritis cohort show DoctorAgents-DS with the strongest mean AUPRC/AUROC among all compared methods. The practical payoff, if the claim is right, is that small clinical cohorts can get end-to-end, inspectable ML pipelines with far less manual feature engineering.

What carries the argument

The load-bearing mechanism is adapted Textual Gradient Descent, an LLM-based analog of backpropagation that turns evaluator comments into localized code edits: the Evaluator writes a natural-language assessment $E$ of the current pipeline $P^*$, the framework derives a language feedback signal $F = \nabla_{P^*}(E)$, and the Optimizer applies $P = A_o(P^*, F)$ as targeted edits that preserve previously successful components. The second central object is the Logger's memory log $L$: inter-iteration notes tie code differences to performance differences, checkpoint summarization keeps the context bounded, and the history prevents repeated failures. In the domain-specialized variant, the key device is the split into a Data Preprocessing agent pair and a Model Development agent pair: the preprocessing loop searches over feature composition and temporal representations, while the modeling loop searches over model families and hyperparameters, with SHAP values linking composed features back to clinical meaning.

What would settle it

Inject a known leakage trap into the raw data, for instance a feature that records a lab value measured after ICU discharge for the readmission task, and check whether the final pipeline code still uses it; if the generated preprocessing survives the trap, the central safety claim fails. A quantitative variant is to re-run all four benchmarks over many seeds and patient splits; if DoctorAgents-DS's edge over ERA reverses on any task, the headline performance claim is unstable.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is that AutoML for small clinical data can be recast as a stateful, reasoning-driven loop rather than a search over a fixed space. DoctorAgents iterates through specialized agents: a Consultant, a Generator, a Repairer, a Logger, an Evaluator, and an Optimizer. The Evaluator produces natural-language feedback; the Optimizer backpropagates that feedback into small, targeted code changes; the Logger carries a summarized memory of what was tried and what happened. DoctorAgents-DS then specializes this loop into separate preprocessing and model-development subtasks, so the preprocessing agent can invent task-specific temporal features (delta-per-hour changes, last-in-window values, measurement counts, recency) before the model agent fits classifiers. The paper reports that this design achieves the highest mean meta-test performance across all four benchmarks and a significantly higher proportion of high-quality features than ERA and AutoML-Agent.

Load-bearing premise

The load-bearing premise is that the LLM agents will build temporally valid, leakage-safe features: every transformation must be fit on the training split only and must never use information unavailable at prediction time, with the Evaluator expected to catch violations such as post-discharge data in readmission or future lab values in mortality.

Editorial extensions

If this is right

  • Small clinical cohorts can get end-to-end ML pipelines with less manual feature engineering: the framework discovers task-specific temporal representations, such as rates of change, recency, and monitoring intensity, rather than relying on a fixed template.
  • The ablation shows both the Consultant's initial task specification and the Logger's persistent memory are needed for stable gains; removing the Consultant lowers the starting point and removing Memory makes validation performance oscillate.
  • Because the final pipeline is executable code with named features and SHAP rankings, the results carry an audit trail that clinicians can inspect before deployment.
  • The framework's per-iteration model and preprocessing choices make the optimization process visible, so users can see exactly when a model-family switch or preprocessing change improves validation performance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the same reasoning-driven loop may transfer to non-clinical small temporal datasets, but the paper only benchmarks clinical data, so transfer is an open hypothesis.
  • Editorial extension: the interpretability claim rests on an LLM judge; a clinician panel rating the same features would be the decisive external test.
  • Editorial extension: the framework's advantage may partly come from the backbone LLM's priors; running the same agents with a smaller open model would separate the framework's structure from model capability.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces DoctorAgents, a multi-agent LLM framework that iteratively constructs and refines AutoML pipelines for small clinical temporal datasets. It consists of specialized agents (Consultant, Generator, Repairer, Logger, Evaluator, Optimizer) that initialize, execute, validate, memorize, and refine pipelines via natural-language feedback, called textual gradient descent. A domain-specialized variant, DoctorAgents-DS, decomposes pipeline construction into dedicated data-preprocessing and model-development agents. The authors evaluate on three MIMIC-IV tasks (ICU mortality, readmission, length-of-stay) and one in-house RA treatment-response cohort, comparing against TabPFN, three LLM baselines, AutoML-Agent, and ERA. They report that DoctorAgents-DS achieves the highest mean AUPRC/AUROC on all four tasks (Table 1), that feature-quality assessments by GPT-5.4 favor DoctorAgents-DS, and that an ablation on readmission supports the value of the Consultant and Memory components.

Significance. If the empirical claims hold, the paper makes a useful engineering contribution: it demonstrates a reasoning-driven, memory-augmented AutoML approach that can construct interpretable, task-specific temporal features without exhaustive search, and the appendix provides unusually transparent intermediate outputs showing how the framework detects and corrects its own preprocessing failures. The design is plausible and the evaluation protocol (meta-train/meta-test, train-only preprocessing, leakage-aware prompts) is generally careful. However, the central claim of consistent outperformance currently rests on small mean differences without significance testing, the feature-quality evidence relies on an LLM judge from the same model family as the generator, and code/seeds are not provided. These gaps make the empirical superiority claim not yet established, although they are addressable in revision.

major comments (5)
  1. [§5.2, Table 1] The headline claim that DoctorAgents 'consistently outperforms established AutoML baselines' is not statistically supported by the reported results. In Table 1, the margins over the strongest baseline are small relative to the fold-level standard deviations (e.g., mortality AUPRC 0.520±0.076 vs TabPFN 0.509±0.034; readmission 0.212±0.047 vs ERA 0.192±0.044; LOS 0.704±0.019 vs ERA 0.692±0.024; RA AUROC 0.577±0.088 vs ERA 0.564±0.121). With only five folds, these differences could easily arise from fold-level noise. The only significance tests in the paper (Sec. 6.2) apply to LLM-judged feature quality, not to the primary performance comparisons. The authors should report paired significance tests (e.g., Wilcoxon signed-rank or paired bootstrap) on the per-fold metric differences, or at minimum report per-fold paired differences and confidence intervals, and should temper the 'consistently outperforms' claim accordingly.
  2. [Abstract vs Table 1] The abstract states that 'DoctorAgents consistently outperforms established AutoML baselines,' but Table 1 shows that only the domain-specialized variant DoctorAgents-DS achieves the highest mean on all tasks; the non-DS DoctorAgents is worse than ERA on readmission (0.190±0.017 vs 0.192±0.044) and on RA treatment response (0.505±0.060 vs 0.564±0.121). The abstract and introduction should be qualified to refer to DoctorAgents-DS, and even then the overperformance is not statistically confirmed (see the previous comment).
  3. [§6.2, Feature-quality evaluation] The feature-reasonability evaluation uses GPT-5.4 as a judge, which belongs to the same model family as the GPT-5 backbone used to generate the pipelines. No human-expert validation, inter-annotator agreement, or calibration against a labeled set of features is reported. The Wilcoxon tests (p = 4.61×10^-8 and 8.00×10^-11) therefore demonstrate that the LLM judge rates DoctorAgents-DS features more highly, but this does not establish that the features are clinically or methodologically superior in a way that is independent of the generator's own prior. The authors should either add a human-clinician evaluation of a sample of features or demonstrate that the judge's ratings correlate with an external criterion (e.g., literature-based feature validity).
  4. [Appendix A.6.2–A.6.5 vs Section 4.1/A.5.1] There is a factual inconsistency in the description of the RA dataset. Section 4.1 and A.5.1 report 105 patients and 82 features, but the appendix's DP/MD evaluator outputs state '55/58 train patients' and 'p >> n regime (371 features vs 58 patients)' (A.6.2, A.6.4). This suggests the appendix example may come from a different data split, a different feature-engineering state, or an earlier version of the dataset. Because the appendix is used to illustrate the framework's self-correction behavior, the authors should reconcile these numbers and specify exactly which split and optimization step the example corresponds to.
  5. [Reproducibility] The paper does not release code, seeds, or detailed LLM inference settings (e.g., temperature, sampling, prompt versions), and the results depend on stochastic LLM generation and 5-fold cross-validation. The reported means in Table 1 therefore cannot be independently verified or reproduced. Given the central claim rests on small performance margins, the authors should release the implementation and all seeds, or at minimum provide detailed run logs and patch-level diffs of the derived pipelines for each fold.
minor comments (5)
  1. [Section 3.7] The abbreviation 'DS' for domain-specialized is used in the abstract and introduction but is not defined until Section 3.7; please define it at first use.
  2. [Figure 3] The caption says 'validation and internal test performance over iteration,' but the plot appears to show only one trajectory; please clearly label which curve corresponds to validation and which to test, or clarify if only validation is shown.
  3. [Section 3.3] The sentence 'A𝑔 is encouraged to follow the suggestion by A𝑐, but it is not constrained to a fixed template' contains a minor grammar issue ('the suggestion by A𝑐' should be 'the suggestions by A𝑐').
  4. [Section 6.1 and Table A1] The feature names in the text (e.g., 'bicarbonate_blood_delta_per_hr') use underscores, while the table uses the same format; please ensure the names in the beeswarm plots exactly match those in Table A1 to avoid ambiguity.
  5. [Section 5.3] The ablation study is conducted on a single dataset (ICU readmission) and for only ten iterations; the authors should state explicitly in the discussion that the generality of the ablation conclusions across other tasks is unverified.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical systems benchmark, not a derivation whose outputs are built from its inputs.

full rationale

The paper's claims rest on an implemented multi-agent AutoML framework and on meta-test benchmarking (Table 1, Sec. 4.3/5.2). No predicted quantity is defined in terms of a fitted parameter, and no 'prediction' is a renamed version of its input. The optimizer loop (validation performance -> Evaluator feedback -> code modification) is the intended mechanism, not a circular reduction. The TrajGPT citation [31] is background related work and is not load-bearing for the central performance claim. The GPT-5.4 feature-reasonability judge (Sec. 6.2) is from the same model family as the generator backbone; the paper itself notes 'the caveat that GPT-5.4 is not perfect,' but this affects the validity/interpretability assessment, not the primary benchmarking claim, and it is not a fitted quantity relabeled as a prediction. The appendix's DP Evaluator discussion of '55/58 train patients have anchor_mode=missing' is an optimization diagnostic, not a circular step. Whether the Table 1 margins are statistically significant is a legitimate evidence concern, but per the review rubric that is a correctness risk, not a circularity finding.

Assumptions & free parameters 8 free parameters · 6 assumptions · 0 invented entities

The central claim rests on many hand-chosen budgets and thresholds, plus background assumptions about LLM agent reliability, TextGrad, SHAP, and an LLM-based judge. No new physical entities are introduced.

free parameters (8)
  • max_optimization_steps = 20 (DoctorAgents, ERA), 15 (DoctorAgents-DS)
    Budget for iterative refinement; chosen by authors, not data-fitted. Different budgets across methods could affect comparisons, though DS gets fewer steps and still wins.
  • repair_retry_limit = 5
    Maximum repair attempts in Algorithm 2; chosen by hand.
  • memory_checkpoint_thresholds = every 15 iterations or 5000 tokens
    Triggers summarization of the memory log; arbitrary settings that affect what the system remembers.
  • feature_missingness_pruning_threshold = 0.80 to 0.95 (agent-chosen)
    The DP agents drop engineered features with missing rate above these values; thresholds vary across iterations and are not derived from theory.
  • select_kbest_k = 10-30 (MD feedback)
    Univariate feature selection range used by the model development agent; high variance in small samples.
  • shap_subsample_size = 300
    Subsample cap for SHAP computation on validation sets.
  • temporal_window_definitions = 6h, 24h, 48h, 0-90d, 91-365d
    Observation and feature aggregation windows chosen by the agents for each task; affect which features can be constructed.
  • paired_ratio_max_gap_hours = 2.0
    In the LOS pipeline code, SpO2 is paired with a preceding FiO2 measurement within 2 hours; chosen in the generated code.
assumptions (6)
  • domain assumption TextGrad (natural-language feedback backpropagation) improves code generation as described in [43].
    The entire refinement loop (Eq. 1-2) assumes that evaluator feedback converted into localized code edits produces steady improvement rather than random drift.
  • domain assumption GPT-5 and the open LLMs follow complex role prompts reliably enough to generate, repair, and evaluate executable pipelines.
    All agent behavior depends on instruction following of the appendix prompts; there is no formal guarantee.
  • domain assumption SHAP values computed on at most 300 validation samples faithfully rank feature importance.
    Used for interpretation and for the 'more interpretable' claim in Sec 6.1.
  • domain assumption GPT-5.4's reasonability ratings (Invalid through Clinically Reasonable) correspond to actual clinical and mathematical validity.
    Sec 6.2 uses this judge to assert superior feature quality; no human clinician validation is included.
  • domain assumption Measurement frequency and recency in EHR data are predictive without leaking future information when restricted to the observation window.
    The framework composes features like measurement counts and recency hours (Table A1), relying on the assumption that these are valid, leakage-safe signals.
  • domain assumption The 5-fold meta-train/meta-test split with internal test based model selection yields unbiased estimates of test performance.
    Nested evaluation is assumed to prevent optimizer overfitting to the meta-test; no theoretical guarantee is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DoctorAgents: an agentic framework to iteratively refine AutoML pipeline for small clinical temporal data." pith.science (2026). https://pith.science/paper/ZLP4Y35I

@misc{pith2026260805375,
  author       = {Pith},
  title        = {Pith review of: DoctorAgents: an agentic framework to iteratively refine AutoML pipeline for small clinical temporal data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZLP4Y35I}},
  note         = {Machine review of arXiv:2608.05375}
}
read the original abstract

Clinical machine learning (ML) has the potential to support high-stakes medical decision-making, but reliable deployment is often constrained by scarce, heterogeneous, and temporal complexity. Developing effective ML pipelines for such data remains time-consuming and error-prone, while existing automated machine learning (AutoML) systems only partially address this challenge because they largely rely on brute-force search over predefined spaces and lack explicit reasoning and memory. We therefore reformulate AutoML for small clinical data from exhaustive search to reasoning-driven refinement. We propose DoctorAgents, an agentic AI framework that autonomously constructs and optimizes end-to-end ML pipelines through specialized large language model (LLM) agents for generation, validation, and refinement. DoctorAgents backpropagates natural-language feedback through textual gradient descent to perform targeted updates without exhaustive search. Experiments across diverse clinical tasks show that DoctorAgents consistently outperforms established AutoML baselines while producing more interpretable task-specific representations.

Figures

Figures reproduced from arXiv: 2608.05375 by the authors.

Figure 1
Figure 1. Overview of DoctorAgents. (a) Initialization produces a solid starting pipeline from user metadata and profiled data statistics using the provided tool environment. (b) Execution validates the given code and returns performance results. (c) Memory records long-term information across optimization iterations. (d) Refinement iteratively improves the pipeline through evaluation-driven updates. The final output to the u… view at source ↗
Figure 2
Figure 2. DoctorAgents Domain-Specific (DS) Design. Both Generator and Evaluator are specialized into Data Prepro￾cessing (DP) and Model Development (MD) to enable deeper DS exploration and targeted implementation. thereby supporting broader exploratory behavior in the preprocess￾ing domain. The data 𝑅 ∗ 𝑖 are then forwarded to A 𝑚𝑑 𝑔 to synthesize the model-development pipeline 𝑃 ∗ 𝑚𝑑 = A 𝑚𝑑 𝑔 (𝐼, 𝑅∗ 𝑖 ) (Sec. A.4.6). Exe￾cu… view at source ↗
Figure 3
Figure 3. Validation and internal test performance over iteration for [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Ablation study on ICU readmission prediction as an [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Distribution of high-quality feature proportions [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

108 extracted references · 64 canonical work pages

  1. [1]

    Deepak Bhaskar Acharya, Karthigeyan Kuppan, and B. Divya. 2025. Agentic AI: Autonomous Intelligence for Complex Goals—A Comprehensive Survey.IEEE Access13 (2025), 18912–18936. doi:10.1109/ACCESS.2025.3532853

    arXiv 2025

  2. [2]

    Arora, Yu Bai, Bowen Baker, Haiming Bao, et al

    Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, et al. 2025. gpt-oss- 120b & gpt-oss-20b Model Card. arXiv:2508.10925 [cs.CL] doi:10.48550/arXiv. 2508.10925

    arXiv doi:10.48550/arxiv.2508.10925 2025

  3. [3]

    Eser Aygün, Anastasiya Belyaeva, Gheorghe Comanici, Marc Coram, Hao Cui, Jake Garrison, Renee Johnston, Anton Kast, Cory Y McLean, Peter Norgaard, et al. 2026. An AI system to help scientists write expert-level empirical software. Nature654 (2026), 909–916. doi:10.1038/s41586-026-10658-6

    doi:10.1038/s41586-026-10658-6 2026

  4. [4]

    Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Haug, Martin Pawel- czyk, and Gjergji Kasneci. 2022. Deep neural networks and tabular data: A survey. IEEE transactions on neural networks and learning systems35, 6 (2022), 7499–7519

    2022

  5. [5]

    Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al . 2020. Language Models are Few-Shot Learners. InAdvances in Neural Information Processing Systems, Vol. 33. Curran Associates, Inc., Red Hook, NY, USA, 1877–1901

    2020

  6. [6]

    Nathalie Carrier, Javier Marrugo, Misti L Paudel, Sophie Roux, Hugues Allard- Chamard, Artur J de Brum-Fernandes, Patrick Liang, and Gilles Boire. 2026. Longitudinal changes in patient characteristics as drivers of clinical outcomes in the Early Undifferentiated PolyArthritis (EUPA) cohort.Rheumatology (Oxford, England)65, 1 (2026), keaf650

    2026

  7. [7]

    Nathalie Carrier, Sophie Roux, Ariel Masetto, Artur J de Brum-Fernandes, Patrick Liang, Meryem Maoui, and Gilles Boire. 2025. Changes at presentation in patients with early rheumatoid arthritis: a 24-year study of the early undifferentiated polyarthritis (EUPA) cohort.The Journal of Rheumatology52, 2 (2025), 119–127

    2025

  8. [8]

    Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. 2018. Recurrent neural networks for multivariate time series with missing values.Scientific reports8, 1 (2018), 6085

    2018

Show all 108 references
  1. [9]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brock- man, et al . 2021. Evaluating Large Language Models Trained on Code. arXiv:2107.03374 [cs.LG] doi:10.48550/arXiv.2107.03374

    2021 doi arXiv

  2. [10]

    Gary S Collins, Karel GM Moons, Paula Dhiman, Richard D Riley, Andrew L Beam, Ben Van Calster, Marzyeh Ghassemi, Xiaoxuan Liu, Johannes B Reitsma, Maarten Van Smeden, et al. 2024. TRIPOD+ AI statement: updated guidance for reporting clinical prediction models that use regressi...

    2024 doi

  3. [11]

    Keyan Ding, Jing Yu, Junjie Huang, Yuchen Yang, Qiang Zhang, and Huajun Chen. 2025. Scitoolagent: a knowledge-graph-driven scientific agent for multitool integration.Nature Computational Science5, 10 (2025), 962–972

    2025

  4. [12]

    Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Tobias Springenberg, Manuel Blum, and Frank Hutter. 2015. Efficient and Robust Automated Machine Learning. InAdvances in Neural Information Processing Systems, Vol. 28. Curran Associates, Inc., Red Hook, NY, USA, 2962–2970

    2015

  5. [13]

    Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Karro, and D. Sculley. 2017. Google Vizier: A Service for Black-Box Optimization. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining(Halifax, NS, Canada)(KDD ’...

    2017

  6. [14]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv:2407.21783 doi:10.48550/ arXiv.2407.21783

    2024 doi arXiv

  7. [15]

    Noah Hollmann, Samuel Müller, Lennart Purucker, Arjun Krishnakumar, Max Körfer, Shi Bin Hoo, Robin Tibor Schirrmeister, and Frank Hutter. 2025. Accurate predictions on small data with a tabular foundation model.Nature637, 8045 (2025), 319–326

    2025

  8. [16]

    Max Horn, Michael Moor, Christian Bock, Bastian Rieck, and Karsten Borgwardt

  9. [17]

    Hoos, and Kevin Leyton-Brown

    Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown. 2011. Sequential Model- Based Optimization for General Algorithm Configuration. InProceedings of the International Conference on Learning and Intelligent Optimization (LION) (Lecture Notes in Computer Science, Vol. 6683). S...

    2011

  10. [18]

    Alistair E W Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, Li-Wei H Lehman, Leo A Celi, and Roger G Mark. 2023. MIMIC-IV, a freely accessible electronic health record dataset.Sci. Data10, 1...

    2023

  11. [19]

    Yujia Li et al. 2022. Competition-Level Code Generation with AlphaCode.Science 378, 6624 (2022), 1092–1097

    2022

  12. [20]

    Scott M Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. InAdvances in Neural Information Processing Systems, Vol. 30. Curran Associates, Inc., Red Hook, NY, USA, 4765–4774

    2017

  13. [21]

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al

  14. [22]

    OpenAI. 2023. GPT-4 Technical Report. arXiv:2303.08774 doi:10.48550/arXiv. 2303.08774

    2023 doi arXiv

  15. [23]

    Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. InProceedings of the 36th annual acm symposium on user interface software and technology (UIST ’23). ...

    2023

  16. [24]

    Trang Pham, Truyen Tran, Dinh Phung, and Svetha Venkatesh. 2016. Deepcare: A deep dynamic memory model for predictive medicine. InPacific-Asia conference on knowledge discovery and data mining (Lecture Notes in Computer Science, Vol. 9652). Springer, Cham, Switzerland, 30–41. ...

    2016 doi

  17. [25]

    Richard D Riley, Kym IE Snell, Joie Ensor, Danielle L Burke, Frank E Harrell Jr, Karel GM Moons, and Gary S Collins. 2019. Minimum sample size for developing a multivariable prediction model: PART II-binary and time-to-event outcomes. Statistics in medicine38, 7 (2019), 1276–1296

    2019

  18. [26]

    Timo Schick, Jane Dwivedi-Yu, et al. 2023. Toolformer: Language Models Can Teach Themselves to Use Tools. InAdvances in Neural Information Processing Systems (NeurIPS), Vol. 36. Neural Information Processing Systems Foundation, Inc., New Orleans, LA, USA, 68539–68551. doi:10.5...

    2023 doi

  19. [27]

    Noah Shinn et al. 2023. Reflexion: Language Agents with Verbal Reinforcement Learning. InAdvances in Neural Information Processing Systems (NeurIPS), Vol. 36. Neural Information Processing Systems Foundation, Inc., New Orleans, LA, USA, 8634–8652. doi:10.52202/075280-0377

    2023 doi

  20. [28]

    Satya Narayan Shukla and Benjamin M. Marlin. 2019. Interpolation-Prediction Networks for Irregularly Sampled Time Series. InInternational Conference on Learning Representations. OpenReview.net, New Orleans, LA, USA, 14 pages. arXiv:1909.07782 doi:10.48550/arXiv.1909.07782

    2019 doi arXiv

  21. [29]

    Satya Narayan Shukla and Benjamin M Marlin. 2021. Multi-time attention net- works for irregularly sampled time series. InInternational Conference on Learning Representations. OpenReview.net, Virtual Conference, 15 pages. arXiv:2101.10318 doi:10.48550/arXiv.2101.10318

    2021 doi arXiv

  22. [30]

    Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, et al

  23. [31]

    Ziyang Song, Qincheng Lu, He Zhu, David Buckeridge, and Yue Li. 2026. TrajGPT: Irregular Time-Series Representation Learning of Health Trajectory.IEEE Journal of Biomedical and Health Informatics30, 5 (2026), 3888–3899. doi:10.1109/JBHI. 2025.3620205

    2026

  24. [32]

    Hoos, and Kevin Leyton-Brown

    Chris Thornton, Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown. 2013. Auto-WEKA: Combined Selection and Hyperparameter Optimization of Clas- sification Algorithms. InProceedings of the ACM SIGKDD International Con- ference on Knowledge Discovery and Data Mining(Chicago, ...

    2013

  25. [33]

    Patara Trirat, Wonyong Jeong, and Sung Ju Hwang. 2025. AutoML-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoML. InForty-second Interna- tional Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 267). PMLR, Vancouver, BC, Canada, 60099–60146

    2025

  26. [34]

    Maarten Van Smeden, Karel GM Moons, Joris AH de Groot, Gary S Collins, Douglas G Altman, Marinus JC Eijkemans, and Johannes B Reitsma. 2019. Sample size for binary logistic prediction models: beyond events per variable criteria. Statistical methods in medical research28, 8 (20...

    2019

  27. [35]

    Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. 2024. A survey on large language model based autonomous agents.Frontiers of Computer Science18, 6 (2024), 186345

    2024

  28. [36]

    Shirly Wang, Matthew BA McDermott, Geeticka Chauhan, Marzyeh Ghassemi, Michael C Hughes, and Tristan Naumann. 2020. Mimic-extract: A data extraction, preprocessing, and representation pipeline for mimic-iii. InProceedings of the ACM conference on health, inference, and learnin...

    2020

  29. [37]

    Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. 2024. Executable code actions elicit better llm agents. InForty-first International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 235). PMLR, Vienna, Austria,...

    2024

  30. [38]

    Xuezhi Wang, Jason Wei, Dale Schuurmans, et al. 2023. Self-Consistency Im- proves Chain of Thought Reasoning in Language Models. InInternational Con- ference on Learning Representations. OpenReview.net, Kigali, Rwanda, 24 pages. arXiv:2203.11171

    2023 arXiv

  31. [39]

    Frank Wilcoxon. 1945. Individual comparisons by ranking methods.Biometrics bulletin1, 6 (1945), 80–83. 9

    1945

  32. [40]

    Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. 2025. The rise and potential of large language model based agents: A survey. arXiv 2023.Science China Information Sciences68, 2 (2025), 121101. doi:10.10...

    2025 doi

  33. [41]

    Feng Xie, Han Yuan, Yilin Ning, Marcus Eng Hock Ong, Mengling Feng, Wynne Hsu, Bibhas Chakraborty, and Nan Liu. 2022. Deep learning for temporal data representation in electronic health records: A systematic review of challenges and methodologies.Journal of biomedical informat...

    2022

  34. [42]

    Shunyu Yao et al. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. InInternational Conference on Learning Representations. OpenReview.net, Kigali, Rwanda, 33 pages. arXiv:2210.03629

    2023 arXiv

  35. [43]

    Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Pan Lu, Zhi Huang, Carlos Guestrin, and James Zou. 2025. Optimizing generative AI by backpropa- gating language model feedback.Nature639, 8055 (2025), 609–616

    2025

  36. [44]

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023. LIMA: Less Is More for Alignment. InAdvances in Neural Information Processing Syst...

    2023

  37. [48]

    Do not include any meta commentary, rationale, or explanations

    Output only the curated notes text. Do not include any meta commentary, rationale, or explanations

  38. [49]

    Do NOT introduce, infer, assume, or add any information that is not explicitly present in the input notes

    The output must be derived strictly and exclusively from the provided notes. Do NOT introduce, infer, assume, or add any information that is not explicitly present in the input notes

  39. [50]

    Preserve information that is most useful for future optimiza- tion decisions:

    Preserve concrete facts such as numeric performance changes, important hyperparameters, step indices and etc. Preserve information that is most useful for future optimiza- tion decisions:

  40. [51]

    Largest performance improvements/regressions and the changes that caused them

  41. [52]

    Repeated failure modes

  42. [53]

    High-impact preprocessing/training strategy shifts

  43. [54]

    Evidence of overfitting/leakage/instability

  44. [55]

    Decisions that narrowed or expanded the search space Remove or compress:

  45. [56]

    Minor refactors with no measurable impact

  46. [57]

    System Prompt for Evaluator You are an expert clinical ML engineer and code reviewer acting as an evaluator for an agentic machine learning system

    Redundant observations repeated across iterations A.4.5 Evaluator. System Prompt for Evaluator You are an expert clinical ML engineer and code reviewer acting as an evaluator for an agentic machine learning system. You will be given the task description, task background infor-...

  47. [61]

    The raw datasets have exactly the same feature space

    (Strict) Load raw train set from {raw_train}; raw valida- tion set from {raw_val}; raw test set from {raw_test}. The raw datasets have exactly the same feature space. All data- dependent preprocessing must bebased on the train set only, then applied to the validation and test ...

  48. [62]

    If exists, You must strictly avoid any feature engineering or preprocessing oper- ation that uses, derives from, or is conditioned on the target label

    Label column existence: ({label_col}). If exists, You must strictly avoid any feature engineering or preprocessing oper- ation that uses, derives from, or is conditioned on the target label. Do not create, modify, or select features based on the label distribution, label corre...

  49. [63]

    Keep ID ({id_col})/Label ({la- bel_col})/Time ({time_col}) column names and case unchanged

    (Strict) At the end, MUST save as{csv} files (Replace if exists) the final preprocessed train set to {prep_train_path}; the final preprocessed validation set to {prep_val_path}; the final prepro- cessed test set to {prep_test_path}. Keep ID ({id_col})/Label ({la- bel_col})/Tim...

  50. [64]

    At the end, MUST save a (Replace if exists) concise and decision-orientedmetadata JSON file to {stats_json_path}, intended for downstream LLM-based agents. The metadata JSON should satisfy: - MUST contain a section have_same_feature_space: (Yes or No) Checks whether the prepro...

  51. [65]

    You must output only one executable Python code

  52. [66]

    In-code Requirements:

    Do not include explanations, comments outside code, markdown, or multiple codes. In-code Requirements:

  53. [67]

    Do Not use ’sparse’ as a keyword argument for OneHotEn- coder

  54. [68]

    The datasets have exactly the same feature space

    Load train set from {prep_train_path}; validation set from {prep_val_path}; test set from {prep_test_path}. The datasets have exactly the same feature space. Use the train set for model training. Use the validation set for tuning hyperparameters, evaluating performance, and se...

  55. [69]

    The capitalization of ID ({id_col})/Label ({label_col})/Time ({time_col}) column names in the preprocessed dataset could vary, identify them regardless of case

  56. [70]

    test_performance (a single float value reporting the best model test performance), and (iv) best_model (ONLY hyperparameters and name of the best model)

    (Strict) At the end, MUST save (Replace if exists) a STRUCTUREDjson fileto {training_stats_json_path} which contains at least four sections: (i) train_performance (a single float value reporting the best model training performance), (ii) val_performance (a single float value r...

  57. [71]

    If all models failed, output empty CSV), and save as a csv file to shap_csv_path with two columns:

    At the end, MUST use the trained best model (IMPORTANT: This should NOT influence the selection scheme of best model above) to compute the SHAP values on validation set (if validation set size≥ 300, then use maximum 300 stratified subsamples for computation) for each feature (...

  58. [72]

    abs_mean_shap (the mean value of absolute SHAP), the CSV file should be ordered by descending abs_mean_shap

    feature_name (the name of each feature in the dataset), 2. abs_mean_shap (the mean value of absolute SHAP), the CSV file should be ordered by descending abs_mean_shap. You should also compute a beeswarm plot of the top 10 (if number of features≤ 10, plot all features) importan...

  59. [73]

    Avoid using any ’Random Guessing’ type of model

  60. [74]

    Use the specified metrics to evaluate the performance

  61. [75]

    accuracy

    Make sure your results are reproducible. MD Evaluator. System Prompt for MD Evaluator You are an expert ML engineer acting as a code reviewer and evaluator formodel trainingonly. You will be given the task description, the specified evaluation metrics, the metadata generated f...

  62. [76]

    Accepts ‘train_path‘ and ‘test_path‘ as strings

  63. [77]

    The same preprocessing strategy should not be blindly applied to all features

    Before training machine learning models, convert the data (this dataset is sparse and provided in time-series format, with time column = ’TIME_COL’) into a clinically meaningful tabular representation using appropriate feature preprocessing strategies guided by clinical insigh...

  64. [78]

    Trains a set of classification models to select the best one

  65. [79]

    Returns the probability predictions for the test set as a numpy array or list

  66. [80]

    IMPORTANT: DO NOT use ‘xgboost‘ or ‘lightgbm‘

    You can use pandas, numpy, scikit-learn, or other packages as needed. IMPORTANT: DO NOT use ‘xgboost‘ or ‘lightgbm‘. Your code must look like this: importpandasaspd importnumpyasnp # ... other imports deftrain_and_predict(train_path, test_path): # Load data train = pd.read_csv...

  67. [81]

    DO NOT use GridSearchCV or RandomizedSearchCV

  68. [82]

    If using RandomForest or Boosting, set ‘n_estimators‘ to maximum 50

  69. [83]

    Invalid",

    Keep the model lightweight (execution time limit is 60 sec- onds). Instruction prompt for classifying Feature Reasonability.The fol- lowing provides the prompt details for feature reasonability judg- ment on each task. The contents inBOLDwill change according to current task. ...

  70. [84]

    intended

    Anchoring logic prevents using those clinically meaningful features for most patients, so the “intended” feature set rarely materializes

  71. [85]

    set to NaN)

    Static-only fallback discards meds/comorbidities entirely (Pred_before_TNF, MTX_before_TNF, etc. set to NaN). Those are not time-series in the same sense and are often known at/around start; dropping them when anchor fails is unneces- sary information loss

  72. [86]

    This dataset historically contains ‘un- known” encodings (e.g., -1) (you even saw Mpoc_-1 show up in Step 6 SHAP)

    Binary coercion is too strict: _coerce_binary drops any- thing not exactly 0,1. This dataset historically contains ‘un- known” encodings (e.g., -1) (you even saw Mpoc_-1 show up in Step 6 SHAP). Current handling likely turns informative 25 ‘unknown/NA-coded” states into missin...

  73. [87]

    days-to-start

    Serology quantile bin feature is computed but effectively unused: * _add_serology_bins() adds FR__latest_pre__qbin / an- tiCCP__latest_pre__qbin * but _fit_preprocessor() determines keep_num from train_pat before those columns exist, so they never enter feature_cols. * Net: ex...

  74. [88]

    Rework anchoring to recover pre-treatment rows for most patients (patient-level start date from DReelle + base- line_time_to_DateDeb, then derive days_to_start)

  75. [89]

    Do not drop meds/comorbidities in static_only mode; keep them as baseline static (with missing indicators if needed)

  76. [90]

    Relax binary coercion to preserve “unknown” state (e.g., map -1 to a separate category or a separate indicator)

  77. [91]

    Either remove serology __qbin or include it properly in the kept numeric/categorical set (right now it’s dead code)

  78. [92]

    If you implement only one change: fix anchoring so you’re not static-only for∼95% of patients—that’s the bottleneck cur- rently limiting validation AUROC

    Given extreme sparsity, avoid dropping engineered features solely due to missingness when missingness is expected; in- stead keep a smaller number of high-value engineered fea- tures (baseline/last + measured_any + days_to_start_last) for CRP/ESR/DAS28/HAQ, and drop the rest (...

  79. [93]

    static-only

    Fix the core failure mode: anchoring is eliminating longitu- dinal signal for∼95% of patients Your own QA (patients_anchor_missing=55/58, no_prestart=55/58) implies almost everyone becomes “static-only”, so the pipeline can’t leverage the clinically strongest predictors (DAS28...

  80. [94]

    Change * Always populate meds/comorbidities from the chosen “base” row (or any available row), even if anchor is uncertain

    Remove unnecessary information loss in static_only fallback (currently harms AUROC) When no pre-start rows are found, you set meds and comor- bidities to NaN: if static_only > 0.0: for c in meds_binary + comorb: row[c] = np.nan But these are not truly time-series dependent on ...

  81. [95]

    un- known/other

    Binary coercion is too strict; you’re silently deleting “un- known/other” states _coerce_binary only accepts exact 0,1; everything else becomes NaN and then median-imputed. If the raw data uses -1 or other sentinel codes (seen in prior steps), this collapses informative ‘unkno...

  82. [96]

    So those bins never enter feature_cols

    Your serology quantile bins are dead code (adds complexity, no signal) You create FR__latest_pre__qbin / antiCCP__latest_pre__qbin in _add_serology_bins() inside _transform(), after _fit_preprocessor() decides keep_num. So those bins never enter feature_cols. Change * Either (...

  83. [97]

    everything missing→ everything pruned

    Reconsider missingness pruning for engineered longitudinal features (currently self-defeating under sparsity) Dropping engineered __ * features when missingness> 0.80 is too aggressive given the extreme sparsity—especially when anchoring is fragile. It causes a feedback loop: ...

  84. [98]

    That can happen when clinical disease activity features are missing, but it can also reflect proxy learning that doesn’t validate well

    Reduce dominance of treatment-choice & socioeconomic proxies (improves generalization, often helps val AUROC) Current SHAP shows AntiTNF_or_JAK_First, SE, Scolar dom- inate. That can happen when clinical disease activity features are missing, but it can also reflect proxy lear...

  85. [99]

    usable longitudi- nal coverage

    Add diagnostics that directly optimize for “usable longitudi- nal coverage” To prevent repeating Step7’s failure mode, add metadata coun- ters like: * %patients with≥1 pre-start row * %patients with baseline_gap_days within 0–30/31–90/91–365 * distribution of inferred start_es...

  86. [100]

    Keep broad coverage, but reject anchors that imply impossible timelines

    Add sanity gates back, but softer than Step 7. Keep broad coverage, but reject anchors that imply impossible timelines. * Pseudocode idea: * compute anchor candidates; choose one only if it yields a reasonable fraction of visits with days_to_start≥ 0 and within a max horizon (...

  87. [101]

    always missing

    Stop emitting “always missing” engineered features. * If n_measured<2 in a window, either: * don’t create std at all, or * set std = 0 and rely on n_measured to indicate reliability (preferable to all-NaN columns that become artifacts after imputation)

  88. [102]

    has_any_measurement

    Revisit the 0–90 / 91–365 split. If most patients have sparse visits, those windows are too granular. Consider: * a single 0–365 window, or * 0–180 and 181–365, or * adaptive windows based on available visits (but be careful to keep it time-safe). B) Reduce care-pattern leakag...

  89. [103]

    model-ready

    Make the modeling pipeline consistent with what prepro- cessing exports (choose one owner of transforms). * If the CSV is truly “model-ready” (already imputed/scaled), then drop training-time SimpleImputer and StandardScaler (at minimum for LR) and keep only minimal guards (e....

  90. [104]

    structurally missing

    Add train-time dropping of “structurally missing” engi- neered columns (especially window stats). * Your metadata shows many window __std_0_90d are missing rate 1.0. Even if imputation makes them constants, they waste degrees of freedom and add selector noise. * Add a simple f...

  91. [105]

    * Try no KBest for LR and rely on ridge/elasticnet regulariza- tion (much more stable in p>>n)

    Replace or de-emphasize SelectKBest (or at least make it much less aggressive). * Try no KBest for LR and rely on ridge/elasticnet regulariza- tion (much more stable in p>>n). * If you keep KBest, push k much higher (think 80–250) and/or use stability selection: compute select...

  92. [106]

    * Your step history already found useful solutions at C≈3 (Step 6), but Step 7’s search tops out at C=0.3 for L2 and elastic

    Expand the LR regularization grid (current grid likely misses good regimes). * Your step history already found useful solutions at C≈3 (Step 6), but Step 7’s search tops out at C=0.3 for L2 and elastic. That’s an unnecessary ceiling. * Also consider solver choices tailored for...

  93. [107]

    mea- sured vs not measured

    Fix HGB to be tree-native (if you keep it). * Let HGB consume more features: remove SelectKBest (or use a very high k) and remove StandardScaler (not used now, good). * Consider using HGB’s native NaN handling by not imputing for HGB at all (your current SimpleImputer erases i...

  94. [108]

    * Set n_splits = min(5, min(n_pos, n_neg)) (or even 3 if minority class is tiny), and/or switch to repeated stratified shuffle split for scoring robustness

    Make CV split settings adaptive to class counts to avoid fold skipping and invalidation. * Set n_splits = min(5, min(n_pos, n_neg)) (or even 3 if minority class is tiny), and/or switch to repeated stratified shuffle split for scoring robustness. * Also: clone the pipeline insi...

  95. [109]

    * Instead of selecting a single ‘best” spec, average predicted probabilities from the top-N diverse specs (e.g., ridge LR + elastic LR + HGB) weighted by CV score

    Use ensembling to reduce variance (very effective at n=58). * Instead of selecting a single ‘best” spec, average predicted probabilities from the top-N diverse specs (e.g., ridge LR + elastic LR + HGB) weighted by CV score. This often improves AUROC in exactly this ‘small samp...

  96. [110]

    no-proxy

    Guard against proxy-dominance as a model-selection crite- rion (to improve hidden/generalization AUROC). * Add an optional ‘proxy blacklist” experiment (education/employment/treatment-choice columns) and treat it as a separate candidate family; select between ‘full” vs “no-pro...

  97. [111]

    balanced

    is effectively doing feature roulette. Any change that (a) removes those columns and (b) reduces reliance on aggressive univariate selection is very likely to move AUROC more than tweaking HGB depth or learning rate. A.6.6 Logger Output. Note Log of DoctorAgents-DS after Summa...

  98. [2020]

    InProceedings of the 37th International Con- ference on Machine Learning (Proceedings of Machine Learning Research, Vol

    Set functions for time series. InProceedings of the 37th International Con- ference on Machine Learning (Proceedings of Machine Learning Research, Vol. 119). PMLR, Virtual, 4353–4363

  99. [2023]

    Self-refine: Iterative refinement with self-feedback.Advances in Neural Information Processing Systems36 (2023), 46534–46594

    2023

  100. [2025]

    OpenAI GPT-5 System Card

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.