{"id":"61bb9f6a-d168-4458-8f87-d7bcc7b7150f","arxiv_id":"2411.15692","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"DrugAgent, a Planner-Instructor multi-agent LLM framework with drug-discovery documentation, beats ReAct by 4.92% ROC-AUC on DTI and exceeds general agents on two of three tasks in its Top3 variant.","lead":"DrugAgent is a multi-agent system where one large language model plans high-level solution ideas and another turns them into code, guided by curated drug-discovery documentation. In tests on three standard tasks, it matched or beat simpler agent baselines on some tasks, though the reported gains come with caveats.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"DrugAgent@Top3's best-of-three test-set reporting, while baselines report one test result, makes the 'consistently outperforms' claim depend on a protocol asymmetry.","rationale":"The reader's weakest assumption identifies exactly the protocol issue I find most load-bearing: DrugAgent@Top3 is allowed to report the best test-set result among three validation-selected solutions, while baselines get one test evaluation per run. The paper's own Table 2 shows that DrugAgent@Top1 does not consistently outperform ResearchAgent on ADMET and HTS, so the headline claim of consistent superiority depends on the Top3 variant. This is not a disagreement about the value of the framework; the case study and ablations suggest the multi-agent design has merit. But the empirical claim as stated is not supported under a fair single-submission protocol, and the invalid-submission exclusion rule further complicates interpretation. The concrete test is straightforward because the needed numbers (Top1 vs Top3) are already in the table; a matched best-of-k baseline would settle whether the advantage is an artifact. I therefore recommend keeping the CONDITIONAL verdict: the framework is promising, but the evaluation protocol must be revised before performance claims are accepted.","tokens_in":12603,"tokens_out":3766,"duration_ms":34351,"concrete_test":"Rerun the ADMET, HTS, and DTI comparisons under a matched protocol: for every method, each run selects exactly one final solution by validation score (no multiple test-set lookups), and all runs are reported including the 'invalid' ones (or with a pre-registered aggregation that does not discard low performers). The paper's Table 2 already contains the key numbers for DrugAgent@Top1; if the aggregated comparison shows DrugAgent@Top1 is not consistently above ResearchAgent on ADMET and HTS, the claimed 'consistent outperformance' is not supported. As a direct test of the selection effect, allow every baseline the same best-of-three validation-selected submission and report the best test result; if baseline ROC-AUC rises by a comparable margin, the advantage is protocol-induced.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"In Section 3.1, DrugAgent@Top3 'submits the top three solutions based on validation results, and reports the best test set outcome,' while CoT, ReAct, and ResearchAgent produce a single submission per run. Table 2 averages over valid runs. On ADMET, DrugAgent@Top3 = 0.8206 vs DrugAgent@Top1 = 0.7667; the Top1 number is below ResearchAgent (0.7957). On HTS, Top3 = 0.8257 vs Top1 = 0.7919 (ResearchAgent = 0.7913, essentially tied). Only on DTI does Top1 (0.8950) beat all AI baselines. Thus the claimed 'consistently outperforms leading baselines' rests on the Top3 variant, which exploits the test set by reporting the maximum over three candidate submissions, an option not given to baselines. Section 3.2 explicitly acknowledges that 'validation set performance does not always strongly correlate with test set performance ... considering multiple submissions can help mitigate this problem,' confirming the selection effect. The additional rule that submissions more than 10% below the human baseline are marked invalid and excluded from averages can also inflate reported means for methods with low valid rates (e.g., ReAct DTI valid rate 50%), but the primary asymmetry is the best-of-three test reporting.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DrugAgent, a multi-agent LLM framework for automating ML programming in drug discovery. DrugAgent combines an LLM Planner that manages an idea space and an LLM Instructor that incorporates curated domain documentation and tools during code generation and debugging. The authors evaluate DrugAgent on three binary classification tasks (ADMET on PAMPA, HTS on HIV, DTI on DAVIS) against CoT, ReAct, ResearchAgent, and a literature-based human baseline, reporting ROC-AUC and valid rate over eight runs per AI method. The central claim is that DrugAgent consistently outperforms the AI baselines, including a 4.92% relative ROC-AUC improvement over ReAct on DTI. The paper also presents an ablation study on DTI, a case study comparing DrugAgent with ReAct, an error-type analysis, and an anonymous code repository.","tokens_in":12828,"tokens_out":5225,"duration_ms":47620,"significance":"If the performance claims held under a fair evaluation protocol, DrugAgent would be a useful contribution to automated ML programming in a domain-specific setting: the framework design is plausible, the curated documentation is a tangible resource, the code is publicly available, and the ablation and trace analyses help isolate the roles of the Planner and Instructor. However, the headline comparisons currently depend on an evaluation asymmetry that is load-bearing. DrugAgent@Top3 reports the best test-set outcome among three validation-selected submissions while baselines submit a single run, and the validity filter excludes low-performing runs from the reported averages. The current evidence supports at most a qualified DTI advantage, not the claimed consistent superiority, so the evaluation needs substantial revision.","major_comments":[{"comment":"The DrugAgent@Top3 protocol reports the best test-set outcome among three validation-selected submissions, while CoT, ReAct, and ResearchAgent submit a single run. This is a test-set selection asymmetry: Table 2's ADMET and HTS advantages (0.8206 and 0.8257) are maxima over three test evaluations, not outcomes of a single predefined submission. The paper's own discussion in Sec. 3.2 states that validation performance does not always correlate with test performance, which is exactly the condition under which best-of-three test reporting inflates results. Notably, DrugAgent@Top1 is below ResearchAgent on ADMET (0.7667 vs. 0.7957) and essentially tied on HTS (0.7919 vs. 0.7913); only DTI shows a Top1 advantage. The claim that DrugAgent 'consistently outperforms leading baselines' therefore rests on the Top3 variant. Please compare all methods under an identical submission rule, or report the full per-run distribution including the first or predefined submission.","section":"Sec. 3.1 and Table 2"},{"comment":"The validity filter is not only about code correctness: a submission is invalid if its performance falls more than 10% below the human baseline. Averaging only over valid submissions therefore removes low-performing runs from the denominator, which can inflate reported means for methods with low valid rates. In Table 2, DTI valid rates are 50% for ReAct, 0% for CoT, and 87.5% for DrugAgent, so the ReAct DTI mean of 0.8530 is computed over a subset of runs that already passed a performance threshold. Please report all eight runs separately, or analyze sensitivity to this threshold; at minimum, separate code and format validity from the performance-based exclusion.","section":"Sec. 3.1, validity definition"},{"comment":"No measures of variance are reported for any method, despite eight independent runs per AI agent. With stochastic LLM agents, point estimates such as ADMET 0.8206 vs. 0.7957 and HTS 0.8257 vs. 0.7913 cannot be interpreted without standard deviations, confidence intervals, or paired significance tests. Please provide per-run results and an appropriate significance analysis for the headline comparisons and for the ablation in Table 3.","section":"Sec. 3.2, Table 2"}],"minor_comments":[{"comment":"The DAVIS dataset is written as 'DA VIS' in Table 2; the spacing should be corrected.","section":"Table 2"},{"comment":"The ReAct baseline is sometimes rendered as 'React' in the body text; the capitalization should be consistent.","section":"Sec. 3.2 and Figure 2"},{"comment":"The caption contains a typo ('impleennted'), and the two subfigures would be easier to read with clearer labels and a consistent line-weight style.","section":"Appendix G, Figure 3"},{"comment":"The human baselines are described without hyperparameters, such as the random forest settings and the DeepDTA architecture details; specifying these would improve reproducibility.","section":"Appendix E"},{"comment":"Table 4 describes DTI as 'multi-instance prediction' while Sec. 3.1 states that all three tasks are binary classification; the task formulation should be clarified.","section":"Table 4 and Sec. 3.1"},{"comment":"The error-mode figure lacks axis labels and explicit counts, making it difficult to judge how the reported percentages relate to the eight runs.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"To the editor: this is a reasonable system paper whose contribution is separable from the current evaluation. The headline claims are not yet supported because the Top3 protocol and the performance-based validity filter both favor DrugAgent. I would support a major revision rather than rejection, since a fair rerun with single predefined submissions and per-run reporting could reasonably confirm at least the DTI result."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read. DrugAgent is a sensible multi-agent system that combines an idea-space Planner with a domain-knowledge Instructor for drug-discovery ML programming. The system is implemented, the docs are curated, and the qualitative case study shows the agents actually learning from failures (e.g., skipping graph-based protein encoding after a crash). That part is worth your time.\n\nThe trouble is the headline 'consistently outperforms' claim. The Top3 variant picks the best of three validation-selected solutions and reports the best test outcome, while CoT/ReAct/ResearchAgent submit one run. Look at Table 2: on ADMET, DrugAgent@Top1 is 0.7667, below ResearchAgent's 0.7957. On HTS, Top1 is 0.7919, essentially tied with ResearchAgent at 0.7913. Only on DTI does Top1 clearly beat all AI baselines. So the 'consistent' advantage is largely a product of the Top3 selection protocol, which the paper itself concedes in Section 3.2. That doesn't sink the framework, but it means the empirical claim as written is not supported.\n\nTwo more issues. Eight runs, no error bars or significance tests. And the validity filter that drops submissions more than 10% below the human baseline makes averages hard to interpret, especially for baselines with low valid rates (ReAct on DTI is 50%). It's a minor-to-moderate issue; the main asymmetry is the test-set selection.\n\nWhat's genuinely good: the ablation on DTI shows both Planner and Instructor contribute; the curated documentation for featurization and models is concrete; the analysis of error types is plausible and consistent with the framework's design. The code is available, though I can't verify it from the text, and the dependence on GPT-4o-2024-08-06 limits reproducibility.\n\nBottom line: this is a legitimate systems paper with an over-claimed evaluation. If a revision reports all eight runs, adds error bars, and either drops the Top3 variant or gives baselines the same top-k option, the claims will be much cleaner. For now, treat 'consistently outperforms' as a hypothesis, not a result. I'd send it to peer review with heavy revision requested, and to a reading group it's a useful example of how to (and how not to) evaluate agent frameworks.","headline":"A useful multi-agent framework whose headline 'consistent outperformance' rests on a best-of-three test-set selection protocol that baselines don't get; the paper needs evaluation fixes before the claims hold.","tokens_in":13386,"tokens_out":2664,"would_cite":true,"duration_ms":23192,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DrugAgent, a two-agent LLM framework, automates end-to-end machine-learning programming for drug-discovery tasks and reports performance comparable to human expert baselines on three benchmarks.","keywords":["drug discovery","multi-agent LLM","automated ML programming","domain knowledge integration","ADMET prediction","drug-target interaction","high-throughput screening","idea-space planning"],"falsifier":"Run both DrugAgent and the comparison methods under a single-submission rule: each run submits its one highest-validation idea, all eight runs are averaged, and no low-scoring runs are discarded, then compare test ROC-AUC on ADMET and HTS. If the margin over the best comparison method narrows to near zero on those tasks, the Top3-and-filter protocol is responsible for the headline gains.","tokens_in":12392,"feed_emoji":"💊","tokens_out":8882,"duration_ms":72972,"temperature":0.7,"pith_summary":"DrugAgent is a multi-agent framework that gets large language models to write complete machine-learning programs for drug-discovery tasks. Its central claim is that a Planner, which generates and filters candidate modeling ideas, plus an Instructor, which consults curated domain documentation while coding, can produce solutions that reach or exceed human-written baselines. The paper tests this on absorption prediction, high-throughput screening, and drug-target interaction, reporting ROC-AUC values of 0.8206, 0.8257, and 0.8950 with 100% valid submissions. The 4.92% relative improvement over the best single-agent baseline on drug-target interaction is the headline quantitative result. A sympathetic reader would take the contribution to be evidence that explicit domain-knowledge integration, not raw code generation, is what unlocks LLM performance in specialized ML workflows.","feed_headline":"Two LLM agents match expert-written drug-discovery models","feed_subtitle":"Beats single-agent baselines on three drug benchmarks, with a 4.9% relative gain on binding prediction.","key_machinery":"The load-bearing mechanism is the planner–instructor division of labor. The Planner maintains an idea space, a set of candidate modeling strategies, generates new options from the task description, selects one per round, and prunes it based on empirical validation or execution failure. The Instructor converts the selected idea into code, detects where specialized knowledge is needed, consults a curated documentation suite covering data acquisition, molecular and protein featurization, and domain-specific model interfaces, and returns a success or failure report that feeds the Planner's next decision. This closed loop turns the documentation library into an externalized domain memory that prevents the kind of encoding mistakes and dead-end debug loops the paper observes in single-agent baselines.","core_discovery":"The paper claims and supports the following: a two-role LLM agent can close the domain-knowledge gap that generic coding agents leave open in drug discovery. The Planner owns the idea space; it proposes several distinct approaches, picks one for implementation, receives validation or failure reports, drops impractical directions, and iterates until a step budget runs out. The Instructor owns implementation; it decides where biological or chemical domain knowledge is required, pulls the relevant facts from a curated documentation library covering data loading, molecular featurization, and pretrained models, writes and debugs the code, and reports back. On the three case studies, the full system, submitting its top three validation-ranked solutions, achieves test ROC-AUC of 0.8206, 0.8257, and 0.8950 and a 100% valid-submission rate, while ablations show each agent contributes. The authors interpret this as showing that systematic idea diversification plus domain-aware implementation lets LLMs match expert-selected models.","pith_inferences":["Under a stricter protocol where each run submits its single best validation idea and low-scoring runs are not excluded, the ADMET and HTS margins over the best baseline would likely shrink or vanish: DrugAgent@Top1 scores 0.7667 and 0.7919 there, close to baseline levels.","The framework's ceiling is set by the documentation library; adding entries for new modality types, such as 3D conformers or dose-response curves, is a natural test of whether the approach transfers beyond the three tasks.","A baseline that retrieves documentation from an open corpus instead of a fixed curated set would separate the value of the knowledge content from the value of the retrieval mechanism.","Comparing DrugAgent@Top3 against three independent single-seed runs of the same agent would test whether the gain comes from idea diversity or simply from more chances to pick a lucky test split."],"forward_implications":["On the three tested tasks, a two-agent loop with validation feedback yields higher ROC-AUC and a higher share of valid submissions than generic reasoning agents, implying that specialization plus exploration, not a stronger LLM, drives the gain.","Allowing three validation-selected submissions instead of one raises final scores on ADMET and HTS, because validation ranking does not perfectly track test ranking; multiple submissions absorb that mismatch.","Both components matter: ablations on the binding task show removing the Instructor cuts ROC-AUC and valid rate, and removing the Planner reduces performance, so the improvement is not due to one agent alone.","Domain-knowledge errors, such as naive encoding of proteins, account for most failures in generic agents and are absent in DrugAgent, indicating that curated documentation is what prevents those errors."],"supporting_citations":[{"why":"Defines the ML programming task formulation, the evaluator setup, and the action set that DrugAgent and the ResearchAgent baseline both use.","marker":"(Huang et al., 2024a)"},{"why":"Supplies the reasoning-acting baseline that DrugAgent is compared against and whose single-agent behavior is analyzed in the case study.","marker":"(Yao et al., 2023)"},{"why":"Provides the expert-selected deep model baseline used as the human reference for drug-target interaction.","marker":"(Öztürk et al., 2018)"},{"why":"Shows tree-based models with fingerprints dominate deep models on ADMET and HTS, justifying the random-forest human baseline choice.","marker":"(Xia et al., 2023)"},{"why":"Supplies the PAMPA permeability dataset used for the ADMET case study.","marker":"(Siramshetty et al., 2021)"},{"why":"Supplies the HIV assay dataset used for the high-throughput screening case study.","marker":"(Wu et al., 2018)"},{"why":"Supplies the kinase-binding dataset used for the drug-target interaction case study.","marker":"(Davis et al., 2011)"}],"fun_headline_variants":["Two LLM agents automate drug discovery coding, beat baselines","DrugAgent: Two-role LLM framework outperforms single agents","4.92% relative gain: LLM planner+instructor for drug binding prediction","LLM multi-agent for drug discovery: matches expert-written models","DrugAgent: Automating AI-aided drug discovery with LLM agents"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported advantage rests on an evaluation protocol in which DrugAgent may submit its top three validation-selected solutions and only the best test result is counted, while each baseline submits one solution, and runs scoring more than 10% below the human baseline are excluded as invalid.","fun_headline_variants_meta":{"raw":{"variants":["Two LLM agents automate drug discovery coding, beat baselines","DrugAgent: Two-role LLM framework outperforms single agents","4.92% relative gain: LLM planner+instructor for drug binding prediction","LLM multi-agent for drug discovery: matches expert-written models","DrugAgent: Automating AI-aided drug discovery with LLM agents"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000308,"raw_usage":{"total_tokens":1742,"prompt_tokens":909,"completion_tokens":833,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":739}},"tokens_in":525,"tokens_out":833,"duration_ms":7866,"temperature":1.0,"reasoning_tokens":739,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:01:17.288632+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run both DrugAgent and the comparison methods under a single-submission rule: each run submits its one highest-validation idea, all eight runs are averaged, and no low-scoring runs are discarded, then compare test ROC-AUC on ADMET and HTS. If the margin over the best comparison method narrows to near zero on those tasks, the Top3-and-filter protocol is responsible for the headline gains.","supporting_citations":[],"review_version":1}