{"id":"56736d29-c20f-4ace-9c7b-11264dfa3421","arxiv_id":"2506.22189","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"On 26 chemistry questions, swapping the LLM, agent type, or prompt in an LLM agent changes its scores so much that the system cannot be treated as modular.","lead":"This paper tests whether you can swap the language model, agent type, or system prompt in an AI agent that answers chemistry questions, and finds that performance changes a lot with every swap. The practical takeaway is that LLM-based agent systems are not modular, so swapping components requires re-testing and re-tuning.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central 'non-modularity' conclusion is measured entirely by GPT-4o as LLM judge, and Appendix A.9's cross-judge check does not validate the judge against human grading or an independent model, so observed differences could be scorer artifacts.","rationale":"The reader identified the LLM judge's reliability as the weakest assumption, and my stress-test converges on the same point: every quantitative result in the paper flows through a single judge, GPT-4o, which is also one of the models being compared. The paper's own cross-judge sanity check in Appendix A.9 is a genuine attempt to address this, and it does show that the broad ranking of models is stable across three judges. That is real supporting evidence and should be credited. However, it is not sufficient to settle the concern: it covers only 10 of 26 questions, uses only 3 runs per condition, uses two judges that are themselves among the tested models, reports no correlation or agreement statistic, and never compares against human expert grading. None of the headline conclusions—LLM ranking, agent-type advantage, prompt effects—are checked against an external ground truth. The central claim is carefully hedged as 'as evaluated by an LLM-as-a-judge system', so the authors are not overclaiming beyond their measurement instrument. But precisely because the claim is tied to that instrument, the instrument's validity is the load-bearing premise. I do not see a stronger internal inconsistency or a more concrete technical flaw: the statistical tests on the main LLM comparison are appropriate, the question-level analyses are exploratory and labeled as such, and the authors explicitly acknowledge the dataset's small size. The absence of code and data release is a reproducibility limitation, but it is secondary to the judge-validity issue because even with code, the scores would remain unvalidated. My recommendation is to keep the reader's CONDITIONAL verdict: the paper is a plausible and honest case study, but its central conclusion should be accepted only conditionally on the judge being shown to align with human expert judgment on this task.","tokens_in":15952,"tokens_out":3879,"duration_ms":47667,"concrete_test":"Have two or three medicinal-chemistry experts independently grade agent outputs using a rubric that separates chemical correctness from formatting and verbosity. The grading should cover at least all 910 CodeAgent/default-prompt outputs (26 questions x 5 runs x 7 LLMs), plus a stratified random sample of the remaining agent-type and prompt conditions large enough to re-run the headline comparisons. Compute inter-annotator agreement and the Spearman correlation between expert grades and GPT-4o judge scores. Then re-run the three headline analyses—the Figure 3 Wilcoxon ranking, the Figure 4 agent-type differences, and the Figure 5 prompt deltas—using expert grades. If the qualitative conclusions survive (top versus bottom LLMs separate significantly, CodeAgent's average advantage persists, and prompt effects remain question- and model-dependent), the judge-validity concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise is that the GPT-4o judge's 0-100 scores validly measure answer quality. All headline results—the LLM ranking in Figure 3, the CodeAgent versus ToolCallingAgent comparison in Figure 4, the prompt-sensitivity analysis in Figure 5, and the concluding non-modularity statement—are computed from these scores. Three issues make this premise insecure. First, the judge is itself one of the compared models (GPT-4o), so the top-tier result for GPT-4o could reflect judge self-preference rather than task competence. Second, the preliminary verification in Appendix A.9 uses only 10 questions and 3 runs, reuses two of the tested models (Claude-3.5-Sonnet and Llama-3.1-8B) as judges, reports only aggregate trends, and provides no statistical agreement measure; it cannot rule out systematic judge bias. Third, the agent-type result (CodeAgent better by 2.5 to 12.9 units) could be an artifact if the judge rewards CodeAgent's more structured or verbose final answers rather than chemical correctness, despite the summarization step in Appendix A.6. The authors explicitly defer human-alignment validation to future work in Section 3.4, and no code or data release allows independent re-scoring. Because the central claim is explicitly conditional on 'as evaluated by an LLM-as-a-judge system', an unvalidated judge is the weakest link: if the judge is biased, every headline comparison loses support.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper investigates whether LLM-based agentic systems for drug discovery are modular, i.e., whether components such as the backbone LLM, agent type, and system prompt can be interchanged without degrading performance. The authors build a smolagents-based system with 17 cheminformatics tools and evaluate seven LLMs (GPT-4o, GPT-3.5-Turbo, Claude-3.5/3.7-Sonnet, Llama-3.1-8B/70B, Nova-Micro) in both CodeAgent and ToolCallingAgent modes across three system prompts on 26 industry-style questions. Performance is measured by an LLM-as-a-judge (GPT-4o) that assigns a 0–100 score. The main findings are that Claude-3.5-Sonnet, Claude-3.7-Sonnet, and GPT-4o significantly outperform the other models (paired one-sided Wilcoxon with Bonferroni correction), that CodeAgent beats ToolCallingAgent on average but with strong question- and model-dependent reversals, and that prompt changes have inconsistent effects. The paper concludes that agentic systems are not modular and that components should not be swapped without re-evaluation.","tokens_in":16253,"tokens_out":6930,"duration_ms":69150,"significance":"If the LLM-judge scores can be trusted, the paper provides useful empirical evidence for a phenomenon—non-modularity of agentic systems—that is often assumed away in system-building practice. The strengths include repeated runs with per-question detail, a reasonable paired statistical test for the model comparison, an explicit acknowledgment of the judge limitations, and an informative question-level analysis of tool usage (Appendices A.7 and A.10). The contribution is, however, entirely contingent on the validity of the GPT-4o judge, which is the same model as one of the evaluated agents; the preliminary cross-judge check in Appendix A.9 is too small and too aggregated to establish reliability. The paper's conclusions would be on firmer ground with human validation or a more extensive and statistically quantified judge comparison. As it stands, the evidence supports a descriptive claim about scores assigned by GPT-4o but not yet a robust claim about the intrinsic performance of the agentic system.","major_comments":[{"comment":"The central claim of non-modularity is derived exclusively from scores assigned by GPT-4o, which is also one of the evaluated models. The cross-judge check in Appendix A.9 uses only 10 questions and 3 runs, employs two judges (Claude-3.5-Sonnet and Llama-3.1-8B), reports only averaged trends without any inter-judge agreement metric (e.g., correlation, Cohen's kappa), and does not compare against human grading. Because the judge's reliability is load-bearing for every headline result (Figures 3–5), the paper should either provide stronger validation or consistently qualify all claims in the abstract and body as “according to GPT-4o” rather than only in the conclusion.","section":"§3.1, Appendix A.9"},{"comment":"The claim that CodeAgent outperforms ToolCallingAgent on average “for each of the models” (by 2.5 to 12.9 units) is made without any measure of uncertainty or significance test, despite the availability of five repetitions per question. A paired test or confidence intervals on the difference would be needed to support this claim, particularly because the heatmap in Figure 4 shows large per-question variability and the paper's own narrative stresses question-dependent reversals.","section":"§3.2"},{"comment":"The prompt engineering analysis reports only mean differences in LLM scores and concludes that effects are inconsistent across questions and models. No statistical analysis is provided, so claims such as “the default prompt performs the best for some questions but worst for others” are based on visual inspection. While this is a secondary result, adding at least a simple effect-size quantification or a paired comparison would strengthen the non-modularity argument.","section":"§3.3 and Figure 5"},{"comment":"No code or data are made available, and the expected answers used in the judge prompt (Appendix A.6) are not listed for the 26 questions. Without these, independent re-scoring is impossible, which is particularly problematic in a study whose main endpoint is an unvalidated subjective score. Releasing the full score matrix would allow readers to assess judge stability and re-run the analyses.","section":"Reproducibility"}],"minor_comments":[{"comment":"The abstract and Section 3.1 state that “Claude-3.5-Sonnet, Claude-3.7-Sonnet and GPT-4o outperform alternative language models” but omit the qualifier “as evaluated by the GPT-4o LLM judge” that appears in the conclusion; this should be consistent throughout.","section":"Abstract and §3.1"},{"comment":"There is a typo in the prompt text: “Agent's Anser” should be “Agent's Answer”.","section":"Appendix A.6"},{"comment":"The SMILES string in question 8 is printed as “C1OCc21c1ccc3S=CC=Cc23?” with an extra “1”; this appears to be a typo (the correct SMILES is likely “C1OCc2c1ccc3S=CC=Cc23”).","section":"Appendix A.10"},{"comment":"The flow diagram is somewhat ambiguous: the arrows do not clearly indicate that the same LLMs are used for both agent types and that the judge scores the adjusted answer; a clearer diagram would help readers follow the experimental design.","section":"Figure 2"},{"comment":"The smolagents version 1.14.0 is mentioned only in Section 3.4; it should be stated in Section 2.1 for reproducibility.","section":"§2.1 and §3.4"},{"comment":"The term “LLM score” is used inconsistently; it should be made explicit throughout that this is the judge's score, not the agent's internal confidence or a chemical property.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is from an industry lab and is well-scoped, but the central evaluation method (LLM-as-a-judge) is unvalidated, and the judge is one of the models under comparison. The authors are aware of the issue and the conclusion is conditional, but the abstract and body often present findings without that qualifier. A revision that adds human validation on a subset or a more thorough multi-judge analysis, and that consistently qualifies the claims, would address the main weakness. The descriptive question-level analyses are a strength and should be preserved."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a small, carefully hedged empirical study worth your time if you work on LLM-agent evaluation or drug-discovery agents. It does not overclaim, and its main weakness is one the authors themselves flag in Section 3.4.\n\nThe new thing is the systematic demolition of modularity assumptions within one framework: 7 LLMs x 2 agent types x 3 prompts, 26 industry-style cheminformatics questions, five repetitions each. The claim is that component swapping changes performance, and the effect is question- and model-dependent. That is a useful cautionary result, and the authors deserve credit for showing detailed question-level heterogeneity rather than just reporting averages. The paired Wilcoxon with Bonferroni correction on repeated runs is a statistically reasonable choice for this scale.\n\nThe obvious soft spot is the evaluation instrument. Everything is scored by GPT-4o, which is also one of the models being compared. The cross-judge check in Appendix A.9 is too small (10 questions, 3 runs) and still lacks human ground truth, so any headline difference could in principle be scorer bias. That said, the paper's central sentence is explicitly \"as evaluated by an LLM-as-a-judge system on a small dataset of 26 questions.\" The authors are not pretending otherwise. I also find it plausible that frontier models genuinely call tools better than Llama-3.1-8B, so this does not look like a systematic fabrication; it looks like a preliminary study whose main point is robust but whose fine-grained agent-type and prompt differences (2.5 to 12.9 points) should be treated with caution. I would ask for human scoring on a subset before relying on the ranking.\n\nThe lack of released code and data is a real reproducibility gap. Given the paper is explicitly a case study, I would still send it to review: the question-level analysis is analytically honest and the negative result is useful. The right audience is people building agentic systems in chemistry or thinking about LLM-as-a-judge validity.","headline":"A careful, clearly-scoped case study that makes a plausible non-modularity point for LLM agents in drug discovery; trust the framing, but treat the LLM-as-a-judge as an unvalidated instrument.","tokens_in":16788,"tokens_out":1929,"would_cite":true,"duration_ms":22357,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM-based agentic systems for drug discovery are not modular: swapping the LLM, agent type, or system prompt changes performance unpredictably, depending on the question and model combination.","keywords":["LLM agent modularity","LLM-as-a-judge","smolagents","CodeAgent","ToolCallingAgent","drug discovery","cheminformatics","system prompts"],"falsifier":"Run the same 26-question evaluation with a judge from a different model family (for example, Claude-3.5-Sonnet or Llama-3.1-70B grading the same agent answers) and compare the model rankings. If the top-three ordering changes or the gap between Claude/GPT and Llama collapses, the observed non-modularity is an artifact of the judge rather than a property of the agentic systems. A second independent check is to have human chemists grade the same 26 answers and compute the correlation with GPT-4o's scores.","tokens_in":15762,"feed_emoji":"🧪","tokens_out":6219,"duration_ms":59190,"temperature":0.7,"pith_summary":"The paper asks whether the parts of an LLM-based agentic system for drug discovery—the backbone LLM, the agent type, and the system prompt—are interchangeable. Its answer is no: on a 26-question cheminformatics task set, scores shift dramatically with each swap, and the direction of the shift depends on the specific question and model combination. That matters because drug-discovery teams are starting to build workflows on these agents; if modularity fails, every new model release or prompt tweak requires re-evaluating the whole system, not just the swapped part.","feed_headline":"Drug-discovery AI agents fail drop-in component swaps","feed_subtitle":"Performance depends on the LLM, agent type, prompt, and even question wording, not on any single component.","key_machinery":"The experimental grid is the central object: seven LLMs crossed with two agent types and three system prompts, evaluated on 26 industry-style cheminformatics questions using 17 RDKit- and PubChem-based tools. The two agent types are CodeAgent, which writes and executes Python code, and ToolCallingAgent, which emits JSON tool calls, both running in the smolagents framework's ReAct-style loop. Scoring is done by an LLM-as-a-judge (GPT-4o) that first summarizes the agent's answer, then assigns a score from 0 to 100 against an expected answer, with five repetitions per question to smooth variance.","core_discovery":"The paper's central discovery is that modularity fails across every axis tested. On a 26-question cheminformatics benchmark scored by GPT-4o as judge, Claude-3.5-Sonnet, Claude-3.7-Sonnet, and GPT-4o beat Llama-3.1-8B, Llama-3.1-70B, GPT-3.5-Turbo, and Nova-Micro; code-generating agents beat tool-calling agents on average, but the advantage reverses or vanishes on specific questions and models; and changing the system prompt helps some model-question pairs while hurting others. The authors conclude that components of an agentic system cannot simply be replaced without re-engineering.","pith_inferences":["Inference: the same non-modularity should be expected in other tool-using agent domains (e.g., general coding or bioinformatics), because the interaction between an LLM's parametric memory and the syntax of tool calls is not chemistry-specific.","Inference: because the judge is itself an LLM, some of the reported score variance may be judge variance rather than agent variance; a judge from a different model family could compress or reorder the observed gaps.","Inference: a direct test of prompt-level modularity would hold the system fixed and rephrase only the question, as the oestrogen/progesterone example does, to quantify how much of the variance is due to wording rather than component choice.","Inference: question-level features, such as whether the compound is in PubChem or whether a SMILES string is supplied, could predict when modularity breaks; the paper's tool-call analysis for questions 5 and 8 is a first step toward such a predictor."],"forward_implications":["A team adopting a new, stronger LLM into an existing drug-discovery agent cannot assume the workflow will improve uniformly; the new model must be re-tested across the same question set and agent type.","Reporting only average benchmark scores is misleading here: the question-level heatmaps show reversals that averages hide, so evaluation reports should include per-question and per-model breakdowns.","Prompt engineering does not transfer: a 'cleaner' or 'domain-relevant' system prompt that helps one LLM can hurt another, so prompts should be treated as part of the configuration space, not as a fixed layer.","The choice between CodeAgent and ToolCallingAgent is not a one-time decision; because the better agent depends on the question and model, the system should either support both or have a selection mechanism.","Agentic systems for drug discovery should be evaluated as full configurations (LLM by agent type by prompt), not as standalone components, before being used in real discovery workflows."],"supporting_citations":[{"why":"Provides the LLM-as-a-judge methodology that the evaluation follows.","marker":"[8]"},{"why":"Motivates the summarization step used to mitigate length bias in the judge's scores.","marker":"[25]"},{"why":"Prior result that code-generating agents outperform tool-calling agents, which the paper tests in the drug-discovery domain.","marker":"[30]"},{"why":"Defines the ReAct reasoning-and-acting loop that the smolagents agent types implement.","marker":"[32]"},{"why":"Supplies RDKit, the cheminformatics toolkit behind most of the tools the agents call.","marker":"[19]"},{"why":"Provides PubChem, the database that the pubchem_tool retrieves compound information from.","marker":"[17]"},{"why":"Benchmark of LLM chemical knowledge used to contextualize why Claude models score well on chemistry tasks.","marker":"[23]"},{"why":"Supplies the acquiescence-bias concept used to explain the question-order effect in the appendix.","marker":"[3]"}],"fun_headline_variants":["Drug AI fails modular swap test","Modular drug AI? Components don't mix","Swapping parts breaks drug-discovery AI","No plug-and-play for drug AI components","Drug AI modularity: a myth for now"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole comparison rests on GPT-4o as the judge giving scores that truly reflect answer quality; the paper only sanity-checks this on 10 questions with no human grading, so a biased judge would make the non-modularity finding an artifact.","fun_headline_variants_meta":{"raw":{"variants":["Drug AI fails modular swap test","Modular drug AI? Components don't mix","Swapping parts breaks drug-discovery AI","No plug-and-play for drug AI components","Drug AI modularity: a myth for now"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000338,"raw_usage":{"total_tokens":1860,"prompt_tokens":932,"completion_tokens":928,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":863}},"tokens_in":548,"tokens_out":928,"duration_ms":10893,"temperature":1.0,"reasoning_tokens":863,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:08:42.123236+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 26-question evaluation with a judge from a different model family (for example, Claude-3.5-Sonnet or Llama-3.1-70B grading the same agent answers) and compare the model rankings. If the top-three ordering changes or the gap between Claude/GPT and Llama collapses, the observed non-modularity is an artifact of the judge rather than a property of the agentic systems. A second independent check is to have human chemists grade the same 26 answers and compute the correlation with GPT-4o's scores.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior result that code-generating agents outperform tool-calling agents, which the paper tests in the drug-discovery domain."}],"review_version":1}