{"id":"95455b89-9080-403b-81da-35d829a55d22","arxiv_id":"2505.23987","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Instruction-tuned LLMs trained on C-MuMOInstruct, a new controllable multi-property molecule optimization dataset, outperform strong baselines on in-distribution and out-of-distribution optimization tasks.","lead":"This paper introduces C-MuMOInstruct, a new instruction-tuning dataset that teaches large language models to optimize drug-like molecules while selectively improving some properties and keeping others stable. The authors' GeLLM4O-C models beat general-purpose and chemistry LLMs on 10 benchmark tasks, with large gains on out-of-distribution property combinations.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The OOD split appears internally contradicted: Algorithm A1 enumerates all property subsets, Table 2 lists training pairs for the OOD combinations, and Section 5.1 says P(10) trains on all possible combinations—so the 0-shot transfer claim is not yet supported.","rationale":"The reader's weakest assumption identifies exactly the load-bearing concern: the OOD property combinations may not have been excluded from training, despite Section 2.6 claiming they were. My reading of the full manuscript strengthens this from a missing-documentation issue to a likely internal inconsistency. Algorithm A1 generates tasks for every property subset; Table 2 reports non-zero training pair and molecule counts for the OOD combinations whose caption explicitly labels those columns as training statistics; and Section 5.1 states P(10) is trained on all possible property combinations up to 10 properties. Collectively, these passages suggest that tasks for CDE, ABMP, BCMQ, BDEQ, and HLMPQ are present in the training data. If so, the headline claim of 0-shot generalization to novel property combinations is not demonstrated, because the model has already seen those combinations during instruction tuning. This is a correctness risk for the central claim, not a disagreement with community consensus. The proposed check is concrete and feasible because the paper points to a public dataset and code URL; inspecting the training task metadata would settle the matter directly. I credit the paper for holding out one instruction per task to test unseen-instruction robustness, but that does not address the property-combination leakage. Since the reader already issued a CONDITIONAL verdict conditioned on confirming the OOD exclusion, my analysis does not change that verdict; it reinforces it. I recommend keeping the CONDITIONAL status: accept only after the authors verify, via released data or a code re-run, that no OOD property combination appears in any training task for the generalist models.","tokens_in":34895,"tokens_out":6014,"duration_ms":59653,"concrete_test":"Download the released C-MuMOInstruct metadata from the GitHub URL and, for the training split used for GeLLM4O-C-P(10), count tasks whose property set exactly matches each OOD combination: {CARC, DRD2, hERG}, {AMP, BBBP, MUT, PlogP}, {BBBP, CARC, MUT, QED}, {BBBP, DRD2, hERG, QED}, {HIA, LIV, MUT, PlogP, QED}. If any count is greater than zero, Section 2.6's exclusion claim is false and Table 4 measures in-distribution rather than OOD performance. An equivalent check is to re-run Algorithm A1 on the documented 100K molecule pairs without any added filter and confirm whether OOD tasks are generated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.6 defines the five OOD tasks (CDE, ABMP, BCMQ, BDEQ, HLMPQ) as property combinations 'not used during training.' Three in-manuscript pieces of evidence contradict this, making the 0-shot generalization claim the paper's most load-bearing weakness. First, Algorithm A1 (Appendix B.1) constructs tasks for 'all possible property combinations involving up to P properties' on sampled molecule pairs, with no documented filter that removes the OOD combinations; Appendix D.1 reports up to 28K tasks and 1M pairs for GeLLM4O-C-P(10), which matches the full dataset size. Second, Table 2's caption states that its #Pairs and #Mols columns denote 'the number of molecule pairs and unique molecules in training, respectively,' yet the OOD rows report non-zero training counts (e.g., CDE 516 pairs, ABMP 1,500, HLMPQ 1,800). If those combinations were truly excluded from training, these counts should be zero. Third, Section 5.1 explicitly says GeLLM4O-C-P(10) is 'trained on all possible property combinations involving up to 10 properties,' which necessarily includes the OOD combinations. If any OOD combination appeared in the P(10) training data, the results in Table 4 are in-distribution performance, not 0-shot generalization to novel combinations, and the claimed 27-35% gains over baselines cannot be attributed to transfer. Since the paper never describes an OOD-exclusion filter, the current manuscript is internally inconsistent rather than merely under-documented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces C-MuMOInstruct, an instruction-tuning dataset for controllable multi-property molecule optimization, where each task specifies which properties to improve up to a threshold and which to maintain. The authors instruction-tune Mistral-7B and Llama-3.1-8B (with LoRA) to produce GeLLM4O-C models in both specialist (per property combination) and generalist (up to N properties) variants. They evaluate on five in-distribution (IND) and five out-of-distribution (OOD) property combinations, reporting large gains in success rate (SR) over general-purpose LLMs and chemistry LLM baselines, as well as 0-shot generalization to unseen instructions. The manuscript includes detailed descriptions of dataset construction, quality control, prompts, and evaluation metrics, and the code/data are publicly released.","tokens_in":35273,"tokens_out":9915,"duration_ms":88060,"significance":"If the results are valid, C-MuMOInstruct is a valuable resource for controllable multi-objective molecule optimization, and the instruction-tuning approach demonstrates a practical path toward generalist molecular optimization LLMs. The paper ships the dataset, code, and model configurations, which supports reproducibility. The reported IND results are detailed and consistent across multiple tasks and backbones. However, the central claim of 0-shot generalization to novel property combinations is compromised by an apparent contamination of the OOD split, and the evaluation protocol gives different sampling budgets to different baseline families. These issues must be resolved before the main conclusions can be accepted.","major_comments":[{"comment":"The claim that CDE, ABMP, BCMQ, BDEQ, and HLMPQ are out-of-distribution tasks is internally contradicted by the dataset construction and model training description. Algorithm A1 enumerates all property subsets C ⊆ P with no documented filter excluding these five combinations. Table 2 reports nonzero #Pairs/#Mols for these combinations under a caption that explicitly states these are training counts (e.g., CDE 516 pairs, ABMP 1,500, HLMPQ 1,800). Section 5.1 states that GeLLM4O-C-P(10) is trained on all possible property combinations involving up to 10 properties, and Appendix D.1 reports training on up to 28K tasks and 1M pairs, which is consistent with a dataset that includes the OOD combinations. If any of these combinations appeared in the P(10) training data, the results in Table 4 are in-distribution performance, not 0-shot generalization to novel combinations. Please either provide the exact exclusion filter and the resulting training task counts, or reclassify these tasks as IND and revise the associated claims (abstract, Sections 5.2, 6).","section":"Section 2.6 vs. Algorithm A1 (Appendix B.1), Table 2, Section 5.1, Appendix D.1"},{"comment":"The success rate (SR) metric defines a test case as successful if at least one of the 20 candidate molecules satisfies all objectives. GeLLM4O-C and open-weights baselines (Mistral, Llama, LlaSMol) generate 20 candidates via beam search, while closed-weights API models (Claude-3.5, GPT-4o) generate only one candidate per input because beam search is not available. This sampling asymmetry gives multi-candidate models an inherent advantage in SR, and the reported gains (e.g., \"up to 126% higher SR\" in the abstract) are not directly comparable across all baseline families. The paper should either generate 20 samples for API models via temperature sampling, or report an additional metric that controls for the number of candidates (e.g., per-candidate success rate or success rate at k=1) for all models.","section":"Section 4.2 and Appendix D.2"},{"comment":"The dataset construction description is inconsistent regarding the number of source molecule pairs. Section 2.3 says task-specific pairs are selected from the 256K molecule pairs in Chen et al. (2021), but Appendix B.1 states that Algorithm A1 is run on a random sample of 100K molecule pairs from the same source. Table 1 reports that C-MuMOInstruct contains 256,185 pairs, which is impossible if only 100K source pairs were considered. Please clarify whether the full 256K or the 100K sample was used, and reconcile the reported dataset size. This is essential for reproducing the dataset and for assessing its scale.","section":"Appendix B.1 vs. Section 2.3 and Table 1"}],"minor_comments":[{"comment":"The property abbreviation for drug-induced liver injury is given as LIV in Section 2.2 and Table 2, but Appendix C uses DILI; please use one consistent abbreviation throughout.","section":"Section 2.2 and Table 2"},{"comment":"Table 2 reports Θp values such as 0.8, 0.4, and 1.5, which appear to be rounded numbers, while Section 2.3 says thresholds are set to the 60th percentile of training molecules. Please clarify whether these are exact empirical percentiles or rounded approximations, and specify which values are used in the task construction algorithm.","section":"Section 2.3 and Table 2"},{"comment":"The sentence \"The generalist variant, GeLLM4O-C-P(N), – trained only on tasks involving BBBP, DRD2, PlogP and QED\" is inaccurate: GeLLM4O-C-P(N) with N=4 is trained on all property combinations up to 4 properties, not only the BDPQ quartet. Please correct this description.","section":"Section 5.1"},{"comment":"The line \"Set dir[p] <- (change[p]>0) if higher p is desirable, else negative\" is ambiguous; it should be rewritten to specify the direction condition clearly (e.g., whether change[p] is positive for desirable direction and negative otherwise).","section":"Appendix B.1, Algorithm A1"},{"comment":"The paper claims 30 distinctly phrased instructions per task, but with 6 general instruction templates and 5 adjustment templates, a task with k sub-optimal properties yields 6×5^k possible prompt combinations. Please clarify what counts as a distinct instruction and how the 30 number is derived.","section":"Section 2.5 and Appendix C"},{"comment":"There are minor typos: \"GeLLMO-C\" in the abstract URL should be \"GeLLM4O-C\", and \"Lipsinki's rule\" in Section 2.4 should be \"Lipinski's rule\".","section":"Abstract and Section 2.4"}],"recommendation":"major_revision","confidential_remarks":"The OOD contamination issue is the most serious concern. If the authors cannot provide an explicit exclusion filter, the 0-shot generalization claim should be withdrawn or substantially weakened, and Table 4 should be reclassified as IND performance. The dataset size inconsistency (100K vs. 256K pairs) also needs reconciliation. The paper otherwise presents a solid empirical study with a released dataset and code, and the IND evaluation is thorough."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The C-MuMOInstruct dataset is the real contribution here. It goes beyond MuMOInstruct and DrugAssist by giving each property its own objective—improve some, keep others stable, with per-property thresholds. That matches real lead optimization better than the usual 'improve everything' setup. The construction is careful, using established predictors and similarity constraints, and the in-distribution results show large, consistent gains over strong baselines. That part is solid.\n\nThe problem is the OOD evaluation. Table 2 lists training pairs for all five 'OOD' combinations—CDE 516, ABMP 1,500, BCMQ 1,398, BDEQ 603, HLMPQ 1,800. Algorithm A1 enumerates every property subset with no documented exclusion, and Section 5.1 says GeLLM4O-C-P(10) is trained on 'all possible property combinations involving up to 10 properties.' So the claim that these are novel, unseen combinations is not supported by the paper's own text. If those combinations were in the training data, Table 4 is measuring in-distribution performance, not 0-shot transfer. The paper also doesn't document any filter that would exclude the specific improve/maintain objectives used in OOD evaluation, so the 'novel objectives' part is unclear too. That's the load-bearing soft spot.\n\nOther issues are minor: no error bars, no commit hash for the code/data URL, and the usual 'first' claim. The limitations section is honest about single-step optimization and predictor noise.\n\nThis paper deserves a serious referee because the dataset and the controllable-objective formulation are useful, and the IND comparisons are informative. But the headline generalization claim needs to be either verified with a real exclusion filter or downgraded to seen-combination results. I'd send it out, with instructions to the referees to push hard on the exact OOD construction and the released split.","headline":"A genuinely useful controllable-optimization dataset, but the OOD split is internally contradicted and undermines the 0-shot transfer claim.","tokens_in":35777,"tokens_out":3667,"would_cite":true,"duration_ms":36965,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Instruction-tuned LLMs learn to improve some drug properties without wrecking others.","keywords":["molecule optimization","instruction tuning","multi-objective optimization","property-specific objectives","large language models","ADMET properties","drug discovery","0-shot generalization"],"falsifier":"Search the released training tasks for any molecule pair or prompt whose property combination matches CDE, ABMP, BCMQ, BDEQ, or HLMPQ; if even one such pair is present, retrain the generalist on the filtered data and check whether the out-of-distribution advantage survives. As a second check, train a generalist on only single- and double-property combinations: strong performance on the five novel combinations would suggest genuine composition of property knowledge, while a collapse would suggest memorization of larger combinations.","tokens_in":1736,"feed_emoji":"🧪","tokens_out":2696,"duration_ms":67407,"temperature":0.7,"pith_summary":"This paper aims to show that instruction-tuned large language models can perform controllable multi-property molecule optimization: given a starting molecule and per-property instructions, the model improves properties below pharmaceutically relevant thresholds while leaving already acceptable properties essentially unchanged. To do this it builds C-MuMOInstruct, a dataset of 256,185 molecule pairs and 28,266 tasks covering up to 10 properties, with property-specific improvement and stability constraints. It then tunes Mistral-7B and Llama3.1-8B on this dataset to produce GeLLM4O-C specialists and generalists. The reported results claim that the generalist models outperform strong LLM baselines by up to 126% in success rate, including 0-shot success on property combinations and instructions held out from training. A sympathetic reader would care because this is the optimization regime actually faced in lead optimization, where some properties are already good enough and should not be degraded while others need targeted improvement.","feed_headline":"LLMs learn to fix one drug property without breaking the rest","feed_subtitle":"Tuned LLMs push sub-optimal drug properties past thresholds while keeping good ones stable, even on unseen tasks.","key_machinery":"The load-bearing object is the C-MuMO task itself, defined by partitioning properties into a sub-optimal set $P_i$ to improve by at least $\\Delta_p$ and a near-optimal set $P_s$ to keep within $\\Delta_p$, with thresholds $\\Theta_p$ marking pharmaceutically relevant levels. C-MuMOInstruct instantiates this as 256,185 molecule pairs filtered from an existing similarity-constrained pair set, while an enumeration algorithm generates tasks for every subset of the 10 properties and 30 instruction templates express the same objectives with varied phrasing. A held-out instruction and held-out property combinations are used to test transfer. The models are trained with LoRA on Mistral-7B and Llama3.1-8B to map a natural-language prompt plus source SMILES to an edited SMILES.","core_discovery":"The central claim is that instruction tuning on carefully constrained molecule pairs teaches LLMs a controllable optimization policy: the model learns to associate small structural edits with directional changes in specified properties, so that a sub-optimal property can be pushed past its threshold without moving near-optimal properties outside their stability bounds. The paper demonstrates this with GeLLM4O-C-P(10), a generalist tuned on all property combinations up to 10 properties, which outperforms the best baseline by 37% average success rate and 102% relative improvement on in-distribution tasks, and by 35% success rate and 76% relative improvement on out-of-distribution tasks, with up to 126% higher success rates on individual tasks. The 0-shot transfer to unseen combinations, such as CDE (CARC, DRD2, hERG) and HLMPQ (HIA, LIV, MUT, PlogP, QED), is presented as evidence that the model composes knowledge about individual properties rather than memorizing task-specific pairs.","pith_inferences":["My inference: the same pair-based instruction-tuning recipe should extend to other molecular properties beyond these ten, as long as reliable predictors and pair data filtered by the same improvement and stability constraints are available.","My inference: a natural next test is iterative optimization, because the current model performs one-step edits; whether repeated single-step calls can climb to thresholds without drifting is an open empirical question.","My inference: because all property scores come from computational predictors rather than wet-lab assays, the practical value of the optimized molecules will only be settled by experimental validation.","My inference: if the apparent 0-shot transfer is real composition of single-property knowledge, out-of-distribution performance should scale with the number of properties covered during training; that scaling curve would be a direct test."],"forward_implications":["If generalist GeLLM4O-C-P(10) performs as reported, a single instruction-tuned LLM can handle diverse optimization tasks without task-specific retraining or handcrafted rewards.","Property-specific thresholds let a user specify both which properties to improve and how far, which is the regime pharmaceutical lead optimization actually operates in.","Generalist models outperform specialists on four of five in-distribution combinations and suffer less from held-out instruction rephrasing, indicating that multi-task tuning transfers across related objectives.","On out-of-distribution tasks the best generalist improves average success rate by about 35% over the strongest baseline, implying the learned strategies are not limited to memorized pair patterns."],"supporting_citations":[{"why":"Supplies the underlying 256K molecule pairs that satisfy the similarity constraint and feed the task construction.","marker":"(Chen et al., 2021)"},{"why":"Provides MuMOInstruct, the prior multi-property instruction-tuning dataset that this work extends by adding property-specific objectives.","marker":"(Dey et al., 2025)"},{"why":"ADMET-AI computes most of the molecular property scores used to annotate the dataset.","marker":"(Swanson et al., 2024)"},{"why":"Provides the computational scoring functions used for the DRD2 and penalized logP properties.","marker":"(You et al., 2018)"},{"why":"ZINC is the source of test molecules for evaluating optimization performance.","marker":"(Sterling and Irwin, 2015)"},{"why":"LoRA is the parameter-efficient fine-tuning method used to adapt the LLM backbones.","marker":"(Hu et al., 2022)"},{"why":"Mistral-7B-Instruct-v0.3 serves as one of the two general-purpose LLM backbones for GeLLM4O-C.","marker":"(AI, 2023)"},{"why":"Llama3.1-8B-Instruct serves as the second LLM backbone for GeLLM4O-C.","marker":"(Grattafiori et al., 2024)"},{"why":"LlaSMolMistral is the strongest chemistry-foundation baseline that the GeLLM4O-C models are compared against.","marker":"(Yu et al., 2024)"}],"fun_headline_variants":["LLMs now tweak one drug property without ruining the rest","C-MuMOInstruct: first dataset for property-specific drug optimization","GeLLMO-C: LLMs that fix specific drug properties on demand","Instruction-tuned LLMs boost drug property success by 126%","0-shot drug optimization: LLMs handle unseen property combos"],"cache_read_input_tokens":37888,"weakest_assumption_plain":"The paper's 0-shot generalization claim rests on the assumption that the five 'out-of-distribution' property combinations never appear in the training data; because the task generator enumerates every property subset and the paper does not document an exclusion filter, a single leaked pair would undercut the transfer result.","fun_headline_variants_meta":{"raw":{"variants":["LLMs now tweak one drug property without ruining the rest","C-MuMOInstruct: first dataset for property-specific drug optimization","GeLLMO-C: LLMs that fix specific drug properties on demand","Instruction-tuned LLMs boost drug property success by 126%","0-shot drug optimization: LLMs handle unseen property combos"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000283,"raw_usage":{"total_tokens":1675,"prompt_tokens":951,"completion_tokens":724,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":567,"completion_tokens_details":{"reasoning_tokens":632}},"tokens_in":567,"tokens_out":724,"duration_ms":6677,"temperature":1.0,"reasoning_tokens":632,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:37:42.338685+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Search the released training tasks for any molecule pair or prompt whose property combination matches CDE, ABMP, BCMQ, BDEQ, or HLMPQ; if even one such pair is present, retrain the generalist on the filtered data and check whether the out-of-distribution advantage survives. As a second check, train a generalist on only single- and double-property combinations: strong performance on the five novel combinations would suggest genuine composition of property knowledge, while a collapse would suggest memorization of larger combinations.","supporting_citations":[],"review_version":1}