{"id":"ae0beb89-baec-4bf3-be1e-f451dbf165b7","arxiv_id":"2502.02810","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Mol-LLM combines SELFIES text, 2D molecular graphs, and preference optimization to build a generalist chemistry LLM that outperforms prior generalist models on most property, reaction, and translation benchmarks.","lead":"Researchers built a general-purpose AI model, Mol-LLM, that reads molecules both as text strings and as 2D graphs, and answers chemistry questions such as predicting solubility or planning reactions. They added a training step that rewards the model for using the graph correctly, and the model beats earlier generalist chemistry AI systems on most benchmarks, especially on out-of-distribution tests.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Graph-utilization evidence is circular: the GDR in Figure 1 is computed on the same MACCS perturbation distribution used to train MolPO, so it may measure artifact detection rather than generalizable chemical graph use.","rationale":"The reader's weakest assumption concerns whether the MACCS-key perturbations are genuinely worse for the target task. My concern is closely related but distinct: even if the perturbations are chemically meaningful, the chosen evaluation of graph utilization is circular because GDR is measured on the same perturbation distribution used to optimize MolPO. High GDR is therefore partly a reproduction of the training objective rather than evidence that the model uses graphs in a chemically generalizable way. The paper's benchmark results are credible and well documented, and I do not question their empirical validity. However, the central novelty of improved graph utilization lacks a non-circular measurement, and the downstream gains in Table 5 are small and not cleanly attributed to the graph pathway. Since this concern is already captured by the existing CONDITIONAL verdict and does not overturn the benchmark claims, I recommend no change to the verdict.","tokens_in":32129,"tokens_out":5388,"duration_ms":54875,"concrete_test":"After training, compute GDR on preference pairs generated by a different perturbation family (e.g., random bond deletion, atom substitution, or functional-group edits chosen to preserve the task label) using the same model. If GDR drops to roughly 0.5 on these held-out perturbation types while remaining high on MACCS-style perturbations, the graph-utilization signal is an artifact of the training perturbation distribution. Additionally, perform a test-time input ablation on the reaction and OOD benchmarks: feed Mol-LLM the same test instances but replace the graph tokens with random noise (or remove the graph pathway entirely). If the metrics change negligibly, the graph modality is not load-bearing for the reported state-of-the-art results.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central novelty is improved graph utilization via MolPO, but the only direct evidence is the graph discrimination ratio (GDR) defined in Section 3.3. GDR compares likelihoods of chosen vs rejected graphs, where rejected graphs come from the MACCS-key removal/addition procedure in Appendix B.4, exactly the same perturbation distribution used to construct MolPO training pairs in Stage 3. MolPO explicitly maximizes a preference margin over this distribution, so a high GDR on this distribution is largely a train-set reproduction effect. The perturbation scheme is task-agnostic and random: 30% of present MACCS keys are removed and absent keys are attached at random positions, with no task-specific validation that the edits actually change the target property. The model could therefore learn to detect editing artifacts (e.g., broken valences, randomly attached groups) rather than chemical semantics. The downstream gains in Table 5 are small (e.g., LogS 1.36 to 1.28 RMSE; FS-Mol-Inst. 0.907 to 0.911 EXACT) and are not isolated from the GNN pretraining, so they do not resolve the concern. Appendix A explicitly acknowledges that MolPO can latch onto spurious patterns, but no experiment tests whether this actually happens.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Mol-LLM, a multimodal generalist molecular LLM that combines a hybrid graph encoder (GINE + TokenGT), a Q-Former cross-modal projector, and a Mistral-7B backbone. The two main technical contributions are (i) Molecular structure Preference Optimization (MolPO), which trains the model to prefer an original molecular graph over a MACCS-key-perturbed graph, and (ii) a GNN pre-training scheme with functional-group prediction and SELFIES reconstruction. The authors construct a ~3.3M-example instruction-tuning dataset and two OOD benchmarks (AqSol for LogS, ORDerly for reaction prediction). The experiments report that Mol-LLM achieves state-of-the-art or comparable results among generalist molecular LLMs across property regression, property classification, reaction prediction, molecule generation, and molecule captioning, with particularly large OOD gains in reaction prediction.","tokens_in":32318,"tokens_out":5734,"duration_ms":53536,"significance":"If the claims hold, the paper is a useful empirical contribution: it demonstrates a single model covering five task groups, introduces OOD benchmarks that the community can reuse, and provides a concrete recipe (graph encoder pre-training + preference optimization) that may help future multimodal molecular LLMs. The paper is transparent about hyperparameters, training resources, and limitations, and it states that code, data, and checkpoints will be released. However, the central evidence for the paper's key novelty—'improved graph utilization'—is not conclusive: the graph discrimination ratio (GDR) is computed on the same perturbation distribution used to train MolPO, so high GDR may reflect successful fitting to a specific perturbation artifact rather than genuine chemical graph understanding. The downstream performance gains attributable to MolPO are small (Table 5) and are reported without error bars, so the significance of the headline claim is currently limited.","major_comments":[{"comment":"The graph discrimination ratio (GDR) defined in Section 3.3 is circular as evidence of graph utilization. The rejected graphs gℓ are generated by the same random MACCS-key removal/addition procedure described in Appendix B.4 that is used to construct MolPO training pairs. Since the MolPO objective in Eq. (1) explicitly increases rw,i relative to rℓ,i on this exact distribution, a high GDR on this distribution is largely a train-set reproduction effect; it does not demonstrate that the model has learned task-relevant chemical structure rather than perturbation-detection artifacts. The authors should either evaluate GDR on held-out perturbation schemes (e.g., different modification ratios, alternative substructure perturbations, or chemically meaningful mutations) or provide independent downstream evidence that MolPO improves genuine graph understanding.","section":"§3.3, Eq. GDR and Appendix B.4"},{"comment":"The limitation section explicitly acknowledges that 'if the training molecular distribution is too narrow or contains spurious patterns unrelated to the given molecular task, the model may inappropriately regard molecules in test set or OOD dataset as rejected molecules,' yet no experiment tests whether this actually occurs. This is a load-bearing concern because the MolPO hypothesis is precisely that the MACCS-perturbed graphs are task-relevant negatives. The paper should include a direct test, for example, applying MolPO with perturbation edits that preserve the target property and measuring whether the model still learns a preference, or measuring whether GDR drops when the perturbation distribution is shifted.","section":"Appendix A"},{"comment":"The paper reports no error bars or statistical significance tests (its own checklist answers 'No'), and many of the claimed improvements are very small: for example, Table 5 shows FS (SMol.) 0.598 to 0.601, T2M (ChEBI-20) 0.426 to 0.443, and LogD 0.96 to 0.91, while Table 1 shows Mol-LLM underperforming LlaSMol on SIDER (76.3 vs 78.4) and BBBP (81.1 vs 82.4). Without multiple seeds or confidence intervals, the key claim that MolPO improves performance on 'most tasks' is not statistically substantiated. The authors should report error bars over at least 3 runs for the ablation study and for the main competitive comparisons where margins are small.","section":"Checklist item 7 and Tables 1–5"}],"minor_comments":[{"comment":"The text states that 'even Mol-LLM (w/o Graph) performs on a par with the full model' on MoleculeNet tasks and attributes this to small molecular sizes; this observation undercuts the paper's emphasis on graph utilization as a general contribution and should be discussed in the introduction or conclusion.","section":"§3.2, paragraph on property tasks"},{"comment":"For molecule captioning, Mol-LLM shows lower ROUGE-L than the w/o Graph variant on both ChEBI-20 and SMolInstruct, and the explanation that it 'produces more concise captions' is post hoc; please provide supporting evidence or temper the claim.","section":"§3.3, Table 3"},{"comment":"The novelty claim that Mol-LLM is 'the first versatile generalist multimodal molecular LLM' should be qualified, since related work UniMoT and 3D-MolT5 also combine structure encoders with instruction tuning; please clarify what distinguishes Mol-LLM from these models.","section":"§1, Introduction"},{"comment":"In the MolPO objective, the margin clipping term min(rw,i − rℓ,i, λclip|rw,i|) may become negative if rw,i is negative; please clarify the intended behavior of the clipping for negative chosen rewards.","section":"§2.2, Eq. (1)"},{"comment":"The hyperparameters λmargin = 0.5, λclip = 1.0, c = 0.25, and the MACCS perturbation ratio (30%) are not ablated; a sensitivity study would help establish the robustness of the proposed method.","section":"Appendix B.4 and Table 7"},{"comment":"There are several typos and formatting issues, including 'bencharks' in the introduction, 'V ALIDITY' in Table 3, 'Heat capavity' in Table 6, 'ROGUE' in Table 13, and inconsistent spacing in Tables 1–4; these should be corrected.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is a well-engineered systems contribution with broad task coverage and a plausible OOD evaluation, but the core claim of 'improved graph utilization' rests on circular evidence. The GDR metric is computed on the same perturbation distribution used for training MolPO, and the downstream attribution to MolPO is undermined by the lack of error bars and by the small margins in Table 5. I would not reject outright—the model and benchmarks could be useful—but the authors need to provide non-circular evidence of graph utilization (e.g., held-out perturbation evaluation, a task-relevance test for the rejected graphs, or error bars that show the MolPO gains are real). The most important fix is to address the circularity and the missing significance testing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: Mol-LLM is a genuine advance in generalist molecular LLMs, with better OOD reaction and solubility results than prior generalists. The evidence for its claimed mechanism, improved graph utilization, is the paper's weakest spot: the GDR metric is partly circular.\n\nWhat's new: MolPO is a sensible adaptation of preference optimization to molecular graphs, using MACCS-key substructure removal/addition to build chosen/rejected pairs, with a task-adaptive reward margin and margin clipping. The hybrid GINE-TokenGT encoder with functional group and SELFIES reconstruction pretraining is a new combination, and the pretraining shows consistent gains over scratch in the Figure 5 learning curves. The OOD benchmarks (AqSol LogS, ORDerly scaffold-split reaction prediction) are a useful contribution, and the OOD margins are large: 0.738 vs 0.473 EXACT for retrosynthesis, 1.02 vs 1.32 RMSE for LogS. The anonymous release of code, model, and data is a plus, as is the candid Appendix A on classification degradation and spurious patterns.\n\nSoft spots: The GDR in Figure 1 measures exactly the chosen-vs-perturbed likelihood gap that MolPO is trained to maximize, using the same perturbation distribution. So high GDR is largely a train-set reproduction effect; it doesn't independently show the model uses graphs for chemistry. This weakens the mechanistic claim but doesn't break the empirical one, because the OOD task gains are the real evidence. More direct evidence (e.g., perturbations not seen in training, or ablations on perturbation type/ratio) would fix it. Second, no error bars; several improvements are tiny (0.911 vs 0.907 exact match) and could be noise. The large OOD differences are likely real, but the paper should say which differences are robust. Third, small reporting problems: lambda_margin is 0.5 in the text and 0.25 in Table 7; the final epoch was added after observing performance, without a clear validation protocol. Fourth, \"first multimodal generalist\" is an overclaim, since Table 10 itself categorizes 3D-MoLM as a generalist.\n\nVerdict: Worth a serious referee and a revision. The empirical core is plausible and useful; the graph-utilization evidence, hyperparameter consistency, and model selection need work. I'd recommend a conditional accept path.","headline":"Mol-LLM is a real empirical advance for generalist molecular LLMs, but the graph-utilization evidence is circular and the reporting needs cleanup.","tokens_in":32952,"tokens_out":2757,"would_cite":true,"duration_ms":24191,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a single multimodal molecular LLM, Mol-LLM, outperforms prior generalist molecular LLMs across property, reaction, generation, and captioning tasks by training it to prefer correct molecular graphs over perturbed ones.","keywords":["molecular language model","multimodal learning","molecular graph representation","preference optimization","instruction tuning","property prediction","reaction prediction","out-of-distribution generalization"],"falsifier":"Build a test where the graph perturbation removes or adds a substructure known to be irrelevant to the target property, and check whether the model still assigns higher likelihood to the original graph; if the graph discrimination ratio stays near 1 on such property-preserving edits, the preference signal is responding to edit artifacts, not to chemistry.","tokens_in":31831,"feed_emoji":"🧪","tokens_out":6982,"duration_ms":59238,"temperature":0.7,"pith_summary":"The paper sets out to show that a molecular large language model can be a true generalist—handling property regression, property classification, reaction prediction, molecule generation, and captioning in one model—if it is explicitly trained to use 2D molecular graphs rather than only SMILES-like strings. The authors diagnose a graph-bypass problem: naive next-token training leaves the model unable to tell a molecule apart from a perturbed version of it, so they propose Molecular structure Preference Optimization (MolPO), which trains on preferences between correct and perturbed graphs, together with a hybrid graph encoder pre-trained to predict functional groups and reconstruct SELFIES. If the central claim holds, a single instruction-tuned model can replace collections of task-specific chemistry models and generalize to new distributions better than sequence-only generalists.","feed_headline":"Single molecular LLM beats generalist rivals on five task families","feed_subtitle":"Mol-LLM adds a hybrid graph encoder and preference optimization to top sequence-only models, even out-of-distribution.","key_machinery":"The load-bearing mechanism is the MolPO objective, which defines per-task rewards $r_{w,i}$ and $r_{\\ell,i}$ from the average log-likelihood of the answer under the chosen and rejected graphs, and optimizes $-\\log\\sigma(\\min(r_{w,i}-r_{\\ell,i},\\lambda_{\\mathrm{clip}}|r_{w,i}|)-\\gamma_i)$ with a task-adaptive target reward margin $\\gamma_i$. Rejected graphs are produced by randomly removing and adding MACCS-key substructures, so the preference pairs require no task-specific perturbation design. Supporting this are a hybrid graph encoder that concatenates GINE local message-passing embeddings with TokenGT global transformer embeddings (node, edge, and graph tokens) for the Q-Former cross-modal projector, and a two-part GNN pre-training objective—functional group prediction plus SELFIES reconstruction—that teaches the encoder which substructures matter and how to preserve global structure.","core_discovery":"The central claim is that the graph-bypass problem—an LLM trained with supervised fine-tuning assigns nearly equal likelihood to a molecule and its perturbed graph—can be fixed by preference optimization over molecular structures, and that this fix is what lets a multimodal generalist outperform sequence-only generalists. The paper reports that Mol-LLM achieves state-of-the-art or comparable results on a benchmark spanning all five task groups and beats prior generalist molecular LLMs by a large margin on out-of-distribution reaction and property prediction. A graph-discrimination ratio confirms that the model now ranks the correct graph above the perturbed one on most tasks, which the authors take as evidence that the graph modality is actually being used.","pith_inferences":["Because the perturbation recipe is task-agnostic and cheap, the same MolPO idea could be applied to other multimodal molecular LLMs or to 3D conformer inputs, a direction the authors only list as future work.","A stress test with property-preserving edits would reveal whether MolPO learns chemistry or simply detects the presence of an edit; the paper does not run this control.","The graph discrimination ratio could serve as a general diagnostic for graph-conditioned LLMs, letting practitioners detect graph bypass before spending compute on full fine-tuning.","If the OOD gains hold beyond the two constructed benchmarks, preference-based graph grounding may be a cheaper route to distributional robustness than scaling sequence-only instruction data."],"forward_implications":["On most benchmark tasks, one Mol-LLM model matches or beats specialist, semi-generalist, and generalist baselines, so a single generalist can cover the five molecular task groups.","The largest gains appear on out-of-distribution reaction and solubility benchmarks, which suggests graph-aware preference training transfers to distributions not seen during instruction tuning.","The graph discrimination ratio provides a label-free check that a multimodal molecular model is actually using the graph modality, and it can be measured per task.","On the small property-classification datasets the MolPO gain largely disappears, indicating that preference optimization needs diverse molecular distributions to help."],"supporting_citations":[{"why":"Supplies the 3.3M-example instruction-tuning dataset and the strongest sequence-only generalist baseline, LlaSMol, that Mol-LLM must beat.","marker":"[5]"},{"why":"Provides the extended SELFIES vocabulary and defines the semi-generalist BioT5+ baselines for regression/reaction and classification/translation.","marker":"[6]"},{"why":"One of the two sources of reaction, generation, and captioning instruction data, and a semi-generalist baseline.","marker":"[7]"},{"why":"Establishes the Q-Former graph-text architecture that Mol-LLM builds on, and serves as a specialist baseline.","marker":"[8]"},{"why":"Supplies the GINE local message-passing graph encoder used in the hybrid encoder.","marker":"[21]"},{"why":"Supplies the TokenGT transformer graph encoder that adds global context and edge embeddings.","marker":"[22]"},{"why":"Inspires MolPO by showing that preference pairs can be built from corrupted inputs in multimodal models.","marker":"[27]"},{"why":"Provides the reference-free reward formulation and target reward margin that the MolPO objective adapts.","marker":"[28]"},{"why":"Provides the AqSol solubility data used to build the out-of-distribution LogS benchmark.","marker":"[32]"},{"why":"Provides the non-USPTO ORDerly reaction data used to build the out-of-distribution reaction benchmark.","marker":"[33]"}],"fun_headline_variants":["Mol-LLM uses preference optimization to actually exploit molecular graphs","Preference optimization makes molecular graphs matter for LLM","Graph-aware multimodal LLM tops out-of-distribution molecular tasks","Multimodal Mol-LLM fixes graph-blind LLMs via MolPO","Mol-LLM is the first multimodal generalist that truly uses graphs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"MolPO assumes that randomly removing and adding MACCS-key substructures produces rejected graphs that are genuinely worse for the target task, so the preference signal teaches task-relevant chemistry rather than artifacts of the editing procedure.","fun_headline_variants_meta":{"raw":{"variants":["Mol-LLM uses preference optimization to actually exploit molecular graphs","Preference optimization makes molecular graphs matter for LLM","Graph-aware multimodal LLM tops out-of-distribution molecular tasks","Multimodal Mol-LLM fixes graph-blind LLMs via MolPO","Mol-LLM is the first multimodal generalist that truly uses graphs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000835,"raw_usage":{"total_tokens":3643,"prompt_tokens":948,"completion_tokens":2695,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":564,"completion_tokens_details":{"reasoning_tokens":2606}},"tokens_in":564,"tokens_out":2695,"duration_ms":17642,"temperature":1.0,"reasoning_tokens":2606,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T11:01:36.387707+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a test where the graph perturbation removes or adds a substructure known to be irrelevant to the target property, and check whether the model still assigns higher likelihood to the original graph; if the graph discrimination ratio stays near 1 on such property-preserving edits, the preference signal is responding to edit artifacts, not to chemistry.","supporting_citations":[{"cited_title":"Pure transformers are powerful graph learners","cited_arxiv_id":null,"evidence_quote":"Supplies the TokenGT transformer graph encoder that adds global context and edge embeddings."},{"cited_title":"Aqsoldb, a curated reference set of aqueous solubility and 2d descriptors for a diverse set of compounds","cited_arxiv_id":null,"evidence_quote":"Provides the AqSol solubility data used to build the out-of-distribution LogS benchmark."},{"cited_title":"Wigh, Joe Arrowsmith, Alexander Pomberger, Kobi C","cited_arxiv_id":null,"evidence_quote":"Provides the non-USPTO ORDerly reaction data used to build the out-of-distribution reaction benchmark."}],"review_version":1}