{"id":"9eed89ba-a76b-48b1-a8ad-7af6f01c6f0f","arxiv_id":"2506.00880","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A modular LLM-centric framework for molecular relational learning that supports 1D, 2D, and 3D molecular inputs and flexible model assembly, benchmarked across DDI, SSI, and CSI tasks.","lead":"ModuLM is a modular software framework for building and comparing molecular relational learning models that combine large language models with 2D graph, 3D conformation, and text inputs. It assembles dozens of encoders, interaction layers, and LLM backbones into thousands of configurations, and it benchmarks them on drug interaction and solvation tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Multimodal/3D improvement claims are drawn from comparisons that change encoder and modality simultaneously; the framework is sound but the component-level guidance is not yet established.","rationale":"The core engineering contribution—a modular framework supporting 1D/2D/3D inputs and dynamic assembly—is credible and supported by the implemented code structure and the reported ability to build a custom model with user-defined encoders and ablation toggles. The stress-test concern is not with the framework's existence but with the causal story told in Section 4.2. The paper claims that multimodal inputs (specifically 3D conformations) improve performance and that interaction layers consistently help; these are the claims a user would rely on to select components. Yet the experimental design does not isolate the factors: 1.4 vs 1.2 changes both encoder (Uni-mol vs GIN) and modality (3D vs 2D), and most interaction-layer comparisons have no matched baseline. The one genuinely controlled pair (1.2 vs 1.3) is insufficient to support 'consistently improves' across backbones and modalities. My recommended minimal factorial ablation would settle whether the 3D advantage is an artifact of Uni-mol's strong pretraining or a genuine modality effect, and whether the interaction-layer gain survives controlled toggles. Since the authors' own Limitations section limits the scope of the experiments, the appropriate verdict remains conditional: accept the framework contribution, but require the controlled ablations and a pinned code release (with commit hash) before the empirical guidance is taken at face value. I therefore leave the reader's CONDITIONAL verdict unchanged.","tokens_in":22141,"tokens_out":8767,"duration_ms":82733,"concrete_test":"Run a minimal factorial ablation on ChChMiner (DDI) and FreeSolv (SSI) using the released framework: fix backbone = Galactica-1.3B and interaction = None, then evaluate GIN, GAT, MPNN, and AttentiveFP on 2D and Uni-mol, EGNN, and SchNet on 3D with identical LoRA rank (r=16), epochs, batch size, and the same five seeds as the paper. Then, for each of these encoders, toggle Cross Attention on/off to obtain a matched interaction baseline; and for Uni-mol only, sweep DeepSeek-1.5B versus DeepSeek-14B. If the top 2D configuration is within one standard deviation of the top 3D configuration, or if the Cross Attention delta is not consistently positive with matched baselines, the paper's component-level claims must be weakened from '3D conformations and interaction layers improve MRL' to 'specific pretrained encoders and interaction layers improve it in the tested configurations.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2 ('Impact of Input Data and Encoders') attributes performance gains to input modality using configurations {1.1,1.2,1.4}, {1.6,1.7,1.8}, and {1.12,1.13} from Table 3, but each pair that differs in modality also differs in encoder: 1.2 uses GIN on 2D while 1.4 uses Uni-mol on 3D, 1.7 uses GIN on 2D while 1.8 uses Uni-mol on 3D, and 1.13 uses CoATGIN on 2D. The consistent superiority of the 3D configurations could be an encoder effect—Uni-mol is a heavily pretrained 3D transformer, whereas GIN and CoATGIN are lightweight 2D GNNs—rather than evidence that 3D conformations per se help. The 'Impact of Interaction Layers' analysis has one controlled comparison (1.2 vs 1.3, GIN with and without Cross Attention) but generalizes using 1.5, 1.9, 1.11, 1.14, and 1.15, none of which has a matched no-interaction baseline with the same encoder, backbone, and modality. The backbone-size claim is partly controlled (1.8 vs 1.10 both use Uni-mol on 3D, DeepSeek-1.5B vs 14B), but the CSI results in Table 12 are post-hoc selected per backbone. Because the manuscript's own Limitations section disclaims exhaustive exploration, the framework contribution survives; what fails is the causal attribution in the component-level analyses.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ModuLM, a modular framework for building and benchmarking LLM-based molecular relational learning (MRL) models. The framework supports 1D SMILES/SelfIES, 2D molecular graphs, and 3D conformations; provides 8 graph encoders, 11 conformation encoders, 7 interaction layers, and 7 LLM backbones; and offers configurable alignment modules and prompts, producing a large space of model configurations. The authors report experiments on DDI, SSI, and CSI datasets, comparing 15 example configurations with existing baselines such as MolTC, and additionally construct a custom model that integrates a redesigned Uni-mol encoder, an interaction layer, and chain-of-thought prompting. The paper includes an anonymous code link, standard public datasets, and five-repeat means with standard deviations.","tokens_in":22418,"tokens_out":5663,"duration_ms":52226,"significance":"If the empirical claims held, ModuLM would fill a real gap: a unified, extensible benchmarking framework that decouples molecular input modality, encoder choice, interaction layer, and LLM backbone for MRL. The breadth of components and the reported ease of dynamic configuration are valuable for the community, and the use of standard public datasets with repeated runs supports reproducibility. However, the component-level conclusions in Section 4.2 are currently drawn from comparisons that change several design dimensions at once, so the paper's central empirical guidance is weaker than the text suggests. The framework contribution itself remains defensible; what needs revision is the attribution of performance gains to individual components.","major_comments":[{"comment":"The claim that 3D conformations improve performance is not supported by controlled comparisons. The sets {1.1, 1.2, 1.4}, {1.6, 1.7, 1.8}, and {1.12, 1.13} (with 1.14 needed for a 3D LLaMA comparison) change both the input modality and the encoder simultaneously: 1.2 uses GIN on 2D while 1.4 uses Uni-mol on 3D, and 1.7 uses GIN on 2D while 1.8 uses Uni-mol on 3D. Uni-mol is a heavily pretrained 3D transformer while GIN and CoATGIN are lightweight 2D GNNs, so the observed gains could be an encoder effect rather than evidence that 3D input per se helps. In fact, Table 4 does not consistently favor 1.4 over 1.2: on ChChMiner AUC-ROC the values are 0.955 and 0.956, essentially tied. The authors should either add matched comparisons that vary only the modality (same encoder and backbone) or substantially soften the claim that 3D conformations are responsible for the improvements.","section":"Section 4.2, Table 3"},{"comment":"The claim that adding interaction layers consistently improves performance is not established by the presented evidence. Among the configurations listed (1.3, 1.5, 1.9, 1.11, 1.14, 1.15), only 1.2 versus 1.3 is a controlled comparison with the same backbone, encoder, and modality. All other interaction-layer configurations lack a matched no-interaction baseline: for example, 1.14 (LLaMA-1B, EGNN, Gated Fusion, 3D) differs from 1.13 (LLaMA-1B, CoATGIN, no interaction, 2D) in encoder, modality, and interaction layer simultaneously. The sentence \"We first conduct experiments with various non-interaction designs\" is also confusing because every listed configuration includes an interaction layer. The authors should report matched ablations, or restrict the conclusion to the single controlled pair.","section":"Section 4.2, \"Impact of Interaction Layers\""},{"comment":"The CSI comparison reports only the best-performing configuration for each backbone, selected post hoc, which biases the comparison in favor of ModuLM and prevents a fixed-configuration assessment. The custom-model comparison also relies on differences that are often within overlapping error bars: for example, ChChMiner accuracy is 0.968 +/- 0.006 for the custom model versus 0.964 +/- 0.005 for configuration 1.8, and the custom model’s Lifetime RMSE of 1.123 +/- 0.082 is worse in mean than 1.8’s 1.119 +/- 0.077. The paper should either provide paired significance tests across the five repeats for the key comparisons or explicitly state which differences are not statistically significant.","section":"Table 12 and Section 4.3"}],"minor_comments":[{"comment":"The claim of \"over 50,000 distinct model configurations\" is not derived from the listed component counts: 8 graph encoders x 11 conformation encoders x 7 interaction layers x 7 backbones equals 4,312 combinations, and even multiplying by the two alignment options and two prompt types gives 17,248. Please provide the counting rule or revise the number.","section":"Abstract and Section 1"},{"comment":"The pretraining prompt template contains typos: \"substructe0\" and \"substructe1\" should be \"substructure0\" and \"substructure1\".","section":"Table 1"},{"comment":"The text says pretraining data are collected from PubChem and DrugBank, but reference [28] is the PubChem 2023 update rather than a DrugBank reference; please check the citation mapping.","section":"Section 3.2 and References"},{"comment":"The statement that all main-text backbones use the Structure Similarity-guided Grouping pretraining approach should be reconciled with Section 3.2, which presents three pretraining strategies and also mentions additional Q-former pretraining; please clarify which pretraining is applied to each reported configuration and to each baseline.","section":"Section 4.1"},{"comment":"In the example JSON configurations, \"max_epochs\" and \"weight_decay\" are quoted as strings (\"30\", \"0.05\") while neighboring fields are numeric; this is likely an artifact of the presentation but should be made consistent to avoid confusion for users.","section":"Appendix A.4"},{"comment":"For the Lifetime column, the model 1.8 has RMSE 1.119 +/- 0.077 while the custom model has RMSE 1.123 +/- 0.082; the text elsewhere presents the custom model as uniformly better, so this row should be discussed explicitly.","section":"Table 12"}],"recommendation":"major_revision","confidential_remarks":"The framework contribution is credible and the code release is a positive element, but the empirical component-level claims need tightening before publication. In particular, the authors should either run controlled ablations that change one design dimension at a time or replace the causal attributions with descriptive statements about configurations. The post-hoc best-per-backbone CSI reporting should also be replaced with a pre-specified comparison. These are fixable within the manuscript's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The real contribution here is the framework: ModuLM is the first LLM-based MRL toolkit that takes 1D, 2D, and 3D molecular inputs and lets you swap encoders, interaction layers, and backbones dynamically. That is a genuinely useful piece of engineering, and the design looks honest — the modular interface, the JSON configuration, the range of supported components. The benchmark effort is also above average for a framework paper: five repeats, error bars, standard public datasets, and more configurations than most people bother to run. If you work in this area, this is a tool you would actually want to try.\n\nThe soft spots are real but localized. Section 4.2's \"impact of input data\" comparisons change encoder and modality at the same time: 1.2 is GIN on 2D, 1.4 is Uni-mol on 3D, and similarly for 1.7 vs 1.8. So the conclusion that 3D conformations help is not supported as stated — it could just be that Uni-mol is a stronger encoder than GIN. The interaction-layer analysis has only one controlled comparison (1.2 vs 1.3); the other configurations lack a matched no-interaction baseline with the same encoder, backbone, and modality. The backbone-size claim (1.8 vs 1.10) is actually controlled, but the CSI results in Table 12 are best-per-backbone selections made after the fact, which weakens any cross-backbone comparison. The paper's own Limitations section explicitly disclaims exhaustive exploration, and that does protect the framework contribution, but it does not protect the causal attributions in Section 4.2, which are stated as if they were established. I also note the anonymous code link has no commit hash, which makes reproducibility auditing harder, and the incremental pretraining used for ModuLM configs is not applied to the baselines, so the headline gains over MolTC could be partly a pretraining effect rather than an architecture effect.\n\nNone of this invalidates the framework itself. The modular design, the breadth of components, and the evaluation infrastructure are solid. What is not yet established is which components actually drive performance. This paper is for anyone who wants to build or benchmark LLM-based MRL models — it gives you a working, extensible platform and a set of initial measurements that should be treated as suggestive, not conclusive.\n\nI would send this to peer review. The framework deserves a place in the literature, but the revision needs controlled ablations that vary one component at a time, a pinned code release, and a clear statement about whether baselines received the same pretraining. With those changes, the empirical claims would match the level of the engineering contribution.","headline":"A genuinely modular MRL framework worth knowing, but the component-level empirical claims are confounded and need controlled ablations before they should be quoted.","tokens_in":23009,"tokens_out":1856,"would_cite":true,"duration_ms":19856,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ModuLM is a unified, extensible framework for LLM-based molecular relational learning that accepts 1D SMILES, 2D graphs, and 3D conformations, and its benchmarks argue that multimodal, 3D-aware configurations outperform established…","keywords":["molecular relational learning","large language models","multimodal molecular representation","3D conformations","modular framework","drug-drug interaction","solvation free energy","benchmarking"],"falsifier":"Hold the LLM backbone and interaction layer fixed and compare a 2D graph encoder against a 3D conformation encoder on the same DDI and SSI datasets; if the 3D configuration does not consistently win, the paper's claim that 3D conformations drive the gains is falsified.","tokens_in":21912,"feed_emoji":"🧪","tokens_out":7295,"duration_ms":64278,"temperature":0.7,"pith_summary":"ModuLM is a proposed framework for molecular relational learning (MRL) that lets a single codebase build LLM-based models from 1D SMILES strings, 2D molecular graphs, or 3D conformations. Its central contribution is a modular assembly mechanism: eight 2D graph encoders, eleven 3D conformation encoders, seven interaction layers, and seven LLM backbones can be combined into over 50,000 model configurations with shared pretraining, prompts, and evaluation metrics. The benchmark results are used to argue that multimodal inputs improve LLM-based MRL, that 3D conformation inputs give the best results, that explicit interaction layers help, and that smaller backbones fine-tune to these tasks better than larger ones. A custom configuration built on Uni-Mol and DeepSeek-1.5B with chain-of-thought prompting outperforms MolTC on the evaluated DDI, SSI, and CSI datasets. The value of the paper is a standardized way to compare and build models in a space where methods currently differ in encoder, modality, and backbone all at once.","feed_headline":"3D conformations push LLM molecule models past established baselines","feed_subtitle":"ModuLM swaps eight graph encoders, eleven 3D encoders, seven interaction layers, and seven backbones in one framework.","key_machinery":"The central mechanism is ModuLM's dynamic model-assembly interface: a configuration-driven pipeline that takes a molecule in 1D, 2D, or 3D form, encodes it with a swappable graph or conformation encoder, optionally combines the two molecules' features through an interaction layer, aligns the fused features into the LLM token space with an MLP or Q-Former, and feeds the result to one of seven backbones under direct or chain-of-thought prompts. This machinery carries the argument because it turns model comparison from a reimplementation problem into a configuration problem, and it is what generates the claimed 50,000-plus distinct model configurations. Three pretraining strategies, namely molecular-interaction-based, substructure-based, and structure-similarity-guided grouping, are also part of the machinery and give all assembled models a shared initialization before fine-tuning.","core_discovery":"ModuLM is, to the authors' knowledge, the first LLM-based MRL framework that supports flexible molecular input formats and dynamic architectural switching. It provides a suite of modular components, including 8 types of 2D molecular graph encoders, 11 types of 3D molecular conformation encoders, 7 interaction layers, 7 LLM backbones, MLP and Q-Former alignment, multiple pretraining strategies, and direct or chain-of-thought prompts, from which over 50,000 model configurations can be assembled. The experiments construct 15 configurations and compare them against ChemT5, MolCA, MolT5, and MolTC on drug-drug interaction, solute-solvent interaction, and chromophore-solvent interaction datasets. The results are used to conclude that adding 2D or 3D structural information improves performance over text-only inputs, that 3D conformations yield the best results, that interaction layers consistently add value, and that smaller LLM backbones adapt better during fine-tuning than larger ones. The paper further shows a user-defined model, built by extending the best 3D configuration with an intra-molecular attention encoder and chain-of-thought prompting, exceeding MolTC on all tested datasets.","pith_inferences":["Beyond the paper: the framework's configuration space is large enough that the same machinery could be used for automated search or Bayesian optimization over molecular encoders and backbones, turning the reported 50,000 configurations from a capacity claim into a search space.","Beyond the paper: the observation that larger LLMs generalize but underperform after fine-tuning points to a capacity-adaptation trade-off that could be studied with scaling curves across more backbone sizes.","Beyond the paper: a single-variable ablation that keeps the encoder family and interaction layer fixed while toggling only 2D versus 3D input would separate the benefit of 3D geometry from the benefit of a stronger encoder, which the current tables are not set up to answer."],"forward_implications":["If ModuLM works as claimed, LLM-based MRL benchmarking can be standardized: the same pipeline, prompts, splits, and metrics apply across encoders and backbones, reducing redundant coding.","The reported 3D advantage implies that geometry-sensitive tasks such as drug-drug interaction and solvation should incorporate 3D conformations rather than relying on SMILES or 2D graphs alone.","The consistent gains from interaction layers imply that future LLM-based MRL models should explicitly model the pair relationship instead of concatenating molecular representations.","The finding that smaller backbones fine-tune better suggests that model scale is not the main driver for these tasks and that task-specific adaptation matters more.","The custom model's results imply that component-level improvements, including a custom encoder, an interaction design, and chain-of-thought prompting, can compound to beat existing methods on these benchmarks."],"supporting_citations":[{"why":"Supplies the multimodal pretraining method, the datasets, and the state-of-the-art baseline that ModuLM's custom model must beat.","marker":"[13]"},{"why":"Provides the Uni-Mol 3D encoder used in the best-performing and in the custom model configurations.","marker":"[80]"},{"why":"Provides the DeepSeek backbone series, whose smaller variants fine-tune best in the benchmarks.","marker":"[2]"},{"why":"An earlier MRL toolkit whose encoder and interaction-layer design motivates ModuLM's modular component suite.","marker":"[57]"},{"why":"Defines the standard metrics, such as MSE, RMSE, MAE, and AUC-ROC, that ModuLM uses for evaluation.","marker":"[22]"},{"why":"Serves as the MolCA baseline compared on DDI and SSI datasets.","marker":"[41]"},{"why":"Serves as the MolT5 baseline compared on DDI and SSI datasets.","marker":"[11]"},{"why":"Serves as the ChemT5 baseline compared on DDI and SSI datasets.","marker":"[4]"},{"why":"Provides the Galactica backbones used in configurations 1.1 through 1.5.","marker":"[60]"}],"fun_headline_variants":["First modular LLM framework for molecular relational learning","50,000+ model configurations in one flexible molecule framework","3D conformations and interaction layers push LLM baselines","Swap-and-play modules enable 50k LLM molecule models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The empirical conclusion that 3D conformations and interaction layers improve performance assumes each comparison changes only the component being tested, but several configurations in the paper's tables change the encoder, the input modality, and the backbone at the same time.","fun_headline_variants_meta":{"raw":{"variants":["First modular LLM framework for molecular relational learning","50,000+ model configurations in one flexible molecule framework","3D conformations and interaction layers push LLM baselines","Swap-and-play modules enable 50k LLM molecule models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000457,"raw_usage":{"total_tokens":2330,"prompt_tokens":1020,"completion_tokens":1310,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":636,"completion_tokens_details":{"reasoning_tokens":1243}},"tokens_in":636,"tokens_out":1310,"duration_ms":9355,"temperature":1.0,"reasoning_tokens":1243,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:54:56.284414+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hold the LLM backbone and interaction layer fixed and compare a 2D graph encoder against a 3D conformation encoder on the same DDI and SSI datasets; if the 3D configuration does not consistently win, the paper's claim that 3D conformations drive the gains is falsified.","supporting_citations":[{"cited_title":"Uni-mol: A universal 3d molecular representation learning framework","cited_arxiv_id":null,"evidence_quote":"Provides the Uni-Mol 3D encoder used in the best-performing and in the custom model configurations."},{"cited_title":"Flexmol: A flexible toolkit for benchmarking molecular relational learning","cited_arxiv_id":null,"evidence_quote":"An earlier MRL toolkit whose encoder and interaction-layer design motivates ModuLM's modular component suite."},{"cited_title":"Coley, Cao Xiao, Jimeng Sun, and Marinka Zitnik","cited_arxiv_id":null,"evidence_quote":"Defines the standard metrics, such as MSE, RMSE, MAE, and AUC-ROC, that ModuLM uses for evaluation."}],"review_version":1}