{"id":"2dd7c257-d04f-4094-9013-cee20d751593","arxiv_id":"2412.06847","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"M3-20M is a new multi-modal molecular dataset with over 20 million molecules, and the paper reports improved molecule generation and property prediction with it.","lead":"This paper introduces M3-20M, a dataset of over 20 million molecules, each with SMILES, 2D graphs, 3D structures, physicochemical properties, and text descriptions. It aims to support AI-driven drug design, and experiments suggest that models using it can generate more diverse molecules and predict properties better than with smaller single-modal sets.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No deduplication described after merging PubChem, ZINC, and QM9; the 'over 20 million molecules' claim may overcount unique molecules if overlapping entries are not removed.","rationale":"The reader's weakest_assumption correctly identifies the lack of deduplication as a central risk. I agree that this is the most load-bearing concern because the paper's headline contribution is the scale of the dataset (20M molecules, 71x larger than PubChemSTM). If overlapping molecules from PubChem, ZINC, and QM9 are counted multiple times, the actual unique-molecule count shrinks and the comparison to existing datasets weakens. This is a concrete, testable issue rather than a matter of scientific taste. The paper does provide some independent support: a described pipeline, a human-expert scoring protocol for generated text, and experiments comparing multi-modal versus single-modal settings. However, none of these validate the uniqueness of the 20M count. The modality-completeness issue is related: the paper claims every molecule has SMILES, 2D graph, 3D structure, properties, and text, but for molecules absent from PubChem the source of 3D structures and properties is not described. I would not escalate the verdict beyond CONDITIONAL because the concern is addressable with dataset-level verification; the reader's conditional verdict already accounts for this uncertainty. The concrete test of canonicalizing SMILES and checking modality coverage would settle whether the scale claim is accurate.","tokens_in":16596,"tokens_out":2624,"duration_ms":28223,"concrete_test":"Download the released M3-20M files from the GitHub repository, canonicalize every SMILES with RDKit (or compute InChI keys), and count unique canonical molecules; also compute the fraction of records with non-null 3D coordinates, at least one physicochemical property value, and at least one text description. If the unique count is below 20 million by more than a few percent, or if any modality is missing from more than ~1% of records, the headline scale and comprehensiveness claims should be revised accordingly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of the paper is that M3-20M contains 'over 20 million molecules', making it 71 times larger than the largest existing multi-modal dataset (PubChemSTM, 280K). For this claim to hold, the 20M count must refer to distinct molecules, not raw records. The Methodology, Data Collection section states that molecules are collected from PubChem, ZINC, and QM9 and then processed into multiple representations, but it never describes removing duplicate molecules across these sources. PubChem and ZINC are known to overlap substantially, and QM9 (134K molecules) is largely contained within PubChem's chemical space. If the released dataset counts entries rather than unique molecules, the headline scale and the '71 times' comparison are inflated, and downstream statistics (e.g., 20,249,090 text descriptions) inherit the same overcounting. A related but distinct issue is per-molecule modality completeness: for molecules 'not present in the PubChem database', the paper does not specify how 3D coordinates and physicochemical properties are obtained, since Table 10 properties are sourced from PubChem. Thus both the uniqueness and the 'each molecule has all five modalities' assertions are load-bearing and currently unverified. The concern is not that the dataset is useless, but that the paper's strongest quantitative claim depends on a count that has not been shown to be a count of unique, fully-annotated molecules.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript presents M3-20M, a multi-modal molecule dataset claimed to contain over 20 million molecules with SMILES strings, 2D graphs, 3D structures, physicochemical properties, and textual descriptions, assembled from PubChem, ZINC, and QM9, with text augmented by GPT-3.5 and filtered by human expert scoring. The paper also constructs seven '-MM' downstream datasets (QM9-MM, MOSES-MM, BACE-MM, BBBP-MM, HIV-MM, ClinTox-MM, Tox21-MM) and reports experiments on molecule generation and property prediction using closed- and open-source LLMs and specialized generative models. The central claims are that M3-20M is 71 times larger than the largest existing multi-modal dataset (PubChemSTM) and that multi-modal training significantly boosts model performance over single-modal data.","tokens_in":16835,"tokens_out":8046,"duration_ms":72483,"significance":"If the scale and modality-completeness claims are verified, M3-20M would be a valuable open resource for molecular machine learning, and the associated '-MM' benchmark datasets would support reproducible comparison. The paper's strengths include the open-data commitment, the explicit expert-scoring protocol for generated text, the breadth of evaluated models (GPT-3.5, GPT-4, GLM4, Llama3-8b, MoFlow, D2L-OMP), and the attempt to provide multiple downstream benchmarks. However, the central claims currently rest on unverified counts and a leakage-control procedure that is not described in sufficient detail, so the contribution's practical value cannot yet be assessed.","major_comments":[{"comment":"The construction pipeline merges PubChem, ZINC, and QM9 but no deduplication step is described. PubChem and ZINC are known to overlap, and QM9 is largely contained in PubChem's chemical space; if the released 20,249,090 entries are record counts rather than unique molecules, the headline 'over 20 million molecules' and the '71 times larger than PubChemSTM' comparison are inflated. Please report the number of unique molecules (e.g., by canonical SMILES or InChIKey) after merging, give the overlap matrix among sources, and recompute all downstream counts if duplicates are found.","section":"Methodology, Data Collection"},{"comment":"The claim that each molecule has all five modalities is not currently supported. For molecules 'not present in the PubChem database', the paper does not say how 3D coordinates or physicochemical properties are obtained; 3D structure generation is described only through PubChem SDF downloads, and Table 10 shows property counts below 20M (e.g., Molecular Weight 19,174,000). Please provide per-modality coverage statistics (SMILES, graph, 3D, properties, text) and specify the procedure for generating 3D coordinates for non-PubChem molecules.","section":"Methodology, Data Collection; Table 10"},{"comment":"The description counts are internally inconsistent. The text lists three sources (direct PubChem texts, property-template conversion, and GPT-3.5), but the reported total 20,249,090 equals exactly the sum of 19,175,245 template descriptions and 1,073,845 GPT-generated descriptions, leaving no room for the 360,133 direct PubChem texts mentioned in the statistics section. In addition, 1,073,845 is 5.30% of 20,249,090, not 0.934% as stated. Please reconcile these numbers and provide a per-source breakdown of descriptions.","section":"Text Description Generation by GPT-3.5; Statistic Analyses and Results"},{"comment":"The leakage-control claim is not verifiable. The paper states that 'we mask all the target property-related information in textual descriptions during training and evaluation', but no masking procedure, list of masked tokens/fields, or code is provided. Since a large fraction of the text was generated by GPT-3.5 and GPT-3.5 is itself one of the evaluated models, the reported gains could be inflated if target properties appear in the text. Please describe the masking protocol precisely and release the masking script; ideally, evaluate on a split where no evaluated model generated the training descriptions.","section":"Molecular Property Prediction Experiments"},{"comment":"The conclusion that multi-modal data 'consistently outperform' single-modal data is not supported by the reported numbers. In Table 6, GPT-4's dipole-moment MAE is worse with SMILES+3D+text (1.27) than with SMILES only (1.17), and adding 3D coordinates alone frequently degrades performance (e.g., GLM-4 alpha: 12.82 to 14.53). In Table 8, 4 of 12 Tox21 subtasks show lower accuracy in the multi-modal setting (NR-AR-LBD, SR-ARE, SR-ATAD5, SR-p53). In Table 7, the BBBP-MM difference (0.228 vs 0.272) is accompanied by standard deviations of 0.02-0.03 and is far below typical supervised baselines for this benchmark. Please report statistical significance tests, discuss the mixed results explicitly, and moderate the wording of the central performance claim.","section":"Experiments, Tables 6-8"},{"comment":"QM9-MM is listed with only 6,830 molecules although QM9 contains 134K molecules; no filtering criterion is given. Because the molecule-generation and regression experiments are run on QM9-MM, the reader cannot tell whether the results reflect the full QM9 chemical space or a small curated subset. Please explain how the 6,830 molecules were selected and whether the same selection affects the comparison to single-modal baselines.","section":"Multi-Modal Datasets for Downstream Tasks, Table 2"}],"minor_comments":[{"comment":"The paragraph beginning 'We employ the FP-ICL method...' is duplicated verbatim; please remove one copy.","section":"Molecule Generation Experiments, Experimental Details"},{"comment":"The caption contains a typo: 'in terns of ACC mean' should be 'in terms of ACC mean'.","section":"Table 9 caption"},{"comment":"The dataset name 'PCdes' is written as both 'PCdes' and 'Pcdes'; please standardize the spelling.","section":"Tables 1 and 5 and related text"},{"comment":"The sentence 'For every 100 generated molecules, 10% are randomly selected' is imprecise; it should state explicitly that 10% of generated descriptions were sampled for expert scoring.","section":"Generated Text Quality Control by Human Expert Scoring"},{"comment":"The Data and Software Availability Statement does not contain the repository URL; please include the URL in that section rather than only in the abstract.","section":"Data and Software Availability Statement"}],"recommendation":"major_revision","confidential_remarks":"The dataset repository was not accessed during review; the authors should be asked to provide a manifest with unique-molecule counts, per-modality coverage, and masking scripts. The on-paper inconsistencies (description counts, QM9-MM size, duplicated paragraph) and the unverified leakage-control procedure need to be addressed before publication. The manuscript fits the scope of a data-report journal if construction details are made available."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a potentially useful resource, not a breakthrough. The genuinely new part is scale: pulling roughly 20 million records from PubChem, ZINC, and QM9 and attaching SMILES, 2D graphs, 3D coordinates, 26 physicochemical properties, and text (mostly templated property lists, with about 1.07M GPT-3.5 descriptions) is real work. The human-expert scoring of generated text is a good touch; 98.8% pass on a 12k stratified sample is evidence that the generated text is not garbage. The seven \"-MM\" downstream datasets are a practical contribution.\n\nThe soft spots are real and load-bearing. First, no deduplication is described after merging PubChem, ZINC, and QM9. Those sources overlap substantially, and QM9 is effectively inside PubChem. As written, the claim of \"over 20 million molecules\" is a count of records, not necessarily unique molecules, and the \"71x larger than PubChemSTM\" comparison inherits the problem. The stress-test note lands. Second, the paper promises five modalities per molecule, but for molecules not in PubChem it doesn't say where 3D structures or physicochemical properties come from; Table 10 says properties are sourced from PubChem, and the 3D pipeline is PubChem API plus RDKit. That gap matters. Third, the text corpus is mostly templated \"property X is value Y\" strings (19.17M of the 20.25M descriptions), so \"comprehensive view\" and \"textual descriptions\" are oversold; the word-frequency plot confirms the templated character.\n\nThe experiments are suggestive but not decisive. Molecule generation gains are real but modest for specialized models; the big gains are in LLM few-shot prompting, which is a narrow setup. Property prediction is mixed: several Tox21 subtasks get worse, HIV drops slightly, and the single-modal BBBP accuracy of 0.228 suggests the evaluation protocol needs scrutiny. The mask-target-property claim is plausible but no code verifies it. The dataset link is unverified.\n\nI'd send this to peer review, but with a clear demand: document deduplication, report per-modality coverage, release the construction and verification code, and fix the evaluation reporting. The dataset could become a useful community resource; right now the headline scale claim is unproven.","headline":"Useful large multimodal molecule resource, but the headline '20M unique molecules' claim and the 'five modalities per molecule' promise need verification before the scale claims are taken at face value.","tokens_in":17408,"tokens_out":2574,"would_cite":false,"duration_ms":28098,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a 20-million-molecule dataset combining SMILES, graphs, 3D structures, properties, and text improves AI molecule generation and property prediction.","keywords":["multi-modal molecule dataset","drug design","molecule generation","molecular property prediction","large language models","SMILES","3D molecular structure","dataset construction"],"falsifier":"Count unique molecules in the released dataset by canonical SMILES or InChI keys; if the union of PubChem, ZINC, and QM9 entries drops below 20 million, the scale claim weakens. Then sample, say, 1,000 entries and test whether each has loadable 2D and 3D structures, non-empty text, and property values consistent with the SMILES; any large failure rate would undercut the claim that every molecule has all five modalities.","tokens_in":16376,"feed_emoji":"🧬","tokens_out":5491,"duration_ms":48538,"temperature":0.7,"pith_summary":"The paper introduces M3-20M, an open-access dataset of more than 20 million molecules in which every molecule is accompanied by a SMILES string, a 2D molecular graph, a 3D structure, physicochemical properties, and a textual description. The authors' central claim is that this multi-modal coverage at this scale meaningfully improves AI models for drug design: using M3-20M as demonstration examples or fine-tuning data makes large language models and specialized generative models produce more valid, more diverse, and more novel molecules, and improves accuracy on molecular property prediction. The dataset is assembled by pulling molecules from PubChem, ZINC, and QM9, computing or fetching the structural and property modalities, and generating missing text with GPT-3.5 under an expert-scored quality check. If the claim holds, the dataset gives the field a common large-scale resource for training text-aware molecular models.","feed_headline":"New dataset pairs 20M molecules with text to boost drug-design AI","feed_subtitle":"M3-20M is 71 times larger than the prior best multimodal set, and LLMs generate more valid, diverse compounds with it.","key_machinery":"The central object is the M3-20M dataset itself and its construction pipeline. Molecules are collected from PubChem, ZINC, and QM9; RDKit converts them to 2D graphs and reads 3D coordinates from PubChem SDF files; 26 physicochemical properties are pulled from PubChem and templated into sentences; missing text is generated by GPT-3.5 using a CO-STAR-style expert-chemist prompt, and a human expert scoring protocol (accuracy, effectiveness, comprehensiveness, simplicity) is used to regenerate low-quality descriptions. What carries the argument is the pairing of text with structure: it gives large language models a shared format in which property names and values become readable sentences, and the experiments compare models given SMILES only versus SMILES plus 3D coordinates plus text.","core_discovery":"M3-20M is claimed to be the largest open-access multi-modal molecule dataset for AI-driven drug design, containing over 20 million molecules and 71 times the molecule count of the largest prior multi-modal dataset. The paper argues that the combination of 1D SMILES, 2D graphs, 3D coordinates, physicochemical properties, and text descriptions changes what models can learn: in experiments, few-shot prompting with multi-modal examples and LoRA fine-tuning with multi-modal data consistently beat single-modal baselines on molecule generation (validity, uniqueness, novelty, and MOSES benchmark metrics) and on property prediction for QM9, BACE, BBBP, ClinTox, HIV, and Tox21. The dataset also ships seven augmented '-MM' sub-datasets for downstream tasks and tools for 2D graph image generation and PubMed text crawling.","pith_inferences":["An implication the authors leave implicit: if the observed gains come mainly from the textual descriptions, then smaller curated multi-modal sets with high-quality text might capture most of the benefit, and the 20-million scale matters less than the text quality; a controlled ablation varying text quality while holding molecule count fixed would test this.","Because the GPT-3.5-generated descriptions cover only about 0.9 percent of the dataset, the reproducibility of the text modality depends on the prompt and scoring protocol; a natural extension would be to release the generation prompts and validator instructions alongside the data.","Because the dataset merges PubChem, ZINC, and QM9 without a described deduplication step, a conservative reading is that the unique-molecule count may be lower than 20 million; computing canonical SMILES or InChI keys over the released files would settle the exact scale."],"forward_implications":["Models trained or prompted with M3-20M should generate molecules with higher validity, uniqueness, and novelty than with single-modal datasets; the paper reports this across GPT-3.5, GPT-4, GLM4, Llama3-8b, MoFlow, and D2L-OMP.","The seven '-MM' sub-datasets provide multi-modal versions of standard benchmarks, so molecule generation and property prediction results become comparable across researchers without each group rebuilding the modality alignment.","Because each molecule has a PubChem CID and text, the dataset can support text-to-molecule and molecule-to-text tasks, including captioning, retrieval, and text-guided editing, not only the two tasks tested.","The dataset is released under GPLv3 with bi-weekly maintenance, so downstream work can build on a stable, updating resource."],"supporting_citations":[{"why":"PubChemSTM, the prior largest multi-modal dataset and the baseline for the 71-times-larger comparison.","marker":"[39]"},{"why":"QM9, a source of molecules and the basis of QM9-MM used in generation and regression experiments.","marker":"[48]"},{"why":"ZINC, one of the three source databases whose molecules are integrated into the dataset.","marker":"[28]"},{"why":"PubChem, the source of molecules, physicochemical properties, and most textual descriptions.","marker":"[30]"},{"why":"igcdata, the prior multi-modal dataset used as a comparison baseline in generation experiments.","marker":"[36]"},{"why":"The earlier structure-text molecular system whose dataset (PCdes) serves as a comparison baseline.","marker":"[65]"},{"why":"MOSES, the generation benchmark whose metrics are used, and the source of MOSES-MM.","marker":"[46]"},{"why":"MoleculeNet, whose task settings are followed for the property prediction experiments.","marker":"[58]"}],"fun_headline_variants":["20M multi-modal molecules power AI drug discovery","M3-20M: 71x larger molecule dataset for drug-design AI","Text plus 3D structure on 20M molecules boosts drug models","Largest open multi-modal molecule dataset: 20M entries","M3-20M: 20M molecules with SMILES, graphs, 3D, and text"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 20 million molecules are distinct and that each one actually has valid data in every modality; the paper does not describe removing molecules shared among PubChem, ZINC, and QM9 or verifying each entry's 3D coordinates and text, so the effective scale and completeness could be lower than the headline claim.","fun_headline_variants_meta":{"raw":{"variants":["20M multi-modal molecules power AI drug discovery","M3-20M: 71x larger molecule dataset for drug-design AI","Text plus 3D structure on 20M molecules boosts drug models","Largest open multi-modal molecule dataset: 20M entries","M3-20M: 20M molecules with SMILES, graphs, 3D, and text"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000328,"raw_usage":{"total_tokens":1866,"prompt_tokens":1012,"completion_tokens":854,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":628,"completion_tokens_details":{"reasoning_tokens":755}},"tokens_in":628,"tokens_out":854,"duration_ms":7867,"temperature":1.0,"reasoning_tokens":755,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:20:18.425578+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Count unique molecules in the released dataset by canonical SMILES or InChI keys; if the union of PubChem, ZINC, and QM9 entries drops below 20 million, the scale claim weakens. Then sample, say, 1,000 entries and test whether each has loadable 2D and 3D structures, non-empty text, and property values consistent with the SMILES; any large failure rate would undercut the claim that every molecule has all five modalities.","supporting_citations":[{"cited_title":"Multi-modal molecule structure–text model for text-based retrieval and editing","cited_arxiv_id":null,"evidence_quote":"PubChemSTM, the prior largest multi-modal dataset and the baseline for the 71-times-larger comparison."},{"cited_title":"Quantum chemistry structures and properties of 134 kilo molecules","cited_arxiv_id":null,"evidence_quote":"QM9, a source of molecules and the basis of QM9-MM used in generation and regression experiments."},{"cited_title":"Zinc- a free database of commercially available compounds for virtual screening","cited_arxiv_id":null,"evidence_quote":"ZINC, one of the three source databases whose molecules are integrated into the dataset."},{"cited_title":"Pubchem substance and compound databases","cited_arxiv_id":null,"evidence_quote":"PubChem, the source of molecules, physicochemical properties, and most textual descriptions."},{"cited_title":"Git- mol: A multi-modal large language model for molecular science with graph, image, and text","cited_arxiv_id":null,"evidence_quote":"igcdata, the prior multi-modal dataset used as a comparison baseline in generation experiments."},{"cited_title":"A deep-learning system bridging molecule structure and biomedical text with comprehension comparable to human professionals","cited_arxiv_id":null,"evidence_quote":"The earlier structure-text molecular system whose dataset (PCdes) serves as a comparison baseline."},{"cited_title":"Molecular sets (moses): a benchmarking platform for molecular generation models","cited_arxiv_id":null,"evidence_quote":"MOSES, the generation benchmark whose metrics are used, and the source of MOSES-MM."},{"cited_title":"Moleculenet: a benchmark for molecular machine learning","cited_arxiv_id":null,"evidence_quote":"MoleculeNet, whose task settings are followed for the property prediction experiments."}],"review_version":1}