{"id":"04c12d86-9c86-4893-a7e9-c26ed5e7297a","arxiv_id":"2502.09571","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"DiffMS, a formula-constrained discrete graph diffusion model conditioned on mass spectra, achieves state-of-the-art de novo molecule generation on NPLIB1 and MassSpecGym.","lead":"DiffMS generates candidate molecular structures from mass spectrometry data using a graph diffusion model constrained by a known chemical formula, and reports state-of-the-art accuracy on two benchmarks. The method's pretraining on millions of fingerprint-structure pairs points to a scaling path for structure elucidation of unknown metabolites.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Decoder pretraining corpus may contain near-identical analogues of benchmark test structures; exact test-molecule removal is insufficient to rule out memorization-based inflation of DiffMS's SOTA results.","rationale":"The reader's weakest assumption correctly identifies the exact-removal-only deduplication as the most load-bearing point, and I find this concern well-founded. What must be true for the central SOTA claim is that the 2.8M fingerprint-structure pretraining corpus teaches a generalizable decoder, not that it contains test-like structures which the decoder can memorize. The paper's own discussion of NPLIB1 near-duplicate train/test pairs (Tanimoto > 0.85) makes it highly plausible that natural-product libraries such as COCONUT and HMDB contain analogues of NPLIB1 test molecules; removing only exact matches does not address this. For MassSpecGym, the benchmark's OOD split is defined relative to the benchmark training set, not relative to the external pretraining libraries, so the pretraining corpus could still contain close analogues of MassSpecGym test molecules. This concern is concrete, testable, and central to the novelty and fairness of the comparison. I do not think it requires rejection: the method is well-ablated, the code is public, and the ablation showing scaling with pretraining size is consistent with a genuine learning signal. But the SOTA claim should be conditioned on a quantitative demonstration that the test sets are also far from the pretraining corpus, or on re-evaluation using a clean subset. Since the reader's CONDITIONAL verdict already captures this uncertainty, no verdict change is needed.","tokens_in":21142,"tokens_out":4722,"duration_ms":45454,"concrete_test":"Compute the maximum pairwise Tanimoto similarity and MCES between every NPLIB1 and MassSpecGym test molecule and the 2.8M pretraining molecules after exact removal; report the distribution and count of test molecules with any pretraining neighbor at Tanimoto > 0.85 or MCES < 10, the same near-duplicate thresholds the paper and MassSpecGym use. Then re-run DiffMS evaluation on the subset of test molecules with no such close pretraining neighbor. If accuracy or Tanimoto on this clean subset drops materially (e.g., NPLIB1 top-1 accuracy falls from 8.34% toward the 5.40% baseline) or the MassSpecGym margin narrows, the pretraining corpus leaks near-duplicate test information and the SOTA claim must be reported conditional on a more aggressive deduplication.","verdict_should_be":"UNCHANGED","load_bearing_attack":"DiffMS's central claim depends on the decoder's fingerprint-to-structure pretraining teaching generalizable chemistry rather than memorizing test-like structures. Section 3.3 removes only exact NPLIB1 and MassSpecGym test/validation molecules from the 2.8M pretraining set, but the pretraining libraries (DSSTox, HMDB, COCONUT, MOSES) are large collections that plausibly contain near-identical analogues of benchmark test molecules. The paper itself notes that NPLIB1 contains cross-split pairs with Tanimoto similarity above 0.85; if such analogues appear in the pretraining corpus, the decoder has effectively seen essentially the same fingerprint-to-adjacency mapping at scale, while most baselines (e.g., MADGEN, Spec2Mol) do not use this pretraining corpus. Exact string removal does not exclude 0.85-similar neighbors. This makes the MassSpecGym 'out-of-distribution' claim unproven and could inflate the reported margins (8.34% vs 5.40% on NPLIB1, 2.30% vs 1.31% on MassSpecGym). The MIST+MSNovelist baseline does use the same pretraining corpus and still loses, so the diffusion decoder may extract more from memorized neighbors; but the comparison to baselines without the corpus remains potentially unfair. This is a testable, substantive concern about whether DiffMS achieves genuine de novo generation or partially retrieves structures from pretraining memory.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents DiffMS, a two-module generative model for predicting molecular structures from tandem mass spectra. A spectrum encoder based on the MIST formula transformer embeds MS/MS peaks and predicted peak formulae, and a discrete graph diffusion decoder (built on DiGress) generates an adjacency matrix under a fixed heavy-atom composition; the decoder is pretrained on 2.8M fingerprint-structure pairs and the encoder on spectrum-to-fingerprint prediction, after which the system is finetuned end-to-end. On NPLIB1 and MassSpecGym, DiffMS is reported to achieve the best top-1/top-10 accuracy, MCES, and Tanimoto metrics among the compared baselines. The paper also reports ablations for encoder pretraining, decoder pretraining scale, diffusion prior, and inferred formulas.","tokens_in":21459,"tokens_out":4012,"duration_ms":36425,"significance":"If the reported results are robust, DiffMS is a meaningful advance for de novo structure elucidation: it is the first formula-constrained diffusion generator for MS/MS, it demonstrates a practical pretraining strategy for the decoder using abundant fingerprint-structure pairs, and it reports consistent scaling with pretraining set size. The empirical evaluation is transparent and includes several ablations, and the code is publicly released. The main caveats are that the out-of-distribution claim on MassSpecGym depends on a leakage-control assumption that is not fully verified, and the headline margins over the strongest baselines are small and reported without variance estimates.","major_comments":[{"comment":"The statement that the authors 'remove all NPLIB1 and MassSpecGym test and validation molecules from our decoder pretraining dataset' only excludes exact matches; it does not exclude near-identical analogues. The paper itself notes in Section 4.3 that NPLIB1 contains train/test pairs with Tanimoto similarity above 0.85, and the pretraining libraries (DSSTox, HMDB, COCONUT, MOSES) are large enough that similar analogues of MassSpecGym test molecules are likely present. Because the decoder is conditioned on Morgan fingerprints, a near-identical analogue seen during pretraining can teach essentially the same fingerprint-to-structure mapping as the test molecule, which would inflate the MassSpecGym 'out-of-distribution' results and make the comparison against MADGEN and Spec2Mol potentially unfair. I ask for a quantitative nearest-neighbor analysis between the pretraining corpus and the benchmark test sets (e.g., maximum and percentile Tanimoto and MCES values), and an ablation in which all pretraining molecules within a threshold (for example Tanimoto > 0.85 or MCES < 10) are removed before retraining the decoder.","section":"Section 3.3"},{"comment":"The description 'We pretrain our encoder on the same datasets used for fine-tuning (i.e., NPLIB1 (CANOPUS) or MassSpecGym)' is ambiguous: if the encoder pretraining includes test-set spectra, the reported end-to-end accuracies would be circular. Please state explicitly that only the training splits of NPLIB1 and MassSpecGym are used for encoder pretraining, and confirm that the finetuning and evaluation splits coincide with the benchmark-defined train/test splits.","section":"Section 3.3"},{"comment":"The headline comparisons in Table 1 are reported as point estimates without standard deviations or confidence intervals. This matters because sampling 100 molecules per spectrum is stochastic and the adopted ranking step is based on frequency, so the top-1 accuracy itself has sampling variance; on MassSpecGym the gap over MADGEN is only 2.30% versus 1.31%, which is small relative to plausible sampling noise. Please report mean plus/minus standard deviation over multiple sampling seeds, and ideally over multiple finetuning seeds, for the main metrics.","section":"Table 1"}],"minor_comments":[{"comment":"The ranking procedure (sampling 100 molecules, removing invalid molecules, and taking the most frequent structures) should be described precisely, including how ties are broken and whether invalid or duplicate samples are counted when computing frequencies; the statement that the post-processing is also applied to baseline methods should clarify how each baseline's native ranking is overridden.","section":"Section 4.1"},{"comment":"The paper states that DiffMS models only heavy atoms and that generated molecules may differ from the true molecule in hydrogen count; please clarify how hydrogen atoms are inferred (or ignored) during training and evaluation, since this affects the definition of exact top-1 matches.","section":"Section 3.1"},{"comment":"Minor formatting: the entry '11.04 %' contains an unusual space, and the table would benefit from a note stating the number of sampling seeds used for all stochastic methods.","section":"Table 1"},{"comment":"For the MIST-CF formula inference ablation, the paper should state what happens when the true formula is not among the top-5 predicted formulae, since the 100 samples are then necessarily generated under incorrect heavy-atom compositions.","section":"Section 4.5"}],"recommendation":"major_revision","confidential_remarks":"The near-neighbor leakage concern is the main risk to the paper's central claim, and it is testable; if the authors provide the nearest-neighbor analysis and the corresponding ablation, the MassSpecGym out-of-distribution claim can be evaluated on solid ground. The encoder pretraining split ambiguity should also be resolved explicitly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth a serious look. DiffMS combines the MIST spectrum encoder with a DiGress-style graph diffusion decoder under a formula constraint, and adds two pretraining stages. That combination is new, and it produces the best published numbers on NPLIB1 and MassSpecGym for de novo structure elucidation from tandem mass spectra. The ablations are honest and useful: encoder pretraining roughly doubles top-1 accuracy on NPLIB1, decoder pretraining scales with dataset size, and the formula-inference ablation shows the method degrades gracefully when the true formula is replaced by MIST-CF predictions. Code is public. These are real strengths.\n\nThe main soft spot is the leakage-control claim. The authors remove exact NPLIB1 and MassSpecGym test and validation molecules from their 2.8M fingerprint-structure pretraining corpus, and describe the evaluation as \"truly novel structures.\" But the pretraining libraries (DSSTox, HMDB, COCONUT, MOSES) are large chemical collections that plausibly contain near-identical analogues of benchmark test molecules. The paper itself notes that NPLIB1 suffers from cross-split Tanimoto similarity above 0.85; the same reasoning applies to the pretraining corpus. Exact removal does not rule out memorization of near neighbours. This is testable: the authors could report a Tanimoto/MCES similarity histogram between the pretraining libraries and the test sets, or retrain with a similarity-based exclusion. Until then, the out-of-distribution claim on MassSpecGym is overstated. I don't think it fully explains the results — the re-implemented MIST+MSNovelist baseline uses the same corpus and still loses to DiffMS — but the comparison to MADGEN and Spec2Mol, which lack the corpus, could indeed be inflated.\n\nA second, milder issue: Table 1 has no error bars. Sampling 100 molecules per spectrum is stochastic, and the MassSpecGym top-1 gap over MADGEN is about one percentage point. Three seeds would settle whether that margin is real.\n\nThe re-implemented baselines are a practical necessity given the original codebases, and the paper is transparent about it. That's acceptable but worth flagging in any revision.\n\nWho is this for? Researchers in metabolomics and ML-for-chemistry who need a strong generative baseline; the ablations also make it useful as a design study for pretraining-conditioned diffusion. I'd bring it to our reading group and cite it in related work. It deserves a proper peer review, with the leakage analysis and error bars as the main asks before acceptance.","headline":"A credible SOTA result in MS-to-structure generation, with a leakage-control gap in the pretraining pipeline that is testable and fixable.","tokens_in":21988,"tokens_out":2295,"would_cite":true,"duration_ms":25353,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DiffMS, a formula-restricted graph diffusion model, outperforms all tested baselines at generating molecular structures from mass spectra.","keywords":["mass spectrometry","de novo structure elucidation","diffusion models","molecular graph generation","formula-constrained generation","fingerprint-structure pretraining","transformer encoder","MS/MS"],"falsifier":"Compute, for every MassSpecGym test molecule, the maximum Tanimoto similarity (or minimum MCES) to any molecule in the 2.8M pretraining set; if a substantial fraction of test molecules have a near neighbour there, say Tanimoto above 0.85 or MCES below 10, the claim that evaluation represents truly novel structures collapses and the comparison to baselines trained without that library would need to be redone.","tokens_in":20955,"feed_emoji":"🧪","tokens_out":9608,"duration_ms":70724,"temperature":0.7,"pith_summary":"DiffMS aims to make de novo structure elucidation from tandem mass spectra practical by generating molecular graphs directly, rather than writing SMILES strings. It takes the chemical formula as given, so the graph decoder only needs to place bonds among a fixed set of heavy atoms, and it conditions that decoder on a spectrum embedding produced by a formula transformer. The paper claims this architecture, trained end-to-end after separate pretraining of encoder and decoder, outperforms every tested baseline on both established benchmarks, with top-1 accuracy of 8.34% versus 5.40% on NPLIB1 and 2.30% versus 1.31% on MassSpecGym. A sympathetic reader would care because exact structure identification is the bottleneck in metabolomics; even when the exact molecule is missed, DiffMS reports the highest rates of structurally meaningful and close Tanimoto matches among methods that report them.","feed_headline":"Diffusion model tops de novo structure elucidation from mass spectra","feed_subtitle":"Formula-constrained graph diffusion plus 2.8M-pair pretraining reaches 8.34% NPLIB1 and 2.30% MassSpecGym top-1.","key_machinery":"The load-bearing mechanism is formula-restricted discrete graph diffusion on the one-hot bond-type adjacency matrix, adapted from the DiGress discrete denoising diffusion framework for graphs. Given the heavy-atom count and types from the chemical formula, the decoder fixes node features and learns to denoise edge types through a Markov transition process with a marginal bond-type prior and cosine noise schedule; training minimizes cross-entropy between predicted and true upper-triangular adjacency entries. The condition is a spectrum embedding from a formula transformer that models each peak's predicted chemical formula and pairwise neutral losses, and decoder pretraining replaces that embedding with the molecule's Morgan fingerprint, exposing the decoder to 2.8M fingerprint-structure pairs before end-to-end finetuning. This design makes generation permutation-invariant, enforces the formula constraint exactly, and guarantees valid graphs by construction.","core_discovery":"The central claim is that conditional molecular generation from mass spectra is better formulated as formula-restricted discrete graph diffusion than as autoregressive text generation. DiffMS's decoder noise-processes the adjacency matrix of a molecule whose heavy-atom composition is fixed by the inferred formula, then denoises it under a spectrum embedding extracted by a transformer that encodes peak formulae and neutral losses. On NPLIB1, DiffMS reaches 8.34% top-1 accuracy versus 5.40% for the best re-implemented baseline; on MassSpecGym it reaches 2.30% versus 1.31% for the next best method, and it also attains the best MCES and Tanimoto similarity at top-1 and top-10. The paper further claims that pretraining the diffusion decoder on 2.8M fingerprint-structure pairs improves performance monotonically with dataset size, and that removing exact benchmark test and validation molecules from that pretraining set makes the evaluation a genuinely novel-structure setting.","pith_inferences":["My inference: the paper's pretraining holdout removes only exact test and validation molecules; a stricter holdout that also excludes near-identical analogues of MassSpecGym test structures would reveal how much of the 2.30% top-1 gain is true generalization versus memorized substructure patterns.","My inference: because every DiffMS sample is a valid graph, candidate lists could be reranked by orthogonal evidence such as retention time, collision cross-section, or forward spectral prediction; the paper stops at generation and does not explore reranking.","My inference: given the one-to-many spectrum-to-structure ambiguity, exact top-1 accuracy has a ceiling; reporting diversity and coverage of the top-k list, which the paper does not formalize, would better capture the value for expert curation.","My inference: the scaling trend suggests the next bottleneck is the spectrum encoder's fingerprint pretraining, which uses only the small benchmark training sets, not the decoder; pretraining the encoder on larger synthetic or retrieved spectra could be a direct follow-up."],"forward_implications":["If DiffMS is right, de novo MS/MS structure elucidation should be built on formula-constrained graph diffusion rather than SMILES or scaffold autoregression; the graph representation yields 100% valid candidates and respects the known heavy-atom composition.","The observed scaling with pretraining set size implies that adding more fingerprint-structure pairs, which are cheap relative to spectrum-structure pairs, should continue to raise exact-match and similarity metrics.","Because predicted formulae from MIST-CF lose only about one point of top-1 accuracy on both benchmarks, a practical pipeline can run DiffMS on the top few formula hypotheses without manual formula assignment and still generate near-target candidates.","On MassSpecGym, where test structures are far from training structures, DiffMS still produces meaningfully similar candidates in the top-1 list while almost all baselines stay near zero, suggesting the method is usable for candidate ranking in discovery workflows, not only for benchmark exact matches."],"supporting_citations":[{"why":"supplies the discrete graph diffusion formulation and noise schedule that DiffMS adapts for the decoder.","marker":"Vignac et al. (2023)"},{"why":"provides the MIST formula transformer encoder and its spectrum-to-fingerprint pretraining objective.","marker":"Goldman et al. (2023b)"},{"why":"provides the MassSpecGym benchmark, evaluation metrics, and the scaffold-based split that makes it an out-of-distribution test.","marker":"Bushuiev et al. (2024)"},{"why":"provides the NPLIB1 benchmark spectra and test split used for evaluation.","marker":"Dührkop et al. (2021b)"},{"why":"defines the MSNovelist fingerprint-to-SMILES approach that DiffMS re-implements as a baseline and extends with a graph decoder.","marker":"Stravs et al. (2022)"},{"why":"provides the MADGEN two-stage scaffold-diffusion baseline that DiffMS outperforms.","marker":"Wang et al. (2025)"},{"why":"supplies SIRIUS, used to assign peak formulae that the spectrum encoder embeds.","marker":"Böcker & Dührkop (2016)"},{"why":"supplies the open-source fingerprint-to-SMILES decoder code used to re-implement the MSNovelist baseline.","marker":"Zhao et al. (2024)"}],"fun_headline_variants":["DiffMS: graph diffusion beats text models for mass spec","Formula-bounded diffusion tops de novo mass spec elucidation","Diffusion + formula constraints sharpen spectrum-to-structure","Pretrained diffusion model sets new bar on mass spec benchmarks","Mass spec to 2D graph: DiffMS diffusion wins top-1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that removing the exact benchmark test and validation molecules from the 2.8M fingerprint-structure pretraining set is enough to keep test information out of the decoder; if near-identical analogues of test molecules remain in libraries such as DSSTox, HMDB, COCONUT, or MOSES, the reported out-of-distribution gains are inflated.","fun_headline_variants_meta":{"raw":{"variants":["DiffMS: graph diffusion beats text models for mass spec","Formula-bounded diffusion tops de novo mass spec elucidation","Diffusion + formula constraints sharpen spectrum-to-structure","Pretrained diffusion model sets new bar on mass spec benchmarks","Mass spec to 2D graph: DiffMS diffusion wins top-1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000852,"raw_usage":{"total_tokens":3716,"prompt_tokens":968,"completion_tokens":2748,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":584,"completion_tokens_details":{"reasoning_tokens":2663}},"tokens_in":584,"tokens_out":2748,"duration_ms":21743,"temperature":1.0,"reasoning_tokens":2663,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T20:58:16.901991+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute, for every MassSpecGym test molecule, the maximum Tanimoto similarity (or minimum MCES) to any molecule in the 2.8M pretraining set; if a substantial fraction of test molecules have a near neighbour there, say Tanimoto above 0.85 or MCES below 10, the claim that evaluation represents truly novel structures collapses and the comparison to baselines trained without that library would need to be redone.","supporting_citations":[{"cited_title":"Digress: Discrete denoising diffusion for graph generation","cited_arxiv_id":null,"evidence_quote":"supplies the discrete graph diffusion formulation and noise schedule that DiffMS adapts for the decoder."},{"cited_title":"u hrkop, K., B \\","cited_arxiv_id":null,"evidence_quote":"defines the MSNovelist fingerprint-to-SMILES approach that DiffMS re-implements as a baseline and extends with a graph decoder."},{"cited_title":"MADGEN : Mass-spec attends to de novo molecular generation","cited_arxiv_id":null,"evidence_quote":"provides the MADGEN two-stage scaffold-diffusion baseline that DiffMS outperforms."},{"cited_title":"How to train your neural network for molecular structure generation from mass spectra? In 2024 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp.\\ 817--822","cited_arxiv_id":null,"evidence_quote":"supplies the open-source fingerprint-to-SMILES decoder code used to re-implement the MSNovelist baseline."}],"review_version":1}