{"id":"4d35bff7-f316-438c-9801-d6126af1520f","arxiv_id":"2507.22514","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"SmilesT5 shows that pretraining a T5 model to reconstruct Murcko scaffolds and predict molecular fragments beats masked-language pretraining on six molecular property classification benchmarks.","lead":"The authors pretrain a T5 language model on molecule strings with chemistry-specific tasks: rebuilding the molecular scaffold and naming molecular fragments. This improved molecular property prediction over masked-language pretraining and published baselines, using far fewer pretraining molecules.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Statistical significance claims rest on non-independent scaffold-split folds and uncorrected Mann-Whitney tests; the central superiority claim needs a paired, multiplicity-corrected re-analysis.","rationale":"The reader's weakest assumption is essentially the same statistical concern; I agree and sharpen it. The unpaired Mann-Whitney U test is doubly problematic here: folds overlap across seeds, and the experiment is paired by construction because all models are finetuned on the same ten splits. The paper's own p-value grids are the only quantitative support for 'statistically outperforms', so if these tests are invalid the central claim is unproven, even if the direction of the medians favors Scaffold+Fragments. This is fixable by re-analysis, so the verdict stays CONDITIONAL rather than moving to reject. Secondary issues, such as inconsistent BLEU values between Section 2.1 and Table S3 and sparse finetuning details, are real but do not by themselves sink the central claim.","tokens_in":12022,"tokens_out":5470,"duration_ms":68947,"concrete_test":"Run a re-analysis on the released per-split F1 scores (GitHub smiles_t5): for each benchmark and each model pair, compute fold-wise differences and test with a paired permutation test (or Wilcoxon signed-rank) across the ten seeds, then apply Benjamini-Hochberg FDR across all six benchmarks and all pairwise comparisons. Also compute the overlap of test-set molecules across the ten scaffold-split seeds; if the Jaccard overlap is substantial, the independence assumption is violated and the unpaired Mann-Whitney U p-values in Figures 3-5 should be replaced by the paired, corrected values. If Scaffold+Fragments still beats MLM in all six benchmarks after correction, the central claim survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—that Scaffold+Fragments pretraining 'statistically outperforms' MLM in all six benchmarks—rests on one-sided Mann-Whitney U p-values computed over ten scaffold-split folds (Section 2.1, Section 4.2). This test assumes the ten F1 scores per model are independent samples. They are not: the ten seeds reshuffle the same 80/10/10 scaffold partitions of the same dataset, so the same molecule can appear in the test set of one fold and the training set of another. The ten test sets are therefore overlapping and correlated, not independent replicates. Moreover, because all models are evaluated on the same ten splits, the correct comparison is paired; an unpaired Mann-Whitney U discards this pairing and its p-values are not a valid significance statement. No multiple-comparison correction is applied across six benchmarks and the many pairwise green cells; at alpha=0.05, several false positives are expected. The paper also does not report effect sizes, so 'statistically greater' is not tied to a practically meaningful margin. If the F1 advantage does not survive a paired, FDR-corrected test, the headline claim of statistically improved performance is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SmilesT5, a T5-style transformer pretrained on SMILES strings with text-to-text objectives: Murcko scaffold reconstruction, fragment presence prediction, and their combination (Scaffold+Fragments). The authors compare four pretraining variants on six molecular property classification benchmarks using ten scaffold-split folds, report F1 scores and Mann-Whitney U significance grids, and claim that the joint Scaffold+Fragments task statistically outperforms masked language modeling on all six benchmarks. They further compare an XL-sized SmilesT5 against graph- and language-based baselines, and present ablations on pretraining corpus size, model size, embedding-based classification, and auxiliary finetuning tasks.","tokens_in":12212,"tokens_out":4593,"duration_ms":52810,"significance":"If the empirical claims hold, the work provides a data-efficient pretraining recipe for molecular language models and demonstrates that frozen encoder embeddings can serve as competitive fixed features, lowering the hardware barrier for practitioners. The paper's strengths include public code and model releases, evaluation against multiple strong baselines, and a systematic ablation of pretraining task design. However, the central statistical claims currently rest on an inappropriate significance testing setup, and the selection of the favored pretraining task on the same benchmarks used for downstream evaluation introduces optimism that must be addressed before the conclusions can be taken at face value.","major_comments":[{"comment":"The Mann-Whitney U tests used to support the headline claim that Scaffold+Fragments 'statistically outperforms' MLM in all six benchmarks are not valid for this experimental design. The ten scaffold-split folds are not independent replicates: the same molecules appear in different roles across folds, and all models are evaluated on the same ten splits, making the observations paired. Mann-Whitney U assumes independent samples and discards this pairing. A paired test (e.g., Wilcoxon signed-rank or a paired permutation test) should be used, and p-values should be adjusted for the multiple comparisons across benchmarks and across model pairs. Without this re-analysis, the green-cell grids in Figures 3b, 4b, and 5b do not establish the claimed statistical superiority.","section":"Section 2.1 and 4.2"},{"comment":"The choice of the Scaffold+Fragments task as the main model is made after comparing the four pretraining variants on the same six benchmarks that are subsequently used to claim superiority over other models and in ablations. This selection procedure means the reported comparisons for the chosen model are optimistic and not independent of the model-selection step. The authors should either report results for all pretraining variants in the comparison against external baselines, or evaluate the selected model on a separate, hold-out benchmark suite, and in any case disclose this selection in the text.","section":"Section 2.1 vs 2.2/2.3"},{"comment":"The F1 computation for multi-label benchmarks is underspecified. For datasets such as SIDER and Tox21 with many tasks, it is unclear whether F1 is macro-averaged, micro-averaged, or computed per-sample, and no decision threshold for converting output probabilities to binary predictions is described. The same applies to Clintox with two tasks. Without these details, the reported F1 numbers cannot be reproduced or interpreted, and the comparison to baselines is not fully verifiable.","section":"Section 4.2, Metrics Calculations"}],"minor_comments":[{"comment":"'Word rate error' should be 'word error rate'.","section":"Section 2.1 and Table S3"},{"comment":"References [16] and [29] appear to cite the same Ross et al. paper with different publication details; please consolidate.","section":"References"},{"comment":"The total parameter row is garbled ('4 198 2 215'); please format the parameter counts separately for the Small, Base, and XL models.","section":"Table 1"},{"comment":"The acronym 'D-MPPN' is used in the text but 'D-MPNN' in the reference list; please standardize.","section":"Throughout"},{"comment":"Because the significance grids only show green/white at p<0.05, consider reporting numeric p-values or a supplementary table with exact p-values and effect sizes.","section":"Figures 3-5"}],"recommendation":"major_revision","confidential_remarks":"The statistical flaws and the model-selection-on-the-test-benchmarks issue are the main blockers. If the authors re-analyze with paired, multiplicity-corrected tests and either hold out benchmark suites or report all pretraining variants in the external comparison, the paper could become publishable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful thing here is the joint Scaffold+Fragments pretraining task for a T5 molecular language model. The components—Murcko scaffold reconstruction and fragment-presence prediction—are not new individually, but combining them as text-to-text tasks before finetuning is a reasonable and reasonably novel recipe. The paper also does a few things right: it ships code and model weights, ablates dataset size and model size, and shows that frozen embeddings from the encoder feed a random forest that is competitive with finetuning at a fraction of the compute. The consistent F1 direction across six benchmarks is encouraging.\n\nThe soft spot is the statistics. The central claim that Scaffold+Fragments 'statistically outperforms' MLM rests on one-sided Mann-Whitney U tests over ten scaffold-split folds. Those folds are not independent—they are reshuffled partitions of the same dataset, so the same molecules appear in different roles across folds, and all models are evaluated on the same splits. That makes the unpaired test inappropriate; the correct comparison is paired. No multiple-comparison correction is applied across six benchmarks and a grid of pairwise cells, and no effect sizes or numeric p-values are reported. At alpha=0.05 you expect false positives. The direction of the results may be real, but the significance statement as written does not survive scrutiny. This is fixable: report paired tests with a correction, plus effect sizes, and the headline will either stand or become appropriately qualified.\n\nThere are also smaller reporting gaps. The BLEU number in Section 2.1 (0.989, word rate error 0.125) disagrees with Table S3 (0.9967, 0.004) for the 1M model; that needs reconciliation. Fine-tuning details are sparse, and the baselines don't include another T5-based SMILES model, so it is hard to separate the pretraining objective from the architecture.\n\nOverall: this is a sensible, reproducible contribution that deserves a serious referee. I would send it out, but with a clear request to redo the statistical analysis with paired tests and multiplicity control, report effect sizes, and fix the BLEU discrepancy. If the paired analysis holds, the paper is a useful practical improvement for molecular property prediction.","headline":"A sensible new pretraining recipe for T5 molecular models with useful ablations and public code, but the significance claims need a paired, corrected re-analysis before they hold.","tokens_in":12768,"tokens_out":2323,"would_cite":true,"duration_ms":26854,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SmilesT5 shows that pretraining a T5 model to reconstruct Murcko scaffolds and detect molecular fragments improves downstream molecular property classification over masked language modelling, and that frozen embeddings from the model…","keywords":["SMILES","molecular property prediction","text-to-text pretraining","T5","Murcko scaffold","molecular fragments","domain-specific pretraining","frozen embeddings"],"falsifier":"A replication using the released code and data but with 50 scaffold-split seeds instead of 10, testing whether the Scaffold+Fragments model achieves higher F1 than the MLM model in all six benchmarks, would settle the claim; any benchmark where the direction flips, or where the advantage loses significance under a corrected threshold, would undercut the headline conclusion.","tokens_in":11793,"feed_emoji":"🧪","tokens_out":9982,"duration_ms":104482,"temperature":0.7,"pith_summary":"The paper argues that the default pretraining objective for molecular language models — masked language modelling on SMILES strings — leaves chemically important structure unused. It introduces two text-to-text pretraining tasks, reconstructing a molecule's Murcko scaffold and listing the molecular fragments it contains, and shows that a T5 model trained on both jointly finetunes to higher F1 scores than an MLM-pretrained counterpart across six classification benchmarks. The authors further report that these gains survive in small models pretrained on only one million molecules, and that the encoder's averaged token embeddings, fed to a random forest classifier, approach finetuning performance at far lower computational cost. If these claims are correct, the joint scaffold-and-fragments objective is a better default recipe for SMILES-based property prediction than likelihood-based pretraining.","feed_headline":"Scaffold-plus-fragment pretraining beats masking on six benchmarks","feed_subtitle":"Chemistry-aware tasks outperform masked-language pretraining; frozen embeddings rival finetuning.","key_machinery":"The load-bearing machinery is the joint Scaffold+Fragments pretraining task implemented in the T5 text-to-text framework, which casts every task as transforming one text string into another. For each SMILES string, the model is asked either to produce the Murcko scaffold (the core ring systems and linkers left after removing side chains) or to output the names of the molecular fragments present, from a fixed set of 86 chemically defined fragments; task-token prefixes such as 'scaffold:' or 'fragments:' tell the model which output to generate. Because scaffolds and fragments are frequently not contiguous substrings of the input SMILES, solving these tasks forces the model to learn latent molecular structure rather than surface token statistics. At finetuning time, class labels are converted into text and the model's language-head probabilities over label tokens form the prediction vector; the encoder's averaged per-token embeddings serve as fixed-length molecular fingerprints in the embedding experiments.","core_discovery":"The central claim is that domain-specific, structure-oriented pretraining transfers better to molecular property prediction than masked-language pretraining. Concretely, a T5 model pretrained to generate the Murcko scaffold of a molecule and to emit the names of the defined molecular fragments present in it, combined through task-token prefixes, statistically outperforms an MLM-pretrained model in all six benchmarks tested (BBBP, BACE, HIV, Clintox, SIDER, and Tox21). The XL version of the model is reported to beat or match graph- and language-based baselines in most comparisons, is never outperformed by another language model, and is bettered only twice overall. The paper also claims that these pretraining tasks are data- and parameter-efficient: increasing pretraining data from one million to one hundred million molecules, or increasing model size, yields only marginal improvements in most benchmarks, and frozen encoder embeddings with a random forest give competitive results in several benchmarks.","pith_inferences":["The same task design — scaffold reconstruction plus fragment-presence prediction — could be adapted to other molecular string notations or to other scientific languages with non-contiguous structure, where token-level masking may be a weaker training signal.","The near-flat scaling from one million to one hundred million pretraining molecules suggests that structure-aware tasks have a steeper returns profile than mask-filling; testing this across more corpus sizes would clarify how much pretraining data molecular language models actually need.","A direct comparison of these frozen embeddings against classical structural fingerprints (for example circular connectivity fingerprints) would quantify how much of the gain is due to the learned representation itself rather than the general strength of a random forest on dense features.","Aggregating the six per-benchmark significance tests into a single joint analysis, rather than reading each pairwise grid separately, would give a more unified statement of when the Scaffold+Fragments advantage holds."],"forward_implications":["The Scaffold+Fragments objective can replace masked-language modelling as the default self-supervised pretraining task for SMILES-based property prediction models on the benchmarks studied.","Small models pretrained on one million molecules are sufficient for strong downstream performance, so pretraining cost for molecular language models can be cut substantially relative to hundred-million-molecule corpora.","Frozen encoder embeddings can act as fixed molecular fingerprints, allowing competitive classifiers to be built with a random forest and CPU-only compute.","Because labels are treated as text, the same model can be finetuned on several tasks at once; adding the pretraining tasks during finetuning helps in some benchmarks (BBBP, Tox21) and hurts in another (Clintox), so task selection matters.","The pretraining tasks transfer to molecules with unseen scaffolds: on a held-out scaffold-split set the joint model reaches high BLEU scores and near-perfect text reconstruction accuracy."],"supporting_citations":[{"why":"Supplies the T5 text-to-text framework, the masked-language modelling variant, and the model size configurations used throughout.","marker":"[21]"},{"why":"Defines the SMILES line notation that serves as the model's input and output language.","marker":"[13]"},{"why":"Supplies the benchmark collection and the scaffold-split protocol used to create the ten train/valid/test splits.","marker":"[2]"},{"why":"Provides the pretraining corpus and benchmark data, with the one-million and one-hundred-million molecule sets used in the ablations.","marker":"[26]"},{"why":"ChemBERTa is the masked-language-pretrained baseline that the domain-specific tasks are compared against.","marker":"[14]"},{"why":"ChemBERTa-2 serves as an additional language-model baseline with larger-scale pretraining and multi-task regression.","marker":"[15]"},{"why":"MolFormer is the large-scale MLM-pretrained language model used as a comparison point for SmilesT5 XL.","marker":"[29]"},{"why":"The D-MPNN graph model is a principal graph-based baseline; the XL model is bettered by it in only one benchmark.","marker":"[6]"},{"why":"KANO is the knowledge-graph-augmented contrastively pretrained baseline; the XL model is bettered by it in only one benchmark.","marker":"[10]"},{"why":"Motivates the choice of statistical testing protocol and the use of F1 over ROC AUC for drawing performance conclusions.","marker":"[27]"}],"fun_headline_variants":["Scaffold-fragment pretraining tops masking on six tasks","Chemistry-aware pretraining beats masked-language in property tests","New pretraining tasks yield better molecular property predictors","T5 with scaffold-fragment tasks wins on six benchmarks","Domain-specific tasks boost molecular property prediction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's 'statistically outperforms' conclusions rest on treating the ten scaffold-split test folds of each benchmark as independent samples for Mann-Whitney U tests, with no correction for the many pairwise comparisons reported across six benchmarks.","fun_headline_variants_meta":{"raw":{"variants":["Scaffold-fragment pretraining tops masking on six tasks","Chemistry-aware pretraining beats masked-language in property tests","New pretraining tasks yield better molecular property predictors","T5 with scaffold-fragment tasks wins on six benchmarks","Domain-specific tasks boost molecular property prediction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000141,"raw_usage":{"total_tokens":1136,"prompt_tokens":892,"completion_tokens":244,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":508,"completion_tokens_details":{"reasoning_tokens":169}},"tokens_in":508,"tokens_out":244,"duration_ms":3248,"temperature":1.0,"reasoning_tokens":169,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T11:35:00.239298+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A replication using the released code and data but with 50 scaffold-split seeds instead of 10, testing whether the Scaffold+Fragments model achieves higher F1 than the MLM model in all six benchmarks, would settle the claim; any benchmark where the direction flips, or where the advantage loses significance under a corrected threshold, would undercut the headline conclusion.","supporting_citations":[],"review_version":1}