{"id":"ddd044be-5fc3-4013-a46a-1e0792e79793","arxiv_id":"2411.15331","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"Scattering-based molecular features plus graph neural networks reach AUC 0.9812 for Ames mutagenicity on a 6,277-compound benchmark in the authors' split.","lead":"Chemicals that damage DNA are hard to screen experimentally; this paper tests whether scattering features from molecular images and graphs improve computer prediction of that effect. Their best model reports higher accuracy and AUC than published baselines, but the evaluation protocol has gaps that make the headline number uncertain.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Pipeline does not state that GIN embeddings, 2D feature selection, and the MOLG3-SAGE meta-graph are built from training molecules only, so the reported test AUCs may be inflated by test-set leakage.","rationale":"I read the paper as claiming that scattering-based featurization yields state-of-the-art Ames mutagenicity prediction, with the strongest evidence being the Lightgbm GGS+GIN AUC of 0.9812 and the MOLG3-SAGE results. For that claim to hold, the test set must be untouched by all learned components. The manuscript never states this. The GIN embedding generation is the most load-bearing gap because it directly feeds the headline model; unlike the meta-graph leakage, which only affects MOLG3-SAGE, GIN embeddings are used by the Lightgbm model that is called 'outstanding' in Table 4. The paper also shows signs of protocol looseness: the abstract and introduction repeatedly call the Hansen mutagenicity benchmark 'ZINC', and Table 4 mixes different published evaluation protocols. The random 80/20 split without scaffold separation makes the test set easier, but that alone is not disqualifying; the unresolved contamination question is. I agree with the reader's meta-graph concern but would elevate the GIN-embedding and feature-selection separation issue as the more central threat to the headline claim. I credit the geometric scattering construction as a plausible and potentially useful representation; the issue is entirely in the evaluation protocol. The rejection is appropriate as written, because the authors do not supply code or a detailed training protocol to rule out leakage; my proposed rerun is the minimal check that would settle it.","tokens_in":14859,"tokens_out":6098,"duration_ms":59284,"concrete_test":"Ask the authors to release code and rerun the full pipeline under a strict train-only protocol: (1) train the GIN embedding generator and the Chi-squared feature selector using only the 80% training split and its labels; (2) construct the MOLG3-SAGE meta-graph edges and Gaussian-kernel normalization using training molecules only, with test molecules inserted afterward for inference; (3) then retrain the Lightgbm classifiers and GraphSAGE and report test AUC and MCC on the held-out 20%. If the reported 0.9812 or 0.9622 AUC drops materially under this protocol, the central SOTA claim fails; if it does not, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central SOTA claim is the Lightgbm GGS+GIN AUC of 0.9812 (Table 4) and the MOLG3-SAGE ACC of 0.9301/AUC of 0.9622. Both rest on an evaluation protocol that never explicitly separates test information from feature construction. Section 4.2.2 selects 2D scattering features with Chi-squared on the full 11,681-dim representation, and Section 5 describes 'GIN embeddings' generated by a GIN architecture with no statement of the loss, label usage, or data split; if GIN is fit on all 6,277 molecules or with test labels, those embeddings encode test information before Lightgbm is trained, directly contaminating the headline number. Independently, Section 4.3 builds a fully connected meta-graph over 'n molecules' using Eqs. (15)-(18) with no exclusion of test molecules; if test nodes participate in GraphSAGE message passing and in the edge-weight normalization (sigma of the distance matrix), the evaluation is transductive, not the inductive comparison made in Table 4. Published baselines like AMPred-CNN are trained and evaluated on the same Hansen dataset but without these extra information channels. The paper does not provide code, seeds, or error bars, so the only way to determine whether the 0.98 AUC is real or an artifact of this leakage is to rerun with a strict train-only protocol. This is an internal protocol gap, not a disagreement about scattering theory.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes scattering-transform-based representations for Ames mutagenicity prediction, combining 2D image scattering, geometric graph scattering (GGS), GIN embeddings, and a fully connected meta-graph GraphSAGE model (MOLG3-SAGE). Using the Hansen et al. benchmark (6,277 compounds after preprocessing and a random 80/20 split), it reports AUC 0.9812 for Lightgbm with GGS+GIN features and ACC 0.9301/AUC 0.9622 for MOLG3-SAGE, and claims state-of-the-art performance against published baselines. The central claim is empirical: scattering features improve mutagenicity screening over descriptors and over published CNN/GNN models.","tokens_in":15168,"tokens_out":4828,"duration_ms":45155,"significance":"Scattering transforms are a plausible and relatively unexplored representation for molecular property prediction, and the paper's claims are concrete and falsifiable: the reported AUCs could in principle be reproduced or refuted with the same data. If the results survive a clean, train-only evaluation protocol, the contribution would be useful to computational toxicology. However, as presented, the evaluation does not rule out test information entering feature construction and meta-graph message passing, and the baseline comparisons are not apples-to-apples. The headline numbers are therefore not currently trustworthy, and the significance of the work hinges on a re-run under a rigorous protocol.","major_comments":[{"comment":"The meta-graph is defined over \"n molecules\" without any statement that test molecules are excluded from the node set and from the edge-weight construction, including the Gaussian-kernel normalization sigma. If test nodes and their pairwise distances participate in GraphSAGE message passing and in the global normalization, the evaluation is transductive, not the inductive comparison made against published models in Table 4. This directly affects the reported MOLG3-SAGE ACC 0.9301 and AUC 0.9622. Please specify and implement a train-only meta-graph construction, with sigma computed from training pairs only, and re-report all affected results.","section":"Section 4.3, Eqs. (15)-(18)"},{"comment":"The Chi-squared feature selection on the 11,681-dimensional 2D scattering representation and the generation of GIN embeddings are not described as restricted to the training split. If feature selection or GIN training uses all 6,277 molecules or the test labels, then the Lightgbm GGS+GIN AUC 0.9812 and the 2D-scattering AUC 0.9128 are inflated by information leakage. The manuscript must state whether the GIN is supervised by mutagenicity labels, on which split it is trained, and whether the 4000-feature subset is selected on the training portion only; the experiments should then be rerun under that strict protocol.","section":"Section 4.2.2 and Section 5"},{"comment":"The comparison to published baselines is not apples-to-apples. Baseline AUC values in Table 4 come from papers with different preprocessing of the Hansen data, different validation schemes (often cross-validation), and possibly different class distributions, whereas this paper uses a single random 80/20 split after its own filtering to 6,277 compounds. A higher AUC than a published number under different protocols does not establish superiority. Baselines should be rerun under the same split and metrics, or standard benchmark splits (e.g., scaffold split) should be used for all models.","section":"Tables 3 and 4"},{"comment":"The central results are reported as point estimates from a single random split, with no standard deviations, no number of seeds, and no code. Given that the best reported AUC (0.9812) is close to other values in the same table (0.9796, 0.9748), the state-of-the-art claim is not statistically supported. Please report means and variances over multiple random splits, and ideally scaffold splits, and make code, seeds, and the exact data split available.","section":"Table 3"}],"minor_comments":[{"comment":"The text repeatedly refers to the \"ZINC dataset\" and \"zinc toxicity,\" but the experiments use the Hansen et al. Ames mutagenicity benchmark; these terms should be corrected throughout.","section":"Abstract and Section 2"},{"comment":"The text says the tight Hann wavelet transform is used with three scales (j = 3), while Table 1 lists the number of tight Hann scales as J = 4; the text and table should be aligned, and the meaning of \"No. Tight HANN coefficients 7\" should be explained.","section":"Section 4.2.1 and Table 1"},{"comment":"The paper says hyperparameters were optimized with 10-fold cross-validation on the training portion, and Table 3 is labeled \"on a test set\"; please clarify whether all Table 3 numbers come from the held-out 20% and, if so, state that the test set was used only once after model selection.","section":"Section 5"},{"comment":"Equation (7) contains an incomplete or malformed trigonometric expression, and the variables R and e_max are not defined in the text; please fix the equation and define all symbols.","section":"Section 3.2, Eq. (7)"},{"comment":"The description of the GIN architecture is unclear: it says three sequential graph isomorphism layers, then mentions input dimension 7 and \"64 hidden layers,\" followed by fully connected layers yielding 128-dimensional embeddings. Please give the exact layer counts and dimensions.","section":"Section 5"},{"comment":"Reference [28] is not the original Graph Isomorphism Network paper; the GIN method should be cited to Xu et al., \"How Powerful are Graph Neural Networks?\" (ICLR 2019).","section":"References"},{"comment":"The descriptor name \"EPCF6\" appears to be a typo for \"ECFP6.\"","section":"Section 5"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is an early preprint-style submission with significant evaluation-protocol gaps. The most serious issue is the lack of any statement that test molecules are excluded from the meta-graph and from feature-construction steps; this is a standard correctness requirement, not a matter of theoretical taste. The authors should be given the opportunity to re-run the experiments under a strict train-only protocol and to provide code, seeds, and proper statistical summaries. If after such a revision the reported AUCs drop substantially, the state-of-the-art claim should be rewritten accordingly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read. The interesting kernel is real: applying wavelet scattering—2D WST on molecular images and geometric graph scattering on molecular graphs—to Ames mutagenicity is a legitimate new application, and the comparison of 2D scattering against RDKit2D/Mordred/MACCS is a useful data point. GGS features alone getting close to AMPred-CNN without a deep net is worth checking. But the paper's central claim, the 0.9812 AUC with GGS+GIN and MOLG3-SAGE, is not supported by the evaluation as written.\n\nThe soft spot is not the scattering theory; it's the training protocol. Section 4.2.2 selects 4000 of 11,681 2D scattering features using Chi-squared with no statement that this is done on training folds only. The GIN embeddings are described with architecture details but no loss, label usage, or data split, so we don't know whether test labels or test molecules influenced them. Section 4.3 builds a fully connected meta-graph over 'n molecules' with edge weights from scattering embeddings; test molecules are never excluded from that graph or from the sigma normalization. If test nodes participate in message passing, the MOLG3-SAGE result is transductive, not comparable to inductive published baselines. The random 80/20 split also makes the test set easier than a scaffold split, and Table 4 mixes published numbers from different preprocessing protocols. No code, seeds, or error bars are given, and Figure 6's reported 0.98-0.99 AUC across all folds only adds to the suspicion that information is leaking.\n\nAlso, the abstract and intro repeatedly call Hansen et al. 'ZINC'; that's a sloppy mislabel, not a fatal flaw, but it doesn't inspire confidence.\n\nWhat's good: the scattering pipeline is well described, the related work is detailed, and if the features were evaluated under a clean train-only protocol, the application-level finding would be a solid incremental contribution to computational toxicology. As it stands, the SOTA claim is unverifiable and likely inflated. I'd want a serious referee to demand a strict split-aware rerun, or let the authors resubmit with code and a clean protocol. This deserves referee time rather than desk rejection, because the underlying idea is worth testing and the leak is fixable.","headline":"Plausible scattering application, but the headline AUC is unverifiable because the pipeline never separates test molecules from feature construction and the meta-graph.","tokens_in":15692,"tokens_out":2721,"would_cite":false,"duration_ms":25419,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","68R10","92E10"],"pacs":[],"model":"deepseek-v4-flash","headline":"Scattering-based molecular features are claimed to lift Ames mutagenicity prediction to a 0.9812 AUC, ahead of a CNN baseline.","keywords":["mutagenicity","graph neural networks","geometric scattering transform","toxicity prediction","wavelet scattering","molecular graphs","GraphSAGE","Ames test"],"falsifier":"Rebuild the meta-graph using only training molecules, assign molecules to train and test by scaffold clusters rather than at random, and rerun both MOLG3-SAGE and the LightGBM-plus-GIN pipeline; if the AUCs stay at roughly 0.9812 and 0.9622 while the CNN baseline stays at 0.954, the claim is supported, and if the gap narrows or inverts, the reported edge depends on split leakage and near-duplicate molecules.","tokens_in":14615,"feed_emoji":"🧬","tokens_out":6697,"duration_ms":60775,"temperature":0.7,"pith_summary":"The paper tries to show that scattering-transform features, computed from molecular graphs and from 2D molecular images, are a more informative input for Ames mutagenicity prediction than standard chemical descriptors, fingerprints, or learned CNN image features. On the paper's benchmark (6,277 compounds after preprocessing), it reports that a LightGBM classifier fed geometric-scattering features plus GIN embeddings reaches an AUC of 0.9812, and that a new graph-of-graphs model called MOLG3-SAGE reaches 0.9301 accuracy and 0.8603 MCC. If these numbers hold under honest splitting, the practical upshot is that a training-free wavelet feature extractor can replace much of the learned featurization in toxicity screening, improving accuracy while reducing model complexity. The paper presents this as a step toward faster and more reliable computational chemical safety assessment.","feed_headline":"Scattering features push Ames mutagenicity AUC to 0.9812","feed_subtitle":"Wavelet-derived molecular features beat CNN and descriptor models on the mutagenicity benchmark, if the split is clean.","key_machinery":"The load-bearing object is the geometric scattering transform on molecular graphs, built from tight Hann wavelets and diffusion wavelets, which produces multiscale, permutation-invariant node embeddings; the 2D scattering transform on molecule images plays a supporting role. The hybrid pipeline concatenates these scattering features with graph-isomorphism-network (GIN) embeddings and feeds the result into standard classifiers, while MOLG3-SAGE replaces the classifier with a graph-of-graphs: a fully connected meta-graph whose nodes are molecules, whose edge weights are Gaussian-kernel cosine similarities between scattering embeddings, and whose message passing is done by two GraphSAGE layers. The meta-graph is the mechanism that lets each molecule's prediction be informed by its similarity to every other molecule in the dataset.","core_discovery":"The central discovery claim is that multiscale wavelet scattering, applied in two versions (a 2D wavelet scattering transform on molecule images and a geometric scattering transform on molecular graphs), captures the structural and spatial information most relevant to mutagenicity better than previous representations. The paper reports that geometric scattering features alone, used with a simple classifier, already rival a CNN model (AUC 0.9507 versus 0.954), and that adding GIN embeddings pushes LightGBM to AUC 0.9812, above all listed baselines. It also claims that organizing the whole dataset as a fully connected meta-graph, in which each molecule is a node and edge weights are Gaussian kernels on cosine similarity of scattering embeddings, and then running GraphSAGE on that meta-graph, yields the best accuracy (0.9301) and MCC (0.8603). The paper frames this as evidence that scattering-based molecular representations set a new performance level for Ames mutagenicity prediction.","pith_inferences":["Editorial inference: the most informative follow-up test, not reported in the paper, is a scaffold-split evaluation in which the meta-graph is rebuilt from training molecules only; this would separate genuine generalization from reliance on near-duplicate molecules between train and test.","Editorial inference: since both 2D and geometric scattering are parameter-light and training-free, the likely practical value is in small-data chemistry settings where deep models overfit; a useful extension would be measuring how the AUC gap changes as training set size shrinks.","Editorial inference: the 0.9812 AUC may approach the ceiling of this benchmark, so the next meaningful comparison is against recent consensus and transformer-based predictors on larger external mutagenicity sets, and on time-split or scaffold-split validation."],"forward_implications":["If the reported results are correct, scattering features offer a training-free, multiscale alternative to learned molecular fingerprints, with simple gradient-boosted trees matching or exceeding a purpose-built CNN.","MOLG3-SAGE's fully connected meta-graph implies that molecule-to-molecule similarity, not just per-molecule structure, carries predictive signal for mutagenicity; retaining that relational information is what the paper credits for its best accuracy and MCC.","The 2D scattering result implies that molecular images, when transformed rather than fed raw to a CNN, can outperform handcrafted descriptors, suggesting image-based screening pipelines need not abandon interpretable feature engineering.","Because the scattering transforms are not mutagenicity-specific, the same pipeline is directly portable to other molecular property and toxicity endpoints, subject to revalidation."],"supporting_citations":[{"why":"Supplies the benchmark mutagenicity dataset (6,512 compounds, preprocessed to 6,277) that every reported number depends on.","marker":"[24]"},{"why":"Provides the main baselines (AMPred-CNN and LightGBM with RDKit2D descriptors) that the proposed features are claimed to outperform.","marker":"[6]"},{"why":"Gives the prior SVM/ECFP4 AUC 0.93 result that the geometric-scattering features are compared against.","marker":"[10]"},{"why":"Defines diffusion wavelets, one of the two wavelet families used to build geometric scattering embeddings.","marker":"[23]"},{"why":"Defines tight Hann wavelets, the spectral wavelet family used for the graph scattering features.","marker":"[25]"},{"why":"Grounds the geometric scattering transform as group-invariant multiscale analysis, justifying permutation invariance.","marker":"[20]"},{"why":"Supplies the GraphSAGE inductive message-passing architecture used by MOLG3-SAGE.","marker":"[27]"},{"why":"Supplies the GIN model whose embeddings are concatenated with scattering features in the hybrid pipeline.","marker":"[28]"}],"fun_headline_variants":["Scattering transforms boost Ames mutagenicity AUC to 0.9812","Geometric scattering plus GIN yields top mutagenicity AUC","Multiscale scattering features hit 0.9812 AUC on Ames","Scattering-based embeddings push Ames prediction to 0.9812 AUC"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire comparison assumes the test molecules never leak into the fully connected meta-graph used to train MOLG3-SAGE, and that an 80/20 random split is a fair approximation of external validation.","fun_headline_variants_meta":{"raw":{"variants":["Scattering transforms boost Ames mutagenicity AUC to 0.9812","Geometric scattering plus GIN yields top mutagenicity AUC","Multiscale scattering features hit 0.9812 AUC on Ames","Scattering-based embeddings push Ames prediction to 0.9812 AUC"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000924,"raw_usage":{"total_tokens":3932,"prompt_tokens":890,"completion_tokens":3042,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":506,"completion_tokens_details":{"reasoning_tokens":2966}},"tokens_in":506,"tokens_out":3042,"duration_ms":19783,"temperature":1.0,"reasoning_tokens":2966,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:25:18.502790+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rebuild the meta-graph using only training molecules, assign molecules to train and test by scaffold clusters rather than at random, and rerun both MOLG3-SAGE and the LightGBM-plus-GIN pipeline; if the AUCs stay at roughly 0.9812 and 0.9622 while the CNN baseline stays at 0.954, the claim is supported, and if the gap narrows or inverts, the reported edge depends on split leakage and near-duplicate molecules.","supporting_citations":[{"cited_title":"Hansen, S","cited_arxiv_id":null,"evidence_quote":"Supplies the benchmark mutagenicity dataset (6,512 compounds, preprocessed to 6,277) that every reported number depends on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the main baselines (AMPred-CNN and LightGBM with RDKit2D descriptors) that the proposed features are claimed to outperform."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the prior SVM/ECFP4 AUC 0.93 result that the geometric-scattering features are compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines diffusion wavelets, one of the two wavelet families used to build geometric scattering embeddings."},{"cited_title":"Jiang, Z","cited_arxiv_id":null,"evidence_quote":"Defines tight Hann wavelets, the spectral wavelet family used for the graph scattering features."},{"cited_title":"Mallat, Group invariant scattering, Communications on Pure and Applied Mathematics 65 (2012) 1331–1398","cited_arxiv_id":null,"evidence_quote":"Grounds the geometric scattering transform as group-invariant multiscale analysis, justifying permutation invariance."}],"review_version":1}