{"id":"59e8ada7-fe4a-4ecc-b0ca-ccff71d5c410","arxiv_id":"2504.20789","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Augmenting SELFIES strings improves LSTM and quantum-kernel LSTM drug side-effect prediction over plain SMILES, but the effect is largely augmentation and lacks proper statistical testing.","lead":"This paper tests whether converting drug molecules into augmented SELFIES strings improves machine-learning models that predict side effects. The reported gains over plain SMILES are mostly due to data augmentation, and the paper's claim of statistical significance is not backed by a proper test.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed SELFIES advantage is an artifact of a confounded contrast: Table I itself shows SELFIES is no better than SMILES once augmentation is held fixed, so the abstract's causal claim is unsupported.","rationale":"The reader's weakest_assumption targeted split/selection bias in the evaluation protocol. That is a real concern, but the more decisive problem is internal: the paper's own Table I and Section III.D provide a factorial decomposition showing that the SELFIES representation is not the source of the reported gain. For the classical LSTM, the only arm with all four cells, holding augmentation fixed makes SELFIES at best neutral or slightly worse than SMILES (0.556 vs 0.562 with augmentation; 0.507 vs 0.525 without). The abstract's causal claim therefore is not merely statistically unverified; it is contradicted by the paper's own numbers. The missing QK-LSTM Augmented SMILES control means the quantum half of the headline is untestable from the reported data. Since the reader's verdict was REJECT and this analysis reinforces that conclusion rather than overturning it, I retain the reader's verdict; no adjustment is needed. I mark agreement as partial because the reader's stated weakest_assumption was the protocol, although the reader's rationale did note the conflation of augmentation and representation.","tokens_in":10140,"tokens_out":8356,"duration_ms":81662,"concrete_test":"Run the missing QK-LSTM Augmented SMILES condition under the same random split, Optuna budget, and top-3 averaging used for the other rows of Table I, then compute paired bootstrap 95% confidence intervals for the two contrasts (Augmented SELFIES - SMILES) and (Augmented SELFIES - Augmented SMILES) across all repeated runs. If the second interval includes zero or favors Augmented SMILES, the headline advantage is an augmentation effect, not a SELFIES effect, and the abstract's causal wording must be withdrawn or reframed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim is that 'augmenting SELFIES yields statistically significant improvements from SMILES' (5.97% classical, 5.91% hybrid). The contrast used, Augmented SELFIES vs SMILES, varies two factors at once: augmentation and representation. Table I contains the controls needed to separate them for the classical LSTM. At no augmentation, SELFIES (0.507) is 0.018 worse than SMILES (0.525); at fixed augmentation, Augmented SELFIES (0.556) is 0.006 worse than Augmented SMILES (0.562). Section III.D itself reports these as a 0.022 worse SMILES-to-SELFIES change and a 0.006 worse augmented-SMILES-to-augmented-SELFIES change, both described as within standard deviation. So the apparent gain in the headline is carried by augmentation, not by SELFIES, and the paper's own discussion concedes that converting to SELFIES gives no statistically significant performance gain. No significance test is reported anywhere; the abstract's 'statistically significant' phrase is unsupported, and for the QK-LSTM arm the paper did not even run the Augmented SMILES control. The central causal claim therefore fails on the paper's own numbers.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether augmenting SELFIES molecular string representations improves drug side-effect prediction over plain SMILES, using LSTM and a hybrid quantum-classical QK-LSTM on the SIDER benchmark. It reports ROC-AUC scores in Table I and claims in the abstract that augmenting SELFIES yields statistically significant improvements of 5.97% (classical) and 5.91% (hybrid) over SMILES. The paper also describes the QK-LSTM architecture, data preprocessing with canonicalization and SMILES augmentation, and an Optuna-driven hyperparameter search with top-three model averaging. The authors conclude that augmentation helps both representations, while conversion to SELFIES yields no statistically significant gain.","tokens_in":10364,"tokens_out":2348,"duration_ms":24462,"significance":"If the central claim were sound, the paper would provide a useful first exploration of augmented SELFIES in both classical and quantum-classical settings, with an architecture-level comparison and a practical benchmark. The work also gives credit for describing the QK-LSTM integration and for publishing its comparison table, which is transparent enough to reveal the main confound. However, the central claim is not supported by the data as presented: the headline contrast varies both augmentation and representation simultaneously, the paper's own Section III.D shows that SELFIES is not better than SMILES once augmentation is held fixed, and no significance test is reported anywhere. The empirical contribution is therefore weakened to a limited ablation study, and the abstract's causal language is unjustified.","major_comments":[{"comment":"The claim that 'augmenting SELFIES yields in statistically significant improvements from SMILES' conflates two factors: augmentation and representation. The comparison used for the headline (LSTM Augmented SELFIES 0.556 vs LSTM SMILES 0.525, and QK-LSTM Augmented SELFIES 0.555 vs QK-LSTM SMILES 0.524) varies both augmentation and the string representation. Table I contains the controls needed to separate these factors for the classical model, but those controls contradict the causal interpretation: at fixed representation, Augmented SELFIES (0.556) is worse than Augmented SMILES (0.562), and at zero augmentation, SELFIES (0.507) is worse than SMILES (0.525). The headline difference is therefore attributable to augmentation, not to SELFIES.","section":"Abstract and Table I"},{"comment":"The paper's own discussion undercuts the abstract's statistical claim. Section III.D states that switching from SMILES to SELFIES gives a 0.022 worse performance that is 'within the standard deviation', and switching from augmented SMILES to augmented SELFIES gives a 0.006 worse performance 'again within the standard deviation'. The section then concludes that there is a 'non-statistically significant performance change' when using SELFIES. This is internally consistent with Table I, but it directly contradicts the abstract's assertion of statistically significant improvements from SMILES. The central claim of the paper is thereby unsupported by the manuscript's own comparisons.","section":"Section III.D, Comparisons"},{"comment":"No significance test is reported anywhere in the paper, despite the abstract and Section III.D using the phrase 'statistically significant'. The reported standard deviations overlap for the relevant comparisons (e.g., SMILES-to-SELFIES differences of 0.022 with standard deviations of 0.023 and 0.026). To support any significance claim, the authors need to specify the number of independent runs, the random seeds, whether the same 80/10/10 split was used across all six conditions, and a proper test such as paired bootstrap or a paired difference test. Without this information, the 'statistically significant' language in the abstract is unsubstantiated.","section":"Section III.C, Results (evaluation protocol)"},{"comment":"The evaluation protocol selects each configuration's top three performing models by validation ROC-AUC and averages their test scores. This top-k selection from a small number of Optuna configurations is a post-hoc model selection procedure that can bias the reported mean upward and understate variance, especially if the same validation split is reused for selection and early stopping. The manuscript does not state how many Optuna trials were run, how many random configurations were sampled, or how the variance across the three selected models relates to the variance across all runs. This makes the reported '±' values difficult to interpret and the comparison between conditions potentially biased if selection operated differently across conditions.","section":"Section III.C, Results (model selection)"},{"comment":"For the QK-LSTM arm, the paper does not run the augmented-SMILES control. Section III.C states that computational cost motivated focusing on the two extremes, SMILES and augmented SELFIES. However, without QK-LSTM augmented SMILES, the claimed 5.91% improvement in the hybrid quantum-classical domain cannot separate the effect of augmentation from the effect of the representation, and the paper's own argument in Section III.D requires that control to make the decomposition. The missing control is a load-bearing gap for the quantum-domain claim.","section":"Section III.C, Results (missing control)"}],"minor_comments":[{"comment":"There is a typo in 'Early stopping iss applied' — 'iss' should be 'is'.","section":"Section III.C, Results"},{"comment":"'QNNs comprises of three core components' should be 'QNNs comprise three core components'.","section":"Section II.G, Quantum Neural Networks"},{"comment":"'Long Short-Term Memory (LSTM) networks [24] is a specialized type' should be 'networks ... are'.","section":"Section II.F, Long Short-Term Memory"},{"comment":"The caption says 'converts classical features ( f1, f4, f3, f4)' but likely should read '(f1, f2, f3, f4)'.","section":"Figure 3 caption"},{"comment":"The sentence 'augmenting SELFIES yields in statistically significant improvements from SMILES' contains a grammatical error: 'yields in' should be 'yields statistically significant improvements over SMILES'.","section":"Abstract"},{"comment":"The paper does not report the number of molecules in the SIDER dataset after canonicalization, the distribution of labels, or the vocabulary size after SELFIES tokenization; these details would help reproducibility and interpretation of the results.","section":"Section III.A, Data Pre-processing"}],"recommendation":"reject","confidential_remarks":"The paper's central claim is contradicted by its own Table I and Section III.D: the only statistically significant-sounding result is a confounded contrast, while the controlled comparisons show no benefit for SELFIES. The missing significance tests and the missing QK-LSTM augmented-SMILES control are not minor omissions; they bear directly on the abstract's headline. I see no fix within the current scope that would preserve the claim, though the paper could be rewritten as a limited negative/ablation result. I would advise the editor that the manuscript, in its present form, does not meet the standard for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a small empirical paper whose headline claim is contradicted by its own results table. The abstract says augmented SELFIES gives a statistically significant ~6% gain over SMILES, but Table I shows that when you hold augmentation constant, SELFIES is actually slightly worse than SMILES (0.507 vs 0.525 for unaugmented; 0.556 vs 0.562 for augmented). The real effect is augmentation, not SELFIES, and the paper's own Section III.D admits the SELFIES conversion produces no statistically significant change. The abstract compares two things at once—augmentation and representation—and the table contains the controls that unravel the claim.\n\nWhat's new: it's the first evaluation of augmented SELFIES, and the first augmentation test in a QK-LSTM. That's a legitimate gap. The paper does a few things right: it canonicalizes SMILES, follows Bjerrum's augmentation recipe, and reports raw ROC-AUC for all six conditions with standard deviations. Section III.D is the most honest part—it reads like the authors know the result is null for SELFIES.\n\nThe soft spots are not subtle. There is no significance test anywhere, despite \"statistically significant\" in the abstract. The top-3-of-four-Optuna-configurations selection is post-hoc and likely inflates results, because choosing the best validation runs and then averaging is a form of selection bias. The random split is not described as identical across conditions, so split luck is a real concern. For the QK-LSTM arm, the augmented-SMILES control was not run, so that comparison is even less interpretable. And the absolute scores (0.52–0.56 ROC-AUC on SIDER) are close to random; the paper doesn't benchmark against any strong baseline, so practical significance is limited.\n\nThat said, the paper is not incoherent. The experimental setup is understandable and the data table is clear. With the abstract rewritten to say \"augmentation improves both SMILES and SELFIES; SELFIES itself gives no significant benefit,\" the paper would be a modest but acceptable negative result. As it stands, the central claim fails on the paper's own numbers.\n\nWho is this for? Someone specifically interested in whether SELFIES helps in recurrent or quantum-hybrid models. A reader would get value from Table I and the honest discussion, but not from the headline. I'd send it to peer review because it's a legitimate empirical question with a reproducible protocol and the flaws are fixable with reframing, not foundational. But I'd expect major revision and a new abstract.","headline":"The paper's own Table I shows the SELFIES-vs-SMILES gain vanishes once augmentation is held fixed, so the abstract's causal claim does not survive contact with the data.","tokens_in":10942,"tokens_out":3106,"would_cite":false,"duration_ms":30394,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Augmented SELFIES improve molecular property prediction over plain SMILES in both classical LSTM and hybrid quantum-classical QK-LSTM models, with relative ROC-AUC gains of about 6 percent.","keywords":["molecular property prediction","SELFIES","SMILES","data augmentation","QK-LSTM","quantum machine learning","side effect prediction","LSTM"],"falsifier":"Rerun all six configurations on a fixed shared split with the same Optuna budget per condition and check whether augmented SELFIES still beats plain SMILES by roughly 0.03 ROC-AUC; as a second check, apply the same pipeline to another MoleculeNet classification dataset, such as BACE or BBBP, where the augmentation advantage can be confirmed or refuted.","tokens_in":9907,"feed_emoji":"🧪","tokens_out":8072,"duration_ms":72122,"temperature":0.7,"pith_summary":"This paper tries to establish that augmenting SELFIES strings—generating multiple valid SELFIES renderings of each molecule instead of a single string—improves molecular property prediction over plain SMILES in both classical LSTM and hybrid quantum-classical models. On the SIDER side-effect dataset, the authors report that LSTM with augmented SELFIES reaches a ROC-AUC of 0.556 versus 0.525 for plain SMILES, and that QK-LSTM (a long short-term memory network with quantum kernel feature maps in its gates) reaches 0.555 versus 0.524, a relative gain of about 6 percent in both domains. The paper presents this as the first test of augmentation for SELFIES and the first application of augmented molecular strings to a quantum kernel LSTM. If correct, it would show that a classical data-level technique transfers to quantum-hybrid models, and that a robust string representation need not cost accuracy.","feed_headline":"Augmented SELFIES beat SMILES for property prediction","feed_subtitle":"Adding SELFIES variations raised ROC-AUC by ~6 percent in LSTM and QK-LSTM side-effect prediction.","key_machinery":"The load-bearing mechanism is an augmentation-conversion pipeline: canonical SMILES strings are first converted to canonical form, then augmented by generating up to twenty alternate SMILES enumerations and retaining the five shortest, and the resulting strings are converted to SELFIES and tokenized at the bracket level into index sequences for the embedding layer. For the hybrid quantum-classical branch, the QK-LSTM replaces the classical gate transformations with quantum kernels built from entangler circuits—angle encoding, a parameterized entangling circuit, measurement, and a fully connected squeeze—so that each LSTM gate (forget, input, output) processes the sequence through a quantum feature map. This pipeline is what lets the paper attribute the ROC-AUC difference to the input representation and augmentation rather than to a new architecture.","core_discovery":"On the paper's own terms, the central discovery is a quantitative comparison: on the SIDER side-effect benchmark, models trained on augmented SELFIES outperform models trained on plain SMILES by 0.031 ROC-AUC in both the classical LSTM (0.556 vs. 0.525) and the hybrid quantum-classical QK-LSTM (0.555 vs. 0.524), which the authors report as statistically significant improvements of 5.97% and 5.91%. The same comparison shows that plain SELFIES without augmentation perform worse than SMILES (0.507 vs. 0.525), which the authors read as evidence that augmentation, rather than the SELFIES grammar alone, drives the learning gain. The paper further reports that QK-LSTM matches classical LSTM within 0.001 ROC-AUC, suggesting the quantum kernel architecture can retain accuracy while reducing the hidden dimension from 32–128 to 8–32.","pith_inferences":["Editorial inference: the headline percentages are relative gains on a low absolute baseline (0.525 to 0.556 is a 0.031 ROC-AUC increase), so the practical importance may be smaller than '5.97%' suggests and depends on replication on other benchmarks.","Editorial inference: since augmented SMILES (0.562) actually scores slightly higher than augmented SELFIES (0.556), the strongest defensible takeaway is that augmentation helps and SELFIES does not hurt, not that SELFIES is intrinsically superior.","Editorial inference: a natural extension is to pair augmented SELFIES with generative or retrosynthesis tasks, where SELFIES' guarantee of valid strings could combine with augmentation to improve both the validity and diversity of proposed molecules.","Editorial inference: the significance claim rests on standard-deviation non-overlap; a paired bootstrap over molecules or multiple random seeds would provide a firmer statistical test."],"forward_implications":["Augmented SELFIES can serve as a drop-in replacement for SMILES in both classical LSTM and QK-LSTM pipelines, improving ROC-AUC by about 0.03 without changing the model architecture.","Hybrid quantum-classical QK-LSTM can match classical LSTM within 0.001 ROC-AUC on molecular property prediction, so near-term quantum hardware constraints do not force an accuracy trade-off on this task.","Plain SELFIES without augmentation does not improve over SMILES, implying that representation choice alone is not the driver; augmentation is the active ingredient.","The augmentation benefit transfers across string formats and across classical/quantum domains, so it behaves like a property of training-data diversity rather than of the specific encoding.","Because the paper finds the largest absolute gains from augmentation for the LSTM, scaling datasets with augmented SELFIES may become increasingly valuable as dataset size grows."],"supporting_citations":[{"why":"Supplies the SMILES enumeration technique used to generate alternate molecular strings for augmentation.","marker":"[15]"},{"why":"Provides the SIDER dataset, the 80/10/10 split methodology, and the ROC-AUC evaluation standard.","marker":"[19]"},{"why":"Defines SELFIES and the bracket-level tokenization that converts molecules into index sequences.","marker":"[23]"},{"why":"Defines the QK-LSTM architecture whose quantum kernels are the hybrid quantum-classical branch of the experiments.","marker":"[28]"},{"why":"Provides the Optuna hyperparameter optimization used to select the four configurations and top-three averaged models.","marker":"[31]"},{"why":"Supplies the recommendation of five augmented samples per molecule that sets the augmentation budget.","marker":"[32]"},{"why":"Defines the LSTM baseline that the classical branch and the QK-LSTM extension build on.","marker":"[24]"}],"fun_headline_variants":["Augmented SELFIES boost molecule prediction by 6%","Quantum LSTM matches classical with augmented SELFIES","SELFIES augmentation lifts ROC-AUC 6% over SMILES","Augmented SELFIES outperform SMILES in hybrid quantum models","Side-effect prediction: augmented SELFIES beat SMILES"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gains rest on the assumption that the random 80/10/10 split, the Optuna hyperparameter search, and the top-three-model averaging treat all six conditions equally; if different conditions saw different splits or different search budgets, the 0.031 ROC-AUC gap could be an artifact of split luck or selection bias rather than an effect of the input representation.","fun_headline_variants_meta":{"raw":{"variants":["Augmented SELFIES boost molecule prediction by 6%","Quantum LSTM matches classical with augmented SELFIES","SELFIES augmentation lifts ROC-AUC 6% over SMILES","Augmented SELFIES outperform SMILES in hybrid quantum models","Side-effect prediction: augmented SELFIES beat SMILES"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000424,"raw_usage":{"total_tokens":2208,"prompt_tokens":1013,"completion_tokens":1195,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":629,"completion_tokens_details":{"reasoning_tokens":1110}},"tokens_in":629,"tokens_out":1195,"duration_ms":9506,"temperature":1.0,"reasoning_tokens":1110,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:19:42.446709+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun all six configurations on a fixed shared split with the same Optuna budget per condition and check whether augmented SELFIES still beats plain SMILES by roughly 0.03 ROC-AUC; as a second check, apply the same pipeline to another MoleculeNet classification dataset, such as BACE or BBBP, where the augmentation advantage can be confirmed or refuted.","supporting_citations":[{"cited_title":"Moleculenet: a benchmark for molecular machine learning,","cited_arxiv_id":null,"evidence_quote":"Provides the SIDER dataset, the 80/10/10 split methodology, and the ROC-AUC evaluation standard."},{"cited_title":"Self-referencing embedded strings (selfies): A 100% robust molecular string representation,","cited_arxiv_id":null,"evidence_quote":"Defines SELFIES and the bracket-level tokenization that converts molecules into index sequences."},{"cited_title":"Optuna: A next-generation hyperparameter optimization framework,","cited_arxiv_id":null,"evidence_quote":"Provides the Optuna hyperparameter optimization used to select the four configurations and top-three averaged models."},{"cited_title":"Faster and more diverse de novo molecular optimization with double-loop reinforcement learning using augmented smiles,","cited_arxiv_id":null,"evidence_quote":"Supplies the recommendation of five augmented samples per molecule that sets the augmentation budget."},{"cited_title":"Long short-term memory,","cited_arxiv_id":null,"evidence_quote":"Defines the LSTM baseline that the classical branch and the QK-LSTM extension build on."}],"review_version":1}