{"id":"e77b25c4-f260-4ff8-8010-c920961c722c","arxiv_id":"2505.08688","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"ChemXploreML, a new modular desktop app, predicts five molecular properties with tree-based models and two embeddings, reaching R^2 up to 0.93 for critical temperature and matching 300-dimensional Mol2Vec with 32-dimensional VICGAE at about 10x lower compute.","lead":"This paper presents ChemXploreML, a desktop application that lets chemists predict molecular properties with machine learning without writing code. It reports R^2 up to 0.93 on critical temperature and finds that a compact 32-dimensional molecular embedding matches a 300-dimensional one at much lower compute cost.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Pre-split cleanlab pruning biases the reported R² and confounds the Mol2Vec vs VICGAE comparison because the cleaned test sets differ per embedder.","rationale":"The reader's weakest assumption correctly identifies the cleanlab pre-processing as the most load-bearing weakness. The central claim of excellent prediction performance (R² up to 0.93) and of VICGAE being comparable to Mol2Vec depends on the reported 5-fold CV metrics being honest estimates of generalization. Applying cleanlab before the split violates that condition in two ways: it can remove hard-to-predict molecules (biasing the absolute R² upward), and it creates different cleaned test sets for the two embeddings (confounding the relative comparison). Evidence for this concern is grounded in the manuscript itself: Section 6 explicitly places cleanlab 'post-embedding' and before model training, Table 1 quantifies the differing retention rates (as low as 80.6% for MP/VICGAE and 94.9% for CT/VICGAE), and Section 8.2 reports all metrics as 5-fold CV on the cleaned sets. I also considered the possibility of hyperparameter-selection leakage from using the same 5-fold CV inside Optuna and for final reporting, which would also inflate R², but the cleanlab issue is more direct because it selectively changes the test set composition per embedder and therefore threatens the VICGAE-vs-Mol2Vec claim, not just the absolute numbers. The proposed concrete test is minimal and decisive: comparing metrics with cleaning nested inside folds versus the reported pre-cleaning approach will reveal whether the bias is material. The reader's CONDITIONAL verdict is appropriate because the concern is specific, easily addressable by re-running the evaluation, and the paper otherwise delivers a reproducible software artifact with real benchmarking data.","tokens_in":27813,"tokens_out":7017,"duration_ms":70678,"concrete_test":"Re-run the full pipeline (embedding, cleanlab cleaning, Optuna tuning, 5-fold CV) for both embeddings on the MP and CT datasets, with cleaning applied only inside each training fold and evaluation on the full held-out test fold (no cleaning applied to the test fold). Compare the resulting R² and RMSE against Table 4. If the CT R² drops by more than ~0.02 or the MP R² by more than ~0.03, the pre-split cleaning is materially inflating the reported performance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 6 states that cleanlab is applied once to each validated dataset before any cross-validation split, and Table 1 shows that the cleaned sets differ substantially by embedder: for CT, Mol2Vec retains 819/819 molecules while VICGAE retains 777/818; for MP, Mol2Vec retains 6167/7476 and VICGAE 6030/7200. Because cleaning happens before the 5-fold CV, the reported metrics in Table 4 are computed on test folds that are not representative of the original CRC data. If cleanlab's outlier score correlates with prediction difficulty, the removed molecules are disproportionately the high-error examples, inflating the headline R² values (0.93 for CT, 0.925 for BP). The paper offers no evidence that the pruned points are true label errors, and the different cleaned-set sizes mean the Mol2Vec and VICGAE pipelines are evaluated on different molecules. The 'VICGAE accuracy comparable to Mol2Vec' claim is therefore not a controlled comparison: the extra 42 CT molecules retained only for Mol2Vec could be exactly those where VICGAE embeddings are weak, artificially narrowing the accuracy gap. No metrics on the uncleaned data or with cleaning nested inside the CV folds are reported, so the magnitude of the bias is unknown.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents ChemXploreML, a modular desktop application for molecular property prediction, and validates it on five CRC Handbook properties (melting point, boiling point, vapor pressure, critical temperature, critical pressure). The pipeline combines SMILES/SELFIES handling, RDKit validation, two molecular embeddings (Mol2Vec, 300-D, and VICGAE, 32-D), UMAP/DBSCAN chemical-space exploration, cleanlab-based outlier removal, Optuna hyperparameter tuning, and four tree-based regressors (GBR, XGBoost, CatBoost, LightGBM). The best reported results are R² = 0.93 for critical temperature and 0.925 for boiling point with Mol2Vec, while VICGAE is reported as comparable at much lower dimensionality and with roughly 10x speedup on the melting-point dataset. The authors claim the platform's modular design makes advanced ML accessible and reproducible.","tokens_in":28114,"tokens_out":6057,"duration_ms":56157,"significance":"If the evaluation were unbiased, the contribution would be a practically useful open-source platform and a favorable efficiency/accuracy trade-off for VICGAE. The manuscript's concrete strengths include public release of code and data, installation packages, a documentation site, and a thorough exploratory analysis of chemical space. The UMAP/DBSCAN clustering analyses are informative and chemically plausible. However, the central quantitative claims rest on an evaluation protocol that removes model-dependent subsets of samples before cross-validation and that reuses the same CV folds for hyperparameter tuning and final reporting. Until the evaluation is made statistically sound, the reported R² values and the Mol2Vec/VICGAE comparison cannot be accepted as stated.","major_comments":[{"comment":"The evaluation protocol applies cleanlab once to each validated dataset before the 5-fold CV split, and the cleaned sets differ substantially by embedder (CT: Mol2Vec 819/819 retained, VICGAE 777/818; MP: Mol2Vec 6167/7476 retained, VICGAE 6030/7200). Because cleanlab scores samples using predictions from a regression model, pruning before the split can selectively remove hard-to-predict molecules, making the test folds in Table 4 artificially easy and non-representative of the original CRC data. The Mol2Vec versus VICGAE comparison is also uncontrolled because the two pipelines are evaluated on different molecule sets. No results on the uncleaned datasets or with cleaning nested inside each CV fold are reported, so the magnitude of this bias is unknown; the headline R² values (0.93 CT, 0.925 BP) and the speed comparison rest on this assumption.","section":"Section 6, Table 1, Table 4"},{"comment":"Hyperparameter optimization is performed with 5-fold CV ('For each model, we performed extensive hyperparameter tuning using Optuna with 5-fold cross-validation'), and the reported metrics in Table 4 are also computed with 5-fold CV on the same cleaned data. No held-out test set or nested CV is described. Selecting hyperparameters by minimizing RMSE on the same folds used to report R² can inflate the reported performance and the ranking between Mol2Vec and VICGAE. Please report an independent final evaluation, e.g., a fixed held-out split or nested CV.","section":"Section 7.2, Section 8.1"},{"comment":"The conclusion states that VICGAE 'even outperformed Mol2Vec for vapor pressure prediction,' but Section 8.2 says the VP difference is likely not statistically significant because uncertainty ranges overlap (e.g., CatBoost VP R² 0.4(2) versus 0.32(7)). This is an internal contradiction in one of the paper's stated advantages of VICGAE and should be corrected to say 'comparable' rather than 'outperformed.'","section":"Section 8.2, Section 9, Table 4"}],"minor_comments":[{"comment":"In most panels the inset labels appear to swap RMSE and MAE relative to Table 4 (e.g., MP GBR inset gives RMSE 30(2) and MAE 39(2), whereas Table 4 lists RMSE 39(2) and MAE 30(2)). Please correct the figure or the table.","section":"Figure 6"},{"comment":"The sentence 'VICGAE showed R² values of 0.4(2) compared to Mol2Vec's 0.32(7)' omits the property name; specify vapor pressure (VP) and the model (CatBoost) for clarity.","section":"Section 9"},{"comment":"The text says the melting point dataset experienced an 18% reduction in Mol2Vec embeddings, while Table 1 shows 6167 of 7476 retained (17.5% reduction); please round consistently or state exact percentages.","section":"Section 6"},{"comment":"The claim that applicability domain (AD) analysis via leverage and Mahalanobis distance is implemented in ChemXploreML is not demonstrated anywhere in the paper; either provide evidence of this functionality or label it as planned.","section":"Section 9"},{"comment":"UMAP and DBSCAN parameters are described as optimized through visual assessment; since these choices affect the exploratory analysis (not the regression), the paper should state more explicitly that they are heuristic and not part of the quantitative evaluation.","section":"Section 5"}],"recommendation":"major_revision","confidential_remarks":"The evaluation protocol is the key gate. If the authors cannot provide results on uncleaned data or with cleaning nested inside the CV folds, I would not certify the reported R² values or the Mol2Vec/VICGAE comparison. The manuscript's main contribution is an engineering platform; the quantitative claims need to be re-derived under a sound protocol."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe short version: this is a real software artifact with a useful benchmark, but the headline accuracy numbers are not to be trusted as reported, and the Mol2Vec vs VICGAE comparison is not a controlled experiment. The problem is exactly where the stress-test points: cleanlab pruning is applied once, before the 5-fold CV split (Section 6, Table 1), and the two embedders are evaluated on different cleaned datasets (e.g., CT: Mol2Vec 819/819, VICGAE 777/818; MP: 6167 vs 6030). If cleanlab removes hard-to-predict molecules rather than true label errors, the test folds are artificially easy, and the R^2 values (0.93 CT, 0.925 BP) are optimistic. No metrics on the uncleaned data or with cleaning nested inside folds are reported, so the magnitude of the bias is unknown. That is a load-bearing flaw for the accuracy claims.\n\nWhat is new and good: ChemXploreML itself is a modular desktop tool with published code, docs, and data on Zenodo/GitHub. That is reproducible and useful for chemists who want to run these pipelines without writing code. The benchmark—two embeddings, four tree-based models, five CRC properties—is new empirical data, and the VICGAE efficiency result (32-d vs 300-d, roughly 10x faster on the MP dataset) is probably robust because it is about dimensionality and compute time, not about the cleaning step. The UMAP/DBSCAN chemical-space analysis is a nice exploratory addition, though not the main claim.\n\nSoft spots beyond the main one: Figure 6 has RMSE and MAE swapped in the text insets (compare to Table 4). The text says \"only a minimal fraction of data is pruned\" right after reporting an 18% reduction in the MP dataset—that is inconsistent. Hyperparameters are tuned on the same cleaned data via Optuna, which is standard but compounds the circularity.\n\nWho this is for: anyone building or choosing molecular property prediction pipelines, especially in astrochemistry-adjacent groups given VICGAE's provenance. It deserves a serious referee, but the referee should ask for the evaluation to be re-run with cleaning nested inside the CV folds, or at least with scores on the uncleaned data. If the cleaned-vs-uncleaned numbers are close, the paper is fine; if not, the conclusions need substantial revision.\n\nMy take: conditional accept is the right disposition. I would not cite the headline R^2 values until the cleaning issue is addressed, but I would cite the tool and the efficiency benchmark.","headline":"Useful modular tool and a real benchmark, but the headline R^2 values are inflated by pre-split cleanlab pruning, and the Mol2Vec vs VICGAE comparison runs on different cleaned datasets.","tokens_in":28624,"tokens_out":2739,"would_cite":false,"duration_ms":24918,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a 32-dimensional molecular embedding can rival a 300-dimensional one, cutting runtime by about tenfold while keeping prediction accuracy within a few points.","keywords":["molecular property prediction","machine learning","molecular embeddings","VICGAE","Mol2Vec","tree-based ensembles","cheminformatics"],"falsifier":"Recompute every reported R² and RMSE by applying the outlier-cleaning step independently inside each cross-validation fold, and compare those numbers with scores from the uncleaned data; a material drop in melting-point R² from its reported 0.86 would show the headline accuracy is an artifact of data selection.","tokens_in":27622,"feed_emoji":"🧪","tokens_out":8956,"duration_ms":79345,"temperature":0.7,"pith_summary":"This paper claims that a modular desktop pipeline can make modern machine-learning property prediction practical for chemists, and that on well-distributed datasets the attainable accuracy is set mostly by the data itself rather than by the choice of molecular embedding. Using five thermodynamic properties drawn from a standard chemistry handbook, the authors report R² values up to 0.93 for critical temperature and 0.925 for boiling point with a 300-dimensional Mol2Vec embedding. They then show that a 32-dimensional VICGAE embedding stays within a few points of that accuracy while running about ten times faster on the largest dataset for gradient boosting. If these results hold, cheminformatics workflows can reduce the computational cost of representation without sacrificing much predictive power, making high-throughput screening more accessible.","feed_headline":"A 32-D molecular embedding rivals 300-D Mol2Vec at 10x speed","feed_subtitle":"ChemXploreML pipeline scores R² 0.93 for critical temperature and 0.925 for boiling point.","key_machinery":"The load-bearing mechanism is the embedding-to-regressor pipeline with a head-to-head embedding comparison. Mol2Vec represents each molecule as a 300-dimensional vector by summing learned fragment embeddings; VICGAE is a GRU autoencoder trained on partially masked SELFIES strings with variance-invariance-covariance regularization, producing 32-dimensional vectors that resist latent-space collapse and keep chemically similar molecules close. Five-fold cross-validation with Bayesian hyperparameter search is what converts those vectors into the reported R², RMSE, and MAE values, and the per-model per-property tables make the comparison traceable.","core_discovery":"The central claim is that ChemXploreML works as an end-to-end platform: given molecular strings, it builds embeddings, cleans the data, tunes four tree-based regressors, and returns cross-validated property predictions. The benchmark portion claims that on well-distributed datasets the results are strong, with CatBoost on Mol2Vec reaching R² = 0.931(7) for critical temperature, 0.925(8) for boiling point, and an RMSE near 36 °C for melting point, which the authors place at the level of published QSPR models. The more transferable discovery is the embedding comparison: VICGAE's 32-dimensional vectors keep accuracy within a few points of Mol2Vec's 300-dimensional vectors while giving roughly a 10-fold speedup for gradient boosting on the largest dataset. The authors therefore present the compact embedding as the practical choice for high-throughput screening and the modular pipeline as the reusable vehicle for such comparisons.","pith_inferences":["Pith inference: if the 32-dimensional embeddings are truly within a few points of 300-dimensional ones, representation cost—not model capacity—is the practical bottleneck for high-throughput screening, and the same speed advantage should transfer to any regressor that consumes embeddings directly.","Pith inference: the vapor-pressure failure suggests neither embedding encodes non-covalent intermolecular interactions such as hydrogen bonding; adding explicit interaction-aware descriptors and checking whether VP R² rises above 0.4 would test that diagnosis.","Pith inference: because outlier removal happens before the cross-validation split, all reported metrics may be optimistic; a re-evaluation that cleans inside each fold, or reports on uncleaned data, is the direct way to establish the true ceiling."],"forward_implications":["Compact 32-dimensional embeddings can substitute for 300-dimensional ones in tree-based property models, cutting training time by roughly an order of magnitude on large datasets.","Data distribution and sample size dominate achievable accuracy: near-normal, well-populated properties such as critical temperature reach R² around 0.93, while the small, heavily skewed vapor-pressure set stalls near R² = 0.4 regardless of embedding.","Because the pipeline is modular, adding a new embedding or regressor does not require rearchitecting the workflow, so the same code can be pointed at classification tasks, larger libraries, or newly integrated representations.","Melting-point RMSE around 36 °C matches published structure-property models, suggesting the platform can serve as a practical screening tool, not just a benchmark harness."],"supporting_citations":[{"why":"Defines the Mol2Vec fragment-embedding method and supplies the 300-dimensional baseline vectors.","marker":"[12]"},{"why":"Introduces VICGAE, the 32-dimensional GRU autoencoder embedding that is the paper's compact alternative.","marker":"[14]"},{"why":"Provides the variance-invariance-covariance regularization that keeps VICGAE embeddings from collapsing.","marker":"[49]"},{"why":"One of the four tree-based regressors benchmarked in the study.","marker":"[17]"},{"why":"One of the four tree-based regressors benchmarked in the study.","marker":"[18]"},{"why":"One of the four tree-based regressors and the model that reaches the top R² values.","marker":"[19]"},{"why":"Supplies the Bayesian hyperparameter search used to tune every model before evaluation.","marker":"[30]"},{"why":"The handbook whose experimental property values form the five datasets.","marker":"[32]"},{"why":"Supplies the confident-learning outlier detection used to clean the datasets before model evaluation.","marker":"[35]"}],"fun_headline_variants":["32-D VICGAE matches 300-D Mol2Vec at 10x speed","ChemXploreML: small embeddings, 10x speedup","10x faster molecular property prediction with 32-D embeddings","Compact embeddings rival Mol2Vec in ChemXploreML","VICGAE: 32-D features, Mol2Vec-level accuracy, 10x faster"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that removing outliers from the full dataset before splitting it into cross-validation folds does not make the test folds artificially easy, since the cleaning step is not repeated inside each fold.","fun_headline_variants_meta":{"raw":{"variants":["32-D VICGAE matches 300-D Mol2Vec at 10x speed","ChemXploreML: small embeddings, 10x speedup","10x faster molecular property prediction with 32-D embeddings","Compact embeddings rival Mol2Vec in ChemXploreML","VICGAE: 32-D features, Mol2Vec-level accuracy, 10x faster"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000405,"raw_usage":{"total_tokens":2150,"prompt_tokens":1033,"completion_tokens":1117,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":649,"completion_tokens_details":{"reasoning_tokens":1016}},"tokens_in":649,"tokens_out":1117,"duration_ms":9997,"temperature":1.0,"reasoning_tokens":1016,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:48:57.633388+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute every reported R² and RMSE by applying the outlier-cleaning step independently inside each cross-validation fold, and compare those numbers with scores from the uncleaned data; a material drop in melting-point R² from its reported 0.86 would show the headline accuracy is an artifact of data selection.","supporting_citations":[{"cited_title":"R., Brunno, T","cited_arxiv_id":null,"evidence_quote":"The handbook whose experimental property values form the five datasets."},{"cited_title":"Mol2vec: Unsupervised Machine Learning Approach with Chemical Intuition","cited_arxiv_id":null,"evidence_quote":"Defines the Mol2Vec fragment-embedding method and supplies the 300-dimensional baseline vectors."},{"cited_title":"N.; Remijan, A","cited_arxiv_id":null,"evidence_quote":"Introduces VICGAE, the 32-dimensional GRU autoencoder embedding that is the paper's compact alternative."},{"cited_title":"XGBoost : A Scalable Tree Boosting System","cited_arxiv_id":null,"evidence_quote":"One of the four tree-based regressors benchmarked in the study."},{"cited_title":"LightGBM : A Highly Efficient Gradient Boosting Decision Tree","cited_arxiv_id":null,"evidence_quote":"One of the four tree-based regressors benchmarked in the study."},{"cited_title":"V.; Gulin, A","cited_arxiv_id":null,"evidence_quote":"One of the four tree-based regressors and the model that reaches the top R² values."},{"cited_title":"2024; https://github.com/cleanlab/cleanlab, original-date: 2018-05-11T01:55:21Z","cited_arxiv_id":null,"evidence_quote":"Supplies the confident-learning outlier detection used to clean the datasets before model evaluation."}],"review_version":1}