{"id":"c53e77d8-5790-4367-962a-1456a6b5bdd9","arxiv_id":"2411.15483","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A probabilistic conditional GAN is claimed to beat standard QSAR models and a transformer on LSD1 inhibitor activity, but the supporting detail is incomplete and the reported numbers conflict.","lead":"This paper presents Prob-cGAN, a conditional GAN that predicts how strongly a molecule inhibits the cancer-related enzyme LSD1, and reports high accuracy on 931 compounds from the ChEMBL database. The result could matter for drug discovery if it reproduces, but the paper lacks code, data, and consistent numbers.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central metric report is internally inconsistent: abstract R2=0.739/RMSE=0.562 contradict Table 1 R2=0.813/RMSE=0.484 for the same comparison, leaving the outperformance claim without a stable evidentiary basis.","rationale":"The reader's weakest_assumption identifies the single random 80/20 split as the key risk. I agree that split design is a serious concern, but the paper has an even more immediate problem: the headline numbers do not agree with each other. The abstract states top-1 R2=0.739 and RMSE=0.562 for Prob-cGAN, while Table 1 states R2=0.8131 and RMSE=0.4838; both are presented as the result of comparing Prob-cGAN with Smiles-Transformer and cGAN. One of these sets must be wrong, or they must correspond to different experimental conditions that the paper never defines. Since the central claim is entirely quantitative superiority, this unresolved inconsistency means the claim is not currently supported by the manuscript. The single-split issue matters for generalization, but the inconsistency is more load-bearing because it undermines the very numbers on which the comparison rests. I therefore recommend keeping the REJECT verdict; this stress-test pass does not change the reader's conclusion. I give partial credit for concrete details such as the RDKit fingerprint settings and dataset source, but those do not resolve the quantitative contradiction.","tokens_in":7758,"tokens_out":2865,"duration_ms":24878,"concrete_test":"Obtain the 931-compound set from ChEMBL 28 and rerun the described pipeline (Morgan r=3, L=512; Word2Vec 300-d; autoencoder to 203-d; Prob-cGAN with f-GAN loss) on the same 80/20 random split used in the paper, computing R2 and RMSE on the holdout. Then determine which of the two reported values (0.739/0.562 or 0.8131/0.4838) is reproduced, or if neither is; if neither matches, the central claim is not reproducible. In the absence of code or data, a minimal alternative is to ask the authors to provide the exact split index, hyperparameters, and per-run predictions, and verify that the abstract and Table 1 numbers arise from the same experimental condition.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim is that Prob-cGAN beats all baselines by a large margin. That claim rests on specific numbers, but the paper gives two incompatible sets. The abstract reports top-1 R2 of 0.739 and RMSE of 0.562 versus Smiles-Transformer (0.591/0.708) and cGAN (0.488/0.791). Section 5.1 and Table 1 report Prob-cGAN at 0.8131/0.4838 and Smiles-Transformer at 0.6097/0.6990. Table 2 gives cGAN 0.488/0.791, matching the abstract for cGAN, yet Prob-cGAN and Smiles-Transformer differ between abstract and body. No explanation is given for a 'top-1' setting or for why the same comparison changes by 0.074 in R2 and 0.078 in RMSE. Because the paper's main quantitative result is self-contradictory, the conclusion of superiority is not supported by the evidence as presented. The single random split, missing scaffold split, absent code/data, and unevaluated probabilistic outputs further weaken the case, but the inconsistency is the most load-bearing: even a perfect experimental protocol cannot establish a claim whose reported outcome is ambiguous.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes Prob-cGAN, a conditional generative adversarial network with a noise-injection generator and f-GAN discriminator, for predicting LSD1 inhibitor activity (pChEMBL) from a dataset of 931 ChEMBL compounds. Input features combine Morgan fingerprints and Word2Vec embeddings, reduced by an autoencoder to 203 dimensions. The authors report that Prob-cGAN substantially outperforms several machine learning baselines and a Smiles-Transformer. The paper does not provide code, data, hyperparameters, or a detailed training protocol.","tokens_in":8065,"tokens_out":4637,"duration_ms":38220,"significance":"The proposed application of a cGAN with probabilistic outputs to QSAR-style activity prediction is topical, and the ablation study (Table 2) attempts to quantify contributions of autoencoder and word embedding components. If the reported results were reproducible and robust to scaffold splits, the method could be a useful addition to the LSD1 modeling toolbox. However, the significance as presented is undercut by an unresolved internal inconsistency in the headline numbers and by an evaluation protocol that does not support the claim of superior generalization. No code or data are released, so the empirical contribution cannot be verified or built upon by other researchers.","major_comments":[{"comment":"The paper reports two incompatible sets of headline results for the same comparison. The abstract states that Prob-cGAN achieves a top-1 R2 of 0.739 and RMSE of 0.562, against Smiles-Transformer (0.591/0.708) and cGAN (0.488/0.791). Table 1 and Section 5.1 report Prob-cGAN at 0.8131 (±0.0008) and 0.4838 (±0.0010), and Smiles-Transformer at 0.6097 (±0.0078) and 0.6990 (±0.0070). The cGAN numbers in Table 2 (0.488/0.791) agree with the abstract but the Prob-cGAN and Smiles-Transformer numbers differ by up to 0.074 in R2 and 0.078 in RMSE. The concept of 'top-1' is never defined, and no explanation is offered for which table the abstract is quoting. Because the paper's central claim is that Prob-cGAN is superior, and the two sets of numbers support different effect sizes, the claim as presented is not supported by a stable evidence base. The authors must reconcile these numbers and specify the exact experimental condition (e.g., best seed vs. mean over runs) for each reported metric.","section":"Abstract vs. Section 4.1/Table 1"},{"comment":"The evaluation uses a single random 80/20 split of 931 ChEMBL compounds, with no scaffold split, temporal split, or repeated cross-validation. Random splitting of a set of inhibitors commonly overestimates performance on new chemical series because structurally similar compounds appear in both training and test sets. The claim in Section 6 that Prob-cGAN is a 'powerful instrument for molecular property predictions' requires demonstration on a scaffold-based or external test set; otherwise the reported R2 cannot be taken as evidence of generalization to novel LSD1 inhibitors.","section":"Section 4"},{"comment":"The manuscript does not provide code, data, hyperparameter values, random seeds, or training details for any of the models, including the deep baselines. The feature pipeline (Morgan fingerprint radius 3, length 512, Word2Vec dimension 300, autoencoder dimension 203) is described, but the network architectures, learning rates, batch sizes, number of epochs, and regularization are omitted. The reported standard deviations (e.g., 0.0008 for Prob-cGAN in Table 1) cannot be interpreted without knowing how many runs and which seeds they are over. This is below the reproducibility standard expected for a machine-learning benchmark paper.","section":"Sections 2-4 / reproducibility"},{"comment":"The paper advertises 'probabilistic predictions' and uncertainty estimation as the key novelty, but the evaluation never assesses these outputs. No calibration plot, coverage probability, interval width, or any uncertainty metric is reported; only point-estimate R2 and RMSE are given. To substantiate the probabilistic claim, the authors should specify how the predictive distribution is derived from the generator and evaluate it with a proper scoring rule or calibration test.","section":"Section 3"},{"comment":"Most baselines in Table 1 (SVR, Ridge, Random Forest, Decision Tree, XGBoost, MLP, K-Neighbours) are reported as single values with no standard deviation or confidence interval, while Prob-cGAN is reported as a mean over some runs. This makes the 'significantly outperforms' statement in Section 5.1 unsupported; without error bars on the baselines, even if the mean differences are large, one cannot assess statistical significance. In addition, the manuscript does not compare against previously published LSD1-specific deep learning models (e.g., the GCN and transfer-learning models cited in the introduction), which limits the benchmark's completeness.","section":"Table 1"}],"minor_comments":[{"comment":"Section 5.1 repeatedly refers to 'Table 3', but the actual results appear in Table 1; Table 3 does not exist in the manuscript.","section":"Section 5.1"},{"comment":"The term 'top-1 R2' is undefined; is it the best performance over multiple runs or something else?","section":"Abstract"},{"comment":"The spelling of the model name is inconsistent ('Prob-cGAN', 'prob-cGAN', 'proposed prob-cGAN'); please unify.","section":"Throughout"},{"comment":"The section heading contains a typo: 'Comparsion' should be 'Comparison'.","section":"Section 5.1 heading"},{"comment":"Section 2 states that a pre-trained Word2Vec model is used, but gives no details about the training corpus or model version; this feature cannot be reproduced by readers.","section":"Section 2"},{"comment":"Section 4 contains a grammatical fragment: 'the corresponding biological activity, To ensure that the dataset is suitable...' — please rephrase the sentence.","section":"Section 4"}],"recommendation":"reject","confidential_remarks":"This manuscript reports an unreproducible empirical claim with two mutually inconsistent sets of headline results. Even setting aside the inconsistency, the absence of code/data and the single-split evaluation are serious limitations for a cs.CE benchmark paper. If the authors can provide code, data, and a reconciled, scaffold-split evaluation, the method might warrant further review, but in its current form the central claim is not verifiable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know that the headline results in this paper do not survive contact with the body. The abstract reports Prob-cGAN at R2=0.739 and RMSE=0.562; Table 1 reports 0.8131 and 0.4838 for the same comparison against Smiles-Transformer. Those are two different claims, and the paper never acknowledges the gap. Since every downstream conclusion leans on that outperformance, the central evidence is currently ambiguous.\n\nWhat is actually new is narrow: applying a conditional GAN regressor to LSD1 inhibitor activity on ChEMBL, combining Morgan fingerprints and SMILES word embeddings with an autoencoder. The ablation study (with/without autoencoder, with/without word embedding) is a reasonable attempt to assign credit. That said, the novelty claim is oversold—probabilistic regression with cGANs is exactly Oskarsson's 2020 thesis, and Aggarwal et al. benchmarked cGAN regression in 2019. Calling this \"the first time\" for small-molecule activity prediction is a stretch.\n\nThe soft spots go beyond the inconsistent numbers. There is no code or data release, one random 80/20 split without scaffold splitting, and no error bars for the classical baselines (the Prob-cGAN ±0.0008 standard deviation is suspiciously tight). The probabilistic predictions—the paper's stated reason for being—are never evaluated: no calibration plot, no interval coverage, nothing. And the random split means the generalization claim is weak out of the box.\n\nNone of this makes the method worthless; the architecture is a plausible combination. But as written, the paper does not give the reader a stable number to trust. The most useful single action would be for the authors to release the data, split, and code, then report a repeated split with the same protocol for every method.\n\nWho is this for? Practitioners working on QSAR for epigenetic targets might find the descriptor combination worth trying, but they should not cite the numerical results yet. I would not bring it to our reading group in its current form, but I would send it to a referee: the claim is consequential enough that it deserves proper scrutiny, and the referee can demand the missing artifacts. Expect heavy revisions.\n\nMy recommendation: engage, but only if the authors can make the numbers consistent and share the code.","headline":"The two headline results disagree with each other, so the claimed outperformance is currently unsupported; worth engaging only if the authors release code and a consistent set of numbers.","tokens_in":8587,"tokens_out":2896,"would_cite":false,"duration_ms":26514,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Prob-cGAN, a probabilistic conditional GAN with noise-injected generation, predicts LSD1 inhibitor activity with reported R2 0.8131 and RMSE 0.4838, beating Random Forest and Smiles-Transformer on ChEMBL data.","keywords":["LSD1","conditional generative adversarial network","activity prediction","probabilistic regression","SMILES","Morgan fingerprint","ChEMBL","QSAR"],"falsifier":"Run the same pipeline with scaffold-based splitting or repeated stratified cross-validation on the same 931 compounds; if the $R^2$ drops near the Random Forest or cGAN baseline under splits that separate chemical series, the claimed generalization advantage is not supported. Also, reproducing the exact configuration should yield one consistent $R^2$; the paper currently reports 0.8131 in Table 1 and 0.739 in the abstract, so a reader can check which number a faithful re-run produces.","tokens_in":7566,"feed_emoji":"🧬","tokens_out":6451,"duration_ms":56174,"temperature":0.7,"pith_summary":"The paper proposes Prob-cGAN, a probabilistic conditional generative adversarial network that predicts the pChEMBL activity of LSD1 inhibitors from a combination of Morgan fingerprints and SMILES word embeddings. On 931 ChEMBL compounds split 80/20, the model reports an $R^2$ of 0.8131 and an RMSE of 0.4838, outperforming Random Forest, SVR, XGBoost, and a Smiles-Transformer baseline. The authors argue that injecting noise into the generator and using an f-GAN discriminator lets the model absorb distributional shift and output an uncertainty estimate rather than a single point. If this holds, it gives drug-discovery teams a way to rank candidate LSD1 inhibitors with a measure of confidence on small, noisy datasets.","feed_headline":"Prob-cGAN lifts LSD1 activity R2 to 0.81","feed_subtitle":"A noise-injecting GAN beats Random Forest and Smiles-Transformer on 931 ChEMBL compounds.","key_machinery":"The load-bearing object is the Prob-cGAN architecture itself. It starts from a fused descriptor: 512-bit Morgan fingerprints concatenated with 300-dimensional Word2Vec SMILES embeddings (812 dimensions total), which an autoencoder compresses to 203 dimensions. The generator concatenates a random noise vector $z$ with the hidden representation at each layer, so the network learns where noise best represents uncertainty; the discriminator uses separate pathways for true and generated samples and is trained with an f-GAN loss, a variational divergence-minimization objective. Together these pieces let the model produce a conditional distribution of activity rather than a single point estimate, which is the mechanism the paper credits for its resilience to distribution shift.","core_discovery":"The paper's central discovery is that recasting inhibitor-activity regression as probabilistic conditional generation—where the generator receives noise at every layer and a dual-path f-GAN discriminator scores real versus generated activity—yields markedly better predictions of LSD1 inhibitor activity than standard regressors. On a random 80/20 split of 931 ChEMBL compounds, the authors report $R^2 = 0.8131$ and RMSE = 0.4838, versus 0.6686 and 0.6441 for Random Forest and 0.6097 and 0.6990 for Smiles-Transformer; the abstract reports a separate best result of $R^2 = 0.739$ and RMSE = 0.562 against the same baselines. The authors attribute the improvement to the model's ability to approximate unknown conditional densities and to its probabilistic output, which gives an uncertainty estimate for each predicted activity.","pith_inferences":["The reported superiority rests on a single random split; a scaffold split or repeated cross-validation would test whether the advantage survives on chemical series absent from training.","Because the abstract and Section 5 give different headline numbers (0.739/0.562 versus 0.8131/0.4838), a faithful re-run needs to state which configuration produced which; the mechanism claim is independent of that discrepancy but the precise magnitude is not.","If noise injection is the active ingredient, the same architecture should generalize to other epigenetic targets with small datasets; comparing Prob-cGAN against Random Forest on, say, EZH2 or HDAC inhibitors would be a direct transfer test.","The f-GAN divergence choice is not isolated in the ablation; swapping in a different divergence term would show whether the probabilistic GAN framing or the specific divergence drives the gain."],"forward_implications":["If the claimed accuracy transfers, Prob-cGAN can rank LSD1 inhibitor candidates by predicted pChEMBL before synthesis, reducing the number of compounds that need experimental testing.","The probabilistic output gives each prediction an uncertainty, which is useful for prioritizing compounds whose predictions are most reliable.","The architecture's tolerance for unknown density functions means it can be applied to other small epigenetic datasets where activity distributions are irregular.","Combining structural fingerprints with SMILES text embeddings appears to matter: ablations removing word embeddings and the autoencoder drop $R^2$ to 0.258, so the fusion pipeline is a key component."],"supporting_citations":[{"why":"Supplies the dataset: 931 LSD1 compounds with pChEMBL values from ChEMBL version 28.","marker":"[26]"},{"why":"Provides the Smiles-Transformer baseline that the paper compares against for molecular property prediction.","marker":"[33]"},{"why":"Establishes the conventional cGAN as a regression method, which serves as the direct ablation baseline for Prob-cGAN.","marker":"[22]"},{"why":"Defines the Morgan fingerprint descriptor used to encode 2D molecular structure.","marker":"[15]"},{"why":"Supplies Word2Vec, the method used to embed SMILES strings as 300-dimensional vectors.","marker":"[23]"},{"why":"Motivates the autoencoder used to reduce 812-dimensional features to 203 dimensions.","marker":"[24]"},{"why":"Introduces the f-GAN framework that the discriminator loss is based on.","marker":"[36]"},{"why":"Provides background on probabilistic regression using conditional generative adversarial networks with unknown densities.","marker":"[21]"}],"fun_headline_variants":["Prob-cGAN hits R2 0.81, tops LSD1 activity baselines","Noise-injected GAN predicts LSD1 inhibitors at R2 0.81","Prob-cGAN delivers R2 0.81 for LSD1 activity prediction","LSD1 inhibitor activity forecast R2 0.81 with Prob-cGAN","Prob-cGAN beats Smiles-Transformer on LSD1 with R2 0.81"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that one random 80/20 split of 931 ChEMBL compounds yields a test set from the same distribution as training, so the reported $R^2$ reflects generalization rather than shared scaffolds or assay artifacts.","fun_headline_variants_meta":{"raw":{"variants":["Prob-cGAN hits R2 0.81, tops LSD1 activity baselines","Noise-injected GAN predicts LSD1 inhibitors at R2 0.81","Prob-cGAN delivers R2 0.81 for LSD1 activity prediction","LSD1 inhibitor activity forecast R2 0.81 with Prob-cGAN","Prob-cGAN beats Smiles-Transformer on LSD1 with R2 0.81"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000754,"raw_usage":{"total_tokens":3335,"prompt_tokens":907,"completion_tokens":2428,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":2319}},"tokens_in":523,"tokens_out":2428,"duration_ms":15929,"temperature":1.0,"reasoning_tokens":2319,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:14:10.518125+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same pipeline with scaffold-based splitting or repeated stratified cross-validation on the same 931 compounds; if the $R^2$ drops near the Random Forest or cGAN baseline under splits that separate chemical series, the claimed generalization advantage is not supported. Also, reproducing the exact configuration should yield one consistent $R^2$; the paper currently reports 0.8131 in Table 1 and 0.739 in the abstract, so a reader can check which number a faithful re-run produces.","supporting_citations":[{"cited_title":"The chembl database in 2017","cited_arxiv_id":null,"evidence_quote":"Supplies the dataset: 931 LSD1 compounds with pChEMBL values from ChEMBL version 28."},{"cited_title":"Benchmarking Regression Methods: A comparison with CGAN","cited_arxiv_id":"1905.12868","evidence_quote":"Establishes the conventional cGAN as a regression method, which serves as the direct ablation baseline for Prob-cGAN."},{"cited_title":"A structure- based platform for predicting chemical reactivity","cited_arxiv_id":null,"evidence_quote":"Defines the Morgan fingerprint descriptor used to encode 2D molecular structure."},{"cited_title":"Word2vec","cited_arxiv_id":null,"evidence_quote":"Supplies Word2Vec, the method used to embed SMILES strings as 300-dimensional vectors."},{"cited_title":"Greedy layer-wise training of deep networks","cited_arxiv_id":null,"evidence_quote":"Motivates the autoencoder used to reduce 812-dimensional features to 203 dimensions."},{"cited_title":"f-gan: Training generative neural samplers using variational divergence minimization","cited_arxiv_id":null,"evidence_quote":"Introduces the f-GAN framework that the discriminator loss is based on."},{"cited_title":"Probabilistic regression using conditional generative adversarial networks, 2020","cited_arxiv_id":null,"evidence_quote":"Provides background on probabilistic regression using conditional generative adversarial networks with unknown densities."}],"review_version":1}