{"id":"a6b14d62-2b15-42fe-9836-42097c0b4dca","arxiv_id":"1908.09426","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A multi-modal neural network using raw promoter DNA and transcription factor expression predicts yeast stress-response gene expression with 79.5% accuracy, and its in-silico MSN2/4 knockout predictions correlate with real microarray data (Spearman 0.486).","lead":"This paper trains a neural network that combines DNA promoter sequences and the activity levels of regulatory proteins to predict how yeast genes respond to stress. It reports that this approach beats older motif-based models and that its simulated gene knockouts partially match real experiments.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Random (gene,condition) split lets the model memorize per-gene and per-condition means, so the reported 79.5% accuracy and 0.845 correlation do not establish generalization to unseen genes or conditions.","rationale":"The reader's weakest_assumption identifies exactly the load-bearing issue: the random pair split allows memorization of gene- and condition-level means, so the headline performance numbers do not measure generalization to unseen genes or unseen cellular states. This concern is central because the paper's stated goal is predicting expression of any target gene in any cellular state. The paper has genuine supporting evidence, including recovery of known cis motifs, ranking of known trans regulators, and an external MSN2/4 knockout correlation of 0.486; these are real independent checks. However, the main performance comparison in Table 1 and Figure 1a is the basis for the claim of substantial outperformance, and that comparison is compromised by the split design. A grouped re-evaluation would settle the matter, and the verdict should remain conditional until such an evaluation is provided. The ad hoc 32-fold knockout perturbation is a secondary concern that could be calibrated or justified, but the split issue is the most load-bearing. Since the reader already recommends conditional acceptance, my stress-test does not move the verdict.","tokens_in":4250,"tokens_out":2639,"duration_ms":25670,"concrete_test":"Recompute classification accuracy and regression correlation under three grouped splits: (i) leave-one-condition-out, holding out all genes for each held-out condition; (ii) leave-one-gene-out, holding out all conditions for each held-out gene; and (iii) hold out a block of genes and a block of conditions jointly. Compare against a simple additive baseline that predicts expression as the sum of a per-gene mean and a per-condition mean (or a linear model with gene and condition indicator variables). If DNN performance drops toward the additive baseline under grouped splits, the reported 79.5% and 0.845 are attributable to memorization of training genes and conditions, and the central generalization claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires predicting expression for any gene in any cellular state. The evaluation in Section 2.4 uses an 80-10-10 split of (g,c) pairs spanning all genes and all stress conditions. Under this split, every test gene and every test condition is represented in training: the same promoter sequence appears in training pairs for that gene, and the same trans-expression vector appears in training pairs for that condition. Thus a model can memorize gene-level and condition-level mean expression and combine them to score test pairs. The 79.5% classification accuracy (Table 1) and Pearson 0.845 (Figure 1a) therefore measure interpolation over seen genes and seen conditions, not out-of-sample prediction of unseen genes or unseen cellular states. Since the cis module sees the same 1kb promoter for a gene in training and test, and the trans module sees the same 472-dimensional condition vector in training and test, no component of the test input is novel; the model does not need to generalize across sequence or across conditions. A held-out-condition or held-out-gene split is required to support the stated goal. The MSN2/4 knockout correlation (Spearman 0.486) is external validation and somewhat mitigates this, but it uses an ad hoc 32-fold perturbation and only one TF; it does not rescue the main performance claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a multi-modal neural network that predicts yeast stress-response gene expression from two inputs: the raw 1 kb promoter sequence of a gene (cis module, a convolutional network) and the expression levels of 472 transcription factors and signaling molecules in a condition (trans module, a dense network). The two modalities are concatenated and integrated by dense layers, with a linear output for regression or a softmax output for three-class classification. The model is trained on the Gasch compendium of 6100 genes under 173 stress conditions, using an 80-10-10 split of (gene, condition) pairs. The authors report 79.5% classification accuracy versus 62.9% for BDTree and 60.9% for GeneClass, a test-set Pearson correlation of 0.845 for regression, recovery of known cis motifs by convolutional filter PWM analysis, and gradient-based ranking of trans regulators that includes known stress-response factors. Finally, they perform an in-silico MSN2/4 knockout by mutating its motif and reducing its expression 32-fold, and report a Spearman correlation of 0.486 between predicted and experimentally measured expression changes.","tokens_in":4514,"tokens_out":2384,"duration_ms":23510,"significance":"If the evaluation supported the claims, this would be a useful contribution: it demonstrates that a convolutional network on raw promoter sequence can be combined with trans-factor expression in a single differentiable model, that the learned cis filters recover known yeast motifs without prior motif annotation, and that the model can be probed by in-silico perturbation to generate falsifiable predictions about knockout effects. The framework is simple and reproducible in principle, and the external knockout comparison is an unusually direct validation step. However, the headline performance numbers are weakened substantially by the data-splitting protocol, which allows per-gene and per-condition memorization, and the in-silico knockout relies on an ad hoc perturbation whose numerical value is not justified. The central claim of predicting expression for any gene in any cellular state therefore requires a re-evaluation under held-out-gene and held-out-condition splits.","major_comments":[{"comment":"The random 80-10-10 split of (gene, condition) pairs does not test the central claim of predicting expression for any gene in any cellular state. Under this split, every test gene appears in training pairs with the same promoter sequence, and every test condition appears in training pairs with the same 472-dimensional trans vector, so the model can memorize gene-level and condition-level mean expression and combine them to score test pairs. The reported 79.5% accuracy and Pearson 0.845 therefore measure interpolation over seen genes and seen conditions, not out-of-sample generalization to unseen genes or unseen cellular states. The authors should report results under held-out-gene, held-out-condition, and held-out-combination splits, or justify why the central claim does not require such generalization.","section":"Section 2.4, Table 1, Figure 1a"},{"comment":"The in-silico MSN2/4 knockout is implemented by replacing every instance of the motif AGGGG with NNNNN and reducing MSN2/4 expression by 32-fold, but the manuscript gives no justification for either the specific motif representation or the 32-fold reduction. The reported Spearman correlation of 0.486 is based on a single TF and a single hand-set perturbation magnitude, so it does not provide strong evidence that the model predicts perturbation effects generally. The authors should justify the 32-fold value from known biology, perform a sensitivity analysis over perturbation magnitudes and motif-alternative choices, and ideally report results for additional TFs.","section":"Section 3.4, Figure 2b"},{"comment":"The comparison with GeneClass and BDTree reports point estimates only, with no error bars, confidence intervals, or significance tests. Because the split is random, the apparent 16.6% improvement over BDTree could be affected by split variability and by the leakage described in the first comment. Repeated stratified splits or a paired test over multiple random seeds are needed before claiming that the deep model substantially outperforms the baselines.","section":"Section 3.1, Table 1"}],"minor_comments":[{"comment":"The label 'grounth truth' is a typo and should read 'ground truth'.","section":"Figure 1"},{"comment":"The discretization thresholds of +/-0.5 fold change are stated but their influence on the classification results is not discussed; a short sensitivity analysis or a reference to previous use of these thresholds would help.","section":"Section 2.4"},{"comment":"The trans module input is described as a vector in 'stress condition C', but uppercase C is inconsistent with the lowercase c used elsewhere; please standardize notation.","section":"Section 2.2"},{"comment":"The TomTom comparison against YEASTRACT is mentioned but the number of significantly matched known motifs and the chosen significance threshold are not reported, making it hard to assess how many of the learned filters correspond to known cis-regulatory elements.","section":"Section 3.2"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know about this paper: the headline numbers are not the story. The 79.5% accuracy and 0.845 correlation come from a split that lets the model memorize per-gene and per-condition means, so they don't show generalization to new genes or new conditions. What is worth your time is the in-silico knockout experiment, which uses real external data and gives a modest but real correlation.\n\nWhat's new: the architecture is a Basset-style convolutional network on 1kb promoters, a dense layer on 472 trans-factor expression values, and a couple of integration layers. That's a straightforward combination of known pieces, but applying it to yeast stress response and evaluating a TF knockout is a sensible contribution. The model recovers known cis motifs (RPN4, SFP1, others) and ranks MSN2/4 and TPK1 as important trans regulators, which is a good sanity check. Most useful, the in-silico MSN2/4 knockout predictions correlate with an actual knockout microarray at Spearman 0.486. Modest, but it's an external perturbation test that a lot of gene-expression models never get.\n\nThe soft spots: Section 2.4 uses an 80-10-10 split of (gene, condition) pairs. Every test gene's promoter appears in training (under other conditions), and every test condition's trans vector appears in training (for other genes). The test inputs are not novel; the model can memorize gene-level and condition-level means and combine them. That makes the reported performance a measure of interpolation, not of predicting any gene in any cellular state, which is exactly the goal stated. This is a load-bearing flaw, not a cosmetic one. The baseline comparison has no error bars or significance tests; the 16.6% gap is large, so it may survive a proper test, but the paper doesn't show it. The 32-fold reduction for MSN2/4 in the knockout is arbitrary, and swapping the motif to NNNNN is crude; I'd want a sensitivity analysis.\n\nMy take: the evaluation split needs to be redone with held-out conditions or held-out genes, with error bars and code/data release. The external knockout validation is the strongest part and partially mitigates the split problem. For someone working on regulatory genomics or on evaluation pitfalls in deep learning, this is worth reading; for citation, the performance numbers shouldn't be used. I'd support sending it to peer review with a request for major revision, not desk-rejecting, because the perturbation-testing idea is valuable and the model's recovered regulators are plausible.","headline":"The performance claim is inflated by a leaky data split, but the in-silico knockout validation is a genuinely useful part.","tokens_in":5050,"tokens_out":3150,"would_cite":false,"duration_ms":31856,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A neural network that reads promoter DNA and regulator expression predicts yeast stress-response gene expression across many conditions, beating motif-based ensemble methods and approximating a real transcription-factor knockout.","keywords":["gene expression prediction","cis-regulatory sequence","trans factor expression","multi-modal neural network","yeast stress response","Saccharomyces cerevisiae","in-silico knockout","transcription factor motif"],"falsifier":"Retrain the identical architecture on the identical data but split by condition instead of by pair: hold out all pairs belonging to, say, 17 of the 173 stress conditions and test only on those held-out conditions. If accuracy on held-out conditions remains near 79.5% and regression correlation near 0.845, the generalization claim holds; if accuracy falls toward the 62.9% motif baseline, the model is largely memorizing per-condition expression means. The mirror-image test holds out entire genes rather than conditions.","tokens_in":4035,"feed_emoji":"🧬","tokens_out":7515,"duration_ms":70668,"temperature":0.7,"pith_summary":"This paper tries to establish that a single multi-modal neural network can predict the stress-response expression of any yeast gene in any stress condition from two inputs: the raw promoter DNA sequence of the gene and the expression levels of 472 transcription factors and signaling molecules in that condition. In the authors' tests on a dataset of 6,100 genes across 173 stress conditions, the classification version reaches 79.5% accuracy, compared with 62.9% for the best motif-based ensemble baseline, and the regression version achieves a Pearson correlation of 0.845 between predicted and measured expression changes. The paper's deeper claim is that cis and trans regulatory information can be learned jointly from raw data, without predefined transcription-factor motif compendia, and that the learned model can simulate regulatory perturbations. This matters because a predictive model of gene regulation would let researchers anticipate how sequence variants or transcription-factor knockouts change expression across cellular states.","feed_headline":"Neural net beats motif methods on yeast stress response at 79.5%","feed_subtitle":"Model reads raw promoter DNA plus regulator expression; its in-silico MSN2/4 knockout tracks a real microarray.","key_machinery":"The load-bearing mechanism is the multi-modal neural network architecture: a convolutional cis module takes 1 kb of promoter DNA as a one-hot encoded four-channel sequence and learns motif-like filters de novo; a dense trans module takes the expression of 472 known regulators and compresses it; an integration module concatenates the two representations through dense layers before a linear neuron (regression) or softmax neuron (classification). Reverse-complement parameter sharing lets each filter recognize a motif and its reverse complement. The integration step is what enables cis-trans interaction: the sequence features learned by the convolutional layers can be modulated by the condition-specific regulator state, which is what allows one gene to be predicted under many different stresses.","core_discovery":"On the paper's own terms, the central discovery is that a convolutional module reading raw promoter sequence, combined with a dense module reading trans-factor expression, learns regulatory programs that outperform existing models built on known motifs. The network's filters recover several known yeast stress-response motifs, and gradient-based importance scoring ranks known regulators such as MSN2/4 and TPK1 at the top. The paper also demonstrates an in-silico knockout of MSN2/4, in which the motif is masked and the trans-factor input is reduced, and shows that the predicted expression changes correlate with a real MSN2/4 knockout microarray experiment at Spearman 0.486. The authors interpret these results as evidence that joint cis-trans modeling from raw inputs can predict expression and perturbation response, laying a foundation for similar models in more complex organisms.","pith_inferences":["The random 80-10-10 split on gene-condition pairs allows every test gene and test condition to appear in training, so the headline numbers measure interpolation between memorized per-gene and per-condition means; a held-out-condition or held-out-gene split would be a stricter test of the 'any gene, any condition' claim.","Because a real knockout measures both direct and indirect effects while the in-silico knockout perturbs the motif and the trans-factor input directly, the reported Spearman 0.486 may understate accuracy on direct targets; comparing predictions to in-vivo TF binding maps, as the authors say they plan to do, would refine this.","The same model could be used for in-silico variant effect scoring: single-nucleotide promoter mutations could be evaluated for their predicted expression effect in every stress condition, an application the paper does not report but the architecture permits."],"forward_implications":["The reported results imply that predefined transcription-factor motif compendia are not required to build competitive regulatory models; raw DNA sequence plus regulator expression can substitute for engineered PWM features.","In-silico knockout becomes a viable screening tool: before running a wet-lab transcription-factor deletion, the model can rank which genes will change and in which direction.","The architecture is transferable to other organisms and other molecular readouts, provided the training data cover enough genes and conditions to learn cis and trans effects.","The gradient-based regulator importance scores provide a data-driven, genome-wide shortlist of candidate trans regulators for follow-up experiments."],"supporting_citations":[{"why":"Defines the GeneClass boosted alternating decision tree baseline on the same classification task that the paper compares against.","marker":"[5]"},{"why":"Defines the BDTree bi-dimensional regression tree baseline trained on known cis-regulatory motif features.","marker":"[6]"},{"why":"Supplies the convolutional-filter-to-PWM interpretation method the paper uses to show its filters recover known motifs.","marker":"[7]"},{"why":"Supplies the yeast stress-response microarray dataset covering 6,100 genes and 173 stress conditions that trains and evaluates the model.","marker":"[8]"},{"why":"Supplies reverse-complement parameter sharing for the convolutional sequence module, letting filters recognize motifs on both DNA strands.","marker":"[9]"},{"why":"Documents MSN2/4 activation under heat shock and informs the design and interpretation of the in-silico knockout experiment.","marker":"[10]"}],"fun_headline_variants":["Multi-modal net beats motif models on yeast stress response","AI predicts yeast stress from raw promoter DNA plus regulator expression","In-silico regulator knockout in neural net mirrors real yeast experiment","Neural net learns cis and trans yeast stress regulation from raw data","Deep learning decodes yeast stress response using sequence and expression"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a random 80-10-10 split of gene-condition pairs measures how well the model predicts genes and stress states it has not seen, even though every test gene and test condition already appeared in training; if that is not true, the reported accuracy and correlation do not support the paper's claim of predicting any gene in any condition.","fun_headline_variants_meta":{"raw":{"variants":["Multi-modal net beats motif models on yeast stress response","AI predicts yeast stress from raw promoter DNA plus regulator expression","In-silico regulator knockout in neural net mirrors real yeast experiment","Neural net learns cis and trans yeast stress regulation from raw data","Deep learning decodes yeast stress response using sequence and expression"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000361,"raw_usage":{"total_tokens":1887,"prompt_tokens":820,"completion_tokens":1067,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":436,"completion_tokens_details":{"reasoning_tokens":983}},"tokens_in":436,"tokens_out":1067,"duration_ms":10457,"temperature":1.0,"reasoning_tokens":983,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:11:08.639736+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the identical architecture on the identical data but split by condition instead of by pair: hold out all pairs belonging to, say, 17 of the 173 stress conditions and test only on those held-out conditions. If accuracy on held-out conditions remains near 79.5% and regression correlation near 0.845, the generalization claim holds; if accuracy falls toward the 62.9% motif baseline, the model is largely memorizing per-condition expression means. The mirror-image test holds out entire genes rather than conditions.","supporting_citations":[{"cited_title":"Predicting genetic regulatory response using classification","cited_arxiv_id":null,"evidence_quote":"Defines the GeneClass boosted alternating decision tree baseline on the same classification task that the paper compares against."},{"cited_title":"A bi-dimensional regression tree approach to the modeling of gene expression regulation","cited_arxiv_id":null,"evidence_quote":"Defines the BDTree bi-dimensional regression tree baseline trained on known cis-regulatory motif features."},{"cited_title":"Basset: learning the regulatory code of the accessible genome with deep convolutional neural networks","cited_arxiv_id":null,"evidence_quote":"Supplies the convolutional-filter-to-PWM interpretation method the paper uses to show its filters recover known motifs."},{"cited_title":"Genomic expression programs in the response of yeast cells to environmental changes","cited_arxiv_id":null,"evidence_quote":"Supplies the yeast stress-response microarray dataset covering 6,100 genes and 173 stress conditions that trains and evaluates the model."},{"cited_title":"Reverse-complement parameter sharing improves deep learning models for genomics","cited_arxiv_id":null,"evidence_quote":"Supplies reverse-complement parameter sharing for the convolutional sequence module, letting filters recognize motifs on both DNA strands."},{"cited_title":"Fine-tuning of the Msn2/4- mediated yeast stress responses as revealed by systematic deletion of Msn2/4 partners","cited_arxiv_id":null,"evidence_quote":"Documents MSN2/4 activation under heat shock and informs the design and interpretation of the in-silico knockout experiment."}],"review_version":1}