{"id":"d15ec471-f8bd-4971-bc69-b0de4479f274","arxiv_id":"2411.12010","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"NAIAD uses single-gene effects plus adaptive embeddings and maximum-predicted-effect sampling to discover the strongest gene pairs in combinatorial CRISPR screens with fewer experimental rounds.","lead":"NAIAD is an active learning framework that predicts which gene pairs in CRISPR screens will have the strongest effects, so researchers can prioritize a few hundred experiments instead of testing millions. It combines measured single-gene effects with adaptive gene embeddings and a maximum-predicted-effect selection rule, and the authors report faster discovery of top gene pairs on four published CRISPR datasets.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MPE discovery gain may be driven by single-gene additive ranking; paper lacks a linear-surrogate MPE baseline, so NAIAD's interaction model may not be necessary for the central discovery claim.","rationale":"I read the paper in good faith. NAIAD is a reasonable framework, and the reported experiments are extensive. The reader's conditional verdict is fair. However, the single most load-bearing gap is not noise in Yi (which the paper itself acknowledges as a limitation) but the missing control that isolates NAIAD's contribution to the discovery result. The MPE acquisition function is model-agnostic; the claimed advantage over uniform and UCB is only meaningful as a property of NAIAD if a simpler surrogate cannot achieve it. Because NAIAD's additive component is explicitly correlated with linear predictions, and because the top-200 pairs in these screens are likely enriched for pairs involving high-effect single genes, the 2x/1.5x discovery numbers may be an artifact of using single-gene effects rather than of the learned embeddings or interaction term. This concern is concrete and falsifiable: one simulation run with a linear-MPE surrogate would settle it. It reinforces the reader's CONDITIONAL verdict rather than changing it, so no verdict adjustment is proposed.","tokens_in":12764,"tokens_out":16297,"duration_ms":168569,"concrete_test":"Run the Section 5.3 active-learning simulation on Norman, Simpson, Horlbeck-K562, and Horlbeck-Jurkat, replacing the NAIAD surrogate with a linear model trained on Yi (or simply ranking unseen pairs by Yi+Yj) under the identical MPE acquisition, cumulative retraining, and top-200 fraction-discovered metric. Compare round-4 fractions and marginal gains (Tables 2 and 3) to NAIAD-MPE. If the linear surrogate achieves at least 90% of NAIAD's discovered count in any dataset, the central discovery claim is not attributable to NAIAD's interaction model, and the paper should be revised to position additive single-gene ranking as the main discovery mechanism.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central discovery claim (Section 5.3, Table 2) is that NAIAD's MPE acquisition finds over twice as many top-200 pairs as uniform sampling and nearly 1.5 times as many as UCB. But MPE ranks pairs by NAIAD's predicted effect, and NAIAD's model is dominated by an overparameterized additive single-gene term that the authors show correlates strongly with the linear model (Fig 5). The paper never benchmarks MPE using a linear surrogate that simply ranks candidate pairs by Yi+Yj or by a linear regression on Yi. Without this control, the MPE advantage could be entirely explained by selecting pairs containing high-effect single genes, not by NAIAD's adaptive embeddings or learned interactions. This is load-bearing because the paper's title and framing attribute the discovery efficiency to NAIAD's architecture; if a linear surrogate recovers the same top-200 coverage, the claimed 2x/1.5x improvements are not evidence for NAIAD specifically. The Discussion's admission that the model assumes accurate single-gene effects makes this alternative explanation particularly plausible.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces NAIAD, an active learning framework for predicting the phenotypic effects of two-gene perturbations in combinatorial CRISPR screens. The model architecture combines an overparameterized single-gene effect term (a high-dimensional projection of the measured single-gene effects Yi and Yj) with learnable gene embeddings whose dimension grows according to a schedule based on training-set size. For active learning, the paper proposes maximum predicted effect (MPE) sampling, which selects the gene pairs with the largest predicted effects for the next experimental round. The authors evaluate NAIAD on four bulk combinatorial perturbation datasets (Norman, Simpson, Horlbeck K562, Horlbeck Jurkat) and compare against linear regression, MLP, GEARS, and RECOVER, reporting that NAIAD outperforms the baselines in small-sample settings, with a claimed 'up to 40%' relative improvement over the second-best model. They further report that MPE sampling uncovers more than twice as many of the top-200 strongest pairs as uniform sampling and nearly 1.5 times as many as UCB by round four of the simulated active learning loop.","tokens_in":12993,"tokens_out":8403,"duration_ms":77898,"significance":"If the empirical claims are correct, NAIAD would be a practically useful tool for designing combinatorial CRISPR libraries in the small-data regime, where active learning is most needed. The adaptive embedding dimension is a sensible and interesting mechanism for trading off model capacity against data availability, and the paper evaluates on unusually large public datasets with held-out test sets, which is a strength. The active learning simulation protocol is also a reasonable proxy for real iterative experimentation. However, the headline quantitative claims are not fully supported by the reported results, and the benchmark and control choices weaken the causal link between the architecture and the reported discovery gains. Specifically, the 40% improvement figure is not reproducible from Table 1, the GEARS baseline is adapted in a way that likely handicaps it, and the MPE acquisition advantage is not compared against a simple linear surrogate, leaving open the possibility that additive single-gene effects, rather than the learned interaction embeddings, drive the discovery performance.","major_comments":[{"comment":"The Abstract and Section 5.2 state that NAIAD outperforms the second-best model by 'over 40%' on average across the four datasets based on Table 1. This is not supported by the reported RMSE values. At gene frequency 4, the relative improvement over the second-best model (the linear model in all four cases) is approximately 18% for Norman, 33% for Simpson, 5% for Horlbeck K562, and 23% for Horlbeck Jurkat, an average of about 20%. Please correct the claim or clarify the exact calculation, since the current phrasing is materially misleading.","section":"Abstract; Section 5.2, Table 1"},{"comment":"The description of the GEARS baseline states that for bulk datasets the authors 'generated synthetic Perturb-seq datasets of normalized gene expression matrices, with a separate Gaussian N(0,1) used for sampling the expression of each gene.' GEARS is designed to take single-cell transcriptomic profiles as input; feeding it random Gaussian features is not a meaningful or fair benchmark and most likely explains GEARS's poor performance. To support the claim that NAIAD outperforms GEARS, the comparison must use a reasonable input representation (for example, pseudobulk expression derived from the same screen or a compatible single-cell dataset), or the GEARS baseline should be removed.","section":"Appendix B (GEARS benchmark)"},{"comment":"The MPE acquisition function is evaluated only with the NAIAD surrogate. The paper does not include a control where MPE ranks candidate pairs by a linear model (e.g., by Yi + Yj or by a linear regression fit on single-gene effects). Given that Appendix Figure 5 shows NAIAD's single-gene component strongly correlates with linear model predictions, the reported MPE advantage over uniform and UCB sampling could be fully explained by additive single-gene ranking rather than by NAIAD's learned interactions or adaptive embeddings. Without such a control, the claim that NAIAD's architecture is responsible for the discovery efficiency is not established.","section":"Section 5.3, Table 2, Figure 4"},{"comment":"The adaptive embedding schedule is described as a general principle that 'hold[s] across all datasets,' but Table 5 gives different embedding dimensions for the same gene occurrence frequency in different datasets. For example, at gene frequency 4, Norman and Simpson use embedding dimension 4 while Horlbeck K562 uses 16; at frequency 10, Norman and Simpson use 16 while Horlbeck uses 32. This indicates that the schedule is dataset-specific rather than a single shared rule. The authors should either present one schedule valid across datasets or explicitly state that the per-dataset values are tuned hyperparameters, and explain how they were chosen without using test information.","section":"Section 3.2, Appendix F.1, Table 5"},{"comment":"The model uses measured single-gene effects Yi as inputs, and the Discussion acknowledges this assumption. However, the paper provides no sensitivity analysis for the impact of noise, bias, or missing values in Yi on either predictive accuracy or the MPE-based discovery results. Since the additive component and the MPE ranking depend directly on these values, a robustness study (e.g., adding Gaussian noise to Yi, or removing a subset of single-gene measurements) is needed to support the practical applicability claimed in the paper.","section":"Section 3.1, Section 6 (Discussion)"}],"minor_comments":[{"comment":"Table 2 and Table 3 report point values without standard errors, while Figure 4 for the same experiment reports SE across three cross-fold replicates. Please clarify whether the tables report a representative fold or an average, and consider adding uncertainty estimates to the tables.","section":"Table 2, Table 3, Figure 4"},{"comment":"The definition of 'Fraction Discovered' concatenates model predictions on unseen data with measured values from seen data and then takes the top P; this is a nonstandard mixture. The main text should explain this more clearly, as it affects the interpretation of the 'top 200 discovered' numbers in Table 2.","section":"Appendix D"},{"comment":"The figures report log(MSE) without specifying the logarithm base (natural or base 10). Please state this in the captions or methods.","section":"Figures 2, 3, 5, 7"},{"comment":"The dimensions of the matrices W1, W2, A1, and A2 are not explicitly defined. Please specify all tensor dimensions so that the architecture is unambiguous.","section":"Section 3.1, model equation"},{"comment":"The caption refers to '4429 training combinations' for the Norman dataset, while the dataset description in Section 4.2 and Appendix A states 6,328 unique combinations. Please clarify the number and the split used.","section":"Figure 2 caption"},{"comment":"The text in Section 5.3 says 'MPE sampling ranked the top 200 gene pairs' but Table 2 reports 'Number of top 200 gene combinations correctly discovered at round 4.' Please ensure the terminology is consistent and that the reader understands whether the table counts pairs that were measured, predicted, or either.","section":"Section 5.3 and Table 2"}],"recommendation":"major_revision","confidential_remarks":"The paper appears to be a NeurIPS workshop submission that has been posted on arXiv. The scale of the empirical study is commendable, but the abstract's headline quantitative claim is not reproducible from the tables, and the GEARS baseline adaptation is likely unfair. If the authors can fix these issues and add the linear-MPE control, the paper could become a solid contribution; as it stands, the evidence for the central claims is incomplete. I would also verify that the GitHub repository is publicly available before publication, as the paper relies on it for reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Jason, quick take on arXiv:2411.12010. The paper is worth your time: it applies active learning to bulk CRISPR pair screens, conditions on measured single-gene effects, and uses an adaptive embedding schedule. The authors run it on four public datasets totaling 350k+ interactions, release code, and evaluate both predictive error and a simulated discovery loop. That is a solid empirical package. The downsampling benchmark is the strongest part: NAIAD is consistently the best or tied across training sizes, and the improvement over linear/RECOVER/MLP at low gene frequencies is real.\n\nNow the soft spots, roughly in order of importance.\n\nThe headline 'up to 40% better than second-best' doesn't hold against Table 1. Averaging the relative RMSE improvement at gene frequency 4 across the four datasets gives roughly 20-30%, and some datasets are in single digits. The abstract and Section 5.2 need a correction.\n\nThe bigger issue is the missing control for the MPE discovery claim. Table 2 shows MPE finds more than twice as many top-200 pairs as uniform and ~1.5x UCB by round four. But MPE ranks pairs by NAIAD's predicted effect, and NAIAD's additive single-gene component correlates strongly with the linear model (Fig 5). Without benchmarking MPE on a simple linear surrogate (e.g., rank by Yi+Yj), you can't tell whether the discovery gain is coming from NAIAD's architecture or just from picking pairs that contain high-effect single genes. That is central to the paper's framing, and the absence of this control is a genuine gap, not a nitpick.\n\nGEARS is also put in a bad position by feeding it synthetic single-cell expression data and keeping default hyperparameters. That makes the GEARS comparison weak, though GEARS isn't the main rival here. The adaptive embedding schedule is dataset-specific in Table 5, which raises a mild tuning concern, but the method's logic is reasonable.\n\nThe paper does acknowledge its main limitation: it assumes accurate single-gene effects as inputs. That's honest, but the authors give no sensitivity analysis. A short experiment with noise added to Yi would tell you how robust the framework is.\n\nMy bottom line: For anyone working on active learning for perturbation screens, this is a useful read and a decent starting benchmark. The predictive benchmarking is credible and the framework is well motivated. The discovery claim is plausible but not yet cleanly attributed, and the 40% number is wrong as written. A serious referee should send this to revision rather than reject; if the authors add a linear MPE baseline and fix the overclaim, the paper would be solid.","headline":"NAIAD is a useful active learning framework for combinatorial CRISPR screens, but the 40% claim is overstated and the MPE discovery result lacks a linear-surrogate control.","tokens_in":13512,"tokens_out":4212,"would_cite":true,"duration_ms":40604,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"NAIAD claims the combinatorial perturbation space becomes tractable when a model is conditioned on measured single-gene effects and uses adaptive gene embeddings, and that a maximum-predicted-effect acquisition strategy finds most of the…","keywords":["active learning","CRISPR combinatorial screens","gene-gene interactions","gene embeddings","maximum predicted effect sampling","small-sample learning","cell viability","acquisition function"],"falsifier":"Run a prospective four-round CRISPR screen where NAIAD's MPE sampling and UCB sampling each choose 100 new pairs per round, then count how many of the top-200 ground-truth pairs each strategy finds; if MPE does not recover more than UCB, the paper's central active-learning claim fails.","tokens_in":12570,"feed_emoji":"🧬","tokens_out":6114,"duration_ms":52266,"temperature":0.7,"pith_summary":"NAIAD is an active learning framework for the combinatorial perturbation space—the set of all two-gene knockouts or activations that could push a cell into a desired state, which is astronomically large (~200 million pairs for the human genome). The paper's central claim is that a model which takes measured single-gene effects as inputs and uses gene embeddings whose dimensionality grows with the training set can predict unseen gene-pair phenotypes accurately from very small samples, beating both simple linear models and deep baselines. It further claims that selecting the next experiments by maximum predicted effect (MPE), rather than by uncertainty or a mixed bound, accelerates discovery: after four rounds the MPE strategy recovers over twice as many of the top-200 strongest pairs as uniform sampling and nearly 1.5 times as many as UCB. If correct, the framework lets labs design CRISPR libraries that converge on potent combinations in a handful of experiment–model iterations instead of exhaustive screens.","feed_headline":"NAIAD finds 150 of top 200 gene pairs with a few CRISPR rounds","feed_subtitle":"An active-learning model beats baselines by up to 40% and uses maximum-effect sampling to slash experiments.","key_machinery":"The model's core object is a decomposition of the pairwise phenotype into two terms: an over-parameterized single-gene effect term $\\phi([Y_i,Y_j]W_1)A_1^T$ that conditions on measured single-gene effects $Y_i, Y_j$, and an interaction term $f(\\phi(W_2X^i_{\\text{gene}}), \\phi(W_2X^j_{\\text{gene}}))A_2^T$ built from adaptive gene embeddings $X_{\\text{gene}}$ with a permutation-invariant combination (sum) and a nonlinear encoder $f$. The embedding dimension starts small and grows on a predetermined schedule tied to the average gene-occurrence frequency, which lets the model behave like a linear model when data are scarce and like a deep network when data accumulate. The acquisition side relies on an ensemble of NAIAD models whose prediction variance estimates uncertainty, and the recommended strategy is Maximum Predicted Effects (MPE) sampling, which selects the gene pairs with the largest absolute predicted effects for the next experimental round.","core_discovery":"On the paper's own terms, the central discovery is that the combinatorial perturbation prediction problem has a favorable small-sample structure when it is decomposed correctly. NAIAD writes the phenotype of a gene pair as the sum of an over-parameterized transformation of the two single-gene effects and a nonlinear interaction term computed from learnable gene embeddings, with the embedding dimension deliberately scheduled to grow with the average number of times each gene has been observed. Across four bulk CRISPR datasets totaling over 350,000 measured interactions, NAIAD trained on small samples achieves the lowest RMSE, highest correlation, and highest true-positive rate among five models, and its advantage is largest when each gene appears only about four times in the training set. The paper also shows that the MPE acquisition function, despite producing worse global MSE because it skews training data toward strong pairs, is the most efficient at finding the strongest gene pairs, which is the stated objective for therapeutic discovery.","pith_inferences":["Because NAIAD's additive term is conditioned on measured single-gene effects, its advantage is likely to shrink as single-gene measurements get noisier; a sensitivity analysis sweeping single-gene noise would test this directly.","The top-k discovery metric used to evaluate acquisition functions is the right objective for therapeutic library design, and its widespread adoption would make comparisons between active learning methods more meaningful than global MSE.","The embedding-growth schedule is a general trick for sparse combinatorial search and could transfer to drug combinations or multi-target screens, though the schedule constants would need re-tuning."],"forward_implications":["At roughly four observations per gene, NAIAD's test RMSE is over 40% better than the second-best model across the four datasets.","The MPE acquisition function identifies about 150 of the top-200 strongest gene pairs by round four in three of the four screens, versus 66–92 for uniform sampling.","Because genome-wide coverage at 20 observations per gene would require roughly 200,000 initial combinations, cutting the needed observations per gene is what makes genome-scale screens feasible.","Adaptive embeddings give a single model that remains competitive in both the small-sample and data-rich regimes, removing the usual need to switch from linear to deep models as data grow."],"supporting_citations":[{"why":"Supplies the 6,328-combination CRISPRa K562 screen used as the primary small benchmark and the motivation for combinatorial perturbation discovery.","marker":"Norman et al. 2019"},{"why":"Supplies the 147,658-combination CRISPRi K562 screen and the viability-normalization procedure used to compute phenotype values.","marker":"Simpson et al. 2023"},{"why":"Supplies the two large CRISPRi screens on K562 and Jurkat T cells used as additional benchmarks.","marker":"Horlbeck et al. 2018"},{"why":"Provides the RECOVER baseline and the UCB acquisition strategy that NAIAD's MPE sampling is compared against.","marker":"Bertin et al. 2023"},{"why":"Provides the GEARS baseline graph-neural-network model for combinatorial perturbation prediction.","marker":"Roohani, Huang, and Leskovec 2023"},{"why":"Supplies the deep-ensemble method used to estimate prediction uncertainty for acquisition functions.","marker":"Lakshminarayanan, Pritzel, and Blundell 2016"},{"why":"Supplies the empirical observation that deep models often fail to beat linear models in small perturbation datasets, motivating the small-sample design.","marker":"Ahlmann-Eltze, Huber, and Anders 2024"}],"fun_headline_variants":["NAIAD finds top gene pairs fast with active learning","NAIAD slashes CRISPR rounds, beats baselines by 40%","Active learning for gene pairs: fewer experiments, better hits","NAIAD: efficient discovery of synergistic gene combos"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"NAIAD assumes the single-gene perturbation effects for every gene in a pair are known and correct; if those measurements are noisy, biased, or missing, the model's additive foundation is corrupted and its advantage could vanish.","fun_headline_variants_meta":{"raw":{"variants":["NAIAD finds top gene pairs fast with active learning","NAIAD slashes CRISPR rounds, beats baselines by 40%","Active learning for gene pairs: fewer experiments, better hits","NAIAD: efficient discovery of synergistic gene combos"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000338,"raw_usage":{"total_tokens":1864,"prompt_tokens":939,"completion_tokens":925,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":555,"completion_tokens_details":{"reasoning_tokens":854}},"tokens_in":555,"tokens_out":925,"duration_ms":8968,"temperature":1.0,"reasoning_tokens":854,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:59:48.908111+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a prospective four-round CRISPR screen where NAIAD's MPE sampling and UCB sampling each choose 100 new pairs per round, then count how many of the top-200 ground-truth pairs each strategy finds; if MPE does not recover more than UCB, the paper's central active-learning claim fails.","supporting_citations":[{"cited_title":"Exploring Genetic Interaction Manifolds Constructed from Rich Single-Cell Phenotypes","cited_arxiv_id":null,"evidence_quote":"Supplies the 6,328-combination CRISPRa K562 screen used as the primary small benchmark and the motivation for combinatorial perturbation discovery."},{"cited_title":"Mapping the Genetic Interaction Network of PARP Inhibitor Response","cited_arxiv_id":null,"evidence_quote":"Supplies the 147,658-combination CRISPRi K562 screen and the viability-normalization procedure used to compute phenotype values."},{"cited_title":"Mapping the Genetic Landscape of Human Cells","cited_arxiv_id":null,"evidence_quote":"Supplies the two large CRISPRi screens on K562 and Jurkat T cells used as additional benchmarks."},{"cited_title":"Predicting Transcriptional Outcomes of Novel Multigene Perturbations with GEARS","cited_arxiv_id":null,"evidence_quote":"Provides the GEARS baseline graph-neural-network model for combinatorial perturbation prediction."}],"review_version":1}