{"id":"21753a1b-6fd3-4720-9298-e1e22ed325a9","arxiv_id":"2506.10271","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new benchmark shows that genomic language models mostly fail to detect loss-of-function mutations in synthetic, evolutionarily implausible DNA, with accuracy tied to how likely the model finds the sequence.","lead":"The authors introduce Nullsettes, a benchmark that tests whether genomic language models can predict loss-of-function mutations in synthetic DNA sequences that look nothing like natural genomes. Most of the 12 models tested fail these tests, and performance drops sharply as the test sequence becomes less familiar to the model. This suggests the models rely on pattern matching rather than a real understanding of how genes work.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The LL-performance correlation may be a floor artifact of the scoring metric rather than evidence of evolutionary-prior reliance.","rationale":"The paper is a serious benchmark with clear value: it assembles functional synthetic cassettes from MPRA and tests 28 model variants, and the raw observation that most models fail to detect LOF mutations is informative. I am not objecting to the benchmark's existence or to the conditional verdict. But the strongest claim—that gLMs rely on pattern-matching rather than mechanistic understanding—depends on the LL-success correlation, and that correlation is vulnerable to a statistical artifact that the paper does not address. The paired permutation test in Eq. 5 asks whether mutant LL is lower than original; when original LL is very low, the possible decrease is bounded, so detection success mechanically falls. This is not a niche worry: Figure 2A shows NT-2.5B-MS failing almost completely below LL=-27, which is precisely the regime where such a floor would bite. The paper's own use of 'length-scaled threshold' shows LL is not a clean functional score; the threshold may be capturing where the model's likelihood has dynamic range, not where its 'understanding' starts. The reader's concern about in-silico LOF labels is real and worth checking, but I see the floor-effect confound as more load-bearing because it directly undermines the interpretive leap from the headline correlation to the mechanism claim. A matched-control experiment can resolve this cleanly. If the drop disappears for control mutants, the paper's central claim should be softened to 'gLMs have limited dynamic range for scoring out-of-distribution sequences'; if it is specific to LOF mutants, the claim is supported. Thus a conditional accept, with the control experiment as a required revision, remains the appropriate verdict.","tokens_in":1163,"tokens_out":838,"duration_ms":96086,"concrete_test":"Recompute the Figure 2B/2C analysis with matched control mutants: for each nonmutant cassette, generate an order-preserving control by shuffling codons within the CDS (or synonymously mutating the CDS) so the canonical promoter-RBS-start-CDS-stop-terminator order is intact, then apply the identical paired permutation test. Plot control success rate against nonmutant LL for Evo2-7B and NT-2.5B-MS. If the success-rate drop replicates for these functional controls, the observed pattern is a generic low-LL artifact, and the inference to 'pattern-matching rather than mechanistic understanding' is not supported; if the drop is specific to Nullsettes mutants, the concern is settled and the original interpretation survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.4 scores each cassette with mean log-likelihood and declares a mutation type 'identified' when a paired permutation test finds mutant LL significantly lower than nonmutant LL. Figure 2 then shows success rate declining with nonmutant LL, and the paper interprets this as evidence that gLMs 'rely heavily on pattern-matching to their evolutionary prior.' The load-bearing assumption is that this decline is caused by model reasoning about sequence plausibility, not by the scoring procedure. If nonmutant LL is already near the bottom of a model's output range, mutant LL cannot move much lower, so paired differences are small and the permutation test loses power regardless of whether the model encodes functional constraints. The paper does not control for this: no matched non-LOF control mutants, no regression with sequence length/GC composition, and no null model. A model with perfect mechanistic understanding but a saturating likelihood function would show exactly the same pattern. Thus the central claim is currently underdetermined by the reported analysis.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Nullsettes, a benchmark of in silico loss-of-function (LOF) mutations in synthetic expression cassettes with little evolutionary precedent, and evaluates 12 genomic language models (gLMs) on zero-shot detection of these mutations. The central finding is that a model's ability to identify LOF mutants declines sharply as the log-likelihood (LL) of the nonmutant sequence decreases, which the authors interpret as evidence that gLMs rely on evolutionary pattern-matching rather than mechanistic understanding of gene expression. The benchmark curates cassettes from five MPRA datasets and applies systematic element translocations defined by a grammar of three ordering rules.","tokens_in":12443,"tokens_out":8105,"duration_ms":100828,"significance":"If the interpretation holds, the paper would be an important cautionary result for using gLMs in synthetic biology, showing that current models do not generalize to engineered sequences outside natural sequence space. The benchmark itself is novel, spans a wide range of model families and scales, and is accompanied by public code and data. The observation that a much smaller model (GENERanno) can match Evo2-7B is also informative about pretraining-data relevance. However, the central claim currently rests on a correlation that has not been separated from a scoring floor/power artifact, and the MLM scoring uses unmasked logits that are not a valid likelihood. The significance is therefore conditional on additional controls and on reanalysis with proper pseudo-likelihood scores.","major_comments":[{"comment":"The central claim that low nonmutant LL causes poor LOF detection is confounded by a floor/power effect. The paired permutation test compares mutant LL against nonmutant LL; for cassettes whose nonmutant LL is already near the model's output floor, the mutant LL cannot move much lower, so the paired differences are small and the test loses power regardless of whether the model encodes functional constraints. A model with a saturating likelihood function and no functional understanding would produce exactly the same pattern. The manuscript provides no null model (e.g., comparing each mutant against a randomly chosen nonmutant), no matched non-LOF control mutations, and no regression controlling for sequence length, GC composition, or other confounds. Without these controls, the decline in success rate with nonmutant LL does not uniquely support the 'evolutionary prior reliance' interpretation. Please add these analyses or explicitly discuss the floor artifact as an alternative explanation.","section":"§4.4 (Eq. 5) and Figure 2A/B"},{"comment":"For masked language models (MLMs), the sequence-level score is computed from unmasked logits, producing P(xt|X) that is not a valid conditional probability because the model sees the token itself. The authors acknowledge that this 'lacks strict probabilistic interpretation,' but 7 of the 12 model families (NT, GENERanno, Caduceus, GPN, GPN-Promoter, DNABERT2, gLM2) are scored this way. The paper's statement that 'all models show a sharp drop in prediction accuracy as the likelihood assigned to the original sequence decreases' is therefore not supported for those models in terms of actual model likelihood. Please recompute scores using a proper pseudo-likelihood (e.g., single-inference PLL or full PLL) or restrict the claim to CLM models and show MLM results separately.","section":"§4.4 (Eqs. 3–4)"},{"comment":"The ground-truth LOF labels are defined entirely in silico by three ordering rules applied to circular permutations. The claim that the resulting mutants are 'completely non-functional' is stronger than the evidence supports, because the rules do not exclude rescue by alternative start codons, cryptic promoters, internal ribosome entry, or translational reinitiation. The authors do note for some mutants that no other in-frame start codon exists, but no experimental validation is provided for any mutant. If some labeled LOF mutants are actually functional, the model evaluation would conflate syntactic violations with true loss of function. Please either provide empirical validation of a subset of mutants, or soften the wording to 'predicted non-functional by the canonical grammar' and discuss the potential impact of label noise on the conclusions.","section":"§4.2"},{"comment":"The cassette curation includes a step that compares LL distributions between the Kosuri and Lagator datasets and prioritizes Kosuri promoter–RBS pairs whose LL distributions surpass those from Lagator. This selects on the independent variable used in the main correlation (nonmutant LL). The authors should test whether the LL-performance correlation persists when this selection step is removed, to verify that it is not an artifact of the curation procedure.","section":"§4.1.3"}],"minor_comments":[{"comment":"The Results state that significance is determined by a paired t-test, while Methods use a one-sided paired permutation test (Eq. 5). Please reconcile this inconsistency and state which test was used for the reported success rates.","section":"§2 and §4.4"},{"comment":"The phrase 'casual language models' should be 'causal language models' in both the main text and the Supplementary Table S4 header.","section":"§4.4 and Table S4"},{"comment":"The caption says 'Each point represents an expression cassette' while the y-axis is described as 'Nullsettes prediction performance.' It is unclear how a binary success/failure is assigned to a single cassette when the permutation test in Methods operates over all cassettes for a mutation type. Please clarify the plotting and the per-cassette success definition.","section":"Figure 2A"},{"comment":"The citation of Gordon et al. [25] for single-inference PLL is misleading because the manuscript does not use that method; it uses unmasked logits. Either implement the cited method or remove the citation.","section":"§4.4"},{"comment":"The claim that the LL threshold 'scales linearly with sequence length' is based on only five datasets and a linear regression with no reported uncertainty or goodness-of-fit. Please report R², confidence intervals, or use a larger set of sequence lengths before drawing this conclusion.","section":"Figure 2C"},{"comment":"The caption says 'four datasets' but lists five (Abf1TATA, pTpA, Zahm, Kosuri, Lagator). Please correct the count.","section":"Supplementary Figure S2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the scope of the journal and the benchmark is potentially useful. The main concern is that the central claim—that gLMs rely on evolutionary prior rather than functional understanding—is underdetermined by the reported analysis because the LL-performance correlation could be a floor artifact, and because the MLM scoring is not a valid likelihood. These are fixable with additional analyses, so I recommend major revision rather than rejection. I did not see any issues with citation or novelty disclosure beyond the minor inconsistencies noted for the authors."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi [Name],\n\nWorth a look. The Nullsettes benchmark is a genuinely useful addition: it takes real MPRA expression cassettes (E. coli, yeast, human), creates in silico LOF mutants by translocating promoter/RBS/start/stop/terminator, and asks 12 gLMs to distinguish mutant from nonmutant via log-likelihood. The data and code are public, the benchmark is clearly specified, and the finding that most models fail badly on these synthetic cassettes is solid. The result that GENERanno, a small model trained on expressed regions, matches Evo2-7B is a nice data point for the \"data quality matters\" debate.\n\nThe main claim—that the sharp drop in accuracy with low nonmutant LL shows gLMs \"rely heavily on pattern-matching to their evolutionary prior\"—goes beyond what the analysis supports. The stress-test concern is real: if a cassette already sits at the bottom of the model's scoring range, a mutant's score cannot move much lower, so paired permutation tests lose power regardless of whether the model understands anything. The paper does not include non-LOF control mutants, does not adjust for sequence length or composition in that correlation, and the MLM scoring uses unmasked logits (the authors admit this lacks strict probabilistic interpretation). The correlation could be partly a floor artifact. That does not kill the paper—the benchmark results stand—but it does mean the \"mechanistic understanding\" language is overreach.\n\nAlso worth noting: the LOF labels are defined in silico by ordering rules, not validated experimentally. Plausible, but if those rules are wrong (e.g., cryptic rescue), the benchmark conflates syntax violations with loss of function. The authors should show robustness of their rules or validate on a subset.\n\nOverall: a solid benchmark with reproducible code, likely to be useful to anyone developing gLMs for synthetic biology. But the central interpretation needs hardening. A serious referee should ask for controls (matched scrambled mutants, a null model, and a validated MLM scoring scheme) before the paper's main claim is accepted.\n\nRecommendation: send to peer review; it deserves referee time, but expect heavy revision.\n\nBest.","headline":"A useful new gLM generalization benchmark, but the headline interpretation—LL-dependence as proof of evolutionary-prior reliance—is undercut by a likely floor effect in the scoring metric.","tokens_in":12950,"tokens_out":2093,"would_cite":true,"duration_ms":22984,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Genomic language models fail to predict loss-of-function in synthetic DNA; their accuracy tracks sequence likelihood, not biological mechanism.","keywords":["genomic language models","loss-of-function prediction","synthetic expression cassettes","evolutionary prior","mutational effect prediction","massively parallel reporter assay","Nullsettes benchmark"],"falsifier":"Run a reporter assay on a random sample of Nullsette translocation mutants in the appropriate host. If many mutants labeled non-functional still produce protein, the ground-truth labels are wrong; if models with high nonmutant likelihood nevertheless miss experimentally confirmed loss-of-function mutants, the claim that predictions rely on the evolutionary prior would need to be revised.","tokens_in":12077,"feed_emoji":"🧬","tokens_out":5795,"duration_ms":66373,"temperature":0.7,"pith_summary":"The paper introduces Nullsettes, a benchmark of synthetic expression cassettes whose control elements have been rearranged into known loss-of-function configurations. Testing 12 genomic language models, it finds that most cannot consistently flag these mutations. The central evidence is a systematic relationship: a model's success at detecting a mutation drops steadily as the log-likelihood it assigns to the original cassette decreases, regardless of model family. The authors read this as pattern-matching to evolutionary prior rather than mechanistic understanding. If correct, current gLMs are unreliable tools for designing functional sequences outside natural sequence space.","feed_headline":"Genomic AI models fail on synthetic DNA design","feed_subtitle":"A new benchmark shows accuracy tracks how natural a sequence looks, not whether the mutation breaks gene expression.","key_machinery":"The load-bearing mechanism is the virtual translocation: each Nullsette is created by moving one regulatory element (promoter, start codon, CDS, stop codon, terminator, and in prokaryotes RBS) to another position, preserving all components while breaking the canonical 5'-3' order. Non-functionality is defined in silico by three ordering rules checked over all circular permutations of the mutant; a mutant is retained only if no permutation satisfies all rules. The evaluation then compares the log-likelihood distribution of nonmutant cassettes with that of the mutants, and the paper's central analytical device is the relationship between the nonmutant log-likelihood and the per-cassette success rate, summarized by a length-scaled threshold.","core_discovery":"The paper's discovery is that genomic language models trained on natural genomes do not understand the regulatory grammar of gene expression well enough to generalize to synthetic cassettes. Nullsettes constructs loss-of-function mutants by translocating one regulatory element in each cassette and defines non-functionality through three ordering rules applied over all circular permutations. Evaluating 12 models, the authors find most identify fewer than half of the mutation types on at least one dataset, and Evo2-7B and GENERanno-0.5B are the only consistent strong performers. The unifying failure is that a model's success at detecting a mutation falls as the log-likelihood it assigns to the original sequence falls, a length-scaled threshold separates reliable from unreliable prediction, and this holds across architectures. The authors conclude that predictions track evolutionary plausibility rather than the biological effect of the mutation.","pith_inferences":["The same likelihood-dependence pattern could be tested directly with experimental measurements: if reporter assays confirm that a specific Nullsette mutant is non-functional while a model predicts it as functional only because the cassette has low likelihood, the paper's interpretation is strengthened.","A natural next step is to fine-tune a gLM on Nullsettes-style rearrangements and ask whether the improvement transfers to natural regulatory variants; transfer would show the missing knowledge is learnable rather than inherently absent.","The ordering-rule definition treats expression as a purely linear grammar, so models that incorporate RNA secondary structure or co-transcriptional folding might show a different failure profile on the same benchmark."],"forward_implications":["Designers should not treat gLM likelihood scores as reliable standalone predictors of function for engineered DNA.","The linear relationship between cassette length and the likelihood threshold for accurate prediction gives a concrete, testable operating rule for when a model's mutation predictions can be trusted.","Pretraining corpus content may matter more than parameter count, since GENERanno-0.5B matched Evo2-7B with far fewer parameters.","Benchmarks centered on natural variants miss a core failure mode; functional-generalization benchmarks need synthetic, out-of-distribution sequences."],"supporting_citations":[{"why":"Supplies the random promoter library in E. coli used to build out-of-distribution nonmutant cassettes.","marker":"[14]"},{"why":"Provides 100 million randomized yeast promoters, the source of the eukaryotic deBoer cassettes.","marker":"[15]"},{"why":"Provides the combinatorial promoter–RBS expression cassettes for the prokaryotic Kosuri dataset.","marker":"[16]"},{"why":"Supplies synthetic mammalian promoter–TRE constructs for the human expression cassettes.","marker":"[17]"},{"why":"Describes GENERanno's curated pretraining, the basis for the data-quality explanation of its strong performance.","marker":"[18]"},{"why":"Introduces Evo2, the strongest large model in the benchmark, whose failure pattern illustrates the likelihood-dependence claim.","marker":"[31]"}],"fun_headline_variants":["Genomic AI fails on synthetic DNA","AI models miss function in synthetic DNA","Synthetic DNA stumps genomic language models","Genomic models can't grasp synthetic gene logic"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark labels a mutant non-functional purely from ordering rules, assuming that no alternative start codon, cryptic promoter, or translational reinitiation can restore expression in the rearranged cassettes.","fun_headline_variants_meta":{"raw":{"variants":["Genomic AI fails on synthetic DNA","AI models miss function in synthetic DNA","Synthetic DNA stumps genomic language models","Genomic models can't grasp synthetic gene logic"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000145,"raw_usage":{"total_tokens":1141,"prompt_tokens":872,"completion_tokens":269,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":488,"completion_tokens_details":{"reasoning_tokens":227}},"tokens_in":488,"tokens_out":269,"duration_ms":3503,"temperature":1.0,"reasoning_tokens":227,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:30:10.987966+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a reporter assay on a random sample of Nullsette translocation mutants in the appropriate host. If many mutants labeled non-functional still produce protein, the ground-truth labels are wrong; if models with high nonmutant likelihood nevertheless miss experimentally confirmed loss-of-function mutants, the claim that predictions rely on the evolutionary prior would need to be revised.","supporting_citations":[{"cited_title":"Predicting bacterial promoter function and evolution from random sequences","cited_arxiv_id":null,"evidence_quote":"Supplies the random promoter library in E. coli used to build out-of-distribution nonmutant cassettes."},{"cited_title":"Deciphering eukaryotic gene-regulatory logic with 100 million ran- dom promoters","cited_arxiv_id":null,"evidence_quote":"Provides 100 million randomized yeast promoters, the source of the eukaryotic deBoer cassettes."},{"cited_title":"Composability of regulatory sequences controlling transcription and translation in escherichia coli","cited_arxiv_id":null,"evidence_quote":"Provides the combinatorial promoter–RBS expression cassettes for the prokaryotic Kosuri dataset."},{"cited_title":"A massively parallel reporter assay library to screen short synthetic promoters in mammalian cells","cited_arxiv_id":null,"evidence_quote":"Supplies synthetic mammalian promoter–TRE constructs for the human expression cassettes."},{"cited_title":"Generanno: A genomic foundation model for metagenomic annotation","cited_arxiv_id":null,"evidence_quote":"Describes GENERanno's curated pretraining, the basis for the data-quality explanation of its strong performance."},{"cited_title":"Genome modeling and design across all domains of life with evo 2","cited_arxiv_id":null,"evidence_quote":"Introduces Evo2, the strongest large model in the benchmark, whose failure pattern illustrates the likelihood-dependence claim."}],"review_version":1}