{"id":"5732a063-f2d6-4ff0-be0a-6c590a08fae6","arxiv_id":"2505.23839","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"GeneBreaker, a new attack framework, steers DNA language models to generate sequences with over 90% identity to human pathogens, with success rates up to 60% on the largest Evo2 model.","lead":"Researchers tested whether DNA language models like Evo2 can be steered to generate pathogen-like DNA sequences, and found that with carefully chosen prompts and search guidance, the largest models produce sequences matching real viruses up to 60% of the time. This matters because it shows generative AI for genomics needs safety screening and monitoring, much like chatbots.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim depends on an unverified training-data exclusion; if JailbreakDNABench targets appear in Evo training data, the reported high-similarity outputs may be memorization rather than a general jailbreak.","rationale":"I read the paper as a first systematic red-teaming effort for DNA language models, and the benchmark, prompt-design agent, and beam-search pipeline are useful contributions regardless of the outcome of this check. The strongest empirical evidence, including the SARS-CoV-2 and HIV-1 case studies and the evolutionary modeling, rests on the same assumption that the model is not simply reproducing sequences seen during training. The paper itself asserts the need for the exclusion check but does not supply the evidence, and the stated evaluation protocol in Section 5.1 makes the memorization concern especially pointed because the model is conditioned on a prefix of the very sequence family used as the BLAST reference. This is not an internal mathematical inconsistency; it is an external-validity threat that is concretely checkable. The authors may well be able to provide the required contamination analysis and control experiments, so the appropriate verdict is conditional rather than rejection. I agree with the reader that this is the weakest assumption in the argument.","tokens_in":15256,"tokens_out":5777,"duration_ms":68943,"concrete_test":"Release the exact JailbreakDNABench sequence set and the exact prompts used in Section 5.1, then run a controlled experiment with four conditions: (a) full GeneBreaker, (b) target-first-half prefix alone with greedy decoding, (c) target-first-half prefix alone with the same beam search, and (d) a random non-homologous prefix with the same beam search. Also compute, for every benchmark target and every generated sequence, the maximum BLASTn/MMseqs2 identity against the public Evo/Evo2 training corpora (including OpenGenome) over a sliding 100 nt window. If any target or generated output has at least 90% identity to a training contig over at least 100 nt, or if condition (b) or (c) alone reaches comparable attack success rate, then the reported ASR is attributable to memorization or to the target-prefix protocol rather than to GeneBreaker's jailbreak mechanism.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that GeneBreaker steers Evo-series models to generate sequences over 90% identical to known pathogens, implying a general jailbreak capability. For that claim to hold, the benchmark targets must be absent from Evo training corpora; otherwise the outputs can be explained by memorization. Section 4 states only that 'a thorough validation' was performed, with no method, thresholds, window size, or results. This is the load-bearing gap. It is sharpened by the evaluation protocol in Section 5.1, which says 'the first half of each DNA sequence is used as input, and the DNA model is asked to generate a subsequent sequence.' If the input prefix comes from the target and the target is represented in training, the model can produce a memorized continuation, and a BLAST hit against JailbreakDNABench then reflects sequence continuation rather than jailbreak. The ablation in Figure 3(c), reporting zero attack success without GeneBreaker, does not settle this unless the no-attack condition uses the identical target prefix and only removes the few-shot prompt and beam search; the paper does not state that. The monotonic increase in success rate with model size is also consistent with larger models memorizing more training data. Without releasing the benchmark sequences and the exact prompts, and without a transparent contamination screen, the headline result is unverifiable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents GeneBreaker, a three-stage framework for red-teaming DNA foundation models. First, an LLM agent retrieves homologous but purportedly non-pathogenic DNA sequences to construct few-shot prompts. Second, a beam search guided by the PathoLM pathogenicity classifier and average log-probability selects generated chunks. Third, success is measured by nucleotide/protein BLAST identity above 90% against a curated database of human pathogenic viruses (JailbreakDNABench). Experiments on Evo1-7B and Evo2-1B/7B/40B report attack success rates up to 60%, case studies on SARS-CoV-2 spike and HIV-1 envelope with AlphaFold3 structural validation, and an evolutionary analysis of SARS-CoV-2 variants.","tokens_in":15467,"tokens_out":6843,"duration_ms":69728,"significance":"If the results are robust, GeneBreaker would be a useful tool for assessing biosecurity risks of genomic foundation models. The benchmark JailbreakDNABench and the observed scaling trend (larger models are more susceptible) are valuable. However, the significance is currently limited by the lack of evidence that the model outputs reflect a general steering capability rather than memorization of training sequences, and by the overstatement inherent in calling the attack a 'jailbreak' when the target models are not safety-aligned.","major_comments":[{"comment":"The claim that \"a thorough validation\" ensured the selected sequences do not appear in Evo training data is unsupported; no procedure, thresholds, or results are given. Because the benchmark sequences are the targets of the BLAST evaluation, any contamination would allow the reported success rates in Table 1 to be explained by memorization. Please provide the exact contamination screen (e.g., exact-match or k-mer search against the Evo training corpus), including the similarity threshold and the fraction of target sequences or fragments that were excluded.","section":"Section 4 (Benchmark Construction)"},{"comment":"The evaluation uses the first half of each target DNA sequence as input, which is a strong memorization cue: if the target or a close relative is present (even partially) in training, the model can produce the continuation from memory. The ablation in Figure 3(c) shows zero success without GeneBreaker, but this only controls for memorization if the no-attack condition uses the identical target prefix and removes only the few-shot prompt and guided beam search. Please specify the exact comparison and add a control with a non-target or shuffled prefix to distinguish generalization from memorization.","section":"Section 5.1 and Figure 3(c)"},{"comment":"The paper calls GeneBreaker a \"jailbreak attack\" on Evo models, but Evo1 and Evo2 are unaligned base generative models; the text does not identify any safety mechanism (e.g., refusal training, output filtering) that is bypassed. The results are more accurately described as targeted prompt-guided generation. This overstatement affects the claimed significance as a security vulnerability, although the underlying methodology and empirical findings remain of interest.","section":"Throughout (title, abstract, Section 1)"},{"comment":"The evolutionary modeling analysis reports that 201 of 10,000 generated sequences match Nextstrain SARS-CoV-2 entries at >99.9% identity. Since Evo2 was trained on a large corpus that likely contains many SARS-CoV-2 genome sequences, the high hit rate may reflect memorization rather than an emergent evolutionary modeling capability. The paper should provide a contamination analysis against the Nextstrain database, or temper the interpretation accordingly.","section":"Section 5.5 (Evolutionary Modeling)"}],"minor_comments":[{"comment":"The citation list for gradient-based attacks contains an unresolved placeholder \"[?]\" and should be completed.","section":"Section 2.1"},{"comment":"Two panels are both labeled \"(b)\" (the correlation with prompt homology and the ablation study); renumber them to avoid confusion.","section":"Figure 3"},{"comment":"There is a typo: \"experineced\" should be \"experienced\", and the model name is inconsistently spelled as \"GENERator\" and \"Generator\"; use one consistent spelling.","section":"Section 2.2 and Table 3"},{"comment":"The category list includes both \"Positive-strand RNA viruses\" and \"Enteric RNA viruses\", but Table 2 lists both as \"+ssRNA\"; clarify whether enteric RNA viruses are a subset or a separate category.","section":"Section 4 and Table 2"},{"comment":"Many entries have very large standard deviations (e.g., 20.0 ± 40.0), so the claim of consistent success across categories should be softened or supported with additional trials.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is relevant to the security and AI-biosafety community, but the 'jailbreak' framing is likely to be contested because the Evo models are not safety-aligned. The decisive issue is the missing contamination analysis: without a rigorous, described screen against the Evo training corpus, the empirical results could be fully explained by memorization. I recommend major revision with emphasis on the contamination screen and a control experiment that isolates memorization from generalization."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know about this one: it is the first systematic attempt to attack DNA foundation models with jailbreak-style prompts, and it ships a benchmark (JailbreakDNABench) plus an attack pipeline that includes an LLM agent for retrieving homologous non-pathogenic sequences, PathoLM-guided beam search, and BLAST evaluation. On Evo2-40B they report up to 60% attack success across viral categories. That headline number is plausible, but I would hold off treating it as a proven jailbreak vulnerability.\n\nWhat is genuinely useful: the benchmark construction (six viral categories, HHS/USDA select agent-inspired) and the combination of techniques. The case studies on SARS-CoV-2 spike and HIV-1 env with AlphaFold3 structural alignment are a nice touch—they show the outputs are not just sequence-close, they fold similarly. The correlation between log-probability and pathogen similarity (Figure 3a) is a real, reproducible observation that could inform detection.\n\nNow the soft spots, in order of seriousness.\n\nFirst, the load-bearing claim that JailbreakDNABench targets are absent from Evo training data is supported by a single sentence: 'we conducted a thorough validation.' No method, no thresholds, no results. This matters because the evaluation protocol feeds the first half of the target sequence as the prompt prefix. If that prefix or near-identical sequence is in training, the model may simply be completing a memorized sequence, and a BLAST hit against the same database is almost guaranteed. The ablation (Figure 3c) shows zero ASR without GeneBreaker, but it does not tell us whether the no-attack baseline uses the same target prefix and simply removes the few-shot prompt and beam search. If it does, the zero result would be surprising—unless the model has no capability to continue from the target prefix at all. The paper needs to release the benchmark sequences, the exact prompts, and a transparency screen.\n\nSecond, the 'jailbreak' framing is overstated. Evo models are base autoregressive models; they have no safety alignment or refusal mechanisms. You cannot bypass a guardrail that does not exist. What GeneBreaker does is steer generation toward pathogen-like sequences using homology and guided search, which is a real dual-use concern, but it is not a jailbreak in the LLM sense. The term invites a misinterpretation of what was shown.\n\nThird, the 'evolutionary modeling' section is really just sampling with a SARS-CoV-2 spike prompt and counting BLAST hits. It is fine as a demonstration, but the title oversells it.\n\nThe bottom line: the method and benchmark deserve referee time, but the paper needs (a) evidence of contamination screening, (b) code/data release with exact protocols, and (c) a more precise framing. I would send it to review, with the expectation of heavy revision. It is a paper for the AI-biosecurity crowd and for developers of genomic foundation models. I would cite the benchmark once it is cleaned up and public.","headline":"First systematic jailbreak-style benchmark for DNA language models, but the 'jailbreak' label oversells and the training-data exclusion claim is unverified.","tokens_in":16031,"tokens_out":3264,"would_cite":true,"duration_ms":32508,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper shows that DNA foundation models can be jailbroken with high-homology prompts and pathogenicity-guided beam search, generating sequences over 90% identical to human pathogens.","keywords":["DNA foundation models","jailbreak attack","biosecurity","beam search","pathogenicity guidance","Evo2","SARS-CoV-2","JailbreakDNABench"],"falsifier":"Audit the training corpus of the Evo2 models for the exact coding sequences, or near-exact (>99% identity) copies, of the viruses in JailbreakDNABench: if any target sequence appears in training, the observed outputs could be retrieval rather than generation, and the jailbreak claim loses its force. A cleaner test is to run GeneBreaker against a DNA language model trained on a corpus that provably excludes all human-pathogenic viral sequences and check whether the attack success rate persists.","tokens_in":15030,"feed_emoji":"🦠","tokens_out":10256,"duration_ms":90143,"temperature":0.7,"pith_summary":"GeneBreaker is a framework that tries to prove DNA foundation models can be jailbroken, not just that they have flaws in isolation. It feeds the victim model a prompt made of non-pathogenic DNA sequences that are highly similar to a human pathogen, then runs a beam search in which each candidate chunk is scored by a pathogenicity classifier plus the model's own log-probability, steering generation toward pathogen-like sequence space. On a curated benchmark of six viral categories, the method reports attack success rates up to 60% on Evo2-40B, and the generation of sequences over 90% identical to viruses such as SARS-CoV-2 and HIV-1. Case studies show the generated viral proteins are structurally faithful to their native counterparts when folded with AlphaFold3. If correct, this result implies that simply excluding pathogens from training data is not a sufficient safeguard for genomic AI models, and that model scaling amplifies the risk.","feed_headline":"Jailbreak attack makes DNA models emit pathogen DNA","feed_subtitle":"GeneBreaker guides Evo2 to generate sequences over 90% identical to SARS-CoV-2 and HIV-1 targets.","key_machinery":"The central mechanism is the jailbreak-oriented scoring function f = PathoLM(x) + α·log p(x), which combines a pathogenicity prediction from PathoLM, a DNA language model fine-tuned to classify viral and bacterial sequences as pathogenic, with the average log-probability the victim model assigns to a candidate sequence chunk. Higher values of f indicate that a chunk is more likely to be pathogen-like, and the function is used inside a beam search that samples chunks of length 128, keeps the top-K' partial sequences, and repeatedly appends the best chunks to the prompt. The load-bearing design choice is the prompt itself: the non-pathogenic but high-homology sequences serve as in-context examples that shift the victim model's distribution toward the target pathogen region, while the scoring function ensures that each beam step prefers continuations that look pathogenic.","core_discovery":"The paper's central claim is that GeneBreaker successfully jailbreaks the latest Evo series DNA language models across six viral categories, with attack success rates reaching 60% for Evo2-40B. The attack works by retrieving non-pathogenic sequences with high homology to the target pathogen (for example bat coronaviruses close to SARS-CoV-2), using them as few-shot prompts, and then guiding an autoregressive beam search with a scoring function f = PathoLM(x) + α·log p(x), where PathoLM is a pathogenicity-classification DNA model and log p(x) is the average log-probability of the candidate under the victim model. Outputs are evaluated with nucleotide and protein BLAST against JailbreakDNABench, a curated database of human-pathogenic viruses built from Select Agents lists, and a jailbreak is scored when identity exceeds 90%. The reported case studies include a SARS-CoV-2 spike protein with 92.77% DNA similarity and 95.29% protein similarity, an HIV-1 envelope protein with 96.09% DNA and 96.47% protein similarity whose AlphaFold3-predicted structure matches the crystal structure at 0.334 RMSD, and an evolutionary experiment in which 201 of 10,000 generated spike sequences match Nextstrain-circulating SARS-CoV-2 variants at over 99.9% identity.","pith_inferences":["Editorial inference: the same recipe should transfer to other generative biological sequence models, such as protein language models, whenever a pathogenicity scorer is available, because the guidance signal is model-agnostic.","Editorial inference: defenders could invert the attack by using the same scoring function as an output filter, turning GeneBreaker into a red-team benchmark for biosecurity screening rather than only an exploit.","Editorial inference: the paper's reported success rates rest on an unverified training-data containment claim, so a third-party audit of Evo2's corpus is the natural next experiment to separate generalization from memorization.","Editorial inference: sequences below the 90% identity threshold can still be structurally and functionally dangerous, so the actual biosecurity risk boundary may lie lower than the paper's flag threshold."],"forward_implications":["A user with only public model access, no fine-tuning, and no DNA-synthesis access can produce sequences that clear the 90% identity threshold used in biosecurity screening for known human pathogens.","Attack success rises monotonically with model size across the tested Evo series, so scaling DNA foundation models without safeguards increases dual-use risk.","Excluding pathogenic sequences from training data is not a sufficient defense, because the attack reconstructs pathogen-like sequences from non-pathogenic homologs.","The method can also act as an evolutionary modeling tool: Evo2-40B generates SARS-CoV-2 spike variants matching real circulating clades such as Alpha, Delta, and Omicron sublineages.","The generated DNA translates into proteins that fold like native viral proteins, so the jailbreak outputs are not just sequence matches but potentially functional sequences."],"supporting_citations":[{"why":"Introduces the Evo2 models that serve as the primary targets of the jailbreak attack and describes the beam-search generation scheme the paper builds on.","marker":"[11]"},{"why":"Introduces the Evo model family and its genome design capabilities, establishing the baseline generation capability the attack targets.","marker":"[39]"},{"why":"Provides PathoLM, the DNA model used to score candidate sequences for pathogenicity during beam search.","marker":"[17]"},{"why":"Supplies the BLAST algorithm used to compare generated sequences against the human pathogen database and evaluate attack success.","marker":"[70]"},{"why":"Defines the Common Mechanism screening standard that motivates the paper's 90% similarity threshold for flagging a jailbreak.","marker":"[1]"},{"why":"Lists the Select Agents and Toxins that guided curation of the six viral categories included in JailbreakDNABench.","marker":"[19]"},{"why":"AlphaFold3 is used to predict the structures of jailbreak-generated proteins and compare them with crystal structures.","marker":"[2]"},{"why":"Provides the Nextstrain global SARS-CoV-2 dataset used to test whether generated spike sequences match circulating variants.","marker":"[23]"},{"why":"Serves as the repository from which the high-homology non-pathogenic sequences are downloaded for prompt construction.","marker":"[53]"}],"fun_headline_variants":["GeneBreaker hijacks DNA models to generate pathogen DNA","Jailbreaking DNA models yields 60% attack success on Evo2","New attack forces DNA models to synthesize viral sequences","DNA models jailbroken to emit SARS-CoV-2, HIV-1 sequences","GeneBreaker: DNA jailbreak creates biosecurity risk"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the target viral sequences in JailbreakDNABench were not part of the Evo series models' training data; if they were, the reported high-similarity outputs could be explained by memorization instead of a generalizable jailbreak capability.","fun_headline_variants_meta":{"raw":{"variants":["GeneBreaker hijacks DNA models to generate pathogen DNA","Jailbreaking DNA models yields 60% attack success on Evo2","New attack forces DNA models to synthesize viral sequences","DNA models jailbroken to emit SARS-CoV-2, HIV-1 sequences","GeneBreaker: DNA jailbreak creates biosecurity risk"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001156,"raw_usage":{"total_tokens":4870,"prompt_tokens":1105,"completion_tokens":3765,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":721,"completion_tokens_details":{"reasoning_tokens":3677}},"tokens_in":721,"tokens_out":3765,"duration_ms":30230,"temperature":1.0,"reasoning_tokens":3677,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:08:26.645918+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Audit the training corpus of the Evo2 models for the exact coding sequences, or near-exact (>99% identity) copies, of the viruses in JailbreakDNABench: if any target sequence appears in training, the observed outputs could be retrieval rather than generation, and the jailbreak claim loses its force. A cleaner test is to run GeneBreaker against a DNA language model trained on a corpus that provably excludes all human-pathogenic viral sequences and check whether the attack success rate persists.","supporting_citations":[{"cited_title":"PathoLM: Identifying pathogenicity from the DNA sequence through the Genome Foundation Model","cited_arxiv_id":"2406.13133","evidence_quote":"Provides PathoLM, the DNA model used to score candidate sequences for pathogenicity during beam search."},{"cited_title":"Blast: improvements for better sequence analysis","cited_arxiv_id":null,"evidence_quote":"Supplies the BLAST algorithm used to compare generated sequences against the human pathogen database and evaluate attack success."},{"cited_title":"Select agents and toxins list, 2025","cited_arxiv_id":null,"evidence_quote":"Lists the Select Agents and Toxins that guided curation of the six viral categories included in JailbreakDNABench."},{"cited_title":"Nextstrain: real-time tracking of pathogen evolution","cited_arxiv_id":null,"evidence_quote":"Provides the Nextstrain global SARS-CoV-2 dataset used to test whether generated spike sequences match circulating variants."},{"cited_title":"Ncbi taxonomy: a comprehensive update on curation, resources and tools","cited_arxiv_id":null,"evidence_quote":"Serves as the repository from which the high-homology non-pathogenic sequences are downloaded for prompt construction."}],"review_version":1}