{"id":"4f8b7aff-dd3c-4f50-a18b-21756d6e10ec","arxiv_id":"2602.17162","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"Adding a JEPA latent objective to DNABERT-2 improves several genomics benchmark scores, but the paper lacks the control needed to prove the JEPA objective—not extra pretraining—is responsible.","lead":"JEPA-DNA adds a JEPA-style latent prediction head to DNABERT-2, training the model to predict a global embedding of masked DNA in addition to reconstructing tokens. The paper reports improved benchmark scores, but a missing continued-pretraining control means the gains cannot yet be attributed to the JEPA objective.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The causal claim that the JEPA objective, not continued pretraining, drives the reported gains is untested: no MLM-only control trained on the same 7.6B bp data and schedule is reported, and the promised ablations are deferred.","rationale":"The reader's weakest assumption identifies exactly the same gap I see: the experimental design cannot attribute the observed improvements to the JEPA branch because the baseline is the original DNABERT-2 checkpoint, not a continued-MLM control. This is not a disagreement with a scientific consensus; it is an internal control problem that directly affects the central claim. The paper's own limitations section confirms the missing pieces: significance analysis and comprehensive ablations are listed as future work. The public code makes the proposed control runnable, but as written the causal claim is unsupported. Therefore I agree with the reader's REJECT verdict and recommend no change. If the proposed control is run and shows JEPA-DNA still wins broadly, the verdict could later move to ACCEPT or CONDITIONAL, but that evidence does not currently exist.","tokens_in":9282,"tokens_out":5576,"duration_ms":50113,"concrete_test":"Run a compute-matched control: initialize from the same DNABERT-2 checkpoint, continue pretraining for exactly 5 epochs on the identical 7.6B bp subset with the same span masking (20-40%), same optimizer, LR schedule, and effective batch size, but with only the MLM loss (no JEPA, no variance/covariance). Apply the identical linear-probing and zero-shot protocols from Tables 1 and 2 to this control. If the control matches or beats JEPA-DNA on the majority of the 17 tasks, or if JEPA-DNA's average AUROC advantage falls within the noise floor, the central causal claim fails; if JEPA-DNA still wins on the tasks where gains are claimed and the differences exceed a paired bootstrap significance threshold, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is causal: the JEPA objective, not the continued pretraining, drives the reported gains. But every comparison pairs JEPA-DNA (DNABERT-2 weights + 5 epochs on a 7.6B bp subset with 20-40% span masking and multi-objective loss) against the original DNABERT-2 checkpoint, which never saw this subset or this masking schedule. The effect of the JEPA branch is therefore fully confounded with the effect of continued training under a different data distribution and masking scheme. No MLM-only control trained on the same 7.6B bp data with the same span masking, optimization schedule, and number of steps is reported anywhere. Absent that control, Tables 1 and 2 cannot support the abstract's claim that JEPA 'establishes a new state-of-the-art.' The internal evidence further weakens the causal claim: Table 1 contains negative entries (VB sQTL -0.529% AUROC) and Table 2 contains substantial drops (LRB Pathogenic OMIM 0.495 to 0.452), contradicting 'consistent gains'; §5 defers comprehensive ablations and significance analysis to future work despite the abstract's promise of 'extensive ablation studies.' Thus, the load-bearing causal premise is untested.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes JEPA-DNA, a continual pre-training framework that adds a Joint-Embedding Predictive Architecture (JEPA) branch to a genomic foundation model such as DNABERT-2. The context encoder processes a span-masked sequence, a target encoder (EMA of the context encoder) processes the unmasked sequence, and a predictor head is trained to match the [CLS] embedding of the target encoder. The total loss combines MLM/NTP loss, a cosine-similarity JEPA loss, and VICReg variance/covariance regularization. The authors evaluate the method by linear probing and zero-shot cosine-similarity ranking on a set of genomic benchmarks and report improved AUROC over the original DNABERT-2 checkpoint on many tasks. The abstract claims consistent gains, a new state of the art for genomic foundation models, and extensive ablation studies, but the body contains no ablation section and the experimental protocol lacks the key continued-pretraining control needed to attribute the gains to the JEPA objective.","tokens_in":9616,"tokens_out":3356,"duration_ms":31775,"significance":"The idea of augmenting genomic foundation models with a latent predictive objective is interesting and potentially impactful: if it is shown to improve representations over continued generative pretraining on the same data, it would be a meaningful contribution with broad applicability across architectures. The paper is also transparent about releasing code and provides a reasonably detailed description of the loss and training schedule. However, the current evidence does not establish the central causal claim. The comparison is confounded, the promised ablations are absent, and several reported results contradict the 'consistent gains' language. The scientific question is well posed and the missing control is well defined, so the work is potentially salvageable, but the manuscript in its present form is not ready for publication.","major_comments":[{"comment":"The central claim is that gains come from the JEPA objective, but every comparison pairs JEPA-DNA (DNABERT-2 weights + 5 epochs on a 7.6B bp subset with 20–40% span masking and a multi-objective loss) against the original DNABERT-2 checkpoint, which never saw this data, masking scheme, or schedule. There is no MLM-only continued-pretraining control trained on the same 7.6B bp with the same span masking and optimization schedule. Without that control, the improvements in Tables 1–2 cannot be attributed to the JEPA branch; they may be entirely due to continued pretraining under a different data distribution. This is the load-bearing flaw of the paper.","section":"§4.1–4.2, Tables 1–2"},{"comment":"The abstract promises 'extensive ablation studies,' but the paper contains no ablation section. Section 5, under 'Comprehensive Ablations,' defers to future work a deeper analysis of the predictor head, loss hyper-parameters, and masking configurations. At minimum, the paper should report ablations for the loss weights (λ1–λ4), EMA momentum schedule, masking ratio/span, and the MLM-only control. As written, the abstract overstates what the manuscript delivers.","section":"Abstract vs. §5"},{"comment":"The paper repeatedly claims 'consistent improvements,' but the data do not support this. Table 1 reports a negative gain on VB sQTL (−0.529%) and a negligible gain on LRB Causal eQTL (+0.142%); Table 2 reports drops on TraitGym Complex (0.499→0.491) and LRB Pathogenic OMIM (0.495→0.452). Additionally, no confidence intervals or significance tests are provided, which is especially important for clinical/biological benchmarks. The authors should either weaken the claim to 'improvements on several tasks' or report per-task statistical significance. The abstract's 'regardless of the underlying GFM architecture or generative objective' is also unsupported because only DNABERT-2 (an MLM Transformer) is evaluated.","section":"§4.2, Tables 1–2"}],"minor_comments":[{"comment":"Typo: 'haracteristic' should be 'characteristic' in 'Area Under the Receiver Operating haracteristic curve.'","section":"§4.1"},{"comment":"Typo: 'design to learn' should be 'designed to learn.' Also, 'preprended' in §3.5 should be 'prepended.'","section":"§2.2"},{"comment":"The notation Z∈R^{B×d} is introduced but it is not explicit whether Z contains context-encoder outputs, predictor outputs, or both; the variance loss paragraph mentions both but the equations use only Z. Please clarify.","section":"§3.4.1"},{"comment":"The '–' marker for 'near-random' performance is used inconsistently: for VB Common vs. Rare, values are shown and then '–' appears in the Gain column, while for other rows it is omitted. Define the threshold for 'near-random' and apply it uniformly.","section":"Tables 1–2"},{"comment":"Reference [10] has a typo: 'V olume' should be 'Volume.' Also, several arXiv references lack version numbers; consider adding them for reproducibility.","section":"References"},{"comment":"The claim that JEPA-DNA is compatible with NTP/SSM architectures is not accompanied by any experiment on those architectures; Section 5 lists testing other GFMs as future work. Please mark this as a claim about design rather than an empirical result.","section":"§3.5"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's central claim is plausible but currently unsubstantiated because of the missing continued-MLM-only control. The required experiments are well defined and feasible, so I would not reject outright. However, the discrepancy between the abstract's promise of extensive ablations and the absence of any ablation section is a serious presentation issue that should be addressed in revision. If the authors cannot run the control or the ablations, then the claims must be substantially weakened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: this is a reasonable application of JEPA to raw DNA, but the headline claim is overreach as written. The experiments compare JEPA-DNA (DNABERT-2 plus 5 epochs of continued pretraining on 7.6B bp with span masking) against the original DNABERT-2 checkpoint. That means the JEPA objective is fully confounded with continued pretraining on a different data distribution. Without a compute-matched MLM-only control on the same data and schedule, Tables 1 and 2 cannot tell us whether the JEPA branch or simply more training on the same data drove the gains. The abstract promises \"extensive ablation studies\"; the body has none, and §5 explicitly defers ablations and significance analysis to future work.\n\nThe novelty is honest: GeneJEPA operates on transcriptomics vectors, LLM-JEPA on text; this is the first application to raw genomic sequence. The architecture is standard JEPA machinery—EMA target encoder, predictor, VICReg—adapted with a CLS token and span masking. The benchmark suite is reasonably broad, and some improvements look real (TF binding +3.2% AUROC, zero-shot expression +6.9%). Code is posted. These are worth taking seriously.\n\nBeyond the confound, the paper's own tables contradict \"consistent gains\": VB sQTL drops -0.5%, TraitGym Complex drops, LRB Pathogenic OMIM drops from 0.495 to 0.452. No confidence intervals anywhere. The claim of architecture-agnostic \"regardless of underlying GFM\" is tested on a single backbone. There's also a citation error: HyenaDNA shares an arXiv ID with DNABERT-2 in the reference list. These are fixable but they matter.\n\nWho is this for? Groups working on GFM pretraining objectives. They'll get a useful idea and a baseline result, but they should not take the state-of-the-art claim at face value.\n\nSend this to peer review—the method and the questions are legitimate, and the missing control is addressable. A revised version with an MLM-only control, ablations, and confidence intervals could be a solid contribution. Desk rejecting would be too harsh.","headline":"A plausible JEPA-for-DNA method that is overclaimed: the reported gains are confounded with continued pretraining, and the promised ablations are absent.","tokens_in":10170,"tokens_out":2413,"would_cite":false,"duration_ms":21693,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that genomic foundation models learn better, more functional DNA representations when they are forced to predict the latent embedding of masked segments rather than only reconstructing nucleotides.","keywords":["JEPA","genomic foundation models","masked language modeling","latent representation learning","DNA sequence","variant effect prediction","self-supervised learning","continual pretraining"],"falsifier":"Train DNABERT-2 on the same 7.6B base pairs with only the MLM loss, using the same masking, optimizer, schedule, and number of epochs; if its linear-probing and zero-shot numbers match or exceed JEPA-DNA's, the central claim collapses. A second check: apply the JEPA branch to a non-transformer backbone (e.g., a state-space model) and see whether the claimed model-agnostic transfer holds.","tokens_in":9183,"feed_emoji":"🧬","tokens_out":2700,"duration_ms":27118,"temperature":0.7,"pith_summary":"The paper introduces JEPA-DNA, a training framework that augments standard genomic language models with a joint-embedding predictive branch. Instead of only asking the model to fill in masked nucleotides, it also asks a predictor head to guess the global [CLS] embedding of the original unmasked sequence from a masked context. The authors claim this latent grounding shifts learning from local token syntax toward global functional semantics, and that doing so improves frozen-feature linear probing and zero-shot variant-effect ranking on top of an existing model like DNABERT-2. The framework is designed to be model-agnostic, compatible with both MLM and next-token-prediction backbones. A sympathetic reader would care because it proposes a relatively cheap, architecture-agnostic way to make existing DNA models more biologically grounded without requiring new data or new architectures.","feed_headline":"JEPA branch lifts DNA model accuracy on functional benchmarks","feed_subtitle":"A latent-space prediction target makes DNABERT-2 better at variant effects without new data or architecture.","key_machinery":"The central object is the latent-prediction branch: a predictor head that takes the context encoder's masked-sequence representation, with masked positions re-masked to prevent trivial copying, and must match the [CLS] embedding produced by an EMA-updated target encoder on the unmasked sequence. Matching is done by cosine similarity, with VICReg-style variance and covariance regularization to prevent representation collapse. Span masking of 20-40% of tokens makes the prediction task nontrivial, and the same masking is shared with the standard LLM loss.","core_discovery":"The authors claim that coupling a generative objective (MLM/NTP) with a latent predictive objective that supervises a [CLS] token produces genomic representations that are more linearly separable and more functionally meaningful. Concretely, they show that continually pre-training DNABERT-2 with this dual objective on about 7.6 billion base pairs improves linear-probing AUROC on transcription factor binding, promoter, splice site, coding and non-coding pathogenicity, and several QTL tasks, and improves zero-shot ranking of expression-effect and Mendelian variants. They attribute these gains to the JEPA branch forcing the model to predict the global functional embedding of masked segments, th","pith_inferences":["The reported gains are measured against the original DNABERT-2 checkpoint, not against a control that continues MLM-only pretraining on the same 7.6B base pairs with the same schedule and epochs; until that control is run, part of the improvement could be attributable to continued pretraining rather than the JEPA objective.","The model-agnostic claim rests on a single backbone, DNABERT-2; a direct transfer test on an NTP or state-space-model backbone would be the natural way to validate the claimed generality.","If the zero-shot gains are real, they suggest a practical cheap use: ranking variants by embedding distance in a frozen grounded model before any fine-tuning, which could aid clinical variant triage.","The paper's own mention of an extra forward pass in evaluation mode to stabilize variance hints that collapse prevention is delicate; scaling the method may require careful length-batching or other stabilization strategies."],"forward_implications":["If the central claim is correct, any existing genomic foundation model can be 'grounded' by a continual JEPA pre-training phase without changing its architecture or generative objective.","The method claims to extend naturally to NTP-based and state-space-model backbones by appending an [EOS] token supervised by the JEPA loss, so the latent-grounding benefit should transfer beyond transformer-encoder MLM models.","Linear-probing gains on frozen features imply the improvement lives in the representation itself, not in task-specific fine-tuning, making the method useful as a drop-in enhancement.","Zero-shot variant-effect ranking gains suggest the embeddings carry functional signal that can be exploited before any downstream training, which is relevant for tasks with scarce labels.","Because the masking is shared between the JEPA and LLM losses, the framework also makes the reconstruction objective more challenging, potentially contributing to the observed improvements."],"fun_headline_variants":["JEPA latent target sharpens DNA model's functional readout","Predicting functional embeddings boosts DNA model benchmarks","Latent-supervised DNA model outscores generative-only peers","JEPA-DNA: latent semantic grounding lifts benchmark scores"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The comparison assumes that continued MLM-only pretraining on the same data with the same schedule would not produce comparable gains, but no such control is reported anywhere in the paper.","fun_headline_variants_meta":{"raw":{"variants":["JEPA latent target sharpens DNA model's functional readout","Predicting functional embeddings boosts DNA model benchmarks","Latent-supervised DNA model outscores generative-only peers","JEPA-DNA: latent semantic grounding lifts benchmark scores"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000839,"raw_usage":{"total_tokens":3478,"prompt_tokens":715,"completion_tokens":2763,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":459,"completion_tokens_details":{"reasoning_tokens":2699}},"tokens_in":459,"tokens_out":2763,"duration_ms":19578,"temperature":1.0,"reasoning_tokens":2699,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T22:18:12.208895+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train DNABERT-2 on the same 7.6B base pairs with only the MLM loss, using the same masking, optimizer, schedule, and number of epochs; if its linear-probing and zero-shot numbers match or exceed JEPA-DNA's, the central claim collapses. A second check: apply the JEPA branch to a non-transformer backbone (e.g., a state-space model) and see whether the claimed model-agnostic transfer holds.","supporting_citations":[],"review_version":1}