{"id":"e4c6f492-4a4e-480d-8fbe-6bb48e4a8b44","arxiv_id":"2512.00283","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"BioArc uses neural architecture search to discover small hybrid models for DNA and protein that outperform far larger pretrained foundation models on several benchmarks.","lead":"This paper introduces BioArc, a framework that automatically searches for small neural-network architectures that work well on DNA and protein tasks, and finds designs that often beat much larger foundation models. It also tests methods that predict good architectures for new biological tasks, giving a practical roadmap for building more efficient biological models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Architecture ranking (Eq. 8) lacks a stated validation/test split; if the same test labels drove selection, the headline gains are selection artifacts rather than intrinsic architecture advantages.","rationale":"The reader's weakest assumption is exactly the load-bearing issue: the architecture ranking protocol in Section 3.4 does not state that ranking was performed on a validation split disjoint from the test sets in Tables 2–3. This is not an internal inconsistency, but it is a missing premise that the headline claims require. The paper's supplementary analyses (layer-wise interpretation, hybrid-vs-single-module comparisons, performance-parameter scatter) provide useful supporting evidence, but none of them establish split hygiene. The 'only-ft' wins with zero pretraining make selection-on-test-set the most plausible alternative explanation for the reported margins. Since the reader already judged this conditional and my read does not move the verdict, I leave it unchanged.","tokens_in":29499,"tokens_out":2798,"duration_ms":33072,"concrete_test":"Inspect the released code (BioArc-9794) and identify, for each of the 18 downstream tasks, the exact data split used when computing Score(a) in Eq. 8. Then re-run the ranking using only a held-out validation split (labels never used to produce Tables 2–3), select the top architectures on validation, and report their test performance. If the top-1 architecture changes or the reported margins over DNABERT-2/VQDNA shrink beyond a few points, the headline claim is a selection artifact. Also report per-seed error bars for the selected architectures.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.4 defines Score(a) over 'downstream tasks T' and selects top architectures A_top using Eq. 8, but nowhere states that T is a validation split disjoint from the test labels reported in Tables 2–3. The strongest claims (RQ1/RQ3) depend on the selected architectures being chosen without access to those test labels. The paper's own Spearman correlation (A.6.13) shows that supernet-initialized ranking correlates with from-scratch ranking (ρ=0.817), but that does not address split hygiene—it only validates the supernet proxy. The 'only-ft' result, where zero-pretraining models outperform heavily pretrained baselines on DNA tasks, is especially suspicious: if architectures were selected on test labels, selection can manufacture large margins even with no pretraining. No passage in the main text or appendices specifies that the 360 architectures were evaluated on a held-out validation set before selecting the top candidates. Without that split, the central claim that BioArc discovers intrinsically superior architectures is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces BioArc, a Neural Architecture Search framework for biological sequence models. It defines a search space over CNN, LSTM, Mamba, Hyena, and Transformer blocks with varying depth and width, prunes the space from 67M combinations to 360 representative paths, trains a weight-sharing supernet under masked modeling, contrastive learning, and next-token prediction, and then evaluates each path either from scratch or from supernet-initialized weights on 12 DNA tasks (GUE) and 6 protein tasks (PEER). Architectures are ranked by a z-score-aggregated score (Eq. 8), and the top performers are reported as task-specific and foundation-model backbones. The paper additionally proposes three architecture-prediction methods — a GNN predictor, LLM+RAG, and a multi-agent LLM system — evaluated in supervised and transfer settings. The central claims are that BioArc-discovered architectures outperform much larger pretrained biological models, that optimal architectures show a Hyena-to-Transformer-to-CNN pattern, and that a BioArc-foundation model achieves state-of-the-art results with roughly 1/20 the model size and 1/10 the training steps of existing DNA foundation models.","tokens_in":29809,"tokens_out":5569,"duration_ms":54301,"significance":"If the central claims hold, BioArc would be a substantial contribution: it provides a systematic, automated alternative to intuition-driven architecture design in biology, yields concrete design principles (Hyena->Transformer->CNN), and proposes methods to predict architectures for new tasks. The paper has real strengths: a broad and carefully pruned search space, three pretraining objectives, a large set of downstream tasks, internal controls such as the Spearman rank correlation (rho=0.817, Appendix A.6.13) between supernet-initialized and from-scratch rankings, layer-wise interpretability analysis (Appendix A.7), and a held-out transfer setting for architecture prediction. However, the empirical claims of state-of-the-art performance are currently undermined by a load-bearing methodological gap: the architecture ranking protocol in Section 3.4 never states whether the downstream tasks used for selection are disjoint from the test labels reported in Tables 2 and 3. Without a validation/test split, the headline gains — especially the zero-pretraining BIOARC(only-ft) results — could be selection artifacts rather than intrinsic architecture advantages. The main experimental tables a","major_comments":[{"comment":"The architecture ranking protocol is the load-bearing step for RQ1 and RQ3, but the paper never states that the set of downstream tasks T used in Eq. (8) is a validation split disjoint from the test sets reported in Tables 2 and 3. Since those tables report performance on exactly the tasks used for ranking, the top architectures may have been selected using test labels. This is especially acute for BIOARC(only-ft), which uses zero pretraining data: with test-label-based selection, a model with no pretraining can appear to beat heavily pretrained baselines merely because it was chosen for that test set. The paper must either state the validation-split protocol explicitly or rerun the selection and evaluation with a held-out split. Without this, the claim that BioArc 'discovers optimal architectures' is unsupported.","section":"Section 3.4, Eq. (8)"},{"comment":"The main SOTA tables report a single number per method and task, with no error bars, no number of seeds, and no significance tests. Many differences among BioArc variants and against baselines are smaller than 1 percentage point (e.g., Table 2, TFP-0: only-ft 84.70 vs mask-ft 84.80; Table 3, Solubility: BioARC(only-ft) 73.34 vs ESM2-8m 73.48). Without uncertainty estimates, the claimed outperformance of much larger models is not quantitatively established. Relatedly, the text in Section 4.2 states that BioARC 'outperform[s] larger counterparts' on sequence-level tasks like Solubility, but Table 3 shows ESM2-8m (8M parameters) scoring 73.48 vs BioARC-only-ft 73.34; the claim is contradicted by the paper's own table. Please report mean +/- std over at least 3 seeds and qualify the claim.","section":"Tables 2 and 3"},{"comment":"The foundation-model experiment selects the BioArc-F architecture by 'top average performance across all tasks' and then evaluates it on the same downstream tasks. This repeats the selection-on-test-set concern: the architecture is not independent of the evaluation tasks. Additionally, the comparison '1/10 training steps' is against full-pretraining baselines; to attribute the gain to the architecture, the authors should compare against a baseline architecture trained for the same 1/10 steps (or matched compute). Without this control, the claim that BioArc-F 'outperforms human-designed architectures' at reduced budget is not cleanly supported.","section":"Figure 3 / RQ3"}],"minor_comments":[{"comment":"Train/test splits are provided for the architecture-prediction experiments, but no analogous split is given for the architecture-ranking protocol of Section 3.4. This is not merely a presentation issue; it directly relates to Major Comment 1.","section":"Appendix A.6.1"},{"comment":"The naming 'BIOARC(only-ft)' is confusing: it denotes training from scratch with no pretraining, not fine-tuning. Consider renaming to 'BIOARC(scratch)' for clarity.","section":"Tables 2 and 3"},{"comment":"Section 3.4 states that each path is 'optimized independently' via fine-tuning or from-scratch training, but Appendix A.6.11/Figure 12 describes evaluating paths 'directly from the pretrained supernet by freezing the backbone weights.' Please clarify which evaluation mode is used for the ranking correlation in A.6.13; if the correlation is computed with frozen weights, it does not validate the ranking used for Tables 2-3.","section":"Section 3.4 vs Appendix A.6.13"},{"comment":"In Figure 10, the task index labels are small and some curves are hard to distinguish. Enlarge fonts or use distinct markers.","section":"Appendix A.6.9"},{"comment":"The sentence 'We use one A100 80G for the pretraining for 10 epoch' should be corrected (grammar and subject-verb agreement).","section":"Appendix A.6.1"},{"comment":"The framework is sometimes spelled 'BIOARC' and sometimes 'BioArc' (e.g., Figure 1 vs Figure 3). Use one consistent spelling throughout.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses an important problem and contains a substantial empirical study, but the central claim currently hinges on the absence of a stated validation/test split in the architecture-ranking protocol. I would encourage the editor to request the authors to clarify or re-run the selection with a disjoint validation set; if they can show that the reported test labels were not used in selecting architectures, the paper could be suitable for acceptance. I do not see evidence of intentional misconduct, but the oversight is consequential."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"BioArc is a substantive empirical contribution: a large NAS testbed for DNA and protein foundation models, 360 architectures evaluated, and a clear set of design principles (hybrid Hyena→Transformer→CNN topologies, tokenizer and training strategy interplay). The agent-based architecture prediction on held-out tasks is a nice extension. But the load-bearing headline—that these architectures outperform much larger pretrained models, including with zero pretraining—rests on an architecture-ranking protocol that never states a validation/test split. Section 3.4 ranks paths by the mean Z-score over 'downstream tasks T' and selects the top candidates. Nothing in the main text or appendix says that T is a validation set disjoint from the test labels reported in Tables 2 and 3. If the same test labels were used to select the architectures, the only-ft and mask-ft rows are selection artifacts, and the '25x smaller' claims are not intrinsic. That doesn't invalidate the search space or the empirical design principles, but it means the central RQ1/RQ3 comparisons are unsupported until the authors clarify the split.\n\nWhat the paper does well: the scale and breadth of the evaluation, the layer-wise contribution analysis, the Spearman 0.817 correlation between supernet-initialized and from-scratch rankings (a good proxy check, though it doesn't address split hygiene), and the hybrid-vs-single-module comparison. The honest reporting that pretraining does not guarantee gains is a useful negative result. The protein structural-task gap is explained by pretraining scale and absolute positional encoding—that analysis seems fair.\n\nSoft spots: besides the split issue, Tables 2-3 are single-point metrics with no error bars or significance tests, and the only-ft wins are surprising enough that they need seeds/CIs. The code is partial and anonymized, which limits reproducibility. The architecture prediction ground truth uses the same ranking, so it inherits the same concern, though the transfer setting is less affected.\n\nThis paper deserves a serious referee. If the authors can show the ranking used only validation labels, and add robustness checks, it becomes a genuinely useful resource for biological model design. I'd bring it to reading group and cite the search space and design principles, but I wouldn't cite the SOTA numbers.","headline":"A useful NAS testbed and design principles for bio foundation models, but the headline SOTA gains need a clean validation/test split in the ranking protocol before they can be believed.","tokens_in":30236,"tokens_out":3794,"would_cite":true,"duration_ms":37809,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that automated architecture search can discover biological foundation-model backbones that are 25x smaller and still outperform human-designed architectures on DNA tasks, with hybrid Hyena-Transformer-CNN topologies as the","keywords":["neural architecture search","foundation models","DNA language models","protein language models","hybrid architectures","supernet","architecture prediction","tokenization"],"falsifier":"Check whether the GUE and PEER test-set labels were accessible during the Eq. (8) ranking step. If they were, rerun the full search on a strictly disjoint validation split and measure whether the top-ranked BioArc architectures still beat DNABERT-2, VQDNA, and ESM-2 at 25x fewer parameters. A simpler proxy: compare BioArc's from-scratch accuracy on CPD-notata with a random architecture of the same size; if random topologies match it, the search is not the cause.","tokens_in":29400,"feed_emoji":"🧬","tokens_out":6712,"duration_ms":58680,"temperature":0.7,"pith_summary":"BioArc is a neural-architecture-search framework for biological sequence models. The paper tries to establish that instead of importing NLP architectures like Transformers wholesale, one can automatically search over combinations of CNN, LSTM, Mamba, Hyena, and Transformer blocks and find models that outperform the largest pretrained DNA and protein foundation models at a fraction of the size. Its headline evidence is that a BioArc-found backbone, pretrained on 1/10 of the steps and with 1/20 of the parameters, beats DNABERT-2, Nucleotide Transformer, and VQDNA on the GUE benchmark. The paper also reports a recurring design pattern—Hyena blocks up front, Transformers in the middle, CNNs at the end—and argues that architecture, tokenization, and pretraining strategy are deeply entangled and must be co-optimized. The practical payoff, if right, is a principled way to generate task-specific and foundation backbones for biology instead of relying on human intuition.","feed_headline":"Searched DNA models beat hand-built foundation models at 1/20 size","feed_subtitle":"Small searched architectures outperform giant pretrained biological models","key_machinery":"The central object is the supernet: a single weight-sharing network whose weights are trained by randomly sampling one 'path'—a sequence of block types and hidden dimensions—per forward pass. Ranking uses Eq. (8), a Z-score aggregate of task performances. The discovered load-bearing pattern is a hybrid topology of the form Hyena → Transformer → CNN, where Hyena reads long-range context, Transformers model contextual syntax, and CNNs extract local motif vocabulary. The architecture predictors (GNN regressor, LLM+RAG, multi-agent system) are secondary machinery for exploiting the search results.","core_discovery":"On the paper's own terms, the central discovery is empirical: a one-shot NAS search over 360 representative module paths—built from CNN, Hyena, LSTM, Mamba, and Transformer blocks—yields architectures that beat far larger pretrained models on DNA tasks, with the best foundation backbone reaching about 82-85% accuracy on transcription-factor and promoter tasks while DNABERT-2 and VQDNA sit in the 66-78% range. The same search on proteins shows a split: BioArc architectures win on sequence-level tasks (solubility, human PPI) but lose on structure-heavy tasks (fold classification, PPI affinity), attributed to limited pretraining data and missing rotary positional embeddings. The paper further c","pith_inferences":["If the ranking-split issue is resolved cleanly, the strongest consequence is that the same Z-score ranking protocol could be applied to other data modalities, such as RNA and single-cell, which the paper names but does not demonstrate.","The Hyena-Transformer-CNN recipe may be read as an inductive-bias claim about biological sequences: long-range dependencies first, contextual syntax second, local motif extraction last. This is a testable design principle for future models even outside the NAS loop.","A practical extension implied by the predictor results: a small multi-agent LLM system could cut the cost of NAS for a new biological task by retrieving good architectures from a library of past searches, but the reported hit rate of about 0.5 at @5 suggests the method still needs substantial improvement before it can replace search.","Cross-modal transfer is the open frontier: the paper suggests the framework generalizes beyond DNA and protein, but its own results show that optimal architectures differ by modality, so a universal biological architecture is unlikely."],"forward_implications":["If the BioArc-F result generalizes, future biological foundation models can be built at 1/20 the parameter count and 1/10 the training steps while matching or beating current large pretrained models on DNA benchmarks.","The reported pattern implies that hybrid topologies, not any single module class, are the right template for DNA sequence modeling: start with long-range context, then attention, then convolutional feature extraction.","The tokenizer finding implies that architecture and tokenization must be selected together; a tokenizer that is optimal for a Transformer is not optimal for a CNN or Mamba.","The training-strategy finding implies that large-scale pretraining is not automatically beneficial for biological tasks; from-scratch training wins on many tasks, so pretraining decisions need task-specific justification.","The architecture-prediction result implies that once an (architecture, task, performance) database exists, optimal architectures for new but similar biological tasks can be retrieved rather than searched from scratch."],"fun_headline_variants":["BioArc: Auto-searched DNA models beat hand-built at 1/20 size","NAS finds compact DNA models that outperform larger pretrained ones","Small searched architectures beat big hand-tuned DNA models","BioArc: Discovered architectures win on DNA, split on proteins"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the architecture ranking used to select the top models was computed on a validation split that never touched the test labels reported in Tables 2 and 3; the paper does not state this split, and if test information leaked into selection the headline comparisons would be selection artifacts rather than intrinsic architecture advantages.","fun_headline_variants_meta":{"raw":{"variants":["BioArc: Auto-searched DNA models beat hand-built at 1/20 size","NAS finds compact DNA models that outperform larger pretrained ones","Small searched architectures beat big hand-tuned DNA models","BioArc: Discovered architectures win on DNA, split on proteins"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000477,"raw_usage":{"total_tokens":2216,"prompt_tokens":773,"completion_tokens":1443,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":517,"completion_tokens_details":{"reasoning_tokens":1369}},"tokens_in":517,"tokens_out":1443,"duration_ms":9824,"temperature":1.0,"reasoning_tokens":1369,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T19:27:27.494033+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Check whether the GUE and PEER test-set labels were accessible during the Eq. (8) ranking step. If they were, rerun the full search on a strictly disjoint validation split and measure whether the top-ranked BioArc architectures still beat DNABERT-2, VQDNA, and ESM-2 at 25x fewer parameters. A simpler proxy: compare BioArc's from-scratch accuracy on CPD-notata with a random architecture of the same size; if random topologies match it, the search is not the cause.","supporting_citations":[],"review_version":1}