{"id":"2e0ed761-cac3-4530-905f-e60982b9d309","arxiv_id":"2412.10347","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"COMET assembles 17 single, cross, and multi-omics tasks and finds that protein language models often transfer well to DNA and RNA tasks while true multi-omics tasks remain unsolved.","lead":"COMET is a new benchmark of 17 tasks that scores DNA, RNA, and protein language models on single, cross, and multi-omics problems. It is built to help researchers choose and design models that work across biological sequence modalities.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cross-omics transfer claims lack a same-architecture random-init baseline; fine-tuning a large transformer on a new task could explain the results without any omics-specific knowledge transfer.","rationale":"The reader's weakest assumption identifies the embedding-replacement adaptation as the fragile point, but my concern is sharper: the experiment as reported lacks a same-architecture randomly initialized control. Without that control, the observed cross-omics performance can be explained by the fine-tuning procedure itself rather than by knowledge stored in the pretrained encoder. This concern is load-bearing because the paper's stated contributions—cross-omics transfer insights and protein models being superior across omics—are derived from these experiments. However, the benchmark resource (the assembled tasks and datasets) remains valuable independent of these transfer claims, and the paper could be revised to add the control experiment, report error bars, and release code. The reader's CONDITIONAL verdict is therefore appropriate; adding this control experiment and the random-init baseline should be part of the conditions. My read does not change the verdict, only sharpens the justification for it.","tokens_in":24961,"tokens_out":4980,"duration_ms":47134,"concrete_test":"Train a randomly initialized transformer with the exact architecture of a representative model (e.g., ESM-2 or DNABERT2) on each cross-omics task (EA, APA, PRS, SSP, Ther, Cont) using the same embedding-replacement and full-fine-tuning protocol, hyperparameters, and data splits as described in the paper. Run at least three random seeds and report mean and standard deviation. If the random-init models achieve performance within a few points of the pretrained models on those tasks, the claim that pretrained encoders encode transferable omics knowledge is not supported; if they perform substantially worse, the transfer conclusion gains credibility.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central insight—that pretrained omics models can transfer across omics by replacing the vocabulary embedding—rests on Section 5.2 and Appendix A.1. The protocol replaces the token embedding matrix and then fully fine-tunes the model on the target task, meaning encoder weights are updated during training. The observed cross-omics performance could therefore reflect the transformer's capacity to learn the new task from the target training data given any reasonable initialization, rather than any pretraining-derived omics knowledge stored in the encoder. The paper does not include the decisive control: a randomly initialized transformer with the identical architecture (depth, width, attention heads) trained on the same tasks with the same fine-tuning protocol and hyperparameters. The naive supervised baselines (CNN, ResNet, LSTM) are far smaller and architecturally different, so they do not control for model capacity. Frozen-model results are also not an adequate control because freezing prevents task-specific adaptation, and the large drop in performance may simply show that fine-tuning is necessary. Without the random-init baseline, the claim that 'the omics knowledge learned during pre-training ... has a considerable proportion in the encoder' and the abstract's 'protein models demonstrating superior performance across various omics' are not established. This is load-bearing because the benchmark's novelty and the paper's headline insights depend on those cross-omics transfer conclusions; if they are artifacts of fine-tuning a large model, the distinctive contribution of the paper is substantially weakened.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents COMET, a benchmark of 17 tasks spanning DNA, RNA, protein, cross-molecule, and multi-molecule settings, and evaluates eight pretrained omics language models plus three simpler supervised baselines under both fully fine-tuned and frozen conditions. The authors report that models can transfer across omics by replacing the vocabulary embedding layer, that protein models often perform well on nucleotide tasks, and that multi-molecular tasks remain challenging. The manuscript also provides a detailed appendix on task pipelines, hyperparameters, and data preprocessing.","tokens_in":25192,"tokens_out":4990,"duration_ms":44622,"significance":"If the central results were fully supported, COMET would fill a real gap: no existing benchmark systematically compares models across DNA, RNA, protein, cross-molecule, and multi-molecule tasks under a common protocol. The task curation and model sweep are extensive, and the appendix documents many implementation details. However, because the cross-omics transfer claim lacks a same-architecture random-initialization control, because the GE input window is selected on test performance, and because the comparisons lack error bars and use an inconsistent tuning protocol (LoRA for LucaOne), the quantitative conclusions are not currently established. With those controls and protocol fixes, this could be a valuable community resource.","major_comments":[{"comment":"The cross-omics transfer conclusion is not supported without a random-initialization baseline. The protocol replaces the token embedding matrix with a randomly initialized one and then fully fine-tunes the model on the target task, so any encoder with sufficient capacity could learn the task from the training data. The frozen-model results do not control for this, because freezing prevents task-specific adaptation and the performance drop may only show that fine-tuning is necessary. A same-architecture transformer with randomly initialized weights, trained under the same protocol and hyperparameters, is required to attribute the observed performance to pretraining. This missing control undermines the headline claim that 'protein models demonstrate superior performance across various omics.'","section":"Section 5.2 and Appendix A.1"},{"comment":"The GE input window is selected on test performance. The authors state that they identify the sequence window at positions 7000-12999 as achieving optimal test performance; using the test set to select input features leaks test information into the model and inflates the GE R2 values in Table 3. The window should be chosen using only the training or validation set, or the analysis should report results for all candidate windows so readers can see the sensitivity.","section":"Appendix A.4.1"},{"comment":"LucaOne is fine-tuned with LoRA (rank 32) while all other pretrained models receive full fine-tuning. This is an inconsistent protocol: any performance difference between LucaOne and the other models in Tables 5 and 6 may be due to the tuning method rather than the multi-omics pretraining. The authors should either fine-tune LucaOne fully or match the parameter budget, or explicitly refrain from comparative claims about LucaOne's relative performance.","section":"Section 5.1 and Appendix A.7"},{"comment":"No error bars or significance tests are reported. Many conclusions depend on small differences (e.g., ESM-2 at 68.75 vs RNA-FM at 68.50 on SSP in Table 3; several MCC differences near 1%). Without multiple seeds and variance estimates, these rankings cannot be distinguished from noise. At minimum, three repeated runs with mean and standard deviation should be reported, and claims of 'comparable' or 'superior' should be based on statistical tests.","section":"Tables 3-6 and Section 5"},{"comment":"The comparisons with literature SOTA are not protocol-matched. The SOTA numbers are taken from their original papers, which use different data splits, model architectures, and sometimes additional inputs (e.g., MSA or structure information). The statement that multi-molecular tasks 'still present significant challenges' relies on these comparisons. The authors should either re-run the SOTA methods under the COMET protocol or narrow the claims to comparisons among the methods evaluated in this paper.","section":"Tables 3, 5, and 6"}],"minor_comments":[{"comment":"In the APA isoform prediction section, the text says 'IPA isoform analysis' but should say 'APA isoform analysis' to match the task name.","section":"Appendix A.4.3"},{"comment":"The task names 'NCProteinTher' and 'SAIS/TIANCHIA' appear to contain typos; the former should likely be 'ProteinTher' or 'Thermostability' and the latter 'SAIS/Tianchi'.","section":"Table 1"},{"comment":"The header 'learing rate' is a typo and should be 'learning rate'.","section":"Table 16"},{"comment":"The headings use 'CROSS-MOLECULER' and 'MULTI-MOLECULER'; these should be 'CROSS-MOLECULAR' and 'MULTI-MOLECULAR'.","section":"Appendix A.4.7 and A.5"},{"comment":"The Range-MAE formula defines the average absolute error but does not explicitly state in the formula that m is the number of samples with predicted values within [0, 30]; consider making this condition explicit in the equation notation.","section":"Appendix A.5.1"},{"comment":"No code repository or public release of evaluation scripts is mentioned; for a benchmark intended for community adoption, providing code to reproduce the reported numbers would significantly improve usability.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about the missing random-initialization baseline is valid and is the most serious issue; it directly affects the paper's central cross-omics claim. The GE window selection on test performance is a clear but fixable leakage problem. In my view the manuscript is a useful benchmark contribution that can be repaired within the scope of a revision, so I recommend major rather than minor revision. I also note the self-citation of BEACON is normal and does not itself raise any concern."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis is a genuinely useful benchmark paper, with one caveat about what you can conclude from its headline. COMET takes a bunch of existing single-omics tasks (BEND, BEACON, PEER, FLIP, TAPE), adds cross- and multi-molecule tasks, and runs eight foundation models through a single protocol. That unified evaluation is the real contribution: the field has been comparing numbers across inconsistent settings, and having one testbed with a common task taxonomy is worth having even if every individual dataset comes from somewhere else. The citation practice is honest – the cross-omics transfer observations were already reported by Boshar et al., Prakash et al., and Outeiral & Deane, and the paper cites all three.\n\nWhat it does well: the task coverage is broad (17 tasks, DNA/RNA/protein, residue-level and sequence-level), the model set includes the usual single-omics suspects plus LucaOne and CaLM, and the appendix documents data preprocessing and hyperparameters in enough detail to replicate most of it. The design of rotating models across omics by replacing the vocab embedding is a sensible way to make cross-omics comparison possible.\n\nWhere the soft spots are, in order of seriousness. First, the GE input window is chosen by test performance (A.4.1): the authors explicitly report that the 7000–12999 bp window was selected because it achieves optimal test performance. That is a test set leak, and it means the GE numbers for every model are computed on a window the authors knew worked well on the test set. Second, no error bars anywhere. With single runs, point differences of a few percent are not meaningful. Third, LucaOne is LoRA-tuned, everyone else fully fine-tuned – an inconsistent protocol that makes the \"LucaOne is competitive\" statements hard to interpret. Fourth, the cross-omics transfer claim. The paper says replacing the vocab embedding shows the encoder holds \"a considerable proportion\" of omics knowledge. But there is no same-architecture random-init baseline. Fine-tuning a 650M-parameter transformer on a new task can produce good performance from capacity alone; the CNN/ResNet/LSTM baselines are much smaller and don't control for that. The frozen results don't answer it either, since freezing prevents adaptation. This doesn't invalidate the benchmark, but it does mean the mechanistic interpretation in Section 5.2 is not established.\n\nWho is this for: anyone choosing a foundation model for a regulatory or structure-prediction task, or building a multi-omics benchmark that wants a reference testbed. It deserves a serious referee, but the revision needs the random-init baseline, error bars, a fix for the GE window selection, and a consistent fine-tuning protocol before the quantitative findings can be taken as reference numbers.\n\nRecommendation: send it out, with the above as required revisions. The benchmark itself is worth having.","headline":"Solid benchmark assembly with a real but fixable flaw: the cross-omics transfer claims lack a same-architecture random-init baseline, and the GE window is chosen on the test set.","tokens_in":25797,"tokens_out":2668,"would_cite":true,"duration_ms":25250,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces COMET, the first comprehensive benchmark for multi-omics biological language models, and reports that protein models transfer best across DNA, RNA, and protein tasks.","keywords":["multi-omics benchmark","biological language models","DNA language models","RNA language models","protein language models","cross-omics transfer","central dogma","embedding replacement"],"falsifier":"A direct test would be to take the same cross-omics setup and replace the encoder weights with randomly initialized weights while keeping the reinitialized vocabulary embedding and classification head, then fine-tune on the target task. If the random-encoder model matches the transferred-pretrained model's performance, the apparent cross-omics knowledge is not coming from the pretrained encoder. A complementary check would be to train the model on target-omics sequences with the vocabulary embedding reinitialized but the encoder frozen, and compare against a model whose vocabulary embedding is learned from the target data from scratch.","tokens_in":24771,"feed_emoji":"🧬","tokens_out":7584,"duration_ms":61002,"temperature":0.7,"pith_summary":"COMET is a benchmark that evaluates biological language models across 17 tasks spanning DNA, RNA, protein, cross-molecule, and multi-molecule prediction. The paper's central claim is that this is the first comprehensive multi-omics benchmark, and that its results reveal systematic cross-omics transfer: models pretrained on one molecule type can handle tasks on another when their vocabulary embeddings are reinitialized. The headline finding is that protein-pretrained models such as ESM-2 perform strongly, sometimes best, on DNA and RNA regulatory tasks, while DNA models can approach protein-model performance on protein tasks. Multi-molecular tasks, such as predicting antibody-antigen neutralizability or RNA-protein interactions, remain difficult for all current models, including the dedicated multi-omics model LucaOne. If these conclusions hold, they imply that omics-general knowledge is stored in the encoders of pretrained models, not only in their token embeddings.","feed_headline":"Protein models lead across DNA, RNA, and protein tasks","feed_subtitle":"A 17-task benchmark finds cross-omics transfer works, but multi-molecule tasks still stump today's models.","key_machinery":"The load-bearing machinery is COMET itself: a curated collection of 17 tasks with fixed splits and metrics, organized into single-molecule, cross-molecule, homo-omics multi-molecule, and hetero-omics multi-molecule categories. The key adaptation protocol is vocabulary-embedding reinitialization: when a model pretrained on one omics is applied to another, its embedding layer is replaced with a randomly initialized embedding for the target vocabulary while the encoder weights are kept, and the whole model is then fine-tuned. This protocol is what makes cross-omics comparisons possible and is what the paper's transfer conclusions rest on.","core_discovery":"The paper presents COMET as the first comprehensive biological multi-omics benchmark, containing 17 tasks organized into single-molecule (DNA, RNA, protein), cross-molecule (codon versus protein sequence), and multi-molecule (homo-omics and hetero-omics pairs) categories. Evaluation of two foundation models per omics, plus the multi-omics model LucaOne and the codon model CaLM, produces three main claims. First, replacing a model's vocabulary embedding layer while keeping its encoder weights allows DNA, RNA, and protein models to achieve comparable results on other omics' tasks, indicating that pretrained encoders store a substantial amount of omics-general knowledge. Second, protein models show superior performance across various omics, including DNA enhancer activity and RNA secondary structure prediction, and DNA models show potential on protein and RNA tasks. Third, multi-molecular tasks such as antibody-antigen neutralizability, RNA-protein interaction, and CRISPR off-target prediction remain significant challenges that neither single-omics combinations nor LucaOne fully solve.","pith_inferences":["A testable extension is to check whether cross-omics transfer survives shuffling or reverse-complementing the input sequences; if it does, the transfer may reflect low-level sequence statistics such as codon usage or GC content rather than deep biological understanding.","The benchmark's multi-omics scope is limited to sequence-level DNA, RNA, and protein tasks; extending it to methylation, chromatin accessibility, or single-cell data could reveal whether the transfer findings generalize beyond central-dogma sequences.","Because the compared models differ in size and pretraining data, the ranking that protein models are superior may be confounded by scale; a controlled comparison matching parameter count and data budget would isolate the omics effect.","The embedding-replacement protocol could serve as a cheap probe to map which encoder layers store omics-general versus omics-specific knowledge, guiding future multi-omics architecture design."],"forward_implications":["If protein models truly transfer to DNA and RNA regulatory tasks, then practitioners can start from protein-pretrained checkpoints for nucleotide tasks, potentially improving performance on regulatory genomics.","If DNA models can handle protein and RNA tasks, then a single nucleotide-pretrained backbone may serve multiple central-dogma tasks, reducing the need for separate domain-specific models.","The benchmark's standardized splits and metrics give the field a common reference for comparing future multi-omics models against each other.","Because multi-molecular tasks remain below specialized state-of-the-art methods, current multi-omics models likely need architectural innovations rather than simply more fine-tuning.","The finding that embedding replacement transfers knowledge suggests that cross-omics transfer can be evaluated systematically across the central dogma, not just within one molecule type."],"supporting_citations":[{"why":"Provides the cross-molecule task settings for Fluorescence and Beta-Lactamase prediction that COMET adopts.","marker":"(Boshar et al., 2024)"},{"why":"Introduces the CaLM codon-embedding model that serves as the cross-omics baseline on CDS tasks.","marker":"(Outeiral & Deane, 2024)"},{"why":"Introduces LucaOne, the multi-omics model that COMET evaluates as the main integrated baseline.","marker":"(He et al., 2024)"},{"why":"Provides ESM-1b, one of the protein foundation models that shows strong cross-omics transfer.","marker":"(Rives et al., 2021)"},{"why":"Provides ESM-2, the protein model that leads several DNA and RNA regulatory tasks in the benchmark.","marker":"(Lin et al., 2023)"},{"why":"Provides DNABERT2, one of the DNA foundation models used for DNA and cross-omics tasks.","marker":"(Zhou et al., 2023)"},{"why":"Provides the Nucleotide Transformer v2, the DNA model that transfers to protein thermostability.","marker":"(Dalla-Torre et al., 2023)"},{"why":"Provides BEACON-B and the RNA task pipeline that COMET extends to multi-omics evaluation.","marker":"(Ren et al., 2024)"},{"why":"Supplies the Xpresso gene-expression dataset and preprocessing that grounds the DNA gene-expression task.","marker":"(Agarwal & Shendure, 2020)"},{"why":"Supplies the DeepSTARR enhancer-activity dataset used for the DNA enhancer task.","marker":"(de Almeida et al., 2022)"}],"fun_headline_variants":["Protein models top DNA and RNA tasks in new benchmark","Cross-omics transfer works, but multi-molecule tasks stump models","COMET benchmark reveals protein models' edge across omics","New multi-omics benchmark finds protein models versatile"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The cross-omics transfer conclusions rest on the assumption that reinitializing the vocabulary embedding layer while keeping the encoder weights is a faithful way to adapt a model to a new omics, so that strong performance on another omics reflects genuine transferred knowledge rather than an artifact of the new embedding layer or the fine-tuning procedure.","fun_headline_variants_meta":{"raw":{"variants":["Protein models top DNA and RNA tasks in new benchmark","Cross-omics transfer works, but multi-molecule tasks stump models","COMET benchmark reveals protein models' edge across omics","New multi-omics benchmark finds protein models versatile"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000252,"raw_usage":{"total_tokens":1581,"prompt_tokens":983,"completion_tokens":598,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":599,"completion_tokens_details":{"reasoning_tokens":532}},"tokens_in":599,"tokens_out":598,"duration_ms":6012,"temperature":1.0,"reasoning_tokens":532,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:55:44.625336+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test would be to take the same cross-omics setup and replace the encoder weights with randomly initialized weights while keeping the reinitialized vocabulary embedding and classification head, then fine-tune on the target task. If the random-encoder model matches the transferred-pretrained model's performance, the apparent cross-omics knowledge is not coming from the pretrained encoder. A complementary check would be to train the model on target-omics sequences with the vocabulary embedding reinitialized but the encoder frozen, and compare against a model whose vocabulary embedding is learned from the target data from scratch.","supporting_citations":[{"cited_title":"Saprot: Protein language modeling with structure-aware vocabulary","cited_arxiv_id":null,"evidence_quote":"Provides DNABERT2, one of the DNA foundation models used for DNA and cross-omics tasks."},{"cited_title":"The nucleotide transformer: Building and evaluating robust foundation models for human genomics","cited_arxiv_id":null,"evidence_quote":"Provides the Nucleotide Transformer v2, the DNA model that transfers to protein thermostability."},{"cited_title":"Genomics-fm: Universal foundation model for versatile and data-efficient functional genomic analysis","cited_arxiv_id":null,"evidence_quote":"Provides BEACON-B and the RNA task pipeline that COMET extends to multi-omics evaluation."}],"review_version":1}