REVIEW 5 major objections 6 minor 22 references
COMET: Benchmark for Comprehensive Biological Multi-omics Evaluation Tasks and Language Models
T0 review · 5 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read The paper introduces COMET, the first comprehensive benchmark for multi-omics biological language models, and reports that protein models transfer best across DNA, RNA, and protein tasks.
desk verdict Solid benchmark assembly with a real but fixable flaw: the cross-omics transfer claims lack a same-architecture random-init baseline, and the GE window is chosen on the test set. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is COMET itself: a curated collection of 17 tasks with fixed splits and metrics, organized into single-molecule, cross-molecule, homo-omics multi-molecule, and hetero-omics multi-molecule categories. The key adaptation protocol is vocabulary-embedding reinitialization: when a model pretrained on one omics is applied to another, its embedding layer is replaced with a randomly initialized embedding for the target vocabulary while the encoder weights are kept, and the whole model is then fine-tuned. This protocol is what makes cross-omics comparisons possible and is what the paper's transfer conclusions rest on.
What would settle it
A direct test would be to take the same cross-omics setup and replace the encoder weights with randomly initialized weights while keeping the reinitialized vocabulary embedding and classification head, then fine-tune on the target task. If the random-encoder model matches the transferred-pretrained model's performance, the apparent cross-omics knowledge is not coming from the pretrained encoder. A complementary check would be to train the model on target-omics sequences with the vocabulary embedding reinitialized but the encoder frozen, and compare against a model whose vocabulary embedding is learned from the target data from scratch.
Extended reading notes
Core claim
The paper presents COMET as the first comprehensive biological multi-omics benchmark, containing 17 tasks organized into single-molecule (DNA, RNA, protein), cross-molecule (codon versus protein sequence), and multi-molecule (homo-omics and hetero-omics pairs) categories. Evaluation of two foundation models per omics, plus the multi-omics model LucaOne and the codon model CaLM, produces three main claims. First, replacing a model's vocabulary embedding layer while keeping its encoder weights allows DNA, RNA, and protein models to achieve comparable results on other omics' tasks, indicating that pretrained encoders store a substantial amount of omics-general knowledge. Second, protein models show superior performance across various omics, including DNA enhancer activity and RNA secondary structure prediction, and DNA models show potential on protein and RNA tasks. Third, multi-molecular tasks such as antibody-antigen neutralizability, RNA-protein interaction, and CRISPR off-target prediction remain significant challenges that neither single-omics combinations nor LucaOne fully solve.
Load-bearing premise
The cross-omics transfer conclusions rest on the assumption that reinitializing the vocabulary embedding layer while keeping the encoder weights is a faithful way to adapt a model to a new omics, so that strong performance on another omics reflects genuine transferred knowledge rather than an artifact of the new embedding layer or the fine-tuning procedure.
Editorial extensions
If this is right
- If protein models truly transfer to DNA and RNA regulatory tasks, then practitioners can start from protein-pretrained checkpoints for nucleotide tasks, potentially improving performance on regulatory genomics.
- If DNA models can handle protein and RNA tasks, then a single nucleotide-pretrained backbone may serve multiple central-dogma tasks, reducing the need for separate domain-specific models.
- The benchmark's standardized splits and metrics give the field a common reference for comparing future multi-omics models against each other.
- Because multi-molecular tasks remain below specialized state-of-the-art methods, current multi-omics models likely need architectural innovations rather than simply more fine-tuning.
- The finding that embedding replacement transfers knowledge suggests that cross-omics transfer can be evaluated systematically across the central dogma, not just within one molecule type.
Reading between the lines
- A testable extension is to check whether cross-omics transfer survives shuffling or reverse-complementing the input sequences; if it does, the transfer may reflect low-level sequence statistics such as codon usage or GC content rather than deep biological understanding.
- The benchmark's multi-omics scope is limited to sequence-level DNA, RNA, and protein tasks; extending it to methylation, chromatin accessibility, or single-cell data could reveal whether the transfer findings generalize beyond central-dogma sequences.
- Because the compared models differ in size and pretraining data, the ranking that protein models are superior may be confounded by scale; a controlled comparison matching parameter count and data budget would isolate the omics effect.
- The embedding-replacement protocol could serve as a cheap probe to map which encoder layers store omics-general versus omics-specific knowledge, guiding future multi-omics architecture design.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents COMET, a benchmark of 17 tasks spanning DNA, RNA, protein, cross-molecule, and multi-molecule settings, and evaluates eight pretrained omics language models plus three simpler supervised baselines under both fully fine-tuned and frozen conditions. The authors report that models can transfer across omics by replacing the vocabulary embedding layer, that protein models often perform well on nucleotide tasks, and that multi-molecular tasks remain challenging. The manuscript also provides a detailed appendix on task pipelines, hyperparameters, and data preprocessing.
Significance. If the central results were fully supported, COMET would fill a real gap: no existing benchmark systematically compares models across DNA, RNA, protein, cross-molecule, and multi-molecule tasks under a common protocol. The task curation and model sweep are extensive, and the appendix documents many implementation details. However, because the cross-omics transfer claim lacks a same-architecture random-initialization control, because the GE input window is selected on test performance, and because the comparisons lack error bars and use an inconsistent tuning protocol (LoRA for LucaOne), the quantitative conclusions are not currently established. With those controls and protocol fixes, this could be a valuable community resource.
major comments (5)
- [Section 5.2 and Appendix A.1] The cross-omics transfer conclusion is not supported without a random-initialization baseline. The protocol replaces the token embedding matrix with a randomly initialized one and then fully fine-tunes the model on the target task, so any encoder with sufficient capacity could learn the task from the training data. The frozen-model results do not control for this, because freezing prevents task-specific adaptation and the performance drop may only show that fine-tuning is necessary. A same-architecture transformer with randomly initialized weights, trained under the same protocol and hyperparameters, is required to attribute the observed performance to pretraining. This missing control undermines the headline claim that 'protein models demonstrate superior performance across various omics.'
- [Appendix A.4.1] The GE input window is selected on test performance. The authors state that they identify the sequence window at positions 7000-12999 as achieving optimal test performance; using the test set to select input features leaks test information into the model and inflates the GE R2 values in Table 3. The window should be chosen using only the training or validation set, or the analysis should report results for all candidate windows so readers can see the sensitivity.
- [Section 5.1 and Appendix A.7] LucaOne is fine-tuned with LoRA (rank 32) while all other pretrained models receive full fine-tuning. This is an inconsistent protocol: any performance difference between LucaOne and the other models in Tables 5 and 6 may be due to the tuning method rather than the multi-omics pretraining. The authors should either fine-tune LucaOne fully or match the parameter budget, or explicitly refrain from comparative claims about LucaOne's relative performance.
- [Tables 3-6 and Section 5] No error bars or significance tests are reported. Many conclusions depend on small differences (e.g., ESM-2 at 68.75 vs RNA-FM at 68.50 on SSP in Table 3; several MCC differences near 1%). Without multiple seeds and variance estimates, these rankings cannot be distinguished from noise. At minimum, three repeated runs with mean and standard deviation should be reported, and claims of 'comparable' or 'superior' should be based on statistical tests.
- [Tables 3, 5, and 6] The comparisons with literature SOTA are not protocol-matched. The SOTA numbers are taken from their original papers, which use different data splits, model architectures, and sometimes additional inputs (e.g., MSA or structure information). The statement that multi-molecular tasks 'still present significant challenges' relies on these comparisons. The authors should either re-run the SOTA methods under the COMET protocol or narrow the claims to comparisons among the methods evaluated in this paper.
minor comments (6)
- [Appendix A.4.3] In the APA isoform prediction section, the text says 'IPA isoform analysis' but should say 'APA isoform analysis' to match the task name.
- [Table 1] The task names 'NCProteinTher' and 'SAIS/TIANCHIA' appear to contain typos; the former should likely be 'ProteinTher' or 'Thermostability' and the latter 'SAIS/Tianchi'.
- [Table 16] The header 'learing rate' is a typo and should be 'learning rate'.
- [Appendix A.4.7 and A.5] The headings use 'CROSS-MOLECULER' and 'MULTI-MOLECULER'; these should be 'CROSS-MOLECULAR' and 'MULTI-MOLECULAR'.
- [Appendix A.5.1] The Range-MAE formula defines the average absolute error but does not explicitly state in the formula that m is the number of samples with predicted values within [0, 30]; consider making this condition explicit in the equation notation.
- [General] No code repository or public release of evaluation scripts is mentioned; for a benchmark intended for community adoption, providing code to reproduce the reported numbers would significantly improve usability.
Circularity Check
No circularity found: COMET is an empirical benchmark whose results are measured from external models on external datasets; cross-omics transfer claims are inferences from those measurements, not re-statements of inputs.
full rationale
COMET contains no derivation whose output is assumed in its input. The benchmark assembles 17 existing datasets (Xpresso, DeepSTARR, FLIP, DeepFRI, etc.) and evaluates externally pretrained models (DNABERT2, NTv2, RNA-FM, BEACON-B, ESM-1b/2, LucaOne, CaLM) under fixed fine-tuning protocols. The reported numbers in Tables 3-6 are direct measurements of those models on held-out splits, so there is no fitted parameter later renamed as a prediction. The claim that pretrained protein language models have strong potential for multi-molecule understanding and that the omics knowledge learned during pre-training has a considerable proportion in the encoder is an inductive interpretation of fine-tuning results, not a quantity obtained by construction; the same measurements could in principle support alternative explanations, such as task-specific adaptation during fine-tuning, but that is a validity concern rather than a circularity. Self-citations, including BEACON (Ren et al., 2024) and Genomics-FM (Ye et al., 2024), which share overlapping authors, are used to reference baseline models and prior benchmarks; no load-bearing argument depends on an unverified self-cited theorem. The first comprehensive multi-omics benchmark claim is a novelty assertion, not a derivation. The paper's own limitation note that the current benchmark involves relatively limited models and tasks is consistent with an empirical contribution and does not reveal a circular step.
Assumptions & free parameters
free parameters (2)
- GE input window position =
6000 bp fragment at positions 7000-12999
- Learning rates =
not reported per task; ranges [1e-6,5e-3] or [2e-5,1e-3]
assumptions (4)
- domain assumption Reinitializing vocabulary embeddings while keeping pretrained encoder weights is a valid cross-omics adaptation
- domain assumption Randomly paired ncRNA and protein sequences form valid negative samples for RPI
- domain assumption Truncating sequences to model max lengths preserves task-relevant signal
- domain assumption Literature SOTA results are comparable to this paper's runs
Cite this review
Pith. "Pith review of COMET: Benchmark for Comprehensive Biological Multi-omics Evaluation Tasks and Language Models." pith.science (2026). https://pith.science/paper/B6WNYEXO
@misc{pith2026241210347,
author = {Pith},
title = {Pith review of: COMET: Benchmark for Comprehensive Biological Multi-omics Evaluation Tasks and Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/B6WNYEXO}},
note = {Machine review of arXiv:2412.10347}
}
read the original abstract
As key elements within the central dogma, DNA, RNA, and proteins play crucial roles in maintaining life by guaranteeing accurate genetic expression and implementation. Although research on these molecules has profoundly impacted fields like medicine, agriculture, and industry, the diversity of machine learning approaches-from traditional statistical methods to deep learning models and large language models-poses challenges for researchers in choosing the most suitable models for specific tasks, especially for cross-omics and multi-omics tasks due to the lack of comprehensive benchmarks. To address this, we introduce the first comprehensive multi-omics benchmark COMET (Benchmark for Biological COmprehensive Multi-omics Evaluation Tasks and Language Models), designed to evaluate models across single-omics, cross-omics, and multi-omics tasks. First, we curate and develop a diverse collection of downstream tasks and datasets covering key structural and functional aspects in DNA, RNA, and proteins, including tasks that span multiple omics levels. Then, we evaluate existing foundational language models for DNA, RNA, and proteins, as well as the newly proposed multi-omics method, offering valuable insights into their performance in integrating and analyzing data from different biological modalities. This benchmark aims to define critical issues in multi-omics research and guide future directions, ultimately promoting advancements in understanding biological processes through integrated and different omics data analysis.
Figures
Reference graph
Works this paper leans on
-
[1]
Following (Angenent-Mari et al., 2020), a stringent quality control process is applied to eliminate artifacts and ensure data reliability. The quality control (QC) framework includes five levels: QC1, QC2, QC3, QC4 and QC5, where QC1 represents the lowest quality and QC5 the highest. Datasets above QC2 are utilized for training, while QC5 is reserved for ...
work page 2020
-
[6]
Simulating 500 million years of evolution with a language model
Tomas Hayes, Roshan Rao, Halil Akin, Nicholas J Sofroniew, Deniz Oktay, Zeming Lin, Robert Verkuil, Vincent Q Tran, Jonathan Deaton, Marius Wiggert, et al. Simulating 500 million years of evolution with a language model. bioRxiv, pp. 2024–07,
2024
-
[9]
Bridging biomolecular modalities for knowledge transfer in bio-language models
Mangal Prakash, Artem Moskalev, Peter A DiMaggio, Steven Combs, Tommaso Mansi, Justin Scheer, and Rui Liao. Bridging biomolecular modalities for knowledge transfer in bio-language models. bioRxiv, pp. 2024–10,
work page 2024
-
[10]
Beacon: Benchmark for comprehensive rna tasks and lan- guage models
Yuchen Ren, Zhiyuan Chen, Lifeng Qiao, Hongtai Jing, Yuchen Cai, Sheng Xu, Peng Ye, Xinzhu Ma, Siqi Sun, Hongliang Yan, et al. Beacon: Benchmark for comprehensive rna tasks and lan- guage models. arXiv preprint arXiv:2406.10391,
-
[11]
Rnabench: A comprehensive library for in silico rna modelling
Frederic Runge, Karim Farid, Jorg KH Franke, and Frank Hutter. Rnabench: A comprehensive library for in silico rna modelling. bioRxiv, pp. 2024–01,
work page 2024
-
[13]
Saprot: Protein language modeling with structure-aware vocabulary
Jin Su, Chenchen Han, Yuyang Zhou, Junjie Shan, Xibin Zhou, and Fajie Yuan. Saprot: Protein language modeling with structure-aware vocabulary. bioRxiv, pp. 2023–10,
work page 2023
-
[14]
Rnacentral 2021: secondary structure integration, improved sequence search and new member databases
Blake A Sweeney, Anton I Petrov, Carlos E Ribas, Robert D Finn, Alex Bateman, Maciej Szyman- ski, Wojciech M Karlowski, Stefan E Seemann, Jan Gorodkin, Jamie J Cannone, et al. Rnacentral 2021: secondary structure integration, improved sequence search and new member databases. Nucleic Acids Research, 49(D1),
work page 2021
-
[16]
Genomics-fm: Universal foundation model for versatile and data-efficient functional genomic analysis
Peng Ye, Weiqiang Bai, Yuchen Ren, Wenran Li, Lifeng Qiao, Chaoqi Liang, Linxiao Wang, Yuchen Cai, Jianle Sun, Zejun Yang, et al. Genomics-fm: Universal foundation model for versatile and data-efficient functional genomic analysis. bioRxiv, pp. 2024–07,
work page 2024
Show all 22 references
-
[17]
Dnabert- 2: Efficient foundation model and benchmark for multi-species genome
Zhihan Zhou, Yanrong Ji, Weijian Li, Pratik Dutta, Ramana Davuluri, and Han Liu. Dnabert- 2: Efficient foundation model and benchmark for multi-species genome. arXiv preprint arXiv:2306.15006,
-
[18]
Cd-gpt: A biological foundation model bridging the gap between molecular sequences through central dogma
Xiao Zhu, Chenchen Qin, Fang Wang, Fan Yang, Bing He, Yu Zhao, and Jianhua Yao. Cd-gpt: A biological foundation model bridging the gap between molecular sequences through central dogma. bioRxiv, pp. 2024–06,
2024
-
[19]
19 Table 14: Configuration settings for EPI, siRNA and AAN Config/Task EPI siRNA AAN optimizer AdamW AdamW AdamW optimizer epsilon 1.00E-08 1.00E-08 1.00E-08 optimizer momentum β1, β2 = 0.9, 0.999 β1, β2 = 0.9, 0.999 β1, β2 = 0.9, 0.999 weight decay 0.01 0.01 0.01 learning rat...
2020
-
[22]
We set the DNA-protein interface distance threshold to 5 ˚A
and map these residues or nucleotides to standard amino acids or DNA sequences using SCOP. We set the DNA-protein interface distance threshold to 5 ˚A. Based on this threshold, we derive the DNA-protein interface information. Subsequently, we match the 23 DNA and protein duple...
2020
-
[1970]
The nucleotide transformer: Building and evaluating robust foundation models for human genomics
Hugo Dalla-Torre, Liam Gonzalez, Javier Mendoza-Revilla, Nicolas Lopez Carranza, Adam Henryk Grzywaczewski, Francesco Oteri, Christian Dallago, Evan Trop, Bernardo P de Almeida, Hassan Sirelkhatim, et al. The nucleotide transformer: Building and evaluating robust foundation mo...
2023
-
[1997]
Overview of strategies for developing high thermostability industrial enzymes: Discovery, mechanism, modification and challenges
Hao Wu, Qiuming Chen, Wenli Zhang, and Wanmeng Mu. Overview of strategies for developing high thermostability industrial enzymes: Discovery, mechanism, modification and challenges. Critical Reviews in Food Science and Nutrition, 63(14):2057–2073,
-
[2012]
Interpretable rna foundation model from unannotated data for highly accurate rna structure and function predictions
Jiayang Chen, Zhihang Hu, Siqi Sun, Qingxiong Tan, Yixuan Wang, Qinze Yu, Licheng Zong, Liang Hong, Jin Xiao, Tao Shen, et al. Interpretable rna foundation model from unannotated data for highly accurate rna structure and function predictions. arXiv preprint arXiv:2204.00300,
-
[2018]
Update on activities at the universal protein resource (uniprot) in 2013.Nucleic acids research, 41(D1):D43–D47,
UniProt Consortium. Update on activities at the universal protein resource (uniprot) in 2013.Nucleic acids research, 41(D1):D43–D47,
2013
-
[2019]
These reads are grouped based on the randomized regions located upstream of the proximal polyadenylation site (pPAS), forming a dictionary of sequence variants for each library
to retain only high-quality, full-length RNA sequences. These reads are grouped based on the randomized regions located upstream of the proximal polyadenylation site (pPAS), forming a dictionary of sequence variants for each library. To expand this dictionary, sequencing is al...
2020
-
[2020]
Accessed: 2024-05-26. Karen S Sarkisyan, Dmitry A Bolotin, Margarita V Meer, Dinara R Usmanova, Alexander S Mishin, George V Sharonov, Dmitry N Ivankov, Nina G Bozhanova, Mikhail S Baranov, Onuralp Soyle- mez, et al. Local fitness landscape of the green fluorescent protein. Na...
2024
-
[2021]
Advancing dna language models: The genomics long-range benchmark
Chia Hsiang Kao, Evan Trop, McKinley Polen, Yair Schiff, Bernardo P de Almeida, Aaron Gokaslan, Thomas Pierrot, and V olodymyr Kuleshov. Advancing dna language models: The genomics long-range benchmark. In ICLR 2024 Workshop on Machine Learning for Genomics Explorations,
2024
-
[2022]
Hotprotein: A novel framework for protein thermostability prediction and editing
11 Tianlong Chen and Chengyue Gong. Hotprotein: A novel framework for protein thermostability prediction and editing. NeurIPS 2022,
2022
-
[2023]
Flip: Benchmark tasks in fitness landscape inference for proteins
Christian Dallago, Jody Mou, Kadina E Johnston, Bruce J Wittmann, Nicholas Bhattacharya, Samuel Goldman, Ali Madani, and Kevin K Yang. Flip: Benchmark tasks in fitness landscape inference for proteins. bioRxiv, pp. 2021–11,
2021
-
[2024]
Lucaone: Generalized biological foundation model with unified nucleic acid and protein language
Yong He, Pan Fang, Yongtao Shan, Yuanfei Pan, Yanhong Wei, Yichang Chen, Yihao Chen, Yi Liu, Zhenyu Zeng, Zhan Zhou, et al. Lucaone: Generalized biological foundation model with unified nucleic acid and protein language. bioRxiv, pp. 2024–05,
2024
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.