{"id":"d4ca82e3-45c8-4749-9da2-6ff0b0404ab7","arxiv_id":"2505.09087","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A head-to-head zero-shot benchmark of 13 RNA language models finds that strong secondary-structure prediction often comes at the cost of poor functional classification performance.","lead":"This paper compares 13 RNA language models, plus DNA and protein controls, using the same two zero-shot tests: RNA secondary structure prediction from attention maps and RNA family classification from embedding similarities. The main finding is a trade-off: models that do well on structure often do poorly on classification, and only one model (AIDO.RNA) is strong on both.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The functional half of the central trade-off rests on an unvalidated overlap-ratio metric; a standard classification benchmark is needed before the trade-off claim can be accepted.","rationale":"The reader's weakest assumption matches mine: the overlap ratio is unvalidated and load-bearing. My review confirms that this is the single most important soft spot. The central claim would be true if OR faithfully measured functional classification, but the paper provides no evidence for that. Internal evidence supports the concern: the Methods introduce OR as a new statistic, the classification experiments compare homologous versus non-homologous pairs rather than functional classes, no external validation is reported, and the functional ranking is summarized without any statistical test. The table also contains patterns consistent with OR not tracking function: RNA-FM, an older and smaller model, is best on OR, while high-SS models such as RNA-km and MP-RNA are among the worst on OR; AIDO.RNA 1.6B is high on both. This could reflect a real trade-off or an artifact of the metric. The concrete test is feasible with the released data and would settle the point. Therefore I keep the reader's CONDITIONAL verdict: the benchmark itself is useful, but the central trade-off claim must be supported by an established classification metric or softened.","tokens_in":13785,"tokens_out":3891,"duration_ms":42621,"concrete_test":"Using the deposited code and data, recompute Table 4's functional column with a standard zero-shot pair-classification metric: for each model, on the same sampled Rfam and ArchiveII pairs, compute the AUC and average precision for separating homologous from non-homologous pairs from the same cosine similarities (or, if functional labels are desired, evaluate a nearest-centroid or linear-probe RNA-family/type classifier trained on the FFT embedding). Then compare model rankings and compute Spearman correlation between SS F1 and OR versus between SS F1 and AUC. If the OR ranking does not match the AUC ranking, or the SS-versus-AUC correlation is not negative, the claimed trade-off is an artifact of the OR metric.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central conclusion—that RNA LMs trade off secondary-structure prediction against function classification—is built entirely on comparing zero-shot SS F1 with the newly introduced overlap ratio (OR). OR is defined in Methods ('Calculation of the overlap ratio') as the intersection over the concatenation of the cosine-similarity distributions of homologous and non-homologous sequence pairs. This is not a validated functional-classification metric. First, it measures distributional overlap of pairwise embedding similarities, not accuracy of any functional category assignment; no baseline classifier, no AUC, and no external benchmark against known RNA functional classes is provided. Second, the two tasks labeled 'functional classification' are really homology detection (Rfam families) and RNA-type discrimination (ArchiveII), both of which can be driven by sequence identity and pretraining-corpus overlap rather than by functional knowledge. Third, Table 4 reports only OR point estimates with effectively zero variance, so the trade-off is asserted from visual rank comparisons; no correlation or significance test between SS F1 and OR is given, and RNA-MSM and ESM2 are absent from the OR columns. If OR does not track genuine functional classification, the central trade-off and the recommendation to rebalance pretraining objectives lose their empirical foundation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper benchmarks 13 RNA language models, three DNA LMs, and one protein LM under a zero-shot protocol. For secondary structure, it extracts attention maps from each model and computes F1 against experimental structures using threshold/head positions chosen on a small validation set. For functional classification, it introduces a new metric, the overlap ratio (OR), which measures the overlap between the cosine-similarity distributions of homologous and non-homologous sequence pairs sampled from Rfam and ArchiveII. The authors report that models strong on secondary structure often have poorer OR values, that AIDO.RNA (1.6B) is the only model with high performance on both tasks, and they conclude that current pretraining objectives are unbalanced.","tokens_in":13997,"tokens_out":3266,"duration_ms":35106,"significance":"If the central claim were supported, this would be a useful comparative resource for the RNA language-model community and would motivate rebalanced pretraining objectives. The paper's strengths include the breadth of models compared, the explicit zero-shot protocol before fine-tuning, the inclusion of DNA and protein controls, the documentation of rRNA/tRNA imbalance in common training and benchmark datasets, and the promise of open data and code on Zenodo. However, the central trade-off conclusion rests entirely on the newly introduced OR metric, which is not externally validated as a measure of functional classification. Because the functional half of the benchmark is load-bearing, the reported trade-off and the recommendation to rebalance pretraining are not yet established.","major_comments":[{"comment":"The OR metric is a distributional overlap between pairwise embedding similarities, not a measure of classification accuracy. It is introduced in this paper without validating it against any standard functional-classification benchmark, a baseline classifier, an AUC value, or a known functional category annotation. Since the central trade-off claim is built entirely on this metric, the authors should validate OR against a genuine supervised classification task (e.g., held-out Rfam family classification or RNA-type classification) and against simple baselines such as sequence identity or k-mer frequency. They should also report the variance of OR across models and runs; the Table 4 entries of 0.000 or 0.001 give the impression of zero variability and do not support the rank comparisons made in the text.","section":"Methods, 'Calculation of the overlap ratio (OR)'; Table 4"},{"comment":"The claim that 'models doing well on secondary structure prediction often perform worse in function classification or vice versa' is supported only by selected examples and visual rank comparison. No correlation coefficient, significance test, or confidence interval is reported between the SS F1 values and OR values across models. The comparison is also incomplete because RNA-MSM and ESM2 are absent from the OR columns and several models are absent from the SS column. The authors should compute a quantitative correlation (with uncertainty), include all models that can be evaluated, and address obvious confounders such as model size and pretraining data composition before asserting a trade-off.","section":"Results, 'Zero-shot model evaluation: RNA classifications'; Table 4"},{"comment":"The text states that Rfam families are easier to separate than ArchiveII families because sequence identity within Rfam families averages 77.1% versus 54.5% in ArchiveII. This admits that the OR values are likely driven by sequence identity and pretraining-corpus overlap rather than by functional knowledge. The paper should control for sequence identity, for example by stratifying homologous pairs into identity bins or by comparing OR with an alignment-free identity baseline. Without such a control, the interpretation of OR as a measure of functional classification ability is not justified.","section":"Results, 'Zero-shot model evaluation: RNA classifications'"},{"comment":"The paper criticizes existing benchmarks for not performing redundancy removal, but its own zero-shot classification test uses Rfam and ArchiveII sequences that may overlap substantially with the RNAcentral or other corpora on which most of the evaluated models were pretrained. This potential leakage could inflate the apparent discrimination ability of the models. The authors should assess leakage directly, for example by removing training-set sequences that are nearly identical to test sequences, or by reporting how OR changes under different redundancy thresholds.","section":"Introduction, discussion of data redundancy; Methods, dataset preparation"}],"minor_comments":[{"comment":"The sentence 'Results shows that...' should be 'Results show that...'.","section":"Abstract"},{"comment":"The column header 'Art.' is unexplained; the authors should either define it or remove it.","section":"Table 1"},{"comment":"The reference list contains duplicate numbering (two entries labeled 32); all subsequent reference numbers need renumbering.","section":"References"},{"comment":"The model name is misspelled as 'DNlABERT-2'; it should be 'DNABERT-2'.","section":"Code Availability"},{"comment":"The meaning of the asterisk entries should be explained in more detail; for example, readers may want to know why DNABERT-2 and NT have no SS F1 values and why RNA-MSM and ESM2 have no OR values.","section":"Table 4"},{"comment":"The axes and legend labels in the cosine-similarity distribution figures are not described in the caption; adding explicit axis titles and a legend for the homologous and non-homologous distributions would improve readability.","section":"Extended Data Figs. 1 and 2"}],"recommendation":"major_revision","confidential_remarks":"The main risk to this manuscript is the unvalidated OR metric, which underpins the central trade-off claim. I recommend asking the authors to supply the validation experiments and quantitative correlation analysis described in the major comments before acceptance. The heavy reliance on the authors' own RNA-MSM protocol is understandable given the continuity of the work, but the novelty of OR should be positioned more carefully. The manuscript fits the scope of the journal and, with the requested revisions, could become a useful comparative resource."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is the broadest zero-shot comparison of RNA language models I've seen, and the data/code are promised on Zenodo. But the central trade-off claim in the abstract is softer than the text supports: the functional half rests on an unvalidated overlap ratio, and there is no correlation or significance test between structure F1 and that ratio. Read it as a benchmark with an interesting hypothesis, not a proven trade-off.\n\nThe genuinely new piece is the side-by-side evaluation: 13 RNA LMs plus DNA and protein controls, scored on the same two zero-shot tasks using a consistent attention-map protocol for structure and a consistent embedding-similarity protocol for classification. That is useful. The authors also document the rRNA/tRNA imbalance in RNAcentral and downstream datasets, which is a real issue for anyone training or evaluating these models. And the specific observation that structure-oriented models like RNA-km and MP-RNA do poorly on the classification proxy is worth taking seriously. The resource itself—a single table with all models on the same tasks—is valuable to the field.\n\nThe soft spots are where the headline conclusion lives. The OR metric is defined as the overlap between cosine-similarity distributions of homologous and non-homologous pairs. That is a reasonable proxy for how separated the embeddings are, but it is not validated against any actual RNA functional classification benchmark. There is no AUC, no classifier, no comparison with established functional labels. So when the paper says \"models doing well on secondary structure often perform worse in function classification,\" the function classification is really distribution separation on Rfam/ArchiveII. That can be informative, but it is not the same as functional classification accuracy. Second, the trade-off is asserted from selected examples rather than a quantitative test. If you rank the models that have both values, the relationship is far from clean—AIDO.RNA (1.6B) is near the top on both structure and OR, which the text acknowledges but which weakens the claim of a general trade-off. Third, RNA-MSM and ESM2 are missing from the classification columns without explanation; they are among the most important references, so their absence leaves the comparison incomplete. The self-citation of the RNA-MSM protocol is not a problem, but the missing rows are a gap.\n\nWho is this for? Anyone building or evaluating RNA LMs. It does not resolve the structure-function question, but it sets a common scaffold for future comparisons. I would send it to serious peer review, and I would recommend acceptance after revision—provided the OR metric is validated against a standard classification benchmark, or the trade-off conclusion is softened to \"some structure-oriented models show weak embedding separation,\" with the correlation analysis made explicit.","headline":"A useful benchmark of RNA language models with a plausible but under-supported trade-off claim; the OR metric needs validation before the headline conclusion is accepted.","tokens_in":14514,"tokens_out":3105,"would_cite":true,"duration_ms":30987,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RNA language models face a structure-versus-function trade-off.","keywords":["RNA language models","zero-shot prediction","secondary structure","RNA classification","overlap ratio","embedding similarity","pretraining objectives","noncoding RNA"],"falsifier":"Take the same set of RNA language models and evaluate them on a supervised RNA family classification task, such as training a linear classifier on their sequence embeddings for Rfam families, then compare ranks with OR ranks. If models with low OR do not systematically achieve higher supervised classification accuracy, or if a model with high OR clearly outperforms one with low OR, the central trade-off claim would be falsified. A simpler check: recompute OR using mean-pooled embeddings instead of FFT-compressed vectors and see whether the pattern of model rankings persists.","tokens_in":13568,"feed_emoji":"🧬","tokens_out":2502,"duration_ms":24958,"temperature":0.7,"pith_summary":"This paper compares 13 RNA language models plus DNA and protein controls under a common zero-shot evaluation, testing each model's ability to predict RNA secondary structure from attention maps and to distinguish homologous from non-homologous RNA sequences using learned embeddings. The central finding is that most models excel at one task at the expense of the other: models with strong secondary-structure prediction, such as RNA-km and MP-RNA, fare poorly at functional classification, while models with strong functional classification, such as RNA-FM, lag in structure. Only AIDO.RNA achieves high performance on both axes. The authors conclude that current pretraining objectives are unbalanced and that RNA language models need training schemes that improve structure prediction without degrading functional representation.","feed_headline":"RNA models trade structure for function","feed_subtitle":"A 13-model zero-shot test finds most RNA language models excel at one task but fail the other; only AIDO.RNA does both.","key_machinery":"Two zero-shot probes carry the comparison. For structure, the paper extracts per-head attention maps from each transformer, symmetrizes and applies APC correction, then thresholds the resulting probabilities into base-pair predictions, selecting the optimal head-layer and threshold on a 40-RNA validation set and scoring on a 70-RNA test set. For function, the paper computes the overlap ratio (OR), defined as the intersection area of the cosine-similarity distributions of homologous versus non-homologous sequence pairs divided by the area of their union; OR ranges from 0 (perfectly separated distributions) to 1 (fully overlapping), so a smaller OR means better functional discrimination. The FFT-compressed sequence embeddings used for cosine similarity are the key methodological choice that makes OR computable for all models regardless of embedding dimension.","core_discovery":"Across a standardized zero-shot comparison, RNA language models show an inverse relationship between secondary-structure prediction quality and embedding-based functional classification quality. Secondary-structure F1 was measured by converting attention maps from each model into base-pair probabilities and scoring against experimentally determined RNA structures, while functional classification was measured by a newly introduced overlap ratio (OR) between cosine-similarity distributions of homologous and non-homologous sequence pairs. Models that rank highest in secondary-structure F1, including RNA-km, MP-RNA, and RNA-MSM, generally show worse OR values, whereas models like RNA-FM and ERNIE-RNA with low OR values show weaker structure prediction. AIDO.RNA (1.6B) is the only model with both strong secondary-structure F1 and low OR, although increasing its size from 650M to 1.6B improved structure but worsened classification. The paper interprets this as evidence that current unsupervised pretraining produces representations that are skewed toward either structural or functional signal, and argues for rebalanced training objectives.","pith_inferences":["The OR metric is introduced here without external validation against supervised functional classification benchmarks; if OR does not track true functional relatedness, the claimed trade-off could be an artifact of embedding geometry rather than a real limitation of RNA language models.","A direct test would be to compare OR rankings with fine-tuned classification accuracy on the same Rfam and ArchiveII splits; a strong negative correlation between OR and supervised accuracy would confirm the metric, while disagreement would weaken the paper's central conclusion.","The FFT compression to 128 dimensions may discard information differently across models of different embedding sizes, potentially biasing OR comparisons; using mean-pooled embeddings or a fixed projection trained on one model could serve as a sensitivity check.","If the trade-off is confirmed, a practical next step is multi-task pretraining that jointly optimizes masked language modeling and a secondary-structure auxiliary objective, then measuring whether both OR and F1 improve together."],"forward_implications":["If the trade-off is real, future RNA language models should be evaluated on both structure and function simultaneously, using a shared protocol like this one, rather than on whichever downstream task each model was tuned to.","Pretraining objectives need rebalancing: strategies that inject structural bias, such as k-mer masking in RNA-km or secondary-structure calibration in MP-RNA, appear to degrade the model's ability to separate RNA families in embedding space.","Scaling model size alone is not a reliable path to balanced performance, since the larger AIDO.RNA improved secondary structure but worsened functional classification relative to the smaller variant.","MSA-based models such as RNA-MSM achieve the best secondary-structure F1 despite being trained on far fewer sequences, suggesting that evolutionary information is a strong complementary signal for structure that single-sequence pretraining does not capture.","Unified RNA-DNA-protein models, represented by LucaOne, fall in the mid-range for both tasks, indicating that a universal biological language model may sacrifice RNA-specific competence."],"supporting_citations":[{"why":"RNA-FM is the best-performing model on the Rfam OR benchmark, establishing the functional-classification side of the trade-off.","marker":"[17]"},{"why":"RNA-MSM provides the top secondary-structure F1 and supplies the VL1/TS dataset protocol used for the zero-shot structure evaluation.","marker":"[20]"},{"why":"AIDO.RNA is the only model with high performance on both axes, and its two sizes demonstrate the scaling-versus-balance effect.","marker":"[33]"},{"why":"RNA-km's k-mer masking example shows that structure-enhancing pretraining can degrade functional classification.","marker":"[26]"},{"why":"MP-RNA's secondary-structure-calibrated pretraining similarly pairs high F1 with poor OR, supporting the trade-off pattern.","marker":"[30]"},{"why":"ERNIE-RNA's structure-enhanced pairing scores produce strong OR but moderate secondary-structure F1, another instance of the trade-off.","marker":"[19]"},{"why":"Rfam is the source of the homologous and non-homologous sequence pairs used to compute OR.","marker":"[32]"},{"why":"ArchiveII serves as the second, more difficult classification dataset and as a standard benchmark for secondary structure.","marker":"[43]"}],"fun_headline_variants":["RNA models split: structure vs function","Only AIDO.RNA does both structure and function","Structure or function: RNA models can't have both","AIDO.RNA beats the RNA structure-function trade-off"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The overlap ratio computed from FFT-compressed embedding cosine similarities is a valid zero-shot measure of an RNA language model's functional classification ability, even though this metric has not been validated against supervised functional benchmarks.","fun_headline_variants_meta":{"raw":{"variants":["RNA models split: structure vs function","Only AIDO.RNA does both structure and function","Structure or function: RNA models can't have both","AIDO.RNA beats the RNA structure-function trade-off"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001265,"raw_usage":{"total_tokens":5139,"prompt_tokens":868,"completion_tokens":4271,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":484,"completion_tokens_details":{"reasoning_tokens":4208}},"tokens_in":484,"tokens_out":4271,"duration_ms":27302,"temperature":1.0,"reasoning_tokens":4208,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:40:16.223242+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same set of RNA language models and evaluate them on a supervised RNA family classification task, such as training a linear classifier on their sequence embeddings for Rfam families, then compare ranks with OR ranks. If models with low OR do not systematically achieve higher supervised classification accuracy, or if a model with high OR clearly outperforms one with low OR, the central trade-off claim would be falsified. A simpler check: recompute OR using mean-pooled embeddings instead of FFT-compressed vectors and see whether the pattern of model rankings persists.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"RNA-FM is the best-performing model on the Rfam OR benchmark, establishing the functional-classification side of the trade-off."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"RNA-MSM provides the top secondary-structure F1 and supplies the VL1/TS dataset protocol used for the zero-shot structure evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"AIDO.RNA is the only model with high performance on both axes, and its two sizes demonstrate the scaling-versus-balance effect."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"RNA-km's k-mer masking example shows that structure-enhancing pretraining can degrade functional classification."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MP-RNA's secondary-structure-calibrated pretraining similarly pairs high F1 with poor OR, supporting the trade-off pattern."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ERNIE-RNA's structure-enhanced pairing scores produce strong OR but moderate secondary-structure F1, another instance of the trade-off."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Rfam is the source of the homologous and non-homologous sequence pairs used to compute OR."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ArchiveII serves as the second, more difficult classification dataset and as a standard benchmark for secondary structure."}],"review_version":1}