Pith. sign in

REVIEW 5 major objections 6 minor 18 references

AnnoDPO: Protein Functional Annotation Learning with Direct Preference Optimization

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Direct Preference Optimization, borrowed from LLM alignment, consistently beats supervised fine-tuning for protein function prediction on Swiss-Prot, with the largest gains on rare Gene Ontology terms.

desk verdict A promising but under-specified application of DPO to GO annotation: the loss is never defined over annotation sets, and the evaluation is too weak to support the SOTA claim. read the letter →

arxiv 2506.07035 v1 pith:3HMCMAW4 submitted 2025-06-08 q-bio.BM cs.AI

Zixuan Jiang , Renjing Xu This is my paper
classification q-bio.BMcs.AI
keywords proteinfunctionpredictiondirectpreferenceoptimizationGeneOntologyannotationcontrastivelearninglanguagemodelsSwiss-Protmulti-labelclassificationlong-taildistribution
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that Direct Preference Optimization (DPO), an alignment technique borrowed from large language models, can improve protein function annotation once supervised fine-tuning (SFT) has already trained a sequence-to-annotation model. On a 9:1 random split of Swiss-Prot, the DPO stage raises F1-Max by 2.7%, 4.1%, and 3.1% over SFT in the three Gene Ontology branches, and the gains are largest for low-frequency annotations. The authors argue that treating curated GO annotations as preferred outcomes and synthetic negatives as dispreferred outcomes teaches the model to rank correct functions above plausible decoys, which is exactly the kind of signal a sparse, imbalanced annotation space needs. If true, DPO offers a cheap, reward-model-free way to inject biological knowledge into protein language models for prediction tasks.

What carries the argument

The framework's engine is the DPO loss, a preference-optimization objective that replaces reward modeling with direct policy optimization on preferred versus non-preferred outputs: a ground-truth Swiss-Prot annotation is the preferred response $y_w$, and a synthetic negative (either mask-noised or model-predicted) is the dispreferred $y_l$. DPO replaces the usual reward-model-then-PPO pipeline with a single-stage objective weighted by $\beta$, with the SFT model acting as the reference policy $\pi_{\mathrm{ref}}$. Before DPO, an SFT stage combines a standard multi-label classification loss over 7,533 GO terms with a sequence-annotation contrastive (SAC) loss that pulls sequence embeddings and annotation embeddings of matching pairs together while pushing decoys apart. The annotation encoder is a de novo trained ProteinBERT-style transformer, and the sequence encoder is the 300M-parameter ESM-C pretrained on UniRef, MGnify, and JGI. Removing the SAC loss cuts F1-Max by 67.9%, so the contrastive alignment is the other load-bearing component of the design.

What would settle it

Train and evaluate the same SFT and DPO checkpoints on a temporal split of Swiss-Prot (sequences added after 2010) and on a homology-reduced test set; if DPO's relative F1-Max improvement over SFT disappears or reverses on either, the claim of consistent preference-alignment gains is refuted.

Watch

Extended reading notes

Core claim

The central claim is that a preference-aligned training stage, built on the DPO loss, consistently outperforms the SFT baseline for protein function prediction, with relative F1-Max improvements of 2.7% in Biological Process, 4.1% in Cellular Component, and 3.1% in Molecular Function. The paper further claims that DPO's advantage is concentrated in rare annotations (8.7% F1-Max gain over SFT in low-frequency GO terms) and that DPO-learned embeddings separate GO categories more cleanly while preserving fine-grained ontological relationships. The authors position this as the first adaptation of DPO to protein function annotation, and the first evidence that preference optimization, not just more supervised data, can address annotation sparsity and category imbalance in biological knowledge integration.

Load-bearing premise

The whole evaluation rests on the assumption that a random 9:1 split of the January 2010 Swiss-Prot release measures real generalization: if ESM-C's pretraining on UniRef, MGnify, and JGI, or the split itself, already leaks information about test sequences, the reported DPO gains may not survive a time-ordered or homology-aware benchmark.

Editorial extensions

If this is right

  • If the claim holds, the DPO stage is a cheap add-on: no reward model, no PPO, just preference pairs, so any SFT-trained protein annotation model can be upgraded with a stable single-stage loss.
  • Since gains are largest for low-frequency GO terms (8.7% over SFT), DPO is a practical lever for the long-tail annotation problem that dominates real Swiss-Prot.
  • The contrastive loss's centrality means cross-modal alignment between sequence space and annotation space is not optional; either component alone underperforms the full pipeline.
  • DPO features split BP/CC/MF categories more cleanly and keep related GO terms close, so downstream retrieval and evidence-based annotation tools could inherit those representations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The random 9:1 split likely inflates all reported numbers because Swiss-Prot entries share homology; on a strict temporal split the absolute F1 scores will drop, and whether the relative DPO advantage survives is an open question the paper does not test.
  • The same preference-alignment recipe could transfer to other sparse biological ontologies (e.g., EC numbers, UniProt keywords) or to structure-aware encoders, since DPO only needs a preferred/dispreferred pair per sequence.
  • Because the best negative set is the model's own predictions, the framework is self-improving: as annotations grow or the model improves, regenerating negatives could yield further gains without architectural change.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. AnnoDPO proposes a three-stage training pipeline for protein function prediction: an ESM-C pretrained encoder, a supervised fine-tuning (SFT) stage combining binary cross-entropy annotation prediction with a sequence-annotation contrastive loss, and a Direct Preference Optimization (DPO) stage that contrasts ground-truth GO annotations against synthetic negatives. The method is evaluated on a random 9:1 split of Swiss-Prot (Jan 2010) with F1-Max, AUPR, and AUROC metrics. The central reported result is that DPO improves over SFT by 2.7-4.1% F1-Max across the BP, CC, and MF subcategories, with the largest gains on low-frequency GO terms. The paper also includes ablation studies, t-SNE visualizations, and a discussion of ontological relationship learning.

Significance. If the claims were validated, the use of DPO for multi-label protein functional annotation would be a novel and potentially useful extension of preference optimization beyond generative language modeling. The paper provides code, and the internal comparison between SFT and DPO is a reasonable starting point. However, the significance is severely limited by an undefined policy distribution in the DPO objective, the absence of any comparison with existing GO predictors, and an evaluation protocol that does not control for homology to pretraining data. The central claim of consistent DPO improvement is not yet supported by the evidence as presented.

major comments (5)
  1. [Section 3, Eq. (3) (DPO Loss)] The DPO objective in Eq. (3) requires a policy πθ(y|x) that assigns probabilities to sets of GO terms, but no such distribution is defined anywhere in the paper. Eq. (1) only defines per-term marginal probabilities p_j via binary cross-entropy; it does not specify a joint distribution over the 7533-dimensional annotation vector. Unless the authors explicitly state, for example, that log πθ(y|x) is the log-product of independent Bernoulli probabilities, the DPO loss cannot be evaluated, the reference policy π_ref is undefined, and the improvements reported in Table 7 cannot be attributed to DPO. This is a load-bearing missing definition that must be added and justified.
  2. [Section 3 (Dataset Curation) and Section 4 (Experiments)] The evaluation uses a random 9:1 split of Swiss-Prot Jan 2010 with no homology-based or temporal splitting. Since the ESM-C encoder was pretrained on UniRef, MGnify, and JGI, the test set is likely to contain sequences homologous to training sequences, which can inflate all reported numbers. Moreover, the paper compares only zero-shot, SFT, and DPO variants; it does not compare against any existing GO prediction method such as DeepGO, DeepGraphGO, or ProteinBERT. The claim of "state-of-the-art performance" in the ablation section is therefore unsupported. The authors should include a CAFA-style temporal or homology-reduced split and report results against established baselines.
  3. [Section 4, Tables 7 and 8] All quantitative claims, including the headline relative improvements of 2.7%, 4.1%, and 3.1% F1-Max, are based on a single run of each model. No error bars, multiple seeds, or statistical significance tests are reported. Given the small margins, the reported improvements could plausibly be within run-to-run variance. The authors should run multiple seeds and report means and standard deviations, or at minimum establish significance with a paired test across the test set.
  4. [Section 3, Eq. (1) (Annotation Prediction Loss)] The text states that L_AP is "a sum of the categorical cross-entropy over the protein sequences and the binary cross-entropy over the annotations," but Eq. (1) contains only the binary cross-entropy term over the N GO terms. The categorical cross-entropy over sequences is never written down, and the summation over training sequences is implicit. This makes the actual training objective incompletely specified and prevents reproduction. Please provide the full loss expression.
  5. [Section 4, Table 1 vs. Table 7] The ablation table reports an aggregate F1-Max of 0.7533 for SFT, while Table 7 lists per-category F1-Max values of 0.7155 (BP), 0.7124 (CC), and 0.8604 (MF). The paper does not state how the aggregate F1-Max in Table 1 is computed (e.g., micro- vs. macro-averaging, or over which label set). Without this definition, the ablations in Table 1, including the claim that removing the contrastive loss degrades F1-Max by 67.9%, cannot be interpreted or reproduced.
minor comments (6)
  1. [Section 3 (Dataset Curation)] The word "spilit" appears twice in the dataset description; it should be "split."
  2. [Section 4 (Evaluation Metrics)] The metric "F1-Max" is used throughout but never defined. It appears to be the maximum F1 over classification thresholds; this should be stated explicitly, along with how AUPR is computed in a multi-label setting.
  3. [Appendix B] The training set (483,285) plus the test set (53,563) totals 536,848, which is inconsistent with the stated ~510,000 Swiss-Prot sequences. Please reconcile the dataset counts.
  4. [Section 4 (Zero-Shot Baseline)] The zero-shot model is not described: it is unclear how a 300M-parameter ESM-C model produces GO-term probabilities without any fine-tuning. The AUROC values near 0.5 suggest near-random behavior, but the recall values around 0.46-0.56 are not explained.
  5. [Appendix A, Table 4] Several DPO hyperparameters are given as ranges (e.g., Number of Augmentations 3-10, Training Weight 0.01-1.0) rather than concrete values. Please specify the exact values used for the reported experiments.
  6. [References] The ESM-C model is cited only to a blog post (ESM Team, 2024); if a peer-reviewed or preprint reference is available, it should be cited instead.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the DPO-over-SFT gains are empirical and not reducible to the training objective or to self-citations.

full rationale

I checked the derivation chain from Eq. (1) to Eq. (3), the training stages in Sec. 3, and the evaluation in Sec. 4 with Tables 1, 7, and 8, plus Appendices A–D. The DPO loss in Eq. (3) is the standard DPO objective with y_w as ground-truth annotations and y_l as synthetic negatives, and the reference policy is the supervised fine-tuned model. The reported F1-Max, AUPR, and AUROC numbers are computed on a held-out Swiss-Prot split and are not terms in any training loss, so the improvements are not forced by construction. The 'DPO w/ pred' ablation uses the model's own predictions as negatives, but this is a hard-negative mining strategy, not a fitted parameter that defines the benchmark; the resulting F1-Max (0.7947 vs 0.7683 for SFT LoRA) is an empirical outcome that could have gone the other way. The paper contains no self-citations by the present authors and no imported uniqueness theorem. The main substantive weakness is that Eq. (3) writes set-level likelihoods πθ(y|x) without specifying a probability distribution over GO-annotation sets; that is an underspecification affecting reproducibility and interpretation, but it is a rigor/correctness issue, not a circular reduction of the outputs to the inputs. I therefore find no circular step and assign a score of 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the validity of a random split for evaluation, on an undefined set-level probability for DPO, on the frozen pretrained ESM-C encoder, and on the quality of Swiss-Prot labels. The free parameters are mostly standard hyperparameters, but the GO vocabulary cutoff and the DPO loss weight directly shape the reported performance.

free parameters (5)
  • GO vocabulary frequency cutoff = 100 occurrences
    GO terms with fewer than 100 occurrences in Swiss-Prot Jan 2010 are excluded, reducing the label space to 7533 terms; this threshold affects which labels are evaluated.
  • DPO beta = 0.1
    Temperature in the DPO loss (Eq. 3) controlling deviation from the reference policy; set by hand.
  • SAC temperature tau = not specified
    Temperature in the contrastive loss (Eq. 2); no value is reported, making the loss under-specified.
  • DPO loss weight = 0.01 to 1.0 (range)
    Training weight reported as a range in Appendix A, Table 4; the actual value used for the reported results is not stated.
  • Number of augmentations = 3 to 10
    Range given for data augmentation in DPO training; the actual count used is unspecified.
assumptions (4)
  • domain assumption ESM-C (300M) provides sequence embeddings that contain enough functional signal for GO prediction without task-specific pretraining.
    Section 3 states ESM-C is the base sequence encoder; the method relies on its representations.
  • domain assumption The 9:1 random split of Swiss-Prot Jan 2010 yields an unbiased test set for measuring functional annotation generalization.
    Used to construct all quantitative results; no temporal or homology-based split is performed.
  • ad hoc to paper The policy pi_theta(y|x) for a set of GO terms y is well-defined and computable, though never defined in the paper.
    Eq. 3 uses log pi_theta(yw|x) for annotation sets, but the distribution over sets is not specified.
  • domain assumption Swiss-Prot annotations are treated as ground truth without error modeling.
    Both SFT labels and DPO positives come from Swiss-Prot; annotation errors or incompleteness are ignored.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AnnoDPO: Protein Functional Annotation Learning with Direct Preference Optimization." pith.science (2026). https://pith.science/paper/3HMCMAW4

@misc{pith2026250607035,
  author       = {Pith},
  title        = {Pith review of: AnnoDPO: Protein Functional Annotation Learning with Direct Preference Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3HMCMAW4}},
  note         = {Machine review of arXiv:2506.07035}
}
read the original abstract

Deciphering protein function remains a fundamental challenge in protein representation learning. The task presents significant difficulties for protein language models (PLMs) due to the sheer volume of functional annotation categories and the highly imbalanced distribution of annotated instances across biological ontologies. Inspired by the remarkable success of reinforcement learning from human feedback (RLHF) in large language model (LLM) alignment, we propose AnnoDPO, a novel multi-modal framework for protein function prediction that leverages Direct Preference Optimization (DPO) to enhance annotation learning. Our methodology addresses the dual challenges of annotation scarcity and category imbalance through preference-aligned training objectives, establishing a new paradigm for biological knowledge integration in protein representation learning.

Figures

Figures reproduced from arXiv: 2506.07035 by the authors.

Figure 1
Figure 1. Model architecture and training objectives of AnnoDPO. The training framework is divided into three stages: Pre-training: Self-supervised learning of ESM-C on protein sequences from UniRef, MGnify, and JGI (ESM Team, 2024); SFT: Dual-objective finetuning with annotation prediction and sequence-annotation contrastive alignment; DPO: Preference optimization through positive annotations against negative ones. adaptatio… view at source ↗
Figure 2
Figure 2. Comprehensive Evaluation of Protein Function Annotation Performance. (a) Cross-category performance comparison (numerical results in Tab. 7). (b) Robustness analysis across label frequency regimes (numerical results in Tab. 8). (c) t-SNE visualization of GO category discriminability in latent space. (d) Hierarchical relationship preservation in tightly-related GO term families (additional examples in Appendix D). up… view at source ↗
Figure 3
Figure 3. Additional results of sequences with biological process related GOs in fine-grained ontological relationship learning task. 13 [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Additional results of sequences with cellular component related GOs in fine-grained ontological relationship learning task. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: Additional results of sequences with molecular function related GOs in fine-grained ontological relationship learning task. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

18 extracted references · 5 canonical work pages

  1. [11]

    Guiding generative pro- tein language models with reinforcement learning

    Stocco, F., Artigues-Lleixa, M., Hunklinger, A., Widatalla, T., Guell, M., and Ferruz, N. Guiding generative pro- tein language models with reinforcement learning. arXiv preprint arXiv:2412.12979,

    arXiv

  2. [12]

    Saprot: Protein language modeling with structure-aware vocabulary

    Su, J., Han, C., Zhou, Y ., Shan, J., Zhou, X., and Yuan, F. Saprot: Protein language modeling with structure-aware vocabulary. bioRxiv, pp. 2023–10,

    work page 2023

  3. [13]

    Aligning protein generative models with experimental fitness via direct preference optimization

    Widatalla, T., Rafailov, R., and Hie, B. Aligning protein generative models with experimental fitness via direct preference optimization. bioRxiv, pp. 2024–05,

    work page 2024

  4. [14]

    Is dpo superior to ppo for llm alignment? a comprehensive study

    Xu, S., Fu, W., Gao, J., Ye, W., Liu, W., Mei, Z., Wang, G., Yu, C., and Wu, Y . Is dpo superior to ppo for llm alignment? a comprehensive study. arXiv preprint arXiv:2404.10719,

    arXiv

  5. [15]

    Rrhf: Rank responses to align language mod- els with human feedback without tears

    6 Yuan, Z., Yuan, H., Tan, C., Wang, W., Huang, S., and Huang, F. Rrhf: Rank responses to align language mod- els with human feedback without tears. arXiv preprint arXiv:2304.05302,

    arXiv

  6. [16]

    Ontoprotein: Pro- tein pretraining with gene ontology embedding

    Zhang, N., Bi, Z., Liang, X., Cheng, S., Hong, H., Deng, S., Lian, J., Zhang, Q., and Chen, H. Ontoprotein: Pro- tein pretraining with gene ontology embedding. arXiv preprint arXiv:2201.11147,

    arXiv

  7. [17]

    Decoding the molecular language of proteins with evolla

    Zhou, X., Han, C., Zhang, Y ., Su, J., Zhuang, K., Jiang, S., Yuan, Z., Zheng, W., Dai, F., Zhou, Y ., et al. Decoding the molecular language of proteins with evolla. bioRxiv, pp. 2025–01,

    work page 2025

  8. [18]

    M., Stiennon, N., Wu, J., Brown, T

    Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593,

    arXiv 1909

Show all 18 references
  1. [2000]

    Training a helpful and harmless assistant with rein- forcement learning from human feedback

    Bai, Y ., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training a helpful and harmless assistant with rein- forcement learning from human feedback. arXiv preprint arXiv:2204.05862,

    arXiv

  2. [2001]

    Improving alignment of dialogue agents via targeted human judgements

    Glaese, A., McAleese, N., Tr˛ ebacz, M., Aslanides, J., Firoiu, V ., Ewalds, T., Rauh, M., Weidinger, L., Chadwick, M., Thacker, P., et al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209.14375,

    arXiv

  3. [2016]

    Raft: Re- ward ranked finetuning for generative foundation model alignment

    Dong, H., Xiong, W., Goyal, D., Zhang, Y ., Chow, W., Pan, R., Diao, S., Zhang, J., Shum, K., and Zhang, T. Raft: Re- ward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767,

    arXiv

  4. [2017]

    Re- warding progress: Scaling automated process verifiers for llm reasoning

    Setlur, A., Nagpal, C., Fisch, A., Geng, X., Eisenstein, J., Agarwal, R., Agarwal, A., Berant, J., and Kumar, A. Re- warding progress: Scaling automated process verifiers for llm reasoning. arXiv preprint arXiv:2410.08146,

    arXiv

  5. [2020]

    and Consortium, U

    Magrane, M. and Consortium, U. Uniprot knowledgebase: a hub of integrated protein data. Database, 2011:bar009,

    2011

  6. [2021]

    Proximal policy optimization algorithms

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,

    arXiv

  7. [2022]

    Glore: When, where, and how to improve llm reason- ing via global and local refinements

    Havrilla, A., Raparthy, S., Nalmpantis, C., Dwivedi-Yu, J., Zhuravinskyi, M., Hambro, E., and Raileanu, R. Glore: When, where, and how to improve llm reason- ing via global and local refinements. arXiv preprint arXiv:2402.10963,

    arXiv

  8. [2023]

    Controllable protein sequence generation with llm preference optimization

    Liu, X., Liu, Y ., Chen, S., and Hu, W. Controllable protein sequence generation with llm preference optimization. arXiv preprint arXiv:2501.15007,

    arXiv

  9. [2024]

    Policy optimization in rlhf: The impact of out-of-preference data

    5 Li, Z., Xu, T., and Yu, Y . Policy optimization in rlhf: The impact of out-of-preference data. arXiv preprint arXiv:2312.10584,

    arXiv

  10. [2025]

    S., Anand, N., Eguchi, R

    Madani, A., McCann, B., Naik, N., Keskar, N. S., Anand, N., Eguchi, R. R., Huang, P.-S., and Socher, R. Progen: Language modeling for protein generation.arXiv preprint arXiv:2004.03497,

    2004 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.