Pith. sign in

REVIEW 4 major objections 5 minor 13 references

Continued domain-specific pre-training of protein language models for pMHC-I binding prediction

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Continued pre-training on HLA epitopes lifts a protein language model to state-of-the-art pMHC-I binding affinity prediction, achieving a median Spearman correlation of 0.62 across 25 common alleles and beating NetMHCpan (0.56) and…

desk verdict A genuinely useful study of continued domain-specific pre-training for pMHC-I prediction, but the central comparison is conditional on the pre-training corpus not leaking the 2020–2025 test set. read the letter →

arxiv 2507.13077 v1 pith:Q6EW6AJR submitted 2025-07-16 q-bio.QM

Sergio E. Mares , Ariel Espinoza Weinberger , Nilah M. Ioannidis This is my paper
classification q-bio.QM
keywords pMHC-Ibindingpredictioncontinuedpre-trainingproteinlanguagemodelsmaskedmodelingHLAallelesIC50affinityneoantigenESMCambrian
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether an additional round of unsupervised masked-language modeling on domain-specific sequences, a technique common in NLP, helps a protein language model predict which peptides bind to MHC class I molecules. The authors show that continued pre-training on HLA-associated epitopes, followed by fine-tuning on quantitative IC50 binding data, produces a model they call ESMCBA that outperforms existing specialized predictors on a held-out test set. The central result is a median Spearman correlation of 0.62 across 25 common HLA alleles, with gains of about 0.10 in correlation for alleles that have between 500 and 2000 binding measurements compared to the same model fine-tuned without continued pre-training. If this holds, it means domain-specific unsupervised pre-training is a practical step for adapting large protein models to data-scarce biological prediction tasks, particularly for a use case like neoantigen vaccine prioritization.

What carries the argument

The core mechanism is a two-stage training protocol. Stage one is masked-language modeling continued pre-training on HLA-associated peptide sequences, using either epitope sequences alone or epitopes concatenated with their HLA heavy chain, with 15% of peptide positions masked while HLA residues remain visible. Stage two is supervised fine-tuning of all 30 transformer blocks plus a final layer norm and a single-unit linear head that reads mean-pooled token embeddings to regress log10 IC50 values. By first learning the residue-level distribution of binding peptides in an unsupervised way, the model acquires allele-relevant sequence motifs before seeing labeled affinity data; the paper proposes that this adjusts the model's biochemical priors and, with concatenated HLA context, lets it learn allele-specific binding contexts.

What would settle it

Check the pre-training corpus for any peptide with an IEDB submission date on or after January 1, 2020; if any such peptide exists, retrain ESMCBA using only pre-2020 epitopes for continued pre-training and re-evaluate on the same held-out test set. If the median Spearman correlation then falls to or below NetMHCpan's 0.56, the paper's central claim of superiority is refuted.

Watch

Extended reading notes

Core claim

The paper's central claim is that continued masked-language-model pre-training on HLA-associated peptide sequences, before supervised fine-tuning, materially improves protein language model performance on pMHC-I binding affinity prediction. Starting from the 300M-parameter ESM Cambrian model, the authors pre-train on epitope sequences alone or epitope sequences concatenated with the corresponding HLA heavy chain, and then fine-tune all 30 transformer layers plus a linear head to predict log10 IC50 values from high-quality functional antagonist assays. On a test set of 1,879 peptides deposited in IEDB between January 2020 and 2025, the resulting model ESMCBA reaches a median Spearman correlation of 0.62 across 25 common HLA alleles, outperforming NetMHCpan (0.56), MHCflurry (0.49), HLApollo (0.44), HLAthena (0.37), and MHCnuggets (0.22). The paper also demonstrates that continued pre-training gives consistent gains over direct fine-tuning without that step, most strongly for alleles with moderate data (500–2000 peptides, ~0.10 correlation improvement), and that ESMCBA generalizes to noisy qualitative annotations with a mean AUROC of 0.79 for Negative vs Positive-Low and 0.97 for Negative vs Positive-High.

Load-bearing premise

The continued pre-training corpus is built from IEDB positive binders with an 80/10/10 split, but the paper does not state that this corpus is filtered by submission date, so it may contain epitopes deposited on or after January 1, 2020, which would mean the held-out test set has already been seen during unsupervised pre-training and the reported performance gains could be inflated.

Editorial extensions

If this is right

  • If the result holds, continued domain-specific pre-training becomes a broadly applicable recipe for adapting protein language models to specialized molecular prediction tasks where labeled data are scarce but unlabeled sequences are abundant.
  • ESMCBA's quantitative affinity predictions could be used for neoantigen vaccine prioritization and large-scale screening across alleles that are underrepresented in current training datasets.
  • The finding that gains are strongest at 500–2000 peptides per allele suggests that continued pre-training mostly helps in a middle-data regime, where the supervised signal is too weak to learn motifs from scratch but strong enough for the pre-trained representations to be useful.
  • Avoiding mass spectrometry-derived training data in favor of functional antagonist assays may be an important design choice that lets the model sidestep systematic biases inherited by other predictors.
  • The framework could be extended to MHC class II binding and TCR–pMHC interactions, as the paper explicitly suggests, if the same continued pre-training benefit carries over.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The most consequential untested assumption is temporal leakage: the unsupervised pre-training corpus is described as built from IEDB positive binders with an 80/10/10 split, and if those binders include epitopes deposited on or after January 1, 2020, the held-out test set used for all benchmark comparisons may have already appeared during pre-training, which would inflate ESMCBA's apparent superio
  • The paper's own limitation data show that continued pre-training actually hurts for alleles with fewer than 500 peptides, so the method's practical value may be confined to alleles with a moderate amount of training data rather than a uniform solution to data scarcity.
  • Because the base ESM Cambrian model was itself trained on large protein corpora that could contain peptide sequences similar to the IEDB epitopes, part of the benefit attributed to continued pre-training might in principle come from the base model's general protein knowledge; a cleaner test would compare against a base model explicitly trained without any HLA-related sequences.
  • The qualitative-label AUROC results suggest that the model's rank ordering is meaningful even when the supervision is noisy, which could make it useful for filtering candidate epitopes before expensive validation, but the paper does not report calibration of the predicted affinities, so the numerical IC50 values should be treated as rankings rather than absolutes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper investigates whether continued domain-specific pre-training of a protein language model (ESM Cambrian, 300M parameters) improves peptide–MHC class I binding affinity prediction. The authors perform masked-language-modeling pre-training on HLA-associated peptides (epitope-only and HLA+epitope formats), then fine-tune on quantitative IC50 data using a temporal split (pre-2020 training, post-2020 held-out test). They report that the resulting model, ESMCBA, achieves a median Spearman correlation of 0.62 across 25 HLA alleles on the held-out test set, outperforming NetMHCpan, MHCflurry, HLApollo, HLAthena, and MHCnuggets, and that continued pre-training adds roughly 0.10 correlation for alleles with 500–2000 training peptides. They also evaluate on a qualitative assay benchmark.

Significance. If the central claims hold, the paper makes a useful contribution: it provides a systematic, controlled test of continued domain-specific pre-training for protein language models in an immunology task, uses high-quality quantitative IC50 data rather than mass-spectrometry-derived labels, and ships public code and a temporal evaluation split. The honest discussion of data-scarce regimes and the explicit limitation statement are commendable. However, the significance hinges on excluding the held-out test peptides from the unsupervised pre-training corpus and on the statistical validity of the reported per-allele correlations; both need to be resolved before the comparative claims can be accepted.

major comments (4)
  1. [§5.2 and §2.3] The unsupervised pre-training corpus is described as 'From the IEDB, we used positive binders as described in the qualitative labels' with an 80/10/10 random split, but no temporal cutoff is stated, while the IEDB download was accessed on 16-01-2025. The held-out test set in §2.3 consists of peptides deposited on or after January 1, 2020 (n=1,879, Table S2). If any post-2020 peptide appears in the Stage 1 MLM training split, the PT models have seen the test peptides during pre-training while the Non-PT models and external baselines have not. This would directly inflate both the SOTA comparison and the ~0.10 continued-pretraining gain in Table 1. The authors must apply the same temporal filter to the Stage 1 corpus or demonstrate by exact sequence matching that no test peptide appears in the MLM training split; the released code makes this check feasible, and both safeguards should be reported.
  2. [Table 1 and Fig 2] The paper states that three replicates were trained, but Table 1 reports only mean Spearman and Pearson values without standard errors or confidence intervals. Fig 2's caption says correlation matrices are computed from 'the top model replicates (by spear man)', which selects the best replicate and can bias the visual comparison. Without variance across replicates, the claimed ~0.10 improvement of PT E over Non-PT E in the 500–2000 peptide bins is not statistically assessable. Report mean±std (or full replicate distributions) for Table 1 and use a pre-specified aggregation rule for Fig 2 rather than selecting the top replicate.
  3. [§2.3 and Table S2] The median Spearman ρ=0.62 is computed across 25 alleles, but Table S2 lists alleles with only 1 or 2 test peptides: HLA-B*44:03, B*39:06, A*32:01, and B*40:01 have n=1, and A*02:05, B*35:01, C*07:01, A*26:01, and A*30:01 have n=2. Spearman correlation is undefined for n=1 and highly unstable for n=2–3, so the reported median across 25 alleles depends on ill-defined or near-arbitrary per-allele values. Specify a minimum n for computing per-allele correlations, or report a pooled/weighted correlation with bootstrap confidence intervals that do not rely on per-allele estimates with n<5.
  4. [§2.4] The qualitative-evaluation section states: 'To prevent any data leakage, we downloaded the original training sets used by ESMCBA and MHCFlurry and verified that none of the qualitative epitopes appeared in their training data.' However, Fig 4 compares ESMCBA against NetMHCpan, HLAthena, HLApollo, MHCnuggets, and MixMHCpred in addition to MHCflurry. The manuscript does not report checking whether the qualitative test entries appear in those models' training sets. Without exclusion checks for all baselines, the claim of a leakage-free qualitative benchmark is not established; either extend the verification to all models or explicitly restrict the leakage claim to the ESMCBA-versus-MHCflurry comparison.
minor comments (5)
  1. [Fig 2 caption] The caption contains the typo 'spear man' and the phrase 'top model replicates (by spear man)' is ambiguous; clarify whether the selection was by per-allele Spearman, by median Spearman, or by some other criterion, and define '0L/30L' in the caption or text.
  2. [References] The HLApollo reference (Thrift et al., 2024) lists DOI 10.1101/2024.01.15.123456, which appears to be a placeholder rather than a real DOI; please verify and provide the correct identifier.
  3. [§5.2] The text says training sequences were split 80:10:10 into train, validation, and evaluation sets, but the evaluation split is never used in the reported results; clarify its role or remove it to avoid confusion.
  4. [Fig 3] For NetMHCpan, the paper uses the 'percentile rank' column but does not explain how the rank was oriented or sign-corrected relative to IC50 before computing Spearman and Pearson correlations; state explicitly whether higher rank was mapped to lower predicted affinity.
  5. [Abstract and §2.3] The phrase '25 common HLA alleles' is imprecise given that several of the alleles in Table S2 have only one or two test peptides; 'common' should be justified or replaced with 'alleles with at least one 2020–2025 measurement'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central results are empirical comparisons against external predictors and direct fine-tuning ablations, with no fitted constants renamed as predictions and no load-bearing self-citations.

full rationale

The paper's derivation chain is empirical rather than definitional. The central claim is that ESMCBA, after continued pre-training and fine-tuning, reaches a median Spearman correlation of 0.62 on a held-out IEDB test set and outperforms NetMHCpan, MHCflurry, and other baselines. These are comparisons against independently published predictors and against an ablation that directly fine-tunes the same ESM Cambrian backbone without continued pre-training. No parameter is fitted to the 2020–2025 test set and then reported as a prediction; the test set is defined by a temporal split described in Section 5.1. The continued-pretraining gain of roughly 0.10 for alleles with 500–2000 peptides is a measured difference between model variants, not an algebraic identity. The unsupervised MLM corpus is built from IEDB data, and the fine-tuning data also come from IEDB, but that is data reuse rather than circular reasoning; it does not make the outcome equivalent to the input by construction. Similarly, the qualitative-assay evaluation uses an explicitly excluded set of 18,269 entries and verifies absence from ESMCBA and MHCflurry training sets. There are no self-citations that carry the argument, no uniqueness theorem imported from the authors' prior work, and no renamed empirical regularity presented as derivation. The potential concern that the pre-training corpus may include post-2020 epitopes would be a data-leakage validity issue, not a circularity issue, and cannot be demonstrated from the text alone. The score is therefore 0.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central claim rests on data curation choices (IEDB IC50, Gaussian subsampling), on the temporal holdout being uncontaminated, and on the ESM Cambrian tokenizer adequately representing HLA+peptide inputs. No new physical or biological entities are introduced.

free parameters (1)
  • Gaussian subsampling kernel center = 10^3 nM
    Methods 5.1: training IC50 values are subsampled with a Gaussian kernel centered at 10^3 nM (approximate mean affinity); this manually chosen center reshapes the training label distribution and is not derived from the test objective.
assumptions (3)
  • domain assumption IEDB quantitative functional antagonist IC50 values are reliable enough to serve as ground truth for both training and held-out evaluation.
    All labels come from heterogeneous IEDB assays; the paper acknowledges noise and outliers but treats IC50 as the target (Sections 2.1, 5.1).
  • domain assumption Held-out peptides deposited after Jan 1, 2020 are unseen by every training stage.
    The temporal split is applied to the quantitative fine-tuning set (Section 5.1), but Section 5.2 does not state that the MLM pre-training corpus follows the same cutoff.
  • domain assumption ESM Cambrian tokenizer and the 1024-token window can represent a concatenated HLA chain plus peptide without loss of binding-relevant signal.
    HLA sequences can exceed the remaining token budget after truncation or padding; the paper does not analyze truncation effects (Section 5.2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Continued domain-specific pre-training of protein language models for pMHC-I binding prediction." pith.science (2026). https://pith.science/paper/Q6EW6AJR

@misc{pith2026250713077,
  author       = {Pith},
  title        = {Pith review of: Continued domain-specific pre-training of protein language models for pMHC-I binding prediction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q6EW6AJR}},
  note         = {Machine review of arXiv:2507.13077}
}
abstract

Predicting peptide--major histocompatibility complex I (pMHC-I) binding affinity remains challenging due to extreme allelic diversity ($\sim$30,000 HLA alleles), severe data scarcity for most alleles, and noisy experimental measurements. Current methods particularly struggle with underrepresented alleles and quantitative binding prediction. We test whether domain-specific continued pre-training of protein language models is beneficial for their application to pMHC-I binding affinity prediction. Starting from ESM Cambrian (300M parameters), we perform masked-language modeling (MLM)-based continued pre-training on HLA-associated peptides (epitopes), testing two input formats: epitope sequences alone versus epitopes concatenated with HLA heavy chain sequences. We then fine-tune for functional IC$_{50}$ binding affinity prediction using only high-quality quantitative data, avoiding mass spectrometry biases that are inherited by existing methods.

Figures

Figures reproduced from arXiv: 2507.13077 by the authors.

Figure 1
Figure 1. (A) Distribution of available pMHC-I training data across 121 HLA alleles from classes A, [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Performance (Spearman correlation) matrices between the top model replicates (by spear [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Predicted versus measured binding affinities for epitopes deposited in IEDB between 2020 [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: ROC-AUC performance across qualitative assay outcomes. Evaluations based on 18,269 qualitative entries from IEDB, excluded from quantitative training sets. 3.3 Limitations and future work Data bias is a recurrent problem in immunopeptidomics and modeling of pMHC intera…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

13 extracted references · 7 canonical work pages

  1. [1]

    P. M. Bruno, R. T. Timms, N. S. Abdelfattah, Y. Leng, F. J. N. Lelis, D. R. Wesemann, X. G. Yu, and S. J. Elledge. High-throughput, targeted mhc class i immunopeptidomics using a functional genetics screening platform. Nature Biotechnology, 41 0 (7): 0 980--992, July 2023. doi:10.1038/s41587-022-01566-x. URL https://doi.org/10.1038/s41587-022-01566-x

    work page doi:10.1038/s41587-022-01566-x 2023

  2. [2]

    B. Chen, D. Gfeller, L. Bassani-Silva, I. Sirois, C. Kesmir, and T. Trolle. Predicting HLA -i binding peptides with position-weight matrices trained on large peptidome datasets. Immunogenetics, 71 0 (6--7): 0 389--400, 2019. doi:10.1007/s00251-019-01121-7

    work page doi:10.1007/s00251-019-01121-7 2019

  3. [3]

    Gururangan, A

    S. Gururangan, A. Marasovi \'c , S. Swayamdipta, K. Lo, I. Beltagy, D. Downey, and N. A. Smith. Don't stop pretraining: Adapt language models to domains and tasks. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020 0 (1): 0 8342--8360, 2020. doi:10.18653/v1/2020.acl-main.740

    doi:10.18653/v1/2020.acl-main.740 2020

  4. [4]

    Hayes, R

    T. Hayes, R. Rao, H. Akin, Z. Lin, and A. Rives. Simulating 500 million years of evolution with a language model. Science, 387 0 (6736): 0 850--858, 2025. doi:10.1126/science.ads0018

    doi:10.1126/science.ads0018 2025

  5. [5]

    Z. Lin, H. Akin, R. Rao, B. Hie, Z. Zhu, W. Lu, N. Smetanin, and C. e. a. Dos Santos. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science, 379 0 (6637): 0 1123--1130, 2023. doi:10.1126/science.ade2574

    doi:10.1126/science.ade2574 2023

  6. [6]

    Nijkamp and E

    E. Nijkamp and E. Team. ESM Cambrian : Revealing the mysteries of proteins with unsupervised learning. EvolutionaryScale Blog, 2024. URL https://evolutionaryscale.ai/blog/esm-cambrian

    work page 2024

  7. [7]

    T. J. O'Donnell, A. Rubinsteyn, and U. Laserson. MHCflurry 2.0: Improved pan-allele prediction of MHC class i-presented peptides by incorporating antigen processing. Cell Systems, 11 0 (1): 0 42--48.e7, 2020. doi:10.1016/j.cels.2020.06.010

    work page doi:10.1016/j.cels.2020.06.010 2020

  8. [8]

    Reynisson, B

    B. Reynisson, B. Alvarez, S. Paul, B. Peters, and M. Nielsen. Netmhcpan 4.1 and netmhciipan 4.0 improve MHC antigen-presentation predictions by integrating mass-spectrometry and affinity data. Nucleic Acids Research, 48 0 (W1): 0 W449--W454, 2020. doi:10.1093/nar/gkaa379

    doi:10.1093/nar/gkaa379 2020

Show all 13 references
  1. [9]

    Rives, J

    A. Rives, J. Meier, T. Sercu, S. Goyal, Z. Lin, J. Liu, D. Guo, M. Ott, C. L. Zitnick, J. Ma, and R. Fergus. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, 118 0 (15...

    2021 doi

  2. [10]

    Sarkizova, S

    S. Sarkizova, S. Klaeger, C. Spielman, J. Daw, N. M. Durham, et al. A large peptidome dataset improves HLA class i epitope prediction with HLAthena . Nature Biotechnology, 38 0 (2): 0 199--209, 2020. doi:10.1038/s41587-019-0322-z

    2020 doi

  3. [11]

    X. Shao, M. Taha, R. M. Ewing, N. S. C. van Oers, D. S. Marks, and A. Rubinsteyn. Mhcnuggets: a deep-learning method for neoantigen prediction that balances accuracy and runtime. Bioinformatics, 36 0 (1): 0 103--111, 2020. doi:10.1093/bioinformatics/btz486

    2020 doi

  4. [12]

    J. C. Thrift, A. Elnaggar, E. C. Alley, and J. A. Greenbaum. Hlapollo predicts peptide– HLA class i binding from paired genotype and peptide sequences using transformers. bioRxiv, 2024. doi:10.1101/2024.01.15.123456

    2024 doi

  5. [13]

    R. Vita, S. Mahajan, J. A. Overton, S. K. Dhanda, S. Martini, J. R. Cantrell, D. K. Wheeler, A. Sette, and B. Peters. The immune epitope database ( IEDB ): 2018 update. Nucleic Acids Research, 47 0 (D1): 0 D339--D343, 2019. doi:10.1093/nar/gky1006

    2018 doi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.