Pith. sign in

REVIEW 3 major objections 5 minor 4 references

TransNRank: Towards Accurate Neoantigen Ranking with Transformer

T0 review · 3 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read TransNRank, a small Transformer, retrieves 51 of 96 immunogenic peptides in per-patient top-20 lists, up from 45 for prior models.

desk verdict The headline 51/96 is the best of 20 runs with no evidence the selection was validation-based, so treat it as an order statistic until the authors report seed-averaged performance. read the letter →

arxiv 2608.01924 v1 pith:3ZQ72363 submitted 2026-08-03 cs.CE cs.AI

Zhiyin An , Yuenan Hou , Shumeng Duan , Yiming Zhou , Yuanting Zheng , Leming Shi This is my paper
classification cs.CEcs.AI
keywords neoantigenpredictionimmunogenicityrankingTransformerself-attentionclassimbalancefeatureablationtop-20recallpersonalizedcancervaccine
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that a standard, small Transformer can rank personalized neoantigen candidates more accurately than the logistic regression and XGBoost baselines that currently dominate the field. On the shared NeoRanking benchmark, the best of 20 trained TransNRank models recovers 51 of 96 immunogenic peptides within per-patient top-20 lists, compared with 45 for the best previously published models, while training in 20 epochs instead of 200. The paper also reports that a seven-feature subset—led by anchor-position mutation and TCGA pan-cancer expression—suffices for most of the performance, and that intersecting predictions from multiple models does not improve recall. If these results hold, they would provide a practical, cheap way to prioritize vaccine candidates.

What carries the argument

The central mechanism is the standard Transformer encoder applied to a peptide's 31 scalar features; self-attention layers model pairwise interactions among features, and a positive-aware BCE loss with weight 10,000 counters the extreme class imbalance. The evaluation protocol—training 20 models and selecting the best by top-20 recall—is the actual carrier of the headline number.

What would settle it

Compute the mean and median top-20 recall across the 20 TransNRank runs shown in the paper's Figure S2E. If the mean is close to 45 (the prior best) or the 95% confidence interval includes 45, the claimed improvement is not robust; alternatively, fix a validation split, train once, and then evaluate on the held-out 96 positives to see whether 51/96 reproduces.

Watch

Extended reading notes

Core claim

TransNRank applies a two-layer, two-head Transformer with dimension 48 to the 31 immunogenicity features of the NeoRanking dataset, using a binary cross-entropy loss with positive-class weight 10,000. With 20 epochs of training, the authors report a top-20 recall of 51/96 across 42 held-out patients—outperforming logistic regression, XGBoost, and an LR-XGBoost voting ensemble (45/96), and ranking first among all TESLA teams on fraction ranked and top-20 immunogenic fraction. Feature ablation shows that SHAP-selected top-15 features retain performance, and that a core set of seven features—including Mutation at Anchor and TCGA Cancer Expression—marks a sharp threshold below which recall drops

Load-bearing premise

The headline number comes from the best of 20 independent training runs, apparently selected on the test-set top-20 recall; if that selection is not a fair estimate of expected performance, the 45-to-51 improvement could be largely selection noise.

Editorial extensions

If this is right

  • If TransNRank generalizes, per-patient vaccine peptide selection can be done with a two-layer Transformer trained for 20 epochs, cutting training cost roughly tenfold.
  • A seven-feature minimal panel—including anchor mutation and TCGA expression—could replace the full 31-feature extraction pipeline without much loss of ranking quality.
  • Intersecting top-100 predictions from multiple models does not improve recall; it can still serve as a negative filter when experimental validation capacity is limited.
  • Adding logistic regression to TransNRank in a voting ensemble does not improve recall, suggesting the Transformer already captures most predictive signal.
  • The model is currently limited to HLA-I 8–12mer peptides with labels from in vitro assays; clinical vaccine response validation is not yet established.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the authors selected the best of 20 runs (and baselines best of 10), the reported 53.1% may overstate expected performance; the mean or median across runs would give a fairer comparison, and that number is not reported.
  • The sharp drop between seven and six features suggests a minimal biological signature; a prospective cohort could test whether a seven-feature assay predicts immunogenicity as accurately as the full panel.
  • If TCGA reference expression indeed outperforms patient RNA-seq, a testable consequence is that public pan-cancer data could replace costly per-patient expression measurement in neoantigen pipelines.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes TransNRank, a Transformer-based model for ranking personalized neoantigen candidates. Using the harmonized NeoRanking dataset (31 features per peptide), the model is trained on NCI-training (57 patients, 82 immunogenic peptides) and evaluated on held-out cohorts: NCI-test, TESLA, and HiTIDE, together containing 96 immunogenic peptides across 42 patients. The central claim is that TransNRank improves top-20 recall from 46.9% (45/96) for LR/XGBoost baselines to 53.1% (51/96), while training in 20 epochs rather than 200. The paper also reports model overlap analyses, intersection filtering experiments, and feature ablation studies, concluding that seven features—including mutation at anchor and TCGA cancer expression—are sufficient for competitive performance.

Significance. If the held-out result is valid, the paper would provide a useful data point: a small Transformer, trained on the same 31 features as prior LR/XGBoost models, can rank immunogenic peptides better in top-20 lists, an operationally relevant threshold for mRNA vaccine design. The authors are to be credited for using publicly available harmonized data, evaluating on independent cohorts not used in training, benchmarking against 25 TESLA teams, and candidly reporting that no marked improvement was seen on TESLA or HiTIDE individually. The feature-ablation analysis, especially the emergence of mutation at anchor and TCGA expression, is also potentially of practical value. However, the strength of the headline comparison is substantially weakened by the model-selection protocol: the reported numbers are best-of-N run maxima, rather than expected-performance estimates, and the selection criterion does not appear to be anchored to a validation split. The 45-to-51 delta is therefore not yet established as a real improvement.

major comments (3)
  1. [§2.1, Figure S2E] The headline result is reported as a best-of-N selection: 'we trained 20 models and chose the optimal one for evaluation (Figure S2E)', and the baselines were similarly chosen as best of 10 retrained runs (Figure S2A–D). Figure S2E plots top-20 recall per run, which appears to be the same metric used for the headline comparison. The paper does not state that this selection was made on a validation split; the statement in §3.2 that test sets were not exposed during training and cross-validation does not rule out selection on the test cohorts. If runs were selected by test-set top-20 recall, the reported 53.1% is an order statistic over 20 fits to the same 96 labels, and comparing it with a best-of-10 baseline confounds model quality with the number of attempts. Please report a validation-selected run, seed-averaged performance with variance, and a confidence interval for the 45-to-51 diff
  2. [§3.3 vs §2.1] The model scoring function in §3.3 is defined as a sum over immunogenic peptide ranks with exponent alpha=0.05, whereas the run selection in §2.1 and Figure S2E is described in terms of top-20 recall. These are different objectives. If hyperparameter optimization used the sum-of-ranks score but run selection used top-20 recall on the test cohorts, the order-statistic concern applies with even more force, because the two stages may have been tuned to different labels. The text should clarify, for both hyperparameter search and final run selection, exactly which metric and which data partition (training/validation/test) were used.
  3. [§2.4 and §3.7] The feature-ablation claim that recall 'drops sharply between the top 7 and top 6 features' is based on training 30 independent models per feature subset and visualizing maximum, mean, and minimum top-20 recall (Figure 4D). No confidence intervals or statistical tests are reported for these differences, and it is unclear whether the plotted values again select the best of 30 runs. Given that the difference between seven and six features may be within run-to-run noise, the 'seven core features' conclusion needs a more statistically grounded presentation, or at least a table of mean ± SD across the 30 runs.
minor comments (5)
  1. [§3.2] Dataset counts are inconsistent: the text says NCI contains 660,818 negative and 103 positive peptides, but the earlier sentence lists 660,121 samples, and NCI-training (487,670) plus NCI-test (172,451) sums to 660,121, not 660,921. Please reconcile these numbers.
  2. [§2.1] The typo 'HCI-test' (should be NCI-test) appears in the dataset description. Please correct.
  3. [§2.4 / Figure S10] Figure S10 is referenced for the input-weight vs SHAP discrepancy, but Figure S10 in the supplementary materials appears to show intersection quantities for ensemble frameworks. The figure reference needs correction or relabeling.
  4. [§3.5] The training data processing is ambiguous: negative peptides are said to be downsampled to 100,000 'when running the Hyperopt optimization', but it is not stated whether the final model also trains on the downsampled set or on all 487,670 negative peptides. This affects the effective class balance and should be clarified.
  5. [§3.7 (duplicate)] There are two consecutive sections numbered 3.7 ('Ablation Experiment' and 'Data Processing and Visualization'). Renumber the latter.

Circularity Check

2 steps flagged · score 5.0 of 10

The reported 53.1% top-20 recall is the best of 20 runs selected by top-20 recall, and the SHAP top-7 feature set is selected by the same metric; the central LR/XGBoost comparison remains an external benchmark.

  1. fitted input called prediction [Section 2.1; Figure S2E; Section 3.2]
    "We therefore adopted top-20 recall as the primary screening criterion after model training. ... With the positive class weight fixed at 10,000, we trained 20 models and chose the optimal one for evaluation (Figure S2E)."

    The headline top-20 recall (51/96) is the value for the best of 20 independently trained Transformers, chosen by top-20 recall (Figure S2E). If that choice was made on the held-out test cohorts rather than a validation fold, then the reported number is the maximum of 20 evaluations against the same 96 labels: the model selection criterion is identical to the reported metric. The paper does not state that the run-selection scores came from the 20% validation split; the only leakage statement (Section 3.2) says independent test sets were not exposed during training/cross-validation, which does not exclude selection on test. The baseline 45/96 is also a best-of-10 selection, so the improvement is an order-statistic comparison rather than an expected-performance estimate.

  2. fitted input called prediction [Section 2.4; Figure 4D; Section 3.7]
    "Given that models trained on the SHAP top 15 features achieved both higher upper bounds and higher mean values in top‑20 recall, we proceeded with this subset for the final elimination analysis. In this analysis, we observed a sharp drop in top‑20 recall between the top 7 and top 6 features (Figure 4D), strongly suggesting that the first 7 features represent the most critical determinants for immunogenicity prediction."

    The feature-ranking method (SHAP top-15) is selected because it gives higher top-20 recall on the evaluation data, and the same top-20 recall from the same held-out cohorts is then used to conclude that the first seven features are the most critical. Each trimmed feature subset is retrained and scored on the same test labels used to choose the SHAP ordering, so the sharp drop between top-7 and top-6 is a characteristic of a feature set selected to maximize the reported metric, not an independent ablation finding.

full rationale

TransNRank is fundamentally an empirical benchmark against external, publicly available cohorts (NCI-test, TESLA, HiTIDE), and the Transformer architecture is a standard external method (Vaswani et al.), not an imported self-citation or uniqueness claim. The central comparison to LR and XGBoost is therefore externally grounded and not circular. The circularity/selection-bias content is in the protocol: the headline 53.1% top-20 recall is obtained by choosing the best of 20 runs using the same top-20 metric that is reported, and Figure S2E displays top-20 recall per run without stating that the selection was pinned to the validation fold. The feature-ablation headline (top-7 features) is likewise obtained by selecting the SHAP-based subset after observing its higher top-20 recall, then reporting the ablation curve on the same test metric. These are not definitional equivalences, but they are statistical fits of the model and feature set to the evaluation metric; the reported numbers are order statistics, not expected-performance estimates. No self-citation chain is load-bearing. Score 5 reflects partial circularity in the reported headline and feature-importance claims, while acknowledging the core external evaluation.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical entities; its claims rest on an extreme positive-weight parameterization plus downsampling, a standard Transformer stack, and the domain assumptions listed. The most consequential implicit inputs are the in-vitro-labels-as-truth assumption and the assumption that 31 precomputed features exhaust the signal; both are acknowledged in the paper's own limitations section.

free parameters (5)
  • positive class weight wp in BCE loss = 10000 (alternative 1314)
    Chosen by hand in Section 3.5; far above the downsampled negative:positive ratio of about 1219:1, and the paper reports results at both 1314 and 10000. The ranking outcome depends on this extreme weighting.
  • negative downsampling cap during hyperparameter search = 100000 negatives
    Section 3.5: negatives downsampled to 100,000 with all 82 positives during Hyperopt; a hand-set cap that changes the effective training distribution and the implied class ratio.
  • architecture dimensions = 48 dim, 2 heads, 2 layers, patience 15, epochs 20
    Section 3.3: discrete hyperparameters chosen by search over a small space; the paper itself notes fewer layers win given the tiny training set.
  • learning rate and weight decay = lr 1e-6 to 5e-4, wd 1e-3 to 5e-2
    Section 3.3: continuous hyperparameters reported as the most influential (Figure S1), tuned per trial; final values are not reported in the text.
  • scoring function exponent alpha = 0.05
    Section 3.3: scoring sum of exp(alpha*r_i); inherited from NeoRanking (ref 26), not fitted here, used for hyperparameter selection.
assumptions (4)
  • domain assumption In vitro immunogenicity assay labels are treated as ground truth for ranking quality
    All training and test labels come from experimental assay readouts (NCI, TESLA, HiTIDE). The Discussion admits "the correlation between in vitro activity and in vivo efficacy remains unclear", so the objective optimizes a surrogate.
  • domain assumption The 31 precomputed features contain the immunogenicity signal
    Every model sees only the 31 feature values; any signal carried by TCR-peptide binding, HLA-II presentation, or variant context is unobservable to all compared methods. Acknowledged partially in the Discussion's limitation on missing TCR features.
  • domain assumption Input feature noise from external tools is symmetric across models
    The features come from predictors such as NetMHCpan, MixMHCpred, and PRIME; their errors are assumed to affect LR, XGBoost, and Transformer equally. No analysis of feature noise is given.
  • domain assumption SHAP values on the best model measure true feature contribution
    Section 2.4 uses SHAP to rank features and drive the ablation, while reporting only r=0.29 correlation with input-layer weights and stating uncertainty about which measure is more faithful.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TransNRank: Towards Accurate Neoantigen Ranking with Transformer." pith.science (2026). https://pith.science/paper/3ZQ72363

@misc{pith2026260801924,
  author       = {Pith},
  title        = {Pith review of: TransNRank: Towards Accurate Neoantigen Ranking with Transformer},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3ZQ72363}},
  note         = {Machine review of arXiv:2608.01924}
}
read the original abstract

Personalized neoantigen prediction is challenging due to the scarcity of positive samples, the noise of the experimental data, the severe class imbalance trait and the complex of immunogenicity features. Prior arts, such as linear regression and XGBoost fail to model long-range dependencies and contextual relationships within peptide features, therefore the performance of neoantigen positive recall rate is limited. In this paper, we present a novel deep learning framework based on Transformer, coined as TransNRank. By leveraging the self-attention mechanism, our model captures both local and global feature contexts, enabling more accurate recognition of immunogenic neoantigens. A positive-aware training objective is utilized to handle the class imbalance problem, assigning more weights to those few positive samples. Extensive experiments are performed on NCI, TESLA and HiTIDE datasets. Notably, our TransNRank can push the upper bound top 20 recall rate of neoantigen prediction from 46.9% (45 from 96) to 53.1% (51 from 96), while reducing the training epochs from 200 epochs to 20 epochs. Furthermore, we analyze the features contribution based on TransNRank and find that the mutation at anchor and TCGA expression level play an unexpected important role in neoantigen prediction, and removing insignificant features to reduce the input dimensionality of peptides does not drastically impair the overall performance of the model. Our paradigm not only streamlines the prediction pipeline but also sets a new state-of-the-art for neoantigen discovery, with broad implications for accurate immuno-oncology.

Figures

Figures reproduced from arXiv: 2608.01924 by the authors.

Figure 1
Figure 1. Dataset composition and model performance benchmarking [PITH_FULL_IMAGE:figures/full_fig_p018_1.png] view at source ↗
Figure 2
Figure 2. Correlation and overlap analysis of LR, Transformer and XGBoost [PITH_FULL_IMAGE:figures/full_fig_p020_2.png] view at source ↗
Figure 3
Figure 3. Immunogenic peptide recall and positive neoantigen enrichment of multi-model intersected candidate peptides. (A–C) Aggregated immunogenic peptide recall across all patients using intersecting Top100 candidate peptides. Recall values are stratified by ranking cutoffs (Top10, Top20, Top50) and grouped by different intersection strategies: peptides unique to each single model and shared candidates between model pairs. … view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Feature importance analysis and ablation experiments guided by [PITH_FULL_IMAGE:figures/full_fig_p022_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

4 extracted references

  1. [1]

    Exploiting the mutanome for tumor vaccination

    Castle JC, Kreiter S, Diekmann J, Löwer M, van de Roemer N, de Graaf J, et al. Exploiting the mutanome for tumor vaccination. Cancer Res. 2012;72(5):1081-91. 2. Sahin U, Derhovanessian E, Miller M, Kloke BP, Simon P, Löwer M, et al. Personalized RNA mutanome vaccines mobilize poly-specific therapeutic immunity against cancer. Nature. 2017;547(7662):222-6....

    2012

  2. [12]

    Computational Pipeline for the PGV-001 Neoantigen Vaccine Trial

    Rubinsteyn A, Kodysh J, Hodes I, Mondet S, Aksoy BA, Finnigan JP, et al. Computational Pipeline for the PGV-001 Neoantigen Vaccine Trial. Front Immunol. 2017;8:1807. 13. Tang S, Madhavan S. neoantigenR: An annotation based pipeline for tumor neoantigen identification from sequencing data. bioRxiv. 2017:171843. 14. Xiao W, Ren L, Chen Z, Fang LT, Zhao Y, L...

    2017

  3. [23]

    Predicting MHC-I ligands across alleles and species: how far can we go? Genome Med

    Tadros DM, Racle J, Gfeller D. Predicting MHC-I ligands across alleles and species: how far can we go? Genome Med. 2025;17(1):25. 24. O'Donnell TJ, Rubinsteyn A, Laserson U. MHCflurry 2.0: Improved Pan-Allele Prediction of MHC Class I-Presented Peptides by Incorporating Antigen Processing. Cell Syst. 2020;11(1):42-8.e7. 25. Gfeller D, Schmidt J, Croce G, ...

    2025

  4. [34]

    Intersection not taken

    Bartolomeo R, Troise F, Allocca S, Sdruscia G, Vitale R, Bignone V, et al. Development of a Potency Assay for Nous-209, a Multivalent Neoantigens-Based Genetic Cancer Vaccine. Vaccines (Basel). 2024;12(3). 35. D’Alise AM, Willis J, Duzagac F, Hall MJ, Cruz-Correa M, Idos GE, et al. Nous-209 neoantigen vaccine for cancer prevention in Lynch syndrome carrie...

    2024

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.