{"id":"6231ad84-8c83-4124-bfa9-f1d11aa62cd8","arxiv_id":"2505.17552","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"RankNovo, a list-wise reranker with mass-deviation supervision, improves de novo peptide sequencing accuracy by selecting among candidates from multiple base models.","lead":"RankNovo is a new machine-learning system that reranks the candidate peptides proposed by several existing de novo sequencing models, picking the best match for each mass spectrum and improving peptide identification accuracy. It shows that combining complementary sequencing models through reranking can beat any single model, including on models never seen during training.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Candidate-pool confound is real: RankNovo's gain over top-1 baselines may be mostly pool expansion, and the paper does not report oracle recall or per-candidate counts.","rationale":"The reader's weakest_assumption identifies the candidate-pool confound, which I agree is the single most load-bearing concern. The paper's strongest claim is about reranking skill establishing SOTA, but the experimental design compares multi-candidate reranking against top-1 baselines without reporting the candidate pool size or oracle recall. My stress-test finds the same gap and proposes a concrete test: compute oracle recall of the beam-5 pool; if it is close to RankNovo's 0.660, the central claim that reranking skill drives the improvement is not supported. I also note the amino acid precision inconsistency (0.829 vs 0.824) as a secondary concern that reinforces the need for revision, but I do not treat it as the primary attack. I recommend keeping the reader's CONDITIONAL verdict: the framework is plausible and the ablations are internally consistent, but the missing candidate-pool baseline and the metric inconsistency prevent full acceptance. I am not moving to REJECT because the paper releases code and the missing numbers are easily obtainable; a conditional accept with a request for oracle recall and candidate counts is appropriate.","tokens_in":30494,"tokens_out":1698,"duration_ms":11531,"concrete_test":"Run the released RankNovo code on the 9-species-V1 test set and report: (1) the mean/median number of candidates per spectrum entering the reranker (with beam size 5 across six base models), and (2) the oracle recall of that candidate pool, i.e., the fraction of spectra where the correct peptide appears among at least one base model's beam. Then compute the peptide recall of a trivial 'oracle' ranker that always selects the correct peptide when present, and compare it with RankNovo's 0.660. If oracle recall is close to 0.660, the reranker's selection skill contributes little beyond pool expansion; if oracle recall is substantially higher, the reranker's selection skill remains unproven unless a best-in-beam per base model baseline is also reported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that RankNovo's reranking skill, not merely pooling more hypotheses, drives the SOTA peptide recall of 0.660 on 9-species-V1. The paper's evaluation compares RankNovo (which reranks all six base models' beam-5 outputs) against base models reported as top-1 predictions, while never reporting how many candidates per spectrum enter the reranker or the oracle recall of that candidate pool. Appendix A confirms this: training uses greedy decoding and evaluation uses beam search with beam size 5, but the number of candidates per spectrum and the fraction of spectra whose correct peptide is present in the pool are not reported. Without an oracle-recall or 'best-in-beam' baseline, the observed improvement could be explained by the expanded candidate pool alone: a reranker that merely picks any candidate from a larger pool would raise recall even with no ranking skill. The ablation in Table 9 partially addresses this by varying model count, but it does not isolate pool expansion from reranking quality, and Table 9's 2-model result (0.586) still uses multiple candidates per model. The paper also reports amino acid precision inconsistently: Section 4.2 says 0.829 while Table 1 shows 0.824. This numerical inconsistency weakens confidence in the reported metrics overall. The conclusion's stated limitation (Appendix E.5) that reranking time is minor and candidate collection dominates is consistent with the pool-expansion interpretation, since the extra candidates are the expensive part. Thus the load-bearing assumption, that the candidate pool contains correct peptides and that RankNovo selects among them better than baselines, is plausible but unverified by the reported experiments.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RankNovo, a list-wise deep reranking framework for de novo peptide sequencing. Six base sequencing models (Casanovo-V2, ContraNovo, ByNovo, R-Casanovo, R-ContraNovo, R-ByNovo) generate candidate peptides; RankNovo represents the candidates as a multiple sequence alignment, applies axial attention (row and column) plus cross-attention to spectrum features, and is trained with two new mass-based losses, PMD (peptide mass deviation) and RMD (residual mass deviation). Evaluation on the 9-species-V1 and 9-species-V2 benchmarks reports average peptide recall of 0.660 and 0.781, respectively, exceeding all base models and prior published methods, together with a zero-shot experiment in which a RankNovo trained on two base models is applied to candidates from additional unseen models.","tokens_in":30795,"tokens_out":5721,"duration_ms":51672,"significance":"If the central claim is established, RankNovo would be a meaningful contribution: it introduces a reranking paradigm to a field dominated by single-model architectures, proposes interpretable mass-deviation supervision that improves over generic pointwise/pairwise/listwise losses, demonstrates transfer to unseen base models, and ships code. The paper also provides extensive ablations (loss terms, column attention, base-model subsets) and reports inference cost. However, the main quantitative claim is currently confounded by candidate-pool expansion: RankNovo reranks beam-5 outputs from up to six models, while base-model numbers are top-1 predictions, and no oracle-recall or best-in-beam control is reported. The significance will be much clearer once the authors quantify how much of the gain comes from having more candidates versus from selecting correctly among them.","major_comments":[{"comment":"The evaluation does not separate reranking skill from candidate-pool expansion. RankNovo reranks the beam-5 outputs of six base models (up to 30 candidates per spectrum), whereas the base-model peptide recalls in Table 1 are top-1 predictions. The paper never reports the number of candidates per spectrum that actually enter the reranker, the oracle recall of that pooled set, or the best-in-beam recall of each individual base model. Without these numbers, the observed gain (0.660 vs 0.623 for ByNovo) could be explained by a selector that benefits only from a larger pool, not from ranking quality. Please report oracle recall and best-in-beam recall for each base model and for the full pool, and add a pool-only baseline such as 'always choose the strongest base model's top-1' or 'choose the longest candidate' to quantify the skill component.","section":"Section 4.2, Appendix A"},{"comment":"The zero-shot experiment is subject to the same confound. Table 16 shows peptide recall rising from 0.586 with 2 inference models to 0.649 with 6 inference models, but the candidate pool also grows from 10 to 30 hypotheses per spectrum. The paper attributes this improvement to RankNovo's zero-shot reranking ability, yet without oracle recall at each inference-model count, or a nonparametric baseline that always selects ByNovo's top-1 (which is already 0.623), the gain could be due mainly to the addition of stronger candidates. Please report oracle recall for each N-infer setting and a 'best base model top-1' baseline.","section":"Section 4.3, Appendix E.1, Table 16"},{"comment":"There is a numerical inconsistency in the reported amino acid precision: the main text states 'RankNovo reaches a precision of 0.829,' while Table 1 and Appendix Table 6 both report an average of 0.824 on 9-species-V1. This discrepancy affects the paper's headline comparison with ByNovo (0.804) and ContraNovo (0.784). Please correct the text and re-verify the summary statistics, including the percentage gains quoted for peptide recall.","section":"Section 4.2, Table 1, Table 6"},{"comment":"The base-model-combination ablations change both the training set and the inference candidate-pool size simultaneously, so they do not isolate the effect of model diversity. A reader cannot tell whether the monotonic improvement from 0.586 (2 models) to 0.660 (6 models) is due to more diverse training candidates, more candidates at inference, or better reranking. Please add a control with a fixed inference pool (e.g., always 6 models) while varying the training set, and a control with a fixed training set while varying the inference pool, and report oracle recall for each condition.","section":"Section 4.4, Tables 9 and 10"}],"minor_comments":[{"comment":"The abstract and introduction call RankNovo 'the first deep reranking framework' for de novo peptide sequencing, but the references include pNovo-3 (Yang et al., 2019), which is described as a learning-to-rank framework for exactly this task. Please qualify the novelty claim by contrasting with pNovo-3 explicitly.","section":"Introduction, Related Work"},{"comment":"The training-set construction excludes 'spectrums that are correctly predicted by all six base models,' leaving 7 million spectra. The potential selection bias introduced by this filter is not discussed; please comment on whether this affects the learned reranking behavior and whether the evaluation set is also filtered.","section":"Appendix A"},{"comment":"The sinusoidal m/z embedding in Equation (1) uses the symbol k both as an index in the sine/cosine argument and as the dimension divisor in the denominator; this is confusing and should be clarified.","section":"Section 3.2, Equation (1)"},{"comment":"The 'contribution of each base model' analysis is based only on spectra where RankNovo is correct and exactly one base model contains the correct peptide in its beam. This is a narrow subset and does not measure each model's marginal contribution to overall recall; a leave-one-out ablation would be more informative.","section":"Appendix E.4, Figure 7"}],"recommendation":"major_revision","confidential_remarks":"The candidate-pool confound is the main risk to the paper's central claim. The authors appear capable of addressing it by reporting oracle recall, best-in-beam numbers, and a pool-only baseline; these are standard additions rather than a change of scope. I would also double-check the consistency of all headline numbers before revision, since the AA-precision discrepancy and percentage claims are easy to fix but currently undermine confidence. The zero-shot experiment is potentially the strongest part of the paper and becomes convincing only with the missing controls."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Net: this is the first deep reranking framework for de novo peptide sequencing, and the core idea is worth taking seriously. The MSA-style list-wise encoding plus axial attention is a sensible way to let candidates compare against each other, and the two mass-based losses, PMD and RMD, are defined cleanly from the amino acid mass table with no fitted parameters. The zero-shot experiment, while limited, does show that a model trained on two base models can rerank candidates from models it never saw, with recall climbing from 0.586 to 0.649 as inference models are added. I also credit the ablations: PMD+RMD beats point-, pair-, and list-wise objectives on the same candidate pool, and column attention helps.\n\nMy main reservation is that the headline \"surpasses its base models\" claim is confounded. Base-model numbers look like top-1 predictions, while RankNovo gets to rerank a pool of up to 30 candidates (six models times beam size 5). The paper never reports how many candidates per spectrum actually enter the reranker, nor the oracle recall of that pool. Without that, part of the gain over ByNovo and ContraNovo could simply be pool expansion. The authors' own ablation shows recall rising from 0.586 with 2 models to 0.660 with 6, which is consistent with \"more candidates helps\" as much as \"reranking helps.\" The right fix is straightforward: report oracle recall, report the best-in-pool baseline, and add a condition that reranks only the top-1 prediction from each base model. That would isolate ranking skill from pool size.\n\nAlso: Section 4.2 says amino acid precision of 0.829, but Table 1 shows 0.824. One of those is wrong. And the \"universal\" framing overreaches: the unseen models are all transformer-based variants trained by the same group, not external third-party models. Finally, there are no error bars, though with 1.5M PSMs that is a minor issue.\n\nBottom line: the framework is novel, the losses are principled, and the reranking-vs-reranking comparisons are internally consistent. The central SOTA claim needs one more experiment before I would trust the number. This deserves a serious referee, not a desk reject—send it out with a request for the oracle-recall analysis.","headline":"A genuinely new reranking idea for de novo peptide sequencing with sensible mass-based losses, but the headline SOTA claim needs an oracle-recall baseline before it fully holds up.","tokens_in":31399,"tokens_out":3264,"would_cite":true,"duration_ms":27364,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"De novo peptide sequencing improves when a reranker selects among candidates from several sequencing models: RankNovo reports 0.660 average peptide recall on the nine-species benchmark, 6.1% above its best base model.","keywords":["de novo peptide sequencing","reranking","mass spectrometry","multiple sequence alignment","axial attention","peptide mass deviation","proteomics","zero-shot generalization"],"falsifier":"Compute, on the 9-species-V1 test set, the oracle peptide recall of the pooled beam candidates from the six base models—the fraction of spectra for which the correct peptide appears anywhere in the pool. If that oracle recall already reaches or exceeds RankNovo's 0.660, or if a trivial selector that always takes the longest or most frequent candidate matches its performance, the claim that learned list-wise reranking drives the improvement is refuted; if oracle recall is high and RankNovo falls far below it, the reranker's selection is the true bottleneck and the architecture claim stands.","tokens_in":30285,"feed_emoji":"🧬","tokens_out":10143,"duration_ms":69832,"temperature":0.7,"pith_summary":"RankNovo treats de novo peptide sequencing—reading a peptide's amino-acid sequence directly from a tandem mass spectrum, without a reference database—as a reranking problem. Instead of trusting any single sequencing model, the paper trains a list-wise reranker that takes the candidate peptides produced by several base models, lets the candidates compare with each other in a single pass, and picks the best one. On the standard nine-species benchmark the reranker reports an average peptide recall of 0.660, beating its strongest base model by 6.1% and the previous state of the art by 4.3%, with similar gains on the newer V2 benchmark. The paper's broader claim is that reranking is a universal layer: trained on just two weak models, RankNovo still improves candidates from unseen models, so it could sit on top of any future sequencer. If this holds, the single-model paradigm of the field is challenged; accuracy becomes something you buy by adding models to the pool and a skilled selector on top.","feed_headline":"Reranker beats best peptide sequencer by 6.1 percent","feed_subtitle":"Picking the best candidate from several sequencing models reaches 0.660 peptide recall and lifts unseen models too.","key_machinery":"The load-bearing object is the candidate panel: for each spectrum, the outputs of several base sequencers are padded to equal length, stacked into a $c \\times \\ell \\times d$ tensor—a multiple sequence alignment of peptides—and processed in one forward pass. Axial attention alternates row attention (within one candidate) and column attention (across candidates), so the score of each candidate depends on its rivals, at cost $O(c\\ell^2 + k^2\\ell)$ instead of full self-attention over all tokens; cross-attention injects the spectrum encoder's features. The training signal is the pair of mass-deviation metrics: PMD runs a Needleman–Wunsch-style dynamic program with residue mismatch cost $|M(r_i)-M(r_j)|$ and a gap penalty set to the average mass divergence between distinct residues, while RMD compares each query residue's prefix mass to the nearest target prefix mass. The joint objective is $L = \\lambda L_{\\mathrm{PMD}} + (1-\\lambda)L_{\\mathrm{RMD}}$ with $\\lambda=0.5$, both terms RMSE. These metrics replace binary correctness labels, giving the reranker a graded, mass-aware notion of how wrong each candidate is.","core_discovery":"The paper's central claim is that the accuracy ceiling of de novo peptide sequencing is not set by any single model: different sequencers make different correct predictions on different spectra, and a meta-model that compares all candidates at once can recover more correct peptides than any of its generators. RankNovo, presented as the first deep reranking framework for this task, stacks the candidate peptides from six base models into a multiple sequence alignment and processes the stack with axial attention—row attention reads each candidate, column attention lets candidates share information—with cross-attention to the spectrum. Supervision comes from two new mass-based scores: PMD, a peptide-level divergence computed by dynamic-programming alignment in which mismatches cost the mass difference between residues, and RMD, a residue-level vector of prefix-mass deviations. Trained to predict both with a joint RMSE loss, RankNovo reports average peptide recall of 0.660 on 9-species-V1 and 0.781 on 9-species-V2, surpassing every base model it was trained on. Ablations attribute part of the gain to the mass-based objectives: identical backbones trained with point-wise, pair-wise, or standard list-wise losses reach only about 0.646–0.648 peptide recall, while PMD+RMD reaches 0.660, and removing column attention drops recall to 0.653. When trained on the two weakest models, the reranker still improves candidates from four unseen models, with recall rising from 0.586 to 0.649 as unseen models are added.","pith_inferences":["The paper never reports oracle recall—how often the correct peptide appears in the candidate pool at all. A decisive follow-up would compute the best-in-pool recall of the six base models' beams on 9-species-V1; if that number is close to 0.660, the reranker's value is modest selection over a rich pool, while a large gap between oracle and achieved recall would show selection skill is the real cei","The MSA-style formulation suggests a natural extension beyond de novo models: the same panel could rerank peptide-spectrum matches produced by database search engines, merging two traditionally separate identification paradigms.","A cheap test of the 'universal reranker' claim would be to train on the two weakest models and then add a qualitatively different sequencer family that was never seen, such as a non-autoregressive or graph-based model; the paper's zero-shot trend predicts continued recall gains.","If the list-wise design is what matters, the number of candidates per spectrum should be reported and varied; the paper fixes beam size at 5 without stating how many candidates actually enter the panel, and that number is the knob controlling both performance and the inference-time trade-off the paper emphasizes."],"forward_implications":["Because the reranker generalizes zero-shot, a new sequencing model can be added to the pool at inference time with no retraining; the paper shows recall rising from 0.586 to 0.649 as unseen models are added to a two-model training set.","Accuracy becomes tunable against compute: more base models cost more candidate-collection time (roughly linear, from 0.258 to 0.949 seconds per spectrum for 2 to 6 models) but yield higher recall, a trade-off no single-model sequencer offers.","The mass-deviation supervision transfers to post-translational-modification settings: the paper reports gains of 5.6% on acetylation, 2.8% on dimethylation, and 6.7% on phosphorylation over the best base model after fine-tuning.","Reranking improvements and single-model improvements compound: any future stronger base model becomes a stronger candidate generator, so the reranking layer's value increases with the diversity and quality of the pool."],"supporting_citations":[{"why":"ContraNovo, the previous state of the art and a base model: it sets the benchmark RankNovo must beat and supplies the prefix/suffix mass embedding design RankNovo adopts.","marker":"Jin et al. 2024"},{"why":"Casanovo V2, a transformer-based base model and the source of the 9-species-V2 benchmark and the sequence-to-sequence translation formulation.","marker":"Yilmaz et al. 2023"},{"why":"DeepNovo, the first deep learning de novo sequencer and the source of the 9-species-V1 benchmark used for the main evaluation.","marker":"Tran et al. 2017"},{"why":"MSA Transformer: establishes axial attention over multiple sequence alignments, the architectural template for RankNovo's multi-peptide encoder.","marker":"Rao et al. 2021"},{"why":"AlphaFold: cited as the precedent for row/column attention over residue matrices, grounding the axial attention machinery.","marker":"Jumper et al. 2021"},{"why":"The dynamic-programming global alignment algorithm that PMD adapts with mass-based substitution costs and a gap penalty.","marker":"Needleman & Wunsch 1970"},{"why":"RankT5: defines the point-wise, pair-wise, and list-wise reranking frameworks that RankNovo is compared against in the framework ablation.","marker":"Zhuang et al. 2023"},{"why":"MassIVE-KB: the large public training corpus for RankNovo and all six base models.","marker":"Wang et al. 2018"},{"why":"Provides the diverse PTM spectra used to test RankNovo on acetylation, dimethylation, and phosphorylation.","marker":"Zolg et al. 2018"}],"fun_headline_variants":["RankNovo reranker lifts peptide recall to 0.660","Universal reranker surpasses all base peptide sequencers","Reranking with mass metrics beats single-model approaches","Reranker for peptide sequencing generalizes to unseen models","First deep reranking framework for de novo peptide sequencing"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the candidate pool usually contains the correct peptide: the paper never reports oracle recall or the number of candidates per spectrum, so part of the measured gain could come from pooling more hypotheses rather than from any learned ranking skill.","fun_headline_variants_meta":{"raw":{"variants":["RankNovo reranker lifts peptide recall to 0.660","Universal reranker surpasses all base peptide sequencers","Reranking with mass metrics beats single-model approaches","Reranker for peptide sequencing generalizes to unseen models","First deep reranking framework for de novo peptide sequencing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000688,"raw_usage":{"total_tokens":3185,"prompt_tokens":1082,"completion_tokens":2103,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":698,"completion_tokens_details":{"reasoning_tokens":2023}},"tokens_in":698,"tokens_out":2103,"duration_ms":16249,"temperature":1.0,"reasoning_tokens":2023,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:44:32.943541+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute, on the 9-species-V1 test set, the oracle peptide recall of the pooled beam candidates from the six base models—the fraction of spectra for which the correct peptide appears anywhere in the pool. If that oracle recall already reaches or exceeds RankNovo's 0.660, or if a trivial selector that always takes the longest or most frequent candidate matches its performance, the claim that learned list-wise reranking drives the improvement is refuted; if oracle recall is high and RankNovo falls far below it, the reranker's selection is the true bottleneck and the architecture claim stands.","supporting_citations":[{"cited_title":"Contranovo: A contrastive learning approach to enhance de novo peptide sequencing","cited_arxiv_id":null,"evidence_quote":"ContraNovo, the previous state of the art and a base model: it sets the benchmark RankNovo must beat and supplies the prefix/suffix mass embedding design RankNovo adopts."},{"cited_title":"E., Bittremieux, W., Melendez, C","cited_arxiv_id":null,"evidence_quote":"Casanovo V2, a transformer-based base model and the source of the 9-species-V2 benchmark and the sequence-to-sequence translation formulation."},{"cited_title":"M., Liu, J., Verkuil, R., Meier, J., Canny, J., Abbeel, P., Sercu, T., and Rives, A","cited_arxiv_id":null,"evidence_quote":"MSA Transformer: establishes axial attention over multiple sequence alignments, the architectural template for RankNovo's multi-peptide encoder."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The dynamic-programming global alignment algorithm that PMD adapts with mass-based substitution costs and a gap penalty."},{"cited_title":"Rankt5: Fine-tuning t5 for text ranking with ranking losses","cited_arxiv_id":null,"evidence_quote":"RankT5: defines the point-wise, pair-wise, and list-wise reranking frameworks that RankNovo is compared against in the framework ablation."},{"cited_title":"P., Wilhelm, M., Schmidt, T., M \\'e dard, G., Zerweck, J., Knaute, T., Wenschuh, H., Reimer, U., Schnatbaum, K., and Kuster, B","cited_arxiv_id":null,"evidence_quote":"Provides the diverse PTM spectra used to test RankNovo on acetylation, dimethylation, and phosphorylation."}],"review_version":1}