Pith. sign in

REVIEW 5 major objections 4 minor 1 cited by

Protriever: End-to-End Differentiable Protein Homology Search for Fitness Prediction

T0 review · 5 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Protriever claims that learning homolog retrieval together with the fitness task outperforms alignment-based retrieval and runs orders of magnitude faster.

desk verdict Solid engineering and a real speed win, but the paper's central claim that learned retrieval beats MSA-based search is confounded by simultaneous reader fine-tuning. read the letter →

arxiv 2506.08954 v1 pith:DLVEO3OK submitted 2025-06-10 q-bio.QM cs.LGq-bio.BM

Ruben Weitzman , Peter M{o}rch Groth , Lood Van Niekerk , Aoi Otani , Yarin Gal , Debora Marks , Pascal Notin This is my paper
classification q-bio.QMcs.LGq-bio.BM
keywords proteinfitnesspredictionhomologysearchdifferentiableretrievaldensepassagemultiplesequencealignmentlanguagemodelsvectorsimilarityzero-shotmutationeffect
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Protriever argues that protein homology search should be learned jointly with the task it serves rather than fixed ahead of time by sequence alignment. It replaces MSA construction with dense vector retrieval: a trained encoder embeds a query protein, searches a 62-million-sequence index, and hands the retrieved homologs to a language-model reader that scores mutations. The paper reports that this end-to-end scheme reaches a Spearman correlation of 0.479 across 217 ProteinGym substitution assays, the best among sequence-based models and ahead of the prior leader PoET (0.470), while retrieving homologs roughly two orders of magnitude faster than MMseqs2 and JackHMMER. If this holds, alignment-free learned retrieval can replace MSA pipelines for fitness prediction and scale to proteome-wide analysis.

What carries the argument

The load-bearing object is the end-to-end retriever-reader loop trained with the EMDR loss. The retriever is a transformer encoder initialized from ESM-2, average-pooled to a 480-dimensional embedding, and pretrained with DPR so homologous sequences sit closer in cosine space. It searches a Faiss inverted-file plus product-quantization index of about 62 million UniRef50 embeddings; the top-K retrieved sequences are filtered and sampled into conditioning sets for a PoET reader, which autoregressively reconstructs the query. The EMDR loss treats the retrieved set as a latent variable: it sums over the top-K sequences the product of the reader's relevance $p_{\mathrm{LM}}(q\mid d_k)$ and the retriever's softmax score $p_{\mathrm{RETR}}(d_k\mid q)$, with a stop-gradient on the reader term so only the retriever is updated. That gradient pulls sequences that helped the reader closer to the query, making homology search itself learn from the fitness task.

What would settle it

For low-MSA-depth ProteinGym queries where the frozen ESM-2/DPR encoder initially ranks a known functional homolog outside the top-K retrieved set, run Protriever's joint training and check whether that homolog ever enters the conditioning set: if it never does, the top-K EMDR sum cannot have supplied the gradient to discover it, and the claim that joint training discovers missed homologs would be falsified for that case.

Watch

Extended reading notes

Core claim

The central discovery is that a retriever trained end-to-end with a reader learns which homologous sequences are informative for reconstructing a query, and that this learned choice improves zero-shot fitness prediction relative to fixed alignment-based retrieval. Protriever initializes the retriever from ESM-2, pretrains it with a Dense Passage Retrieval contrastive objective on UniRef50, and then trains it with the EMDR loss, which weighs each retrieved sequence by how much it helps the reader predict the query while backpropagating only into the retriever. The ablations show each ingredient contributes: frozen ESM embeddings score 0.432, adding DPR reaches 0.440, joint training without DPR reaches 0.466, and the full combination reaches 0.479, with the largest gains at low MSA depth. The paper also reports that the trained retriever can be paired with different reader architectures and databases without retraining, and that vector search retrieves homologs in milliseconds rather than seconds or hours.

Load-bearing premise

The method's gain rests on informative homologs already sitting inside the top-K embedding neighborhood at training time, because the EMDR gradient only re-ranks currently retrieved sequences and the 62-million-sequence index is refreshed only every 5,000 steps.

Editorial extensions

If this is right

  • On the 217-assay ProteinGym substitution benchmark, Protriever's 0.479 Spearman correlation is the best sequence-based result, ahead of PoET at 0.470, with the largest gains in low-MSA-depth families.
  • Per-query retrieval drops to under 5 milliseconds, two orders of magnitude faster than MMseqs2-GPU and far faster still than JackHMMER, which makes proteome-wide zero-shot fitness scoring practical.
  • Because the index is separate from the model, new sequences can be encoded and added without retraining, and focused databases such as viral or proprietary collections can be swapped in at inference time.
  • The framework is architecture-agnostic: with a Fusion-in-Decoder reader at 150M parameters it reaches fitness accuracy comparable to much larger single-sequence models, showing that learned retrieval context can substitute for model scale.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A consequence the paper leaves implicit is that the EMDR gradient only re-ranks sequences already in the top-K neighborhood, so Protriever's ceiling depends on the frozen ESM-2/DPR embedding space already placing informative homologs near the query; exploration beyond top-K would be a direct stress test.
  • The same retriever-reader loop should transfer to other conditioning tasks such as masked language modeling or supervised property prediction, with the reader's loss redefining what counts as a useful homolog.
  • A testable extension is to evaluate on deliberately remote homologs (for example, below 30% sequence identity) to see whether learned retrieval finds functional relationships that alignment significance thresholds miss, which is the paper's qualitative claim about distant homologs.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. Protriever replaces MSA-based homolog retrieval with a learned dense retriever initialized from ESM-2 and pretrained with DPR, backed by a Faiss index, and feeds the retrieved top-K sequences to a PoET reader. The retriever and reader are trained jointly with a hybrid reconstruction-plus-EMDR objective. On the 217-assay ProteinGym substitution benchmark, the paper reports a Spearman correlation of 0.479, compared with 0.470 for PoET, and claims retrieval that is two orders of magnitude faster than MMseqs2 and JackHMMER. Table 2 presents ablations across ESM/DPR initialization and end-to-end training; Table 3 compares retrieval methods with respect to fitness prediction and speed.

Significance. If the central claim were established, Protriever would be a meaningful advance: it would show that task-aware retrieval can replace alignment-based homology search for fitness prediction, with large inference-time speedups and competitive zero-shot accuracy. The manuscript has clear strengths: it provides a reproducible code link, a detailed appendix on Faiss index construction and parameter selection, retrieval-speed benchmarks, and an alternative Fusion-in-Decoder reader experiment. However, the experimental design as reported does not isolate the effect of learned retrieval from the effect of reader fine-tuning, and the low-depth performance relative to PoET is inconsistent with the paper's stated narrative. The contribution is therefore interesting but not yet supported by the evidence presented.

major comments (5)
  1. [Table 2 and §5.1] The ablation protocol does not isolate the retriever. In Table 2, the Frozen ESM and Frozen DPR rows evaluate the reader at its pretrained initialization, while both Protriever rows retrain the PoET reader for the full end-to-end procedure under the hybrid reconstruction-plus-EMDR objective. Because the retriever probability cancels out of the fitness score whenever the mutated and wild-type sequences share the same conditioning set (§4.1), all per-position predictive signal in the final ranking comes from the reader. The improvements from 0.432/0.440 to 0.466/0.479 can therefore be explained by reader fine-tuning on the retrieved-context distribution rather than by improved retrieval. A control that trains the reader end-to-end while keeping the ESM or DPR retriever frozen is needed to attribute the gain to the retriever.
  2. [Table 3 and §5.2] Table 3 compares the Protriever row, whose PoET reader is trained end-to-end, with MMseqs2, MMseqs2-GPU, and JackHMMER rows that use a frozen off-the-shelf PoET reader. A reader fine-tuned on Protriever's retrieval distribution will score better on that distribution than an unmodified pretrained reader, independent of whether the retrieved set is more informative. To support the claim that joint retrieval training causes the speed-quality advantage, the authors should retrain or fine-tune the same reader on the MMseqs2 and JackHMMER retrieved sets, or alternatively evaluate Protriever's retrieval with the frozen PoET reader.
  3. [Equation (3) and §5.1] The EMDR loss in Eq. (3) sums only over the current top-K retrieved set D_K and applies a stop-gradient to p_LM, so the retriever receives learning signal only for re-ranking documents already in its top-K; it is never rewarded for moving a below-K homolog into the conditioning set. Consequently, the paper's claims that joint training lets the model 'discover informative homologs' and 'adjust sequence relationships' (§1, §5.1) overstate what the loss can do. At most, the retriever can re-weight its existing neighborhood. The authors should either provide an analysis of homologs that enter the top-K after end-to-end training and that are absent from alignment-based retrievals, or soften the discovery claim.
  4. [Table C.1 and §5.1] The low-depth results contradict the narrative that learned retrieval is most valuable where MSAs are scarce. Table C.1 shows Protriever at 0.464 Spearman on low-depth assays versus 0.478 for PoET, while Table 2's low-depth gains are relative only to Frozen ESM/DPR baselines. Since the paper's stated advantage over alignment-based methods is central, the authors should explain this discrepancy or provide per-depth comparisons that control for reader training.
  5. [Table 1 and §4.3] The headline state-of-the-art claim rests on a 0.479 versus 0.470 gap over PoET, but no confidence intervals, bootstrap standard errors, or paired per-assay significance tests are reported. Given the small magnitude of the difference and the known heterogeneity of ProteinGym assays, some uncertainty quantification is needed before claiming state-of-the-art performance.
minor comments (4)
  1. [Appendix B.1] The word 'Proriever' appears in the sentence describing the FiD training setup; it should be 'Protriever'.
  2. [§4.1] The text says 'we first build an index of all protein sequences in our database' and then 'At inference time, we use the trained retriever from Protriever to encode all 62 million UniRef50 sequences.' This is ambiguous about whether the inference-time index is rebuilt with the final trained retriever or reused from training; please clarify the indexing procedure at evaluation time.
  3. [Table 2 caption] The caption reports average Spearman correlation by MSA depth but does not state the number of assays in each depth bin; adding these counts would help the reader assess the stability of the per-depth numbers.
  4. [§2.4] The statement that 'no prior work has achieved end-to-end joint training of retrieval and protein sequence modeling' is strong; please cite or explicitly distinguish the most relevant concurrent or closely related works (e.g., AIDO.RAG and differentiable MSA construction methods discussed earlier in the same section).

Circularity Check

0 steps flagged · score 2.0 of 10

No material circularity: fitness labels are external and unused in training; the retriever-score cancellation in Section 4.1 is an explicit reduction, not a hidden equivalence.

full rationale

Protriever's derivation chain is self-contained rather than circular. Fitness predictions are evaluated on ProteinGym DMS measurements that originate from external experimental assays, and these labels are never used as a training signal. The retriever is pretrained with DPR using BLAST-derived homology labels, and the reader is trained with autoregressive reconstruction; the EMDR loss in Equation (3) updates only the retriever via a stop-gradient on pLM, so both components are trained with self-supervised objectives that do not incorporate the target fitness labels. The scoring identity in Section 4.1, where the retriever probability cancels when mutant and wild-type share the same conditioning set DK, is an explicit mathematical reduction, not a hidden circularity: the retriever's influence is limited to the discrete selection of DK, and the fitness ranking is produced by the reader's likelihood ratio. The comparisons against MSA-based retrieval in Table 3 use the same PoET reader architecture, and the ablations in Table 2 show monotone gains; although the jointly trained reader is also updated in the Protriever rows, that is an experimental confound about attribution of the gain, not a circular dependence. The only self-citations (ProteinGym, Tranception, TranceptEVE) are benchmark and baseline references that are externally checkable, and the underlying DMS data are third-party experimental measurements, so they are not load-bearing. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors, and no ansatz is smuggled via citation. Score 2 reflects the presence of minor, non-load-bearing self-citations; the central derivation is independent.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The framework relies on several hyperparameters (temperature, sampling weights, context sizes, index parameters) tuned on validation sets, and on domain assumptions shared with all homology-based fitness models. No invented physical entities are introduced. The main risk is that the EMDR loss only re-ranks within the top-K neighborhood, limiting the claim that the retriever discovers novel distant homologs.

free parameters (5)
  • Retriever softmax temperature theta
    In pRETR(d | q) in Equation 2; no value or tuning procedure is reported in the paper.
  • Cluster sampling weights a and T = five hand-chosen combinations
    Appendix E defines cluster sampling weights sqrt(s) * (1 + exp(-a d / T))^{-1}; five combinations are ensembled at inference.
  • Conditioning set sizes = 6,144 / 12,288 / 24,576 tokens
    Three context lengths ensembled at inference (Section 4.1, Appendix E).
  • Sequence identity filter threshold = 15%
    Sequences below 15% identity to the query are removed before sampling (Appendix E).
  • Faiss index parameters K_IVF, P_IVF, code size M, bits per sub-vector = K_IVF in {3941, 15764, 31528}, P_IVF=2048, M in {32, 48, 96}, 8 bits
    Selected by recall and speed validation in Appendix A.2; these affect retrieval quality and therefore downstream fitness scores.
assumptions (5)
  • domain assumption The conditional likelihood ratio log P(x_mut | D_K) / P(x_wt | D_K) is a valid zero-shot fitness score.
    Invoked in Section 4.1 following Frazer et al. and Notin et al.; the model is never trained on DMS labels, so the validity of this score is assumed.
  • domain assumption Homologous sequences in the retrieval database capture evolutionary constraints that transfer to functional fitness.
    Central premise of homology-based fitness prediction (Section 1).
  • ad hoc to paper The EMDR loss with a stop-gradient on pLM provides a useful training signal for the retriever.
    Equation 3 is the paper's core training objective; the paper does not prove convergence or that this signal improves retrieval for fitness, only reports empirical gains in Table 2.
  • ad hoc to paper x_mut and x_wt share the same retrieved conditioning set D_K so the retriever probabilities cancel.
    Stated as an approximation in Section 4.1 ('in practice'); it can fail for multi-mutants or indels, making the simplified fitness score inexact.
  • standard math Approximate nearest neighbor search via IVF and PQ preserves retrieval quality sufficient for fitness prediction.
    Appendix A.2 measures recall for index configurations; the choice of IVFPQ96x8 is an approximation to exact search.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Protriever: End-to-End Differentiable Protein Homology Search for Fitness Prediction." pith.science (2026). https://pith.science/paper/DLVEO3OK

@misc{pith2026250608954,
  author       = {Pith},
  title        = {Pith review of: Protriever: End-to-End Differentiable Protein Homology Search for Fitness Prediction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DLVEO3OK}},
  note         = {Machine review of arXiv:2506.08954}
}
read the original abstract

Retrieving homologous protein sequences is essential for a broad range of protein modeling tasks such as fitness prediction, protein design, structure modeling, and protein-protein interactions. Traditional workflows have relied on a two-step process: first retrieving homologs via Multiple Sequence Alignments (MSA), then training models on one or more of these alignments. However, MSA-based retrieval is computationally expensive, struggles with highly divergent sequences or complex insertions & deletions patterns, and operates independently of the downstream modeling objective. We introduce Protriever, an end-to-end differentiable framework that learns to retrieve relevant homologs while simultaneously training for the target task. When applied to protein fitness prediction, Protriever achieves state-of-the-art performance compared to sequence-based models that rely on MSA-based homolog retrieval, while being two orders of magnitude faster through efficient vector search. Protriever is both architecture- and task-agnostic, and can flexibly adapt to different retrieval strategies and protein databases at inference time -- offering a scalable alternative to alignment-centric approaches.

Figures

Figures reproduced from arXiv: 2506.08954 by the authors.

Figure 1
Figure 1. Protriever. The Protriever framework is composed of three parts: a learned retriever, an index, and a reader. The two neural networks work together to produce a conditional sequence likelihood. The retriever selects a set of sequences DK = {dk}1,...,K to be passed on to the reader using vector similarity search between the embedded query sequence and retrieval index. This set of sequences is then passed on to the re… view at source ↗
Figure 2
Figure 2. Retrieval speed and quality. Left: Retrieval time per query sequence (mean and standard error) at different query sizes using embedding similarity search, MMseqs2, and GPU-accelerated MMseqs2. Benchmarking details and tabulated values can be found in Appendix A.3 and Table A.2, respectively. Right: Distribution of sequence identities between queries and retrieved sequences for Protriever, MMSeqs2, and JackHMMER, usi… view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Knowledge before Reasoning: EC-Reason-Bench, a Training-Free Diagnostic Benchmark for LLM Enzyme Classification

    cs.CL 2026-07 conditional novelty 6.0 of 10

    For LLM enzyme classification, external homology evidence is what drives accuracy; reasoning wrappers add almost nothing on aggregate, giving opposite gains and losses on adversarial versus multi-functional enzyme splits.

Reference graph

Works this paper leans on

13 extracted references · 7 canonical work pages · cited by 1 Pith paper

  1. [2]

    Publisher: Nature Publishing Group

    URL https://www.nature.com/articles/ s41592-022-01700-2 . Publisher: Nature Publishing Group. Ma, C., Zhao, H., Zheng, L., Xin, J., Li, Q., Wu, L., Deng, Z., Lu, Y . Y ., Liu, Q., Wang, S., and Kong, L. Retrieved Se- quence Augmentation for Protein Representation Learning. In Al-Onaizan, Y ., Bansal, M., and Chen, Y .-N. (eds.),Proceed- ings of the 2024 C...

    arXiv 2024

  2. [8]

    doi: 10.1016/j.cels.2023.10

    ISSN 2405-4712, 2405-4720. doi: 10.1016/j.cels.2023.10

    work page doi:10.1016/j.cels.2023.10 2023

  3. [9]

    Publisher: Else- vier

    URL https://www.cell.com/cell-systems/ abstract/S2405-4712(23)00272-7. Publisher: Else- vier. Notin, P., Dias, M., Frazer, J., Marchena-Hurtado, J., Gomez, A., Marks, D. S., and Gal, Y . Tranception: protein fitness pre- diction with autoregressive transformers and inference-time re- trieval, May 2022. URL http://arxiv.org/abs/2205. 13760. arXiv:2205.1376...

    arXiv 2022

  4. [13]

    Publisher: Nature Publishing Group

    URL https://www.nature.com/articles/ s42256-022-00465-9 . Publisher: Nature Publishing Group. Wang, S., Yu, M., Guo, X., Wang, Z., Klinger, T., Zhang, W., Chang, S., Tesauro, G., Zhou, B., and Jiang, J. R3: re- inforced ranker-reader for open-domain question answering. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thir...

    2024

  5. [63]

    doi: 10.1016/S0167-7306(08)60461-5

    January 1998. doi: 10.1016/S0167-7306(08)60461-5. URL https://www.sciencedirect.com/science/ article/pii/S0167730608604615. Lee, K., Chang, M.-W., and Toutanova, K. Latent Retrieval for Weakly Supervised Open Domain Question Answering. In Ko- rhonen, A., Traum, D., and M`arquez, L. (eds.), Proceedings of the 57th Annual Meeting of the Association for Comp...

    work page doi:10.1016/s0167-7306(08)60461-5 1998

  6. [1696]

    URL https://www.nature

    doi: 10.1038/nbt.3988. URL https://www.nature. com/articles/nbt.3988. Publisher: Nature Publishing Group. Suzek, B. E., Wang, Y ., Huang, H., McGarvey, P. B., and Wu, C. H. UniRef clusters: a comprehensive and scalable alter- native for improving sequence similarity searches. Bioinfor- matics, 31(6):926–932, March 2015. ISSN 1367-4803. doi: 10.1093/bioinf...

    doi:10.1038/nbt.3988

  7. [1994]

    doi: 10.1002/prot.340180402

    ISSN 1097-0134. doi: 10.1002/prot.340180402. URL https://onlinelibrary.wiley.com/ doi/abs/10.1002/prot.340180402. eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/prot.340180402. Haddox, H. K., Dingens, A. S., Hilton, S. K., Overbaugh, J., and Bloom, J. D. Mapping mutational effects along the evolutionary landscape of HIV envelope. eLife, 7:e34420,...

    arXiv

  8. [2015]

    Dao, T., Fu, D

    doi: 10.1038/nmeth.3176. Dao, T., Fu, D. Y ., Ermon, S., Rudra, A., and R ´e, C. FlashAt- tention: Fast and Memory-Efficient Exact Attention with IO- Awareness. In Advances in Neural Information Processing Systems (NeurIPS), 2022. Douze, M., Guzhva, A., Deng, C., Johnson, J., Szilvasy, G., Mazar´e, P.-E., Lomeli, M., Hosseini, L., and J ´egou, H. The Fais...

    arXiv

Show all 13 references
  1. [2020]

    ISSN: 2640-3498

    URL https://proceedings.mlr.press/ v119/guu20a.html. ISSN: 2640-3498. G¨obel, U., Sander, C., Schneider, R., and Valencia, A. Corre- lated mutations and residue contacts in proteins. Proteins: Structure, Function, and Bioinformatics , 18(4):309–317,

  2. [2022]

    doi: 10.1038/s42256-022-00465-

    ISSN 2522-5839. doi: 10.1038/s42256-022-00465-

    doi

  3. [2023]

    doi: 10.1038/s41592-022-01700-

    ISSN 1548-7105. doi: 10.1038/s41592-022-01700-

    doi

  4. [2105]

    URL https://doi

    doi: 10.1186/1471-2105-11-431. URL https://doi. org/10.1186/1471-2105-11-431 . Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ron- neberger, O., Tunyasuvunakool, K., Bates, R., ˇZ´ıdek, A., Potapenko, A., Bridgland, A., Meyer, C., Kohl, S. A. A., Bal- lard, A. J....

  5. [4673]

    URL https://doi.org/10.1093/nar/22.22. 4673. Tripathi, A., Gupta, K., Khare, S., Jain, P. C., Patel, S., Kumar, P., Pulianmackal, A. J., Aghera, N., and Varadarajan, R. Molecular Determinants of Mutant Phenotypes, Inferred from Saturation Mutagenesis Data. Molecular Biology an...

    doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.