{"id":"6e079c10-2428-418c-bdea-aca764f49de8","arxiv_id":"2506.08954","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Protriever trains a retriever and a protein language model together so the model learns which homologs to retrieve, reaching state-of-the-art zero-shot fitness prediction on ProteinGym with much faster retrieval.","lead":"Protriever is a protein AI system that learns, during training, which similar sequences to pull from a database to help predict how mutations affect protein function. It matches or beats standard alignment-based methods on a common fitness benchmark while retrieving those supporting sequences roughly 100 times faster.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed benefit of end-to-end retrieval is not yet isolated: every end-to-end row jointly retrains the PoET reader, so the 0.479 gains over frozen DPR and frozen-PoET baselines could be reader warm-start rather than learned retrieval.","rationale":"I read the paper's central claim as: joint optimization of retrieval and reading improves zero-shot fitness prediction beyond fixed-retrieval pipelines, and does so by making the retriever select better homologs. The DPR pretraining, Faiss speedup measurements, and qualitative retrieval-identity distributions are independent supporting evidence and are not in dispute. The EMDR top-K/stale-index limitation flagged by the reader is real but secondary: with a DPR-initialized retriever, informative homologs are plausibly already inside the top-K neighborhood, so the practical claim does not require reaching sequences that currently rank below K. The load-bearing gap is that no experiment separates retriever training from reader training. Because the final fitness score is a ratio of reader log-likelihoods under one conditioning set, and because Table 3 gives Protriever a freshly trained reader while MSA baselines use frozen PoET, the reported improvements are compatible with a better-calibrated reader rather than a better retriever. This does not refute the method; it means the core attribution is unproven as written. A 2x2 ablation would settle it. The reader's conditional verdict already captures this uncertainty, so I recommend no change.","tokens_in":25581,"tokens_out":9330,"duration_ms":100762,"concrete_test":"Run a 2x2 ablation on ProteinGym with a single fixed inference/ensembling scheme: retriever in {frozen DPR, joint-trained Protriever} x reader in {frozen pretrained PoET, jointly trained PoET}. If the frozen-reader/trained-retriever cell is close to frozen-reader/frozen-DPR, the Table 2 gain is reader-side; if it reproduces most of the 0.479 gap, the retriever is the active ingredient. Report Spearman by MSA depth and a paired bootstrap confidence interval over the 217 assays, and use the same interval to state whether Protriever actually exceeds PoET (0.470).","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Table 2, the End-to-End column turns on retriever training and reader (PoET) training simultaneously. The Frozen ESM and Frozen DPR rows keep the reader at its pretrained initialization, while the two Protriever rows update both the retriever and the reader under the hybrid reconstruction + EMDR objective. The 0.479 vs 0.440 gap therefore cannot be attributed to the retriever component: the reader is also trained for many additional steps with retrieved-context conditioning. This matters because in the scoring equation of §4.1 the retriever probability cancels when all variants share the same conditioning set DK; the retriever influences the final fitness ranking only through the discrete choice of conditioning set, while the reader supplies the entire per-position log-likelihood. Table 3 compounds the confound by comparing a jointly trained Protriever reader against a frozen PoET reader for the MMseqs2 and JackHMMER rows. A reader fine-tuned on Protriever's retrieval distribution will score better on that distribution than a frozen reader, independent of whether the retrieval is better. The low-MSA-depth result (Protriever 0.464 vs published PoET 0.478, Table C.1) is consistent with the alternative reading that the retriever adds little where alignment-based homologs are scarce. Thus the central novelty, that differentiable retrieval discovers more useful homologs rather than the jointly trained reader being better calibrated, is not yet supported by the reported experiments.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"Protriever replaces MSA-based homolog retrieval with a learned dense retriever initialized from ESM-2 and pretrained with DPR, backed by a Faiss index, and feeds the retrieved top-K sequences to a PoET reader. The retriever and reader are trained jointly with a hybrid reconstruction-plus-EMDR objective. On the 217-assay ProteinGym substitution benchmark, the paper reports a Spearman correlation of 0.479, compared with 0.470 for PoET, and claims retrieval that is two orders of magnitude faster than MMseqs2 and JackHMMER. Table 2 presents ablations across ESM/DPR initialization and end-to-end training; Table 3 compares retrieval methods with respect to fitness prediction and speed.","tokens_in":25919,"tokens_out":4757,"duration_ms":52834,"significance":"If the central claim were established, Protriever would be a meaningful advance: it would show that task-aware retrieval can replace alignment-based homology search for fitness prediction, with large inference-time speedups and competitive zero-shot accuracy. The manuscript has clear strengths: it provides a reproducible code link, a detailed appendix on Faiss index construction and parameter selection, retrieval-speed benchmarks, and an alternative Fusion-in-Decoder reader experiment. However, the experimental design as reported does not isolate the effect of learned retrieval from the effect of reader fine-tuning, and the low-depth performance relative to PoET is inconsistent with the paper's stated narrative. The contribution is therefore interesting but not yet supported by the evidence presented.","major_comments":[{"comment":"The ablation protocol does not isolate the retriever. In Table 2, the Frozen ESM and Frozen DPR rows evaluate the reader at its pretrained initialization, while both Protriever rows retrain the PoET reader for the full end-to-end procedure under the hybrid reconstruction-plus-EMDR objective. Because the retriever probability cancels out of the fitness score whenever the mutated and wild-type sequences share the same conditioning set (§4.1), all per-position predictive signal in the final ranking comes from the reader. The improvements from 0.432/0.440 to 0.466/0.479 can therefore be explained by reader fine-tuning on the retrieved-context distribution rather than by improved retrieval. A control that trains the reader end-to-end while keeping the ESM or DPR retriever frozen is needed to attribute the gain to the retriever.","section":"Table 2 and §5.1"},{"comment":"Table 3 compares the Protriever row, whose PoET reader is trained end-to-end, with MMseqs2, MMseqs2-GPU, and JackHMMER rows that use a frozen off-the-shelf PoET reader. A reader fine-tuned on Protriever's retrieval distribution will score better on that distribution than an unmodified pretrained reader, independent of whether the retrieved set is more informative. To support the claim that joint retrieval training causes the speed-quality advantage, the authors should retrain or fine-tune the same reader on the MMseqs2 and JackHMMER retrieved sets, or alternatively evaluate Protriever's retrieval with the frozen PoET reader.","section":"Table 3 and §5.2"},{"comment":"The EMDR loss in Eq. (3) sums only over the current top-K retrieved set D_K and applies a stop-gradient to p_LM, so the retriever receives learning signal only for re-ranking documents already in its top-K; it is never rewarded for moving a below-K homolog into the conditioning set. Consequently, the paper's claims that joint training lets the model 'discover informative homologs' and 'adjust sequence relationships' (§1, §5.1) overstate what the loss can do. At most, the retriever can re-weight its existing neighborhood. The authors should either provide an analysis of homologs that enter the top-K after end-to-end training and that are absent from alignment-based retrievals, or soften the discovery claim.","section":"Equation (3) and §5.1"},{"comment":"The low-depth results contradict the narrative that learned retrieval is most valuable where MSAs are scarce. Table C.1 shows Protriever at 0.464 Spearman on low-depth assays versus 0.478 for PoET, while Table 2's low-depth gains are relative only to Frozen ESM/DPR baselines. Since the paper's stated advantage over alignment-based methods is central, the authors should explain this discrepancy or provide per-depth comparisons that control for reader training.","section":"Table C.1 and §5.1"},{"comment":"The headline state-of-the-art claim rests on a 0.479 versus 0.470 gap over PoET, but no confidence intervals, bootstrap standard errors, or paired per-assay significance tests are reported. Given the small magnitude of the difference and the known heterogeneity of ProteinGym assays, some uncertainty quantification is needed before claiming state-of-the-art performance.","section":"Table 1 and §4.3"}],"minor_comments":[{"comment":"The word 'Proriever' appears in the sentence describing the FiD training setup; it should be 'Protriever'.","section":"Appendix B.1"},{"comment":"The text says 'we first build an index of all protein sequences in our database' and then 'At inference time, we use the trained retriever from Protriever to encode all 62 million UniRef50 sequences.' This is ambiguous about whether the inference-time index is rebuilt with the final trained retriever or reused from training; please clarify the indexing procedure at evaluation time.","section":"§4.1"},{"comment":"The caption reports average Spearman correlation by MSA depth but does not state the number of assays in each depth bin; adding these counts would help the reader assess the stability of the per-depth numbers.","section":"Table 2 caption"},{"comment":"The statement that 'no prior work has achieved end-to-end joint training of retrieval and protein sequence modeling' is strong; please cite or explicitly distinguish the most relevant concurrent or closely related works (e.g., AIDO.RAG and differentiable MSA construction methods discussed earlier in the same section).","section":"§2.4"}],"recommendation":"major_revision","confidential_remarks":"The central issue is experimental attribution rather than methodological validity. A reader-trained-with-frozen-retriever control and reader-retrained MSA baselines would directly address the main confound; the low-depth discrepancy in Table C.1 also needs a clear explanation. I would be receptive to a revised version that includes these controls."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a well-engineered system with a genuine speed advantage, but the headline intellectual claim—that end-to-end retrieval discovers better homologs than MSA search—is not yet isolated by the experiments. The two-orders-of-magnitude speedup is real; the 0.479 vs 0.470 Spearman gain over PoET is small, has no error bars, and is confounded.\n\nWhat's new and good: Protriever is the first system I know that trains a dense retriever and a protein sequence model jointly for fitness prediction, using an EMDR-style loss with PoET as the reader. The DPR pretraining on UniRef50, the Faiss IVF-PQ index with periodic re-indexing, and the inference ensembling are substantial and clearly described. The speed benchmark is honest and reproducible-looking: 0.005 s/query vs 0.6 s for MMseqs2-GPU and 15 s for CPU MMseqs2 is a genuine advance for proteome-scale scanning. The FiD results in Appendix B show the framework is not PoET-specific, which is real plus. Code is released.\n\nThe soft spots are real and central. In Table 2, the End-to-End column turns on joint training of the retriever and the reader simultaneously. The Frozen ESM and Frozen DPR rows keep the reader at its pretrained initialization, while the Protriever rows fine-tune the reader on retrieved-context conditioning. So the 0.479 vs 0.440 gap could be mostly reader warm-start, not better retrieval. Table 3 compounds this: the MMseqs2 and JackHMMER rows use a frozen PoET reader, while Protriever's row uses a reader fine-tuned on its own retrieval distribution. A matched comparison—fine-tuning PoET on MMseqs2-retrieved contexts, or evaluating frozen PoET on Protriever retrieval—is missing. The low-MSA-depth result (0.464 vs published PoET's 0.478 in Table C.1) also undercuts the claim that learned retrieval helps most where alignments are scarce.\n\nNone of this sinks the speed contribution, and the paper is honest about its components. But the central narrative—that differentiable retrieval finds more useful homologs rather than the jointly trained reader being better calibrated—is not yet supported. The fix is straightforward: additional ablations that separate reader fine-tuning from retriever learning, plus uncertainty estimates.\n\nI'd send this to review. It is a serious and well-engineered system, and the confound is addressable. I'd also bring it to reading group, because it is a great example of how easy it is to conflate retrieval quality with reader adaptation in RAG-style systems.","headline":"Solid engineering and a real speed win, but the paper's central claim that learned retrieval beats MSA-based search is confounded by simultaneous reader fine-tuning.","tokens_in":26512,"tokens_out":4273,"would_cite":true,"duration_ms":45256,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Protriever claims that learning homolog retrieval together with the fitness task outperforms alignment-based retrieval and runs orders of magnitude faster.","keywords":["protein fitness prediction","homology search","differentiable retrieval","dense passage retrieval","multiple sequence alignment","protein language models","vector similarity search","zero-shot mutation effect prediction"],"falsifier":"For low-MSA-depth ProteinGym queries where the frozen ESM-2/DPR encoder initially ranks a known functional homolog outside the top-K retrieved set, run Protriever's joint training and check whether that homolog ever enters the conditioning set: if it never does, the top-K EMDR sum cannot have supplied the gradient to discover it, and the claim that joint training discovers missed homologs would be falsified for that case.","tokens_in":25334,"feed_emoji":"🧬","tokens_out":9781,"duration_ms":97874,"temperature":0.7,"pith_summary":"Protriever argues that protein homology search should be learned jointly with the task it serves rather than fixed ahead of time by sequence alignment. It replaces MSA construction with dense vector retrieval: a trained encoder embeds a query protein, searches a 62-million-sequence index, and hands the retrieved homologs to a language-model reader that scores mutations. The paper reports that this end-to-end scheme reaches a Spearman correlation of 0.479 across 217 ProteinGym substitution assays, the best among sequence-based models and ahead of the prior leader PoET (0.470), while retrieving homologs roughly two orders of magnitude faster than MMseqs2 and JackHMMER. If this holds, alignment-free learned retrieval can replace MSA pipelines for fitness prediction and scale to proteome-wide analysis.","feed_headline":"Learned retrieval tops MSA pipelines for protein fitness","feed_subtitle":"Protriever scores 0.479 Spearman across 217 ProteinGym assays and retrieves homologs ~100x faster than MMseqs2.","key_machinery":"The load-bearing object is the end-to-end retriever-reader loop trained with the EMDR loss. The retriever is a transformer encoder initialized from ESM-2, average-pooled to a 480-dimensional embedding, and pretrained with DPR so homologous sequences sit closer in cosine space. It searches a Faiss inverted-file plus product-quantization index of about 62 million UniRef50 embeddings; the top-K retrieved sequences are filtered and sampled into conditioning sets for a PoET reader, which autoregressively reconstructs the query. The EMDR loss treats the retrieved set as a latent variable: it sums over the top-K sequences the product of the reader's relevance $p_{\\mathrm{LM}}(q\\mid d_k)$ and the retriever's softmax score $p_{\\mathrm{RETR}}(d_k\\mid q)$, with a stop-gradient on the reader term so only the retriever is updated. That gradient pulls sequences that helped the reader closer to the query, making homology search itself learn from the fitness task.","core_discovery":"The central discovery is that a retriever trained end-to-end with a reader learns which homologous sequences are informative for reconstructing a query, and that this learned choice improves zero-shot fitness prediction relative to fixed alignment-based retrieval. Protriever initializes the retriever from ESM-2, pretrains it with a Dense Passage Retrieval contrastive objective on UniRef50, and then trains it with the EMDR loss, which weighs each retrieved sequence by how much it helps the reader predict the query while backpropagating only into the retriever. The ablations show each ingredient contributes: frozen ESM embeddings score 0.432, adding DPR reaches 0.440, joint training without DPR reaches 0.466, and the full combination reaches 0.479, with the largest gains at low MSA depth. The paper also reports that the trained retriever can be paired with different reader architectures and databases without retraining, and that vector search retrieves homologs in milliseconds rather than seconds or hours.","pith_inferences":["A consequence the paper leaves implicit is that the EMDR gradient only re-ranks sequences already in the top-K neighborhood, so Protriever's ceiling depends on the frozen ESM-2/DPR embedding space already placing informative homologs near the query; exploration beyond top-K would be a direct stress test.","The same retriever-reader loop should transfer to other conditioning tasks such as masked language modeling or supervised property prediction, with the reader's loss redefining what counts as a useful homolog.","A testable extension is to evaluate on deliberately remote homologs (for example, below 30% sequence identity) to see whether learned retrieval finds functional relationships that alignment significance thresholds miss, which is the paper's qualitative claim about distant homologs."],"forward_implications":["On the 217-assay ProteinGym substitution benchmark, Protriever's 0.479 Spearman correlation is the best sequence-based result, ahead of PoET at 0.470, with the largest gains in low-MSA-depth families.","Per-query retrieval drops to under 5 milliseconds, two orders of magnitude faster than MMseqs2-GPU and far faster still than JackHMMER, which makes proteome-wide zero-shot fitness scoring practical.","Because the index is separate from the model, new sequences can be encoded and added without retraining, and focused databases such as viral or proprietary collections can be swapped in at inference time.","The framework is architecture-agnostic: with a Fusion-in-Decoder reader at 150M parameters it reaches fitness accuracy comparable to much larger single-sequence models, showing that learned retrieval context can substitute for model scale."],"supporting_citations":[{"why":"It supplies the PoET reader backbone that conditions on retrieved homologs and serves as the strongest sequence-based baseline.","marker":"(Truong Jr & Bepler, 2023)"},{"why":"It supplies the Dense Passage Retrieval objective used to pretrain the retriever's embedding space.","marker":"(Karpukhin et al., 2020)"},{"why":"It supplies the EMDR loss whose stop-gradient on the reader turns reader usefulness into retriever updates.","marker":"(Sachan et al., 2021)"},{"why":"It supplies ESM-2 weights from which the retriever encoder is initialized.","marker":"(Lin et al., 2023)"},{"why":"It supplies the MMseqs2 alignment-search baseline for speed and fitness comparisons.","marker":"(Steinegger & Soding, 2017)"},{"why":"It supplies the JackHMMER baseline that anchors the sensitivity-versus-cost comparison.","marker":"(Potter et al., 2018)"},{"why":"It supplies the Faiss vector-search implementation (IVF plus product quantization) that makes 62-million-sequence retrieval fast.","marker":"(Johnson et al., 2021)"},{"why":"It supplies the ProteinGym benchmark and its 217 deep mutational scanning assays used for evaluation.","marker":"(Notin et al., 2023)"},{"why":"It supplies the log-likelihood-ratio fitness scoring method and the low/medium/high MSA-depth stratification used in all evaluations.","marker":"(Frazer et al., 2021)"}],"fun_headline_variants":["Learned retrieval tops MSA for protein fitness prediction","Protriever: end-to-end homolog search speeds fitness tasks","Differentiable retrieval boosts fitness prediction over alignment","Protriever finds better homologs 100x faster than MSA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's gain rests on informative homologs already sitting inside the top-K embedding neighborhood at training time, because the EMDR gradient only re-ranks currently retrieved sequences and the 62-million-sequence index is refreshed only every 5,000 steps.","fun_headline_variants_meta":{"raw":{"variants":["Learned retrieval tops MSA for protein fitness prediction","Protriever: end-to-end homolog search speeds fitness tasks","Differentiable retrieval boosts fitness prediction over alignment","Protriever finds better homologs 100x faster than MSA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000202,"raw_usage":{"total_tokens":1381,"prompt_tokens":941,"completion_tokens":440,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":557,"completion_tokens_details":{"reasoning_tokens":373}},"tokens_in":557,"tokens_out":440,"duration_ms":5247,"temperature":1.0,"reasoning_tokens":373,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:58:39.733038+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"For low-MSA-depth ProteinGym queries where the frozen ESM-2/DPR encoder initially ranks a known functional homolog outside the top-K retrieved set, run Protriever's joint training and check whether that homolog ever enters the conditioning set: if it never does, the top-K EMDR sum cannot have supplied the gradient to discover it, and the claim that joint training discovers missed homologs would be falsified for that case.","supporting_citations":[],"review_version":1}