Protein Language Model Zero-Shot Fitness Predictions are Improved by Inference-only Dropout

Aditya Ravuri; Neil D. Lawrence

REVIEW 4 major objections 5 minor 8 references

Protein Language Model Zero-Shot Fitness Predictions are Improved by Inference-only Dropout

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Inference-only dropout between embedding and transformer, averaged over forward passes, improves zero-shot protein fitness predictions on ProteinGym.

desk verdict A cheap and plausible trick that might help zero-shot fitness ranking, but the current evaluation rests on an unusual scoring function and no validation split, so the headline claim is not yet established. read the letter →

arxiv 2506.14793 v1 pith:22YIME4L submitted 2025-05-31 cs.LG

Aditya Ravuri , Neil D. Lawrence This is my paper

classification cs.LG

keywords proteinlanguagemodelszero-shotfitnesspredictioninference-onlydropoutMonteCarloESM2Gymmodelcalibrationout-of-domaindetection

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a frozen protein language model can be turned into a better zero-shot fitness predictor without any retraining: insert a dropout layer between the embedding and transformer blocks, run several stochastic forward passes, and average the output log-probabilities. On a 50-family subset of the ProteinGym DMS substitution benchmark, this inference-only Monte-Carlo dropout improves the median Spearman rank correlation across the ESM2 model sizes tested, with the largest relative gain for the 35M-parameter model. A dropout rate of 0.1 appears to work across models, and for the larger 150M and 15B models dropout layers also had to be inserted into early transformer layers. The authors suggest the gain comes from improved calibration of the model's output distributions. If correct, the result upgrades existing frozen PLMs at near-zero cost.

What carries the argument

The mechanism is inference-only Monte-Carlo dropout: a dropout layer inserted between the embedding and transformer blocks of an ESM2 model, with 100 stochastic forward passes averaged at test time. The scores come from $S(L)=\sum_{ij}L_{ij}$, the sum of per-position log-probabilities, which the authors interpret as a proxy for output entropy and out-of-domainness. Dropout-averaging smooths this proxy, and for the largest models additional dropout must be placed in the first fifth of transformer layers for the 150M model and the first third for the 15B model to realize the gain.

What would settle it

Evaluate the same protocol on the full ProteinGym DMS substitution benchmark with the wildtype-relative scoring from Hsu et al. (2022) rather than $S(L)=\sum_{ij}L_{ij}$; if the dropout-averaged model's median Spearman correlation is not above the no-dropout baseline under that score, the claimed fitness improvement would be refuted.

Watch

Extended reading notes

Core claim

The paper's central claim is that injecting a dropout layer at inference time between the featurizer/embedding layer and the transformer of a protein language model, then averaging the output log-probability matrices over 100 stochastic forward passes, increases zero-shot fitness prediction performance on a subset of the ProteinGym dataset. This holds even for models that were never trained with dropout, so no retraining or finetuning is required. The effect is observed for every ESM2 model size tested, with the largest relative improvement for the 35M-parameter model; in the 150M and 15B cases, dropout also had to be added to some early transformer layers. The authors use the scoring function $S(L)=\sum_{ij}L_{ij}$ on the log-probability matrix $L$, and report the median Spearman rank correlation per protein family, with a dropout rate of 0.1 performing best across all models.

Load-bearing premise

The load-bearing premise is that $S(L)=\sum_{ij}L_{ij}$, the sum of all per-position log-probabilities, is a valid proxy for protein fitness on the ProteinGym benchmark. If this entropy-like sum mostly measures output dispersion or out-of-domainness rather than mutation effects, the dropout gains may not reflect improved fitness prediction.

Editorial extensions

If this is right

Frozen ESM2 models can be upgraded to better zero-shot fitness predictors by averaging stochastic forward passes, with no changes to pretraining or finetuning.
A dropout rate of 0.1 transfers across model sizes, making the method easy to apply without per-model tuning.
The improvement holds even for models not trained with dropout, so the benefit comes from inference-time stochasticity rather than matching the training distribution.
The method is computationally modest: 100 forward passes per sequence at inference time, replacing a single deterministic pass.
Applying the same inference-only dropout to larger models requires placing dropout in early transformer layers, not just at the embedding output.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper's gains are measured with an entropy-like score, so a natural extrapolation is that dropout-averaging primarily improves calibration and out-of-domain detection; whether it improves true mutation-effect ranking should be tested with wildtype-relative joint log-probability scoring.
The mechanism likely transfers beyond ESM2 and ProteinGym: any masked language model used for zero-shot property prediction could be averaged over dropout-injected forward passes.
A cheap extension would be to measure how the median Spearman correlation depends on the number of Monte-Carlo samples; 100 may be far more than necessary, and a smaller budget would make the method practical for high-throughput screens.
Because dropout-averaging behaves like a lightweight ensemble, it may combine with other post-hoc calibration methods rather than replace them.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Let me say plainly: this is a short, simple empirical claim that could be a useful trick, but the current evaluation doesn't support the headline. The new thing is applying Monte-Carlo dropout at inference to frozen PLMs for zero-shot fitness, and showing that on their chosen metric it helps, especially for the small ESM2 35M model. The method is trivial to implement and requires no retraining. They also deserve credit for admitting in footnote 2 that a wildtype-relative score would be more appropriate.

The soft spots are not small. First, their fitness proxy S(L)=sum_ij L_ij is not the standard mutation-effect score; it's a sum of log-probs over all tokens, effectively an entropy-like measure. The paper itself says a proper score should compare joint log-probability relative to the wildtype. If the improvement only shows up with this proxy, then "zero-shot fitness predictions are improved" is misleading. They never test an alternative scoring. Second, the 50-family subset is unspecified, there are no error bars or significance tests, and the dropout rate and insertion location are selected after looking at the same test set, so "0.1 seems performant" may be overfitting. Third, for the 150M and 15B models they added dropout in early transformer layers, which the abstract omits. That makes the headline less general than it sounds.

I'm not saying the effect is fake. MC dropout can improve calibration, and that might help ranking under a proper scoring rule too. But a paper like this needs at least the standard test: use the wildtype-relative log-likelihood score, report SRCC with confidence intervals, and hold out a validation split for hyperparameters. As it stands, the central claim is not established.

Still, it's a reasonable candidate for peer review. The idea is cheap to test, and a good referee could push the authors to fix the evaluation. It's short, not a major advance, but not empty either. I'd send it out, with a strong recommendation to add the missing checks. If the effect survives, it's a nice practical note; if not, the paper fails cleanly.

Referee Report

4 major / 5 minor

Summary. The paper proposes an inference-only modification to protein language models: insert a dropout layer between the embedding layer and the transformer block of ESM2, run multiple stochastic forward passes, and average the resulting log-probability matrices. The averaged outputs are then scored with S(L) = sum_ij L_ij, and the Spearman rank correlation of this proxy against fitness labels is measured on a subset of 50 ProteinGym DMS families. The authors report that this Monte-Carlo-style dropout averaging improves median SRCC for all ESM2 model sizes tested, with dropout rate 0.1 performing well, and they suggest the improvement stems from improved calibration or out-of-domainness ranking. No retraining or fine-tuning is required.

Significance. The proposed intervention is simple, cheap, and potentially useful: if correct, it would allow frozen ESM2 checkpoints to be converted into better zero-shot fitness predictors at inference time only. The paper has several concrete strengths: it evaluates on multiple model sizes (35M, 150M, 15B), uses 100 Monte-Carlo samples, provides graphical summaries, and states that minimal reproducing code is available. However, the central claim is currently supported only through a nonstandard scoring function that the authors themselves acknowledge is a rough proxy, and the experimental protocol leaves several load-bearing details underspecified. If the improvement does not transfer to wildtype-relative mutation-effect scoring, the abstract's claim that 'zero-shot fitness predictions are improved' would be misleading. The result is promising but not yet established to the standard required for a benchmark claim.

major comments (4)

[Section 1, definition of S(L) and footnote 2] The central evaluation uses S(L) = sum_ij L_ij as the fitness proxy, and footnote 2 explicitly acknowledges that 'a more appropriate scoring function should be constructed by quantifying how the joint log-probability of a sequence changes w.r.t. the wildtype sequence under the presence of a mutation.' ProteinGym DMS labels measure the functional effect of mutations relative to a wildtype, not absolute sequence likelihood. As written, dropout-averaging may improve ranking on an entropy-like out-of-domainness proxy without improving the model's ability to distinguish fit from unfit mutations. The paper should at minimum evaluate the same dropout intervention with a wildtype-relative scoring function (e.g., the masked-marginal or mutant-vs-wildtype log-ratio used in Hsu et al. 2022) and report whether the improvement persists. Without this check, the central claim that zero-shot fitness predictions are improved is not established.
[Section 2, out-of-domainness argument] The justification that S(L) is a fitness proxy rests on the conjecture that out-of-domainness correlates with fitness, with the example of thermostability. This is presented as plausible reasoning, not as a demonstrated property of the ProteinGym benchmark. Since the inputs are mutants of a wildtype rather than arbitrary out-of-domain sequences, the connection between summed log-probability and fitness needs direct empirical support on the benchmark itself. A concrete test would be to compare dropout-averaged SRCC under S(L) with SRCC under a standard mutation-effect score within the same families; if the entropy-like proxy drives the gain, the fitness claim should be revised.
[Section 2, experimental protocol] The dropout rate and insertion locations appear to be selected after inspecting the same test-set SRCC values. For the 150M and 15B models, the text states that dropout layers 'had to be added to some of the early transformer layers too, for performance,' with the first fifth and first third respectively. This is a model-selection procedure performed on the test set, with no validation split and no error bars or significance tests. Consequently, the reported improvements, especially the claim that 'dropout 0.1 seems performant across all models,' may reflect overfitting to the evaluation families. The authors should either fix a single protocol a priori for all model sizes or select hyperparameters on a validation split and report confidence intervals or per-family distributions.
[Section 1, dataset description] The paper evaluates on 'a subset of the ProteinGym DMS substitution dataset with 50 protein families' but does not specify which 50 families or the selection criteria. This is a reproducibility and potential-bias issue: the subset could be chosen to favor the method, and readers cannot assess whether the result generalizes. The full list of families (or a standard benchmark split) and the inclusion criteria should be provided.

minor comments (5)

[Section 2, last paragraph] There is a typo: 'A dropout of 0.1 seemes performant' should read 'seems performant.'
[Figure 2 caption] The caption says 'improvement in performance for every model size,' but the text explains that for 150M and 15B the dropout also had to be inserted inside early transformer layers. The caption should be precise about the difference between inserting only at the embedding-transformer boundary and inserting at additional locations.
[Section 1, footnote 2] The scoring function S(L) = sum_ij L_ij is described as a proxy for entropy, but summing all log-probabilities over positions and tokens is not the standard entropy of the per-position predictive distribution. The relationship should be stated more carefully, and the role of masked positions (mentioned in footnote 2) should be made explicit in the definition.
[Section 2, 'Minimal reproducing code'] The text says code is available on GitHub, but no URL is provided in the manuscript. Please include the repository link.
[Section 2, MC dropout comparison] The method is described as 'akin to Monte-Carlo dropout,' but unlike standard MC dropout the model was not trained with dropout. The paper should clarify whether the inserted dropout layer affects only the embedding output or also the subsequent transformer layers, and how the 'dropout = 0' baseline is defined when the layer is present but inactive.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper's central claim is an empirical benchmark result evaluated on external ProteinGym labels, not a derivation that reduces to its inputs.

full rationale

The paper reports an empirical finding: inserting an inference-only dropout layer before the transformer block of ESM2 and averaging log-probability outputs improves median SRCC against measured fitness values on a 50-family ProteinGym subset. ProteinGym labels are external to the model, so the central comparison does not reduce to the model's own outputs or to a fitted parameter renamed as a prediction. The acknowledged simplification in footnote 2—that S(L)=sum_ij L_ij is a proxy for entropy/out-of-domainness rather than a wildtype-relative mutation score—is a threat to construct validity, but the SRCC is still computed against true fitness labels, so it is not a definitional equivalence. Similarly, the choice of dropout rate 0.1 and per-model insertion depths appears to have been selected after inspecting the same test SRCC values; this is a selection-on-test/overfitting concern, not a circularity, because the paper does not derive those values from the benchmark labels. The one self-citation (Ravuri et al. 2024) is used only to support the general observation that OOD inputs have higher entropy and is independent evidence, not load-bearing. No circular step can be exhibited.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The evaluation rests on ESM2 pretrained features, a crude sum-of-log-probabilities fitness proxy, median SRCC over an unspecified 50-family subset, and the application of MC dropout to a model that was never trained with dropout. No new unobserved entities are introduced; the main debt is the scoring proxy and the test-set-selected hyperparameters.

free parameters (3)

dropout probability = 0.1
Chosen as the value that 'seems performant across all models' on the ProteinGym subset; no held-out validation is described, so it is effectively fitted to the evaluation set.
number of Monte-Carlo samples = 100
Fixed for all experiments; no sensitivity analysis is reported.
dropout insertion location = between embedding and transformer; for ESM2-150M and ESM2-15B also in the first fifth and first third of transformer…
Adjusted per model size to obtain improvements, which means the method includes model-specific tuning not stated in the abstract.

assumptions (5)

domain assumption ESM2 pretrained representations support zero-shot fitness scoring through output log-probabilities.
The evaluation uses ESM2 as a frozen feature extractor and never retrains it.
domain assumption S(L) = sum_ij L_ij is an adequate fitness proxy on the ProteinGym DMS families.
Introduced in the 'classical setup' paragraph; footnote 2 admits a wildtype-relative scoring function would be more appropriate.
domain assumption Median SRCC over the unspecified 50-family subset is a sufficient performance summary.
No per-family variance, error bars, or significance tests are reported.
ad hoc to paper Inference-only dropout in a model never trained with dropout behaves like MC dropout.
MC dropout theory assumes dropout during training (Gal and Ghahramani, 2016); the paper applies it to a frozen model without this property.
ad hoc to paper Dropout rate 0.1 and the chosen insertion layers generalize beyond the evaluated subset.
These values are selected on the same 50 families used to measure the result, with no held-out validation.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Protein Language Model Zero-Shot Fitness Predictions are Improved by Inference-only Dropout." pith.science (2026). https://pith.science/paper/22YIME4L

@misc{pith2026250614793,
  author       = {Pith},
  title        = {Pith review of: Protein Language Model Zero-Shot Fitness Predictions are Improved by Inference-only Dropout},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/22YIME4L}},
  note         = {Machine review of arXiv:2506.14793}
}

read the original abstract

Protein Language Models (PLMs) such as ESM2 have been shown to be capable of zero-shot prediction of critical scalar properties of proteins (fitness). In this work, we show that injecting a dropout layer at inference time between a PLM's featurizer/embedding layer and its transformer, and averaging its output akin to Monte-Carlo dropout increases zero-shot performance on a subset of the ProteinGym dataset. This is the case even when the model was not trained with dropouts to begin with, and does not require retraining or finetuning of the PLM. A dropout of 0.1 seems performant across all models.

Figures

Figures reproduced from arXiv: 2506.14793 by the authors.

**Figure 1.** Graphical abstract: we simply introduce a dropout between the embedding layer and transformer block of a PLM, run many forward passes through the model and average the output log probabilities. These dropout-averaged outputs are more performant for zero-shot fitness prediction, even though the PLM was not trained with dropout. The classical setup: Let s denote a protein’s amino acid sequence, represented as an integ… view at source ↗

**Figure 2.** Graphs showing the zero-shot fitness performance of ESM2 with dropout added at inference-time [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

8 extracted references · 2 canonical work pages

[1]

Dropout as a bayesian approximation: Representing model uncertainty in deep learning, 2016

Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning, 2016. URL https://arxiv.org/abs/1506.02142

arXiv 2016
[2]

A baseline for detecting misclassified and out-of-distribution examples in neural networks, 2018

Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks, 2018. URL https://arxiv.org/abs/1610.02136

arXiv 2018
[3]

Learning protein fitness models from evolutionary and assay-labeled data

Chloe Hsu, Hunter Nisonoff, Clara Fannjiang, and Jennifer Listgarten. Learning protein fitness models from evolutionary and assay-labeled data. Nature Biotechnology, 40 0 (7): 0 1114--1122, Jul 2022. ISSN 1546-1696. doi:10.1038/s41587-021-01146-5. URL https://doi.org/10.1038/s41587-021-01146-5

doi:10.1038/s41587-021-01146-5 2022
[4]

Evolutionary-scale prediction of atomic-level protein structure with a language model

Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, Allan dos Santos Costa, Maryam Fazel-Zarandi, Tom Sercu, Salvatore Candido, and Alexander Rives. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science, 379 0 (6637): 0 1123--1130, 202...

doi:10.1126/science.ade2574 2023
[5]

Language models enable zero-shot prediction of the effects of mutations on protein function

Joshua Meier, Roshan Rao, Robert Verkuil, Jason Liu, Tom Sercu, and Alex Rives. Language models enable zero-shot prediction of the effects of mutations on protein function. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp.\ 29287--29303. Curran Associates...

work page 2021
[6]

Proteingym: Large-scale benchmarks for protein fitness prediction and design

Pascal Notin, Aaron Kollasch, Daniel Ritter, Lood van Niekerk, Steffanie Paul, Han Spinner, Nathan Rollins, Ada Shaw, Rose Orenbuch, Ruben Weitzman, Jonathan Frazer, Mafalda Dias, Dinko Franceschi, Yarin Gal, and Debora Marks. Proteingym: Large-scale benchmarks for protein fitness prediction and design. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Ha...

work page 2023
[7]

Uncertainty as a predictor: Leveraging self-supervised learning for zero-shot mos prediction

Aditya Ravuri, Erica Cooper, and Junichi Yamagishi. Uncertainty as a predictor: Leveraging self-supervised learning for zero-shot mos prediction. In 2024 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW), pp.\ 580--584, 2024. doi:10.1109/ICASSPW62465.2024.10626267

arXiv 2024
[8]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Dropout as a bayesian approximation: Representing model uncertainty in deep learning, 2016

Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning, 2016. URL https://arxiv.org/abs/1506.02142

arXiv 2016

[2] [2]

A baseline for detecting misclassified and out-of-distribution examples in neural networks, 2018

Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks, 2018. URL https://arxiv.org/abs/1610.02136

arXiv 2018

[3] [3]

Learning protein fitness models from evolutionary and assay-labeled data

Chloe Hsu, Hunter Nisonoff, Clara Fannjiang, and Jennifer Listgarten. Learning protein fitness models from evolutionary and assay-labeled data. Nature Biotechnology, 40 0 (7): 0 1114--1122, Jul 2022. ISSN 1546-1696. doi:10.1038/s41587-021-01146-5. URL https://doi.org/10.1038/s41587-021-01146-5

doi:10.1038/s41587-021-01146-5 2022

[4] [4]

Evolutionary-scale prediction of atomic-level protein structure with a language model

Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, Allan dos Santos Costa, Maryam Fazel-Zarandi, Tom Sercu, Salvatore Candido, and Alexander Rives. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science, 379 0 (6637): 0 1123--1130, 202...

doi:10.1126/science.ade2574 2023

[5] [5]

Language models enable zero-shot prediction of the effects of mutations on protein function

Joshua Meier, Roshan Rao, Robert Verkuil, Jason Liu, Tom Sercu, and Alex Rives. Language models enable zero-shot prediction of the effects of mutations on protein function. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp.\ 29287--29303. Curran Associates...

work page 2021

[6] [6]

Proteingym: Large-scale benchmarks for protein fitness prediction and design

Pascal Notin, Aaron Kollasch, Daniel Ritter, Lood van Niekerk, Steffanie Paul, Han Spinner, Nathan Rollins, Ada Shaw, Rose Orenbuch, Ruben Weitzman, Jonathan Frazer, Mafalda Dias, Dinko Franceschi, Yarin Gal, and Debora Marks. Proteingym: Large-scale benchmarks for protein fitness prediction and design. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Ha...

work page 2023

[7] [7]

Uncertainty as a predictor: Leveraging self-supervised learning for zero-shot mos prediction

Aditya Ravuri, Erica Cooper, and Junichi Yamagishi. Uncertainty as a predictor: Leveraging self-supervised learning for zero-shot mos prediction. In 2024 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW), pp.\ 580--584, 2024. doi:10.1109/ICASSPW62465.2024.10626267

arXiv 2024

[8] [8]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...