Pith. Sign in

REVIEW

recAMPLIFY: Large-Scale Pretraining of Recursive Protein Language Models

PaperPDF from arXiv

data files

ReviewThe objections and verdict, as JSON Paper breakdownThe authors' claims, conditions and measurements, as JSON Signed receiptProof this review has not been altered
X Bluesky LinkedIn Reddit Hacker News
Saurav Jha, Qianli Liao, Robert Vernon, Sarath Chandar, Quentin Fournier

Machine Learning · Submitted 19 Oct 2026 · 70 references

Pith's verdict: A genuinely honest, large-scale recursive-pLM study with real artifacts; the headline gains hold as measurements, but the parameter-efficiency framing needs the matched-depth feedforward control it never runs.

T0 · AI review

How this review was made: An AI referee (deepseek/deepseek-v4-flash) read the paper on October 20, 2026 and answered a fixed list of questions about it. T0 is the starting level every paper gets. The levels describe how deeply Pith has checked the paper, never the authors.

  1. T0 AI review An AI read the paper and answered a fixed list of questions.
  2. T1 Audit Automatic checks that the references and files the paper relies on exist and match.
  3. T2 Compute The paper's code or calculations re-run with the same results.
  4. T3 Formal statement The claims written out in a formal math language a computer can read.
  5. T4 Proof Those claims proved, and the proof checked by computer.

How Pith checks papers →

· 2 major objections · 3 minor · Ask about this paper · The compiled record · Read the full review →
Challenge this review Comment (0) Read on arXiv ↗
Cite ▾
arXiv ID2610.26747 (version 1)
Pith IDJ6HEC462, Pith's permanent record of this paper and its review
CitationQianli Liao, Quentin Fournier, Robert Vernon, Sarath Chandar, Saurav Jha. "recAMPLIFY: Large-Scale Pretraining of Recursive Protein Language Models." arXiv:2610.26747 (2026). https://pith.science/paper/J6HEC462
Cite this reviewBibTeX for Pith's review of this paper
Claim paper

Why Pith says this

Highlight any text to comment on it.

Referee report

How Pith checks papers → · Full audit of this paper →

The paper

The paper claims

A 155M-parameter protein language model built around a weight-tied recursive core matches or beats a 354M-parameter model of the same lineage on the same training data, and a learned gate lets each protein spend only the recurrent cycles…

The reading

This paper claims that a masked protein language model can buy test-time compute without buying parameters: replace most of the encoder with a single weight-tied Transformer block that refines the same residue-aligned hidden states over an adjustable number of cycles. Instantiating this as recAMPLIFY, a 155M-parameter model trained on the same 2T tokens as the 354M-parameter AMPLIFY-350M, the authors report gains on CASP contact recovery and on three of four supervised probes, most notably fold accuracy rising from 56.8% to 69.6%. Because no single cycle count suits every protein, they add Gated Mixture Recurrence, a halting-and-gating scheme that decodes from a mixture of intermediate states and reduces expected recurrent cycles by 32–40% while keeping most of the gains. If correct, parameter-tied recurrence becomes a third scaling axis for protein encoders, alongside model size and data.

What carries the argument

The load-bearing object is the weight-tied recursive core: a Transformer block whose parameters are reused on every cycle, with the input embedding re-injected at each cycle (input injection) to keep the iterated map bounded. The architecture is prelude → core$^L$ → coda, so effective depth $p + rL + c$ is a test-time variable independent of the 155M unique parameters. The second mechanism is Gated Mixture Recurrence (GMR), which reads the hidden trajectory, emits a conditional halt probability and a residual gate at scalar, low-rank, or per-dimension channel resolution, and decodes the masked-token prediction from the survival-weighted mixture $\bar{z} = \sum_k p_k z_k$ of visited states; with the gate removed, GMR reduces to PonderNet-style halting. GMR does the work of making recursion input-adaptive: it decides both when to stop and how much of each cycle's proposed update to accept, and its monotonically non-increasing accumulated survival $\prod_{j<k} g_j$ gives the gate the same commitment semantics as halting.

Extended reading notes

Core claim

The paper's central claim is that weight-tied depth recurrence is a viable pretraining strategy for masked protein encoders, separating effective depth from unique parameter count. recAMPLIFY keeps the AMPLIFY recipe—same 2T tokens, same hidden width 960, same masking objective—but compresses the architecture into a 2-layer prelude, a 10-layer weight-tied recurrent core, and a 2-layer coda, giving 155M unique parameters whose effective depth $p + rL + c$ is whatever the user chooses at inference. Evaluated at $L = 4$ cycles (44 effective layer applications), it matches or beats AMPLIFY-350M: fold accuracy 69.6% versus 56.8%, localization 74.1% versus 71.6%, stability Spearman 0.703 versus 0.585, and higher CASP16 contact precision, with small losses on zero-shot DMS correlation and KL calibration. The paper acknowledges that larger ESM-2 checkpoints still outperform it on three of four supervised probes, a comparison it flags as confounded by different pretraining data. On top of recAMPLIFY, Gated Mixture Recurrence (GMR) replaces the fixed cycle count with a self-supervised halt head plus a learned residual gate at scalar, low-rank, or per-dimension resolution; gated variants cut expected depth from 52 to 33–37 layer applications and improve CASP and most supervised probes. Mechanistic analyses argue that residual gating, not halting alone, drives the recurrent state to an empirical fixed point beyond the trained horizon, and that expected halt depth, once length and prediction sharpness are controlled, tracks sequence entropy and hydrophobic composition.

Load-bearing premise

The gains are credited to weight-tied recurrence, yet recAMPLIFY is read out at 44 effective layer applications versus AMPLIFY-350M's 32, and the paper trains no matched-width, same-data feedforward encoder at 44 layers to show that the same compute without weight tying would not produce the same results.

Editorial extensions

If this is right

  • Weight-tied recurrence gives protein encoders a third scaling lever: with the same 2T tokens, 155M unique parameters arranged as a tied core match or beat a 354M feedforward model on fold, localization, stability, and CASP contacts, so parameter count alone does not bound pretraining quality.
  • Adaptive test-time depth works for masked encoders without any external verifier: GMR is trained purely from the masked-language-modeling objective plus a halt-entropy term, and its expected cycle count drops by 32–40% while improving CASP contact recovery and most supervised probes.
  • Fixed-depth recursion is brittle past the trained horizon, while variable-depth pretraining (sampling cycles uniformly per minibatch) stabilizes extrapolation; the practical recipe is to vary depth during training and set test-time depth near the training mean.
  • Residual gating, not halting alone, is what drives the recurrent state toward an empirical fixed point beyond the trained horizon—the mechanistic basis for claiming the update loop can be safely cut without losing the representation.
  • Expected halt depth, after controlling for sequence length and model confidence, tracks biological composition—hydrophobic and entropy-rich proteins receive more cycles—making the compute allocation interpretable rather than opaque.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The cleanest test the paper leaves open is a depth-matched control: a non-recurrent encoder with 44 total layers at the same width, data, and step budget. If that control reproduced the fold and stability gains, the parameter-efficiency framing would reduce to a plain depth effect; if not, weight tying itself carries the benefit.
  • Gating is per-protein rather than per-residue, which the paper acknowledges as a limitation. Extending GMR's survival to per-token channels would let a single protein spend extra cycles on structurally demanding regions, a natural next experiment given the per-assay depth optima the paper reports.
  • The halt-depth analysis suggests a cheap compute-routing heuristic: sequences with hydrophobic composition, which correlate with greater depth, could be assigned larger cycle budgets at inference while polar-dominated sequences could be truncated early—a use of the halt signal the paper does not itself propose.
  • Because the ESM-2 comparison is acknowledged to be confounded, the externally decisive test of the scaling claim is a recursive encoder versus a depth-matched feedforward encoder under the same UniRef100 mixture—the one control that isolates recurrence from raw compute.
Original abstract

Protein language models (pLMs) are powerful tools for protein annotation and mutation-effect prediction, but progress has largely depended on scaling model size and data, making them expensive and prone to memorization. Test-time scaling offers a complementary path, but methods developed for autoregressive LLMs do not transfer directly: masked encoders lack a generation loop, and amino acids do not verbalize in a reasoning language. Instead, we adapt recursive pretraining to masked pLMs, where a weight-tied Transformer core refines residue-aligned hidden states over an adjustable number of cycles. We instantiate this recipe as recAMPLIFY, a pLM with 155M unique parameters trained on the same 2T tokens as AMPLIFY-350M. Despite 56% fewer parameters, it improves CASP contact recovery and three out of four supervised tasks, notably raising fold accuracy by 22.5%. Because optimal depth for recAMPLIFY varies across tasks and assays, we then introduce Gated Mixture Recurrence (GMR), a framework that decodes each protein from a probability-weighted mixture of hidden states visited during recursion. GMR employs a learned residual gate at scalar, low-rank, or per-dimension resolution to control how much of each cycle's proposed update is accepted into the state. This unifies halting and gating into a single adaptive computation objective while recovering PonderNet-style halting as a special case without gates. Layered on top of recAMPLIFY, GMR further reduces expected layer depth by up to 37%, although the accuracy trade-off varies by task. Mechanistic analyses show that residual gating controls recurrence stability beyond the trained depth and that halt depth tracks biological properties.

The compiled record

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-20: 1 request, 7,757 input tokens.

What the abstract itself states, as probabilities: claims first 0.07 · states limitation 0.92 · human participants 0.03 · controlled comparison 0.80 · randomized assignment 0.01 · states quantitative result 0.99 · compares with prior results 0.95 · mentions code or data availability 0.03.

The abstract, sentence by sentence

arXiv listing abstract v1. Results and the main claim are underlined; .

Protein language models (pLMs) are powerful tools for protein annotation and mutation-effect prediction, but progress has largely depended on scaling model size and data, making them expensive and prone to memorization. Test-time scaling offers a complementary path, but methods developed for autoregressive LLMs do not transfer directly: masked encoders lack a generation loop, and amino acids do not verbalize in a reasoning language. Instead, we adapt recursive pretraining to masked pLMs, where a weight-tied Transformer core refines residue-aligned hidden states over an adjustable number of cycles. We instantiate this recipe as recAMPLIFY, a pLM with 155M unique parameters trained on the same 2T tokens as AMPLIFY-350M. Despite 56% fewer parameters, it improves CASP contact recovery and three out of four supervised tasks, notably raising fold accuracy by 22.5%. Because optimal depth for recAMPLIFY varies across tasks and assays, we then introduce Gated Mixture Recurrence (GMR), a framework that decodes each protein from a probability-weighted mixture of hidden states visited during recursion. GMR employs a learned residual gate at scalar, low-rank, or per-dimension resolution to control how much of each cycle's proposed update is accepted into the state. This unifies halting and gating into a single adaptive computation objective while recovering PonderNet-style halting as a special case without gates. Layered on top of recAMPLIFY, GMR further reduces expected layer depth by up to 37%, although the accuracy trade-off varies by task. Mechanistic analyses show that residual gating controls recurrence stability beyond the trained depth and that halt depth tracks biological properties.

background contribution method result interpretation main claim

What the classifier reports for each sentence. The number is the probability it gave the label.
#sentencerolethe authors' own workstates the main conclusionstrength
s0 Protein language models (pLMs) are powerful tools for protein annotation and mutation-effect prediction, but progress has largely depended on scaling model size and data, making them expensive and prone to memorization. background 0.98 0.05 0.04 asserted 1.00
s1 Test-time scaling offers a complementary path, but methods developed for autoregressive LLMs do not transfer directly: masked encoders lack a generation loop, and amino acids do not verbalize in a reasoning language. gap 0.64 0.07 0.03 asserted 1.00
s2 Instead, we adapt recursive pretraining to masked pLMs, where a weight-tied Transformer core refines residue-aligned hidden states over an adjustable number of cycles. method 0.89 0.86 0.07 asserted 0.99
s3 We instantiate this recipe as recAMPLIFY, a pLM with 155M unique parameters trained on the same 2T tokens as AMPLIFY-350M. method 0.64 0.94 0.09 asserted 1.00
s4 Despite 56% fewer parameters, it improves CASP contact recovery and three out of four supervised tasks, notably raising fold accuracy by 22.5%. result 1.00 0.95 0.49 asserted 1.00
s5 Because optimal depth for recAMPLIFY varies across tasks and assays, we then introduce Gated Mixture Recurrence (GMR), a framework that decodes each protein from a probability-weighted mixture of hidden states visited during recursion. contribution 0.73 0.95 0.13 asserted 1.00
s6 GMR employs a learned residual gate at scalar, low-rank, or per-dimension resolution to control how much of each cycle's proposed update is accepted into the state. method 0.99 0.84 0.05 asserted 1.00
s7 This unifies halting and gating into a single adaptive computation objective while recovering PonderNet-style halting as a special case without gates. method 0.58 0.83 0.08 asserted 1.00
s8 Layered on top of recAMPLIFY, GMR further reduces expected layer depth by up to 37%, although the accuracy trade-off varies by task. result 0.99 0.94 0.14 asserted 0.90
s9 Mechanistic analyses show that residual gating controls recurrence stability beyond the trained depth and that halt depth tracks biological properties. result 0.58 0.94 0.16 asserted 1.00

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from simulation or computation 0.61.

What was examined, and what was not

sources read
  • arXiv listing abstract v1
not examined
  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers
levels run
  • P abstract
not run
  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. Every number in the measurements table is the authors', checked by code. Record as JSON

Figures
Figure 1. recAMPLIFY architecture. A non-recurrent prelude maps input embeddings to a residue-aligned latent state 𝑧𝑘, a weight-tied Transformer core iteratively refines this state, and a non-recurrent coda maps the refined state to the output space. Gated Mixture Recurrence (GMR) decodes from a probability-weighted mixture over the recurrence trajectory, with a per-sequence residual gate at scalar, low-rank, or per-dimension channel resolution controlling how much of each cycle’s proposed update is accepted. Blue circles illustrate these channel-level parameterizations. pLM computation.…
Figure 2. recAMPLIFY’s per-assay optimal recurrence depth on ProteinGym DMS. Desired properties of a recursive pLM. Proteins vary widely in biological complexity, from short signal peptides to multi-domain enzymes and from structured domains to intrinsically disordered chains. pLMs must accommodate diverse proteins, and the per￾assay optima suggest that no single recurrence depth is optimal across all proteins (
Figure 3. recAMPLIFY at varying recurrence depths. (a) For 1M-step recAMPLIFY, the coda embeddings become progressively more class-separable (5-NN accuracy) with larger test-time depth 𝐿eval. (b) For 100K-step recAMPLIFY, as 𝐿eval increases past the trained depth (𝐿 = 6), the fixed-𝐿 model (red) degrades while the variable-𝐿 variant (blue) remains largely stable. 0 50 100 150 200 Training tokens (B) 6 8 10 12 14 16 UniProt dev perplexity 1 2 3 4 5 6 Training recurrence L 5.1 5.2 5.3 5.4 5.5 Final UniProt dev PPL UniProt SCOP 160 200 5.2 5.4 5.6 6.8 7.0 7.2 7.4 Final SCOP dev PPL 1 2 3 4 5 6 T…
Figure 4. Effect of recurrence depth on sequence likelihood and downstream separability at 100K steps. (a) UniProt perplexity for fixed-𝐿 pretraining, evaluated at 𝐿eval = 𝐿. (b) EC level-1 and level-2 5-NN macro-F1 across 𝐿eval for recAMPLIFY trained with 𝐿 = 6, using 14K Swiss-Prot enzymes balanced across the 7 top-level classes relabeled into 46 level-2 subclasses (≥ 30 samples each). (c) SCL 10-way 5-NN macro-F1 across five FLIP DeepLoc-1.0 splits [Didi et al., 2026], fitted on 11,181 shared training sequences and evaluated on each held-out test set. deeper recursion yields macro-F1 gains…
Figure 5. Dynamics of recurrent block layer ℓ on stability prediction upon training with 𝐿 = 6 cycles (vertical dashed lines). (a) The successive difference norm ∥z ℓ 𝑘 − z ℓ 𝑘−1 ∥2 shows gating drives each layer’s step size toward zero beyond the trained depth. (b) The normalized difference to each layer’s deepest recorded state highlights the rate and timing of decay as key convergence diagnostics. depth as the mean of the model’s predicted halting distribution: 𝔼[𝑘] = Í 𝑘 𝑘 𝑝𝑘. This characterizes the learned allocation policy rather than measured wall-clock savings, since our evaluation…
Figure 6. ColSum concentration through the recurrent block at 𝐿eval = 50. Cycles colored light (early) to dark (late). Dashed red line is the feedforward reference. Feedforward Transformer layers exhibit heterogeneous compute profiles, where early layers often cluster residues by local context, middle layers integrate over increasing spans, and late layers commit to global structural pre￾dictions [Vig et al., 2021, Siji et al., 2026]. We thus ask whether, for each recAMPLIFY variant, the tied block pre￾serves this depth-heterogeneous mixing schedule within the trained recurrence depth 𝐿 = 6…
Figure 7. Per-protein expected depth 𝔼[𝑘] vs mean per￾position KL to MSA consensus (halt-only GMR, 6,868 human proteins). Colors code the model’s prediction sharp￾ness −𝐻[𝑝𝜃], averaged over positions. Left: raw scatter; right: residuals of 𝔼[𝑘] and KL after regressing out log 𝑇 and sharpness
Figure 8. Performance comparison of different representation choices used for training downstream supervised task probes: 𝑧 uses the output of the recurrent core while coda is the standard pre-LM head output 𝑓𝑐(ez). We sweep 𝐿eval ∈ {1, . . . , 9} on the same recAMPLIFY trained with ( 𝑝, 𝑟, 𝑐) = (2, 8, 2), and 𝐿 = 3 (★ denotes 𝐿eval with matched 𝐿). 20
Figure 9. Ablation of physical layer counts: we train each variant for 100K steps and evaluate at its training depth: 𝐿 = 𝐿eval = 4 in (a) and 𝐿 = 𝐿eval = 3 in (b). Overall, 𝑟 = 10 performs best in (a) on DMS, CASP, fold and SSP, while 𝑟 = 12 is best on KL (NT). The prelude/coda split in (b) changes every probe by less than 1 pp. curve signals fidelity to the feedforward reference in both shape and amplitude. F. Additional Results on GMR Depth Encoding Biological Signal Compositional axes of the GMR halt signal. Main-paper
Figure 10. Effect of truncated backpropagation (BPTT) steps on supervised probes. We sweep across 𝐿eval ∈ {1, . . . , 9} for four checkpoints that are trained with 𝐿 = 6 but differ in the truncated backprop-through-time window 𝜏: full BPTT (𝜏 =full, gradients through every cycle) and truncated windows 𝜏 ∈ {3, 4, 5}. Here, the last 𝜏 cycles receive gradients, while the earlier 𝐿 − 𝜏 do not. Full-BPTT holds up across the sweep and remains competitive with AMPLIFY-350M (main-paper
Figure 11. Varying context type for 𝐿 = 4. All models are evaluated at the matched 𝐿eval = 4 depth [Liao and Poggio, 2026]. The simplest context type, ‘zero’, which uses an identity transformation of the previous cycle’s latent 𝑧𝑘−1, performs surprisingly well throughout all tasks. 0 10 20 30 40 50 recurrence cycle k 0.5 0.0 0.5 1.0 cosine to deepest state cos(zk, zL) input injection on input injection off 0 10 20 30 40 50 recurrence cycle k 10 2 10 3 10 4 z k z k 1 2 absolute step zk zk 1 2 (log) Input injection ablation fixed depth
Figure 12. Input-injection ablation for recAMPLIFY. Blue line shows a normal forward pass while red denotes injection stream removed from the recurrent core. Left: cos(z𝑘, z𝐿). Right: ∥z𝑘−z𝑘−1 ∥2 on log-𝑦. With input injection, the state is a bounded iterated map (∥Δz∥2 plateaus below 103 ). Without the injection, the state diverges by three orders of magnitude within a few cycles. Input injection thus helps keep the recurrence bounded, matching the finding of Blayney et al. [2026] on causal-LM looped Transformers. We note that this boundedness does not imply convergence. The blue plateau in…
Figure 13. Per-layer dynamics of each recurrent block layer ℓ on the stability prediction task (median over 32 probes). (a) Successive-difference norm: gating drives each layer’s step size toward zero past the 𝐿 = 6 cycles seen during training. (b) Normalized operator residual to each layer’s deepest recorded state z ℓ 𝐿 (taken as an approximate fixed point) over 𝑘 ∈ {1, . . . , 𝐿 − 1} cycles shows the key diagnostic being the rate and timing of decay. Scalar and per-dim gates reach near-zero residual within a few cycles of the trained horizon, whereas halt-only and rank-16 decay only slowly…
Figure 14. Post-halt attention stability. Head-averaged attention Frobenius distance between recurrent cycles for three representative variants at 𝐿eval = 50. Each cell (𝑘, 𝑘′ ) is ∥𝐴𝑘 − 𝐴𝑘 ′ ∥𝐹 averaged over 𝐵 = 4 probe sequences from the stability prediction task, and the dashed line marks the trained halt horizon at 𝑘 ≈ 7. For the variant with fixed test-time depth and varying-𝐿 pretraining (left), attention keeps drifting through all 50 cycles, despite the recurrence being bounded (
Figure 15. ColSum concentration vs. physical layer position across five recAMPLIFY variants, evaluated at 𝐿eval = 50. Solid lines with circles are the trained cycles ([1, 6]). Dashed lines with diamonds: the six evenly spaced samples from the extrapolation range [7, 50], orange square: prelude, black triangle: coda. The dashed red line is the feedforward reference which is a recAMPLIFY trained with the same ( 𝑝, 𝑟, 𝑐) = (2, 8, 2) physical layer counts without recursion, i.e., 𝐿 = 1. trained under the AMPLIFY data recipe, and generalization to substantially different scales, architectures, or…
Figure 16. Median residual halt depth 𝔼[𝑘] binned by composition-diversity × hydrophobicity. Left: Shannon entropy of the AA distribution (rows) against % of FILMVWY (columns). Right: Jensen–Shannon divergence of the AA distribution to the uniform 1/20 prior (rows) against % of FILMVWY (columns). Both panels are computed on the 10K UniProt held-out sequences of §5.1.3 and use 𝔼[𝑘] residualized against log 𝑇 and the model’s prediction sharpness −𝐻[𝑝𝜃]. Diverging color scale is centered at 0. Positive (red) means the model spends more cycles than a length- and sharpness-matched null. 25
Figure 17. Median residual halt depth 𝔼[𝑘] by amino acid × per-AA composition decile. Rows are grouped into the seven hydrophobic/aromatic residues (bottom) and the remaining residues (top). The horizontal line separates the hydrophobic block (I, L, V, F, M, W, Y) from the non-hydrophobic block above. Columns are within-AA composition deciles (bin 1: lowest fraction, bin 10: highest). Hydrophobic AAs show increases over most of the range: heavier hydrophobic content in general drives more GMR cycles. Non-hydrophobic AAs show an inverted U-shape with extreme composition dominance driving fewer…
Figure 18. LM-head top-1 probability per real token on stability prediction, evaluated with (ON) and without (OFF) input-injection. 26

Figures reproduced from arXiv:2610.26747 by the authors.

References

Discussion

Disagree with the review, or know something it missed? Leave a comment, highlight any passage above to comment on it, or challenge this review formally.