Two Stages of Folding: Convergent Mechanisms in AI Protein Folding Trunks

Aaron Mueller; Chris Wendler; David Bau; Jannik Brinkmann; Kevin Lu; Stefan Huber; Yonatan Belinkov

REVIEW 3 major objections 4 minor 1 cited by

By patching donor hairpin activations into ESMFold's trunk, this paper shows that protein folding in the model runs through two causally verifiable stages—early blocks write biochemical features into pairwise space, late blocks build the ge

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 04:02 UTC pith:NTUPU3ZY

load-bearing objection The ESMFold two-stage story is real and well-supported; the abstract's three-model convergence claim is not in the body and should be rewritten before publication. the 3 major comments →

arxiv 2602.06020 v3 pith:NTUPU3ZY submitted 2026-02-05 cs.LG q-bio.BM

Two Stages of Folding: Convergent Mechanisms in AI Protein Folding Trunks

Kevin Lu , Jannik Brinkmann , Stefan Huber , Aaron Mueller , Yonatan Belinkov , David Bau , Chris Wendler This is my paper

classification cs.LG q-bio.BM

keywords protein structure predictionmechanistic interpretabilityactivation patchingbeta-hairpin foldingfolding trunkpairwise representationcausal interventionESMFold

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that ESMFold's folding trunk—the network component that turns a sequence representation into a structure—computes a fold in two separable stages, and that the same two-stage organization is shared by other protein-folding trunks such as OpenFold and Boltz-1. The evidence comes from activation patching: transplanting a donor protein's hairpin sequence or pairwise activations into a target protein at a single trunk block, then asking whether the output folds as a hairpin. Sequence patches succeed only in early blocks (0–7), while pairwise patches succeed only from about block 25 onward, localizing where folding decisions are made. The paper further shows these stages are causally real: steering a linear charge direction in early blocks induces hairpin formation, steering a learned distance direction in late blocks induces cross-strand hydrogen bonds, and scaling the pairwise state proportionally expands or contracts the predicted structure. A sympathetic reader would care because this turns a black-box predictor into a testable two-stage computational model with concrete handles for intervention.

Core claim

On the paper's own terms: ESMFold folds a beta hairpin in two causally separable stages. During early blocks (0–7), residue identity and biochemical features such as charge flow from the sequence representation into the pairwise representation through the seq2pair pathway; sequence patching is effective only there. During late blocks (roughly 25 onward), the pairwise representation accumulates distance and contact information, modulates sequence attention through the pair2seq bias, and directly controls the structure module; pairwise patching is effective only there. The causal interventions are charge steering, distance steering, attention redirection, and pairwise scaling. The abstract ext

What carries the argument

The central object is the pairwise representation z—an L×L tensor over residue pairs that the trunk refines block by block—together with its two communication pathways: seq2pair, which projects sequence features into z via elementwise products and differences, and pair2seq, which turns z into attention biases on the sequence. Activation patching at individual blocks localizes where each representation type carries folding-relevant information; linear probes and steering vectors expose that charge and distance are stored linearly; scaling interventions show z's magnitude acts as a global distance map consumed by the structure module.

Load-bearing premise

The results assume that activation patching is a valid counterfactual: that a donor hairpin's activations, transplanted into a target at one block, reveal what those representations normally cause, rather than being off-distribution artifacts—a risk the paper itself acknowledges in its low-confidence counterfactual structures, steric clashes, and chain breaks.

What would settle it

A concrete test: rerun the single-block patching experiments with donor activations replaced by random perturbations of the target activations matched for norm and covariance. If such perturbations induce hairpins at a comparable rate (roughly 40% for full patches, 20% for late-block pairwise patches), the donor-specific content is not doing the causal work and the two-stage localization collapses. A second check: find any donor–target pair where patching the pairwise representation alone before block 25, or the sequence representation alone after block 8, reliably folds a hairpin, breaking th

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Folding decisions can be localized to specific trunk blocks: intervention windows differ sharply for sequence versus pairwise representations, so future analyses and edits can target the block range that actually carries a given decision.
The pairwise representation is a manipulable distance map: scaling z proportionally expands or contracts the predicted structure, and steering a linear distance direction induces targeted cross-strand hydrogen bonds.
Charge is encoded along a linear direction, and steering that direction toward electrostatic complementarity induces hairpin formation—demonstrating that a known biochemical prior can causally steer a learned folding model.
The same two-stage pattern appears in reverse patching (helix into hairpin), suggesting the chemistry-then-geometry organization is a general property of the ESMFold trunk rather than a hairpin-specific artifact.
If the cross-model convergence holds, pairwise states from different folding trunks can be linearly aligned and substituted, making mechanistic findings transferable across architectures.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

If the same patching methods were applied to an MSA-based trunk, the stage boundary might shift because sequence information arrives through a different encoder; the paper's own appendix flags that transfer is untested, so the cross-model convergence should be read as a hypothesis as much as a result.
Because scaling z alone controls global compactness, a practical design handle suggests itself: modulate the pairwise scale to tune protein compactness without changing sequence—though the paper does not test whether such structures are physically foldable.
The near-zero probe selectivity for residue identity suggests seq2pair's early write-in is architectural (products and differences of identity embeddings) rather than a deeply learned strategy; if so, stage 1's timing may be a structural consequence of the trunk design, not an emergent learned strategy.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

The ESMFold two-stage story is real and well-supported; the abstract's three-model convergence claim is not in the body and should be rewritten before publication.

read the letter

The actual contribution here is the ESMFold trunk analysis, and it holds up better than the abstract does. If you ignore the title and the first abstract, this is a careful mechanistic study showing that early trunk blocks transfer sequence-level biochemistry into the pairwise representation, and late blocks encode and manipulate spatial geometry. That is genuinely new: prior interpretability work on protein models stop at the encoder, and nobody has causally dissected the folding trunk itself. The evidence is multi-pronged—single-block patching, seq2pair ablations, charge steering with same-charge controls, distance probes, pair2seq attention redirection, and z-scaling—and the controls address the obvious patching-artifact worry. The limitation appendix is unusually honest: it concedes the counterfactual structures are physically unrealistic and that inverse folding fails. That honesty buys a lot of goodwill.

The soft spots are real but mostly surface-level. The convergence claim is the load-bearing one: the title and abstract say ESMFold, OpenFold, and Boltz-1 share a two-stage organization, but every experiment runs on ESMFold, and Appendix A explicitly says the stages may not transfer to other architectures. That is not a minor framing issue—it is the difference between a solid single-model result and an unsupported architectural claim. Fixing it is easy: rewrite the title/abstract to say 'in ESMFold' and either add the other models or drop them. Second, the single-block localization restricts to donor–target pairs where full patching already succeeded, which conditions on the outcome and could inflate early-block sequence effects; the reverse patching and pathway-contribution analyses mitigate this, but the paper should say so explicitly. Third, the compaction filter (Rg < 0.9×) is applied without reporting how many cases were excluded, and steering strengths (α = 3σ, 20σ) are arbitrary; these are minor, but reviewers will want the numbers.

None of this undercuts the ESMFold core. The two-stage pattern is supported by converging causal interventions, the charge direction is genuinely linear and steerable, and the distance-map property of z is demonstrated both by probes and by scaling. The paper deserves a serious referee; it just needs substantial revision to match the claims to the evidence.

Recommendation: send it to peer review, but make the reframing a condition. The ESMFold result is worth refereeing; the convergence claim as written is not.

Referee Report

3 major / 4 minor

Summary. This paper studies the folding trunk of ESMFold by transplanting activations from a beta-hairpin donor into an alpha-helical target and measuring whether a hairpin is formed. On the basis of single-block patching, the authors identify two regimes: sequence patches are effective in blocks 0–7 and pairwise patches in blocks ~25–35. They then trace stage 1 to the seq2pair pathway, show that a linear 'charge direction' exists and that steering it induces cross-strand H-bonds, and trace stage 2 to pairwise distance/contact information: linear probes decode Cα distances from z, steering z along probe gradients induces H-bond formation, the pair2seq bias carries contact information, and scaling z scales output geometry. The main body is limited to ESMFold, with code and data availability. The paper as a whole, however, contains an early abstract/title claiming convergence across ESMFold, OpenFold, and Boltz-1 and functional interchangeability, which is not supported by any experiment in the body.

Significance. The ESMFold-specific contribution is significant and generally well-executed: it is one of the first direct mechanistic analyses of a folding trunk, and it uses externally grounded physical metrics (DSSP, N–O H-bonds, Cα distances) rather than only model-internal correlations. The paper includes multiple converging controls — reverse patching, seq2pair/pair2seq ablations, Rg filtering, same-charge steering, and z-scaling — and ships code/data. If the ESMFold claims are correct, they provide a useful map of where and when folding decisions are made in this architecture. The broader 'convergent mechanisms across architectures' claim would be a significant generalization, but the body contains no OpenFold or Boltz-1 experiments and the appendix explicitly disclaims transfer.

major comments (3)

[Abstract; §1; §7; App. A] The title and the first abstract claim a shared two-stage structure across ESMFold, OpenFold, and Boltz-1 and linearly interchangeable representations. Every experiment in the body (§§3–5 and appendices) is run on ESMFold only; there are no OpenFold or Boltz-1 experiments and no cross-model alignment or substitution. Appendix A explicitly states 'the computational stages we observe may not directly transfer to other architectures.' This is a load-bearing scope mismatch, not a presentational detail: a reader of the title/abstract would attribute to the paper a claim the manuscript itself disclaims. The title and abstract must be changed to the ESMFold-specific claim (as in the full-text abstract) unless the authors provide cross-architecture evidence.
[§3; App. A] The block-localization result (Fig. 1), the seq2pair necessity claim (Fig. 7b), and the attention-redirection result (Fig. 11) all rely on activation patching as a counterfactual. Appendix A concedes that the resulting structures have low pLDDT, that inverse folding fails to recover viable sequences, and that interventions can create steric clashes or chain breaks. These facts support an alternative interpretation in which early/late patches act as off-distribution artifacts rather than revealing the normal causal role of the representations. The authors should report pLDDT/clash rates for the patched outputs and add an explicit control (e.g., patching random or noise activations of comparable norm) or qualify the causal conclusions in the main text. The current sentence in App. A is a serious limitation that is not integrated into the main text's causal language.
[Fig. 1; §3] The two-regime separation rests on single-block patching success rates of roughly 40% (sequence, early) and 20% (pairwise, late). No confidence intervals, error bars, or significance tests are reported for these rates, and the thresholds (blocks 0–7 and 25+) are read off the curves. Please provide per-block CIs or bootstrap intervals and a quantitative test of the regime boundary, to confirm that the separation is robust rather than a consequence of smoothing or arbitrary thresholds.

minor comments (4)

[Abstract; §1] The abstract refers to Boltz-1 while the introduction refers to Boltz-2; please make the model references consistent.
[§7] The phrase 'first to analyze the computational mechanisms of the folding trunk' is stronger than needed given prior behavioral and probing analyses of AlphaFold-family models; suggest softening.
[§5.1] The linear distance-probe result (R² ≈ 0.9) would benefit from a control task (e.g., predicting sequence separation) and reported confidence intervals, since z is initialized with positional embeddings that correlate with sequence separation.
[§4.2; §5.1] The steering strengths α = 3σ (charge) and α = 20σ (distance) are presented without sensitivity analysis; please report the range of α over which the effects hold or add a brief robustness discussion.

Circularity Check

0 steps flagged

No significant circularity: the causal interventions are evaluated against external PDB/DSSP outcomes, and fitted probes/vectors are not used to define the target findings.

full rationale

The paper's derivation chain is empirical throughout. The two-stage localization (Fig. 1) comes from single-block activation patching with success measured by DSSP secondary-structure assignment against PDB-derived donor/target structures. The charge mechanism uses a difference-of-means vector fitted on 200 independent helical proteins (Eq. 5); the causal steering (Eq. 6) is evaluated by backbone N-O H-bond criteria (<3.5 Å) on separate target proteins, and the same-charge control provides a sign test. The distance mechanism trains linear probes on held-out proteins (400 train/200 test) against PDB Cα distances; steering uses the probe gradient, and success is measured by H-bond formation, not by the probe's training objective. The z-scaling result directly measures output geometry under representation scaling. None of these steps fits the target finding into the instrument used to produce it. The only concerns—the abstract's three-model convergence claim and Appendix A's admission that stages 'may not directly transfer to other architectures'—are about generalization support, not circularity. No load-bearing self-citation or uniqueness import appears. Score 0.

Axiom & Free-Parameter Ledger

6 free parameters · 6 axioms · 0 invented entities

The central empirical claims rest on: (1) counterfactual validity of activation patching (conceded by the authors to be partly off-distribution, App. A); (2) single-recycle equivalence (justified only for short proteins, App. D); (3) DSSP as ground truth for motif detection; (4) the charge direction being causally specific to electrostatics; (5) linear probes on z implying the structure module reads distances from z; (6) representativeness of the curated all-α / hairpin datasets. Free parameters are intervention hyperparameters (steering strengths, window sizes, compaction threshold) and fitted probe/direction vectors; the steering strengths are not tuned to the outcome, but their values materially affect the reported success magnitudes.

free parameters (6)

Charge steering strength α = 3σ = 3σ, with σ the std of residue projections onto v_charge
Hand-set intervention magnitude in Eq. (6); it determines how far sequence representations are moved along the charge direction and hence the strength of the reported H-bond induction (Fig. 8b).
Distance steering strength α = 20σ = 20σ, with σ the std of pair projections onto the probe direction ŵ
Hand-set magnitude for z′ij = zij − αŵ (Section 5.1); success rates (Fig. 9b) depend on it.
Charge direction v_charge (DoM vector) = Difference of mean representations of K,R,H vs D,E over 200 α-helical proteins
Derived from a fitted dataset (Eq. 5); the stage-1 interpretation and all charge steering depend on this vector being a causally specific direction.
Distance probe parameters (per-block w, b) = Linear ridge probes trained on 400 proteins against Cα distances; R²≈0.9 by late blocks
The probe defines the steering direction in z (Section 5.1); probe fit is evaluated on 200 held-out proteins but the steering target (5.5Å) is a domain constant from the literature.
Compaction filter threshold (Rg < 0.9× baseline) = 0.9
Post-hoc exclusion of steering cases where global collapse could inflate the H-bond metric (Appendix F); excluded count unreported.
Steering window sizes = 15 blocks (charge steering, seq2pair ablation); 10 blocks (distance steering)
Sliding-window hyperparameters for interventions (Figs. 7b, 8b, 9b); wider windows generally increase measured effects.

axioms (6)

domain assumption Patching donor activations into a target forward pass yields outputs that reveal the causal role of the patched representation (counterfactual validity on or near the model's manifold).
Load-bearing for every intervention conclusion; Appendix A concedes counterfactual structures show low pLDDT, steric clashes, and fail inverse folding, indicating patched states are partly off-distribution.
domain assumption Disabling recycling does not change the folding mechanism for the studied proteins.
Section 3 disables recycling and runs the trunk once; Appendix D justifies this only for short (<100 aa) proteins — medium and long proteins continue to refine with recycling, which bounds the claim's regime.
standard math DSSP secondary-structure assignment on predicted structures is a valid detector of induced hairpins and helix content.
Hairpin success (Fig. 1), reverse-patching success (App. E.3), and dataset curation all rely on DSSP (Kabsch & Sander 1983; Hekkelman et al. 2025), Appendix B.
domain assumption The difference-of-means 'charge direction' is causally specific to residue charge rather than to correlated physico-chemical properties.
Partially supported by the same-charge control (Fig. 8c), but the vector is fitted on 200 α-helical proteins (Eq. 5) and may encode correlated properties such as size or identity; the steering interpretation depends on this specificity.
domain assumption Linear decodability of Cα distance from z (R²≈0.9) licenses the conclusion that the structure module reads geometry out of z.
Section 5.1 and 5.3 combine probe R², gradient steering, and z-scaling to infer z acts as a distance map; the probes are correlational and the causal step is inferred from the intervention outcomes.
domain assumption The curated all-α targets and ~80k extracted hairpins are representative of the folding behaviors being localized.
Dataset construction (Section 3, Appendix C) restricts analysis to short motifs and helical targets; the generality of the block windows to other motifs is explicitly untested (Appendix A).

pith-pipeline@v1.3.0-alltime-deepseek · 19620 in / 21120 out tokens · 197058 ms · 2026-08-03T04:02:10.235368+00:00 · methodology

0 comments

read the original abstract

How do protein structure prediction models fold proteins? We investigate this question through causal interventions on the folding trunks of ESMFold, OpenFold, and Boltz-1. Across all three models, we find a shared two-stage computational structure. In the first stage, early blocks initialize pairwise biochemical signals: features like charge propagate from sequence into pairwise representations through architecture-specific pathways. In the second stage, late blocks develop pairwise spatial features: distance and contact information accumulate in the pairwise representation. We verify these mechanisms causally by showing that steering charge and distance features induces predictable structural changes. Furthermore, these representations are functionally interchangeable: pairwise states can be linearly aligned and substituted across models. Together, these results suggest that folding trunks with different architectures, inputs, and training procedures converge on a shared representational organization for mapping sequence chemistry into spatial geometry.

Figures

Figures reproduced from arXiv: 2602.06020 by Aaron Mueller, Chris Wendler, David Bau, Jannik Brinkmann, Kevin Lu, Stefan Huber, Yonatan Belinkov.

**Figure 1.** Figure 1: Two computational stages in the ESMFold folding trunk. We identify which latent representations in the model influence hairpin formation by patching activations from a hairpin protein into a helical protein at each block of the trunk, then measuring whether the output folds as a hairpin. Sequence patches (orange) induce hairpin formation in early blocks (0–7); pairwise patches (green) are effective in late… view at source ↗

**Figure 2.** Figure 2: Sequence view of a β-hairpin (highlighted) within a protein: two β-strand segments separated by a loop. Colored braces label the strand–loop–strand decomposition. From sequence to structure. The amino acid sequence ( [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** Figure 3: 3D cartoon diagram of a beta-hairpin, a common secondary-structure motif consisting of two antiparallel β-strands (side chain V and side chain W) connected by a short turn/loop. Interactions between side chains on opposing strands and backbone hydrogen bonding together determine the folded hairpin geometry. Beta-hairpins constitute a non-trivial minimal example of a protein folding motif, and have a long h… view at source ↗

**Figure 5.** Figure 5: Folding block. A sequence update consisting of Pair2Seq and a sequence transformer layer (attention plus MLP) followed by a pairwise update consisting of Seq2Pair and multiplicative and attention based updates of the pairwise representation (triangular update, see (Jumper et al., 2021)) plus MLP. Adapted from (Lin et al., 2023b). In this section, we treat f (1) and f (3) as black boxes and focus on the fo… view at source ↗

**Figure 6.** Figure 6: Activation patching setup. We run a donor protein containing a beta hairpin through ESMFold and extract the sequence (orange top) and pairwise representations (green top) of the hairpin (highlighted in purple) within the folding trunk. During the forward pass of the target protein containing a helix-turn-helix motif, within a single block of the trunk, we replace helix-turn-helix (highlighted in purple) se… view at source ↗

**Figure 7.** Figure 7: (a) Shift in representations in the target forward pass under different interventions (sequence patching at block 0, sequence patching at block 0 plus freezing seq2pair at blocks 0-10, pairwise patching at block 0). Sequence patching leads to representation alignment between intervened and donor forward pass as measured using Eq. (4). Freezing seq2pair while doing so prevents the representation alignmen… view at source ↗

**Figure 8.** Figure 8: Electrostatic complementarity steering induces hairpin formation. (a) Intervention setup: we steer the sequence representation toward opposite charges on the two helical regions flanking the loop, mimicking the electrostatic complementarity of natural hairpins. (b) We measure hairpin induction by the percentage of cross-strand residue pairs forming backbone hydrogen bonds (N-O distance < 3.5Å). Steering w… view at source ↗

**Figure 9.** Figure 9: Distance information is linearly accessible in intermediate blocks and can be manipulated via steering. (a) R 2 score of linear distance probes, reaching ≈ 0.9 at late blocks. (b) Intervention success measured by the percentage of cross-strand residue pairs forming backbone hydrogen bonds (N-O distance <3.5Å) after steering with window size = 10. Hydrogen bond formation peaks when steering blocks 20–35. (c… view at source ↗

**Figure 10.** Figure 10: The pair2seq bias encourages communication among structural contacts during the sequence update. (a) ROC-AUC for classifying contacts (Cα < 8Å) vs. non-contacts using bias values alone. The pair2seq bias cleanly separates contacts in hairpins in middle and late blocks. (b) Pair-to-sequence bias vs. structural contacts (Block 32). Heatmap: attention bias values averaged across heads (red = positive, blue… view at source ↗

**Figure 11.** Figure 11: Patching z causally redirects sequence attention toward donor contacts. (a) For each block k after patching, we compute the percent change in mean attention probability A¯ (k) ij (averaged across heads) to residue pairs (i, j) that are contacts uniquely in the donor structure (green) versus uniquely in the target structure (red). Percent change is computed relative to the unpatched baseline. After patchin… view at source ↗

**Figure 12.** Figure 12: Scaling the pairwise representation produces proportionally scaled 3D structures. (a) Effect of scaling the pairwise (z) versus sequence (s) representations on mean pairwise CA distance, shown as deviation from baseline (scale=1.0). Shaded regions indicate standard deviation across 600 proteins. (b) Example structures at different pairwise scaling factors, with the gradient ∂(distance)/∂(scale) compute… view at source ↗

**Figure 13.** Figure 13: Mean Cα RMSD to the highest-recycle ESMFold prediction as a function of the number of recycles, shown for short (< 100 aa), medium (100–300 aa), and long (> 300 aa) proteins sampled from UniProt. For short sequences, the RMSD drops sharply between 0 and 1 recycle and decreases only modestly thereafter, indicating that most structural convergence occurs within the first recycle. In contrast, longer protei… view at source ↗

**Figure 14.** Figure 14: We apply activation patching to the ESM protein language model encoder, folding trunk, and structure module. Patching in the folding trunk is the only place where hairpins consistently transfer. Input intervention: alter the sequence residue text to match hairpin residues before inputting into ESMFold model [PITH_FULL_IMAGE:figures/full_fig_p015_14.png] view at source ↗

**Figure 15.** Figure 15: Single-block patching with touch mask. Pairwise patching using the touch mask (green) shows a broader effective window extending into late blocks (25–47), compared to sequence patching (orange) which is effective only in early blocks (0–8). The touch mask patches interactions between hairpin residues and flanking regions, bypassing the need for information to propagate from the patched region to neighbori… view at source ↗

**Figure 16.** Figure 16: Reverse patching reproduces the two-stage structure. We patch alpha-helical representations into beta-hairpin targets and measure successful conversion (≥50% alpha-helix content in the patched region). Sequence patching (orange) is effective in early blocks (0–7), while pairwise patching (green) is effective in late blocks (25–40), mirroring the pattern observed in forward (hairpin → helix) patching. This… view at source ↗

**Figure 17.** Figure 17: Tracking residue identity transfer into the pairwise representation. (a) We train linear probes to predict residue i’s identity from the seq2pair output and pairwise representation zij . Probing accuracy on z (green) starts near chance and rises through blocks 0–15 as information transfers from s; ablating seq2pair (yellow) prevents this rise. (b) Sequence patching at block 0 (top) causes probes to read d… view at source ↗

**Figure 18.** Figure 18: Relative contributions of inter-representation pathways across blocks. The seq2pair pathway (orange) contributes most strongly to pairwise updates in early blocks (0–10), while the pair2seq pathway (green) contributes minimally in early blocks but increases substantially in late blocks (30–40). Values are min-max normalized within each pathway for visualization. This asymmetry supports the two-stage mo… view at source ↗

**Figure 21.** Figure 21: Charge information is linearly accessible in pairwise representations. Balanced accuracy of linear probes predicting positive charge (K, R, H; blue) and negative charge (D, E; red) from (a) the seq2pair output and (b) the pairwise representation z. Probes predict the charge of residue i from pairwise features at position (i, j) for sequence-distant pairs (|i − j| ≥ 4). The seq2pair output maintains high a… view at source ↗

**Figure 19.** Figure 19: Charge is linearly encoded in the sequence representation. Projections of positively charged residues (K, R, H; blue) and negatively charged residues (D, E; red) onto the charge direction (difference of means) at each block of the folding trunk. Vertical lines indicate group means. Clean separation emerges by block 4 and is maintained throughout the trunk, confirming that charge information is encoded in … view at source ↗

**Figure 20.** Figure 20: Pair2seq attention bias across blocks (head-averaged). Each panel shows the bias term βij from Eq. (2), averaged across all 8 attention heads, for protein 6rwc at selected blocks of the folding trunk. Red indicates positive bias (encouraging attention); blue indicates negative bias. Green contours mark structural contacts (Cα < 8Å). In early blocks, the bias is near-uniform; by the middle blocks, it begin… view at source ↗

**Figure 22.** Figure 22: Per-head pair2seq bias at block 32. Individual attention head bias values for beta sheet protein 6rwc at block 32. Green contours mark structural contacts (Cα < 8Å). Different heads exhibit distinct patterns: some heads show strong contact-aligned bias (e.g., heads that highlight the off-diagonal contact structure), while others capture different spatial relationships or show more diffuse patterns. This s… view at source ↗

**Figure 23.** Figure 23: Per-head pair2seq bias at block 32. Individual attention head bias values for beta protein 1l0s at block 32. Green contours mark structural contacts (Cα < 8Å). Different heads exhibit distinct patterns: some heads show strong contact-aligned bias (e.g., heads that highlight the off-diagonal contact structure), while others capture different spatial relationships or show more diffuse patterns. This special… view at source ↗

**Figure 24.** Figure 24: Example proteins used throughout the analysis. PyMOL visualization of PDB 6rwc, a beta-sheet protein with antiparallel strands connected by turns, and PDB: 1l0s, a left-handed beta-helix with repeating triangular coils. These two structures illustrate distinct beta-rich folds and serve as representative examples for the pair2seq bias visualizations in Appendix 5.2. 20 [PITH_FULL_IMAGE:figures/full_fig_p0… view at source ↗

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Design-CP: Context Parallelism for Design of Protein Nanoparticles
cs.LG 2026-07 conditional novelty 6.0

Context-parallel inference for RFdiffusion 3 enables end-to-end all-atom design of large symmetric protein nanoparticles on multi-GPU hardware without retraining.

Reference graph

Works this paper leans on

12 extracted references · 1 canonical work pages · cited by 1 Pith paper

[1]

Secondary structure assignment: We run DSSP (Hekkelman et al., 2025) on the predicted structure to identifyβ-strand regions (DSSP codes E or B)

2025
[2]

Geiger, A., Ibeling, D., Zur, A., Chaudhary, M., Chauhan, S., Huang, J., Arora, A., Wu, Z., Goodman, N., Potts, C., et al

URL https://openreview.net/forum? id=zjGwoJaVPS. Geiger, A., Ibeling, D., Zur, A., Chaudhary, M., Chauhan, S., Huang, J., Arora, A., Wu, Z., Goodman, N., Potts, C., et al. Causal abstraction: A theoretical foundation for mechanistic interpretability.Journal of Machine Learning Research, 26(83):1–64, 2025. Glusker, J. P., Lewis, M., and Rossi, M.Crystal st...

2025
[3]

Loop length: The connecting loop must be 0–5 residues
[4]

URL https://www.biorxiv.org/content/ early/2025/04/17/2025.04.11.648460

doi: 10.1101/2025.04.11.648460. URL https://www.biorxiv.org/content/ early/2025/04/17/2025.04.11.648460. Hewitt, J. and Liang, P. Designing and interpreting probes with control tasks.arXiv preprint arXiv:1909.03368, 2019. Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, D., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Zidek, A., Potapenko, A.,...

Pith/arXiv arXiv 2025
[6]

doi: 10.1162/COLI.a.572

ISSN 0891-2017. doi: 10.1162/COLI.a.572. URL https://doi.org/10.1162/COLI.a.572. Munoz, V ., Henry, E. R., Hofrichter, J., and Eaton, W. A. A statistical mechanical model for β-hairpin kinetics.Pro- ceedings of the National Academy of Sciences, 95(11): 5872–5879, 1998. Murakami, T., Hashidate, Y ., and Matsunaga, Y . Mechanis- tic interpretability of fine...

Pith/arXiv arXiv 2017
[10]

Strand length: Each strand must contain at least 2 residues
[12]

mainly α

Adjacency: The two strands must be sequential in the primary sequence (no intervening strands). C. Experiment Datasets Alpha-helical target dataset (patching targets).For acti- vation patching experiments, we curated a separate set of 95 all-α proteins from the PDB spanning diverse structural families (globins, helix bundles, helix–turn–helix proteins, co...

2019
[41]

URL https: //doi.org/10.1093/bioadv/vbae187

doi: 10.1093/bioadv/vbae187. URL https: //doi.org/10.1093/bioadv/vbae187. Hekkelman, M. L., Álvarez Salmoral, D., Perrakis, A., and Joosten, R. P. Dssp 4: Fair annota- tion of protein secondary structure.bioRxiv,

work page doi:10.1093/bioadv/vbae187
[48]

findings-acl.48/

URL https://aclanthology.org/2022. findings-acl.48/. Vig, J., Gehrmann, S., Belinkov, Y ., Qian, S., Nevo, D., Singer, Y ., and Shieber, S. Investigating gender bias in language models using causal mediation analysis. In Larochelle, H., Ranzato, M., Had- sell, R., Balcan, M., and Lin, H. (eds.),Advances in Neural Information Processing Systems, vol- ume 3...

2022
[2020]

cc/paper_files/paper/2020/file/ 92650b2e92217715fe312e6fa7b90d82-Paper

URL https://proceedings.neurips. cc/paper_files/paper/2020/file/ 92650b2e92217715fe312e6fa7b90d82-Paper. pdf. Wang, N., Nalmpantis, C., Fang, C., Niki, P., Bis- sell, M., Kathail, P., Hazra, D., Karailiev, A., Pearce, M., Ganbat, J.-O., Jain, A., Giacomoni, L., Balsam, D., Wan, J., and Solanki, R. Using interpretabil- ity to identify a novel class of Alzh...

2020
[2021]

Kabsch, W

doi: 10.1038/s41586-021-03819-2. Kabsch, W. and Sander, C. Dictionary of protein secondary structure: pattern recognition of hydrogen-bonded and geometrical features.Biopolymers, 22:2577–2637, 1983. doi: 10.1002/bip.360221211. Koga, N., Tatsumi Koga, R., Liu, G., Xiao, R., Acton, T. B., Montelione, G. T., and Baker, D. Principles for designing ideal prote...

Pith/arXiv arXiv 1983
[2025]

Ahdritz, G., Bouatta, N., Floristean, C., Kadyan, S., Xia, Q., Gerecke, W., O’Donnell, T

URL https://openreview.net/forum? id=zdOGBRQEbz. Ahdritz, G., Bouatta, N., Floristean, C., Kadyan, S., Xia, Q., Gerecke, W., O’Donnell, T. J., Berenberg, D., Fisk, I., Zanichelli, N., Zhang, B., Nowaczynski, A., Wang, B., Stepniewska-Dziubinska, M. M., Zhang, S., Ojew- ole, A., Guney, M. E., Biderman, S., Watkins, A. M., Ra, S., Lorenzo, P. R., Nivon, L.,...

2024

[1] [1]

Secondary structure assignment: We run DSSP (Hekkelman et al., 2025) on the predicted structure to identifyβ-strand regions (DSSP codes E or B)

2025

[2] [2]

Geiger, A., Ibeling, D., Zur, A., Chaudhary, M., Chauhan, S., Huang, J., Arora, A., Wu, Z., Goodman, N., Potts, C., et al

URL https://openreview.net/forum? id=zjGwoJaVPS. Geiger, A., Ibeling, D., Zur, A., Chaudhary, M., Chauhan, S., Huang, J., Arora, A., Wu, Z., Goodman, N., Potts, C., et al. Causal abstraction: A theoretical foundation for mechanistic interpretability.Journal of Machine Learning Research, 26(83):1–64, 2025. Glusker, J. P., Lewis, M., and Rossi, M.Crystal st...

2025

[3] [3]

Loop length: The connecting loop must be 0–5 residues

[4] [4]

URL https://www.biorxiv.org/content/ early/2025/04/17/2025.04.11.648460

doi: 10.1101/2025.04.11.648460. URL https://www.biorxiv.org/content/ early/2025/04/17/2025.04.11.648460. Hewitt, J. and Liang, P. Designing and interpreting probes with control tasks.arXiv preprint arXiv:1909.03368, 2019. Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, D., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Zidek, A., Potapenko, A.,...

Pith/arXiv arXiv 2025

[5] [6]

doi: 10.1162/COLI.a.572

ISSN 0891-2017. doi: 10.1162/COLI.a.572. URL https://doi.org/10.1162/COLI.a.572. Munoz, V ., Henry, E. R., Hofrichter, J., and Eaton, W. A. A statistical mechanical model for β-hairpin kinetics.Pro- ceedings of the National Academy of Sciences, 95(11): 5872–5879, 1998. Murakami, T., Hashidate, Y ., and Matsunaga, Y . Mechanis- tic interpretability of fine...

Pith/arXiv arXiv 2017

[6] [10]

Strand length: Each strand must contain at least 2 residues

[7] [12]

mainly α

Adjacency: The two strands must be sequential in the primary sequence (no intervening strands). C. Experiment Datasets Alpha-helical target dataset (patching targets).For acti- vation patching experiments, we curated a separate set of 95 all-α proteins from the PDB spanning diverse structural families (globins, helix bundles, helix–turn–helix proteins, co...

2019

[8] [41]

URL https: //doi.org/10.1093/bioadv/vbae187

doi: 10.1093/bioadv/vbae187. URL https: //doi.org/10.1093/bioadv/vbae187. Hekkelman, M. L., Álvarez Salmoral, D., Perrakis, A., and Joosten, R. P. Dssp 4: Fair annota- tion of protein secondary structure.bioRxiv,

work page doi:10.1093/bioadv/vbae187

[9] [48]

findings-acl.48/

URL https://aclanthology.org/2022. findings-acl.48/. Vig, J., Gehrmann, S., Belinkov, Y ., Qian, S., Nevo, D., Singer, Y ., and Shieber, S. Investigating gender bias in language models using causal mediation analysis. In Larochelle, H., Ranzato, M., Had- sell, R., Balcan, M., and Lin, H. (eds.),Advances in Neural Information Processing Systems, vol- ume 3...

2022

[10] [2020]

cc/paper_files/paper/2020/file/ 92650b2e92217715fe312e6fa7b90d82-Paper

URL https://proceedings.neurips. cc/paper_files/paper/2020/file/ 92650b2e92217715fe312e6fa7b90d82-Paper. pdf. Wang, N., Nalmpantis, C., Fang, C., Niki, P., Bis- sell, M., Kathail, P., Hazra, D., Karailiev, A., Pearce, M., Ganbat, J.-O., Jain, A., Giacomoni, L., Balsam, D., Wan, J., and Solanki, R. Using interpretabil- ity to identify a novel class of Alzh...

2020

[11] [2021]

Kabsch, W

doi: 10.1038/s41586-021-03819-2. Kabsch, W. and Sander, C. Dictionary of protein secondary structure: pattern recognition of hydrogen-bonded and geometrical features.Biopolymers, 22:2577–2637, 1983. doi: 10.1002/bip.360221211. Koga, N., Tatsumi Koga, R., Liu, G., Xiao, R., Acton, T. B., Montelione, G. T., and Baker, D. Principles for designing ideal prote...

Pith/arXiv arXiv 1983

[12] [2025]

Ahdritz, G., Bouatta, N., Floristean, C., Kadyan, S., Xia, Q., Gerecke, W., O’Donnell, T

URL https://openreview.net/forum? id=zdOGBRQEbz. Ahdritz, G., Bouatta, N., Floristean, C., Kadyan, S., Xia, Q., Gerecke, W., O’Donnell, T. J., Berenberg, D., Fisk, I., Zanichelli, N., Zhang, B., Nowaczynski, A., Wang, B., Stepniewska-Dziubinska, M. M., Zhang, S., Ojew- ole, A., Guney, M. E., Biderman, S., Watkins, A. M., Ra, S., Lorenzo, P. R., Nivon, L.,...

2024