Pith. sign in

REVIEW 3 major objections 4 minor 34 references

Transformers trained on proteins can learn to attend to Euclidean distance

T0 review · 3 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Standard Transformers can learn to gate their attention by Euclidean distance when coordinates are linearly embedded, and structural pretraining improves protein function prediction beyond dedicated structure models.

desk verdict Solid theory and clean simulations; the protein benchmark leaks DeepFRI test chains into pretraining and needs a redo before the practical claims are credible. read the letter →

arxiv 2502.01533 v1 pith:7XDAMJWA submitted 2025-02-03 cs.LG cs.AIq-bio.BM

Isaac Ellmen , Constantin Schneider , Matthew I.J. Raybould , Charlotte M. Deane This is my paper · ORCID
classification cs.LGcs.AIq-bio.BM
keywords transformersproteinlanguagemodelsspatialattentionGaussianEuclideandistancemaskedtokenpredictionfunctionSE(3)invariance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that ordinary Transformers do not need graph neural networks or custom structural modules to reason about 3D structure: given linearly embedded coordinates, their attention can act as a Gaussian filter on Euclidean distance. A short derivation shows why: LayerNorm turns a linear coordinate embedding into a quadratic function of pairwise distance, which softmax attention then turns into a Gaussian. Simulations with random 3D point sets and protein masked-token pretraining both confirm the predicted distance filters, and the structure-pretrained protein encoder improves downstream function prediction beyond a dedicated structural model. If correct, this gives a simple route to hybrid sequence-structure models.

What carries the argument

The load-bearing machinery is the nonlinearity of LayerNorm acting on a constant coordinate embedding. For the linear embedding $E_{\mathrm{lin}}(x)=(x,-x,1,-1)$, LayerNorm leaves the mean at zero but divides by a standard deviation $\sqrt{(1+x^2)/2}$; expanding this for small $x$ turns the constant $\pm 1$ entries into $\pm(1-x^2/2)$ and leaves the $x$ entries approximately linear, so the dot product of two such normalized embeddings becomes $4-2(x_i-x_j)^2$. Feeding this through scaled softmax attention yields a Gaussian filter $\exp(-a|x_i-x_j|^2/\sqrt{d})$. The proof requires only $n+2$ embedding dimensions per head in $\mathbb{R}^n$, and the appendix shows that gated linear units can compute the quadratic $x^2$ exactly, giving a higher-order approximation.

What would settle it

Remove every chain in the function-prediction evaluation set from the pretraining corpus and rerun the finetuned model; if the performance advantage of structure over sequence-only training collapses, the downstream claim is inflated by data leakage.

Watch

Extended reading notes

Core claim

The paper's central claim is that a standard pre-norm Transformer can act as its own structure model when each token receives a linear embedding of its 3D coordinates. Choosing an embedding $E$ with $\mathrm{LN}(E(\vec x_i))\cdot \mathrm{LN}(E(\vec x_j))\approx -a|\vec x_i-\vec x_j|^2 + b$ makes the unnormalized attention weight approximately $\exp(-a|\vec x_i-\vec x_j|^2/\sqrt{d})$, a 3D Gaussian in distance, so each head can attend selectively to spatially close tokens and tune its variance through LayerNorm gains or learned Q/K maps. The authors verify this in simulated point clouds and in protein masked-token prediction, where the coordinate-trained model reaches training perplexity 6.5 versus 11.9 without coordinates, and its finetuned molecular-function model reaches AUPRC 0.566 versus 0.446 for a graph-convolutional baseline.

Load-bearing premise

The argument hinges on coordinates staying small enough for the Taylor approximation to hold, and on the same protein structures not having been seen during pretraining; if either fails, the reported Gaussian filtering and function-prediction gains would not follow as presented.

Editorial extensions

If this is right

  • A standard pre-norm Transformer given linearly embedded coordinates can produce a 3D Gaussian spatial filter in each attention head without graph edges or custom structural modules.
  • Only $n+2$ head dimensions are needed to measure distance in $\mathbb{R}^n$, so even small Transformers can represent 3D structure compactly.
  • Masked-token protein pretraining with coordinates reaches training perplexity 6.5 versus 11.9 without, and structure-trained attention is visibly Gaussian in early layers.
  • Finetuning the structure-pretrained encoder for GO molecular function prediction beats a graph-convolutional baseline, with AUPRC 0.566 versus 0.446.
  • Because standard attention admits linear-memory implementations, structure-aware Transformers can attend over fully connected sets of residues instead of being limited to k-nearest-neighbour edge graphs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same linear-coordinate Gaussian attention should transfer to other coordinate-token problems such as small-molecule property prediction or 3D object reasoning; the paper does not run those experiments.
  • Because each head can set its own LayerNorm gains, the model can represent multiple spatial resolutions at once; inspecting per-head fitted Gaussian widths would test this directly.
  • The random-rotation augmentation result suggests SE(3) invariance can be trained into a standard Transformer without architectural constraints, so equivalent augmentation may suffice for other coordinate-based sequence models.
  • The GLU calculation implies Transformers with gated linear units could approximate distance filters accurately over larger coordinate magnitudes, a prediction about newer architectures that is not demonstrated here.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper claims that standard Transformers can learn to attend to Euclidean distance when coordinates are passed as linear embeddings. The theoretical section derives conditions under which LayerNorm-normalized embeddings produce attention weights approximately equal to exp(-a/sqrt(d) |x_i - x_j|^2), i.e., a 3D Gaussian filter of Euclidean distance. Simulated experiments support the predicted squared-distance dependence, show an empirical requirement of roughly n+2 head dimensions for R^n, and demonstrate that random rotations improve SE(3)-invariance in a low-data regime. Protein experiments show that an ESM/BERT-style model pretrained with coordinates reaches lower masked-token perplexity than a sequence-only counterpart, and finetuning this model for GO molecular function gives AUPRC 0.566 versus 0.446 for DeepFRI. The paper concludes that standard Transformers can operate as hybrid structure-language models without custom structural modules.

Significance. If the downstream benchmark claim held, this would be a significant result: it gives a simple mechanism for why linearly embedded coordinates can work in structure-aware Transformers, connects it to Gaussian attention, and suggests that standard Transformers, which admit linear-memory attention implementations, could replace memory-intensive structural GNNs. The paper also ships reproducible code and makes a falsifiable prediction (Gaussian attention) that is cleanly tested in simulation; the protein attention diagnostics provide direct qualitative evidence that the trained model uses distance-based filtering. The principal weakness is that the headline function-prediction comparison is currently not trustworthy because the pretraining corpus likely contains the DeepFRI evaluation chains, and the comparison is reported without error bars.

major comments (3)
  1. [3.2.1, 3.2.3, Table 1] The pretraining corpus is described in Section 3.2.1 as the entire GO PDB dataset from DeepFRI, while Section 3.2.3 states that downstream evaluation uses the same data splits as DeepFRI. Since the DeepFRI test split is a subset of that corpus, the coordinate-pretrained model has seen the evaluation chains and their coordinates during 100 epochs of masked-token pretraining. The finetuned model can therefore memorize chain-specific sequence/structure features correlated with GO labels, potentially inflating the reported AUPRC 0.566 versus 0.446 and the structure gain of 0.185 over DeepFRI. This is load-bearing for the paper's practical claim that standard Transformers outperform custom structural models. The comparison must be rerun after excluding DeepFRI evaluation chains (ideally with additional sequence-identity clustering) from the pretraining corpus, and the results should be reported with multiple seeds or ensembles to quantify run-to-run variability.
  2. [2.1, Appendix A.1, Equations (6) and (A11)] The theoretical derivation of Gaussian attention requires the small-coordinate assumption: small |x_i - x_j| for E_trig and small |x_i|, |x_j| for E_lin and E_quad, and it also assumes the trained network discovers Q = K = Id with appropriately tuned LayerNorm gains. Appendix A.1 asserts that a learned rescaling can enforce small coordinates, but no quantitative evidence is provided that the trained protein model operates in this asymptotic regime or that its attention is governed by this mechanism rather than by other learned Q/K structure. The simulated experiments deliberately impose the rescaling (coordinates are rescaled by 1/16), so they confirm the mechanism only under the paper's stated conditions. Please report the effective coordinate scale after the learned embedding, the LayerNorm gain values, and/or a controlled comparison of learned Q/K deviations from identity to assess whether the approximation is actually satisfied in the protein model.
  3. [3.1.2, Figure 2b] The paper claims that Transformers need n+2 embedding dimensions to learn distance in R^n, but the supporting experiment varies head dimension rather than embedding dimension, and the theoretical construction in Section 2.2 supplies a 4-dimensional embedding for n = 1, one more than n+2 = 3. The text should reconcile this discrepancy, for example by stating that n+2 is an empirically sufficient head dimension and explaining why the theoretical embedding uses one extra dimension, or by softening the language from 'need' to an observed threshold. As written, the claim is stronger than the evidence presented.
minor comments (4)
  1. [Figure 1] The caption contains the unclear phrase 'Partial Attention (N-1)x' and the equation label 'max(QK^T i)' appears to have a typo; please clarify the intended notation and the meaning of the partial attention block.
  2. [2.2 versus A.2.1] The embedding E_trig is defined in Section 2.2 as (sin(x), -sin(x), cos(x), -cos(x)) but in Appendix A.2.1 as (cos(x), -cos(x), sin(x), -sin(x)); the two orderings are equivalent up to permutation, but the inconsistency should be fixed for clarity.
  3. [3.2.3, Tables 1-3] All function-prediction results are reported as single numbers without error bars or multiple runs; given the small size of the finetuning datasets and the reported gaps, it would be helpful to report standard deviations or at least the number of seeds used.
  4. [Section 3.2.1] The text reports 'final training perplexity' and 'final validation loss' but Figure 4a only shows perplexity curves; please state explicitly whether the reported validation numbers are perplexities or losses.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the Gaussian-attention result follows from explicit embedding algebra, and the protein experiments provide independent support.

full rationale

The paper's theoretical derivation is self-contained. Section 2.1 defines a target condition (Equation 2) and shows that if an embedding satisfies it, attention becomes Gaussian (Equations 3–4). The paper then explicitly constructs embeddings E_trig, E_lin, and E_quad and proves in Appendix A.2 that they satisfy the condition to second order (Equation 6). This is constructive algebra, not a reduction of the conclusion to the assumption: the embeddings are given in closed form and the LayerNorm calculations are explicit. The small-coordinate assumption is stated as a condition, not smuggled in through a self-citation. The simulated experiments train a truncated Transformer to reproduce a Gaussian attention target, which demonstrates capability rather than circularity; because the target is externally chosen and the model is optimized against it, the low loss is a fitting result, but the paper does not present this as an independent prediction. The protein masked-token-pretraining experiment is the genuinely predictive test: the model is never trained to produce Gaussian attention, yet the post hoc attention analysis shows Gaussian dependence on Euclidean distance. This is independent evidence for the theory. There are no load-bearing self-citations, no imported uniqueness theorems, and no fitted parameter is renamed as a prediction. The potential inclusion of DeepFRI evaluation chains in pretraining is a legitimate data-hygiene and benchmark-fairness concern, but it does not make the central derivation circular; it would affect the external-validity interpretation of the downstream comparison, not the logical relation between the paper's equations and conclusions. Overall, the derivation chain is not circular and the score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central derivation rests on explicit LayerNorm algebra with no fitted parameters; the hand-set scales (1/16 input rescaling, Gaussian target sigma 200) are not fitted to the conclusion. The main unstated premises are the small-coordinate approximation, the assumption that training finds the simple Q=K=I configuration, and the assumption that pretraining on the evaluation dataset does not leak structure into the finetuned benchmark.

free parameters (2)
  • Input coordinate rescaling factor = 1/16
    Hand-chosen normalization applied to all structures before embedding (Section 3.1). The theory only requires coordinates to be small, so the exact value is not fitted to the target result, but it is a hand-set scale used in every experiment.
  • Simulated Gaussian target scale = sigma = 200
    The simulated attention target uses exp(-(x_i-x_j)^2/200^2) with coordinates in [0,200]. This scale is chosen by the authors for the experiment and is not a model parameter, but it is a hand-picked target for the p-sweep and head-dimension experiments.
assumptions (4)
  • domain assumption There exists a rescaling of input coordinates such that the Taylor expansions used for LayerNorm(linear embedding) are accurate.
    Section 2 states "we assume that all coordinates are small" and Appendix A.1 claims scaling can be learned. The proof of Gaussian attention depends on this approximation, and the protein model is trained with a fixed 1/16 rescaling rather than a learned one.
  • domain assumption A trained Transformer can learn to set Q and K approximately to identity and to use LayerNorm gains to tune the Gaussian variance.
    Section 2.1 assumes Q=K=Id for the derivation and then asserts individual heads can tune variance. No theorem or controlled experiment shows the protein model arrives at exactly this configuration, though the attention analyses are consistent with it.
  • domain assumption The Gaussian fits in Section 3.2.2 isolate distance effects because amino acids were fixed to alanine and linear positions fixed to a constant.
    Section 3.2.2 describes this isolation. It assumes the remaining attention variation is dominated by Euclidean distance and not by residual correlations among coordinates.
  • domain assumption Including downstream evaluation proteins in the self-supervised pretraining corpus does not materially inflate finetuned function prediction results.
    The paper pretrains on the same GO PDB dataset used for DeepFRI splits (Sections 3.2.1 and 3.2.3) and does not report excluding evaluation chains from pretraining. This is an unstated assumption for a fair comparison.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Transformers trained on proteins can learn to attend to Euclidean distance." pith.science (2026). https://pith.science/paper/7XDAMJWA

@misc{pith2026250201533,
  author       = {Pith},
  title        = {Pith review of: Transformers trained on proteins can learn to attend to Euclidean distance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7XDAMJWA}},
  note         = {Machine review of arXiv:2502.01533}
}
read the original abstract

While conventional Transformers generally operate on sequence data, they can be used in conjunction with structure models, typically SE(3)-invariant or equivariant graph neural networks (GNNs), for 3D applications such as protein structure modelling. These hybrids typically involve either (1) preprocessing/tokenizing structural features as input for Transformers or (2) taking Transformer embeddings and processing them within a structural representation. However, there is evidence that Transformers can learn to process structural information on their own, such as the AlphaFold3 structural diffusion model. In this work we show that Transformers can function independently as structure models when passed linear embeddings of coordinates. We first provide a theoretical explanation for how Transformers can learn to filter attention as a 3D Gaussian with learned variance. We then validate this theory using both simulated 3D points and in the context of masked token prediction for proteins. Finally, we show that pre-training protein Transformer encoders with structure improves performance on a downstream task, yielding better performance than custom structural models. Together, this work provides a basis for using standard Transformers as hybrid structure-language models.

Figures

Figures reproduced from arXiv: 2502.01533 by the authors.

Figure 1
Figure 1. Overview of the simulated experiment model. Coordinates are passed through a Transformer [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. (a) Validation loss as a function of the exponent [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. (a) Training and validation loss per epoch in a low data setting. The rolling average of the last [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: (a) Masked token prediction loss per epoch. Adding coordinates to the model hugely improves [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Average attention paid per layer to linear and 3D positional information (a-d). Fit Gaussian [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

34 extracted references · 12 canonical work pages

  1. [1]

    Ballard, Joshua Bambrick, Sebastian W

    Josh Abramson, Jonas Adler, Jack Dunger, Richard Evans, Tim Green, Alexander Pritzel, Olaf Ronneberger, Lindsay Willmore, Andrew J. Ballard, Joshua Bambrick, Sebastian W. Bodenstein, David A. Evans, Chia-Chun Hung, Michael O’Neill, David Reiman, Kathryn Tunyasuvunakool, Zachary Wu, Akvilė Žemgulytė, Eirini Arvaniti, Charles Beattie, Ottavia Bertolli, Alex...

    2024

  2. [2]

    Ball, Judith A

    Michael Ashburner, Catherine A. Ball, Judith A. Blake, David Botstein, Heather Butler, J. Michael Cherry, Allan P. Davis, Kara Dolinski, Selina S. Dwight, Janan T. Eppig, Midori A. Harris, David P. Hill, Laurie Issel-Tarver, Andrew Kasarskis, Suzanna Lewis, John C. Matese, Joel E. Richardson, Martin Ringwald, Gerald M. Rubin, and Gavin Sherlock. Gene Onto...

    doi:10.1038/75556

  3. [3]

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer Normalization , July 2016. URL http://arxiv.org/abs/1607.06450. arXiv:1607.06450 [cs, stat]

    arXiv 2016

  4. [4]

    David A. C. Beck, Darwin O. V. Alonso, Daigo Inoyama, and Valerie Daggett. The intrinsic conformational propensities of the 20 naturally occurring amino acids and reflection of these propensities in proteins. Proceedings of the National Academy of Sciences, 105 0 (34): 0 12259--12264, August 2008. doi:10.1073/pnas.0706527105. URL https://www.pnas.org/doi/...

    work page doi:10.1073/pnas.0706527105 2008

  5. [5]

    FlashAttention -2: Faster Attention with Better Parallelism and Work Partitioning , July 2023

    Tri Dao. FlashAttention -2: Faster Attention with Better Parallelism and Work Partitioning , July 2023. URL http://arxiv.org/abs/2307.08691. arXiv:2307.08691 [cs]

    arXiv 2023

  6. [6]

    Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

    Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention : Fast and Memory - Efficient Exact Attention with IO - Awareness , June 2022. URL http://arxiv.org/abs/2205.14135. arXiv:2205.14135 [cs]

    arXiv 2022

  7. [7]

    Understanding the Representation Power of Graph Neural Networks in Learning Graph Topology

    Nima Dehmamy, Albert-Laszlo Barabasi, and Rose Yu. Understanding the Representation Power of Graph Neural Networks in Learning Graph Topology . In Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019. URL https://papers.nips.cc/paper_files/paper/2019/hash/73bf6c41e241e28b89d0fb9e0c82f9ce-Abstract.html

    work page 2019

  8. [8]

    BERT : Pre -training of Deep Bidirectional Transformers for Language Understanding , May 2019

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT : Pre -training of Deep Bidirectional Transformers for Language Understanding , May 2019. URL http://arxiv.org/abs/1810.04805. arXiv:1810.04805 [cs]

    arXiv 2019

Show all 34 references
  1. [9]

    An Image is Worth 16x16 Words : Transformers for Image Recognition at Scale , June 2021

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words : Transformers for Image Recognition a...

    2021 arXiv

  2. [10]

    Fuchs, Daniel E

    Fabian B. Fuchs, Daniel E. Worrall, Volker Fischer, and Max Welling. SE (3)- Transformers : 3D Roto - Translation Equivariant Attention Networks , November 2020. URL http://arxiv.org/abs/2006.10503. arXiv:2006.10503 [cs, stat]

    2020 arXiv

  3. [11]

    Douglas Renfrew, Tomasz Kosciolek, Julia Koehler Leman, Daniel Berenberg, Tommi Vatanen, Chris Chandler, Bryn C

    Vladimir Gligorijević, P. Douglas Renfrew, Tomasz Kosciolek, Julia Koehler Leman, Daniel Berenberg, Tommi Vatanen, Chris Chandler, Bryn C. Taylor, Ian M. Fisk, Hera Vlamakis, Ramnik J. Xavier, Rob Knight, Kyunghyun Cho, and Richard Bonneau. Structure-based protein function pre...

    2021 doi

  4. [12]

    Generating Sequences With Recurrent Neural Networks , June 2014

    Alex Graves. Generating Sequences With Recurrent Neural Networks , June 2014. URL http://arxiv.org/abs/1308.0850. arXiv:1308.0850 [cs]

    2014 arXiv

  5. [13]

    Sofroniew, Deniz Oktay, Zeming Lin, Robert Verkuil, Vincent Q

    Thomas Hayes, Roshan Rao, Halil Akin, Nicholas J. Sofroniew, Deniz Oktay, Zeming Lin, Robert Verkuil, Vincent Q. Tran, Jonathan Deaton, Marius Wiggert, Rohil Badkundri, Irhum Shafkat, Jun Gong, Alexander Derry, Raul S. Molina, Neil Thomas, Yousuf Khan, Chetan Mishra, Carolyn K...

    2024

  6. [14]

    Gaussian Error Linear Units ( GELUs ), June 2023

    Dan Hendrycks and Kevin Gimpel. Gaussian Error Linear Units ( GELUs ), June 2023. URL http://arxiv.org/abs/1606.08415. arXiv:1606.08415 [cs] version: 5

    2023 arXiv

  7. [15]

    Ho and Robert Brasseur

    Bosco K. Ho and Robert Brasseur. The Ramachandran plots of glycine and pre-proline. BMC Structural Biology, 5 0 (1): 0 14, August 2005. ISSN 1472-6807. doi:10.1186/1472-6807-5-14. URL https://doi.org/10.1186/1472-6807-5-14

    2005 doi

  8. [16]

    Learning inverse folding from millions of predicted structures

    Chloe Hsu, Robert Verkuil, Jason Liu, Zeming Lin, Brian Hie, Tom Sercu, Adam Lerer, and Alexander Rives. Learning inverse folding from millions of predicted structures. preprint, Systems Biology, April 2022. URL http://biorxiv.org/lookup/doi/10.1101/2022.04.10.487779

    2022 doi

  9. [17]

    Generative Models for Graph - Based Protein Design

    John Ingraham, Vikas Garg, Regina Barzilay, and Tommi Jaakkola. Generative Models for Graph - Based Protein Design . In Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019. URL https://papers.nips.cc/paper_files/paper/2019/hash/f3a4ff48...

    2019

  10. [18]

    John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon A. A. Kohl, Andrew J. Ballard, Andrew Cowie, Bernardino Romera-Paredes, Stani...

    2021

  11. [19]

    Kipf and Max Welling

    Thomas N. Kipf and Max Welling. Semi- Supervised Classification with Graph Convolutional Networks . July 2022. URL https://openreview.net/forum?id=SJU4ayYgl

    2022

  12. [20]

    DeepGO : predicting protein functions from sequence and interactions using a deep ontology-aware classifier

    Maxat Kulmanov, Mohammed Asif Khan, and Robert Hoehndorf. DeepGO : predicting protein functions from sequence and interactions using a deep ontology-aware classifier. Bioinformatics, 34 0 (4): 0 660--668, February 2018. ISSN 1367-4803. doi:10.1093/bioinformatics/btx624. URL ht...

    2018 doi

  13. [21]

    Guzmán-Vega, Paula Duek Roggli, Lydie Lane, Stefan T

    Maxat Kulmanov, Francisco J. Guzmán-Vega, Paula Duek Roggli, Lydie Lane, Stefan T. Arold, and Robert Hoehndorf. Protein function prediction as approximate semantic entailment. Nature Machine Intelligence, 6 0 (2): 0 220--228, February 2024. ISSN 2522-5839. doi:10.1038/s42256-0...

    2024 doi

  14. [22]

    Convolutional networks for images, speech, and time series

    Yann LeCun, Yoshua Bengio, and others . Convolutional networks for images, speech, and time series. Publisher: Citeseer

  15. [23]

    ProSST : Protein Language Modeling with Quantized Structure and Disentangled Attention , May 2024

    Mingchen Li, Pan Tan, Xinzhu Ma, Bozitao Zhong, Huiqun Yu, Ziyi Zhou, Wanli Ouyang, Bingxin Zhou, Liang Hong, and Yang Tan. ProSST : Protein Language Modeling with Quantized Structure and Disentangled Attention , May 2024. URL https://www.biorxiv.org/content/10.1101/2024.04.15...

    2024 doi

  16. [24]

    Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphs , February 2023

    Yi-Lun Liao and Tess Smidt. Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphs , February 2023. URL http://arxiv.org/abs/2206.11990. arXiv:2206.11990 [physics]

    2023 arXiv

  17. [25]

    EquiformerV2 : Improved Equivariant Transformer for Scaling to Higher - Degree Representations , December 2023

    Yi-Lun Liao, Brandon Wood, Abhishek Das, and Tess Smidt. EquiformerV2 : Improved Equivariant Transformer for Scaling to Higher - Degree Representations , December 2023. URL http://arxiv.org/abs/2306.12059. arXiv:2306.12059 [physics]

    2023 arXiv

  18. [26]

    Evolutionary-scale prediction of atomic-level protein structure with a language model

    Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, Allan dos Santos Costa, Maryam Fazel-Zarandi, Tom Sercu, Salvatore Candido, and Alexander Rives. Evolutionary-scale prediction of atomic-level p...

    2023 doi

  19. [27]

    Lawrence Zitnick, Jerry Ma, and Rob Fergus

    Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C. Lawrence Zitnick, Jerry Ma, and Rob Fergus. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the...

    2021 doi

  20. [28]

    GLU Variants Improve Transformer , February 2020

    Noam Shazeer. GLU Variants Improve Transformer , February 2020. URL http://arxiv.org/abs/2002.05202. arXiv:2002.05202 [cs, stat]

    2020 arXiv

  21. [29]

    MMseqs2 enables sensitive protein sequence searching for the analysis of massive data sets

    Martin Steinegger and Johannes Söding. MMseqs2 enables sensitive protein sequence searching for the analysis of massive data sets. Nature Biotechnology, 35 0 (11): 0 1026--1028, November 2017. ISSN 1546-1696. doi:10.1038/nbt.3988. URL https://www.nature.com/articles/nbt.3988. ...

    2017 doi

  22. [30]

    RoFormer : Enhanced Transformer with Rotary Position Embedding , August 2022

    Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. RoFormer : Enhanced Transformer with Rotary Position Embedding , August 2022. URL http://arxiv.org/abs/2104.09864. arXiv:2104.09864 [cs]

    2022 arXiv

  23. [31]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need , December 2017. URL http://arxiv.org/abs/1706.03762. arXiv:1706.03762 [cs]

    2017 arXiv

  24. [32]

    Varshney, Caiming Xiong, Richard Socher, and Nazneen Rajani

    Jesse Vig, Ali Madani, Lav R. Varshney, Caiming Xiong, Richard Socher, and Nazneen Rajani. BERTology Meets Biology : Interpreting Attention in Protein Language Models . October 2020. URL https://openreview.net/forum?id=YWtLZvLmud7

    2020

  25. [33]

    On Layer Normalization in the Transformer Architecture

    Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On Layer Normalization in the Transformer Architecture . In Proceedings of the 37th International Conference on Machine Learning , pp.\ 10524--10533....

    2020

  26. [34]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.