Pith. sign in

REVIEW 5 major objections 5 minor 76 references

Do Protein Transformers Have Biological Intelligence?

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that small Transformers trained from scratch on protein sequences can capture biologically meaningful functional patterns, and that its Sequence Score method reveals those patterns in known catalytic motifs.

desk verdict Useful dataset and an honest efficiency win, but the 'biological intelligence' claim collapses without redundancy-controlled splits. read the letter →

arxiv 2506.06701 v1 pith:JNERSZE4 submitted 2025-06-07 cs.LG cs.AIq-bio.BM

Fudong Lin , Wanrou Du , Jinchan Liu , Tarikul Milon , Shelby Meche , Wu Xu , Xiaoqi Qin , Xu Yuan This is my paper
classification cs.LGcs.AIq-bio.BM
keywords proteinfunctionpredictionsequencetransformerProtein-FNdatasetScoreexplainableAIsaliencymapscatalyticmotifsfrom-scratchtraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to answer whether a Transformer trained only on protein sequences can capture what it calls biological intelligence—the functional patterns that matter to how proteins work. It introduces a new six-class protein function dataset, Protein-FN, a from-scratch Transformer family called SPT, and a linear-time explanation method called Sequence Score. The central claim is that even a 5.4M-parameter model can match or beat pretrained protein language models on three benchmarks, and that the model's highlighted amino acids coincide with known catalytic motifs such as the carbonic anhydrase zinc-binding site and the serine protease catalytic triad. If true, the result would make biologically valid protein function prediction and interpretation available to groups without large-scale pretraining resources.

What carries the argument

The central machinery is the Sequence Protein Transformer (SPT), a standard Transformer encoder with a protein-specific input embedding: each amino acid is one-hot encoded over the 20 standard residues, linearly projected to a hidden dimension $D$, and added to a learnable positional embedding before a stack of 12 self-attention blocks with a classification token. The accompanying explanation tool is Sequence Score, a gradient-based attribution method: it computes the gradient of a class logit with respect to the feature map, global-average-pools the gradient over sequence positions to obtain a per-channel weight $w^c$, and forms each residue's score as a weighted sum of activations, followed by a ReLU and max-normalization. Its cost scales as $O(D\cdot P)$, linear in protein length, in contrast to attention-aggregation XAI methods that scale as $O(L^2\cdot P^4)$.

What would settle it

Cluster Protein-FN by sequence identity, for example at 30% or 50%, and split so that no cluster spans training and test, then retrain SPT-Tiny from scratch. If the near-perfect accuracies persist, the conclusion survives; if the error rate jumps sharply, the reported accuracy was carried by family-level similarity. A second observation would settle attribution: train the same model on randomly shuffled labels and run Sequence Score—if the known catalytic motifs still receive high scores, the method is highlighting sequence statistics rather than learned function.

Watch

Extended reading notes

Core claim

The central discovery the paper argues for is that protein sequence alone, encoded through a simple one-hot embedding plus learned positional information and processed by a standard Transformer encoder, is enough for a small model trained from scratch to predict protein function accurately—0.31% test error on Protein-FN with the largest variant, 5.7% test error on Antibiotic Resistance with the smallest variant—and to produce explanations that match known biology. In the paper's telling, the model captures biological intelligence because Sequence Score assigns high importance to the same conserved motifs that biochemists use to explain enzyme mechanism: the His94-His96-His119 and Ser29-His107-Tyr194 motifs in carbonic anhydrases and the His57-Asp102-Ser195 catalytic triad in serine proteases. The authors take this alignment as evidence that the representation learned from primary structure is not a superficial sequence artifact but reflects functional biology.

Load-bearing premise

The whole conclusion rests on the assumption that the Protein-FN train/test split separates proteins that share little enough sequence identity that near-perfect accuracy means genuine function generalization, not recognition of conserved family signatures present in both splits.

Editorial extensions

If this is right

  • Protein function prediction no longer requires self-supervised pretraining on millions of sequences: a 5.4M-parameter model trained from scratch reaches 99.6% test accuracy on Protein-FN and 94.3% on Antibiotic Resistance.
  • Long proteins become interpretable: Sequence Score explains a full sequence in time linear in its length, where attention-flow methods cost $O(L^2\cdot P^4)$.
  • The explanation maps are testable biological hypotheses: the model highlights the zinc-binding motif of carbonic anhydrases and the catalytic triad of serine proteases, which are known to be load-bearing for those activities.
  • Larger SPT variants consistently improve accuracy across all three benchmarks, from 0.41% to 0.31% test error on Protein-FN, so the architecture scales predictably without pretraining.
  • The released Protein-FN dataset offers a compact, expert-labeled benchmark for comparing sequence-only protein function models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not report an identity-clustered train/test split for Protein-FN; a natural stress test would cluster sequences by percent identity and hold out whole families. If accuracy collapses under that split, the reported numbers would reflect memorizing conserved family signatures rather than generalizable biological knowledge.
  • Because Sequence Score is a linear gradient-times-activation saliency map, an obvious next check is to compare its top-ranked residues against experimentally measured mutation effects, such as deep mutational scanning, to test whether the highlighted sites are truly causal.
  • If the central claim holds, the same from-scratch architecture and linear-time attribution could be carried to other single-sequence protein tasks—stability, localization, or interaction prediction—where pretraining corpora are small or unavailable.
  • A scrambled-label control, where the same SPT architecture is trained on shuffled class labels, would show whether Sequence Score still flags the same motifs; if it does, those flags are a property of sequence statistics, not learned biology.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces Protein-FN, a dataset of 9,014 PDB-sourced protein sequences in six functional families; proposes Sequence Protein Transformers (SPT), a from-scratch transformer architecture with 5.4M-85.5M parameters; and presents Sequence Score, a gradient-based attribution method. The authors report that SPT-Tiny achieves 94.3% accuracy on the AR dataset and 99.6% on Protein-FN, outperforming TAPE and ESM baselines, and that Sequence Score highlights known biological motifs such as the carbonic anhydrase zinc-binding histidines and the serine protease catalytic triad. They conclude that protein transformers can 'capture biological intelligence underlined in protein sequences.' Code and dataset are publicly released.

Significance. If the central claims held, the paper would be significant: a 5.4M-parameter model trained from scratch that matches or beats much larger pretrained protein language models, together with a linear-time attribution method that reliably recovers known catalytic motifs, would be a useful contribution for resource-constrained protein-function research. The released dataset and code are concrete assets. However, the central interpretive claim—that near-perfect family-level classification plus saliency maps matching known motifs constitutes 'biological intelligence'—is not supported by the evidence as presented, because the evaluation does not control for sequence-family redundancy and the motif validation is circular.

major comments (5)
  1. [Section 3, Table 1, Table 3] The Protein-FN train/test split is documented only by per-class sample counts; the paper never states whether sequences were clustered or de-replicated by sequence identity before splitting. Because the six labels are functional families whose classification-relevant residues are conserved motifs, random PDB splits leave near-identical homologs in both train and test. The near-zero train/test error gap for SPT-Tiny (0.39% versus 0.41%, Table 3) is exactly the pattern expected under such redundancy. Without an identity-controlled split, the reported accuracies and the claim that the model 'discovers' biology-aligned patterns are not evidence of biological intelligence; they may simply reflect memorization of family signatures.
  2. [Tables 3 and 4] All accuracy results are single runs with no standard deviations, confidence intervals, or multiple random seeds. This is particularly problematic for the near-ceiling Protein-FN numbers (0.31-0.41% test error): a single favorable split cannot be distinguished from robust generalization. The comparisons to TAPE and ESM therefore lack the statistical support needed for the conclusion in Section 5.2 that SPT models show 'excellent generalization abilities.'
  3. [Sections 5.3 and 5.4] The motif validation is circular as evidence of biological intelligence. The training labels are the same six functional families whose defining conserved motifs (zinc-binding histidines, catalytic triad) are used for validation, and the authors select a handful of proteins after training whose heatmaps match these motifs. This selection-after-the-fact cannot establish that the model made a discovery; a model that memorized family-specific sequence signatures would produce exactly these saliency maps. The faithfulness experiments (Figure 3) show only that removing high-score residues hurts accuracy more than removing low-score residues; they do not compare Sequence Score against random baselines or alternative attribution methods, nor do they test whether the highlighted motifs are enriched beyond what a trivial sequence-matching baseline would find.
  4. [Section 4.3, Eqs. (4)-(6)] Sequence Score is gradient-weighted feature-map pooling, i.e., Grad-CAM applied to a Transformer layer, and the paper does not cite or compare against prior gradient-based transformer interpretability work that uses the same operation. More importantly, the complexity comparison with Attention Flow (O(L^2 P^4)) is not like-for-like: Sequence Score uses a single layer's feature maps and gradients, whereas Attention Flow aggregates information flow through all layers. The linear-time claim is therefore not a novel algorithmic contribution as framed.
  5. [Abstract and Section 1] The central term 'biological intelligence' is never defined or operationalized, so the conclusion that SPT models 'can capture biological intelligence underlined in protein sequences' is unfalsifiable as stated. The paper's evidence can at most support the weaker claim that the models' attributions are correlated with some known functional residues, and even that requires an identity-controlled evaluation before it can be accepted.
minor comments (5)
  1. [Throughout] There are numerous typos and inconsistent terms: 'bided' in the Abstract and Section B.2 should be 'embedded' or 'hidden'; 'underlined' in Sections 1 and 2 should be 'embedded'; 'ProFunc-9K' appears in Section 3 but the dataset is elsewhere called Protein-FN; 'Traning Error' appears in the header of Table 3; 'dimentional' appears in Section 4.2; and 'TM-aligh' appears in Appendix A.1.
  2. [Appendix B.1, Figure 6] The proteins in each panel of Figure 6 have identical displayed residue windows (e.g., 'RFIAVGYVDDTQFVFRFDSDA'), so the near-identical importance scores may reflect identical input segments rather than stability over structurally analogous proteins. The full-sequence similarity and the alignment method should be stated.
  3. [Appendix A.1, Table 5] It is unclear how MUSCLE and TM-align are converted into protein-function predictors. Without a description of the adaptation protocol, the comparison in Table 5 is not reproducible.
  4. [Section 5.1] The statement that GPU memory usage is 'consistently ≤ 9.6%' is ambiguous; the paper should specify the metric (for example, fraction of the 24 GB RTX 4090 memory) and report training time or total FLOPs per dataset.
  5. [Abstract and Tables 3-4] The Abstract reports 94.3% accuracy on AR and 99.6% on Protein-FN, which correspond to error rates of 5.7% (SPT-Tiny) and 0.41% (SPT-Tiny), respectively, but the text should explicitly map these numbers to the tables to avoid ambiguity about which model variant is being referenced.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the model is trained on sequence labels, the motif recovery is a post-hoc attribution result rather than a training input, and the AR/MIB comparisons provide independent external benchmarks.

full rationale

The paper's derivation chain is: (1) construct Protein-FN with six expert-assigned functional labels; (2) train SPT from scratch on one-hot encoded amino acid sequences to predict those labels; (3) compute Sequence Score attributions from the trained model; (4) observe that high-attribution positions coincide with known conserved motifs such as the zinc-binding histidines and the serine protease catalytic triad. No step reduces by construction to its own inputs. The motifs are not provided to the model during training; the model receives only sequences and class labels, and the attribution is computed post-hoc from the fitted parameters. The fact that the functional labels are family-level categories whose conserved motifs are later used for validation is a benchmark-construction concern, not a circular derivation: the model could in principle solve the classification using non-motif features, and the AR and MIB results are external benchmarks with established splits. The faithfulness evaluation (masking high-score residues produces larger accuracy drops) is a standard gradient-attribution sanity check and not a fitted parameter renamed as a prediction. The paper contains self-citations, but none is load-bearing for the central claim, and no uniqueness theorem or ansatz is imported from the authors' prior work. The very low Protein-FN train/test gap may indicate sequence-identity leakage, but leakage is a validity threat rather than a circularity under the taxonomy used here. No equation or fitted quantity is reused as its own prediction, so no specific circular step can be exhibited.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

The central claims rest on a small number of hand-set hyperparameters, on an unverified dataset split, and on an interpretive premise that equates family classification with biological intelligence. No new physical or mathematical entity is needed; the only invented entity is the paper's biological intelligence construct, which lacks an operational definition.

free parameters (3)
  • SPT model hyperparameters = depth 12; hidden sizes 192/384/768; heads 4/6/12
    Chosen by hand following ViT-B configurations; these settings define the models whose accuracy is reported in Tables 3-4 and affect all results.
  • Training hyperparameters = lr=1e-3, weight decay=0.05, layer-wise decay=0.75, label smoothing=0.1, path dropping=0.1
    Selected from prior vision transformer work and applied to all runs; not ablated except hidden size and positional embedding in Appendix A.2.
  • Feature map layer for Sequence Score = unspecified, e.g., the last block
    The choice of A in Eq. (4) determines the attribution map; the paper does not ablate which Transformer block is used, so the reported faithfulness may depend on this choice.
assumptions (4)
  • domain assumption Protein-FN class labels are biologically accurate and assigned by expert review.
    Section 3 states labels were 'carefully examined by biological experts in our team'; no label provenance or inter-annotator agreement is provided.
  • domain assumption The Protein-FN train/test split is i.i.d. with no near-duplicate sequences between folds.
    Sections 3 and 5.1 give only sample counts; no sequence-identity clustering or redundancy removal is described. This assumption is needed for the reported 0.31% to 0.41% test errors to reflect generalization.
  • ad hoc to paper Accurate classification into six functional families plus alignment with known motifs demonstrates biological intelligence.
    This equivalence is asserted in Sections 1 and 6 but never defined or tested; the motifs are discriminative features of the class labels, so a supervised classifier is expected to use them.
  • standard math First-order gradient weighting in Eqs. (4)-(5) yields faithful importance estimates for transformer feature maps.
    This is the standard Grad-CAM linear approximation; its validity for protein transformers is tested only indirectly through deletion and mutation accuracy drops, not against causal ground truth.
invented entities (1)
  • Biological intelligence as a property of Protein Transformers
    purpose: Central claim and title question; used to interpret accurate classification and motif saliency as understanding biology.
    No operational definition, no falsifiable prediction, and no distinction from ordinary supervised learning of family-specific sequence statistics is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Do Protein Transformers Have Biological Intelligence?." pith.science (2026). https://pith.science/paper/JNERSZE4

@misc{pith2026250606701,
  author       = {Pith},
  title        = {Pith review of: Do Protein Transformers Have Biological Intelligence?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JNERSZE4}},
  note         = {Machine review of arXiv:2506.06701}
}
read the original abstract

Deep neural networks, particularly Transformers, have been widely adopted for predicting the functional properties of proteins. In this work, we focus on exploring whether Protein Transformers can capture biological intelligence among protein sequences. To achieve our goal, we first introduce a protein function dataset, namely Protein-FN, providing over 9000 protein data with meaningful labels. Second, we devise a new Transformer architecture, namely Sequence Protein Transformers (SPT), for computationally efficient protein function predictions. Third, we develop a novel Explainable Artificial Intelligence (XAI) technique called Sequence Score, which can efficiently interpret the decision-making processes of protein models, thereby overcoming the difficulty of deciphering biological intelligence bided in Protein Transformers. Remarkably, even our smallest SPT-Tiny model, which contains only 5.4M parameters, demonstrates impressive predictive accuracy, achieving 94.3% on the Antibiotic Resistance (AR) dataset and 99.6% on the Protein-FN dataset, all accomplished by training from scratch. Besides, our Sequence Score technique helps reveal that our SPT models can discover several meaningful patterns underlying the sequence structures of protein data, with these patterns aligning closely with the domain knowledge in the biology community. We have officially released our Protein-FN dataset on Hugging Face Datasets https://huggingface.co/datasets/Protein-FN/Protein-FN. Our code is available at https://github.com/fudong03/BioIntelligence.

Figures

Figures reproduced from arXiv: 2506.06701 by the authors.

Figure 1
Figure 1. Illustration of two key conserved motifs, [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The architecture of our Sequence Protein Transformers (SPT) model. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Comparisons of prediction performance by masking amino acids with the highest [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Illustration of the catalytic triad, i.e., “His57-Asp102-Ser195”, in the two pro￾teins of serine proteases. Here, “His”, “Asp”, and “Ser” are abbreviated as “H”, “D” and “S”, respectively. The figure in the upper section utilizes the protein “1TX6” as an example to sho…
Figure 5
Figure 5. Figure 5: Illustration of how (a) the hidden size and (b) the positional embedding affect [PITH_FULL_IMAGE:figures/full_fig_p020_5.png]
Figure 6
Figure 6. Figure 6: The heatmaps of importance scores for (a) the “receptor” and (b) the “isomerase” [PITH_FULL_IMAGE:figures/full_fig_p021_6.png]
Figure 7
Figure 7. Figure 7: Comparisons of prediction performance by simulating protein mutations that [PITH_FULL_IMAGE:figures/full_fig_p022_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

76 extracted references · 69 canonical work pages

  1. [1]

    Samira Abnar and Willem H. Zuidema. Quantifying attention flow in transformers. In Annual Meeting of the Association for Computational Linguistics (ACL), 2020

    work page 2020

  2. [2]

    Unified rational protein engineering with sequence-based deep representation learn- ing

    Ethan C Alley, Grigory Khimulya, Surojit Biswas, Mohammed AlQuraishi, and George M Church. Unified rational protein engineering with sequence-based deep representation learn- ing. Nature methods, 2019

    work page 2019

  3. [3]

    Jaakkola

    David Alvarez-Melis and Tommi S. Jaakkola. Towards robust interpretability with self- explaining neural networks. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors,Neural Information Processing Systems 2018 (NeurIPS), 2018

    work page 2018

  4. [4]

    Vivit: A video vision transformer

    Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lucic, and Cordelia Schmid. Vivit: A video vision transformer. In International Conference on Computer Vision (ICCV), 2021

    work page 2021

  5. [5]

    Layer normalization

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016

    arXiv 2016

  6. [6]

    Accurate prediction of protein structures and interactions using a three-track neural network

    Minkyung Baek, Frank DiMaio, Ivan Anishchenko, Justas Dauparas, Sergey Ovchinnikov, Gyu Rie Lee, Jue Wang, Qian Cong, Lisa N Kinch, R Dustin Schaeffer, et al. Accurate prediction of protein structures and interactions using a three-track neural network. Science, 2021

    work page 2021

  7. [7]

    Beit: BERT pre-training of image transformers

    Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: BERT pre-training of image transformers. In International Conference on Learning Representations (ICLR), 2022

    work page 2022

  8. [8]

    GAM: explainable visual similarity and classification via gradient ac- tivation maps

    Oren Barkan, Omri Armstrong, Amir Hertz, Avi Caciularu, Ori Katz, Itzik Malkiel, and Noam Koenigstein. GAM: explainable visual similarity and classification via gradient ac- tivation maps. In International Conference on Information and Knowledge Management (CIKM), 2021

    work page 2021

Show all 76 references
  1. [9]

    Emerging properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Interna- tional Conference on Computer Vision (ICCV), 2021

    2021

  2. [10]

    Balasubramanian

    Aditya Chattopadhyay, Anirban Sarkar, Prantik Howlader, and Vineeth N. Balasubramanian. Grad-cam++: Generalized gradient-based visual explanations for deep convolutional net- works. In Winter Conference on Applications of Computer Vision (WACV), 2018. Do Protein Transformers H...

    2018

  3. [11]

    Generic attention-model explainability for interpret- ing bi-modal and encoder-decoder transformers

    Hila Chefer, Shir Gur, and Lior Wolf. Generic attention-model explainability for interpret- ing bi-modal and encoder-decoder transformers. In International Conference on Computer Vision (ICCV), 2021

    2021

  4. [12]

    Transformer interpretability beyond attention visual- ization

    Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visual- ization. In Conference on Computer Vision and Pattern Recognition (CVPR), 2021

    2021

  5. [13]

    Explain the explainer: Interpreting model-agnostic counterfactual explanations of a deep re- inforcement learning agent

    Ziheng Chen, Fabrizio Silvestri, Gabriele Tolomei, Jia Wang, He Zhu, and Hongshik Ahn. Explain the explainer: Interpreting model-agnostic counterfactual explanations of a deep re- inforcement learning agent. IEEE Transactions on Artificial Intelligence , 5(4):1443–1457, 2022

    2022

  6. [14]

    Grease: Generate factual and counterfactual explanations for gnn-based recommendations

    Ziheng Chen, Fabrizio Silvestri, Jia Wang, Yongfeng Zhang, Zhenhua Huang, Hongshik Ahn, and Gabriele Tolomei. Grease: Generate factual and counterfactual explanations for gnn-based recommendations. arXiv preprint arXiv:2208.04222, 2022

    2022 arXiv

  7. [15]

    The dark side of explanations: Poisoning recommender systems with counterfactual examples

    Ziheng Chen, Fabrizio Silvestri, Jia Wang, Yongfeng Zhang, and Gabriele Tolomei. The dark side of explanations: Poisoning recommender systems with counterfactual examples. In Proceedings of the 46th International ACM SIGIR conference on Research and Development in Information ...

    2023

  8. [16]

    Bonemet: An open large-scale multi-modal murine dataset for breast cancer bone metastasis diagnosis and prognosis

    Tiankuo Chu, Fudong Lin, Shubo Wang, Jason Jiang, Wiley Jia-Wei Gong, Xu Yuan, and Liyun Wang. Bonemet: An open large-scale multi-modal murine dataset for breast cancer bone metastasis diagnosis and prognosis. In The Thirteenth International Conference on Learning Representati...

    2025

  9. [17]

    Electra: Pre- training text encoders as discriminators rather than generators

    Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. Electra: Pre- training text encoders as discriminators rather than generators. In International Conference on Learning Representations, 2019

    2019

  10. [18]

    Real time image saliency for black box classifiers

    Piotr Dabkowski and Yarin Gal. Real time image saliency for black box classifiers. InNeural Information Processing Systems (NeurIPS), 2017

    2017

  11. [19]

    Ramaswamy

    Saurabh Desai and Harish G. Ramaswamy. Ablation-cam: Visual explanations for deep convolutional network via gradient-free localization. In Winter Conference on Applications of Computer Vision (WACV), 2020

    2020

  12. [20]

    BERT: pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL- HLT), 2019

    2019

  13. [21]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at...

    2021

  14. [22]

    Muscle: a multiple sequence alignment method with reduced time and space complexity

    Robert C Edgar. Muscle: a multiple sequence alignment method with reduced time and space complexity. BMC bioinformatics, 5(1):1–19, 2004

    2004

  15. [23]

    Prottrans: To- ward understanding the language of life through self-supervised learning.IEEE transactions on pattern analysis and machine intelligence, 44(10):7112–7127, 2021

    Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, et al. Prottrans: To- ward understanding the language of life through self-supervised learning.IEEE transactions on pattern...

    2021

  16. [24]

    Backward recursive class activation map refinement for high resolution saliency map

    Alexandre Englebert, Olivier Cornu, and Christophe De Vleeschouwer. Backward recursive class activation map refinement for high resolution saliency map. In International Confer- ence on Pattern Recognition (ICPR), 2022

    2022

  17. [25]

    Protein complex prediction with alphafold-multimer

    Richard Evans, Michael O’Neill, Alexander Pritzel, Natasha Antropova, Andrew Senior, Tim Green, Augustin Žídek, Russ Bates, Sam Blackwell, Jason Yim, et al. Protein complex prediction with alphafold-multimer. biorxiv, 2021

    2021

  18. [26]

    Multiscale vision transformers

    Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In International Conference on Computer Vision (ICCV), 2021. 16 F. Lin, W. Du, et al

    2021

  19. [27]

    Pfam: the protein families database

    Robert D Finn, Alex Bateman, Jody Clements, Penelope Coggill, Ruth Y Eberhardt, Sean R Eddy, Andreas Heger, Kirstie Hetherington, Liisa Holm, Jaina Mistry, et al. Pfam: the protein families database. Nucleic acids research, 2014

    2014

  20. [28]

    Understanding deep networks via extremal perturbations and smooth masks

    Ruth Fong, Mandela Patrick, and Andrea Vedaldi. Understanding deep networks via extremal perturbations and smooth masks. In International Conference on Computer Vision (ICCV), 2019

    2019

  21. [29]

    Fong and Andrea Vedaldi

    Ruth C. Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturbation. In International Conference on Computer Vision (ICCV), 2017

    2017

  22. [30]

    Axiom- based grad-cam: Towards accurate visualization and explanation of cnns

    Ruigang Fu, Qingyong Hu, Xiaohu Dong, Yulan Guo, Yinghui Gao, and Biao Li. Axiom- based grad-cam: Towards accurate visualization and explanation of cnns. InBritish Machine Vision Conference (BMVC), 2020

    2020

  23. [31]

    Girshick

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked autoencoders are scalable vision learners. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022

    2022

  24. [32]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Computer Vision and Pattern Recognition (CVPR), 2016

    2016

  25. [33]

    Learning inverse folding from millions of predicted structures

    Chloe Hsu, Robert Verkuil, Jason Liu, Zeming Lin, Brian Hie, Tom Sercu, Adam Lerer, and Alexander Rives. Learning inverse folding from millions of predicted structures. In International Conference on Machine Learning (ICML), 2022

    2022

  26. [34]

    Exploring evolution-aware & -free protein language models as protein function predictors

    Mingyang Hu, Fajie Yuan, Kevin Yang, Fusong Ju, Jin Su, Hui Wang, Fei Yang, and Qi- uyang Ding. Exploring evolution-aware & -free protein language models as protein function predictors. In Neural Information Processing Systems (NeurIPS), 2022

    2022

  27. [35]

    Exploring evolution-aware &-free protein language models as protein function predictors

    Mingyang Hu, Fajie Yuan, Kevin Yang, Fusong Ju, Jin Su, Hui Wang, Fei Yang, and Qi- uyang Ding. Exploring evolution-aware &-free protein language models as protein function predictors. 2022

    2022

  28. [36]

    Deep networks with stochastic depth

    Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In ECCV, 2016

    2016

  29. [37]

    Mohammad A. A. K. Jalwana, Naveed Akhtar, Mohammed Bennamoun, and Ajmal Mian. CAMERAS: enhanced resolution and sanity preserving class activation mapping for image saliency. In Conference on Computer Vision and Pattern Recognition (CVPR), 2021

    2021

  30. [38]

    Lay- ercam: Exploring hierarchical class activation maps for localization

    Peng-Tao Jiang, Chang-Bin Zhang, Qibin Hou, Ming-Ming Cheng, and Yunchao Wei. Lay- ercam: Exploring hierarchical class activation maps for localization. IEEE Trans. Image Process., 30:5875–5888, 2021

    2021

  31. [39]

    Highly accurate protein structure prediction with alphafold

    John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ron- neberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583–589, 2021

    2021

  32. [40]

    Cai, James Wexler, Fernanda B

    Been Kim, Martin Wattenberg, Justin Gilmer, Carrie J. Cai, James Wexler, Fernanda B. Viégas, and Rory Sayres. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCA V). InInternational Conference on Machine Learning (ICML), 2018

    2018

  33. [41]

    Development of a tsr-based method for protein 3-d structural comparison with its applications to protein classification and motif discovery

    Sarika Kondra, Titli Sarkar, Vijay Raghavan, and Wu Xu. Development of a tsr-based method for protein 3-d structural comparison with its applications to protein classification and motif discovery. Frontiers in Chemistry, 8:602291, 2021

    2021

  34. [42]

    Mmst-vit: Climate change-aware crop yield prediction via multi-modal spatial-temporal vision trans- former

    Fudong Lin, Summer Crawford, Kaleb Guillot, Yihe Zhang, Yan Chen, Xu Yuan, Li Chen, Shelby Williams, Robert Minvielle, Xiangming Xiao, Drew Gholson, Nicolas Ashwell, Tri Setiyono, Brenda Tubana, Lu Peng, Magdy Bayoumi, and Nian-Feng Tzeng. Mmst-vit: Climate change-aware crop y...

    2023

  35. [43]

    An open and large-scale dataset for multi-modal climate change-aware crop yield predic- tions

    Fudong Lin, Kaleb Guillot, Summer Crawford, Yihe Zhang, Xu Yuan, and Nian-Feng Tzeng. An open and large-scale dataset for multi-modal climate change-aware crop yield predic- tions. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), p...

    2024

  36. [44]

    Towards robust vision trans- former via masked adaptive ensemble

    Fudong Lin, Jiadong Lou, Xu Yuan, and Nian-Feng Tzeng. Towards robust vision trans- former via masked adaptive ensemble. In Proceedings of the 33rd ACM International Con- ference on Information and Knowledge Management (CIKM), pages 1389–1399, 2024

    2024

  37. [45]

    Comprehensive transformer-based model architecture for real-world storm predic- tion

    Fudong Lin, Xu Yuan, Yihe Zhang, Purushottam Sigdel, Li Chen, Lu Peng, and Nian-Feng Tzeng. Comprehensive transformer-based model architecture for real-world storm predic- tion. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases (ECML-PKDD), ...

    2023

  38. [46]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Interna- tional Conference on Computer Vision (ICCV), 2021

    2021

  39. [47]

    Sgdr: Stochastic gradient descent with warm restarts

    Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations (ICLR), 2016

    2016

  40. [48]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), 2019

    2019

  41. [49]

    Lundberg and Su-In Lee

    Scott M. Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Neural Information Processing Systems (NeurIPS), 2017

    2017

  42. [50]

    Antigen- specific antibody design and optimization with diffusion-based generative models for protein structures

    Shitong Luo, Yufeng Su, Xingang Peng, Sheng Wang, Jian Peng, and Jianzhu Ma. Antigen- specific antibody design and optimization with diffusion-based generative models for protein structures. 2022

    2022

  43. [51]

    Visualizing deep convolutional neural networks using natural pre-images

    Aravindh Mahendran and Andrea Vedaldi. Visualizing deep convolutional neural networks using natural pre-images. Int. J. Comput. Vis. (IJCV), 120(3):233–255, 2016

    2016

  44. [52]

    The comprehensive antibiotic resistance database

    Andrew G McArthur, Nicholas Waglechner, Fazmin Nizam, Austin Yan, Marisa A Azad, Al- ison J Baylay, Kirandeep Bhullar, Marc J Canova, Gianfranco De Pascale, Linda Ejim, et al. The comprehensive antibiotic resistance database. Antimicrobial agents and chemotherapy, 57(7):3348–3...

    2013

  45. [53]

    Language models enable zero-shot prediction of the effects of mutations on protein function

    Joshua Meier, Roshan Rao, Robert Verkuil, Jason Liu, Tom Sercu, and Alex Rives. Language models enable zero-shot prediction of the effects of mutations on protein function. 2021

    2021

  46. [54]

    Protein data bank (pdb), 2023

    PDB. Protein data bank (pdb), 2023

    2023

  47. [55]

    RISE: randomized input sampling for explanation of black-box models

    Vitali Petsiuk, Abir Das, and Kate Saenko. RISE: randomized input sampling for explanation of black-box models. In British Machine Vision Conference (BMVC), page 151, 2018

    2018

  48. [56]

    Evaluating protein transfer learning with tape

    Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Peter Chen, John Canny, Pieter Abbeel, and Yun Song. Evaluating protein transfer learning with tape. Neural In- formation Processing Systems (NeurIPS), 2019

    2019

  49. [57]

    Canny, Pieter Abbeel, Tom Sercu, and Alexander Rives

    Roshan Rao, Jason Liu, Robert Verkuil, Joshua Meier, John F. Canny, Pieter Abbeel, Tom Sercu, and Alexander Rives. MSA transformer. In International Conference on Machine Learning (ICML), 2021

    2021

  50. [58]

    Trans- former protein language models are unsupervised structure learners

    Roshan Rao, Joshua Meier, Tom Sercu, Sergey Ovchinnikov, and Alexander Rives. Trans- former protein language models are unsupervised structure learners. In International Con- ference on Learning Representations (ICLR), 2021

    2021

  51. [59]

    Deep generative models of genetic variation capture the effects of mutations

    Adam J Riesselman, John B Ingraham, and Debora S Marks. Deep generative models of genetic variation capture the effects of mutations. Nature methods, 2018

    2018

  52. [60]

    Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences

    Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C Lawrence Zitnick, Jerry Ma, et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National ...

    2021

  53. [61]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Computer Vision and Pat- tern Recognition (CVPR), 2022

    2022

  54. [62]

    Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra

    Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient- based localization. In International Conference on Computer Vision (ICCV), 2017

    2017

  55. [63]

    Deep inside convolutional net- works: Visualising image classification models and saliency maps

    Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional net- works: Visualising image classification models and saliency maps. In International Confer- ence on Learning Representations Workshop (ICLRW), 2014

    2014

  56. [64]

    Full-gradient representation for neural network visual- ization

    Suraj Srinivas and Francois Fleuret. Full-gradient representation for neural network visual- ization. In Neural Information Processing Systems (NeurIPS), 2019

    2019

  57. [65]

    Axiomatic attribution for deep networks

    Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International Conference on Machine Learning (ICML), 2017

    2017

  58. [66]

    Re- thinking the inception architecture for computer vision

    Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In Computer Vision and Pattern Recognition (CVPR), 2016

    2016

  59. [67]

    Training data-efficient image transformers & distillation through attention

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In Marina Meila and Tong Zhang, editors, International Conference on Machine Learning (ICML), 2021

    2021

  60. [68]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Neural Infor- mation Processing Systems (NeurIPS), 2017

    2017

  61. [69]

    Analyzing multi- head self-attention: Specialized heads do the heavy lifting, the rest can be pruned

    Elena V oita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi- head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Annual Meeting of the Association for Computational Linguistics (ACL), 2019

    2019

  62. [70]

    Score-cam: Score-weighted visual explanations for convolutional neural networks

    Haofan Wang, Zifan Wang, Mengnan Du, Fan Yang, Zijian Zhang, Sirui Ding, Piotr Mardziel, and Xia Hu. Score-cam: Score-weighted visual explanations for convolutional neural networks. In Conference on Computer Vision and Pattern Recognition Workshop (CVPRW), 2020

    2020

  63. [71]

    Pyramid vision transformer: A versatile backbone for dense prediction without convolutions

    Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In International Conference on Computer Vision (ICCV), 2021

    2021

  64. [72]

    Weiyan Xie, Xiao-Hui Li, Caleb Chen Cao, and Nevin L. Zhang. Vit-cx: Causal explanation of vision transformers. In International Joint Conference on Artificial Intelligence (IJCAI), 2023

    2023

  65. [73]

    Zeiler and Rob Fergus

    Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European Conference on Computer Vision (ECCV), 2014

    2014

  66. [74]

    Tm-align: a protein structure alignment algorithm based on the tm-score

    Yang Zhang and Jeffrey Skolnick. Tm-align: a protein structure alignment algorithm based on the tm-score. Nucleic acids research, 33(7):2302–2309, 2005

    2005

  67. [75]

    Learning deep features for discriminative localization

    Bolei Zhou, Aditya Khosla, Àgata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016. Do Protein Transformers Have Biological Intelligence? 19 Table 5: Compar...

    2016

  68. [192]

    residue number

    This indicates that the model’s learning ability is bottlenecked by other factors, e.g., the data itself. Positional Embedding. Unlike fixed start points for conventional vision tasks, the start point of the protein sequence, indicated by the “residue number” in the PDB data, ...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.