Exploring evolution-aware & -free protein language models as protein function predictors

Fajie Yuan; Fei Yang; Fusong Ju; Hui Wang; Jin Su; Kevin K. Yang; Mingyang Hu; Qiuyang Ding

RECORD

Exploring evolution-aware & -free protein language models as protein function predictors

Mingyang Hu, Fajie Yuan, Kevin K. Yang, Fusong Ju, Jin Su, Hui Wang, Fei Yang, Qiuyang Ding

Quantitative Methods · Submitted 14 Jun 2022

abstract compiled · 2026-09-23 · no language-model review

The authors' main claim We compare these models by empirical study along with new insights and conclusions.

how it is stated · asserted 0.55 read the record →

Comment (0) Read on arXiv ↗

Cite ▾

arXiv ID2206.06583 (version 2)

Pith IDCLAFCQAG, Pith's permanent record of this paper and its review

CitationFajie Yuan, Fei Yang, Fusong Ju, Hui Wang, Jin Su, Kevin K. Yang, Mingyang Hu, Qiuyang Ding. "Exploring evolution-aware & -free protein language models as protein function predictors." arXiv:2206.06583 (2022). https://pith.science/paper/CLAFCQAG

Claim paper

The compiled record

The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.

Open the compiled record

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 7,443 input tokens.

What the abstract itself states, as probabilities: claims first 0.11 · states limitation 0.12 · human participants 0.03 · controlled comparison 0.53 · randomized assignment 0.01 · states quantitative result 0.03 · compares with prior results 0.78 · mentions code or data availability 0.99.

The abstract, sentence by sentence

arXiv listing abstract v2. Results and the main claim are underlined; show every role.

Large-scale Protein Language Models (PLMs) have improved performance in protein prediction tasks, ranging from 3D structure prediction to various function predictions. In particular, AlphaFold, a ground-breaking AI system, could potentially reshape structural biology. However, the utility of the PLM module in AlphaFold, Evoformer, has not been explored beyond structure prediction. In this paper, we investigate the representation ability of three popular PLMs: ESM-1b (single sequence), MSA-Transformer (multiple sequence alignment) and Evoformer (structural), with a special focus on Evoformer. Specifically, we aim to answer the following key questions: (i) Does the Evoformer trained as part of AlphaFold produce representations amenable to predicting protein function? (ii) If yes, can Evoformer replace ESM-1b and MSA-Transformer? (ii) How much do these PLMs rely on evolution-related protein data? In this regard, are they complementary to each other? We compare these models by empirical study along with new insights and conclusions. All code and datasets for reproducibility are available at https://github.com/elttaes/Revisiting-PLMs.

background contribution method result interpretation main claim

What the classifier reports for each sentence. The number is the probability it gave the label.
#	sentence	role	the authors' own work	states the main conclusion	strength
s0	Large-scale Protein Language Models (PLMs) have improved performance in protein prediction tasks, ranging from 3D structure prediction to various function predictions.	background 1.00	0.05	0.02	asserted 1.00
s1	In particular, AlphaFold, a ground-breaking AI system, could potentially reshape structural biology.	background 1.00	0.03	0.02	hedged possibility 1.00
s2	However, the utility of the PLM module in AlphaFold, Evoformer, has not been explored beyond structure prediction.	gap 1.00	0.05	0.03	asserted 0.99
s3	In this paper, we investigate the representation ability of three popular PLMs: ESM-1b (single sequence), MSA-Transformer (multiple sequence alignment) and Evoformer (structural), with a special focus on Evoformer.	contribution 0.85	0.58	0.03	asserted 0.92
s4	Specifically, we aim to answer the following key questions: (i) Does the Evoformer trained as part of AlphaFold produce representations amenable to predicting protein function?	contribution 0.93	0.17	0.03	unclear 0.69
s5	(ii) If yes, can Evoformer replace ESM-1b and MSA-Transformer?	contribution 0.74	0.06	0.02	unclear 0.68
s6	(ii) How much do these PLMs rely on evolution-related protein data?	contribution 0.51	0.13	0.02	unclear 0.92
s7	In this regard, are they complementary to each other?	contribution 0.45	0.08	0.02	unclear 0.97
s8	We compare these models by empirical study along with new insights and conclusions.	contribution 0.55	0.76	0.05	asserted 0.55
s9	All code and datasets for reproducibility are available at https://github.com/elttaes/Revisiting-PLMs.	contribution 0.63	0.42	0.02	asserted 0.98

What kind of paper the classifier reports this is: empirical study 1.00 · evidence from observation or experiment 0.68.

What was examined, and what was not

sources read

arXiv listing abstract v2

not examined

full text
figures and tables
references and citing sentences
relations to other papers

levels run

P abstract

not run

S sections
T body sentences
C citations
R cross-paper relations

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. Every number in the measurements table is the authors', checked by code. Record as JSON

The paper

Original abstract

Large-scale Protein Language Models (PLMs) have improved performance in protein prediction tasks, ranging from 3D structure prediction to various function predictions. In particular, AlphaFold, a ground-breaking AI system, could potentially reshape structural biology. However, the utility of the PLM module in AlphaFold, Evoformer, has not been explored beyond structure prediction. In this paper, we investigate the representation ability of three popular PLMs: ESM-1b (single sequence), MSA-Transformer (multiple sequence alignment) and Evoformer (structural), with a special focus on Evoformer. Specifically, we aim to answer the following key questions: (i) Does the Evoformer trained as part of AlphaFold produce representations amenable to predicting protein function? (ii) If yes, can Evoformer replace ESM-1b and MSA-Transformer? (ii) How much do these PLMs rely on evolution-related protein data? In this regard, are they complementary to each other? We compare these models by empirical study along with new insights and conclusions. All code and datasets for reproducibility are available at https://github.com/elttaes/Revisiting-PLMs.

Human review

How Pith checks papers → · Full audit of this paper →

References

How the literature uses it

3 arXiv papers cite this in 3 sentences

Each citing sentence is taken from the citing paper's LaTeX source and typed by Pith's classifier. A label appears at probability 0.8 or more (0.95 for support), a contest at 0.7 or more; the quoted words are the citing authors'. How each label was checked · JSON

Papers citing it · 2 sentences without a confident label

Cites it as background 1 paper

It is trained on extensive data encompassing protein sequences and structures to predict the 3D structure of a given protein sequence [ref] .

T Cell Receptor Protein Sequences and Sparse Coding: A Novel Approach to Cancer Classification · Related Work · 0.91

What it cites · 3 sentences without a confident label

Uses the method or tool of 1 paper

On the other hand, given the powerful function representation of ESM-1b, we design a fast homology retrieval framework using Siamese ESM-1b, following [ref] .

Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks · Relationship between Evolution-aware & -free PLMs · 0.91

Cites as background 2 papers

Recently, with the advent of large transformer models in natural language processing (NLP), large PLMs using the Transformer [ref] architecture and BERT [ref] denoising task have been widely adopted [ref] .[Note that the bioRxiv version of [ref] is concurrent with [ref] and [ref] . ] PLMs trained on large sequence databases have been successfully applied for various protein related tasks, including secondary structure prediction [ref] , contact prediction [ref] , 3D structure prediction [ref] , annotation prediction [ref] , signal peptide prediction [ref] , intracellular localization prediction [ref] , protein-protein interaction prediction [ref] , and fitness prediction [ref] .

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding · Related Work · 0.95
Recently, with the advent of large transformer models in natural language processing (NLP), large PLMs using the Transformer [ref] architecture and BERT [ref] denoising task have been widely adopted [ref] .[Note that the bioRxiv version of [ref] is concurrent with [ref] and [ref] . ] PLMs trained on large sequence databases have been successfully applied for various protein related tasks, including secondary structure prediction [ref] , contact prediction [ref] , 3D structure prediction [ref] , annotation prediction [ref] , signal peptide prediction [ref] , intracellular localization prediction [ref] , protein-protein interaction prediction [ref] , and fitness prediction [ref] .

ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance Computing · Related Work · 0.95

Discussion

Know something about this paper? Leave a comment, or highlight any passage above to comment on it.