Pith. sign in

REVIEW 3 major objections 4 minor 52 references

Task- and dataset-specific information in protein language models

T0 review · 3 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read For protein language models, the most useful embeddings for a downstream task usually sit in an intermediate layer, and which layer that is depends more on the dataset than on the task itself.

desk verdict A broad, well-built empirical map of layer-wise PLM embeddings, but the 'dataset, not task' claim is not yet disentangled from label type. read the letter →

arxiv 2608.12090 v2 pith:WISN6CK5 submitted 2026-08-12 cs.LG q-bio.BM

Roman Joeres , Ilya Senatorov , Anastasia Kolchina , Dietrich Klakow , Olga V. Kalinina This is my paper · ORCID
classification cs.LGq-bio.BM
keywords proteinlanguagemodelslayerprobingembeddingsdownstreamtasksdeepmutationalscanninglatentspaceanalysispropertypredictiontransferlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks where, across the layers of a protein language model, the most useful information for a downstream prediction task actually lives. Probing 13 PLMs on 15 tasks from 11 datasets, it finds that the conventional choice, the model's final layer, is the best layer only about 18% of the time; performance typically rises over early layers, peaks mid-network, and falls again near the output. The central claim is that for whole-protein tasks the dataset, not the task itself, decides where this peak sits: datasets built from deep mutational scans of one or few proteins are best served by shallow embeddings, while datasets spanning many diverse natural proteins are best served by deeper ones. For residue-level tasks, performance rises monotonically toward the last layer, matching the models' token-level pre-training objective. The authors also report a sharp drop in prediction quality when the inputs are artificial, designed proteins rather than natural sequences.

What carries the argument

The central object is the per-layer probe: a linear classifier or regressor (plus k-nearest-neighbor probes as a nonlinear check) trained on mean-pooled embeddings from each layer, producing a performance-versus-depth curve for every model-task pair. These curves are compared across paired tasks on the same dataset (fluorescence regression vs. binary classification, Meltome temperature vs. species, DeepLoc binary vs. 10-class, SCOPe 3-class vs. 8-class secondary structure) to isolate dataset effects from task effects. Three latent-space metrics — intrinsic dimension estimated by TwoNN, neighborhood overlap between consecutive layers, and variance explained by the first ten principal components — corroborate the probe results. Fine-tuning ESM-2 150M on six tasks and re-probing all layers, plus evaluating the original pre-training losses per layer, supplies the mechanistic link between pre-training objective and layer trends.

What would settle it

Re-run the paired-task comparison with label decorrelation: for fluorescence, move the binary boundary or include the omitted middle interval and check whether classification and regression layer curves diverge; for Meltome, predict species after regressing out temperature (or vice versa). If the curves separate under either intervention, the shared dataset alone is not controlling the layer profile.

Watch

Extended reading notes

Core claim

Across 13 protein language models (five families, four architectures) and 15 downstream tasks, the paper establishes that the informativeness of embeddings varies non-monotonically with layer depth for whole-protein tasks: probe performance improves over the first layers, peaks between roughly the 10th and 90th percentile of depth, and declines in the deepest layers, so the last layer is best in only 17.92% of cases. The location of the peak is controlled by dataset composition, not by whether the task is regression or classification, binary or multi-class: variants-heavy deep mutational scanning datasets (fluorescence, GB1, parts of Rocklin, Tsuboyama) peak in shallow layers, whereas diverse multi-protein datasets (DeepLoc2.0, DeepSol, SCOPe40) peak in deeper layers. Paired tasks on the same dataset produce highly correlated layer-performance curves, which the paper reads as evidence that dataset identity, rather than task objective or difficulty, governs the layer profile. Residue-level tasks instead improve monotonically as layers deepen, which the authors attribute to alignment with the masked-language-model or next-token-prediction pre-training objectives. Fine-tuning a PLM on a downstream task makes every layer improve toward that task while degrading the pre-training objective, and performance on artificial proteins (Rosetta designs and ProGen-generated lysozymes) is markedly worse than on natural ones.

Load-bearing premise

The dataset-not-task conclusion requires that the two tasks paired on each dataset are effectively independent experiments; if correlated labels — such as the GMM-derived fluorescence classes and their parent regression values, or the closely related Meltome species and temperature labels — actually drive the similar layer curves, then the dataset identity conclusion loses its footing.

Editorial extensions

If this is right

  • In most protein-level tasks, last-layer embeddings are outperformed by embeddings from an intermediate layer; the best layer usually sits between the 10th and 90th percentile of model depth, and in some model-task pairs the relative improvement is several-fold.
  • For residue-level tasks such as secondary-structure and binding-site prediction, the last layer is typically the right choice because these tasks align with the token-level pre-training objective.
  • Dataset composition, not task type, determines the best layer for whole-protein tasks: deep-mutational-scan data favor shallow embeddings, while diverse natural protein sets favor deep embeddings.
  • Fine-tuning a PLM on a downstream task makes layers monotonically better for that task, so the non-monotonic profiles of frozen models are a signature of the pre-training objective rather than a fixed property of the model.
  • Only 15-20% of the training data is needed to identify a layer that achieves at least 95% of the best layer's performance, making layer selection feasible in practice.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An open extension: predicting mutational effects at the residue level on the same DMS datasets would show whether the shallow-layer preference comes from local sequence context rather than from how whole-protein embeddings are pooled; the paper does not run this control.
  • If the artificial-protein result generalizes, PLM-based scoring functions in generative design pipelines will tend to rank natural-like designs above equally functional novel sequences; this bias can be quantified by comparing model scores against measured activity on existing Rocklin and lysozyme data.
  • The authors' data also imply that pretraining data composition, not architecture alone, controls where task information sits: comparing models with identical architectures but different training corpora (e.g., ESM-2 versus ESMC) on the same datasets would make that explicit.
  • A neighboring problem that should inherit this result is remote homology and function search with embeddings, where layer choice is usually fixed; the same 15-20% subsampling trick could identify the best layer for retrieval tasks.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This manuscript presents a systematic probing study of 13 protein language models (PLMs) across 15 downstream tasks from 11 datasets. For each model and task, the authors train linear probes on per-layer embeddings, measure layer-wise performance, and supplement these measurements with three latent-space metrics (intrinsic dimension, variance@10, and neighborhood overlap). The main empirical findings are: (i) the deepest layer is rarely the best embedding source (reported in 17.92% of cases), with performance usually peaking in intermediate layers for protein-level tasks; (ii) residue-level tasks show a steady performance increase with depth, which the authors attribute to alignment with the masked-language-model pretraining objective; (iii) for whole-protein tasks, the paper claims that the dataset, rather than the task, determines the layer profile, with DMS-centered datasets favoring shallow layers and diverse-protein datasets favoring deeper layers; and (iv) performance on artificially generated proteins is markedly worse than on natural proteins. The paper also includes fine-tuning experiments showing that fine-tuning makes layer-wise performance monotonic toward the fine-tuning objective, and sparse-data experiments showing that 15-20% of training data often suffices to identify a near-best layer.

Significance. If the central findings hold, they are practically and scientifically useful: practitioners would be guided away from the default last-layer embedding and toward layer selection based on dataset structure, and the paper adds to the growing evidence that PLM representations are not uniformly organized by task relevance across depth. The study is unusually broad in model and task coverage, and the authors have made code and processed data available, which strengthens reproducibility. The paper is also careful in several secondary analyses, such as reporting three-seed stability for the sparse-probe layer selection and using DataSAIL for stratified splits. However, the headline distinction between dataset-driven and task-driven layer profiles is not yet established to journal standard, because the within-dataset task pairs used to separate these factors share heavily correlated labels, and the across-dataset comparison confounds dataset identity with label type. The central dichotomy therefore needs additional evidence or a more carefully scoped claim before the paper can be accepted.

major comments (3)
  1. [§2.1 and Figure 3A-D; §3] The claim that the layer-performance profile depends on the dataset, not the task, rests on the four within-dataset task pairs in Figure 3A-D, but these pairs do not provide independent label information. In §4.3 and Figure 6, the Fluorescence binary task is created by fitting a GMM to the same log-fluorescence values used for regression and discarding the interval between the two 2-sigma boundaries, so it is a thresholded coarsening of the regression label. The Meltome species and Tm tasks are acknowledged as closely related [42]; the DeepLoc binary membrane and 10-class localization labels are nested; and the SCOPe 3-class and 8-class secondary-structure labels are nested DSSP assignments. The high curve correlations (0.681-0.996) can therefore be explained by label correlation rather than by dataset identity. The paper should either provide task pairs with orthogonal labels on the same protein set or explicitly acknowledge and statistically control for the shared label information before claiming independence from objective, split, and difficulty.
  2. [§3 and Table 3] The grouping of the Tsuboyama stability dataset with the DMS-based, shallow-layer-favoring datasets is internally inconsistent. Section 3 describes these datasets as 'centered around one or several proteins and comprise many variants of them,' but Table 3 reports that the Tsuboyama set comprises 64 species, 63 CD-HIT clusters, and a mean sequence identity of 0.2192, which is far closer to the diverse-protein datasets than to Fluorescence or GB1 (which have 1 cluster each). This undermines the proposed dichotomy between 'DMS datasets' and 'diverse natural protein datasets.' The authors should either reclassify Tsuboyama, demonstrate that its multi-species DMS structure behaves like single-protein DMS for the relevant layer metrics, or otherwise revise the explanatory claim in Section 3.
  3. [Figure 1 and Table 1] The central quantitative summary that the last layer is best in only 17.92% of cases is reported without error bars, confidence intervals, or per-seed variance, and Figure 1B,C show only means over models. Because the best-layer selection is a discrete claim and the performance curves are close in some regions, the 17.92% figure could be sensitive to probe initialization, data splits, or other minor variations. The authors report three seeds only for the sparse-probe experiments in Figure 3E-H; the same rigor should be applied to the main best-layer analysis, or a sensitivity analysis should be provided. This is a load-bearing summary statistic for the first main claim.
minor comments (4)
  1. [Supplementary Figures 2 and 3] The label '2-NN Intrindic Dimension' contains a typo; it should read '2-NN Intrinsic Dimension.'
  2. [Figure 1] The legend states that Figure 1B and C show the mean over the 13 PLMs, but no dispersion measure is provided; adding per-layer standard deviations or model-specific faint curves would improve interpretability.
  3. [Table 1] Several entries are listed as NaN for ESM-2 3B, ProGen2-large, and ProtGPT2 on residue-level tasks; the table would be clearer if a footnote explained that these models were excluded from residue-level experiments, consistent with §4.4.
  4. [§2.2] The statement that the fine-tuned models 'show a clear effect' in Figure 4B-G is supported by visual inspection, but the main text does not report the magnitude of the improvement or a statistical comparison; a numeric summary would strengthen the claim.

Circularity Check

1 steps flagged · score 3.0 of 10

The 'dataset, not task' dichotomy rests on within-dataset task pairs whose labels are co-derived or nested, so the comparison is confounded with label structure rather than circular in its derivation.

  1. other [Section 2.1, Figure 3A-D; Section 4.3, Fluorescence paragraph, Figure 6]
    "The conversion to a classification dataset was performed by fitting a Gaussian Mixture Model to the data, computing 2-standard-deviation intervals around the means, and separating into the upper bound of the lower mode and the lower bound of the upper mode (Figure 6). The data between these two bounds is omitted because, when converting a regression problem to classification, the task is easier to learn when the two classes are separated by an interval rather than by a single threshold."

    By construction, the binary active/inactive labels are a thresholded function of the same log-fluorescence values used in the regression task. Thus the two 'different tasks' compared in Figure 3A share their label source, so any similarity in layer-performance curves (Pearson r=0.681) is partly forced by the label construction rather than by dataset identity. The same nesting holds for DeepLoc2.0 (membrane status is one of the localization classes), SCOPe40 SSP (3-class is a coarsening of 8-class), and Meltome (species and Tm are acknowledged to be closely related [42]). Since the 'dataset, not task' conclusion rests on these four within-dataset pairs, the comparison cannot separate dataset identity from task/label structure; the central dichotomy is confounded with label co-derivation.

full rationale

The paper's main quantitative results—last layer rarely best (17.92%), layer-performance curves, latent-space metrics, and fine-tuning shifts—are empirical measurements rather than derivations from fitted parameters, and the study is self-contained against external benchmarks (TAPE, ProteinGym, DeepLoc, etc.). There is no load-bearing self-citation chain: the only author self-citations (DataSAIL for stratified splits [43]; Zenodo archive [52]) are procedural. The one place where a conclusion is partially built into its evidence is the 'dataset, not task' claim (Section 3, Figure 3A-D). To show task-independence, the paper compares two tasks per dataset; but in each pair the labels are co-derived or nested: fluorescence binary labels are a GMM-thresholded coarsening of the same log-fluorescence values used for regression (Section 4.3, Figure 6), DeepLoc membrane status is a subset of the 10-class localization labels, SCOPe 3-class SSP is a coarsening of 8-class SSP, and Meltome Tm/species are acknowledged as closely related [42]. High curve correlations (0.68-0.996) are therefore expected from shared label information and cannot cleanly separate 'dataset identity' from 'task/label structure'; the dichotomy remains confounded. This is a validity threat to the central interpretation rather than a derivation that reduces to its input, so the circularity score is moderate (3), not high.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central empirical claims do not rest on a parametric derivation; the main free parameters are the GMM thresholds used to construct the fluorescence classification task and the k=10 hyperparameter for probes and metrics. The axioms are the standard probing assumption (probe accuracy equals information), mean pooling adequacy, no pretraining leakage, the questionable independence of within-dataset task pairs, and the TwoNN distributional assumption. No invented entities are introduced.

free parameters (2)
  • Fluorescence GMM classification thresholds = two class boundaries at 2-sigma from GMM means; data between bounds omitted
    Section 4.3 and Figure 6: the binary active/dead task is created by fitting a Gaussian Mixture Model to log-fluorescence and discarding the interval between the two 2-sigma bounds; the thresholds are fitted to the data and the exclusion changes the task.
  • k for k-NN probes and neighborhood overlap = k=10
    Section 4.1.1 and 4.1.2: k=10 is chosen for both the k-NN probe and the neighborhood overlap metric; the authors state results are stable, following Valeriani et al., but this is a hand-set hyperparameter that influences the metrics.
assumptions (5)
  • domain assumption Linear probe accuracy measures the information a layer contains for a downstream task
    Section 4.1.2 defines layer probes following Alain & Bengio; the interpretation of probe performance as 'informativeness' throughout Results assumes linear separability is a faithful proxy for stored information.
  • domain assumption Mean pooling of residue embeddings is an adequate representation for whole-protein downstream tasks
    Section 4.1 defines h_l_i as the mean over tokens; all protein-level tasks use this pooling, so layer comparisons are conditional on this aggregation.
  • domain assumption Benchmark datasets do not substantially overlap with PLM pretraining corpora
    SCOPe40, DeepLoc, DeepSol, and DMS datasets contain sequences that may appear in UniRef/BFD pretraining data; the paper does not test for leakage, which could inflate shallow-layer probe performance.
  • domain assumption Within-dataset task pairs in Figure 3 are sufficiently independent to attribute shared layer trends to the dataset
    Section 2.1 compares regression vs classification, binary vs 10-class, Tm vs species, and 3- vs 8-class SSP; the fluorescence pair is derived from the same continuous labels and Meltome tasks are acknowledged as related, so the independence assumption is questionable.
  • standard math The TwoNN intrinsic dimension estimator is valid for PLM latent spaces
    Section 4.1.1 uses TwoNN assuming local distance ratios follow a Pareto distribution; this is an unproved statistical assumption for these high-dimensional embeddings.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Task- and dataset-specific information in protein language models." pith.science (2026). https://pith.science/paper/WISN6CK5

@misc{pith2026260812090,
  author       = {Pith},
  title        = {Pith review of: Task- and dataset-specific information in protein language models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WISN6CK5}},
  note         = {Machine review of arXiv:2608.12090}
}
read the original abstract

Protein language models (PLMs) have transferred the latest advances from natural language processing to computational biology. These models, trained on large corpora of protein sequence data, are widely used to translate amino acid sequences into latent-space embeddings, ready for use in diverse downstream tasks (DTs). By a common consensus, embeddings from the model's last layer are used, and the model's internal behavior remains poorly understood. We analyzed 13 PLMs across 15 DTs from 11 datasets to investigate the informativeness of embeddings created in intermediate PLM layers. We trained probe models on embeddings from each layer, compared their performance, and computed characteristics of the latent spaces they span to estimate the information they contain, and found that the last layers of PLMs rarely contained embeddings that led to the best results on downstream tasks. Furthermore, we identified a connection between DTs and the distribution across PLMs' layers of the relevant information to predict that task. For example, similarity between the pre-training objective and the objective of predicting properties of individual residues leads to a steady increase in understanding of such tasks across the layers of PLMs. On the other hand, for whole-protein tasks, we observe that the dataset, rather than the task itself, defines PLMs' ability to perform well on a DT. Embeddings from shallow layers of PLMs perform better for datasets that contain deep mutational scan (DMS) data, while datasets containing diverse natural proteins find most useful embeddings in the models' deeper layers. Additionally, we discover that the performance of PLMs drops significantly when tasks are introduced for artificial proteins.

Figures

Figures reproduced from arXiv: 2608.12090 by the authors.

Figure 1
Figure 1. Performance Overview. A: Relative improvement of the best layer of each PLM over the last layer on all DTs. B and C: Average performance for each dataset across all models for regression (B) and classification (C) tasks. We show the mean over the 13 PLMs. Supplementary [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Changes of latent spaces spanned by individual layers of PLMs. The metrics are defined in the Methods section (rows: Intrinsic Dimension, Neighborhood Overlap, and Explained Variance by the first 10 principal components) and evaluated on the latent spaces spanned by the 13 PLMs. Supplementary Figures 2 and 3 visualize these metrics for the individual PLMs. across different PLMs, datasets, and DTs. Moderate exception… view at source ↗
Figure 3
Figure 3. Ablation studies. A-D: Pairwise comparisons of two tasks for one dataset, showing that model performance patterns do not change across DTs within the same model and dataset. E-H: Subsampling of the DT training dataset shows that relatively little data is sufficient to find a PLM layer that achieves close-to-optimal performance. These experiments were computed over three seeds; in each of them, the same layer was ide… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: The effect of fine-tuning on model performance. Subfigure A shows that native PLMs improve in each layer on their pretraining objective. This is true for both the MLM objective (ESM-2 models) and the NTP objective (ProGen2 models). It also shows that fine-tuning the PL…
Figure 5
Figure 5. Figure 5: Artificial protein sets A shows the comparison of the composition of the Rocklin stability dataset into performance on the DMS and the designed proteins. Importantly, we did not train separate models but only one and evaluated this on the DMS and artificial sequences s…
Figure 6
Figure 6. Figure 6: Label distribution in the Fluorescence dataset. The blue histogram shows the distribution of log-fluorescence values. The dashed red lines high￾light the decision boundaries for the conversion from regression to classification. The data falling in be￾tween have been om…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

52 extracted references · 42 canonical work pages

  1. [42]

    Supervised learning of protein melting temperature: Cross-species vs

    Sebastián García López, Jesper Salomon, and Wouter Boomsma. Supervised learning of protein melting temperature: Cross-species vs. species-specific prediction.Proteins: Structure, Function, and Bioinformatics, 93(12):2158– 2166, 2025

    work page 2025

  2. [1]

    Alexander Rives, Joshua Meier, Tom Sercu, Sid- dharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C Lawrence Zitnick, Jerry 14 Ma, et al. Biological structure and func- tion emerge from scaling unsupervised learn- ing to 250 million protein sequences.Pro- ceedings of the national academy of sciences, 118(15):e2016239118, 2021

    work page 2021

  3. [2]

    Comparative assessment of protein large language models for enzyme commission number prediction.BMC bioinfor- matics, 26(1):68, 2025

    João Capela, Maria Zimmermann-Kogadeeva, Aalt DJ van Dijk, Dick de Ridder, Oscar Dias, and Miguel Rocha. Comparative assessment of protein large language models for enzyme commission number prediction.BMC bioinfor- matics, 26(1):68, 2025

    work page 2025

  4. [3]

    Language mod- els can identify enzymatic binding sites in pro- tein sequences.Computational and structural biotechnology journal, 23:1929–1937, 2024

    Yves Gaetan Nana Teukam, Loïc Kwate Dassi, Matteo Manica, Daniel Probst, Philippe Schwaller, and Teodoro Laino. Language mod- els can identify enzymatic binding sites in pro- tein sequences.Computational and structural biotechnology journal, 23:1929–1937, 2024

    work page 1929

  5. [4]

    Iglm: Infilling language model- ing for antibody sequence design.Cell systems, 14(11):979–989, 2023

    Richard W Shuai, Jeffrey A Ruffolo, and Jef- frey J Gray. Iglm: Infilling language model- ing for antibody sequence design.Cell systems, 14(11):979–989, 2023

    work page 2023

  6. [5]

    Protein language models: Applications and perspectives

    Mickael Leclercq and Arnaud Droit. Protein language models: Applications and perspectives. Journal of Proteome Research, 25(2):507–524, 2025

    work page 2025

  7. [6]

    Under- standing intermediate layers using linear classi- fier probes.arXiv preprint arXiv:1610.01644, 2016

    Guillaume Alain and Yoshua Bengio. Under- standing intermediate layers using linear classi- fier probes.arXiv preprint arXiv:1610.01644, 2016

    arXiv 2016

  8. [7]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InProceedings of the IEEE con- ference on computer vision and pattern recogni- tion, pages 770–778, 2016

    work page 2016

Show all 52 references
  1. [8]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In2009 IEEE Con- ference on Computer Vision and Pattern Recog- nition, pages 248–255, 2009

    2009

  2. [9]

    Bert: Pre-training of deep bidirectional transformers for language un- derstanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language un- derstanding. InProceedings of the 2019 con- ference of the North American chapter of the association for computational linguistics: hu- man ...

    2019

  3. [10]

    The geometry of hidden representations of large transformer models.Ad- vances in Neural Information Processing Sys- tems, 36:51234–51252, 2023

    Lucrezia Valeriani, Diego Doimo, Francesca Cu- turello, Alessandro Laio, Alessio Ansuini, and Alberto Cazzaniga. The geometry of hidden representations of large transformer models.Ad- vances in Neural Information Processing Sys- tems, 36:51234–51252, 2023

    2023

  4. [11]

    Bertology meets bi- ology: Interpreting attention in protein language models.arXiv preprint arXiv:2006.15222, 2020

    Jesse Vig, Ali Madani, Lav R Varsh- ney, Caiming Xiong, Richard Socher, and Nazneen Fatema Rajani. Bertology meets bi- ology: Interpreting attention in protein language models.arXiv preprint arXiv:2006.15222, 2020

    2006 arXiv

  5. [12]

    Layer by layer: Uncovering hid- den representations in language models.arXiv preprint arXiv:2502.02013, 2025

    Oscar Skean, Md Rifat Arefin, Dan Zhao, Niket Patel, Jalal Naghiyev, Yann LeCun, and Ravid Shwartz-Ziv. Layer by layer: Uncovering hid- den representations in language models.arXiv preprint arXiv:2502.02013, 2025

    2025 arXiv

  6. [13]

    The underlying structures of self- attention: symmetry, directionality, and emer- gent dynamics in transformer training.arXiv preprint arXiv:2502.10927, 2025

    Matteo Saponati, Pascal Sager, Pau Vilimelis Aceituno, Thilo Stadelmann, and Benjamin Grewe. The underlying structures of self- attention: symmetry, directionality, and emer- gent dynamics in transformer training.arXiv preprint arXiv:2502.10927, 2025

    2025 arXiv

  7. [14]

    Generative pretraining from pixels

    Mark Chen, Alec Radford, Rewon Child, Jef- frey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. InInternational conference on machine learn- ing, pages 1691–1703. PMLR, 2020

    2020

  8. [15]

    Evolutionary-scale prediction of atomic-level protein structure with a language model.Sci- ence, 379(6637):1123–1130, 2023

    Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, et al. Evolutionary-scale prediction of atomic-level protein structure with a language model.Sci- ence, 379(6637):1123–1130, 2023

    2023

  9. [16]

    Layer probing improves kinase functional prediction with protein language models.arXiv preprint arXiv:2512.00376, 2025

    Ajit Kumar and IndraPrakash Jha. Layer probing improves kinase functional prediction with protein language models.arXiv preprint arXiv:2512.00376, 2025. 15

    2025

  10. [17]

    Visualizing and understanding convolutional networks

    Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pages 818–833. Springer, 2014

    2014

  11. [18]

    Quantify- ing uncertainty in protein representations across models and tasks.Nature Methods, pages 1–9, 2026

    R Prabakaran and Yana Bromberg. Quantify- ing uncertainty in protein representations across models and tasks.Nature Methods, pages 1–9, 2026

    2026

  12. [19]

    Local fitness land- scape of the green fluorescent protein.Nature, 533(7603):397–401, 2016

    Karen S Sarkisyan, Dmitry A Bolotin, Mar- garita V Meer, Dinara R Usmanova, Alexan- der S Mishin, George V Sharonov, Dmitry N Ivankov, Nina G Bozhanova, Mikhail S Bara- nov, Onuralp Soylemez, et al. Local fitness land- scape of the green fluorescent protein.Nature, 533(7603):3...

    2016

  13. [20]

    A comprehensive biophysical description of pairwise epistasis throughout an entire protein domain.Current biology, 24(22):2643–2651, 2014

    C Anders Olson, Nicholas C Wu, and Ren Sun. A comprehensive biophysical description of pairwise epistasis throughout an entire protein domain.Current biology, 24(22):2643–2651, 2014

    2014

  14. [21]

    Global analysis of protein folding using mas- sively parallel design, synthesis, and testing.Sci- ence, 357(6347):168–175, 2017

    Gabriel J Rocklin, Tamuka M Chidyausiku, Inna Goreshnik, Alex Ford, Scott Houliston, Alexan- der Lemak, Lauren Carter, Rashmi Ravichan- dran, Vikram K Mulligan, Aaron Chevalier, et al. Global analysis of protein folding using mas- sively parallel design, synthesis, and testing...

    2017

  15. [22]

    Mega-scale experimental analysis of protein folding stability in biology and design.Nature, 620(7973):434–444, 2023

    Kotaro Tsuboyama, Justas Dauparas, Jonathan Chen, Elodie Laine, Yasser Mohseni Behba- hani, Jonathan J Weinstein, Niall M Mangan, Sergey Ovchinnikov, and Gabriel J Rocklin. Mega-scale experimental analysis of protein folding stability in biology and design.Nature, 620(7973):43...

    2023

  16. [23]

    Large language models generate functional protein se- quences across diverse families.Nature biotech- nology, 41(8):1099–1106, 2023

    Ali Madani, Ben Krause, Eric R Greene, Subu Subramanian, Benjamin P Mohr, James M Holton, Jose Luis Olmos Jr, Caiming Xiong, Zachary Z Sun, Richard Socher, et al. Large language models generate functional protein se- quences across diverse families.Nature biotech- nology, 41(8...

    2023

  17. [24]

    Meltome atlas—thermal proteome stability across the tree of life.Nature methods, 17(5):495–503, 2020

    Anna Jarzab, Nils Kurzawa, Thomas Hopf, Matthias Moerch, Jana Zecha, Niels Lei- jten, Yangyang Bian, Eva Musiol, Melanie Maschberger, Gabriele Stoehr, et al. Meltome atlas—thermal proteome stability across the tree of life.Nature methods, 17(5):495–503, 2020

    2020

  18. [25]

    Deepsol: a deep learning framework for sequence-based protein solubility prediction.Bioinformatics, 34(15):2605–2613, 2018

    Sameer Khurana, Reda Rawi, Khalid Kunji, Gwo-Yu Chuang, Halima Bensmail, and Raghvendra Mall. Deepsol: a deep learning framework for sequence-based protein solubility prediction.Bioinformatics, 34(15):2605–2613, 2018

    2018

  19. [26]

    Deeploc 2.0: multi-label subcellular localization prediction using protein language models.Nucleic acids research, 50(W1):W228–W234, 2022

    Vineet Thumuluri, José Juan Almagro Ar- menteros, Alexander Rosenberg Johansen, Hen- rik Nielsen, and Ole Winther. Deeploc 2.0: multi-label subcellular localization prediction using protein language models.Nucleic acids research, 50(W1):W228–W234, 2022

    2022

  20. [27]

    John-Marc Chandonia, Lindsey Guan, Shiangyi Lin, Changhua Yu, Naomi K Fox, and Steven E Brenner. Scope: improvements to the structural classification of proteins–extended database to facilitate variant interpretation and machine learning.Nucleic acids research, 50(D1):D553– D559, 2022

    2022

  21. [28]

    Protein embeddings and deep learning pre- dict binding residues for various ligand classes

    Maria Littmann, Michael Heinzinger, Christian Dallago, Konstantin Weissenow, and Burkhard Rost. Protein embeddings and deep learning pre- dict binding residues for various ligand classes. Scientific reports, 11(1):23916, 2021

    2021

  22. [29]

    Wolfgang Kabsch and Christian Sander. Dic- tionary of protein secondary structure: pattern recognition of hydrogen-bonded and geometri- cal features.Biopolymers: Original Research on Biomolecules, 22(12):2577–2637, 1983

    1983

  23. [30]

    ESM Cambrian: Revealing the mysteries of proteins with unsupervised learn- ing.EvolutionaryScale Website, 12 2024

    ESM Team. ESM Cambrian: Revealing the mysteries of proteins with unsupervised learn- ing.EvolutionaryScale Website, 12 2024

    2024

  24. [31]

    Prottrans: to- ward understanding the language of life through self-supervised learning.IEEE transactions on pattern analysis and machine intelligence, 44(10):7112–7127, 2021

    Ahmed Elnaggar, Michael Heinzinger, Chris- tian Dallago, Ghalia Rehawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph An- gerer, Martin Steinegger, et al. Prottrans: to- ward understanding the language of life through self-supervised learning.IEEE transactions on pat...

    2021

  25. [32]

    Bilingual language model for protein sequence and structure.NAR Genomics and Bioinformat- ics, 6(4):lqae150, 2024

    Michael Heinzinger, Konstantin Weissenow, Joaquin Gomez Sanchez, Adrian Henkel, Milot Mirdita, Martin Steinegger, and Burkhard Rost. Bilingual language model for protein sequence and structure.NAR Genomics and Bioinformat- ics, 6(4):lqae150, 2024

    2024

  26. [33]

    Progen2: exploring the boundaries of protein language models.Cell systems, 14(11):968–978, 2023

    Erik Nijkamp, Jeffrey A Ruffolo, Eli N Wein- stein, Nikhil Naik, and Ali Madani. Progen2: exploring the boundaries of protein language models.Cell systems, 14(11):968–978, 2023

    2023

  27. [34]

    Protgpt2 is a deep unsupervised lan- guage model for protein design.Nature commu- nications, 13(1):4348, 2022

    Noelia Ferruz, Steffen Schmidt, and Birte Höcker. Protgpt2 is a deep unsupervised lan- guage model for protein design.Nature commu- nications, 13(1):4348, 2022

    2022

  28. [35]

    A new algorithm for data com- pression.The C Users Journal, 12(2):23–38, 1994

    Philip Gage. A new algorithm for data com- pression.The C Users Journal, 12(2):23–38, 1994

    1994

  29. [36]

    Attention is all you need.Advances in neural information processing systems, 30, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017

    2017

  30. [37]

    Exploring the limits of transfer learning with a unified text- to-text transformer.Journal of machine learning research, 21(140):1–67, 2020

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text- to-text transformer.Journal of machine learning research, 21(140):1–67, 2020

    2020

  31. [38]

    Improving language understanding by generative pre-training, 2018

    Alec Radford, Karthik Narasimhan, Tim Sali- mans, and Ilya Sutskever. Improving language understanding by generative pre-training, 2018

    2018

  32. [39]

    Estimating the intrinsic dimension of datasets by a minimal neighbor- hood information.Scientific reports, 7(1):12140, 2017

    Elena Facco, Maria d’Errico, Alex Rodriguez, and Alessandro Laio. Estimating the intrinsic dimension of datasets by a minimal neighbor- hood information.Scientific reports, 7(1):12140, 2017

    2017

  33. [40]

    Hierarchical nucleation in deep neural networks.Advances in Neural Information Processing Systems, 33:7526–7536, 2020

    Diego Doimo, Aldo Glielmo, Alessio Ansuini, and Alessandro Laio. Hierarchical nucleation in deep neural networks.Advances in Neural Information Processing Systems, 33:7526–7536, 2020

    2020

  34. [41]

    Karl Pearson. Liii. on lines and planes of closest fit to systems of points in space.The London, Edinburgh, and Dublin philosophical magazine and journal of science, 2(11):559–572, 1901

    1901

  35. [43]

    Data splitting to avoid information leakage with datasail.Nature Communications, 16(1):3337, 2025

    Roman Joeres, David B Blumenthal, and Olga V Kalinina. Data splitting to avoid information leakage with datasail.Nature Communications, 16(1):3337, 2025

    2025

  36. [44]

    Bert rediscovers the classical nlp pipeline

    Ian Tenney, Dipanjan Das, and Ellie Pavlick. Bert rediscovers the classical nlp pipeline. In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 4593–4601, 2019

    2019

  37. [45]

    Uniprot: the universal protein knowledgebase in 2025.Nucleic acids research, 52(D 1):D609– D617, 2024

    Alex Bateman, Maria-Jesus Martin, Sandra Or- chard, Michele Magrane, Aduragbemi Adesina, Shadab Ahmad, Emily H Bowler-Barnett, Hema Bye-A-Jee, David Carpentier, Paul Denny, et al. Uniprot: the universal protein knowledgebase in 2025.Nucleic acids research, 52(D 1):D609– D617, 2024

    2025

  38. [46]

    Mgnify: the microbiome sequence data analy- sis resource in 2023.Nucleic acids research, 51(D1):D753–D759, 2023

    Lorna Richardson, Ben Allen, Germana Baldi, Martin Beracochea, Maxwell L Bileschi, Tony Burdett, Josephine Burgin, Juan Caballero- Pérez, Guy Cochrane, Lucy J Colwell, et al. Mgnify: the microbiome sequence data analy- sis resource in 2023.Nucleic acids research, 51(D1):D753–D...

    2023

  39. [47]

    The genome portal of the de- partment of energy joint genome institute: 2014 updates.Nucleic acids research, 42(D1):D26– D31, 2014

    Henrik Nordberg, Michael Cantor, Serge Dusheyko, Susan Hua, Alexander Poliakov, Igor Shabalov, Tatyana Smirnova, Igor V Grigoriev, and Inna Dubchak. The genome portal of the de- partment of energy joint genome institute: 2014 updates.Nucleic acids research, 42(D1):D26– D31, 2014

    2014

  40. [48]

    Protein-level assembly increases pro- tein sequence recovery from metagenomic sam- 17 ples manyfold.Nature methods, 16(7):603–606, 2019

    Martin Steinegger, Milot Mirdita, and Johannes Söding. Protein-level assembly increases pro- tein sequence recovery from metagenomic sam- 17 ples manyfold.Nature methods, 16(7):603–606, 2019

    2019

  41. [49]

    Cd-hit: a fast program for clustering and comparing large sets of protein or nucleotide sequences.Bioinformat- ics, 22(13):1658–1659, 2006

    Weizhong Li and Adam Godzik. Cd-hit: a fast program for clustering and comparing large sets of protein or nucleotide sequences.Bioinformat- ics, 22(13):1658–1659, 2006

    2006

  42. [50]

    Evaluating protein transfer learning with tape.Advances in neural information processing systems, 32, 2019

    Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Peter Chen, John Canny, Pieter Abbeel, and Yun Song. Evaluating protein transfer learning with tape.Advances in neural information processing systems, 32, 2019

    2019

  43. [51]

    Proteingym: Large- scale benchmarks for protein fitness prediction and design.Advances in neural information processing systems, 36:64331–64379, 2023

    Pascal Notin, Aaron Kollasch, Daniel Ritter, Lood Van Niekerk, Steffanie Paul, Han Spin- ner, Nathan Rollins, Ada Shaw, Rose Orenbuch, Ruben Weitzman, et al. Proteingym: Large- scale benchmarks for protein fitness prediction and design.Advances in neural information processing...

    2023

  44. [52]

    Kalinina

    Roman Joeres, Ilya Senatorov, and Olga V . Kalinina. Task- and Dataset-Specific Informa- tion in Protein Language Models, August 12 2026. 18 0.2 0.3 0.4 0.5 0.6 0.7MCC ( ) A Fluorescence binary 0.2 0.3 0.4 0.5 0.6 0.7MCC ( ) D Meltome Atlas Species 0.40 0.45 0.50 0.55 0.60 0.6...

    2026

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.