Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

S$^2$ALM: Sequence-Structure Pre-trained Large Language Model for Comprehensive Antibody Representation Learning

T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper introduces S2ALM, a large pre-trained antibody language model that jointly encodes amino-acid sequences and 3D structures via Foldseek's 3Di tokens, and claims state-of-the-art performance across antigen binding prediction, B…

desk verdict A real large-scale structure-aware antibody pretraining effort with a plausible core idea, but the SOTA claims outrun the reported statistics and the paper as posted is not reproducible. read the letter →

arxiv 2411.15215 v1 pith:43ZSTLS7 submitted 2024-11-20 cs.LG cs.AIq-bio.BM

Mingze Yin , Hanjing Zhou , Jialu Wu , Yiheng Zhu , Yuxuan Zhan , Zitai Kong , Hongxia Xu , Chang-Yu Hsieh
show 3 more authors
Jintai Chen Tingjun Hou Jian Wu
This is my paper · ORCID
classification cs.LGcs.AIq-bio.BM MSC 68T0792D20
keywords antibodylanguagemodelpre-trainingproteinstructure3Ditokenssequence-structurematchingcross-levelreconstructionantigenbindingpredictionCDRdesign
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that existing antibody language models are limited because they read only the amino-acid sequence, ignoring the 3D structure that actually determines binding. It proposes S2ALM, a large pre-trained model that jointly encodes antibody sequences and structures, using Foldseek's 3Di tokens to turn structure into a sequence-like input. Pre-trained on 75 million sequences and 11.7 million structures, S2ALM claims state-of-the-art results on antigen binding prediction, B cell maturation classification, paratope prediction, binding affinity prediction, and CDR design. The implication, if correct, is that a single general-purpose antibody model can improve both understanding and generation of antibodies for therapeutic development.

What carries the argument

The key mechanism is the hybrid token vocabulary of 20 standard amino acids plus Foldseek's 20 3Di tokens, which discretize each residue's local 3D interaction into a single symbol. Foldseek converts a 3D structure into a 1D 3Di sequence, allowing the transformer to process sequence and structure in a unified format. Two pre-training objectives then bind the levels: Sequence-Structure Matching (SSM) is a binary classification of whether a 1D and a 3Di sequence come from the same antibody, and Cross-Level Reconstruction (CLR) masks tokens in one level and reconstructs them from the other. This machinery is what lets a single model absorb both modalities during masked language modeling.

What would settle it

Train S2ALM with the same pipeline but with 3Di sequences randomly shuffled or replaced by a fixed token per residue, so the structural channel carries no real geometry. If performance on antigen binding, paratope, and affinity tasks stays at the reported levels, the structural encoding is not the cause of the gains; if performance drops, the structural information is causative.

Watch

Extended reading notes

Core claim

The central claim is that injecting structural information into antibody pre-training, through a hybrid vocabulary of amino-acid and 3Di tokens, yields a general antibody foundation model that outperforms sequence-only antibody and protein language models. The paper demonstrates this with a two-stage pre-training pipeline: first general protein sequence-structure learning, then antibody-specific multi-level learning with two new objectives, Sequence-Structure Matching and Cross-Level Reconstruction. The resulting representations capture functional specificity, species, and isotype information, and the model sets new state-of-the-art numbers across four understanding tasks and one generation task.

Load-bearing premise

The claim depends on Foldseek's 20-state 3Di tokens preserving the structural details that determine antibody binding; if those discretized tokens discard the geometry that matters, the structural advantage claimed over sequence-only models would not be genuine.

Editorial extensions

If this is right

  • A single pre-trained model can serve both understanding tasks (binding, maturation, paratope, affinity) and generation (CDR infilling), replacing task-specific antibody models.
  • Structure-aware representations improve paratope prediction, the token-level task most directly tied to 3D binding interfaces, as shown in the paper's ablation where removing SSM hurts paratope prediction most.
  • The two-stage curriculum (general protein then antibody-specific) lets scarce antibody structure data be supplemented by abundant protein structure data without losing antibody specificity.
  • With 3Di tokens, existing sequence-model architectures can be adapted to structure without geometric deep nets, lowering the barrier for structure-aware antibody models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same 3Di-based hybrid pretraining could transfer to other protein families with few experimental structures, such as T-cell receptors or nanobodies, where predicted structures from AlphaFold-like models are plentiful.
  • As Foldseek's 3Di vocabulary improves or is replaced by finer structural tokens, S2ALM-style models should scale in performance without architectural change; the current 20-state discretization is likely a bottleneck, as the authors acknowledge.
  • A testable extension is to apply the SSM and CLR objectives to paired sequence-structure data from proteins other than antibodies to see if the structural benefit generalizes beyond the immune system.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents S2ALM, a 650M-parameter antibody language model pre-trained with a two-stage hierarchical scheme on 75 million sequences and 11.7 million structures. Structures are discretized with Foldseek into 3Di tokens. Stage I performs masked language modeling on 1D and 3Di sequences from general proteins; stage II adds two antibody-specific objectives, Sequence-Structure Matching (SSM) and Cross-Level Reconstruction (CLR). The model is evaluated on antigen-binding classification, B-cell maturation classification, paratope prediction, binding-affinity regression, and CDR-H3 design, reporting state-of-the-art results against sequence-only antibody and protein language models. The paper also presents t-SNE analyses, attention-based structural interpretability, and AlphaFold3-based structural evaluation of designed CDRs.

Significance. If the reported results are statistically robust, S2ALM would be a substantial advance: it is among the first antibody-specific foundation models to incorporate structure at scale, and the two-stage design with the SSM/CLR objectives is a reasonable and potentially reusable approach. The paper provides a broad set of downstream benchmarks, an ablation showing that both objectives contribute, and interpretability analyses; these are strengths that should be credited. However, the lack of error bars in Table 1, overlapping error bars in Table 2, and the absence of a matched sequence-only control for the structure stream mean that the central 'new state-of-the-art' claim is not yet firmly established. The contribution is promising but conditional on stronger statistical and ablative evidence.

major comments (3)
  1. [Table 1; §3.4–3.6] Table 1 reports no error bars or significance tests, and the margins over the best baseline are small (e.g., antigen binding AUC 0.931 vs 0.922, maturation ACC 0.588 vs 0.581, paratope AUC 0.898 vs 0.887). Without variance estimates or paired significance tests, the abstract's claim of 'new state-of-the-art performance' and the statement in Section 3.1 that S2ALM 'outperforms well-established and renowned baselines' are not statistically supported.
  2. [Table 2; §3.7] In Table 2 the provided error bars overlap for key comparisons: 14H Pearson is 0.650±0.013 for S2ALM versus 0.642±0.012 for A2binder, and BioMap Spearman is 0.749±0.028 versus 0.746±0.025. The text states that S2ALM 'significantly surpasses all baselines' except BioMap Spearman, but at the reported precision the differences are within one standard error. Please report the number of runs, the meaning of the parenthetical values, and appropriate significance tests or confidence intervals.
  3. [Table S2; §2.3] The ablation in Table S2 does not isolate the contribution of structural input. The 'w/o SSM&CLR' condition still includes stage-I MLM on 3Di tokens and stage-II 1D MLM, so it removes only the two auxiliary objectives, not the structural modality. Without a same-architecture, same-data sequence-only 650M-parameter control, the paper cannot support the claim that structure-aware pretraining (rather than model scale, data volume, or the auxiliary objectives) drives the observed gains. Please add such a control or clearly qualify the claim.
minor comments (6)
  1. [§2.1.2] The text states that 216,437,989 unique antibody sequences were obtained but then says 10 million sequences were randomly selected for training; please clarify why only a subset was used and whether the random selection was stratified or otherwise controlled.
  2. [§3.7] The sentence 'S2ALM significantly surpasses all baselines, except for the BioMap dataset, where it achieves the same Spearman correlation as A2binder' is internally inconsistent; please rephrase to distinguish 'best' from 'significantly better'.
  3. [Data Availability] The Data Availability statement mentions 'biotext-protein paired dataset' and 'CLIP-informed protein editing', which do not correspond to the antibody data and models in this paper and appear to be copied from another manuscript; please replace with an accurate statement for S2ALM.
  4. [Fig. 4 caption] The caption refers to 'the 3th head' of the last hidden layer; this should be 'the 3rd head'.
  5. [Table 2 caption] The parenthetical values in Table 2 are not defined in the caption or text; please state whether they are standard deviations over random seeds, data splits, or something else.
  6. [§3.3] The structural interpretability analysis relies on a single antibody (STE90-C11, PDB 7B3O); consider quantifying how often high-attention residue pairs coincide with known hydrogen bonds or contacts across a larger set of antibody structures.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: pretraining is self-supervised and downstream benchmarks are external; score reflects a non-load-bearing self-citation artifact and an unverified decontamination claim, not a derivational circle.

full rationale

Walking the derivation chain, no load-bearing step reduces to its own input. The pretraining losses (Eqs. 1-7) are self-supervised: 1D/3Di MLM, sequence-structure matching, and cross-level reconstruction use only the unlabeled pretraining corpus and Foldseek-encoded structures; no downstream label, benchmark metric, or test-set value appears in the pretraining objective. All reported downstream evaluations use external, fixed benchmark datasets (antigen binding, B cell maturation, paratope, affinity, and CoV-AbDab CDR design) and independent scorers (ProGen for perplexity, AlphaFold3 for structural metrics), so the state-of-the-art claims are empirical comparisons rather than fitted inputs renamed as predictions. The only citation-linked concern is the Data Availability section, which reuses the availability statement of the authors' own CLIP protein-editing preprint (ref [56]) and is unrelated to S2ALM; this is a non-load-bearing manuscript artifact, not a load-bearing self-citation. The possible overlap between OAS pretraining sequences and benchmark sequences is not demonstrated by the paper (only 70% clustering is described), so it is a data-leakage risk, not a proven circular reduction. Similarly, the Table S2 ablation confound (the 'w/o SSM&CLR' model still trains on 3Di tokens in stage I) is an experimental-design limitation, not a derivational circle. Hence the score is low.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests on several modeling and data choices the reader does not pay for upstream: the chosen Foldseek structure discretization, reliance on computationally predicted structures, and absence of a decontamination analysis. These are not fitted constants in a derivation, but they are load-bearing design choices that determine whether the structure-aware pretraining actually helps.

free parameters (5)
  • 3Di token vocabulary size (K) = 20
    Foldseek default setting; sets the resolution of structure encoding. The paper uses it without an ablation of K.
  • Masked token fraction in MLM and CLR = 0.15
    Standard BERT choice; no sensitivity analysis is reported.
  • Stage II loss weights = Equal weights: L_SSM + L_1D-CLR + L_3Di-CLR
    No weights are reported; equal weighting is a design choice that affects the learned representation.
  • Pretraining learning rates = 4e-4 (stage I), 1e-3 (stage II)
    Hand-chosen optimizer hyperparameters; no tuning study is presented.
  • OAS clustering threshold = 70% sequence identity
    Used with MMseqs2 to deduplicate antibody pretraining data; the threshold affects data diversity and contamination risk.
assumptions (4)
  • domain assumption Foldseek 3Di tokens are a faithful enough encoding of 3D structure for antibody function learning.
    The whole method converts structures into 20 discrete tokens (Section 2.2) and pretrains on them; if this encoding discards hydrogen-bond or side-chain geometry, the structural signal is lost.
  • domain assumption Computationally predicted structures (AlphaFoldDB, IgFold, ABodyBuilder2) are accurate enough for large-scale pretraining.
    11.5 million of 11.7 million structures are predicted (Sections 2.1.1 and 2.1.3); errors in predicted structures propagate into the 3Di tokens.
  • domain assumption Downstream benchmark labels are not contaminated by pretraining data.
    No decontamination analysis is provided (Section 2.1.2); the 70% clustering is applied only within the pretraining set, not against test sets from references [13, 16, 33, 48, 49].
  • standard math ESM-2's transformer architecture and BERT-style MLM provide a strong base for antibody representation learning.
    The model is built on ESM-2 (Section 3.1); this is a background design assumption, not derived in the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of S$^2$ALM: Sequence-Structure Pre-trained Large Language Model for Comprehensive Antibody Representation Learning." pith.science (2026). https://pith.science/paper/43ZSTLS7

@misc{pith2026241115215,
  author       = {Pith},
  title        = {Pith review of: S$^2$ALM: Sequence-Structure Pre-trained Large Language Model for Comprehensive Antibody Representation Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/43ZSTLS7}},
  note         = {Machine review of arXiv:2411.15215}
}
abstract

Antibodies safeguard our health through their precise and potent binding to specific antigens, demonstrating promising therapeutic efficacy in the treatment of numerous diseases, including COVID-19. Recent advancements in biomedical language models have shown the great potential to interpret complex biological structures and functions. However, existing antibody specific models have a notable limitation that they lack explicit consideration for antibody structural information, despite the fact that both 1D sequence and 3D structure carry unique and complementary insights into antibody behavior and functionality. This paper proposes Sequence-Structure multi-level pre-trained Antibody Language Model (S$^2$ALM), combining holistic sequential and structural information in one unified, generic antibody foundation model. We construct a hierarchical pre-training paradigm incorporated with two customized multi-level training objectives to facilitate the modeling of comprehensive antibody representations. S$^2$ALM's representation space uncovers inherent functional binding mechanisms, biological evolution properties and structural interaction patterns. Pre-trained over 75 million sequences and 11.7 million structures, S$^2$ALM can be adopted for diverse downstream tasks: accurately predicting antigen-antibody binding affinities, precisely distinguishing B cell maturation stages, identifying antibody crucial binding positions, and specifically designing novel coronavirus-binding antibodies. Remarkably, S$^2$ALM outperforms well-established and renowned baselines and sets new state-of-the-art performance across extensive antibody specific understanding and generation tasks. S$^2$ALM's ability to model comprehensive and generalized representations further positions its potential to advance real-world therapeutic antibody development, potentially addressing unmet academic, industrial, and clinical needs.

Figures

Figures reproduced from arXiv: 2411.15215 by the authors.

Figure 1
Figure 1. Overview of the proposed hierarchical pre-training paradigm containing two stages. a, In stage I, S 2ALM aims at general sequence-structure learning with protein sequences and structures. In stage II, S 2ALM learns antibody specific multi-level knowledge using antibody sequences and structures. b, Masked Language Modeling (MLM) reconstructs the masked tokens based on the contextualized information. c, Sequence-Struc… view at source ↗
Figure 2
Figure 2. Illustrations of compositional ratios of the pre-training data and the struc￾tural encoding protocol. a, The protein data contains three parts: sequences, experimentally￾determined structures, computationally-predicted structures. b, The antibody data contains four parts: sequences, experimentally-determined structures, computationally-predicted structures from ABodyBuilder2 [20] and IgFold [21]. c, Efficient encodi… view at source ↗
Figure 3
Figure 3. The t-SNE visualization results. Different colors indicate antibodies with different categories correspondingly. Untrained S 2ALM and pre-trained ESM-2 are included for comparison. The visualization analyses demonstrate that S 2ALM contains information about functional speci￾ficity, biological species and evolutionary isotypes in its comprehensive encoded representations. sample and collect 15,000 antibody sequences… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: S 2ALM exhibits superior performance on antibody understanding and gener￾ation tasks. a, Interpretability analysis of S 2ALM in capturing antibody structural interaction patterns. The heatmap reveals the self-attention values of the STE90-C11’s heavy chain, derived fro…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Antigen-specific Antibody Multi-modal Foundation Model for Functional Antibody Design

    q-bio.BM 2026-07 reject novelty 5.0 of 10

    AAMFM combines ESM3, an antigen-geometry adapter, and Cal-DPO preference optimization rewarded by AlphaFold3-style scores to design antibody CDRs and structures, reporting higher predicted binding scores than prior methods.

Reference graph

Works this paper leans on

68 extracted references · 63 canonical work pages · cited by 1 Pith paper

  1. [1]

    Neutralizing monoclonal antibodies for treatment of COVID-19

    Taylor PC, Adams AC, Hufford MM, De La Torre I, Winthrop K, and Gottlieb RL. Neutralizing monoclonal antibodies for treatment of COVID-19. Nature Reviews Immunology 2021;21:382– 93

    work page 2021

  2. [2]

    SARS-CoV-2 RBD and Its Variants Can Induce Platelet Activa- tion and Clearance: Implications for Antibody Therapy and Vaccinations against COVID-19

    Ma X, Liang J, Zhu G, et al. SARS-CoV-2 RBD and Its Variants Can Induce Platelet Activa- tion and Clearance: Implications for Antibody Therapy and Vaccinations against COVID-19. Research 2023;6:0124

    work page 2023

  3. [3]

    Anti-TNFR2 Antibody-Conjugated PLGA Nanoparticles for Targeted Delivery of Adriamycin in Mouse Colon Cancer

    Li P, Yang Y, Wang Y, et al. Anti-TNFR2 Antibody-Conjugated PLGA Nanoparticles for Targeted Delivery of Adriamycin in Mouse Colon Cancer. Research 2024;7:0444

    work page 2024

  4. [4]

    FDA approves 100th monoclonal antibody product

    Mullard A. FDA approves 100th monoclonal antibody product. Nature reviews. Drug discovery 2021;20:491–5

    work page 2021

  5. [5]

    Generative language modeling for antibody design

    Shuai R W, Ruffolo JA, and Gray JJ. Generative language modeling for antibody design. Cell Systems 2023;14:979–89

    work page 2023

  6. [6]

    De novo generation of SARS-CoV-2 antibody CDRH3 with a pre-trained generative large language model

    He H, He B, Guan L, et al. De novo generation of SARS-CoV-2 antibody CDRH3 with a pre-trained generative large language model. Nature Communications 2024;15:6867

    work page 2024

  7. [7]

    Toward Unified AI Drug Discovery with Multimodal Knowledge

    Luo Y, Liu XY, Yang K, et al. Toward Unified AI Drug Discovery with Multimodal Knowledge. Health Data Science 2024;4:0113

    work page 2024

  8. [8]

    Accurate prediction of antibody function and structure using bio-inspired antibody language model

    Jing H, Gao Z, Xu S, et al. Accurate prediction of antibody function and structure using bio-inspired antibody language model. Briefings in Bioinformatics 2024;25:bbae245

    work page 2024

Show all 68 references
  1. [9]

    Sequence modeling and design from molecular to genome scale with Evo

    Nguyen E, Poli M, Durrant MG, et al. Sequence modeling and design from molecular to genome scale with Evo. Science 2024;386:eado9336. 23

    2024

  2. [10]

    A Transformer-Based Ensemble Framework for the Prediction of Protein-Protein Interaction Sites

    Mou M, Pan Z, Zhou Z, et al. A Transformer-Based Ensemble Framework for the Prediction of Protein-Protein Interaction Sites. Research 2023;6:0240

    2023

  3. [11]

    DeepSecE: A Deep-Learning-Based Framework for Multiclass Prediction of Secreted Proteins in Gram-Negative Bacteria

    Zhang Y, Guan J, Li C, et al. DeepSecE: A Deep-Learning-Based Framework for Multiclass Prediction of Secreted Proteins in Gram-Negative Bacteria. Research 2023;6:0258

    2023

  4. [12]

    Inferring the Effects of Protein Variants on Protein-Protein Interactions with Interpretable Transformer Representations

    Liu Z, Qian W, Cai W, et al. Inferring the Effects of Protein Variants on Protein-Protein Interactions with Interpretable Transformer Representations. Research 2023;6:0219

    2023

  5. [13]

    Deciphering the language of antibodies using self-supervised learning

    Leem J, Mitchell LS, Farmery JH, Barton J, and Galson JD. Deciphering the language of antibodies using self-supervised learning. Patterns 2022;3:100513

    2022

  6. [14]

    Deciphering antibody affinity maturation with language models and weakly supervised learning

    Ruffolo JA, Gray JJ, and Sulam J. Deciphering antibody affinity maturation with language models and weakly supervised learning. arXiv preprint arXiv:2112.07782 2021

    2021 arXiv

  7. [15]

    AbLang: an antibody language model for completing antibody sequences

    Olsen TH, Moal IH, and Deane CM. AbLang: an antibody language model for completing antibody sequences. Bioinformatics Advances 2022;2:vbac046

    2022

  8. [16]

    On Pre-trained Language Models for Antibody

    Wang D, Ye F, and Zhou H. On Pre-trained Language Models for Antibody. In: International Conference on Learning Representations. 2023

    2023

  9. [17]

    UniRef clusters: a comprehensive and scalable alternative for improving sequence similarity searches

    Suzek BE, Wang Y, Huang H, McGarvey PB, Wu CH, and Consortium U. UniRef clusters: a comprehensive and scalable alternative for improving sequence similarity searches. Bioinfor- matics 2015;31:926–32

    2015

  10. [18]

    The protein data bank

    Berman HM, Westbrook J, Feng Z, et al. The protein data bank. Nucleic acids research 2000;28:235–42

    2000

  11. [19]

    AlphaFold Protein Structure Database: massively expanding the structural coverage of protein-sequence space with high-accuracy models

    Varadi M, Anyango S, Deshpande M, et al. AlphaFold Protein Structure Database: massively expanding the structural coverage of protein-sequence space with high-accuracy models. Nucleic Acids Research 2022;50:439–44

    2022

  12. [20]

    ImmuneBuilder: Deep-Learning models for predicting the structures of immune proteins

    Abanades B, Wong WK, Boyles F, Georges G, Bujotzek A, and Deane CM. ImmuneBuilder: Deep-Learning models for predicting the structures of immune proteins. Communications Bi- ology 2023;6:575

    2023

  13. [21]

    Fast, accurate antibody structure prediction from deep learning on massive set of natural antibodies

    Ruffolo JA, Chu LS, Mahajan SP, and Gray JJ. Fast, accurate antibody structure prediction from deep learning on massive set of natural antibodies. Nature Communications 2023;14:2389

    2023

  14. [22]

    Fast and accurate protein structure search with Foldseek

    Van Kempen M, Kim SS, Tumescheit C, et al. Fast and accurate protein structure search with Foldseek. Nature Biotechnology 2024;42:243–6

    2024

  15. [23]

    Observed Antibody Space: A diverse database of cleaned, annotated, and translated unpaired and paired antibody sequences

    Olsen TH, Boyles F, and Deane CM. Observed Antibody Space: A diverse database of cleaned, annotated, and translated unpaired and paired antibody sequences. Protein Science 2022;31:141– 6

    2022

  16. [24]

    Clustering huge protein sequence sets in linear time

    Steinegger M and S¨ oding J. Clustering huge protein sequence sets in linear time. Nature Com- munications 2018;9:2542

    2018

  17. [25]

    SAbDab: the structural antibody database

    Dunbar J, Krawczyk K, Leem J, et al. SAbDab: the structural antibody database. Nucleic Acids Research 2014;42:D1140–D1146. 24

    2014

  18. [26]

    Neural discrete representation learning

    Aaron vdO, Oriol V, and Koray K. Neural discrete representation learning. In: Advances in Neural Information Processing Systems. 2017

    2017

  19. [27]

    SaProt: Protein Language Modeling with Structure-aware Vocabulary

    Su J, Han C, Zhou Y, Shan J, Zhou X, and Yuan F. SaProt: Protein Language Modeling with Structure-aware Vocabulary. In: International Conference on Learning Representations. 2024

    2024

  20. [28]

    BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

    Devlin J, Chang MW, Lee K, and Toutanova K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint arXiv:1810.04805 2019

    2019 arXiv

  21. [29]

    Evolutionary-scale prediction of atomic-level protein structure with a language model

    Lin Z, Akin H, Rao R, et al. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science 2023;379:1123–30

    2023

  22. [30]

    Visualizing data using t-SNE

    Van der Maaten L and Hinton G. Visualizing data using t-SNE. Journal of machine learning research 2008;9

    2008

  23. [31]

    Reprogramming pretrained language models for antibody sequence infilling

    Melnyk I, Chenthamarakshan V, Chen PY, et al. Reprogramming pretrained language models for antibody sequence infilling. In: International Conference on Machine Learning. 2023

    2023

  24. [32]

    In situ class switching and differentiation to IgA-producing cells in the gut lamina propria

    Fagarasan S, Kinoshita K, Muramatsu M, Ikuta K, and Honjo T. In situ class switching and differentiation to IgA-producing cells in the gut lamina propria. Nature 2001;413:639–43

    2001

  25. [33]

    Optimization of therapeutic antibodies by pre- dicting antigen specificity from antibody sequence via deep learning

    Mason DM, Friedensohn S, Weber CR, et al. Optimization of therapeutic antibodies by pre- dicting antigen specificity from antibody sequence via deep learning. Nature Biomedical Engi- neering 2021;5:600–12

    2021

  26. [34]

    Biological structure and function emerge from scaling unsu- pervised learning to 250 million protein sequences

    Rives A, Meier J, Sercu T, et al. Biological structure and function emerge from scaling unsu- pervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences 2021;118:e2016239118

    2021

  27. [35]

    MSA Transformer

    Rao RM, Liu J, Verkuil R, et al. MSA Transformer. In: International Conference on Machine Learning. 2021

    2021

  28. [36]

    Different B cell subpopulations show distinct patterns in their IgH repertoire metrics

    Ghraichy M, Niederh¨ ausern V von, Kovaltsuk A, Galson JD, Deane CM, and Tr¨ uck J. Different B cell subpopulations show distinct patterns in their IgH repertoire metrics. Elife 2021;10:e73111

    2021

  29. [37]

    Antibody regulation of B cell development

    Meffre E, Casellas R, and Nussenzweig MC. Antibody regulation of B cell development. Nature immunology 2000;1:379–85

    2000

  30. [38]

    Differences in the composition of the human antibody repertoire by b cell subsets in the blood

    Mroczek ES, Ippolito GC, Rogosch T, et al. Differences in the composition of the human antibody repertoire by b cell subsets in the blood. Frontiers in immunology 2014;5:96

    2014

  31. [39]

    Analysis of the B cell receptor repertoire in six immune-mediated diseases

    Bashford-Rogers R, Bergamaschi L, McKinney E, et al. Analysis of the B cell receptor repertoire in six immune-mediated diseases. Nature 2019;574:122–6

    2019

  32. [40]

    Paratome: an online tool for systematic identification of antigen-binding regions in antibodies based on sequence or structure

    Kunik V, Ashkenazi S, and Ofran Y. Paratome: an online tool for systematic identification of antigen-binding regions in antibodies based on sequence or structure. Nucleic acids research 2012;40:W521–W524

    2012

  33. [41]

    BioPhi: A platform for antibody design, humanization, and humanness evaluation based on natural antibody repertoires and deep learning

    Prihoda D, Maamary J, Waight A, et al. BioPhi: A platform for antibody design, humanization, and humanness evaluation based on natural antibody repertoires and deep learning. MAbs 2022;14:2020203. 25

    2022

  34. [42]

    ProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning

    Elnaggar A, Heinzinger M, Dallago C, et al. ProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning. IEEE Transactions on Pattern Analysis and Machine Intelligence 2022;44:7112–27

    2022

  35. [43]

    Antibody complementarity determining region design using high-capacity machine learning

    Liu G, Zeng H, Mueller J, et al. Antibody complementarity determining region design using high-capacity machine learning. Bioinformatics 2020;36:2126–33

    2020

  36. [44]

    Learning the language of antibody hypervariability

    Singh R, Im C, Qiu Y, et al. Learning the language of antibody hypervariability. bioRxiv 2023:2023–4

    2023

  37. [45]

    Enhancing antibody language models with structural infor- mation

    Barton J, Galson JD, and Leem J. Enhancing antibody language models with structural infor- mation. bioRxiv 2024:2023–12

    2024

  38. [46]

    Antibody Representation Learning for Drug Discovery

    Li L, Gupta E, Spaeth J, Shing L, Bepler T, and Caceres RS. Antibody Representation Learning for Drug Discovery. arXiv preprint arXiv:2210.02881 2022

    2022 arXiv

  39. [47]

    Large scale paired antibody language models

    Kenlay H, Dreyer F A, Kovaltsuk A, Miketa D, Pires D, and Deane CM. Large scale paired antibody language models. arXiv preprint arXiv:2403.17889 2024

    2024 arXiv

  40. [48]

    A dataset comprised of binding interactions for 104,972 antibodies against a SARS-CoV-2 peptide

    Engelhart E, Emerson R, Shing L, et al. A dataset comprised of binding interactions for 104,972 antibodies against a SARS-CoV-2 peptide. Scientific Data 2022;9:653

    2022

  41. [49]

    xTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering the Language of Protein

    Chen B, Cheng X, Geng Ya, et al. xTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering the Language of Protein. bioRxiv 2024

    2024

  42. [50]

    Iterative Refinement Graph Neural Net- work for Antibody Sequence-Structure Co-design

    Jin W, Wohlwend J, Barzilay R, and Jaakkola TS. Iterative Refinement Graph Neural Net- work for Antibody Sequence-Structure Co-design. In: International Conference on Learning Representations. 2022

    2022

  43. [51]

    CoV-AbDab: the Coronavirus Antibody Database

    Raybould MIJ, Kovaltsuk A, Marks C, and Deane CM. CoV-AbDab: the Coronavirus Antibody Database. Bioinformatics 2021;37:734–5

    2021

  44. [52]

    Large language models generate functional protein sequences across diverse families

    Madani A, Krause B, Greene ER, et al. Large language models generate functional protein sequences across diverse families. Nature Biotechnology 2023;41:1099–106

    2023

  45. [53]

    RosettaAntibodyDesign (RAbD): A general framework for computational antibody design

    Adolf-Bryfogle J, Kalyuzhniy O, Kubitz M, et al. RosettaAntibodyDesign (RAbD): A general framework for computational antibody design. Public Library of Science Computational Biology 2018;14:e1006112

    2018

  46. [54]

    ProGen2: Exploring the bound- aries of protein language models

    Nijkamp E, Ruffolo JA, Weinstein EN, Naik N, and Madani A. ProGen2: Exploring the bound- aries of protein language models. Cell Systems 2023;14:968–978.e3

    2023

  47. [55]

    Accurate structure prediction of biomolecular interactions with AlphaFold 3

    Abramson J, Adler J, Dunger J, et al. Accurate structure prediction of biomolecular interactions with AlphaFold 3. Nature 2024:1–3

    2024

  48. [56]

    Multi-Modal CLIP-Informed Protein Editing

    Yin M, Zhou H, Zhu Y, et al. Multi-Modal CLIP-Informed Protein Editing. arXiv preprint arXiv:2407.19296 2024

    2024 arXiv

  49. [57]

    Bridge-IF: Learning Inverse Protein Folding with Markov Bridges

    Zhu Y, Wu J, Li Q, et al. Bridge-IF: Learning Inverse Protein Folding with Markov Bridges. arXiv preprint arXiv:2411.02120 2024

    2024 arXiv

  50. [58]

    Enzyme Commission Number Prediction and Benchmarking with Hierarchical Dual-core Multitask Learning Framework

    Shi Z, Deng R, Yuan Q, et al. Enzyme Commission Number Prediction and Benchmarking with Hierarchical Dual-core Multitask Learning Framework. Research 2023;6:0153. 26

    2023

  51. [59]

    GPT-4 technical report

    Achiam J, Adler S, Agarwal S, et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774 2023

    2023 arXiv

  52. [60]

    Learning transferable visual models from natural lan- guage supervision

    Radford A, Kim JW, Hallacy C, et al. Learning transferable visual models from natural lan- guage supervision. In: International Conference on Machine Learning. 2021:8748–63

    2021

  53. [61]

    PyTorch: An Imperative Style, High-Performance Deep Learning Library

    Paszke A, Gross S, Massa F, et al. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In: Advances in Neural Information Processing Systems. 2019

    2019

  54. [62]

    ZeRO: Memory optimizations toward training trillion parameter models

    Rajbhandari S, Rasley J, Ruwase O, and He Y. ZeRO: Memory optimizations toward training trillion parameter models. In: SC20: International Conference for High Performance Comput- ing, Networking, Storage and Analysis. 2020

    2020

  55. [63]

    Multi-level protein structure pre-training with prompt learning

    Wang Z, Zhang Q, Yu H, et al. Multi-level protein structure pre-training with prompt learning. In: International Conference on Learning Representations. 2023

    2023

  56. [64]

    Structure-aware protein self-supervised learning

    Chen C, Zhou J, Wang F, Liu X, and Dou D. Structure-aware protein self-supervised learning. Bioinformatics 2023;39:btad189

    2023

  57. [65]

    Enhancing protein lan- guage model with structure-based encoder and pre-training

    Zhang Z, Xu M, Lozano A, Chenthamarakshan V, Das P, and Tang J. Enhancing protein lan- guage model with structure-based encoder and pre-training. In: ICLR 2023-Machine Learning for Drug Discovery workshop. 2023

    2023

  58. [66]

    Pre-training Sequence, Structure, and Surface Features for Comprehensive Protein Representation Learning

    Lee Y, Yu H, Lee J, and Kim J. Pre-training Sequence, Structure, and Surface Features for Comprehensive Protein Representation Learning. In: International Conference on Learning Representations. 2023

    2023

  59. [67]

    ProstT5: Bilingual Language Model for Protein Sequence and Structure

    Heinzinger M, Weissenow K, Sanchez JG, Henkel A, Steinegger M, and Rost B. ProstT5: Bilingual Language Model for Protein Sequence and Structure. bioRxiv 2023

    2023

  60. [68]

    Pre-training Antibody Language Models for Antigen-Specific Computational Antibody Design

    Gao K, Wu L, Zhu J, et al. Pre-training Antibody Language Models for Antigen-Specific Computational Antibody Design. In: ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2023. 27

    2023

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.