Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Diffusion Sequence Models for Enhanced Protein Representation and Generation

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read The paper claims that replacing masked-language modeling with masked diffusion in an ESM2 protein model yields one architecture that both matches similarly sized models on representation benchmarks and generates biomimetic sequences and…

desk verdict Retrofitting ESM2 with masked diffusion gives a solid representation-plus-reconstruction result, but the binder-generation headline rests on an unvalidated closed-source scorer and a winner's-curse comparison. read the letter →

arxiv 2506.08293 v1 pith:CWQSZ5GB submitted 2025-06-09 q-bio.BM

classification q-bio.BM
keywords maskeddiffusionproteinlanguagemodelrepresentationlearninggenerationbinderdesignprotein-proteininteractionBenchBBsequencereconstruction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Protein language models trained by masked language modeling learn good representations but generate poorly, while autoregressive models generate but embed worse. This paper proposes DSM, a retrofit of the ESM2 architecture with a masked-diffusion training objective in the style of LLaDA, and claims that the same model can do both jobs. DSM650 matches or outperforms similarly sized pLMs on linear-probe benchmarks, reconstructs sequences from up to 90 percent masked input with alignment scores far above random, and generates proteins whose amino-acid, secondary-structure, and predicted-function distributions closely track natural ones. A fine-tuned variant, DSM(ppi), attends to a target sequence and produces binder candidates whose predicted binding affinities exceed the best known BenchBB binders for several targets. If correct, masked diffusion gives the protein-modeling field one architecture for representation and design instead of two.

What carries the argument

The load-bearing object is the masked-diffusion training objective and its reverse sampling. During training, a random fraction of tokens is replaced by the mask token and the model must predict every masked token in a single forward pass, with the loss scaled by $1/(t+\epsilon)$ so that heavily corrupted examples dominate; this is the LLaDA recipe applied to an ESM2 backbone with a modified head that caps logits before softmax and ties the final projection to the token-embedding matrix. At generation time, the model starts from all masks and progressively unmasks a chosen number of tokens per step, randomly re-masking the rest, until a full sequence emerges. The same denoising operation lets DSM both reconstruct 90 percent masked inputs and serve as a representation encoder, and the conditional variant DSM(ppi) simply concatenates a target sequence before the masked partner, making the diffusion objective target-aware.

What would settle it

Express and measure, by surface plasmon resonance or biolayer interferometry, the top DSM(ppi) designs for EGFR and PD-L1 alongside the known literature binders that the predictor ranks below them; if the measured affinities do not reproduce the predicted ordering, the claim of superior binders fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that masked diffusion is a natural learning objective for protein language models, unifying high-quality representation with generative design in a single architecture. Concretely, the authors take pretrained ESM2 checkpoints and continue training them with a LLaDA-style masked-diffusion loss that masks tokens at a random fraction $t$ and asks the model to fill all masked positions in one pass, weighted by $1/(t+\epsilon)$. The resulting DSM650 reconstructs 90 percent masked validation and test sequences with an Alignment Score around $0.277$, over four standard deviations above randomly paired natural proteins, while matching or beating similarly sized models on frozen-embedding linear probes and outperforming an autoregressive pLM almost twice its size. Generated sequences are biomimetic but distinct: amino-acid k-mer, predicted secondary-structure, and predicted-function distributions overlap natural ones with low Jensen-Shannon divergence even though chi-square tests reject identical distributions. When fine-tuned on high-confidence protein-protein interaction pairs as DSM(ppi), the model reconstructs an interacting partner better when the target is provided, and template-guided screening over 100,000 candidates per BenchBB target yields designs with predicted binding affinities above the best known literature binder for multiple targets. The authors explicitly note that all structural, functional, and affinity evaluations used in-silico proxies rather than wet-lab measurements.

Load-bearing premise

The binder-generation claims rest on the premise that the closed-source predictor's ppKd values faithfully rank real binding affinity; if that predictor is miscalibrated or exploitable, only the reconstruction and representation evidence remains.

Editorial extensions

If this is right

  • Any MLM-based pLM can be retrained with a few changes to the masking scheme, loss, and head, turning an encoder into a generator without losing representation quality.
  • Diffusion-based pLMs show a different sequence-reconstruction scaling: performance stays high at 90 percent masking where ESM2 collapses, so single-forward-pass generation becomes practical.
  • A single 650M-parameter model can serve both annotation tasks and de novo generation, potentially replacing separate representation and design models at that scale.
  • Target-conditioned fine-tuning on PPI data improves partner reconstruction and produces candidates predicted to bind more strongly than known BindCraft, AlphaProteo, and Adaptyv binders, suggesting template-guided screening as a viable design loop.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: because the same closed-source predictor both screened the designs and scored the known binders, the 'superior binder' result is only as strong as that predictor's calibration; a wet-lab affinity panel on a few top candidates would be the decisive check.
  • Beyond the paper: the diffusion objective is agnostic to supervision source, so a direct next step is to feed structure-aware or experimentally measured fitness into the same scaffold; a testable prediction is that such variants improve wet-lab hit rates over the current in-silico-only versions.
  • Beyond the paper: the random remasking strategy was chosen because logit-based selection underperformed locally, but search-based unmasking schedules are an open knob that could push generation quality and diversity beyond what the paper reports.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces the Diffusion Sequence Model (DSM), a masked-diffusion training objective applied to ESM2 checkpoints, and claims that one pLM can simultaneously provide high-quality frozen representations and generative protein design. DSM is evaluated on linear-probe representation benchmarks, on sequence reconstruction with mask rates from 5% to 90%, on unconditional generation statistics (amino acid k-mers, predicted secondary structure, predicted Annotation Vocabulary terms), and on a binder-design case study using the BenchBB targets. A PPI-conditioned variant, DSM(ppi), is fine-tuned with LoRA on StringDB pairs. The central reported results are: DSM650 representations are competitive with or better than similarly sized pLMs; DSM reconstructs heavily masked sequences with alignment scores around 0.27; generated sequences match several distributional statistics of natural proteins; and DSM(ppi) produces binders with higher Synteract2-predicted affinity than literature binders on the BenchBB targets.

Significance. If the claims hold, the paper would be significant: a single architecture that reconciles representation quality with controllable generation would be a useful step beyond MLM-only pLMs and autoregressive models. The paper has real strengths: reconstruction comparisons across models use identical masked positions via shared random seeds; the representation probes include both a random-vector negative control and a randomized-transformer control; the PPI fine-tuning includes a no-context control model to show the effect of target conditioning; and code, selected datasets, and weights are promised on GitHub. The main limitation is that the most practically salient claim, superior binder generation, currently rests on a closed-source scorer used both to screen and to evaluate, and on a comparison of the maximum of 100,000 scored candidates to a single literature value. The in-silico nature of all functional and binding evaluations is acknowledged in Section 5, but the missing null model means the binder case study does not yet support the abstract's claim of biologically coherent generation.

major comments (4)
  1. [Section 3.7, Table S4, Figure S9] The binder screen has no null baseline, so the 'superior predicted binding affinity' claim is not secured. For each target, 100,000 template-masked candidates are generated and scored, and the maximum Synteract2 ppKd is compared against a single literature binder's pKd. Even for a perfectly calibrated oracle, the maximum over many noisy predictions will exceed a single reference value by chance (winner's curse). A necessary control is a non-DSM baseline, for example randomly mutating or shuffling the template sequences and running the identical 100,000-candidate screen, or reporting the full distribution of ppKd values for template-only random perturbations. The Section 5 caveat that binding affinity is an in-silico proxy is honest, but it does not address this missing null model.
  2. [Section 4.1, Figure 3, Table S1] The distributional evidence for 'biologically plausible' generation is weakened by the use of Translator-predicted Annotation Vocabulary terms as ground truth for function, without a null generative model. Table S1 reports low F1 for several Translator aspects, e.g., BP F1 = 0.23 in the case study and EC F1 = 0.33 on the test set, so the AV-term comparison in Figure 3 inherits substantial predictor noise. Moreover, no baseline such as length-matched random sequences, a Markov model of order 1-3, or an MLM-only ESM2 is compared. Without such a null, the low JS between natural and generated k-mer distributions does not establish that DSM specifically captures function-related statistics rather than trivial amino acid composition.
  3. [Section 3.2, Section 4.3, Figure 5] The representation improvement of DSM over its ESM2 base is potentially confounded by continued pretraining on OMG prot50. Since DSM starts from ESM2 checkpoints and then trains for 100,000 steps on 207 million additional sequences, the correct control for isolating the diffusion objective is an ESM2 model trained with standard MLM on the same OMG data for the same number of steps and the same compute budget. Without this control, the gains in Figure 5 could be due to additional data exposure and fine-tuning rather than to masked diffusion. The paper currently compares only against frozen, un-continued ESM2 checkpoints.
  4. [Section 4.2, Figure 4] The reconstruction claim that ASc ~0.27 is 'over four standard deviations above the mean of randomly paired natural protein sequences' needs more support. The random-pair null distribution should be defined for length-matched sequences and the standard deviation should be reported, ideally with confidence intervals for the ASc at each mask rate. As written, the single-number comparison is hard to interpret, and the reader cannot tell whether the reported DSM/DPLM advantage over ESM2 is statistically meaningful at the individual mask rates.
minor comments (6)
  1. [Section 3.3.1, Section 4.1] The secondary structure distributions in Figure 2 are produced by a predictor whose base model is ESMC-600 (Supplemental Figure S7), not by DSM itself; this should be stated explicitly in the main text near Figure 2 so readers do not attribute these predictions to DSM.
  2. [Figure 5, Supplemental Figure S4] The normalized score in Figure 5 can amplify small raw F1 differences; the raw scores in Figure S4 show that DSM650 is not uniformly ahead of DPLM and ESM2 baselines across tasks, so the headline 'match or exceed' should be supported by reporting per-task raw scores with error bars or a significance test.
  3. [Section 3.2] The 'soft-logit cap' and the modified language modeling head are described only verbally; an equation or pseudocode would be needed for reproducibility.
  4. [Section 3.7, Table S4] The column labeled 'Template ppKd error' is not defined in the main text or table caption; please state what quantity it measures and how it is computed.
  5. [Section 2.3] For the ASc formula, the paper should clarify the handling of possibly negative Needleman-Wunsch scores and report the standard deviation of the random-sequence null distribution, not only the mean.
  6. [Figure 6 and Figure S7 captions] There are typos in the captions: 'Adapytv' should be 'Adaptyv' and 'None-class' should be 'nine-class'.

Circularity Check

1 steps flagged · score 6.0 of 10

The binder-generation claim reduces to a self-scored maximum over 100,000 Synteract2 predictions with no null baseline; representation and reconstruction evidence are independent.

  1. fitted input called prediction [Section 3.7, Section 4.4, Table S4, Supplemental A.1.1]
    "Target proteins and their newly designed binders were sent to Synteract2 through the Synthyra API to predict their binding affinity Supplemental section A.1.1(56). ... Reported is the average predicted binding affinity (ppKd), success rate (percentage higher than best known binder ppKd)."

    Synteract2 is the authors' own closed-source fitted model, and it is both the source of every ppKd and the definition of 'success' (ppKd greater than the known binder's pKd). With 100,000 random maskings of a known strong binder per target and no shuffled-template or random-sequence null control, the maximum Synteract2 score can exceed a single literature reference value purely by selection over a large noisy sample, even if DSM contributes no generative skill. The reported 'superior predicted binding affinity' is therefore a property of the max-over-100k statistic and Synteract2's calibration, not an independent measurement of DSM's generative ability; the binder claim reduces to Synteract2 ranking its own top outputs highly.

full rationale

The representation and reconstruction evaluations are self-contained and not circular: DSM is benchmarked against ESM2, DPLM, and other pLMs on public datasets with controlled linear probes, and the high-corruption reconstruction uses held-out sequences with identical masked positions. The function-distribution comparison relies on the authors' Translator model for annotations, which is a validity and independence concern but not a by-construction reduction. The binder case study, however, is structurally circular: 100,000 template mutants are generated per target and all are scored by Synteract2, the authors' closed-source PPI model; the 'success rate' is the fraction with Synteract2 ppKd above one literature pKd. Because the maximum of a large sample of noisy predictions exceeds a single reference value by selection even under a null model of random mutation, the reported superiority does not establish that DSM's generation improved on the template. The paper's Section 5 caveat that binding affinity is an in-silico proxy and wet-lab validation is future work is honest but does not supply the missing null control, so the most practically significant generation claim remains unsecured.

Assumptions & free parameters 5 free parameters · 8 assumptions · 0 invented entities

The central claims rest on fitted sampling hyperparameters (temperature and step size chosen on the validation set), on the closed-source Synteract2 and Translator predictors treated as ground truth, and on standard assumptions about alignment scores, linear probes, and protein interaction data. The binder results are the most assumption-heavy part of the paper.

free parameters (5)
  • sampling temperature T = 1.0
    Chosen to minimize Jensen-Shannon divergence between generated and natural 3-mer distributions (Section 3.5).
  • tokens unmasked per step s = 5
    Selected because JS at s=5 was nearly unchanged from s=1, giving faster generation (Section 3.5).
  • LoRA rank and alpha = r=8, alpha=32, dropout=0.01
    Hyperparameters for DSMppi fine-tuning (Section 3.7).
  • PPI dataset confidence threshold = StringDB combined score >= 900
    Data-selection cutoff for high-confidence interactions; changes the training distribution (Section 3.7).
  • CD-hit sequence identity cutoff = 90%
    Clustering threshold for PPI dataset representatives (Section 3.7).
assumptions (8)
  • domain assumption Pretrained ESM2 weights and architecture provide a sufficiently strong prior that masked diffusion fine-tuning preserves representation quality while adding generative ability.
    DSM is initialized from ESM2 checkpoints (Section 3.2); the paper's representation results are evidence that this holds, not a derivation.
  • standard math The LLaDA masked-diffusion objective with 1/t loss is Fisher-consistent and scales favorably, making it suitable for protein sequences.
    Adopted from LLaDA (Section 2.1) and used to justify the training objective.
  • domain assumption Normalized Needleman-Wunsch alignment with BLOSUM62 (ASc) is a valid measure of biological sequence similarity.
    Defined in Section 2.3 and used for reconstruction quality.
  • domain assumption Jensen-Shannon divergence and chi-square tests on k-mer and annotation-term frequency distributions capture whether generated proteins are biomimetic.
    Section 3.4 uses these statistics to conclude generated distributions are biomimetic.
  • ad hoc to paper Synteract2 predicted pKd is a valid proxy for true binding affinity.
    Section 3.7 and Supplemental A.1.1 use Synteract2 to screen and score all binder designs; the model is closed-source and from the authors' company.
  • ad hoc to paper Translator-predicted Annotation Vocabulary terms are accurate enough to compare generated and natural proteins.
    Supplemental A.1.2; Table S1 shows low precision for EC, BP, CC, and MF, so this assumption is weak.
  • domain assumption StringDB entries with combined score at least 900 represent true high-confidence protein-protein interactions.
    Section 3.7 constructs the PPI training set using this threshold and CD-hit clustering.
  • domain assumption Random transformer embeddings provide a meaningful baseline for correlation based on basic sequence homology.
    Section 3.3 uses this control to normalize probe scores.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Diffusion Sequence Models for Enhanced Protein Representation and Generation." pith.science (2026). https://pith.science/paper/CWQSZ5GB

@misc{pith2026250608293,
  author       = {Pith},
  title        = {Pith review of: Diffusion Sequence Models for Enhanced Protein Representation and Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CWQSZ5GB}},
  note         = {Machine review of arXiv:2506.08293}
}
read the original abstract

Proteins are fundamental to biology, executing diverse functions through complex physicochemical interactions, and they hold transformative potential across medicine, materials science, and environmental applications. Protein Language Models (pLMs) aim to unlock insights from the vast space of unlabeled protein sequences by learning rich, semantic representations from primary sequences via masked language modeling. However, these models typically exhibit limited generative capacity. In this work, we introduce the Diffusion Sequence Model (DSM), a novel pLM trained with masked diffusion to enable both high-quality representation learning and generative protein design. DSM builds upon the ESM2 architecture by incorporating a masked forward diffusion process inspired by the LLaDA framework. After training, DSM is capable of generating diverse, biomimetic sequences that align with expected amino acid compositions, secondary structures, and predicted functions, even with 90\% token corruption. Furthermore, DSM's learned representations match or exceed those of similarly sized pLMs on downstream tasks. We also introduce DSM(ppi), a variant fine-tuned to generate protein binders by attending to target sequences. We demonstrate DSM(ppi)'s effectiveness on the challenging Bench-tested Binder Benchmark (BenchBB), where both DSM and DSM(ppi) produce candidates with superior predicted binding affinity compared to known binders. Our results establish masked diffusion as a powerful paradigm for unifying protein representation and generation in a single framework.

Figures

Figures reproduced from arXiv: 2506.08293 by the authors.

Figure 1
Figure 1. Model architecture with DSM and DSM-PPI training scheme highlighted. Amino acids are masked uniformly and filled in completely with each forward pass, either with or without interacting protein context. For DSM pretaining, we used OMGprot50, a protein-only dataset containing over 207 million sequences after clustering the Open MetaGenomic dataset (OMG) at 50% sequence identity (36). We randomly removed 10,000 sequen… view at source ↗
Figure 2
Figure 2. Plot of natural vs. generated 1-mer, 2-mer, and 3-mer comparisons with associated χ 2 values and JS. Amino acid frequencies (top row), secondary structure [four-class] token frequencies (middle row), and secondary structure [nine-class] token frequencies (bottom row). We conducted a grid search over sam￾pling temperature and s, identifying that temperature = 1.0, no gumbel softmax, and filling in one token at a time… view at source ↗
Figure 3
Figure 3. (Left) Plot of natural vs. generation AV token 1-mers with χ 2 values and JS. (Right) Word cloud of natural and generated sequence annotations. Higher frequency terms have a bigger font, and terms are colored based on similarity between natural and generated frequencies via log2 fold change. there is some clear nuance in the distribution of unconditional DSM outputs. Whereas a few secondary structure k-mers are over… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Loss, weighted F1, and ASc for DSM, DPLM, and ESM2 models across a range of mask percentages for the validation (top row) and test sets (bottom row). Hallee et al. | arXiv | September 20, 2025 | In review | 6–20 [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Protein representation probe scores, reported and colored by their relative F1 score increase over the random vector control. Hallee et al. | arXiv | September 20, 2025 | In review | 7–20 [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Designed EGFR binders folded alongside EGFR using AlphaFold3, associated plots from web server rendered (57). (A) Unconditional generation at a high mask rate with a very high ppKd and compliance with AlphaFold3. (B) Conditional generation with the highest recorded Alp…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Survey on Diffusion Language Models

    cs.CL 2025-08 unverdicted novelty 3.0 of 10

    A comprehensive survey of diffusion language models covering taxonomy, training and inference techniques, and comparisons with autoregressive models.

Reference graph

Works this paper leans on

76 extracted references · 41 canonical work pages · cited by 1 Pith paper

  1. [1]

    Synonymous mutations in representative yeast genes are mostly strongly non-neutral.Nature, 606(7915): 725–731, June 2022

    Xukang Shen, Siliang Song, Chuan Li, and Jianzhi Zhang. Synonymous mutations in representative yeast genes are mostly strongly non-neutral.Nature, 606(7915): 725–731, June 2022. ISSN 1476-4687. doi:10.1038/s41586-022-04823-w

  2. [2]

    Laurence Loewe and William G. Hill. The population genetics of mutations: good, bad and indifferent.Philosophical Transactions of the Royal Society B: Biological Sciences, 365(1544):1153–1167, April 2010. ISSN 0962-8436. doi:10.1098/rstb.2009.0317

  3. [3]

    Gleghorn

    Logan Hallee, Niko Rafailidis, Colin Horger, David Hong, and Jason P . Gleghorn. Annotation vocabulary (might be) all you need.bioRxiv, page 2024.07.30.605924, July

  4. [4]

    Uniprot: the universal protein knowledgebase in 2023.Nucleic Acids Research, 51(D1):D523–D531, January 2023

    The UniProt Consortium. Uniprot: the universal protein knowledgebase in 2023.Nucleic Acids Research, 51(D1):D523–D531, January 2023. ISSN 0305-1048. doi: 10.1093/nar/gkac1052

  5. [5]

    Estimating the annotation error rate of curated GO database sequence annotations.BMC Bioinformatics, 8:170, 2007

    Craig E Jones, Alfred L Brown, and Ute Baumann. Estimating the annotation error rate of curated GO database sequence annotations.BMC Bioinformatics, 8:170, 2007. ISSN 1471-2105. doi:10.1186/1471-2105-8-170

  6. [6]

    Enzymap: Exploiting protein annotation for modeling and predicting ec number changes in uniprot/swiss-prot.PLOS ONE, 9(2):e89162, February 2014

    Sabrina de Azevedo Silveira, Raquel Cardoso de Melo-Minardi, Carlos Henrique da Silveira, Marcelo Matos Santoro, and Wagner Meira Jr. Enzymap: Exploiting protein annotation for modeling and predicting ec number changes in uniprot/swiss-prot.PLOS ONE, 9(2):e89162, February 2014. ISSN 1932-6203. doi:10.1371/journal.pone. 0089162

  7. [7]

    Accelerating biocatalysis discovery with machine learning: A paradigm shift in enzyme engineering, discovery, and design.ACS Catalysis, 13(21):14454–14469, November 2023

    Braun Markus, Gruber Christian C, Krassnigg Andreas, Kummer Arkadij, Lutz Stefan, Oberdorfer Gustav, Siirola Elina, and Snajdrova Radka. Accelerating biocatalysis discovery with machine learning: A paradigm shift in enzyme engineering, discovery, and design.ACS Catalysis, 13(21):14454–14469, November 2023. doi: 10.1021/acscatal.3c03417

  8. [8]

    Millar-Haskell, Allyson M

    Catherine S. Millar-Haskell, Allyson M. Dang, and Jason P . Gleghorn. Coupling synthetic biology and programmable materials to construct complex tissue ecosystems. MRS communications, 9(2):421–432, June 2019. ISSN 2159-6859. doi:10.1557/mrc.2019.69

Show all 76 references
  1. [9]

    Edwards, Elias McComb, Jason P

    Vonetta L. Edwards, Elias McComb, Jason P . Gleghorn, Larry Forney, Patrik M. Bavoil, and Jacques Ravel. Three-dimensional models of the cervicovaginal epithelia to study host-microbiome interactions and sexually transmitted infections.Pathogens and Disease, 80(1):ftac026, Aug...

  2. [10]

    DeLong, Maitham A

    John P . DeLong, Maitham A. Al-Sammak, Zeina T. Al-Ameeli, David D. Dunigan, Kyle F . Edwards, Jeffry J. Fuhrmann, Jason P . Gleghorn, Hanqun Li, Kona Haramoto, Amelia O. Harrison, Marcia F . Marston, Ryan M. Moore, Shawn W. Polson, Barbra D. Ferrell, Miranda E. Salsbery, Chri...

  3. [11]

    Nelson, N’Dea Irvin-Choy, Matthew K

    Katherine M. Nelson, N’Dea Irvin-Choy, Matthew K. Hoffman, Jason P . Gleghorn, and Emily S. Day. Diseases and conditions that impact maternal and fetal health and the potential for nanomedicine therapies.Advanced Drug Delivery Reviews, 170:425–438, March 2021. ISSN 1872-8294. ...

  4. [12]

    Gilbert and Jason P

    Rachel M. Gilbert and Jason P . Gleghorn. Connecting clinical, environmental, and genetic factors point to an essential role for vitamin a signaling in the pathogenesis of congenital diaphragmatic hernia.American Journal of Physiology. Lung Cellular and Molecular Physiology, 3...

  5. [13]

    Gleghorn

    Yuanjun Shen and Jason P . Gleghorn. Class iii phosphatidylinositol-3 kinase/vacuolar protein sorting 34 in cardiovascular health and disease.Journal of Cardiovascular Translational Research, 18(2):392–407, April 2025. ISSN 1937-5395. doi:10.1007/s12265-024-10581-z

  6. [14]

    Roberts, Logan Hallee, and Chi Keung Lam

    Richard J. Roberts, Logan Hallee, and Chi Keung Lam. The potential of hsp90 in targeting pathological pathways in cardiac diseases.J Pers Med, 11(12):1373, 2021. ISSN 2075-4426. doi:10.3390/jpm11121373

  7. [15]

    The potential of gamma secretase as a therapeutic target for cardiac diseases.J Pers Med, 11(12):1294, 2021

    Sujoita Sen, Logan Hallee, and Chi Keung Lam. The potential of gamma secretase as a therapeutic target for cardiac diseases.J Pers Med, 11(12):1294, 2021. doi: 10.3390/jpm11121294. Number: 12 Publisher: Multidisciplinary Digital Publishing Institute

  8. [16]

    Enrique Herrero Acero, Doris Ribitsch, Anita Dellacher, Sabine Zitzenbacher, Annemarie Marold, Georg Steinkellner, Karl Gruber, Helmut Schwab, and Georg M. Guebitz. Surface engineering of a cutinase from thermobifida cellulosilytica for improved polyester hydrolysis.Biotechnol...

  9. [17]

    The catalytic role of rubisco for in situ co2 recycling in escherichia coli.Frontiers in Bioengineering and Biotechnology, 8, November 2020

    Ju-Jiun Pang, Jong-Shik Shin, and Si-Yu Li. The catalytic role of rubisco for in situ co2 recycling in escherichia coli.Frontiers in Bioengineering and Biotechnology, 8, November 2020. ISSN 2296-4185. doi:10.3389/fbioe.2020.543807

  10. [18]

    Protein-based biological materials: Molecular design and artificial production.Chemical Reviews, 123(5):2049–2111, March 2023

    Ali Miserez, Jing Yu, and Pezhman Mohammadi. Protein-based biological materials: Molecular design and artificial production.Chemical Reviews, 123(5):2049–2111, March 2023. ISSN 0009-2665. doi:10.1021/acs.chemrev.2c00621

  11. [19]

    ProtTrans: Toward understanding the language of life through self-supervised learning.IEEE Trans Pattern Anal Mach Intell, 44 (10):7112–7127, 2022

    Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, Debsindhu Bhowmik, and Burkhard Rost. ProtTrans: Toward understanding the language of life through self-supervised learning...

  12. [20]

    Gleghorn

    Logan Hallee, Nikolaos Rafailidis, and Jason P . Gleghorn. cdsBERT - extending protein language models with codon awareness.bioRxiv, 2023. doi:10.1101/2023.09.15. 558027. Pages: 2023.09.15.558027 Section: New Results

  13. [21]

    CodonBERT: Large language models for mRNA design and optimization.bioRxiv, 2023

    Sizhen Li, Saeed Moayedpour, Ruijiang Li, Michael Bailey, Saleh Riahi, Milad Miladi, Jacob Miner, Dinghai Zheng, Jun Wang, Akshay Balsubramani, Khang Tran, Minnie Zacharia, Monica Wu, Xiaobo Gu, Ryan Clinton, Carla Asquith, Joseph Skalesk, Lianne Boeglin, Sudha Chivukula, Anus...

  14. [22]

    CodonBERT: a BERT-based architecture tailored for codon optimization using the cross-attention mechanism.Bioinformatics, page btae330, 2024

    Zilin Ren, Lili Jiang, Y axin Di, Dufei Zhang, Jianli Gong, Jianting Gong, Qiwei Jiang, Zhiguo Fu, Pingping Sun, Bo Zhou, and Ming Ni. CodonBERT: a BERT-based architecture tailored for codon optimization using the cross-attention mechanism.Bioinformatics, page btae330, 2024. I...

  15. [23]

    Durrant, Armin W

    Eric Nguyen, Michael Poli, Matthew G. Durrant, Armin W. Thomas, Brian Kang, Jeremy Sullivan, Madelena Y . Ng, Ashley Lewis, Aman Patel, Aaron Lou, Stefano Ermon, Stephen A. Baccus, Tina Hernandez-Boussard, Christopher Ré, Patrick D. Hsu, and Brian L. Hie. Sequence modeling and...

  16. [24]

    Esm all-atom: Multi-scale protein language model for unified molecular modeling.arXiv, June 2024

    Kangjie Zheng, Siyu Long, Tianyu Lu, Junwei Y ang, Xinyu Dai, Ming Zhang, Zaiqing Nie, Wei-Ying Ma, and Hao Zhou. Esm all-atom: Multi-scale protein language model for unified molecular modeling.arXiv, June 2024. doi:10.48550/arXiv.2403.12995. arXiv:2403.12995 [q-bio]

  17. [25]

    Durrant, Jerome Ku, Michael Poli, Greg Brockman, Daniel Chang, Gabriel A

    Garyk Brixi, Matthew G. Durrant, Jerome Ku, Michael Poli, Greg Brockman, Daniel Chang, Gabriel A. Gonzalez, Samuel H. King, David B. Li, Aditi T. Merchant, Mohsen Naghipourfar, Eric Nguyen, Chiara Ricci-Tam, David W. Romero, Gwanggyu Sun, Ali Taghibakshi, Anton Vorontsov, Bran...

  18. [26]

    Lawrence Zitnick, Jerry Ma, and Rob Fergus

    Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C. Lawrence Zitnick, Jerry Ma, and Rob Fergus. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences.Proceedings of the ...

  19. [27]

    Amini, Yisong Yue, Kevin K

    Francesca-Zhoufan Li, Ava P . Amini, Yisong Yue, Kevin K. Y ang, and Alex X. Lu. Feature reuse and scaling: Understanding transfer learning with protein language models. bioRxiv, 2024. doi:10.1101/2024.02.05.578959

  20. [28]

    Ankh: Optimized protein language model unlocks general-purpose modelling.arXiv, January 2023

    Ahmed Elnaggar, Hazem Essam, Wafaa Salah-Eldin, Walid Moustafa, Mohamed Elkerdawy, Charlotte Rochereau, and Burkhard Rost. Ankh: Optimized protein language model unlocks general-purpose modelling.arXiv, January 2023. doi:10.48550/arXiv.2301.06568. arXiv:2301.06568 [cs]

  21. [30]

    Vernon, Almer van der Sloot, Benjamin Schulz, Sarath Chandar, and Christopher James Langmead

    Quentin Fournier, Robert M. Vernon, Almer van der Sloot, Benjamin Schulz, Sarath Chandar, and Christopher James Langmead. Protein language models: Is scaling necessary?bioRxiv, page 2024.09.23.614603, September 2024. doi:10.1101/2024.09.23.614603. Halleeet al.| arXiv | Septemb...

  22. [31]

    Siddiqui, Albert Gu, Michael D

    Krithik Ramesh, Sameed M. Siddiqui, Albert Gu, Michael D. Mitzenmacher, and Pardis C. Sabeti. Lyra: An efficient and expressive subquadratic architecture for modeling biological sequences.arXiv, March 2025. doi:10.48550/arXiv.2503.16351. arXiv:2503.16351 [cs]

  23. [32]

    Large language diffusion models.arXiv, February 2025

    Shen Nie, Fengqi Zhu, Zebin Y ou, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Y ankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models.arXiv, February 2025. doi:10.48550/arXiv.2502.09992. arXiv:2502.09992 [cs]

  24. [33]

    Evolutionary-scale prediction of atomic-level protein structure with a language model.Science, 379 (6637):1123–1130, March 2023

    Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Y aniv Shmueli, Allan dos Santos Costa, Maryam Fazel-Zarandi, Tom Sercu, Salvatore Candido, and Alexander Rives. Evolutionary-scale prediction of atomic-level ...

  25. [34]

    Calia, Chance A

    Tudor-Stefan Cotet, Igor Krawczuk, Filippo Stocco, Noelia Ferruz, Anthony Gitter, Y oichi Kurumida, Lucas de Almeida Machado, Francesco Paesani, Cianna N. Calia, Chance A. Challacombe, Nikhil Haas, Ahmad Qamar, Bruno E. Correia, Martin Pacesa, Lennart Nickel, Kartic Subr, Leon...

  26. [35]

    R. A. Fisher and Edward John Russell. On the mathematical foundations of theoretical statistics.Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character, 222(594-604):309–368, 1922. doi:10.1098/rsta.1922.0009

  27. [36]

    The omg dataset: An open metagenomic corpus for mixed-modality genomic language modeling.bioRxiv, August 2024

    Andre Cornman, Jacob West-Roberts, Antonio Pedro Camargo, Simon Roux, Martin Beracochea, Milot Mirdita, Sergey Ovchinnikov, and Yunha Hwang. The omg dataset: An open metagenomic corpus for mixed-modality genomic language modeling.bioRxiv, August 2024. doi:10.1101/2024.08.14.60...

  28. [37]

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan...

  29. [38]

    Le, Mohammad Norouzi, and Samy Bengio

    Irwan Bello, Hieu Pham, Quoc V. Le, Mohammad Norouzi, and Samy Bengio. Neural combinatorial optimization with reinforcement learning.arXiv, January 2017. doi: 10.48550/arXiv.1611.09940. arXiv:1611.09940 [cs]

  30. [39]

    Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing.arXiv, March 2023

    Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing.arXiv, March 2023. doi:10.48550/arXiv.2111.09543. arXiv:2111.09543 [cs]

  31. [40]

    Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference.arXiv, December 2024

    Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin Adams, Jeremy Howard, and Iacopo Poli. Smarter, better, faster, longer: A modern bidirectional e...

  32. [41]

    Sgdr: Stochastic gradient descent with warm restarts.arXiv, May 2017

    Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts.arXiv, May 2017. doi:10.48550/arXiv.1608.03983. arXiv:1608.03983 [cs]

  33. [42]

    Decoupled weight decay regularization.arXiv, January 2019

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization.arXiv, January 2019. doi:10.48550/arXiv.1711.05101. arXiv:1711.05101 [cs]

  34. [43]

    Algorithmic capabilities of random transformers.arXiv, October 2024

    Ziqian Zhong and Jacob Andreas. Algorithmic capabilities of random transformers.arXiv, October 2024. doi:10.48550/arXiv.2410.04368. arXiv:2410.04368v1

  35. [44]

    Training compute-optimal protein language models.bioRxiv, 2024

    Xingyi Cheng, Bo Chen, Pan Li, Jing Gong, Jie Tang, and Le Song. Training compute-optimal protein language models.bioRxiv, 2024. doi:10.1101/2024.06.06.597716. Pages: 2024.06.06.597716 Section: New Results

  36. [45]

    Esm cambrian: Revealing the mysteries of proteins with unsupervised learning.Evolutionary Scale Blog, December 2024

    ESM Team. Esm cambrian: Revealing the mysteries of proteins with unsupervised learning.Evolutionary Scale Blog, December 2024. Evolutionary Scale, blog post

  37. [46]

    Gleghorn

    Logan Hallee, David Bichara, and Jason P . Gleghorn. Esm++: Efficient and hugging face compatible versions of the esm cambrian models.Hugging Face, 2024. doi: 10.57967/hf/3726

  38. [47]

    Diffusion language models are versatile protein learners.arXiv, October 2024

    Xinyou Wang, Zaixiang Zheng, Fei Y e, Dongyu Xue, Shujian Huang, and Quanquan Gu. Diffusion language models are versatile protein learners.arXiv, October 2024. doi:10.48550/arXiv.2402.18567

  39. [48]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need.arXiv, 2017. doi:10.48550/arXiv.1706.03762. Number: arXiv:1706.03762

  40. [49]

    Roformer: Enhanced transformer with rotary position embedding.arXiv, November 2023

    Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding.arXiv, November 2023. arXiv:2104.09864 [cs]

  41. [50]

    Hu, Y elong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Y elong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv, October 2021. doi:10.48550/arXiv.2106.09685. arXiv:2106.09685 [cs]

  42. [51]

    Karl Pearson. X. on the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling, July 1900

  43. [52]

    Divergence measures based on the shannon entropy.IEEE Transactions on Information Theory, 37(1):145–151, January 1991

    Jianhua Lin. Divergence measures based on the shannon entropy.IEEE Transactions on Information Theory, 37(1):145–151, January 1991

  44. [53]

    Introduction to information retrieval.Cambridge UP, 2009

    Christopher Manning, Prabhakar Raghavan, and Hinrich Schuetze. Introduction to information retrieval.Cambridge UP, 2009

  45. [54]

    Gable, Tao Fang, Nadezhda T

    Damian Szklarczyk, Rebecca Kirsch, Mikaela Koutrouli, Katerina Nastou, Farrokh Mehryary, Radja Hachilif, Annika L. Gable, Tao Fang, Nadezhda T. Doncheva, Sampo Pyysalo, Peer Bork, Lars J. Jensen, and Christian von Mering. The string database in 2023: protein-protein associatio...

  46. [55]

    Cd-hit: a fast program for clustering and comparing large sets of protein or nucleotide sequences.Bioinformatics (Oxford, England), 22 (13):1658–1659, July 2006

    Weizhong Li and Adam Godzik. Cd-hit: a fast program for clustering and comparing large sets of protein or nucleotide sequences.Bioinformatics (Oxford, England), 22 (13):1658–1659, July 2006. ISSN 1367-4803. doi:10.1093/bioinformatics/btl158

  47. [56]

    Gleghorn

    Logan Hallee and Jason P . Gleghorn. Protein-protein interaction prediction is achievable with large language models.bioRxiv, page 2023.06.07.544109, June 2023. doi: 10.1101/2023.06.07.544109

  48. [57]

    Ballard, Joshua Bambrick, Sebastian W

    Josh Abramson, Jonas Adler, Jack Dunger, Richard Evans, Tim Green, Alexander Pritzel, Olaf Ronneberger, Lindsay Willmore, Andrew J. Ballard, Joshua Bambrick, Sebastian W. Bodenstein, David A. Evans, Chia-Chun Hung, Michael O’Neill, David Reiman, Kathryn Tunyasuvunakool, Zachar...

  49. [58]

    SaProt: Protein language modeling with structure-aware vocabulary.bioRxiv, 2023

    Jin Su, Chenchen Han, Yuyang Zhou, Junjie Shan, Xibin Zhou, and Fajie Yuan. SaProt: Protein language modeling with structure-aware vocabulary.bioRxiv, 2023. doi: 10.1101/2023.10.01.560349. Pages: 2023.10.01.560349 Section: New Results

  50. [59]

    Protokens: Probabilistic vocabulary for compact and informative encodings of all-atom protein structures.bioRxiv, page 2023.11.27.568722, July 2024

    Xiaohan Lin, Zhenyu Chen, Y anheng Li, Zicheng Ma, Chuanliu Fan, Ziqiang Cao, Shihao Feng, Yi Qin Gao, and Jun Zhang. Protokens: Probabilistic vocabulary for compact and informative encodings of all-atom protein structures.bioRxiv, page 2023.11.27.568722, July 2024. doi:10.110...

  51. [60]

    Khomtchouk

    Logan Hallee and Bohdan B. Khomtchouk. Machine learning classifiers predict key genomic and evolutionary traits across the kingdoms of life.Scientific Reports, 13(1): 2088, 2023. ISSN 2045-2322. doi:10.1038/s41598-023-28965-7

  52. [61]

    Gleghorn, and Bohdan B

    Logan Hallee, Rohan Kapur, Arjun Patel, Jason P . Gleghorn, and Bohdan B. Khomtchouk. Contrastive learning and mixture of experts enables precise vector embeddings in biological databases.Scientific Reports, 15(1):14953, April 2025. ISSN 2045-2322. doi:10.1038/s41598-025-98185-8

  53. [62]

    Carter, Samuel Curtis, Claudia Emerson, Jeffrey Gray, Ian C

    Sarah R. Carter, Samuel Curtis, Claudia Emerson, Jeffrey Gray, Ian C. Haydon, Andrew Hebbeler, Claire Qureshi, Nicholas Randolph, Alexander Rives, and Lynda Stuart. Community values, guiding principles, and commitments for the responsible development of ai for protein design, ...

  54. [63]

    Michael L. Waskom. seaborn: statistical data visualization.Journal of Open Source Software, 6(60):3021, 2021. doi:10.21105/joss.03021

  55. [64]

    Deep learning using rectified linear units (relu).arXiv, February 2019

    Abien Fred Agarap. Deep learning using rectified linear units (relu).arXiv, February 2019. doi:10.48550/arXiv.1803.08375. arXiv:1803.08375 [cs]

  56. [65]

    Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R

    Geoffrey E. Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R. Salakhutdinov. Improving neural networks by preventing co-adaptation of feature detectors.arXiv, July 2012. doi:10.48550/arXiv.1207.0580. arXiv:1207.0580 [cs]

  57. [66]

    J. A. Cuff and G. J. Barton. Evaluation and improvement of multiple sequence methods for protein secondary structure prediction.Proteins, 34(4):508–519, March 1999. ISSN 0887-3585. doi:10.1002/(sici)1097-0134(19990301)34:4<508::aid-prot10>3.0.co;2-4

  58. [67]

    Sixty-five years of the long march in protein secondary structure prediction: the final stretch?Briefings in Bioinformatics, 19(3):482–494, May 2018

    Yuedong Y ang, Jianzhao Gao, Jihua Wang, Rhys Heffernan, Jack Hanson, Kuldip Paliwal, and Y aoqi Zhou. Sixty-five years of the long march in protein secondary structure prediction: the final stretch?Briefings in Bioinformatics, 19(3):482–494, May 2018. ISSN 1477-4054. doi:10.1...

  59. [68]

    Ghamary, Laura Vinué, Brahm J

    Martin Pacesa, Lennart Nickel, Joseph Schmidt, Ekaterina Pyatova, Christian Schellhaas, Lucas Kissling, Ana Alcaraz-Serna, Y ehlin Cho, Kourosh H. Ghamary, Laura Vinué, Brahm J. Y achnin, Andrew M. Wollacott, Stephen Buckley, Sandrine Georgeon, Casper A. Goverde, Georgios N. H...

  60. [69]

    Chu, Harshnira Patani, Amy E

    Vinicius Zambaldi, David La, Alexander E. Chu, Harshnira Patani, Amy E. Danson, Tristan O. C. Kwan, Thomas Frerix, Rosalia G. Schneider, David Saxton, Ashok Thillaisundaram, Zachary Wu, Isabel Moraes, Oskar Lange, Eliseo Papa, Gabriella Stanton, Victor Martin, Sukhdeep Singh, ...

  61. [70]

    Hutter, Fabian M

    Iwan Zimmermann, Pascal Egloff, Cedric A.J. Hutter, Fabian M. Arnold, Peter Stohler, Nicolas Bocquet, Melanie N. Hug, Sylwia Huber, Martin Siegrist, Lisa Hetemann, Jennifer Gera, Samira Gmür, Peter Spies, Daniel Gygax, Eric R. Geertsma, Roger J.P . Dawson, and Markus A. Seeger...

  62. [71]

    C. D. Suraweera, M. G. Hinds, and M. Crystal Kvansakul. Crystal structures of epstein–barr virus bcl-2 homolog bhrf1 bound to bid and puma bh3 motif peptides.Viruses, 14(2222), 2022. ISSN 10. doi:10.3390/v14102222

  63. [72]

    Desbien, John W

    Anthony L. Desbien, John W. Kappler, and Philippa Marrack. The epstein–barr virus bcl-2 homolog, bhrf1, blocks apoptosis by binding to a limited amount of bim. Proceedings of the National Academy of Sciences, 106:5663–5668, 2009. ISSN 14. doi:10.1073/pnas.0901036106

  64. [73]

    De novo design protein binders for mbp and gst tags.Biochemical and Biophysical Research Communications, 748:151322, February 2025

    Jinlong Zhou, Yue Xiao, Quian Tang, Yunjun Y an, Dongqi Liu, and Houjin Zhang. De novo design protein binders for mbp and gst tags.Biochemical and Biophysical Research Communications, 748:151322, February 2025. ISSN 0006-291X. doi:10.1016/j.bbrc.2025.151322. Publisher: Elsevier

  65. [74]

    Hicks, Agnidipta Ghosh, Tristin A

    Wei Y ang, Derrick R. Hicks, Agnidipta Ghosh, Tristin A. Schwartze, Brian Conventry, Inna Goreshnik, Aza Allen, Samer F . Halabiya, Chan Johng Kim, Cynthia S. Hinck, David S. Lee, Asim K. Bera, Zhe Li, Yujia Wang, Thomas Schlichthaerle, Longxing Cao, Buwei Huang, Sarah Garrett...

  66. [75]

    E. R. DeLong, D. M. DeLong, and D. L. Clarke-Pearson. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach.Biometrics, 44(3):837–845, September 1988. ISSN 0006-341X

  67. [76]

    Xu Sun and Weichao Xu. Fast implementation of delong’s algorithm for comparing the areas under correlated receiver operating characteristic curves.IEEE Signal Processing Letters, 21(11):1389–1393, November 2014. ISSN 1558-2361. doi:10.1109/LSP.2014.2337313. Halleeet al.| arXiv...

  68. [2024]

    doi:10.1101/2024.07.30.605924

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.