Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

In-Context Learning can distort the relationship between sequence likelihoods and biological fitness

T0 review · 3 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Masked language models score repeat-containing sequences as near-perfect because they copy the duplicate residue, and this distorts the likelihood–fitness link.

desk verdict Solid empirical demonstration that transformer masked protein/RNA LMs collapse pseudo-perplexity on repeated sequences via copy-based retrieval; the title overstates the fitness implication, but the core finding is real and worth engaging. read the letter →

arxiv 2504.17068 v1 pith:2FAJZM2N submitted 2025-04-23 cs.LG q-bio.BM

Pranav Kantroo , Günter P. Wagner , Benjamin B. Machta This is my paper · ORCID
classification cs.LGq-bio.BM
keywords in-contextlearningproteinlanguagemodelspseudo-perplexityrepeatedmotifsfitnesspredictionmaskedRNAhairpinssequencelikelihood
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that in-context learning, usually treated as a useful emergent ability, corrupts likelihood scores in biological sequence models. Sequences built from repeated motifs receive pseudo-perplexity scores near one—the minimum possible value—because a transformer-based masked language model resolves a masked position by looking up the equivalent position in another copy of the motif, rather than by judging biological plausibility. The authors argue this collapses the relationship between likelihood and fitness: repeat-laden sequences, functional or not, look maximally fit. They show the effect across model architectures, for imperfect repeats and small repeated needles embedded in long contexts, and for RNA hairpin reversed complements. The concern matters because repeated domains are common and physiologically important in real proteins.

What carries the argument

The load-bearing mechanism is in-context retrieval: the masked language model uses an attention-based look-up to copy the identity of a residue from a duplicated copy of a motif into the masked position. The paper measures this through pseudo-perplexity, defined from the one-at-a-time masked profile (following the One Fell Swoop approach), where a value near one indicates complete certainty. The retrieval is identified by double-masking experiments: masking the equivalent position in the second copy destroys the confidence, while masking a non-equivalent position does not, and substituting the equivalent residue changes the prediction to match the substitution. The mechanism also shows a contra-lateral preference at sequence ends and an operational memory of roughly a thousand residues in ESM2.

What would settle it

Take a library of repeat-containing variants with experimentally measured fitness (for example a deep mutational scanning dataset on a repeat-rich protein) and compare ESM2 pseudo-perplexity rankings against measured fitness. If repeat-containing variants with low measured fitness nevertheless receive pseudo-perplexity near one and are ranked above high-fitness non-repeat variants, the distortion is confirmed in fitness terms; if not, the collapse does not propagate to fitness estimates.

Watch

Extended reading notes

Core claim

The central claim is that ESM2 and other transformer-based masked language models implement an in-context look-up operation: when a sequence contains two copies of a motif, the model answers the masked position by retrieving the identity of the residue at the equivalent position in the other copy. This retrieval collapses the model's uncertainty to a pseudo-perplexity of approximately one, the lowest attainable value, so the sequence receives a near-maximal likelihood score. The paper shows the collapse occurs for natural protein domains and for random sequences, persists when the second copy is up to half-mutated, works for needles as short as ten residues across haystacks of hundreds, and can override the model's learned amino-acid preferences when the retrieved residue clashes with the local context. The same mechanism extends to RNA language models for reversed-complement hairpin motifs. Repetition also degrades the information content of residue-level embeddings, as measured by the One Fell Swoop regression task.

Load-bearing premise

The title-level claim is that likelihood–fitness relationships are distorted, yet the paper measures no fitness values; the load-bearing premise is that the pseudo-perplexity scores studied are the same likelihood scores practitioners use for fitness ranking, so a collapse in those scores directly translates into distorted fitness predictions.

Editorial extensions

If this is right

  • Any likelihood-based fitness ranking that uses ESM2-style pseudo-perplexity will systematically over-rate sequences containing repeated motifs, whether or not those sequences are functional.
  • The distortion strengthens with model scale: larger masked language models collapse more completely and extend retrieval to more complex patterns such as hairpin motifs.
  • Imperfect and non-contiguous repeats still trigger retrieval, so filtering exact duplicates will not remove the artifact.
  • Downstream tasks built on ESM2 embeddings inherit the distortion, since repeat-containing sequences yield lower-information residue embeddings.
  • Architectural choice matters: convolutional CARP and BiMamba-based LC-PLM show weaker or absent collapse, suggesting the artifact is not inherent to likelihood scoring but to transformer context mixing.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the retrieval is an attention circuit analogous to induction heads, attention-pattern probes on ESM2 could localize the circuit and perhaps disable it without retraining, offering a surgical fix for fitness scoring.
  • A testable extension: score repeat-containing variants against experimentally measured fitness landscapes; the paper predicts the likelihood rank correlation should degrade specifically at repeat positions, which a fitness dataset could confirm.
  • The same effect may afflict structure-based or multimodal models when symmetric subunits are present, since any architecture with context-dependent retrieval could copy from symmetric copies; this is a conjecture beyond the paper's sequence-domain evidence.
  • Practitioners could calibrate likelihoods by masking out equivalent duplicate positions or by penalizing low pseudo-perplexity caused by repeats; this is not proposed by the paper.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The manuscript reports that masked language models trained on biological sequences assign near-minimal pseudo-perplexity scores to sequences containing repeated motifs. The authors show this phenomenon across ESM2, Progen2, CARP, LC-PLM, and RiNALMo; characterize it mechanistically as in-context retrieval from a duplicate copy of the masked position; demonstrate persistence for imperfect repeats, one-skip sequences, and RNA hairpin-like reversed complements; and show that repetition degrades the information content of ESM2 embeddings in a regression proxy. The paper concludes that in-context learning can distort the relationship between sequence likelihoods and biological fitness.

Significance. If the core empirical observation holds, it is an important caveat for likelihood-based fitness estimation and representation learning in protein and RNA language models. The paper's strengths are its controlled experimental design: one-at-a-time masking controls, double-masking at equivalent versus non-equivalent positions, random-sequence baselines, and comparisons across architectures. These controls make the pseudo-perplexity collapse on repeated sequences robust and reproducible. However, the title-level claim that this collapse distorts the relationship between likelihood and biological fitness is not directly tested, because no fitness measurements appear in the manuscript. The significance is therefore conditional: the mechanism and score distortion are well supported, but the biological-fitness conclusion requires reframing or additional evidence.

major comments (3)
  1. [Abstract and Section 3] The central claim that in-context learning 'distorts the relationship between sequence likelihoods and biological fitness' is not tested. No fitness measurements, fitness benchmarks, or variant-effect assays appear in the manuscript. The reported result is a change in model scores (pseudo-perplexity collapse), not a change in the score-fitness relationship. To support the title-level claim, the authors would need to show, for example, that repeat-containing sequences receive inflated fitness estimates relative to their measured fitness, or that likelihood-based fitness rankings are systematically corrupted for such sequences. A concrete remedy is to reframe the paper's claim as 'repetition can collapse masked-LM pseudo-perplexity regardless of biological plausibility' or to add a fitness benchmark such as ProteinGym-style variant effect predictions on repeat-containing versus non-repeat proteins.
  2. [Section 3, first paragraph; Figure 2B] The statement that this effect 'makes otherwise nonsensical sequences seem natural' assumes that the sequences receiving collapsed scores are biologically nonsensical or low-fitness. This premise is not demonstrated: random repeats are not assayed, and natural repeat-containing proteins are often functional. The claim conflates model confidence with biological plausibility. If natural repeat proteins are functional, then a high likelihood for such sequences is not necessarily a distortion of the fitness-likelihood relationship. The manuscript should either provide evidence about the fitness of the affected sequences or soften the 'nonsensical' language.
  3. [Section 3 and Figure 2A] The propagation of the score collapse to fitness predictions also depends on how practitioners use likelihood scores. Many variant-effect workflows, including ProteinGym benchmarks, score variants relative to a wild-type sequence; in such a setting, a repeat-induced baseline offset may cancel out, and the distortion may not affect downstream fitness estimates. The manuscript does not address this usage pattern, yet it is load-bearing for the claim that the observed score distortion translates to a fitness distortion. The authors should either analyze a realistic scoring pipeline or explicitly restrict their conclusions to raw pseudo-perplexity scores.
minor comments (4)
  1. [Figure 2A caption] The caption refers to 'Progen-M', while the main text uses 'Progen2-M'; please standardize the model name.
  2. [Section 2.4 and Figure 5] The embedding-quality analysis is interesting but uses a regression loss as a proxy for information content. This is reasonable, but the connection to the paper's fitness-related thesis is not established; consider stating explicitly that this is a representation-quality observation rather than a fitness prediction result.
  3. [Methods] No code or data availability statement is included. Given that the experiments involve several large models and custom sequence manipulations, a reproducibility statement or a link to the analysis scripts would strengthen the paper.
  4. [Section 2.2, Figure 3J] The contra-lateral retrieval preference is shown only for positions near the sequence ends. The proposed explanation (the model 'sides with the longer contiguous repeating unit') is not directly tested; this interpretation should be labeled as speculative or tested with further experiments.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central uncertainty-collapse finding is established directly from model outputs with independent controls, and the only self-citation (OFS) is explicitly checked against one-at-a-time masking.

full rationale

The paper's core claim is empirical: transformer-based masked language models assign near-minimal pseudo-perplexity to repeated sequences because the model looks up the masked residue from a duplicate copy. This is demonstrated directly from raw model behavior under controlled perturbations: doubling natural and random sequences (Figure 2), masking equivalent versus non-equivalent positions (Figure 3C-E), changing the equivalent-position residue and observing the prediction flip (Figure 3F,I), needle-in-a-haystack tests, and one-skip controls (Figure 4). No parameter is fitted to a subset of data and then repackaged as a prediction; the pseudo-perplexity values are direct functions of the models' output distributions. The sole self-citation is ref. 19, the authors' One Fell Swoop (OFS) pseudo-perplexity estimator, used as a convenience scorer in Figures 1, 4, and 5. This is not load-bearing for the central claim because the authors explicitly rule out an OFS artifact: 'In order to rule out the possibility that our observations may be a quirk of the One Fell Swoop calculation (OFS) [19], as opposed to an inherent property of the model, we compute pseudo-perplexity by masking the residues of a sequence one at a time for this experiment.' The title-level claim about fitness distortion is an extrapolation without fitness measurements, and the absence of such measurements is a limitation or external-validity concern, not a circular reduction: the paper does not derive fitness values from the scores, nor does it define fitness in terms of the scores. Thus there is no self-definitional step, no fitted-input-called-prediction step, and no load-bearing self-citation chain.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

This is an empirical characterization paper; the central numbers are measurements from public model checkpoints rather than parameters fitted to data. The hand-set thresholds above are analysis choices that do not drive the main result. The assumptions that matter are conceptual: pseudo-perplexity as a fitness proxy, faithful residue-level masking, training-data enrichment of hairpins, and representativeness of single checkpoints per architecture.

free parameters (3)
  • Pseudo-perplexity inclusion threshold = 5
    Domains with OFS pseudo-perplexity below 5 are excluded from Figures 3-4 to remove sequences already containing repeats. This is a hand-set analysis filter; it does not influence the core collapse observation in Figure 2.
  • Domain length bounds = 20 to 1000 residues
    Swissprot domains outside this range are filtered during preprocessing. Standard filtering; not load-bearing.
  • One-at-a-time masking position exclusion = position 1 excluded
    The first position is excluded due to known methionine bias; a standard correction that does not affect the repeated-motif results.
assumptions (4)
  • domain assumption Pseudo-perplexity computed from masked language models is a valid proxy for the likelihood scores used in fitness prediction.
    The paper interprets low pseudo-perplexity as an inflated fitness signal, relying on prior validation (refs 9-12) that higher likelihood correlates with higher fitness; no fitness data are analyzed here.
  • domain assumption The OFS and one-at-a-time masking implementations faithfully evaluate ESM2 residue-level likelihoods without tokenization artifacts.
    ESM2 uses byte-level BPE tokenization; the authors do not detail residue-to-token mask alignment, deferring to their prior OFS method (ref 19).
  • domain assumption Hairpin (reversed-complement) motifs are enriched in the RNA language model training data relative to reversed or complemented sequences.
    Used to explain why RiNALMo retrieves from reversed complements but not reversed or complemented sequences; the training distribution is not measured in the paper.
  • domain assumption Public pretrained checkpoints of ESM2, Progen2, CARP, LC-PLM, and RiNALMo behave as representative instances of their architectures.
    The study's architecture-level conclusions (transformer vs CNN vs Mamba) are drawn from single checkpoints per architecture without training-from-scratch controls.

how reviews work

0 comments
Cite this review

Pith. "Pith review of In-Context Learning can distort the relationship between sequence likelihoods and biological fitness." pith.science (2026). https://pith.science/paper/2FAJZM2N

@misc{pith2026250417068,
  author       = {Pith},
  title        = {Pith review of: In-Context Learning can distort the relationship between sequence likelihoods and biological fitness},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2FAJZM2N}},
  note         = {Machine review of arXiv:2504.17068}
}
read the original abstract

Language models have emerged as powerful predictors of the viability of biological sequences. During training these models learn the rules of the grammar obeyed by sequences of amino acids or nucleotides. Once trained, these models can take a sequence as input and produce a likelihood score as an output; a higher likelihood implies adherence to the learned grammar and correlates with experimental fitness measurements. Here we show that in-context learning can distort the relationship between fitness and likelihood scores of sequences. This phenomenon most prominently manifests as anomalously high likelihood scores for sequences that contain repeated motifs. We use protein language models with different architectures trained on the masked language modeling objective for our experiments, and find transformer-based models to be particularly vulnerable to this effect. This behavior is mediated by a look-up operation where the model seeks the identity of the masked position by using the other copy of the repeated motif as a reference. This retrieval behavior can override the model's learned priors. This phenomenon persists for imperfectly repeated sequences, and extends to other kinds of biologically relevant features such as reversed complement motifs in RNA sequences that fold into hairpin structures.

Figures

Figures reproduced from arXiv: 2504.17068 by the authors.

Figure 2
Figure 2. Repetition can induce an uncertainty collapse in masked language models Pseudo-perplexity is a measure of the model’s uncertainty in its predictions with a value of 1 signifying complete and absolute certainty. (A) We show violin plots of the distribution of pseudo-perplexity scores for 1000 protein domains. The 1x distribution in blue denotes natural protein sequences and the 2x distribution in orange denotes doubl… view at source ↗
Figure 3
Figure 3. In-context retrieval can override learned priors (A) A depiction of a Calmodulin-binding motif (Q622K8:757-779) that is masked at position 13. The model’s prediction for the masked position indicates a preference for residues with large hydrophobic side-chains. This aligns with the identity of the actual residue in the position – W. (B) Adding the second domain dramatically reduces the uncertainty of the model. The … view at source ↗
Figure 4
Figure 4. In-context learning extends well beyond perfect repeats (A) A representation of imperfectly repeated sequences. Mutations are added in the appended copy of the protein domain to assess if the in-context learning effect persists for imperfect repeats. (B) The local ESM2 OFS pseudo-perplexity of the mutated protein domain is significantly lower when it appears alongside the natural domain as compared to when it appear… view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Repetition can deteriorate the quality of embeddings (A) A depiction of the One Fell Swoop regression task as it is used in this setting. The embedding vectors are used to regress on the one-at-a-time masked profile of the sequence, and the loss achieved by the model i…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Induction Meets Biology: Mechanisms of Repeat Detection in Protein Language Models

    cs.LG 2026-02 conditional novelty 7.0 of 10

    Protein language models complete repeats by combining induction heads that copy the aligned residue from the other repeat copy with neurons encoding amino-acid similarity; the approximate-repeat circuit contains and g...

Reference graph

Works this paper leans on

46 extracted references · 29 canonical work pages · cited by 1 Pith paper

  1. [1]

    Evolutionary-scale prediction of atomic-level protein structure with a language model,

    Z. Lin, H. Akin, R. Rao, B. Hie, Z. Zhu, W. Lu, N. Smetanin, R. Verkuil, O. Kabeli, Y. Shmueli, et al. , “Evolutionary-scale prediction of atomic-level protein structure with a language model,” Science, vol. 379, no. 6637, pp. 1123–1130, 2023

    work page 2023

  2. [2]

    Dnabert-2: Efficient foundation model and benchmark for multi-species genome,

    Z. Zhou, Y. Ji, W. Li, P. Dutta, R. Davuluri, and H. Liu, “Dnabert-2: Efficient foundation model and benchmark for multi-species genome,” 2023

    work page 2023

  3. [3]

    Rinalmo: General-purpose rna language models can generalize well on structure prediction tasks,

    R. J. Peni´ c, T. Vlaˇ si´ c, R. G. Huber, Y. Wan, and M.ˇSiki´ c, “Rinalmo: General-purpose rna language models can generalize well on structure prediction tasks,” arXiv preprint arXiv:2403.00043 , 2024

    arXiv 2024

  4. [4]

    Transfer learning enables predictions in network biology,

    C. V. Theodoris, L. Xiao, A. Chopra, M. D. Chaffin, Z. R. Al Sayed, M. C. Hill, H. Mantineo, E. M. Brydon, Z. Zeng, X. S. Liu, et al., “Transfer learning enables predictions in network biology,”Nature, vol. 618, no. 7965, pp. 616–624, 2023

    work page 2023

  5. [5]

    Multi-megabase scale genome interpretation with genetic language models

    F. Tr¨ auble, L. Stuart, A. Georgiou, P. Notin, A. Mehrjou, R. Schwessinger, M. Chevalley, K. Branson, B. Sch¨ olkopf, C. van Duijn,et al. , “Multi-megabase scale genome interpretation with genetic language mod- els,” arXiv preprint arXiv:2501.07737 , 2025

    work page Pith review arXiv 2025

  6. [6]

    Progen2: exploring the boundaries of protein language models,

    E. Nijkamp, J. A. Ruffolo, E. N. Weinstein, N. Naik, and A. Madani, “Progen2: exploring the boundaries of protein language models,” Cell systems , vol. 14, no. 11, pp. 968–978, 2023

    work page 2023

  7. [7]

    Sequence modeling and design from molecular to genome scale with evo,

    E. Nguyen, M. Poli, M. G. Durrant, B. Kang, D. Katrekar, D. B. Li, L. J. Bartie, A. W. Thomas, S. H. King, G. Brixi, et al. , “Sequence modeling and design from molecular to genome scale with evo,” Science, vol. 386, no. 6723, p. eado9336, 2024

    2024

  8. [8]

    Rna language models predict mutations that improve rna function,

    Y. Shulgina, M. I. Trinidad, C. J. Langeberg, H. Nisonoff, S. Chithrananda, P. Skopintsev, A. J. Nissley, J. Patel, R. S. Boger, H. Shi, et al. , “Rna language models predict mutations that improve rna function,” Nature Communications, vol. 15, no. 1, pp. 1–17, 2024

    work page 2024

Show all 46 references
  1. [9]

    Designing proteins with language models,

    J. A. Ruffolo and A. Madani, “Designing proteins with language models,” Nature Biotechnology, vol. 42, no. 2, pp. 200–202, 2024

    2024

  2. [10]

    Proteingym: large-scale benchmarks for protein fitness prediction and design,

    P. Notin, A. Kollasch, D. Ritter, L. Van Niekerk, S. Paul, H. Spinner, N. Rollins, A. Shaw, R. Orenbuch, R. Weitzman, et al., “Proteingym: large-scale benchmarks for protein fitness prediction and design,” Advances in Neural Information Processing Systems , vol. 36, 2024

    2024

  3. [11]

    Benchmarking dna sequence models for causal regulatory variant prediction in human genetics,

    G. Benegas, G. Eraslan, and Y. S. Song, “Benchmarking dna sequence models for causal regulatory variant prediction in human genetics,” bioRxiv, pp. 2025–02, 2025

    2025

  4. [12]

    Genome modeling and design across all domains of life with evo 2,

    G. Brixi, M. G. Durrant, J. Ku, M. Poli, G. Brockman, D. Chang, G. A. Gonzalez, S. H. King, D. B. Li, A. T. Merchant, et al. , “Genome modeling and design across all domains of life with evo 2,” bioRxiv, pp. 2025–02, 2025

    2025

  5. [13]

    Protein language models are biased by unequal sequence sampling across the tree of life,

    F. Ding and J. Steinhardt, “Protein language models are biased by unequal sequence sampling across the tree of life,” BioRxiv, pp. 2024–03, 2024. 12

    2024

  6. [14]

    Removing bias in sequence models of protein fitness,

    A. Y. Shaw, H. B. Spinner, S. Gurev, J.-E. Shin, N. Rollins, and D. S. Marks, “Removing bias in sequence models of protein fitness,” bioRxiv, pp. 2023–09, 2023

    2023

  7. [15]

    Protein language model fitness is a matter of preference,

    C. Gordon, A. X. Lu, and P. Abbeel, “Protein language model fitness is a matter of preference,” bioRxiv, pp. 2024–10, 2024

    2024

  8. [16]

    Masked language model scoring,

    J. Salazar, D. Liang, T. Q. Nguyen, and K. Kirchhoff, “Masked language model scoring,” arXiv preprint arXiv:1910.14659, 2019

    1910 arXiv

  9. [17]

    Cd-search: protein domain annotations on the fly,

    A. Marchler-Bauer and S. H. Bryant, “Cd-search: protein domain annotations on the fly,” Nucleic acids research, vol. 32, no. suppl 2, pp. W327–W331, 2004

    2004

  10. [18]

    Cdd/sparcle: the conserved domain database in 2020,

    S. Lu, J. Wang, F. Chitsaz, M. K. Derbyshire, R. C. Geer, N. R. Gonzales, M. Gwadz, D. I. Hurwitz, G. H. Marchler, J. S. Song, et al. , “Cdd/sparcle: the conserved domain database in 2020,” Nucleic acids research , vol. 48, no. D1, pp. D265–D268, 2020

    2020

  11. [19]

    Pseudo-perplexity in one fell swoop for protein fitness estimation,

    P. Kantroo, G. Wagner, and B. Machta, “Pseudo-perplexity in one fell swoop for protein fitness estimation,” bioRxiv, pp. 2024–07, 2024

    2024

  12. [20]

    The mechanistic basis of data dependence and abrupt learning in an in-context classification task,

    G. Reddy, “The mechanistic basis of data dependence and abrupt learning in an in-context classification task,” arXiv preprint arXiv:2312.03002 , 2023

    2023 arXiv

  13. [21]

    Data distributional properties drive emergent in-context learning in transformers,

    S. Chan, A. Santoro, A. Lampinen, J. Wang, A. Singh, P. Richemond, J. McClelland, and F. Hill, “Data distributional properties drive emergent in-context learning in transformers,” Advances in neural information processing systems, vol. 35, pp. 18878–18891, 2022

    2022

  14. [22]

    The transient nature of emergent in-context learning in transformers,

    A. Singh, S. Chan, T. Moskovitz, E. Grant, A. Saxe, and F. Hill, “The transient nature of emergent in-context learning in transformers,” Advances in Neural Information Processing Systems, vol. 36, pp. 27801–27819, 2023

    2023

  15. [23]

    A mathematical framework for transformer circuits,

    N. Elhage, N. Nanda, C. Olsson, T. Henighan, N. Joseph, B. Mann, A. Askell, Y. Bai, A. Chen, T. Conerly, et al., “A mathematical framework for transformer circuits,” Transformer Circuits Thread, vol. 1, no. 1, p. 12, 2021

    2021

  16. [24]

    In-context learning and induction heads,

    C. Olsson, N. Elhage, N. Nanda, N. Joseph, N. DasSarma, T. Henighan, B. Mann, A. Askell, Y. Bai, A. Chen, et al. , “In-context learning and induction heads,” arXiv preprint arXiv:2209.11895 , 2022

    2022 arXiv

  17. [25]

    Language models are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., “Language models are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020

    1901

  18. [26]

    What learning algorithm is in-context learning? investigations with linear models,

    E. Aky¨ urek, D. Schuurmans, J. Andreas, T. Ma, and D. Zhou, “What learning algorithm is in-context learning? investigations with linear models,” arXiv preprint arXiv:2211.15661 , 2022

    2022 arXiv

  19. [27]

    Transformers learn in-context by gradient descent,

    J. Von Oswald, E. Niklasson, E. Randazzo, J. Sacramento, A. Mordvintsev, A. Zhmoginov, and M. Vladymy- rov, “Transformers learn in-context by gradient descent,” in International Conference on Machine Learning , pp. 35151–35174, PMLR, 2023

    2023

  20. [28]

    Competition dynamics shape algorithmic phases of in-context learning,

    C. F. Park, E. S. Lubana, I. Pres, and H. Tanaka, “Competition dynamics shape algorithmic phases of in-context learning,” arXiv preprint arXiv:2412.01003 , 2024

    2024 arXiv

  21. [29]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems , vol. 30, 2017

    2017

  22. [30]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805 , 2018

    2018 arXiv

  23. [31]

    Language models are unsupervised multitask learners,

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. , “Language models are unsupervised multitask learners,” OpenAI blog, vol. 1, no. 8, p. 9, 2019. 13

    2019

  24. [32]

    Convolutions are competitive with transformers for protein sequence pretraining,

    K. K. Yang, N. Fusi, and A. X. Lu, “Convolutions are competitive with transformers for protein sequence pretraining,” Cell Systems , vol. 15, no. 3, pp. 286–294, 2024

    2024

  25. [33]

    Long-context protein language model,

    Y. Wang, Z. Wang, G. Sadeh, L. Zancato, A. Achille, G. Karypis, and H. Rangwala, “Long-context protein language model,” bioRxiv, pp. 2024–10, 2024

    2024

  26. [34]

    Larger language models do in-context learning differently,

    J. Wei, J. Wei, Y. Tay, D. Tran, A. Webson, Y. Lu, X. Chen, H. Liu, D. Huang, D. Zhou, et al. , “Larger language models do in-context learning differently,” arXiv preprint arXiv:2303.03846 , 2023

    2023 arXiv

  27. [35]

    Amino acid metabolism conflicts with protein diversity,

    T. Krick, N. Verstraete, L. G. Alonso, D. A. Shub, D. U. Ferreiro, M. Shub, and I. E. S´ anchez, “Amino acid metabolism conflicts with protein diversity,” Molecular biology and evolution , vol. 31, no. 11, pp. 2905–2912, 2014

    2014

  28. [36]

    Lost in the middle: How language models use long contexts,

    N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang, “Lost in the middle: How language models use long contexts,” Transactions of the Association for Computational Linguistics , vol. 12, pp. 157–173, 2024

    2024

  29. [37]

    Hairpin rna: a secondary structure of primary importance,

    P. Svoboda and A. D. Cara, “Hairpin rna: a secondary structure of primary importance,” Cellular and Molecular Life Sciences CMLS , vol. 63, pp. 901–908, 2006

    2006

  30. [38]

    Protein codes promote selective subcellular compartmentalization,

    H. R. Kilgore, I. Chinn, P. G. Mikhael, I. Mitnikov, C. Van Dongen, G. Zylberberg, L. Afeyan, S. F. Banani, S. Wilson-Hawken, T. I. Lee, et al. , “Protein codes promote selective subcellular compartmentalization,” Science, p. eadq2634, 2025

    2025

  31. [39]

    Netgo 3.0: protein language model improves large-scale functional annotations,

    S. Wang, R. You, Y. Liu, Y. Xiong, and S. Zhu, “Netgo 3.0: protein language model improves large-scale functional annotations,” Genomics, Proteomics & Bioinformatics , vol. 21, no. 2, pp. 349–358, 2023

    2023

  32. [40]

    Protein repeats: structures, functions, and evolution,

    M. A. Andrade, C. Perez-Iratxeta, and C. P. Ponting, “Protein repeats: structures, functions, and evolution,” Journal of structural biology , vol. 134, no. 2-3, pp. 117–131, 2001

    2001

  33. [41]

    Parallel structures in pre-training data yield in-context learning,

    Y. Chen, C. Zhao, Z. Yu, K. McKeown, and H. He, “Parallel structures in pre-training data yield in-context learning,” arXiv preprint arXiv:2402.12530 , 2024

    2024 arXiv

  34. [42]

    A practical review of mechanistic interpretability for transformer-based language models,

    D. Rai, Y. Zhou, S. Feng, A. Saparov, and Z. Yao, “A practical review of mechanistic interpretability for transformer-based language models,” arXiv preprint arXiv:2407.02646 , 2024

    2024

  35. [43]

    Skill-mix: A flexible and expandable family of evaluations for ai models,

    D. Yu, S. Kaur, A. Gupta, J. Brown-Cohen, A. Goyal, and S. Arora, “Skill-mix: A flexible and expandable family of evaluations for ai models,” arXiv preprint arXiv:2310.17567 , 2023

    2023 arXiv

  36. [44]

    Can models learn skill composition from examples?,

    H. Zhao, S. Kaur, D. Yu, A. Goyal, and S. Arora, “Can models learn skill composition from examples?,” Advances in Neural Information Processing Systems , vol. 37, pp. 102393–102427, 2024

    2024

  37. [45]

    Uniprot: a hub for protein information,

    U. Consortium, “Uniprot: a hub for protein information,” Nucleic acids research , vol. 43, no. D1, pp. D204– D212, 2015

    2015

  38. [46]

    Mmseqs2 enables sensitive protein sequence searching for the analysis of massive data sets,

    M. Steinegger and J. S¨ oding, “Mmseqs2 enables sensitive protein sequence searching for the analysis of massive data sets,” Nature biotechnology, vol. 35, no. 11, pp. 1026–1028, 2017. 14

    2017

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.