{"id":"d6466565-0fd8-4d34-be20-2eddbff92eb0","arxiv_id":"2504.17068","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Repeated motifs collapse pseudo-perplexity to near one in transformer protein language models because the model retrieves the masked residue from the duplicate copy, a behavior that can distort fitness rankings.","lead":"Protein and RNA language models give suspiciously high likelihood scores to sequences with repeated motifs, because the model copies the identity of a masked residue from the duplicate copy. This in-context lookup can override the model's learned priors, which matters for anyone using likelihood scores to rank biological sequences by fitness.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No fitness measurements appear anywhere in the manuscript, so the title-level claim that in-context retrieval 'distorts the relationship between sequence likelihoods and biological fitness' rests on the untested premise that the pseudo-perplexity collapse propagates to fitness estimates.","rationale":"The reader's weakest assumption is the same one I would flag, so I agree with the CONDITIONAL verdict. The paper's controlled experiments convincingly establish that ESM2 (and related models) can collapse pseudo-perplexity by copying an unmasked duplicate, and the double-masking and mutation-flip controls rule out the most obvious artifacts. However, the central claim in the title and abstract is about biological fitness, and the manuscript never measures fitness. The inference from 'score is near 1' to 'fitness ranking is distorted' requires an external premise that the repeated sequences are low-fitness and that raw pseudo-perplexity is the score used for fitness decisions. Both are plausible, and the first is likely true for random repeats, but the paper does not test either. This is not an internal inconsistency or a fatal flaw; it is a missing experimental link that can be supplied by a fitness benchmark. If the authors add such a comparison and it confirms the distortion, the claim stands; if it does not, the paper should be reframed. The empirical contribution would remain valuable either way.","tokens_in":12928,"tokens_out":7710,"duration_ms":76357,"concrete_test":"Run a fitness-correlation check on existing benchmark data. Take ProteinGym DMS variant sets whose wild-type proteins contain an identifiable tandem repeat or low-complexity region, and a matched set without repeats; compute ESM2 one-at-a-time pseudo-perplexity for every variant exactly as in Figure 2. Compare (a) the Spearman correlation between pseudo-perplexity and measured fitness within each set and (b) the absolute pseudo-perplexity distributions. If repeat-containing sets show no significantly lower correlation after controlling for protein length, or if a length-matched non-repeat control shows the same gap, then the title-level 'distortion of the fitness-likelihood relationship' is unsupported; the manuscript should either add such a comparison or narrow the claim to likelihood-score collapse.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The core empirical observation, that transformer masked language models assign near-minimal pseudo-perplexity to doubled or repeated sequences through copy-based look-up, is well supported by the controls in Figures 2-4 (random sequences, double masking, equivalent-position mutation flips, and architecture comparisons). The load-bearing gap is the step from score collapse to fitness distortion. The abstract and Section 3 state that the effect 'distorts the relationship between fitness and likelihood scores' and 'makes otherwise nonsensical sequences seem natural,' but the manuscript contains no fitness measurements. For the title claim to hold, two premises must hold: (i) the sequences receiving collapsed scores are actually low fitness, and (ii) practitioners use the raw pseudo-perplexity values that collapse, rather than calibrated or relative scores, when making fitness judgments. Premise (i) is plausible but not demonstrated: random repeats are not assayed, and natural repeat proteins are often functional. Premise (ii) is also untested: common variant-effect workflows, such as ProteinGym benchmarks, score variants relative to a wild-type sequence, so a repeat-induced baseline offset could cancel. The paper shows a change in model scores; it does not show that the score-fitness relationship is distorted. The finding would still be valuable if reframed as 'repetition collapses masked-LM pseudo-perplexity regardless of biological plausibility,' but the current abstract overclaims.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript reports that masked language models trained on biological sequences assign near-minimal pseudo-perplexity scores to sequences containing repeated motifs. The authors show this phenomenon across ESM2, Progen2, CARP, LC-PLM, and RiNALMo; characterize it mechanistically as in-context retrieval from a duplicate copy of the masked position; demonstrate persistence for imperfect repeats, one-skip sequences, and RNA hairpin-like reversed complements; and show that repetition degrades the information content of ESM2 embeddings in a regression proxy. The paper concludes that in-context learning can distort the relationship between sequence likelihoods and biological fitness.","tokens_in":13165,"tokens_out":3153,"duration_ms":32531,"significance":"If the core empirical observation holds, it is an important caveat for likelihood-based fitness estimation and representation learning in protein and RNA language models. The paper's strengths are its controlled experimental design: one-at-a-time masking controls, double-masking at equivalent versus non-equivalent positions, random-sequence baselines, and comparisons across architectures. These controls make the pseudo-perplexity collapse on repeated sequences robust and reproducible. However, the title-level claim that this collapse distorts the relationship between likelihood and biological fitness is not directly tested, because no fitness measurements appear in the manuscript. The significance is therefore conditional: the mechanism and score distortion are well supported, but the biological-fitness conclusion requires reframing or additional evidence.","major_comments":[{"comment":"The central claim that in-context learning 'distorts the relationship between sequence likelihoods and biological fitness' is not tested. No fitness measurements, fitness benchmarks, or variant-effect assays appear in the manuscript. The reported result is a change in model scores (pseudo-perplexity collapse), not a change in the score-fitness relationship. To support the title-level claim, the authors would need to show, for example, that repeat-containing sequences receive inflated fitness estimates relative to their measured fitness, or that likelihood-based fitness rankings are systematically corrupted for such sequences. A concrete remedy is to reframe the paper's claim as 'repetition can collapse masked-LM pseudo-perplexity regardless of biological plausibility' or to add a fitness benchmark such as ProteinGym-style variant effect predictions on repeat-containing versus non-repeat proteins.","section":"Abstract and Section 3"},{"comment":"The statement that this effect 'makes otherwise nonsensical sequences seem natural' assumes that the sequences receiving collapsed scores are biologically nonsensical or low-fitness. This premise is not demonstrated: random repeats are not assayed, and natural repeat-containing proteins are often functional. The claim conflates model confidence with biological plausibility. If natural repeat proteins are functional, then a high likelihood for such sequences is not necessarily a distortion of the fitness-likelihood relationship. The manuscript should either provide evidence about the fitness of the affected sequences or soften the 'nonsensical' language.","section":"Section 3, first paragraph; Figure 2B"},{"comment":"The propagation of the score collapse to fitness predictions also depends on how practitioners use likelihood scores. Many variant-effect workflows, including ProteinGym benchmarks, score variants relative to a wild-type sequence; in such a setting, a repeat-induced baseline offset may cancel out, and the distortion may not affect downstream fitness estimates. The manuscript does not address this usage pattern, yet it is load-bearing for the claim that the observed score distortion translates to a fitness distortion. The authors should either analyze a realistic scoring pipeline or explicitly restrict their conclusions to raw pseudo-perplexity scores.","section":"Section 3 and Figure 2A"}],"minor_comments":[{"comment":"The caption refers to 'Progen-M', while the main text uses 'Progen2-M'; please standardize the model name.","section":"Figure 2A caption"},{"comment":"The embedding-quality analysis is interesting but uses a regression loss as a proxy for information content. This is reasonable, but the connection to the paper's fitness-related thesis is not established; consider stating explicitly that this is a representation-quality observation rather than a fitness prediction result.","section":"Section 2.4 and Figure 5"},{"comment":"No code or data availability statement is included. Given that the experiments involve several large models and custom sequence manipulations, a reproducibility statement or a link to the analysis scripts would strengthen the paper.","section":"Methods"},{"comment":"The contra-lateral retrieval preference is shown only for positions near the sequence ends. The proposed explanation (the model 'sides with the longer contiguous repeating unit') is not directly tested; this interpretation should be labeled as speculative or tested with further experiments.","section":"Section 2.2, Figure 3J"}],"recommendation":"major_revision","confidential_remarks":"The core empirical finding—that repeat-induced pseudo-perplexity collapse is a robust and mechanistically interpretable failure mode of transformer masked language models—is well supported and likely publishable after reframing. The main weakness is the disconnect between the title claim about biological fitness and the absence of any fitness data. I would encourage the authors to either add a fitness-related validation or substantially temper the title and abstract claims. The self-citation to the authors' prior OFS work (ref 19) is acceptable but perhaps should be clearly marked as not yet peer-reviewed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core result here is real and well shown: ESM2 and RiNALMo, transformer masked LMs, assign pseudo-perplexity near one to doubled or repeated sequences because the model copies the residue from the matching position in the duplicate. The controls are the right ones — double masking the equivalent position restores uncertainty, mutating the equivalent position flips the prediction, random sequences collapse just like natural ones, and CARP and LC-PLM behave differently. That architecture dependence, plus the clean mechanistic experiments in Figure 3, makes this a genuine contribution. The RNA hairpin result, where reversed complements trigger retrieval but plain reverses or complements do not, is also a nice, believable data-dependence story.\n\nWhat the paper does not do is test fitness. The title says in-context learning distorts the relationship between likelihoods and biological fitness, but there are no fitness measurements anywhere. The abstract claims repeated motifs get ‘anomalously high likelihood scores’ and the discussion says this ‘makes otherwise nonsensical sequences seem natural,’ but the jump from score collapse to fitness distortion rests on an untested premise: that practitioners use raw pseudo-perplexity as a fitness proxy. Many variant-effect workflows score relative to wild-type or calibrate, which could cancel a repeat-induced baseline offset. Also, the paper never shows that the repeat-containing sequences receiving collapsed scores are actually low fitness — many natural repeat proteins are functional. So the title overclaims; the finding would be just as valuable reframed as ‘repetition collapses masked-LM pseudo-perplexity regardless of biological plausibility.’\n\nMinor points: no code or data released, the internalized-alignment interpretation is asserted rather than ablated, and the OFS tool is from the authors’ own prior work. None of these undercut the main empirical finding. The self-citation is mild and not circular — the collapse is visible in vanilla one-at-a-time masking too.\n\nThis deserves a serious referee. The fix is mostly framing and one additional experiment: take a small set of repeat proteins and their point mutants with known fitness effects, and show the score collapse actually inverts or corrupts fitness rankings in practice. Without that, the title should be toned down.\n\nReading group: yes, the double-masking controls and architecture comparison are worth discussing. I’d cite it if the fitness framing gets corrected; as is, I’d cite it for the mechanics of pseudo-perplexity collapse.","headline":"Solid empirical demonstration that transformer masked protein/RNA LMs collapse pseudo-perplexity on repeated sequences via copy-based retrieval; the title overstates the fitness implication, but the core finding is real and worth engaging.","tokens_in":13720,"tokens_out":603,"would_cite":true,"duration_ms":7761,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Masked language models score repeat-containing sequences as near-perfect because they copy the duplicate residue, and this distorts the likelihood–fitness link.","keywords":["in-context learning","protein language models","pseudo-perplexity","repeated motifs","fitness prediction","masked language models","RNA hairpins","sequence likelihood"],"falsifier":"Take a library of repeat-containing variants with experimentally measured fitness (for example a deep mutational scanning dataset on a repeat-rich protein) and compare ESM2 pseudo-perplexity rankings against measured fitness. If repeat-containing variants with low measured fitness nevertheless receive pseudo-perplexity near one and are ranked above high-fitness non-repeat variants, the distortion is confirmed in fitness terms; if not, the collapse does not propagate to fitness estimates.","tokens_in":12692,"feed_emoji":"🧬","tokens_out":4939,"duration_ms":44525,"temperature":0.7,"pith_summary":"The paper sets out to show that in-context learning, usually treated as a useful emergent ability, corrupts likelihood scores in biological sequence models. Sequences built from repeated motifs receive pseudo-perplexity scores near one—the minimum possible value—because a transformer-based masked language model resolves a masked position by looking up the equivalent position in another copy of the motif, rather than by judging biological plausibility. The authors argue this collapses the relationship between likelihood and fitness: repeat-laden sequences, functional or not, look maximally fit. They show the effect across model architectures, for imperfect repeats and small repeated needles embedded in long contexts, and for RNA hairpin reversed complements. The concern matters because repeated domains are common and physiologically important in real proteins.","feed_headline":"Repeated motifs make protein language models falsely certain","feed_subtitle":"A duplicated motif gives ESM2 near-perfect likelihood scores by copying the duplicate residue, inflating fitness rankings.","key_machinery":"The load-bearing mechanism is in-context retrieval: the masked language model uses an attention-based look-up to copy the identity of a residue from a duplicated copy of a motif into the masked position. The paper measures this through pseudo-perplexity, defined from the one-at-a-time masked profile (following the One Fell Swoop approach), where a value near one indicates complete certainty. The retrieval is identified by double-masking experiments: masking the equivalent position in the second copy destroys the confidence, while masking a non-equivalent position does not, and substituting the equivalent residue changes the prediction to match the substitution. The mechanism also shows a contra-lateral preference at sequence ends and an operational memory of roughly a thousand residues in ESM2.","core_discovery":"The central claim is that ESM2 and other transformer-based masked language models implement an in-context look-up operation: when a sequence contains two copies of a motif, the model answers the masked position by retrieving the identity of the residue at the equivalent position in the other copy. This retrieval collapses the model's uncertainty to a pseudo-perplexity of approximately one, the lowest attainable value, so the sequence receives a near-maximal likelihood score. The paper shows the collapse occurs for natural protein domains and for random sequences, persists when the second copy is up to half-mutated, works for needles as short as ten residues across haystacks of hundreds, and can override the model's learned amino-acid preferences when the retrieved residue clashes with the local context. The same mechanism extends to RNA language models for reversed-complement hairpin motifs. Repetition also degrades the information content of residue-level embeddings, as measured by the One Fell Swoop regression task.","pith_inferences":["If the retrieval is an attention circuit analogous to induction heads, attention-pattern probes on ESM2 could localize the circuit and perhaps disable it without retraining, offering a surgical fix for fitness scoring.","A testable extension: score repeat-containing variants against experimentally measured fitness landscapes; the paper predicts the likelihood rank correlation should degrade specifically at repeat positions, which a fitness dataset could confirm.","The same effect may afflict structure-based or multimodal models when symmetric subunits are present, since any architecture with context-dependent retrieval could copy from symmetric copies; this is a conjecture beyond the paper's sequence-domain evidence.","Practitioners could calibrate likelihoods by masking out equivalent duplicate positions or by penalizing low pseudo-perplexity caused by repeats; this is not proposed by the paper."],"forward_implications":["Any likelihood-based fitness ranking that uses ESM2-style pseudo-perplexity will systematically over-rate sequences containing repeated motifs, whether or not those sequences are functional.","The distortion strengthens with model scale: larger masked language models collapse more completely and extend retrieval to more complex patterns such as hairpin motifs.","Imperfect and non-contiguous repeats still trigger retrieval, so filtering exact duplicates will not remove the artifact.","Downstream tasks built on ESM2 embeddings inherit the distortion, since repeat-containing sequences yield lower-information residue embeddings.","Architectural choice matters: convolutional CARP and BiMamba-based LC-PLM show weaker or absent collapse, suggesting the artifact is not inherent to likelihood scoring but to transformer context mixing."],"supporting_citations":[{"why":"Supplies the primary transformer-based masked protein language model, ESM2, on which the uncertainty-collapse and retrieval experiments are run.","marker":"[1]"},{"why":"Defines masked language model scoring and pseudo-perplexity, the uncertainty measure used throughout the paper.","marker":"[16]"},{"why":"Provides the One Fell Swoop pseudo-perplexity method and its regression task used to score domains and measure embedding information content.","marker":"[19]"},{"why":"Connects the retrieval behavior to induction heads, the transformer circuit proposed for in-context learning.","marker":"[24]"},{"why":"Supplies CARP, the convolutional masked language model whose unit-size-dependent collapse defines the architectural contrast.","marker":"[32]"},{"why":"Supplies LC-PLM, the BiMamba-based long-context model that shows progressive decline rather than collapse.","marker":"[33]"},{"why":"Supplies RiNALMo, the RNA language model used to test reversed-complement hairpin retrieval.","marker":"[3]"},{"why":"Provides the flipped-label example showing in-context exemplars can override learned priors in language models.","marker":"[34]"},{"why":"Supplies ProGen2, the autoregressive transformer used to show perplexity drop under sequence doubling.","marker":"[6]"}],"fun_headline_variants":["Repeated motifs trick protein models into false fitness scores","Protein models inflate fitness predictions for repeated sequences","Duplicated motifs cause protein models to overestimate fitness","Repeated patterns break likelihood-fitness link in protein models","Protein models falsely score repeated sequences as high fitness"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The title-level claim is that likelihood–fitness relationships are distorted, yet the paper measures no fitness values; the load-bearing premise is that the pseudo-perplexity scores studied are the same likelihood scores practitioners use for fitness ranking, so a collapse in those scores directly translates into distorted fitness predictions.","fun_headline_variants_meta":{"raw":{"variants":["Repeated motifs trick protein models into false fitness scores","Protein models inflate fitness predictions for repeated sequences","Duplicated motifs cause protein models to overestimate fitness","Repeated patterns break likelihood-fitness link in protein models","Protein models falsely score repeated sequences as high fitness"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001059,"raw_usage":{"total_tokens":4431,"prompt_tokens":922,"completion_tokens":3509,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":3434}},"tokens_in":538,"tokens_out":3509,"duration_ms":23439,"temperature":1.0,"reasoning_tokens":3434,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:50:02.258507+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a library of repeat-containing variants with experimentally measured fitness (for example a deep mutational scanning dataset on a repeat-rich protein) and compare ESM2 pseudo-perplexity rankings against measured fitness. If repeat-containing variants with low measured fitness nevertheless receive pseudo-perplexity near one and are ranked above high-fitness non-repeat variants, the distortion is confirmed in fitness terms; if not, the collapse does not propagate to fitness estimates.","supporting_citations":[{"cited_title":"Evolutionary-scale prediction of atomic-level protein structure with a language model,","cited_arxiv_id":null,"evidence_quote":"Supplies the primary transformer-based masked protein language model, ESM2, on which the uncertainty-collapse and retrieval experiments are run."},{"cited_title":"Long-context protein language model,","cited_arxiv_id":null,"evidence_quote":"Supplies LC-PLM, the BiMamba-based long-context model that shows progressive decline rather than collapse."},{"cited_title":"Progen2: exploring the boundaries of protein language models,","cited_arxiv_id":null,"evidence_quote":"Supplies ProGen2, the autoregressive transformer used to show perplexity drop under sequence doubling."}],"review_version":1}