{"id":"38e49255-5bd5-4003-ae7c-83d8930563b8","arxiv_id":"2501.07747","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Long and quantized ESM2 models process up to 2,048 amino acids and show small Fmax gains on protein function prediction, though the comparison lacks a properly matched baseline.","lead":"Researchers modified the ESM2 protein language model to accept sequences twice as long, up to 2,048 amino acids, by using windowed attention and quantization. The adapted models matched or beat the original on protein function prediction in most tests, but the gain may come from extra training, not the new architecture.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Performance claim confounded: long/quantized models received 5 epochs of Swiss-Prot continued pretraining while standard ESM2 did not, so the 'superior' Fmax in Tables 4 and 5 may reflect extra training, not architecture.","rationale":"The reader's weakest_assumption correctly identifies that the comparison between standard ESM2 and the long/quantized versions is confounded by unequal continued pretraining. This is indeed the most load-bearing concern for the paper's central performance claim. The paper delivers a real engineering contribution: it releases model weights and demonstrates a working 2,048-amino-acid input extension with local attention and int4 quantization, which is independently valuable. However, the abstract and introduction explicitly claim superior performance 'in most cases', and that claim is only supported by comparisons against an untrained-on-Swiss-Prot baseline. The continued-pretraining confound directly undermines the attribution of the gains to the architectural changes. The proposed concrete test—continued pretraining the standard models on the same data—would settle whether the architecture or simply additional training drives the results. Because the reader already flagged this exact issue and assigned a CONDITIONAL verdict, I agree with that verdict and see no need to change it. The engineering claim stands; the performance claim should be re-evaluated once a controlled baseline is added. I found no additional internal inconsistencies or misrepresentations beyond this confound, and I credit the authors for releasing the models and using a standard evaluation pipeline with AutoML and Fmax on CAFA5-derived data.","tokens_in":9373,"tokens_out":2969,"duration_ms":30637,"concrete_test":"Continue pretraining the original standard ESM2 checkpoints (T6, T12, T30, T33, and T36) for 5 epochs on the same 569,793 Swiss-Prot proteins, using the same learning rate, optimizer, and sequence length as the long/quantized variants, but keeping the original global attention and 1,024-token context. Then rerun the protein-function-prediction pipeline (embedding extraction, AutoML classifier, Fmax on test set) for BPO, CCO, and MFO, and compare against the reported long and quantized results in Tables 4 and 5. If the continued-pretrained standard models match or exceed the long/quantized Fmax values, the superiority claim collapses. As a complementary check, evaluate the long models without any continued pretraining to see whether the architecture alone changes performance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central performance claim ('in most cases, these architectures demonstrated superior performance compared to the standard ESM2 architecture') rests on an unfair comparison. Section 2 states that the long and quantized models were pre-trained for 5 epochs on 569,793 Swiss-Prot proteins, with a learning rate of 10^-5 and AdamW. The standard ESM2 baselines in Tables 4 and 5 are the original released checkpoints, which received no such continued pretraining. Therefore, any observed Fmax gains could be entirely due to additional domain-specific training on Swiss-Prot, not to the architectural changes (local attention with window 1,024, extended context to 2,050, or int4/LoRA quantization). This is especially problematic because the reported differences are small (e.g., T12 BPO Long 0.532 vs Standard 0.505; T30 BPO Long 0.527 vs Standard 0.539 in the other direction) and no significance testing is provided. The engineering claim of a 2,048-amino-acid input limit is credible and reproducible, but the headline performance claim is not supported without a controlled baseline that receives the same continued pretraining.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents extended-context versions of the ESM2 protein language models, increasing the maximum input length to 2,048 amino acids, and int4/LoRA quantized variants of these long models. The adaptation copies the original positional embeddings to 2,050 positions and replaces global self-attention with local windowed attention over a window of 1,024 tokens. The authors continue pretraining the long and quantized models on 569,793 Swiss-Prot proteins for 5 epochs, then extract embeddings and train AutoML classifiers for Gene Ontology protein function prediction on a CAFA5-derived dataset. They report Fmax scores in Tables 4 and 5 and claim that the long and quantized embeddings are superior to the standard ESM2 embeddings in most cases.","tokens_in":9735,"tokens_out":5280,"duration_ms":49977,"significance":"If the engineering contribution holds, the released models are a useful practical resource: they allow protein language model embeddings to be computed for sequences up to roughly twice the original ESM2 limit, with local attention reducing the quadratic cost of global attention, and the quantized variants lower memory use for the larger architectures. The authors explicitly release the model weights on HuggingFace, which is a concrete reproducibility advantage. The architectural modifications are clearly described and the memory comparison in Table 2 is informative. However, the headline performance claim is currently not supported because the comparison is confounded by an asymmetric continued-pretraining protocol. With a controlled baseline, the engineering and performance claims could both be assessed fairly.","major_comments":[{"comment":"The comparison supporting the 'superior performance' claim is confounded. The long and quantized models were continued-pretrained for 5 epochs on 569,793 Swiss-Prot proteins with learning rate 10^-5 and the AdamW optimizer, whereas the standard ESM2 baselines in Tables 4 and 5 are the original released checkpoints with no such continued pretraining. Reported differences such as T12 BPO Long 0.532 vs Standard 0.505 and T33 BPO Quantized 0.549 vs Standard 0.540 may therefore be caused by additional domain-specific training rather than by the windowed attention or quantization. The authors should add a controlled baseline: either continue pretraining the standard ESM2 architecture on the same Swiss-Prot data for 5 epochs, or evaluate the long and quantized versions after architectural adaptation without continued pretraining. Without this control, the abstract and Section 1 claim of superior performance is not supported.","section":"Section 2, pretraining description; Tables 4 and 5"},{"comment":"The maximum input length is stated inconsistently. Section 2 says the context representation is copied to 2,050 positions with 2,048 amino acids and 2 special tokens, while Section 3 says the long and quantized models accept 2,046 amino acids and the example uses a slice of 2,046 amino acids. Since the central contribution is 'doubling the input size limit to 2,048 amino acids,' the implemented and evaluated limits must be stated unambiguously and consistently.","section":"Section 2, paragraph 4; Section 3, paragraph 1"},{"comment":"The performance comparison reports only single Fmax values with no confidence intervals, repeated runs, or significance testing. Many differences are small, such as T6 MFO Long 0.757 vs Standard 0.754 and T30 CCO Long 0.742 vs Standard 0.739, and the conclusion 'superior in most cases' rests on margins of about 0.01 to 0.03. Repeating the classification experiments or reporting significance tests would strengthen the claim considerably.","section":"Tables 4 and 5"}],"minor_comments":[{"comment":"The sentence stating that quantization reduces memory by approximately four times for the largest ESM2 architecture is only true for T33; for T6 and T12 the quantized models require more memory than the standard and long versions. The text does acknowledge this, but stating the crossover point explicitly would improve clarity.","section":"Table 2"},{"comment":"The reference to Lin et al. contains a typo: 'Language Nodel' should be 'Language Model'.","section":"References"},{"comment":"The pipeline and dataset are inherited from Oliveira et al. 2024, but the paper does not describe how the CAFA5-derived test set was separated from the training set. A sentence on the splitting procedure would improve reproducibility, and releasing the evaluation code would allow readers to rerun the comparisons.","section":"Section 3"},{"comment":"The description of loading models in int4 format with LoRA and bfloat16 does not state whether the LoRA adapters are kept separate during inference or merged into the base model. Clarifying this would make the quantized deployment story more complete.","section":"Section 2"}],"recommendation":"major_revision","confidential_remarks":"The engineering contribution with released model weights is plausible and fits the scope of a machine learning applications venue. The main issue is the unfair comparison: the long and quantized models received 5 epochs of continued pretraining on Swiss-Prot while the standard baselines did not. This is fixable by adding a controlled baseline. I also note that the evaluation protocol, dataset, and pipeline come from the authors' own prior work; this is not disqualifying, but it means the results should be judged on the controlled comparison rather than on absolute Fmax values."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper delivers a real engineering artifact: ESM2 variants that accept 2,048 amino acids instead of 1,022, using local attention (window 1,024), copied context embeddings, and int4/LoRA quantization. The weights are released, so the core capability is testable and reproducible. That part works and is genuinely useful.\n\nWhat is new here is borrowed: the long attention is Longformer's mechanism, context copying is from Beltagy et al., and int4 quantization follows Dettmers and Zettlemoyer. The contribution is applying these to the ESM2 family and continued pretraining on Swiss-Prot. That is a legitimate extension, and the released model artifacts are the actual product.\n\nThe soft spot is the headline performance claim. The long and quantized models were pretrained for 5 epochs on 569,793 Swiss-Prot proteins (Section 2, paragraph 5). The standard ESM2 baselines in Tables 4 and 5 are the original released checkpoints, which received none of that continued pretraining. So the Fmax differences may simply reflect extra training on domain data rather than the architectural changes. The gaps are small (for example, T12 BPO Long 0.532 vs Standard 0.505) and no significance testing is provided. This is a real confound, not a stylistic objection. The fix is straightforward: continued-pretrain the standard ESM2 models on the same Swiss-Prot data for 5 epochs and compare again. Until that control exists, the performance claim should be read as not established.\n\nThat said, the engineering claim does not depend on the performance claim. A 2,048-token input limit is credible, and the released models make it directly testable. The evaluation is not circular—the test set is held out from the CAFA5-derived split—and the citations properly credit the source techniques. The self-citation to the authors' own evaluation pipeline is moderate but not a problem.\n\nVerdict: this paper is for practitioners who work with long proteins and want to avoid sliding-window preprocessing for ESM2. It deserves a serious referee, but that referee should demand the controlled baseline before accepting the central comparison. With that addition, the paper's conclusions would be much stronger.","headline":"Useful engineering extension of ESM2 to 2,048 amino acids with released weights, but the 'superior performance' headline is undercut by unequal continued pretraining.","tokens_in":10134,"tokens_out":2827,"would_cite":true,"duration_ms":26586,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that ESM2 protein language models can be retargeted to accept 2,048 amino acids by switching to local attention, and that the resulting long and quantized variants match or beat the standard models on protein function…","keywords":["ESM2","protein function prediction","long protein sequences","local attention","quantization","LoRA","CAFA5","Fmax"],"falsifier":"Run the standard ESM2 T12, T30, and T33 models through the same five-epoch continued pretraining on the same 569,793 reviewed proteins and evaluate Fmax on the same test sets; if the re-pretrained standard models match or exceed the long and quantized results, the performance advantage is not caused by local attention or int4 quantization.","tokens_in":9142,"feed_emoji":"🧬","tokens_out":5740,"duration_ms":52559,"temperature":0.7,"pith_summary":"This paper claims that the standard ESM2 protein language models, limited to 1,022 amino acids, can be adapted to accept 2,048 amino acids by replacing global self-attention with local attention and continuing pretraining on reviewed protein sequences. It also introduces an int4-quantized version, using LoRA, that cuts memory use for larger models. On a CAFA5-derived protein function prediction benchmark, the long and quantized embeddings achieve Fmax at or above the standard models in most architecture and ontology combinations. The practical point is that proteins longer than the old limit can be represented without truncation, and the released variants can be used as drop-in embedding extractors.","feed_headline":"ESM2 variants for long proteins beat standard in most function tests","feed_subtitle":"Local attention and int4 quantization double the input limit and often lift Fmax over standard ESM2.","key_machinery":"The load-bearing mechanism is local self-attention with a fixed window of 1,024 tokens, replacing global attention and reducing complexity from $O(n^2)$ to $O(nk)$, where $n$ is sequence length and $k$ is the window size. Position embeddings are copied to cover 2,050 positions, following the observation that copying context representations is more effective than random initialization. The quantized variants combine int4 weight representation with LoRA adaptation and bfloat16 computation, reducing the memory footprint of larger models by roughly four times.","core_discovery":"The central claim is that the ESM2 architecture family can be extended to twice its original input length without retraining from scratch, by copying the position embeddings to 2,050 positions and changing the attention modules from global to local form with a window of 1,024 tokens. The authors further claim that long versions and int4-quantized LoRA versions of ESM2 T6, T12, T30, and T33, after five epochs of continued pretraining on 569,793 reviewed proteins, produce embeddings whose protein function prediction Fmax is equal to or better than standard ESM2 in most settings, with especially consistent gains on proteins longer than 1,024 amino acids.","pith_inferences":["The performance gains attributed to the long and quantized architectures are confounded with the extra continued pretraining: a controlled comparison that runs standard ESM2 through the same five-epoch Swiss-Prot pretraining would separate architectural benefit from data benefit.","The int4 quantization strategy appears to pay off only at larger scales, since the smallest quantized model (T6) loads in more memory than its standard or long counterpart.","For proteins beyond 2,048 amino acids, the pipeline still falls back to sliding-window averaging, so the advantage of the long models may shrink on very long sequences even if it holds for the 1,024–2,048 range.","The Fmax differences are small, so the practical value may lie mainly in the larger input limit and lower memory footprint rather than in large accuracy improvements."],"forward_implications":["Protein function prediction pipelines can feed sequences up to 2,048 amino acids directly, avoiding sliding-window fragmentation for most proteins longer than the original limit.","The released long and quantized ESM2 variants can serve as embedding backbones for other protein tasks, such as secondary structure and contact map prediction, without truncation.","Quantized variants make larger ESM2 models practical in memory-constrained environments; the T33 quantized model loads in about a quarter of the memory of the long version.","Because the long and quantized models were pretrained for five epochs on a large reviewed protein set, downstream users inherit that continued pretraining, not only the architectural change.","The same local-attention adaptation could be applied to other transformer-based protein models, an extension the authors explicitly flag for future work."],"supporting_citations":[{"why":"supplies the local-attention mechanism that the long models adopt to reduce complexity from $O(n^2)$ to $O(nk)$","marker":"[Beltagy et al. 2020]"},{"why":"defines the ESM2 architectures whose context embeddings and attention modules are modified","marker":"[Lin et al. 2023]"},{"why":"provides the int4 precision format used for the quantized model versions","marker":"[Dettmers and Zettlemoyer 2023]"},{"why":"provides LoRA, the parameter-efficient adaptation method used during quantized pretraining","marker":"[Yu et al. 2023]"},{"why":"supplies the embedding-extraction and classification pipeline and the dataset derived from it","marker":"[Oliveira et al. 2024]"},{"why":"provides the Swiss-Prot reviewed protein corpus used for five-epoch continued pretraining","marker":"[The UniProt Consortium 2023]"},{"why":"defines the CAFA5 benchmark underlying the protein function prediction evaluation sets","marker":"[Friedberg et al. 2023]"}],"fun_headline_variants":["Long and quantized ESM2 doubles protein length limit","Local attention lets ESM2 handle 2048-amino-acid proteins","ESM2 extended to 2048 residues via local attention and int4","Quantized ESM2 matches or beats standard on long proteins","Doubling ESM2 input limit to 2048 improves long-protein prediction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that the standard ESM2 baselines needed no additional pretraining, while the long and quantized models were pretrained for five epochs on 569,793 reviewed proteins, so if that extra pretraining alone drove the Fmax gains, the superior-performance claim collapses even though the 2,048-amino-acid engineering claim would still hold.","fun_headline_variants_meta":{"raw":{"variants":["Long and quantized ESM2 doubles protein length limit","Local attention lets ESM2 handle 2048-amino-acid proteins","ESM2 extended to 2048 residues via local attention and int4","Quantized ESM2 matches or beats standard on long proteins","Doubling ESM2 input limit to 2048 improves long-protein prediction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000449,"raw_usage":{"total_tokens":2197,"prompt_tokens":808,"completion_tokens":1389,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":424,"completion_tokens_details":{"reasoning_tokens":1306}},"tokens_in":424,"tokens_out":1389,"duration_ms":11136,"temperature":1.0,"reasoning_tokens":1306,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:35:26.059953+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the standard ESM2 T12, T30, and T33 models through the same five-epoch continued pretraining on the same 569,793 reviewed proteins and evaluate Fmax on the same test sets; if the re-pretrained standard models match or exceed the long and quantized results, the performance advantage is not caused by local attention or int4 quantization.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"defines the ESM2 architectures whose context embeddings and attention modules are modified"},{"cited_title":"and Zettlemoyer, L","cited_arxiv_id":null,"evidence_quote":"provides the int4 precision format used for the quantized model versions"},{"cited_title":"B., Pedrini, H., and Dias, Z","cited_arxiv_id":null,"evidence_quote":"supplies the embedding-extraction and classification pipeline and the dataset derived from it"},{"cited_title":"UniProt: the universal protein knowledgebase in 2023","cited_arxiv_id":null,"evidence_quote":"provides the Swiss-Prot reviewed protein corpus used for five-epoch continued pretraining"},{"cited_title":"D., Piovesan, D., Joshi, P., Reade, W., and Howard, A","cited_arxiv_id":null,"evidence_quote":"defines the CAFA5 benchmark underlying the protein function prediction evaluation sets"}],"review_version":1}