{"id":"994e5297-5c84-493f-9b96-f8806d0906e6","arxiv_id":"2607.22777","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"Long-range contact supervision during ESM2 adaptation improves sequence-only protein representations on eight protein-level benchmarks, most strongly remote homology (+0.0647 macro-F1).","lead":"This paper adds a training step to a protein language model, teaching it which distant parts of a protein chain end up touching when the protein folds, using predicted 3D structures. The adapted model improves several protein-analysis tasks, especially remote-homology detection, while still using only the amino-acid sequence at inference.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The all-eight-task claim is unverifiable until the paper reconciles Table 5's full split sizes with Figure 6's identity-filtering audit; if Table 2 used unfiltered splits, adaptation-set leakage alone could explain the remote-homology gain.","rationale":"The reader's weakest assumption is exactly the load-bearing concern: the adaptation set and downstream benchmark sets may overlap, and Figure 6's identity-filtering audit is never connected to the reported Table 2 results. This concern is internal to the manuscript, not a matter of outside consensus: the paper itself reports that strict filtering removes 91–98% of evaluation entries, yet Table 5 lists unfiltered split sizes and no sentence says which split was used. If the final numbers came from the unfiltered splits, the reported remote-homology improvement could be an artifact of the model having seen near-identical Swiss-Prot sequences during contact-supervised adaptation. This directly undermines the abstract's strongest claim and the paper's central interpretation. Other issues—GO-MF reported on the validation gate, ESM-S evaluated at epoch 3, small deltas within noise—are secondary but reinforce the need for caution. The paper does have genuine strengths: the method is clearly specified, the remote-homology gain is consistent across checkpoints in Figure 4, and the authors explicitly bound their claim by noting failures on local diagnostics. Those strengths justify keeping the verdict conditional rather than rejecting outright. The concrete test—rerunning the final comparison on the 30% identity-filtered splits and releasing the overlap statistics—would settle whether the central claim survives. Therefore the reader's CONDITIONAL verdict should stand unchanged until that check is performed.","tokens_in":10979,"tokens_out":3025,"duration_ms":33506,"concrete_test":"Recompute Table 2 using the strictly identity-filtered evaluation entries from Figure 6 (30% identity, query coverage ≥80%) for the seven tasks with validation partitions, using the same five-classifier-seed protocol; report remote-homology macro-F1 for ESM2 and LC-SEPLM epoch 10. If the +0.0647 delta shrinks to near zero, the central claim fails. Also compute the Jaccard overlap between the 500k adaptation training protein accessions/sequences and each downstream test split; release the filtered split sizes so Table 5 can be reconciled with Figure 6. The minimal check: state explicitly whether Table 2 used full or filtered splits.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that LC-SEPLM improves all eight protein-level tasks, with the largest gain in remote homology—depends on the downstream evaluation being uncontaminated by the 500k AlphaFold/Swiss-Prot adaptation set. The paper never establishes this. Figure 6 reports that even exact-accession filtering removes 91% of pooled evaluation entries, and 30% identity filtering leaves only 1.7%; Table 5 nonetheless lists full original split sizes (e.g., 12,517 remote-homology test proteins). Nowhere does the text state which split produced Table 2 and Figures 3/8. If Table 2 used the unfiltered splits, the +0.0647 remote-homology gain could simply reflect memorization of Swiss-Prot sequences seen during contact-supervised adaptation, not transferable structural knowledge. The independent ESM-S comparison does not resolve this: it uses epoch 3 rather than the selected epoch 10, compares against reported official scores rather than a retrained control, and shows null/negative results on fold and GO-MF splits. The paper's own limitation statement admits AlphaFold bias and ESM-S benchmark-design differences, but the unstated filtering protocol is the load-bearing gap. Without the filtered-split numbers, the strongest empirical claim is unverifiable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LC-SEPLM, a method that adapts ESM2-150M using LoRA and binary long-range contact labels (|i-j|>=12, Cα distance <= 8 Å) derived from AlphaFold Swiss-Prot structures, with an auxiliary MLM loss. During downstream inference the contact head is removed and the adapted encoder is used sequence-only. The authors report that LC-SEPLM improves all eight protein-level tasks relative to ESM2 at the downstream-selected epoch-10 checkpoint, with the largest gain in Pfam-clan remote homology (macro-F1 0.6122 → 0.6769). They also report an independent comparison with ESM-S on the official benchmark, with gains on EC splits and mixed results elsewhere. Two local diagnostics (ProteinGym and PTM-site prediction) do not improve.","tokens_in":11290,"tokens_out":2453,"duration_ms":27716,"significance":"If the empirical claims are reliable, the paper makes a useful contribution: it demonstrates a bounded, sequence-only route for injecting pairwise structural supervision into a protein language model via LoRA, with explicit attention to checkpoint selection and an honest statement of failure cases (local diagnostics, several ESM-S splits). The design is clean and the paper is candid about limitations. However, the headline claim—improvement on all eight protein-level tasks—currently rests on two unverified pillars: the statistical significance of small deltas in Table 2, and the absence of leakage between the 500k AlphaFold/Swiss-Prot adaptation set and downstream benchmarks derived from Swiss-Prot. The manuscript's own filtering audit (Figure 6) raises more questions than it answers because the final evaluation split is never identified. These issues are load-bearing and must be resolved before the central claim can be accepted.","major_comments":[{"comment":"The manuscript never states whether Table 2 and Figures 3/8 used the original splits listed in Table 5 or the identity-filtered splits quantified in Figure 6. Figure 6 reports that exact-accession filtering retains only 9.0% of pooled evaluation entries and 30% identity filtering retains 1.7%, yet Table 5 lists full original partition sizes (e.g., 12,517 remote-homology test proteins). Because the adaptation set is AlphaFold/Swiss-Prot and several downstream benchmarks are also Swiss-Prot-derived, using unfiltered splits could explain the remote-homology gain as memorization of adaptation sequences rather than transferable contact supervision. The authors must state which split produced the final numbers and provide the corresponding results on the filtered split. This is the load-bearing gap for the all-eight-task claim.","section":"Supporting configuration and filtering analyses; Figure 6; Table 5"},{"comment":"The claim 'LC-SEPLM improved all eight protein-level tasks' is not statistically supported for several tasks. Deltas of +0.0015 (EC), +0.0020 (subcellular localization), +0.0004 (membrane), and +0.0022 (HOU) are within or below the reported standard deviations (e.g., EC: 0.885186±0.005664 vs 0.883669±0.002586; HOU: 0.534068±0.016696 vs 0.531879±0.018151). Only remote homology, GO-MF, Pfam family, and strict membrane show deltas that clearly exceed the variability across seeds. The authors should either provide paired significance tests or appropriately weaken the claim to 'improved on the majority of protein-level tasks, with significant gains on...'.","section":"Table 2"},{"comment":"The ESM-S comparison uses epoch 3 rather than the selected epoch-10 checkpoint, compares against official reported scores rather than a retrained control, and yields null/negative results on fold and GO-MF splits. The text acknowledges that differences 'may reflect both representation quality and benchmark design,' but the abstract still highlights a 'maximum absolute gain of 0.1771.' The abstract's phrasing is misleading because that gain is EC-only. The authors should either restrict the abstract claim to 'EC prediction' or provide a properly controlled comparison (same epoch, same training pipeline, retrained ESM-S).","section":"Table 3; Discussion paragraph 'The comparison with ESM-S...'"}],"minor_comments":[{"comment":"The statement that a protein of length L contains O(L!) candidate residue pairs is incorrect; the number of pairs is O(L²). This typo should be fixed.","section":"Introduction, paragraph 4"},{"comment":"The broken horizontal scale for remote homology is useful but should be explicitly described in the main text so readers do not infer a false visual magnitude for the other tasks.","section":"Figure 3 legend"},{"comment":"The exclusion of ClinVar and metal-ion diagnostics is reasonable, but the fact that archived ClinVar scores are exactly zero for every model suggests a possible data-path issue. The authors should audit these pipelines and state whether any other evaluation split or metric may be affected by a similar problem.","section":"Table 7"},{"comment":"The data availability statement says processed data are available 'upon reasonable request' and code will be released 'after publication.' For a paper whose central claim depends on filtering and checkpoint-selection details, the authors should provide reproducible code and processed filtered splits at submission, at least for review.","section":"Data and code availability"}],"recommendation":"major_revision","confidential_remarks":"The central idea is sound and the paper is unusually candid about its limitations. The decision rests entirely on whether the authors can show that Table 2 and Figures 3/8 were produced from filtered splits (or, if not, that the results are robust to filtering). Without that information, the remote-homology gain cannot be distinguished from adaptation-set leakage. The statistical weakness of several small deltas is a second, independent issue that also needs a fix before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is new and the paper is honest about its limits. LC-SEPLM adapts ESM2 with LoRA and long-range contact labels from AlphaFold, using a pair-query cross-attention head that is removed at inference. That is a clean way to inject structural supervision without changing the downstream sequence-only interface. The training curves show the contact objective is learned without collapsing MLM, and the remote-homology gain (+0.0647 macro-F1) persists across checkpoints, which is the strongest evidence in the paper. Credit is also due for reporting ProteinGym and PTM failures and the mixed ESM-S results instead of hiding them.\n\nThe soft spots are real but not all equal. The biggest is the filtering protocol. Figure 6 shows that strict identity filtering leaves only 1.7–9.0% of evaluation entries, while Table 5 lists full original split sizes, and the text never says which split produced Table 2 and Figures 3/8. That is load-bearing: if the reported gains come from unfiltered splits, adaptation-set leakage could explain much of the remote-homology improvement. The stress-test note lands exactly here. Second, the \"all eight tasks\" claim overreaches the statistics. Four of the eight deltas are under 0.003 with standard deviations of similar magnitude; there are no significance tests. GO-MF is reported on the validation gate used for checkpoint selection, not a held-out test. Third, the ESM-S comparison uses epoch 3 instead of the selected epoch 10, and against official reported scores rather than a retrained control. That is odd and needs justification. Fourth, code and processed data are not released, so the central numbers cannot be checked independently.\n\nNone of this means the method is fake. The remote-homology result is large and consistent, and the architecture is described clearly enough to reimplement. But the paper is not ready as-is. A serious referee could sort this out, and the authors should be asked to state the filtering protocol explicitly, report results on filtered splits, add significance testing, explain epoch 3, and release code and data. This deserves peer review, not desk rejection, but with expectation of major revision. I would not cite it until the leakage question is resolved.","headline":"A genuinely new contact-supervision adaptation for sequence-only protein LMs, with a large remote-homology gain that looks real but is currently unverifiable due to an unresolved filtering ambiguity and overreaching all-eight-task claim.","tokens_in":11821,"tokens_out":1558,"would_cite":false,"duration_ms":20718,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Long-range contact supervision during adaptation, removed before inference, improves all eight protein-level tasks of a sequence-only protein language model, with the largest gain in remote-homology recognition.","keywords":["protein language models","residue-contact supervision","low-rank adaptation","remote-homology recognition","sequence-only inference","ESM2","AlphaFold Swiss-Prot","transfer learning"],"falsifier":"Rerun the eight controlled tasks on the strictly 30%-identity filtered evaluation sets described in Figure 6 and compare ESM2 against LC-SEPLM; if the remote-homology gain shrinks toward zero once adaptation-set homologs are removed, the contact-supervision effect is not established. A second check: recompute the ESM-S EC gain with the same filtering rule applied to the adaptation set and the benchmark test proteins.","tokens_in":10823,"feed_emoji":"🧬","tokens_out":4510,"duration_ms":46572,"temperature":0.7,"pith_summary":"LC-SEPLM asks whether a sequence-only protein language model can be improved by supervising it, during adaptation only, with which residue pairs that are far apart along the chain end up close in three-dimensional space. The paper reports that this long-range contact supervision, applied to ESM2 through low-rank adapters, improves all eight protein-level benchmark tasks relative to the original model. The clearest gain is remote-homology recognition, where macro-F1 rises from about 0.61 to 0.68. The benefit is bounded: mutation-effect prediction and post-translational-modification site prediction do not improve, and the model does not beat a structure-aware competitor on every ESM-S benchmark split. If correct, the result shows a practical middle path between sequence-only and structure-requiring protein representations.","feed_headline":"Contact-supervised training lifts all eight protein-level tasks","feed_subtitle":"Teaching a protein language model which distant residues touch improves sequence-only predictions, led by remote homology.","key_machinery":"The central object is the pair-specific query contact head. For a sampled pair (i,j), the model forms a descriptor from the two ESM2 residue states, their absolute difference, their element-wise product, and normalized sequence distance; a small MLP projects this descriptor into a query that attends over every residue in the sequence via cross-attention. The attended vector is classified as contact or non-contact. This head is trained together with an auxiliary MLM loss (weight 0.6) while LoRA adapters (rank 4) modify the last eight layers of ESM2-150M. After adaptation the head and structural labels are discarded, leaving a sequence-only encoder.","core_discovery":"The paper introduces a pair-query contact head that, for each sampled residue pair, combines the two residue representations with their difference, element-wise product, and normalized sequence distance, then lets the resulting query attend to the full sequence before predicting whether the pair forms a long-range contact. Positive contacts are defined by sequence separation at least 12 residues and C-alpha distance at most 8 Å, using AlphaFold-derived Swiss-Prot structures as labels. The contact head and an auxiliary masked-language-modeling head are used only during adaptation; after training, they are removed and downstream inference uses the adapted ESM2 encoder on amino-acid sequences a","pith_inferences":["If the reported gains survive strict sequence-identity filtering, contact supervision may be a lighter-weight alternative to structure-token models for sequence-only annotation pipelines.","The method could be tested on larger base encoders; if the remote-homology gain scales, contact supervision may complement rather than replace evolutionary-scale pretraining.","Confidence-aware weighting of predicted contacts, or use of experimental structures, could separate genuine transferable signal from AlphaFold prediction bias.","The absence of improvement on residue-level tasks suggests a natural test: add a residue-pair objective that supervises local contacts or variant effects to see whether the global/local boundary is due to supervision granularity rather than the encoder."],"forward_implications":["Remote-homology recognition can be improved substantially by contact-supervised adaptation without requiring structures at test time.","A masked-language-modeling objective can be retained during structural adaptation without collapsing sequence modeling accuracy.","Protein-level tasks that depend on global fold organization are the ones that benefit; local diagnostics such as variant effect and PTM-site prediction do not.","Checkpoint selection based on downstream validation, rather than raw contact accuracy, is necessary because the objective that peaks upstream is not the one that transfers best.","Contact-supervised adaptation can be combined with LoRA, so the added parameters remain in the encoder and no structural branch is needed during deployment."],"fun_headline_variants":["Contact-trained protein model improves all eight tasks over ESM2","Remote homology gains 6.47 points with contact-supervised ESM2","AlphaFold contacts teach protein model sequence-only wins","Sequence-based model gains structure hints, tops ESM2 everywhere","Long-range contact supervision lifts protein model on every benchmark"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The main claim assumes that the downstream comparisons are not inflated by overlap between the 500,000 Swiss-Prot adaptation proteins and the benchmark test sets; the manuscript reports identity-filtering audits showing only 1.7-9.0% of evaluation entries remain after strict filtering, yet it never states which filtered split produced Table 2.","fun_headline_variants_meta":{"raw":{"variants":["Contact-trained protein model improves all eight tasks over ESM2","Remote homology gains 6.47 points with contact-supervised ESM2","AlphaFold contacts teach protein model sequence-only wins","Sequence-based model gains structure hints, tops ESM2 everywhere","Long-range contact supervision lifts protein model on every benchmark"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001202,"raw_usage":{"total_tokens":4790,"prompt_tokens":740,"completion_tokens":4050,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":484,"completion_tokens_details":{"reasoning_tokens":3966}},"tokens_in":484,"tokens_out":4050,"duration_ms":32352,"temperature":1.0,"reasoning_tokens":3966,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T05:49:07.785684+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the eight controlled tasks on the strictly 30%-identity filtered evaluation sets described in Figure 6 and compare ESM2 against LC-SEPLM; if the remote-homology gain shrinks toward zero once adaptation-set homologs are removed, the contact-supervision effect is not established. A second check: recompute the ESM-S EC gain with the same filtering rule applied to the adaptation set and the benchmark test proteins.","supporting_citations":[],"review_version":1}