{"id":"a39d05ab-f0d9-4799-9500-c27d31ac67e7","arxiv_id":"2505.20301","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A sequence-only cross-attention model, ProtAttBA, yields competitive and robust predictions of antibody-antigen binding affinity changes across three benchmarks.","lead":"ProtAttBA is a deep learning model that predicts how mutations alter antibody-antigen binding affinity from protein sequences alone, without any structural input. The paper reports competitive accuracy against structure-based methods and greater robustness when only predicted structures are available.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unspecified mutation-chain handling makes antigen-side mutations invisible to the model; S1131 may contain such cases.","rationale":"The architecture in Section 2.2.1 and Fig. 1 admits only one mutated sequence, the antibody, while the antigen is always wild type. For the central claim 'sequence-only prediction of binding affinity changes' to hold on the evaluated benchmarks, every mutation must lie on the input channel that is actually varied. S1131 is a SKEMPI-derived interface mutation set, and SKEMPI contains mutations on both chains of a complex; the manuscript does not state that all S1131 mutations were filtered or relabeled to a single chain, nor does it acknowledge this as a limitation. If the released code simply treats one PDB chain as 'antibody' and the other as 'antigen', any antigen-side mutation would leave the mutant input identical to the wild-type input, making those predictions uninformatative. This is more than a missing baseline or a robustness caveat: it would mean the model is not actually scoring those mutations. The reader's weakest_assumption identified this same issue, so I agree. The conditional verdict remains appropriate because the concern is checkable with the released code and data: the authors should demonstrate that the input pipeline correctly encodes the mutated residue for every benchmark example, or the central claim should be restricted to antibody-side mutations. I also note that the reader's strongest_claim misstates Table 1: the ESM2 model's S1131 RMSE is 1.31, not 1.70, and DDGPred-PDB outperforms ProtAttBA on S1131 and AB645 under cross-validation, but this does not change the paper's actual claim of competitive performance and robustness with uncertain structures.","tokens_in":21741,"tokens_out":11266,"duration_ms":119010,"concrete_test":"Using the released GitHub pipeline, regenerate the exact sequence inputs for every record in S1131, AB645, and AB1101, and count how many records have the mutated residue on the chain passed as 'antibody' versus the chain passed as 'antigen'. If any antigen-side records exist, rerun the ESM2 variant with the true mutant antigen sequence (or with those records removed) and compare RMSE, PCC, and R2 on S1131. If the count is zero and the metrics are unchanged, the concern is refuted; if the input is invariant for any nonzero subset, the reported numbers overstate the model's capability on those mutations.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that ProtAttBA predicts antibody-antigen binding affinity changes from sequence alone requires the model input to actually contain the mutation. In Section 2.2.1 the four inputs are defined as 'the wild-type antibody, the wild-type antigen, the mutated antibody, and the wild-type antigen' (also shown in Fig. 1), so there is no input channel for a mutation on the antigen chain. For any antigen-side mutation, the 'mutant' input would be identical to the wild-type input, and the predicted value would be invariant to that mutation. The paper never states that all evaluated mutations are on the antibody chain, and the benchmark descriptions do not guarantee it: S1131 is derived from SKEMPI, a general protein-protein interaction mutation database containing mutations on either partner, and AB645/AB1101 are not explicitly restricted to antibody-chain mutations in the text. If the released pipeline did not filter or silently relabel such mutations, then a subset of the reported S1131 and AB1101 results is not computed on a meaningful representation of the corresponding mutation. This directly affects the headline claim rather than only its interpretation, because those predictions would not be predictions of the labeled mutation at all.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript describes ProtAttBA, a sequence-only deep learning model that predicts binding-affinity changes (ΔΔG) for antibody-antigen complexes. The model encodes wild-type and mutant antibody/antigen sequences with a frozen protein language model (ESM2, ESM-1b, ProtBert, or Ankh), then applies 1D convolutions, dual multi-head cross-attention with rotary embeddings, and convolutional pooling before a regression head. The authors evaluate on AB645, S1131, and AB1101 under K-fold cross-validation, a 30% sequence-identity split, and a mutation-depth split, comparing with sequence- and structure-based baselines. They report that ProtAttBA is competitive or better than baselines, robust to structural uncertainty, and that attention weights highlight functionally relevant residues in two case studies.","tokens_in":21995,"tokens_out":9105,"duration_ms":83292,"significance":"Accurate sequence-only prediction of mutation effects would be practically valuable for antibody engineering, especially when structures are unavailable; the paper also provides a useful comparison of four protein language models and reports an attention ablation. Strengths include a clear architecture, code/data availability, and evaluation under multiple protocols. However, the headline performance claim is not fully supported by the reported tables: DDGPred-PDB outperforms ProtAttBA on S1131 in Table 1, and the cross-validation protocol may permit same-complex leakage. If the mutation-chain restriction is handled and the performance claims are revised to be accurate, the framework could still make a modest contribution.","major_comments":[{"comment":"The input is defined as wild-type antibody, wild-type antigen, mutated antibody, and wild-type antigen, so a mutation on the antigen chain is not represented in any input channel; for such a mutation the model output is invariant to the mutation. Since S1131 is derived from SKEMPI, which contains mutations on either interaction partner, and the text does not state that all evaluated mutations occur on the antibody chain, the reported results may include predictions for inputs that do not encode the labeled mutation. Please state the filtering rule, report the per-benchmark count of antibody-side versus antigen-side mutations, and either exclude antigen-side mutations or extend the architecture to encode them.","section":"Section 2.2.1 and Fig. 1"},{"comment":"The text says ProtAttBA 'consistently outperforms all baseline methods,' but this is contradicted by the table. DDGPred-PDB achieves RMSE 0.95±0.13, R2 0.84±0.04, PCC 0.92±0.02, and Spearman 0.85±0.02 on S1131, while ProtAttBA-ESM2 achieves 1.31±0.09, 0.69±0.09, 0.84±0.05, and 0.75±0.06; DDGPred-PDB is also better on AB645's R2, PCC, and Spearman. The claim should be limited to what the table supports, and the comparison should include significance testing over folds.","section":"Section 3.2, Table 1"},{"comment":"The K-fold cross-validation description does not state whether the splits are grouped by complex. Because AB645, S1131, and AB1101 contain multiple mutations per complex, random splitting can place mutations from the same complex in both training and test folds, which inflates correlation and R2. Please clarify whether folds are complex-aware; if they are not, report complex-grouped cross-validation results.","section":"Section 3.1"},{"comment":"The sequence-identity and mutation-depth results are given as single numbers with no standard deviations or confidence intervals, so statements such as 'ProtAttBA maintained relatively stable and reliable performance' are not quantitatively supported. Please report repeated runs with different random seeds or otherwise provide uncertainty estimates for these splits.","section":"Section 3.2, Table 2"},{"comment":"The paper calls S1131 an antibody-antigen benchmark, but S1131 from SKEMPI is a general protein-protein interaction mutation set. If S1131 includes non-antibody complexes, then the terms 'antibody' and 'antigen' are misnomers for those entries, and conclusions about antibody engineering from S1131 are not justified. Please clarify the composition of S1131 and restrict the claims or the dataset accordingly.","section":"Section 2.1"},{"comment":"The interpretability claim relies on two selected complexes and visual inspection of attention maps, with no quantitative measure of whether attention-highlighted residues coincide with known functional sites beyond the two examples. The conclusion that 'attention scores can identify critical residues' is therefore not established. Please add quantitative validation (e.g., overlap with known epitope residues across many complexes) or weaken the claim to a qualitative observation.","section":"Section 3.3"}],"minor_comments":[{"comment":"In the abstract, 'Therefore, We propose' has an unnecessary capital 'W' in 'We'; this should be lowercase.","section":"Abstract"},{"comment":"The sentence 'Hl o denotes the lth column in Ho, i.e., the ith position of the protein' uses both l and i for the same index; the index notation should be made consistent.","section":"Equation (7)"},{"comment":"The text says the joint representation is 'summarized by summing the information from the wild-type and mutant complexes,' but the equation f = concat(f_wt_ab + f_wt_ag, f_mt_ab + f_mt_ag) is a concatenation of two sums; this wording should be clarified.","section":"Section 2.2.3"},{"comment":"Several result entries are typeset without spaces (e.g., 'ProtAttBA-ESM21.70±0.25'), making the table difficult to read; the LaTeX spacing should be fixed.","section":"Table 1"},{"comment":"The sentence fragment 'Using early stopping to avoid overfitting.' should be integrated into the preceding sentence.","section":"Section 3.1"}],"recommendation":"major_revision","confidential_remarks":"The most serious issues are the unsupported 'consistently outperforms' claim in view of Table 1 and the unclarified antigen-mutation handling; both are fixable in revision. I would also ask the editor to ensure that the authors release the exact data-split indices with the code, since the GitHub link alone is not sufficient to verify the complex-grouping question."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a competent empirical paper, better than the novelty score suggests, with a load-bearing caveat the authors didn't state. The architecture is not conceptually new — frozen pLM embeddings, 1D conv, RoPE, dual cross-attention, MLP head — but it is a sensible configuration and the comparisons are useful. The three split protocols (cross-validation, sequence-identity, mutation-depth) are a real strength; the robustness to structure quality is the correct framing and the numbers support it. Code and data are promised at the GitHub link, which is the right way to do this kind of work.\n\nWhere I would push back: Section 2.2.1 defines the input as wild-type antibody, wild-type antigen, mutated antibody, wild-type antigen. There is no channel for an antigen-side mutation. The paper never states that all mutations in AB645, S1131, or AB1101 are on the antibody chain. S1131 comes from SKEMPI, which is a general PPI mutation database, and the text does not describe a filter. If any rows in the benchmarks are antigen mutations, then those test inputs are literally identical to the wild-type pair, the model returns a value that is invariant to the mutation, and the reported RMSE/R2/PCC are averaged over some samples for which the model never saw the mutation. That is a direct threat to the headline claim, not a fine point. The fix is easy: state the assumption, filter to antibody-chain mutations, or add a fourth input for mutant antigen.\n\nSmaller issues: Table 2 has no error bars, there is no significance testing in Table 1, random K-fold by row can leak complexes between folds, and the interpretability claim rests on two examples. All minor relative to the chain issue. I also missed baselines like MCSM-AB or recent immune-specific pLM scorers, but the included comparisons are fair.\n\nOverall: I would send this to a serious referee. The empirical setup is above average, the code/data availability is a plus, and the mutation-chain question is resolvable in review. People building antibody engineering pipelines will want to read this, but I would not cite the S1131 numbers until that question is answered.","headline":"ProtAttBA is a clean sequence-only ΔΔG predictor with competitive numbers and a real robustness story, but it never states that mutations are restricted to the antibody chain, and if any antigen-side mutations are in S1131 or AB1101 the reported averages are partly for inputs that do not contain the mutation.","tokens_in":22454,"tokens_out":2634,"would_cite":false,"duration_ms":28119,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ProtAttBA predicts antibody–antigen binding affinity changes from protein sequences alone, without structural input.","keywords":["antibody engineering","binding affinity change prediction","sequence-only model","cross-attention","protein language model","ΔΔG regression","interpretability","attention weights"],"falsifier":"Take any mutation in the benchmark data that changes the antigen rather than the antibody chain, and feed ProtAttBA the wild-type antigen sequence for both the wild-type and mutant complexes, as the architecture prescribes. If the model's predicted $\\Delta\\Delta G$ is unchanged or uncorrelated with the measured value for those cases, then the paper's sequence-only claim is limited to antibody-side mutations.","tokens_in":21591,"feed_emoji":"🧬","tokens_out":14188,"duration_ms":126170,"temperature":0.7,"pith_summary":"The paper introduces ProtAttBA, a deep-learning model that predicts the change in antibody–antigen binding free energy ($\\Delta\\Delta G$) from protein sequences alone, with no structure input. Its central thesis is that sequence-only prediction can be competitive with structure-based methods in standard benchmarks and more dependable when complex structures are absent or predicted with low confidence. On three open benchmarks under cross-validation, sequence-identity splits, and mutation-depth splits, the best ProtAttBA variant reaches $R^2=0.69$ and Pearson correlation $0.84$ on a single-site benchmark, and it retains accuracy when structure-based baselines collapse after switching from crystal structures to predicted ones. If the claim holds, antibody engineers can rank mutant designs quickly from sequence and use the model's attention scores to choose which residues to test experimentally.","feed_headline":"No structure needed: model predicts antibody binding changes","feed_subtitle":"Sequence pairs and attention let ProtAttBA match structure-based energy prediction, even without reliable structures.","key_machinery":"The engine is a frozen pre-trained protein language model feeding a dual multi-head cross-attention block. In each block, one chain's representation is transformed into queries while the partner chain supplies keys and values, so residue-level information from the antigen can reshape the antibody representation and vice versa; rotary position embeddings preserve residue order. A 1D convolution with kernel size 1 gates the incoming residue features, and convolutional pooling compresses the attended matrices into chain-level vectors. The same machinery runs on wild-type and mutant pairs, and the final regression head concatenates the two pooled pairs before three fully connected layers. The cross-attention score matrices are also the interpretability output: the paper shows that high-attention residue pairs align with positions whose mutation disrupts hydrogen bonds.","core_discovery":"ProtAttBA's core claim is that a quartet of sequences—wild-type antibody, wild-type antigen, mutant antibody, and again wild-type antigen—contains enough information to regress $\\Delta\\Delta G$ accurately, provided the chains are allowed to attend to each other. A frozen pretrained protein language model produces residue embeddings; a convolution gates local residue features; then symmetric multi-head cross-attention lets the antibody representation read from the antigen and vice versa, for both wild-type and mutant pairs. The pooled wild-type and mutant features are concatenated and passed through three fully connected layers to output $\\Delta\\Delta G$. On the single-site benchmark under cross-validation, the best variant reaches RMSE 1.31, $R^2=0.69$, PCC 0.84, and Spearman 0.75. This is competitive with the strongest structure-based method when that method receives crystal structures, and clearly better when that method is forced to use predicted structures instead. The paper's conclusion is that the bottleneck for practical binding-affinity prediction is often structural input quality—a bottleneck sequence-only models sidestep.","pith_inferences":["Editorial extension: because the current input already uses the wild-type antigen twice, extending ProtAttBA to predict antigen-side mutations would require adding a mutant-antigen input; the symmetric cross-attention design could plausibly support this without architectural changes, but the paper does not test it.","Editorial extension: attention mass on residue pairs could be converted into a mutagenesis-prioritization score, and its agreement with alanine-scanning or deep-mutational-scanning data could be quantified experimentally.","Editorial extension: the evaluation design of feeding the same structure-based model crystal versus predicted structures suggests a general standard—any structure-aware binding model should report how much its accuracy drops when structures come from folding tools rather than crystals.","Editorial extension: if attention between antibody and antigen is the reason these sequence-only predictions work, then zero-shot protein language models that score mutants without antigen context may be leaving a large amount of signal on the table."],"forward_implications":["A researcher with only antibody and antigen sequences can obtain useful $\\Delta\\Delta G$ estimates without building or predicting a complex structure.","On multi-site mutations, the model still performs well, so it can rank multi-residue variants, not just single point mutants.","Because all four tested pretrained language model backbones give similar results, the cross-attention interaction module, not the choice of embedding model, appears to carry the predictive signal.","When structure-based methods are given predicted rather than experimental structures, their accuracy drops; ProtAttBA's stability suggests it is the safer choice in screening pipelines that lack crystallography.","The attention weights can serve as a residue-level map that points to positions worth targeted mutagenesis, giving a mechanistic handle on the model's predictions."],"supporting_citations":[{"why":"Supplies the SKEMPI database, the experimental source of the single-site benchmark values.","marker":"Moal and Fernández-Recio (2012)"},{"why":"Supplies the AB-bind database, the experimental source of the single- and multi-site benchmark values.","marker":"Sirin et al. (2016)"},{"why":"Curates the single-site benchmark S1131 from SKEMPI and defines its mutation set.","marker":"Xiong et al. (2017)"},{"why":"Provides the AB645 and AB1101 benchmarks, including multi-site mutations across 1 to 7 residues.","marker":"Wang et al. (2020)"},{"why":"Defines the sequence-only attention baseline and the cross-validation protocol the model is compared against.","marker":"Jin et al. (2024)"},{"why":"Supplies the structure-based deep learning baseline whose performance drops when fed predicted rather than crystal structures.","marker":"Shan et al. (2022)"},{"why":"Provides one of the pretrained protein language models used for the frozen embedding module.","marker":"Rives et al. (2021)"},{"why":"Provides the pretrained language model whose embeddings give the best overall ProtAttBA variant.","marker":"Lin et al. (2023)"}],"fun_headline_variants":["Sequence-only model predicts antibody affinity changes without structures","Attention-based model matches structure-based predictions using only sequences","ProtAttBA: sequence-based affinity prediction rivals structural methods","No structure? No problem: sequences alone predict binding changes","Interpretable attention spots key residues from sequences alone"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that every mutation to be predicted lies on the antibody chain: the model is always given the wild-type antigen for both the wild-type and mutant complex, so an antigen-side mutation would be invisible to it and the prediction would be undefined.","fun_headline_variants_meta":{"raw":{"variants":["Sequence-only model predicts antibody affinity changes without structures","Attention-based model matches structure-based predictions using only sequences","ProtAttBA: sequence-based affinity prediction rivals structural methods","No structure? No problem: sequences alone predict binding changes","Interpretable attention spots key residues from sequences alone"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000534,"raw_usage":{"total_tokens":2580,"prompt_tokens":972,"completion_tokens":1608,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":588,"completion_tokens_details":{"reasoning_tokens":1532}},"tokens_in":588,"tokens_out":1608,"duration_ms":12364,"temperature":1.0,"reasoning_tokens":1532,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:31:02.377510+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take any mutation in the benchmark data that changes the antigen rather than the antibody chain, and feed ProtAttBA the wild-type antigen sequence for both the wild-type and mutant complexes, as the architecture prescribes. If the model's predicted $\\Delta\\Delta G$ is unchanged or uncorrelated with the measured value for those cases, then the paper's sequence-only claim is limited to antibody-side mutations.","supporting_citations":[],"review_version":1}