{"id":"9a88f1a6-474d-4b92-a73f-1420c0c23a90","arxiv_id":"2505.20036","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A systematic benchmark and architecture comparison shows that hierarchical pooling and pooled cross-attention usually beat concatenation for PLM-based protein-protein binding affinity prediction, although statistical significance is not established.","lead":"Researchers release a cleaned, split-on-30%-sequence-identity benchmark of 8,207 protein-protein complexes and compare four architectures that adapt protein language models for binding affinity prediction. The two more elaborate architectures (hierarchical pooling, cross-attention addition) beat simple concatenation in most settings, which matters for anyone building sequence-based predictors for drug discovery.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Claim that HP/PAD 'consistently outperform' concatenation is contradicted by multiple entries in the paper's own Supplementary Table 3, and the 12% figure is a selected contrast rather than a systematic effect.","rationale":"The central claim of the paper is the consistent superiority of HP/PAD over EC/SC for PLM-based PPI affinity prediction. For this claim to hold, the architecture ranking would need to be robust across the evaluated PLMs and training schemes. The paper's own full results (Supplementary Table 3) contain multiple cells in which SC or EC matches or exceeds HP/PAD (e.g., ESM2-3B FT-SC, Ankh2-Ext2 CV-SC, ESM2-650M FT-SC, Ankh2-Ext1 CV-SC, ESM3 CV-SC). With three seeds, the error bars overlap substantially, so even the favorable differences are not statistically secured. The 'up to 12%' phrasing is a selected maximum, not a typical effect. This makes the headline claim unsupported as stated, which is a more direct threat to the paper's central contribution than the dataset reconstruction issue. The dataset concern is real—the authors themselves note that SEQRES-based missing-residue recovery creates inconsistencies with original PDB structures (Section 4)—but it affects all architectures equally and would primarily challenge absolute performance, whereas the relative ranking is the paper's main message. A conditional accept with a required rewrite of the abstract/conclusion to a weaker claim is appropriate; hence the reader's verdict stands unchanged.","tokens_in":16452,"tokens_out":5984,"duration_ms":56951,"concrete_test":"Using the published code and dataset, recompute the 3-seed test Spearman values for all cells in Supplementary Table 3. For each PLM×training-method cell, determine which architecture achieves the highest mean Spearman; count how many cells HP, PAD, EC, and SC win. Then run a paired Wilcoxon signed-rank test (or a permutation test) on the cell-level means contrasting HP+PAD against EC+SC. If HP/PAD win in fewer than ~70% of cells, or the paired test gives p>0.05, the 'consistently outperform' claim is not supported. Merely re-aggregating the published table's means and standard deviations provides this check without new experiments.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Supplementary Table 3 lists test-set Spearman means over 3 seeds for every PLM/training/architecture cell. Under 'consistently outperform,' each HP/PAD entry should exceed each EC/SC entry in its cell, or at least across nearly all cells. Instead, the table shows counterexamples: ESM2-3B FT-SC (0.44±0.01) ties FT-PAD (0.44±0.01) and beats FT-HP (0.41±0.02) and FT-EC (0.41±0.02); Ankh2-Ext2 CV-SC (0.41±0.02) beats CV-HP (0.39±0.03) and CV-PAD (0.40±0.01); ESM2-650M FT-SC (0.43±0.03) beats FT-PAD (0.39±0.02); Ankh2-Ext1 CV-SC (0.41±0.02) ties CV-PAD (0.41±0.02) and beats CV-HP (0.39±0.02); ESM3 CV-SC (0.35±0.02) is statistically indistinguishable from CV-HP (0.35±0.01) and CV-PAD (0.36±0.04). With only 3 seeds, standard deviations overlap for nearly all architecture pairs, so the mean ordering is not established as significant. The 'up to 12%' improvement appears to be the best single contrast (ProtT5 CV-PAD 0.48 vs CV-EC 0.43 ≈ 12%), not a representative effect. The central claim therefore overstates the evidence; the data support 'HP/PAD often help, depending on PLM and training scheme,' not universal superiority.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces a curated version of the PPB-Affinity dataset for sequence-based multi-chain protein-protein interaction (PPI) binding affinity prediction, with a pipeline that corrects annotation errors, fills missing residues from SEQRES records, removes short chains, consolidates duplicates, and applies a stringent ≤30% sequence identity split. The authors then compare four PLM adaptation architectures—embedding concatenation (EC), sequence concatenation (SC), hierarchical pooling (HP), and pooled attention addition (PAD)—under full fine-tuning and a frozen-PLM ConvBERT head across several PLMs (ProtT5, ESM2 variants, Ankh, Ankh2, and ESM3). The central claim is that HP and PAD consistently outperform conventional concatenation methods, achieving up to 12% improvement in Spearman correlation. The paper also releases its code and dataset.","tokens_in":16848,"tokens_out":4712,"duration_ms":49663,"significance":"If fully supported, the paper would provide useful practical guidance for adapting protein language models to multi-chain PPI affinity prediction, and the released dataset with a low-leakage split would be a valuable community resource. The study's strengths include its reproducible pipeline and public code, the breadth of the PLM/training-scheme matrix, and the explicit attention to data leakage. However, the headline claim is not supported by the paper's own full results in Supplementary Table 3; the evidence supports a more modest conclusion that HP and PAD often help but not consistently, and that gains depend heavily on the PLM and training scheme.","major_comments":[{"comment":"The claim that HP and PAD 'consistently outperform' EC and SC is contradicted by the paper's full results. For example, ESM2-3B FT-SC reaches test Spearman 0.44±0.01, tying FT-PAD (0.44±0.01) and exceeding FT-HP (0.41±0.02) and FT-EC (0.41±0.02); ESM2-650M FT-SC (0.43±0.03) exceeds FT-PAD (0.39±0.02); Ankh2-Ext1 CV-SC (0.41±0.02) ties CV-PAD (0.41±0.02) and beats CV-HP (0.39±0.02); and Ankh2-Ext2 CV-SC (0.41±0.02) beats CV-HP (0.39±0.03). The 'up to 12%' increase appears to be a single best contrast (ProtT5 CV-PAD 0.48 vs CV-EC 0.43), not a systematic effect. The abstract and Section 3 should be revised to state that HP and PAD tend to help in many settings, with the specific favorable and unfavorable configurations identified.","section":"Abstract and Section 3, Table 3"},{"comment":"No significance tests are reported for the architecture comparisons. With only three seeds and the reported standard deviations, many differences are likely within noise (e.g., ESM3-SM-Open CV-SC 0.35±0.02 vs CV-HP 0.35±0.01 vs CV-PAD 0.36±0.04). Since the central claim is about consistent superiority, the authors should report paired tests across seeds (or a mixed-effects model with architecture as a factor and PLM/setup interactions) and effect sizes. The current presentation of means without inference does not establish the claimed ranking.","section":"Section 3, Table 3"},{"comment":"The benchmark's validity rests on the reconstructed sequences: missing residues are filled from SEQRES records, duplicate affinity values are averaged across experimental methods, and Section 4 concedes that these repairs created inconsistencies with original PDB structures. Because every architecture comparison occurs on this reconstructed dataset, the possibility of sequence-label mismatches or artifacts is load-bearing. The authors should provide validation analyses, such as the distribution of recovered residues, a comparison of model performance with and without affected entries, or a consistency check of reconstructed sequences against independent structural information. Without such checks, the dataset's reliability for benchmarking is not fully established.","section":"Sections 2.1.2 and 4"}],"minor_comments":[{"comment":"The caption begins 'Table presents...' and should read 'The table presents...' for grammatical consistency.","section":"Table 3 caption"},{"comment":"The term 'ConvBERT' is used in the abstract but introduced only in Section 2.3; please define or briefly describe it at first use.","section":"Abstract and Section 2.3"},{"comment":"The description of marginal means would be clearer if it stated whether they are unweighted averages across all cells; this affects how readers interpret the per-PLM and per-setup summaries.","section":"Figure 2"},{"comment":"The 'fairly assessing' framing would be strengthened by including at least one external baseline (e.g., a published PPI affinity prediction method evaluated on the same split), even if it is not the main focus of the architecture comparison.","section":"Section 3"}],"recommendation":"major_revision","confidential_remarks":"The paper has a useful empirical contribution—a carefully split, public PPI affinity dataset and a broad internal architecture comparison—but the abstract overstates the evidence. The contradiction with Table 3 is easy to address by softening the claims and adding significance testing; I do not see a fundamental flaw in the study design. The dataset validation issue in Section 4 is the most substantive remaining concern and should be addressed in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read it. The useful part is the curated PPB-Affinity subset and the systematic comparison grid. 8,207 entries, a real strict 30% identity split with PDB coherency, public code and data. That is a legitimate benchmark contribution, and the four architectures across five PLMs and two training modes is a reasonable first map of the design space. If I worked on sequence-based PPI affinity prediction, I would start from this dataset.\n\nThe main claim, though, does not hold as written. \"HP and PAD consistently outperform concatenation\" is contradicted by their own Table 3. ESM2-3B FT-SC ties PAD at 0.44 and beats HP at 0.41; Ankh2-Ext2 CV-SC beats CV-HP and ties PAD; ESM2-650M FT-SC beats FT-PAD by 0.04. There are enough such cells that \"consistently\" is simply false. The 12% figure is the best single contrast (ProtT5 CV-PAD vs CV-EC), not a representative effect. And with only three seeds, the standard deviations overlap for most architecture pairs, so even the mean ordering is not established. The paper needs significance tests, or at least paired comparisons with confidence intervals, and it needs to either soften the abstract or show an aggregated effect.\n\nThe dataset reconstruction is a softer spot. Recovering missing residues from SEQRES and then averaging affinities across experimental methods is defensible, and they flag the structural inconsistency in the conclusion. But because all experiments run on this one dataset, a quick external sanity check (e.g., compare against the raw PPB-Affinity, or reproduce a few known entries) would make the benchmark easier to trust. Some baselines outside the four architectures would also help: D-SCRIPT is cited but not run, and no existing PPI affinity method is compared.\n\nThat said, the work is honest, the limitations are acknowledged, and the reproducibility is real. The central tendency—HP/PAD often help, especially with fine-tuning—is plausible. The paper just needs its claims reigned in and a bit more statistical discipline. I would send it out for review: the dataset alone deserves a serious look, and a referee can push on the significance issue. A desk reject would waste a useful benchmark.","headline":"Useful curated PPI affinity benchmark, but the 'consistently outperform' claim is contradicted by the paper's own numbers.","tokens_in":17364,"tokens_out":3022,"would_cite":true,"duration_ms":33288,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pooled and cross-attention adapters beat concatenation for protein-protein affinity prediction.","keywords":["protein-protein interaction","binding affinity prediction","protein language models","hierarchical pooling","cross-attention","PPB-Affinity dataset","sequence identity splitting","multi-chain complexes"],"falsifier":"Retrain the four architectures on the original PPB-Affinity entries without SEQRES residue recovery and without averaging affinities; if HP and PAD no longer consistently beat EC and SC, the claimed architectural advantage is an artifact of the dataset reconstruction rather than a general property of the models.","tokens_in":16286,"feed_emoji":"🧬","tokens_out":4383,"duration_ms":44661,"temperature":0.7,"pith_summary":"The paper argues that how a protein language model's per-chain representations are combined matters more than the choice of model for sequence-based protein-protein binding affinity prediction. It introduces a cleaned version of the PPB-Affinity benchmark with 8,207 entries and a ≤30% sequence-identity split, then compares four adapter architectures across five PLM families and two training regimes. The central result is that hierarchical pooling (HP) and pooled attention addition (PAD) consistently outperform simple embeddings or sequence concatenation, with up to a 12% relative gain in Spearman correlation. If correct, this means standard concatenation pipelines are leaving predictive signal on the table for multi-chain interactions.","feed_headline":"Pooling beats concatenation for protein affinity","feed_subtitle":"Hierarchical pooling and cross-attention addition rank binding strengths better across five protein language models.","key_machinery":"The load-bearing mechanism is the two-step pooling hierarchy and the cross-attention residual block. HP's two-level attention pooler (intra-chain then inter-chain) keeps each chain's representation distinct before forming a partner-level vector, avoiding the dilution that follows pooling a long concatenated chain sequence. PAD's shared multi-head cross-attention lets each partner's residues attend to the other partner's residues, then adds that context back and sums the pooled partner embeddings, an explicit inductive bias for intermolecular interaction. The same 1D global attention pooling is reused in all architectures, so the comparisons isolate the aggregation strategy.","core_discovery":"The authors claim that for adapting PLMs to multi-chain PPI affinity prediction, concatenating chains before or after encoding is systematically worse than architectures that preserve chain identity and model ligand-receptor interaction explicitly. HP encodes each chain separately, pools per-chain embeddings, then pools across chains; PAD applies cross-attention between ligand and receptor hidden states, adds the attention output back residually, pools, and sums the two partner embeddings instead of concatenating. Across ProtT5, ESM2, Ankh, Ankh2, and ESM3, with both full fine-tuning and frozen-feature ConvBERT heads, HP and PAD occupy the top tier while EC and SC rarely do, up to a 12% Spearman improvement.","pith_inferences":["If the architecture ranking holds on other affinity benchmarks, the same adapter designs could be adopted by tools that currently use concatenation.","The HP advantage should be largest for complexes with more than one chain per partner; a per-subset breakdown by number of chains would test this directly and is not reported in the paper.","The competitive frozen-feature ConvBERT results suggest that for very large PLMs, avoiding fine-tuning may make sophisticated adapters affordable in practice."],"forward_implications":["For sequence-only PPI affinity pipelines, switching from concatenation to HP or PAD should improve ranking of candidate binders by Spearman correlation, by up to 12% relative in the tested setups.","Full PLM fine-tuning is not always worth the cost: for ESM2, ESM3, and ProtT5, a frozen PLM with a ConvBERT head matches or nearly matches fine-tuning, while for Ankh and Ankh2 fine-tuning is needed.","No single PLM is uniformly best; ProtT5 has the highest average correlation, but ESM2-650M and Ankh2-Ext1 win in specific HP setups, and larger ESM2-3B does not beat ESM2-650M.","The curated dataset with ≤30% sequence identity splitting should reduce leakage, so models evaluated on it are being tested for generalization rather than memorization."],"supporting_citations":[{"why":"Provides the source PPB-Affinity dataset from which the curated 8,207-entry benchmark is derived.","marker":"[23]"},{"why":"Supplies the SKEMPI2 data underlying many PPB-Affinity entries and affinity labels.","marker":"[25]"},{"why":"MMseqs2 clustering and search are the tools used to enforce the ≤30% sequence identity split.","marker":"[33]"},{"why":"Establishes the embeddings concatenation baseline that the proposed architectures are compared against.","marker":"[16]"},{"why":"Another concatenation-based sequence PPI approach that motivates the architectural comparison.","marker":"[19]"},{"why":"Provides the multi-head attention mechanism used for cross-attention in the PAD architecture.","marker":"[37]"},{"why":"Defines the ConvBERT model used as the lightweight frozen-feature prediction head.","marker":"[38]"},{"why":"Supplies the ESM2 PLM evaluated in the experiments.","marker":"[11]"}],"fun_headline_variants":["Pooling beats concatenation for protein affinity","Protein PPI models: attention and pooling outperform","Concatenation loses: new PLM architectures rank higher","12% better Spearman: pooling over concatenation for PPI","Redesigning PLM heads: better protein interaction scores"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation rests on the reconstructed PPB-Affinity dataset: missing residues were filled from SEQRES records and affinity values were averaged across methods, and the paper itself notes these repairs create inconsistencies with the original PDB structures, so the architecture ranking is only as valid as that reconstruction.","fun_headline_variants_meta":{"raw":{"variants":["Pooling beats concatenation for protein affinity","Protein PPI models: attention and pooling outperform","Concatenation loses: new PLM architectures rank higher","12% better Spearman: pooling over concatenation for PPI","Redesigning PLM heads: better protein interaction scores"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000126,"raw_usage":{"total_tokens":1133,"prompt_tokens":993,"completion_tokens":140,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":609,"completion_tokens_details":{"reasoning_tokens":62}},"tokens_in":609,"tokens_out":140,"duration_ms":2187,"temperature":1.0,"reasoning_tokens":62,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:00:53.667990+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the four architectures on the original PPB-Affinity entries without SEQRES residue recovery and without averaging affinities; if HP and PAD no longer consistently beat EC and SC, the claimed architectural advantage is an artifact of the dataset reconstruction rather than a general property of the models.","supporting_citations":[{"cited_title":"Ppb-affinity: Protein-protein binding affinity dataset for ai-based protein drug discovery.Scientific Data, 11(1):1–11, 2024","cited_arxiv_id":null,"evidence_quote":"Provides the source PPB-Affinity dataset from which the curated 8,207-entry benchmark is derived."},{"cited_title":"Skempi 2.0: an updated benchmark of changes in protein–protein binding energy, kinetics and thermodynamics upon mutation.Bioinformatics, 35(3):462–469, 2019","cited_arxiv_id":null,"evidence_quote":"Supplies the SKEMPI2 data underlying many PPB-Affinity entries and affinity labels."},{"cited_title":"Con- vbert: Improving bert with span-based dynamic convolution.Advances in Neural Information Processing Systems, 33:12837–12848, 2020","cited_arxiv_id":null,"evidence_quote":"Defines the ConvBERT model used as the lightweight frozen-feature prediction head."}],"review_version":1}