Pith. sign in

REVIEW 1 major objections 2 minor 19 references

Protein pocket markers and ESM-2 embeddings improve binding affinity predictions on standard splits but reduce performance on strict no-leak test tiers.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-28 08:06 UTC pith:WON5QLML

load-bearing objection The reversal on no-leak splits is the main point but the protocol details are missing so the claim stays unverified. the 1 major comments →

arxiv 2606.03422 v1 pith:WON5QLML submitted 2026-06-02 cs.CE

HonestAffinity: Leak-Aware Evaluation of Protein and Pocket Priors for Binding Affinity Prediction

Junhao Wei , Baili Lu , Zhenhong Peng , Wanyan Li , Zhirong Huang , Yanxiao Li , Yifu Zhao , Dexing Yao
show 5 more authors
Haochen Li Xudong Ye Sio-Kei Im Yapeng Wang Xu Yang
This is my paper
classification cs.CE
keywords binding affinity predictionleak-aware evaluationsequence-based modelsprotein embeddingspocket markerno-leak splitsprotein-ligand complexesdeep learning predictors
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper evaluates three variants of a compact sequence-based model for protein-ligand binding affinity under a protocol meant to block similarity leaks between train and test data. It reports that adding a binary pocket-position marker and frozen ESM-2 embeddings raises mean Pearson correlation on validation and CASF-2016 splits, yet the version that drops the ESM-2 input performs best on every strict no-leak tier of the LP-PDBBind hold-out. The reversal indicates that the value of each prior is conditional on how much the test proteins resemble those seen in training. A reader would care because most published benchmarks use splits that allow such resemblance, so they may select priors that fail when truly novel targets appear.

Core claim

HonestAffinity-Pocket achieves the best mean Pearson R on validation and CASF-2016 splits, whereas HonestAffinity-Pocket-NoESM achieves the best mean Pearson R on every strict LP no-leak tier (test_cl1-cl3). Both the pocket marker and ESM-2 input improve performance on familiar splits but reduce Pearson R on strict no-leak tiers. The authors conclude that models should report paired canonical and leak-proof ablations and that deployment-regime-matched variants describe performance more accurately than any single default choice.

What carries the argument

The HonestAffinity 1D multi-scale convolutional/Transformer predictor, run in three ablated forms to isolate the contribution of a learned binary pocket-position marker and frozen ESM-2 protein embeddings under a three-tier no-leak hold-out protocol.

Load-bearing premise

The LP-PDBBind three-tier no-leak hold-out protocol removes every relevant similarity between training and test complexes.

What would settle it

Re-evaluating the same three model variants after redefining the no-leak tiers with stricter sequence-identity cutoffs or with structure-based clustering instead of the original method, and finding that the performance reversal between Pocket and Pocket-NoESM disappears.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Both the pocket marker and ESM-2 embeddings raise correlation when test data shares detectable similarity with training data.
  • The same priors lower correlation once all detectable similarity is removed from the test tiers.
  • Reporting both standard and leak-proof splits reveals conditional behavior that a single split conceals.
  • Selecting a model variant matched to the expected similarity regime of deployment gives a clearer performance picture than a universal default.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • In practice a user expecting mostly novel proteins might prefer the lighter Pocket-NoESM variant while a user working inside a well-sampled protein family might keep both priors.
  • Sequence embeddings can capture patterns that improve interpolation inside similarity clusters but add noise once those clusters are excluded.
  • The same reversal pattern could be checked for other common priors such as ligand graph features or additional structural inputs.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 2 minor

Summary. The paper introduces HonestAffinity, a compact 1D convolutional/Transformer model for protein-ligand binding affinity prediction that isolates the effects of frozen ESM-2 (650M) embeddings and a learned binary pocket-position marker. Three variants (Pocket, NoPocket, Pocket-NoESM) are trained on 11,513 LP-PDBBind complexes and evaluated against five baselines on the LP-PDBBind 3-tier no-leak hold-out (test_cl1-cl3), CASF-2016, and a CASF-2016 non-train subset. The central claim is a split-conditioned reversal: HonestAffinity-Pocket yields the highest mean Pearson R on validation and CASF-2016, while HonestAffinity-Pocket-NoESM is best on every strict no-leak tier; both the pocket marker and ESM-2 input improve familiar splits but degrade performance under strict no-leak conditions. The authors advocate reporting paired canonical and leak-proof ablations and deployment-matched variants.

Significance. If the no-leak protocol holds, the reversal demonstrates that architectural priors can produce opposite rankings under different similarity regimes, supporting the call for leak-aware evaluation protocols in affinity prediction. The work ships linked code and scripts (with checkpoints promised), enabling direct reproduction of the ~3 GPU-hour training runs and the multi-scale template. This is a concrete strength for a computational paper.

major comments (1)
  1. [Methods (LP-PDBBind construction and splits)] LP-PDBBind 3-tier no-leak hold-out protocol (Methods): the reversal claim (Pocket best on CASF/validation; Pocket-NoESM best on cl1-cl3) is load-bearing on the assertion that test_cl1-cl3 contain zero relevant sequence, structural, or pocket similarity to the 11,513 training complexes. The abstract states only that a '3-tier no-leak hold-out' is used; if the methods section supplies no quantitative cutoffs (sequence identity, TM-score, pocket RMSD) and no clustering algorithm/parameters, residual leaks remain possible and could artifactually produce the observed reversal.
minor comments (2)
  1. [Abstract / Results] The five baselines are mentioned but not named or referenced in the abstract; a table or explicit list with citations would clarify the comparison.
  2. [Model description] Notation for the three variants (HonestAffinity-Pocket, etc.) is clear, but the multi-scale convolutional/Transformer template architecture is only sketched; a diagram or layer-count table would aid readers.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for highlighting the need for explicit quantitative details on the LP-PDBBind no-leak protocol. This is a valid point that strengthens reproducibility, and we address it directly below.

read point-by-point responses

  1. Referee: [Methods (LP-PDBBind construction and splits)] LP-PDBBind 3-tier no-leak hold-out protocol (Methods): the reversal claim (Pocket best on CASF/validation; Pocket-NoESM best on cl1-cl3) is load-bearing on the assertion that test_cl1-cl3 contain zero relevant sequence, structural, or pocket similarity to the 11,513 training complexes. The abstract states only that a '3-tier no-leak hold-out' is used; if the methods section supplies no quantitative cutoffs (sequence identity, TM-score, pocket RMSD) and no clustering algorithm/parameters, residual leaks remain possible and could artifactually produce the observed reversal.

    Authors: We agree that the no-leak claim requires explicit quantitative thresholds to rule out residual similarity as a confounder for the observed reversal. The current Methods section outlines the 3-tier construction (sequence clustering, structural filtering, and pocket-level checks) but does not list the precise cutoffs and algorithm parameters. We will revise the Methods to include the exact values (e.g., sequence identity <30%, TM-score <0.5, pocket RMSD >2 Å) along with the clustering tools and parameters used. This addition will make the protocol fully reproducible and directly support that the split-conditioned reversal is not an artifact of incomplete leak removal. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical evaluation with no derivations or self-referential fits

full rationale

The paper reports an empirical comparison of three model variants on multiple data splits (LP-PDBBind tiers, CASF-2016). No equations, fitted parameters repurposed as predictions, or derivation chain appear in the provided text. The central claim (split-conditioned reversal in Pearson R) is an observed experimental outcome, not equivalent to its inputs by construction. Absence of similarity-threshold details in the no-leak protocol is a validity issue, not circularity. No self-citation load-bearing or ansatz smuggling is present.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Abstract-only; no explicit free parameters, axioms, or invented entities are stated. The evaluation protocol itself rests on an unstated assumption about leak removal.

axioms (1)
  • domain assumption LP-PDBBind 3-tier splits eliminate similarity-based data leakage between train and test
    Central to the reversal claim; invoked when contrasting 'familiar splits' vs 'strict LP no-leak tier'

pith-pipeline@v0.9.1-grok · 5858 in / 1244 out tokens · 16306 ms · 2026-06-28T08:06:12.807522+00:00 · methodology

0 comments
read the original abstract

Sequence-based deep learning offers a scalable alternative to structure-based scoring for protein-ligand binding affinity prediction. However, progress is hard to interpret when architectural priors are evaluated on canonical PDBbind-style splits that leak similarity classes across folds. We present HonestAffinity, a compact 1D-input predictor to isolate two priors under a leak-aware protocol: frozen ESM-2 (650M) protein embeddings and a learned binary pocket-position marker. We evaluate a multi-scale convolutional/Transformer template in three variants: HonestAffinity-Pocket, HonestAffinity-NoPocket, and HonestAffinity-Pocket-NoESM. All three train on 11,513 LP-PDBBind complexes in ~3 GPU-hours. We benchmark against five baselines on the LP-PDBBind 3-tier no-leak hold-out, CASF-2016, and a CASF-2016 non-train subset. Our central finding is a split-conditioned reversal rather than a uniformly best prior: HonestAffinity-Pocket achieves the best mean Pearson R on validation and CASF-2016 splits, whereas HonestAffinity-Pocket-NoESM achieves the best mean Pearson R on every strict LP no-leak tier (test_cl1-cl3). Both the pocket marker and ESM-2 input improve performance on familiar splits but reduce Pearson R on strict no-leak tiers. We argue models should report paired canonical and leak-proof ablations, and that deployment-regime-matched variants better describe these reversals than a single default. Code and scripts are linked in the footnote; checkpoints will be released upon acceptance.

Figures

Figures reproduced from arXiv: 2606.03422 by Baili Lu, Dexing Yao, Haochen Li, Junhao Wei, Sio-Kei Im, Wanyan Li, Xudong Ye, Xu Yang, Yanxiao Li, Yapeng Wang, Yifu Zhao, Zhenhong Peng, Zhirong Huang.

Figure 1
Figure 1. Figure 1: Architecture of HONESTAFFINITY. The schematic shows the shared 1D-input template and the optional paths that define the three deployment variants. The protein branch uses either frozen ESM-2 (650M) per-residue embeddings projected from 1280 to d=256, or, for HONESTAFFINITY-POCKET￾NOESM, a learned residue embedding. When pocket annotations are available, a learned pocket-marker embedding is added to the pro… view at source ↗
Figure 2
Figure 2. Figure 2: Per-split generalization curves (3-seed mean ± std error bars). Pearson R (left, higher better) and RMSE (right, lower better) are shown across the six evaluation splits. HONESTAFFINITY-POCKET (red) leads on validation and the two CASF splits; HONESTAFFINITY-POCKET-NOESM (purple) leads on the strict LP no-leak tiers (cl1–cl3); HONESTAFFINITY-NOPOCKET (orange) is the variant for targets without pocket annot… view at source ↗
Figure 3
Figure 3. Figure 3: Training dynamics of the three HONESTAFFINITY variants. Validation RMSE / Pearson R / CI per epoch for HONESTAFFINITY-POCKET, HONESTAFFINITY-NOPOCKET, and HONESTAFFINITY-POCKET-NOESM, shown for single representative seeds. This figure is diagnostic rather than a model-selection result: the final comparison uses validation-RMSE-selected checkpoints and three-seed mean±std reporting in Tables I–III. change s… view at source ↗
Figure 4
Figure 4. Figure 4: CASF-2016 scatter. Seed-averaged predicted vs. experimental − log K for DeepDTAF [2] (left), the strongest non-HONESTAFFINITY CASF competitor in Table I, and HONESTAFFINITY-POCKET (right). The scatter is qualitative; the numerical comparison and paired uncertainty check are reported in Tables I–III and Section V-B. discount components that are useful for in-distribution scoring on familiar targets. The mec… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

19 extracted references · 1 canonical work pages

  1. [1]

    DeepDTA: Deep drug–target binding affinity prediction,

    H. ¨Ozt¨urk, A. ¨Ozg¨ur, and E. Ozkirimli, “DeepDTA: Deep drug–target binding affinity prediction,”Bioinformatics, vol. 34, no. 17, pp. i821–i829, 2018

    2018

  2. [2]

    DeepDTAF: a deep learning method to predict protein–ligand binding affinity,

    K. Wang, R. Zhou, Y . Li, and M. Li, “DeepDTAF: a deep learning method to predict protein–ligand binding affinity,”Brief. Bioinform., vol. 22, no. 5, p. bbab072, 2021

    2021

  3. [3]

    DeepBindAffinity: A ResNet-augmented hybrid CNN–Transformer for protein–ligand binding affinity prediction,

    G. A. Girmay, Q. A. Aljanabi, and L. Deng, “DeepBindAffinity: A ResNet-augmented hybrid CNN–Transformer for protein–ligand binding affinity prediction,” inProc. IEEE BIBM, 2025, pp. 126–129

    2025

  4. [4]

    Evolutionary-scale prediction of atomic-level protein structure,

    Z. Lin, H. Akin, R. Rao, B. Hie, Z. Zhu, W. Lu, N. Smetanin, R. Verkuil, O. Kabeli, Y . Shmueli, A. dos Santos Costa, M. Fazel-Zarandi, T. Sercu, S. Candido, and A. Rives, “Evolutionary-scale prediction of atomic-level protein structure,”Science, vol. 379, no. 6637, pp. 1123–1130, 2023

    2023

  5. [5]

    Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences,

    A. Rives, J. Meier, T. Sercu, S. Goyal, Z. Lin, J. Liu, D. Guo, M. Ott, C. L. Zitnick, J. Maet al., “Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences,” PNAS, vol. 118, no. 15, 2021

    2021

  6. [6]

    Leak-proof PDBBind: A reorganized dataset of protein–ligand complexes for more generalizable binding affinity prediction,

    J. Li, X. Guan, O. Zhang, K. Sun, Y . Wang, D. Bagni, and T. Head- Gordon, “Leak-proof PDBBind: A reorganized dataset of protein–ligand complexes for more generalizable binding affinity prediction,”ArXiv, vol. 2308.09639, 2023

    work page arXiv 2023

  7. [7]

    Development and evaluation of a deep learning model for protein– ligand binding affinity prediction,

    M. M. Stepniewska-Dziubinska, P. Zielenkiewicz, and P. Siedlecki, “Development and evaluation of a deep learning model for protein– ligand binding affinity prediction,”Bioinformatics, vol. 34, no. 21, pp. 3666–3674, 2018

    2018

  8. [8]

    DEAttentionDTA: protein–ligand binding affinity prediction based on dynamic embedding and self-attention,

    X. Chen, J. Huang, T. Shen, H. Zhang, L. Xu, M. Yang, X. Xie, Y . Yan, and J. Yan, “DEAttentionDTA: protein–ligand binding affinity prediction based on dynamic embedding and self-attention,”Bioinformatics, vol. 40, no. 6, p. btae319, 2024

    2024

  9. [9]

    CAPLA: improved prediction of protein–ligand binding affinity by a deep learning approach based on a cross-attention mechanism,

    Z. Jin, T. Wu, T. Chen, D. Pan, X. Wang, J. Xie, L. Quan, and Q. Lyu, “CAPLA: improved prediction of protein–ligand binding affinity by a deep learning approach based on a cross-attention mechanism,”Bioinformatics, vol. 39, no. 2, p. btad049, 2023

    2023

  10. [10]

    Uni-Mol: A universal 3d molecular representation learning framework,

    G. Zhou, Z. Gao, Q. Ding, H. Zheng, H. Xu, Z. Wei, L. Zhang, and G. Ke, “Uni-Mol: A universal 3d molecular representation learning framework,” inICLR, 2023

    2023

  11. [11]

    Large-scale chemical language representations capture molecular structure and properties,

    J. Ross, B. Belgodere, V . Chenthamarakshan, I. Padhi, Y . Mroueh, and P. Das, “Large-scale chemical language representations capture molecular structure and properties,” inNature Machine Intelligence, 2022

    2022

  12. [12]

    The PDBbind database: methodologies and updates,

    R. Wang, X. Fang, Y . Lu, C.-Y . Yang, and S. Wang, “The PDBbind database: methodologies and updates,”J. Med. Chem., vol. 48, no. 12, pp. 4111–4119, 2005

    2005

  13. [13]

    Comparative assessment of scoring functions: the CASF-2016 update,

    M. Su, Q. Yang, Y . Du, G. Feng, Z. Liu, Y . Li, and R. Wang, “Comparative assessment of scoring functions: the CASF-2016 update,”J. Chem. Inf. Model., vol. 59, no. 2, pp. 895–913, 2019

    2016

  14. [14]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” inNeurIPS, 2017

    2017

  15. [15]

    Decoupled weight decay regularization,

    I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” ICLR, 2019

    2019

  16. [16]

    SGDR: Stochastic gradient descent with warm restarts,

    ——, “SGDR: Stochastic gradient descent with warm restarts,” inICLR, 2017

    2017

  17. [17]

    Biopython: freely available Python tools for computational molecular biology and bioinformatics,

    P. J. A. Cock, T. Antao, J. T. Chang, B. A. Chapman, C. J. Cox, A. Dalke, I. Friedberg, T. Hamelryck, F. Kauff, B. Wilczynski, and M. J. L. de Hoon, “Biopython: freely available Python tools for computational molecular biology and bioinformatics,”Bioinformatics, vol. 25, no. 11, pp. 1422– 1423, 2009

    2009

  18. [18]

    Dictionary of protein secondary structure: pattern recognition of hydrogen-bonded and geometrical features,

    W. Kabsch and C. Sander, “Dictionary of protein secondary structure: pattern recognition of hydrogen-bonded and geometrical features,” Biopolymers, vol. 22, no. 12, pp. 2577–2637, 1983

    1983

  19. [19]

    RDKit: Open-source cheminformatics,

    G. Landrumet al., “RDKit: Open-source cheminformatics,”https://www. rdkit.org, 2024

    2024