{"id":"abd35cfa-3250-425b-8dc8-48bad4e62fa6","arxiv_id":"2504.16261","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"IPBind, a frame-averaged graph neural network, predicts binding affinity as a sum of atomic energy differences and claims state-of-the-art accuracy on the low-sequence-identity LBA30 and LBA60 benchmarks.","lead":"This paper introduces IPBind, a geometric deep learning model that predicts protein-ligand binding affinity by summing learned per-atom energy differences between bound and unbound states. The authors report new state-of-the-art results on generalization benchmarks, which matters for drug discovery against novel protein targets.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA generalization claim relies on LBA60/LBA30 test sets being disjoint from IPBind's PDBbind2020 training set, but the paper never verifies this and the sets share the same source; if contaminated, the reported improvements are in-sample.","rationale":"The reader's weakest assumption matches my primary concern: the LBA60/LBA30 splits must be disjoint from IPBind's training set and baselines must be evaluated identically. I find this to be the most load-bearing issue because the paper's entire contribution is the empirical generalization result; without a valid split, there is no result. The paper gives no evidence of filtering the PDBbind2020 refined set against the Atom3D test sets, even though both derive from PDBbind. I also note the zero standard deviations for IPBind in Table II as a secondary reproducibility red flag, but the split overlap is the decisive question. I agree with the reader's conditional verdict: the paper should not be accepted until the split audit and protocol details are provided. If the audit fails, the verdict should move to REJECT; if it passes, the empirical claim would be substantially supported. The independent support in the paper is limited: no code, no data, no formal verification. The frame-averaging equations (3)-(4) are also mathematically questionable, but that affects the method's theoretical novelty rather than the central generalization claim.","tokens_in":10624,"tokens_out":7559,"duration_ms":67407,"concrete_test":"Download the Atom3D LBA60/LBA30 test set definitions (from [36]) and the list of PDB IDs used for IPBind training (PDBbind2020 refined set minus the 285 CASF2016 complexes). Compute the intersection of PDB IDs between the training list and each test list. Then run MMseqs2 (or BLAST) to compute the maximum sequence identity between every protein chain in the training set and every chain in each test set. If any test complex has an identical PDB ID in training, or any protein chain shares >30% sequence identity with a training chain, the split is contaminated. Additionally, for the five self-run baselines, verify that all methods used the identical training/validation/test split and the same affinity labels; for the adopted baselines, confirm that their source protocol matches. This audit would settle whether the claimed SOTA improvements are valid.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Section III-C, Table II) is that IPBind establishes state-of-the-art generalization on the Atom3D LBA60/LBA30 benchmarks, with Pearson correlations of 0.843 and 0.732. For this to be true, the evaluation complexes must be disjoint from the 5,050 complexes used for training (Section III-A). The paper states only that the LBA splits were 'adopted from Atom3D [36]' and does not report any filtering of the PDBbind2020 refined training set against these test sets. Because the Atom3D LBA benchmark is itself constructed from the PDBbind refined set, and because IPBind trains on the PDBbind2020 refined set, the same complexes (or proteins with >30% sequence identity) are very likely present in both training and test. If so, the reported 19.6% (LBA30) and 8.9% (LBA60) improvements over prior methods measure memorization, not generalization. The comparison is further clouded by the fact that only five baselines in Table II were re-run by the authors; the remaining numbers are 'adopted from [20]', which may involve different training data or protocols. The load-bearing assumption is the disjointness of splits and protocol equivalence; if it is false, the headline claim collapses.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes IPBind, a geometric deep learning method for protein-ligand binding affinity prediction. The model constructs three atom-level graphs (bound complex, unbound protein pocket, unbound ligand), applies frame averaging in the preprocessing module, and predicts affinity as the sum of per-atom energy differences between bound and unbound states. The authors report state-of-the-art results on CASF2016 and on the Atom3D LBA60/LBA30 low-sequence-identity benchmarks, with claims of improved generalization to dissimilar proteins, robustness to predicted structures, interpretability, and training efficiency. The paper includes ablations on symmetry group choice, graph construction, and network architecture, as well as visualizations of atomic contributions.","tokens_in":10925,"tokens_out":5961,"duration_ms":57294,"significance":"If the central claims are correct, the LBA30 result (Pearson 0.732, a 19.6% relative improvement over prior methods) would be a meaningful advance in generalization for structure-based affinity prediction, and the bound/unbound potential difference is an interesting inductive bias. The paper also provides useful empirical insights, including an ablation showing that symmetry treatment and the three-graph decomposition matter for performance. The claimed robustness to predicted structures is practically important. However, the validity of the headline generalization claims depends on the LBA splits being disjoint from the PDBbind2020 training set, a point the manuscript does not verify, and on the correctness of the symmetry labeling. The empirical evidence is otherwise plausible but not fully verifiable because several key numbers are reported with zero standard deviation and robustness results appear only in figures.","major_comments":[{"comment":"The LBA60 and LBA30 test splits are adopted from Atom3D [36], but the paper does not demonstrate that these test complexes are disjoint from the 5,050 PDBbind2020 refined-set complexes used for training IPBind. Because both the Atom3D LBA benchmark and the PDBbind2020 refined set are derived from PDBbind, the same complexes or proteins with >30% sequence identity could appear in both training and test. If so, the reported Pearson correlations of 0.843 (LBA60) and 0.732 (LBA30) and the stated 8.9% and 19.6% improvements over baselines would measure memorization rather than generalization. Please report the maximum sequence identity between IPBind's training set and each LBA test set (and for the baselines' training sets), or retrain IPBind on the original Atom3D training split and re-evaluate.","section":"Section III-A and III-C, Table II"},{"comment":"The frame set F_SE(3)(X) defined in Eq. (3) includes matrices with determinant -1, e.g., α=-1, β=1 gives columns [-u1, u2, -u3] with determinant -1. Averaging over this set produces output invariant to reflections, i.e., E(3)-invariant, not SE(3)-invariant. The claim that IPBind is 'SE(3)-invariant' and that this 'respects biochemical reality in chirality' is therefore not supported by the stated construction. This also confounds the interpretation of the ablation in Table III, where 'IPBind SE(3)' is compared with 'IPBind E(3)'. Please either correct the frame construction to use only proper rotations (e.g., choose frames with det=+1) or revise the symmetry claims and the discussion of chirality.","section":"Section II-B, Eq. (3)"},{"comment":"Several IPBind results are reported with zero standard deviation: 0.732±0.000 (LBA30 Pearson), 1.119±0.000 (LBA60 RMSE), and 0.843±0.000 (LBA60 Pearson). With five independent runs, an exact zero standard deviation is implausible and likely results from rounding to too few significant digits or an aggregation error. The standard deviations are needed to assess the stability of the state-of-the-art claims, particularly because the improvements over EHIGN on LBA60 are about 8.9% in Pearson. Please report means and standard deviations with sufficient precision (e.g., 0.843±0.003) and verify the computation.","section":"Table II"},{"comment":"The robustness experiments with redocked and co-folded structures are presented only as figures, without a numerical table. The text claims that IPBind 'uniquely matching or exceeding PSICHIC across conditions' and that GIGN/EHIGN/Tankbind suffer 'sharp performance drops' under co-folded structures, but the exact values are not given. Since this is a key practical claim, please provide a table reporting mean and standard deviation for all models under each input-structure condition, so readers can verify the comparisons.","section":"Section III-D, Figure 3"},{"comment":"The comparison with baselines is not fully controlled: only five baselines (OnionNet-2, PSICHIC, Tankbind, GIGN, EHIGN) were re-run by the authors, while the remaining numbers are 'adopted from [20]'. If those baselines were trained on a different training set or with different splits, the relative improvements may be partly due to protocol differences. Please clarify the training data and evaluation protocol for the adopted baselines, or re-run them under IPBind's training conditions.","section":"Section III-C, Table II"}],"minor_comments":[{"comment":"There are several typographical errors: 'Interatmoic Potential' in the Introduction, 'provids' in the Abstract, 'colding structures' and 'unbder' in Figure 3, and 'massage passing' in Section II-A. These should be corrected.","section":"Throughout"},{"comment":"In the frame-averaging equation, φ takes two arguments: ((X - 1t^T)R, X). The role of the second argument X is unclear; please clarify whether it is the original coordinates used for edge construction or something else.","section":"Eq. (4)"},{"comment":"The text says 'we first cropped the protein pocket to concentrate on the most critical interaction sites, which was defined as maximum to the closest 50 residues to ligand's heavy atoms.' The wording 'maximum to' is awkward; please rephrase for clarity.","section":"Section II-A"},{"comment":"The total loss in Eq. (10) sums LBME and Lrank without any weighting coefficients. Please state explicitly whether equal weighting was used and whether any sensitivity analysis was performed.","section":"Section II-E"},{"comment":"Reference [40] appears as 'C. D. team, J. Boitreaud, ...' — the author name 'C. D. team' should be the official consortium name (e.g., Chai Discovery team). Please check the citation format.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The primary risk is the train/test overlap between IPBind's PDBbind2020 refined-set training data and the Atom3D LBA splits. This should be resolved before the paper is accepted, as the entire generalization claim rests on it. The SE(3) mislabeling is also concerning because it is a stated contribution and because the ablation in Table III is presented as evidence for the importance of chirality-aware modeling. I would ask the authors to provide explicit sequence-identity verification and to re-evaluate the symmetry claims with corrected terminology."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"IPBind is a geometric GNN that predicts binding affinity by summing bound/unbound atomic energy differences, built on FAENet-style frame averaging. The architecture is a reasonable combination of known pieces, and some parts are genuinely well done: the ablation study is thorough, the robustness tests with redocked and co-folded structures are a useful addition, and the writing is mostly clear.\n\nThe problems sit exactly where the paper's value claims sit. First, the frame averaging in Eq. (3) is mislabeled. The four frames generated by flipping the signs of u1 and u2 include reflections as well as rotations — for each sign flip, the determinant becomes -1. Averaging over reflections gives E(3)-invariance, not SE(3)-invariance. So the paper's chirality-aware modeling claim, and the ablation against an 'E(3) IPBind', do not support what they appear to. Also, the set of frames is not a group, so the frame-averaging theory from Puny et al. does not directly apply.\n\nSecond, the LBA60/LBA30 results are very likely contaminated. The authors train on the PDBbind2020 refined set after removing the CASF-2016 test complexes (5050 complexes). The Atom3D LBA splits are built from the same PDBbind refined set. The paper never states that LBA test complexes were excluded from the 5050 training complexes, and it reports no sequence-identity filtering between training and these test sets. Given the 30% and 60% thresholds, overlap is almost certain. If that is true, the claimed 19.6% and 8.9% improvements are partly in-sample, and the paper's central generalization claim collapses. The authors need to show explicit split verification, and ideally retrain on the LBA training splits.\n\nSmaller issues: several standard deviations are printed as 0.000, which is implausible across five seeds; no code or data is provided. Baselines adopted from [20] may have used different protocols, though the five re-run baselines are reassuring.\n\nI would send this to peer review, because the questions are important and addressable. But I would ask the authors to make the training/test splits explicit, correct the symmetry analysis, and release code and data. As written, the headline results are not trustworthy.","headline":"Useful architecture, but the headline generalization results are probably contaminated and the SE(3) claim is wrong as written.","tokens_in":11400,"tokens_out":4523,"would_cite":false,"duration_ms":41846,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"IPBind predicts protein-ligand binding affinity as the summed per-atom energy difference between bound and unbound states, and reports state-of-the-art generalization on proteins with low sequence identity to the training set.","keywords":["protein-ligand binding affinity","geometric deep learning","interatomic potential","SE(3) invariance","frame averaging","generalization to unseen proteins","drug discovery"],"falsifier":"Re-run EHIGN, GIGN, and IPBind from identical code, splits, and seeds on the same 30% and 60% sequence-identity partitions; if the baseline Pearson values shift materially or complexes overlap the training set at the stated thresholds, the claimed 19.6% and 8.9% improvements are not established. A second check is to feed IPBind co-folded structures generated by a different tool than the one used in the paper; if the Pearson advantage over sequence-based baselines disappears, the robustness claim is specific to the chosen structure generator.","tokens_in":10456,"feed_emoji":"🧬","tokens_out":7461,"duration_ms":66110,"temperature":0.7,"pith_summary":"The paper proposes IPBind, a geometric deep learning model that predicts how tightly a small molecule binds a protein by summing per-atom energy differences between the protein and ligand alone and the protein–ligand complex. The central aim is to make affinity prediction generalize to proteins that share little sequence identity with the training data, where existing structure-based models degrade sharply. On the hardest 30% sequence-identity benchmark, the paper reports a Pearson correlation of 0.732, a 19.6% relative improvement over the previous best method, and reports that the model keeps its advantage when input structures are predicted by docking or co-folding rather than crystal structures. If these results hold, IPBind would give drug-discovery pipelines an affinity scorer that stays reliable for novel targets and imperfect structures.","feed_headline":"IPBind beats prior affinity models by 19.6% on hardest splits","feed_subtitle":"By scoring per-atom energy differences between bound and unbound states, it keeps working on co-folded structures too.","key_machinery":"The load-bearing mechanism is the energy-difference decomposition with shared atom-level encoders. The input complex, protein pocket, and ligand are each built as atom graphs with a 5 Å radius cutoff; a common message-passing encoder produces per-atom embeddings, and an output head predicts each atom's energy contribution in the bound and unbound graphs. Binding affinity is the summed difference. Frame averaging over the four signed eigenbases of the coordinate covariance matrix enforces $SE(3)$ invariance, which the ablation table credits with up to 7.6% higher Pearson correlation than $E(3)$ invariance at 30% identity. The loss combines Balanced MSE with a ranking term, and training reaches the main result in about 14 minutes.","core_discovery":"IPBind treats binding affinity as a machine-learned interatomic potential: each atom is assigned an energy contribution, and the predicted affinity is the difference between the summed atomic energies of the unbound protein plus ligand and of the bound complex. The model is made invariant to rotations and translations but not reflections, using frame averaging over signed eigenbases, so molecular chirality is respected. Trained on the PDBbind2020 refined set, it reports Pearson correlations of 0.843 at 60% sequence identity and 0.732 at 30%, surpassing listed baselines by at least 8.9% and 19.6% respectively, and remains competitive when input structures come from redocking or co-folding. The paper also shows atom-level contribution maps that align with known interaction profiler outputs, suggesting the energy-difference decomposition gives interpretable predictions.","pith_inferences":["Ablating the 50-residue pocket crop and the 5 Å edge cutoff on the 30% identity benchmark would separate the contribution of the energy-difference idea from the preprocessing choices; the paper does not report such ablations.","Because the model predicts per-atom energies, one can extend it to predict enthalpy-entropy components or to output an uncertainty over the summed affinity; neither is tested here.","The chirality claim could be probed directly with enantiomeric ligand pairs: if the $SE(3)$ model does not rank mirror-image ligands correctly, the observed gap between $SE(3)$ and $E(3)$ may come from something other than chirality-awareness.","The ranking loss suggests IPBind may shine more in virtual screening, where ranking candidates matters, than in absolute affinity estimation; evaluating with ranking metrics like enrichment would test that."],"forward_implications":["If the reported results are correct, drug-discovery pipelines can score candidate ligands against novel targets without retraining, since the model's advantage is largest exactly where sequence similarity to training data is lowest.","Predicted structures from redocking or co-folding can be used as input without the sharp performance drop seen for most structure-based baselines, which matters because crystal structures are rarely available early in discovery.","The atom-level contribution maps let a researcher see which interactions the model weights, so failures can be traced to specific missing interaction types such as water bridges.","The use of $SE(3)$ rather than $E(3)$ symmetry measurably improves accuracy, supporting the view that chirality-aware modeling matters for affinity.","Training to competitive accuracy in about 14 minutes with only atomic numbers and coordinates as input means the method can be embedded in iterative screening loops at low compute cost."],"supporting_citations":[{"why":"Supplies the LBA60 and LBA30 sequence-identity splits used for the generalization benchmarks.","marker":"[36]"},{"why":"Provides the frame averaging method that gives the SE(3)-invariant output.","marker":"[27]"},{"why":"Provides the message-passing layer reused for atom interaction updates.","marker":"[29]"},{"why":"Defines the CASF-2016 benchmark.","marker":"[34]"},{"why":"Provides the Balanced MSE loss used for skewed label distributions.","marker":"[30]"},{"why":"Provides the approximate NDCG ranking loss.","marker":"[31]"},{"why":"Is the E(3)-invariant GIGN baseline used for the symmetry comparison.","marker":"[22]"},{"why":"Is the source of most baseline numbers in the low-sequence-identity comparison table.","marker":"[20]"}],"fun_headline_variants":["IPBind predicts affinity via per-atom energies, beating prior models by 19.6%","Geometric deep learning uses bound-unbound energy gaps for generalizable affinity","IPBind's interatomic potential gives interpretable, chiral-aware affinity scores","Predicting binding affinity: IPBind leverages unbound-to-bound energy differences","IPBind: machine-learned potentials improve affinity prediction on novel proteins"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim rests on the assumption that the low-sequence-identity test splits are truly disjoint from the training complexes at the stated identity thresholds and that the baseline numbers were obtained under the same training, validation, and evaluation protocol, so the reported relative improvements are fair.","fun_headline_variants_meta":{"raw":{"variants":["IPBind predicts affinity via per-atom energies, beating prior models by 19.6%","Geometric deep learning uses bound-unbound energy gaps for generalizable affinity","IPBind's interatomic potential gives interpretable, chiral-aware affinity scores","Predicting binding affinity: IPBind leverages unbound-to-bound energy differences","IPBind: machine-learned potentials improve affinity prediction on novel proteins"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000266,"raw_usage":{"total_tokens":1568,"prompt_tokens":857,"completion_tokens":711,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":473,"completion_tokens_details":{"reasoning_tokens":611}},"tokens_in":473,"tokens_out":711,"duration_ms":6664,"temperature":1.0,"reasoning_tokens":611,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:08:33.536690+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run EHIGN, GIGN, and IPBind from identical code, splits, and seeds on the same 30% and 60% sequence-identity partitions; if the baseline Pearson values shift materially or complexes overlap the training set at the stated thresholds, the claimed 19.6% and 8.9% improvements are not established. A second check is to feed IPBind co-folded structures generated by a different tool than the one used in the paper; if the Pearson advantage over sequence-based baselines disappears, the robustness claim is specific to the chosen structure generator.","supporting_citations":[{"cited_title":"Faenet: Frame averaging equivariant gnn for materials modeling,","cited_arxiv_id":null,"evidence_quote":"Provides the message-passing layer reused for atom interaction updates."},{"cited_title":"Comparative assessment of scoring functions: the casf-2016 update,","cited_arxiv_id":null,"evidence_quote":"Defines the CASF-2016 benchmark."},{"cited_title":"Balanced mse for imbalanced visual regression,","cited_arxiv_id":null,"evidence_quote":"Provides the Balanced MSE loss used for skewed label distributions."},{"cited_title":"Learning to rank by optimizing ndcg measure,","cited_arxiv_id":null,"evidence_quote":"Provides the approximate NDCG ranking loss."},{"cited_title":"Geometric interaction graph neural network for predicting protein–ligand binding affinities from 3d structures (gign),","cited_arxiv_id":null,"evidence_quote":"Is the E(3)-invariant GIGN baseline used for the symmetry comparison."},{"cited_title":"ProFSA: Self-supervised Pocket Pretraining via Protein Fragment-Surroundings Alignment","cited_arxiv_id":"2310.07229","evidence_quote":"Is the source of most baseline numbers in the low-sequence-identity comparison table."}],"review_version":1}