REVIEW 4 major objections 6 minor 26 references
Deep-Learning Based Docking Methods: Fair Comparisons to Conventional Docking Workflows
T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read DiffDock's reported docking superiority is largely an artifact of near-neighbor memorization from its training set, and conventional docking workflows outperform it when run properly.
desk verdict A useful, reproducible re-benchmark that puts DiffDock's reported advantage in doubt, but the 'table-lookup' claim overreaches because the near-neighbor split is never run on conventional baselines. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The argument is carried by a contamination diagnostic that splits the test set by near-neighbor status. A training case counts as a near neighbor when the 2D ligand topological similarity Gsim is in the top 1% or ≥0.3 and the binding-pocket similarity PSIM is ≥0.65, with Gsim ≥0.80 defining an extreme subset. The complementary machinery is a fully automatic conventional docking pipeline, including automated protein and ligand preparation, protomol-based site definition, docking, pose-family clustering, and automorph-corrected RMSD calculation, applied to a Clean Test Set of 290 complexes curated from DiffDock's 363-case test set to ensure the conventional baselines use the methods as intended.
What would settle it
Delete from DiffDock's training set all complexes that meet the near-neighbor criteria for the 290 clean test cases, retrain, and re-dock; if success on the previously 'hard' cases rises from about 21% toward the 57% seen on near-neighbor cases, the table-lookup claim is refuted, whereas if it stays near 21%, the claim is confirmed.
Extended reading notes
Core claim
The paper's central claim is that DiffDock's reported performance cannot be taken as evidence that deep learning solves molecular docking. Recomputing DiffDock's own benchmark with a fully automatic Surflex-Dock workflow shows conventional docking is far better at redocking cognate ligands when the binding site is known (Top-1/Top-5 success 68/81% vs. 45/51% at 2.0 Å RMSD) and no worse when the site is unknown. The decisive new result is a diagnostic: 191 of the 290 clean test complexes have a near-neighbor training case, defined by high topological ligand similarity (Gsim) and high binding-pocket similarity (PSIM), and DiffDock's success is dichotomized by that label, 57/65% on near-neighbor cases versus 21/28% on the 99 hard cases, with extreme near-neighbors above 90%. The paper therefore states that DiffDock has apparently encoded a type of table-lookup and that its comparisons to other methods used those methods in a nonstandard, blinded way that handicapped them.
Load-bearing premise
The memorization conclusion rests on the chosen similarity thresholds (Gsim top 1% or ≥0.3 and PSIM ≥0.65) that split the test set into near-neighbor and hard cases; if those cutoffs do not validly capture 'near-identical', the 40-point gap and the table-lookup reading could be overstated.
Editorial extensions
If this is right
- DiffDock's apparent gains over conventional docking do not survive a properly run comparison: conventional tools win by 20–30 points at the 2.0 Å threshold in the known-site condition.
- Benchmark claims for learning-based docking methods should be audited for near-neighbor contamination before being read as evidence of generalization.
- Temporal splits of 98% training / 2% testing are too skewed to prevent memorization from dominating results; more balanced splits (the paper suggests 25/75) would better reveal true capabilities.
- The roughly 40-point gap between near-neighbor and hard cases means predictions for targets and ligands without near-identical precedents are the real challenge, and on those DiffDock is weak.
- When binding sites are unknown, a conventional automated workflow with pocket detection remains competitive or better, so blind-docking comparisons should be run with the conventional method's own site-finding tools.
Reading between the lines
- The Gsim/PSIM contamination audit could be adopted as a standard reporting metric for any machine-learned docking or structure-prediction model, making memorization effects visible without additional experiments.
- Because the test set is drawn from PDBBind complexes released after 2019, other deep-learning methods trained on the same pre-2019 PDBBind data may show the same split; re-running this audit on them is a direct next test.
- The paper's addendum notes that DiffDock-L and a new DockGen benchmark appeared after its analyses; applying this same near-neighbor audit to DiffDock-L would reveal whether its claimed generalization to new pockets is real or a similar artifact.
- If this finding is correct, reported state-of-the-art docking accuracy in the deep-learning literature may be substantially overstated, and progress should be judged on truly novel-scaffold and novel-pocket test sets.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper benchmarks four conventional docking methods (Surflex-Dock, Glide, AutoDock Vina, Gnina) against the deep-learning method DiffDock on the PDBBind 2020 temporal split, using a cleaned 290-case subset of DiffDock's 363-case test set. The authors report that, with a known binding site, Surflex-Dock and Glide succeed at 2.0 Å RMSD at rates of 68/81% and 67/73% (Top-1/Top-5), compared with DiffDock's 45/51%, and that Vina and Gnina also outperform DiffDock. In the blind-docking condition, Surflex-Dock shows a smaller advantage. The paper's second main claim is that DiffDock's success is concentrated in the 191/290 test cases that have near-neighbor protein-ligand complexes in the training set, with success rates of 57/65% on those versus 21/28% on the 99 remaining 'hard' cases, leading the authors to conclude that DiffDock has 'apparently encoded a type of table-lookup' and that its reported results are artifactual.
Significance. This is a timely and potentially useful benchmark contribution. The authors provide fully automatic, scripted workflows for four conventional docking methods on a cleaned version of the DiffDock test set, together with a public data archive, which is a valuable reproducibility resource for the docking community. The known-site results for conventional methods, if properly compared, would be an important reference point. The near-neighbor analysis raises a legitimate concern about temporal-split benchmarking at an extreme 98/2 ratio. However, the central memorization conclusion is not yet supported because the analysis lacks a critical control, and the known-site comparison is asymmetric. If the missing analyses are provided, the paper could become a significant corrective to optimistic assessments of deep-learning docking; in its current form, the evidence is incomplete.
major comments (4)
- [§2.2.1, Figure 2 (left)] The 'known binding site' comparison is asymmetric. The DiffDock results shown are those from the original DiffDock report, which used a blind docking protocol, as the paper itself notes in §2.2 for the corresponding Glide baseline. Surflex-Dock, Glide, Vina, and Gnina, by contrast, are given the binding site defined by the cognate ligand. The paper never reports DiffDock's performance when it also receives the pocket as input. Consequently, the headline gap of 68/81% versus 45/51% conflates method quality with the amount of information provided. To support the claim that conventional methods 'far exceed' DiffDock in the known-site condition, the authors should either run DiffDock in its pocket-conditioned mode or explicitly frame the comparison as 'conventional docking with known site versus DiffDock blind' and temper the wording accordingly.
- [§2.3, Figure 7] The conclusion that DiffDock's success is 'inextricably linked' to near-neighbor training cases and that it 'has apparently encoded a type of table-lookup' is not supported because no conventional docking method is evaluated on the same near-neighbor/hard split. The 57/65% versus 21/28% success rates for DiffDock are equally consistent with the simple explanation that near-neighbor cases are easier docking problems for any method. The overall Clean Test Set success rates reported for Surflex-Dock and Glide do not serve as the needed control, since they mix the two populations. The paper should report the same split for Surflex-Dock, Glide, Vina, and Gnina, all of which have already been run on the Clean Test Set, and test whether the performance gap between the near-neighbor and hard subsets is significantly larger for DiffDock than for conventional methods.
- [§2.3 and Appendix 'Finding Near-Neighbor Training Cases'] The classification into 191 near-neighbor and 99 hard cases depends on ad hoc similarity thresholds: Gsim in the top 1% or ≥0.3, followed by PSIM ≥0.65, with 'extreme' cases defined by Gsim ≥0.80. These thresholds are not validated against an external benchmark, and no sensitivity analysis is provided. Since the roughly 40-percentage-point performance gap is the central evidence for the memorization claim, the authors should vary the thresholds (for example, PSIM from 0.5 to 0.8 and alternative Gsim cutoffs) and show that the easy/hard partition and the performance gap are stable, or report how the conclusions change under reasonable alternative definitions.
- [§2.2.1, unknown binding site] The claim that Surflex-Dock is statistically superior to DiffDock in blind docking is based on a paired t-test computed on a post-hoc subset of 160/290 cases where both methods achieved Top-5 RMSD ≤4.0 Å. Selecting the subset based on outcomes biases the test and invalidates the p-values as a statement about the full test set. The authors should provide a pre-specified analysis on all 290 cases, for example by treating non-converged dockings as 20 Å or using a non-parametric test on ranked RMSDs, or should explicitly limit the statistical claim to the 160-case subset and present the full-set comparison descriptively.
minor comments (6)
- [§2.1, Figure 1 caption] The paper should state explicitly whether the DiffDock RMSD values shown for the Clean Test Set were recomputed by the authors or provided by the DiffDock team, and confirm that the same RMSD definition was used for DiffDock and for the conventional methods.
- [§2.2.1] The paired t-test is applied to binary success indicators at a single RMSD threshold; McNemar's test or a bootstrap of the paired difference would be more appropriate for these binary outcomes, even though the large gaps likely remain significant.
- [§2.2.1] The statement that the Surflex-Dock approach 'made no use of prior information from pre-2019 structures' is overstated, since the Surflex-Dock scoring function is empirically parameterized on protein-ligand complexes. The more defensible claim is that no specific pre-2019 co-crystal structures near to the test cases were used for binding-site identification or pose selection.
- [§2.3, Appendix] The exclusion of identical training ligands from the near-neighbor definition is not explained. If any test complex has a fully identical protein-ligand training complex, that would be the clearest possible memorization case, and its handling should be stated explicitly.
- [§4, Conclusions] The recommendation that a 25/75% temporal split is 'more reasonable' is presented as a general benchmarking guideline without supporting evidence beyond a citation to a prior paper; it would benefit from being framed as an opinion or from being supported by additional analysis.
- [§5, Addendum] The Addendum notes the later release of DockGen and DiffDock-L, but the main text does not discuss whether those developments affect the near-neighbor criticism. A brief discussion of whether the same issue persists in DiffDock-L would strengthen the paper's relevance.
Circularity Check
No significant circularity found: the benchmark results are new measurements against external methods, and the near-neighbor analysis is an empirical partition rather than a self-fulfilling construction.
full rationale
The paper's central comparisons are empirical benchmarks: Surflex-Dock, Glide, Vina, and Gnina are run on the Clean Test Set and their RMSD-based success rates are compared directly with DiffDock's reported results. These measurements do not depend on any fitted parameter or on the authors' prior conclusions, so the claim that conventional docking outperforms DiffDock is self-contained and externally checkable. The near-neighbor analysis partitions the 290 test cases into 191 near-neighbor and 99 hard cases using ligand and pocket similarity thresholds (Gsim and PSIM), and then reports DiffDock success rates of 57/65% versus 21/28% within those groups. This is an observational correlation between an independently defined partition and DiffDock's outputs; near-neighbor status is not defined by DiffDock success, nor is DiffDock success computed from the similarity scores. The 'table-lookup' conclusion is an interpretation of that correlation, not a result forced by construction. Concerns about ad hoc thresholds and the absence of a conventional-docking control on the same hard/easy split are validity or generalization issues, not circularity. The paper does cite the authors' own Surflex-Dock and similarity methods, but those citations are method attributions for tools used to generate new results, not load-bearing logical premises that presuppose the paper's conclusions. No fitted input is renamed as a prediction, and no uniqueness theorem from the authors' prior work is invoked. Overall, the derivation chain does not reduce to its inputs.
Assumptions & free parameters
free parameters (4)
- near-neighbor Gsim threshold =
>=0.3 or top 1% (about 160 compounds)
- near-neighbor PSIM threshold =
>=0.65
- extreme near-neighbor Gsim threshold =
>=0.80
- unknown-site subset criterion =
top-5 RMSD <= 4.0 Å for both methods (160/290)
assumptions (4)
- domain assumption The 290-complex Clean Test Set is representative of DiffDock's 363-case test set for comparing methods.
- domain assumption Automorph-corrected RMSD computed by the BioPharmics tools is directly comparable to the RMSD values reported by DiffDock.
- domain assumption Performance data for DiffDock, provided directly by the original authors, correspond exactly to the published figures.
- domain assumption High ligand similarity and binding-site similarity to a training case is the mechanism behind DiffDock's success on those test cases.
Cite this review
Pith. "Pith review of Deep-Learning Based Docking Methods: Fair Comparisons to Conventional Docking Workflows." pith.science (2026). https://pith.science/paper/H5Y56HY6
@misc{pith2026241202889,
author = {Pith},
title = {Pith review of: Deep-Learning Based Docking Methods: Fair Comparisons to Conventional Docking Workflows},
year = {2026},
howpublished = {\url{https://pith.science/paper/H5Y56HY6}},
note = {Machine review of arXiv:2412.02889}
}
read the original abstract
The diffusion learning method, DiffDock, for docking small-molecule ligands into protein binding sites was recently introduced. Results included comparisons to more conventional docking approaches, with DiffDock showing superior performance. Here, we employ a fully automatic workflow using the Surflex-Dock methods to generate a fair baseline for conventional docking approaches. Results were generated for the common and expected situation where a binding site location is known and also for the condition of an unknown binding site. For the known binding site condition, Surflex-Dock success rates at 2.0 Angstroms RMSD far exceeded those for DiffDock (Top-1/Top-5 success rates, respectively, were 68/81% compared with 45/51%). Glide performed with similar success rates (67/73%) to Surflex-Dock for the known binding site condition, and results for AutoDock Vina and Gnina followed this pattern. For the unknown binding site condition, using an automated method to identify multiple binding pockets, Surflex-Dock success rates again exceeded those of DiffDock, but by a somewhat lesser margin. DiffDock made use of roughly 17,000 co-crystal structures for learning (98% of PDBBind version 2020, pre-2019 structures) for a training set in order to predict on 363 test cases (2% of PDBBind 2020) from 2019 forward. DiffDock's performance was inextricably linked with the presence of near-neighbor cases of close to identical protein-ligand complexes in the training set for over half of the test set cases. DiffDock exhibited a 40 percentage point difference on near-neighbor cases (two-thirds of all test cases) compared with cases with no near-neighbor training case. DiffDock has apparently encoded a type of table-lookup during its learning process, rendering meaningful applications beyond its reach. Further, it does not perform even close to competitively with a competently run modern docking workflow.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Diffdock: Diffusion steps, twists, and turns for molecular docking
Gabriele Corso, Hannes Stärk, Bowen Jing, Regina Barzilay, and Tommi Jaakkola. Diffdock: Diffusion steps, twists, and turns for molecular docking. arXiv preprint arXiv:2210.01776, 2022
arXiv 2022
-
[2]
Martin Buttenschoen, Garrett M Morris, and Charlotte M Deane. Posebusters: Ai-based docking methods fail to generate physically valid poses or generalise to novel sequences. arXiv preprint arXiv:2308.05777, 2023
work page Pith review arXiv 2023
-
[3]
Ann E Cleves and Ajay N Jain. Knowledge-guided docking: Accurate prospective prediction of bound config- urations of novel ligands using Surflex-Dock. Journal of Computer-Aided Molecular Design , 29(6):485–509, 2015
work page 2015
-
[4]
Surflex-Dock: Docking benchmarks and real-world application
Russell Spitzer and Ajay N Jain. Surflex-Dock: Docking benchmarks and real-world application. Journal of Computer-Aided Molecular Design, 26(6):687–699, 2012
work page 2012
-
[5]
Ajay N. Jain. Surflex: Fully automatic flexible molecular docking using a molecular similarity-based search engine. Journal of Medicinal Chemistry, 46(4):499–511, 2003
work page 2003
-
[6]
Ajay N. Jain. Surflex-Dock 2.1: Robust performance from ligand energetic modeling, ring flexibility, and knowledge-based search. Journal of Computer-Aided Molecular Design , 21:281–306, 2007. ISSN 0920-654X
work page 2007
-
[7]
Ajay N Jain. Effects of protein conformation in docking: Improved pose prediction through protein pocket adaptation. Journal of Computer-Aided Molecular Design , 23(6):355–374, 2009
work page 2009
-
[8]
R. A. Friesner, J. L. Banks, R. B. Murphy, T. A. Halgren, J. J. Klicic, D. T. Mainz, M. P. Repasky, E. H. Knoll, M. Shelley, J. K. Perry, D. E. Shaw, P. Francis, and P. S. Shenkin. Glide: A new approach for rapid, accurate docking and scoring. 1. method and assessment of docking accuracy. J Med Chem, 47(7):1739–1749, 2004
work page 2004
Show all 26 references
-
[9]
Glide: A new approach for rapid, accurate docking and scoring
Thomas A Halgren, Robert B Murphy, Richard A Friesner, Hege S Beard, Leah L Frye, W Thomas Pollard, and Jay L Banks. Glide: A new approach for rapid, accurate docking and scoring. 2. enrichment factors in database screening. Journal of Medicinal Chemistry, 47(7):1750–1759, 2004
2004
-
[10]
Autodock Vina: Improving the speed and accuracy of docking with a new scoring function, efficient optimization, and multithreading
Oleg Trott and Arthur J Olson. Autodock Vina: Improving the speed and accuracy of docking with a new scoring function, efficient optimization, and multithreading. Journal of Computational Chemistry, 31(2):455–461, 2010
2010
-
[11]
Autodock vina 1.2
Jerome Eberhardt, Diogo Santos-Martins, Andreas F Tillack, and Stefano Forli. Autodock vina 1.2. 0: New docking methods, expanded force field, and Python bindings. Journal of Chemical Information and Modeling , 61 (8):3891–3898, 2021
2021
-
[12]
Gnina 1.0: Molecular docking with deep learning
Andrew T McNutt, Paul Francoeur, Rishal Aggarwal, Tomohide Masuda, Rocco Meli, Matthew Ragoza, Jocelyn Sunseri, and David Ryan Koes. Gnina 1.0: Molecular docking with deep learning. Journal of Cheminformatics, 13(1):43, 2021
2021
-
[13]
Cleves, Rocco Varela, and Ajay N
Russell Spitzer, Ann E. Cleves, Rocco Varela, and Ajay N. Jain. Protein function annotation by local binding site surface similarity. Proteins, 82(4):679–694, 2014
2014
-
[14]
Cleves, Rocco Varela, and Ajay N
Russell Spitzer, Ann E. Cleves, Rocco Varela, and Ajay N. Jain. Protein function annotation by local binding site surface similarity. Proteins: Structure, Function, and Bioinformatics , 82(4):679–694, 2014
2014
-
[15]
Lessons learned in empirical scoring with smina from the csar 2011 benchmarking exercise
David Ryan Koes, Matthew P Baumgartner, and Carlos J Camacho. Lessons learned in empirical scoring with smina from the csar 2011 benchmarking exercise. Journal of Chemical Information and Modeling , 53(8): 1893–1904, 2013
2011
-
[16]
ForceGen 3D structure and conformer generation: From small lead-like molecules to macrocyclic drugs
Ann E Cleves and Ajay N Jain. ForceGen 3D structure and conformer generation: From small lead-like molecules to macrocyclic drugs. Journal of Computer-Aided Molecular Design , 31(5):419–439, 2017. 10 Conventional Docking vs. Deep Learning BRIEF PERSPECTIVE
2017
-
[17]
Complex macrocycle exploration: Parallel, heuristic, and constraint-based conformer generation using forcegen
Ajay N Jain, Ann E Cleves, Qi Gao, Xiao Wang, Yizhou Liu, Edward C Sherer, and Mikhail Y Reibarkh. Complex macrocycle exploration: Parallel, heuristic, and constraint-based conformer generation using forcegen. Journal of Computer-Aided Molecular Design, 33(6):531–558, 2019
2019
-
[18]
Structure-and ligand-based virtual screening on DUD-E+: Performance dependence on approximations to the binding pocket
Ann E Cleves and Ajay N Jain. Structure-and ligand-based virtual screening on DUD-E+: Performance dependence on approximations to the binding pocket. Journal of Chemical Information and Modeling , 60(9):4296–4310, 2020
2020
-
[19]
Electrostatic-field and surface-shape similarity for virtual screening and pose prediction
Ann E Cleves, Stephen R Johnson, and Ajay N Jain. Electrostatic-field and surface-shape similarity for virtual screening and pose prediction. Journal of Computer-Aided Molecular Design , 33(10):865–886, 2019
2019
-
[20]
ANI-1: An extensible neural network potential with dft accuracy at force field computational cost
Justin S Smith, Olexandr Isayev, and Adrian E Roitberg. ANI-1: An extensible neural network potential with dft accuracy at force field computational cost. Chemical Science, 8(4):3192–3203, 2017
2017
-
[21]
max 1 " gsim | head -1 | awk ’{ print $1 } ’ | sed s /:// | 15 awk ’{ print
Gabriele Corso, Arthur Deng, Benjamin Fry, Nicholas Polizzi, Regina Barzilay, and Tommi Jaakkola. Deep Confident Steps to New Pockets: Strategies for Docking Generalization. arXiv preprint arXiv:2402.18396, 2024. 11 Conventional Docking vs. Deep Learning BRIEF PERSPECTIVE Appe...
2024 arXiv
-
[22]
Prepare the proteins: 01_protein_prep.py (note: original PDB file copied to protein.pdb)
-
[23]
Remove the ligands from the prepared proteins: $SCHRODINGER/run 02_extract_ligand_from_mae_protein.py
-
[24]
Build the Glide grids: 03_build_glide_grid.py
-
[25]
Dock the molecules: 04_run_glide.py
-
[26]
$ S C H R O D I N G E R / run / opt / schrodinger2022 -3/ mmshare - v5 .9/ python / scripts / p r e p w i z a r d 2 _ d r i v e r . py
Analyze the data: 05_evaluate_docking.ipynb 01_protein_prep.py 1 # !/ usr / bin / env python 3 from glob import glob import os 5 def r u n _ p r o t e i n _ p r e p ( infile_name , o u t f i l e _ n a m e ) : 7 cmd = " $ S C H R O D I N G E R / run / opt / schrodinger2022 -3/ ...
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.