{"id":"779952c8-1858-43e4-a689-ee2ceffb5b04","arxiv_id":"2411.12597","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A graph neural network selects among nine docking programs per protein-ligand pair and reports better average RMSD than any single program on PDBBind 2020 refined set.","lead":"This paper trains graph neural networks to predict which of nine docking programs will give the lowest RMSD for a given protein-ligand pair, then selects that program. On the PDBBind 2020 refined set, the accuracy-oriented model reports a lower mean RMSD (1.74 Å) than any single tested docking program, with the best single program DiffDock at 2.95 Å.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1 selects the GNN architecture on the test set and Table 2 reports final metrics on that same test set, so the headline 1.74 Å mean RMSD is an in-sample, optimistically biased estimate.","rationale":"I read the paper as an empirical claim that a GNN-based algorithm selection model can beat every individual docking tool on mean RMSD, with a secondary efficiency claim. For that claim to hold, the reported test-set numbers must be unbiased estimates of the selected model's performance on new protein-ligand pairs. The weakest point is exactly where the reader pointed: the test set is used both to choose the architecture (Table 1) and to report final results (Table 2). This is not a theoretical flaw in the algorithm-selection idea; it is an evaluation-protocol flaw that makes the headline numbers optimistically biased. The bias is not necessarily fatal, because the architecture search is small and the gap to DiffDock is large, but it is load-bearing because every headline number, 1.74 Å, 81.8%, 92%, 29.05 s, comes from this same test set. I considered alternative concerns, such as the lack of code/data or the absence of error bars. Those are real limitations but they do not by themselves invalidate the reported numbers; the test-set selection issue directly does. I also note the paper includes an Oracle baseline and per-algorithm comparisons, which is useful context, but those do not mitigate the selection-bias problem. A clean three-way split with sequence-identity control would settle the question, so the reader's CONDITIONAL verdict remains appropriate.","tokens_in":12077,"tokens_out":4297,"duration_ms":49184,"concrete_test":"Re-run the full pipeline with a three-way split: 70% train, 15% validation, 15% held-out test. Select the architecture pair from Table 1 using only validation-set RMSD, then finalize that architecture and report Table 2 metrics on the untouched held-out test. Additionally, cluster proteins at 30% sequence identity (e.g., with MMseqs2 or CD-HIT) and ensure no test protein shares a cluster with any training protein. If the held-out mean RMSD remains near 1.74 Å and below DiffDock's 2.95 Å, the concern is resolved; if it rises toward the 1.85–2.0 Å range seen across Table 1, the headline improvement overstates real-world performance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim, that GNNAS-Dock reaches a mean RMSD of 1.74 Å versus 2.95 Å for DiffDock (Table 2), rests on an evaluation protocol that is not unbiased. Section 4.1 and Table 1 explicitly evaluate 21 protein/ligand GNN architecture pairs on the test set and select the pair with the lowest test-set RMSD (GCN_GAT_GINE for protein, GAT for ligand). Section 4.2 then reports the final GNNAS-Dock results on that same test set. Because the test set was used to choose among 21 configurations, the reported 1.74 Å is the best test-set outcome among a small menu of architectures; on a fresh test set the expected performance regresses toward the typical values in Table 1 (roughly 1.85–2.0 Å), not necessarily below DiffDock's 2.95 Å. The split is also described only as 'training and testing set with a ratio of 0.3' (Section 4.1), with no separate validation set and no sequence-identity filtering for the random split. PDBBind contains homologous protein families, so train/test leakage could further inflate the estimates. The qualitative claim that algorithm selection can improve mean docking accuracy may survive, but the specific numbers in the abstract and Table 2 are not supported by the current protocol.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GNNAS-Dock, a graph-neural-network-based algorithm selection system for blind molecular docking. It builds separate GNN encoders for ligand and protein graphs, concatenates their learned features, and trains two meta-models: an accuracy model that predicts the RMSD of each candidate docking algorithm and selects the predicted best, and an efficiency model that predicts binary success (RMSD < 2 Å) and runtime, then selects the fastest predicted-successful algorithm. The system is evaluated on the PDBBind 2020 refined set against a portfolio of nine docking configurations (DiffDock, DSDP, TankBind, GNINA, SMINA, Qvina-W, and three p2rank-augmented variants). The central quantitative claims are a mean RMSD of about 1.74 Å for the accuracy model versus 2.95 Å for DiffDock, and a mean RMSD of 2.75 Å with an average runtime of 29.05 seconds for the efficiency model.","tokens_in":12373,"tokens_out":2177,"duration_ms":22414,"significance":"The paper addresses a practically important and relatively underexplored problem: automated selection among docking algorithms for individual protein-ligand pairs. The idea of end-to-end learned algorithm selection with graph representations of both partners is sensible, and the two-model formulation (accuracy versus efficiency) is a useful framing. If the reported improvements held under a properly unbiased evaluation, the work would be a meaningful step toward robust and cost-aware docking pipelines. The paper also provides a relatively broad algorithmic portfolio and reports symmetry-corrected RMSD, which is good practice. However, the current evaluation protocol does not support the headline numbers, so the significance is currently prospective rather than established.","major_comments":[{"comment":"The GNN architecture pair is selected using the test set, and the same test set is then used to report the final results. Table 1 evaluates 21 protein/ligand architecture combinations on the test set and selects the pair with the lowest test-set RMSD (GCN_GAT_GINE for protein, GAT for ligand). Table 2 then reports the accuracy model's mean RMSD of 1.74 Å on that same test set. Because the test set was used for model selection among 21 configurations, the reported 1.74 Å is the minimum of a small family of test-set estimates, not an unbiased estimate of performance on new data. The expected performance on a fresh test set would regress toward the typical values in Table 1 (approximately 1.83–2.0 Å), which may still be competitive but is not established by the current protocol. The authors need a separate validation set for architecture selection, or nested cross-validation, so that the final test-set numbers are not used in any model-selection decision.","section":"§4.1, Table 1 and §4.2, Table 2"},{"comment":"The train/test split is described only as a ratio of 0.3, with no statement about sequence-identity filtering. PDBBind contains homologous protein families, and a random split can place near-identical proteins in both training and test partitions, inflating the reported accuracy. The authors should cluster proteins by sequence or structure similarity (for example, using BLAST or MMseqs2 clustering) before splitting, and report the resulting split statistics. Without this control, the generalization claim for the 1.74 Å mean RMSD is not supported.","section":"§4.1, dataset split"},{"comment":"No error bars, standard deviations, or significance tests are reported for any of the RMSD or success-rate values. GNN training is stochastic, and the differences between some entries are small (for example, 81.80% versus 84.36% below 2 Å, and 79.73% versus 84.36% in the efficiency comparison). The authors should run multiple independent training runs with different seeds and report means with standard deviations, and use a paired statistical test (e.g., paired bootstrap or Wilcoxon signed-rank test) for the comparisons that support the abstract's claim of significant improvement. As it stands, the reader cannot assess whether the differences are reproducible or within noise.","section":"§4.2, Tables 1–2"}],"minor_comments":[{"comment":"The sentence \"no single algorithm consistently outperforms the others\" would be clearer as \"no single algorithm consistently outperforms all others.\"","section":"§1, Introduction"},{"comment":"The text says the protein GNN architecture is derived from graphLambda and that \"different combination of GNNs should be experimented to determine an effective GNN architecture,\" but the actual search is only described in §4.1. A brief forward reference would improve readability.","section":"§3.2, Protein Graph"},{"comment":"The sentence \"it should be noted that any of these pairs outperform the overall, single best docking algorithm of DiffDock\" is misleading: the values in Table 1 are model-selection metrics on the test set, not independent performance estimates, and the statement should be qualified accordingly.","section":"§4.1, Table 1"},{"comment":"In the Oracle row, the value under the 5 Å column is written as \"99.65\" without a percent sign; it should be \"99.65%.\"","section":"§4.2, Table 2"},{"comment":"There are several typographical errors that should be corrected: \"structual\" (§1), \"diagnoal\" (§3.1), \"Efficieny\" (§4.2), and \"it due it its success\" (§1).","section":"Throughout"},{"comment":"The text says PDBBind contains \"more than 19,000 pairs\" while the abstract and §4 state the refined set has about 5,300 pairs; the distinction between the general and refined sets should be stated explicitly to avoid confusion.","section":"§2.1, Background"}],"recommendation":"major_revision","confidential_remarks":"The core idea is publishable if the evaluation is made unbiased. The test-set architecture selection is a straightforward fix (hold out a validation fold), and sequence-similarity splitting is a standard requirement for protein datasets. I would ask for these before considering the paper further. The paper also does not mention code or data availability; for a reproducibility-oriented venue, providing the trained models and evaluation scripts would strengthen the submission. The self-citation to the authors' prior ALORS-based docking work is appropriate background and not a concern."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper demonstrates a sensible GNN-based algorithm-selection wrapper over nine blind docking tools and shows it can beat the best single tool in mean RMSD. I think the qualitative claim is real and worth following up. The specific number in the abstract is not supported by the protocol.\n\nWhat's actually new: this is the first GNN-based AS for blind docking that uses both protein and ligand graph representations. The design is clean: amino-acid-level protein graph, atom-level ligand graph, GNN feature extraction, stacking meta-model, and two variants (accuracy- and efficiency-oriented). Testing on ~5,300 PDBBind refined complexes against nine algorithms is useful. The observation that selection can beat every individual algorithm aligns with the no-free-lunch framing.\n\nWhere it gets shaky: Table 1 evaluates 21 architecture pairs on the test set and picks the best; Table 2 reports final performance on that same test set. So the 1.74 Å mean RMSD is an in-sample best over a menu of 21 configurations. Expected performance on a fresh set regresses toward the typical Table 1 values (roughly 1.85–2.0 Å), which might still beat DiffDock's 2.95 but not by as much and with real uncertainty. There are no error bars, no significance tests, no sequence-similarity filtering for the random split, and no code or data. The prior AS work by the same group is mentioned but not cited. These are fixable, but they mean the headline should be read as preliminary.\n\nThe efficiency model is a nice addition, and the runtime comparison is broadly reasonable. The same evaluation concerns apply to its accuracy numbers.\n\nWho this is for: people working on docking pipelines, automation, or meta-learning for structural biology. They will get a clear proof-of-concept that algorithm selection is worth trying. The specific accuracy claims need a re-run with a proper held-out validation split before being quoted.\n\nRecommendation: worth a serious referee. The idea is legitimate and the paper is a real contribution, but it needs an unbiased evaluation before the numbers can be trusted.","headline":"The core idea is sound—GNN-based algorithm selection for blind docking—but the headline 1.74 Å result is an in-sample number from test-set architecture selection, not a clean estimate.","tokens_in":12867,"tokens_out":3110,"would_cite":true,"duration_ms":30800,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"No single docking tool wins every case; a GNN that picks per pair cuts mean RMSD to 1.74 Å.","keywords":["algorithm selection","graph neural networks","molecular docking","blind docking","RMSD","PDBBind","runtime prediction","drug discovery"],"falsifier":"Re-run the comparison on a hold-out set built by clustering PDBBind proteins at 30% sequence identity, with architecture selection done on a validation split; if the selected model's mean RMSD is not materially below DiffDock's, the claimed advantage is an artifact of test-set reuse and near-duplicate proteins.","tokens_in":11854,"feed_emoji":"🧬","tokens_out":5575,"duration_ms":47907,"temperature":0.7,"pith_summary":"This paper claims that no single docking program is best for every protein-ligand pair, and that a graph neural network trained to predict each candidate program's RMSD can pick the right tool per case. On the PDBBind 2020 refined set, the selection model's chosen poses average 1.74 Å RMSD, lower than the best individual program (DiffDock at 2.95 Å), with 81.8% of cases below 2 Å. A second efficiency-oriented model keeps 79.73% below 2 Å and averages 29.05 seconds per docking, faster than DiffDock's 37.31 seconds. If the result holds, per-instance algorithm selection becomes a practical way to improve blind docking without designing a new docking engine.","feed_headline":"GNN picks best docking tool: mean RMSD 1.74 Å","feed_subtitle":"Nine docking tools, one GNN pick per protein–ligand pair: 1.74 Å mean RMSD, and the fast variant keeps 79.7% under 2 Å","key_machinery":"The machinery is a stacked graph-neural-network selector. Ligands are represented as atom-bond graphs and proteins as amino-acid-residue graphs with peptide-bond edges; the selected architecture pair is GAT for the ligand graph and a GCN+GAT+GINE combination for the protein graph, the latter adapted from graphLambda. The two GNNs' learned feature vectors are concatenated and fed into a small dense meta-model that outputs either predicted RMSD per algorithm (accuracy model) or predicted binary success at the 2 Å threshold plus predicted runtime per algorithm (efficiency model). The selection rule is to choose the algorithm with the lowest predicted RMSD, or, in the efficiency variant, the fastest algorithm among those predicted to succeed, falling back to the fastest overall when none is predicted to succeed.","core_discovery":"The paper's central claim is that algorithm selection by learned performance prediction can beat every tested single docking program on average RMSD. The accuracy model, GNNAS-Dock (Accuracy), predicts the RMSD each of nine docking algorithms would produce for a given protein-ligand pair and selects the predicted best; on the test set this yields a mean RMSD of 1.74 Å, compared with 2.95 Å for DiffDock, the strongest individual tool. The efficiency model predicts both whether each algorithm will succeed (RMSD below 2 Å) and how long it will take, then chooses the fastest predicted-successful algorithm; it reaches a mean RMSD of 2.75 Å, keeps 79.73% of results below 2 Å, and averages 29.05 seconds per docking. The paper also reports an oracle that always picks the true best tool, reaching 0.67 Å mean RMSD, indicating that the remaining gap is due to imperfect performance prediction rather than to the portfolio itself.","pith_inferences":["A direct test of generalization would be to split PDBBind by protein sequence identity and choose the GNN architecture on a validation set, then measure mean RMSD on a held-out cluster; this would show whether the reported advantage survives contact with unseen protein families.","The same stacked-GNN selection idea could be transferred to docking settings where no single tool dominates, such as flexible-side-chain docking, covalent docking, or docking into cryo-EM density maps.","Because labeling each protein-ligand pair requires running all nine docking tools, active learning over the label budget could make the approach scale to larger datasets at lower computational cost.","The accuracy model averages 66.85 seconds per task, slower than DiffDock, because it sometimes selects slow traditional tools paired with p2rank; a budget-constrained variant that excludes tools over a time limit would directly probe the accuracy-efficiency frontier."],"forward_implications":["If the accuracy model's estimates are unbiased, algorithm selection over existing docking tools can lower mean RMSD below every tested single tool on the same distribution of protein-ligand pairs.","The efficiency model shows that per-instance selection can reduce average runtime relative to DiffDock while keeping roughly four out of five poses under 2 Å, making it relevant for high-throughput screening.","Because the selector uses only the structures of the protein and ligand, it can in principle recommend a docking tool for any new blind docking pair without retraining.","The accuracy model's 92.00% success rate below 5 Å suggests that selection mostly removes poor docking outcomes rather than only sharpening already-good poses.","The oracle result of 0.67 Å mean RMSD defines an upper bound: further improvements in the performance predictor could roughly halve the mean RMSD of the current accuracy model."],"supporting_citations":[{"why":"DiffDock is the best standalone baseline, supplying the 2.95 Å mean RMSD and 37.31 s average runtime that the selection models must beat.","marker":"Corso et al., 2023"},{"why":"PDBind 2020 refined set, the dataset of about 5,300 protein-ligand pairs used for training and testing.","marker":"Liu et al., 2017"},{"why":"TankBind is a fast geometric-deep-learning docking candidate in the portfolio, with 0.03 s runtime.","marker":"Lu et al., 2022"},{"why":"DSDP is the GPU-accelerated blind docking candidate in the portfolio.","marker":"Huang et al., 2023"},{"why":"GNINA is the CNN-scored docking candidate in the portfolio, also combined with p2rank.","marker":"McNutt et al., 2021"},{"why":"SMINA is the customizable AutoDock Vina variant used as a candidate and combined with p2rank.","marker":"Koes et al., 2013"},{"why":"Qvina-W is the faster Vina variant used as a candidate and combined with p2rank.","marker":"Hassan et al., 2017"},{"why":"p2rank provides the docking-site predictions that are combined with GNINA, SMINA, and Qvina-W.","marker":"Krivák & Hoksza, 2018"},{"why":"graphLambda is the source of the protein GNN architecture (GCN, GAT, GINE combination) adapted in GNNAS-Dock.","marker":"Mqawass & Popov, 2024"},{"why":"spyrmsd provides the symmetry-corrected RMSD metric used to score all docking poses.","marker":"Meli & Biggin, 2020"}],"fun_headline_variants":["GNN picks best docking tool: 1.74 Å RMSD vs 2.95 for DiffDock","GNN algorithm selection: mean RMSD 1.74 Å, beats every single tool","GNN picks per-case winner among nine docking algorithms: 1.74 Å","GNN decides docking algorithm per pair: mean RMSD drops to 1.74 Å"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported improvement assumes that the GNN architecture selected in Table 1 and the final results in Table 2 come from the same random test split, with no filtering of similar proteins across train and test; if the architecture had been chosen on a separate validation set and the split clustered by sequence identity, the 1.74 Å mean RMSD could shrink toward DiffDock's level.","fun_headline_variants_meta":{"raw":{"variants":["GNN picks best docking tool: 1.74 Å RMSD vs 2.95 for DiffDock","GNN algorithm selection: mean RMSD 1.74 Å, beats every single tool","GNN picks per-case winner among nine docking algorithms: 1.74 Å","GNN decides docking algorithm per pair: mean RMSD drops to 1.74 Å"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0008,"raw_usage":{"total_tokens":3530,"prompt_tokens":968,"completion_tokens":2562,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":584,"completion_tokens_details":{"reasoning_tokens":2464}},"tokens_in":584,"tokens_out":2562,"duration_ms":15630,"temperature":1.0,"reasoning_tokens":2464,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:21:27.278961+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the comparison on a hold-out set built by clustering PDBBind proteins at 30% sequence identity, with architecture selection done on a validation split; if the selected model's mean RMSD is not materially below DiffDock's, the claimed advantage is an artifact of test-set reuse and near-duplicate proteins.","supporting_citations":[],"review_version":1}