{"id":"7c673704-fa8a-4bf3-b8b7-daebe2a400c0","arxiv_id":"2501.15631","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"BoKDiff fine-tunes a diffusion model for 3D ligand generation on the highest-scoring candidates using QED, SA, and Vina rewards, and shows that best-of-N sampling alone improves property metrics.","lead":"This paper applies a known AI alignment trick, called best-of-K reward-ranked fine-tuning, to 3D molecule generation for drug design. It reports modest gains over the DecompDiff baseline on the CrossDocked2020 benchmark, plus a cheaper inference-time variant that selects the best of 20 generated molecules.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline SOTA claim is not yet established: the reward weights [1,0,0] were selected on the same 30 test proteins used for ablations, and the reported DecompDiff-vs-BoKDiff gains look within noise. A held-out evaluation with confidence intervals is required.","rationale":"The paper's main claim is that BoKDiff achieves state-of-the-art results on CrossDocked2020, specifically -8.58 average Vina Dock and 26% success. For this claim to hold, the reported numbers must be a fair, low-bias estimate of performance on unseen test proteins. The paper explicitly uses the test set to select the reward weights: the ablation section says evaluations were limited to 30 test samples, Tables 4 and 5 compare five weight configurations on those samples, and the Main Results table is 'conducted using our optimal weight configuration'. That is a textbook model-selection-on-the-test-set protocol. The differences between configurations in the ablation are tiny, so the selection is effectively picking among noise. The DecompDiff-to-BoKDiff deltas in Table 1 are also small, and since no measures of variance are given, there is no way to tell whether the improvement is real or a random fluctuation. This is more load-bearing than the CoM-relocation concern: even a perfect CoM relocation would not rescue a headline result whose evaluation protocol is biased. However, the issue is fixable with a proper held-out split and significance testing, so conditional acceptance is the right call. I credit the paper for releasing code and for using a sensible base model (DecompDiff), and if the held-out test confirms the gains, the central claim would be substantially strengthened. The reader identified the CoM-relocation assumption as the weakest point, but also noted the test-set selection and same-sample ablation in their rationale; I agree with the reader's verdict but place the weight on the evaluation-protocol problem.","tokens_in":14430,"tokens_out":6091,"duration_ms":55693,"concrete_test":"Split the 100-protein test set into selection (30) and held-out (70) splits. Run the full BoKDiff pipeline for each candidate weight set ([1,1,1], [1,0,0], [0,1,0], [0,0,1], [1,1,2]) using only the 30 selection proteins to pick the best weight, then evaluate that single pre-registered configuration on the held-out 70 alongside DecompDiff on the same proteins. Report per-protein paired differences for Vina Dock and success rate with bootstrap 95% confidence intervals or a Wilcoxon signed-rank test. If [1,0,0] is no longer best, or the CIs include zero, the SOTA claim fails; if it remains best with the CI excluding zero, the concern is resolved. A cheaper check is to recompute Table 4 with repeated random 30/70 splits of the 100 test proteins.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on selecting the reward-weight configuration [QED:1, SA:0, Vina:0] from ablations run on 30 test-set proteins (Section 'Weight Configurations', Tables 4-5), then reporting Table 1 with this 'optimal' configuration on the full 100-protein test set. This is test-set selection: the hyperparameter that drives the headline -8.58 Vina Dock and 26.0% success was chosen by comparing configurations on the same proteins used to measure final performance. The selection signal is weak: accumulated scores in Table 4 differ by only 0.013 (1.880 vs 1.867) between [1,0,0] and [1,1,1], and no confidence intervals, standard deviations, or paired significance tests are reported anywhere. The core comparisons, DecompDiff -8.39 vs BoKDiff -8.58 and 24.5% vs 26.0%, are single mean/median summaries over roughly 1000 molecules; without per-protein paired analysis, these differences are well within plausible sampling noise. The CoM-relocation step may be sound, but it is not what currently carries the SOTA claim; the evaluation protocol does not.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes BoKDiff, a fine-tuning framework built on DecompDiff for target-specific 3D molecule generation. BoKDiff applies the RAFT reward-ranked fine-tuning recipe: for each protein in a batch, it generates K ligand samples, ranks them with a weighted sum of QED, SA, and Vina docking score, relocates the best sample's center of mass to that of its docking pose, extracts sub-pockets and substructures from the relocated sample, and fine-tunes the base diffusion model on the curated high-reward set. The paper also evaluates a Best-of-N sampling strategy that skips fine-tuning and directly selects the highest-reward candidate from N samples. The headline claims are a -8.58 average Vina docking score and a 26% success rate on CrossDocked2020, presented as state-of-the-art results, together with a BoN success rate above 35%. The main methodological novelty is the application of best-of-K alignment and BoN sampling to structure-based drug design, plus a center-of-mass relocation step intended to fix ligand-protein misalignment before substructure extraction.","tokens_in":14657,"tokens_out":5284,"duration_ms":50399,"significance":"If the evaluation were sound, this would be a useful and clearly described contribution: it transplants a successful LLM alignment strategy (RAFT) into structure-based drug design, provides a concrete fix for the misalignment problem in decomposed diffusion models, and includes a public code repository, which is valuable for reproducibility. The center-of-mass relocation idea is simple and potentially transferable to other SBDD pipelines. However, the current manuscript does not establish the state-of-the-art claim because the central evaluation protocol selects hyperparameters on the same test set used for final reporting, reports no uncertainty estimates, and relies on small aggregate differences over a single run. The core alignment mechanism may well be sound, but the evidence as presented is not yet sufficient for the claimed strength of the results.","major_comments":[{"comment":"The optimal weight configuration [QED:1, SA:0, Vina:0] is selected by comparing five configurations on 30 samples taken from the test set (Tables 4 and 5), and the same configuration is then used to produce the headline results in Table 1. This is test-set selection: the hyperparameter that drives the -8.58 Vina Dock and 26.0% success rate is chosen by looking at the evaluation proteins. Moreover, the selection signal in Table 4 is very weak, with accumulated mean scores of 1.880 for [1,0,0] versus 1.867 for [1,1,1], a difference of 0.013, and no confidence intervals or significance tests are given. Please re-run the weight selection on a validation split that is disjoint from the reported test set, and report the variability of the selection criterion.","section":"Weight Configurations, Tables 4-5"},{"comment":"The ablation studies are described as 'conducted on 30 test set samples,' and the table captions state that the results use '30 data samples selected from the test set.' Since the BoKDiff pipeline fine-tunes on the collected samples, this means the model is fine-tuned and evaluated on the same test proteins. This contamination undermines the conclusions about weight configurations, batch size, iteration number, and the docking-pose versus reference-ligand relocation comparison. Please use a separate validation set for all ablations and report results on a truly held-out test set.","section":"Ablation Studies, Tables 4-6"},{"comment":"Table 1 reports a single mean and median per metric with no error bars, no per-protein paired analysis, and no statement about the number of random seeds or independent runs. The differences that support the state-of-the-art claim are small: BoKDiff versus DecompDiff gives Vina Dock -8.58 versus -8.39 and success rate 26.0% versus 24.5%. With roughly 1000 generated molecules, these differences may be within sampling noise, especially if docking seeds and generation randomness are not controlled. Please report repeated runs, per-protein paired comparisons, and confidence intervals, and clarify whether the baseline numbers in Table 1 were re-run under identical conditions or taken from prior publications.","section":"Main Results, Table 1"},{"comment":"The center-of-mass relocation step is central to the method, but the paper does not directly validate its key assumption: that translating a generated ligand so that its center of mass coincides with the center of mass of its Vina docking pose preserves the ligand's structural integrity and allows Alphaspace2 to extract the correct sub-pockets and substructures. The initial attempt with raw docking poses is said to alter ligand structure, and the proposed fix avoids that alteration, yet no quantitative evidence is given that the relocated ligand still corresponds to a valid, extractable decomposition. Table 6 compares end metrics for two relocation sources, but it does not show extraction success rates, substructure assignment fidelity, or validity before and after relocation. Please add direct validation of the extraction step.","section":"Data Preparation"},{"comment":"The BoN results in Table 3 are also affected by selection on the evaluation data: the text states that the weight set [1.1,1,0.9] 'was identified through a grid search centered around the baseline [1,1,1],' and then Table 3 reports performance for that selected configuration. The claim that BoN surpasses 35% success rate is therefore based on the best configuration found on the same data used to report the result. In addition, the sentence 'other combinations emphasizing SA or Vina Dock are omitted due to their similarity to [1,0,0]' is inconsistent with the table, which includes [1,1,1] and [1.1,1,0.9]. Please clarify the selection procedure and report BoN results on a held-out set.","section":"Best-of-N, Table 3"}],"minor_comments":[{"comment":"The manuscript still contains placeholder header text ('Journal Title Here', 'FOR PUBLISHER ONLY', '© The Author 2022') and several typographical issues such as 'F or permissions' and 'Y ear'; these should be cleaned before submission.","section":"Title page"},{"comment":"The description of Vina score normalization is underspecified: min-max scaling across the K=10 samples per input depends on the particular set of generated samples and can be unstable when K is small; please state the exact formula and how ties are handled.","section":"Data Ranking section"},{"comment":"The notation for the DecompDiff equations is not fully self-contained: the variables H, Σ, µ, and the tilde notation for prior-centered positions are introduced only briefly. Readers not already familiar with DecompDiff would benefit from explicit definitions.","section":"Equations (1)-(3)"},{"comment":"The abstract and Section 5 both claim QED values exceeding 0.6, SA above 0.75, and success above 35% for BoN, but these numbers only hold for the larger N values and for selected weight sets; the text should make clear that these are properties of a particular, data-dependent configuration rather than of BoN sampling per se.","section":"Best-of-N section"},{"comment":"The implementation details state that fine-tuning is run for '1,000 epochs' but earlier text says '1,000 iterations' and the original model is trained for 500,000 iterations; please use consistent terminology and specify what one epoch or iteration means.","section":"Experimental Setup"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a relevant problem in computational drug discovery and the core idea is potentially interesting, but the evaluation protocol is the main obstacle: hyperparameter selection and ablations are performed on the test set, the headline comparisons have no uncertainty quantification, and the relocation step lacks direct validation. These issues are fixable with a held-out validation protocol and additional experiments, so I would not recommend rejection, but I cannot support acceptance or minor revision in the current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"BoKDiff is a sensible engineering contribution: it transfers RAFT's reward-ranked fine-tuning and best-of-N sampling from LLMs to structure-based drug design, using DecompDiff as the base. The center-of-mass relocation step is a real fix for a genuine practical problem—generated ligands land far from the pocket and DecompDiff's sub-component extraction fails. The paper is honest about its limitations and ships code, which helps reproducibility.\n\nWhat is actually new is limited. RAFT and BoN are not new; the application to SBDD is new, and the CoM relocation is the one genuinely fresh piece. The BoN results are also interesting but they are not an apples-to-apples comparison: BoN metrics are computed on the single best sample per protein, while the BoKDiff row in Table 1 averages over all 10 generated samples. That discrepancy is acknowledged in the text but makes the 'success rate surpassing 35%' claim misleading next to the 26% number.\n\nThe bigger problem is the evaluation protocol. The reward weights [1,0,0] were chosen by comparing configurations on 30 test-set proteins (Tables 4 and 5), and then the same optimal configuration is used to report Table 1 on the 100-protein test set. That is test-set selection. The differences are also small—Vina Dock -8.39 to -8.58, success 24.5% to 26%—and there are no confidence intervals or paired significance tests anywhere. The ablation study fine-tunes and evaluates on the same 30 samples, which is direct contamination. As a result, the SOTA claim is not established. The method might genuinely help; the paper just doesn't prove it.\n\nThe math and the citation pattern look fine. The method section is clear and the implementation details are enough to reproduce the pipeline. The soft spots are all in the evaluation design, not in the core idea.\n\nThis paper deserves a serious referee, but the referee should press hard on a held-out validation split, error bars, and a paired per-protein analysis. The authors should also either drop the 'state-of-the-art' phrasing or back it with a properly designed comparison. I'd cite the CoM relocation if I were working on preprocessing tricks, but I would not cite the headline numbers.","headline":"Useful engineering recipe for aligning DecompDiff, but the evaluation protocol undermines the SOTA claim; the headline numbers are partly selected on the test set.","tokens_in":15298,"tokens_out":2002,"would_cite":true,"duration_ms":17517,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A best-of-K fine-tuning loop over a base diffusion model's own samples improves 3D molecule generation for drug targets, reaching -8.58 average Vina docking score and 26% success rate.","keywords":["3D drug design","Best-of-N","best-of-K alignment","diffusion models","structure-based drug design","molecular generation","docking score","reward-ranked fine-tuning"],"falsifier":"A reader could test the mechanism by taking a fixed batch of generated ligands and comparing substructure-extraction success with and without the center-of-mass relocation; if extraction succeeds at similar rates without relocation, or fails often even with it, the alignment step is not doing the work attributed to it. The paper does not report extraction success rates, only the final aggregate metrics.","tokens_in":14169,"feed_emoji":"🧬","tokens_out":9598,"duration_ms":76091,"temperature":0.7,"pith_summary":"The paper proposes BoKDiff, a framework that takes an existing target-specific 3D molecule diffusion model (DecompDiff) and aligns it to drug-design objectives through reward-ranked fine-tuning. The authors generate several candidate ligands per protein pocket, rank them with a weighted reward of drug-likeness (QED), synthesizability (SA), and a Vina docking score, and fine-tune the base model on the best candidate. Because generated ligands often sit far from the protein in 3D space, which breaks the substructure extraction the model relies on, the paper introduces a data-preparation step that relocates each winning ligand's center of mass to that of its Vina docking pose before re-extracting sub-pockets and substructures. On the CrossDocked2020 benchmark, BoKDiff reports a -8.58 average Vina docking score and a 26% success rate, outperforming the base DecompDiff model (-8.39 and 24.5%). A separate Best-of-N variant, which skips fine-tuning and simply selects the best of N generated samples, reports QED above 0.6, SA above 0.75, and success rates above 35%.","feed_headline":"Best-of-K fine-tuning lifts 3D drug design to -8.58 dock score","feed_subtitle":"On CrossDocked2020, BoKDiff beats the base DecompDiff model's success rate (26% vs 24.5%) and average Vina Dock (-8.58 vs -8.39).","key_machinery":"The central object is the DecompDiff generator, a diffusion model that produces atoms and bonds conditioned on data-dependent priors built from protein sub-pockets and ligand fragments (arms and a scaffold). Around it, BoKDiff wraps a Best-of-K reward-ranked fine-tuning loop (RAFT): sample K candidate ligands per pocket, score them with a weighted reward of QED, SA, and Vina dock, keep the top one, relocate its center of mass to the docking pose's center of mass, re-extract sub-pockets and substructures with adjusted Alphaspace2 parameters, and fine-tune the generator on the curated sample. The relocation is the paper's key adaptation; without it, generated ligands are often too far from the protein for substructure extraction to succeed.","core_discovery":"The central claim is that a reward-ranked fine-tuning loop, adapted from RAFT, can push a strong diffusion generator beyond its original performance on binding and drug-likeness metrics. Concretely, BoKDiff samples K=10 ligands per training pocket, scores them with a weighted reward (with the best configuration found to be QED weight 1, SA weight 0, Vina weight 0), keeps the top-scoring ligand, relocates its center of mass to the center of mass of its Vina docking pose, and fine-tunes DecompDiff for 1,000 iterations at a learning rate of 1e-6 on the resulting curated set. The paper reports that this procedure raises mean Vina Dock from -8.39 to -8.58, success rate from 24.5% to 26.0%, and mean QED from 0.45 to 0.48 relative to DecompDiff, and it argues that the center-of-mass relocation is what makes the fine-tuning signal usable by enabling reliable sub-pocket and substructure extraction. It also claims that a no-fine-tuning Best-of-N sampling strategy, selecting the best of 20 generated samples by the same reward, achieves higher QED, SA, and success rate than the fine-tuned model, at greater inference cost.","pith_inferences":["An untested extension is to relocate to multiple docking poses per ligand and aggregate the reward, which the paper lists as future work but does not run; if the relocation mechanism is what matters, pose diversity should sharpen the reward signal.","The success-rate threshold comes from Jin et al. and is permissive (QED > 0.25, SA > 0.59, Vina Dock < -8.18); adopting a more stringent threshold could reorder the relative standing of BoKDiff and Best-of-N.","The paper's comparison of CoM relocation to the docking pose versus the reference ligand suggests that docking-pose alignment is the more useful signal; a natural next probe is whether partial rotations toward the pose add further gains without breaking substructure extraction.","The finding that a QED-only reward also lifts Vina metrics may be specific to DecompDiff's inductive bias (it favors simpler arms), so transferring BoKDiff to a different generator may not reproduce the same correlation."],"forward_implications":["A base diffusion model for structure-based drug design can be improved on docking and success-rate metrics by fine-tuning on its own best-ranked generated samples, with no change to the model architecture and a fine-tuning cost of roughly 10 minutes on one GPU.","Because the optimal reward configuration was QED-only, the paper implies that optimizing drug-likeness alone can transfer to improved Vina metrics and success rate for this model.","The Best-of-N results indicate that simply drawing N=20 samples and picking the highest-reward one yields success rates above 35% and QED above 0.65, outperforming the fine-tuned model on those metrics at higher inference cost.","The success of the pipeline depends on having a docking pose for each generated sample; every improvement in the paper is therefore conditional on the availability and quality of those poses."],"supporting_citations":[{"why":"Supplies the reward-ranked fine-tuning (RAFT) procedure that BoKDiff adapts.","marker":"[5]"},{"why":"DecompDiff is the base diffusion model being aligned; its checkpoint, decomposed priors, and sampling code are used throughout.","marker":"[11]"},{"why":"BoNBoN provides the theoretical grounding that Best-of-N sampling approximates RLHF sampling, underpinning the BoN variant.","marker":"[12]"},{"why":"CrossDocked2020 is the dataset used for both fine-tuning data collection and final evaluation.","marker":"[8]"},{"why":"AutoDock Vina computes the docking scores used in the reward function and in the reported Vina metrics.","marker":"[6]"},{"why":"Defines the success-rate thresholds (QED > 0.25, SA > 0.59, Vina Dock < -8.18) that the paper reports.","marker":"[13]"},{"why":"Defines QED, one of the three reward components.","marker":"[2]"},{"why":"Defines SA, one of the three reward components.","marker":"[7]"}],"fun_headline_variants":["BoKDiff: Best-of-K fine-tuning hits -8.58 dock score","Best-of-K alignment lifts 3D drug design to SOTA","BoKDiff: Reward-ranked diffusion beats DecompDiff","Best-of-N sampling delivers 35% success in SBDD","BoKDiff: Refining 3D molecules with Best-of-K"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The data-preparation step assumes that translating a generated ligand so its center of mass coincides with that of its docking pose preserves the ligand's structural integrity and lets the modified Alphaspace2 extraction reliably identify sub-pockets and substructures; if that alignment corrupts or misassigns components, the fine-tuning signal is degraded and the gains would not generalize.","fun_headline_variants_meta":{"raw":{"variants":["BoKDiff: Best-of-K fine-tuning hits -8.58 dock score","Best-of-K alignment lifts 3D drug design to SOTA","BoKDiff: Reward-ranked diffusion beats DecompDiff","Best-of-N sampling delivers 35% success in SBDD","BoKDiff: Refining 3D molecules with Best-of-K"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000199,"raw_usage":{"total_tokens":1476,"prompt_tokens":1153,"completion_tokens":323,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":769,"completion_tokens_details":{"reasoning_tokens":230}},"tokens_in":769,"tokens_out":323,"duration_ms":3428,"temperature":1.0,"reasoning_tokens":230,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:05:15.745585+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could test the mechanism by taking a fixed batch of generated ligands and comparing substructure-extraction success with and without the center-of-mass relocation; if extraction succeeds at similar rates without relocation, or fails often even with it, the alignment step is not doing the work attributed to it. The paper does not report extraction success rates, only the final aggregate metrics.","supporting_citations":[{"cited_title":"Three-dimensional convolutional neural networks and a cross-docked data set for structure-based drug design","cited_arxiv_id":null,"evidence_quote":"CrossDocked2020 is the dataset used for both fine-tuning data collection and final evaluation."},{"cited_title":"Autodock vina 1.2","cited_arxiv_id":null,"evidence_quote":"AutoDock Vina computes the docking scores used in the reward function and in the reported Vina metrics."},{"cited_title":"Multi-objective molecule generation using interpretable substructures","cited_arxiv_id":null,"evidence_quote":"Defines the success-rate thresholds (QED > 0.25, SA > 0.59, Vina Dock < -8.18) that the paper reports."},{"cited_title":"Quantifying the chemical beauty of drugs","cited_arxiv_id":null,"evidence_quote":"Defines QED, one of the three reward components."},{"cited_title":"Estimation of synthetic accessibility score of drug-like molecules based on molecular complexity and fragment contributions","cited_arxiv_id":null,"evidence_quote":"Defines SA, one of the three reward components."}],"review_version":1}