{"id":"564acf84-86fb-4c77-8f86-dca39485b9b1","arxiv_id":"2508.16521","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"RLPF applies PPO-style reinforcement learning with DFT or xTB force-field rewards to fine-tune equivariant diffusion models, improving molecule and atom stability on QM9 and GEOM-drug beyond prior generative baselines.","lead":"Researchers fine-tuned 3D molecular diffusion models with reinforcement learning, using force-field-based rewards to generate more physically stable molecules. On QM9 and GEOM-drug benchmarks, the method lifts molecule stability well beyond prior generative baselines, showing that physics feedback can guide generation after standard training.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Size-invariant log-likelihood in §4.5 replaces the true trajectory density ratio with a per-atom average; reported gains may depend on this nonstandard objective rather than on standard RL.","rationale":"The reader's weakest_assumption points to base-model diversity (Appendix D). That is a real limitation but not the most load-bearing for the reported results: the base EDM on QM9 already produces 18% unstable molecules, so the reward variance needed for advantage estimation is present. The reader's rationale also mentions the size-invariant log-likelihood as a technical gap, but does not identify it as the primary threat. I think the sharper issue is that §4.5 changes the objective itself: the importance ratio used in the PPO surrogate is not the density ratio of the trajectory distribution but a per-atom geometric mean. This means the method is not optimizing the stated expected-reward objective, and the empirical gains could be an artifact of size reweighting rather than evidence that physical-feedback RL improves molecular stability. This is testable with a single re-run using the unnormalized log-ratio. Because the existing verdict is already CONDITIONAL, my concern reinforces that condition rather than requiring a different verdict; I therefore recommend UNCHANGED.","tokens_in":15859,"tokens_out":14608,"duration_ms":197334,"concrete_test":"Re-run the QM9 EDM-RLPF experiment (same pretrained checkpoint, same DFT or xTB reward, same PPO hyperparameters, same 51,200 sampled trajectories) using the correct full-trajectory importance ratio in Eq. (9): log r = Σ_t Σ_{i valid} [log p_θ(x_{t-1,i}|x_t) − log p_θold(x_{t-1,i}|x_t)], with no 1/|M_i| factor in Eq. (13). Compare molecule stability, validity, and V·U to Table 1. Also report these metrics separately for molecules with ≤4 heavy atoms vs ≥5 heavy atoms. If the corrected objective reproduces the Table 1 gains, the size-invariant modification is not load-bearing. If performance drops materially or the small-molecule advantage disappears, the central claim must be re-scoped to the modified objective.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In §4.5 (Eq. 13), RLPF defines the reverse-transition log-probability as an average over valid atoms, dividing by |M_i|. For a molecule with N valid atoms, this makes the importance ratio in Eq. (9) the N-th root of the true trajectory density ratio: log r = (1/N) Σ_{i,t} log(p_θ/p_θold). Consequently the PPO surrogate in Eq. (11) is not an estimator for J_RLPF in Eq. (6); the policy-gradient update is scaled by 1/N per trajectory, so the effective reward becomes R/N rather than R. On QM9, molecules range from a few to 29 atoms, so this normalization reweights the objective strongly toward small molecules. If force rewards correlate with molecular size, the advertised stability/validity gains could largely reflect this size reweighting rather than the physical reward per se. The paper says the Gaussian normalization constant cancels; that part is plausible because EDM's reverse variance is schedule-fixed and independent of θ, but the missing factor of N does not cancel and is not a constant across molecules. No derivation or ablation shows that the corrected, unnormalized trajectory ratio behaves the same. This is a concrete, testable departure from standard DDPO, and it is introduced as 'critical' to RLPF, so the central empirical claim is not yet cleanly attributable to RL on the stated objective.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RLPF (Reinforcement Learning with Physical Feedback), a post-training procedure that fine-tunes pretrained 3D equivariant diffusion models with PPO-style policy gradient updates. The reward is derived from atomic force residuals computed by DFT (B3LYP/6-31G(2df,p)) or GFN2-xTB, with a valency-based stability reward as an additional variant. The method is applied to EDM, GeoLDM, and UniGEM on the QM9 and GEOM-drug datasets. The main reported result is a large improvement in molecule stability (e.g., EDM on QM9: 82.0% to 93.37%) and validity, with smaller gains on GEOM-drug. The appendix contains a continued-training control, reward-function ablations, sampling-step ablations, a clipping-threshold ablation, conditional generation experiments, and an explicit statement of a limitation regarding base-model sampling diversity.","tokens_in":16240,"tokens_out":4725,"duration_ms":54433,"significance":"If the central claim holds, RLPF provides a practical and model-agnostic way to align 3D molecular diffusion models with physical force-field feedback. The method's rewards are external (DFT/xTB), the evaluation metrics are standard, and the paper includes a fairness control against continued training. The code is released, and the paper honestly states a key limitation in Appendix D. However, the policy-gradient objective is modified in a way that is not derived and is not equivalent to the stated RL objective; this affects the attribution of the reported gains. The experimental results are also presented with inconsistencies in key tables. The idea is promising, but the current manuscript requires substantial revision before the claims can be accepted.","major_comments":[{"comment":"The masked per-atom average of the reverse-transition log-probability changes the importance-sampling ratio in Eq. (9). For a molecule with N atoms, log r becomes (1/N) times the true trajectory log-ratio. Consequently, the PPO surrogate in Eq. (11) does not estimate J_RLPF in Eq. (6); it rescales the effective reward by 1/N per trajectory. Since QM9 molecules range from a few to 29 atoms, this size normalization can reweight the objective toward small molecules if force rewards correlate with size. The paper's statement that the Gaussian normalization constant cancels is plausible, but the factor 1/N does not cancel and is not constant across molecules. No derivation or ablation with the unnormalized trajectory ratio is provided. This is load-bearing because the averaged likelihood is described as 'critical' to RLPF. Please provide a derivation, and either use the true trajectory log-pr","section":"§4.5, Eq. (13)"},{"comment":"The EDM-RLPF results on QM9 are reported inconsistently across the two tables. Table 1 lists validity 98.22% and V·U 92.87%, while Table 3 lists validity 97.02% and a garbled V·U value for the same EDM-RLPF configuration. These are not the same numbers, and the paper does not explain the discrepancy. This makes the headline result difficult to reproduce. Please reconcile the tables or state explicitly which configuration/run each number corresponds to.","section":"§5.1 vs. §5.3, Tables 1 and 3"},{"comment":"The fairness control is a useful experiment, but it does not fully isolate the effect of RL versus continued training. The control trains only on rejection-sampled valid molecules, whereas RLPF trains on all sampled trajectories and assigns a penalty to invalid structures. Thus the training distributions differ both in reward signal and in data composition. Please discuss this asymmetry and, if possible, add a control that uses the same accepted/penalized trajectory split without policy-gradient updates, or clarify why the current control is sufficient.","section":"Appendix A.2, Table 6"}],"minor_comments":[{"comment":"The notation is under-specified: the meaning of M_i, the index ranges for s and j, and the exact average (over atoms, features, or both) are not fully defined. Please clarify the formula and its relationship to the per-timestep transition distribution.","section":"§4.5, Eq. (13)"},{"comment":"The baseline 'END [20]' should be 'EDN [20]' to match the reference and the rest of the text. The same typo appears in Table 2.","section":"Table 1"},{"comment":"The 'Data (Ground Truth)' row is ambiguous: the column alignment suggests only one of Atom Stability or Validity is reported, but the entry '86.5' is not labeled. Please clarify which metric it refers to.","section":"Table 2"},{"comment":"Several numerical values in these tables are missing or corrupted (e.g., polarizability MAE values, all rows in Table 5, and parts of Table 4). Ablation results cannot be assessed from the current text. Please ensure all reported numbers are visible and precise.","section":"Appendix A.1, Tables 4 and 5"},{"comment":"The Stability and Force (xTB) rows report identical molecule stability (96.45%) and identical atom stability/validity values; please verify whether this is a formatting error or a genuine result, and explain why two different reward functions produce identical values on all metrics.","section":"Appendix A.5, Table 9"},{"comment":"The pseudocode mentions the Advantage A but does not define it; the definition appears only in Eq. (10). Please add a reference to the equation in the algorithm.","section":"Algorithm 1"},{"comment":"The paper uses N both for the number of denoising steps and later for the number of sampled trajectories, and Eq. (13) uses N for the number of atoms. Please use distinct symbols to avoid confusion.","section":"§5.1"}],"recommendation":"major_revision","confidential_remarks":"The size-invariant log-likelihood in Eq. (13) is the main technical concern. If the authors can supply a derivation or an unnormalized-likelihood ablation and reconcile the table inconsistencies, the paper could become acceptable. The external reward and fairness control are strengths; the explicit limitation in Appendix D is honest. No concerns about novelty disclosure or citation patterns beyond what is stated in the report."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core claim here is believable: fine-tuning 3D diffusion models with force-field-based rewards improves stability and validity across multiple backbones, and the continued-training control in Table 6 is the right check. The method is a fairly direct transfer of DDPO/PPO to equivariant diffusion with a domain-specific reward, and the multi-backbone results (EDM, GeoLDM, UniGEM) plus the code release make it easy to take seriously. I credit the authors for stating their own limitation in Appendix D about dependence on base-model diversity; that's an honest and real weakness, not a hidden one.\n\nThe main soft spot is Eq. (13) in §4.5. The paper defines the reverse-transition log-probability as an average over valid atoms, dividing by |M_i|. That makes the importance ratio the N-th root of the actual trajectory density ratio, so the PPO surrogate is not estimating the J_RLPF in Eq. (6). This is not a constant scaling across molecules—N varies—so the effective reward is R/N, which reweights the objective toward small molecules. The paper calls this 'size-invariant' and says it's critical, but gives no derivation and no ablation comparing against the unnormalized ratio. If force rewards correlate with molecular size, some of the reported gains could come from this reweighting rather than from the physical feedback per se. This is testable and should be resolved before the central claim is clean.\n\nSecond, several tables are missing numbers. Tables 4, 5, and 9 have blank entries in the version I read. That's a rendering or compilation failure, but it makes key ablations unverifiable. Also, the paper doesn't compare against existing RL-based molecular diffusion methods like GDPO; since the novelty rests on the specific combination, at least one such baseline would help isolate the effect of the physics reward.\n\nNone of this is fatal. The paper is systematic, the fairness control is good, and the honest limitation in Appendix D suggests the authors understand their method's boundary. The missing derivations and numbers are fixable in revision. I'd send it to peer review, but I'd ask for a derivation of Eq. (13), an ablation with and without the per-atom normalization, and completed tables. A reader working on RL for generative models or molecule generation would get value from this.","headline":"RLPF is a credible, useful extension of DDPO to 3D molecular diffusion with physical rewards, but the size-invariant log-likelihood in §4.5 changes the objective in a way the paper doesn't justify, and several ablation tables are missing numbers.","tokens_in":16724,"tokens_out":1691,"would_cite":false,"duration_ms":20827,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning diffusion models with force-field rewards substantially improves molecular stability, validity, and equilibrium quality.","keywords":["3D molecular generation","equivariant diffusion models","reinforcement learning","force-field reward","molecular stability","PPO fine-tuning","QM9","GEOM-drug"],"falsifier":"Before fine-tuning, draw a large sample from the pretrained model and compute the variance of force-RMSD rewards. If that variance is near zero, RLPF predicts negligible stability gain; if variance is large, molecule stability should jump by roughly the reported margin. A second check is to re-optimize generated post-RLPF molecules with the same force field: if the low residual forces are genuine equilibrium, energy minimization should change them little, and a different force field should also report low forces.","tokens_in":15785,"feed_emoji":"🧪","tokens_out":5487,"duration_ms":61624,"temperature":0.7,"pith_summary":"This paper claims that a pretrained 3D diffusion model for molecules can be fine-tuned with reinforcement learning whose reward is a physical force-field evaluation, and that this post-training step substantially improves structural stability. The proposed method, RLPF, treats each denoising step as an action in a Markov decision process and scores the final molecule by the root-mean-square deviation of its atomic forces under DFT or GFN2-xTB. On QM9, fine-tuning EDM with DFT force rewards raises molecule stability from 82.0% to 93.37% and validity from 91.9% to 98.22%; on GEOM-drug, atom stability rises from 81.3% to 87.52%. The same recipe improves GeoLDM and UniGEM, and a control experiment shows continued supervised training on valid samples does not produce comparable gains. If correct, this makes physics-based RL fine-tuning a general post-training strategy for molecular diffusion models.","feed_headline":"Force-field rewards lift molecule stability to 93.4%","feed_subtitle":"Physics-based rewards retrain 3D diffusion models to produce equilibrium structures, not just valid ones.","key_machinery":"The load-bearing object is the force-RMSD reward, the root-mean-square deviation of atomic forces computed on the final generated molecule. It is attached to a DDPO-style Markov decision process over denoising steps and optimized with PPO-style clipping and standardized advantages. A size-invariant log-likelihood, implemented as a masked average over valid atoms, keeps gradient contributions balanced across variable-size molecules. The force-field reward is what converts chemically valid generation into physically relaxed generation.","core_discovery":"RLPF's central claim is that direct physical feedback, in the form of atomic force residuals, is a usable reward for aligning diffusion-generated molecules with equilibrium structures. The paper formulates the reverse diffusion process as an MDP in which the state is the noised molecular latent, the action is the denoising step, and a scalar reward is assigned only at the terminal step: the force RMSD from DFT at the B3LYP/6-31G(2df,p) level or the semi-empirical GFN2-xTB method, or a valency-based stability signal. Policy updates use a PPO-style clipped surrogate objective with advantage normalization and a size-invariant masked log-likelihood so molecules with different atom counts contrib","pith_inferences":["A natural next step is to reuse the same pipeline with rewards beyond force residuals, such as binding energy, strain energy, or solvation free energy, to steer generation toward function rather than equilibrium.","Because RLPF's advantage signal depends on reward variance, monitoring that variance during training could serve as a diagnostic: when it collapses, stability gains should plateau. This follows directly from the paper's stated limitation.","The reduced novelty of the DFT-fine-tuned model may not be a defect, since QM9 is a nearly complete enumeration of small stable molecules; lower novelty may partly reflect stronger confinement to valid chemical space.","A stress test of the reward would be to re-minimize post-RLPF molecules with a different force field; if the forces are genuinely near equilibrium, the alternative force field should also report low residuals, and a mismatch would suggest overfitting to one energy surrogate."],"forward_implications":["RLPF can be dropped onto any pretrained 3D diffusion generator without collecting new data or retraining the backbone from scratch.","Force-field rewards can be chosen by cost: DFT for small molecules, GFN2-xTB for drug-sized molecules, with similar stability gains at much lower compute.","RLPF fine-tuning roughly halves both the number of samples and the wall-clock time required for rejection sampling of stable molecules.","Adding a property-prediction term to the force reward improves conditional generation accuracy while preserving stability gains.","Supervised continued training on valid samples does not reproduce RLPF's stability gains, indicating that the reinforcement signal, not extra data, drives the improvement."],"supporting_citations":[{"why":"Supplies the pretrained equivariant diffusion backbone that RLPF fine-tunes.","marker":"[9]"},{"why":"Provides the denoising-as-MDP policy optimization formulation that RLPF adapts to 3D molecules.","marker":"[23]"},{"why":"Defines the GFN2-xTB semi-empirical force field used to compute cheap force-deviation rewards.","marker":"[27]"},{"why":"Supplies the QM9 benchmark and the training/validation/test split used for the small-molecule experiments.","marker":"[15]"},{"why":"Supplies the GEOM-drug benchmark of drug-like conformers used for the large-molecule experiments.","marker":"[16]"},{"why":"One of the alternative diffusion backbones whose fine-tuning demonstrates RLPF's generalizability.","marker":"[13]"},{"why":"The second alternative backbone showing RLPF improves generation quality beyond EDM.","marker":"[14]"},{"why":"The REINFORCE-style policy gradient that underlies the optimized objective.","marker":"[12]"}],"fun_headline_variants":["Force-field rewards guide diffusion models to stable molecules","Physics-based rewards fine-tune diffusion for molecule stability","RL with force-field feedback yields equilibrium 3D structures","PPO fine-tunes diffusion models using atomic force residuals","Direct physical feedback improves 3D molecule generation"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"RLPF's gains depend on the base model having enough spread between stable and unstable outputs; if the pretrained model generates molecules of similar quality, all advantages shrink toward zero and the fine-tuning signal disappears.","fun_headline_variants_meta":{"raw":{"variants":["Force-field rewards guide diffusion models to stable molecules","Physics-based rewards fine-tune diffusion for molecule stability","RL with force-field feedback yields equilibrium 3D structures","PPO fine-tunes diffusion models using atomic force residuals","Direct physical feedback improves 3D molecule generation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000212,"raw_usage":{"total_tokens":1240,"prompt_tokens":715,"completion_tokens":525,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":459,"completion_tokens_details":{"reasoning_tokens":463}},"tokens_in":459,"tokens_out":525,"duration_ms":5324,"temperature":1.0,"reasoning_tokens":463,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T17:15:02.091078+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Before fine-tuning, draw a large sample from the pretrained model and compute the variance of force-RMSD rewards. If that variance is near zero, RLPF predicts negligible stability gain; if variance is large, molecule stability should jump by roughly the reported margin. A second check is to re-optimize generated post-RLPF molecules with the same force field: if the low residual forces are genuine equilibrium, energy minimization should change them little, and a different force field should also report low forces.","supporting_citations":[{"cited_title":"Equivariant diffusion for molecule generation in 3D","cited_arxiv_id":null,"evidence_quote":"Supplies the pretrained equivariant diffusion backbone that RLPF fine-tunes."},{"cited_title":"Training diffusion models with reinforcement learning","cited_arxiv_id":null,"evidence_quote":"Provides the denoising-as-MDP policy optimization formulation that RLPF adapts to 3D molecules."},{"cited_title":"Gfn2-xtb—an accurate and broadly parametrized self-consistent tight-binding quantum chemical method with multipole electrostatics and density-dependent dispersion contributions","cited_arxiv_id":null,"evidence_quote":"Defines the GFN2-xTB semi-empirical force field used to compute cheap force-deviation rewards."},{"cited_title":"Quantum chemistry structures and properties of 134 kilo molecules","cited_arxiv_id":null,"evidence_quote":"Supplies the QM9 benchmark and the training/validation/test split used for the small-molecule experiments."},{"cited_title":"Geom, energy-annotated molecular conformations for property prediction and molecular generation","cited_arxiv_id":null,"evidence_quote":"Supplies the GEOM-drug benchmark of drug-like conformers used for the large-molecule experiments."},{"cited_title":"Geometric latent diffusion models for 3d molecule generation","cited_arxiv_id":null,"evidence_quote":"One of the alternative diffusion backbones whose fine-tuning demonstrates RLPF's generalizability."},{"cited_title":"Unigem: A unified approach to generation and property prediction for molecules","cited_arxiv_id":null,"evidence_quote":"The second alternative backbone showing RLPF improves generation quality beyond EDM."},{"cited_title":"Simple statistical gradient-following algorithms for connectionist reinforcement learning","cited_arxiv_id":null,"evidence_quote":"The REINFORCE-style policy gradient that underlies the optimized objective."}],"review_version":1}