Pith. sign in

REVIEW 5 major objections 5 minor 88 references

DrugImproverGPT: A Large Language Model for Drug Optimization with Fine-Tuning via Structured Policy Optimization

T0 review · 5 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuning a drug-optimization LLM with Structured Policy Optimization makes it generate analogs that outperform the original molecule and competing baselines on normalized multi-objective reward.

desk verdict A plausible RL fine-tuning recipe and a valuable docking dataset, but the docking claim is not independently validated because the same fitted surrogate serves as both reward and evaluator. read the letter →

arxiv 2502.07237 v1 pith:HJ4SO7VP submitted 2025-02-11 cs.LG cs.CLq-bio.BMstat.ML

Xuefeng Liu , Songhao Jiang , Siyu Chen , Zhuoran Yang , Yuxin Chen , Ian Foster , Rick Stevens This is my paper · ORCID
classification cs.LGcs.CLq-bio.BMstat.ML
keywords drugoptimizationlargelanguagemodelsreinforcementlearningStructuredPolicyadvantagepreferencedockingscoresurrogateSMILESgenerationrepurposing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a large language model, fine-tuned with a new reinforcement-learning algorithm called Structured Policy Optimization (SPO), can take an existing drug molecule and generate analogs that score better than the original across several drug-relevant objectives while staying chemically similar. The practical payoff would be substantial: adapting an approved drug to a fast-evolving virus or a drug-resistant cancer is far cheaper and faster than inventing a new drug from scratch, and most generative models either build molecules from nothing or reproduce biases from their training data. The paper reports that on the SARS-CoV-2 protease 3CLPro and the human cancer target RTCB, the generated analogs beat the original drug and all compared baselines on average normalized reward, with Tanimoto similarity held above 0.6. It also releases a dataset of one million compounds with docking scores against multiple viral and cancer protein targets, so other methods can be fine-tuned and benchmarked on the same grounds. The central mechanism is an advantage preference: the policy is pushed to increase a generated molecule's normalized multi-objective score relative to the input molecule, and the reward signal is additionally densified by scoring partial molecules.

What carries the argument

The load-bearing object is the advantage-preference reward $r_{AP}(Y_{1:T}, X) = R_c(Y_{1:T}) - R_c(X)$, where $R_c$ is a min-max normalized sum of four critics (drug-likeness, solubility, synthesizability, docking score) plus a Tanimoto-similarity term relative to the input drug. SPO uses this signed difference as the policy-gradient coefficient, so sequences that beat the original are reinforced and sequences that fall short are suppressed. To handle sparse rewards, it also samples a random partial prefix, completes the prefix with roll-in/roll-out using best-of-N decoding, and forms a blended advantage $R_{AP}$ that averages the partial-molecule advantage and the full-sequence advantage. The theoretical lemmas show that, when best-of-N strictly improves suboptimal molecules, maximizing the SPO objective is equivalent to maximizing the standard reward, and that the blended gradient decomposes into per-prefix terms, which is the formal sense in which the reward signal is densified.

What would settle it

Rescore a random sample of the molecules generated for 3CLPro and RTCB with the actual docking program, and compare their scores with the original drugs' scores. If the true docking scores of the generated set are not better than the originals, or are systematically worse than the surrogate predicted, the paper's central claim is refuted.

Watch

Extended reading notes

Core claim

The paper's central claim is that SPO fine-tuning turns a pretrained sequence-to-sequence LLM into a drug-optimization engine: given a source drug, it outputs valid chemical analogs with higher normalized reward than the source and higher than every baseline tested, while preserving similarity to the source. In the main experiments, the method's average normalized reward is 0.601 on 3CLPro versus 0.524 for the original and 0.575 for the best baseline variant, and 0.694 on RTCB versus 0.538 for the original and 0.642 for the best baseline. The complete comparison also shows higher drug-likeness, better synthesizability, and improved solubility on both targets; the docking component improves on RTCB (-9.462 vs. -8.538) but not on 3CLPro (-8.163 vs. -8.687), which is why the authors phrase the headline result as outperforming across most performance metrics. An ablation attributes part of the gain to the partial-molecule term, which raises validity and most rewards compared to optimizing only complete sequences.

Load-bearing premise

The load-bearing premise is that the learned docking surrogate's predicted scores are accurate for the newly generated molecules; the same surrogate is used both inside the reward and in the reported evaluation, so if it is systematically optimistic for generated analogs, the main improvement claim fails.

Editorial extensions

If this is right

  • If the central claim is right, drug optimization becomes a sequence-to-sequence LLM task: input a known drug and get back a structurally similar analog with better computed properties, without enumerating chemical space.
  • The partial-molecule advantage term densifies the reward signal, so SPO should train more stably and converge faster on sparse-reward molecular tasks than full-sequence-only policy gradients.
  • The released docking dataset and surrogate models give the field a shared benchmark for fine-tuning and comparing generative drug-optimization methods on viral and cancer targets.
  • Because the reward is a weighted sum of arbitrary critics, the same algorithm extends to other objectives such as toxicity or metabolic stability whenever those critics can be computed for generated molecules.
  • Reported validity of 0.844 on 3CLPro and 0.964 on RTCB indicates the fine-tuned policy keeps producing syntactically plausible SMILES while optimizing, so the gains are not just a side effect of generating nonsense strings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: since the same learned surrogate supplies both the reward and the reported docking numbers, the most direct test of the paper's central claim is to re-score the generated molecules with the actual docking program; the paper does not report such a check.
  • Editorial inference: the advantage-preference construction does not depend on SMILES tokenization, so it should transfer to graph-based molecular generative models or synthesis-planning policies with only the critics changed.
  • Editorial inference: the best-of-N completion term makes partial-molecule advantage estimates optimistic by construction, so part of the measured densification gain could come from sampling bias rather than from better policy learning; separating the two would require comparing best-of-N-completed rewards against full-sequence rewards under the same policy.
  • Editorial inference: if surrogate optimism is concentrated in exactly the region the policy explores, interleaving SPO updates with occasional true docking scores would turn the method into an active-learning loop and could correct for the drift.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes DRUGIMPROVER, an LLM-based framework for drug optimization, and SPO, a structured policy optimization algorithm that fine-tunes the LLM using an advantage preference comparing generated analogs with the original drug across multiple critics (druglikeness, solubility, synthesizability, docking). The authors report improvements over several baselines on SARS-CoV-2 (3CLPro) and cancer (RTCB) benchmarks, provide an ablation of the partial-molecule component, and present two theoretical lemmas supporting the SPO objective. They also release a dataset of one million docked compounds and plan to release code.

Significance. If the empirical claims hold, the paper would be a useful contribution to RL-based drug optimization: it addresses a practically important task (optimizing existing drugs rather than de novo design), introduces a plausible policy-gradient objective with partial-molecule rollouts, and releases a large docking dataset that could benefit the community. The paper also deserves credit for comparing against six REINVENT 4 variants and several other baselines, and for reporting wall-clock times. However, the central docking-improvement claim currently rests on evaluation with the same fitted surrogate that supplies the reward, and the theoretical lemmas are conditional on an unverified best-of-N assumption; these issues need to be resolved before the claims can be accepted as stated.

major comments (5)
  1. [§4.2, Eq. (11); Table 1; Appendix A.5] The reported docking improvements are circular: the docking score used as a reward component in Eq. (11) is the transformer surrogate described in Appendix A.5, and the same surrogate appears to produce the Docking column in Table 1. With validation R2 = 0.842 for 3CLPro and 0.73 for RTCB, the surrogate may be optimistic on the chemical region explored by the policy, so the headline claim that DRUGIMPROVER generates analogs with better docking than the originals and baselines is not established. Please evaluate the generated molecules with the actual OEDOCK/FRED docking program, or at minimum with an independent surrogate not used in the reward, and report surrogate calibration on the generated distribution.
  2. [Table 1 caption; §6.1] All reported results are restricted to generated molecules with Tanimoto similarity above 0.6. This filter can exclude the most aggressively optimized molecules and may differentially affect methods that explore lower similarity; without unfiltered statistics or a justification that the filter is not responsible for the observed gains, the comparative claims are vulnerable. Please report the fraction of generated molecules excluded and provide unfiltered results for all metrics.
  3. [Eq. (15); Definition 4.1] The partial-molecule advantage r^AP_BON(j) is under-specified. It is not stated how the best-of-N rollouts are drawn when the partial SMILES is invalid or cannot be completed, how the N candidates are formed, or whether the X and Y partial sequences are aligned by token position; moreover, Eq. (5) defines BON as an arg-max over rewards while Eq. (15) writes an expectation over BON completions. Please clarify the exact sampling procedure used in the experiments, since this term is central to the SPO update in Eq. (17).
  4. [§5, Lemma 5.1; Appendix C] The proof of Lemma 5.1 appears to contain an inequality with the wrong direction: from J(π) ≥ J0(π) and J0(π*) ≥ J(π) it does not follow that J0(π*) − J(π) ≥ (1/2)(J0(π*) − J0(π)); the latter would require an additional bound that is not proved. In addition, the lemma is conditional on the BON strict-improvement assumption in Eq. (19), which is not verified for the trained policy. Please correct the proof or state precisely under what verified conditions SPO and the standard RL objective share the same optimizers.
  5. [Eq. (11)] The reward weighting is inconsistent as written. If the list C(yT) contains the four non-similarity critics, then the total weight on those critics is 4(1−β)/5, so the coefficients do not sum to 1 except at β = 1; if C includes Tanimoto, then Tanimoto is double-counted. Please state the intended normalization and give the exact reward formula used to produce Tables 1 and 2.
minor comments (5)
  1. [Eq. (10)] The pretraining loss is written as λ·NLL(X,Y)/((1−λ)·Similarity(X,Y)); this form is unusual and likely a typo for a weighted sum such as λ·NLL + (1−λ)·(1−Similarity) or similar. Please clarify the exact loss used.
  2. [Appendix A.7, Table 5] There is a discrepancy between the hyperparameter table ('# of Molecules Optimized 256') and the text in Appendix A.7 ('A selection of 1280 molecules from each of the RTCB and 3CLPro datasets ... is used for SPO finetuning'). Please reconcile these numbers.
  3. [Table 5] The handling of invalid generated SMILES is listed ambiguously as '[0, −Rc(X)]' with two alternatives described in the text; please specify which option was used for the results in Tables 1 and 2 and report the invalid-SMILES rate per method.
  4. [Abstract and §1] The prose contains several grammatical and typographical errors (e.g., 'introduce a novel reinforcement learning algorithm', 'while retains the beneficial chemical properties'), which should be corrected in a thorough language edit.
  5. [Fig. 2] Figure 2 shows Tanimoto similarity over epochs without error bars or standard deviations despite the text referring to five experimental runs; please add uncertainty information or state why it is omitted.

Circularity Check

1 steps flagged · score 6.0 of 10

Reported docking improvements are read from the same fitted surrogate that serves as the docking critic in the SPO reward, making the central docking prediction an output of the optimized model rather than an independent OEDOCK validation.

  1. fitted input called prediction [Section 4.2 (Docking Score and Eq. 11); Appendix A.5; Table 1 and Section 6.1]
    "Docking Score: The docking score assesses the drug’s potential to bind and inhibit the target site. To enable efficient computation, we employ a docking surrogate model (See Appendix A.5) to output this score. ... The validation r2 values are 0.842 for 3CLPro and 0.73 for the RTCB dataset. ... Table 1 demonstrates that the DRUGIMPROVER algorithm outperforms all competing baselines, including the original and six variants from the current leading method, REINVENT 4, across most performance metrics for both viral and cancer-related benchmarks."

    The CDocking term entering the SPO reward in Eq. (11) is the same fitted transformer surrogate described in Appendix A.5. The policy gradient (17) is trained to maximize Rc, which contains CDocking, and the Docking column in Table 1 (plus the AvgNormReward headline) is then computed from that same CDocking. Hence the reported 'improved docking' of generated molecules is an output of the fitted model used as the training reward, not an OEDOCK measurement of the generated SMILES. The moderate R2 does not remove the identity: the comparison is surrogate-vs-surrogate, so the docking prediction is, by the paper's own equations, a projection of the optimized reward function rather than an external test.

full rationale

The paper contains one central circular step: the docking evaluation is the same fitted surrogate that is optimized inside SPO. Eq. (11) defines the reward with CDocking supplied by a surrogate model, and Section 6 uses the metrics introduced in section 4.2 as evaluation metrics, so the Docking column and AvgNormReward in Table 1 are values of the trained reward function. This makes the headline docking claim partially reduce to the fitted input by construction. The non-docking critics (druglikeness, synthesizability, solubility) come from RDKit and are not circular, and the comparison against baselines under the same surrogate retains some independent algorithmic content, so the paper is not entirely circular. The theoretical lemmas do not repair this: Lemma 5.1 explicitly conditions on the unverified BON-improvement inequality (19), and Lemma 5.2 is a standard policy-gradient rewrite. Self-citations to the same group's prior work provide the dataset and the TOPPK/reward construction, but these are not the load-bearing circular step here. Score 6 reflects one central 'prediction' that reduces to the fitted reward function by construction.

Assumptions & free parameters 7 free parameters · 6 assumptions · 0 invented entities

The central claim rests on four domain assumptions (Tanimoto as similarity, surrogate as docking, equal critics as quality, valid partial rollouts), one ad hoc BON-improvement assumption, and standard policy-gradient math, plus several tunable reward and decoding parameters. No new physical entities are introduced.

free parameters (7)
  • Tanimoto reward weight beta = range [0.2, 0.4, 0.6, 0.8] listed in Table 5; final value not specified
    Controls trade-off between similarity preservation and property improvement in Eq. 11.
  • Pretraining loss weight lambda = lambda in (0,1), no exact value given
    Balances NLL and similarity penalty in Eq. 10; no concrete value is reported.
  • Top-K, Top-P, Best-of-N = TopK [10,15,20], TopP [0.85,0.9,0.95], BON [4,6,8]; exact values not given
    Decoding and rollout hyperparameters used in TOP-PK and BON; the specific values behind Table 1 are not stated.
  • Min-max normalization bounds = [-10, 10]
    Norm 2 scaling of critics in Eq. 11; underlying min and max are not described.
  • SPO finetuning molecule selection = 1280 molecules with docking scores from -14 to -6
    Defines the RL training distribution; arbitrary selection criterion based on prior work.
  • Evaluation Tanimoto filter = 0.6
    Table 1 caption reports results only for molecules with Tanimoto similarity above 0.6; unfiltered results are not shown.
  • Invalid SMILES advantage handling = [0, -Rc(X)]
    Two options are listed in Table 5; the final choice used for main results is not specified.
assumptions (6)
  • ad hoc to paper Best-of-N rollouts strictly improve over suboptimal molecules (Eq. 19)
    Used in Lemma 5.1 to prove that SPO and standard reward objectives share maximizers; not verified empirically in the paper.
  • domain assumption SMILES prefix rollouts define a meaningful partial-molecule improvement signal
    Definition 4.1 rolls out from prefixes X_1:j and Y_1:j; it is not specified how original-molecule prefixes are completed or why this is a valid baseline.
  • domain assumption Tanimoto similarity from Morgan fingerprints captures chemical similarity and property preservation
    Used in pretraining pair selection and reward Eq. 11; a standard cheminformatics assumption.
  • domain assumption Surrogate docking model generalizes to SPO-generated molecules
    Appendix A.5 reports validation R2 of 0.842 and 0.73 on held-out docked molecules; no direct docking on generated molecules is reported.
  • domain assumption The four critics with equal weights are a valid proxy for drug quality
    Reward Eq. 11 assigns equal weight to normalized docking, drug-likeness, synthesizability, and solubility after removing the Tanimoto share.
  • standard math Score function identity for conditional language model distributions
    Lemma 5.2 uses E[grad log pi(Y_t+1:T | Y_1:t,X) | Y_1:t,X] = 0, the standard zero-mean property of score functions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DrugImproverGPT: A Large Language Model for Drug Optimization with Fine-Tuning via Structured Policy Optimization." pith.science (2026). https://pith.science/paper/HJ4SO7VP

@misc{pith2026250207237,
  author       = {Pith},
  title        = {Pith review of: DrugImproverGPT: A Large Language Model for Drug Optimization with Fine-Tuning via Structured Policy Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HJ4SO7VP}},
  note         = {Machine review of arXiv:2502.07237}
}
read the original abstract

Finetuning a Large Language Model (LLM) is crucial for generating results towards specific objectives. This research delves into the realm of drug optimization and introduce a novel reinforcement learning algorithm to finetune a drug optimization LLM-based generative model, enhancing the original drug across target objectives, while retains the beneficial chemical properties of the original drug. This work is comprised of two primary components: (1) DrugImprover: A framework tailored for improving robustness and efficiency in drug optimization. It includes a LLM designed for drug optimization and a novel Structured Policy Optimization (SPO) algorithm, which is theoretically grounded. This algorithm offers a unique perspective for fine-tuning the LLM-based generative model by aligning the improvement of the generated molecule with the input molecule under desired objectives. (2) A dataset of 1 million compounds, each with OEDOCK docking scores on 5 human proteins associated with cancer cells and 24 binding sites from SARS-CoV-2 virus. We conduct a comprehensive evaluation of SPO and demonstrate its effectiveness in improving the original drug across target properties. Our code and dataset will be publicly available at: https://github.com/xuefeng-cs/DrugImproverGPT.

Figures

Figures reproduced from arXiv: 2502.07237 by the authors.

Figure 1
Figure 1. DRUGIMPROVER framework. It comprises two major components: (1) A large language model designed for drug optimization. (2) A Structured Policy Optimization (SPO) algorithm aims to fine-tune the LLM-based generator for drug improvement across desired properties. V π (s), which represents the expected reward at a given state s under policy π. The value function can be expressed as follows: V (s) = Ea∼πθ(s) [Q(s, a)] = … view at source ↗
Figure 3
Figure 3. The binding sites of proteins 3CLPro (PDB ID: 7BQY) ( [PITH_FULL_IMAGE:figures/full_fig_p017_3.png] view at source ↗
Figure 4
Figure 4. Training corpus example and visualization [PITH_FULL_IMAGE:figures/full_fig_p020_4.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

88 extracted references · 56 canonical work pages

  1. [1]

    De novo drug design using reinforcement learning with graph-based deep generative models

    Sara Romeo Atance, Juan Viguera Diez, Ola Engkvist, Simon Olsson, and Rocío Mercado. De novo drug design using reinforcement learning with graph-based deep generative models. Journal of Chemical Information and Modeling, 62(20):4863–4872, 2022. 3, 5

    2022

  2. [2]

    DrugCentral 2023 extends human clinical data and integrates veterinary drugs

    Sorin Avram, Thomas B Wilson, Ramona Curpan, Liliana Halip, Ana Borota, Alina Bora, Cristian G Bologa, Jayme Holmes, Jeffrey Knockel, Jeremy J Yang, et al. DrugCentral 2023 extends human clinical data and integrates veterinary drugs. Nucleic Acids Research, 51(D1): D1276–D1287, 2023. 1

    2023

  3. [3]

    MolGPT: Molecular generation using a transformer-decoder model

    Viraj Bagal, Rishal Aggarwal, PK Vinod, and U Deva Priyakumar. MolGPT: Molecular generation using a transformer-decoder model. Journal of Chemical Information and Modeling, 62(9):2064–2076, 2021. 3

    2021

  4. [4]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862,

    arXiv

  5. [5]

    Constitutional ai: Harmlessness from ai feedback

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022. 3

    arXiv 2022

  6. [6]

    Why is tanimoto index an appropriate choice for fingerprint-based similarity calculations? Journal of cheminformatics, 7:1–13, 2015

    Dávid Bajusz, Anita Rácz, and Károly Héberger. Why is tanimoto index an appropriate choice for fingerprint-based similarity calculations? Journal of cheminformatics, 7:1–13, 2015. 5

    2015

  7. [7]

    Molecular similarity: a key technique in molecular informatics

    Andreas Bender and Robert C Glen. Molecular similarity: a key technique in molecular informatics. Organic & biomolecular chemistry, 2(22):3204–3218, 2004. 2

    2004

  8. [8]

    Better rewards yield better summaries: Learning to summarise without references

    Florian Böhm, Yang Gao, Christian M Meyer, Ori Shapira, Ido Dagan, and Iryna Gurevych. Better rewards yield better summaries: Learning to summarise without references. arXiv preprint arXiv:1909.01214, 2019. 3

    arXiv 1909

Show all 88 references
  1. [9]

    Paccmannrl: De novo generation of hit-like anticancer molecules from transcriptomic data via reinforcement learning

    Jannis Born, Matteo Manica, Ali Oskooei, Joris Cadow, Greta Markert, and María Rodríguez Martínez. Paccmannrl: De novo generation of hit-like anticancer molecules from transcriptomic data via reinforcement learning. Iscience, 24(4), 2021. 2, 3

    2021

  2. [10]

    Transformers and large language models for chemistry and drug discovery

    Andres M Bran and Philippe Schwaller. Transformers and large language models for chemistry and drug discovery. arXiv preprint arXiv:2310.06083, 2023. 3

    2023 arXiv

  3. [11]

    Offline rl without off-policy evaluation

    David Brandfonbrener, Will Whitney, Rajesh Ranganath, and Joan Bruna. Offline rl without off-policy evaluation. Advances in neural information processing systems, 34:4933–4946, 2021. 7

    2021

  4. [12]

    Safe learning in robotics: From learning-based control to safe reinforcement learning

    Lukas Brunke, Melissa Greeff, Adam W Hall, Zhaocong Yuan, Siqi Zhou, Jacopo Panerati, and Angela P Schoellig. Safe learning in robotics: From learning-based control to safe reinforcement learning. Annual Review of Control, Robotics, and Autonomous Systems, 5:411–444, 2022. 2

    2022

  5. [13]

    Policy improvement via imitation of multiple oracles

    Ching-An Cheng, Andrey Kolobov, and Alekh Agarwal. Policy improvement via imitation of multiple oracles. Advances in Neural Information Processing Systems, 33:5587–5598, 2020. 7 11

    2020

  6. [14]

    Deep reinforcement learning from human preferences

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017. 3

    2017

  7. [15]

    Ai-accelerated protein-ligand docking for sars-cov-2 is 100-fold faster with no significant change in detection

    Austin Clyde, Xuefeng Liu, Thomas Brettin, Hyunseung Yoo, Alexander Partin, Yadu Babuji, Ben Blaiszik, Jamaludin Mohd-Yusof, Andre Merzky, Matteo Turilli, et al. Ai-accelerated protein-ligand docking for sars-cov-2 is 100-fold faster with no significant change in detection. Sc...

    2023

  8. [16]

    The cost of new drug discovery and development

    Michael Dickson and Jean Paul Gagnon. The cost of new drug discovery and development. Discovery medicine, 4(22):172–179, 2009. 1

    2009

  9. [17]

    Raft: Reward ranked finetuning for generative foundation model alignment

    Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767, 2023. 3

    2023 arXiv

  10. [18]

    Estimation of synthetic accessibility score of drug-like molecules based on molecular complexity and fragment contributions

    Peter Ertl and Ansgar Schuffenhauer. Estimation of synthetic accessibility score of drug-like molecules based on molecular complexity and fragment contributions. Journal of cheminfor- matics, 1:1–11, 2009. 6

    2009

  11. [19]

    Neural scaling of deep chemical models

    Nathan C Frey, Ryan Soklaski, Simon Axelrod, Siddharth Samsi, Rafael Gomez-Bombarelli, Connor W Coley, and Vijay Gadepally. Neural scaling of deep chemical models. Nature Machine Intelligence, 5(11):1297–1305, 2023. 3

    2023

  12. [20]

    Mimosa: Multi-constraint molecule sampling for molecule optimization

    Tianfan Fu, Cao Xiao, Xinhao Li, Lucas M Glass, and Jimeng Sun. Mimosa: Multi-constraint molecule sampling for molecule optimization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 125–133, 2021. 3, 8, 9, 19

    2021

  13. [21]

    A new algorithm for data compression

    Philip Gage. A new algorithm for data compression. The C Users Journal, 12(2):23–38, 1994. 9

    1994

  14. [22]

    Scaling laws for reward model overoptimization

    Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages 10835–10866. PMLR, 2023. 4

    2023

  15. [23]

    Learning to navigate the synthetically accessible chemical space using reinforcement learning

    Sai Krishna Gottipati, Boris Sattarov, Sufeng Niu, Yashaswi Pathak, Haoran Wei, Shengchao Liu, Simon Blackburn, Karam Thomas, Connor Coley, Jian Tang, et al. Learning to navigate the synthetically accessible chemical space using reinforcement learning. In International confere...

    2020

  16. [24]

    Objective-reinforced generative adversarial networks (ORGAN) for sequence generation models

    Gabriel Lima Guimaraes, Benjamin Sanchez-Lengeling, Carlos Outeiral, Pedro Luis Cunha Farias, and Alán Aspuru-Guzik. Objective-reinforced generative adversarial networks (ORGAN) for sequence generation models. arXiv preprint arXiv:1705.10843, 2017. 2, 3

    2017 arXiv

  17. [25]

    Covid-19 vaccines and variants of concern: A review

    Ikbel Hadj Hassine. Covid-19 vaccines and variants of concern: A review. Reviews in medical virology, 32(4):e2313, 2022. 1

    2022

  18. [26]

    Learning from dialogue after deployment: Feed yourself, chatbot! arXiv preprint arXiv:1901.05415, 2019

    Braden Hancock, Antoine Bordes, Pierre-Emmanuel Mazare, and Jason Weston. Learning from dialogue after deployment: Feed yourself, chatbot! arXiv preprint arXiv:1901.05415, 2019. 3

    1901 arXiv

  19. [27]

    Molecular optimization by capturing chemist’s intuition using deep neural networks

    Jiazhen He, Huifang You, Emil Sandström, Eva Nittinger, Esben Jannik Bjerrum, Christian Tyr- chan, Werngard Czechtizky, and Ola Engkvist. Molecular optimization by capturing chemist’s intuition using deep neural networks. Journal of cheminformatics, 13(1):1–17, 2021. 3, 9, 17, 19, 20

    2021

  20. [28]

    Transformer-based molecular optimization beyond matched molecular pairs

    Jiazhen He, Eva Nittinger, Christian Tyrchan, Werngard Czechtizky, Atanas Patronov, Es- ben Jannik Bjerrum, and Ola Engkvist. Transformer-based molecular optimization beyond matched molecular pairs. Journal of cheminformatics, 14(1):18, 2022. 3, 9, 17, 19, 20

    2022

  21. [29]

    Reward learning from human preferences and demonstrations in atari

    Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. Advances in neural information processing systems, 31, 2018. 3

    2018

  22. [30]

    The distribution of the flora in the alpine zone

    Paul Jaccard. The distribution of the flora in the alpine zone. 1. New phytologist, 11(2):37–50,

  23. [31]

    Way off-policy batch deep reinforcement learning of implicit human preferences in dialog

    Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint arXiv:1907.00456, 2019. 3

    1907 arXiv

  24. [32]

    A graph-based genetic algorithm and generative model/monte carlo tree search for the exploration of chemical space

    Jan H Jensen. A graph-based genetic algorithm and generative model/monte carlo tree search for the exploration of chemical space. Chemical science, 10(12):3567–3572, 2019. 3

    2019

  25. [33]

    Multi-objective molecule generation using interpretable substructures

    Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Multi-objective molecule generation using interpretable substructures. In International conference on machine learning, pages 4849–4859. PMLR, 2020. 2, 3

    2020

  26. [34]

    Posit: flexible shape-guided docking for pose prediction

    Brian P Kelley, Scott P Brown, Gregory L Warren, and Steven W Muchmore. Posit: flexible shape-guided docking for pose prediction. Journal of Chemical Information and Modeling, 55 (8):1771–1780, 2015. 2

    2015

  27. [35]

    Giraffe: Using deep reinforcement learning to play chess

    Matthew Lai. Giraffe: Using deep reinforcement learning to play chess. arXiv preprint arXiv:1509.01549, 2015. 2

    2015 arXiv

  28. [36]

    RDkit: Open-source cheminformatics software

    Greg Landrum et al. RDkit: Open-source cheminformatics software. https://www.rdkit. org. Accessed Oct 2023. 6

    2023

  29. [37]

    Rdkit: A software suite for cheminformatics, computational chemistry, and predictive modeling

    Greg Landrum et al. Rdkit: A software suite for cheminformatics, computational chemistry, and predictive modeling. Greg Landrum, 8(31.10):5281, 2013. 5

    2013

  30. [38]

    Scalable agent alignment via reward modeling: A research direction

    Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: A research direction. arXiv preprint arXiv:1811.07871,

    arXiv

  31. [39]

    Drugimprover: Utilizing reinforcement learning for multi-objective alignment in drug optimization

    Xuefeng Liu, Songhao Jiang, Archit Vasan, Alexander Brace, Ozan Gokdemir, Thomas Bret- tin, and Fangfang Xia. Drugimprover: Utilizing reinforcement learning for multi-objective alignment in drug optimization. In NeurIPS 2023 Workshop on New Frontiers of AI for Drug Discovery a...

    2023

  32. [40]

    Blend- ing imitation and reinforcement learning for robust policy improvement

    Xuefeng Liu, Takuma Yoneda, Rick L Stevens, Matthew R Walter, and Yuxin Chen. Blend- ing imitation and reinforcement learning for robust policy improvement. arXiv preprint arXiv:2310.01737, 2023. 7

    2023 arXiv

  33. [41]

    Active policy improvement from multiple black-box oracles

    Xuefeng Liu, Takuma Yoneda, Chaoqi Wang, Matthew R Walter, and Yuxin Chen. Active policy improvement from multiple black-box oracles. In Proceedings of the International Conference on Machine Learning (ICML), pages 22320–22337, 2023. 3, 7

    2023

  34. [42]

    Entropy-reinforced planning with large language models for de novo drug discovery

    Xuefeng Liu, Chih-Chan Tien, Peng Ding, Songhao Jiang, and Stevens Rick. Entropy-reinforced planning with large language models for de novo drug discovery. ICML, 2024. 3, 4, 6, 17, 18

    2024

  35. [43]

    Drugex v3: scaffold-constrained drug design with graph transformer-based reinforcement learning

    Xuhan Liu, Kai Ye, Herman WT van Vlijmen, Adriaan P IJzerman, and Gerard JP van Westen. Drugex v3: scaffold-constrained drug design with graph transformer-based reinforcement learning. Journal of Cheminformatics, 15(1):24, 2023. 3, 8, 9, 19

    2023

  36. [44]

    Reinvent 4: Modern ai–driven generative molecule design

    Hannes H Loeffler, Jiazhen He, Alessandro Tibo, Jon Paul Janet, Alexey V oronov, Lewis H Mervin, and Ola Engkvist. Reinvent 4: Modern ai–driven generative molecule design. Journal of Cheminformatics, 16(1):20, 2024. 3, 8, 9, 19, 20

    2024

  37. [45]

    The different mechanisms of cancer drug resistance: a brief review

    Behzad Mansoori, Ali Mohammadi, Sadaf Davudian, Solmaz Shirjang, and Behzad Baradaran. The different mechanisms of cancer drug resistance: a brief review. Advanced pharmaceutical bulletin, 7(3):339, 2017. 1

    2017

  38. [46]

    Playing atari with deep reinforcement learning

    V olodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013. 2

    2013 arXiv

  39. [47]

    Exploring deep recurrent models with reinforcement learning for molecule design

    Daniel Neil, Marwin Segler, Laura Guasch, Mohamed Ahmed, Dean Plumbley, Matthew Sellwood, and Nathan Brown. Exploring deep recurrent models with reinforcement learning for molecule design. In ICLR, 2018. 2, 3 13

    2018

  40. [48]

    Molecular de-novo design through deep reinforcement learning

    Marcus Olivecrona, Thomas Blaschke, Ola Engkvist, and Hongming Chen. Molecular de-novo design through deep reinforcement learning. Journal of cheminformatics, 9(1):1–14, 2017. 3

    2017

  41. [49]

    Advantage-weighted regression: Simple and scalable off-policy reinforcement learning

    Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177, 2019. 7

    1910 arXiv

  42. [50]

    Combinatorial enumeration of groups, graphs, and chemical compounds

    Georg Polya and Ronald C Read. Combinatorial enumeration of groups, graphs, and chemical compounds. Springer Science & Business Media, 2012. 2

    2012

  43. [51]

    Alvinn: An autonomous land vehicle in a neural network

    Dean A Pomerleau. Alvinn: An autonomous land vehicle in a neural network. Advances in neural information processing systems, 1, 1988. 3

    1988

  44. [52]

    Deep reinforcement learning for de novo drug design

    Mariya Popova, Olexandr Isayev, and Alexander Tropsha. Deep reinforcement learning for de novo drug design. Science advances, 4(7):eaap7885, 2018. 3, 5

    2018

  45. [53]

    Drug repurposing: Progress, challenges and recommendations

    Sudeep Pushpakom, Francesco Iorio, Patrick A Eyers, K Jane Escott, Shirley Hopper, Andrew Wells, Andrew Doig, Tim Guilliams, Joanna Latimer, Christine McNamee, Alan Norris, Philippe Sanseau, David Cavalla, and Munir Pirmohamed. Drug repurposing: Progress, challenges and recomm...

    2019

  46. [54]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290, 2023. 3

    2023 arXiv

  47. [55]

    Learning by playing solving sparse reward tasks from scratch

    Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Wiele, Vlad Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing solving sparse reward tasks from scratch. In International conference on machine learning , pages 4344–4353. P...

    2018

  48. [56]

    Extended-connectivity fingerprints

    David Rogers and Mathew Hahn. Extended-connectivity fingerprints. Journal of chemical information and modeling, 50(5):742–754, 2010. 6

    2010

  49. [57]

    Reinforcement and imitation learning via interactive no-regret learning

    Stephane Ross and J Andrew Bagnell. Reinforcement and imitation learning via interactive no-regret learning. arXiv preprint arXiv:1406.5979, 2014. 3, 7

    2014 arXiv

  50. [58]

    A reduction of imitation learning and structured prediction to no-regret online learning

    Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth interna- tional conference on artificial intelligence and statistics, pages 627–635. JMLR Workshop and C...

    2011

  51. [59]

    C5T5: Control- lable generation of organic molecules with transformers

    Daniel Rothchild, Alex Tamkin, Julie Yu, Ujval Misra, and Joseph Gonzalez. C5T5: Control- lable generation of organic molecules with transformers. arXiv preprint arXiv:2108.10307,

    arXiv

  52. [60]

    Neural machine translation of rare words with subword units

    Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909, 2015. 9

    2015 arXiv

  53. [61]

    Preference ranking optimization for human alignment

    Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. Preference ranking optimization for human alignment. arXiv preprint arXiv:2306.17492, 2023. 3

    2023 arXiv

  54. [62]

    Deep reinforcement learning for multiparameter optimization in de novo drug design

    Niclas Ståhl, Goran Falkman, Alexander Karlsson, Gunnar Mathiason, and Jonas Bostrom. Deep reinforcement learning for multiparameter optimization in de novo drug design. Journal of chemical information and modeling, 59(7):3166–3176, 2019. 3

    2019

  55. [63]

    ZINC15–ligand discovery for everyone

    Teague Sterling and John J Irwin. ZINC15–ligand discovery for everyone. Journal of Chemical Information and Modeling, 55(11):2324–2337, 2015. 5

    2015

  56. [64]

    Learning to summarize with human feedback

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea V oss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021, 2020. 3 14

    2020

  57. [65]

    Molsearch: search-based multi-objective molecular generation and property optimization

    Mengying Sun, Jing Xing, Han Meng, Huijun Wang, Bin Chen, and Jiayu Zhou. Molsearch: search-based multi-objective molecular generation and property optimization. In Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining, pages 4724–4732,

  58. [66]

    Policy gradient meth- ods for reinforcement learning with function approximation

    Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999. 6

    1999

  59. [67]

    Reinforcement learning for systems pharmacology-oriented and personalized drug design

    Ryan K Tan, Yang Liu, and Lei Xie. Reinforcement learning for systems pharmacology-oriented and personalized drug design. Expert Opinion on Drug Discovery, 17(8):849–863, 2022. 3

    2022

  60. [68]

    Drlinker: Deep reinforcement learning for optimization in fragment linking design

    Youhai Tan, Lingxue Dai, Weifeng Huang, Yinfeng Guo, Shuangjia Zheng, Jinping Lei, Hong- ming Chen, and Yuedong Yang. Drlinker: Deep reinforcement learning for optimization in fragment linking design. Journal of Chemical Information and Modeling, 62(23):5907–5917,

  61. [69]

    DeepMind control suite

    Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy Lillicrap, and Martin Riedmiller. DeepMind control suite. arXiv preprint arXiv:1801.00690, 2018. 2

    2018 arXiv

  62. [70]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. 3

    2023 arXiv

  63. [71]

    Matched molecular pair analysis in short: algorithms, applications and limitations

    Christian Tyrchan and Emma Evertsson. Matched molecular pair analysis in short: algorithms, applications and limitations. Computational and structural biotechnology journal, 15:86–90,

  64. [72]

    Benchmarking language-based docking models

    Archit Vasan, Rick Stevens, Arvind Ramanathan, and Vishwanath Venkatram. Benchmarking language-based docking models. 2023. 18

    2023

  65. [73]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. 5

    2017

  66. [74]

    Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards

    Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint arXiv:170...

    2017 arXiv

  67. [75]

    A reinforcement learning approach for protein–ligand binding pose prediction

    Chenran Wang, Yang Chen, Yuan Zhang, Keqiao Li, Menghan Lin, Feng Pan, Wei Wu, and Jinfeng Zhang. A reinforcement learning approach for protein–ligand binding pose prediction. BMC bioinformatics, 23(1):1–18, 2022. 3

    2022

  68. [76]

    Smiles, a chemical language and information system

    David Weininger. Smiles, a chemical language and information system. 1. introduction to methodology and encoding rules. Journal of chemical information and computer sciences, 28 (1):31–36, 1988. 2, 4, 5

    1988

  69. [77]

    Simple statistical gradient-following algorithms for connectionist reinforce- ment learning

    Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine learning, 8(3):229–256, 1992. 5, 19

    1992

  70. [78]

    Recursively summarizing books with human feedback

    Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. Recursively summarizing books with human feedback. arXiv preprint arXiv:2109.10862, 2021. 3

    2021 arXiv

  71. [79]

    Rlcg: When reinforce- ment learning meets coarse graining

    Shenghao Wu, Tianyi Liu, Zhirui Wang, Wen Yan, and Yingxiang Yang. Rlcg: When reinforce- ment learning meets coarse graining. In NeurIPS 2022 AI for Science: Progress and Promises,

    2022

  72. [80]

    Towards coherent and engaging spoken dialog response generation using automatic conversation evaluators

    Sanghyun Yi, Rahul Goel, Chandra Khatri, Alessandra Cervone, Tagyoung Chung, Behnam Hedayatnia, Anu Venkatesh, Raefer Gabriel, and Dilek Hakkani-Tur. Towards coherent and engaging spoken dialog response generation using automatic conversation evaluators. arXiv preprint arXiv:1...

    1904 arXiv

  73. [81]

    Population-based de novo molecule generation, using grammatical evolution

    Naruki Yoshikawa, Kei Terayama, Masato Sumita, Teruki Homma, Kenta Oono, and Koji Tsuda. Population-based de novo molecule generation, using grammatical evolution. Chemistry Letters, 47(11):1431–1434, 2018. 3

    2018

  74. [82]

    Graph convolutional policy network for goal-directed molecular graph generation

    Jiaxuan You, Bowen Liu, Zhitao Ying, Vijay Pande, and Jure Leskovec. Graph convolutional policy network for goal-directed molecular graph generation. Advances in neural information processing systems, 31, 2018. 3

    2018

  75. [83]

    Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning

    Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pages 1094–1100. PMLR, 2020. 2

    2020

  76. [84]

    Rrhf: Rank responses to align language models with human feedback without tears

    Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint arXiv:2304.05302, 2023. 3

    2023 arXiv

  77. [85]

    Covid-19 pathophysiology: A review

    Koichi Yuki, Miho Fujiogi, and Sophia Koutsogiannaki. Covid-19 pathophysiology: A review. Clinical immunology, 215:108427, 2020. 1

    2020

  78. [86]

    Universal approach to de novo drug design for target proteins using deep reinforcement learning

    Yunjiang Zhang, Shuyuan Li, Miaojuan Xing, Qing Yuan, Hong He, and Shaorui Sun. Universal approach to de novo drug design for target proteins using deep reinforcement learning. ACS omega, 8(6):5464–5474, 2023. 2, 3, 5

    2023

  79. [87]

    Optimization of molecules via deep reinforcement learning

    Zhenpeng Zhou, Steven Kearnes, Li Li, Richard N Zare, and Patrick Riley. Optimization of molecules via deep reinforcement learning. Scientific reports, 9(1):10752, 2019. 3

    2019

  80. [88]

    Fine-tuning language models from human preferences

    Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. 3 16 A Appendix A.1 Pre-training and fine-tuning dataset We u...

    1909 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.