Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

Scalable Autoregressive 3D Molecule Generation

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Quetzal builds 3D molecules atom-by-atom and matches diffusion quality.

desk verdict Solid autoregressive alternative to diffusion for 3D molecules; core generation and speed claims hold, but NLL comparisons and flexible-task generality need sharpening. read the letter →

arxiv 2505.13791 v2 pith:57O6P2HH submitted 2025-05-20 cs.LG physics.chem-ph

Austin H. Cheng , Chong Sun , Alán Aspuru-Guzik This is my paper · ORCID
classification cs.LGphysics.chem-ph
keywords autoregressivegeneration3DmoleculediffusionlosscausaltransformerDiffMLPhydrogendecorationscaffoldcompletionexactlikelihood
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Quetzal treats each 3D molecule as an ordered sequence of atoms and generates it one atom at a time. A causal transformer predicts the next atom's type, while a small per-atom diffusion model, the DiffMLP, predicts the continuous 3D position conditioned on the already-built prefix. The paper claims this simple hybrid closes most of the quality gap between autoregressive and diffusion 3D molecule generation: on QM9 and GEOM it beats all autoregressive baselines and reaches diffusion-level validity, while sampling substantially faster. Because the coordinate diffusion model acts on a single 3D position at a time, exact likelihood scores are tractable, and variable-size tasks like hydrogen decoration and scaffold completion work without any architectural change.

What carries the argument

The load-bearing object is the Diffusion Loss, applied as a per-token Diffusion MLP (DiffMLP) that models the distribution of the next atom's 3D position. The DiffMLP is a conditional denoising diffusion model over a single coordinate vector, trained with denoising score matching and sampled with the Heun integrator on geometrically spaced time steps. A causal transformer processes all prefixes in one forward pass to produce prefix embeddings; a second transformer stack incorporates the next atom type and produces a conditioning vector; the DiffMLP then maps noisy coordinates, timestep, and conditioning vector to the clean position. This separation means only O(n) transformer forward passes are needed per molecule, while all diffusion steps go through a cheap MLP, which is what makes fast sampling and exact 3x3 Jacobian likelihood computation tractable.

What would settle it

Permute the atom order of molecules in the QM9 test set, or reorder the bare heavy-atom scaffolds before hydrogen decoration, and measure generation quality or decoration accuracy. The paper itself reports xyz2mol validity falling from 99.1% to 80.7% under random atomic permutations; if a different ordering scheme restores high validity, then the ordering is a modeling choice rather than a hard limitation.

Watch

Extended reading notes

Core claim

The central claim is that a standard causal transformer combined with Diffusion Loss—a per-token diffusion model for continuous coordinates—is enough to make autoregressive 3D molecule generation competitive with state-of-the-art diffusion models. Quetzal factorizes the molecule distribution into a categorical next-atom-type prediction from a GPT-style transformer and a continuous next-position prediction from a lightweight diffusion MLP over 3D coordinates. On QM9, Quetzal reaches xyz2mol validity of 98.6% and validity-by-uniqueness of 94.0%, ahead of previous autoregressive models and comparable to or better than strong diffusion baselines. On GEOM, it is the first autoregressive model demonstrated at that scale, with lookup validity-by-uniqueness of 95.3% and the best reported negative log-likelihood of -313.63. It also samples 22.5x faster than SymDiff on QM9 at 30 diffusion steps, and 128x faster than EDM on GEOM. The same trained model performs hydrogen decoration with 99.8% correct hydrogen counts and low RMSD, and completes molecular scaffolds, because generation is conditioned on an arbitrary prefix.

Load-bearing premise

The model depends on a consistent, chemically localized atom order inherited from .xyz files, and if real inputs arrive in a different order, generation quality collapses.

Editorial extensions

If this is right

  • Autoregressive models can be made competitive with diffusion models for 3D molecular generation without discretizing coordinates or using equivariant layers.
  • Generation speed improves by one to two orders of magnitude because the expensive transformer runs once per atom rather than once per diffusion step.
  • Exact per-atom log-likelihood becomes computable, enabling direct likelihood-based model comparison and potential use in importance sampling.
  • Because the model accepts arbitrary-size inputs and outputs, tasks like hydrogen decoration and scaffold completion can be solved with the same checkpoint and no retraining.
  • The approach scales with standard transformer infrastructure, so larger models and datasets can be leveraged directly.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the ordering assumption transfers to other ordered 3D data, the same architecture could apply to proteins or crystal structures where a canonical atom order exists, but the paper only tests molecules with inherited .xyz ordering.
  • Exact per-token likelihood opens a concrete path to importance-sampled Boltzmann generators and reward-based finetuning, which the paper names as future work.
  • A testable extension would be learning or inferring the atom generation order (for example via masked diffusion), removing the reliance on fixed .xyz order; the paper explicitly flags this as the main open problem.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces Quetzal, an autoregressive generative model for 3D molecules. The model treats a molecule as an ordered sequence of atoms and factorizes p(M) = product over i of ptype(a_{i+1} | prefix) pcoord(x_{i+1} | prefix, a_{i+1}). A causal transformer predicts the next atom type, and a per-token Diffusion MLP trained with the DiffLoss objective of Li et al. (2024a) predicts the continuous next position. The authors report state-of-the-art results among autoregressive baselines, near-diffusion-level quality on QM9 and GEOM, faster sampling, exact per-atom divergence-based likelihoods, and zero-training adaptation to hydrogen decoration and scaffold completion. The paper includes ablation studies and openly released code.

Significance. If the central claims hold, this is a meaningful step for autoregressive 3D molecule generation: it shows that a simple hybrid of causal transformer and per-token diffusion can close much of the quality gap with equivariant diffusion models while offering exact likelihoods and flexible output sizes. Strengths include the clean decomposition, the use of standard benchmarks with several metrics and multiple evaluation runs, the frank disclosure of the model's order sensitivity and of the GEOM split issue, and the public code. The main reservations concern the strength of the generality claims in light of the demonstrated order dependence and the interpretability of the GEOM NLL result under a leaky split.

major comments (3)
  1. [Section 5, Section 4.4, Table 4 (Appendix B.2)] The paper correctly discloses that Quetzal is not permutation-invariant and that random atom permutations lower xyz2mol validity from 99.1% to 80.7% and molecule stability from 87.6% to 25.9%. However, the abstract and Section 3 claim that Quetzal 'accepts any 3D structure as input' and 'natively handles variable-size tasks.' These claims are stronger than the evidence supports, since hydrogen decoration and scaffold completion are demonstrated only for the specific atom orderings seen in the training data (e.g., hydrogens last in QM9). Please either add an experiment showing robustness under realistic input orderings (e.g., RDKit canonical order, PDB order) or qualify the claims to say the model requires the training distribution's atom order. This issue is load-bearing for the paper's third contribution.
  2. [Section 4.3 and Appendix B] The GEOM evaluation uses a random split in which up to 30 conformers per molecule are distributed across train and test; the paper itself notes that 'most molecules in the test set have conformers that are seen in the training set.' Consequently the reported GEOM NLL (-313.63) and generation metrics may largely reflect memorization rather than generalization. I ask for an additional evaluation on a molecule-disjoint split or, at minimum, a clear statement that the reported numbers are not a test of generalization to unseen molecules. Without this, the 'state-of-the-art NLL on GEOM' claim is not fully supportable.
  3. [Table 1 and Section 3.1] The NLL column in Table 1 mixes different estimators across models: Quetzal numbers come from the exact per-atom ODE likelihood computed with Ndiff=60, while baseline numbers are copied from prior papers and may be ELBOs or use different solvers. Please report the estimator and discretization for every entry, or restrict the comparison to models evaluated with the same protocol. Also, the phrase 'exact likelihood' in Section 3.1 should be qualified as exact only in the limit of the ODE integrator, since Ndiff=60 is a finite discretization.
minor comments (5)
  1. [Throughout] There are recurring formatting errors from missing spaces (e.g., 'we presentQuetzal', 'we proposeQuetzal', 'Quetzaluses') and some table headers are cramped; please fix these presentation issues.
  2. [Table 2 caption] The caption states 'We assume uniqueness is 100%.' This assumption should be justified, since it directly affects the reported validity x uniqueness values for the baselines.
  3. [Section 4.2] The paper explains Quetzal's reduced validity x uniqueness on QM9 as 'signs of overfitting' on the fixed atom orderings. Consider adding an explicit analysis, such as nearest-neighbor distances to the training set, to support this interpretation rather than leaving it as a hypothesis.
  4. [Section 4.5] The scaffold completion results are purely qualitative and the paper defers quantitative evaluation to future work. Please clearly label this section as a demonstration rather than an evaluated capability, or provide a quantitative metric.
  5. [Section 5] Given that 'we need to know the best order in which to generate atoms' is described as a central limitation, a short discussion of possible remedies (e.g., order learning, canonicalization, or order-robust training) would strengthen the paper.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's claims are empirical evaluations of a trained model, with standard diffusion and autoregressive objectives credited to external prior work.

full rationale

Quetzal's central claims are empirical: a causal transformer plus a per-token Diffusion MLP is trained on QM9/GEOM and evaluated on held-out samples by validity, uniqueness, stability, NLL, and speed. Equations (1)-(10) are the standard autoregressive factorization and the standard denoising score-matching objective; Eq. (9) explicitly credits the Diffusion Loss to Li et al. (2024a), an external prior work, and the ODE likelihood formula in Eq. (11) is the standard change-of-variables result from Chen et al. (2018), Song et al. (2020a), and Karras et al. (2022). No hyperparameter or fitted constant is tuned to the reported evaluation metrics and then relabeled as a prediction; the reported NLL and validity numbers are post-hoc measurements on held-out data. The paper's self-citations (e.g., Flam-Shepherd and Aspuru-Guzik 2023) appear only in related work and are not load-bearing. The atom-ordering dependence is candidly disclosed in Sections 4.4, 5, and B.2 as a limitation; it concerns generalization scope rather than circularity. The hydrogen-decoration success relies on QM9's '.xyz' ordering with hydrogens last, but this is an empirical conditional claim, not an equivalence between an input and an output.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claims are empirical, so the ledger tracks dataset and ordering assumptions rather than physical constants. The main load-bearing choices are the fixed .xyz atom ordering, the hydrogen-last property of QM9, and the use of data augmentation in place of architectural equivariance. Standard diffusion theory is treated as background. I list N_diff and sigma_data as hand-selected hyperparameters because they directly affect reported results, but I do not treat standard Karras defaults as invented inputs.

free parameters (4)
  • Learned model weights (transformer + DiffMLP) = 165M parameters (86M transformer, 79M DiffMLP)
    Fitted by AdamW on QM9 and GEOM; they determine behavior and are not derived analytically, but they are standard learned parameters, not ad hoc constants.
  • N_diff (diffusion steps per atom at inference) = 30 (speed test), 60 (QM9), 120 (GEOM and NLL)
    Chosen per task; controls the quality and speed tradeoff and the reported likelihood estimates.
  • sigma_data = 1.4 (QM9), 2.5 (GEOM)
    Coordinate standard deviation used in diffusion preconditioning; set to dataset statistics, not derived from the model.
  • DiffMLP timestep sampling distribution = ln t ~ N(-1.2, 1.2^2)
    Training hyperparameter from Karras et al. (2022); affects the denoising loss weighting and learned score.
assumptions (5)
  • domain assumption The .xyz atom ordering provides a learnable, localized generation order.
    Section 3 defines the ordered sequence; ablation B.2 shows random permutations collapse validity, so this is load-bearing.
  • domain assumption Hydrogens are last in the QM9 .xyz files used for hydrogen decoration.
    Section 4.4: Quetzal's hydrogen-decorating success relies on this; reordering the bare molecule degrades performance.
  • domain assumption Random rotations and translations during training are sufficient substitutes for architectural equivariance.
    Section 3; ablation B.2 without translations and rotations drops xyz2mol validity from 99.1 to 63.0.
  • standard math Standard score-matching and probability-flow ODE theory applies to the per-atom coordinate model.
    Section 3, Equations 5 through 9 rely on established diffusion theory; no novel derivation is attempted.
  • domain assumption GEOM test conformers are valid evaluation targets despite possible train overlap.
    Section 4.3 notes up to 30 conformers per molecule and random splitting, so test molecules may have near-identical training conformers; this inflates NLL and uniqueness estimates.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Scalable Autoregressive 3D Molecule Generation." pith.science (2026). https://pith.science/paper/57O6P2HH

@misc{pith2026250513791,
  author       = {Pith},
  title        = {Pith review of: Scalable Autoregressive 3D Molecule Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/57O6P2HH}},
  note         = {Machine review of arXiv:2505.13791}
}
read the original abstract

Generative models of 3D molecular structure play a rapidly growing role in the design and simulation of molecules. Diffusion models currently dominate the space of 3D molecule generation, while autoregressive models have trailed behind. In this work, we present Quetzal, a simple but scalable autoregressive model that builds molecules atom-by-atom in 3D. Treating each molecule as an ordered sequence of atoms, Quetzal combines a causal transformer that predicts the next atom's discrete type with a smaller Diffusion MLP that models the continuous next-position distribution. Compared to existing autoregressive baselines, Quetzal achieves substantial improvements in generation quality and is competitive with the performance of state-of-the-art diffusion models. In addition, by reducing the number of expensive forward passes through a dense transformer, Quetzal enables significantly faster generation speed, as well as exact divergence-based likelihood computation. Finally, without any architectural changes, Quetzal natively handles variable-size tasks like hydrogen decoration and scaffold completion. We hope that our work motivates a perspective on scalability and generality for generative modelling of 3D molecules.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Atomic Design Transformer: Scaffold-Conditioned 3D Molecule Generation via xTB-Reward Reinforcement Learning

    physics.comp-ph 2026-07 conditional novelty 6.0 of 10

    A plain causal transformer that tokenizes atom positions in local frames generates 3D molecules directly; RL against an xTB relaxation reward lifts topology-preserving valid yield from ~50% to ~95%.

  2. InertialAR: Autoregressive 3D Molecule Generation with Inertial Frames

    cs.LG 2025-10 conditional novelty 6.0 of 10

    An autoregressive transformer with inertial-frame tokenization and geometric rotary positional encoding reports state-of-the-art validity and stability on QM9, GEOM-Drugs, and B3LYP, plus strong functional-group-condi...

Reference graph

Works this paper leans on

88 extracted references · 16 canonical work pages · cited by 2 Pith papers

  1. [1]

    Accurate structure prediction of biomolecular interactions with alphafold 3

    Josh Abramson, Jonas Adler, Jack Dunger, Richard Evans, Tim Green, Alexander Pritzel, Olaf Ronneberger, Lindsay Willmore, Andrew J Ballard, Joshua Bambrick, et al. Accurate structure prediction of biomolecular interactions with alphafold 3. Nature, pages 1--3, 2024

    2024

  2. [2]

    De novo design of protein structure and function with rfdiffusion

    Joseph L Watson, David Juergens, Nathaniel R Bennett, Brian L Trippe, Jason Yim, Helen E Eisenach, Woody Ahern, Andrew J Borst, Robert J Ragotte, Lukas F Milles, et al. De novo design of protein structure and function with rfdiffusion. Nature, 620 0 (7976): 0 1089--1100, 2023

    2023

  3. [3]

    A generative model for inorganic materials design

    Claudio Zeni, Robert Pinsler, Daniel Z \"u gner, Andrew Fowler, Matthew Horton, Xiang Fu, Zilong Wang, Aliaksandra Shysheya, Jonathan Crabb \'e , Shoko Ueda, et al. A generative model for inorganic materials design. Nature, pages 1--3, 2025

    2025

  4. [4]

    Equivariant diffusion for molecule generation in 3d

    Emiel Hoogeboom, V ctor Garcia Satorras, Cl \'e ment Vignac, and Max Welling. Equivariant diffusion for molecule generation in 3d. In International conference on machine learning, pages 8867--8887. PMLR, 2022

    2022

  5. [5]

    Equivariant flow matching with hybrid probability transport for 3d molecule generation

    Yuxuan Song, Jingjing Gong, Minkai Xu, Ziyao Cao, Yanyan Lan, Stefano Ermon, Hao Zhou, and Wei-Ying Ma. Equivariant flow matching with hybrid probability transport for 3d molecule generation. Advances in Neural Information Processing Systems, 36, 2024 a

    2024

  6. [6]

    Symdiff: Equivariant diffusion via stochastic symmetrisation

    Leo Zhang, Kianoosh Ashouritaklimi, Yee Whye Teh, and Rob Cornish. Symdiff: Equivariant diffusion via stochastic symmetrisation. arXiv preprint arXiv:2410.06262, 2024

    arXiv 2024

  7. [7]

    All-atom diffusion transformers: Unified generative modelling of molecules and materials

    Chaitanya K Joshi, Xiang Fu, Yi-Lun Liao, Vahe Gharakhanyan, Benjamin Kurt Miller, Anuroop Sriram, and Zachary W Ulissi. All-atom diffusion transformers: Unified generative modelling of molecules and materials. arXiv preprint arXiv:2503.03965, 2025

    arXiv 2025

  8. [8]

    Symmetry-adapted generation of 3d point sets for the targeted discovery of molecules

    Niklas Gebauer, Michael Gastegger, and Kristof Sch\" u tt. Symmetry-adapted generation of 3d point sets for the targeted discovery of molecules. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d Alch\' e -Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https://proceedi...

    2019

Show all 88 references
  1. [9]

    An autoregressive flow model for 3d molecular geometry generation from scratch

    Youzhi Luo and Shuiwang Ji. An autoregressive flow model for 3d molecular geometry generation from scratch. In International conference on learning representations (ICLR), 2022

    2022

  2. [10]

    Symphony: Symmetry-equivariant point-centered spherical harmonics for molecule generation

    Ameya Daigavane, Song Kim, Mario Geiger, and Tess Smidt. Symphony: Symmetry-equivariant point-centered spherical harmonics for molecule generation. arXiv preprint arXiv:2311.16199, 2023

    2023 arXiv

  3. [11]

    Language models can generate molecules, materials, and protein binding sites directly in three dimensions as xyz, cif, and pdb files

    Daniel Flam-Shepherd and Al \'a n Aspuru-Guzik. Language models can generate molecules, materials, and protein binding sites directly in three dimensions as xyz, cif, and pdb files. arXiv preprint arXiv:2305.05708, 2023

    2023 arXiv

  4. [12]

    Tokenizing 3d molecule structure with quantized spherical coordinates

    Kaiyuan Gao, Yusong Wang, Haoxiang Guan, Zun Wang, Qizhi Pei, John E Hopcroft, Kun He, and Lijun Wu. Tokenizing 3d molecule structure with quantized spherical coordinates. arXiv preprint arXiv:2412.01564, 2024

    2024 arXiv

  5. [13]

    Autoregressive image generation without vector quantization

    Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. arXiv preprint arXiv:2406.11838, 2024 a

    2024 arXiv

  6. [14]

    E (n) equivariant normalizing flows

    Victor Garcia Satorras, Emiel Hoogeboom, Fabian Fuchs, Ingmar Posner, and Max Welling. E (n) equivariant normalizing flows. Advances in Neural Information Processing Systems, 34: 0 4181--4192, 2021

    2021

  7. [15]

    Unified generative modeling of 3d molecules via bayesian flow networks

    Yuxuan Song, Jingjing Gong, Yanru Qu, Hao Zhou, Mingyue Zheng, Jingjing Liu, and Wei-Ying Ma. Unified generative modeling of 3d molecules via bayesian flow networks. arXiv preprint arXiv:2403.15441, 2024 b

    2024 arXiv

  8. [16]

    Geometric latent diffusion models for 3d molecule generation

    Minkai Xu, Alexander S Powers, Ron O Dror, Stefano Ermon, and Jure Leskovec. Geometric latent diffusion models for 3d molecule generation. In International Conference on Machine Learning, pages 38592--38610. PMLR, 2023

    2023

  9. [17]

    Geometric representation condition improves equivariant molecule generation

    Zian Li, Cai Zhou, Xiyuan Wang, Xingang Peng, and Muhan Zhang. Geometric representation condition improves equivariant molecule generation. arXiv preprint arXiv:2410.03655, 2024 b

    2024 arXiv

  10. [18]

    Fast 3d molecule generation via unified geometric optimal transport

    Haokai Hong, Wanyu Lin, and Kay Chen Tan. Fast 3d molecule generation via unified geometric optimal transport. arXiv preprint arXiv:2405.15252, 2024

    2024 arXiv

  11. [19]

    3d molecule generation by denoising voxel grids

    Pedro O O Pinheiro, Joshua Rackers, Joseph Kleinhenz, Michael Maser, Omar Mahmood, Andrew Watkins, Stephen Ra, Vishnu Sresht, and Saeed Saremi. 3d molecule generation by denoising voxel grids. Advances in Neural Information Processing Systems, 36, 2024

    2024

  12. [20]

    Score-based 3d molecule generation with neural fields

    Matthieu Kirchmeyer, Pedro O Pinheiro, and Saeed Saremi. Score-based 3d molecule generation with neural fields. arXiv preprint arXiv:2501.08508, 2025

    2025 arXiv

  13. [21]

    Generating 3 D molecules for target protein binding

    Meng Liu, Youzhi Luo, Kanji Uchino, Koji Maruhashi, and Shuiwang Ji. Generating 3 D molecules for target protein binding. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference o...

    2022

  14. [22]

    Fine-tuned language models generate stable inorganic materials as text

    Nate Gruver, Anuroop Sriram, Andrea Madotto, Andrew Gordon Wilson, C Lawrence Zitnick, and Zachary Ulissi. Fine-tuned language models generate stable inorganic materials as text. arXiv preprint arXiv:2402.04379, 2024

    2024 arXiv

  15. [23]

    Bindgpt: A scalable framework for 3d molecular design via language modeling and reinforcement learning

    Artem Zholus, Maksim Kuznetsov, Roman Schutski, Rim Shayakhmetov, Daniil Polykovskiy, Sarath Chandar, and Alex Zhavoronkov. Bindgpt: A scalable framework for 3d molecular design via language modeling and reinforcement learning. arXiv preprint arXiv:2406.03686, 2024

    2024 arXiv

  16. [24]

    Large language models are innate crystal structure generators

    Jingru Gan, Peichen Zhong, Yuanqi Du, Yanqiao Zhu, Chenru Duan, Haorui Wang, Carla P Gomes, Kristin A Persson, Daniel Schwalbe-Koda, and Wei Wang. Large language models are innate crystal structure generators. arXiv preprint arXiv:2502.20933, 2025

    2025

  17. [25]

    3dsmiles-gpt: 3d molecular pocket-based generation with token-only large language model

    Jike Wang, Hao Luo, Rui Qin, Mingyang Wang, Xiaozhe Wan, Meijing Fang, Odin Zhang, Qiaolin Gou, Qun Su, Chao Shen, et al. 3dsmiles-gpt: 3d molecular pocket-based generation with token-only large language model. Chemical Science, 16 0 (2): 0 637--648, 2025

    2025

  18. [26]

    Autoregressive denoising diffusion models for multivariate probabilistic time series forecasting

    Kashif Rasul, Calvin Seward, Ingmar Schuster, and Roland Vollgraf. Autoregressive denoising diffusion models for multivariate probabilistic time series forecasting. In International conference on machine learning, pages 8857--8868. PMLR, 2021

    2021

  19. [27]

    Diffusion forcing: Next-token prediction meets full-sequence diffusion

    Boyuan Chen, Diego Marti Monso, Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitzmann. Diffusion forcing: Next-token prediction meets full-sequence diffusion. arXiv preprint arXiv:2407.01392, 2024

    2024 arXiv

  20. [28]

    Jetformer: An autoregressive generative model of raw images and text

    Michael Tschannen, Andr \'e Susano Pinto, and Alexander Kolesnikov. Jetformer: An autoregressive generative model of raw images and text. arXiv preprint arXiv:2411.19722, 2024

    2024 arXiv

  21. [29]

    Trans-dimensional generative modeling via jump diffusion models

    Andrew Campbell, William Harvey, Christian Weilbach, Valentin De Bortoli, Thomas Rainforth, and Arnaud Doucet. Trans-dimensional generative modeling via jump diffusion models. Advances in Neural Information Processing Systems, 36, 2024

    2024

  22. [30]

    Fourier features let networks learn high frequency functions in low dimensional domains

    Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in neural information proc...

    2020

  23. [31]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017

    2017

  24. [32]

    Elucidating the design space of diffusion-based generative models

    Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems, 35: 0 26565--26577, 2022

    2022

  25. [33]

    Score-based generative modeling through stochastic differential equations

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020 a

    2011 arXiv

  26. [34]

    A connection between score matching and denoising autoencoders

    Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation, 23 0 (7): 0 1661--1674, 2011

    2011

  27. [35]

    Tweedie’s formula and selection bias

    Bradley Efron. Tweedie’s formula and selection bias. Journal of the American Statistical Association, 106 0 (496): 0 1602--1614, 2011

    2011

  28. [36]

    Film: Visual reasoning with a general conditioning layer

    E Perez, F Strub, H De Vries, V Dumoulin, and A Courville. Film: Visual reasoning with a general conditioning layer. arxiv. arXiv preprint arXiv:1709.07871, 2017

    2017 arXiv

  29. [37]

    Scalable diffusion models with transformers

    William S Peebles and Saining Xie. Scalable diffusion models with transformers. 2023 ieee. In CVF International Conference on Computer Vision (ICCV), volume 4172, 2022

    2023

  30. [38]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. arxiv e-prints. arXiv preprint arXiv:1512.03385, 10, 2015

    2015 arXiv

  31. [39]

    Flashattention-2: Faster attention with better parallelism and work partitioning

    Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691, 2023

    2023 arXiv

  32. [40]

    Pytorch 2: Faster machine learning through dynamic python bytecode transformation and graph compilation

    Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky, Bin Bao, Peter Bell, David Berard, Evgeni Burovski, et al. Pytorch 2: Faster machine learning through dynamic python bytecode transformation and graph compilation. In Proceedings of the...

    2024

  33. [41]

    Bio2token: All-atom tokenization of any biomolecular structure with mamba

    Andrew Liu, Axel Elaldi, Nathan Russell, and Olivia Viessmann. Bio2token: All-atom tokenization of any biomolecular structure with mamba. arXiv preprint arXiv:2410.19110, 2024

    2024 arXiv

  34. [42]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1 0 (8): 0 9, 2019

    2019

  35. [43]

    karpathy/ nanoGPT , January 2025

    Andrej Karpathy. karpathy/ nanoGPT , January 2025. URL https://github.com/karpathy/nanoGPT. original-date: 2022-12-28T00:51:12Z

    2025

  36. [44]

    Palm: Scaling language modeling with pathways

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24 0 (240): 0 1--113, 2023

    2023

  37. [45]

    Small-scale proxies for large-scale transformer training instabilities

    Mitchell Wortsman, Peter J Liu, Lechao Xiao, Katie Everett, Alex Alemi, Ben Adlam, John D Co-Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, et al. Small-scale proxies for large-scale transformer training instabilities. arXiv preprint arXiv:2309.14322, 2023

    2023 arXiv

  38. [46]

    Efficient sequence packing without cross-contamination: Accelerating large language models without impacting performance

    Mario Michael Krell, Matej Kosec, Sergio P Perez, and Andrew Fitzgibbon. Efficient sequence packing without cross-contamination: Accelerating large language models without impacting performance. arXiv preprint arXiv:2107.02027, 2021

    2021 arXiv

  39. [47]

    Neural ordinary differential equations

    Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018

    2018

  40. [48]

    A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines

    Michael F Hutchinson. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communications in Statistics-Simulation and Computation, 18 0 (3): 0 1059--1076, 1989

    1989

  41. [49]

    Quantum chemistry structures and properties of 134 kilo molecules

    Raghunathan Ramakrishnan, Pavlo O Dral, Matthias Rupp, and O Anatole Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data, 1 0 (1): 0 1--7, 2014

    2014

  42. [50]

    GEOM , energy-annotated molecular conformations for property prediction and molecular generation

    Simon Axelrod and Rafael Gomez-Bombarelli. GEOM , energy-annotated molecular conformations for property prediction and molecular generation. Scientific Data, 9 0 (1): 0 1--14, 2022

    2022

  43. [51]

    E (n) equivariant graph neural networks

    V ctor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E (n) equivariant graph neural networks. In International conference on machine learning, pages 9323--9332. PMLR, 2021

    2021

  44. [52]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020 b

    2010 arXiv

  45. [53]

    Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps

    Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35: 0 5775--5787, 2022

    2022

  46. [54]

    Geometry-complete diffusion for 3d molecule generation and optimization

    Alex Morehead and Jianlin Cheng. Geometry-complete diffusion for 3d molecule generation and optimization. Communications Chemistry, 7 0 (1): 0 150, 2024

    2024

  47. [55]

    Practical suggestions for better crystal structures

    Peter M \"u ller. Practical suggestions for better crystal structures. Crystallography Reviews, 15 0 (1): 0 57--83, 2009

    2009

  48. [56]

    Olex2: a complete structure solution, refinement and analysis program

    Oleg V Dolomanov, Luc J Bourhis, Richard J Gildea, Judith AK Howard, and Horst Puschmann. Olex2: a complete structure solution, refinement and analysis program. Journal of applied crystallography, 42 0 (2): 0 339--341, 2009

    2009

  49. [57]

    Open babel: An open chemical toolbox

    Noel M O'Boyle, Michael Banck, Craig A James, Chris Morley, Tim Vandermeersch, and Geoffrey R Hutchison. Open babel: An open chemical toolbox. Journal of cheminformatics, 3: 0 1--14, 2011

    2011

  50. [58]

    Adding hydrogen atoms to molecular models via fragment superimposition

    Patrick Kunzmann, Jacob Marcel Anter, and Kay Hamacher. Adding hydrogen atoms to molecular models via fragment superimposition. Algorithms for Molecular Biology, 17 0 (1): 0 7, 2022

    2022

  51. [59]

    Diffdec: structure-aware scaffold decoration with an end-to-end diffusion model

    Junjie Xie, Sheng Chen, Jinping Lei, and Yuedong Yang. Diffdec: structure-aware scaffold decoration with an end-to-end diffusion model. Journal of Chemical Information and Modeling, 64 0 (7): 0 2554--2564, 2024

    2024

  52. [60]

    Transformer language models without positional encodings still learn positional information

    Adi Haviv, Ori Ram, Ofir Press, Peter Izsak, and Omer Levy. Transformer language models without positional encodings still learn positional information. arXiv preprint arXiv:2203.16634, 2022

    2022 arXiv

  53. [61]

    The impact of positional encoding on length generalization in transformers

    Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. The impact of positional encoding on length generalization in transformers. Advances in Neural Information Processing Systems, 36, 2024

    2024

  54. [62]

    Causal language modeling can elicit search and reasoning capabilities on logic puzzles

    Kulin Shah, Nishanth Dikkala, Xin Wang, and Rina Panigrahy. Causal language modeling can elicit search and reasoning capabilities on logic puzzles. arXiv preprint arXiv:2409.10502, 2024

    2024 arXiv

  55. [63]

    Do large language models need sensory grounding for meaning and understanding

    Yann LeCun. Do large language models need sensory grounding for meaning and understanding. In Workshop on Philosophy of Deep Learning, NYU Center for Mind, Brain, and Consciousness and the Columbia Center for Science and Society, 2023

    2023

  56. [64]

    The pitfalls of next-token prediction

    Gregor Bachmann and Vaishnavh Nagarajan. The pitfalls of next-token prediction. arXiv preprint arXiv:2403.06963, 2024

    2024 arXiv

  57. [65]

    Train for the worst, plan for the best: Understanding token ordering in masked diffusions

    Jaeyeon Kim, Kulin Shah, Vasilis Kontonis, Sham Kakade, and Sitan Chen. Train for the worst, plan for the best: Understanding token ordering in masked diffusions. arXiv preprint arXiv:2502.06768, 2025

    2025 arXiv

  58. [66]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35: 0 24824--24837, 2022

    2022

  59. [67]

    Training large language models to reason in a continuous latent space

    Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason in a continuous latent space. arXiv preprint arXiv:2412.06769, 2024

    2024 arXiv

  60. [68]

    Boltzmann generators: Sampling equilibrium states of many-body systems with deep learning

    Frank No \'e , Simon Olsson, Jonas K \"o hler, and Hao Wu. Boltzmann generators: Sampling equilibrium states of many-body systems with deep learning. Science, 365 0 (6457): 0 eaaw1147, 2019

    2019

  61. [69]

    Equivariant flow matching

    Leon Klein, Andreas Kr \"a mer, and Frank No \'e . Equivariant flow matching. Advances in Neural Information Processing Systems, 36: 0 59886--59910, 2023

    2023

  62. [70]

    Transferable boltzmann generators

    Leon Klein and Frank No \'e . Transferable boltzmann generators. arXiv preprint arXiv:2406.14426, 2024

    2024 arXiv

  63. [71]

    Scalable equilibrium sampling with sequential boltzmann generators

    Charlie B Tan, Avishek Joey Bose, Chen Lin, Leon Klein, Michael M Bronstein, and Alexander Tong. Scalable equilibrium sampling with sequential boltzmann generators. arXiv preprint arXiv:2502.18462, 2025

    2025

  64. [72]

    Dual use of artificial-intelligence-powered drug discovery

    Fabio Urbina, Filippa Lentzos, C \'e dric Invernizzi, and Sean Ekins. Dual use of artificial-intelligence-powered drug discovery. Nature machine intelligence, 4 0 (3): 0 189--191, 2022

    2022

  65. [73]

    Python reference manual, volume 111

    Guido Van Rossum, Fred L Drake, et al. Python reference manual, volume 111. Centrum voor Wiskunde en Informatica Amsterdam, 1995

    1995

  66. [74]

    Python for scientific computing

    Travis E Oliphant. Python for scientific computing. Computing in science & engineering, 9 0 (3): 0 10--20, 2007

    2007

  67. [75]

    Py T orch: An imperative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Py T orch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32, 2019

    2019

  68. [76]

    PyTorch Lightning , March 2019

    William Falcon and The PyTorch Lightning team . PyTorch Lightning , March 2019. URL https://github.com/Lightning-AI/lightning

    2019

  69. [77]

    Scalfani, guillaume godin, Juuso Lehtivarjo, Rachel Walker, Axel Pahl, Francois Berenger, jasondbiggs, and strets123

    Greg Landrum, Paolo Tosco, Brian Kelley, Ric, David Cosgrove, sriniker, gedeck, Riccardo Vianello, NadineSchneider, Eisuke Kawashima, Dan N, Gareth Jones, Andrew Dalke, Brian Cole, Matt Swain, Samo Turk, AlexanderSavelyev, Alain Vaucher, Maciej Wójcikowski, Ichiru Take, Daniel...

    2023

  70. [78]

    3Dmol .js: Molecular visualization with WebGL

    Nicholas Rego and David Koes. 3Dmol .js: Molecular visualization with WebGL . Bioinformatics, 31 0 (8): 0 1322--1324, 2015

    2015

  71. [79]

    Jupyter notebooks-a publishing format for reproducible computational workflows

    Thomas Kluyver, Benjamin Ragan-Kelley, Fernando P \'e rez, Brian E Granger, Matthias Bussonnier, Jonathan Frederic, Kyle Kelley, Jessica B Hamrick, Jason Grout, Sylvain Corlay, et al. Jupyter notebooks-a publishing format for reproducible computational workflows. Elpub, 2016: ...

    2016

  72. [80]

    Matplotlib: A 2d graphics environment

    John D Hunter. Matplotlib: A 2d graphics environment. Computing in science & engineering, 9 0 (03): 0 90--95, 2007

    2007

  73. [81]

    Seaborn: statistical data visualization

    Michael L Waskom. Seaborn: statistical data visualization. Journal of Open Source Software, 6 0 (60): 0 3021, 2021

    2021

  74. [82]

    Harris, K

    Charles R. Harris, K. Jarrod Millman, Stéfan J van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fer...

    2020

  75. [83]

    Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, St \'e fan J

    Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, St \'e fan J. van der Walt , Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nels...

    2020

  76. [84]

    pandas-dev/pandas: Pandas

    The pandas development team . pandas-dev/pandas: Pandas . URL https://github.com/pandas-dev/pandas

  77. [85]

    Flex attention: A programming model for generating optimized attention kernels

    Juechu Dong, Boyuan Feng, Driss Guessous, Yanbo Liang, and Horace He. Flex attention: A programming model for generating optimized attention kernels. arXiv preprint arXiv:2412.05496, 2024

    2024 arXiv

  78. [86]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  79. [87]

    Analyzing and improving the training dynamics of diffusion models

    Tero Karras, Miika Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Analyzing and improving the training dynamics of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24174--24184, 2024

    2024

  80. [88]

    Universal structure conversion method for organic molecules: from atomic connectivity to three-dimensional geometry

    Yeonjoon Kim and Woo Youn Kim. Universal structure conversion method for organic molecules: from atomic connectivity to three-dimensional geometry. Bulletin of the Korean Chemical Society, 36 0 (7): 0 1769--1777, 2015

    2015

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.