Pith. sign in

REVIEW 3 major objections 9 minor 50 references

A standard transformer and attention with linear biases for molecular conformer generation

T0 review · 3 major / 9 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A standard transformer with ALiBi-style linear graph-distance biases reaches higher conformer recall on GEOM-DRUGS at 25M parameters than a 64M non-equivariant baseline.

desk verdict S23D's recall numbers over MCF-B look real as reported, but the paper over-attributes the gain to its positional encoding when other architectural changes are bundled in; still worth peer review. read the letter →

arxiv 2506.19834 v1 pith:FYDHI4NK submitted 2025-06-24 q-bio.BM cs.AIcs.LG

Viatcheslav Gurev , Timothy Rumbell This is my paper · ORCID
classification q-bio.BMcs.AIcs.LG
keywords molecularconformergenerationtransformerALiBipositionalencodingrelativediffusionmodelGEOM-DRUGSnon-equivariantchiralitycorrection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the main reason non-equivariant transformer models for molecular conformer generation need so many parameters is a poorly chosen positional encoding. It introduces S23D, a standard LLaMA-style transformer whose attention subtracts a head-specific slope times the shortest-path distance between atoms, analogous to ALiBi in NLP. At 24.8M parameters it reports recall coverage of 84.6% and AMR of 0.412 on GEOM-DRUGS, beating the 64M MCF baseline at 84.0% and 0.427 without chirality features, and with a post hoc chirality correction it reaches 87.0% recall coverage. If the comparison is right, a cheap fixed graph bias supplies enough inductive bias to make non-equivariant transformers competitive at a fraction of the size.

What carries the argument

The load-bearing object is the ALiBi-style linear graph bias inserted into every multi-head attention layer. In place of Graphormer's learnable bias lookup, S23D uses $A^{\mathrm{head}}_{ij} = (h_i W^{\mathrm{head}}_Q)(h_j W^{\mathrm{head}}_K)^T/\sqrt{d_{\mathrm{head}}} - m_{\mathrm{head}}\phi(v_i,v_j)$, where $\phi(v_i,v_j)$ is the shortest-path distance between atoms in the molecular graph and $m_{\mathrm{head}}$ follows ALiBi's geometric slope schedule. This fixed bias costs nothing to learn, involves no lookup table, and lets different heads specialize: heads with large slopes can encode graph structure while heads with near-zero slopes can process atom coordinates. The rest of the architecture is a standard LLaMA transformer block split into a graph encoder and a structural subnetwork, with coordinate encodings injected after the first normalization layer and a VPSDE diffusion score network on top.

What would settle it

Take the S23D-B-1/13 model and training protocol exactly as described but replace the ALiBi shortest-path bias with MCF's eigenvector positional encoding; if recall coverage stays around 84.6% instead of dropping toward the eigenvector-PE results in the ablation, then the linear bias is not the causal ingredient. The complementary check is to add the same linear bias to the MCF architecture without any of the other changes and see whether it reaches S23D-B's numbers.

Watch

Extended reading notes

Core claim

The central claim is that a well-chosen relative positional encoding, rather than model scale, is what lets a non-equivariant diffusion transformer compete with larger and more specialized models for conformer generation. The paper implements relative encoding as a negative attention bias linear in graph shortest-path distance, with head-specific slopes taken from the ALiBi geometric sequence. With this bias, S23D-B-1/13 at 25M parameters surpasses MCF-B at 64M parameters on GEOM-DRUGS recall metrics and comes within 0.1% recall coverage of MCF-L at 242M parameters, establishing a new recall SOTA among models under 64M parameters. The paper also shows that a two-stage protocol of hydrogen-free pretraining followed by finetuning on complete molecules is viable, and that a simple RDKit chirality correction improves both recall and precision, narrowing the precision gap relative to the equivariant ET-Flow model.

Load-bearing premise

The claim that a well-chosen positional encoding removes the need for a large non-equivariant model rests on the assumption that the other simultaneous differences between S23D and MCF—backbone, graph encoder, two-stage training, augmentation, coordinate encodings, and training budget—are not the actual source of the benchmark improvement.

Editorial extensions

If this is right

  • S23D-B-1/13 (24.8M parameters) reports recall coverage of 84.6% and AMR of 0.412 on GEOM-DRUGS without chirality features, beating the 64M MCF-B baseline at 84.0% COV and 0.427 AMR.
  • With a post hoc RDKit chirality correction, the same model reaches 87.0% recall coverage and 0.380 AMR, outperforming the 242M MCF-L on recall while using about a tenth of the parameters.
  • The two-stage hydrogen-free pretraining protocol reduces the number of atom tokens by roughly 44% in the first stage, and a hydrogen-free model already surpasses the previous small-model recall SOTA.
  • The ALiBi-style linear bias is about 35% faster to train than the learnable shortest-path bias of Graphormer and gives comparable or better metrics in the ablation, and S23D is several times faster per diffusion step than MCF at similar batch sizes.
  • Chirality correction increases recall coverage by more than 2 points and precision coverage by roughly 6 points for the small model, showing that chirality handling needs to be reported for fair cross-model comparison.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The headline comparison changes several ingredients at once—backbone, graph encoder, two-stage training, O(3) augmentation, coordinate encoders, and training budget—so the paper's attribution of the gain to the positional encoding would only be confirmed by an ablation that changes only the PE while holding the rest fixed.
  • If the linear shortest-path bias is the active ingredient, it should transfer to equivariant backbones and to other molecular graph tasks; a cheap test would be adding the same bias to a flow-matching or message-passing model and measuring the parameter savings.
  • The ALiBi slope schedule is copied from language modeling; a graph-specific schedule, for example one that also depends on degree or on the diameter of the molecule, is a natural extension that could improve the already small models.
  • The authors note that current GEOM benchmark splits are random rather than scaffold-based, so a scaffold-split evaluation would clarify whether the parameter efficiency holds on molecules that are less similar to training data.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 9 minor

Summary. This paper introduces S23D, a non-equivariant diffusion model for molecular conformer generation built from a LLaMA-style transformer backbone. The main technical proposal is a relative positional encoding implemented as a negative attention bias that grows linearly with the shortest-path distance between graph nodes, using head-specific slopes analogous to ALiBi. The model is trained with a two-stage protocol that first pretrains on hydrogen-free molecules and then finetunes on complete molecules, and a post-hoc chirality correction is applied in some variants. On GEOM-DRUGS, the S23D-B model (24.8M parameters) is reported to surpass the 64M-parameter MCF-B baseline on recall coverage and AMR (COV-R 84.6 vs 84.0; AMR-R 0.412 vs 0.427), and with chirality correction it reaches COV-R 87.0, exceeding the 242M MCF-L model on recall. A positional-encoding ablation in Appendix B.1.1 shows that both the ALiBi-style bias and a learnable shortest-path bias outperform Laplacian-eigenvector positional encodings within the S23D architecture. The paper also analyzes the effect of chirality correction on precision metrics and proposes a two-stage training protocol that reduces computational cost.

Significance. If the reported results are reproducible, the paper provides a useful data point that a simple relative positional encoding can make non-equivariant transformers parameter-efficient for conformer generation, and the two-stage training protocol and chirality-correction analysis are practical contributions to the MCG literature. The paper does not ship code, trained models, or seed-averaged results, and the causal attribution of the benchmark gains to the specific positional encoding is not fully isolated from other architectural and training differences. Nevertheless, the question the paper addresses is timely, the comparisons with published MCF and ET-Flow numbers are transparently tabulated, and the internal PE ablation, while not controlling the cross-model comparison, is a step in the right direction. The work is likely to be of interest to the conformer-generation and graph-transformer communities.

major comments (3)
  1. [Abstract; §4.2, Table 1] The abstract states that a well-chosen positional encoding effectively addresses model-size limitations, but the reported experiments do not isolate the positional encoding from the other differences between S23D and MCF. S23D-B differs from MCF-B in backbone (LLaMA vs PerceiverIO), graph-encoder/structural split, coordinate encoding, O(3) augmentation, masking, and training schedule, so the 0.6 percentage-point COV-R margin (84.6 vs 84.0) and 0.015 Å AMR-R margin in Table 1 cannot be attributed specifically to the linear-bias PE. The ablation in Appendix B.1.1 varies the PE only within S23D-S, and the S23D-B eigenvector run is single-stage and without hydrogens, so it does not hold the rest of the pipeline fixed against MCF. A controlled experiment that replaces MCF's eigenvector PE with this shortest-path bias in the original MCF code, or a clear reframing of the abstract claim to refer to the S23D architecture as a whole, is needed to support the attribution.
  2. [Table 1; Appendix B.1.1] No error bars, confidence intervals, or repeated-seed results are reported for any S23D result. The headline margins over MCF-B are small (0.6 pp COV-R and 0.015 Å AMR-R), and both diffusion training and the RMSD-based evaluation metrics are stochastic. Without variance estimates, the claim that S23D-B 'surpasses' MCF-B cannot be distinguished from run-to-run noise. Please report mean and standard deviation over at least three seeds, or otherwise quantify the noise floor, for the central comparisons in Table 1.
  3. [Appendix B.1.1, Table 3] The ablation shows that a standard learnable Graphormer-style bias performs essentially identically to the ALiBi-style linear bias (COV-R 81.5 vs 81.4; AMR-R 0.468 vs 0.472) at small scale. This undercuts the emphasis in the abstract and contributions on the specific linear-slope schedule as the source of the improvement; the distinguishing result is the improvement over the Laplacian-eigenvector PE, not the particular functional form of the bias. Please either demonstrate an advantage of the fixed linear schedule over the learnable baseline in some setting or revise the contribution claim to focus on the relative-PE-vs-eigenvector-PE comparison.
minor comments (9)
  1. [Section 3.1.1, Eq. (6)] The description of the ALiBi slope schedule is ambiguous; please state explicitly how m_head is computed as a function of the number of heads and head index, and confirm that it matches the original ALiBi paper.
  2. [Section 2 vs B.1.1] The eigenvector positional encoding is described as ordered by ascending eigenvalues in Section 2 but by largest eigenvalues in B.1.1; please reconcile these descriptions.
  3. [B.1.1, Table 3 caption] The PE ablation rows are single-stage training with hydrogens, so they are not directly comparable to the two-stage rows in Table 1; please state this explicitly in the table caption.
  4. [B.2.1] The QM9 evaluation uses 995 test molecules, with an assumption that MCF's published numbers are on the same subset; please verify this or note the discrepancy.
  5. [B.3.2 and Figure 3] The text and Figure 3 refer to S23D-B-1/9 and S23D-S-1/13 configurations that do not appear in Table 2; please correct the model naming, which is likely S23D-B-1/13 and S23D-S-1/9.
  6. [Section A.2] There is a typo in the noise schedule description: 'β_in (0.0,18.0)' should presumably be 'β_s' or 'β_start'.
  7. [Section 3.1.2] There is a typo in 'atomi coordinate encodings'; it should be 'atom i coordinate encodings'.
  8. [Section 4.2] The sentence 'we estimate that further scaling of the S23D-B model will surpass precision metric performance' is speculative and not a result; consider removing it or clearly labeling it as extrapolation.
  9. [General] No code or model release is mentioned; please add a data and code availability statement, since the reproducibility of the reported benchmark numbers would be greatly enhanced by releasing the evaluation code and training configurations.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper's claims are empirical benchmark comparisons with no fitted constant being renamed as a prediction; the main interpretive concerns are experimental confound issues, not circular reasoning.

full rationale

The paper's derivation chain is an empirical comparison rather than a derived law. The central claim is that a standard LLaMA-style transformer with a fixed ALiBi-style linear shortest-path attention bias reaches higher GEOM-DRUGS recall than MCF-B at 25M parameters. No parameter is fitted to the benchmark and then re-reported as a prediction: the per-head slopes are taken directly from the ALiBi geometric sequence described in Section 3.1.1 ('We used slopes that were used in the ALiBi positional encoding [31], a geometric sequence that starts with 2^-8/n for n attention heads'), and the diffusion loss in Eq. (11) is a standard score-matching objective. Comparisons in Table 1 use externally published numbers for MCF and ET-Flow, and the chirality correction is checked against independent RDKit Cahn-Ingold-Prelog labels, not against the model's own outputs. The internal positional-encoding ablation in Appendix B.1.1 varies the PE inside the S23D architecture and shows that eigenvector PE is worse than the ALiBi-style bias, which supports rather than presupposes the paper's attribution. That the cross-model difference against MCF-B is not fully isolated from other architectural and training differences is a legitimate benchmark-design and correctness concern, but it is not circularity: the reported benchmark numbers could be true even if the causal attribution to the positional encoding is under-supported, and no equation in the paper reduces the claimed result to its own input by construction. Accordingly, the honest finding is no significant circularity, with a score of 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim is empirical and needs no new physical entities. It rests on standard diffusion theory, acceptance of CREST/GEOM as ground truth, reliance on RDKit chirality labels, and an unvalidated transfer of ALiBi slopes from token positions to molecular graph distances. Hand-chosen hyperparameters (head slopes, coordinate scale, noise endpoints, block split) set the operating point for all benchmark numbers.

free parameters (4)
  • ALiBi head slopes m_head = 2^(-8/n) geometric sequence; fixed, not learned
    Chosen from the NLP ALiBi paper rather than fitted to GEOM data; the paper's central PE claim depends on this schedule.
  • Coordinate normalization constant b = 20
    Atom coordinates are divided by 20 in the noise schedule and input scaling (Appendix A.2); chosen by hand and affects all sampled coordinates.
  • Noise schedule endpoints beta_s, beta_e = 0.0 and 18.0
    Linear VPSDE schedule in Appendix A.2; these endpoints are selected by hand and affect the difficulty of the diffusion task.
  • Graph encoder / structural block split = 1/9, 4/6, or 1/13 depending on variant
    Model capacity allocation between the 2D encoder and 3D structural module; the 4/6 split changes precision metrics in Table 1, so the choice matters for the reported results.
assumptions (4)
  • standard math Score-matching training with the VPSDE (equation 11) yields a valid sampler for the conditional distribution of coordinates given the molecular graph.
    Standard diffusion theory from Song et al. [38]; the paper does not prove or modify this, it relies on it.
  • domain assumption GEOM reference ensembles generated by CREST are treated as ground truth for low-energy conformations.
    All benchmark metrics in Tables 1, 5, and 6 compare against these ensembles; the paper notes benchmark saturation on QM9 but does not question DRUGS labels.
  • domain assumption RDKit CIP labels correctly determine whether a generated conformer has the target chirality and that mirror reflection fixes it.
    Section 3.1.3 uses RDKit CIP labels for post-hoc chirality correction; the paper itself notes the correction rarely succeeds for multiple chiral centers.
  • ad hoc to paper The ALiBi geometric slope schedule designed for token positions in text is an appropriate bias schedule for graph shortest-path distances in molecules.
    No derivation is given for why 2^(-8/n) slopes transfer from NLP to molecular graphs; the ablation supports it empirically relative to alternatives, but it is a paper-specific choice.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A standard transformer and attention with linear biases for molecular conformer generation." pith.science (2026). https://pith.science/paper/FYDHI4NK

@misc{pith2026250619834,
  author       = {Pith},
  title        = {Pith review of: A standard transformer and attention with linear biases for molecular conformer generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FYDHI4NK}},
  note         = {Machine review of arXiv:2506.19834}
}
read the original abstract

Sampling low-energy molecular conformations, spatial arrangements of atoms in a molecule, is a critical task for many different calculations performed in the drug discovery and optimization process. Numerous specialized equivariant networks have been designed to generate molecular conformations from 2D molecular graphs. Recently, non-equivariant transformer models have emerged as a viable alternative due to their capability to scale to improve generalization. However, the concern has been that non-equivariant models require a large model size to compensate the lack of equivariant bias. In this paper, we demonstrate that a well-chosen positional encoding effectively addresses these size limitations. A standard transformer model incorporating relative positional encoding for molecular graphs when scaled to 25 million parameters surpasses the current state-of-the-art non-equivariant base model with 64 million parameters on the GEOM-DRUGS benchmark. We implemented relative positional encoding as a negative attention bias that linearly increases with the shortest path distances between graph nodes at varying slopes for different attention heads, similar to ALiBi, a widely adopted relative positional encoding technique in the NLP domain. This architecture has the potential to serve as a foundation for a novel class of generative models for molecular conformations.

Figures

Figures reproduced from arXiv: 2506.19834 by the authors.

Figure 1
Figure 1. A: S23D score network architecture. The backbone of the model is a standard transformer network. The first set of transformer blocks (Graph encoder) encode the 2D graph and atom features, which are projected into hidden states in the first transformer block. Coordinate encodings are projected into hidden states of the first transformer block in the second set of blocks (structural module), where time tokens are also… view at source ↗
Figure 2
Figure 2. (A-B) Recall and (C-D) precision coverage as a function of coverage threshold, and E recall and F precision AMR as a function of number of atoms in a test molecule, for S23D-S-1/9 (S) and S23D-B-1/13 (S) (without chirality correction). of the coverage threshold (Fig. 2A-D) and number of atoms in the test molecules (Fig. 2E-F). This breakdown can be compared with [PITH_FULL_IMAGE:figures/full_fig_p017_2.png] view at source ↗
Figure 3
Figure 3. Inference time per diffusion step for S23D-S-1/9 (M), S23D-B-1/9 (M), MCF-S, MCF-B [PITH_FULL_IMAGE:figures/full_fig_p018_3.png] view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

50 extracted references · 31 canonical work pages

  1. [1]

    GEOM, energy-annotated molecular conformations for property prediction and molecular generation.Scientific Data, 9(185), 2022

    Simon Axelrod and Gómez-Bombarelli. GEOM, energy-annotated molecular conformations for property prediction and molecular generation.Scientific Data, 9(185), 2022

    work page 2022

  2. [2]

    Equivariant energy-guided SDE for inverse molecular design

    Fan Bao, Min Zhao, Zhongkai Hao, Peiyao Li, Chongxuan Li, and Jun Zhu. Equivariant energy-guided SDE for inverse molecular design. InInternational Conference on Learning Representations, 2022

    work page 2022

  3. [3]

    How attentive are graph attention networks?arXiv preprint arXiv:2105.14491, 2021

    Shaked Brody, Uri Alon, and Eran Yahav. How attentive are graph attention networks?arXiv preprint arXiv:2105.14491, 2021

    arXiv 2021

  4. [4]

    Memory transformer

    Mikhail S Burtsev, Yuri Kuratov, Anton Peganov, and Grigory V Sapunov. Memory transformer. arXiv preprint arXiv:2006.11527, 2020

    arXiv 2006

  5. [5]

    Morris, and Charlotte M

    Martin Buttenschoen, Garrett M. Morris, and Charlotte M. Deane. Posebusters: Ai-based docking methods fail to generate physically valid poses or generalise to novel sequences.Chem. Sci., 15:3130–3139, 2024

    work page 2024

  6. [6]

    On the importance of noise scheduling for diffusion models.arXiv preprint arXiv:2301.10972, 2023

    Ting Chen. On the importance of noise scheduling for diffusion models.arXiv preprint arXiv:2301.10972, 2023

    arXiv 2023

  7. [7]

    Principal neighbourhood aggregation for graph nets.Advances in Neural Information Processing Systems, 33:13260–13271, 2020

    Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Liò, and Petar Veliˇckovi´c. Principal neighbourhood aggregation for graph nets.Advances in Neural Information Processing Systems, 33:13260–13271, 2020

    work page 2020

  8. [8]

    A hitchhiker’s guide to geometric GNNs for 3D atomic systems.arXiv preprint arXiv:2312.07511, 2023

    Alexandre Duval, Simon V Mathis, Chaitanya K Joshi, Victor Schmidt, Santiago Miret, Fragkiskos D Malliaros, Taco Cohen, Pietro Lio, Yoshua Bengio, and Michael Bronstein. A hitchhiker’s guide to geometric GNNs for 3D atomic systems.arXiv preprint arXiv:2312.07511, 2023

    arXiv 2023

Show all 50 references
  1. [9]

    A generalization of transformer networks to graphs

    Vijay Prakash Dwivedi and Xavier Bresson. A generalization of transformer networks to graphs. InAAAI Workshop on Deep Learning on Graphs: Methods and Applications, 2021

    2021

  2. [10]

    Coley, Regina Barzilay, Klavs F

    Octavian-Eugen Ganea, Lagnajit Pattanaik, Connor W. Coley, Regina Barzilay, Klavs F. Jensen, William H. Green, and Tommi S. Jaakkola. GeoMol: torsional geometric generation of molecular 3D conformer ensembles. InAdvances in Neural Information Processing Systems, pages 13757–13...

    2021

  3. [11]

    Symmetry-adapted generation of 3d point sets for the targeted discovery of molecules

    Niklas Gebauer, Michael Gastegger, and Kristof Schütt. Symmetry-adapted generation of 3d point sets for the targeted discovery of molecules. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors,Advances in Neural Information Processing ...

    2019

  4. [12]

    On embeddings for numerical features in tabular deep learning.Advances in Neural Information Processing Systems, 35:24991–25004, 2022

    Yury Gorishniy, Ivan Rubachev, and Artem Babenko. On embeddings for numerical features in tabular deep learning.Advances in Neural Information Processing Systems, 35:24991–25004, 2022. 10

    2022

  5. [13]

    Scaffold splits overesti- mate virtual screening performance

    Qianrong Guo, Saiveth Hernandez-Hernandez, and Pedro J Ballester. Scaffold splits overesti- mate virtual screening performance. InInternational Conference on Artificial Neural Networks, pages 58–72. Springer, 2024

    2024

  6. [14]

    ET-Flow: Equivariant flow-matching for molecular conformer generation

    Majdi Hassan, Nikhil Shenoy, Jungyoon Lee, Hannes Stark, Stephan Thaler, and Dominique Beaini. ET-Flow: Equivariant flow-matching for molecular conformer generation. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  7. [15]

    Havel, Irwin D

    Timothy F. Havel, Irwin D. Kuntz, and Gordon M. Crippen. The combinatorial distance geometry method for the calculation of molecular conformation. I. a new approach to an old problem.Journal of Theoretical Biology, 104(3):359–381, 1983

    1983

  8. [16]

    Paul C. D. Hawkins. Conformation generation: The state of the art.Journal of Chemical Information and Modeling, 57(8):1747–1756, 2017

    2017

  9. [17]

    Paul C. D. Hawkins, A. Geoffrey Skillman, Gregory L. Warren, Benjamin A. Ellingson, and Matthew T. Stahl. Conformer generation with OMEGA: Algorithm and validation using high quality structures from the Protein Databank and Cambridge Structural Database.Journal of Chemical Inf...

    2010

  10. [18]

    Ballester

    Saiveth Hernández-Hernández and Pedro J. Ballester. On the best way to cluster NCI-60 molecules.Biomolecules, 13(3), 2023

    2023

  11. [19]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Red Hook, NY , USA, 2020. Curran Associates Inc

    2020

  12. [20]

    Cascaded diffusion models for high fidelity image generation.Journal of Machine Learning Research, 23(47):1–33, 2022

    Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation.Journal of Machine Learning Research, 23(47):1–33, 2022

    2022

  13. [21]

    Perceiver IO: A general architecture for structured inputs & outputs

    Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al. Perceiver IO: A general architecture for structured inputs & outputs. InInternational Conference on Learning ...

    2021

  14. [22]

    Torsional diffusion for molecular conformer generation.Advances in Neural Information Processing Systems, 35:24240–24253, 2022

    Bowen Jing, Gabriele Corso, Jeffrey Chang, Regina Barzilay, and Tommi Jaakkola. Torsional diffusion for molecular conformer generation.Advances in Neural Information Processing Systems, 35:24240–24253, 2022

    2022

  15. [23]

    Eigenfold: Generative protein structure prediction with diffusion models

    Bowen Jing, Ezra Erives, Peter Pao-Huang, Gabriele Corso, Bonnie Berger, and Tommi S Jaakkola. Eigenfold: Generative protein structure prediction with diffusion models. InICLR 2023-Machine Learning for Drug Discovery workshop, 2023

    2023

  16. [24]

    Rethinking graph transformers with spectral attention.Advances in Neural Information Pro- cessing Systems, 34:21618–21629, 2021

    Devin Kreuzer, Dominique Beaini, Will Hamilton, Vincent Létourneau, and Prudencio Tossou. Rethinking graph transformers with spectral attention.Advances in Neural Information Pro- cessing Systems, 34:21618–21629, 2021

    2021

  17. [25]

    Graph inductive biases in transformers without message passing

    Liheng Ma, Chen Lin, Derek Lim, Adriana Romero-Soriano, Puneet K Dokania, Mark Coates, Philip Torr, and Ser-Nam Lim. Graph inductive biases in transformers without message passing. InInternational Conference on Machine Learning, pages 23321–23337. PMLR, 2023

    2023

  18. [26]

    Molecular geometry prediction using a deep generative graph neural network.Scientific reports, 9(1):20381, 2019

    Elman Mansimov, Omar Mahmood, Seokho Kang, and Kyunghyun Cho. Molecular geometry prediction using a deep generative graph neural network.Scientific reports, 9(1):20381, 2019

    2019

  19. [27]

    Transformer for graphs: An overview from architecture perspective.arXiv preprint arXiv:2202.08455, 2022

    Erxue Min, Runfa Chen, Yatao Bian, Tingyang Xu, Kangfei Zhao, Wenbing Huang, Peilin Zhao, Junzhou Huang, Sophia Ananiadou, and Yu Rong. Transformer for graphs: An overview from architecture perspective.arXiv preprint arXiv:2202.08455, 2022

    2022 arXiv

  20. [28]

    Miteva, Frederic Guyon, and Pierre Tufféry

    Maria A. Miteva, Frederic Guyon, and Pierre Tufféry. Frog2: Efficient 3D conformation ensemble generator for small compounds.Nucleic Acids Research, 38(Suppl 2):W622–W627, 2010. 11

    2010

  21. [29]

    Automated exploration of the low-energy chemical space with fast quantum chemical methods.Physical Chemistry Chemical Physics, 22:7169–7192, 2020

    Philipp Pracht, Fabian Bohle, and Stefan Grimme. Automated exploration of the low-energy chemical space with fast quantum chemical methods.Physical Chemistry Chemical Physics, 22:7169–7192, 2020

    2020

  22. [30]

    Wesołowski, and Felix Zeller

    Philipp Pracht, Stefan Grimme, Christoph Bannwarth, Fabian Bohle, Sebastian Ehlert, Gereon Feldmann, Johannes Gorges, Marcel Müller, Tim Neudecker, Christoph Plett, Sebastian Spicher, Pit Steinbach, Patryk A. Wesołowski, and Felix Zeller. CREST — a program for the exploration ...

    2024

  23. [31]

    Train short, test long: Attention with linear biases enables input length extrapolation

    Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. InInternational Conference on Learning Representations, 2021

    2021

  24. [32]

    Recipe for a general, powerful, scalable graph transformer.Advances in Neural Information Processing Systems, 35:14501–14515, 2022

    Ladislav Rampášek, Michael Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, and Dominique Beaini. Recipe for a general, powerful, scalable graph transformer.Advances in Neural Information Processing Systems, 35:14501–14515, 2022

    2022

  25. [33]

    Sereina Riniker and Gregory A. Landrum. Better informed distance geometry: Using what we know to improve conformation generation.Journal of Chemical Information and Modeling, 55(12):2562–2574, 2015

    2015

  26. [34]

    E (n) equivariant graph neural networks

    Vıctor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E (n) equivariant graph neural networks. InInternational conference on machine learning, pages 9323–9332. PMLR, 2021

    2021

  27. [35]

    Learning gradient fields for molecular conformation generation

    Chence Shi, Shitong Luo, Minkai Xu, and Jian Tang. Learning gradient fields for molecular conformation generation. InInternational conference on machine learning, pages 9558–9568. PMLR, 2021

    2021

  28. [36]

    A generative model for molecular distance geometry

    Gregor Simm and Jose Miguel Hernandez-Lobato. A generative model for molecular distance geometry. InInternational Conference on Machine Learning, pages 8949–8958. PMLR, 2020

    2020

  29. [37]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2020

    2020

  30. [38]

    Score-based generative modeling through stochastic differential equations

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2020

    2020

  31. [39]

    Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063, 2024

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063, 2024

    2024

  32. [40]

    MLP- mixer: An all-MLP architecture for vision.Advances in neural information processing systems, 34:24261–24272, 2021

    Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. MLP- mixer: An all-MLP architecture for vision.Advances in neural information processing systems, 34:242...

    2021

  33. [41]

    LLaMA: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  34. [42]

    Vainio and Mark S

    Mikko J. Vainio and Mark S. Johnson. Generating conformer ensembles using a multiobjective genetic algorithm.Journal of Chemical Information and Modeling, 47(6):2462–2474, 2007

    2007

  35. [43]

    Swallowing the bitter pill: Simplified scalable conformer generation

    Yuyang Wang, Ahmed AA Elhag, Navdeep Jaitly, Joshua M Susskind, and Miguel Ángel Bautista. Swallowing the bitter pill: Simplified scalable conformer generation. InForty-first International Conference on Machine Learning, 2024

    2024

  36. [44]

    Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S

    Zhenqin Wu, Bharath Ramsundar, Evan N. Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S. Pappu, Karl Leswing, and Vijay Pande. MoleculeNet: a benchmark for molecular machine learning.Chem. Sci., 9:513–530, 2018

    2018

  37. [45]

    Learning neural generative dynamics for molecular conformation generation

    Minkai Xu, Shitong Luo, Yoshua Bengio, Jian Peng, and Jian Tang. Learning neural generative dynamics for molecular conformation generation. InInternational Conference on Learning Representations, 2021. 12

    2021

  38. [46]

    GeoDiff: A geometric diffusion model for molecular conformation generation

    Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang. GeoDiff: A geometric diffusion model for molecular conformation generation. InInternational Conference on Learning Representations, 2022

    2022

  39. [47]

    Do transformers really perform badly for graph representation?Advances in neural information processing systems, 34:28877–28888, 2021

    Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Liu. Do transformers really perform badly for graph representation?Advances in neural information processing systems, 34:28877–28888, 2021

    2021

  40. [48]

    Do deep learning methods really perform better in molecular conformation generation? InICLR 2023-Machine Learning for Drug Discovery workshop, 2023

    Gengmo Zhou, Zhifeng Gao, Zhewei Wei, Hang Zheng, and Guolin Ke. Do deep learning methods really perform better in molecular conformation generation? InICLR 2023-Machine Learning for Drug Discovery workshop, 2023

    2023

  41. [49]

    Direct molecular conformation generation

    Jinhua Zhu, Yingce Xia, Chang Liu, Lijun Wu, Shufang Xie, Yusong Wang, Tong Wang, Tao Qin, Wengang Zhou, Houqiang Li, et al. Direct molecular conformation generation. Transactions on Machine Learning Research, 2022

    2022

  42. [50]

    intramolecular validity

    Peiye Zhuang, Samira Abnar, Jiatao Gu, Alex Schwing, Joshua M Susskind, and Miguel Angel Bautista. Diffusion probabilistic fields. InThe Eleventh International Conference on Learning Representations, 2023. 13 A Model details A.1 Model and training parameters We trained the mod...

    2023

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.