Pith. sign in

REVIEW 4 major objections 6 minor 3 cited by

Graph Generative Pre-trained Transformer

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that graph generation can be cast as next-token prediction over a node-then-edge token sequence, and that a Transformer trained this way matches or surpasses diffusion-based and other autoregressive baselines on generic…

desk verdict A solid, honest empirical paper that makes a real case for edge-list tokenization for graph generation, but the load-bearing degree-based ordering is a dataset-specific design choice that undercuts the 'general foundation model' claim. read the letter →

arxiv 2501.01073 v2 pith:5ILDVX4K submitted 2025-01-02 cs.LG cs.AI

Xiaohui Chen , Yinkai Wang , Jiaxing He , Yuanqi Du , Soha Hassoun , Xiaolin Xu , Li-Ping Liu This is my paper · ORCID
classification cs.LGcs.AI
keywords graphgenerationnext-tokenpredictionTransformerdecoderedge-listrepresentationmolecularpropertyfine-tuningtokenization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that graph generation can be treated as a language-modeling problem: a graph is written out as a token sequence that lists all nodes first and then all edges, and a Transformer decoder is trained to predict the next token. The proposed model, G2PT, learns graph structure by maximizing the likelihood of such sequences, which the authors show is equivalent to maximizing a lower bound on the graph likelihood and requires only O(n + m) tokens instead of the O($n^{2}$) entries of an adjacency matrix. Across four generic graph datasets and three molecular benchmarks, G2PT matches or surpasses permutation-invariant diffusion and autoregressive baselines, and it can be fine-tuned by rejection sampling or PPO for goal-oriented molecular generation and by a supervised head for graph property prediction. If these results hold, sparse graph generation, molecular design, and graph representation learning can share one Transformer-based sequence model.

What carries the argument

The load-bearing object is a token-based graph representation: every graph is serialized as node-type/index tokens, a separator token, then edge triples (source index, destination index, edge type), with special start and end tokens. Edge order is fixed by Algorithm 1, which repeatedly removes the edge whose source has the minimum degree, breaks ties by destination degree, and reverses the removal list, so generation proceeds from a dense core to the low-degree periphery. This ordering turns an unordered graph into a one-dimensional language over a unified vocabulary, giving the Transformer a next-token prediction task and reducing the number of variables from O($n^{2}$) adjacency entries to O(m) actual edges; the ordering choice is what carries the model's ability to learn structure.

What would settle it

Take a family of regular graphs, where every node has the same degree and Algorithm 1's min-degree choices are arbitrary tie-breaks, and train G2PT with any fixed tie-breaking rule; if validity and MMD statistics fall to the level of uniform ordering, then the degree-based ordering is not what makes the representation work.

Watch

Extended reading notes

Core claim

The central claim is that an autoregressive Transformer trained with next-token prediction over node-then-edge token sequences is a general and competitive graph generative model. The paper identifies the edge-ordering rule as the main design decision: the reverse of a degree-based edge-removal process (Algorithm 1) produces a compact dense-core-first construction order, and the model learns the distribution over these ordered token sequences. G2PT is shown to outperform strong discrete-diffusion and flow baselines on QM9, MOSES, and GuacaMol for molecules and on Planar, Tree, Lobster, and SBM for generic graphs, while the same pre-trained model, fine-tuned, improves QED, SA, and GSK3-beta objectives and produces graph-level embeddings competitive with self-supervised baselines on eight classification tasks.

Load-bearing premise

The load-bearing premise is that the reverse of a degree-based edge-removal process is a good, general way to order the edges of a graph for a language model to learn; if different graph domains need different edge orderings, then G2PT is a dataset-specific encoder rather than a general graph foundation.

Editorial extensions

If this is right

  • Sparse graphs are generated with O(n + m) tokens instead of the O(n^2) entries of an adjacency matrix, making sequence length proportional to the actual number of edges.
  • Once graphs are token sequences, standard Transformer techniques—model scaling, data augmentation, rejection sampling fine-tuning, and PPO—apply directly to graph generation without new architecture design.
  • The same pre-trained sequence model can be fine-tuned for property classification using the final-token embedding, matching self-supervised graph learners on eight molecule classification tasks.
  • Edge-ordering choice is a first-class design decision: BFS and degree-based orderings support strong generation, while uniform ordering collapses validity, so any claim about the representation must be paired with an ordering strategy.
  • Larger models and multiple sequences per graph improve validity until saturation, with G2PT scaling from 1M to 1.5B parameters across molecular datasets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The order sensitivity the paper concedes suggests that a learned or canonical 'assembly grammar' for graphs could be the next step; if such a universal ordering exists, G2PT-style models could become cross-domain foundation models rather than per-dataset encoders.
  • Because the representation uses only existing edges, very large sparse graphs such as social networks or knowledge graphs are a natural stress test the paper does not run; sequence length grows with edges, not with the square of nodes.
  • Property prediction via the final-token embedding could be extended to regression, link prediction, or conditional generation with prompts, since the paper only demonstrates classification.
  • The comparison between rejection sampling fine-tuning and PPO suggests that distribution-shaping under hard constraints may be more reliably achieved by iterative filtering than by reward optimization when the target region is rare.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. G2PT represents a graph as a token sequence that lists all node tokens, a delimiter token, and then all edge tokens, with the edge order defined by the reverse of a degree-based edge-removal process (Algorithm 1). A Transformer decoder is trained with a standard next-token prediction objective on this serialization, and the paper shows that maximizing sequence likelihood maximizes a lower bound on graph likelihood. The same pre-trained model is fine-tuned for goal-oriented molecular generation via rejection-sampling fine-tuning and PPO, and for graph property prediction by attaching a linear classifier to the last-token representation. Experiments cover four generic graph datasets (Planar, Tree, Lobster, SBM) and three molecular datasets (QM9, MOSES, GuacaMol), plus eight MoleculeNet classification benchmarks, with scaling and edge-ordering sensitivity analyses.

Significance. The work addresses a timely question: whether a plain GPT-style Transformer can serve as a graph generator through an edge-list serialization. The lower-bound derivation in Section 3.2 is sound, the empirical protocol broadly follows established benchmarks, and the paper includes useful analyses of model/data scaling and edge-ordering sensitivity, as well as an ablation showing the benefit of pre-training for property prediction. The code release supports reproducibility. If the claims are confirmed with appropriate uncertainty quantification and with the ordering-sensitivity issue addressed, the edge-token serialization would be a valuable alternative to adjacency-matrix diffusion and autoregressive baselines, with clear computational advantages on sparse graphs.

major comments (4)
  1. [Algorithm 1, Table 9, Section 6] The degree-based edge-removal order in Algorithm 1 is the only bridge between a graph and the token sequence learned by the model, and the paper's own sensitivity analysis shows this bridge is fragile: on MOSES, replacing it with uniform ordering drops G2PTsmall validity from 95.1 to 62.9 and FCD from 1.10 to 7.0 (Table 9), and the Conclusion states that 'G2PT is order-sensitive, where different graph domains may prefer different edge orderings.' No experiment demonstrates that a single ordering transfers across the generic graph and molecular domains used to support the 'general-purpose foundation model' claim in Section 1. I ask the authors to add cross-domain ordering-transfer evidence (for example, training on one domain and testing ordering choices on another, or comparing the chosen ordering across all four generic and three molecular datasets) or to explicitly reframe the contribution as a dataset-specific encoding rather than a general graph encoding.
  2. [Section 5.3, Table 2] The abstract and Section 5.3 claim 'superior generative performance' on generic graphs, but Table 2 shows G2PTbase is best on 11 of 24 metrics and top-two on 17 of 24, and on Planar degree G2PTsmall (4.7e-3) is an order of magnitude worse than DeFoG (5e-4). Because no standard deviations or repeated-seed results are reported for any MMD metric, differences among near-zero entries (e.g., Orbit 0.00 for several methods) cannot be judged statistically. Please report uncertainty estimates and either soften the superiority claim to 'competitive' or identify the specific metrics on which superiority is claimed.
  3. [Section 5.5, Figure 2] The goal-oriented generation results are presented only as density plots, with no quantitative metrics (e.g., fraction of accepted samples, mean property values, validity, uniqueness, or reward) and no baseline method for comparison beyond visual inspection of the pre-trained distribution. The text makes strong statements such as 'RFT overcomes the barrier in the second round' and 'PPO fails to sample data with high rewards' without numerical support. Please add a quantitative evaluation of the fine-tuning procedures, including the validity and diversity of the optimized samples, so that the fine-tuning contribution can be assessed.
  4. [Section 5.4, Table 4, Section B.6] The molecular generation comparison is incomplete in two respects. First, the QM9 table reports FCD for G2PT but leaves DiGress's FCD as '-' even though DiGress is a primary baseline, so the headline FCD advantage on QM9 cannot be compared against it. Second, the GuacaMol column headers use 'KL Div.↑' and 'FCD↑', which conflicts with the standard GuacaMol evaluation where these quantities are lower-is-better; if the arrows are correct, the definitions should be clarified. Please reconcile the protocol with the cited toolkits and state the validity definition used for each baseline (Section B.6 mentions that the RDKit validity definition can differ from the lenient definition used in some prior work).
minor comments (6)
  1. [Section 3.2] The sentence 'Although pθ(s|G) is not directly compute' has a grammatical error; it should read 'not directly computed'.
  2. [Section C.2] The sentence 'As discussed in ??' contains an unresolved cross-reference; please fill in the section number.
  3. [Section A.3] The phrase 'o mitigate potential degeneration' should be 'to mitigate potential degeneration'.
  4. [Table 7, Section 5.7] Table 7 reports hyperparameters only for the 10M, 85M, and 300M models, but Section 5.7 and Figure 3 include 707M and 1.5B models; please provide the architecture and training details for these larger models.
  5. [Section 5.7] The data-scaling experiment states that multiple sequences are generated from the same graph but does not specify how the multiple sequences are produced (random node orders, random tie-breaking in Algorithm 1, or both); please document the augmentation procedure.
  6. [Table 3] The row labels 'A s (Ours)' appear garbled; the two rows labeled 'A' and 's (Ours)' should be clearly distinguished in the table or its caption.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: sequence MLE is a standard lower-bound objective, and the degree-based edge ordering is a fixed design choice rather than a fitted quantity.

full rationale

The derivation chain is self-contained. The training objective in Eq. (1) is ordinary next-token maximum likelihood over token sequences, and the §3.2 lower-bound argument is a standard ELBO/chain-rule identity: it relates graph likelihood to sequence likelihood without defining the target in terms of the training loss or renaming a fitted quantity as a prediction. Algorithm 1's degree-based edge ordering is a fixed serialization heuristic, not a parameter fitted to the evaluation metric; Table 9 is an honest sensitivity analysis showing BFS is competitive and uniform ordering is poor, and the Conclusion explicitly concedes that G2PT is order-sensitive. That is a generality limitation, not a circular step. The paper's self-citations (Order Matters, EDGE, DAGG, Edge++) appear as background, baselines, or as support for BFS node ordering in a baseline setup; none supplies the central claim that next-token prediction on edge-list sequences generates graphs well, which is supported by external benchmarks (MOSES, GuacaMol, MoleculeNet) and comparisons to non-self baselines. No fitted parameter is disguised as a prediction, and no uniqueness theorem is invoked to force the chosen ordering.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central generative claim does not depend on fitted constants; the listed free parameters concern fine-tuning and data augmentation. The only significant load-bearing assumption is the degree-based edge ordering, which is a design heuristic rather than an axiom. No new physical or conceptual entities are introduced.

free parameters (3)
  • RFT acceptance tolerances omega = QED: 0.4/0.6/0.8/0.9; SA: 3.0/2.0/1.5; GSK3beta: 0.2/0.4/0.6/0.8
    Manual thresholds in Section B.3 that define the target distribution for rejection-sampling fine-tuning; they are chosen by hand rather than derived.
  • PPO loss coefficients rho1, rho2, rho3 = 0.5, 0.03/0.05, 0.03 (Table 8)
    Chosen per task in Section B.3; they affect the RL fine-tuning behavior but not the core generative claims.
  • Number of sequences per graph augmentation = 1, 10, 100
    Data scaling experiment in Section 5.7; the paper states that one sequence per graph is insufficient, so the augmentation count is a required choice for good performance.
assumptions (5)
  • standard math The token sequence uniquely determines the graph, and p(G, s) = p(s).
    Used in the lower-bound derivation in Section 3.2; true because the tokenization maps one-to-one to node-labeled and edge-labeled graphs.
  • domain assumption A fixed maximum node count nmax per dataset bounds the vocabulary and sequence space.
    Section 3.2 assigns node indices 1 through nmax, so graphs with more than nmax nodes cannot be represented.
  • ad hoc to paper The reverse of degree-based edge removal creates sequences that are easier to model, and this ordering generalizes across graphs.
    Algorithm 1 and Table 9; uniform ordering performs far worse, so the method's success depends on this heuristic.
  • domain assumption Training on multiple random sequences from the same graph tightens the lower bound on graph likelihood.
    Section 3.2 and Figure 3; plausible and empirically supported, but not proven formally in the paper.
  • domain assumption The graphs are undirected, and edge tokens store both endpoints as node indices.
    Section 3.1; directed graphs would require separate handling and are not discussed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Graph Generative Pre-trained Transformer." pith.science (2026). https://pith.science/paper/5ILDVX4K

@misc{pith2026250101073,
  author       = {Pith},
  title        = {Pith review of: Graph Generative Pre-trained Transformer},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5ILDVX4K}},
  note         = {Machine review of arXiv:2501.01073}
}
read the original abstract

Graph generation is a critical task in numerous domains, including molecular design and social network analysis, due to its ability to model complex relationships and structured data. While most modern graph generative models utilize adjacency matrix representations, this work revisits an alternative approach that represents graphs as sequences of node set and edge set. We advocate for this approach due to its efficient encoding of graphs and propose a novel representation. Based on this representation, we introduce the Graph Generative Pre-trained Transformer (G2PT), an auto-regressive model that learns graph structures via next-token prediction. To further exploit G2PT's capabilities as a general-purpose foundation model, we explore fine-tuning strategies for two downstream applications: goal-oriented generation and graph property prediction. We conduct extensive experiments across multiple datasets. Results indicate that G2PT achieves superior generative performance on both generic graph and molecule datasets. Furthermore, G2PT exhibits strong adaptability and versatility in downstream tasks from molecular design to property prediction. Code available at https://github.com/tufts-ml/G2PT,

Figures

Figures reproduced from arXiv: 2501.01073 by the authors.

Figure 1
Figure 1. Illustration of our proposed graph sequence representation. This representation can be viewed as a sequence of actions: first generating all nodes (node type, node index), then explicitly adding edges (source node index, destination node index, edge type) step by step until completion. A unified vocabulary is used to map different types of actions into a shared token space. We denote the tokenized sequence s = [s1, … view at source ↗
Figure 2
Figure 2. Goal-oriented molecule generation using QED, SA and GSK3β scores. Top row (a) shows the results using RFT, and bottom row (b) shows the results using RL. The quantitative results are presented in [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Model and data scaling effects. as 3D information or chirality, G2PT is trained exclusively on 2D graph structural information. Overall, these results indicate that G2PT not only excels in generation but also learns effective graph representations. 5.7. Scaling Effects We analyze how scaling the model size and data size will affect the model performance using the three molecular datasets. We use the validity score t… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The visualization of generic graph datasets MOSES Train G2PTsmall G2PTbase GuacaMol Train G2PTsmall G2PTbase [PITH_FULL_IMAGE:figures/full_fig_p022_4.png]
Figure 5
Figure 5. Figure 5: The visualization of molecular datasets 22 [PITH_FULL_IMAGE:figures/full_fig_p022_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. PROVCREATOR: Synthesizing Complex Heterogenous Graphs with Node and Edge Attributes

    cs.LG 2025-07 conditional novelty 6.0 of 10

    ProvCreator is a framework that serializes complex heterogeneous graphs into token sequences and fine-tunes LLaMA 3.2 3B to generate new graphs with structure and attributes generated jointly.

  2. NGTM: Substructure-based Neural Graph Topic Model for Interpretable Graph Generation

    cs.LG 2025-07 conditional novelty 5.0 of 10

    NGTM generates graphs by sampling substructures from learned topic-specific distributions and assembling them, achieving competitive quality with interpretable, controllable topics.

  3. Scalable Interference Graph Learning for Low-Latency Wi-Fi Networks using Hashing-based Evolution Strategy

    eess.SP 2025-02 conditional novelty 5.0 of 10

    A neural graph model trained with evolution strategies and deep hashing assigns Wi-Fi 7 scheduled transmission slots, using about 25% fewer slots and losing up to 30% fewer packets in 1,000-device simulations.

Reference graph

Works this paper leans on

51 extracted references · 19 canonical work pages · cited by 3 Pith papers

  1. [1]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774,

    arXiv

  2. [2]

    Among them, GRAN (Liao et al., 2019), BiGG (Dai et al., 2020), and BwR (Diamant et al.,

    We consider baselines mainly from two categories: auto-regressive and diffusion graph models. Among them, GRAN (Liao et al., 2019), BiGG (Dai et al., 2020), and BwR (Diamant et al.,

    work page 2019

  3. [6]

    Order Matters: Probabilistic Modeling of Node Sequence for Graph Generation

    Chen, D., O’Bray, L., and Borgwardt, K. Structure-aware transformer for graph representation learning. In Interna- tional Conference on Machine Learning, pp. 3469–3489. PMLR, 2022a. Chen, X., Han, X., Hu, J., Ruiz, F. J., and Liu, L. Order mat- ters: Probabilistic modeling of node sequence for graph generation. arXiv preprint arXiv:2106.06189,

    work page Pith review arXiv

  4. [8]

    and Kipf, T

    De Cao, N. and Kipf, T. Molgan: An implicit genera- tive model for small molecular graphs. arXiv preprint arXiv:1805.11973,

    arXiv

  5. [9]

    2024; Xu et al., 2024; Wang et al.,

    Sensitivity analysis on edge orderings. 2024; Xu et al., 2024; Wang et al.,

    work page 2024

  6. [11]

    The llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    arXiv

  7. [12]

    Dwivedi, V . P. and Bresson, X. A generalization of transformer networks to graphs. arXiv preprint arXiv:2012.09699,

    arXiv 2012

  8. [13]

    A., Welling, M., and van de Meent, J.-W

    Eijkelboom, F., Bartosh, G., Naesseth, C. A., Welling, M., and van de Meent, J.-W. Variational flow matching for graph generation. arXiv preprint arXiv:2406.04843,

    arXiv

Show all 51 references
  1. [14]

    Gao, Z., Dong, D., Tan, C., Xia, J., Hu, B., and Li, S. Z. A graph is worth k words: Euclideanizing graph using pure transformer. arXiv preprint arXiv:2402.02464,

    arXiv

  2. [15]

    T., Syn- naeve, G., Adi, Y ., and Lipman, Y

    Gat, I., Remez, T., Shaul, N., Kreuk, F., Chen, R. T., Syn- naeve, G., Adi, Y ., and Lipman, Y . Discrete flow match- ing. arXiv preprint arXiv:2407.15595,

    arXiv

  3. [16]

    K., Martinkus, K., Perraudin, N., and Wat- tenhofer, R

    Haefeli, K. K., Martinkus, K., Perraudin, N., and Wat- tenhofer, R. Diffusion models for graphs benefit from discrete state spaces. arXiv preprint arXiv:2210.01549,

    arXiv

  4. [17]

    Graphmae: Self-supervised masked graph autoencoders

    Hou, Z., Liu, X., Cen, Y ., Dong, Y ., Yang, H., Wang, C., and Tang, J. Graphmae: Self-supervised masked graph autoencoders. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 594–604, 2022a. Hou, Z., Liu, X., Cen, Y ., Dong, Y ., Yang...

    1905 arXiv

  5. [18]

    A simple and scalable representation for graph generation

    10 Graph Generative Pre-trained Transformer Jang, Y ., Lee, S., and Ahn, S. A simple and scalable representation for graph generation. arXiv preprint arXiv:2312.02230,

    arXiv

  6. [19]

    Jo, J., Lee, S., and Hwang, S. J. Score-based generative modeling of graphs via the system of stochastic differen- tial equations. In International conference on machine learning, pp. 10362–10383. PMLR, 2022a. Jo, J., Lee, S., and Hwang, S. J. Score-based generative modeling o...

    arXiv

  7. [20]

    Learning deep generative models of graphs

    Li, Y ., Vinyals, O., Dyer, C., Pascanu, R., and Battaglia, P. Learning deep generative models of graphs. arXiv preprint arXiv:1803.03324,

    arXiv

  8. [21]

    T., Ben-Hamu, H., Nickel, M., and Le, M

    Lipman, Y ., Chen, R. T., Ben-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747,

    arXiv

  9. [22]

    Pre-training molecular graph representation with 3d geometry, 2022a

    Liu, S., Wang, H., Liu, W., Lasenby, J., Guo, H., and Tang, J. Pre-training molecular graph representation with 3d geometry, 2022a. URL https://arxiv.org/abs/ 2110.07728. Liu, X., Gong, C., and Liu, Q. Flow straight and fast: Learning to generate and transfer data with rectifi...

    arXiv

  10. [23]

    Luo, Y ., Yan, K., and Ji, S

    URL https://arxiv.org/abs/ 1711.05101. Luo, Y ., Yan, K., and Ji, S. Graphdf: A discrete flow model for molecular graph generation. In International conference on machine learning, pp. 7192–7203. PMLR,

    arXiv

  11. [24]

    Graph- nvp: An invertible flow model for generating molecular graphs

    Madhawa, K., Ishiguro, K., Nakago, K., and Abe, M. Graph- nvp: An invertible flow model for generating molecular graphs. arXiv preprint arXiv:1905.11600,

    1905 arXiv

  12. [25]

    Min, E., Chen, R., Bian, Y ., Xu, T., Zhao, K., Huang, W., Zhao, P., Huang, J., Ananiadou, S., and Rong, Y

    URL https://arxiv.org/abs/2204.01613. Min, E., Chen, R., Bian, Y ., Xu, T., Zhao, K., Huang, W., Zhao, P., Huang, J., Ananiadou, S., and Rong, Y . Trans- former for graphs: An overview from architecture per- spective. arXiv preprint arXiv:2202.08455,

    arXiv

  13. [26]

    Preuer, K., Renz, P., Unterthiner, T., Hochreiter, S., and Klambauer, G

    URL https://arxiv.org/abs/1811.12823. Preuer, K., Renz, P., Unterthiner, T., Hochreiter, S., and Klambauer, G. Fr ´echet chemnet distance: A metric for generative models for molecules in drug discovery,

    arXiv

  14. [27]

    11 Graph Generative Pre-trained Transformer Qin, Y ., Madeira, M., Thanou, D., and Frossard, P

    URL https://arxiv.org/abs/1803.09518. 11 Graph Generative Pre-trained Transformer Qin, Y ., Madeira, M., Thanou, D., and Frossard, P. Defog: Discrete flow matching for graph generation,

    arXiv

  15. [28]

    Radford, A

    URL https://arxiv.org/abs/2410.04263. Radford, A. Improving language understanding by genera- tive pre-training

    arXiv

  16. [29]

    High-dimensional continuous control using generalized advantage estimation

    Schulman, J., Moritz, P., Levine, S., Jordan, M., and Abbeel, P. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438,

    arXiv

  17. [31]

    and Komodakis, N

    Simonovsky, M. and Komodakis, N. Graphvae: Towards generation of small graphs using variational autoencoders. In Artificial Neural Networks and Machine Learning– ICANN 2018: 27th International Conference on Artificial Neural Networks, Rhodes, Greece, October 4-7, 2018, Proceed...

    2018

  18. [32]

    D., and Morris, C

    Siraudin, A., Malliaros, F. D., and Morris, C. Cometh: A continuous-time discrete-state graph diffusion model. arXiv preprint arXiv:2406.06449,

    arXiv

  19. [33]

    URL https://arxiv.org/abs/1908. 01000. Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems,

    1908

  20. [36]

    N., Gomes, J., Ge- niesse, C., Pappu, A

    Wu, Z., Ramsundar, B., Feinberg, E. N., Gomes, J., Ge- niesse, C., Pappu, A. S., Leswing, K., and Pande, V . Moleculenet: a benchmark for molecular machine learn- ing. Chemical science, 9(2):513–530, 2018a. Wu, Z., Ramsundar, B., Feinberg, E. N., Gomes, J., Ge- niesse, C., Pap...

    arXiv

  21. [37]

    Discrete-state continuous- time diffusion for graph generation

    Xu, Z., Qiu, R., Chen, Y ., Chen, H., Fan, X., Pan, M., Zeng, Z., Das, M., and Tong, H. Discrete-state continuous- time diffusion for graph generation. arXiv preprint arXiv:2405.11416,

    arXiv

  22. [38]

    URL https://arxiv.org/abs/2010.13902. Zang, C. and Wang, F. Moflow: an invertible flow model for generating molecular graphs. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 617–626,

    2010 arXiv

  23. [39]

    Secrets of rlhf in large language models part i: Ppo

    Zheng, R., Dou, S., Gao, S., Hua, Y ., Shen, W., Wang, B., Liu, Y ., Jin, S., Liu, Q., Zhou, Y ., et al. Secrets of rlhf in large language models part i: Ppo. arXiv preprint arXiv:2307.04964,

    arXiv

  24. [40]

    falls off the grid

    Dataset statistics. Value Function Approximation. The critic model Vψ(s<l) in PPO algorithm is used to approximate the actual value function V p(s<l). We use the mean absolute value loss to minimize the difference between the predicted values and the actual return values. Spec...

    2023

  25. [41]

    Lr scheduler Cosine Weight decay 1e-1 #iterations 300000 Batch size 60 60 30 #Gradient Accumulation 8 8 16 Grad Clipping Value 1 #Warmup Iterations 2000 Table

    2000

  26. [42]

    Training transformers on adjacency matrices

    We employ BFS to determine the node orderings, which is used to permute the rows and columns of the adjacency matrix to reduce the learning complexity (as uniform orderings are generally harder to fit (Chen et al., 2021)). Training transformers on adjacency matrices. After obt...

    2021

  27. [43]

    Hyperparameters used for PPO training. B.3. Fine-tuning G2PT for Goal-oriented Generation For the goal-oriented generation, we fine-tune G2PT to generate molecules with desired characteristics. Specifically, we consider three properties that are commonly used for molecule opti...

    2022

  28. [44]

    Fine-tuning G2PT for Graph Property Prediction Datasets

    B.4. Fine-tuning G2PT for Graph Property Prediction Datasets. We use eight classification tasks in MoleculeNet (Wu et al., 2018a) following Zhu et al. (2024) to validate the predictive capability of our learned representations. The datasets cover two types of molecular propert...

    2024

  29. [46]

    are built based on diffusion frameworks. DiGress is the first approach that uses a discrete diffusion model to iteratively modify graphs, while HSpectra focuses on multi-scale graph construction by progressively generating graphs through localized denoising diffusion. Molecule...

    2022

  30. [47]

    The dataset-agnostic metrics evaluate the alignment between the distributions of the generated graphs and the training data by analyzing general graph properties

    and (Bergmeister et al., 2024), incorporating both dataset-agnostic and dataset-specific metrics. The dataset-agnostic metrics evaluate the alignment between the distributions of the generated graphs and the training data by analyzing general graph properties. Specifically, we...

    2024

  31. [48]

    On the contrary, 19 Graph Generative Pre-trained Transformer Algorithm 4 Depth-First search edge order generation Input: Graph G = (V, E), neighborhood function Nei.(·)

    prefix a canonical ordering (e.g., breath-first search) for the nodes and edges and generates nodes and edges associated with them step by step. On the contrary, 19 Graph Generative Pre-trained Transformer Algorithm 4 Depth-First search edge order generation Input: Graph G = (...

    2020

  32. [49]

    both leverage pre-trained auto-regressive models to fine-tune with a reward model to generate molecules with desired properties. C.2. Non-auto-regressive Graph Generative Models In addition to auto-regressive models, non-auto-regressive graph generative models can be categoriz...

    2018

  33. [50]

    Nevertheless, one-shot graph generative models often suffer from the decoding strategies such that it requires an expressive decoder to map from latent vectors to graphs

    and normalizing flows (Madhawa et al., 2019; Zang & Wang, 2020). Nevertheless, one-shot graph generative models often suffer from the decoding strategies such that it requires an expressive decoder to map from latent vectors to graphs. On the other side, iterative refinement m...

    2019

  34. [52]

    As discussed in ??, they often require a prefixed number of refinement steps and they need to maintain an adjacency matrix over the trajectory which is computationally intensive

    and flow matching models (Qin et al., 2024; Eijkelboom et al., 2024; Lipman et al., 2022; Liu et al., 2022b; Campbell et al., 2024; Gat et al., 2024). As discussed in ??, they often require a prefixed number of refinement steps and they need to maintain an adjacency matrix ove...

    2022

  35. [2015]

    Proximal policy optimization algorithms

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,

    arXiv

  36. [2017]

    Digress: Discrete denoising diffusion for graph generation

    Vignac, C., Krawczuk, I., Siraudin, A., Wang, B., Cevher, V ., and Frossard, P. Digress: Discrete denoising diffusion for graph generation. arXiv preprint arXiv:2209.14734,

    arXiv

  37. [2018]

    Bert: Pre-training of deep bidirectional trans- formers for language understanding

    Devlin, J. Bert: Pre-training of deep bidirectional trans- formers for language understanding. arXiv preprint arXiv:1810.04805,

    arXiv

  38. [2019]

    doi: 10.1021/acs.jcim.8b00839

    ISSN 1549- 960X. doi: 10.1021/acs.jcim.8b00839. URL http: //dx.doi.org/10.1021/acs.jcim.8b00839. Campbell, A., Benton, J., De Bortoli, V ., Rainforth, T., Deli- giannidis, G., and Doucet, A. A continuous time frame- work for discrete denoising models. Advances in Neural Inform...

    doi

  39. [2021]

    Nvdiff: Graph generation through the diffusion of node vectors

    Chen, X., Li, Y ., Zhang, A., and Liu, L.-p. Nvdiff: Graph generation through the diffusion of node vectors. arXiv preprint arXiv:2211.10794, 2022b. 9 Graph Generative Pre-trained Transformer Chen, X., He, J., Han, X., and Liu, L.-P. Efficient and degree- guided graph generati...

    arXiv

  40. [2022]

    Generative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design

    Campbell, A., Yim, J., Barzilay, R., Rainforth, T., and Jaakkola, T. Generative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design. arXiv preprint arXiv:2402.04997,

    arXiv

  41. [2023]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Dosovitskiy, A. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint arXiv:2010.11929,

    2010 arXiv

  42. [2024]

    Brown, N., Fiscato, M., Segler, M

    URL https: //arxiv.org/abs/2312.11529. Brown, N., Fiscato, M., Segler, M. H., and Vaucher, A. C. Guacamol: Benchmarking models for de novo molec- ular design. Journal of Chemical Information and Modeling, 59(3):1096–1108, March

    arXiv

  43. [2025]

    Wu, M., Chen, X., and Liu, L.-P

    URL https://arxiv.org/abs/2501.01950. Wu, M., Chen, X., and Liu, L.-P. Edge++: Improved training and sampling of edge. arXiv preprint arXiv:2310.14441,

    arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.