{"id":"abb0af5c-d968-40e5-8d39-94a445b4dc69","arxiv_id":"2507.20967","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"ProvCreator is a framework that serializes complex heterogeneous graphs into token sequences and fine-tunes LLaMA 3.2 3B to generate new graphs with structure and attributes generated jointly.","lead":"ProvCreator turns the problem of generating complex graphs with rich text attributes into the problem of writing text with a language model, using special tokens to represent graph structure. Its key experiments show synthetic cybersecurity provenance graphs and knowledge graphs that beat a diffusion baseline on structure and attribute fidelity metrics.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim depends on a canonical edge ordering; for non-provenance graphs that ordering is arbitrary, so the learned distribution may be order-specific rather than graph-general.","rationale":"Reader's weakest assumption aligns with the main risk I see. The paper's central claim is that a transformer can jointly generate structure and attributes by autoregressive sequence generation. The mechanism for converting graphs to sequences is Algorithm 1, which is parameterized by an edge ordering. For the primary cybersecurity domain this is natural (timestamps), so the results there are not threatened. But the paper extends the claim to general graphs and supports it with IntelliGraph results. There, the ordering is chosen by a topological sort, which is not defined for cyclic graphs and is only one of many possible orders for DAGs. Because the model never sees alternative serializations of the same graph, it cannot learn an order-invariant distribution; it can only learn P(sequence | chosen order). The reported validity rates on IntelliGraph (Table 4) do not measure whether the generated graphs are structurally faithful to the training distribution, so the ordering concern is not refuted by existing experiments. I therefore agree with the reader's weakest assumption. Other concerns, such as the third-party GDSS baseline and missing error bars, are real but less central: they affect the strength of the comparison, not the validity of the framework's own construction. The proposed cross-entropy check would directly test whether the ordering assumption is load-bearing. If the model assigns similar loss to alternative orderings, the concern is resolved; if not, the conclusion should be conditional on datasets with a canonical edge ordering.","tokens_in":13137,"tokens_out":4957,"duration_ms":61325,"concrete_test":"Use the IntelliGraph training graphs, which lack timestamps, and serialize each graph under two different edge orders: the current topological-sort-based order and a reversed (or random) topological order. Compute per-token cross-entropy of the fine-tuned PROVCREATOR model on the alternative ordering without any retraining. If loss on the unseen ordering is substantially higher than on the training ordering, the model has memorized the serialization convention; the framework should then be reported as conditional on a meaningful edge ordering, and the general-graph claim would need qualification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 3.1 the framework assumes 'a well-defined order to visit every edge.' For provenance graphs this is provided by timestamps, but for general graphs the paper proposes a topological sort. A topological sort orders nodes, not edges, and is undefined for cyclic graphs; for the IntelliGraph knowledge graphs no canonical edge order is identified. Because Algorithm 1 trains the autoregressive model on a single deterministic serialization, the model's distribution is over sequences in that specific order. If a user applies PROVCREATOR to a graph dataset without timestamps, the encoder imposes an arbitrary ordering, and the learned sequence distribution may reflect that ordering convention rather than an order-invariant graph distribution. This is load-bearing: the central claim that graph generation is reducible to sequence generation is only valid if the chosen serialization preserves graph semantics. The IntelliGraph evaluation reports only validity percentages, not MMD or other structure metrics, so it cannot detect order-induced artifacts.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents PROVCREATOR, a framework that casts synthetic graph generation for heterogeneous graphs with rich node and edge attributes as autoregressive sequence generation with a transformer-based LLM. It introduces a tokenization scheme with special tokens for graph structure and attributes, a state-machine parser that recovers from invalid tokens, and uses LoRA fine-tuning of LLaMA-3 on serialized graphs. The method is evaluated on provenance graphs from Firefox and PowerShell and on IntelliGraph knowledge graphs, reporting structural MMD metrics, attribute validity rates, embedding cosine similarity, and downstream classification performance. The paper argues that joint generation of structure and semantics yields better fidelity than a diffusion baseline and better validity than IntelliGraph baselines.","tokens_in":13205,"tokens_out":5362,"duration_ms":58461,"significance":"If the results hold, PROVCREATOR is a valuable contribution: it is among the first to apply LLM-based sequence generation to complex heterogeneous graphs with high-dimensional textual attributes, and it demonstrates that joint modeling of topology and attributes is feasible. The compact tokenization and lossless reconstruction (subject to the maximum-node limitation) are useful engineering contributions, and the downstream classification experiment suggests practical utility for data augmentation in security domains. The authors provide a reproducibility code link, which strengthens the paper. However, the significance is tempered by the evaluation gaps detailed below; the central claim of order-invariant general graph generation is not yet fully supported.","major_comments":[{"comment":"The encoding requires a well-defined edge order, and for general graphs the paper proposes a topological sort. A topological sort is defined only for acyclic graphs and it orders nodes rather than edges; many knowledge graphs (including those in IntelliGraph) are cyclic and have no natural edge ordering. The tokenization algorithm therefore imposes an arbitrary, dataset-specific ordering, and the learned autoregressive distribution may reflect that ordering rather than an order-invariant graph distribution. The paper should either restrict the claim to timestamped graphs or provide an order-invariant encoding and demonstrate that generated structures are independent of the chosen order.","section":"3.1, Algorithm 1"},{"comment":"The MMD values are reported without any error bars, confidence intervals, or statistical tests, and the computation of MMD is not described (e.g., the kernel, bandwidth, number of graph samples, and whether the metrics are computed on graphs of matched size). The structural-fidelity claim rests on these numbers, so the reader cannot determine whether the differences between GDSS and PROVCREATOR are significant. Please provide the full MMD estimation protocol and repeat experiments over multiple random seeds.","section":"4.1, Table 2"},{"comment":"Attribute validity is measured solely by regular-expression matching. A generated string can match a regex while being semantically incoherent (for example, an executable name that is syntactically a Windows path but does not correspond to any real binary or to the process's behavior). The paper's claim of capturing 'semantic fidelity' is therefore not substantiated by the reported metric. Consider supplementing with human evaluation or a semantic plausibility check (e.g., known program paths, embedding-based similarity to real attribute distributions).","section":"4.2, Table 3 and Table 6"},{"comment":"The comparison against the IntelliGraph baselines is confounded: the baselines generate attributes as sampled numbers, whereas PROVCREATOR generates attributes as free-form text, and only the best-performing baselines are shown. The footnote acknowledges the difference but the conclusion that PROVCREATOR 'outperforms the baseline models by a large margin' is not justified under a mismatched protocol. Moreover, PROVCREATOR receives 0.00 on syn-tipr and wd-articles; the explanation that evaluation criteria are 'strict' is not quantified. A fair evaluation would use the same attribute representation or, at minimum, report performance under identical validity criteria.","section":"4.3, Table 4"},{"comment":"The tokenizer injects exactly N node tokens to represent 'up to N nodes,' but the paper never states what happens when a graph contains more than N nodes. If such graphs are truncated or dropped, the claimed lossless encoding (Section 1, Section 3.1) is false for a nontrivial portion of the input distribution. Please specify the value of N, the maximum graph size in the datasets, and the handling of oversized graphs.","section":"3.1"}],"minor_comments":[{"comment":"The model name is inconsistent: 'LLama3-3.2-3B' in Section 3 and 'LLaMA3-3.2B' in Section 3.3; please correct and specify the exact checkpoint.","section":"3, 3.3"},{"comment":"There is a typo in the last line of the algorithm: 'Suence S' should be 'sequence S'.","section":"A.1, Algorithm 1"},{"comment":"The header '% Novel & Vaild Graphs' has a typo ('Vaild' should be 'Valid').","section":"Table 4"},{"comment":"The caption does not explain what the 'Baseline' bar represents beyond 'randomly sampled node attributes'; please clarify that the baseline uses real structures with random attributes and report error bars.","section":"Figure 3"},{"comment":"The relationship types are capitalized inconsistently ('READ,WRITE,CREATE andEXECUTE'); please fix the formatting.","section":"2.2"},{"comment":"The in-text citation 'Feng et al.' and some other reference entries are incomplete; ensure all references have full author lists and years.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is promising and the provenance experiments are suggestive, but the evaluation in its current form is not rigorous enough for acceptance. The authors should be encouraged to clarify the scope of the method (timestamped graphs vs. general graphs), to provide a full MMD estimation protocol with error bars, and to make the attribute validity and IntelliGraph comparisons more defensible. I recommend major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"ProvCreator is a credible and useful contribution: it casts heterogeneous attributed graph generation as LLM sequence generation with a compact special-token encoding, and backs it up with real experiments on provenance graphs and knowledge graphs. The downstream fact that training a GNN on synthetic graphs retains 65-70% of real-data F1 is the strongest evidence in the paper. The tokenizer, the recovery parser, and the joint structure-attribute interleaving are all new relative to the cited prior work; I believe the claim that this is the first synthetic provenance graph generator targets a real gap.\n\nWhat the paper does well: it describes the encoding and decoding unambiguously, gives dataset statistics and training cost, and is honest about the IntelliGraph comparison being confounded (footnote), which I appreciate. The evaluation protocol is standard MMD for structure, a regex-based attribute validity check, and a downstream task. The paper is well written and the results are reproducible in principle.\n\nThe soft spots are real but mostly not fatal. The attribute validity metric is only syntactic well-formedness; a random string of the right shape counts as valid, so Table 3 overstates semantic fidelity. There are no error bars or significance tests across the multiple sampled graphs; the MMD numbers in Table 2 look clearly better than GDSS for firefox but are close for powershell on some metrics, so I'd want confidence intervals before trusting the ordering of those columns. The GDSS baseline is a third-party modified implementation, and the paper never cites its source, which makes the comparison hard to audit. The 'privacy-aware' claim in the abstract is not demonstrated: the authors say they will release data after applying privacy-enhancing techniques, which is a plan, not a result.\n\nThe stress-test concern about edge ordering is worth taking seriously, but I'd phrase it as a limitation, not a refutation. For provenance graphs timestamps give a natural order. For general graphs, topological sort of nodes doesn't directly order edges, and for cyclic graphs it's undefined; the paper doesn't discuss this. This means the general-graph claims (IntelliGraph) are the least well-supported part. It's a fair question whether the learned distribution is order-invariant, and the IntelliGraph evaluation only reports validity rates, which don't test semantic generalization. But this is a standard property of autoregressive graph generation and does not undermine the central claim that serialization works for domains where a consistent ordering exists.\n\nFor the provenance domain, the paper delivers. For general graphs, read the IntelliGraph results as preliminary. I'd recommend accepting it for peer review with the expectation of a major revision that adds error bars, documents the baseline, and either drops or relabels the 'privacy-aware' claim.","headline":"ProvCreator is a solid engineering contribution with genuine downstream evidence; the weak spots are evaluation rigor (no error bars, regex validity) and an overbroad general-graph claim, not the core method.","tokens_in":13880,"tokens_out":2449,"would_cite":true,"duration_ms":28811,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PROVCREATOR shows that complex heterogeneous graphs with rich node and edge attributes can be generated by serializing them into token sequences and fine-tuning a transformer LLM, producing synthetic provenance and knowledge graphs that…","keywords":["heterogeneous graph generation","sequence generation","large language models","graph tokenization","provenance graphs","knowledge graphs","synthetic data","attribute fidelity"],"falsifier":"Take a provenance dataset without timestamps and train PROVCREATOR twice on the same graphs: once with the topological-sort ordering and once with a fixed but random edge order. If validity and downstream classification F1 are statistically indistinguishable between the two, the edge-order assumption is not load-bearing; if the random-order model collapses, the ordering is essential. Alternatively, remove the token-filtering and parser recovery during sampling and measure validity; if validity drops sharply, the reported validity rates are a property of the constrained decoding, not of the generation model itself.","tokens_in":12790,"feed_emoji":"🕸️","tokens_out":8776,"duration_ms":86123,"temperature":0.7,"pith_summary":"PROVCREATOR claims that generating complex heterogeneous graphs with rich node and edge attributes can be treated as a language generation problem. The framework serializes each graph into a compact sequence of special tokens and attribute text, then fine-tunes a transformer-based LLM to produce such sequences. On system provenance graphs and IntelliGraph knowledge graphs, it reports lower structural distribution distances than the GDSS diffusion baseline and far higher rates of valid graphs than benchmark baselines. The practical payoff is synthetic graphs that preserve structure-semantics dependencies, enabling privacy-aware data augmentation for security applications.","feed_headline":"LLMs generate complex graphs by treating them as token sequences","feed_subtitle":"Serializing nodes and edges into tokens lets fine-tuned LLMs generate realistic provenance and knowledge graphs","key_machinery":"The central mechanism is the graph-to-sequence encoder and its inverse decoder built on a custom token vocabulary (<bog>, <eog>, <bon>, <eon>, <boe>, <nodex>, <ntypej>, <etypek>, <bof>, <eof>). It serializes a graph by iterating edges in a fixed order—timestamps in provenance data, topological sort otherwise—and interleaving structural markers with attribute text, so the sequence losslessly determines the graph. This compact representation (35% of tokens vs plain JSON on provenance data, 22% on IntelliGraph) fits large graphs into the LLM context window and enables end-to-end learnable generation. Decoding relies on token filtering and a state-machine parser that uses anchor tokens to recover from invalid tokens.","core_discovery":"The central claim is that a lossless graph-to-sequence encoder, combined with a pretrained transformer LLM and a constraint-aware decoder, can jointly model graph topology and high-dimensional node and edge attributes well enough to produce realistic synthetic graphs. The paper operationalizes this by encoding each edge and its attributes into tokens, interleaving node attributes when nodes first appear, using a deterministic edge order (timestamps for provenance graphs, topological sort for general graphs). The LLM is fine-tuned with LoRA on these sequences; at generation time, token filtering and a state-machine parser with anchor tokens ensure structural validity. Evaluations on firefox.exe and powershell.exe provenance graphs show lower maximum-mean-discrepancy scores than GDSS on six structural metrics, attribute validity rates above 93%, and on IntelliGraph datasets, 86–94% valid graphs compared with near zero for the benchmark baselines. A downstream GNN classification task trained on PROVCREATOR-generated graphs retains 65–70% of real-data F1, indicating preserved class signal.","pith_inferences":["An open question the paper leaves implicit is how much of the reported graph-validity rate comes from the constrained decoding (token filtering plus parser recovery) rather than from the generative model itself; an ablation that disables these guards would isolate the model's true capability.","Because the serialization depends on a total edge order, using PROVCREATOR on graphs without timestamps requires imposing a topological order; a robustness test varying the ordering convention would show whether the learned distribution depends on that choice.","The reported graph sizes are in the thousands of tokens, so scaling to enterprise-scale provenance graphs (millions of edges) will likely require hierarchical or chunked generation; the paper does not yet demonstrate this.","The embedding-fidelity comparison uses doc2vec on random walks, which is sensitive to the serialized text; a structure-aware graph embedding would provide a complementary check of whether the synthetic graphs truly match real graph topology."],"forward_implications":["Organizations can generate synthetic provenance graphs for training security ML detectors, sidestepping privacy and sharing restrictions on real syscall traces.","The same serialization recipe should apply to any graph with serializable node and edge attributes and a natural edge order—event logs, knowledge graphs, temporal networks—making LLM-based generation a general graph tool.","Because the backbone is an LLM, users can condition generation with prompts to produce graphs matching specific contexts, such as particular command-line arguments, which diffusion baselines cannot easily do.","The downstream classification result suggests synthetic graphs can be used for data augmentation to mitigate class imbalance, not merely as a stand-in for real graphs."],"supporting_citations":[{"why":"Baseline GDSS diffusion model; PROVCREATOR's structural-fidelity improvement is measured against its MMD scores on the same provenance graphs.","marker":"Jo et al., 2022"},{"why":"IntelliGraph benchmark provides the knowledge-graph datasets and baseline percentages (validity, novelty, empty graphs) that PROVCREATOR's results are compared with.","marker":"Thanapalasingam et al., 2023"},{"why":"Earlier work representing graphs as plain text; cited to justify PROVCREATOR's more compact token vocabulary.","marker":"Chen et al., 2023"},{"why":"Also prior text-based graph representation; part of the plain-text encoding approach that PROVCREATOR's tokenizer improves upon.","marker":"Wang et al., 2023"},{"why":"Defines doc2vec, used to compute the graph-embedding cosine-similarity fidelity measure in the paper's evaluation.","marker":"Le & Mikolov, 2014a"}],"fun_headline_variants":["LLMs synthesize graphs by tokenizing nodes and edges","ProvCreator turns graph synthesis into a sequence task","Tokenizing graphs unlocks LLM generation of complex data","Transformers generate heterogeneous graphs from token sequences","ProvCreator: LLM-based graph generator with high fidelity"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The model assumes that a fixed edge ordering (by timestamps or topological sort) encodes the semantic dependencies between structure and attributes; if the ordering is arbitrary or uninformative, the LLM must learn a convention that may not generalize, and the validity results could be an artifact of the decoding constraints rather than of the learned distribution.","fun_headline_variants_meta":{"raw":{"variants":["LLMs synthesize graphs by tokenizing nodes and edges","ProvCreator turns graph synthesis into a sequence task","Tokenizing graphs unlocks LLM generation of complex data","Transformers generate heterogeneous graphs from token sequences","ProvCreator: LLM-based graph generator with high fidelity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000176,"raw_usage":{"total_tokens":1289,"prompt_tokens":944,"completion_tokens":345,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":560,"completion_tokens_details":{"reasoning_tokens":271}},"tokens_in":560,"tokens_out":345,"duration_ms":4087,"temperature":1.0,"reasoning_tokens":271,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T13:03:52.923179+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a provenance dataset without timestamps and train PROVCREATOR twice on the same graphs: once with the topological-sort ordering and once with a fixed but random edge order. If validity and downstream classification F1 are statistically indistinguishable between the two, the edge-order assumption is not load-bearing; if the random-order model collapses, the ordering is essential. Alternatively, remove the token-filtering and parser recovery during sampling and measure validity; if validity drops sharply, the reported validity rates are a property of the constrained decoding, not of the generation model itself.","supporting_citations":[{"cited_title":"IntelliGraphs: Datasets for Benchmarking Knowledge Graph Generation","cited_arxiv_id":"2307.06698","evidence_quote":"IntelliGraph benchmark provides the knowledge-graph datasets and baseline percentages (validity, novelty, empty graphs) that PROVCREATOR's results are compared with."}],"review_version":1}