PluRel: Synthetic Data unlocks Scaling Laws for Relational Foundation Models

Carlos Guestrin; Johannes Hoffart; Jure Leskovec; Rishabh Ranjan; Valter Hudovernik; Vignesh Kothapalli; Vijay Prakash Dwivedi

REVIEW 4 major objections 4 minor 8 cited by

Synthetic relational databases, generated from scratch, unlock power-law scaling for relational foundation models and improve real-world predictions.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 04:46 UTC pith:CDAQLDRN

load-bearing objection Worth engaging: PLUREL is a real from-scratch synthetic relational database generator, the scaling-law fits are honest but describe the generator prior, and the real-data transfer evidence is suggestive rather than conclusive. the 4 major comments →

arxiv 2602.04029 v2 pith:CDAQLDRN submitted 2026-02-03 cs.DB cs.AIcs.LG

PluRel: Synthetic Data unlocks Scaling Laws for Relational Foundation Models

Vignesh Kothapalli , Rishabh Ranjan , Valter Hudovernik , Vijay Prakash Dwivedi , Johannes Hoffart , Carlos Guestrin , Jure Leskovec This is my paper

classification cs.DB cs.AIcs.LG

keywords synthetic relational databasesrelational foundation modelsscaling lawsdata diversitystructural causal modelshierarchical stochastic block modelspretrainingmasked token prediction

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Relational foundation models need large, diverse databases for pretraining, but such databases are rarely public due to privacy. This paper introduces PLUREL, a three-stage framework that synthesizes new relational databases from scratch, generating schemas, primary–foreign key connections, and feature values. Using PLUREL to pretrain a relational foundation model, the authors report that pretraining loss follows a power law in both the number of databases (diversity) and the total token count (size), and that scaling the number of synthetic databases improves zero-shot generalization to real databases. They also claim that initializing on synthetic data before continued pretraining on real data beats real-data-only pretraining, with average gains of +1.2% AUROC and +3.0% R². If true, this offers a privacy-compatible path to building the large-scale relational training corpora that foundation models need.

Core claim

On its own terms, the paper claims that relational foundation model pretraining loss obeys separable power laws in database diversity and token count: L(N)=A_N N^{-alpha_N}+C_N and L(S)=A_S S^{-alpha_S}+C_S. Using PLUREL to generate databases at scale, the authors fit these laws on up to 1024 databases and 32B tokens, hold out the largest points, and show that extrapolation predicts loss at 2x data scale. They further show that scaling the number of synthetic databases consistently lowers validation loss on real-world tasks and improves zero-shot AUROC and R², and that synthetic pretraining followed by continued pretraining on real data outperforms real-data-only pretraining on the majority

What carries the argument

PLUREL is a three-stage generative pipeline: (1) schemas are sampled from directed acyclic graph priors (e.g., Barabasi-Albert, random trees, Watts-Strogatz); (2) primary–foreign key connectivity between rows is modeled with hierarchical stochastic block models, creating clustered, event-like dependencies; (3) table features are generated by structural causal models whose source nodes carry temporal trend/cycle/noise inputs and whose mechanisms project parent values into a shared latent space before reconstructing numeric or categorical cells. This pipeline is CPU-only and lightweight, and each database is controlled by a scalar seed, so arbitrarily many diverse databases can be generated fo

Load-bearing premise

The load-bearing premise is that PLUREL's hand-chosen generator prior—DAG-only schemas, hierarchical-block foreign keys, and SCM features with the hyperparameter ranges listed in Table 2—is representative enough of real relational databases that scaling behavior measured on held-out synthetic data transfers to real-world data; this is verified only indirectly through 18 real tasks, with no coverage or distribution-shift analysis.

What would settle it

Train a relational foundation model under PLUREL at 2048 or 4096 synthetic databases and 64B tokens; if the measured validation loss clearly deviates from the fitted power-law line, the claimed scaling law is falsified. Alternatively, if a synthetic-only model trained on a much larger and more diverse corpus fails to improve (or worsens) zero-shot AUROC and R² on the same real tasks, the claim that synthetic diversity transfers to real data is called into question.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Scaling both diversity and size matters: increasing the number of databases while fixing tokens, or tokens while fixing databases, leads to non-monotonic loss; the two axes must be scaled together.
The fitted power laws extrapolate reasonably to held-out larger scales (2x beyond the training grid), suggesting predictable returns from more synthetic data.
Increasing the number of synthetic databases improves zero-shot performance on real-world relational tasks, indicating transfer from synthetic to real data.
Synthetic pretraining is a useful warm start: continued pretraining on real data from a synthetic checkpoint beats real-data-only pretraining on average, with the largest gains on regression tasks.
Synthetic-only pretraining is insufficient on its own; models trained only on synthetic data underperform real-data-only baselines on most tasks, so real-data alignment is still needed.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

If the scaling laws hold beyond the tested grid, synthetic generation could become the standard pretraining corpus for relational foundation models, bypassing privacy constraints that currently block data sharing.
The three-stage design invites ablation experiments: isolating the effect of schema topology, foreign-key clustering, or temporal structure on the fitted exponents could reveal which aspects of relational data most drive downstream transfer.
A direct testable extension is widening the generator prior to include cyclic schemas, self-referencing foreign keys, or text columns; the paper already identifies these as unsupported, so measuring whether the power-law exponents change would sharpen the theory.
The non-monotonic behavior when one axis is fixed suggests a compute-optimal frontier analogous to language-model scaling; fitting a joint scaling law with interaction terms could yield a principled recipe for allocating tokens across synthetic databases.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Worth engaging: PLUREL is a real from-scratch synthetic relational database generator, the scaling-law fits are honest but describe the generator prior, and the real-data transfer evidence is suggestive rather than conclusive.

read the letter

Quick take: worth engaging. The new thing is PLUREL — a from-scratch generator for multi-table relational databases, covering schema graph, primary–foreign key connectivity, and feature columns, instead of regenerating one real database. That fills a real hole for RFM pretraining, and the generator is cheap and produces visibly diverse schemas and data. The first RFM scaling-law fits are also genuinely new: they fit L(N) and L(S) as power laws, hold out N=1024 and S=32B, and the extrapolations land reasonably. The continued-pretraining result on RelBench (+1.2% AUROC, +3.0% R2 vs real-only) is plausible with standard errors in Appendix D.1, and the appendix's second base model makes the post-hoc base selection a minor blemish rather than a serious flaw.

The soft spots, in proportion. The scaling laws are fit on held-out synthetic RDBs drawn from the same hand-specified generator prior. They are honest scaling laws for PLUREL, not established laws for relational data in general; the paper doesn't quantify the shift to real databases. Appendix A admits self-loops and cyclic schemas, present in real RelBench (rel-stack posts.ParentID), are excluded. The real-data bridge is 18 tasks on six RelBench databases, with mean gains but also negative cases, and Figure 4c shows real-data validation loss saturating, non-monotonic at small N, which the authors attribute to out-of-distribution data. That is a fair caveat, but it means the claim 'scaling synthetic databases improves real-data generalization' rests on downstream task metrics, not on the scaling-law fit itself. The scaling fits also use few points with no uncertainty quantification; I'd call that standard but worth stating.

Overall, the paper is not circular: the scaling-law parameters are fitted to observed loss, the held-out points are real extrapolations under the functional form, and RelBench evaluation grounds the transfer. The citation pattern is appropriate — self-cites to RT and RelBench are on point.

This is a paper for the RFM/tabular-synthesis community and for anyone who wants to pretrain a relational model without real data. I'd send it to reviewers; the main request would be a distribution-shift or coverage analysis between PLUREL schemas and real databases, plus honest caveats on the scope of the scaling laws. With that, it's a solid contribution.

Referee Report

4 major / 4 minor

Summary. The paper introduces PLUREL, a three-stage generator for synthetic relational databases: (1) schema graphs sampled from DAG priors, (2) row-level primary–foreign key connectivity via hierarchical stochastic block models, and (3) feature values generated by table-wise Structural Causal Models with temporal exogenous inputs. The authors pretrain a Relational Transformer (RT) on up to 1024 synthetic databases and 32B tokens, and report two scaling laws: pretraining loss L(N) = A_N N^{-α_N} + C_N and L(S) = A_S S^{-α_S} + C_S (Eqs. 6-7). They further report that models pretrained on larger/more diverse synthetic corpora achieve lower MTP loss and better zero-shot AUROC/R² on 18 RelBench tasks, and that continued pretraining on real data from a synthetic checkpoint improves over real-only pretraining by +1.2 AUROC and +3.0 R² on average (Table 1). The main positive results are the synthetic-data scaling curves and the continued-pretraining gains, while the main limitation is that the scaling laws are established only on data from the hand-designed PLUREL prior.

Significance. If the transfer claims hold, this is a significant contribution: it attacks the data bottleneck for relational foundation models and provides a lightweight, reproducible synthetic-data pipeline. The paper is unusually thorough on compute: 56 pretraining runs, a held-out extrapolation point for each scaling curve, standard errors for the main results (App. D.1), and a robustness check with an alternative base model. The code, models, and data are released, and the authors are explicit about limitations (no cycles/self-loops, no text, no formal proofs). The main risk is that the scaling laws are fit to the synthetic prior and their connection to real-world behavior is only qualitative; the paper's own Figure 4c shows saturation and non-monotonicity on real data. This does not invalidate the synthetic-data findings, but it does mean the central 'unlocks scaling laws' claim needs either a quantitative transfer analysis or a more careful rephrasing.

major comments (4)

[Sec. 3.1 (Eqs. 6-7) and Fig. 4] The scaling laws in Eqs. (6)-(7) are fit to held-out synthetic RDBs sampled from the same PLUREL generator. The abstract's claim that scaling N improves generalization to real databases is supported only by Fig. 4, where the real-data validation loss saturates and is non-monotonic for N∈{8,16,32} (the authors attribute this to RelBench being out-of-distribution, Sec. 3.2). This leaves open the possibility that the laws describe the generator rather than relational data in general. Please provide a direct transfer test: fit the same power-law form to the real-data loss from Fig. 4c and compare exponents, or at least report the correlation between synthetic held-out loss and real-data loss across the grid. If that is not possible, the abstract/conclusion should separate the synthetic scaling law from the qualitative transfer result.
[Sec. 3.1 and Fig. 1] The power-law fits use 6-7 points with no uncertainty intervals on A_N, α_N, C_N (and similarly for S). Because L(N)=min_S L(N,S) is taken over a discrete 8×7 grid, each frontier point for N depends on a single S value; grid sparsity can materially change the fitted exponents. The held-out extrapolation to N=1024/S=32B (2× the fitted range) is reported without confidence bands. Please provide bootstrap or profile-likelihood intervals for the parameters, report the full L(N,S) table or surface, and state how many grid points determine each frontier point.
[Sec. 3.3 and Table 1] The base model for continued pretraining (1024 RDBs, 4B tokens) is selected post hoc to maximize the worst validation metric (Sec. 3.3). Although App. D.1 reports robustness with one alternative base (512 RDBs, 32B tokens), the selection rule is not prespecified and the main table reports only the selected model. Please report the distribution of synthetic+real gains over all scaling checkpoints, or a prespecified selection rule, to quantify selection bias.
[Appendix A and Table 2] PLUREL excludes self-loops/cycles (e.g., rel-stack posts.ParentID) and text/semantic columns, and the authors themselves connect this to performance drops (Sec. 3.3: 'lack of textual information and column semantics'). Because the transfer claim rests on representativeness of the prior, this limitation should be quantified (e.g., what fraction of RelBench schemas contain cycles/self-loops or text columns) and discussed as a boundary condition of the scaling law, not only in the appendix.

minor comments (4)

[Sec. 2.3.2, Eq. (5)] The set M(i) is defined, but Eq. (5) uses |P(i)| in the sum index. Should be |M(i)|, or P(i) must be defined.
[Appendix B.1] Typo: 'W A graphs' should be 'WS graphs' (Watts-Strogatz).
[Table 2] Watts-Strogatz rewire probability is listed as 'constant uniform [0.1, 0.3]'. This is contradictory; it should be 'uniform'.
[Sec. 3.2 and Fig. 4c] The statement that the loss 'saturates' should be quantified (e.g., plateau threshold or slope), since the same figure shows non-monotonic behavior for small N.

Circularity Check

0 steps flagged

Scaling laws are empirical fits with held-out extrapolation and external RelBench validation; no circular reduction found.

full rationale

The paper's central claims (Eqs. 6-7) are power-law fits to measured validation losses, not quantities defined into existence. Section 3.1 explicitly fits A_N, alpha_N, C_N on 7 (N, L(N)) points and the size-law parameters on 6 (S, L(S)) points while keeping N=1024 and S=32B held out, so those points are genuine extrapolations under the assumed functional form rather than predictions forced by construction. The transfer claim is tested against an external benchmark: 18 RelBench tasks not drawn from the PLUREL prior (Section 3.2, Table 1), with mean AUROC/R2 gains reported over real-only baselines. The principal self-citations (RT architecture, leave-one-DB-out protocol, and the Hudovernik et al. 2025 observation motivating HSBM connectivity) are background or methodological references and do not force the scaling result; no uniqueness theorem is imported and no fitted parameter is renamed as a prediction. The paper also admits limitations: Appendix A excludes self-loops and cyclic schemas, and Figure 4c shows saturation attributed to RelBench being out-of-distribution. These are external-validity and prior-representativeness concerns, not circularity; they may weaken the generality of the scaling law but do not make the derivation equivalent to its own inputs. A caveat remains that if the hand-specified PLUREL prior is unrepresentative, the measured scaling behavior describes PLUREL rather than real-world RDBs, but this is a generalization risk, not a circular reduction.

Axiom & Free-Parameter Ledger

9 free parameters · 6 axioms · 0 invented entities

The central scaling and transfer claims rest on a large hand-chosen hyperparameter prior (Table 2) and on the assumption that synthetic databases resemble real ones. The only numbers actually fitted to data are the six scaling-law constants in Fig. 1. No new physical entities are introduced.

free parameters (9)

Schema graph prior families = uniform {Barabasi-Albert, Reverse Random-Tree, Watts-Strogatz}
Hand-chosen set determines structural diversity of generated schemas; Table 2.
Table, row, and column metadata ranges = tables [3,20]; entity rows [500,1000]; activity rows [2000,5000]; columns power-law [3,40]; NULL rate [0.01,0.1]
Hand-chosen ranges in Stage 1 metadata; control database size and shape.
SCM causal graph prior and feature-node fraction = uniform {Layered, Erdos-Renyi, Barabasi-Albert, Random-Tree, Reverse Random-Tree}; feature node % [0.3,0.9]
Determines the causal structure of generated feature columns.
MLP and embedding hyperparameters = hidden dim 32, depth 2, activations {relu, elu, silu, softsign, tanh}, multiple initializations
Randomly initialized projections affect feature distributions but are not tuned.
Exogenous input prior = Beta(0.5,0.5), Beta(2,2), Beta(2,3), Beta(2,4), Beta(4,1); edge weights N(0,1)
Hand-picked latent noise distributions for SCM aggregation.
HSBM connectivity parameters = levels [1,5]; blocks per level [1,3]; diagonal 0.9, off-diagonal U(0.001,0.002)
Controls row-level foreign-key locality; motivated by real-world observations.
Temporal pattern parameters = trend exponent [0,2]; scales -1,1; cycle frequencies {0.1,...,1.0}; noise scales 0.05/1.0
Hand-specified temporal priors for activity vs entity tables.
Scaling-law constants = A_N, alpha_N, C_N, A_S, alpha_S, C_S (fit in Fig. 1)
Fitted to observed losses using the Kaplan et al. procedure; central to the power-law claim.
RT pretraining hyperparameters = 12 layers, batch 128, context 1024, BFS width 128, lr 5e-4, weight decay 0.1, warmup 0.2
Fixed architecture and optimizer choices; scaling behavior may depend on them.

axioms (6)

ad hoc to paper Power-law functional forms L(N)=A_N N^-alpha_N + C_N and L(S)=A_S S^-alpha_S + C_S hold for RFM pretraining on PLUREL data.
Assumed in Eqs. (6)-(7), motivated by Kaplan et al.; not derived. Fit to 6-7 points with two held-out points.
domain assumption Validation loss on held-out synthetic RDBs is a meaningful proxy for RFM capability and real-world generalization.
Scaling curves in Section 3.1 are computed on synthetic held-out RDBs; Section 3.2 separately checks real RelBench, so the proxy assumption is only indirectly supported.
domain assumption DAG-only schemas, with no cycles or self-loops, cover enough real relational structure.
Explicit limitation in Section 2.1 and Appendix A; real rel-stack has a self-loop on posts, yet transfer is measured on such data.
domain assumption HSBM hierarchies and SCM causal mechanisms capture the multi-scale dependencies RFMs need.
Stage 2 and Stage 3 design choices based on prior observations; not validated directly against real connectivity.
domain assumption Row indices as primary keys and abstract column names do not block transfer to semantically named real tables.
Authors themselves hypothesize that lack of textual semantics explains task degradations in Section 3.3 and Appendix D.1.
domain assumption Temporal exogenous inputs (trend, cycle, fluctuation) are sufficient to model row correlations.
Introduced in Section 2.3.1; no comparison to other temporal correlation models.

pith-pipeline@v1.3.0-alltime-deepseek · 20525 in / 13989 out tokens · 127555 ms · 2026-08-03T04:46:23.894141+00:00 · methodology

0 comments

read the original abstract

Relational Foundation Models (RFMs) facilitate data-driven decision-making by learning from complex multi-table databases. However, the diverse relational databases needed to train such models are rarely public due to privacy constraints. While there are methods to generate synthetic tabular data of arbitrary size, incorporating schema structure and primary-foreign key connectivity for multi-table generation remains challenging. Here we introduce PLUREL, a framework to synthesize multi-tabular relational databases from scratch. In a step-by-step fashion, PLUREL models (1) schemas with directed graphs, (2) inter-table primary-foreign key connectivity with bipartite graphs, and, (3) feature distributions in tables via conditional causal mechanisms. The design space across these stages supports the synthesis of a wide range of diverse databases, while being computationally lightweight. Using PLUREL, we observe for the first time that (1) RFM pretraining loss exhibits power-law scaling with the number of synthetic databases and total pretraining tokens, (2) scaling the number of synthetic databases improves generalization to real databases, and (3) synthetic pretraining yields strong base models for continued pretraining on real databases. Overall, our framework and results position synthetic data scaling as a promising paradigm for RFMs.

Figures

Figures reproduced from arXiv: 2602.04029 by Carlos Guestrin, Johannes Hoffart, Jure Leskovec, Rishabh Ranjan, Valter Hudovernik, Vignesh Kothapalli, Vijay Prakash Dwivedi.

**Figure 1.** Figure 1: (Left) Pretraining loss L scales as a power law with both (1) the number of synthetic databases N and (2) the pretraining dataset size S, when not bottle-necked by the other. See Section 3.1 for details. (Right) On real-world predictive tasks, PLUREL-based synthetic pretraining followed by continued pretraining on real data outperforms real data pretraining alone. See Section 3.3 for details. through a com… view at source ↗

**Figure 2.** Figure 2: The PLUREL framework. Stage 1 generates a schema by sampling a directed graph G and populating the metadata with row and column counts. In Stage 2, the foreign key columns are populated using a bipartite graph between rows of parent–child table pairs, each edge representing a primary–foreign key (P→F) link. In Stage 3, we follow a topological ordering of tables in G and leverage Structural Causal Models (S… view at source ↗

**Figure 3.** Figure 3: Synthesizing RDBs with PLUREL results in diverse data distributions across feature column values. current implementation focuses on numeric and categorical features, the framework can naturally extend to these richer data modalities by augmenting the SCM mechanisms. 2.3.1. MODELING TEMPORAL PATTERNS Features in real-world databases often exhibit correlations across rows due to temporally related events. We… view at source ↗

**Figure 4.** Figure 4: Validation loss and zero-shot performance on RelBench tasks. The synthetic pretraining dataset sizes (in billions of tokens) are varied along with the number of PLUREL RDBs to obtain the scaling curves. (↓)/(↑) indicates that lower/higher values are better. curated task tables to predict future outcomes. For example, masking cells in the item-churn table of rel-amazon trains the model to predict whether a … view at source ↗

**Figure 5.** Figure 5: QK-Norm mitigates early overfitting with leave-one-db-out pretraining during the baseline runs and also improves the peak performance. AUROC (%) on the val/test splits of rel-stack/user-engagement (a, b) and rel-stack/user-badge (c, d) tasks highlights the mitigation of overfitting. R2 (%) on the val/test splits of rel-stack/post-votes (e, f) and rel-f1/driver-position (g, h) tasks shows improvements to pe… view at source ↗

discussion (0)

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Towards Anomaly Detection on Relational Data
cs.LG 2026-06 unverdicted novelty 7.0

RelAD is a reconstruction-based framework for anomaly detection on relational data that combines conditional sparse-gated attribute reconstruction with dual-view multi-relational edge reconstruction and outperforms ba...
RelBench v2: A Large-Scale Benchmark and Repository for Relational Data
cs.LG 2026-02 unverdicted novelty 7.0

RelBench v2 expands a relational deep learning benchmark with four new large datasets and autocomplete tasks, showing models that use table relationships outperform single-table baselines.
Curriculum Matters: Data-Efficient Relational PFN Pretraining with Synthetic Data
cs.LG 2026-07 conditional novelty 6.0

A width-progressive curriculum over synthetic tables reaches 0.703 ROC-AUC with ~13,300 tables on 23 tabular tasks, while the same data all-at-once reaches 0.541; a single-table model also nearly matches relational pr...
Parameter-Free Encoders Remain Viable for RDB Foundation Models
cs.LG 2026-07 accept novelty 6.0

Trainable RDB encoders cannot robustly exploit neighborhood labels as fixed foundation-model features or feature-importance signals, so simple parameter-free encoders stay near-SOTA.
Universal Encoders for Modular Relational Deep Learning
cs.LG 2026-06 unverdicted novelty 6.0

Proposes a pretrained Universal Row Encoder using transformers and global statistics to generate table-width invariant row embeddings for modular relational graph models, claiming improved transfer, convergence, and m...
PluRel-to-RDB-PFN: Schema-Guided Synthetic Relational Pretraining
cs.LG 2026-07 conditional novelty 5.0

External synthetic relational data from PluRel, when curated with a real-world-schema-first curriculum, recovers 87.6-93.8% of RDB-PFN's performance using ~33K tasks instead of ~1.8M.
Parameter-Free Encoders Remain Viable for RDB Foundation Models
cs.LG 2026-07 conditional novelty 5.0

A parameter-free relational encoder combined with a single-table foundation model matches or beats parameterized, pre-trained RDB encoders on six benchmarks.
RelGT-AC: A Relational Graph Transformer for Autocomplete Tasks in Relational Databases
cs.AI 2026-06 unverdicted novelty 5.0

RelGT-AC adds column masking, unified task head, and TF-IDF encoding to RelGT, outperforming GraphSAGE on regression autocomplete tasks and gaining up to 10 AUROC on text-heavy tasks across RelBench v2 datasets.

Reference graph

Works this paper leans on

17 extracted references · 8 linked inside Pith · cited by 7 Pith papers

[1]

When pretrained using synthetic data followed by real data, the model consistently improves upon the performance of RT trained solely on real data

We report the mean and standard error across three random seeds. When pretrained using synthetic data followed by real data, the model consistently improves upon the performance of RT trained solely on real data. On certain tasks, we observe slight degradations in performance. Notably, these tasks align with those for which the original authors report per...

2025
[2]

Parameter Kind Sampling Choices Database Schema graph priors (PG)set uniform{Barabasi-Albert, Reverse Random-Tree, Watts-Strogatz} Num tablesrange uniform [3, 20] Num rows (entity tables)range uniform [500, 1000] Num rows (activity tables)range uniform [2000, 5000] Num columnsrange power-law [3, 40] Min timestampconstant-1990-01-01 Max timestampconstant-2...

2000
[4]

Gueye, M., Attabi, Y ., and Dumas, M

URL https://arxiv.org/ abs/2511.08667. Gueye, M., Attabi, Y ., and Dumas, M. Row conditional-tgan for generating synthetic relational databases. InICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5. IEEE,

Pith/arXiv arXiv 2023
[5]

R., Pawar, S

Henry, A., Dachapally, P. R., Pawar, S. S., and Chen, Y . Query-key normalization for transformers. InFindings of the Association for Computational Linguistics: EMNLP 2020, pp. 4246–4253,

2020
[6]

Relational data generation with graph neural networks and latent diffusion models

Hudovernik, V . Relational data generation with graph neural networks and latent diffusion models. InNeurIPS 2024 Third Table Representation Learning Workshop,

2024
[8]

Ma, J., Thomas, V ., Hosseinzadeh, R., Labach, A., Cress- well, J

URL https://arxiv.org/ abs/2412.19437. Ma, J., Thomas, V ., Hosseinzadeh, R., Labach, A., Cress- well, J. C., Golestan, K., Yu, G., Caterini, A. L., and V olkovs, M. TabDPT: Scaling tabular foundation models on real data. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems,

Pith/arXiv arXiv
[9]

Robinson, J., Ranjan, R., Hu, W., Huang, K., Han, J., Dobles, A., Fey, M., Lenssen, J

URL https://arxiv.org/abs/ 2510.06377. Robinson, J., Ranjan, R., Hu, W., Huang, K., Han, J., Dobles, A., Fey, M., Lenssen, J. E., Yuan, Y ., Zhang, Z., et al. Relbench: A benchmark for deep learning on relational databases.Advances in Neural Information Processing Systems, 37:21330–21341,

arXiv
[10]

URL https://arxiv.org/abs/2505. 15668. Schambach, M., Paul, D., and Otterbach, J. Scaling ex- periments in self-supervised cross-table representation learning. InNeurIPS 2023 Second Table Representation Learning Workshop,

2023
[11]

Team, G., Kamath, A., Ferret, J., Pathak, S., Vieillard, N., Merhej, R., Perrin, S., Matejovicova, T., Ram ´e, A., Rivi`ere, M., et al

URL https://arxiv.org/ abs/2405.09818. Team, G., Kamath, A., Ferret, J., Pathak, S., Vieillard, N., Merhej, R., Perrin, S., Matejovicova, T., Ram ´e, A., Rivi`ere, M., et al. Gemma 3 technical report,

Pith/arXiv arXiv
[12]

Wang, M., Gan, Q., Wipf, D., Zhang, Z., Faloutsos, C., Zhang, W., Zhang, M., Cai, Z., Li, J., Mao, Z., et al

URL https://arxiv.org/abs/2503.19786. Wang, M., Gan, Q., Wipf, D., Zhang, Z., Faloutsos, C., Zhang, W., Zhang, M., Cai, Z., Li, J., Mao, Z., et al. 4dbinfer: A 4d benchmarking toolbox for graph-centric predictive modeling on rdbs.Advances in Neural Infor- mation Processing Systems, 37:27236–27273,

Pith/arXiv arXiv
[13]

Zhai, X., Kolesnikov, A., Houlsby, N., and Beyer, L

URL https://arxiv.org/ abs/2505.09388. Zhai, X., Kolesnikov, A., Houlsby, N., and Beyer, L. Scaling vision transformers. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12104–12113,

Pith/arXiv arXiv
[15]

The binary mask M∈ {0,1}n×n specifies allowable token interactions, with M[q, k] = 1indicating that token q can attend to token k

(9) Here Q,K∈R n×dk and V∈R n×dv denote the query, key, and value matrices, wheren is the context length. The binary mask M∈ {0,1}n×n specifies allowable token interactions, with M[q, k] = 1indicating that token q can attend to token k. For example, causal language models use Mcausal[q, k] =1{k≤q} .Column Attention.Restricts attention to tokens within the...

2025
[2020]

URL https://arxiv.org/abs/2001. 08361. Kent, W. Consequences of assuming a universal relation. ACM Transactions on Database Systems (TODS), 6(4): 539–556,

2001
[2022]

Cai, K., Xiao, X., and Cormode, G

URL https://arxiv.org/ abs/2108.07258. Cai, K., Xiao, X., and Cormode, G. Privlava: synthesizing relational data with foreign keys under differential privacy. Proceedings of the ACM on Management of Data, 1(2): 1–25,

Pith/arXiv arXiv
[2023]

Barab´asi, A.-L

URL https://arxiv.org/abs/2303.08774. Barab´asi, A.-L. and Albert, R. Emergence of scaling in random networks.science, 286(5439):509–512,

Pith/arXiv arXiv
[2024]

and suggesting that post-hoc model selection would be ideal. D.2. Architectural Improvements: Query-Key Normalization The RT architecture supports multi-modal input representations for text, numeric, and boolean cell tokens, with type-specific encoders. During synthetic pretraining with such multi-modal(type) input tokens, we observed that zero-shot gener...

2020
[2025]

URL https://arxiv.org/abs/ 2405.13396. Dove, E. S. and Phillips, M. Privacy law, data sharing policies, and medical data: a comparative perspective. In Medical data privacy handbook, pp. 639–678. Springer,

Pith/arXiv arXiv

[1] [1]

When pretrained using synthetic data followed by real data, the model consistently improves upon the performance of RT trained solely on real data

We report the mean and standard error across three random seeds. When pretrained using synthetic data followed by real data, the model consistently improves upon the performance of RT trained solely on real data. On certain tasks, we observe slight degradations in performance. Notably, these tasks align with those for which the original authors report per...

2025

[2] [2]

Parameter Kind Sampling Choices Database Schema graph priors (PG)set uniform{Barabasi-Albert, Reverse Random-Tree, Watts-Strogatz} Num tablesrange uniform [3, 20] Num rows (entity tables)range uniform [500, 1000] Num rows (activity tables)range uniform [2000, 5000] Num columnsrange power-law [3, 40] Min timestampconstant-1990-01-01 Max timestampconstant-2...

2000

[3] [4]

Gueye, M., Attabi, Y ., and Dumas, M

URL https://arxiv.org/ abs/2511.08667. Gueye, M., Attabi, Y ., and Dumas, M. Row conditional-tgan for generating synthetic relational databases. InICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5. IEEE,

Pith/arXiv arXiv 2023

[4] [5]

R., Pawar, S

Henry, A., Dachapally, P. R., Pawar, S. S., and Chen, Y . Query-key normalization for transformers. InFindings of the Association for Computational Linguistics: EMNLP 2020, pp. 4246–4253,

2020

[5] [6]

Relational data generation with graph neural networks and latent diffusion models

Hudovernik, V . Relational data generation with graph neural networks and latent diffusion models. InNeurIPS 2024 Third Table Representation Learning Workshop,

2024

[6] [8]

Ma, J., Thomas, V ., Hosseinzadeh, R., Labach, A., Cress- well, J

URL https://arxiv.org/ abs/2412.19437. Ma, J., Thomas, V ., Hosseinzadeh, R., Labach, A., Cress- well, J. C., Golestan, K., Yu, G., Caterini, A. L., and V olkovs, M. TabDPT: Scaling tabular foundation models on real data. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems,

Pith/arXiv arXiv

[7] [9]

Robinson, J., Ranjan, R., Hu, W., Huang, K., Han, J., Dobles, A., Fey, M., Lenssen, J

URL https://arxiv.org/abs/ 2510.06377. Robinson, J., Ranjan, R., Hu, W., Huang, K., Han, J., Dobles, A., Fey, M., Lenssen, J. E., Yuan, Y ., Zhang, Z., et al. Relbench: A benchmark for deep learning on relational databases.Advances in Neural Information Processing Systems, 37:21330–21341,

arXiv

[8] [10]

URL https://arxiv.org/abs/2505. 15668. Schambach, M., Paul, D., and Otterbach, J. Scaling ex- periments in self-supervised cross-table representation learning. InNeurIPS 2023 Second Table Representation Learning Workshop,

2023

[9] [11]

Team, G., Kamath, A., Ferret, J., Pathak, S., Vieillard, N., Merhej, R., Perrin, S., Matejovicova, T., Ram ´e, A., Rivi`ere, M., et al

URL https://arxiv.org/ abs/2405.09818. Team, G., Kamath, A., Ferret, J., Pathak, S., Vieillard, N., Merhej, R., Perrin, S., Matejovicova, T., Ram ´e, A., Rivi`ere, M., et al. Gemma 3 technical report,

Pith/arXiv arXiv

[10] [12]

Wang, M., Gan, Q., Wipf, D., Zhang, Z., Faloutsos, C., Zhang, W., Zhang, M., Cai, Z., Li, J., Mao, Z., et al

URL https://arxiv.org/abs/2503.19786. Wang, M., Gan, Q., Wipf, D., Zhang, Z., Faloutsos, C., Zhang, W., Zhang, M., Cai, Z., Li, J., Mao, Z., et al. 4dbinfer: A 4d benchmarking toolbox for graph-centric predictive modeling on rdbs.Advances in Neural Infor- mation Processing Systems, 37:27236–27273,

Pith/arXiv arXiv

[11] [13]

Zhai, X., Kolesnikov, A., Houlsby, N., and Beyer, L

URL https://arxiv.org/ abs/2505.09388. Zhai, X., Kolesnikov, A., Houlsby, N., and Beyer, L. Scaling vision transformers. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12104–12113,

Pith/arXiv arXiv

[12] [15]

The binary mask M∈ {0,1}n×n specifies allowable token interactions, with M[q, k] = 1indicating that token q can attend to token k

(9) Here Q,K∈R n×dk and V∈R n×dv denote the query, key, and value matrices, wheren is the context length. The binary mask M∈ {0,1}n×n specifies allowable token interactions, with M[q, k] = 1indicating that token q can attend to token k. For example, causal language models use Mcausal[q, k] =1{k≤q} .Column Attention.Restricts attention to tokens within the...

2025

[13] [2020]

URL https://arxiv.org/abs/2001. 08361. Kent, W. Consequences of assuming a universal relation. ACM Transactions on Database Systems (TODS), 6(4): 539–556,

2001

[14] [2022]

Cai, K., Xiao, X., and Cormode, G

URL https://arxiv.org/ abs/2108.07258. Cai, K., Xiao, X., and Cormode, G. Privlava: synthesizing relational data with foreign keys under differential privacy. Proceedings of the ACM on Management of Data, 1(2): 1–25,

Pith/arXiv arXiv

[15] [2023]

Barab´asi, A.-L

URL https://arxiv.org/abs/2303.08774. Barab´asi, A.-L. and Albert, R. Emergence of scaling in random networks.science, 286(5439):509–512,

Pith/arXiv arXiv

[16] [2024]

and suggesting that post-hoc model selection would be ideal. D.2. Architectural Improvements: Query-Key Normalization The RT architecture supports multi-modal input representations for text, numeric, and boolean cell tokens, with type-specific encoders. During synthetic pretraining with such multi-modal(type) input tokens, we observed that zero-shot gener...

2020

[17] [2025]

URL https://arxiv.org/abs/ 2405.13396. Dove, E. S. and Phillips, M. Privacy law, data sharing policies, and medical data: a comparative perspective. In Medical data privacy handbook, pp. 639–678. Springer,

Pith/arXiv arXiv