Pith. sign in

REVIEW 2 major objections 5 minor 26 references

Under one fair protocol, the Relational Transformer beats graph models and even TabPFN 2.5 on multi-table and single-table RDB tasks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 00:51 UTC pith:6WOTWITU

load-bearing objection Solid first controlled bake-off of RT vs Griffin vs DBFormer under matched hops and budgets; RT wins, multi-hop gains diminish, and the forced protocol is the main caveat rather than a fatal flaw. the 2 major comments →

arxiv 2607.03659 v1 pith:6WOTWITU submitted 2026-07-04 cs.DB cs.AI

A Fair Benchmarking of Deep Relational Database Learning Models

Kazi F. Akhter , Bharath Ajendla , Manar D. Samad This is my paper
classification cs.DB cs.AI
keywords relational databasesdeep learningbenchmarkingrelational transformergraph neural networkshop-based neighborhoodtabular foundation models
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Relational databases store most enterprise data as linked tables, but recent deep models for them have been compared under mismatched training budgets, encoders, and neighborhood schemes, so it has been hard to know which approach actually wins. This paper rebuilds three deep RDB models (DBFormer, Griffin, RT) and two strong tabular baselines (LightGBM, TabPFN 2.5) under a single controlled protocol: same encoder, same embedding size, same learning rate and batch size, same epoch budget, and an explicit hop-based expansion that lets every method use 0-, 1-, or 2-hop neighborhoods of a target row. Across five RelBench databases and ten tasks (classification and regression), the Relational Transformer posts the best average ranks on both task types. Even when restricted to a single table (hop = 0), the RDB-designed models already beat the leading tabular foundation model TabPFN 2.5. Pulling in neighboring tables usually helps, but the jump from hop 1 to hop 2 is small relative to the large rise in training time. The result is a concrete ranking and a practical warning: multi-hop context is useful, but only up to a point.

Core claim

When every deep RDB model is forced onto the same data encoding, hop-controlled neighborhood construction, and training budget, the Relational Transformer is the strongest overall performer on both classification and regression tasks across five RelBench databases; the same models also beat TabPFN 2.5 even at hop = 0, while gains from hop 2 over hop 1 are modest and costly.

What carries the argument

Hop-based relational neighborhood expansion: for a target row, hop 0 uses only that row, hop 1 adds rows linked by a single primary-foreign-key edge, and hop 2 adds the next ring of tables; every model is evaluated at the same hop depth under identical MiniLM-v2 encoding and training hyperparameters.

Load-bearing premise

That locking every architecture to the same encoder, embedding size, learning rate, batch size and epoch budget produces a fair ranking rather than systematically hurting models whose original designs used different encoders or stopping rules.

What would settle it

Re-run the same five databases and ten tasks with each model's original encoder, embedding size and early-stopping regime (or a larger matched budget) and check whether RT still holds the best average ranks on classification and regression.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper presents a controlled benchmarking study of deep learning methods for relational databases (RDBs). It evaluates three deep RDB architectures (DBFormer, Griffin, Relational Transformer/RT) plus two strong tabular baselines (LightGBM, TabPFN 2.5) on five RelBench databases, each with one classification and one regression task. The authors introduce a hop-based neighborhood expansion (Eqs. 1–4) that standardizes how multi-table context is supplied, refactor all models onto a common encoder (MiniLM-v2), embedding size (256), learning rate, batch size and 100-epoch budget (Table 2), and report AUROC/RMSE tables (Tables 3–4), win–loss matrices (Fig. 2) and wall-clock times (Table 5). Under this protocol RT obtains the best average ranks on both task types; even at hop = 0 the RDB-native models often beat TabPFN 2.5; gains from hop 1 to hop 2 are modest relative to the increase in training cost.

Significance. If the ranking holds under the stated protocol, the work supplies the first systematic, publicly reproducible comparison of the emerging deep-RDB literature and gives practitioners concrete guidance: prefer the relational transformer, treat hop-2 expansion cautiously, and do not assume that tabular foundation models automatically dominate large multi-table enterprise data. Strengths that raise the contribution above a routine bake-off include the public code release, the architecture-agnostic hop formulation, explicit temporal-leakage controls, and the side-by-side reporting of both predictive metrics and wall-clock cost. These elements make the empirical claims falsifiable and reusable by subsequent work.

major comments (2)
  1. [Table 2, §3.1.2] Table 2 and §3.1.2: the central ranking claim rests on a single forced hyper-parameter regime (MiniLM-v2, dim 256, lr 1e-4, batch 256, 100 epochs, best-on-val selection). The original papers used different encoders, embedding sizes and early-stopping rules; without at least a limited sensitivity sweep (or an ablation that restores each model’s published settings) it remains possible that the protocol systematically favors RT’s sparse cell-attention design over Griffin’s message-passing or DBFormer’s hybrid layers. This is the load-bearing fairness assumption of the study.
  2. [Tables 3–4, Fig. 2] Tables 3–4 and Fig. 2 report only single-seed point estimates; no standard deviations, multi-seed averages or statistical significance tests accompany the average-rank and win–loss claims. With only five databases the ranking of RT as “strongest overall” is therefore sensitive to sampling noise. Adding even three random seeds (or a simple paired Wilcoxon test across the 15 task–hop cells) would make the superiority claim robust rather than suggestive.
minor comments (5)
  1. [Table 1] Table 1 caption and §4.1: “Sample Size” column is never defined; clarify whether it equals task rows or something else.
  2. [§3.1.2] §3.1.2: numerical-feature encoding is described for Griffin (quantile + MLP) but left vague for RT and DBFormer; a short sentence stating that both use a linear projection would improve reproducibility.
  3. [Figure 1] Figure 1: the final concatenated row still contains the class label column; a note that the label is held out during feature construction would avoid confusion.
  4. [References] References [4] and [22] appear to be the same arXiv preprint; consolidate to a single entry.
  5. [Table 5] Table 5 reports times only for the small rel-f1 task; a one-sentence remark on how the relative ordering scales to the large amazon/stack databases would strengthen the cost discussion.

Circularity Check

0 steps flagged

No circularity: pure empirical bake-off whose rankings are measured on held-out RelBench splits under an explicitly standardized protocol.

full rationale

The paper's central claims (RT strongest overall; multi-hop yields diminishing returns; RDB methods beat TabPFN 2.5 even at hop=0) are obtained by training and evaluating five models on the public RelBench train/val/test splits under a single controlled protocol (MiniLM-v2 encoder, dim 256, lr 1e-4, batch 256, 100 epochs, hop-controlled neighborhoods defined by Eqs. 1-4). Performance numbers (Tables 3-4, Fig. 2 win matrices) are therefore independent measurements, not algebraic rearrangements of fitted parameters or self-defined quantities. Citations to RelBench, Griffin, RT and DBFormer supply the models and data; none of those citations is used as a uniqueness theorem or load-bearing premise that forces the ranking. Self-citations of the authors' earlier tabular-learning papers appear only in the related-work section and do not underwrite any experimental result. Consequently the derivation chain contains no self-definitional step, no fitted-input-called-prediction, and no self-citation that collapses the claim.

Axiom & Free-Parameter Ledger

5 free parameters · 3 axioms · 0 invented entities

As an empirical benchmarking study the paper rests on standard ML evaluation assumptions plus a small set of protocol choices that define “fairness.” No new physical or mathematical entities are postulated; free parameters are the usual training hyper-parameters fixed by the authors for all models.

free parameters (5)
  • embedding_size = 256
    Fixed at 256 for all deep models (Table 2); original papers used 64/256/512. Choice affects capacity and ranking.
  • learning_rate = 1e-4
    Fixed at 1e-4 for all deep models; originals used 1e-5 / 3e-4 / 1e-4.
  • batch_size = 256
    Fixed at 256; originals used 16 or 256.
  • train_epochs = 100
    Fixed at 100 with best-on-validation selection; originals used 1000, 50 or none.
  • text_encoder = MiniLM-v2
    All models forced to MiniLM-v2 (384-dim) instead of their original GloVe / Nomic / MiniLM choices.
axioms (3)
  • domain assumption RelBench train/validation/test splits and temporal leakage controls are correct and representative of real RDB prediction tasks.
    All reported metrics inherit the RelBench split protocol (Section 4.1).
  • ad hoc to paper Hop-based neighborhood expansion (Eqs. 1–4) is a faithful and architecture-agnostic way to control relational context.
    Introduced by the authors to unify sampling strategies that originally differed across models (Section 3.1.1).
  • domain assumption Aggregating multi-hop tables into flat features for LightGBM and TabPFN is a fair baseline comparison.
    Standard practice in RelBench; adopted here for hop-controlled fairness (Section 3.1.3).

pith-pipeline@v1.1.0-grok45 · 16802 in / 2484 out tokens · 22398 ms · 2026-07-12T00:51:13.617417+00:00 · methodology

0 comments
read the original abstract

Relational databases (RDBs) are the primary data infrastructure in many enterprises, yet recent deep learning methods designed for RDBs have been evaluated under inconsistent experimental protocols, making fair comparison difficult. We present one of the first systematic benchmarking studies of recently released deep learning methods for RDBs, evaluating them across five relational databases, with one classification and one regression task for each. We refactor all deep RDB models to allow the full range of experimental procedures to be applied consistently across all methods. Our findings indicate that the relational transformer (RT) approach delivers the strongest overall performance on both classification and regression tasks compared to the state-of-the-art graph-based modeling and learning of RDBs. Even for single-table learning tasks, deep learning methods designed for RDBs outperform the leading tabular foundation model, TabPFN 2.5. Extending learning from a single table (hop = 0) to multiple tables (hop = 1, 2) by connecting neighboring tables in relational databases enhances performance, but the additional benefit from higher hops diminishes as computational overhead grows. Deep RDB learning methods have the potential to challenge state-of-the-art tabular foundation models, especially on large-scale enterprise data. The source code for this benchmarking study is publicly available.

Figures

Figures reproduced from arXiv: 2607.03659 by Bharath Ajendla, Kazi F. Akhter, Manar D. Samad.

Figure 1
Figure 1. Figure 1: Multi-table relationships in a relational database to illustrate hop-based relational neighborhood expansion [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Pairwise model comparison using the win–loss matrix. (a) For classification tasks using AUROC scores and [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

26 extracted references · 1 canonical work pages

  1. [1]

    Robinson, R

    J. Robinson, R. Ranjan, W. Hu, K. Huang, J. Han, A. Dobles, M. Fey, J. E. Lenssen, Y . Yuan, Z. Zhang, X. He, J. Leskovec, Relbench: A benchmark for deep learning on relational databases, in: 38th Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URLhttps://relbench.stanford.eduhttps://openreview.net/forum?id=WEFxOm3...

    2024

  2. [2]

    M. Fey, W. Hu, K. Huang, J. E. Lenssen, R. Ranjan, J. Robinson, R. Ying, J. You, J. Leskovec, Position: Relational deep learning – graph representation learning on relational databases, in: International Conference on Machine Learning (ICML), 2024

    2024

  3. [3]

    Fey, et al., Relational deep learning: Graph representation learning on relational databases, in: ICML, 2024

    M. Fey, et al., Relational deep learning: Graph representation learning on relational databases, in: ICML, 2024

    2024

  4. [5]

    Grinsztajn, E

    L. Grinsztajn, E. Oyallon, G. Varoquaux, Why do tree-based models still outperform deep learning on tabular data?, in: NeurIPS, 2022

    2022

  5. [6]

    Borisov, et al., Deep neural networks and tabular data: A survey, IEEE Transactions on Neural Networks and Learning Systems (2022)

    V . Borisov, et al., Deep neural networks and tabular data: A survey, IEEE Transactions on Neural Networks and Learning Systems (2022)

    2022

  6. [7]

    Shwartz-Ziv, A

    R. Shwartz-Ziv, A. Armon, Tabular data: Deep learning is not all you need, Information Fusion 81 (2022) 84–90

    2022

  7. [8]

    Gorishniy, I

    Y . Gorishniy, I. Rubachev, V . Khrulkov, A. Babenko, Revisiting deep learning models for tabular data, in: Ad- vances in Neural Information Processing Systems, V ol. 34, 2021, pp. 18932–18943

    2021

  8. [9]

    Borisov, T

    V . Borisov, T. Leemann, K. Sessler, J. Haug, M. Pawelczyk, G. Kasneci, Deep Neural Networks and Tabular Data: A Survey, IEEE Transactions on Neural Networks and Learning Systems (2022) 1–21doi:10.1109/ TNNLS.2022.3229161. URLhttps://ieeexplore.ieee.org/document/9998482/

    arXiv 2022

  9. [10]

    Grinsztajn, E

    L. Grinsztajn, E. Oyallon, G. Varoquaux, Why do tree-based models still outperform deep learning on typical tabular data?, in: Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URLhttps://openreview.net/forum?id=Fp7__phQszn

    2022

  10. [11]

    S. B. Rabbani, I. V . Medri, M. D. Samad, Attention versus contrastive learning of tabular data: a data-centric benchmarking, International Journal of Data Science and Analytics 20 (4) (2025) 3069–3091. REFERENCES 10

    2025

  11. [12]

    Holzm ¨uller, L

    D. Holzm ¨uller, L. Grinsztajn, I. Steinwart, Better by default: Strong pre-tuned mlps and boosted trees on tabular data, Advances in Neural Information Processing Systems 37 (2024) 26577–26658

    2024

  12. [13]

    Hollmann, et al., Accurate predictions on small data with a tabular foundation model, Nature (2025)

    N. Hollmann, et al., Accurate predictions on small data with a tabular foundation model, Nature (2025)

    2025

  13. [14]

    Ye, S.-Y

    H.-J. Ye, S.-Y . Liu, W.-L. Chao, A closer look at tabpfn v2: Understanding its strengths and extending its capabilities, in: The Thirty-ninth Annual Conference on Neural Information Processing Systems

  14. [15]

    J. You, G. Liu, Beyond graphs: Learning with relational DBs (2024). URLhttps://openreview.net/forum?id=ZQIgnzxMKJ

    2024

  15. [16]

    Cvitkovic, Supervised learning on relational databases with graph neural networks (2020).arXiv:2002

    M. Cvitkovic, Supervised learning on relational databases with graph neural networks (2020).arXiv:2002. 02046. URLhttps://arxiv.org/abs/2002.02046

    Pith/arXiv arXiv 2020

  16. [17]

    T. Chen, C. Kanatsoulis, J. Leskovec, Relgnn: Composite message passing for relational deep learning, in: ICML, 2025

    2025

  17. [18]

    V . P. Dwivedi, et al., Relational graph transformer, arXiv preprint arXiv:2505.10960 (2025)

    arXiv 2025

  18. [19]

    Y . Wang, X. Wang, Q. Gan, M. Wang, Q. Yang, D. Wipf, M. Zhang, Griffin: Towards a graph-centric relational database foundation model, in: Proceedings of the 42nd International Conference on Machine Learning (ICML), 2025

    2025

  19. [20]

    Pele ˇska, G

    J. Pele ˇska, G. ˇS´ır, Tabular transformers meet relational databases, ACM Trans. Intell. Syst. Technol. 16 (5) (Sep. 2025).doi:10.1145/3749991. URLhttps://doi.org/10.1145/3749991

    work page doi:10.1145/3749991 2025

  20. [21]

    S. B. Rabbani, I. V . Medri, M. D. Samad, Attention versus contrastive learning of tabular data: a data-centric benchmarking, International Journal of Data Science and Analytics 20 (4) (2025) 3069–3091

    2025

  21. [22]

    A. information requires manual verification, Relational transformer: Toward zero-shot foundation models for relational data, arXiv preprint arXiv:2510.06377Needs manual verification for final venue and full author list (2025)

    arXiv 2025

  22. [23]

    Pennington, R

    J. Pennington, R. Socher, C. D. Manning, Glove: Global vectors for word representation, in: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), 2014, pp. 1532–1543

    2014

  23. [24]

    Nussbaum, J

    Z. Nussbaum, J. X. Morris, B. Duderstadt, A. Mulyar, Nomic embed: Training a reproducible long context text embedder

  24. [25]

    W. Wang, H. Bao, S. Huang, L. Dong, F. Wei, Minilmv2: Multi-head self-attention relation distillation for com- pressing pretrained transformers, in: Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, 2021, pp. 2140–2151

    2021

  25. [26]

    Reimers, I

    N. Reimers, I. Gurevych, Sentence-bert: Sentence embeddings using siamese bert-networks, in: Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), 2019, pp. 3982–3992

    2019

  26. [27]

    Pedregosa, G

    F. Pedregosa, G. Varoquaux, A. Gramfort, V . Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V . Dubourg, et al., Scikit-learn: Machine learning in python, the Journal of machine Learning research 12 (2011) 2825–2830

    2011