Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Large Language Models are Good Relational Learners

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Rel-LLM claims that a frozen LLM conditioned on GNN-derived soft prompts outperforms relational deep learning baselines on RelBench.

desk verdict Solid empirical architecture with real average gains on RelBench, but the title claim is under-supported because the LLM is never ablated, and "consistently outperforms" is false on the paper's own tables. read the letter →

arxiv 2506.05725 v1 pith:H3BSZPVH submitted 2025-06-06 cs.CL cs.AIcs.LG

Fang Wu , Vijay Prakash Dwivedi , Jure Leskovec This is my paper
classification cs.CLcs.AIcs.LG
keywords relationaldeeplearninglargelanguagemodelsgraphneuralnetworksretrieval-augmentedgenerationprompttuningRelBenchdatabasessoftprompts
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that large language models can be effective relational learners when the structure of a relational database is handed to them as compact graph-derived soft prompts instead of as flattened text. It introduces Rel-LLM, which uses a temporal-aware GNN encoder to summarize the local subgraph around an entity, projects those embeddings into the LLM's embedding space, and feeds them to a frozen LLM as a structured prompt. On the RelBench benchmark, the authors report the highest average test AUROC of 77.82 among compared methods, surpassing the relational deep learning baseline (75.83) and the in-context-learning-plus-MLP baseline (76.83), and the lowest average regression MAE of 12.306. They also report that the zero-shot version, without fine-tuning labels, reaches an average AUROC of 63.42, comparable to LightGBM's 63.66. If true, this gives a scalable way to bring LLM reasoning to relational databases without serializing entire tables into text.

What carries the argument

The load-bearing object is the graph prompt $H^*$, a nested structure of projected GNN node embeddings built by denormalizing links from the seed entity up to $n_{\text{nest}}$ neighbors and depth $\zeta$. It is assembled from the subgraph encoder output and concatenated with the text embedder output as $[H^*; h_{\text{text}}]$, then fed through the frozen LLM's self-attention layers. This mechanism replaces text serialization with trainable soft prompts, which keeps context length small and lets the LLM attend to relational structure directly; because the LLM is frozen, only the graph encoder, projection MLP, and mask token receive gradients.

What would settle it

Train the same graph encoder and projection with a linear or MLP head in place of the frozen LLM, using identical subgraph sampling, pretraining, and hyperparameters; if the replacement matches or exceeds Rel-LLM's average AUROC and MAE on RelBench, the claim that the LLM contributes relational reasoning is falsified.

Watch

Extended reading notes

Core claim

Rel-LLM's central claim is that a frozen LLM can reason over relational structures if those structures are pre-digested by a graph encoder and delivered as soft prompts. For each target entity, the model samples a temporal subgraph that excludes future events, encodes it with a heterogeneous GraphSAGE-style message-passing network, and projects the node and pooled embeddings through an MLP into LLM embedding space. These projections are arranged by a denormalization process into a nested JSON-like graph prompt $H^* = \{\hat{h}^* : \{\{\hat{h}^*_i : \{\hat{h}^*_{i,j} : \{\ldots\}\}\}\}\}$, which is concatenated with the text embedding of the task and question and passed through a frozen LLM; gradients update only the graph encoder, projection, and mask token. The model is pretrained by masked attribute prediction, in which whole entities are masked and the LLM must regenerate their attribute text, aligning graph and text spaces. Empirically, the paper reports that this pipeline achieves the best average AUROC (77.82) and lowest average MAE (12.306) on RelBench test sets, with zero-shot AUROC of 63.42 before any fine-tuning.

Load-bearing premise

The load-bearing premise is that a frozen LLM genuinely reasons over projected GNN node embeddings when they are presented as soft prompts, so that the reported gains come from LLM relational reasoning rather than only from the GNN encoder and projection layers.

Editorial extensions

If this is right

  • Rel-LLM's average gains over RDL and ICL+MLP on RelBench imply that graph-derived soft prompts are a viable alternative to text serialization for relational prediction tasks.
  • The zero-shot AUROC of 63.42, close to LightGBM's 63.66, implies the pretrained graph-text alignment can support predictions without task labels in low-data regimes.
  • Because the LLM is frozen and only the encoder and projection are trained, Rel-LLM keeps fine-tuning cost low while remaining compatible with larger frozen LLMs, which the paper shows give modest further gains at 3B parameters.
  • Entity-level masked attribute prediction is reported to outperform cell-level masking, suggesting that forcing the model to reconstruct whole entities builds stronger relational representations.
  • The many-shot in-context experiments reported in the appendix show that adding many demonstration embeddings hurts Rel-Zero, so the model's relational reasoning is best used with few or no in-context examples.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the GNN already summarizes all relational structure before the LLM sees it, the paper's headline claim would be tested more sharply by an ablation that replaces the frozen LLM with a simple decoder; if the decoder matches Rel-LLM, the relational ability lives in the GNN, not the LLM.
  • The same graph-prompt recipe should transfer to knowledge graphs, biomedical records, or financial networks, where heterogeneous graphs and text descriptions coexist, but the paper does not test those settings.
  • If pretrained textual knowledge matters, Rel-LLM should show larger gains on tasks whose entities are well represented in text, such as the Formula 1 tasks, than on tasks with little textual footprint; this is testable by correlating per-task gains with textual coverage.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Rel-LLM, a framework for relational deep learning that combines a heterogeneous GNN encoder with a frozen LLM. The GNN produces temporal-aware subgraph embeddings, which are projected and presented to the LLM as soft prompts; the LLM generates predictions conditioned on these prompts and a task description. The model is pretrained with masked attribute prediction on RelBench databases and then fine-tuned on RelBench classification and regression tasks. The authors report the highest average test AUROC (77.82) and lowest average MAE (12.306) among compared baselines, and claim that Rel-LLM consistently outperforms or matches all baselines. They also report a zero-shot variant (Rel-Zero).

Significance. If the main claim is supported, Rel-LLM would be a useful practical architecture for relational databases, avoiding the context-length and redundancy problems of text serialization while retaining a frozen LLM's text interface. The paper ships code, uses temporal splits to avoid leakage, and includes a detailed failure analysis in the appendix. At the same time, the paper's own tables show that the advantage is average rather than per-task, and the architecture lacks the control needed to attribute the gain to the LLM. The work is therefore interesting, but its current form overstates both the consistency and the mechanism of the claimed improvement.

major comments (4)
  1. [§3.3, Eq. (6)] No ablation isolates the contribution of the frozen pretrained LLM. In Eq. (6), the LLM conditions only on [H*; h_text], where H* from Eq. (4) is a sequence of projected GNN embeddings; entity identifiers and attribute text never appear as tokens. The paper reports no control replacing the frozen LLM with a randomly initialized transformer or with a non-pretrained MLP decoder of matched capacity, so the average gains (test AUROC 77.82 vs. 75.83 for RDL; MAE 12.306 vs. 12.631 for RDL) could in principle come entirely from the trainable GNN encoder, projection layer, temporal sampling, and masked-attribute pretraining. Please add such an ablation, or temper the title claim 'Large Language Models are Good Relational Learners' accordingly.
  2. [§4.2 and §4.3, Tables 1 and 2] The sentence 'Rel-LLM consistently outperforms or matches all baselines across different datasets and tasks' is contradicted by the paper's own tables. In Table 1, ICL+MLP exceeds Rel-LLM on rel-avito user-clicks (71.31 vs. 66.74), rel-event user-ignore (84.02 vs. 83.74), rel-f1 driver-dnf (78.41 vs. 77.15), and rel-f1 driver-top3 (87.36 vs. 82.22) on the test split. In Table 2, ICL+MLP has a lower MAE than Rel-LLM on rel-f1 driver-position (3.539 vs. 3.967). The averaged results support a claim of 'on average, Rel-LLM achieves the best mean performance,' but not per-task consistency. Please revise the wording and report a per-task win/loss summary.
  3. [§4.2] The explanation that Rel-LLM excels on REL-F1 because 'Formula 1 drivers and their historical performance are well-documented in large-scale text corpora used for LLM pretraining' is not compatible with the input representation in §3.2–§3.3. The LLM receives only projected GNN embeddings H* and the text embedder output for the task/query; it does not receive driver names or attribute values as text tokens, so there is no textual surface on which pretrained F1 knowledge could be retrieved. Either provide direct evidence (e.g., an experiment exposing entity identifiers as text) or remove this attribution.
  4. [§3.4 and §4.4] Rel-Zero is described as zero-shot performance, but the pretraining in §3.4 is performed on the same RelBench databases used for downstream evaluation. While the temporal split avoids label leakage, the pretraining corpus still contains the entities and relations of the evaluation databases, so the Rel-Zero numbers measure within-benchmark adaptation, not generalization to unseen relational databases. Rename this setting (e.g., 'self-supervised pretraining, no downstream labels') or add an evaluation on databases held out from pretraining to support the zero-shot vocabulary.
minor comments (5)
  1. [Abstract and §1] The abstract and introduction claim 'outperforms existing methods' without acknowledging the per-task losses listed in Table 1; qualify this claim as 'on average.'
  2. [Tables 1 and 2] Tables 1 and 2 report no standard deviations for LightGBM, RDL, ICL, and ICL+MLP, while Appendix Tables 6 and 7 give standard deviations only for Rel-LLM; adding baseline variance would make the comparisons statistically interpretable.
  3. [§3.2, Eq. (4)] The section calls the prompt a 'JSON object' and a 'serialized' format, but Eq. (4) defines H* as a nested set of embedding vectors, not serialized text; this terminology is misleading and should be clarified.
  4. [Appendix B.3] Appendix B.3 lists 'Text Embedder – [Glove, MPNet]' and 'Fine-tuning strategy – [Freeze, LoRA]' in the search space, but §3.3 defines the text embedder as the frozen LLM's first layer and the main text reports only the frozen strategy; please reconcile the two descriptions or report results for the searched variants.
  5. [Various] Minor typos and wording issues: 'Comparision' in the Figure 2 caption, 'REF-F1' in Appendix B.5, 'hyperparameyter' in Appendix B.3, and 'suboptimal' in the Table 2 caption should be 'second best.'

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the core evaluation is trained on labeled temporal splits and tested on held-out splits; self-cited benchmark use and the missing LLM ablation are non-circular concerns.

full rationale

The paper's derivation chain is not circular. The predictive model is defined in Eq. (6) as a frozen-LLM distribution conditioned on [H*; h_text], with H* produced by the GNN/projection equations (2)-(4), and the pretraining objective Eq. (8) is a self-supervised attribute reconstruction loss. The main empirical claims (test AUROC 77.82 vs RDL 75.83; MAE 12.306 vs 12.631) come from training on labeled temporal splits of RelBench and evaluating on held-out future splits, so the reported numbers are not equal to any fitted quantity by construction. The paper does lean on self-authored infrastructure (RelBench, RDL, GraphSAGE) and calls Rel-Zero zero-shot after masked-attribute pretraining on the same relational databases; the latter is a data-contamination/terminology concern, and the lack of an ablation replacing the frozen LLM weakens the title attribution, but neither step is an equation-level reduction of a prediction to its input. No self-citation imposes a uniqueness or ansatz choice, and no known result is merely renamed as a derivation. Under the hard rule requiring an exhibited Eq. X = Eq. Y reduction, there is no significant circularity.

Assumptions & free parameters 8 free parameters · 4 assumptions · 0 invented entities

The architecture introduces no new physical or geometric entities. The central extras are tuned hyperparameters and the ad hoc pretraining-on-evaluation-data assumption. The most important free parameters are the per-task answer generation strategy and the focal loss weights, both selected on validation and not controlled across baselines.

free parameters (8)
  • focal loss class weight α_t = per task, searched in [0.1, 0.2, 0.4, 0.8, 0.9]
    Chosen by random search on the validation set; directly changes the classification loss and therefore the reported AUROC values (Section B.5, Table 5).
  • focal loss focusing parameter γ = 2.0
    Fixed constant, but it interacts with the severity of class imbalance and is not used by the baseline models (Section B.5).
  • number of nested entities n_nest = searched in [0, 128]
    Controls how much relational context enters the graph prompt; selected per task on validation (Section 3.2, Table 5).
  • denormalization recursion depth ζ = searched in [0, 1]
    Limits multi-hop relational context and is selected per task (Section 3.2, Table 5).
  • answer generation strategy = per task: plain text, token distribution, or MLP transformation
    The paper states different tasks benefit from different strategies; this is a per-task model selection degree of freedom not available to baselines (Section 3.3, Appendix B.1).
  • pretraining mask ratio |V_mask|/|V| = 0.5
    Chosen design point for masked attribute prediction (Table 5).
  • GNN layers L = 2
    Fixed after hyperparameter search (Table 5).
  • GNN output dimension d_g = 128
    Fixed graph encoder width (Table 5).
assumptions (4)
  • standard math Message passing on heterogeneous graphs and transformer self-attention correctly model relational structure.
    The paper relies on standard GraphSAGE-style aggregation (Eq. 9-10) and pretrained LLM attention (Eq. 6) without proving these models capture relational semantics; this is accepted background math.
  • domain assumption Timestamps τ(v) are reliable and exhaustive, so temporal neighbor sampling with τ(w) ≤ t* yields causally valid subgraphs.
    Section 3.1 and Eq. 11: if timestamps are incomplete or noisy, the reported gains could come from temporal leakage.
  • ad hoc to paper Projected GNN embeddings H* can be prepended to textual token embeddings and understood by a frozen LLM as prompts.
    Section 3.2-3.3, Eq. 4 and Eq. 6: the architecture assumes the frozen LLM can meaningfully attend to the projected graph vectors; the paper provides no independent check that the LLM contributes beyond the GNN features.
  • ad hoc to paper Masked attribute prediction on the same RelBench databases transfers to downstream tasks and is a legitimate zero-shot pretraining basis.
    Section 3.4 and Section 4.4: Rel-Zero is called zero-shot despite pretraining on the evaluation data, which is a strong transfer assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Large Language Models are Good Relational Learners." pith.science (2026). https://pith.science/paper/H3BSZPVH

@misc{pith2026250605725,
  author       = {Pith},
  title        = {Pith review of: Large Language Models are Good Relational Learners},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H3BSZPVH}},
  note         = {Machine review of arXiv:2506.05725}
}
read the original abstract

Large language models (LLMs) have demonstrated remarkable capabilities across various domains, yet their application to relational deep learning (RDL) remains underexplored. Existing approaches adapt LLMs by traversing relational links between entities in a database and converting the structured data into flat text documents. Still, this text-based serialization disregards critical relational structures, introduces redundancy, and often exceeds standard LLM context lengths. We introduce Rel-LLM, a novel architecture that utilizes a graph neural network (GNN)- based encoder to generate structured relational prompts for LLMs within a retrieval-augmented generation (RAG) framework. Unlike traditional text-based serialization approaches, our method preserves the inherent relational structure of databases while enabling LLMs to effectively process and reason over complex entity relationships. Specifically, the GNN encoder extracts a local subgraph around an entity to build feature representations that contain relevant entity relationships and temporal dependencies. These representations are transformed into structured prompts using a denormalization process, effectively allowing the LLM to reason over relational structures. Through extensive experiments, we demonstrate that Rel-LLM outperforms existing methods on key RDL tasks, offering a scalable and efficient approach to integrating LLMs with structured data sources. Code is available at https://github.com/smiles724/Rel-LLM.

Figures

Figures reproduced from arXiv: 2506.05725 by the authors.

Figure 1
Figure 1. Overview of the Rel-LLM framework. Input from a relational database is fed to an LLM through a graph [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Comparision of model prediction variation across different classification tasks. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. illustrates the comparison, revealing that the 3B model generally outperforms the 1B model across most tasks. However, the performance gap between the two models is relatively modest, sug￾gesting that while increasing model size provides benefits, the returns may diminish at this scale. Notably, the 3B model demonstrates a more pro￾amazon-item amazon-user avito-clicks avito-visits event-ignore event-repeat f1-dnf f1… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Visualization of different algorithms to compose the task-specific prompts. [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: Illustration of many-shot ICL with graph neural prompt. [PITH_FULL_IMAGE:figures/full_fig_p019_5.png]
Figure 6
Figure 6. Figure 6: Difference between entity-level and cell-level masking strategies for pretraining. [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ASARL: Autonomous Social-Aware Relevance Learning for QQ Search

    cs.IR 2026-07 conditional novelty 4.0 of 10

    An agent-loop data-curation pipeline with social-aware chain-of-thought, preference, and distillation training improves QQ group/channel search relevance in offline and online evaluation.

Reference graph

Works this paper leans on

12 extracted references · 3 canonical work pages · cited by 1 Pith paper

  1. [4]

    InInternational Conference on Artificial Intelligence and Statistics, pages 5549–5581

    Tabllm: Few-shot classification of tabular data with large language models. InInternational Conference on Artificial Intelligence and Statistics, pages 5549–5581. PMLR. Yupeng Hou, Junjie Zhang, Zihan Lin, Hongyu Lu, Ruobing Xie, Julian McAuley, and Wayne Xin Zhao

    work page

  2. [5]

    InEuropean Conference on Information Retrieval, pages 364–381

    Large language models are zero-shot rankers for recommender systems. InEuropean Conference on Information Retrieval, pages 364–381. Springer. Weihua Hu, Yiwen Yuan, Zecheng Zhang, Akihiro Nitta, Kaidi Cao, Vid Kocijan, Jure Leskovec, and Matthias Fey. 2024. Pytorch frame: A modular framework for multi-modal tabular learning.arXiv preprint arXiv:2404.00776...

    arXiv 2024

  3. [11]

    arXiv preprint arXiv:2402.02244

    Beyond the limits: A survey of techniques to extend the context length in large language models. arXiv preprint arXiv:2402.02244. Zhiruo Wang, Haoyu Dong, Ran Jia, Jia Li, Zhiyi Fu, Shi Han, and Dongmei Zhang. 2021. Tuta: Tree- based transformers for generally structured table pre- training. InProceedings of the 27th ACM SIGKDD Conference on Knowledge Dis...

    arXiv 2021

  4. [12]

    YES" and

    Retrieval-reasoning large language model- based synthetic clinical trial generation.arXiv preprint arXiv:2410.12476. Yazheng Yang, Yuqi Wang, Guang Liu, Ledell Wu, and Qi Liu. 2023. Unitabe: Pretraining a unified tabu- lar encoder for heterogeneous tabular data.arXiv preprint arXiv:2307.09249. Yunhu Ye, Binyuan Hui, Min Yang, Binhua Li, Fei Huang, and Yon...

    work page arXiv 2023

  5. [607]

    Ananya Singha, José Cambronero, Sumit Gulwani, Vu Le, and Chris Parnin

    Springer. Ananya Singha, José Cambronero, Sumit Gulwani, Vu Le, and Chris Parnin. 2023. Tabular represen- tation, noisy operators, and impacts on table struc- ture understanding tasks in llms.arXiv preprint arXiv:2310.10358. Jan Struyf and Hendrik Blockeel. 2010. Relational learning. Yuan Sui, Mengyu Zhou, Mingjie Zhou, Shi Han, and Dongmei Zhang. 2024. T...

    arXiv 2023

  6. [2015]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hein- rich Küttler, Mike Lewis, Wen-tau Yih, Tim Rock- täschel, and 1 others

    Deep learning.nature, 521(7553):436–444. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hein- rich Küttler, Mike Lewis, Wen-tau Yih, Tim Rock- täschel, and 1 others. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks.Ad- vances in Neural Information Processing Systems, 33:9459–9474. Weich...

    arXiv 2020

  7. [2017]

    Diederik P Kingma

    Lightgbm: A highly efficient gradient boost- ing decision tree.Advances in neural information processing systems, 30. Diederik P Kingma. 2014. Adam: A method for stochas- tic optimization.arXiv preprint arXiv:1412.6980. Thomas N Kipf and Max Welling. 2016. Semi- supervised classification with graph convolutional networks.arXiv preprint arXiv:1609.02907. Y...

    arXiv 2014

  8. [2018]

    InThe semantic web: 15th inter- national conference, ESWC 2018, Heraklion, Crete, Greece, June 3–7, 2018, proceedings 15, pages 593–

    Modeling relational data with graph convolu- tional networks. InThe semantic web: 15th inter- national conference, ESWC 2018, Heraklion, Crete, Greece, June 3–7, 2018, proceedings 15, pages 593–

    work page 2018

Show all 12 references
  1. [2021]

    Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettle- moyer

    A comprehensive survey on graph anomaly detection with deep learning.IEEE Transactions on Knowledge and Data Engineering, 35(12):12012– 12038. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettle- moyer. 2022. Rethinking the role ...

    2022 arXiv

  2. [2023]

    arXiv preprint arXiv:2310.07875

    Tablib: A dataset of 627m tables with context. arXiv preprint arXiv:2310.07875. Xi Fang, Weijie Xu, Fiona Anting Tan, Jiani Zhang, Ziqing Hu, Yanjun Jane Qi, Scott Nickleach, Diego Socolinsky, Srinivasan Sengamedu, Christos Falout- sos, and 1 others. 2024. Large language model...

    arXiv

  3. [2024]

    Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gre- gory Valiant

    Large scale transfer learning for tabu- lar data via language modeling.arXiv preprint arXiv:2406.12031. Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gre- gory Valiant. 2022. What can transformers learn in-context? a case study of simple function classes. Advances in Neura...

    2022 arXiv

  4. [2025]

    Mubashara Akhtar, Abhilash Shankarampeta, Vivek Gupta, Arpit Patil, Oana Cocarascu, and Elena Sim- perl

    Many-shot in-context learning.Advances in Neural Information Processing Systems, 37:76930– 76966. Mubashara Akhtar, Abhilash Shankarampeta, Vivek Gupta, Arpit Patil, Oana Cocarascu, and Elena Sim- perl. 2023. Exploring the numerical reasoning capa- bilities of language models:...

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.