{"id":"70a43e34-0339-4bb4-9fdb-4a1339523796","arxiv_id":"2506.05725","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"Rel-LLM combines a GNN encoder with a frozen LLM via soft prompts and masked attribute pretraining, reporting improved average performance on RelBench relational database tasks.","lead":"This paper introduces Rel-LLM, a system that turns relational database rows into graph embeddings and feeds those embeddings to a frozen large language model as soft prompts. On the RelBench benchmark it reports higher average accuracy and lower average error than standard relational deep learning baselines, though the advantage is not consistent on every task.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing control: Rel-LLM never ablates the frozen LLM, so the title claim that LLMs are the relational learners is unverified; per-task tables also contradict 'consistently outperforms.'","rationale":"The paper is a plausible engineering contribution: it couples a heterogeneous GraphSAGE encoder, temporal subgraph sampling, a projection MLP, and a frozen Llama-3.2-1B via soft prompts, with code released and evaluations on RelBench. The reader's conditional verdict is appropriate. My stress-test focus is the attribution step. For the title to hold, the frozen LLM must be doing nontrivial relational reasoning. The architecture feeds the LLM only projected GNN embeddings plus task text (Eqs. 4-6), so the LLM is effectively a fixed nonlinear decoder. Nothing in the paper compares against a same-capacity non-pretrained or non-LLM decoder, so the reported average gains could be due entirely to the trainable GNN/projection/pretraining pipeline. This is an internal missing control rather than a disagreement with external consensus. The paper's own tables also contradict the word 'consistently' (e.g., Table 1 rel-avito user-clicks and rel-f1 driver-top3; Table 2 rel-f1 driver-position), which lowers confidence in the strength of the empirical claim even though the averages favor Rel-LLM. I therefore agree with the reader's weakest assumption and recommend keeping the verdict conditional pending the LLM ablation and corrected per-task wording.","tokens_in":20911,"tokens_out":6867,"duration_ms":68371,"concrete_test":"Replace the frozen pretrained LLM in Eq. (6) with a randomly initialized transformer decoder of identical architecture and, separately, with a trainable MLP head of comparable parameter count, keeping the GNN encoder, temporal sampling, projection, and masked-attribute pretraining exactly as in the paper; compare average RelBench test AUROC/MAE to Rel-LLM. If either replacement matches the reported 77.82 AUROC / 12.306 MAE within one standard deviation, the LLM's pretrained weights are not the source of relational ability. For the Rel-F1 factual-knowledge claim in Section 4.2, additionally rerun with driver names replaced by anonymous IDs; a null result would show that explanation is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"For the title claim 'LLMs are Good Relational Learners' to hold, the frozen LLM must contribute nontrivial relational ability. In Eq. (6), the LLM conditions only on [H*; h_text], where H* from Eq. (4) is a sequence of projected GNN embeddings; the LLM never receives entity identifiers or attribute text as tokens. The paper reports no ablation that replaces the frozen pretrained LLM with a randomly initialized transformer or a non-pretrained MLP decoder of matched capacity, so the average gains (test AUROC 77.82 vs RDL 75.83; MAE 12.306 vs 12.631) could come entirely from the trainable GNN encoder, projection layer, temporal sampling, and masked-attribute pretraining. This is an internal missing control, not a dispute about consensus. The paper's own tables also weaken the empirical claim: Table 1 test rows show Rel-LLM below ICL+MLP on rel-avito user-clicks (66.74 vs 71.31), rel-event user-ignore (83.74 vs 84.02), rel-f1 driver-dnf (77.15 vs 78.41), and driver-top3 (82.22 vs 87.36); Table 2 shows higher MAE than ICL+MLP on rel-f1 driver-position (3.967 vs 3.539). Thus 'consistently outperforms or matches all baselines' is false in the paper's own data. The Section 4.2 explanation that Rel-F1 gains come from F1 knowledge in LLM pretraining is also unsupported, because the input representation does not expose driver names as text tokens.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Rel-LLM, a framework for relational deep learning that combines a heterogeneous GNN encoder with a frozen LLM. The GNN produces temporal-aware subgraph embeddings, which are projected and presented to the LLM as soft prompts; the LLM generates predictions conditioned on these prompts and a task description. The model is pretrained with masked attribute prediction on RelBench databases and then fine-tuned on RelBench classification and regression tasks. The authors report the highest average test AUROC (77.82) and lowest average MAE (12.306) among compared baselines, and claim that Rel-LLM consistently outperforms or matches all baselines. They also report a zero-shot variant (Rel-Zero).","tokens_in":21258,"tokens_out":5502,"duration_ms":45234,"significance":"If the main claim is supported, Rel-LLM would be a useful practical architecture for relational databases, avoiding the context-length and redundancy problems of text serialization while retaining a frozen LLM's text interface. The paper ships code, uses temporal splits to avoid leakage, and includes a detailed failure analysis in the appendix. At the same time, the paper's own tables show that the advantage is average rather than per-task, and the architecture lacks the control needed to attribute the gain to the LLM. The work is therefore interesting, but its current form overstates both the consistency and the mechanism of the claimed improvement.","major_comments":[{"comment":"No ablation isolates the contribution of the frozen pretrained LLM. In Eq. (6), the LLM conditions only on [H*; h_text], where H* from Eq. (4) is a sequence of projected GNN embeddings; entity identifiers and attribute text never appear as tokens. The paper reports no control replacing the frozen LLM with a randomly initialized transformer or with a non-pretrained MLP decoder of matched capacity, so the average gains (test AUROC 77.82 vs. 75.83 for RDL; MAE 12.306 vs. 12.631 for RDL) could in principle come entirely from the trainable GNN encoder, projection layer, temporal sampling, and masked-attribute pretraining. Please add such an ablation, or temper the title claim 'Large Language Models are Good Relational Learners' accordingly.","section":"§3.3, Eq. (6)"},{"comment":"The sentence 'Rel-LLM consistently outperforms or matches all baselines across different datasets and tasks' is contradicted by the paper's own tables. In Table 1, ICL+MLP exceeds Rel-LLM on rel-avito user-clicks (71.31 vs. 66.74), rel-event user-ignore (84.02 vs. 83.74), rel-f1 driver-dnf (78.41 vs. 77.15), and rel-f1 driver-top3 (87.36 vs. 82.22) on the test split. In Table 2, ICL+MLP has a lower MAE than Rel-LLM on rel-f1 driver-position (3.539 vs. 3.967). The averaged results support a claim of 'on average, Rel-LLM achieves the best mean performance,' but not per-task consistency. Please revise the wording and report a per-task win/loss summary.","section":"§4.2 and §4.3, Tables 1 and 2"},{"comment":"The explanation that Rel-LLM excels on REL-F1 because 'Formula 1 drivers and their historical performance are well-documented in large-scale text corpora used for LLM pretraining' is not compatible with the input representation in §3.2–§3.3. The LLM receives only projected GNN embeddings H* and the text embedder output for the task/query; it does not receive driver names or attribute values as text tokens, so there is no textual surface on which pretrained F1 knowledge could be retrieved. Either provide direct evidence (e.g., an experiment exposing entity identifiers as text) or remove this attribution.","section":"§4.2"},{"comment":"Rel-Zero is described as zero-shot performance, but the pretraining in §3.4 is performed on the same RelBench databases used for downstream evaluation. While the temporal split avoids label leakage, the pretraining corpus still contains the entities and relations of the evaluation databases, so the Rel-Zero numbers measure within-benchmark adaptation, not generalization to unseen relational databases. Rename this setting (e.g., 'self-supervised pretraining, no downstream labels') or add an evaluation on databases held out from pretraining to support the zero-shot vocabulary.","section":"§3.4 and §4.4"}],"minor_comments":[{"comment":"The abstract and introduction claim 'outperforms existing methods' without acknowledging the per-task losses listed in Table 1; qualify this claim as 'on average.'","section":"Abstract and §1"},{"comment":"Tables 1 and 2 report no standard deviations for LightGBM, RDL, ICL, and ICL+MLP, while Appendix Tables 6 and 7 give standard deviations only for Rel-LLM; adding baseline variance would make the comparisons statistically interpretable.","section":"Tables 1 and 2"},{"comment":"The section calls the prompt a 'JSON object' and a 'serialized' format, but Eq. (4) defines H* as a nested set of embedding vectors, not serialized text; this terminology is misleading and should be clarified.","section":"§3.2, Eq. (4)"},{"comment":"Appendix B.3 lists 'Text Embedder – [Glove, MPNet]' and 'Fine-tuning strategy – [Freeze, LoRA]' in the search space, but §3.3 defines the text embedder as the frozen LLM's first layer and the main text reports only the frozen strategy; please reconcile the two descriptions or report results for the searched variants.","section":"Appendix B.3"},{"comment":"Minor typos and wording issues: 'Comparision' in the Figure 2 caption, 'REF-F1' in Appendix B.5, 'hyperparameyter' in Appendix B.3, and 'suboptimal' in the Table 2 caption should be 'second best.'","section":"Various"}],"recommendation":"major_revision","confidential_remarks":"The missing LLM ablation is the decisive issue. If the authors can show that replacing the frozen LLM with a non-pretrained decoder of matched capacity degrades performance, the paper would make a solid contribution; without that control, the title claim is not supported. The paper is otherwise carefully evaluated with temporal splits and a useful failure analysis, and the average results are publishable with appropriately hedged claims. The per-task overclaim is easy to fix by rewording."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The architecture is well-engineered and the average results are believable: a temporal-aware heterogeneous GNN encoder, denormalized JSON-style soft prompts, and masked attribute pretraining is a sensible combination, and the code is released. The paper does real work on a serious benchmark, and the appendix is honest about failures (many-shot ICL, cell-level masking). That is worth credit.\n\nThe load-bearing problem is the missing control for the title claim. In Eq. (6), the frozen LLM sees only projected GNN embeddings plus the task text; it never sees attribute text, entity identifiers, or raw rows. The paper never replaces the frozen LLM with a randomly initialized transformer or a matched-capacity MLP decoder. The 1B-vs-3B ablation suggests the LLM size matters, but it does not answer whether the LLM is doing relational reasoning or just acting as an interchangeable head that the GNN already feeds. Without that ablation, \"Large Language Models are Good Relational Learners\" is not established. This is an internal missing control, not a deep conceptual flaw, but it is central.\n\nThe empirical overclaim is real but easily fixed. \"Consistently outperforms or matches all baselines\" is contradicted by Table 1 (Rel-LLM below ICL+MLP on rel-avito user-clicks, rel-event user-ignore, rel-f1 driver-dnf and driver-top3) and Table 2 (driver-position MAE 3.967 vs 3.539). The average AUROC and MAE do favor Rel-LLM, so the honest claim is \"better on average across tasks,\" not consistency.\n\nOther soft spots are minor. The Rel-Zero label is misleading because pretraining uses the same RelBench databases, so it is not independent zero-shot. Baseline standard deviations are missing while Rel-LLM's are reported, making comparisons uneven. The per-task answer generation strategy selection could give a model-selection advantage; the paper should clarify how those choices were made (validation-based or test-based).\n\nNone of this sinks the core contribution. The average gains are plausible and the architecture is a practical advance for combining GNNs with frozen LLMs on relational data. The fix is a focused revision: add the LLM-ablation, soften the consistency claim, and tighten the zero-shot wording. This paper deserves a serious referee; I would accept it with major revision expectations.\n\nRecommendation: send to peer review, and ask for the ablation and rewording before acceptance.","headline":"Solid empirical architecture with real average gains on RelBench, but the title claim is under-supported because the LLM is never ablated, and \"consistently outperforms\" is false on the paper's own tables.","tokens_in":21858,"tokens_out":1373,"would_cite":true,"duration_ms":15308,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Rel-LLM claims that a frozen LLM conditioned on GNN-derived soft prompts outperforms relational deep learning baselines on RelBench.","keywords":["relational deep learning","large language models","graph neural networks","retrieval-augmented generation","graph prompt tuning","RelBench","relational databases","soft prompts"],"falsifier":"Train the same graph encoder and projection with a linear or MLP head in place of the frozen LLM, using identical subgraph sampling, pretraining, and hyperparameters; if the replacement matches or exceeds Rel-LLM's average AUROC and MAE on RelBench, the claim that the LLM contributes relational reasoning is falsified.","tokens_in":20602,"feed_emoji":"🧠","tokens_out":6734,"duration_ms":59814,"temperature":0.7,"pith_summary":"The paper tries to establish that large language models can be effective relational learners when the structure of a relational database is handed to them as compact graph-derived soft prompts instead of as flattened text. It introduces Rel-LLM, which uses a temporal-aware GNN encoder to summarize the local subgraph around an entity, projects those embeddings into the LLM's embedding space, and feeds them to a frozen LLM as a structured prompt. On the RelBench benchmark, the authors report the highest average test AUROC of 77.82 among compared methods, surpassing the relational deep learning baseline (75.83) and the in-context-learning-plus-MLP baseline (76.83), and the lowest average regression MAE of 12.306. They also report that the zero-shot version, without fine-tuning labels, reaches an average AUROC of 63.42, comparable to LightGBM's 63.66. If true, this gives a scalable way to bring LLM reasoning to relational databases without serializing entire tables into text.","feed_headline":"Graph prompts let frozen LLMs beat relational baselines","feed_subtitle":"Rel-LLM averages 77.82 AUROC and 12.306 MAE on RelBench, passing text-serialization and graph baselines.","key_machinery":"The load-bearing object is the graph prompt $H^*$, a nested structure of projected GNN node embeddings built by denormalizing links from the seed entity up to $n_{\\text{nest}}$ neighbors and depth $\\zeta$. It is assembled from the subgraph encoder output and concatenated with the text embedder output as $[H^*; h_{\\text{text}}]$, then fed through the frozen LLM's self-attention layers. This mechanism replaces text serialization with trainable soft prompts, which keeps context length small and lets the LLM attend to relational structure directly; because the LLM is frozen, only the graph encoder, projection MLP, and mask token receive gradients.","core_discovery":"Rel-LLM's central claim is that a frozen LLM can reason over relational structures if those structures are pre-digested by a graph encoder and delivered as soft prompts. For each target entity, the model samples a temporal subgraph that excludes future events, encodes it with a heterogeneous GraphSAGE-style message-passing network, and projects the node and pooled embeddings through an MLP into LLM embedding space. These projections are arranged by a denormalization process into a nested JSON-like graph prompt $H^* = \\{\\hat{h}^* : \\{\\{\\hat{h}^*_i : \\{\\hat{h}^*_{i,j} : \\{\\ldots\\}\\}\\}\\}\\}$, which is concatenated with the text embedding of the task and question and passed through a frozen LLM; gradients update only the graph encoder, projection, and mask token. The model is pretrained by masked attribute prediction, in which whole entities are masked and the LLM must regenerate their attribute text, aligning graph and text spaces. Empirically, the paper reports that this pipeline achieves the best average AUROC (77.82) and lowest average MAE (12.306) on RelBench test sets, with zero-shot AUROC of 63.42 before any fine-tuning.","pith_inferences":["Because the GNN already summarizes all relational structure before the LLM sees it, the paper's headline claim would be tested more sharply by an ablation that replaces the frozen LLM with a simple decoder; if the decoder matches Rel-LLM, the relational ability lives in the GNN, not the LLM.","The same graph-prompt recipe should transfer to knowledge graphs, biomedical records, or financial networks, where heterogeneous graphs and text descriptions coexist, but the paper does not test those settings.","If pretrained textual knowledge matters, Rel-LLM should show larger gains on tasks whose entities are well represented in text, such as the Formula 1 tasks, than on tasks with little textual footprint; this is testable by correlating per-task gains with textual coverage."],"forward_implications":["Rel-LLM's average gains over RDL and ICL+MLP on RelBench imply that graph-derived soft prompts are a viable alternative to text serialization for relational prediction tasks.","The zero-shot AUROC of 63.42, close to LightGBM's 63.66, implies the pretrained graph-text alignment can support predictions without task labels in low-data regimes.","Because the LLM is frozen and only the encoder and projection are trained, Rel-LLM keeps fine-tuning cost low while remaining compatible with larger frozen LLMs, which the paper shows give modest further gains at 3B parameters.","Entity-level masked attribute prediction is reported to outperform cell-level masking, suggesting that forcing the model to reconstruct whole entities builds stronger relational representations.","The many-shot in-context experiments reported in the appendix show that adding many demonstration embeddings hurts Rel-Zero, so the model's relational reasoning is best used with few or no in-context examples."],"supporting_citations":[{"why":"Supplies the RelBench benchmark, its 30 predictive tasks, and the RDL baseline that Rel-LLM is compared against.","marker":"(Robinson et al., 2024)"},{"why":"Defines relational deep learning on relational databases and the graph representation that the GNN encoder builds on.","marker":"(Fey et al., 2024)"},{"why":"Provides the text-serialization ICL baseline and the denormalization idea that Rel-LLM adapts into graph prompts.","marker":"(Wydmuch et al., 2024)"},{"why":"Supplies the temporal neighbor sampling and GraphSAGE message-passing encoder used to extract subgraph embeddings.","marker":"(Hamilton et al., 2017)"},{"why":"Motivates the JSON serialization format used in the structured graph prompt.","marker":"(Singha et al., 2023)"},{"why":"Establishes the retrieval-augmented generation framework that Rel-LLM extends to graph-derived prompts.","marker":"(Lewis et al., 2020)"},{"why":"Provides the multimodal column encoders that produce initial entity embeddings for the relational graph.","marker":"(Hu et al., 2024)"},{"why":"Inspires graph prompt tuning by showing graph-structured prompts can condition frozen LLMs.","marker":"(He et al., 2024)"}],"fun_headline_variants":["GNN-crafted prompts unlock LLM relational reasoning","Rel-LLM: frozen LLMs reason via graph-encoded prompts","Graph prompts let frozen LLMs rival relational models","Rel-LLM beats text-serialization with graph-structured prompts","GNN prompts for frozen LLM beat relational baselines"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a frozen LLM genuinely reasons over projected GNN node embeddings when they are presented as soft prompts, so that the reported gains come from LLM relational reasoning rather than only from the GNN encoder and projection layers.","fun_headline_variants_meta":{"raw":{"variants":["GNN-crafted prompts unlock LLM relational reasoning","Rel-LLM: frozen LLMs reason via graph-encoded prompts","Graph prompts let frozen LLMs rival relational models","Rel-LLM beats text-serialization with graph-structured prompts","GNN prompts for frozen LLM beat relational baselines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000538,"raw_usage":{"total_tokens":2628,"prompt_tokens":1035,"completion_tokens":1593,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":651,"completion_tokens_details":{"reasoning_tokens":1510}},"tokens_in":651,"tokens_out":1593,"duration_ms":10044,"temperature":1.0,"reasoning_tokens":1510,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:13:20.588514+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same graph encoder and projection with a linear or MLP head in place of the frozen LLM, using identical subgraph sampling, pretraining, and hyperparameters; if the replacement matches or exceeds Rel-LLM's average AUROC and MAE on RelBench, the claim that the LLM contributes relational reasoning is falsified.","supporting_citations":[],"review_version":1}