Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

GL-Fusion: Rethinking the Combination of Graph Neural Network and Large Language model

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read GL-Fusion claims that one transformer can match or beat both GNN-centered and LLM-centered methods on graph tasks by fusing message passing, full-text cross-attention, and twin GNN/LLM prediction heads—reaching state of the art on…

desk verdict A promising GNN-LLM fusion architecture, but the ogbn-arxiv SOTA is compromised by an unshared label-input signal. read the letter →

arxiv 2412.06849 v1 pith:3C5RZIWO submitted 2024-12-08 cs.LG cs.AIcs.CL

Haotong Yang , Xiyuan Wang , Qian Tao , Shuxian Hu , Zhouchen Lin , Muhan Zhang This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords graphneuralnetworklargelanguagemodeltext-attributedstructure-awaretransformergraph-textcross-attentiontwinpredictorOGBN-ArxivOGBG-Code2
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a single architecture can combine graph structure with natural language better than either a GNN-centered pipeline (LLM encodes text into fixed vectors, GNN predicts) or an LLM-centered pipeline (graph serialized into tokens). It argues that prior hybrids lose information: GNN-centered models compress text, LLM-centered models compress graph structure. GL-Fusion keeps both uncompressed: message passing runs inside the transformer layers, cross-attention reads the full node text, and a twin predictor produces both parallel GNN-style labels and autoregressive language output. If the claim holds, one model handles node classification, link prediction, commonsense QA, and graph-to-text generation without task-specific format engineering, outperforming specialized baselines on several benchmarks.

What carries the argument

The structure-aware transformer layer is the load-bearing component: a causal attention mask is modified so graph tokens can attend to each other regardless of node order, all graph tokens share one positional index, and a message-passing sublayer with mean, max, and standard-deviation aggregators (gated to start at zero) updates node token representations at selected layers. Graph-text cross-attention supplies full node text to those tokens without bloating the context window, reducing the cost from quadratic in the concatenated text length to O(nLn · Lt), and the twin predictor attaches a graph readout and a language head to the same final representations.

What would settle it

A controlled replication where every baseline gets the exact same node text (including any added labels and distance annotations), and where ogbg-code2 is scored under identical generation-level decoding; if GL-Fusion's margins vanish, the SOTA claim fails. Separately, shuffling node order in the input should leave identical outputs; if it changes, the permutation-invariance claim fails.

Watch

Extended reading notes

Core claim

GL-Fusion's central claim is that a causal LLM can be made graph-aware without giving up text generation. By giving graph tokens a permutation-invariant attention mask and a shared positional index, inserting message passing with multiple aggregators inside selected layers, and adding cross-attention over full uncompressed node text, the same backbone learns both structural and semantic representations. The twin predictor then decouples output mode: a graph readout classifies all nodes in one pass, while the LLM head generates free text. On ogbn-arxiv the combined predictions reach 78.20% accuracy, surpassing baselines such as GLEM, XRT, and OneForAll; on ogbg-code2, generating function names directly as text achieves 40.97% F1 versus 22.22% for the best classification-style baseline; and on the inductive KG completion benchmark FB15k-237-ind it sets the highest MRR on all four splits.

Load-bearing premise

The reported gains come from the architecture itself, not from giving GL-Fusion extra task information—like training-label text on ogbn-arxiv or distance annotations on FB15k-237-ind—that baseline methods did not receive.

Editorial extensions

If this is right

  • Node classification and text generation can come from the same forward pass, so a model does not need a manually reformatted task description per task.
  • Uncompressed node text demonstrably matters: without cross-attention, GL-Fusion's node-text retrieval accuracy falls to 0% and ogbn-arxiv accuracy drops by roughly 2–3 percentage points.
  • In few-shot node classification, GL-Fusion's margin over prompt-based baselines grows as the training set shrinks (56.44% vs 52.48% at 10 shots per class), suggesting the architecture extracts more from limited labels.
  • Both prediction heads are needed: ablations show removing the GNN loss hurts final text accuracy and removing the text loss hurts GNN accuracy, so joint supervision is what drives the ensemble result.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The shared single positional index only works because cross-attention supplies per-node identity from the text; the two mechanisms are coupled, so the design may not transfer to graphs with no text attributes without an alternative identity source.
  • The FB15k-237-ind results lean on distance-to-head/tail annotations in the input text; a fair re-run without that annotation could narrow the gap between GL-Fusion and the GNN baselines, since those baselines did not receive it.
  • The ogbg-code2 comparison is language-generation versus classification: a head-to-head with a generation-trained baseline under identical decoding would clarify whether the gain is architectural or a consequence of the task reframing.
  • The pattern suggests a scaling path the paper names as future work: multi-task pretraining of one GL-Fusion backbone across graph and text corpora could amortize the cost of per-task fine-tuning.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes GL-Fusion, a hybrid architecture that combines a GNN and an LLM. Its three main design elements are structure-aware transformer layers that insert message passing into causal self-attention, graph-text cross-attention blocks that let node and text tokens read from uncompressed node text, and a twin GNN/LLM predictor that supports both one-pass graph predictions and autoregressive text generation. The model is evaluated on synthetic graph property prediction, node classification (ogbn-arxiv, Cora, CSTAG), inductive knowledge graph completion (FB15k-237-ind), commonsense QA, and code graph-to-text generation (ogbg-code2). The central empirical claims are state-of-the-art accuracy on ogbn-arxiv (78.20 versus 77.51 for OneForAll) and state-of-the-art F1 on ogbg-code2 (40.97 versus 22.22 for SAT++). The authors also report strong results on few-shot arxiv and on the inductive KG completion splits.

Significance. If the empirical claims were cleanly established, GL-Fusion would be a meaningful architectural contribution: it combines scalable parallel graph prediction with flexible language output, uses full node text rather than compressed embeddings, and demonstrates that cross-attention can keep text information at a manageable cost. The idea of a twin predictor and the explicit attention-mask design for jointly handling causality and permutation invariance are genuinely interesting and worth publishing in some form. However, the headline state-of-the-art claims rest on benchmark comparisons that are not currently controlled. In particular, the label-in-input procedure for ogbn-arxiv and the distance annotations for FB15k-237-ind give GL-Fusion information that the cited baselines do not receive, and the ogbg-code2 comparison appears to mix different task formulations. These issues are load-bearing: removing or matching the extra input signals could shrink or reverse the reported gaps. The paper does not release code or data, and it reports no variance for GL-Fusion, which further limits the ability to assess whether the improvements are robust.

major comments (4)
  1. [Appendix B.3 / §5.2, Tables 2 and 3] The ogbn-arxiv comparison is confounded by an input-signal asymmetry. Appendix B.3 states: 'We also add label of non-target nodes in training set to input node text.' This means the training nodes' ground-truth classes are present as text in the input sequence. Because the structure-aware attention mask allows all nodes in the same graph to attend to one another, and because message passing propagates information along edges, the model can at inference time diffuse these labels to test nodes, effectively performing learned label propagation. The standard baselines (GCN, GLEM, XRT, OneForAll, GPT4graph, GraphGPT) receive labels only through the training loss, not as input features. The reported margin over OneForAll is only 0.69 points on the full training set, which is small enough that this extra signal could plausibly explain it. The few-shot results in Table 3 are even more directly affected: with 10 or 100 labels per class, embedding the exact labels into the node text makes the task substantially easier, and the reported jumps (52.48 to 56.44 at 10 shots, 61.67 to 68.18 at 100 shots) may largely reflect label memorization and propagation rather than architectural superiority. The authors must either remove the label augmentation, or give all baselines the same input signal, and then re-report the comparisons.
  2. [Appendix B.1 / §5.3, Table 5] The FB15k-237-ind results may also rest on an unshared input feature. Appendix B.1 says that 'Following approaches like NBFNet and UniLP, we annotate nodes with distances to their corresponding head or tail nodes for prediction tasks,' and the example node text includes a 'dist to head' field. NBFNet and UniLP are designed to use such structural annotations, but GraiL, KG-BERT, BertRL, and KRST are not, and the reported MRR improvements over UniLP (for example, 0.8558 versus 0.754 on v1) could be inflated by this extra distance information. The authors should either remove the distance annotations, apply the same annotation protocol to all baselines, or provide an ablation that quantifies the contribution of the distance fields to the reported MRR/H@10 numbers.
  3. [§5.5, Table 7] The ogbg-code2 claim of state-of-the-art performance is based on a comparison that mixes task formulations. The OGB leaderboard baselines (GAT, GraphTrans, SAT++, DAGformer) are evaluated under the standard classification-style protocol of predicting function names from a limited candidate set, whereas GL-Fusion generates free text. The reported F1 of 40.97 may not be comparable to the baselines' F1 values if the label space, decoding procedure, and evaluation metric are not identical. The paper should either run the baselines under the same generation protocol, use the official OGB evaluation for all methods, or clearly explain why the comparison is apples-to-apples. Additionally, no standard deviation or multiple-seed result is reported for GL-Fusion, while baseline values in Table 7 include error bars, so it is unclear whether the margin is statistically meaningful.
  4. [§3.1, Figure 2] The claim that assigning a single shared positional index to all graph tokens preserves permutation invariance while not harming language generation is asserted but not independently validated. The attention mask in Figure 2b does make the node tokens mutually visible, which plausibly restores permutation equivariance of the node representations, but the effect of this positional scheme on language modeling quality, and the interaction with cross-attention, is not tested. A simple ablation comparing the shared positional encoding against distinct positional indices (or sinusoidal positional offsets) on the synthetic property-prediction tasks and on ogbn-arxiv would strengthen the architectural justification.
minor comments (6)
  1. [Abstract] There is a typo in the abstract: 'achieves outstand performance' should read 'achieves outstanding performance.'
  2. [§5.5] In §5.5 the text says 'The results are shown in Table 6' but the code2 results are in Table 7; the table numbering is inconsistent throughout the paper (for example, Table 4 is split across pages, and the CSTAG table is not numbered in the text).
  3. [Table 2] The Cora row in Table 2 contains several missing entries and an array of dashes that is difficult to parse; the authors should present the available baseline numbers cleanly or explain which values are unavailable.
  4. [§5.2] The text mentions OFA as a baseline, but Table 2 lists OneForAll; the relationship between 'OFA' and 'OneForAll' is not clarified, and the same reference (Liu et al., 2023) is cited for both.
  5. [Appendix A] No code or data release is mentioned, which makes it difficult to reproduce the reported results, especially because several preprocessing details (label augmentation, distance annotations, subgraph sampling) are nonstandard and central to the comparisons.
  6. [§7] The limitations section acknowledges that each task uses a separately trained model and that no unified pretrained parameters are established; this is a useful caveat, but it should be connected to the generality claims in the introduction and abstract.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: all central claims are empirical comparisons against external test sets; the label-injection issue is a fairness confound, not a tautological derivation.

full rationale

GL-Fusion makes no derivational claim of the kind that can be circular: there is no equation whose output is defined by a fitted parameter, and no result is imported from a same-author theorem. The headline results are accuracy/F1 numbers on external test sets (ogbn-arxiv, ogbg-code2, FB15k-237-ind, CommonsenseQA) compared against published baselines. The inclusion of OFA (Liu et al. 2023), a paper co-authored by Muhan Zhang, is as a baseline in Table 2 and an external published result, not as justification for the architecture, so it does not make the argument self-supporting. The statement in Appendix B.3 that labels of non-target training nodes are added to the input node text is a genuine input-information asymmetry relative to baselines and should be examined as a fairness/validity concern, but it is not circularity in the defined sense: the model's output is not defined by the label text, and the SOTA claim is not equivalent to an input by construction. Similarly, the distance annotation for FB15k-237-ind is standard practice cited from external methods and is an input-feature choice, not a reduction. No fitted value is renamed as a prediction; no uniqueness theorem from the authors is invoked; no ansatz is smuggled in via citation. The paper is self-contained against external benchmarks, so the internal circularity burden is not met.

Assumptions & free parameters 4 free parameters · 6 assumptions · 1 invented entities

The central empirical claim rests on pretrained 8B backbones, hand-chosen rank and layer schedules, per-task predictor selection, and dataset-specific input enrichments such as training labels and distance annotations. No code is provided to test sensitivity to these choices. No new physical entities are introduced; the only invented objects are special graph tokens.

free parameters (4)
  • LoRA rank per dataset = 64 for ogbn-arxiv, 4 for CSTAG, 32 for others
    Appendix A selects r per dataset; no sensitivity analysis or selection procedure is given.
  • Layer schedule for message passing and cross-attention = Message passing at layers 0,4,8,12,16,20,24,28; cross-attention at 3,7,11,15,19,23,27,31
    Hand-chosen placement; only full removal was ablated, not alternative schedules.
  • Optimizer hyperparameters = AdamW, lr=3e-5, weight decay=0.1, one epoch
    Fixed choices in Appendix A; no tuning protocol or multiple seeds reported.
  • Twin-predictor selection rule per task = Ensembling for node classification and CSQA; GNN-only for link; text-only for graph and synthetic tasks
    The per-task prediction rule is a choice that affects the final reported numbers; its validation-based selection is not described.
assumptions (6)
  • domain assumption Pretrained Llama-3-8B and LLM2Vec provide a suitable text encoder and language decoder for graph tasks.
    Section 3 and Appendix A use these as fixed or LoRA-tuned backbones; if they are inappropriate, all results are affected.
  • ad hoc to paper Shared positional encoding for all graph tokens preserves permutation invariance without harming language generation.
    Section 3.1 introduces this to avoid graph-node order dependence; no independent test of the language-side effect is reported.
  • ad hoc to paper Appending training-node labels to node text on ogbn-arxiv is a fair and non-leaking input.
    Appendix B.3 states the practice; comparability with baselines is not discussed.
  • domain assumption Distance annotations to head/tail nodes for KG completion are available and do not unfairly advantage GL-Fusion.
    Appendix B.1 follows NBFNet/UniLP; the four baseline families are not shown to use identical annotations.
  • standard math Standard attention and message-passing math is valid.
    Section 3 uses softmax attention, MPNN aggregation, and backpropagation without formal proof.
  • ad hoc to paper Each node should cross-attend only to its own text to avoid node representations collapsing.
    Section 3.2: 'we constrain each node to extract information solely from its own text to avoid this problem.' The constraint is justified by observed failure, not by analysis.
invented entities (1)
  • Special graph token embeddings (<graph start>, <graph end>, <node>)
    purpose: Mark graph structure inside the LLM input sequence and give node tokens a shared positional identity.
    These tokens are learned only within this paper's models; there is no external evidence that they generalize across tasks or models.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GL-Fusion: Rethinking the Combination of Graph Neural Network and Large Language model." pith.science (2026). https://pith.science/paper/3C5RZIWO

@misc{pith2026241206849,
  author       = {Pith},
  title        = {Pith review of: GL-Fusion: Rethinking the Combination of Graph Neural Network and Large Language model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3C5RZIWO}},
  note         = {Machine review of arXiv:2412.06849}
}
read the original abstract

Recent research on integrating Large Language Models (LLMs) with Graph Neural Networks (GNNs) typically follows two approaches: LLM-centered models, which convert graph data into tokens for LLM processing, and GNN-centered models, which use LLMs to encode text features into node and edge representations for GNN input. LLM-centered models often struggle to capture graph structures effectively, while GNN-centered models compress variable-length textual data into fixed-size vectors, limiting their ability to understand complex semantics. Additionally, GNN-centered approaches require converting tasks into a uniform, manually-designed format, restricting them to classification tasks and preventing language output. To address these limitations, we introduce a new architecture that deeply integrates GNN with LLM, featuring three key innovations: (1) Structure-Aware Transformers, which incorporate GNN's message-passing capabilities directly into LLM's transformer layers, allowing simultaneous processing of textual and structural information and generating outputs from both GNN and LLM; (2) Graph-Text Cross-Attention, which processes full, uncompressed text from graph nodes and edges, ensuring complete semantic integration; and (3) GNN-LLM Twin Predictor, enabling LLM's flexible autoregressive generation alongside GNN's scalable one-pass prediction. GL-Fusion achieves outstand performance on various tasks. Notably, it achieves state-of-the-art performance on OGBN-Arxiv and OGBG-Code2.

Figures

Figures reproduced from arXiv: 2412.06849 by the authors.

Figure 1
Figure 1. Workflow of the GL-Fusion model. (a) An example of a text-attributed graph, where each [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Design of Structure-Aware Transformer layer. (a) Structure-aware Transformer layer. The [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The attention mask in cross-attention layers. For text tokens before the graph, they do not involve cross-attention to maintain causal￾ity (red dashed line with the ×). For node tokens <node>, each token only has access to its own text (orange lines). For text tokens after the graph, they have access to all node text (black lines). One significant drawback of previous GNN￾centered models is that they compress node, … view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Lost in Serialization: Invariance and Generalization of LLM Graph Reasoners

    cs.LG 2025-11 conditional novelty 6.0 of 10

    Fine-tuned graph reasoners become more stable to node relabeling but more brittle to edge ordering and format, and they do not consistently beat base models on new spectral graph tasks.

Reference graph

Works this paper leans on

15 extracted references · 5 canonical work pages · cited by 1 Pith paper

  1. [1]

    Llm2vec: Large language models are secretly powerful text encoders

    Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Cha- pados, and Siva Reddy. Llm2vec: Large language models are secretly powerful text encoders. CoRR, abs/2404.05961,

    arXiv

  2. [9]

    One for all: Towards training one graph model for all classification tasks

    11 Hao Liu, Jiarui Feng, Lecheng Kong, Ningyue Liang, Dacheng Tao, Yixin Chen, and Muhan Zhang. One for all: Towards training one graph model for all classification tasks. arXiv preprint arXiv:2310.00149,

    arXiv

  3. [10]

    Commonsenseqa: A question answering challenge targeting commonsense knowledge

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In NAACL, pp. 4149–4158. Yanchao Tan, Hang Lv, Xinyi Huang, Jiawei Zhang, Shiping Wang, and Carl Yang. Musegraph: Graph-oriented instruction tuning of large language models for generic graph mining. CoRR, abs/2...

    arXiv

  4. [11]

    Human parity on commonsenseqa: Augmenting self-attention with external attention

    Yichong Xu, Chenguang Zhu, Shuohang Wang, Siqi Sun, Hao Cheng, Xiaodong Liu, Jianfeng Gao, Pengcheng He, Michael Zeng, and Xuedong Huang. Human parity on commonsenseqa: Augmenting self-attention with external attention. arXiv preprint arXiv:2112.03254,

    arXiv

  5. [13]

    [reverse] xxx

    #links #nodes #links v1 train 183 2000 5226 ind-test 146 1500 2404 v2 train 203 3000 12085 ind-test 176 2000 5092 v3 train 218 4000 22397 ind-test 187 3000 9137 v4 train 222 5000 33916 ind-test 204 3500 14554 KG completion work on GNN, we also added reverse relation for each relation and label them as “[reverse] xxx”. The total graph is too large to input...

    work page 2000

  6. [15]

    cite” or “cited

    For ogbn-arxiv, we directly use official split. For node 14 text, we use paper title and abstract. We also add label of non-target nodes in training set to input node text. The edge text are “cite” or “cited”.U Table 11: Statistics of citation datasets. Dataset #Nodes #Edges #Class Split Ratio Metric text length ogbn-arxiv 169,343 1,166,243 40 Time 54/18/...

    work page 2021

  7. [32]

    Some parameters are loaded from pretrained LLM

    GNN in our model in implemented with torch geometric (Fey & Lenssen, 2019). Some parameters are loaded from pretrained LLM. • For structure-aware Transformer Layers, we leverage the pretrained Llama-3-8B model as the backbone. As detailed in Section 3.1, we introduce new parameters through modifica- tions to positional encoding and attention masks. Additi...

    work page 2019

  8. [500]

    title”: “Jane Krakowski

    An example of our dataset: The input question: The Adventures of Tintin (2011 film directed by Steven Spielberg) –/film/film/productioncompanies− →?. The graph:<graphstart><node><node>. . .<node><graphend>. The node text features in the graph: {“title”: “Jane Krakowski”, “desc”: “American actress”, “dist to head”: 6} {“title”: “Shochiku”, “desc”: “Japanes...

    work page 2023

Show all 15 references
  1. [1998]

    ISBN 0897919653

    Association for Computing Machinery. ISBN 0897919653. doi: 10.1145/276675.276685. URL https://doi.org/10.1145/276675.276685. Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In ICML, pp. 1263–1272,

  2. [2008]

    ISBN 9781605581026

    Association for Computing Machinery. ISBN 9781605581026. doi: 10.1145/1376616.1376746. URL https://doi.org/10.1145/1376616.1376746. Ziwei Chai, Tianjie Zhang, Liang Wu, Kaiqiao Han, Xiaohai Hu, Xuanwen Huang, and Yang Yang. Graphllm: Boosting graph reasoning ability of large l...

  3. [2017]

    Gpt4graph: Can large language models understand graph structured data? an empirical evaluation and benchmarking

    Jiayan Guo, Lun Du, and Hengyu Liu. Gpt4graph: Can large language models understand graph structured data? an empirical evaluation and benchmarking. CoRR, abs/2305.15066, 2023a. Jiayan Guo, Lun Du, Hengyu Liu, Mengyu Zhou, Xinyi He, and Shi Han. Gpt4graph: Can large language m...

    arXiv

  4. [2020]

    Ogb-lsc: A large-scale challenge for machine learning on graphs

    Weihua Hu, Matthias Fey, Hongyu Ren, Maho Nakata, Yuxiao Dong, and Jure Leskovec. Ogb-lsc: A large-scale challenge for machine learning on graphs. arXiv preprint arXiv:2103.09430,

    arXiv

  5. [2021]

    Prompt-based node feature extractor for few-shot learning on text-attributed graphs

    Xuanwen Huang, Kaiqiao Han, Dezheng Bao, Quanjin Tao, Zhisheng Zhang, Yang Yang, and Qi Zhu. Prompt-based node feature extractor for few-shot learning on text-attributed graphs. CoRR, abs/2309.02848,

    arXiv

  6. [2023]

    Fast graph representation learning with pytorch geometric

    Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. arXiv preprint arXiv:1903.02428,

    1903 arXiv

  7. [2024]

    Freebase: a collab- oratively created graph database for structuring human knowledge

    Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. Freebase: a collab- oratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data, SIGMOD ’08, pp. 1247–1250, 1...

    2008

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.