REVIEW 5 major objections 6 minor 24 references
Navigating the Black Box: Leveraging LLMs for Effective Text-Level Graph Injection Attacks
T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read A black-box attacker who sees only a small labeled sample can make GNNs misclassify targeted nodes by injecting LLM-written text that stays on topic.
desk verdict A real new black-box text-level GIA method, but the paper's universal superiority claim is contradicted by its own Table II. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the Target-Injected node Similarity (T-I Similarity), Eq. (5), which scores each candidate injected text by $(1-\lambda)c(o,a)+\lambda\sum_{v_i\in N_o^k} w_i c(i,a)$, where $w_i$ is a GNNExplainer-derived importance of neighbor $v_i$ for the target node and $c(\cdot,\cdot)$ is cosine similarity. This score is used three ways: it selects the generated text with the lowest similarity, it ranks existing node texts to serve as positive few-shot examples, and it feeds the replay buffer for example sampling. The prompting side supplies three strategies -- stay-away, contrast, and fusion -- that constrain the LLM's exploration so generated text stays within the topic and category while being semantically distant from the target and its important neighbors. GNNExplainer, a post-hoc explanation method that assigns importance to nodes, also selects the injection position, deciding whether the injected node connects directly to the target or to important neighbors in an indirect attack.
What would settle it
Take the Fitness-small setup, train the surrogate on labels drawn from a different class distribution than the target model, and run ATAG-LLM's text selection; if the attack success rate drops sharply compared with the reported values, the T-I similarity proxy is not transferring to the unseen target, and the method's success depends on the surrogate rather than the generated text.
Extended reading notes
Core claim
The central claim is that ATAG-LLM, a framework using LLM text generation with three prompting strategies and a Target-Injected node Similarity (T-I Similarity) score, achieves state-of-the-art attack success on text-attributed graphs. T-I Similarity is a weighted combination of cosine similarity between the injected node and the target node, and between the injected node and the target's important k-hop neighbors, with node importance obtained from GNNExplainer on a surrogate model trained with 10% of test labels. Selecting candidate texts with the lowest T-I similarity, while keeping the LLM constrained to the dataset's topic and categories, gives injected nodes that disrupt graph homophily without looking out of place. The paper reports that this outperforms embedding-level attacks (TDGIA, G2A2C) and text-level attacks (RandLLM, WTGIA) on GCN and EGNNGuard, with the largest margins on the Fitness-small dataset, and that combined strategies beat any single strategy.
Load-bearing premise
The method's selection loop assumes the surrogate model trained on 10% of test labels gives node-importance and T-I similarity scores that remain valid for LLM-generated text and transfer to the unseen target model.
Editorial extensions
If this is right
- If the paper is right, a strict black-box attacker needs no gradient access and no full surrogate training: 10% of test labels plus an LLM API is enough to degrade GCN and EGNNGuard classification.
- The attack transfers to a defense mechanism that explicitly restores homophily: on EGNNGuard the reported ASR for ATAG-LLM is higher than against plain GCN, whereas baseline methods often drop, suggesting that natural-looking on-topic text is what defeats the defense.
- Text length constrains attack quality: RandLLM is competitive on Books-History where text can be long, but falls behind on Fitness-small where text length is limited; ATAG-LLM's constrained generation does well in both.
- Combining the three prompting strategies outperforms any single one, indicating that exploration and reliability are complementary rather than competing.
- Direct and indirect attacks both work, meaning the method can be applied even when the attacker cannot link injected nodes directly to the target.
Reading between the lines
- The paper validates the T-I similarity proxy using copied existing node texts, not LLM-generated texts under topic constraints; if the proxy fails to transfer to generative output, the attack's advantage would shrink in domains with broad categories or drifting LLM output.
- The framework is modular: the LLM, the text encoder, and the node-importance estimator could each be swapped for cheaper alternatives, and testing those swaps would reveal which component actually carries the attack.
- A defense direction not explored in the paper would be to check whether injected nodes form a detectable cluster of near-duplicate topic text with unusually low similarity to their neighbors, which the T-I similarity metric itself could flag.
- The Fitness-small results are much stronger than the Books-History results; the paper attributes this to text-length constraints, but the pattern also suggests that category breadth and dataset homophily modulate attack success, which a targeted study could confirm.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes ATAG-LLM, a black-box text-level graph injection attack for text-attributed graphs. The method uses an LLM with three prompting strategies (stay-away, contrast, fusion) to generate natural candidate texts, a T-I similarity measure based on GNNExplainer node importances to select text and few-shot examples, and a GNNExplainer-based injection position selector. Experiments on Books-History and Fitness-small against GCN and EGNNGuard compare ATAG-LLM with embedding-level (TDGIA, G2A2C) and text-level (RandLLM, WTGIA) baselines. The abstract and conclusion claim superior or state-of-the-art performance over both embedding-level and text-level methods across various scenarios.
Significance. If the claims were fully supported, the paper would make a useful contribution: it addresses a realistic threat model where the attacker can inject only interpretable raw text, avoids full surrogate retraining, and provides a concrete LLM-prompting recipe with an explainability-based selection criterion. The Fitness-small results are striking, and the consistent advantage against EGNNGuard suggests the method is genuinely effective at producing text that evades a homophily-aware defense. The paper also gives a relatively transparent experimental setup and includes ablations over the three generation strategies. However, the breadth of the headline claim is not supported by the paper's own Table II, and several methodological choices need additional validation before the central contribution can be accepted.
major comments (5)
- [Abstract and Section VI (Conclusion) vs. Table II] The claim that ATAG-LLM 'outperforms existing state-of-the-art methods, both at the embedding level and text level, across various scenarios' is directly contradicted by Table II. On Books-History against GCN, TDGIA achieves 40.43±2.64, 42.71±2.16, and 42.71±1.35 at ΔN=5, 10, and 20, whereas ATAG-LLM achieves 25.12±3.62, 33.33±3.13, and 36.23±3.13. The gap is approximately 15, 9, and 6 points, respectively, so the claim of embedding-level superiority is false in this scenario. The authors should either narrow the claim to text-level methods or to the specific datasets/models where the advantage holds, or provide additional evidence that resolves this inconsistency.
- [Section IV-E and Table II] No statistical significance tests are reported, and several comparisons the text calls 'significantly outperforming' have overlapping or nearly overlapping standard deviations. For example, on Books-History against GCN at ΔN=20, ATAG-LLM reports 36.23±3.13 versus RandLLM's 36.72±4.48, and at ΔN=5 the two are 25.12±3.62 versus 25.60±3.88. Since the results are repeated over 10 target-node groups, paired tests or a bootstrap over those groups should be reported to justify the comparative claims.
- [Section III-A, Figure 4, and Figure 8] The T-I similarity proxy is validated only for copied existing node texts in Figure 4, not for LLM-generated candidates under topic and naturalness constraints. The generator's distribution is different from the random-copy distribution, so the correlation in Figure 4 may not transfer. Figure 8 shows that combining the three strategies beats each single strategy, but it does not test whether selecting the candidate with the lowest T-I similarity is better than random selection among valid candidates. An ablation that compares random candidate selection with T-I-similarity-based selection on the generated texts is needed to support the central mechanism.
- [Sections II-C, IV-C, and IV-D] The 'strict black-box' claim needs clarification and additional analysis. Target nodes are selected as nodes correctly classified by the target model, using 25% highest and 25% lowest confidence margins from the target model's predictions, which presumes access to target-model confidences; a strict black-box attacker would not know which test nodes are correctly classified. In addition, the surrogate is trained with 10% of true test labels plus pseudo-labels above 0.75, and no label-budget sensitivity is reported. Please state exactly what the attacker observes and show how performance varies with the number of available labels.
- [Section IV-D] Several hyperparameters are set without sensitivity analysis: λ=0.1, m=10, Nn=1, Np=2, Nr=2, Ne=1000, and the pseudo-label confidence threshold of 0.75. Because the method's effectiveness depends on these choices and the paper claims generalizability across scenarios, the authors should report sensitivity to at least λ, Nr/Ne, and the confidence threshold, and preferably also to the label budget.
minor comments (6)
- [Section II-C] The symbol V′ is used both for the node set of the perturbed graph G′ and for the small subset of Vtest whose labels the attacker can access; this is confusing and should be renamed, for example V_labeled.
- [Figure 4] The caption says 'T-I Similarities' while the text uses 'T-I similarity'; please make the capitalization consistent across the manuscript.
- [Table III] The indirect-attack results are reported only for ΔN=5 and 10; please report ΔN=20 as well for consistency with Table II.
- [Section IV-D] The description '10 groups of target nodes for repeated experiments' should specify whether each group is an independent sample of target nodes with separate injection executions, and how the standard deviations in Tables II and III are computed.
- [Figures 7 and 8] The figures do not state the number of repeated runs or show error bars; adding this information would make the ablation results easier to interpret.
- [References] Reference [15] is cited as a preprint; if a peer-reviewed version exists, it should be cited instead, and the relationship between WTGIA and the implemented 'TGDIA' baseline should be clarified.
Circularity Check
No significant circularity: ATAG-LLM's T-I similarity is an empirically motivated selection heuristic, not an input-equivalent derivation of attack success.
full rationale
The paper's central chain is: define T-I similarity in Eq. (5), show in Figure 4 that lower T-I similarity correlates with higher ASR on the same datasets/models, then select LLM-generated text and injection positions to minimize T-I similarity. This is a heuristic optimization loop, not a derivation in which the target quantity (ASR, Eq. (8)) is defined as T-I similarity or inferred from it by construction. No fitted parameter is renamed as a prediction: lambda and m are hyperparameters, and the Figure 4 correlation is presented as empirical motivation rather than as a theorem or a fitted ASR model. The only apparent self-citation, reference [1] (which includes co-author Chaozhuo Li), is used to source the Amazon Books-History and Sports-Fitness datasets and to motivate TAGs; it is not load-bearing for the attack method's correctness. The Books-History GCN result, where TDGIA outperforms ATAG-LLM, undermines the conclusion's universal 'across various scenarios' claim, but that is a consistency/overclaim issue, not circularity. No specific equation or fitted value reduces to another equation or to the evaluation metric, so no circular step can be exhibited.
Assumptions & free parameters
free parameters (7)
- lambda (T-I similarity trade-off) =
0.1
- m (truncation coefficient) =
10
- Nn (number of negative examples) =
1
- Np (number of positive examples) =
2
- Nr (generation rounds per strategy) =
2
- Ne (positive example sampling pool) =
1000
- pseudo-label confidence threshold =
0.75
assumptions (5)
- standard math Cosine similarity between node text embeddings is a meaningful semantic distance for attacking GNNs.
- domain assumption Disrupting graph homophily by injecting dissimilar nodes degrades GNN classification, and defenses like EGNNGuard try to restore homophily.
- domain assumption GNNExplainer importance weights computed on a surrogate model transfer to the unseen target model and identify attack-relevant neighbors.
- ad hoc to paper An LLM can generate text fitting topic/category constraints while minimizing T-I similarity, and such text remains semantically natural.
- ad hoc to paper A strict black-box attacker can obtain true labels for a subset of the test set and use them to train a surrogate.
invented entities (1)
-
T-I Similarity (Target-Injected Node Similarity)
Cite this review
Pith. "Pith review of Navigating the Black Box: Leveraging LLMs for Effective Text-Level Graph Injection Attacks." pith.science (2026). https://pith.science/paper/BOUXZVGG
@misc{pith2026250613276,
author = {Pith},
title = {Pith review of: Navigating the Black Box: Leveraging LLMs for Effective Text-Level Graph Injection Attacks},
year = {2026},
howpublished = {\url{https://pith.science/paper/BOUXZVGG}},
note = {Machine review of arXiv:2506.13276}
}
read the original abstract
Text-attributed graphs (TAGs) integrate textual data with graph structures, providing valuable insights in applications such as social network analysis and recommendation systems. Graph Neural Networks (GNNs) effectively capture both topological structure and textual information in TAGs but are vulnerable to adversarial attacks. Existing graph injection attack (GIA) methods assume that attackers can directly manipulate the embedding layer, producing non-explainable node embeddings. Furthermore, the effectiveness of these attacks often relies on surrogate models with high training costs. Thus, this paper introduces ATAG-LLM, a novel black-box GIA framework tailored for TAGs. Our approach leverages large language models (LLMs) to generate interpretable text-level node attributes directly, ensuring attacks remain feasible in real-world scenarios. We design strategies for LLM prompting that balance exploration and reliability to guide text generation, and propose a similarity assessment method to evaluate attack text effectiveness in disrupting graph homophily. This method efficiently perturbs the target node with minimal training costs in a strict black-box setting, ensuring a text-level graph injection attack for TAGs. Experiments on real-world TAG datasets validate the superior performance of ATAG-LLM compared to state-of-the-art embedding-level and text-level attack methods.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[1]
A comprehensive study on text-attributed graphs: Benchmarking and rethinking,
H. Yan, C. Li, R. Long, C. Yan, J. Zhao, W. Zhuang, J. Yin, P. Zhang, W. Han, H. Sun, W. Deng, Q. Zhang, L. Sun, X. Xie, and S. Wang, “A comprehensive study on text-attributed graphs: Benchmarking and rethinking,” in Proc. Advances in Neural In- formation Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, New Orleans, ...
work page 2023
-
[2]
Adversarial attack on graph structured data,
H. Dai, H. Li, T. Tian, X. Huang, L. Wang, J. Zhu, and L. Song, “Adversarial attack on graph structured data,” in Proc. 35th International Conference on Machine Learning, Stockholm, Sweden, Jul. 2018, pp. 1123–1132
work page 2018
-
[3]
Adversarial attacks on neural networks for graph data,
D. Z ¨ugner, A. Akbarnejad, and S. G ¨unnemann, “Adversarial attacks on neural networks for graph data,” in Proc. 28th In- ternational Joint Conference on Artificial Intelligence , Macao, China, Aug. 2019, pp. 6246–6250
work page 2019
-
[4]
Textbugger: Generating adversarial text against real-world applications,
J. Li, S. Ji, T. Du, B. Li, and T. Wang, “Textbugger: Generating adversarial text against real-world applications,” arXiv preprint, vol. arXiv:1812.05271, 2018
arXiv 2018
-
[5]
B. He, M. Ahamad, and S. Kumar, “Petgen: Personalized text generation attack on deep sequence embedding-based classifica- tion models,” in Proc. 27th ACM SIGKDD Conference on Knowl- edge Discovery and Data Mining, Virtual Event, Singapore, Aug. 2021, pp. 575–584
work page 2021
-
[6]
X. Hu, G. Liu, B. Zheng, L. Zhao, Q. Wang, Y . Zhang, and M. Du, “Fasttextdodger: Decision-based adversarial attack against black-box nlp models with extremely high efficiency,” IEEE Trans. Inf. Forensics Secur., vol. 19, pp. 2398–2411, 2024
work page 2024
-
[7]
Adversarial attack and defense on graph data: A survey,
L. Sun, Y . Dou, C. Yang, K. Zhang, J. Wang, S. Y . Philip, L. He, and B. Li, “Adversarial attack and defense on graph data: A survey,”IEEE Trans. Knowl. Data Eng., vol. 35, no. 8, pp. 7693– 7711, 2022
work page 2022
-
[8]
Adversarial attacks on node embeddings via graph poisoning,
A. Bojchevski and S. G ¨unnemann, “Adversarial attacks on node embeddings via graph poisoning,” in Proc. 36th International Conference on Machine Learning , Long Beach, CA, USA, Jun. 2019, pp. 695–704
work page 2019
Show all 24 references
-
[9]
Let graph be the go board: Gradient-free node injection attack for graph neural networks via reinforcement learning,
M. Ju, Y . Fan, C. Zhang, and Y . Ye, “Let graph be the go board: Gradient-free node injection attack for graph neural networks via reinforcement learning,” in Proc. Thirty-Seventh AAAI Confer- ence on Artificial Intelligence, Washington, DC, USA, Feb. 2023, pp. 4383–4390
2023
-
[10]
Tdgia: Effective injection attacks on graph neural networks,
X. Zou, Q. Zheng, Y . Dong, X. Guan, E. Kharlamov, J. Lu, and J. Tang, “Tdgia: Effective injection attacks on graph neural networks,” in Proc. 27th ACM SIGKDD Conference on Knowl- edge Discovery and Data Mining, Virtual Event, Singapore, Aug. 2021, pp. 2461–2471
2021
-
[11]
Understanding and improving graph injection attack by promoting unnoticeability,
Y . Chen, H. Yang, Y . Zhang, K. Ma, T. Liu, B. Han, and J. Cheng, “Understanding and improving graph injection attack by promoting unnoticeability,” in Proc. The Tenth International Conference on Learning Representations , Apr. 2022
2022
-
[12]
Adversarial camouflage for node injection attack on graphs,
S. Tao, Q. Cao, H. Shen, Y . Wu, L. Hou, F. Sun, and X. Cheng, “Adversarial camouflage for node injection attack on graphs,” Inf. Sci., vol. 649, p. 119611, 2023
2023
-
[13]
Single node injection attack against graph neural networks,
S. Tao, Q. Cao, H. Shen, J. Huang, Y . Wu, and X. Cheng, “Single node injection attack against graph neural networks,” in Proc. 30th ACM International Conference on Information and Knowledge Management , Virtual Event, Queensland, Australia, Nov. 2021, pp. 1794–1803
2021
-
[14]
Scalable attack on graph data by injecting vicious nodes,
J. Wang, M. Luo, F. Suya, J. Li, Z. Yang, and Q. Zheng, “Scalable attack on graph data by injecting vicious nodes,” Data Min. Knowl. Discov., vol. 34, no. 5, pp. 1363–1389, 2020
2020
-
[15]
Intruding with words: Towards understanding graph injection attacks at the text level,
R. Lei, Y . Hu, Y . Ren, and Z. Wei, “Intruding with words: Towards understanding graph injection attacks at the text level,” CoRR, vol. abs/2405.16405, 2024
2024 arXiv
-
[16]
How does heterophily impact the robustness of graph neural networks?: Theoretical connections and practical implications,
J. Zhu, J. Jin, D. Loveland, M. T. Schaub, and D. Koutra, “How does heterophily impact the robustness of graph neural networks?: Theoretical connections and practical implications,” in Proc. 28th ACM SIGKDD Conference on Knowledge Discov- ery and Data Mining , Washington, DC, ...
2022
-
[17]
Gn- nexplainer: Generating explanations for graph neural networks,
Z. Ying, D. Bourgeois, J. You, M. Zitnik, and J. Leskovec, “Gn- nexplainer: Generating explanations for graph neural networks,” in Proc. Advances in Neural Information Processing Systems 32 , 2019, pp. 9240–9251
2019
-
[18]
Semi-supervised classification with graph convolutional networks,
T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” in Proc. Intl. Conf. Learning Representations, 2017
2017
-
[19]
Node injection attacks on graphs via reinforcement learning,
Y . Sun, S. Wang, X. Tang, T. Hsieh, and V . G. Honavar, “Node injection attacks on graphs via reinforcement learning,” CoRR, vol. abs/1909.06543, 2019
1909 arXiv
-
[20]
Message injection attack on rumor detection under the black-box evasion setting using large language model,
Y . Luo, Y . Li, D. Wen, and L. Lan, “Message injection attack on rumor detection under the black-box evasion setting using large language model,” in Proc. ACM Web Conference 2024 , Singapore, May 2024, pp. 4512–4522
2024
-
[21]
A survey of graph meets large language model: Progress and future directions,
Y . Li, Z. Li, P. Wang, J. Li, X. Sun, H. Cheng, and J. X. Yu, “A survey of graph meets large language model: Progress and future directions,” in Proc. Thirty-Third International Joint Conference on Artificial Intelligence (IJCAI) . Jeju, South Korea: ijcai.org, Aug. 2024, pp....
2024
-
[22]
Learning on graphs with large language models(llms): A deep dive into model robustness,
K. Guo, Z. Liu, Z. Chen, H. Wen, W. Jin, J. Tang, and Y . Chang, “Learning on graphs with large language models(llms): A deep dive into model robustness,” CoRR, vol. abs/2407.12068, 2024
2024 arXiv
-
[23]
Graphedit: Large language models for graph structure learning,
Z. Guo, L. Xia, Y . Yu, Y . Wang, Z. Yang, W. Wei, L. Pang, T. Chua, and C. Huang, “Graphedit: Large language models for graph structure learning,” CoRR, vol. abs/2402.15183, 2024
2024 arXiv
-
[24]
Can large language models improve the adversarial ro- bustness of graph neural networks?
Z. Zhang, X. Wang, H. Zhou, Y . Yu, M. Zhang, C. Yang, and C. Shi, “Can large language models improve the adversarial ro- bustness of graph neural networks?” in Proc. 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining , Toronto, ON, Canada, Aug. 2025, pp. 2008–2019
2025
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.