REVIEW 4 major objections 5 minor 33 references
LLMs Between the Nodes: Community Discovery Beyond Vectors
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read CommLLM claims a prompted GPT-4o recovers community structure from neighbor lists alone, beating four LLM baselines on six graphs.
desk verdict Plausible idea, sloppy evaluation: the undocumented partial-node merging for Cora/CiteSeer is a load-bearing gap, and the baselines are too inconsistent to support the headline. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is graph-to-text conversion combined with an instruction-tuned prompt. Adjacency lists such as 'Node 0 is connected to: 1, 2, 3 ...' preserve local topology as a token sequence, and the prompt defines a community, states the task, demands the exact 'Node:<id>; Community:<id>' format, and forbids extra text. This combination lets GPT-4o produce a parseable community assignment, with the strict formatting constraint doing the work of turning free-form reasoning into evaluable labels.
What would settle it
Run CommLLM on Cora with the same prompt but with nodes presented in three different random orders; if the parsed, merged partition's NMI against ground truth swings beyond the reported ±0.03 or collapses toward zero, the claimed performance depends on serialization order and the merging strategy rather than on graph structure the model recovered.
Extended reading notes
Core claim
CommLLM's central claim is that GPT-4o's reasoning ability, guided by a carefully engineered prompt and an adjacency-list text encoding, recovers meaningful community structure directly from connectivity. The method converts an undirected graph G=(V,E) into text lines of the form 'Node 0 is connected to: 1, 2, 3 ...', then prompts the model with a definition of community, the task instruction, and the requirement to output 'Node:<id>; Community:<id>' with no additional text. Across Karate Club, Football, WebKB, Terrorist Attacks, Cora, and CiteSeer, CommLLM achieves the best NMI on five datasets and best ARI on four, with Table 4 values such as NMI=0.90 on Karate Club and NMI=0.91 on Football; it is also the only method that completes inference on the two largest graphs. The paper frames this as an implicit function f:T(G)->C, where the model infers clusters without optimizing any explicit objective.
Load-bearing premise
The method assumes that when the model returns community labels for only part of a large graph, those partial outputs can be stitched into a complete, aligned partition whose quality scores still mean what they normally mean.
Editorial extensions
If this is right
- LLMs can perform zero-shot community detection on graphs up to a few thousand nodes without any graph-specific training or structural assumptions.
- Prompt phrasing is decisive: on Karate Club, CommLLM's full prompt reaches NMI=0.90 while stripped-down variants score 0.55-0.78, showing that definition, task framing, and output constraints materially change results.
- The approach is token-bound, with input tokens growing from 777 for Karate Club to roughly 58,000 for Cora and CiteSeer, so larger graphs will require batching, summarization, or a more compact encoding.
- On Cora and CiteSeer, rival LLMs either exhaust their token budgets or score below 0.1, while CommLLM completes inference, giving it a practical edge on mid-size networks.
- Since no modularity, spectral, or other graph objective is used, the results suggest community structure can be recovered from local connectivity descriptions alone.
Reading between the lines
- The paper never compares against cheap classical algorithms such as Louvain, Infomap, or label propagation, so a natural test is whether CommLLM's margins over other LLMs survive when those non-LLM baselines run on the same six graphs.
- Adjacency-list order is likely to matter because LLM attention is not permutation-invariant; shuffling node order or neighbor order and measuring NMI variance would reveal how much of the result comes from graph structure rather than serialization luck.
- For Cora and CiteSeer, the paper reports partial predictions without describing the merging procedure, leaving open whether community labels are aligned across calls; a concrete extension is to force a global label scheme across batches and check whether the reported NMI and ARI hold.
- The paper asserts but does not ablate that its encoding preserves topology; testing edge-list or incident encodings against adjacency lists would show which part of the pipeline is responsible for the gains.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes CommLLM, a two-step community detection method that converts a graph's adjacency structure into a text description and prompts GPT-4o to return a node-to-community assignment. The method is evaluated on six real-world datasets (Karate Club, Football, WebKB, Terrorist Attacks, Cora, CiteSeer) against four LLM baselines (gemini-1.5-pro, gpt-3.5-turbo/gpt-4-turbo, llama3-3-70b-instruct, claude-3.5-sonnet) using NMI, ARI, VOI, and purity. The authors report that CommLLM outperforms the LLM baselines on most datasets, present a prompt-comparison study on Football, and discuss scaling behavior in terms of time and token usage. The central claim is that an LLM prompted with serialized graph topology can recover meaningful community structure on small to medium graphs without graph-specific training.
Significance. If the reported results were reliable, the paper would provide a useful empirical data point showing that a zero-shot prompted LLM can perform community detection on small graphs, complementing existing work on LLM graph reasoning. The public availability of the dataset and conversion script, the repeated execution with reported variance, and the explicit prompt-ablation study are positive features that aid reproducibility. However, the significance is currently limited by the small graph sizes, the absence of any classical community detection baseline, and the evaluation-protocol issues detailed below. The head-to-head comparison with other LLM prompting approaches is potentially useful, but the load-bearing Cora and CiteSeer results are not verifiable from the manuscript as written.
major comments (4)
- [§4.4, Table 2 caption] The caption of Table 2 states that for Cora and CiteSeer the model can only predict partial nodes at a time, and Table 4 reports NMI, ARI, VOI, and purity for these datasets. The manuscript never describes how these partial predictions are merged into a single complete community assignment. NMI and ARI require one labeling over all nodes, so the reported scores are undefined unless the batching scheme, overlap between batches, community-ID alignment across calls, and duplicate-resolution strategy are fully specified. This missing protocol is load-bearing because the claimed advantage of CommLLM on Cora and CiteSeer rests entirely on these numbers, and the results are not reproducible without it.
- [§4.4, Terrorist Attacks paragraph vs. Table 4] The text says CommLLM 'achieves competitive scores' and that 'Claude outperforms in NMI and Purity, but CommLLM remains close while offering better balance across all metrics.' In Table 4, however, claude-3.5-sonnet has NMI 0.50 vs. 0.37, ARI 0.36 vs. 0.34, Purity 0.73 vs. 0.56, and VOI 1.12 vs. 1.14 on Terrorist Attacks, meaning Claude is better on every reported metric. This directly contradicts the paper's central claim that CommLLM outperforms the baseline LLM methods, and the discrepancy must be resolved rather than explained away.
- [§4.4, Prompt comparisons] The prompt-selection experiment is performed on the Football dataset, and the selected Prompt 4 is then used for all six datasets, including Football, in the final results of Table 4. This is a form of selection on the evaluation set: the Football CommLLM result is not an independent test of the prompt. The authors should either use a separate validation dataset for prompt selection or clearly report the selection procedure as part of the method development and avoid presenting the chosen prompt's performance on Football as an unbiased estimate.
- [§4.2 vs. Table 4] The baselines are inconsistently identified across the method description and the results table. Section 4.2 lists gpt-3.5-turbo as a baseline, but Table 4 uses gpt-4-turbo for Terrorist Attacks, Cora, and CiteSeer, with no explanation for the switch. In addition, the model named 'llama3-3-70b-instruct' is cited to the Llama 2 paper [29], not to any Llama 3 release. These inconsistencies make it unclear whether the same protocol was applied to all models and datasets, and they weaken the comparison that supports the main claim.
minor comments (5)
- [§4.4, Karate Club paragraph] The text says 'TribeLLM achieves NMI = 0.90' although the method is called CommLLM; this appears to be a typo.
- [§4.1] The word 'avaiable' should be 'available' in the sentence about the public dataset link.
- [Table 4 caption] The caption calls the baselines 'state-of-art methods,' but the table contains only LLM-based methods and no classical community detection algorithm from the literature reviewed in Section 2.1. The phrasing is therefore overbroad.
- [§4.4, Football heading] The heading 'F ootball' contains an erroneous space and should be 'Football.'
- [§4.4, Prompt 2 discussion] The discussion says Prompt 2 'lacks the actual node connections or network structure to base decisions on,' but all prompts are presumably appended to the same graph details; this claim needs clarification about what exactly differs.
Circularity Check
No significant circularity: the paper is an empirical LLM-prompting study, not a derivation that reduces to its inputs, and its self-citations are background-only.
full rationale
CommLLM makes no formal derivation claim. Its pipeline is graph-to-text serialization followed by a GPT-4o prompt, and its reported numbers are measured against external ground-truth communities and external baselines. No equation in Section 3 defines the output in terms of the evaluation metrics, and no parameter is fitted to the reported NMI/ARI values. The prompt-selection experiment (Table 3) chooses Prompt 4 because it performed best on Football and then reuses Football in Table 4; this is a genuine evaluation-protocol concern (selection bias on one dataset), but it is not circularity because the prompt choice does not by construction determine the reported score. The self-citations (refs. 2, 3, 11, 14, 15, 16) appear only in the literature review and are not load-bearing for the method's validity. The undocumented partial-output merging for Cora/CiteSeer and the inconsistent baseline model names are reproducibility and correctness issues, not instances of a derivation reducing to its own inputs. The central empirical comparison is self-contained against external data, so no circular step is exhibited.
Assumptions & free parameters
free parameters (3)
- Prompt template (Prompt 4) =
selected as best of four variants in Table 3
- Graph-to-text serialization format =
Node i is connected to: neighbor list
- Partial-node batching for Cora and CiteSeer =
not specified
assumptions (5)
- domain assumption Communities are groups of nodes more densely connected internally than to the rest of the network.
- domain assumption Adjacency-list serialization preserves enough graph topology for the LLM to recover community structure.
- standard math NMI, ARI, VOI, and purity are valid when computed between LLM output and ground truth partitions.
- ad hoc to paper Partial node predictions from multiple LLM calls can be merged into a single coherent assignment.
- domain assumption Dataset ground-truth community labels are reliable references.
Cite this review
Pith. "Pith review of LLMs Between the Nodes: Community Discovery Beyond Vectors." pith.science (2026). https://pith.science/paper/JQ4SBLMT
@misc{pith2026250722955,
author = {Pith},
title = {Pith review of: LLMs Between the Nodes: Community Discovery Beyond Vectors},
year = {2026},
howpublished = {\url{https://pith.science/paper/JQ4SBLMT}},
note = {Machine review of arXiv:2507.22955}
}
read the original abstract
Community detection in social network graphs plays a vital role in uncovering group dynamics, influence pathways, and the spread of information. Traditional methods focus primarily on graph structural properties, but recent advancements in Large Language Models (LLMs) open up new avenues for integrating semantic and contextual information into this task. In this paper, we present a detailed investigation into how various LLM-based approaches perform in identifying communities within social graphs. We introduce a two-step framework called CommLLM, which leverages the GPT-4o model along with prompt-based reasoning to fuse language model outputs with graph structure. Evaluations are conducted on six real-world social network datasets, measuring performance using key metrics such as Normalized Mutual Information (NMI), Adjusted Rand Index (ARI), Variation of Information (VOI), and cluster purity. Our findings reveal that LLMs, particularly when guided by graph-aware strategies, can be successfully applied to community detection tasks in small to medium-sized graphs. We observe that the integration of instruction-tuned models and carefully engineered prompts significantly improves the accuracy and coherence of detected communities. These insights not only highlight the potential of LLMs in graph-based research but also underscore the importance of tailoring model interactions to the specific structure of graph data.
Figures
Reference graph
Works this paper leans on
-
[29]
Llama 2: Open foundation and fine-tuned chat models
Hugo Touvron, Louis Martin, Kevin Stone, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023
arXiv 2023
-
[1]
Claude 3.5 sonnet, 2024
work page 2024
-
[2]
t-pine: Tensor-based predictable and interpretable node embeddings
Saba Al-Sayouri, Ekta Gujral, Danai Koutra, Evangelos E Papalexakis, and Sarah S Lam. t-pine: Tensor-based predictable and interpretable node embeddings. Social Network Analysis and Mining, 10:1–11, 2020
work page 2020
-
[3]
t-pne: tensor-based predictable node embeddings
Saba A Al-Sayouri, Ekta Gujral, Danai Koutra, Evangelos E Papalexakis, and Sarah S Lam. t-pne: tensor-based predictable node embeddings. In 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), pages 491–494. IEEE, 2018
work page 2018
-
[4]
Gemini 1.5: Unlocking multimodal understanding across millions of tokens
Rohan Anil et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens. arXiv preprint arXiv:2403.05530, 2024
arXiv 2024
-
[5]
Fast unfold- ing of communities in large networks
Vincent D Blondel, Jean-Loup Guillaume, Renaud Lambiotte, and Etienne Lefebvre. Fast unfold- ing of communities in large networks. Journal of Statistical Mechanics: Theory and Experiment, 2008(10):P10008, 2008
work page 2008
-
[6]
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021
arXiv 2021
-
[7]
A complete survey on llm- based ai chatbots
Sumit Kumar Dam, Choong Seon Hong, Yu Qiao, and Chaoning Zhang. A complete survey on llm- based ai chatbots. arXiv preprint arXiv:2406.16937, 2024
arXiv 2024
Show all 33 references
-
[8]
Mntd: An efficient dynamic community detector based on nonnegative tensor decomposition
Hao Fang, Qu Wang, Qicong Hu, and Hao Wu. Mntd: An efficient dynamic community detector based on nonnegative tensor decomposition. arXiv preprint arXiv:2407.18849, 2024
2024 arXiv
-
[9]
Talk like a graph: Encoding graphs for large language models
Bahare Fatemi, Jonathan Halcrow, and Bryan Perozzi. Talk like a graph: Encoding graphs for large language models. arXiv preprint arXiv:2310.04560, 2023
2023 arXiv
-
[10]
Community structure in social and biological networks
Michelle Girvan and Mark EJ Newman. Community structure in social and biological networks. Proceedings of the National Academy of Sciences, 99(12):7821–7826, 2002
2002
-
[11]
Larc: Learning activity-regularized overlapping communities across time
Alexander Gorovits, Ekta Gujral, Evangelos E Papalexakis, and Petko Bogdanov. Larc: Learning activity-regularized overlapping communities across time. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1465–1474, 2018
2018
-
[12]
Linqs lab, 2024
LINQS Research Group. Linqs lab, 2024
2024
-
[13]
A survey on llm-as-a-judge
Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594, 2024
2024 arXiv
-
[14]
Smacd: Semi-supervised multi-aspect community detection
Ekta Gujral and Evangelos E Papalexakis. Smacd: Semi-supervised multi-aspect community detection. In Proceedings of the 2018 SIAM International Conference on Data Mining, pages 702–710. SIAM, 2018
2018
-
[15]
Hacd: Hierarchical agglomerative community detection in social networks
Ekta Gujral, Evangelos E Papalexakis, Georgios Theocharous, and Anup Rao. Hacd: Hierarchical agglomerative community detection in social networks. In 2019 IEEE 29th International Workshop on Machine Learning for Signal Processing (MLSP), pages 1–6. IEEE, 2019
2019
-
[16]
Beyond rank-1: Discovering rich commu- nity structure in multi-aspect graphs
Ekta Gujral, Ravdeep Pasricha, and Evangelos Papalexakis. Beyond rank-1: Discovering rich commu- nity structure in multi-aspect graphs. In Proceedings of The Web Conference 2020, pages 452–462, 2020
2020
-
[17]
Gpt4graph: Can large language models understand graph structured data? an empirical evaluation and benchmarking
Jiayan Guo, Lun Du, Hengyu Liu, Mengyu Zhou, Xinyi He, and Shi Han. Gpt4graph: Can large language models understand graph structured data? an empirical evaluation and benchmarking. arXiv preprint arXiv:2305.15066, 2023
2023 arXiv
-
[18]
Stochastic blockmodels: First steps
Paul W Holland, Kathryn Blackmond Laskey, and Samuel Leinhardt. Stochastic blockmodels: First steps. Social Networks, 5(2):109–137, 1983
1983
-
[19]
Variational graph auto-encoders
Thomas N Kipf and Max Welling. Variational graph auto-encoders. arXiv preprint arXiv:1611.07308, 2016
2016 arXiv
-
[20]
Modularity and community structure in networks
Mark EJ Newman. Modularity and community structure in networks. Proceedings of the National Academy of Sciences, 103(23):8577–8582, 2006
2006
-
[21]
Comgpt: Detecting local community structure with large language models
Li Ni, Haowen Shen, Lin Mu, Yiwen Zhang, and Wenjian Luo. Comgpt: Detecting local community structure with large language models. arXiv preprint arXiv:2408.06658, 2024
2024
-
[22]
Gpt-3.5 turbo technical overview, 2023
OpenAI. Gpt-3.5 turbo technical overview, 2023. Accessed via OpenAI API
2023
- [23]
-
[24]
Hierarchical block structures and high-resolution model selection in large networks
Tiago P Peixoto. Hierarchical block structures and high-resolution model selection in large networks. Physical Review X, 4(1):011047, 2014
2014
-
[25]
Deepwalk: Online learning of social representations
Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: Online learning of social representations. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 701–710. ACM, 2014
-
[26]
Near linear time algorithm to detect community structures in large-scale networks
Usha Nandini Raghavan, R´ eka Albert, and Soundar Kumara. Near linear time algorithm to detect community structures in large-scale networks. Physical Review E, 76(3):036106, 2007
2007
-
[28]
Identification of overlapping communities via constrained egonet tensor decomposition
Fatemeh Sheikholeslami and Georgios B Giannakis. Identification of overlapping communities via constrained egonet tensor decomposition. IEEE Transactions on Signal Processing, 66(21):5730–5745, 2018
2018
-
[30]
Can language models solve graph problems in natural language?Advances in Neural Information Processing Systems, 36:30840–30861, 2023
Heng Wang, Shangbin Feng, Tianxing He, Zhaoxuan Tan, Xiaochuang Han, and Yulia Tsvetkov. Can language models solve graph problems in natural language?Advances in Neural Information Processing Systems, 36:30840–30861, 2023
2023
-
[31]
Graphtool-instruction: Revolutionizing graph reasoning in llms through decomposed subtask instruction
Rongzheng Wang, Shuang Liang, Qizhi Chen, Jiasheng Zhang, and Ke Qin. Graphtool-instruction: Revolutionizing graph reasoning in llms through decomposed subtask instruction. arXiv preprint arXiv:2412.12152, 2024
2024 arXiv
-
[32]
Llm4graph: Large language models are strong graph reasoners
Xiao Wang, Yu Liu, Qitian Zhang, Meng Liu, Dawei Yin, and Chuxu Zhang. Llm4graph: Large language models are strong graph reasoners. In arXiv preprint arXiv:2306.09382, 2023
2023 arXiv
-
[33]
Plangenllms: A modern survey of llm planning capabilities
Hui Wei, Zihao Zhang, Shenghua He, Tian Xia, Shijia Pan, and Fei Liu. Plangenllms: A modern survey of llm planning capabilities. arXiv preprint arXiv:2502.11221, 2025
2025 arXiv
-
[34]
Chain-of-thought prompting elicits reasoning in large language models
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022. Authors Ekta Gujral is staff d...
2022
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.