Pith. sign in

REVIEW 4 major objections 5 minor 33 references

LLMs Between the Nodes: Community Discovery Beyond Vectors

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read CommLLM claims a prompted GPT-4o recovers community structure from neighbor lists alone, beating four LLM baselines on six graphs.

desk verdict Plausible idea, sloppy evaluation: the undocumented partial-node merging for Cora/CiteSeer is a load-bearing gap, and the baselines are too inconsistent to support the headline. read the letter →

arxiv 2507.22955 v1 pith:JQ4SBLMT submitted 2025-07-29 cs.SI cs.LG

classification cs.SIcs.LG
keywords communitydetectionlargelanguagemodelsgraph-to-textconversionpromptengineeringGPT-4osocialnetworkgraphszero-shotreasoningadjacencylistencoding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether a generic large language model can solve community detection by reading a graph as plain text, without any graph-specific training or structural assumptions. Its answer is a qualified yes: a two-step method called CommLLM serializes each node's neighbor list into lines like 'Node i is connected to ...', asks GPT-4o to assign every node to a community with a strict output format, and reports higher normalized mutual information and adjusted Rand index than four other LLM baselines on six real social networks. The authors' point is that prompt engineering and instruction tuning are enough to make an LLM act as a community detector for small-to-medium graphs. The result matters because it recasts a classic graph-mining task as a language-reasoning problem and opens the door to injecting semantic cues into structural analysis.

What carries the argument

The load-bearing mechanism is graph-to-text conversion combined with an instruction-tuned prompt. Adjacency lists such as 'Node 0 is connected to: 1, 2, 3 ...' preserve local topology as a token sequence, and the prompt defines a community, states the task, demands the exact 'Node:<id>; Community:<id>' format, and forbids extra text. This combination lets GPT-4o produce a parseable community assignment, with the strict formatting constraint doing the work of turning free-form reasoning into evaluable labels.

What would settle it

Run CommLLM on Cora with the same prompt but with nodes presented in three different random orders; if the parsed, merged partition's NMI against ground truth swings beyond the reported ±0.03 or collapses toward zero, the claimed performance depends on serialization order and the merging strategy rather than on graph structure the model recovered.

Watch

Extended reading notes

Core claim

CommLLM's central claim is that GPT-4o's reasoning ability, guided by a carefully engineered prompt and an adjacency-list text encoding, recovers meaningful community structure directly from connectivity. The method converts an undirected graph G=(V,E) into text lines of the form 'Node 0 is connected to: 1, 2, 3 ...', then prompts the model with a definition of community, the task instruction, and the requirement to output 'Node:<id>; Community:<id>' with no additional text. Across Karate Club, Football, WebKB, Terrorist Attacks, Cora, and CiteSeer, CommLLM achieves the best NMI on five datasets and best ARI on four, with Table 4 values such as NMI=0.90 on Karate Club and NMI=0.91 on Football; it is also the only method that completes inference on the two largest graphs. The paper frames this as an implicit function f:T(G)->C, where the model infers clusters without optimizing any explicit objective.

Load-bearing premise

The method assumes that when the model returns community labels for only part of a large graph, those partial outputs can be stitched into a complete, aligned partition whose quality scores still mean what they normally mean.

Editorial extensions

If this is right

  • LLMs can perform zero-shot community detection on graphs up to a few thousand nodes without any graph-specific training or structural assumptions.
  • Prompt phrasing is decisive: on Karate Club, CommLLM's full prompt reaches NMI=0.90 while stripped-down variants score 0.55-0.78, showing that definition, task framing, and output constraints materially change results.
  • The approach is token-bound, with input tokens growing from 777 for Karate Club to roughly 58,000 for Cora and CiteSeer, so larger graphs will require batching, summarization, or a more compact encoding.
  • On Cora and CiteSeer, rival LLMs either exhaust their token budgets or score below 0.1, while CommLLM completes inference, giving it a practical edge on mid-size networks.
  • Since no modularity, spectral, or other graph objective is used, the results suggest community structure can be recovered from local connectivity descriptions alone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper never compares against cheap classical algorithms such as Louvain, Infomap, or label propagation, so a natural test is whether CommLLM's margins over other LLMs survive when those non-LLM baselines run on the same six graphs.
  • Adjacency-list order is likely to matter because LLM attention is not permutation-invariant; shuffling node order or neighbor order and measuring NMI variance would reveal how much of the result comes from graph structure rather than serialization luck.
  • For Cora and CiteSeer, the paper reports partial predictions without describing the merging procedure, leaving open whether community labels are aligned across calls; a concrete extension is to force a global label scheme across batches and check whether the reported NMI and ARI hold.
  • The paper asserts but does not ablate that its encoding preserves topology; testing edge-list or incident encodings against adjacency lists would show which part of the pipeline is responsible for the gains.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes CommLLM, a two-step community detection method that converts a graph's adjacency structure into a text description and prompts GPT-4o to return a node-to-community assignment. The method is evaluated on six real-world datasets (Karate Club, Football, WebKB, Terrorist Attacks, Cora, CiteSeer) against four LLM baselines (gemini-1.5-pro, gpt-3.5-turbo/gpt-4-turbo, llama3-3-70b-instruct, claude-3.5-sonnet) using NMI, ARI, VOI, and purity. The authors report that CommLLM outperforms the LLM baselines on most datasets, present a prompt-comparison study on Football, and discuss scaling behavior in terms of time and token usage. The central claim is that an LLM prompted with serialized graph topology can recover meaningful community structure on small to medium graphs without graph-specific training.

Significance. If the reported results were reliable, the paper would provide a useful empirical data point showing that a zero-shot prompted LLM can perform community detection on small graphs, complementing existing work on LLM graph reasoning. The public availability of the dataset and conversion script, the repeated execution with reported variance, and the explicit prompt-ablation study are positive features that aid reproducibility. However, the significance is currently limited by the small graph sizes, the absence of any classical community detection baseline, and the evaluation-protocol issues detailed below. The head-to-head comparison with other LLM prompting approaches is potentially useful, but the load-bearing Cora and CiteSeer results are not verifiable from the manuscript as written.

major comments (4)
  1. [§4.4, Table 2 caption] The caption of Table 2 states that for Cora and CiteSeer the model can only predict partial nodes at a time, and Table 4 reports NMI, ARI, VOI, and purity for these datasets. The manuscript never describes how these partial predictions are merged into a single complete community assignment. NMI and ARI require one labeling over all nodes, so the reported scores are undefined unless the batching scheme, overlap between batches, community-ID alignment across calls, and duplicate-resolution strategy are fully specified. This missing protocol is load-bearing because the claimed advantage of CommLLM on Cora and CiteSeer rests entirely on these numbers, and the results are not reproducible without it.
  2. [§4.4, Terrorist Attacks paragraph vs. Table 4] The text says CommLLM 'achieves competitive scores' and that 'Claude outperforms in NMI and Purity, but CommLLM remains close while offering better balance across all metrics.' In Table 4, however, claude-3.5-sonnet has NMI 0.50 vs. 0.37, ARI 0.36 vs. 0.34, Purity 0.73 vs. 0.56, and VOI 1.12 vs. 1.14 on Terrorist Attacks, meaning Claude is better on every reported metric. This directly contradicts the paper's central claim that CommLLM outperforms the baseline LLM methods, and the discrepancy must be resolved rather than explained away.
  3. [§4.4, Prompt comparisons] The prompt-selection experiment is performed on the Football dataset, and the selected Prompt 4 is then used for all six datasets, including Football, in the final results of Table 4. This is a form of selection on the evaluation set: the Football CommLLM result is not an independent test of the prompt. The authors should either use a separate validation dataset for prompt selection or clearly report the selection procedure as part of the method development and avoid presenting the chosen prompt's performance on Football as an unbiased estimate.
  4. [§4.2 vs. Table 4] The baselines are inconsistently identified across the method description and the results table. Section 4.2 lists gpt-3.5-turbo as a baseline, but Table 4 uses gpt-4-turbo for Terrorist Attacks, Cora, and CiteSeer, with no explanation for the switch. In addition, the model named 'llama3-3-70b-instruct' is cited to the Llama 2 paper [29], not to any Llama 3 release. These inconsistencies make it unclear whether the same protocol was applied to all models and datasets, and they weaken the comparison that supports the main claim.
minor comments (5)
  1. [§4.4, Karate Club paragraph] The text says 'TribeLLM achieves NMI = 0.90' although the method is called CommLLM; this appears to be a typo.
  2. [§4.1] The word 'avaiable' should be 'available' in the sentence about the public dataset link.
  3. [Table 4 caption] The caption calls the baselines 'state-of-art methods,' but the table contains only LLM-based methods and no classical community detection algorithm from the literature reviewed in Section 2.1. The phrasing is therefore overbroad.
  4. [§4.4, Football heading] The heading 'F ootball' contains an erroneous space and should be 'Football.'
  5. [§4.4, Prompt 2 discussion] The discussion says Prompt 2 'lacks the actual node connections or network structure to base decisions on,' but all prompts are presumably appended to the same graph details; this claim needs clarification about what exactly differs.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical LLM-prompting study, not a derivation that reduces to its inputs, and its self-citations are background-only.

full rationale

CommLLM makes no formal derivation claim. Its pipeline is graph-to-text serialization followed by a GPT-4o prompt, and its reported numbers are measured against external ground-truth communities and external baselines. No equation in Section 3 defines the output in terms of the evaluation metrics, and no parameter is fitted to the reported NMI/ARI values. The prompt-selection experiment (Table 3) chooses Prompt 4 because it performed best on Football and then reuses Football in Table 4; this is a genuine evaluation-protocol concern (selection bias on one dataset), but it is not circularity because the prompt choice does not by construction determine the reported score. The self-citations (refs. 2, 3, 11, 14, 15, 16) appear only in the literature review and are not load-bearing for the method's validity. The undocumented partial-output merging for Cora/CiteSeer and the inconsistent baseline model names are reproducibility and correctness issues, not instances of a derivation reducing to its own inputs. The central empirical comparison is self-contained against external data, so no circular step is exhibited.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

There is no derivation to audit. What the claim rests on is three hand-made choices: prompt wording, serialization format, and the unstated partial-node merging rule. In addition, the evaluation assumes the LLM output is a complete, parseable partition and that the ground-truth labels are reliable. These choices are reasonable to explore but none are validated independently, so the contribution is an empirical prompt recipe rather than a tested method.

free parameters (3)
  • Prompt template (Prompt 4) = selected as best of four variants in Table 3
    The final prompt combines a community definition, task instruction, output format, and a 'no extra text' constraint; it was chosen by comparing variants on the evaluation datasets and materially affects all reported scores.
  • Graph-to-text serialization format = Node i is connected to: neighbor list
    The adjacency-list encoding is the only structural information the LLM receives; ordering and truncation of neighbor lists are hand-chosen and not validated against alternative encodings.
  • Partial-node batching for Cora and CiteSeer = not specified
    Table 2 notes the model can only predict partial nodes at a time for these datasets; the batch size and merging procedure are unstated, yet they determine the final assignment used for metric computation.
assumptions (5)
  • domain assumption Communities are groups of nodes more densely connected internally than to the rest of the network.
    This is the definition written into the prompt (Section 3.2), and it is the criterion against which ground-truth labels are compared.
  • domain assumption Adjacency-list serialization preserves enough graph topology for the LLM to recover community structure.
    Section 3.1: the entire structural input to GPT-4o is a list of neighbor sets; if this text loses too much topology, the method cannot succeed.
  • standard math NMI, ARI, VOI, and purity are valid when computed between LLM output and ground truth partitions.
    Section 4.3 applies these metrics; they assume each node is assigned exactly one community and that label permutations are handled, which the paper does not verify for partial outputs.
  • ad hoc to paper Partial node predictions from multiple LLM calls can be merged into a single coherent assignment.
    Table 2 caption states that for Cora and CiteSeer the model can only predict partial nodes at a time, but the merge rule is never described; the evaluation depends on this unstated procedure.
  • domain assumption Dataset ground-truth community labels are reliable references.
    Section 4.1 uses six public datasets and treats their community labels as ground truth for all metrics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLMs Between the Nodes: Community Discovery Beyond Vectors." pith.science (2026). https://pith.science/paper/JQ4SBLMT

@misc{pith2026250722955,
  author       = {Pith},
  title        = {Pith review of: LLMs Between the Nodes: Community Discovery Beyond Vectors},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JQ4SBLMT}},
  note         = {Machine review of arXiv:2507.22955}
}
read the original abstract

Community detection in social network graphs plays a vital role in uncovering group dynamics, influence pathways, and the spread of information. Traditional methods focus primarily on graph structural properties, but recent advancements in Large Language Models (LLMs) open up new avenues for integrating semantic and contextual information into this task. In this paper, we present a detailed investigation into how various LLM-based approaches perform in identifying communities within social graphs. We introduce a two-step framework called CommLLM, which leverages the GPT-4o model along with prompt-based reasoning to fuse language model outputs with graph structure. Evaluations are conducted on six real-world social network datasets, measuring performance using key metrics such as Normalized Mutual Information (NMI), Adjusted Rand Index (ARI), Variation of Information (VOI), and cluster purity. Our findings reveal that LLMs, particularly when guided by graph-aware strategies, can be successfully applied to community detection tasks in small to medium-sized graphs. We observe that the integration of instruction-tuned models and carefully engineered prompts significantly improves the accuracy and coherence of detected communities. These insights not only highlight the potential of LLMs in graph-based research but also underscore the importance of tailoring model interactions to the specific structure of graph data.

Figures

Figures reproduced from arXiv: 2507.22955 by the authors.

Figure 1
Figure 1. Illustration of CommLLM that includes two main steps: graph-to-text conversion and then LLM Reasoning to detect the communities in the graph In contrast, our work introduces a novel approach CommLLM (as shown in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Illustration of Karate Club Network The Graph-to-text conversion will be as follow: Node 0 is connected to: 1, 2, 3, 4, 5, 6, 7, 8, . . . . Node 1 is connected to: 0, 2, 3, 7, 13, 17, . . . . . . . Node 33 is connected to: 8, 9, 13, 14, 15, 18, . . . . 3.2 Community Detection via LLM Reasoning We propose a novel community detection method CommLLM that formulates the problem as a text-based reasoning task. The input … view at source ↗
Figure 3
Figure 3. Illustration of community assignment by CommLLM for the graphs 4.4 Experiment Results Results on networks We evaluate CommLLM, our reasoning-based community detec￾tion method, across six benchmark datasets and compare its performance against leading large language models (LLMs). The results are shown in [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

33 extracted references · 20 canonical work pages

  1. [29]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

  2. [1]

    Claude 3.5 sonnet, 2024

  3. [2]

    t-pine: Tensor-based predictable and interpretable node embeddings

    Saba Al-Sayouri, Ekta Gujral, Danai Koutra, Evangelos E Papalexakis, and Sarah S Lam. t-pine: Tensor-based predictable and interpretable node embeddings. Social Network Analysis and Mining, 10:1–11, 2020

  4. [3]

    t-pne: tensor-based predictable node embeddings

    Saba A Al-Sayouri, Ekta Gujral, Danai Koutra, Evangelos E Papalexakis, and Sarah S Lam. t-pne: tensor-based predictable node embeddings. In 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), pages 491–494. IEEE, 2018

  5. [4]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens

    Rohan Anil et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens. arXiv preprint arXiv:2403.05530, 2024

  6. [5]

    Fast unfold- ing of communities in large networks

    Vincent D Blondel, Jean-Loup Guillaume, Renaud Lambiotte, and Etienne Lefebvre. Fast unfold- ing of communities in large networks. Journal of Statistical Mechanics: Theory and Experiment, 2008(10):P10008, 2008

  7. [6]

    Hudson, Ehsan Adeli, et al

    Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021

  8. [7]

    A complete survey on llm- based ai chatbots

    Sumit Kumar Dam, Choong Seon Hong, Yu Qiao, and Chaoning Zhang. A complete survey on llm- based ai chatbots. arXiv preprint arXiv:2406.16937, 2024

Show all 33 references
  1. [8]

    Mntd: An efficient dynamic community detector based on nonnegative tensor decomposition

    Hao Fang, Qu Wang, Qicong Hu, and Hao Wu. Mntd: An efficient dynamic community detector based on nonnegative tensor decomposition. arXiv preprint arXiv:2407.18849, 2024

  2. [9]

    Talk like a graph: Encoding graphs for large language models

    Bahare Fatemi, Jonathan Halcrow, and Bryan Perozzi. Talk like a graph: Encoding graphs for large language models. arXiv preprint arXiv:2310.04560, 2023

  3. [10]

    Community structure in social and biological networks

    Michelle Girvan and Mark EJ Newman. Community structure in social and biological networks. Proceedings of the National Academy of Sciences, 99(12):7821–7826, 2002

  4. [11]

    Larc: Learning activity-regularized overlapping communities across time

    Alexander Gorovits, Ekta Gujral, Evangelos E Papalexakis, and Petko Bogdanov. Larc: Learning activity-regularized overlapping communities across time. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1465–1474, 2018

  5. [12]

    Linqs lab, 2024

    LINQS Research Group. Linqs lab, 2024

  6. [13]

    A survey on llm-as-a-judge

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594, 2024

  7. [14]

    Smacd: Semi-supervised multi-aspect community detection

    Ekta Gujral and Evangelos E Papalexakis. Smacd: Semi-supervised multi-aspect community detection. In Proceedings of the 2018 SIAM International Conference on Data Mining, pages 702–710. SIAM, 2018

  8. [15]

    Hacd: Hierarchical agglomerative community detection in social networks

    Ekta Gujral, Evangelos E Papalexakis, Georgios Theocharous, and Anup Rao. Hacd: Hierarchical agglomerative community detection in social networks. In 2019 IEEE 29th International Workshop on Machine Learning for Signal Processing (MLSP), pages 1–6. IEEE, 2019

  9. [16]

    Beyond rank-1: Discovering rich commu- nity structure in multi-aspect graphs

    Ekta Gujral, Ravdeep Pasricha, and Evangelos Papalexakis. Beyond rank-1: Discovering rich commu- nity structure in multi-aspect graphs. In Proceedings of The Web Conference 2020, pages 452–462, 2020

  10. [17]

    Gpt4graph: Can large language models understand graph structured data? an empirical evaluation and benchmarking

    Jiayan Guo, Lun Du, Hengyu Liu, Mengyu Zhou, Xinyi He, and Shi Han. Gpt4graph: Can large language models understand graph structured data? an empirical evaluation and benchmarking. arXiv preprint arXiv:2305.15066, 2023

  11. [18]

    Stochastic blockmodels: First steps

    Paul W Holland, Kathryn Blackmond Laskey, and Samuel Leinhardt. Stochastic blockmodels: First steps. Social Networks, 5(2):109–137, 1983

  12. [19]

    Variational graph auto-encoders

    Thomas N Kipf and Max Welling. Variational graph auto-encoders. arXiv preprint arXiv:1611.07308, 2016

  13. [20]

    Modularity and community structure in networks

    Mark EJ Newman. Modularity and community structure in networks. Proceedings of the National Academy of Sciences, 103(23):8577–8582, 2006

  14. [21]

    Comgpt: Detecting local community structure with large language models

    Li Ni, Haowen Shen, Lin Mu, Yiwen Zhang, and Wenjian Luo. Comgpt: Detecting local community structure with large language models. arXiv preprint arXiv:2408.06658, 2024

  15. [22]

    Gpt-3.5 turbo technical overview, 2023

    OpenAI. Gpt-3.5 turbo technical overview, 2023. Accessed via OpenAI API

  16. [23]

    Gpt-4o system card

    OpenAI. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

  17. [24]

    Hierarchical block structures and high-resolution model selection in large networks

    Tiago P Peixoto. Hierarchical block structures and high-resolution model selection in large networks. Physical Review X, 4(1):011047, 2014

  18. [25]

    Deepwalk: Online learning of social representations

    Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: Online learning of social representations. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 701–710. ACM, 2014

  19. [26]

    Near linear time algorithm to detect community structures in large-scale networks

    Usha Nandini Raghavan, R´ eka Albert, and Soundar Kumara. Near linear time algorithm to detect community structures in large-scale networks. Physical Review E, 76(3):036106, 2007

  20. [28]

    Identification of overlapping communities via constrained egonet tensor decomposition

    Fatemeh Sheikholeslami and Georgios B Giannakis. Identification of overlapping communities via constrained egonet tensor decomposition. IEEE Transactions on Signal Processing, 66(21):5730–5745, 2018

  21. [30]

    Can language models solve graph problems in natural language?Advances in Neural Information Processing Systems, 36:30840–30861, 2023

    Heng Wang, Shangbin Feng, Tianxing He, Zhaoxuan Tan, Xiaochuang Han, and Yulia Tsvetkov. Can language models solve graph problems in natural language?Advances in Neural Information Processing Systems, 36:30840–30861, 2023

  22. [31]

    Graphtool-instruction: Revolutionizing graph reasoning in llms through decomposed subtask instruction

    Rongzheng Wang, Shuang Liang, Qizhi Chen, Jiasheng Zhang, and Ke Qin. Graphtool-instruction: Revolutionizing graph reasoning in llms through decomposed subtask instruction. arXiv preprint arXiv:2412.12152, 2024

  23. [32]

    Llm4graph: Large language models are strong graph reasoners

    Xiao Wang, Yu Liu, Qitian Zhang, Meng Liu, Dawei Yin, and Chuxu Zhang. Llm4graph: Large language models are strong graph reasoners. In arXiv preprint arXiv:2306.09382, 2023

  24. [33]

    Plangenllms: A modern survey of llm planning capabilities

    Hui Wei, Zihao Zhang, Shenghua He, Tian Xia, Shijia Pan, and Fei Liu. Plangenllms: A modern survey of llm planning capabilities. arXiv preprint arXiv:2502.11221, 2025

  25. [34]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022. Authors Ekta Gujral is staff d...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.