{"id":"ddcbf76f-9b61-4d17-8ab8-9fccd766caf7","arxiv_id":"2507.22955","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"CommLLM, a two-step graph-to-text plus LLM prompting method, reports high NMI on six small networks, but its evaluation omits standard community-detection baselines and relies on a prompt tuned on one test set.","lead":"This paper tests whether large language models can find communities in social networks by turning each node's connections into text and asking GPT-4o to assign groups. It reports that its prompt-based method beats other LLMs on small benchmark graphs, but the comparison omits standard community-detection algorithms and has several reproducibility gaps.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Cora/CiteSeer claims hinge on an undocumented partial-output merging procedure; the reader caught this, and it is the single most load-bearing empirical gap.","rationale":"The reader's weakest_assumption already identified the partial-output merging problem as load-bearing, and the paper text confirms it: Table 2 says 'For Cora and CiteSeer model can only predict partial nodes at a time,' but neither §3 nor §4 gives a batching or merging procedure. This is the most concrete correctness risk because the numeric results on two of six datasets cannot be reproduced or even interpreted without that procedure. The baselines inconsistency (gpt-3.5-turbo vs gpt-4-turbo across datasets; 'llama3-3-70b-instruct' vs the cited Llama 2 paper) is a second real weakness, but it is less load-bearing because the headline claim also fails without the Cora/CiteSeer rows. I agree with the reader's REJECT verdict, because the evaluation as presented does not support the claimed results for the two largest datasets, and the merged outputs are never shown or released in a form that would let a reviewer verify the partition. A concrete re-run with an explicit merging protocol would settle whether the claimed numbers are reproducible and meaningful; absent that, the central claim remains unsupported. I would not manufacture additional objections about the absence of classical baselines, since the paper frames the comparison as against LLM baselines and the strongest claim is about LLM methods; however, the paper's own wording 'outperforms the baseline LLM methods' is accurately the scope of its claim, so the reader's decision to reject on the merging issue is sufficient.","tokens_in":9960,"tokens_out":1793,"duration_ms":17700,"concrete_test":"Re-run CommLLM on Cora and CiteSeer with a fixed merging protocol: split nodes into the same token-limited batches, prompt each batch, then independently (a) concatenate raw responses without relabeling and compute metrics, and (b) align community IDs across batches by maximizing pairwise overlap of shared nodes or by using a global community-numbering instruction; report whether the Table 4 NMI/ARI values are reproduced by either protocol, and state the exact merging code used. Also check whether any node appears in zero batches or multiple batches and how duplicates were resolved.","verdict_should_be":"REJECT","load_bearing_attack":"The central empirical claim is that CommLLM outperforms baseline LLMs, with the strongest advantage claimed on Cora and CiteSeer (Table 4). Table 2 explicitly states that for these two graphs 'the model can only predict partial nodes at a time,' yet the paper never describes how the partial outputs are merged into the complete partition used to compute NMI, ARI, VOI, and purity. NMI and ARI require a single labeling over all nodes; if the partial batches assign independently drawn community IDs (e.g., each batch restarts community numbering at 1, or the model uses different labels for the same node in different batches), the reported scores are not well-defined without a documented label-alignment and duplicate-resolution step. The method section (§3) presents only the single-shot prompt for the full graph and gives no batching, overlap, or reconciliation procedure. Because the prompt/response parsing and merging are not specified, the Cora and CiteSeer results are not reproducible, and it is unclear whether the comparison with baselines on those datasets uses the same protocol. Additionally, for the baselines the table uses gpt-3.5-turbo on three datasets but gpt-4-turbo on Terrorist Attacks, Cora, and CiteSeer with no explanation, and the llama3 model is named inconsistently as 'llama3-3-70b-instruct' while the cited reference is Llama 2; this weakens the interpretation of 'outperforming baseline LLM methods.' These are correctness risks in the evaluation protocol, not merely missing ablations. The central claim that CommLLM outperforms baselines can already be checked from Table 4 even without the Cora/CiteSeer rows, but the paper's headline and analysis lean on those rows, so the undocumented merge is load-bearing.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CommLLM, a two-step community detection method that converts a graph's adjacency structure into a text description and prompts GPT-4o to return a node-to-community assignment. The method is evaluated on six real-world datasets (Karate Club, Football, WebKB, Terrorist Attacks, Cora, CiteSeer) against four LLM baselines (gemini-1.5-pro, gpt-3.5-turbo/gpt-4-turbo, llama3-3-70b-instruct, claude-3.5-sonnet) using NMI, ARI, VOI, and purity. The authors report that CommLLM outperforms the LLM baselines on most datasets, present a prompt-comparison study on Football, and discuss scaling behavior in terms of time and token usage. The central claim is that an LLM prompted with serialized graph topology can recover meaningful community structure on small to medium graphs without graph-specific training.","tokens_in":10192,"tokens_out":4853,"duration_ms":54639,"significance":"If the reported results were reliable, the paper would provide a useful empirical data point showing that a zero-shot prompted LLM can perform community detection on small graphs, complementing existing work on LLM graph reasoning. The public availability of the dataset and conversion script, the repeated execution with reported variance, and the explicit prompt-ablation study are positive features that aid reproducibility. However, the significance is currently limited by the small graph sizes, the absence of any classical community detection baseline, and the evaluation-protocol issues detailed below. The head-to-head comparison with other LLM prompting approaches is potentially useful, but the load-bearing Cora and CiteSeer results are not verifiable from the manuscript as written.","major_comments":[{"comment":"The caption of Table 2 states that for Cora and CiteSeer the model can only predict partial nodes at a time, and Table 4 reports NMI, ARI, VOI, and purity for these datasets. The manuscript never describes how these partial predictions are merged into a single complete community assignment. NMI and ARI require one labeling over all nodes, so the reported scores are undefined unless the batching scheme, overlap between batches, community-ID alignment across calls, and duplicate-resolution strategy are fully specified. This missing protocol is load-bearing because the claimed advantage of CommLLM on Cora and CiteSeer rests entirely on these numbers, and the results are not reproducible without it.","section":"§4.4, Table 2 caption"},{"comment":"The text says CommLLM 'achieves competitive scores' and that 'Claude outperforms in NMI and Purity, but CommLLM remains close while offering better balance across all metrics.' In Table 4, however, claude-3.5-sonnet has NMI 0.50 vs. 0.37, ARI 0.36 vs. 0.34, Purity 0.73 vs. 0.56, and VOI 1.12 vs. 1.14 on Terrorist Attacks, meaning Claude is better on every reported metric. This directly contradicts the paper's central claim that CommLLM outperforms the baseline LLM methods, and the discrepancy must be resolved rather than explained away.","section":"§4.4, Terrorist Attacks paragraph vs. Table 4"},{"comment":"The prompt-selection experiment is performed on the Football dataset, and the selected Prompt 4 is then used for all six datasets, including Football, in the final results of Table 4. This is a form of selection on the evaluation set: the Football CommLLM result is not an independent test of the prompt. The authors should either use a separate validation dataset for prompt selection or clearly report the selection procedure as part of the method development and avoid presenting the chosen prompt's performance on Football as an unbiased estimate.","section":"§4.4, Prompt comparisons"},{"comment":"The baselines are inconsistently identified across the method description and the results table. Section 4.2 lists gpt-3.5-turbo as a baseline, but Table 4 uses gpt-4-turbo for Terrorist Attacks, Cora, and CiteSeer, with no explanation for the switch. In addition, the model named 'llama3-3-70b-instruct' is cited to the Llama 2 paper [29], not to any Llama 3 release. These inconsistencies make it unclear whether the same protocol was applied to all models and datasets, and they weaken the comparison that supports the main claim.","section":"§4.2 vs. Table 4"}],"minor_comments":[{"comment":"The text says 'TribeLLM achieves NMI = 0.90' although the method is called CommLLM; this appears to be a typo.","section":"§4.4, Karate Club paragraph"},{"comment":"The word 'avaiable' should be 'available' in the sentence about the public dataset link.","section":"§4.1"},{"comment":"The caption calls the baselines 'state-of-art methods,' but the table contains only LLM-based methods and no classical community detection algorithm from the literature reviewed in Section 2.1. The phrasing is therefore overbroad.","section":"Table 4 caption"},{"comment":"The heading 'F ootball' contains an erroneous space and should be 'Football.'","section":"§4.4, Football heading"},{"comment":"The discussion says Prompt 2 'lacks the actual node connections or network structure to base decisions on,' but all prompts are presumably appended to the same graph details; this claim needs clarification about what exactly differs.","section":"§4.4, Prompt 2 discussion"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is an extended abstract with an interesting empirical question, but the evaluation protocol contains a load-bearing undocumented merging procedure for Cora and CiteSeer, a direct contradiction between text and Table 4 for Terrorist Attacks, and prompt selection on an evaluation dataset. These issues are fixable in principle, but if the authors cannot supply the missing merging details and corrected comparisons, the paper should be rejected. I recommend major revision rather than outright rejection because the central idea is defensible and the identified problems are specific enough to address in a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about arXiv:2507.22955. First, the core idea—serialize adjacency lists, prompt an LLM to assign communities—is not new; ComGPT and the graph-to-text encoding papers already do this. What CommLLM adds is a slightly different prompt and a small empirical study. Second, the evaluation as written does not back the headline claim: for Cora and CiteSeer, the model only predicts partial nodes, and the paper never says how those partial outputs are merged into a complete partition. NMI, ARI, VOI, and purity all need a single labeling over all nodes. Without a documented label-alignment or duplicate-resolution step, those two rows are not interpretable. That is not a minor missing ablation; it is the basis for the paper's strongest claimed advantage on larger graphs.\n\nWhat the paper does reasonably well: it is readable, the graph-to-text conversion is clearly illustrated, the prompt comparison on Karate/Football is a sensible sanity check, and the authors provide a code/data link. On the smallest two datasets, CommLLM clearly beats the other LLM baselines, which suggests the method has some real signal for tiny graphs.\n\nThe soft spots are proportionally serious. Table 4 itself contradicts the headline on Terrorist Attacks: Claude 3.5 Sonnet gets higher NMI and purity than CommLLM. Baseline models are inconsistently named and cited (gpt-3.5-turbo on some datasets, gpt-4-turbo on others, Llama 3.3 with a Llama 2 reference), and no classical community-detection algorithm (Louvain, Infomap, label propagation) is included, so there is no sense of how this compares to standard practice. The prompt was chosen after seeing results on one of the evaluation datasets, which is mild test-set selection. The novelty is modest, and the scalability limits are acknowledged, but the empirical gaps are the real problem.\n\nWho gets value from this paper? Someone surveying how far zero-shot prompting can go on very small graphs might find the numbers suggestive, but no one should rely on the Cora/CiteSeer results until the merging protocol is specified. A serious referee could push the authors to fix the evaluation, add classical baselines, and clarify baselines. I would accept it for peer review with the expectation of major revision, but I would not cite it in its current form.","headline":"Plausible idea, sloppy evaluation: the undocumented partial-node merging for Cora/CiteSeer is a load-bearing gap, and the baselines are too inconsistent to support the headline.","tokens_in":10865,"tokens_out":1677,"would_cite":false,"duration_ms":21732,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CommLLM claims a prompted GPT-4o recovers community structure from neighbor lists alone, beating four LLM baselines on six graphs.","keywords":["community detection","large language models","graph-to-text conversion","prompt engineering","GPT-4o","social network graphs","zero-shot reasoning","adjacency list encoding"],"falsifier":"Run CommLLM on Cora with the same prompt but with nodes presented in three different random orders; if the parsed, merged partition's NMI against ground truth swings beyond the reported ±0.03 or collapses toward zero, the claimed performance depends on serialization order and the merging strategy rather than on graph structure the model recovered.","tokens_in":1540,"feed_emoji":"🕸️","tokens_out":5045,"duration_ms":92469,"temperature":0.7,"pith_summary":"The paper asks whether a generic large language model can solve community detection by reading a graph as plain text, without any graph-specific training or structural assumptions. Its answer is a qualified yes: a two-step method called CommLLM serializes each node's neighbor list into lines like 'Node i is connected to ...', asks GPT-4o to assign every node to a community with a strict output format, and reports higher normalized mutual information and adjusted Rand index than four other LLM baselines on six real social networks. The authors' point is that prompt engineering and instruction tuning are enough to make an LLM act as a community detector for small-to-medium graphs. The result matters because it recasts a classic graph-mining task as a language-reasoning problem and opens the door to injecting semantic cues into structural analysis.","feed_headline":"A prompted LLM beats other LLMs at finding network communities","feed_subtitle":"If it holds, community labels can be inferred by prompting, with no graph-specific training.","key_machinery":"The load-bearing mechanism is graph-to-text conversion combined with an instruction-tuned prompt. Adjacency lists such as 'Node 0 is connected to: 1, 2, 3 ...' preserve local topology as a token sequence, and the prompt defines a community, states the task, demands the exact 'Node:<id>; Community:<id>' format, and forbids extra text. This combination lets GPT-4o produce a parseable community assignment, with the strict formatting constraint doing the work of turning free-form reasoning into evaluable labels.","core_discovery":"CommLLM's central claim is that GPT-4o's reasoning ability, guided by a carefully engineered prompt and an adjacency-list text encoding, recovers meaningful community structure directly from connectivity. The method converts an undirected graph G=(V,E) into text lines of the form 'Node 0 is connected to: 1, 2, 3 ...', then prompts the model with a definition of community, the task instruction, and the requirement to output 'Node:<id>; Community:<id>' with no additional text. Across Karate Club, Football, WebKB, Terrorist Attacks, Cora, and CiteSeer, CommLLM achieves the best NMI on five datasets and best ARI on four, with Table 4 values such as NMI=0.90 on Karate Club and NMI=0.91 on Football; it is also the only method that completes inference on the two largest graphs. The paper frames this as an implicit function f:T(G)->C, where the model infers clusters without optimizing any explicit objective.","pith_inferences":["The paper never compares against cheap classical algorithms such as Louvain, Infomap, or label propagation, so a natural test is whether CommLLM's margins over other LLMs survive when those non-LLM baselines run on the same six graphs.","Adjacency-list order is likely to matter because LLM attention is not permutation-invariant; shuffling node order or neighbor order and measuring NMI variance would reveal how much of the result comes from graph structure rather than serialization luck.","For Cora and CiteSeer, the paper reports partial predictions without describing the merging procedure, leaving open whether community labels are aligned across calls; a concrete extension is to force a global label scheme across batches and check whether the reported NMI and ARI hold.","The paper asserts but does not ablate that its encoding preserves topology; testing edge-list or incident encodings against adjacency lists would show which part of the pipeline is responsible for the gains."],"forward_implications":["LLMs can perform zero-shot community detection on graphs up to a few thousand nodes without any graph-specific training or structural assumptions.","Prompt phrasing is decisive: on Karate Club, CommLLM's full prompt reaches NMI=0.90 while stripped-down variants score 0.55-0.78, showing that definition, task framing, and output constraints materially change results.","The approach is token-bound, with input tokens growing from 777 for Karate Club to roughly 58,000 for Cora and CiteSeer, so larger graphs will require batching, summarization, or a more compact encoding.","On Cora and CiteSeer, rival LLMs either exhaust their token budgets or score below 0.1, while CommLLM completes inference, giving it a practical edge on mid-size networks.","Since no modularity, spectral, or other graph objective is used, the results suggest community structure can be recovered from local connectivity descriptions alone."],"supporting_citations":[{"why":"Supplies the GPT-4o model that CommLLM uses as its reasoning engine.","marker":"[23]"},{"why":"Provides the graph-encoding approach (integer nodes, incident-style edge encoding) that the paper adapts for community detection.","marker":"[9]"},{"why":"The prior LLM-based community detection work that this paper extends by comparing multiple LLMs on social network graphs.","marker":"[21]"},{"why":"Introduces NLGraph and evidence on LLMs' graph-reasoning abilities, motivating the prompt-based route taken here.","marker":"[30]"},{"why":"Chain-of-thought prompting is the reference prompt strategy the authors consider and contrast with their constrained output prompt.","marker":"[34]"},{"why":"Source of the six real-world datasets used for evaluation.","marker":"[12]"},{"why":"Gemini 1.5 is one of the four LLM baselines.","marker":"[4]"},{"why":"GPT-3.5 Turbo is another LLM baseline against which CommLLM is compared.","marker":"[22]"},{"why":"Llama-2 is a baseline that fails on the largest graphs due to token limits, supporting CommLLM's efficiency claim.","marker":"[29]"},{"why":"Claude 3.5 Sonnet is a baseline that outperforms CommLLM only on the Terrorist Attacks dataset.","marker":"[1]"}],"fun_headline_variants":["GPT-4o prompts find network communities without graph training","Prompted LLM detects communities from plain adjacency lists","CommLLM: GPT-4o reasoning beats other LLMs at community detection","LLM prompting alone recovers community structure in graphs","No training needed: GPT-4o prompts for community detection"],"cache_read_input_tokens":12800,"weakest_assumption_plain":"The method assumes that when the model returns community labels for only part of a large graph, those partial outputs can be stitched into a complete, aligned partition whose quality scores still mean what they normally mean.","fun_headline_variants_meta":{"raw":{"variants":["GPT-4o prompts find network communities without graph training","Prompted LLM detects communities from plain adjacency lists","CommLLM: GPT-4o reasoning beats other LLMs at community detection","LLM prompting alone recovers community structure in graphs","No training needed: GPT-4o prompts for community detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000238,"raw_usage":{"total_tokens":1520,"prompt_tokens":965,"completion_tokens":555,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":581,"completion_tokens_details":{"reasoning_tokens":470}},"tokens_in":581,"tokens_out":555,"duration_ms":6126,"temperature":1.0,"reasoning_tokens":470,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T11:55:40.221931+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run CommLLM on Cora with the same prompt but with nodes presented in three different random orders; if the parsed, merged partition's NMI against ground truth swings beyond the reported ±0.03 or collapses toward zero, the claimed performance depends on serialization order and the merging strategy rather than on graph structure the model recovered.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Claude 3.5 Sonnet is a baseline that outperforms CommLLM only on the Terrorist Attacks dataset."},{"cited_title":"Chain-of-thought prompting elicits reasoning in large language models","cited_arxiv_id":null,"evidence_quote":"Chain-of-thought prompting is the reference prompt strategy the authors consider and contrast with their constrained output prompt."},{"cited_title":"Linqs lab, 2024","cited_arxiv_id":null,"evidence_quote":"Source of the six real-world datasets used for evaluation."},{"cited_title":"Gpt-3.5 turbo technical overview, 2023","cited_arxiv_id":null,"evidence_quote":"GPT-3.5 Turbo is another LLM baseline against which CommLLM is compared."}],"review_version":1}