{"id":"6b7a0971-5e6d-4223-b0bc-ceaeb94fdd95","arxiv_id":"2501.15755","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A 55-template prompt benchmark showing general LLMs can beat specialized graph LLMs and GNNs on node classification and link prediction, though the reported margins are inflated by test-set prompt selection.","lead":"This paper introduces GraphICL, a set of 55 prompt templates that convert text-attributed graphs into structured text, including neighbor information and labeled examples, for large language models. The authors report that prompt-only LLMs outperform specialized graph LLMs and GNNs on node classification and link prediction, especially with limited labels and across domains.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cross-domain outperformance is confounded: GraphICL's Table 2 prompts include target training labels (Table 8), while graph-LLM baselines have none; the out-of-domain claim is unsupported.","rationale":"The reader's weakest assumption correctly identifies the decisive flaw: Table 2's cross-domain protocol gives GraphICL labeled demonstrations from the target training split while the specialized graph LLMs have no target labels. This is confirmed by the paper's own notation: Table 8 lists the S1/S2 settings used for both Table 1 and Table 2, and A.2 defines 'G' (global) and 'C' (class-aware) demonstration selection as drawing from the training set. Therefore the comparison in Table 2 is not a test of out-of-domain capability but of few-shot in-context learning on the target distribution, an information advantage that fully explains the large relative gains. The best-of-55 test-set template selection is a second, independent validity threat, but the leakage alone is sufficient to reject the cross-domain claim as stated. The in-domain semi-supervised results, where target labels are legitimately available to both GraphICL and the baselines, are not invalidated by this concern, but the headline claim in the abstract explicitly includes out-of-domain tasks, so the paper's central conclusion is unsupported. A source-only demonstration rerun would settle the question; if the advantage persists under that protocol, the out-of-domain claim would be restored. Given no code or error bars are provided, the appropriate verdict remains REJECT, matching the reader.","tokens_in":20185,"tokens_out":4522,"duration_ms":39459,"concrete_test":"Rerun Table 2 with GraphICL prompts built exclusively from labeled demonstrations (and neighbor text) sampled from the training splits of the source datasets only, never from the target dataset; for each target dataset, select the template on a source-domain validation set rather than on target test labels. If GraphICL no longer surpasses the graph-LLM baselines (or drops substantially), the cross-domain advantage is an artifact of target-label leakage. Also report all 55 template scores on the validation set to rule out test-set template selection as an alternative explanation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that GraphICL outperforms specialized graph LLMs in out-of-domain tasks rests on Table 2. Appendix A.3.4 and Table 8 state that the GraphICL settings used for Table 2 are the same S1/S2 configurations as Table 1. These settings include demonstrations selected from the training split of each target dataset, e.g., '1RGR' (global random demonstrations) and '1RCP' (class-aware PageRank demonstrations) on Cora and Arxiv. Class-aware and global demos are explicitly defined in A.2 as drawn from the training set. Thus, for every cross-domain test, GraphICL receives in-distribution labeled examples from the target dataset. The graph LLM baselines (LLaGA, GraphPrompter, GraphTranslator, GraphGPT) were trained only on other datasets and have zero access to target labels. The comparison is therefore not out-of-domain for GraphICL: it is few-shot in-context learning on the target distribution versus zero-shot transfer for the baselines. The reported gains, e.g., +487% on Computers and +2459% on Sports, are consistent with label leakage rather than graph reasoning ability. Unless GraphICL is restricted to source-domain demonstrations only, the out-of-domain superiority claim in the abstract and Observation 2 is invalid.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces GraphICL, a set of 55 prompt templates for using frozen LLMs on text-attributed graph tasks, with components for task description, anchor-node text, structure-aware neighbor information, and few-shot demonstrations. The authors evaluate LLaMA2-13B-Chat, LLaMA3-70B-Instruct, and GPT-4o on node classification and link prediction across 9 datasets, comparing against GNNs, specialized graph LLMs, and zero-shot LLMs, and claim that prompt design alone lets general-purpose LLMs outperform specialized graph LLMs in both in-domain and out-of-domain settings. The paper also reports ablations of neighbor-selection strategy, demonstration-selection strategy, structural depth, and chain-of-thought prompting.","tokens_in":20392,"tokens_out":7558,"duration_ms":71209,"significance":"If the claims held, the paper would be an important result: it would show that a carefully designed set of prompts can close or reverse the gap between frozen general-purpose LLMs and specialized graph LLMs, with implications for how graph LLM research is evaluated. The taxonomy of prompt components, the 55-template design space, and the ablations are potentially useful resources for future benchmarking. However, the central comparison is currently undermined by two evaluation-protocol problems: the reported GraphICL numbers appear to be per-dataset optima over 55 template choices, and the cross-domain experiments allow GraphICL to use labeled demonstrations from the target dataset while the graph-LLM baselines have no such access. These issues mean the headline claims of in-domain and out-of-domain superiority are not established as written. The empirical infrastructure could still be valuable after a corrected evaluation with validation-based prompt selection and symmetric information access.","major_comments":[{"comment":"The reported GraphICL results for each dataset are the first- and second-best of 55 prompt templates, but no validation-based selection procedure is described. The caption of Table 1 states that S1 and S2 denote the first- and second-best GraphICL prompts, and Appendix A.3.4 refers to \"the best results from the 55 configurations tested.\" If the best configuration was chosen using test-set accuracy, then the accuracies in Tables 1, 2, 10, 11, and 12 are post-hoc optima and do not represent a single well-defined prompting method. Comparisons against fixed graph-LLM baselines are inflated by this selection. Please either select prompts on a validation split, fix one template per task in advance, or report the full distribution of accuracies across all 55 configurations, and adjust the conclusions accordingly.","section":"Section 5.2.1, Appendix A.2, Table 8, Figure 3"},{"comment":"The cross-domain comparison is not out-of-domain for GraphICL. Table 8 shows that the settings used for Table 2 are the same S1/S2 settings as in the in-domain experiments, and Appendix A.2 defines \"G\" and \"C\" demonstrations as selected from the training set. Thus, for example, the \"1RCP\" setting on Computers and Cora uses class-aware PageRank demonstrations drawn from the target dataset's training split. The graph-LLM baselines, by contrast, were trained only on other datasets and have no access to target labels. GraphICL therefore receives in-distribution labeled examples that the baselines do not, so the large relative gains in Table 2 (e.g., +487% on Computers and +2459% on Sports) are consistent with label-information asymmetry rather than with graph reasoning ability. The cross-domain experiment must be rerun with demonstrations restricted to source-domain data, or the baselines must be given the same few-shot target examples, before any out-of-domain superiority claim can be made.","section":"Section 5.2.2, Table 2, Appendix A.3.4, Table 8"},{"comment":"The central claim that GraphICL outperforms specialized graph LLMs in resource-constrained settings and out-of-domain tasks rests on the two evaluation issues above. In particular, Observation 2 and the abstract should not state that GraphICL enables LLaMA to outperform specialized graph LLMs without training until the test-set prompt selection and the target-label demonstration advantage are addressed. The relative improvement percentages over near-zero baselines (e.g., +2459.94% on Sports) are also misleading; please report absolute accuracy differences alongside any relative percentages.","section":"Abstract and Section 5.2.2, Observation 2"}],"minor_comments":[{"comment":"There is a typo in \"Ad dtional details\" in the opening of Section 5.1; it should read \"Additional details.\"","section":"Section 5.1"},{"comment":"The notation S1/S2 and template codes such as \"1RCP\" are used in the main text but defined only in the appendix; please define or briefly explain the notation in the main text for readability.","section":"Table 1 and Appendix A.2"},{"comment":"Relative improvements over very small baselines are hard to interpret; please also report absolute accuracy gaps, especially in the cross-domain tables.","section":"Tables 1, 2, and 10-12"},{"comment":"The paper alternates between the terms \"GICL\" and \"GraphICL\" without a clear distinction; please make the terminology consistent throughout.","section":"Problem Statement and Section 4"},{"comment":"For reproducibility of a benchmark paper, please state whether code and configuration files for all 55 templates, prompt texts, and sampling procedures will be released.","section":"Appendix A.3"},{"comment":"The OGB-Products system prompt in Table 13 says \"(Categories omitted due to length)\"; the actual category list is needed for reproducibility and should be provided in full or in a supplementary file.","section":"Table 13"}],"recommendation":"major_revision","confidential_remarks":"The two principal concerns are the test-set selection of the best prompt configuration and the information asymmetry in the cross-domain comparison. Both are load-bearing for the paper's central claims, but both are in principle fixable by re-running experiments with a validation-based selection rule and with source-domain-only demonstrations or equivalent few-shot access for baselines. I therefore recommend major revision rather than rejection, but the paper should not be accepted until the corrected evaluation is presented and the claims are re-derived from it."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe headline: GraphICL is a useful prompt-suite artifact, but the paper's central claim—that prompt-only LLMs beat specialized graph LLMs and GNNs in semi-supervised and cross-domain settings—is not supported by the experiments as run. The cross-domain comparison is confounded, and the headline numbers are selected from 55 configurations on the test set.\n\nWhat's new and good: They build a 55-template prompt space combining anchor text, structure-aware neighbors (1/2-hop, random/PageRank/similarity), and demonstrations (global/class-aware, random/PageRank/similarity). They evaluate across 9 TAG datasets with multiple LLMs and run ablations on structural info, CoT, and selection strategies. They are transparent that Huang et al. 2023 and Li et al. 2024a prompts are subsets of their templates. As a benchmark artifact, this is a contribution.\n\nSoft spots: First, the per-dataset best (S1/S2) is chosen based on test accuracy across the 55 templates (Table 8). That's post-hoc selection on the test labels; the reported numbers are maxima over a large configuration space, not predictions from a fixed protocol. Second, the cross-domain results in Table 2 use the same settings as Table 1, which include demonstrations drawn from the target dataset's training split (e.g., 1RGR, 1RCP). So GraphICL gets in-distribution labeled examples in its prompt while the graph LLM baselines are trained only on other datasets and see no target labels. The comparison is few-shot ICL on the target distribution versus zero-shot transfer, not out-of-domain vs out-of-domain. The huge gains on Computers and Sports are consistent with that leakage. Third, no error bars, no released code, and no direct comparison against the prompt-only methods they review (Huang et al., Li et al.), which would be the natural baselines for a prompt benchmark.\n\nThe in-domain semi-supervised results are less affected by the leakage issue, but still suffer from the test-set selection. The paper's core contribution is the prompt suite and the observation that LLMs can be competitive with proper prompts; that is plausible and worth testing rigorously. As submitted, the load-bearing comparisons are invalid.\n\nRecommendation: I'd send this to peer review with the clear expectation of heavy revision. The artifact deserves referee time, and the evaluation protocol is fixable (validation-based template selection, no target-label demos in cross-domain, add error bars and code). A serious reviewer should not desk-reject it, but should push hard on the experimental protocol.","headline":"Useful prompt-suite artifact undermined by post-hoc test-set selection and target-label leakage in the cross-domain comparison; worth refereeing but the empirical claims need a major rework.","tokens_in":20997,"tokens_out":3515,"would_cite":false,"duration_ms":30384,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Structured prompt templates let frozen LLMs beat trained graph models in low-label and cross-domain settings.","keywords":["graph in-context learning","text-attributed graphs","prompt engineering","node classification","link prediction","large language models","semi-supervised learning","graph reasoning"],"falsifier":"Run the cross-domain comparison again with GraphICL demonstrations restricted to the source or mixed training sets only, with no target-distribution labels in the prompt. If accuracy then falls to or below the specialized graph LLMs, the reported out-of-domain advantage is an artifact of in-prompt label leakage rather than graph reasoning.","tokens_in":19916,"feed_emoji":"🕸️","tokens_out":4756,"duration_ms":40551,"temperature":0.7,"pith_summary":"The paper's central claim is that a general-purpose LLM, given the right prompt template, can do graph reasoning tasks on text-attributed graphs as well as or better than specialized graph LLMs and graph neural networks that have been trained for those tasks. The authors build a benchmark of 55 prompt templates assembled from four components: task description, anchor node text, structure-aware neighbor information, and labeled demonstrations. Across nine datasets, the best GraphICL prompts improve accuracy by roughly 20 percent relative over specialized baselines in semi-supervised node classification and also win in out-of-domain tests without any weight updates. The intended message is a warning and a baseline: prior comparisons that pitted trained graph LLMs against naive zero-shot LLM prompts understated what frozen LLMs can do. If the claim holds, prompt design is a serious alternative to instruction tuning for graph learning under label scarcity.","feed_headline":"Prompts alone let frozen LLMs beat specialized graph models","feed_subtitle":"One prompt benchmark lifts LLM accuracy about 20 percent over trained graph models in label-scarce tests.","key_machinery":"The central object is the GraphICL prompt template, which combines four components: task description, anchor node text, structure-aware neighbor information, and labeled demonstrations. Structure-aware information simulates message passing at the text level by adding 1-hop or 2-hop neighbor texts, selected randomly, by cosine similarity, or by PageRank. Demonstrations are chosen globally or class-aware using the same three selection strategies, and the paper evaluates all 55 combinations. The mechanism is that a frozen LLM performs in-context inference from these textual signals alone; no gradient updates are applied.","core_discovery":"GraphICL establishes that in-context prompting can release graph reasoning from frozen LLMs. The paper reports that LLaMA3-70B-Instruct with GraphICL reaches 87.37 percent accuracy on Amazon-Computers, 91.39 percent on Amazon-Sports, 93.18 percent on PubMed, 83.58 percent on Cora, 73.68 percent on OGB-Arxiv, and 81.48 percent on OGB-Products in semi-supervised node classification, beating trained GNNs and specialized graph LLMs. In link prediction on Cora it reports 88.08 percent, above the specialized LLaGA-HO's 86.82 percent in the supervised setup. The authors interpret these numbers as evidence that the benefit of specialized graph LLMs has been overstated because their baseline comparisons used simplistic prompts; with structured prompts, general LLMs become the stronger option.","pith_inferences":["If the cross-domain gains come mainly from in-prompt demonstrations sampled from the target dataset's own training labels, the reported out-of-domain superiority is closer to in-context label propagation than to transferable graph reasoning; re-running with only source-domain demonstrations would separate these explanations.","The 55-template sweep treats prompt design as a searchable hyperparameter space, so an automated search over neighbor and demonstration selection strategies could extend the benchmark further.","Similarity-based neighbor selection is likely exploiting label homophily, which suggests a testable prediction: GraphICL should degrade on heterophilous graphs where neighbors frequently carry different labels.","The strong semi-supervised performance hints that frozen LLMs may be effective as data-efficient annotators or priors for graph models, a use the paper leaves implicit."],"forward_implications":["Specialized graph LLMs should be re-evaluated against structured prompts rather than naive zero-shot prompts before their gains are attributed to training.","In label-scarce settings, prompt engineering can replace training: GraphICL's semi-supervised results match or beat supervised trained models on several datasets.","Cross-domain deployment can be handled by a frozen LLM plus in-prompt labeled demonstrations, avoiding retraining when the target graph changes.","Similarity-based neighbor selection and class-aware demonstrations are the most consistently useful prompt choices, while chain-of-thought prompting is unreliable and often neutral or harmful.","The link prediction results indicate the template transfers beyond node classification to edge-level reasoning tasks."],"supporting_citations":[{"why":"Supplies the in-context learning premise that demonstrations let frozen LLMs adapt to new tasks.","marker":"[Brown et al., 2020]"},{"why":"Defines the GCN baseline and the semi-supervised node classification setting GraphICL is compared against.","marker":"[Kipf and Welling, 2016a]"},{"why":"Provides the inductive GNN baseline (GraphSAGE) used in the node classification and link prediction tables.","marker":"[Hamilton et al., 2017]"},{"why":"GraphGPT is the instruction-tuned specialized graph LLM baseline that GraphICL outperforms in cross-domain tests.","marker":"[Tang et al., 2023]"},{"why":"LLaGA is the specialized graph LLM baseline whose trained mixed-set strategy is contrasted with GraphICL's no-training prompting.","marker":"[Chen et al., 2024b]"},{"why":"GraphPrompter is the strongest semi-supervised specialized baseline; the reported 39.88 percent Computers gain is computed against it.","marker":"[Liu et al., 2024b]"},{"why":"GraphTranslator supplies the cross-domain graph LLM baseline trained on mixed source sets.","marker":"[Zhang et al., 2024]"},{"why":"Provides OGB-Arxiv and OGB-Products datasets and their standard splits used in the evaluation.","marker":"[Hu et al., 2020]"}],"fun_headline_variants":["Frozen LLMs beat specialized graph models with better prompts","Prompt engineering unlocks graph learning in LLMs without training","Structured prompts lift LLMs above tuned graph networks","GraphICL: Prompt-only method outperforms trained graph models","No training needed: LLMs outdo graph specialists via prompts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"In the cross-domain experiments, the strongest GraphICL prompts draw their few-shot demonstrations from the target dataset's own training labels, while the specialized graph LLMs must work without any target labels; the claimed out-of-domain advantage depends on treating that label access as fair rather than as leakage.","fun_headline_variants_meta":{"raw":{"variants":["Frozen LLMs beat specialized graph models with better prompts","Prompt engineering unlocks graph learning in LLMs without training","Structured prompts lift LLMs above tuned graph networks","GraphICL: Prompt-only method outperforms trained graph models","No training needed: LLMs outdo graph specialists via prompts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00056,"raw_usage":{"total_tokens":2665,"prompt_tokens":954,"completion_tokens":1711,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":570,"completion_tokens_details":{"reasoning_tokens":1630}},"tokens_in":570,"tokens_out":1711,"duration_ms":9868,"temperature":1.0,"reasoning_tokens":1630,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T13:57:37.244393+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the cross-domain comparison again with GraphICL demonstrations restricted to the source or mixed training sets only, with no target-distribution labels in the prompt. If accuracy then falls to or below the specialized graph LLMs, the reported out-of-domain advantage is an artifact of in-prompt label leakage rather than graph reasoning.","supporting_citations":[],"review_version":1}