{"id":"41cbe3ed-72eb-4118-bb49-47f5680bcf0d","arxiv_id":"2505.02027","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"GraphPrompter improves few-shot graph in-context learning by reconstructing prompt subgraphs, selecting prompts with kNN and learned importance, and adding cached pseudo-labeled test samples.","lead":"GraphPrompter is a multi-stage prompt optimization pipeline for graph in-context learning: it re-weights sampled subgraph edges, selects few-shot prompts by k-nearest-neighbor retrieval combined with a learned importance score, and augments the prompt set with cached test predictions. The paper reports average accuracy gains over Prodigy and other baselines on citation and knowledge-graph benchmarks without updating model parameters at inference.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Paper's own Table III contradicts 'consistently surpasses all baselines' and the claimed >8% average: at arXiv 40-way, ProG scores 47.50 vs GraphPrompter 29.47, and the 48%/54.43% numbers are not reproducible from the table.","rationale":"The reader's formal weakest_assumption is the transferability of pretrained embeddings for cosine retrieval in Eq. 6, which is a legitimate mechanistic concern. However, the more immediately load-bearing problem is that the paper's own tables contradict the headline claims. Table III shows ProG beating GraphPrompter by a large margin at arXiv 40-way, while the text says GraphPrompter 'consistently surpasses all other baselines.' The abstract's 'over 8%' claim is computed, if at all, against Prodigy rather than the per-setting best baseline, and the '48%' and '54.43%' statements in Section V-B1 are numerically opaque. These issues do not prove the method is ineffective, and a corrected comparison could in principle preserve a positive average gain, so the appropriate verdict remains CONDITIONAL pending reconciliation of the reported numbers rather than outright rejection. I therefore leave the reader's verdict unchanged, while noting that the stated central claim is not currently supported by the evidence as printed.","tokens_in":20208,"tokens_out":15121,"duration_ms":146229,"concrete_test":"Recompute the headline from Tables III–VI: for each row, identify the best baseline accuracy (max over NoPretrain, Contrastive, Finetune, Prodigy, ProG, and OFA where reported), compute GraphPrompter's absolute and relative gain over that per-setting best, and average over all settings using the paper's claimed weighting. Also derive the '48%' and '54.43%' numbers from Table III explicitly. If the arXiv-40way row (ProG 47.50 vs GraphPrompter 29.47) is included and the per-setting best baseline is used, the 'consistently surpasses all baselines' claim is false; if the corrected average falls below the claimed 8% or the 48% figure cannot be derived, the quantitative central claim is unsupported as written.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that GraphPrompter beats state-of-the-art baselines, on average by over 8%, and 'consistently surpasses all other baselines.' The load-bearing condition is that the reported aggregate is computed against the correct best baseline, and that the stated margins can be derived from the tables. That condition fails in the paper's own Table III. In the 40-way arXiv setting, ProG, one of the listed baselines, achieves 47.50±6.67 while GraphPrompter achieves 29.47±3.82, an 18-point gap. Section V-B1 nevertheless claims GraphPrompter 'consistently surpasses all other baselines,' and the abstract claims an average >8% over state-of-the-art baselines. The same section's 'improving by 48% on average and up to 54.43%' is not traceable: 54.43% is GraphPrompter's mean accuracy on arXiv, not an improvement, and no straightforward relative or absolute computation from Table III yields 48%. If the headline average is recomputed over the per-setting best baseline, including ProG and OFA, the arXiv block alone contributes a large negative margin at 40-way, which materially changes the reported average. This is an internal inconsistency in the evidence supporting the central claim, not a disagreement with external consensus.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"GraphPrompter proposes a multi-stage prompt optimization pipeline for graph in-context learning, consisting of a Prompt Generator that reweights edges via a reconstruction layer, a Prompt Selector that combines pre-trained selection layers with kNN retrieval, and a Prompt Augmenter that maintains an LFU cache of pseudo-labeled test samples to expand the prompt set at inference. The method is evaluated on arXiv node classification and ConceptNet, FB15K-237, and NELL edge classification under 3-shot cross-domain settings against NoPretrain, Contrastive, Finetune, Prodigy, ProG, and OFA. The paper claims an average improvement of over 8% over state-of-the-art baselines and consistent superiority across settings, and it releases code at a public repository.","tokens_in":1675,"tokens_out":2806,"duration_ms":79317,"significance":"Prompt optimization is a relevant and comparatively underexplored direction for graph in-context learning, and the three proposed components are intuitive and well-motivated. The paper ships code, includes ablations of each component, analyzes cache size, number of shots, multi-hop subgraphs, and inference time, and reports several large gains, such as NELL 40-way from 60.02 to 75.74. If the aggregate claims were corrected and the online augmentation protocol were cleared of circularity, this would be a useful empirical contribution to graph in-context learning. However, the headline quantitative claims are not supported by the paper's own tables, and one load-bearing component needs a precise description of its data flow.","major_comments":[{"comment":"In Section V-B1, the text states that 'our approach consistently surpasses all other baselines' and that on arXiv it improves 'by 48% on average and up to 54.43% over the best-performing Prodigy baseline.' Table III contradicts these statements: at 40-way arXiv, ProG reaches 47.50 plus or minus 6.67 while GraphPrompter reaches 29.47 plus or minus 3.82, an 18-point deficit, and no straightforward computation from the table yields a 48% average gain or a 54.43% improvement. The value 54.43 is instead approximately GraphPrompter's mean accuracy across the five arXiv settings. A similar unreproducible number appears in the same section, 'ProG is average 48.24%', which does not match Table V. The abstract's 'over 8%' claim therefore lacks a clear and reproducible definition, and the claim of consistent superiority is false as stated. Please recompute all aggregates against the best baseline in each setting, including ProG and OFA, report per-setting margins, and either remove or qualify the 'consistently surpasses' and '>8%' statements.","section":"V-B1, Tables III-V"},{"comment":"The Prompt Augmenter stores 'test samples with predicted labels (pseudo-labels)' in a cache, and the inference algorithm updates the cache before predicting the label in Algorithm 2, lines 9 through 13. If the current query's own pseudo-label is inserted before that query is predicted, the augmenter uses the model's own output as part of the prompt for that same query, which is circular and would inflate the measured benefit of the module. Please state explicitly whether the cache contains only samples from earlier inference steps, with the current query excluded, and adjust Algorithm 2 so the update occurs strictly after prediction. If pseudo-labels from earlier steps are used, report the exact online protocol and verify in the ablation that the gain is not an artifact of this ordering.","section":"IV-C and Algorithm 2"},{"comment":"The kNN retrieval criterion assumes that subgraph embeddings produced by the pre-trained GNN are comparable between the pretraining graph and the downstream graphs, so that cosine similarity ranks true task relevance across domains. The paper itself emphasizes the pretraining/testing gap in Section I, yet no experiment validates this transferability, for example through retrieval precision on downstream graphs, a comparison against random retrieval, or a sensitivity analysis of the embedding space. Because both the Prompt Selector and the cache-hit mechanism in the Prompt Augmenter rely on this assumption, please add such an analysis or explicitly discuss the conditions under which Eq. (6) is expected to hold.","section":"IV-B.2, Eq. (6)"},{"comment":"The ablation study is reported only graphically, without numeric values or error bars, and the text's characterization, 'the w/o kNN model only performs 1% better than the baseline', is not tied to a table. Given that the kNN and selection-layer contributions are central to the claimed advantage over Prodigy, please provide the full ablation numbers for each component on each dataset and way count, together with an appropriate significance assessment, such as paired tests over the 500-sample evaluation.","section":"V-C, Figure 3"}],"minor_comments":[{"comment":"The spelling is inconsistent: 'Prompt Augmenter' in the abstract and Section IV-C alternates with 'Prompt Augmentor' in Section V-C and Algorithm 2, and 'ConcepNet' appears in Table IV and Section V-A while 'ConceptNet' is used elsewhere.","section":"Throughout"},{"comment":"Equation (7) uses I_q, the importance of the query, but I_q is never defined in Section IV-B; please define it or revise the score formula.","section":"IV-B.1, Eq. (7)"},{"comment":"The top-k set T(q) is defined per query, but the final prompt set S-hat is described as a single global set; clarify whether all queries share the same selected prompts or each query receives its own prompt set, since this affects the construction of the task graph in Algorithm 2.","section":"IV-B.2, Eq. (8)"},{"comment":"The OFA results are taken from external claims rather than from runs under the exact protocol described here; please report the OFA checkpoint, hyperparameters, and dataset splits used, or run OFA under the same random category selection procedure.","section":"V-A.3 and Table VI"},{"comment":"The comparison with GAT as a prompt generator lacks configuration details and numeric values; please report the GAT architecture, number of layers, and the exact accuracy numbers behind the figure.","section":"V-D.2, Figure 4"},{"comment":"The paper says GraphPrompter outperforms Finetune by 19.6% on arXiv and over 15% on KGs, but no calculation path is given; please specify whether these are absolute or relative improvements and define the aggregation over ways and datasets.","section":"V-B1"}],"recommendation":"major_revision","confidential_remarks":"The paper's core method is plausible and the experimental coverage is broad, but the headline quantitative claims are internally inconsistent with Table III, and the Prompt Augmenter's inference ordering creates a circularity risk that must be resolved. The novelty statement 'first research to explore the concept of prompt optimization on graph models' is broader than the related work supports and should be tempered. If the authors correct the aggregate statistics, clarify the cache protocol, and add the requested validation of the embedding-space assumption, the paper could become suitable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"GraphPrompter is a sensible three-stage prompt-optimization pipeline for graph in-context learning, and the combination — edge-weight reconstruction, kNN-plus-learned prompt selection, and LFU cache augmentation — is genuinely new relative to Prodigy and OFA. But the paper's central claim does not survive contact with its own Table III. At arXiv 40-way, ProG scores 47.50 ± 6.67 and GraphPrompter 29.47 ± 3.82, an 18-point deficit. The abstract's \"over 8%\" average and Section V-B1's \"48% average improvement / up to 54.43%\" are not derivable from the reported numbers. That is a load-bearing internal inconsistency, not a cosmetic issue.\n\nWhat the paper does well: the framework is modular, the ablations in Figure 3 suggest each component contributes, and combining kNN retrieval with a learned selection layer is a reasonable adaptation of retrieval-augmented ICL to graphs. The reconstruction layer is a simple structure-learning trick but applied cleanly. Pretraining at the scale of MAG240M and Wiki is non-trivial. The code link exists, though without a hash or run protocol.\n\nSoft spots beyond the numerical contradiction: the cache size in Section V-D1 is tuned on test accuracy, which leaks target-domain information into a few-shot evaluation. The Prompt Augmenter also feeds the model's own high-confidence predictions back into the prompt set — that is test-time adaptation/pseudo-labeling, not purely in-context learning, and the ablation doesn't separate the cache's contribution from the effect of seeing test data. Several figures lack error bars. The \"first to explore prompt optimization on graph models\" claim needs a broader literature search.\n\nNone of this kills the method. The components are well motivated and the direction is plausible, but the evidence in the paper does not support the headline. The authors need to reconcile the aggregates, report per-setting comparisons honestly, provide seeds and code artifacts, add error bars, and re-run cache-size selection in a validation-only split.\n\nThis is a paper for the graph foundation model / ICL subfield. With the numbers fixed it deserves a serious refereeing; in its current form it should be revised before acceptance. I'd still send it to peer review rather than desk reject — the idea is worth the referee time.","headline":"A new prompt-optimization recipe for graph ICL with real components, undermined by its own inconsistent tables and test-set-tuned hyperparameters.","tokens_in":21069,"tokens_out":3493,"would_cite":false,"duration_ms":33000,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GraphPrompter optimizes prompt generation, selection, and use, improving graph in-context learning by over 8% with no parameter updates.","keywords":["graph in-context learning","prompt optimization","graph neural networks","few-shot learning","retrieval-augmented selection","test-time adaptation","node classification","edge classification"],"falsifier":"Pre-train GraphPrompter on a citation graph, then test on a structurally and featurally distant graph such as a social network or a graph with only numeric features; if kNN-retrieved prompts do not outperform randomly selected prompts (nullifying the score function in Eq. 7) or if the reported average 8% gain over Prodigy disappears, the cross-domain transfer assumption behind the selector fails.","tokens_in":19960,"feed_emoji":"🧩","tokens_out":7906,"duration_ms":72846,"temperature":0.7,"pith_summary":"Graph in-context learning (GICL) adapts a pre-trained graph model to new graphs by conditioning it on a few labeled prompt subgraphs, with no weight updates. The paper's central claim is that existing GICL methods are held back by their prompts: random subgraph sampling and random prompt selection inject noise, and the gap between pretraining and test graphs makes accuracy collapse when the number of downstream classes grows large. To fix this, GraphPrompter reworks all three stages of prompting: a reconstruction layer denoises the sampled subgraphs, a kNN-plus-importance selector chooses query-specific prompts, and an LFU cache augments the prompt set with confident pseudo-labeled test samples. The authors report that across node and edge classification benchmarks, this multi-stage pipeline surpasses state-of-the-art baselines by over 8% on average, with the largest gains in 40- to 100-class settings.","feed_headline":"Multi-stage prompt tuning lifts graph in-context learning by 8%","feed_subtitle":"GraphPrompter denoises, retrieves, and caches prompt subgraphs so pretrained models generalize without parameter updates.","key_machinery":"The load-bearing object is the task-graph formulation inherited from Prodigy, where each query is classified by comparing its task-graph node embedding with label embeddings obtained by aggregating prompt subgraphs. GraphPrompter's three modules improve the inputs to that aggregation: the reconstruction layer (an MLP-plus-sigmoid edge re-weighting followed by a GraphSAGE encoder) produces denoised subgraph embeddings; the selector computes $score(p,q)=\\cos(G_p,G_q)+I_p\\cdot I_q$ and uses a voting scheme to choose the top-$k$ prompts; and the LFU cache injects pseudo-labeled test embeddings into the prompt set. The key property is that no parameter updates occur at test time: all adaptation comes from changing which subgraphs feed into the task graph, making the method a pure prompt-optimization procedure.","core_discovery":"GraphPrompter is a three-stage, parameter-free-at-test-time pipeline for improving graph in-context learning. Stage one, the Prompt Generator, replaces the random walk subgraphs used by existing approaches with a reconstruction layer that learns an edge weight $w_{uv}=\\mathrm{sigmoid}(\\mathrm{MLP}(\\text{node embeddings}, \\text{edge embedding}))$ for every edge in the sampled $l$-hop subgraph, then runs a GNN over the re-weighted graph to produce a cleaner subgraph embedding. Stage two, the Prompt Selector, scores each candidate prompt $p$ for query $q$ by combining a pre-trained importance value $I_p$ with cosine similarity between subgraph embeddings, $score(p,q)=\\cos(G_p,G_q)+I_p\\cdot I_q$, and uses a voting mechanism across all queries to pick the top-$k$ prompts. Stage three, the Prompt Augmenter, maintains a small cache of the most confident test-time predictions (pseudo-labels) and adds them to the prompt set using an LFU replacement policy. All model parameters are learned in the pretraining phase and no gradient updates happen during inference. Across arXiv node classification and ConceptNet/FB15K-237/NELL edge classification under 3-shot settings, the method surpasses Prodigy, OFA, ProG, and fine-tuned baselines by an average of over 8% accuracy, with the largest gains when the number of classes grows to 40-100.","pith_inferences":["An implication the paper leaves implicit is that the cross-domain relevance signal is entirely the cosine similarity between pre-trained subgraph embeddings; any better alignment of that embedding space, such as a learned cross-domain metric fitted on a handful of labeled examples, would directly strengthen the kNN selector and likely widen the reported gains.","A testable extension is to make the LFU cache label-noise-aware: filtering pseudo-labels by predictive entropy or prediction agreement across augmented views rather than raw confidence should reduce the performance drop the paper observes when the cache grows beyond size 3.","The reconstruction layer is a form of graph structure learning, and since GraphSAGE with learned edge weights beat GAT's attention weighting, an implication is that lightweight edge reweighting is a portable complement to any prompt-graph model, not only the Prodigy-style architecture tested here."],"forward_implications":["Without any gradient updates, GraphPrompter consistently outperforms Prodigy, OFA, and ProG across node and edge classification, by about 8% average accuracy in 3-shot settings.","Ablations show each of the three components helps: removing the reconstruction layer causes the largest drop, while kNN and selection layers are complementary and the LFU cache benefits both few-class and many-class settings.","The approach transfers the prompt-optimization paradigm from large language models to graphs, suggesting that graph foundation models can adapt to new domains purely through better prompt engineering rather than fine-tuning.","High-class-count robustness improves: on 50- to 100-way edge classification, GraphPrompter holds accuracy better than Prodigy, which degrades sharply beyond the 15-class pretraining regime.","The added inference cost is bounded: with a cache of size 3, per-query time grows roughly two- to three-fold but stays in the tens-to-hundreds of milliseconds range."],"supporting_citations":[{"why":"The Prodigy framework that GraphPrompter builds on provides the task-graph architecture and the random prompt selection baseline it replaces.","marker":"[3]"},{"why":"OFA, the One-For-All prompt-graph baseline that GraphPrompter outperforms in cross-domain few-shot settings.","marker":"[5]"},{"why":"Learning-to-retrieve prompts for in-context learning supplies the retrieval-augmented prompt selection idea.","marker":"[13]"},{"why":"Selective annotation results justify the claim that choosing better in-context examples improves few-shot generalization.","marker":"[14]"},{"why":"GraphSAGE is the encoder used for subgraph embeddings in the Prompt Generator.","marker":"[22]"},{"why":"TENT's test-time adaptation by entropy minimization inspires the online Prompt Augmenter.","marker":"[45]"},{"why":"Provides the Wiki, FB15K-237, and NELL datasets used for pretraining and evaluation.","marker":"[48]"},{"why":"The O(1) LFU cache eviction scheme implements the Prompt Augmenter's replacement strategy.","marker":"[51]"}],"fun_headline_variants":["Parameter-free graph learning gets 8% boost via adaptive prompts","Multi-stage prompt optimization lifts graph in-context learning","Denoise, select, cache: new prompt strategy beats baselines by 8%","Adaptive prompts for graph in-context learning: +8% average","GraphPrompter: smarter prompt selection improves graph learning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pre-trained GNN's embedding space is comparable across the pretraining graph and the downstream graph, so cosine similarity between a query subgraph and candidate prompt subgraphs reliably ranks which prompts are actually useful.","fun_headline_variants_meta":{"raw":{"variants":["Parameter-free graph learning gets 8% boost via adaptive prompts","Multi-stage prompt optimization lifts graph in-context learning","Denoise, select, cache: new prompt strategy beats baselines by 8%","Adaptive prompts for graph in-context learning: +8% average","GraphPrompter: smarter prompt selection improves graph learning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000237,"raw_usage":{"total_tokens":1601,"prompt_tokens":1136,"completion_tokens":465,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":752,"completion_tokens_details":{"reasoning_tokens":385}},"tokens_in":752,"tokens_out":465,"duration_ms":4414,"temperature":1.0,"reasoning_tokens":385,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:05:07.223091+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Pre-train GraphPrompter on a citation graph, then test on a structurally and featurally distant graph such as a social network or a graph with only numeric features; if kNN-retrieved prompts do not outperform randomly selected prompts (nullifying the score function in Eq. 7) or if the reported average 8% gain over Prodigy disappears, the cross-domain transfer assumption behind the selector fails.","supporting_citations":[{"cited_title":"Prodigy: Enabling in-context learning over graphs,","cited_arxiv_id":null,"evidence_quote":"The Prodigy framework that GraphPrompter builds on provides the task-graph architecture and the random prompt selection baseline it replaces."},{"cited_title":"One for all: Towards training one graph model for all classification tasks,","cited_arxiv_id":null,"evidence_quote":"OFA, the One-For-All prompt-graph baseline that GraphPrompter outperforms in cross-domain few-shot settings."},{"cited_title":"Learning to retrieve prompts for in- context learning,","cited_arxiv_id":null,"evidence_quote":"Learning-to-retrieve prompts for in-context learning supplies the retrieval-augmented prompt selection idea."},{"cited_title":"Selective anno- tation makes language models better few-shot learners,","cited_arxiv_id":null,"evidence_quote":"Selective annotation results justify the claim that choosing better in-context examples improves few-shot generalization."},{"cited_title":"Inductive representation learning on large graphs,","cited_arxiv_id":null,"evidence_quote":"GraphSAGE is the encoder used for subgraph embeddings in the Prompt Generator."},{"cited_title":"One-shot relational learning for knowledge graphs,","cited_arxiv_id":null,"evidence_quote":"Provides the Wiki, FB15K-237, and NELL datasets used for pretraining and evaluation."},{"cited_title":"An O(1) algorithm for implementing the LFU cache eviction scheme,","cited_arxiv_id":null,"evidence_quote":"The O(1) LFU cache eviction scheme implements the Prompt Augmenter's replacement strategy."}],"review_version":1}