{"id":"e212a23c-b141-4944-930f-b822d6d452cd","arxiv_id":"2506.07168","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"GAGA matches or exceeds state-of-the-art accuracy on several text-attributed graph benchmarks while requiring large language model annotations for only 1% of nodes or edges.","lead":"Researchers built GAGA, a framework that trains graph neural networks on text-rich networks using large language model annotations for only 1% of the data. It matches or beats existing methods on citation and product graphs while cutting annotation time and cost by 3 to 100 times.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never specifies whether the 1% annotated nodes are also the only ground-truth labels used in downstream fine-tuning, so the comparison to 1%-label baselines in Section 5.2 is ambiguous and the central label-efficiency claim is unverified.","rationale":"The load-bearing assumption is exactly the one the Reader identified: the paper never says whether the 1% annotation budget is also the label budget for downstream fine-tuning. This matters because the central claim is efficiency with 1% data, and Section 5.2 explicitly benchmarks against GCN* and TAPE* 'using only 1% of labeled data.' If GAGA actually trains its GNN on the full training split, the comparison is apples-to-oranges and the headline 'only 1% annotated' claim must be re-scoped to LLM annotation cost only. The concern is concrete, testable, and directly connected to the paper's central claim; no other issue I examined would change the verdict. The method is otherwise plausible: the alignment idea is coherent, the ablations show robustness to backbone and seed ratio, and the cost savings over TAPE are large even if only LLM annotation is reduced. Because the missing label budget is an omitted experimental detail that could be fixed by reporting or code release, the appropriate disposition remains CONDITIONAL rather than outright rejection.","tokens_in":20476,"tokens_out":7495,"duration_ms":80341,"concrete_test":"Obtain the training harness (from authors or a code release) and count the ground-truth node labels used in the downstream fine-tuning loss for node classification, separately from the 1% LLM-annotated nodes. Then run GAGA on ogbn-arxiv and PubMed in two settings: (i) fine-tune only with labels of the 1% selected seed nodes, and (ii) fine-tune with the full official training split, and compare both to GCN* and TAPE* trained with the same 1% label budget. If setting (i) drops substantially below the reported 76.21/94.62 and does not beat GCN*/TAPE*, the label-efficiency claim is unsupported; if setting (i) matches the reported numbers, the concern is resolved. Also report the exact train/validation/test split sizes used for Table 2.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that GAGA needs only 1% of data annotated. The method section defines annotation as LLM-generated predictions/explanations for 1% of nodes (Sections 4.1, 5.1), but it never states how many ground-truth labels supervise the downstream fine-tuning described in Section 4.3. If the downstream classifier is trained on the full training split, then the reported accuracies (76.21 on ogbn-arxiv, 94.62 on PubMed) reflect full supervision plus 1% LLM annotations, and the comparison in Section 5.2 with GCN* and TAPE*—explicitly described as 'using only 1% of labeled data'—is not a label-efficiency comparison. The near-flat accuracy from 0.1% to 1.0% seed ratio in Table 4 is consistent with full-label fine-tuning, since the annotation ratio would then not be the limiting factor. Because the abstract and Section 5.2 use 'annotated' and 'labeled' in overlapping ways, the headline efficiency claim is ambiguous between LLM annotation cost and ground-truth label cost. The strongest experimental evidence for label efficiency therefore rests on an unstated label budget.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"GAGA is a framework for node classification and link prediction on text-attributed graphs. It selects a small set of representative nodes (or edges) via k-means density scoring, prompts an LLM to generate predictions/explanations for those nodes, builds a k-NN annotation graph from the LLM outputs, aligns subgraphs of this annotation graph with subgraphs of the original TAG through two-level contrastive learning (subgraph-level and prototype-level), and then fine-tunes only a GNN on the downstream task while keeping the language model frozen. The paper reports strong accuracies on six datasets, claims state-of-the-art or near-state-of-the-art results with only 1% of data annotated, and presents time/cost comparisons against prior LLM-based methods.","tokens_in":20699,"tokens_out":3900,"duration_ms":43439,"significance":"If the central efficiency claim holds, GAGA is a practically useful contribution: it would substantially reduce the LLM annotation cost for TAG representation learning while retaining competitive accuracy. The paper is commendably empirical, with ablations over backbone choice, annotation ratio, prototype size, alpha, and language model; these help establish robustness of the proposed components. The strongest experimental evidence is the time/cost table and the accuracy table against strong baselines. However, the central label-efficiency claim is not yet verified because the manuscript never states how many ground-truth labels supervise the downstream fine-tuning stage. This is a load-bearing gap, not a cosmetic one, and it must be resolved before the efficiency conclusion is reliable.","major_comments":[{"comment":"The manuscript does not state how many ground-truth labels are used when fine-tuning the GNN for downstream tasks. Section 5.1 says 'we use 1% nodes for annotation', but annotation here is the LLM-generated prediction/explanation; it is not stated whether those same 1% nodes are the only labeled examples available to the downstream classifier. Table 2 compares GAGA with GCN* and TAPE*, which are described as 'using only 1% of labeled data', but if GAGA's downstream fine-tuning uses the full training split, this is not a label-efficiency comparison. The central claim in the abstract and Section 5.2 that GAGA 'requires only 1% of the data to be annotated' is therefore ambiguous between LLM annotation cost and ground-truth label cost. The authors must specify the exact label budget for the downstream classifier and, if it differs from 1%, rerun the comparisons under matched label budgets.","section":"§5.1, §5.2, §4.3"},{"comment":"The subgraph alignment loss in Eq. (1) assumes that the sub-text graph G_T and the sub-annotation graph G_A contain the same set of nodes, since it computes pairwise distances between h_i^t and h_i^a for matched indices i. However, Section 4.2 says that for each selected node v* the method samples its k-hop neighbors in 'both the TAG and annotation graph G*_A'. The annotation graph contains only the selected nodes V*_a, so its k-hop neighborhood generally has far fewer nodes than the k-hop neighborhood in the original TAG, which includes unselected neighbors. The paper does not explain how the node sets are made to correspond, or whether unmatched nodes are discarded. This is a central step of the method and needs a precise description, otherwise Eq. (1) is undefined for subgraphs of different sizes.","section":"§4.2, Eq. (1)"},{"comment":"The near-flat accuracy from 0.1% to 1.0% seed ratio in Table 4 is consistent with the hypothesis that the downstream classifier is trained on the full training split, since the seed ratio would then affect only the alignment stage, not the label supervision. The authors should report the downstream label budget alongside this table and, if the classifier does use full labels, provide an additional experiment where the number of downstream labels is also restricted to 1% (or another fixed small fraction). Without this, the paper does not establish that 0.1%–1.0% annotated nodes are sufficient for the complete learning pipeline.","section":"Table 4"},{"comment":"The time and cost comparison in Table 1 is informative, but the caption in Appendix A.3.2 states that money usage includes only the cost of LLM annotation, not any other labeling cost. If the downstream stage uses a substantial number of human ground-truth labels, those costs are omitted from the efficiency comparison. The paper should clarify whether any human labeling cost is involved in GAGA and, if so, include it or explicitly scope the efficiency claim to LLM annotation only. The current wording of the abstract and Section 5.2 ('requiring only 1% of the data to be annotated') overstates what Table 1 actually measures.","section":"§5.2, Table 1, Appendix A.3.2"}],"minor_comments":[{"comment":"In the OneForAll* row the accuracy value is printed as '0.6983' while all other accuracies in the same table are given in percent; this is almost certainly a typo for 69.83 (or an inconsistent formatting choice) and should be corrected.","section":"Table 1"},{"comment":"The sentence 'we use 1% nodes for annotation and √nedges for link prediction annotation with edge number nedges' is ambiguous: it should state explicitly that the number of selected edges is the square root of the number of edges in the graph, and it should define the notation consistently.","section":"§5.1"},{"comment":"Table 6 reports test accuracy 76.65% for kp=40 on ogbn-arxiv, while Table 2 reports 76.21% for GAGA on the same dataset with kp=40 specified in Section 5.1. The authors should explain this discrepancy, for instance by stating whether these are different runs, different seeds, or different settings.","section":"Table 6 vs. Table 2"},{"comment":"The section heading is 'Impact of Text Encoder' but the table title says 'different LLMs for annotation'; the first two rows are MiniLM variants and the latter rows are much larger models. The terminology should be aligned so that the reader knows whether the language model is used as the encoder, the annotation generator, or both.","section":"Appendix B, Table 8"},{"comment":"The prompt for ogbn-products lists 46 categories, while Section A.1 states the dataset has 47 classes. The authors should verify that the prompt covers all possible labels, or explain why the omitted label is not needed.","section":"Appendix D"}],"recommendation":"major_revision","confidential_remarks":"The paper has a promising empirical design and the ablations are a real strength. The main issue is the unstated downstream label budget, which is central to the efficiency claim; if it turns out that full ground-truth labels are used for fine-tuning, the paper would need substantial reframing and re-analysis. I would ask the authors to clarify this point and, if necessary, rerun the label-matched comparison before the work is accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know about this paper: it presents GAGA, a three-stage pipeline for text-attributed graph learning—select representative nodes/edges via density-based k-means, prompt an LLM to annotate just 1% of them, build an annotation graph, and align that graph with the original TAG through two-level contrastive learning (subgraph and prototype). Downstream, the LM stays frozen and a GNN is fine-tuned with cross-attention over prototype embeddings. The empirical results on six datasets look plausible and the method is simple enough to be useful.\n\nWhat is actually new: the specific integration. TAPE annotates all nodes, SimTeG fine-tunes on all nodes, ENGINE adds LLM layers. GAGA's selective annotation plus annotation-graph construction plus two-level alignment is a reasonable combination I have not seen in the cited prior work. The paper also does real ablation work—backbone, annotation ratio, prototype size, alpha, and feature combination—and reports time and money costs. That is honest, reproducible-to-the-extent-possible effort.\n\nThe soft spots, in proportion. The big one is experimental transparency. The paper never states how many ground-truth labels are used for downstream fine-tuning. It says \"we use 1% nodes for annotation\" in Section 5.1, and compares against GCN* and TAPE* described as \"using only 1% of labeled data.\" If the downstream GNN is trained on the full training split, then the 1% is only about LLM annotation cost, not label cost, and the comparison with 1%-label baselines is apples-to-oranges. The near-flat accuracy from 0.1% to 1.0% in Table 4 is exactly what you'd expect if the annotation ratio is not the limiting factor—the model is already seeing plenty of ground-truth labels. This is a load-bearing omission, not a nitpick. A second, smaller issue: Appendix E admits alignment must be run per dataset, so the method does not transfer to new graphs without retraining. And there is no code release, which makes the alignment details hard to reproduce.\n\nWho this is for: graph ML researchers working on LLM-augmented TAG learning would get value from reading this, especially the contrastive alignment idea. But the central efficiency claim is unsupported until the label budget is clarified.\n\nI would send this to peer review—it deserves serious refereeing—but I would make the authors state the exact number of ground-truth labels in Section 4.3 and rerun the low-label comparison with matched budgets. If they confirm downstream fine-tuning uses only the selected 1% labels, the claim stands. If not, they need to reframe the paper as an annotation-cost saving, not a label-efficiency saving.","headline":"GAGA is a sensible contrastive-alignment method with real experiments, but the central '1% data' claim conflates LLM annotation cost with ground-truth label budget, so the headline efficiency result is unverified as written.","tokens_in":21289,"tokens_out":2726,"would_cite":false,"duration_ms":27557,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GAGA matches or beats LLM-augmented graph methods while annotating only 1% of nodes, using a sparse annotation graph plus two-level contrastive alignment to generalize to the full text-attributed graph.","keywords":["text-attributed graphs","node classification","link prediction","selective annotation","graph alignment","contrastive learning","large language models","vector quantization"],"falsifier":"Replace the density-based selector with random node selection while holding the annotation budget, alignment, and downstream fine-tuning fixed, and run on ogbn-arxiv. If random selection keeps test accuracy around 76%, the claim that representative selection is the source of GAGA's efficiency is false. Independently, count the ground-truth labels used in the final fine-tuning step: if the full standard training split is required to reach 76%, then the '1% annotated' framing does not describe the method's actual label consumption.","tokens_in":20204,"feed_emoji":"📉","tokens_out":8250,"duration_ms":76481,"temperature":0.7,"pith_summary":"GAGA is an attempt to make text-attributed graph learning cheap by annotating only a tiny, representative slice of the graph instead of every node. The paper claims that with about 1% of nodes annotated (or a sparse set of edges for link prediction), the framework matches or beats LLM-augmented state-of-the-art methods on node classification and link prediction benchmarks while cutting annotation time and cost by 3 to 100 times. It does so by prompting an LLM to annotate selected nodes or edges, assembling those annotations into an annotation graph, and then teaching a GNN, with the language model kept frozen, to align subgraphs of the full network with these annotations. If the claim holds, GAGA would remove the main cost barrier, paid LLM annotation and per-node fine-tuning, that keeps LLM-based graph methods off large industrial graphs.","feed_headline":"1% of nodes annotated: match or beat full-label graph models","feed_subtitle":"A sparser annotation graph plus two-level alignment cuts ogbn-arxiv query cost from $113 to under $3.","key_machinery":"The load-bearing object is the annotation graph, built from the selected 1% of nodes or edges after LLM annotation, with edges connecting each annotation to its $k'$-nearest neighbors by embedding similarity. This graph is used in a two-level alignment: a subgraph-level contrastive loss aligns sampled two-hop neighborhoods from the text-attributed graph with matching sub-annotation graphs, while a prototype-level loss, derived from vector quantization of annotation embeddings into $k_p$ prototypes, aligns textual embeddings with prototype embeddings. A cross-attention layer between the fine-tuned GNN's node embedding and the prototype matrix fuses annotation knowledge at test time. This machinery lets semantic and structural information from sparse annotations propagate to the whole graph while the LLM encoder stays frozen.","core_discovery":"The paper's central claim is that the topological structure among a small set of LLM annotations can substitute for annotation of the whole graph. GAGA first scores each node by its Euclidean distance to the nearest $k$-means cluster center in language-model embedding space and keeps the top 1% as representative, using the density score $\\phi_{\\mathrm{density}}(v_i) = 1/(1+\\|\\mathrm{Emb}(v_i)-C_{e(v_i)}\\|)$; for edges it sums endpoint densities. An LLM prompted for predictions, concepts, and reasoning supplies annotations, and $k'$-nearest-neighbor links among annotation embeddings form an annotation graph. Two contrastive objectives then align two-hop sub-text graphs with corresponding sub-annotation graphs and with vector-quantized prototypes, so a frozen MiniLM encoder plus a fine-tuned GCN can be deployed downstream. On ogbn-arxiv the paper reports 76.21% accuracy at $2.87 annotation cost versus TAPE's 75.20% at $113.39, and on PubMed 94.62% versus 94.31%; for link prediction, annotating only $\\sqrt{n_{\\mathrm{edges}}}$ edges reaches 99.24% AUC on PubMed.","pith_inferences":["Beyond the paper's reported experiments, the selection rule is a testable design choice: replacing $k$-means-density selection with random or uncertainty-based selection under the same 1% budget would quantify how much of the gain comes from choosing high-density nodes rather than from the alignment module itself.","The two-level alignment and prototype cross-attention are not specific to node classification; they could be dropped into active-learning pipelines where a human, not an LLM, provides the sparse annotations, or into graph retrieval where prototypes act as a compact memory of recurring concepts.","A cheap audit is to count how many ground-truth labels enter the final GNN fine-tuning: if only the 1% annotated nodes supply labels, the method is fully label-efficient, whereas if the standard full training split is used, the efficiency claim should be read as annotation-cost efficiency rather than total label efficiency. This distinction is not spelled out in the paper."],"forward_implications":["On ogbn-arxiv, PubMed, and tape-arxiv23, the paper reports the best node-classification accuracy among the compared methods while using only 1% annotated nodes, with gains over TAPE of 1.01 points on ogbn-arxiv and 0.31 points on PubMed.","Annotation cost drops to $2.87 on ogbn-arxiv and $0.49 on PubMed, against $113.39 and $17.63 for TAPE, so LLM-augmented graph pipelines become affordable on large graphs.","Accuracy on ogbn-arxiv stays at 75.71% even when only 0.1% of nodes are selected, indicating the method tolerates a further tenfold reduction in the annotation budget.","Because the GNN backbone is interchangeable across GCN, SAGE, GAT, and RevGAT, the alignment stage can be reused with whatever message-passing model is already deployed.","For link prediction, annotating only a sparse set of edges yields results that lead MRR@10 on Cora and Citeseer and AUC on PubMed, so the same recipe transfers from node classification to edge-level supervision."],"supporting_citations":[{"why":"Supplies the main annotation-heavy baseline and the tape-arxiv23 benchmark; GAGA's cost and accuracy comparisons center on it.","marker":"[23]"},{"why":"Baseline that fine-tunes an LLM for node representations; GAGA claims to replace full fine-tuning with alignment.","marker":"[13]"},{"why":"Baseline whose self-supervised neighborhood prediction is the main time-consuming alternative on large graphs.","marker":"[9]"},{"why":"Baseline that couples an LLM with a GNN via variational EM; compared in time, cost, and accuracy.","marker":"[84]"},{"why":"Baseline that inserts GNN structure into every LLM layer; its OOM and timeout outcomes drive the efficiency comparison.","marker":"[89]"},{"why":"The message-passing backbone used in GAGA's GNN encoder and in the label-limited GCN* and TAPE* baselines.","marker":"[37]"},{"why":"Supplies ogbn-arxiv and ogbn-products datasets and their standard splits.","marker":"[32]"},{"why":"Supplies vector quantization, the mechanism GAGA uses to compress annotation embeddings into prototypes.","marker":"[62]"},{"why":"Provides the link-prediction benchmark splits and baseline suite that GAGA compares against.","marker":"[41]"},{"why":"The GPT-3.5 model used for generating node and edge annotations.","marker":"[3]"}],"fun_headline_variants":["Match full-label accuracies with only 1% node annotations","GAGA: 1% annotation cuts graph learning cost to $3","Sparse annotation graph outperforms dense labeling on graphs","1% annotations, full-label performance in graph classification"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on the assumption that the few nodes closest to $k$-means cluster centres in language-model embedding space carry enough of the graph's semantic and structural variety that aligning their annotations with the full network teaches a model that generalizes to all the unannotated nodes.","fun_headline_variants_meta":{"raw":{"variants":["Match full-label accuracies with only 1% node annotations","GAGA: 1% annotation cuts graph learning cost to $3","Sparse annotation graph outperforms dense labeling on graphs","1% annotations, full-label performance in graph classification"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00022,"raw_usage":{"total_tokens":1457,"prompt_tokens":963,"completion_tokens":494,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":579,"completion_tokens_details":{"reasoning_tokens":425}},"tokens_in":579,"tokens_out":494,"duration_ms":5665,"temperature":1.0,"reasoning_tokens":425,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:40:32.011037+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the density-based selector with random node selection while holding the annotation budget, alignment, and downstream fine-tuning fixed, and run on ogbn-arxiv. If random selection keeps test accuracy around 76%, the claim that representative selection is the source of GAGA's efficiency is false. Independently, count the ground-truth labels used in the final fine-tuning step: if the full standard training split is required to reach 76%, then the '1% annotated' framing does not describe the method's actual label consumption.","supporting_citations":[{"cited_title":"Evaluating graph neural networks for link prediction: Current pitfalls and new benchmarking","cited_arxiv_id":null,"evidence_quote":"Provides the link-prediction benchmark splits and baseline suite that GAGA compares against."}],"review_version":1}