pith. sign in

arxiv: 2605.12827 · v2 · pith:B6X7YNKJnew · submitted 2026-05-12 · 💻 cs.CR · cs.AI· cs.LG

GraphIP-Bench: How Hard Is It to Steal a Graph Neural Network, and Can We Stop It?

Kaixiang Zhao , Bolin Shen , Yuyang Dai , Shayok Chakraborty , Yushun Dong This is my paper

Pith reviewed 2026-05-14 19:22 UTC · model grok-4.3

classification 💻 cs.CR cs.AIcs.LG
keywords graph neural networksmodel extraction attacksownership verificationwatermarkingbenchmarkadversarial machine learninggraph classificationmodel stealing
0
0 comments X

The pith

Stealing a graph neural network is straightforward at medium query budgets, and existing defenses rarely prevent extraction or preserve ownership signals on surrogates.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper establishes a comprehensive benchmark to determine how easily graph neural networks can be stolen through model extraction attacks and whether defenses can effectively counter them. By standardizing evaluations across multiple attacks, defenses, graphs, backbones, and tasks, it reveals that extraction succeeds with moderate resources in most cases. The benchmark also shows that while some defenses like watermarks protect the original model, they lose effectiveness once an attacker trains a surrogate. Heterophilic graphs present greater challenges for extraction compared to others, and mismatches in model architecture between target and surrogate hinder but do not stop the process. These findings matter because GNNs are increasingly offered as cloud services, making them vulnerable to unauthorized replication without strong protections.

Core claim

The paper claims that under a unified black-box protocol, model extraction attacks can produce surrogates with high fidelity to the target GNN at medium query budgets, and that defenses spanning watermarking, output perturbation, and query detection largely fail to increase extraction difficulty or maintain reliable ownership verification on the resulting surrogates, with heterophilic graphs being systematically harder to steal and cross-architecture extraction succeeding at reduced performance.

What carries the argument

GraphIP-Bench, the unified benchmark integrating twelve extraction attacks, twelve defenses, ten graphs, three backbones, and three tasks to evaluate fidelity, utility, verification, and cost under shared conditions.

If this is right

  • Extraction is easy at medium query budgets across most configurations.
  • Watermarks verify on protected models but lose most signal on extracted surrogates.
  • Heterophilic graphs are harder to steal than homophilic ones.
  • Cross-architecture mismatch reduces extraction success but does not prevent it.
  • Joint attack-defense evaluation exposes gaps missed by single-model tests.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Service providers may require defenses designed to survive the extraction process itself rather than just marking the original model.
  • The results point to heterophily as a potential natural barrier worth exploiting in future designs.
  • Extending the benchmark to include adaptive attacks targeting specific defenses could reveal additional vulnerabilities.
  • Real-world GNN services might benefit from monitoring query patterns more aggressively if detection proves more robust than watermarking.

Load-bearing premise

The chosen set of twelve attacks, twelve defenses, ten graphs, three backbones, and three tasks is representative of the broader space of GNN services and threats.

What would settle it

A defense that achieves high watermark verification accuracy on surrogates extracted from defended targets across the benchmark's diverse setups would contradict the finding that most defenses lose their verification signal after extraction.

Figures

Figures reproduced from arXiv: 2605.12827 by Bolin Shen, Kaixiang Zhao, Shayok Chakraborty, Yushun Dong, Yuyang Dai.

Figure 2
Figure 2. Figure 2: Budget–metric curves on six representative datasets (columns) for accuracy and fidelity [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 1
Figure 1. Figure 1: Sample efficiency across ten datasets and four regimes. Extension to large-scale and heterophilic graphs. The seven graphs in the original protocol are small and ho￾mophilic, which limits the conclusions to one structural regime. To extend the protocol along three independent axes, we evaluate the same twelve attacks on three addi￾tional graphs: OGBN-Arxiv (169,343 nodes, 40 classes, edge homophily 0.699) … view at source ↗
Figure 3
Figure 3. Figure 3: Surrogate fidelity (%) at budget 0.25× on the three additional graphs for all twelve attacks (mean over three seeds, whiskers are ± one std). Among the watermarking and integrity defenses, BackdoorWM offers the best protection-utility balance: it reaches the highest median fidelity (80.07%), per￾fect verification, and only a 3.27 pp median utility drop. ImperceptibleWM also reaches perfect ver￾ification, b… view at source ↗
Figure 4
Figure 4. Figure 4: Protection-utility scatter across ten datasets and three seeds per defense (one point per dataset-seed run). Upper-left is best. To answer RQ3, we evaluate the defended model’s task utility and its alignment with the original target together with ownership verification on a fixed ver￾ification set under the unified protocol in Section 3. All defenses protect the same architecture and use identical splits; … view at source ↗
Figure 5
Figure 5. Figure 5: Defense effectiveness across all ten datasets: (a) utility drop and (b) ownership verification. [PITH_FULL_IMAGE:figures/full_fig_p032_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Radar profile of the five watermarking and integrity defenses across six axes. F1, fidelity, [PITH_FULL_IMAGE:figures/full_fig_p034_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Heatmap view of the seven information-limiting and query-detection defenses across all [PITH_FULL_IMAGE:figures/full_fig_p037_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Peak GPU memory (GB) on a symmetric-log scale, aggregated across all ten datasets. Bars [PITH_FULL_IMAGE:figures/full_fig_p038_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Wall-clock cost summary on a log scale. (a) Total attack time (min) at [PITH_FULL_IMAGE:figures/full_fig_p038_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Budget–metric curves on all ten datasets (columns) for accuracy, fidelity, and macro [PITH_FULL_IMAGE:figures/full_fig_p039_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Regime sensitivity across budgets. Cells show the ratio between the average fidelity in a [PITH_FULL_IMAGE:figures/full_fig_p040_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Per-attack surrogate-fidelity curves on all ten datasets in the [PITH_FULL_IMAGE:figures/full_fig_p040_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Regime sensitivity from two complementary views: differential (left) and absolute (right). [PITH_FULL_IMAGE:figures/full_fig_p041_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Distribution of per-run attack wall-clock time (log scale, in minutes) across all (dataset, [PITH_FULL_IMAGE:figures/full_fig_p041_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Pairwise Pearson correlation between the twelve attacks, where each attack is represented [PITH_FULL_IMAGE:figures/full_fig_p042_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Per-dataset utility loss of the seven information-limiting and query-detection defenses [PITH_FULL_IMAGE:figures/full_fig_p042_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: RQ5 joint evaluation on Computers at 0.25× (mean over three seeds). (a) Surrogate fidelity (%) against the five watermarks. (b) Surrogate fidelity (%) against the seven information￾limiting defenses; PRADA and AdaptMisinfo reduce the strongest attacks from 80–92% to 25–55%. (c) Watermark verification rate (%) on the extracted surrogate; Integrity survives at near 100% on most attacks, whereas SurviveWM, R… view at source ↗
Figure 18
Figure 18. Figure 18: Distribution and structural correlates of watermark survival. [PITH_FULL_IMAGE:figures/full_fig_p043_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Distributional views of surrogate fidelity. (a) shows that fidelity distributions are nearly [PITH_FULL_IMAGE:figures/full_fig_p045_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: Per-dataset heatmap grid of joint surrogate fidelity (%). Each panel is a [PITH_FULL_IMAGE:figures/full_fig_p052_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: Cross-task heatmap of surrogate fidelity (%) on three task settings: link prediction on [PITH_FULL_IMAGE:figures/full_fig_p054_21.png] view at source ↗
read the original abstract

Graph neural networks (GNNs) deployed as cloud services can be stolen through model-extraction attacks, which train a surrogate from query responses to reproduce the target's behavior, and a growing line of ownership defenses tries to prevent or trace such theft. This paper asks two questions: how hard is it to steal a GNN, and can we stop it? Prior work cannot answer either, because experiments use inconsistent datasets, threat models, and metrics. We introduce GraphIP-Bench, a unified benchmark that evaluates both sides under a single black-box protocol. GraphIP-Bench integrates twelve extraction attacks, twelve defenses spanning watermarking, output perturbation, and query-pattern detection, ten public graphs covering homophilic, heterophilic, and large-scale regimes, three GNN backbones, and three graph-learning tasks. It reports fidelity, task utility, ownership verification, and computational cost on shared splits, queries, and budgets. We further add a joint attack-and-defense track that runs every attack on every defended target and measures watermark verification on the resulting surrogate, exposing how much protection a defense retains after extraction. The empirical picture is clear: stealing a GNN is easy at medium query budgets and most defenses do not change this; several watermarks verify reliably on the protected model but lose most of their verification signal on the extracted surrogate, exposing a gap that single-model evaluations miss; and heterophilic graphs are systematically harder to steal, while a cross-architecture mismatch between target and surrogate reduces but does not prevent extraction. We release GraphIP-Bench with reproducible scripts and configurations, and integrate the attacks and defenses into the PyGIP library. Code: https://github.com/LabRAI/GraphIP-Bench. Library: https://labrai.github.io/PyGIP/index.html.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 3 minor

Summary. The paper introduces GraphIP-Bench, a unified benchmark for evaluating model extraction attacks on GNNs and corresponding ownership defenses under a consistent black-box protocol. It integrates twelve extraction attacks, twelve defenses spanning watermarking, output-perturbation, and query-pattern-detection families, ten public graphs covering homophilic, heterophilic, and large-scale regimes, three GNN backbones, and three graph-learning tasks. The benchmark reports fidelity, task utility, ownership verification, and computational cost on shared splits and query budgets. A joint attack-and-defense track evaluates every attack against every defended target and measures watermark verification on the resulting surrogates. The main empirical findings are that stealing a GNN is easy at medium query budgets, most defenses do not substantially alter this, watermarks verify reliably on protected models but lose most verification signal on extracted surrogates, heterophilic graphs are systematically harder to steal, and cross-architecture mismatch reduces but does not prevent extraction.

Significance. If the selected components prove representative, the benchmark supplies a standardized, reproducible platform that resolves inconsistencies in prior GNN extraction studies and enables direct comparison of attacks and defenses. The joint attack-defense track is a particular strength, as it reveals protection gaps that single-model evaluations miss. Public code release supports reproducibility and future extensions. These elements could usefully guide development of more robust GNN IP mechanisms by quantifying current limitations in watermarking and perturbation approaches.

major comments (3)
  1. [§3] §3 (Benchmark Design): The selection of the twelve attacks, twelve defenses, ten graphs, three backbones, and three tasks lacks any coverage argument, sensitivity analysis, or comparison to omitted methods. Because the headline claims that stealing is easy at medium budgets and that most defenses fail rest entirely on results from this specific set, the absence of justification for representativeness directly weakens generalization to real-world GNN services.
  2. [§5.3] §5.3 (Joint Attack-and-Defense Track): The observation that watermarks lose most verification signal on surrogates is central to the defense-effectiveness claim, yet the manuscript does not specify the exact verification threshold, how it is applied uniformly across methods, or whether results include multiple random seeds; without these details the reported gap could be sensitive to post-hoc choices.
  3. [Table 2] Table 2 (Fidelity vs. Budget): The statement that heterophilic graphs are systematically harder to steal would be stronger if accompanied by statistical tests or variance across query selections, as single-run fidelity numbers may not establish the difference reliably.
minor comments (3)
  1. [Abstract] Abstract: The phrase 'several watermarks verify reliably' would be clearer if the specific watermarking methods were enumerated.
  2. [Figure 4] Figure 4: The x-axis scaling on query-budget plots makes it difficult to visually separate the medium-budget regime where the main claims are made; a log scale or inset would improve readability.
  3. [§2] §2 (Related Work): A few 2023-2024 GNN extraction papers are missing; adding them would better situate the benchmark.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive comments and for recognizing the value of the unified benchmark and joint attack-defense track. We address each major point below and will incorporate revisions to improve clarity and rigor.

read point-by-point responses

  1. Referee: The selection of the twelve attacks, twelve defenses, ten graphs, three backbones, and three tasks lacks any coverage argument, sensitivity analysis, or comparison to omitted methods. The headline claims rest on this specific set, weakening generalization.

    Authors: We selected components to represent the primary families in the GNN extraction literature (query-efficient, gradient-based attacks; watermarking, perturbation, and detection defenses) while covering homophilic, heterophilic, and large-scale graphs. We will add a dedicated paragraph in §3 with references to prior surveys justifying representativeness and a limited sensitivity check on two omitted methods. Full enumeration of every variant is infeasible, but the chosen set enables direct comparison under a consistent protocol. revision: yes

  2. Referee: The observation that watermarks lose verification signal on surrogates lacks the exact verification threshold, uniform application details, and multiple random seeds; results may be sensitive to post-hoc choices.

    Authors: We agree these details are essential. In the revision we will state the precise threshold for each watermark (taken from the original papers), confirm uniform application, and report means and standard deviations over five random seeds in §5.3 and the experimental setup. This will demonstrate that the observed verification drop is robust rather than threshold-dependent. revision: yes

  3. Referee: The claim that heterophilic graphs are systematically harder to steal would be stronger with statistical tests or variance across query selections, as single-run fidelity numbers may not establish the difference reliably.

    Authors: We will augment Table 2 and §5 with fidelity variance across three independent query-selection seeds and add paired t-tests comparing homophilic versus heterophilic graphs at each budget. These additions will quantify the systematic gap with statistical support. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical benchmark relies on external measurements

full rationale

The paper introduces GraphIP-Bench as a unified empirical evaluation of 12 attacks, 12 defenses, 10 public graphs, 3 backbones and 3 tasks under a fixed black-box protocol. All reported results (fidelity, utility, verification success, cost) are obtained by direct execution on standard public datasets and open-source GNN implementations; no equations, fitted parameters, or predictions are derived from the benchmark itself. Central claims rest on observed experimental outcomes rather than any self-referential reduction, self-citation chain, or ansatz smuggled from prior author work. The benchmark is therefore self-contained against external, reproducible inputs.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

The benchmark rests on standard public graph datasets, off-the-shelf GNN implementations, and conventional black-box query assumptions; no new free parameters, axioms, or invented entities are introduced.

pith-pipeline@v0.9.0 · 5646 in / 1122 out tokens · 34125 ms · 2026-05-14T19:22:32.061097+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.