pith. sign in

arxiv: 2605.07133 · v1 · submitted 2026-05-08 · 💻 cs.LG · cs.AI

GAD in the Wild: Benchmarking Graph Anomaly Detection under Realistic Deployment Challenges

Jingjing Zhou , Shiyu Huang , Qing Qing , Zuquan Yuan , Huafei Huang , Ziqi Xu , Mingliang Hou , Xikun Zhang
show 2 more authors
Renqiang Luo Ivan Lee
This is my paper

Pith reviewed 2026-05-11 02:13 UTC · model grok-4.3

classification 💻 cs.LG cs.AI
keywords graph anomaly detectionbenchmarkinglarge-scale graphsanomaly scarcitymissing attributesgraph neural networksscalabilityreal-world deployment
0
0 comments X

The pith

Graph anomaly detection models fail on large-scale graphs with rare anomalies and missing attributes

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper shows that graph anomaly detection models successful in small settings struggle when faced with million-node graphs, very rare anomalies, and missing node attributes. Using variants of five graphs including two industrial ones over 3.7 million nodes, nine models are tested and found to have memory problems, near-zero detection rates at 0.1% anomaly levels, and high dependence on how missing data is handled. This gap between lab and real conditions means current methods may not work for practical uses such as fraud detection on large networks. The authors release their benchmark to encourage creation of more reliable GAD systems.

Core claim

We derive a family of controlled benchmark variants from five diverse graphs, including two native industrial-scale datasets with over 3.7 million nodes. Our extensive evaluation of nine representative GAD models reveals three major limitations: most GNN-based methods fail to scale to million-node graphs due to prohibitive memory requirements, detection performance drops sharply under realistic anomaly ratios often resulting in zero recall, and reconstruction-based models are highly sensitive to attribute imputation strategies. Our findings suggest that strong performance in laboratory settings does not guarantee robustness in production environments.

What carries the argument

The family of controlled benchmark variants derived from five diverse graphs that isolate million-scale size, extreme anomaly scarcity, and missing node attributes for systematic testing.

If this is right

  • GNN-based methods require new techniques to reduce memory use for million-node graphs.
  • Evaluation must use realistic low anomaly ratios to avoid inflated performance estimates.
  • Reconstruction-based models need imputation approaches that do not dominate their results.
  • The released benchmark serves as a testbed for building GAD systems that work on imperfect large graphs.
  • Practical applications like fraud detection should re-test models under deployment conditions rather than lab ones.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Sampling or distributed methods could be tested to address the memory barriers for large graphs.
  • Tighter integration of attribute handling within detection models might lessen sensitivity to imputation.
  • The benchmarking method could extend to other graph learning tasks facing similar scale and data issues.
  • Persistent limitations might favor simpler scalable detectors over complex neural approaches in practice.

Load-bearing premise

The controlled benchmark variants from the five graphs accurately represent the main real-world deployment challenges of very large size, extreme anomaly rarity, and missing node attributes.

What would settle it

A GAD model that processes the 3.7 million node graphs within standard memory limits, maintains non-zero recall at 0.1% anomaly ratio, and shows stable results across different attribute imputation strategies would show the reported limitations do not hold universally.

Figures

Figures reproduced from arXiv: 2605.07133 by Huafei Huang, Ivan Lee, Jingjing Zhou, Mingliang Hou, Qing Qing, Renqiang Luo, Shiyu Huang, Xikun Zhang, Ziqi Xu, Zuquan Yuan.

Figure 1
Figure 1. Figure 1: Anomaly detection results across datasets based on varying anomaly ratios. Only success [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Anomaly detection results across datasets under varying missing rate. [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
read the original abstract

Graph Anomaly Detection (GAD) is a critical task in graph machine learning with vital applications in financial fraud detection and social platform governance. However, existing GAD benchmarks are often restricted to small-scale, curated graphs with relatively balanced anomaly ratios, leaving a substantial gap between academic evaluation and real-world deployment. To bridge this gap, we present a multi-dimensional benchmark that systematically evaluates GAD models under three deployment-relevant challenges: million-scale graphs, extreme anomaly scarcity, and missing node attributes. We derive a family of controlled benchmark variants from five diverse graphs, including two native industrial-scale datasets with over 3.7 million nodes. Our extensive evaluation of nine representative GAD models reveals three major limitations: (1) most GNN-based methods fail to scale to million-node graphs due to prohibitive memory requirements; (2) detection performance drops sharply under realistic anomaly ratios (e.g., 0.1\%), often resulting in zero recall; and (3) reconstruction-based models are highly sensitive to attribute imputation strategies. Our findings suggest that strong performance in laboratory settings does not guarantee robustness in production environments. We release this benchmark and empirical evaluation as a diagnostic testbed to promote the development of robust and scalable GAD systems for large-scale, imperfect graphs encountered in practice. Code is available at https://anonymous.4open.science/r/Benchmark_GAD-E7A3.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper introduces a multi-dimensional benchmark for Graph Anomaly Detection (GAD) derived from five source graphs (including two industrial datasets exceeding 3.7 million nodes). It systematically generates controlled variants to evaluate nine representative GAD models under three deployment challenges: million-scale size, extreme anomaly scarcity (e.g., 0.1% ratios), and missing node attributes. The evaluation reports that most GNN-based methods fail to scale due to memory limits, detection performance (including recall) drops sharply at realistic anomaly ratios, and reconstruction-based models are sensitive to attribute imputation strategies. The authors release the benchmark and code as a diagnostic testbed.

Significance. If the benchmark variants faithfully instantiate the claimed deployment conditions, the work is significant for highlighting gaps between laboratory GAD performance and real-world robustness in applications like fraud detection. Explicit credit is due for the release of reproducible code and the benchmark itself, which enables future falsifiable testing. The empirical scope across diverse graphs and models provides a concrete diagnostic resource, though its impact hinges on the representativeness of the constructed variants.

major comments (2)
  1. [Benchmark variant derivation] Benchmark construction (described in the section deriving variants from the five source graphs): the paper does not report any statistical validation (e.g., comparison of homophily, degree distributions, or community structure around anomalies) that the 0.1% anomaly-ratio variants—whether via subsampling or synthetic injection—preserve the placement properties of the original industrial graphs. Without such checks, the reported zero-recall outcomes may not securely demonstrate intrinsic model failure under realistic scarcity rather than an artifact of variant construction.
  2. [Missing node attributes evaluation] Attribute-missingness experiments (in the evaluation of reconstruction-based models): the sensitivity results are presented without explicit comparison of the missingness mechanism (MCAR vs. MNAR) to patterns observed in the native industrial datasets. This weakens the tie between the observed imputation sensitivity and the claimed deployment challenge of missing attributes.
minor comments (2)
  1. [Abstract and §3] The abstract and introduction could more precisely define the nine models and the exact anomaly injection procedure for the controlled variants to improve reproducibility.
  2. [Tables and figures] Table captions and figure legends should explicitly state the anomaly ratios and graph sizes used in each experiment for immediate clarity.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive and detailed review. We address each major comment point by point below and indicate the planned revisions.

read point-by-point responses

  1. Referee: [Benchmark variant derivation] Benchmark construction (described in the section deriving variants from the five source graphs): the paper does not report any statistical validation (e.g., comparison of homophily, degree distributions, or community structure around anomalies) that the 0.1% anomaly-ratio variants—whether via subsampling or synthetic injection—preserve the placement properties of the original industrial graphs. Without such checks, the reported zero-recall outcomes may not securely demonstrate intrinsic model failure under realistic scarcity rather than an artifact of variant construction.

    Authors: We thank the referee for this observation on benchmark fidelity. The variant construction employs controlled subsampling and injection that aim to retain original topology and anomaly placement by preserving degree sequences and local connectivity. We acknowledge that explicit statistical validations (e.g., homophily ratios, KS tests on degree distributions, and modularity around anomalies) were not reported in the submitted version. In the revision we will add a dedicated subsection presenting these quantitative comparisons between source graphs and derived variants to confirm that performance drops arise from the scarcity condition rather than construction artifacts. revision: yes

  2. Referee: [Missing node attributes evaluation] Attribute-missingness experiments (in the evaluation of reconstruction-based models): the sensitivity results are presented without explicit comparison of the missingness mechanism (MCAR vs. MNAR) to patterns observed in the native industrial datasets. This weakens the tie between the observed imputation sensitivity and the claimed deployment challenge of missing attributes.

    Authors: We agree that stronger linkage to the industrial missingness patterns would improve the deployment relevance of the results. The reported experiments used MCAR to isolate imputation effects in a controlled setting. In the revision we will add an analysis of available missingness statistics from the industrial graphs and include MNAR simulations derived from observed patterns (where metadata permits) to directly compare mechanisms and discuss implications for the sensitivity findings. revision: yes

Circularity Check

0 steps flagged

No circularity: pure empirical benchmarking with no derivations or fitted predictions

full rationale

This is a standard empirical benchmarking paper. It selects five source graphs (two industrial), constructs controlled variants to simulate million-scale size, low anomaly ratios, and missing attributes, then runs nine existing GAD models and reports observed performance drops. No equations, no fitted parameters renamed as predictions, no self-citation chains supporting core claims, and no ansatz or uniqueness theorems are invoked. The three reported limitations are direct experimental observations, not reductions to inputs by construction. The skeptic concern about variant fidelity is a validity question, not a circularity issue.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the domain assumption that the chosen graphs and derived variants faithfully represent realistic deployment conditions; no free parameters are fitted and no new entities are postulated because the contribution is evaluative rather than generative.

axioms (1)
  • domain assumption The selected graphs and derived variants represent realistic deployment scenarios for GAD.
    Invoked when constructing the multi-dimensional benchmark to bridge the gap between academic evaluation and real-world use.

pith-pipeline@v0.9.0 · 5571 in / 1305 out tokens · 48816 ms · 2026-05-11T02:13:26.440523+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

42 extracted references · 42 canonical work pages

  1. [1]

    Journal of Machine Learning Research , volume =

    Roel Bouman and Zaharah Bukhsh and Tom Heskes , title =. Journal of Machine Learning Research , volume =

    work page

  2. [2]

    Tsourakakis , title =

    Tianyi Chen and Charalampos E. Tsourakakis , title =. Proceedings of the

    work page

  3. [3]

    Proceedings of the

    Yuxuan Cao and Jiarong Xu and Chen Zhao and Jiaan Wang and Carl Ji Yang and Chunping Wang and Yang Yang , title =. Proceedings of the

    work page

  4. [4]

    Proceedings of the SIAM International Conference on Data Mining,

    Deep Anomaly Detection on Attributed Networks , author=. Proceedings of the SIAM International Conference on Data Mining,

    work page

  5. [5]

    Proceedings of the Fifteenth

    Yingtong Dou , title =. Proceedings of the Fifteenth

    work page

  6. [6]

    Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing,

    AnomalyDAE: Dual Autoencoder for Anomaly Detection on Attributed Networks , author=. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing,. 2020 , organization=

    work page 2020

  7. [7]

    Kipf and Max Welling , title =

    Thomas N. Kipf and Max Welling , title =. Proceedings of the International Conference on Learning Representations,

    work page

  8. [8]

    Advances in Neural Information Processing Systems , year =

    Sunwoo Kim and Soo Yong Lee and Fanchen Bu and Shinhwan Kang and Kyungho Kim and Jaemin Yoo and Kijung Shin , title =. Advances in Neural Information Processing Systems , year =

    work page

  9. [9]

    Foundations and Trends in Signal Processing , volume =

    Feng Xia and Ciyuan Peng and Jing Ren and Falih Gozi Febrinanto and Renqiang Luo and Vidya Saikrishna and Shuo Yu and Xiangjie Kong , title =. Foundations and Trends in Signal Processing , volume =

    work page

  10. [10]

    Xiangjie Kong and Siyue Shuai and Hui Wang and Guojiang Shen and Feng Xia , title =

    work page

  11. [11]

    Das , title =

    Xiangjie Kong and Wenyi Zhang and Hui Wang and Mingliang Hou and Xin Chen and Xiaoran Yan and Sajal K. Das , title =

    work page

  12. [12]

    IEEE Transactions on Neural Networks and Learning Systems , volume=

    Anomaly Detection on Attributed Networks via Contrastive Self-Supervised Learning , author=. IEEE Transactions on Neural Networks and Learning Systems , volume=. 2022 , publisher=

    work page 2022

  13. [13]

    Chen and Zhihao Jia and Philip S

    Kay Liu and Yingtong Dou and Yue Zhao and Xueying Ding and Xiyang Hu and Ruitong Zhang and Kaize Ding and Canyu Chen and Hao Peng and Kai Shu and Lichao Sun and Jundong Li and George H. Chen and Zhihao Jia and Philip S. Yu , title =. Advances in Neural Information Processing Systems , year =

    work page

  14. [14]

    Proceedings of the

    Renqiang Luo and Huafei Huang and Tao Tang and Jing Ren and Ziqi Xu and Mingliang Hou and Enyan Dai and Feng Xia , title =. Proceedings of the

    work page

  15. [15]

    Cheng Wang and Hangyu Zhu , title =

    work page

  16. [16]

    Neural Computing and Applications , volume=

    One-Class Graph Neural Networks for Anomaly Detection in Attributed Networks , author=. Neural Computing and Applications , volume=. 2021 , publisher=

    work page 2021

  17. [17]

    Proceedings of the ACM International Conference on Web Search and Data Mining,

    ComGA: Community-Aware Attributed Graph Anomaly Detection , author=. Proceedings of the ACM International Conference on Web Search and Data Mining,

    work page

  18. [18]

    Proceedings of the Seventeenth ACM International Conference on Web Search and Data Mining , pages=

    GAD-NR: Graph Anomaly Detection via Neighborhood Reconstruction , author=. Proceedings of the Seventeenth ACM International Conference on Web Search and Data Mining , pages=

    work page

  19. [19]

    Proceedings of the ACM on Web Conference,

    SmoothGNN: Smoothing-aware GNN for Unsupervised Node Anomaly Detection , author=. Proceedings of the ACM on Web Conference,

    work page

  20. [20]

    Neural Networks , volume=

    Graph Anomaly Detection Based on Hybrid Node Representation Learning , author=. Neural Networks , volume=. 2025 , publisher=

    work page 2025

  21. [21]

    Proceedings of the AAAI Conference on Artificial Intelligence , pages=

    A Label-free Heterophily-guided Approach for Unsupervised Graph Fraud Detection , author=. Proceedings of the AAAI Conference on Artificial Intelligence , pages=

    work page

  22. [22]

    Advances in Neural Information Processing Systems , volume=

    GADBench: Revisiting and Benchmarking Supervised Graph Anomaly Detection , author=. Advances in Neural Information Processing Systems , volume=

    work page

  23. [23]

    Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining,

    GraphWorld: Fake Graphs Bring Real Insights for GNNs , author=. Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining,

    work page

  24. [24]

    Advances in Neural Information Processing Systems , volume=

    GOOD: A Graph Out-of-Distribution Benchmark , author=. Advances in Neural Information Processing Systems , volume=

    work page

  25. [25]

    Jingye Zhao and Jianan Shen and Jianwei Wang and Tianyuan Zhou and Ruijia Wu and Kai Wang and Xuemin Lin , title =

    work page

  26. [26]

    Proceedings of the NeurIPS 2021 Datasets and Benchmarks Track , year=

    Graph Robustness Benchmark: Benchmarking the Adversarial Robustness of Graph Machine Learning , author=. Proceedings of the NeurIPS 2021 Datasets and Benchmarks Track , year=

    work page 2021

  27. [27]

    Reconstructed Graph Neural Network With Knowledge Distillation for Lightweight Anomaly Detection , journal =

    Xiaokang Zhou and Jiayi Wu and Wei Liang and Kevin I. Reconstructed Graph Neural Network With Knowledge Distillation for Lightweight Anomaly Detection , journal =

    work page

  28. [28]

    Proceedings of the International Wireless Communications and Mobile Computing,

    Qing Qing and Huafei Huang and Mingliang Hou and Renqiang Luo and Mohsen Guizani , title =. Proceedings of the International Wireless Communications and Mobile Computing,

    work page

  29. [29]

    Proceedings of the International Conference on Learning Representations,

    IGL-Bench: Establishing the Comprehensive Benchmark for Imbalanced Graph Learning , author=. Proceedings of the International Conference on Learning Representations,

    work page

  30. [30]

    Hezhe Qiao and Hanghang Tong and Bo An and Irwin King and Charu Aggarwal and Guansong Pang , title =

    work page

  31. [31]

    Advances in Neural Information Processing Systems , volume=

    Handling Missing Data with Graph Representation Learning , author=. Advances in Neural Information Processing Systems , volume=

    work page

  32. [32]

    Proceedings of the

    Xu Yuan and Na Zhou and Shuo Yu and Huafei Huang and Zhikui Chen and Feng Xia , title =. Proceedings of the

    work page

  33. [33]

    Proceedings of the AAAI Conference on Artificial Intelligence,

    Yiming Xu and Zhen Peng and Bin Shi and Xu Hua and Bo Dong and Song Wang and Chen Chen , title =. Proceedings of the AAAI Conference on Artificial Intelligence,

    work page

  34. [34]

    Aggarwal , title =

    Jing Ren and Feng Xia and Ivan Lee and Azadeh Noori Hoshyar and Charu C. Aggarwal , title =

    work page

  35. [35]

    Proceedings of the Learning on Graphs Conference , volume=

    On the Unreasonable Effectiveness of Feature Propagation in Learning on Graphs with Missing Node Features , author=. Proceedings of the Learning on Graphs Conference , volume=

    work page

  36. [36]

    Proceedings of the International Conference on Learning Representations,

    Confidence-Based Feature Imputation for Graphs with Partially Known Features , author=. Proceedings of the International Conference on Learning Representations,

    work page

  37. [37]

    Proceedings of the ACM International Conference on Information and Knowledge Management,

    Towards Fair Graph Anomaly Detection: Problem, Benchmark Datasets, and Evaluation , author=. Proceedings of the ACM International Conference on Information and Knowledge Management,

    work page

  38. [38]

    AI Magazine , volume=

    Collective Classification in Network Data , author=. AI Magazine , volume=

    work page

  39. [39]

    Expert Systems with Applications , volume=

    The Comparisons of Data Mining Techniques for the Predictive Accuracy of Probability of Default of Credit Card Clients , author=. Expert Systems with Applications , volume=. 2009 , publisher=

    work page 2009

  40. [40]

    Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining,

    Individual Fairness for Graph Neural Networks: A Ranking Based Approach , author=. Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining,

    work page

  41. [41]

    Advances in Neural Information Processing Systems , volume=

    DGraph: A Large-Scale Financial Dataset for Graph Anomaly Detection , author=. Advances in Neural Information Processing Systems , volume=

    work page

  42. [42]

    Proceedings of the International Conference on Machine Learning,

    Rethinking Graph Neural Networks for Anomaly Detection , author=. Proceedings of the International Conference on Machine Learning,

    work page