Pith. sign in

REVIEW 4 major objections 5 minor 28 references

Are GNNs Actually Effective for Multimodal Fault Diagnosis in Microservice Systems?

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Pith's one-line read Plain MLP matches or beats GNNs in microservice fault diagnosis

desk verdict Good question, sensible baseline, but the ablation isn't controlled enough to back the headline claim. read the letter →

arxiv 2501.02766 v2 pith:2IT7EJTS submitted 2025-01-06 cs.SE cs.AI

classification cs.SEcs.AI
keywords microservicefaultdiagnosisgraphneuralnetworksmultimodaltelemetrymultilayerperceptronbaselineablationstudylocalizationdependencypreprocessing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether graph neural networks actually earn their place in multimodal fault diagnosis for microservice systems. The authors build DiagMLP, a deliberately minimal, topology-agnostic baseline that keeps the same preprocessing, embedding, and task modules as existing GNN-based pipelines but replaces the graph module with two concatenation-based MLPs. Across five public datasets covering fault detection, localization, and classification, DiagMLP matches or outperforms state-of-the-art GNN-based methods. The paper's claim is that the reported gains of GNN frameworks come mostly from preprocessing and multimodal fusion, not from modeling the service dependency graph. If correct, this means simple MLP baselines and standardized evaluation protocols should be standard before any graph-based innovation is claimed.

What carries the argument

The central object is DiagMLP, an ablation baseline defined by two single-layer MLPs joined by concatenation. It takes per-node multimodal embeddings for metrics, logs, and traces, appends a learnable position embedding per node, fuses the modalities node-by-node, then concatenates all nodes and fuses them into a system-level representation for detection, localization, and classification. Its role is to replace a GNN module inside an existing pipeline with a topology-agnostic module that retains all multimodal fusion, so any performance difference can be attributed to graph modeling alone. Learnable position embeddings give nodes distinct identities without an explicit dependency graph, and the single-layer design keeps the baseline deliberately minimal, so parity with GNNs is evidence against the value of topology.

What would settle it

Re-run the original GNN frameworks (Eadro, TVDiag, DeepHunt) under the paper's corrected protocol—fixed window splitting and an added validation set—and compare against DiagMLP on the same five datasets; if any baseline then beats DiagMLP by a clear margin, the parity result is an artifact of protocol fixes rather than evidence against graph modeling. A complementary test on a synthetic dataset with faults injected along known dependency paths would show whether topology ever matters.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that graph structure is not the source of performance in current GNN-based fault diagnosis pipelines. DiagMLP is inserted into Eadro, TVDiag, and DeepHunt in place of their GNN modules—graph attention, GraphSAGE, and graph autoencoder backbones—while every other component stays fixed. The resulting model performs at parity or better: fault detection F1 of 96.7 percent versus 92.1 percent on one dataset and 90.8 percent versus 90.7 percent on another; classification within one standard deviation on a third; and localization Top-1 accuracy that is substantially higher on the two largest comparisons. UMAP visualizations show that preprocessed multimodal features already separate fault classes by root cause before any graph modeling, and that neither MLP nor GNN embeddings add much separation. The authors conclude that preprocessing pipelines already encode critical dependency information, making explicit graph modeling at best redundant and at worst a source of instability.

Load-bearing premise

The conclusion rests on the fairness of the ablation: DiagMLP and each GNN baseline must differ only in the graph module, with identical preprocessing, embeddings, data splits, and evaluation, so that observed parity reflects the redundancy of graph modeling rather than protocol fixes.

Editorial extensions

If this is right

  • If the central claim is right, reported accuracy gains in recent GNN-based fault diagnosis papers should be re-assigned to preprocessing and multimodal fusion, not to graph architecture.
  • Simple MLP baselines like DiagMLP should be added to every future fault-diagnosis evaluation before a GNN is claimed to help.
  • Standardized evaluation protocols—including fixed window splitting, validation sets, and re-running baselines under identical conditions—are needed to prevent preprocessing artifacts from masquerading as model gains.
  • Larger datasets with realistic dependency-driven fault propagation are needed to test whether any architecture can exploit topology; current datasets with dozens of service instances may simply be too small for graphs to matter.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct testable extension would be to construct synthetic microservice data where faults are injected along known dependency edges; if DiagMLP still matches GNNs there, the case for graph modeling shrinks further, and if it fails there, that defines the regime where GNNs earn their complexity.
  • The parity result, if it generalizes, suggests that many multimodal deep-learning pipelines may be carrying their performance in feature preprocessing and fusion, a hypothesis worth probing in adjacent domains such as cloud infrastructure monitoring and network failure diagnosis.
  • The learnable position embeddings are a quiet part of the result: they give the MLP a way to distinguish nodes without topology, so a follow-up could test whether those embeddings implicitly recover service-dependency structure by comparing them to the true dependency graph.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes DiagMLP, a deliberately topology-agnostic MLP baseline for multimodal fault diagnosis in microservice systems, and evaluates whether GNN modules in existing frameworks add value beyond preprocessing and multimodal fusion. The authors replace the GNN components in Eadro, TVDiag, and DeepHunt with DiagMLP while purportedly keeping all other pipeline elements, and compare performance on five datasets (SN, TT, GAIA, D1, D2) across fault detection, localization, and classification. They report that DiagMLP matches or exceeds the GNN-based baselines, concluding that graph structures contribute marginally to diagnostic performance. UMAP visualizations are used both as motivation and as supporting evidence.

Significance. If substantiated, this is a valuable negative result for the microservice fault diagnosis community. It challenges the prevailing assumption that explicit service-dependency graphs are necessary, and it advocates for standardized baselines and protocols. The paper is well-motivated and the DiagMLP design is admirably simple. However, the current experimental evidence does not support the central claim because the ablation is not fully controlled: protocol changes (window-splitting correction, validation set, reimplemented preprocessing) are conflated with the removal of the GNN module. The significance of the paper depends on whether the authors can provide a rigorous controlled comparison.

major comments (4)
  1. [Section IV.A, Notes (3) and Table IV footnote] The central ablation is not controlled. The paper states that it 'rectified issues in prior frameworks, such as the window-splitting method in Eadro, which risked data leakage, and incorporated a validation set,' while Table IV's footnote says that results without standard deviations are 'directly extracted from the cited papers.' This means several baseline numbers (e.g., DiagFusion, CHASE, and possibly parts of DeepHunt and TVDiag) were produced under the original, potentially flawed protocol, whereas DiagMLP was evaluated under the corrected protocol. The large SN Top-1 localization gap (80.2±3.1% vs. 41.8±14.5%) is exactly the kind of difference that fixing a data-leaking window split could produce. To support the claim that GNN modules contribute marginally, all baselines must be re-run under the identical corrected protocol (same window splitting, same validation set) or the claims must be explicitly restricted to the re-run baselines.
  2. [Section IV.A(b)] For SN and TT, the paper says it 'reimplemented preprocessing code due to missing original scripts.' This implies the input features for DiagMLP and for the Eadro baseline may differ from those in the original Eadro paper. If the Eadro results in Tables II and IV are taken from the original publication, then the comparison conflates preprocessing differences with architecture differences, contradicting the paper's statement in Section IV that 'we preserve all other pipeline elements.' The authors should either demonstrate that their reimplemented preprocessing reproduces the original Eadro results (a reproducibility check) or re-run Eadro using the reimplemented preprocessing under the corrected protocol.
  3. [Tables II and III] The detection and classification results are reported without standard deviations, yet the text in Section IV.B claims 'differences within standard deviations' for GAIA classification and treats the SN/TT detection improvements as meaningful. A parity claim is a negative result and requires statistical evidence that the observed differences are not significant. Please provide standard deviations, confidence intervals, or significance tests for all key comparisons, especially the detection results in Table II where DiagMLP appears to outperform Eadro by a large margin (F1 96.7 vs. 92.1 on SN).
  4. [Section III.B and Section IV.C] The UMAP visualizations in Figs. 3 and 5 are used both to motivate the DiagMLP design and to explain the observed parity (see 'Key Insight 2'). This is circular: the same plots that justify the baseline are offered as evidence that GNNs do not improve separability. Moreover, visual inspection is not a quantitative evaluation. Please report objective separability metrics (e.g., silhouette score, k-nearest-neighbor accuracy) computed on the preprocessed input features, MLP embeddings, and GNN embeddings to support the claim that preprocessing already encodes dependency information.
minor comments (5)
  1. [Section I] In the sentence 'we introduce DiagMLP, a ablation experimental baseline,' the article should be 'an ablation experimental baseline.'
  2. [Section III.B] The sentence 'we project Xt from from R3dN to R2' contains a duplicated 'from.'
  3. [References] Reference [13] has a typo: 'F ound.' should be 'Found.'
  4. [Table II] The detection results in Table II lack standard deviations; if available from the re-run experiments, please include them to allow readers to assess variability.
  5. [Figure 5] The three panels in Figure 5 are not labeled with (a), (b), (c) in the caption or on the figures themselves; please add labels for clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation; the parity claim is an empirical ablation result, and the protocol concerns are experimental confounds rather than circularity.

full rationale

The paper's central claim, that DiagMLP matches GNN-based performance, is an empirical claim established by running a new model on published datasets. No equation in the paper defines DiagMLP's output in terms of the GNN baselines' outputs; Eq. (2) and Eq. (3) simply define an MLP. There is no fitted parameter that is then renamed as a prediction, no uniqueness theorem imported from the authors' prior work, and no ansatz smuggled in via citation. The UMAP visualizations (Fig. 3 and Fig. 5) are used both to motivate and to explain the result, but they are exploratory evidence rather than a derivation, so this is consistency, not circularity. The paper does cite prior work by overlapping authors, but none of these citations carries the central argument; the parity claim is supported by the experimental tables. The concern raised in the review about changed preprocessing and validation protocol (Section IV.A note (3) and the Table IV footnote) is a threat to the validity of the comparison, not a circularity: it concerns whether GNN baselines were evaluated under the same protocol, not whether the conclusion is presupposed by its inputs. Therefore no circular step can be identified under the required standard.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper's central claim rests on experimental control (identical protocol across methods), on the representativeness of five small datasets, and on qualitative UMAP evidence that preprocessed features separate faults. The MLP itself introduces learned position embeddings per node and unspecified training hyperparameters, which are free choices not disclosed.

free parameters (2)
  • Per-node learnable position embedding p_i = learned during training; dimension equals embedding dimension d
    DiagMLP assigns a learnable vector to each service instance, allowing the model to identify nodes without topology. This is a deliberate design choice and a source of parameters that may substitute for graph structure (Eq. (2), Section III-C).
  • Fusion MLP hidden dimension, dropout rate, and training hyperparameters = not reported in the paper
    The MLP architecture is described structurally as a single layer with LayerNorm, ReLU, and dropout, but concrete capacity and training settings are not given, so a replication would have to choose them by hand.
assumptions (3)
  • domain assumption The published baseline numbers for Eadro, TVDiag, and DeepHunt are directly comparable to the re-run DiagMLP results despite protocol changes (window splitting, validation set).
    The paper mentions rectifying Eadro's window-splitting and adding validation (Section IV.A, note 3), yet several baseline values in Tables II and IV are extracted from cited papers. This assumption underlies every parity or superiority claim.
  • domain assumption UMAP visualizations are a valid indicator of class separability and hence of whether topology modeling adds information.
    The motivating example and Section IV.C use qualitative UMAP plots to argue that preprocessed features already separate root causes; no quantitative separability metric is given.
  • domain assumption The five datasets (SN, TT, GAIA, D1, D2) are representative enough to support a general claim about GNN utility in microservice fault diagnosis.
    All datasets have at most 46 service instances, and larger proprietary datasets were excluded; the paper acknowledges this but still concludes GNN modules contribute marginally.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Are GNNs Actually Effective for Multimodal Fault Diagnosis in Microservice Systems?." pith.science (2026). https://pith.science/paper/2IT7EJTS

@misc{pith2026250102766,
  author       = {Pith},
  title        = {Pith review of: Are GNNs Actually Effective for Multimodal Fault Diagnosis in Microservice Systems?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2IT7EJTS}},
  note         = {Machine review of arXiv:2501.02766}
}
read the original abstract

Graph Neural Networks (GNNs) are widely adopted for fault diagnosis in microservice systems, premised on their ability to model service dependencies. However, the necessity of explicit graph structures remains underexamined, as existing evaluations conflate preprocessing with architectural contributions. To isolate the true value of GNNs, we propose DiagMLP, a deliberately minimal, topology-agnostic baseline that retains multimodal fusion capabilities while excluding graph modeling. Through ablation experiments across five datasets, DiagMLP achieves performance parity with state-of-the-art GNN-based methods in fault detection, localization, and classification. These findings challenge the prevailing assumption that graph structures are indispensable, revealing that: (i) preprocessing pipelines already encode critical dependency information, and (ii) GNN modules contribute marginally beyond multimodality fusion. Our work advocates for systematic re-evaluation of architectural complexity and highlights the need for standardized baseline protocols to validate model innovations.

Figures

Figures reproduced from arXiv: 2501.02766 by the authors.

Figure 1
Figure 1. Overview of GNNs ablation methods. (a) shows fault diagnosis using GNNs as the backbone to model the service dependency graphs. (b) shows the model with GNNs replaced by a topology-agnostic MLPs while keeping the other components. pipelines as GNN-based methods, DiagMLP preserves the multimodal fusion capabilities inherent to existing frameworks while explicitly removing service dependency graph mod￾eling. Furthermo… view at source ↗
Figure 2
Figure 2. Pipeline of Existing GNN-Based Multimodal Fault Diagnosis Models. (a) Preprocessing transforms raw multimodal data (metrics, logs, traces) into standardized formats, such as time series, text, and graphs; (b) Embedding modules, often pretrained, encode these features into dense representations; (c) GNN modules, varying across methods, integrate multimodal features with dependency graphs for diagnosis. II. GNN-BASED … view at source ↗
Figure 3
Figure 3. Motivating Example. UMAP visualization of preprocessed multi￾modal features from the SN and TT datasets [2]. Each point represents a fault window, colored by its root cause, showing clear clustering without topology modeling. pipeline, without imposing restrictions on the specific methods employed. The goal is to learn a mapping f : {xi} N i=1 → R c that determines whether the system is anomalous (for c = 2), identi… view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: UMAP Visualization of GAIA Dataset. Fault windows are depicted using node-wise Original Features (from FastText), MLP Embeddings (from DiagMLP), and GNN Embeddings (from TVDiag). Key Insight 1: The superior or comparable performance of DiagMLP highlights the limited be…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 24 canonical work pages

  1. [1]

    R. Xin, P. Chen, and Z. Zhao, ‘Causalrca: Causal inference based pre- cise fine-grained root cause localization for microservice applications’, Journal of Systems and Software , vol. 203, p. 111724, 2023

  2. [2]

    Eadro: An End-to-End Troubleshooting Framework for Microservices on Multi-Source Data,

    C. Lee, T. Yang, Z. Chen, Y . Su, and M. R. Lyu, "Eadro: An End-to-End Troubleshooting Framework for Microservices on Multi-Source Data," in Proc. 45th Int. Conf. Softw. Eng. , 2023, pp. 1750–1762

  3. [3]

    Interpretable Failure Localization for Microservice Sys- tems Based on Graph Autoencoder,

    Y . Sun et al., "Interpretable Failure Localization for Microservice Sys- tems Based on Graph Autoencoder," ACM Trans. Softw. Eng. Methodol., 2024

  4. [4]

    Fault-Aware Service Scheduling Optimization Frame- work in Edge Data Center,

    Y . Zhang et al., "Fault-Aware Service Scheduling Optimization Frame- work in Edge Data Center," in Proc. 2024 IEEE 21st Int. Conf. Mobile Ad-Hoc Smart Syst. (MASS) , 2024, pp. 514–519

  5. [5]

    TVDiag: A Task-oriented and View-invariant Fail- ure Diagnosis Framework with Multimodal Data,

    S. Xie et al., "TVDiag: A Task-oriented and View-invariant Fail- ure Diagnosis Framework with Multimodal Data," arXiv preprint arXiv:2407.19711, 2024

  6. [6]

    Robust Failure Diagnosis of Microservice System Through Multimodal Data,

    S. Zhang et al., "Robust Failure Diagnosis of Microservice System Through Multimodal Data," IEEE Trans. Serv. Comput.(TSC) , vol. 16, no. 6, pp. 3851–3864, 2023

  7. [7]

    CHASE: A Causal Heterogeneous Graph based Frame- work for Root Cause Analysis in Multimodal Microservice Systems,

    Z. Zhao et al., "CHASE: A Causal Heterogeneous Graph based Frame- work for Root Cause Analysis in Multimodal Microservice Systems," arXiv preprint arXiv:2406.19711 , 2024

  8. [8]

    Graph neural networks: A review of methods and applications,

    J. Zhou et al., "Graph neural networks: A review of methods and applications," AI Open , vol. 1, pp. 57–81, 2020

Show all 28 references
  1. [9]

    DeepTraLog: Trace-Log Combined Microservice Anomaly Detection through Graph-based Deep Learning,

    C. Zhang et al., "DeepTraLog: Trace-Log Combined Microservice Anomaly Detection through Graph-based Deep Learning," in Proc. 2022 IEEE/ACM 44th Int. Conf. Softw. Eng. (ICSE) , 2022, pp. 623–634

  2. [10]

    Twin Graph-Based Anomaly Detection via Attentive Multi-Modal Learning for Microser- vice System,

    J. Huang, Y . Yang, H. Yu, J. Li, and X. Zheng, "Twin Graph-Based Anomaly Detection via Attentive Multi-Modal Learning for Microser- vice System," in Proc. 2023 38th IEEE/ACM Int. Conf. Autom. Softw. Eng. (ASE) , 2023, pp. 66–78

  3. [11]

    Drain: An Online Log Parsing Approach with Fixed Depth Tree,

    P. He, J. Zhu, Z. Zheng, and M. R. Lyu, "Drain: An Online Log Parsing Approach with Fixed Depth Tree," in Proc. 2017 IEEE Int. Conf. Web Services (ICWS) , 2017, pp. 33–40

  4. [12]

    MULAN: Multi-modal Causal Structure Learning and Root Cause Analysis for Microservice Systems,

    L. Zheng, Z. Chen, J. He, and H. Chen, "MULAN: Multi-modal Causal Structure Learning and Root Cause Analysis for Microservice Systems," in Proc. ACM Web Conf. 2024 , 2024, pp. 4107–4116

  5. [13]

    Nezha: Interpretable Fine-Grained Root Causes Analysis for Microservices on Multi-modal Observability Data,

    G. Yu, P. Chen, Y . Li, H. Chen, X. Li, and Z. Zheng, "Nezha: Interpretable Fine-Grained Root Causes Analysis for Microservices on Multi-modal Observability Data," in Proc. 31st ACM Joint Eur . Softw. Eng. Conf. Symp. F ound. Softw. Eng. , 2023, pp. 553–565

  6. [14]

    An empirical evaluation of generic convolutional and recurrent networks for sequence modeling,

    S. Bai, J. Z. Kolter, and V . Koltun, "An empirical evaluation of generic convolutional and recurrent networks for sequence modeling," arXiv preprint arXiv:1803.01271, 2018

  7. [15]

    Attention is All you Need,

    A. Vaswani et al., "Attention is All you Need," in Proc. Neural Inf. Process. Syst. (NeurIPS) , 2017

  8. [16]

    Enriching Word Vectors with Subword Information,

    P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov, "Enriching Word Vectors with Subword Information," Trans. Assoc. Comput. Linguistics , vol. 5, pp. 135–146, 2017

  9. [17]

    Glove: Global vectors for word representation,

    J. Pennington, R. Socher, and C. D. Manning, "Glove: Global vectors for word representation," in Proc. 2014 Conf. Empir . Methods Nat. Lang. Process. (EMNLP), 2014, pp. 1532–1543

  10. [18]

    BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding," in Proc. 2019 Conf. North Amer . Chapter Assoc. Comput. Linguistics: Human Lang. Technol. , V ol. 1, 2019, pp. 4171–4186

  11. [19]

    DGERCL: A Dynamic Graph Embedding Approach for Root Cause Localization in Microser- vice Systems,

    H. Cheng, Q. Li, B. Liu, S. Liu, and L. Pan, "DGERCL: A Dynamic Graph Embedding Approach for Root Cause Localization in Microser- vice Systems," IEEE Trans. Serv. Comput. (TSC) , vol. 17, no. 1, pp. 1–12, 2024

  12. [20]

    Deep sets,

    M. Zaheer, S. Kottur, S. Ravanbakhsh, B. Poczos, R. R. Salakhutdinov, and A. J. Smola, "Deep sets," Adv. Neural Inf. Process. Syst. (NeurIPS) , vol. 30, 2017

  13. [21]

    Set transformer: A framework for attention-based permutation-invariant neu- ral networks,

    J. Lee, Y . Lee, J. Kim, A. Kosiorek, S. Choi, and Y . W. Teh, "Set transformer: A framework for attention-based permutation-invariant neu- ral networks," in Proc. Int. Conf. Mach. Learn. (ICML) , 2019, pp. 3744–3753

  14. [22]

    Characterizing Microservice Dependency and Perfor- mance: Alibaba Trace Analysis,

    S. Luo et al., "Characterizing Microservice Dependency and Perfor- mance: Alibaba Trace Analysis," in Proc. ACM Symp. Cloud Comput. , 2021, pp. 412–426

  15. [23]

    CloudRCA: A Root Cause Analysis Framework for Cloud Computing Platforms,

    Y . Zhang et al., "CloudRCA: A Root Cause Analysis Framework for Cloud Computing Platforms," in Proc. 30th ACM Int. Conf. Inf. Knowl. Manage. (CIKM), 2021, pp. 4373–4382

  16. [24]

    Failure Diagnosis in Microservice Systems: A Compre- hensive Survey and Analysis,

    S. Zhang et al., "Failure Diagnosis in Microservice Systems: A Compre- hensive Survey and Analysis," arXiv preprint arXiv:2407.01710 , 2024

  17. [25]

    Graph Neural Networks with Learnable Structural and Positional Representa- tions,

    V . P. Dwivedi, A. T. Luu, T. Laurent, Y . Bengio, and X. Bresson, "Graph Neural Networks with Learnable Structural and Positional Representa- tions," in Proc. Int. Conf. Learn. Represent. (ICLR) , 2022

  18. [26]

    Inductive Representation Learning on Large Graphs,

    W. Hamilton, Z. Ying, and J. Leskovec, "Inductive Representation Learning on Large Graphs," in Adv. Neural Inf. Process. Syst. (NeurIPS) , vol. 30, 2017

  19. [27]

    Graph Attention Networks,

    P. Veli ˇckovi´c, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y . Bengio, "Graph Attention Networks," in Proc. Int. Conf. Learn. Represent. (ICLR), 2018

  20. [28]

    UMAP: Uniform Manifold Approximation and Projection,

    L. McInnes, J. Healy, N. Saul, and L. Großberger, "UMAP: Uniform Manifold Approximation and Projection," J. Open Source Softw. , vol. 3, no. 29, p. 861, 2018

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.