Pith. sign in

REVIEW 56 references

Balancing Graph Embedding Smoothness in Self-Supervised Learning via Information-Theoretic Decomposition

T0 review · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read BSG adds neighbor, minimal, and divergence losses to graph self-supervised learning, balancing embedding smoothness and improving downstream node and link tasks.

arxiv 2504.12011 v1 pith:SSKFW4XQ submitted 2025-04-16 cs.LG cs.AI

Heesoo Jung , Hogun Park This is my paper · ORCID
classification cs.LGcs.AI
keywords graphlossbalancingmethodssmoothnessexistinglearningrepresentation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Graph self-supervised learning trains a Graph Neural Network without labels by giving it a made-up task, such as hiding some edges and asking the model to reconstruct them. This paper starts from a standard information-theory identity: the information a learned representation carries about the pretext signal can be split into a part shared with neighboring nodes, a part explained by the reconstruction signal, and a part that only appears when neighbors are considered. From this split, the authors define three losses. The neighbor loss pulls a node's embedding toward the average embedding of its neighbors. The minimal loss pulls an embedding toward the representation obtained from a masked version of the graph. The divergence loss pushes apart nodes whose embeddings are already too similar to their neighbors. The first and third losses pull in opposite directions, which the authors say lets the model sit between over-smoothing, where all nodes become identical, and under-smoothing, where neighbors share no useful information.

The paper reports that adding these three losses to contrastive, feature-reconstruction, and edge-reconstruction graph SSL methods improves node classification and link prediction on standard benchmarks, and that the resulting embeddings have intermediate smoothness values. The experiments are broad: eight datasets for node classification, five for link prediction, and three for graph classification. The main limitations are that the theoretical proofs are sketchy, the balance point is tuned per dataset, and one graph-classification claim is contradicted by the paper's own table.

Extended reading notes

Core claim

The paper's load-bearing assertion is that decomposing the SSL objective into neighbor, minimal, and divergence terms, and balancing them with the BSG losses, improves performance across a wider range of downstream tasks. The abstract states: "A framework, BSG ... introduces novel loss functions designed to supplement the representation quality in graph-based SSL by balancing the derived three terms: neighbor loss, minimal loss, and divergence loss" and "Extensive experiments ... consistently demonstrate that BSG achieves state-of-the-art performance." If the paper is correct, adding these three losses to existing graph SSL objectives should improve node classification, link prediction, and graph classification without requiring labels.

Load-bearing premise

The downstream relevance of the whole decomposition rests on the multi-view assumption in Definition 1 (Section 3.2) that maximizing I(ZX;S) positively correlates with maximizing I(ZX;Y), and on the homophily assumption used in Theorem 2 (Section 4.8.1) that neighbors act as a distinct view of a node. The proof of Theorem 2 in Appendix A.4.2 additionally assumes a Markov chain S↔Y↔X→ZX and a Dirac condition P(Z_BSG_X|X), neither of which is justified for edge-masked graph SSL. If these conditions fail, for example on heterophilic graphs, the neighbor loss may increase smoothness without increasing task-relevant information.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

No new particles, mediators, or physical entities are introduced. The neighbor random variable ZN is a mathematical construct derived from the graph structure, not an independent postulated entity. The central claim relies instead on the assumptions above plus per-dataset tuned loss weights.

free parameters (5)
  • lambda_1 (neighbor loss weight) = 0.0002 (Cora), varies per dataset
    Grid-searched per dataset in A.2 over 0.0001 to 0.001; controls the I(ZX;ZN) neighbor term.
  • lambda_2 (minimal loss weight) = 0.001 (Cora), varies per dataset
    Chosen per dataset from {0.1, 0.01, 0.001, 0.0001, 0.00001} in A.2.
  • lambda_3 (divergence loss weight) = 0.0009 (Cora), varies per dataset
    Grid-searched per dataset over 0.0001 to 0.001 in A.2.
  • margin m in divergence loss = -0.2 (Cora), varies per dataset
    Tuned per dataset; Table 6 lists values from -0.4 to 0.5.
  • edge mask ratio = 0.7
    Set to 0.7 for all experiments, as stated in Section 4.6; not tuned but hand-chosen.
assumptions (5)
  • domain assumption Multi-view assumption: maximizing I(ZX;S) positively correlates with maximizing I(ZX;Y)
    Definition 1 in Section 3.2; load-bearing for using pretext-task mutual information to improve downstream tasks.
  • ad hoc to paper The Markov chain S↔Y↔X→ZX holds in self-supervised learning
    Invoked in A.4.2 proof of Theorem 2; not generally justified for edge-masked graphs.
  • domain assumption ZX and ZN are zero-mean, unit-variance Gaussian random vectors
    Used to derive Equation (8) from MSE; learned representations are not necessarily Gaussian in practice.
  • domain assumption The input graph is homophilic, so neighbors act as a distinct view of a node
    Used in Theorem 2 and Section 4.3; fails on heterophilic graphs.
  • domain assumption Small-MSE approximation MSE << Demb in Equation (8)
    The second-order term in Equation (25) is dropped; this may hold near the optimum but not early in training.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Balancing Graph Embedding Smoothness in Self-Supervised Learning via Information-Theoretic Decomposition." pith.science (2026). https://pith.science/paper/SSKFW4XQ

@misc{pith2026250412011,
  author       = {Pith},
  title        = {Pith review of: Balancing Graph Embedding Smoothness in Self-Supervised Learning via Information-Theoretic Decomposition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SSKFW4XQ}},
  note         = {Machine review of arXiv:2504.12011}
}
read the original abstract

Self-supervised learning (SSL) in graphs has garnered significant attention, particularly in employing Graph Neural Networks (GNNs) with pretext tasks initially designed for other domains, such as contrastive learning and feature reconstruction. However, it remains uncertain whether these methods effectively reflect essential graph properties, precisely representation similarity with its neighbors. We observe that existing methods position opposite ends of a spectrum driven by the graph embedding smoothness, with each end corresponding to outperformance on specific downstream tasks. Decomposing the SSL objective into three terms via an information-theoretic framework with a neighbor representation variable reveals that this polarization stems from an imbalance among the terms, which existing methods may not effectively maintain. Further insights suggest that balancing between the extremes can lead to improved performance across a wider range of downstream tasks. A framework, BSG (Balancing Smoothness in Graph SSL), introduces novel loss functions designed to supplement the representation quality in graph-based SSL by balancing the derived three terms: neighbor loss, minimal loss, and divergence loss. We present a theoretical analysis of the effects of these loss functions, highlighting their significance from both the SSL and graph smoothness perspectives. Extensive experiments on multiple real-world datasets across node classification and link prediction consistently demonstrate that BSG achieves state-of-the-art performance, outperforming existing methods. Our implementation code is available at https://github.com/steve30572/BSG.

Figures

Figures reproduced from arXiv: 2504.12011 by the authors.

Figure 1
Figure 1. A comparison of existing graph SSL baselines, with [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Illustration of the representations and loss functions of BSG. The figure first shows the process of obtaining three [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The effect of Lnei and Ldiv respect to graph smooth￾ness. The y-axis denotes the normalized graph embedding smoothness score, and the low values indicate oversmooth￾ing. loss functions: neighbor loss and divergence loss. The neighbor loss promotes graph smoothing, while the divergence loss coun￾teracts this effect, preventing the representation from becoming oversmoothed. To quantify the impact of each loss function… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Ablation study of the proposed loss functions. We [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Extended experiment of BSG in the Cora dataset, [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Extended experiment of BSG on the Cora dataset, [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

56 extracted references · 46 canonical work pages

  1. [1]

    Deli Chen, Yankai Lin, Wei Li, Peng Li, Jie Zhou, and Xu Sun. 2020. Measuring and relieving the over-smoothing problem for graph neural networks from the topological view. In Proceedings of the AAAI conference on artificial intelligence

    work page 2020

  2. [2]

    Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. 2020. Simple and deep graph convolutional networks. InProceedings of the International Conference on Machine Learning

    work page 2020

  3. [3]

    Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, and Guoping Hu

  4. [4]

    Kaveh Hassani and Amir Hosein Khasahmadi. 2020. Contrastive multi-view representation learning on graphs. In Proceedings of the International Conference on Machine Learning

    work page 2020

  5. [5]

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick

  6. [6]

    Zhenyu Hou, Yufei He, Yukuo Cen, Xiao Liu, Yuxiao Dong, Evgeny Kharlamov, and Jie Tang. 2023. GraphMAE2: A Decoding-Enhanced Masked Self-Supervised Graph Learner. In Proceedings of the ACM Web Conference

    work page 2023

  7. [7]

    Zhenyu Hou, Xiao Liu, Yukuo Cen, Yuxiao Dong, Hongxia Yang, Chunjie Wang, and Jie Tang. 2022. Graphmae: Self-supervised masked graph autoencoders. In Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining

    work page 2022

  8. [8]

    Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. 2020. Open graph benchmark: Datasets for machine learning on graphs. In Proceedings of the Advances in Neural Information Processing Systems

    work page 2020

Show all 56 references
  1. [9]

    Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Baner- jee, and Fillia Makedon. 2020. A survey on contrastive self-supervised learning. Technologies (2020)

    2020

  2. [10]

    Cheng Ji, Zixuan Huang, Qingyun Sun, Hao Peng, Xingcheng Fu, Qian Li, and Jianxin Li. 2024. ReGCL: Rethinking Message Passing in Graph Contrastive Learning. In Proceedings of the conference on Artifical Intelligence

    2024

  3. [11]

    Heesoo Jung Jongwon Park and Hogun Park. 2025. CIMAGE: Exploiting the Conditional Independence in Masked Graph Auto-encoders. In Proceedings of the ACM International Conference on Web Search and Data Mining

    2025

  4. [12]

    Nicolas Keriven. 2022. Not too little, not too much: a theoretical analysis of graph (over) smoothing. In Proceedings of the Advances in Neural Information Processing Systems

    2022

  5. [13]

    Minseon Kim, Jihoon Tack, and Sung Ju Hwang. 2020. Adversarial self-supervised contrastive learning. In Proceedings of the Advances in Neural Information Pro- cessing Systems

    2020

  6. [14]

    Thomas N Kipf and Max Welling. 2016. Variational graph auto-encoders. arXiv preprint arXiv:1611.07308 (2016)

    2016 arXiv

  7. [15]

    Thomas N Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In Proceedings of the International Conference on Learning Representations

    2017

  8. [16]

    Guohao Li, Matthias Muller, Ali Thabet, and Bernard Ghanem. 2019. Deepgcns: Can gcns go as deep as cnns?. In Proceedings of the IEEE/CVF Conference on Computer Vision

    2019

  9. [17]

    Jintang Li, Ruofan Wu, Wangbin Sun, Liang Chen, Sheng Tian, Liang Zhu, Changhua Meng, Zibin Zheng, and Weiqiang Wang. 2023. What’s Behind the Mask: Understanding Masked Graph Modeling for Graph Autoencoders. In Pro- ceedings of the ACM SIGKDD Conference on Knowledge Discovery ...

    2023

  10. [18]

    Qimai Li, Zhichao Han, and Xiao-Ming Wu. 2018. Deeper insights into graph con- volutional networks for semi-supervised learning. InProceedings of the conference on Artifical Intelligence

    2018

  11. [19]

    Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang. 2021. Self-supervised learning: Generative or contrastive.IEEE Transactions on Knowledge and Data Engineering (2021)

    2021

  12. [20]

    Yixin Liu, Ming Jin, Shirui Pan, Chuan Zhou, Yu Zheng, Feng Xia, and S Yu Philip

  13. [21]

    Annamalai Narayanan, Mahinthan Chandramohan, Rajasekar Venkatesan, Lihui Chen, Yang Liu, and Shantanu Jaiswal. 2017. graph2vec: Learning distributed representations of graphs. International Workshop on Mining and Learning with Graphs (2017)

    2017

  14. [22]

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018)

    2018 arXiv

  15. [23]

    IEEE Transactions on Knowledge and Data Engineering (2022)

    Graph self-supervised learning: A survey. IEEE Transactions on Knowledge and Data Engineering (2022)

    2022

  16. [24]

    Yu Rong, Wenbing Huang, Tingyang Xu, and Junzhou Huang. 2020. Dropedge: Towards deep graph convolutional networks on node classification. InProceedings of the International Conference on Learning Representations

    2020

  17. [25]

    Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad. 2008. Collective classification in network data. AI magazine (2008)

    2008

  18. [26]

    Shirui Pan, Ruiqi Hu, Guodong Long, Jing Jiang, Lina Yao, and Chengqi Zhang

  19. [27]

    Yucheng Shi, Yushun Dong, Qiaoyu Tan, Jundong Li, and Ninghao Liu. 2023. Gigamae: Generalizable graph masked autoencoder via collaborative latent space reconstruction. In Proceedings of the ACM International Conference on Information and Knowledge Management

    2023

  20. [28]

    Ravid Shwartz Ziv and Yann LeCun. 2024. To compress or not to compress—self- supervised learning and information theory: A review. Entropy (2024)

    2024

  21. [29]

    Karthik Sridharan and Sham M. Kakade. 2008. An Information Theoretic Frame- work for Multi-view Learning. In Proceedings of the Conference on Learning Theory

    2008

  22. [30]

    Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. 2018. Pitfalls of graph neural network evaluation. arXiv preprint arXiv:1811.05868 (2018)

    2018 arXiv

  23. [31]

    Qiaoyu Tan, Ninghao Liu, Xiao Huang, Soo-Hyun Choi, Li Li, Rui Chen, and Xia Hu. 2023. S2GAE: self-supervised graph autoencoders are generalizable learners with graph masking. In Proceedings of the ACM International Conference on Web Search and Data Mining

    2023

  24. [32]

    Naftali Tishby, Fernando C Pereira, and William Bialek. 2000. The information bottleneck method. arXiv preprint physics/0004057 (2000)

    2000 arXiv

  25. [33]

    Yao-Hung Hubert Tsai, Yue Wu, Ruslan Salakhutdinov, and Louis-Philippe Morency. 2021. Self-supervised Learning from a Multi-view Perspective. In Proceedings of the International Conference on Learning Representations

    2021

  26. [34]

    Fan-Yun Sun, Jordan Hoffmann, Vikas Verma, and Jian Tang. 2019. Infograph: Unsupervised and semi-supervised graph-level representation learning via mu- tual information maximization. In Proceedings of the International Conference on Learning Representations

    2019

  27. [35]

    Congcong Wang, Shouhang Du, Wenbin Sun, and Deqin Fan. 2023. Self- supervised learning for high-resolution remote sensing images change detection with variational information bottleneck. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing (2023)

    2023

  28. [36]

    Liang Wang, Xiang Tao, Qiang Liu, and Shu Wu. 2024. Rethinking Graph Masked Autoencoders through Alignment and Uniformity. InProceedings of the conference on Artifical Intelligence

    2024

  29. [37]

    Boris Weisfeiler and Andrei Leman. 1968. The reduction of a graph to canonical form and the algebra which appears therein. nti, Series (1968)

    1968

  30. [38]

    Petar Veličković, William Fedus, William L Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. 2019. Deep graph infomax. (2019)

    2019

  31. [39]

    Xinyi Wu, Amir Ajorlou, Zihui Wu, and Ali Jadbabaie. 2024. Demystifying oversmoothing in attention-based graph neural networks. In Proceedings of the Advances in Neural Information Processing Systems

    2024

  32. [40]

    Dongkuan Xu, Wei Cheng, Dongsheng Luo, Haifeng Chen, and Xiang Zhang

  33. [41]

    Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. 2019. How powerful are graph neural networks?. In Proceedings of the International Conference on Learning Representations

    2019

  34. [42]

    Lirong Wu, Haitao Lin, Cheng Tan, Zhangyang Gao, and Stan Z Li. 2021. Self- supervised learning on graphs: Contrastive, generative, or predictive. IEEE Transactions on Knowledge and Data Engineering (2021)

    2021

  35. [43]

    Hou Yifan, Zhang Jian, Cheng James, Ma Kaili, Ma Richard TB, Chen Hongzhi, and Yang Ming-Chang. 2020. Measuring and improving the use of graph information in graph neural network. InProceedings of the International Conference on Learning Representations

    2020

  36. [44]

    Zhitao Ying, Jiaxuan You, Christopher Morris, Xiang Ren, Will Hamilton, and Jure Leskovec. 2018. Hierarchical graph representation learning with differentiable pooling. In Proceedings of the Advances in Neural Information Processing Systems

    2018

  37. [45]

    Yuning You, Tianlong Chen, Yang Shen, and Zhangyang Wang. 2021. Graph contrastive learning automated. In Proceedings of the International Conference on Machine Learning. PMLR

    2021

  38. [46]

    Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. 2020. Graph contrastive learning with augmentations. In Proceedings of the Advances in Neural Information Processing Systems

    2020

  39. [47]

    Pinar Yanardag and SVN Vishwanathan. 2015. Deep graph kernels. InProceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining

    2015

  40. [48]

    Lingxiao Zhao and Leman Akoglu. 2019. Pairnorm: Tackling oversmoothing in gnns. In Proceedings of the International Conference on Learning Representations

    2019

  41. [49]

    Ziwen Zhao, Yuhua Li, Yixiong Zou, Jiliang Tang, and Ruixuan Li. 2024. Masked Graph Autoencoder with Non-discrete Bandwidths. In Proceedings of the ACM Web Conference

    2024

  42. [50]

    Zexian Zhou and Xiaojing Liu. 2023. Masked Autoencoders in Computer Vision: A Comprehensive Survey. IEEE Access (2023)

    2023

  43. [51]

    Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. 2020. Deep Graph Contrastive Representation Learning. In ICML Workshop on Graph Representation Learning and Beyond . WWW ’25, April 28-May 2, 2025, Sydney, NSW, Australia Heesoo Jung, and Hogun Park. Table 5: ...

    2020

  44. [52]

    Hengrui Zhang, Qitian Wu, Junchi Yan, David Wipf, and Philip S Yu. 2021. From canonical correlation analysis to self-supervised graph neural networks. In Pro- ceedings of the Advances in Neural Information Processing Systems

    2021

  45. [2018]

    Pro- ceedings of the International Joint Conference on Artificial Intelligence (2018)

    Adversarially regularized graph autoencoder for graph embedding. Pro- ceedings of the International Joint Conference on Artificial Intelligence (2018)

    2018

  46. [2020]

    In Findings of the Empirical Methods in Natural Language Processing

    Revisiting pre-trained models for Chinese natural language processing. In Findings of the Empirical Methods in Natural Language Processing

  47. [2021]

    In Proceedings of the Advances in Neural Information Processing Systems

    Infogcl: Information-aware graph contrastive learning. In Proceedings of the Advances in Neural Information Processing Systems

  48. [2022]

    In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.