REVIEW 4 major objections 5 minor 20 references
A Metric for the Balance of Information in Graph Learning
T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read NNRD measures whether a graph dataset carries more useful information in its structure or in its node features.
desk verdict A well-posed idea with an unvalidated core: NNRD's sign depends on the arbitrary relative scaling of the two noise schedules, so it is not yet a dataset property. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The object carrying the argument is the ratio $h_X(t)/h_E(t)$ between two performance curves, one under feature noise and one under structure noise, averaged over ten noise levels and logged. Structure noise removes $p_t|V|$ random edges and adds the same number of random edges, transferring removed edge features to the new edges, while feature noise randomly permutes the feature vectors of $p_t N$ nodes across the dataset, preserving marginal distributions. The noise schedules are intended to destroy useful information monotonically, so equal information in the two sources should make the curves fall at equal rates and the log-ratio sit at zero. The experimental setup fixes a three-layer Graph Isomorphism Network as the model and uses the Open Graph Benchmark molecular tasks as the testbed.
What would settle it
On a synthetic dataset engineered so features and structure carry equal mutual information with the target, NNRD should be 0; if it is not, or if halving the per-step edge count flips NNRD's sign on a fixed real dataset, the equal-destruction assumption is broken.
Extended reading notes
Core claim
The central claim is that the balance of useful information between graph structure and node features can be quantified by perturbing each source separately and comparing the resulting degradation of model performance. For a fixed model, let $h_X(t)$ be performance after $t$ steps of feature noise with structure intact and $h_E(t)$ performance after $t$ steps of structure noise with features intact; the metric is $\mathrm{NNRD} = \log\!\left(\frac{1}{|T|} \sum_t \frac{h_X(t)}{h_E(t)}\right)$. The paper asserts that NNRD is $0$ when features and structure hold equal information, positive when structure holds more, and negative when features hold more, and that the value is comparable across tasks with different performance metrics because the ratio cancels the metric's scale. Across the molecular benchmarks tested, most datasets come out feature-reliant, CLINTOX comes out structure-reliant, and ESOL sits near parity, matching the visual shape of the degradation curves better than endpoints-only aggregates.
Load-bearing premise
The load-bearing premise is that one step of structure noise and one step of feature noise destroy comparable amounts of information, so that the ratio of the two performance drops has a meaningful zero point; if that calibration is arbitrary, NNRD's sign can change without the dataset changing.
Editorial extensions
If this is right
- A dataset publisher can report NNRD for one or a few fixed models, giving users an early signal of whether to design structure-heavy or feature-heavy graph learning pipelines.
- NNRD can be compared across datasets even when one uses ROC-AUC and another uses RMSE, because the ratio inside the log removes the metric's scale.
- Endpoint-only aggregates can mislead: on ESOL, the maximum-noise scores suggest structure dominates, while NNRD's near-zero value reflects the nearly equal degradation rates visible in the full curves.
- The metric changes with the chosen baseline model, so NNRD is a property of a dataset-model pair rather than a purely intrinsic dataset property.
Reading between the lines
- Because NNRD's sign depends on treating one structure-noise step and one feature-noise step as equally destructive, an obvious extension is to calibrate the two schedules by equalizing the amount of information removed, for example through mutual-information estimates, before comparing datasets.
- The BACE result, where structure noise improves performance, suggests structure noise can add useful shortcut edges rather than only destroy information; testing a variant that only removes edges would clarify whether NNRD's monotonic premise holds for that dataset.
- The same measurement could be applied beyond molecules to any graph domain, but only after the noise-schedule comparability question is settled; otherwise cross-domain NNRD values are not yet comparable.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Noise-Noise Ratio Difference (NNRD), a scalar metric intended to quantify whether the useful information in a graph dataset for a given prediction task resides more in the node/edge features or in the graph structure. NNRD is defined in Eq. (1) as the logarithm of the average, over noise levels, of the ratio h_X(t)/h_E(t), where h_X and h_E are the model's performance under feature noise and structure noise, respectively. The authors instantiate the metric with a GIN model on OGB molecular datasets, applying random edge removal/addition as structure noise and random feature permutation as feature noise, and report that NNRD yields intuitive results and corresponds to a loss of information. The paper also introduces NNRD_e, computed only at the maximum noise level.
Significance. If the metric were properly validated, it could be practically useful for deciding whether to invest in structural or feature-based modeling for a given graph dataset. The core idea of comparing performance degradation under two noise channels is intuitive and readily computable. However, the paper does not establish that NNRD measures a property of the dataset: there is no synthetic ground truth, no external benchmark, and the metric's sign is not invariant under an arbitrary rescaling of either noise schedule. The authors' own evidence is partly tautological because NNRD is defined from the same performance curves shown in Figure 2. The manuscript is clearly written and covers several OGB datasets, but the central claim is currently unsupported.
major comments (4)
- [Experiments, Eq. (1)] The two noise schedules are not calibrated, so NNRD's sign and magnitude are not invariant under an arbitrary rescaling of one schedule. Structure noise at step t removes and adds pt|V| edges, while feature noise at step t permutes the features of pt·N nodes. If the feature-noise step were redefined to permute twice as many nodes per step, the sequence h_X(t) would decay roughly twice as quickly, changing the average ratio in Eq. (1) and potentially flipping the sign of NNRD on the same dataset. Since no calibration or invariance argument is provided, NNRD cannot be interpreted as a dataset-level information balance.
- [NNRD, Results, Figure 2] The claim that NNRD "corresponds well to a loss of information" is supported only by visual agreement with Figure 2, but NNRD is defined as the log average ratio of the very curves shown in that figure. This correspondence is therefore at least partly tautological. There is no evaluation against synthetic datasets with known information balance, no external ground-truth metric, and no test of whether NNRD predicts a model design choice that actually improves performance. Independent validation is load-bearing for the paper's central claim.
- [NNRD, Table 1] The assumption that "t and h_X(t), h_E(t) should monotonically increase together" is contradicted by the paper's own results. Table 1 shows that structure noise on BACE raises ROC-AUC from 0.575 to 0.601, and the text notes that on CLINTOX feature noise also improves performance. When the performance curve is non-monotonic, the average ratio in Eq. (1) does not have the interpretation stated in the paper, because the noise is not necessarily removing useful information at every step.
- [Discussion] The paper acknowledges that NNRD "varies on a per-model basis" and proposes reporting it for a few fixed models. However, no sensitivity analysis is provided, and the choice of GIN is not justified beyond expressivity. Since the metric is defined through a specific model's performance curves, the dependence on model capacity, training procedure, and hyperparameters could be as large as the signal being measured. At minimum, the paper should show that the qualitative ordering of datasets by NNRD is stable across a small set of reasonable base models.
minor comments (5)
- [Experiments] The sentence beginning "Att, we remove..." contains a typo; it should read "At t, we remove...".
- [Experiments] In the feature-noise description, "N points" should read "N nodes" for consistency with the rest of the paper.
- [Results] The notation NNRD_e is introduced as "calculated only at the extreme noise level t = T" but is not defined with an equation, and Table 1 does not clearly label which column corresponds to NNRD_e versus NNRD.
- [Eq. (1)] The notation |T| is used for the number of noise levels, but T is also used as the final time step; please distinguish the set of time indices from its cardinality to avoid ambiguity.
- [Figure 2] The caption states "molecular regression benchmark" but most of the listed datasets (BACE, BBBP, CLINTOX, TOX21, HIV, SIDER) are classification tasks; please correct the caption.
Circularity Check
NNRD's validation is definitional: Eq. (1) is a log-mean ratio of h_X and h_E, and the paper's evidence that NNRD 'corresponds well' is that it tracks those same curves in Figure 2.
-
self definitional
[NNRD section, Eq. (1)]
"NNRD = log( 1/|T | sum_t hX (t) / hE(t) ) (1) ... NNRD is 0 when features and structure contain equal information, > 0 when structure contains more information, and < 0 when features contain more information."
NNRD is defined as the log-mean ratio of the two performance curves h_X(t) and h_E(t). Its sign is fixed by which curve is larger on average, so the interpretive claim that the sign reveals which source contains more useful information is an assumption attached to the definition, not a derived result. No independent ground-truth quantity (synthetic dataset with known information balance, or a different model) is used to test whether the sign actually tracks information balance.
-
self definitional
[Discussion]
"While NNRD corresponds with the visualisations in Figure 2, there are some cases where it fails."
Figure 2 plots h_X(t) and h_E(t), exactly the quantities whose log-mean ratio defines NNRD in Eq. (1). Therefore NNRD's 'correspondence' with those visualisations is guaranteed by construction and cannot serve as independent evidence that NNRD measures the balance of information. The admitted failure on clintox further shows that the sign interpretation is not universally valid.
full rationale
The paper contains no load-bearing self-citation chain: the only self-citation (Davies et al. 2022) appears in the introduction and does no argumentative work. The central issue is different. NNRD is defined in Eq. (1) as the logarithm of the mean ratio of h_X(t) to h_E(t), and Figure 2 plots exactly those h_X and h_E curves. The paper's main positive evidence, stated in the Abstract ('corresponds well to a loss of information') and in the Discussion ('NNRD corresponds with the visualisations in Figure 2'), is therefore a restatement of the definition rather than an independent empirical check. There is no synthetic or hand-constructed dataset with a known feature/structure information balance against which the sign of NNRD is validated. The metric's sign interpretation also depends on the monotonicity assumption 't and h_X(t), h_E(t) should monotonically increase together', which is contradicted by the paper's own BACE result (Table 1: structure noise raises ROC-AUC from 0.575 to 0.601). This is a correctness concern rather than a circularity, but it compounds the definitional nature of the validation. Separately, the ratio's magnitude and sign are sensitive to the arbitrary scaling of the two noise schedules (one structure-noise step removes/adds p_t|V| edges while one feature-noise step permutes p_t N nodes, with no calibration), which means NNRD is not a dataset-only property; that too is a validity concern beyond circularity. Overall, the derivation chain is not one of fitted parameters renamed as predictions, and the metric definition itself is a legitimate proposal, so the circularity is partial: the claimed empirical support reduces to the definition, but the paper does not disguise a fit as a prediction.
Assumptions & free parameters
free parameters (2)
- structure vs feature noise step calibration =
edge changes per step = pt*|V|; feature permutations per step = pt*N; pt = t/10
- number of noise levels T =
10
assumptions (3)
- ad hoc to paper Random edge removal/addition and feature permutation monotonically reduce the useful information in structure and features respectively.
- domain assumption Performance of one fixed 3-layer GIN model is a faithful proxy for the mutual information I(y; X) and I(y; E) of the dataset.
- domain assumption At maximum structure noise, a random graph with the same density retains no useful information.
Cite this review
Pith. "Pith review of A Metric for the Balance of Information in Graph Learning." pith.science (2026). https://pith.science/paper/PW6XVFDN
@misc{pith2026250119137,
author = {Pith},
title = {Pith review of: A Metric for the Balance of Information in Graph Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/PW6XVFDN}},
note = {Machine review of arXiv:2501.19137}
}
read the original abstract
Graph learning on molecules makes use of information from both the molecular structure and the features attached to that structure. Much work has been conducted on biasing either towards structure or features, with the aim that bias bolsters performance. Identifying which information source a dataset favours, and therefore how to approach learning that dataset, is an open issue. Here we propose Noise-Noise Ratio Difference (NNRD), a quantitative metric for whether there is more useful information in structure or features. By employing iterative noising on features and structure independently, leaving the other intact, NNRD measures the degradation of information in each. We employ NNRD over a range of molecular tasks, and show that it corresponds well to a loss of information, with intuitive results that are more expressive than simple performance aggregates. Our future work will focus on expanding data domains, tasks and types, as well as refining our choice of baseline model.
Figures
Reference graph
Works this paper leans on
-
[1]
, " * write output.state after.block = add.period write newline
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Alon, U.; and Yahav, E. 2021. On the Bottleneck of Graph Neural Networks and its Practical Implications . ArXiv:2006.05205
arXiv 2021
-
[4]
Bechler-Speicher, M.; Amos, I.; Gilad-Bachrach, R.; and Globerson, A. 2024. Graph Neural Networks Use Graphs When They Shouldn 't. ArXiv:2309.04332
arXiv 2024
-
[5]
Chen, Y.; Coskunuzer, B.; and Gel, Y. R. 2021. Topological relational learning on graphs. Advances in Neural Information Processing Systems, 34: 27029--27042
work page 2021
-
[6]
Davies, A.; Ajmeri, N.; and Silva Filho, T. 2022. Realistic synthetic social networks with graph neural networks. arXiv Pre-Print, 1(1): 1--12. Tex.arxivid: 2212.07843
arXiv 2022
-
[7]
Gilmer, J.; Schoenholz, S. S.; Riley, P. F.; Vinyals, O.; and Dahl, G. E. 2017. Neural message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning, (ICML) 2017, 3: 2053--2070. ISBN: 9781510855144 Publisher: International Machine Learning Society (IMLS) tex.arxivid: 1704.01212
arXiv 2017
-
[8]
N.; Duvenaud, D.; Hernández-Lobato, J
Gómez-Bombarelli, R.; Wei, J. N.; Duvenaud, D.; Hernández-Lobato, J. M.; Sánchez-Lengeling, B.; Sheberla, D.; Aguilera-Iparraguirre, J.; Hirzel, T. D.; Adams, R. P.; and Aspuru-Guzik, A. 2018. Automatic chemical design using a data-driven continuous representation of molecules. ACS Central Science, 4(2): 268--276. Publisher: American Chemical Society tex....
arXiv 2018
Show all 20 references
-
[9]
Horn, M.; De Brouwer, E.; Moor, M.; Moreau, Y.; Rieck, B.; and Borgwardt, K. 2021. Topological graph neural networks. ICLR 2022 - 10th International Conference on Learning Representations. Publisher: International Conference on Learning Representations, ICLR tex.arxivid: 2102.07835
2021 arXiv
-
[10]
Hu, W.; Fey, M.; Zitnik, M.; Dong, Y.; Ren, H.; Liu, B.; Catasta, M.; Leskovec, J.; Barzilay, R.; Battaglia, P.; Bengio, Y.; Bronstein, M.; Günnemann, S.; Hamilton, W.; Jaakkola, T.; Jegelka, S.; Nickel, M.; Re, C.; Song, L.; Tang, J.; Welling, M.; and Zemel, R. 2020. Open gra...
2020
-
[11]
Jin, W.; Derr, T.; Wang, Y.; Ma, Y.; Liu, Z.; and Tang, J. 2021. Node Similarity Preserving Graph Convolutional Networks . ArXiv:2011.09643
2021 arXiv
-
[12]
J.; Bollegala, D.; and Kell, D
Khemchandani, Y.; O'Hagan, S.; Samanta, S.; Swainston, N.; Roberts, T. J.; Bollegala, D.; and Kell, D. B. 2020. DeepGraphMolGen , a multi-objective, computational strategy for generating molecules with desirable properties: a graph convolution and reinforcement learning approa...
2020
-
[13]
Khodayar, M.; Wang, J.; Wang, Z.; Khodayar, M.; Wang, J.; and Wang, Z. 2019. Deep generative graph distribution learning for synthetic power grids. arXiv, arXiv:1901.09674. Tex.arxivid: arXiv:1901.09674
2019 arXiv
-
[14]
M.; Huang, K.; and Zitnik, M
Li, M. M.; Huang, K.; and Zitnik, M. 2022. Graph representation learning in biomedicine and healthcare. Nature Biomedical Engineering, 6(12): 1353--1369. Publisher: Nature Research
2022
-
[15]
Popova, M.; Shvets, M.; Oliva, J.; Isayev, O.; Popova, M.; Shvets, M.; Oliva, J.; and Isayev, O. 2019. MolecularRNN : Generating realistic molecular graphs with optimized properties. arXiv, arXiv:1905.13372. Tex.arxivid: arXiv:1905.13372
2019 arXiv
-
[16]
Vignac, C.; Krawczuk, I.; Siraudin, A.; Wang, B.; Cevher, V.; and Frossard, P. 2023. DiGress : Discrete Denoising diffusion for graph generation. In The eleventh international conference on learning representations
2023
-
[17]
Wang, J.; Guo, J.; Sun, Y.; Gao, J.; Wang, S.; Yang, Y.; and Yin, B. 2024. DGNN : Decoupled Graph Neural Networks with Structural Consistency between Attribute and Graph Embedding Representations . ArXiv:2401.15584
2024 arXiv
-
[18]
Wu, F.; Li, S.; Wu, L.; Radev, D.; and Li, S. Z. 2023. Discovering and Explaining the Representation Bottleneck of Graph Neural Networks from Multi -order Interactions . ArXiv:2205.07266
2023
-
[19]
Wu, T.; Ren, H.; Li, P.; and Leskovec, J. 2020. Graph Information Bottleneck . In Advances in Neural Information Processing Systems , volume 33, 20437--20448. Curran Associates, Inc
2020
-
[20]
Xu, K.; Jegelka, S.; Hu, W.; and Leskovec, J. 2019. How powerful are graph neural networks? In 7th international conference on learning representations, ICLR 2019 . International Conference on Learning Representations, ICLR. Tex.arxivid: 1810.00826
2019 arXiv
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.