Pith. sign in

REVIEW 4 major objections 5 minor 20 references

A Metric for the Balance of Information in Graph Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Pith's one-line read NNRD measures whether a graph dataset carries more useful information in its structure or in its node features.

desk verdict A well-posed idea with an unvalidated core: NNRD's sign depends on the arbitrary relative scaling of the two noise schedules, so it is not yet a dataset property. read the letter →

arxiv 2501.19137 v1 pith:PW6XVFDN submitted 2025-01-31 cs.LG cs.AI

classification cs.LGcs.AI
keywords graphneuralnetworksmoleculargraphsinformationbalancefeaturenoisestructureNNRDdatasetcharacterization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes Noise-Noise Ratio Difference (NNRD), a single summary number intended to state whether a graph dataset carries more useful information in its structure or in its node and edge features. The idea is to degrade one information source at a time—adding or removing random edges for structure, permuting feature vectors for features—and compare how a fixed model's performance falls under the two noise schedules. NNRD is the log of the average ratio of these two performance curves, so it is zero at parity, positive when structure carries more useful information, and negative when features do. The paper demonstrates the metric on eight molecular benchmarks and argues that it captures trends that simple endpoint aggregates misrepresent, such as near-parity on ESOL. A reader should care because such a metric could tell dataset publishers and model designers up front whether to invest in structure-aware or feature-aware architectures.

What carries the argument

The object carrying the argument is the ratio $h_X(t)/h_E(t)$ between two performance curves, one under feature noise and one under structure noise, averaged over ten noise levels and logged. Structure noise removes $p_t|V|$ random edges and adds the same number of random edges, transferring removed edge features to the new edges, while feature noise randomly permutes the feature vectors of $p_t N$ nodes across the dataset, preserving marginal distributions. The noise schedules are intended to destroy useful information monotonically, so equal information in the two sources should make the curves fall at equal rates and the log-ratio sit at zero. The experimental setup fixes a three-layer Graph Isomorphism Network as the model and uses the Open Graph Benchmark molecular tasks as the testbed.

What would settle it

On a synthetic dataset engineered so features and structure carry equal mutual information with the target, NNRD should be 0; if it is not, or if halving the per-step edge count flips NNRD's sign on a fixed real dataset, the equal-destruction assumption is broken.

Watch

Extended reading notes

Core claim

The central claim is that the balance of useful information between graph structure and node features can be quantified by perturbing each source separately and comparing the resulting degradation of model performance. For a fixed model, let $h_X(t)$ be performance after $t$ steps of feature noise with structure intact and $h_E(t)$ performance after $t$ steps of structure noise with features intact; the metric is $\mathrm{NNRD} = \log\!\left(\frac{1}{|T|} \sum_t \frac{h_X(t)}{h_E(t)}\right)$. The paper asserts that NNRD is $0$ when features and structure hold equal information, positive when structure holds more, and negative when features hold more, and that the value is comparable across tasks with different performance metrics because the ratio cancels the metric's scale. Across the molecular benchmarks tested, most datasets come out feature-reliant, CLINTOX comes out structure-reliant, and ESOL sits near parity, matching the visual shape of the degradation curves better than endpoints-only aggregates.

Load-bearing premise

The load-bearing premise is that one step of structure noise and one step of feature noise destroy comparable amounts of information, so that the ratio of the two performance drops has a meaningful zero point; if that calibration is arbitrary, NNRD's sign can change without the dataset changing.

Editorial extensions

If this is right

  • A dataset publisher can report NNRD for one or a few fixed models, giving users an early signal of whether to design structure-heavy or feature-heavy graph learning pipelines.
  • NNRD can be compared across datasets even when one uses ROC-AUC and another uses RMSE, because the ratio inside the log removes the metric's scale.
  • Endpoint-only aggregates can mislead: on ESOL, the maximum-noise scores suggest structure dominates, while NNRD's near-zero value reflects the nearly equal degradation rates visible in the full curves.
  • The metric changes with the chosen baseline model, so NNRD is a property of a dataset-model pair rather than a purely intrinsic dataset property.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because NNRD's sign depends on treating one structure-noise step and one feature-noise step as equally destructive, an obvious extension is to calibrate the two schedules by equalizing the amount of information removed, for example through mutual-information estimates, before comparing datasets.
  • The BACE result, where structure noise improves performance, suggests structure noise can add useful shortcut edges rather than only destroy information; testing a variant that only removes edges would clarify whether NNRD's monotonic premise holds for that dataset.
  • The same measurement could be applied beyond molecules to any graph domain, but only after the noise-schedule comparability question is settled; otherwise cross-domain NNRD values are not yet comparable.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Noise-Noise Ratio Difference (NNRD), a scalar metric intended to quantify whether the useful information in a graph dataset for a given prediction task resides more in the node/edge features or in the graph structure. NNRD is defined in Eq. (1) as the logarithm of the average, over noise levels, of the ratio h_X(t)/h_E(t), where h_X and h_E are the model's performance under feature noise and structure noise, respectively. The authors instantiate the metric with a GIN model on OGB molecular datasets, applying random edge removal/addition as structure noise and random feature permutation as feature noise, and report that NNRD yields intuitive results and corresponds to a loss of information. The paper also introduces NNRD_e, computed only at the maximum noise level.

Significance. If the metric were properly validated, it could be practically useful for deciding whether to invest in structural or feature-based modeling for a given graph dataset. The core idea of comparing performance degradation under two noise channels is intuitive and readily computable. However, the paper does not establish that NNRD measures a property of the dataset: there is no synthetic ground truth, no external benchmark, and the metric's sign is not invariant under an arbitrary rescaling of either noise schedule. The authors' own evidence is partly tautological because NNRD is defined from the same performance curves shown in Figure 2. The manuscript is clearly written and covers several OGB datasets, but the central claim is currently unsupported.

major comments (4)
  1. [Experiments, Eq. (1)] The two noise schedules are not calibrated, so NNRD's sign and magnitude are not invariant under an arbitrary rescaling of one schedule. Structure noise at step t removes and adds pt|V| edges, while feature noise at step t permutes the features of pt·N nodes. If the feature-noise step were redefined to permute twice as many nodes per step, the sequence h_X(t) would decay roughly twice as quickly, changing the average ratio in Eq. (1) and potentially flipping the sign of NNRD on the same dataset. Since no calibration or invariance argument is provided, NNRD cannot be interpreted as a dataset-level information balance.
  2. [NNRD, Results, Figure 2] The claim that NNRD "corresponds well to a loss of information" is supported only by visual agreement with Figure 2, but NNRD is defined as the log average ratio of the very curves shown in that figure. This correspondence is therefore at least partly tautological. There is no evaluation against synthetic datasets with known information balance, no external ground-truth metric, and no test of whether NNRD predicts a model design choice that actually improves performance. Independent validation is load-bearing for the paper's central claim.
  3. [NNRD, Table 1] The assumption that "t and h_X(t), h_E(t) should monotonically increase together" is contradicted by the paper's own results. Table 1 shows that structure noise on BACE raises ROC-AUC from 0.575 to 0.601, and the text notes that on CLINTOX feature noise also improves performance. When the performance curve is non-monotonic, the average ratio in Eq. (1) does not have the interpretation stated in the paper, because the noise is not necessarily removing useful information at every step.
  4. [Discussion] The paper acknowledges that NNRD "varies on a per-model basis" and proposes reporting it for a few fixed models. However, no sensitivity analysis is provided, and the choice of GIN is not justified beyond expressivity. Since the metric is defined through a specific model's performance curves, the dependence on model capacity, training procedure, and hyperparameters could be as large as the signal being measured. At minimum, the paper should show that the qualitative ordering of datasets by NNRD is stable across a small set of reasonable base models.
minor comments (5)
  1. [Experiments] The sentence beginning "Att, we remove..." contains a typo; it should read "At t, we remove...".
  2. [Experiments] In the feature-noise description, "N points" should read "N nodes" for consistency with the rest of the paper.
  3. [Results] The notation NNRD_e is introduced as "calculated only at the extreme noise level t = T" but is not defined with an equation, and Table 1 does not clearly label which column corresponds to NNRD_e versus NNRD.
  4. [Eq. (1)] The notation |T| is used for the number of noise levels, but T is also used as the final time step; please distinguish the set of time indices from its cardinality to avoid ambiguity.
  5. [Figure 2] The caption states "molecular regression benchmark" but most of the listed datasets (BACE, BBBP, CLINTOX, TOX21, HIV, SIDER) are classification tasks; please correct the caption.

Circularity Check

2 steps flagged · score 5.0 of 10

NNRD's validation is definitional: Eq. (1) is a log-mean ratio of h_X and h_E, and the paper's evidence that NNRD 'corresponds well' is that it tracks those same curves in Figure 2.

  1. self definitional [NNRD section, Eq. (1)]
    "NNRD = log( 1/|T | sum_t hX (t) / hE(t) ) (1) ... NNRD is 0 when features and structure contain equal information, > 0 when structure contains more information, and < 0 when features contain more information."

    NNRD is defined as the log-mean ratio of the two performance curves h_X(t) and h_E(t). Its sign is fixed by which curve is larger on average, so the interpretive claim that the sign reveals which source contains more useful information is an assumption attached to the definition, not a derived result. No independent ground-truth quantity (synthetic dataset with known information balance, or a different model) is used to test whether the sign actually tracks information balance.

  2. self definitional [Discussion]
    "While NNRD corresponds with the visualisations in Figure 2, there are some cases where it fails."

    Figure 2 plots h_X(t) and h_E(t), exactly the quantities whose log-mean ratio defines NNRD in Eq. (1). Therefore NNRD's 'correspondence' with those visualisations is guaranteed by construction and cannot serve as independent evidence that NNRD measures the balance of information. The admitted failure on clintox further shows that the sign interpretation is not universally valid.

full rationale

The paper contains no load-bearing self-citation chain: the only self-citation (Davies et al. 2022) appears in the introduction and does no argumentative work. The central issue is different. NNRD is defined in Eq. (1) as the logarithm of the mean ratio of h_X(t) to h_E(t), and Figure 2 plots exactly those h_X and h_E curves. The paper's main positive evidence, stated in the Abstract ('corresponds well to a loss of information') and in the Discussion ('NNRD corresponds with the visualisations in Figure 2'), is therefore a restatement of the definition rather than an independent empirical check. There is no synthetic or hand-constructed dataset with a known feature/structure information balance against which the sign of NNRD is validated. The metric's sign interpretation also depends on the monotonicity assumption 't and h_X(t), h_E(t) should monotonically increase together', which is contradicted by the paper's own BACE result (Table 1: structure noise raises ROC-AUC from 0.575 to 0.601). This is a correctness concern rather than a circularity, but it compounds the definitional nature of the validation. Separately, the ratio's magnitude and sign are sensitive to the arbitrary scaling of the two noise schedules (one structure-noise step removes/adds p_t|V| edges while one feature-noise step permutes p_t N nodes, with no calibration), which means NNRD is not a dataset-only property; that too is a validity concern beyond circularity. Overall, the derivation chain is not one of fitted parameters renamed as predictions, and the metric definition itself is a legitimate proposal, so the circularity is partial: the claimed empirical support reduces to the definition, but the paper does not disguise a fit as a prediction.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

No fitted constants in the usual sense, but the metric's meaning rests on hand-chosen noise calibration and strong assumptions about monotonic degradation, model faithfulness, and the absence of information in maximally noised graphs. These are not derived from first principles and are not validated against external ground truth.

free parameters (2)
  • structure vs feature noise step calibration = edge changes per step = pt*|V|; feature permutations per step = pt*N; pt = t/10
    This hand-chosen equivalence determines whether the h_X/h_E ratio is above or below 1. NNRD's sign can flip if the rate of edge rewiring per step is changed relative to the feature permutation rate, even on identical data.
  • number of noise levels T = 10
    T is chosen by hand. NNRD averages over this grid, and the result can depend on the number and spacing of noise levels.
assumptions (3)
  • ad hoc to paper Random edge removal/addition and feature permutation monotonically reduce the useful information in structure and features respectively.
    Invoked in the NNRD section to justify that h_X(t) and h_E(t) descend with t. Contradicted on BACE, where performance rises with structure noise (Results section).
  • domain assumption Performance of one fixed 3-layer GIN model is a faithful proxy for the mutual information I(y; X) and I(y; E) of the dataset.
    The whole metric is computed through one model's test performance. The paper itself acknowledges model dependence in the Discussion, which undercuts the dataset-level reading.
  • domain assumption At maximum structure noise, a random graph with the same density retains no useful information.
    Stated in the Experiments section. Random graphs may still preserve size and degree signals that correlate with targets, so 'no useful information after NT' is not guaranteed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Metric for the Balance of Information in Graph Learning." pith.science (2026). https://pith.science/paper/PW6XVFDN

@misc{pith2026250119137,
  author       = {Pith},
  title        = {Pith review of: A Metric for the Balance of Information in Graph Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PW6XVFDN}},
  note         = {Machine review of arXiv:2501.19137}
}
read the original abstract

Graph learning on molecules makes use of information from both the molecular structure and the features attached to that structure. Much work has been conducted on biasing either towards structure or features, with the aim that bias bolsters performance. Identifying which information source a dataset favours, and therefore how to approach learning that dataset, is an open issue. Here we propose Noise-Noise Ratio Difference (NNRD), a quantitative metric for whether there is more useful information in structure or features. By employing iterative noising on features and structure independently, leaving the other intact, NNRD measures the degradation of information in each. We employ NNRD over a range of molecular tasks, and show that it corresponds well to a loss of information, with intuitive results that are more expressive than simple performance aggregates. Our future work will focus on expanding data domains, tasks and types, as well as refining our choice of baseline model.

Figures

Figures reproduced from arXiv: 2501.19137 by the authors.

Figure 1
Figure 1. A molecule undergoing structure noise through edge removal and addition. Each noise step is applied on the original [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Performance variation for supervised training of our GIN models on each molecular regression bench￾mark dataset with increasing noise on structure and features. All datasets ex￾cept LIPO and ESOL are classifica￾tion, and we report ROC-AUC. This is very close to the procedure taken by graph diffusion models as forward noising processes (Vignac et al. 2023). At t = T, pt = 1, and the structure is a random graph with t… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 7 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Alon, U.; and Yahav, E. 2021. On the Bottleneck of Graph Neural Networks and its Practical Implications . ArXiv:2006.05205

  4. [4]

    Bechler-Speicher, M.; Amos, I.; Gilad-Bachrach, R.; and Globerson, A. 2024. Graph Neural Networks Use Graphs When They Shouldn 't. ArXiv:2309.04332

  5. [5]

    Chen, Y.; Coskunuzer, B.; and Gel, Y. R. 2021. Topological relational learning on graphs. Advances in Neural Information Processing Systems, 34: 27029--27042

  6. [6]

    Davies, A.; Ajmeri, N.; and Silva Filho, T. 2022. Realistic synthetic social networks with graph neural networks. arXiv Pre-Print, 1(1): 1--12. Tex.arxivid: 2212.07843

  7. [7]

    S.; Riley, P

    Gilmer, J.; Schoenholz, S. S.; Riley, P. F.; Vinyals, O.; and Dahl, G. E. 2017. Neural message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning, (ICML) 2017, 3: 2053--2070. ISBN: 9781510855144 Publisher: International Machine Learning Society (IMLS) tex.arxivid: 1704.01212

  8. [8]

    N.; Duvenaud, D.; Hernández-Lobato, J

    Gómez-Bombarelli, R.; Wei, J. N.; Duvenaud, D.; Hernández-Lobato, J. M.; Sánchez-Lengeling, B.; Sheberla, D.; Aguilera-Iparraguirre, J.; Hirzel, T. D.; Adams, R. P.; and Aspuru-Guzik, A. 2018. Automatic chemical design using a data-driven continuous representation of molecules. ACS Central Science, 4(2): 268--276. Publisher: American Chemical Society tex....

Show all 20 references
  1. [9]

    Horn, M.; De Brouwer, E.; Moor, M.; Moreau, Y.; Rieck, B.; and Borgwardt, K. 2021. Topological graph neural networks. ICLR 2022 - 10th International Conference on Learning Representations. Publisher: International Conference on Learning Representations, ICLR tex.arxivid: 2102.07835

  2. [10]

    Hu, W.; Fey, M.; Zitnik, M.; Dong, Y.; Ren, H.; Liu, B.; Catasta, M.; Leskovec, J.; Barzilay, R.; Battaglia, P.; Bengio, Y.; Bronstein, M.; Günnemann, S.; Hamilton, W.; Jaakkola, T.; Jegelka, S.; Nickel, M.; Re, C.; Song, L.; Tang, J.; Welling, M.; and Zemel, R. 2020. Open gra...

  3. [11]

    Jin, W.; Derr, T.; Wang, Y.; Ma, Y.; Liu, Z.; and Tang, J. 2021. Node Similarity Preserving Graph Convolutional Networks . ArXiv:2011.09643

  4. [12]

    J.; Bollegala, D.; and Kell, D

    Khemchandani, Y.; O'Hagan, S.; Samanta, S.; Swainston, N.; Roberts, T. J.; Bollegala, D.; and Kell, D. B. 2020. DeepGraphMolGen , a multi-objective, computational strategy for generating molecules with desirable properties: a graph convolution and reinforcement learning approa...

  5. [13]

    Khodayar, M.; Wang, J.; Wang, Z.; Khodayar, M.; Wang, J.; and Wang, Z. 2019. Deep generative graph distribution learning for synthetic power grids. arXiv, arXiv:1901.09674. Tex.arxivid: arXiv:1901.09674

  6. [14]

    M.; Huang, K.; and Zitnik, M

    Li, M. M.; Huang, K.; and Zitnik, M. 2022. Graph representation learning in biomedicine and healthcare. Nature Biomedical Engineering, 6(12): 1353--1369. Publisher: Nature Research

  7. [15]

    Popova, M.; Shvets, M.; Oliva, J.; Isayev, O.; Popova, M.; Shvets, M.; Oliva, J.; and Isayev, O. 2019. MolecularRNN : Generating realistic molecular graphs with optimized properties. arXiv, arXiv:1905.13372. Tex.arxivid: arXiv:1905.13372

  8. [16]

    Vignac, C.; Krawczuk, I.; Siraudin, A.; Wang, B.; Cevher, V.; and Frossard, P. 2023. DiGress : Discrete Denoising diffusion for graph generation. In The eleventh international conference on learning representations

  9. [17]

    Wang, J.; Guo, J.; Sun, Y.; Gao, J.; Wang, S.; Yang, Y.; and Yin, B. 2024. DGNN : Decoupled Graph Neural Networks with Structural Consistency between Attribute and Graph Embedding Representations . ArXiv:2401.15584

  10. [18]

    Wu, F.; Li, S.; Wu, L.; Radev, D.; and Li, S. Z. 2023. Discovering and Explaining the Representation Bottleneck of Graph Neural Networks from Multi -order Interactions . ArXiv:2205.07266

  11. [19]

    Wu, T.; Ren, H.; Li, P.; and Leskovec, J. 2020. Graph Information Bottleneck . In Advances in Neural Information Processing Systems , volume 33, 20437--20448. Curran Associates, Inc

  12. [20]

    Xu, K.; Jegelka, S.; Hu, W.; and Leskovec, J. 2019. How powerful are graph neural networks? In 7th international conference on learning representations, ICLR 2019 . International Conference on Learning Representations, ICLR. Tex.arxivid: 1810.00826

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.