{"id":"db236d67-4d3c-444e-ad1e-362415193c87","arxiv_id":"2502.02379","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"RINGS evaluates graph-learning datasets through mode perturbations, using performance separability and mode complementarity to classify 13 benchmarks into keep, realign, or deprecate.","lead":"This paper introduces RINGS, a framework for judging whether a graph-learning dataset is a good benchmark by corrupting its graph structure or node features and measuring how model performance and data geometry change. It then ranks 13 popular graph datasets, proposing that some (like COLLAB and IMDB) be deprecated and others (like MolHIV and NCI1) be kept.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Deprecate labels for COLLAB/IMDB rest on low structural diversity, yet these datasets are structure-informative under the paper's own separability test; the diversity metric is unvalidated and a different structural distance could flip the taxonomy.","rationale":"The paper is a well-engineered framework contribution with reproducible code, extensive experiments, and honest limitations. The central claim I scrutinized is the normative taxonomy, not the framework itself. The reader's weakest assumption already identifies the unvalidated metric choice for mode complementarity, and my reading agrees: the Deprecate verdict for COLLAB and IMDB follows from equating low structural diversity with low benchmark utility, while the paper's own separability results show these datasets do require graph structure. That internal tension makes the metric choice load-bearing. The proposed check is deliberately narrow: holding all other design decisions fixed, replace only the structural distance and see whether taxonomy labels move. This would settle whether the recommendations are properties of the datasets or of the chosen diffusion distance. Since the reader's CONDITIONAL verdict already flags this vulnerability, no adjustment to the verdict is needed.","tokens_in":44058,"tokens_out":10209,"duration_ms":103251,"concrete_test":"Recompute Table 2's Delta_S (and, for completeness, Delta_F) for all 13 datasets with the pipeline fixed except for the structural metric: use heat-kernel distance at t=1 and diffusion distance at t=10, then apply the same Section 4.3 binning and taxonomy rules. Report how many datasets change Keep/Realign/Deprecate labels. If COLLAB, IMDB-B, or IMDB-M move out of Deprecate, or if any other label flips, the taxonomy is an artifact of the chosen structural distance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.3's taxonomy is the load-bearing output, and its Deprecate categories are decided by structural diversity Delta_S (Definition 2.14) computed from diffusion distance at t=1 with the L1,1 comparator (Definition 2.12). This score is assumed to proxy whether a dataset contains interesting structural variation that graph-learning methods can exploit. The paper's own P1 results contradict that proxy: Table 1 lists COLLAB, IMDB-B, and IMDB-M as having informative structure (the original separably outperforms structural perturbations), yet Table 2 reports low Delta_S (0.30, 0.18, 0.09), and Section 4.3 consequently deprecates them. The same datasets are thus shown to require graph structure and then labeled as not probing graph-learning capabilities. The reason is visible in the metric: an all-pairs spectral distance on diameter-2 ego-networks is nearly degenerate, even though local adjacency and neighborhood structure is exactly what message-passing GNNs consume; all-pairs metric comparison does not measure that. The complementarity-performance correlation in Figure 5 is computed on the same 13 datasets and therefore does not validate the metric as a predictor of benchmark utility. The one alternative-metric illustration (Appendix B.2, Figure 7) covers only Peptides and does not re-derive the taxonomy. If a different structural distance changes Delta_S enough to move COLLAB/IMDB out of Deprecate, the headline recommendations are an artifact of metric choice.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces RINGS, a mode-perturbation framework for evaluating the quality of graph-learning datasets. The framework applies perturbations to the graph structure or node features and uses two measures: performance separability, which compares performance distributions of models trained on original versus perturbed data via permutation tests, and mode complementarity, which computes a normalized L1,1 distance between metric spaces induced by the structure and features. The authors evaluate 13 graph-classification datasets, report that only COLLAB, MolHIV, and NCI1 satisfy the expected separability relations, and derive a taxonomy classifying datasets as Keep, Realign, or Deprecate. They also demonstrate extensions to regression, transformer architectures, and graph-level analyses, and publicly release code.","tokens_in":44362,"tokens_out":11981,"duration_ms":101647,"significance":"If the framework is valid, it addresses a real need in data-centric graph learning: systematic evaluation of benchmark datasets rather than models. The authors provide a reproducible pipeline (Zenodo/GitHub), thorough statistical testing (10,000-permutation tests, Bonferroni correction, robustness checks across test statistics and alpha levels), and a clean theoretical result in Theorem 2.15. The proposed mode-complementarity measure could serve as a fast, task-independent diagnostic. However, the practical value hinges on the unvalidated assumption that the chosen structural metric (diffusion distance at t=1 with the L1,1 norm) captures the structural variation that graph-learning methods actually exploit; the paper's own performance-separability results for COLLAB and IMDB contradict the taxonomy's Deprecate labels. The headline taxonomy should be treated as suggestive rather than established.","major_comments":[{"comment":"The Deprecate (‡) category for COLLAB, IMDB-B, and IMDB-M is based on low structural diversity Δ_S, computed from the diffusion distance at t=1 with the L1,1 comparator. This conflicts with the paper's own P1 results: Table 1 reports COLLAB as having informative structure and IMDB-B/IMDB-M as having at least separable structure. The authors never establish that Δ_S measures the structural information used by message-passing GNNs; for diameter-2 ego-networks the all-pairs diffusion distance is nearly degenerate even though local adjacency patterns are informative. Appendix B.2 (Figure 7) illustrates alternative metrics only on Peptides and does not re-derive Δ_S or the taxonomy. As written, the Deprecate recommendations are an artifact of a particular, unvalidated metric choice.","section":"Section 4.3, Table 2, Definition 2.14"},{"comment":"The claim that mode complementarity is a useful task-independent diagnostic is supported by a correlation computed on the same 13 datasets used to construct the taxonomy. This is a circular validation: the data that motivated the measure are also used to show it predicts performance. An external validation—for example, on the OGB datasets shown in Figure 9, or on held-out datasets with known rankings—is needed before the diagnostic claim can be accepted.","section":"Figure 5, Section 4.2"},{"comment":"The comparator C_{p,q} in Definition 2.11 divides by the q-th root of n^2−n, but the proof of Theorem 2.15 uses the norm without this factor, and the numerical results in Figure 4 are consistent with the unnormalized version. This is not merely cosmetic; it changes all complementarity values and the derived diversity scores. The definition should be corrected or the proof and results reconciled.","section":"Definition 2.11 and proof of Theorem 2.15"},{"comment":"The conclusion that Reddit-M 'lacks performance separability' is based on experiments where the complete-graph and complete-features perturbations failed to train, and Reddit-B has missing GIN runs. Missing two of six conditions on Reddit-M means the separability ordering is only partially observed; the failure may reflect model or resource constraints rather than dataset quality. Please provide the missing runs or temper the conclusion.","section":"Figure 3, Tables 6 and 7"}],"minor_comments":[{"comment":"The term 'Rings' appears in the abstract but should be 'RINGS' for consistency with the rest of the paper.","section":"Abstract"},{"comment":"The word 'unformative' should be 'uninformative'.","section":"Table 14"},{"comment":"The text says 'AURUC' but should say 'AUROC'.","section":"Appendix C.2.2"},{"comment":"The text states that Δ_* scores the ability of d_* to produce non-trivial geometric structure, but for * = f the quantity γ_{e*} measures the structure metric (since φ_{ef} zeroes the features). Clarify the subscript convention to avoid a swapped interpretation of Δ_S and Δ_F.","section":"Definition 2.14"},{"comment":"The threshold for 'high' is based on coarse bins (e.g., ◦ is grouped with + and ++); a sensitivity analysis over these thresholds would strengthen the taxonomy.","section":"Section 4.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a valuable contribution to dataset evaluation, but the taxonomy is load-bearing and currently rests on metric choices that are not validated against the paper's own separability evidence. The authors should be asked to either re-derive the taxonomy under multiple structural metrics and thresholds, or to present the taxonomy as a case study rather than a definitive recommendation. The missing Reddit-M runs should also be addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, it is a real contribution: a mode-perturbation framework with two complementary measures—performance separability (task-dependent) and mode complementarity (task-independent)—plus an associated taxonomy for graph-learning datasets. The math is clean, the experiments are extensive and mostly robust, the code and data are public, and the paper is unusually honest about its limitations. Second, the soft spot is the leap from the mode-diversity measure to the \"deprecate\" recommendations. That leap is not internally contradictory, but it is less grounded than the rest of the work.\n\nWhat is genuinely new: the formalization of mode perturbations into two measures, especially mode complementarity as a task-independent metric-space distance, and the resulting framework for dataset evaluation. The perturbation idea exists in prior work (Errica et al., Bechler-Speicher et al.), but the framework framing and the two-measure decomposition are new. The paper also does the right things with robustness: permutation tests with two statistics and two alpha levels, additional architectures, regression datasets, graph-level analyses, and a reproducible package. Credit where due: this is a well-engineered, careful paper.\n\nWhere it is softer: the structural diversity measure (Diffusion distance at t=1, L1,1 comparator) is assumed, not shown, to capture structural variation that matters for graph learning. The paper's own separability results show COLLAB and the IMDB datasets are structure-informative, yet they are labeled deprecated because Delta_S is low. That is not a contradiction by the paper's own definitions, but it is a large normative step. Appendix B.2 tests alternative metrics on Peptides only, which does not tell you whether the taxonomy would flip for ego-networks. The complementarity-performance correlation in Figure 5 is computed on the same 13 datasets used to build the taxonomy, so it is suggestive, not predictive. Some Reddit-M runs failed; the authors report this, but it weakens the otherwise broad empirical base.\n\nWho this is for: anyone building or choosing graph benchmarks, and anyone working on data-centric graph learning. It deserves a serious referee and will likely be cited. My recommendation: send it to review, but push the authors to validate the taxonomy on a held-out set of datasets or with alternative structural distances before the deprecate/realign labels are treated as guidance for the community. As written, the framework is solid; the verdicts are hypotheses.","headline":"A genuinely useful mode-perturbation framework for evaluating graph datasets, with clean math and extensive experiments; the taxonomy recommendations rest on an unvalidated structural-diversity metric and should be treated as hypotheses, not verdicts.","tokens_in":44898,"tokens_out":1667,"would_cite":true,"duration_ms":19237,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RINGS tests whether graph benchmarks need both structure and features; most of 13 popular datasets fail its quality test.","keywords":["graph-learning datasets","dataset quality evaluation","mode perturbations","performance separability","mode complementarity","benchmark datasets","graph neural networks","evaluation of evaluations"],"falsifier":"Build a dataset whose labels depend on only one mode (or neither) but whose structure and features are geometrically complementary: if mode complementarity still ranks it alongside MolHIV and NCI1, the diagnostic is measuring geometric variety rather than task-relevant information. A cheaper check is to recompute the taxonomy with resistance distance or shortest-path distance as the structural metric; any dataset that switches category under that swap exposes the verdicts' dependence on the diffusion-distance choice.","tokens_in":2067,"feed_emoji":"🕸️","tokens_out":2561,"duration_ms":114882,"temperature":0.7,"pith_summary":"This paper tries to establish a principled way to judge the quality of graph-learning benchmark datasets, rather than judging the models trained on them. Its starting point is that attributed graphs combine two modes — graph structure and node features — and a good benchmark should require information from both. To test this, the paper introduces RINGS, which replaces one mode with an uninformative surrogate (empty, complete, random, or shuffled) and measures two things: performance separability, whether models trained on the original data statistically outperform models trained on the perturbations, and mode complementarity, the geometric distance between the graph's diffusion-distance space and the features' Euclidean space. Applied to 13 popular graph-classification datasets, the framework finds that only 3 satisfy the expected separability relations and that several originals are separably outperformed by their own perturbed versions, and it distills these findings into an actionable taxonomy: keep MolHIV, NCI1, and Peptides; realign five datasets; deprecate five. The contribution matters because it offers a task-independent, model-agnostic diagnostic for dataset quality to complement costly model-based evaluation.","feed_headline":"Only 3 of 13 graph benchmarks pass a new quality test","feed_subtitle":"Randomizing edges or features often matches the original, so those benchmarks don't show graph structure matters.","key_machinery":"The central object is the mode perturbation — a map that replaces the edge set or feature set of an attributed graph with a controlled surrogate, applied element-wise to an entire dataset. Two measures sit on top of it: performance separability, the statistically tested distance between performance distributions of tuned models on the original versus perturbed datasets, and mode complementarity, the normalized $L_{1,1}$ norm between the graph's diffusion-distance matrix and the features' Euclidean distance matrix. A duality theorem, $\\gamma_{c*} = 1 - \\gamma_{e*}$, links the complete and empty perturbations under this comparator, and the derived mode diversity $\\Delta = 1 - |1 - 2\\gamma_{e*}|$ scores how much non-trivial geometric structure a single mode carries. The diffusion distance is sampled over $t \\in [10]$ diffusion steps, so complementarity is a curve rather than a point estimate.","core_discovery":"On its own terms, the paper's central claim is that a good graph-learning dataset is one whose structure and features both carry task-relevant information and whose structural and feature geometries are complementary — and that both properties can be tested by perturbing one mode while leaving the other intact. The paper formalizes mode perturbations (empty, complete, random, and shuffled graphs and features) and two measures built on them. Performance separability compares performance distributions of tuned GNNs on original versus perturbed datasets using permutation tests with the Kolmogorov-Smirnov statistic; when a perturbation matches or beats the original, the perturbed mode is not carrying the information the task needs. Mode complementarity compares the normalized distance matrices of the two modes using the $L_{1,1}$ norm, with a duality theorem relating complete and empty perturbations and a derived mode-diversity score. Across 13 real-world graph-classification datasets, only COLLAB, MolHIV, and NCI1 (with Peptides close behind) pass the expected separability relations, while DD is separably outperformed by its random-graph perturbation, Enzymes by its empty-graph perturbation, and Reddit-B by complete-features and random-graph perturbations. Higher mean mode complementarity is associated with higher AUROC across perturbations, supporting complementarity as a task-independent diagnostic.","pith_inferences":["Because mode complementarity scores depend on the chosen structural and feature metrics, a different metric pair could reorder the taxonomy; the paper tests a small set of alternatives, so its verdicts on individual datasets are less stable than the framework itself.","The framework suggests a constructive use the authors only gesture at: steer dataset construction by maximizing the drop in complementarity under perturbation, actively building benchmarks that make structural and feature information non-redundant rather than auditing existing ones afterward.","The correlation between complementarity and performance is established on 13 datasets and one perturbation family; a stress test on adversarially constructed graphs with high complementarity but useless labels would show whether the diagnostic reflects benchmark utility or just geometric diversity.","A two-stage pipeline is left implicit: screen candidate datasets with complementarity (cheap), then verify with separability (expensive), dividing labor between the task-independent and task-dependent measures."],"forward_implications":["Claims of new GNN capability should come with separability evidence: a model that performs well on the original data but also on random-graph or complete-features perturbations has not shown that it needs the graph structure.","Mode complementarity can serve as a cheap screening diagnostic — it correlates with AUROC across perturbations — so dataset builders can assess candidates before expensive hyperparameter tuning.","The taxonomy gives concrete guidance: keep MolHIV, NCI1, and Peptides as-is; realign AIDS, DD, MUTAG, Reddit-B, and Reddit-M toward better-aligned tasks; deprecate COLLAB, IMDB-B, IMDB-M, Enzymes, and Proteins as graph benchmarks.","The framework extends beyond classification: the appendix applies it to regression datasets (QM9, ZINC-12k) and transformer architectures without changing the core conclusions.","Graph-level analysis of correctly classified sets can expose shortcut learning — Proteins' complete-features perturbation correctly classifies a distinct outlier subset, a sign that models exploit superficial cues rather than meaningful patterns."],"supporting_citations":[{"why":"Shows that some graph-classification tasks can be solved without the graph structure, the observation that performance separability is designed to detect.","marker":"Errica et al. (2020)"},{"why":"Demonstrates that GNNs use graph structure even when it hurts the task, the direct precedent for empty-graph perturbations outperforming the original.","marker":"Bechler-Speicher et al. (2024)"},{"why":"Supplies the diffusion-maps construction behind the diffusion distance used as the structural metric in mode complementarity.","marker":"Coifman & Lafon (2006)"},{"why":"Source of the COLLAB, IMDB-B, IMDB-M, Reddit-B, and Reddit-M datasets evaluated in the paper.","marker":"Yanardag & Vishwanathan (2015)"},{"why":"Introduces the Open Graph Benchmark and the MolHIV dataset, one of the datasets the framework rates as worth keeping.","marker":"Hu et al. (2020)"},{"why":"Provides the TUDataset encodings and node-feature conventions the paper adopts for its evaluated datasets.","marker":"Morris et al. (2020)"},{"why":"Introduces the Peptides dataset and the long-range graph benchmark the paper classifies as worth keeping.","marker":"Dwivedi et al. (2022)"},{"why":"A companion critique of graph-classification benchmarks whose model-centric analysis RINGS extends to the datasets themselves.","marker":"Li et al. (2024)"},{"why":"Documents that purported performance gaps vanish with proper tuning, motivating the paper's perturbation-based standard of proof.","marker":"Tönshoff et al. (2023)"}],"fun_headline_variants":["Only 3 of 13 graph datasets pass a new quality test","Most graph benchmarks fail a test for structural relevance","New test says only 3 of 13 graph datasets need true structure","Rings: perturbation test reveals weak graph datasets"],"cache_read_input_tokens":46976,"weakest_assumption_plain":"The load-bearing premise is that the geometric gap between node-feature distances and graph-diffusion distances, summed with the $L_{1,1}$ norm, captures the complementary information that graph-learning models actually exploit, so that a dataset scoring well on mode complementarity is genuinely a better benchmark; if another pair of metric choices reverses the complementarity orderings, the dataset verdicts would change.","fun_headline_variants_meta":{"raw":{"variants":["Only 3 of 13 graph datasets pass a new quality test","Most graph benchmarks fail a test for structural relevance","New test says only 3 of 13 graph datasets need true structure","Rings: perturbation test reveals weak graph datasets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000651,"raw_usage":{"total_tokens":3045,"prompt_tokens":1067,"completion_tokens":1978,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":683,"completion_tokens_details":{"reasoning_tokens":1910}},"tokens_in":683,"tokens_out":1978,"duration_ms":14465,"temperature":1.0,"reasoning_tokens":1910,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T12:21:04.433148+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a dataset whose labels depend on only one mode (or neither) but whose structure and features are geometrically complementary: if mode complementarity still ranks it alongside MolHIV and NCI1, the diagnostic is measuring geometric variety rather than task-relevant information. A cheaper check is to recompute the taxonomy with resistance distance or shortest-path distance as the structural metric; any dataset that switches category under that swap exposes the verdicts' dependence on the diffusion-distance choice.","supporting_citations":[{"cited_title":"A fair comparison of graph neural networks for graph classification","cited_arxiv_id":null,"evidence_quote":"Shows that some graph-classification tasks can be solved without the graph structure, the observation that performance separability is designed to detect."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the diffusion-maps construction behind the diffusion distance used as the structural metric in mode complementarity."},{"cited_title":"and Vishwanathan, S","cited_arxiv_id":null,"evidence_quote":"Source of the COLLAB, IMDB-B, IMDB-M, Reddit-B, and Reddit-M datasets evaluated in the paper."},{"cited_title":"M., Bause, F., Kersting, K., Mutzel, P., and Neumann, M","cited_arxiv_id":null,"evidence_quote":"Provides the TUDataset encodings and node-feature conventions the paper adopts for its evaluated datasets."},{"cited_title":"P., Ramp \\'a s ek, L., Galkin, M., Parviz, A., Wolf, G., Luu, A","cited_arxiv_id":null,"evidence_quote":"Introduces the Peptides dataset and the long-range graph benchmark the paper classifies as worth keeping."}],"review_version":1}