{"id":"f5e0ad65-568b-4e0c-9a08-85ea8f64b9ac","arxiv_id":"2504.18353","paper_version":1,"verdict":"UNVERDICTED","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"A doctoral proposal to adapt individual fairness testing techniques to graph neural networks, containing no experimental results.","lead":"This paper describes a PhD research plan for building a framework to test whether graph-based AI models treat similar people similarly. It reviews existing fairness-testing methods and proposes adapting them to graph neural networks, with planned industry case studies.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The proposed structure-preserving IDI generation may produce out-of-distribution counterfactuals because graph edges depend on features, undermining the naturalness assumption central to the framework.","rationale":"The reader identified the assumption that changing only the sensitive attribute while preserving structure will expose fairness violations in natural cases. My concern sharpens this: the bottleneck is not just whether predictions flip, but whether the generated counterfactuals are natural (i.e., on the data manifold). Because graph structure and node features are interdependent, forcing topology to remain fixed likely creates out-of-distribution instances that invalidate the test oracle. This is a load-bearing risk for the proposed framework's feasibility. However, the paper is a PhD proposal with no implemented system or empirical results, so the appropriate verdict remains UNVERDICTED, consistent with the reader. The concern does not change the verdict; it adds a specific technical risk that the proposal must address. I partially agree with the reader's weakest assumption because the reader focused on prediction flips while I focus on the prior condition of naturalness, which is where the framework's feasibility is least secure.","tokens_in":11760,"tokens_out":5435,"duration_ms":58412,"concrete_test":"Implement the proposed IDI generation on a dataset with known feature-structure dependence, e.g., the German credit graph or a stochastic block model with homophily. Train a GNN, then for each node flip only the sensitive attribute while preserving the original graph structure. Use a learned conditional density model or a kernel two-sample test to measure whether the flipped feature vector given the unchanged neighborhood is out-of-distribution relative to observed nodes with that sensitive attribute. If, for example, more than 5% of generated IDIs yield a significant likelihood drop, the naturalness assumption fails and the framework's test cases cannot be interpreted as valid individual fairness violations. A complementary check: compare prediction flips against a baseline that also rewires edges to match the flipped features and see whether the violation rate changes substantially.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The framework's central claim (Sec. 4.2) requires that structure-preserving perturbations generate natural, realistic individual discriminatory instances (IDIs) for GNNs. The paper asserts in Sec. 2 (Phase 2) that 'generating IDIs for GNNs requires preserving both the node's features and the structural dependencies induced by the graph topology.' This implies keeping the graph topology fixed while flipping the sensitive attribute. However, in many real and semi-synthetic graph datasets, edges are constructed from feature similarity (the paper itself flags this for German and Credit in Sec. 2.1). Under such generative models, a node's features and its neighborhood are statistically dependent: a node with the opposite sensitive attribute but identical remaining features is unlikely to have the same neighbors. Preserving degree and neighborhood while flipping only the sensitive attribute therefore creates a counterfactual that lies in a low-density region of the joint (features, structure) distribution. A prediction change on such out-of-distribution inputs may reflect OOD artifacts rather than genuine discrimination, making the test oracle unreliable. The paper does not define how naturalness will be measured or guaranteed, nor does it discuss the tension between preserving structure and remaining on the data manifold. Without addressing this, the core contribution of 'effective individual fairness testing' is not established, even as a plan.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript is a PhD-project research plan (submitted to EASE 2025) rather than a completed technical study. It argues that individual fairness testing for Graph Neural Networks is underexplored and proposes a three-phase program: (i) a systematic literature review that produces an individual-fairness taxonomy, reported as substantially complete (1,118 papers retrieved and 136 selected); (ii) a GNN-specific fairness-testing framework whose components are structure-preserving test-case generation of individual discriminatory instances, a layer-wise fairness neuron coverage adequacy criterion, test oracles, execution, and retraining; and (iii) validation through industrial case studies with Deloitte on graph-based LLMs. No algorithms, formal definitions, experiments, or evaluation results are presented; the only concrete results are the SLR process statistics and the agreed industrial use case.","tokens_in":11982,"tokens_out":5941,"duration_ms":55966,"significance":"The planned framework targets a real and recognized problem: existing fairness test-case generation methods operate on IID data and do not account for message passing, so GNN-specific testing approaches are needed. The paper's strengths are its clear research questions, its explicit acknowledgement that naive graph conversions from tabular data are problematic (Sec. 2.1, citing Qian et al. [42]), and the planned industrial validation through action research. If the framework were implemented and evaluated, the proposed structure-preserving IDI generation and GNN-specific adequacy metric could be useful contributions. However, the current manuscript contains no technical results, no formal definitions, and two central design choices—the constraints for IDI generation and the test oracle—raise concerns that must be resolved before the plan can be considered viable. The paper also explicitly acknowledges limitations in existing datasets and commits to using synthetic generation methods, which is commendable, but the overall contribution is at the proposal stage.","major_comments":[{"comment":"The paper asserts that generating IDIs for GNNs requires preserving both the node's features and the structural dependencies induced by graph topology, and that structure-preserving perturbations should maintain node degree and neighborhood consistency. For the German and Credit datasets discussed in Sec. 2.1, however, edges are constructed from feature similarity, so a node's neighborhood is a function of its features. Flipping the sensitive attribute while holding everything else, including the edge set, fixed creates counterfactuals that lie in a low-density region of the joint (features, structure) distribution; a prediction flip on such inputs may reflect out-of-distribution artifacts rather than discrimination. The manuscript does not define how naturalness will be measured or how the tension between structure preservation and remaining on the data manifold will be resolved. This directly affects the central claim in Sec. 4.2 that the framework enables effective individual fairness testing in GNNs.","section":"Sec. 2 (Phase 2) and Sec. 2.1"},{"comment":"The statement 'no dedicated research on individual or group fairness testing specifically for GNNs' is contradicted by the paper's own related-work section, which lists REDRESS [8], IFMR [29], FairGAE [12], and InFoRM [23] as individual-fairness methods for GNNs. If the intended distinction is that none of these is a testing approach as opposed to a mitigation or measurement approach, that distinction should be made explicit and defended; as written, the research-gap claim is overstated.","section":"Sec. 1.2 and Sec. 4.2"},{"comment":"The test oracle described in the framework uses statistical parity difference, a group-fairness metric that compares favorable outcome rates across protected groups. The paper's own definition of an individual discriminatory instance is an input whose prediction changes when only the protected attribute changes; the corresponding oracle should compare the model's predictions on the original and perturbed instances. Using a group-level parity metric is misaligned with individual fairness testing and would not detect the individual-level violations the framework aims to find.","section":"Sec. 2 (Phase 2)"},{"comment":"The proposed 'layer-wise fairness neuron coverage' adequacy criterion is presented as a novel contribution but is never defined. The manuscript does not state what makes a neuron fairness-relevant, how layer-wise coverage is computed, or how this criterion extends existing neuron-coverage and fairness-adequacy metrics such as those in [64]. Without these details, the criterion is only a name and cannot yet be evaluated as a contribution.","section":"Sec. 2 (Phase 2) and Sec. 4.2"}],"minor_comments":[{"comment":"The author email address contains a typo ('tilbuurguniversity.edu'), and the advisor email shown in the header is inconsistent with the spelled-out affiliation details.","section":"Title page"},{"comment":"References [53] and [54] are the same paper (Wang, Narasimhan, Yao, and Zhang, ICDM 2023) but are listed as if they were distinct sources.","section":"References"},{"comment":"The third research question is labeled RQ2 immediately after SQ2; using the same numbering style for sub-questions and the application-level question is confusing.","section":"Sec. 1.3"},{"comment":"The text attributes 'gradient-guided adversarial sampling' to reference [55] (MAFT), but that paper proposes a zero-order gradient search; the canonical white-box adversarial sampling method is [63]. The citation should be checked.","section":"Sec. 2 (Phase 2)"},{"comment":"The conclusion says 'In the section phase' where 'the second phase' is intended; this appears to be a typo.","section":"Sec. 5"}],"recommendation":"major_revision","confidential_remarks":"This is a research-plan paper with no technical results, formal definitions, or evaluation. For a venue that publishes vision or roadmap papers, the issues above are addressable in a revision. The editor may also wish to consider whether the venue expects preliminary results; if so, the paper is borderline. The research-gap claim relies on the author's unpublished SLR, and the paper's own related work partially contradicts the gap statement; the SLR findings should be made available or described in enough detail for verification."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a research plan, not a results paper, and it is honest about that. The structure is sensible: SLR, proposed framework, industry evaluation. The writing is clear and the plan is executable. Credit where due: the author flags the known issues with feature-similarity graph datasets (Sec. 2.1), and the idea of adapting fairness testing to GNNs via structure-preserving perturbations is a plausible extension of the IID testing literature, even if not a fundamental new insight. The taxonomy from the SLR, if delivered, would be a useful service to the field.\n\nThat said, the central claim that no dedicated fairness testing exists for GNNs is overstated. The paper's own related work lists InFoRM, REDRESS, and FairGAE, which are individual fairness methods for graphs. They are not testing frameworks in the software-engineering sense, so the gap is partly real, but the blanket statement needs qualification.\n\nThe bigger soft spot is the naturalness assumption, which the stress-test note correctly identifies. The plan says IDIs for GNNs should preserve node degree and neighborhood consistency while flipping the sensitive attribute. But on datasets like German and Credit, edges are constructed from feature similarity, so features and neighborhood are statistically dependent. Flipping the sensitive attribute while fixing the rest and the topology produces a point in a low-density region of the joint distribution. A prediction change at such an out-of-distribution point may reflect OOD artifacts, not discrimination. The paper does not say how naturalness will be defined or measured, and it does not connect its own criticism of synthetic datasets to this problem. That is a genuine unresolved tension, and it is the load-bearing part of the proposed framework.\n\nMinor issues: no formal definitions, no algorithms, no experiments. That is expected for a plan paper, but it means the framework is currently a description, not a method. The proposed layer-wise fairness neuron coverage is named but not specified.\n\nFor a PhD progress paper at a venue like EASE, this is acceptable. It is coherent, and the author is not pretending to have results. If the venue publishes research-in-progress, I would send it to a referee who can push on the naturalness problem and the gap claim. I would not cite it as a contribution in the next year, since nothing is implemented yet.","headline":"A clearly written PhD research plan that overstates its gap claim and leaves the naturalness of its proposed test cases unresolved; worth a desk read, not a results paper.","tokens_in":12456,"tokens_out":1340,"would_cite":false,"duration_ms":14775,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims individual fairness testing for GNNs must use structure-preserving test generation and layer-wise neuron coverage, because message passing makes isolated feature perturbations miss how bias propagates through the graph.","keywords":["individual fairness","graph neural networks","fairness testing","individual discriminatory instances","structure-preserving perturbations","neuron coverage","test adequacy","bias mitigation"],"falsifier":"Run the proposed generator on a trained GNN over a dataset with strong same-group clustering: if flipping a node's sensitive attribute with degree and neighborhood held fixed changes no predictions across a large sample of nodes, the framework's central generation mechanism fails on that setting.","tokens_in":11540,"feed_emoji":"⚖️","tokens_out":5197,"duration_ms":52897,"temperature":0.7,"pith_summary":"This PhD project paper proposes a testing framework for individual fairness in graph neural networks (GNNs), an area where existing fairness tests for independent data do not apply. The author argues that GNN predictions are shaped by neighborhood aggregation, so a fair test must create individual discriminatory instances by changing only the sensitive attribute of a node while preserving its degree and neighborhood structure. It also introduces layer-wise fairness neuron coverage as a test adequacy criterion, letting test suites be judged by how much of the fairness-relevant network they exercise. Alongside the framework, the project contributes a taxonomy of individual fairness definitions, metrics, testing methods, and mitigation strategies from a systematic literature review. If the plan succeeds, practitioners would gain a concrete way to detect and mitigate cases where a single node's protected attribute changes its outcome, including in graph-based LLM applications.","feed_headline":"GNN fairness test flips sensitive traits without breaking the graph","feed_subtitle":"Most fairness tests assume independent data, but GNN outcomes spread through neighbors; this plan tests nodes the way GNNs learn.","key_machinery":"The central objects are individual discriminatory instances (IDIs), defined as inputs where changing only a protected attribute flips the model's prediction, together with structure-preserving perturbations that keep node degree and neighborhood consistency so the generated graph instances remain natural. Around these, the framework builds four more components: a test oracle such as statistical parity difference to decide whether a violation occurred, a retraining step to mitigate found violations, and a layer-wise fairness neuron coverage metric to judge whether a test suite has exercised fairness-relevant parts of the network. The structural constraint is what distinguishes the technique from IID fairness test generation, which perturbs feature vectors in isolation and cannot capture GNN message-passing behavior.","core_discovery":"The paper's central claim is that fairness testing for GNNs has to be topology-aware: because predictions spread through message passing, a test input cannot be judged in isolation. The proposed fix is a GNN-specific testing pipeline: generate individual discriminatory instances (IDIs) by flipping only a node's protected attribute while preserving its degree and neighborhood distribution; use test oracles such as statistical parity difference to flag violations; and measure test completeness with a new layer-wise fairness neuron coverage criterion. The author argues these techniques can be adapted from existing approaches such as gradient-guided adversarial sampling and GAN-based instance generation by adding topological constraints. The paper is a research plan rather than a completed empirical study; it reports an SLR as progress and proposes to evaluate the framework on synthetic and industrial graph applications, including graph-based LLMs. It also acknowledges that test case generation has high complexity and scalability issues, which it leaves to future work.","pith_inferences":["A natural extension is to treat surviving many structure-preserving sensitive-attribute flips as a certification signal: a model with no flips on a large sample may be robustly fair at the individual level, not just fair on the tested cases.","The framework could be validated on synthetic graphs with deliberately planted bias; if the generator reliably finds the planted discriminatory instances, both the generation and coverage criteria are confirmed.","Layer-wise neuron coverage alone does not indicate how severe a fairness violation is, so pairing it with a measure of prediction-change magnitude or confidence shift would strengthen the adequacy metric.","Because the generated instances differ only in a protected attribute while keeping graph structure, they could double as counterfactual explanations, connecting fairness testing to model explainability."],"forward_implications":["If the framework works, GNN auditors can point to specific nodes whose protected attribute flip changes a prediction while the surrounding graph stays intact, giving concrete evidence of individual unfairness.","The layer-wise neuron coverage criterion would give a quantitative measure of how complete a fairness test suite is, analogous to coverage criteria in software testing.","Structure-preserving generation would catch biases that propagate through neighborhoods, which IID test-case methods would miss.","Retraining on the generated discriminatory instances offers a mitigation path that aims to reduce individual unfairness without sacrificing model accuracy.","Applied to graph-based LLMs, the framework could turn fairness testing into a routine audit step for enterprise knowledge-graph question answering systems."],"supporting_citations":[{"why":"Supplies the gradient-guided zero-order search method for fairness test generation that the framework adapts to graph-structured data.","marker":"[55]"},{"why":"Provides the white-box adversarial sampling approach that serves as a baseline fairness testing technique to extend.","marker":"[63]"},{"why":"GAN-based generation of natural individual discriminatory instances is one of the techniques the framework extends with topological constraints.","marker":"[57]"},{"why":"Survey that supplies the definitions of test oracles and test adequacy and documents existing fairness testing approaches.","marker":"[5]"},{"why":"Establishes that GNN data is non-IID, which is the central reason IID fairness tests cannot transfer directly.","marker":"[66]"},{"why":"Survey of fairness in graph neural networks that identifies the research gap in individual fairness testing.","marker":"[4]"},{"why":"Synthetic graph dataset generation method used to avoid poorly constructed semi-synthetic benchmarks in fairness evaluation.","marker":"[42]"}],"fun_headline_variants":["Topology-aware fairness testing for GNNs keeps connections intact","Flip protected traits, preserve degrees: new GNN fairness oracle","Graph-structure-preserving fairness checks for neural net predictions","Fairness testing that treats GNNs like graphs, not tabular data","Individual fairness for GNNs: test by flipping with neighborhood constraints"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The plan rests on the idea that in realistic GNNs, changing only a node's sensitive attribute while preserving its graph neighborhood will often change the prediction; if such flips rarely change outcomes in practice, the generated test cases will fail to expose unfairness.","fun_headline_variants_meta":{"raw":{"variants":["Topology-aware fairness testing for GNNs keeps connections intact","Flip protected traits, preserve degrees: new GNN fairness oracle","Graph-structure-preserving fairness checks for neural net predictions","Fairness testing that treats GNNs like graphs, not tabular data","Individual fairness for GNNs: test by flipping with neighborhood constraints"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000692,"raw_usage":{"total_tokens":3121,"prompt_tokens":924,"completion_tokens":2197,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":540,"completion_tokens_details":{"reasoning_tokens":2108}},"tokens_in":540,"tokens_out":2197,"duration_ms":16120,"temperature":1.0,"reasoning_tokens":2108,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:17:50.417586+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the proposed generator on a trained GNN over a dataset with strong same-group clustering: if flipping a node's sensitive attribute with degree and neighborhood held fixed changes no predictions across a large sample of nodes, the framework's central generation mechanism fails on that setting.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes that GNN data is non-IID, which is the central reason IID fairness tests cannot transfer directly."}],"review_version":1}