{"id":"142c02ad-7877-4532-ba17-eb620330446a","arxiv_id":"2503.18001","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"Z-REx is a perturbation-based explainer for heterogeneous link prediction that finds important city features and co-clicked city edges, and claims large Fidelity gains over prior GNN explainers.","lead":"Z-REx explains Zillow's GNN-based real estate recommendations by zeroing out city features and removing graph edges to find what the model relies on. The paper claims Z-REx beats existing GNN explainers by 61% on a self-defined Fidelity metric, but the evaluation is circular, the dataset is private, and the headline numbers are internally inconsistent.","discovery_kind":"new_application","skeptic_critique":null,"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Z-REx, a post-hoc explanation method for user-city link prediction in a heterogeneous GNN recommender (ZiGNN). It identifies important city features by zeroing out each feature and measuring the change in nDCG@K (Eq. 3), and important graph edges by removing candidate edges, including domain-derived co-clicked city edges, and measuring the change in cosine similarity (Eq. 4). The method is evaluated on Zillow interaction data and compared against PaGE-Link, GNNExplainer, and SubgraphX, with the claim that Z-REx outperforms these baselines by 61% in the Fidelity metric.","tokens_in":17760,"tokens_out":6319,"duration_ms":62513,"significance":"If the method were rigorously validated, this would be a useful practical contribution: Z-REx is one of few explainers targeted at heterogeneous link prediction, and the use of domain-specific co-clicked city edges to prune the search space is a sensible engineering idea. The paper also uses a real industrial dataset and reports a scaling analysis (Table 8). However, the central quantitative claim is undermined by a circular evaluation and by inconsistent reported numbers; the current evidence does not establish that Z-REx produces better explanations than the baselines.","major_comments":[{"comment":"The Fidelity metric is identical to the objective used for selection. Z-REx ranks features by ΔnDCG and edges by |Δsim|, and Table 2 then reports as “Fidelity” the drop in nDCG and Δsim after removing the selected items. Any method that greedily chooses high-impact perturbations will dominate this metric, so the reported 29/85/70 differences do not by themselves demonstrate superior explanation accuracy. A minimal control would be to compare against a perturbation-based explainer using the same greedy objective, or to evaluate fidelity on a held-out perturbation set not used during selection.","section":"§5.1, Eq. (3); §5.2, Eq. (4); §6.1"},{"comment":"The reported numbers are internally inconsistent. Table 2 labels the row “nDCG (% decr.)” with Z-REx at 92%, but §6.4 says Z-REx “preserves 92% of the original nDCG”; §6.5 says PaGE-Link “retains only 63% of nDCG” while the table lists 63% in the same row. If the row is percent decrease, then Z-REx preserves 8%/6%, not 92%/94%; if it is retention, the differences in §8 (29%, 85%, 70%) do not follow from the table. Additionally, the 30-day table values imply an 83-percentage-point gap for GNNExplainer, not the 85% stated in §8 and used in the abstract’s 61% average. The headline numbers are therefore not reproducible from the paper’s own table.","section":"§6.4, Table 2; §6.5; §8"},{"comment":"The “ground truth” explanation is not established. §4.2 says ground-truth explanations are generated by identifying the most relevant subgraphs and comparing node features, but the actual evaluation treats observed positive edges as ground truth. Observed interactions are labels for link prediction, not evidence about which substructures the model used. Without a separate annotation study or a causal validation, the claim that Z-REx explanations are “aligned with ground truth” (Sections 1 and 8) is unsupported.","section":"§4.2, §6.4"},{"comment":"The hyperparameter sensitivity study does not validate the explainer. Table 3 reports nDCG@1 of the ZiGNN recommender under different settings, not explanation fidelity or human interpretability, so it cannot justify the choices of hop distance k or edge removal strategy in Z-REx. Moreover, all entries in Table 2 appear to be single-run values with no error bars, confidence intervals, or significance tests, so the reported margins (e.g., 9% vs 92% for GNNExplainer) cannot be distinguished from noise.","section":"§6.6, Table 3"}],"minor_comments":[{"comment":"The function defined on line 17 is named “StructurePerturb” but is called as “StructuralPerturb” on line 3; the names should be consistent.","section":"Algorithm 1"},{"comment":"The parenthetical values such as “(-13%)”, “(-73%)”, and “(-70%)” are not explained in the caption or text; clarify whether they denote relative differences from Z-REx or another quantity.","section":"Table 2"},{"comment":"The URL for the Zillow website is missing the protocol: “https:www.zillow.com” should be “https://www.zillow.com”.","section":"Reference [30]"},{"comment":"The phrase “outperforms them by 61%” should be stated as a percentage-point difference in the reported metric, not as a percent improvement, and should be made consistent with the corrected table values.","section":"§8, abstract"},{"comment":"The axis label “Difference in nDCG ×10^2” is likely a scaling typo; as printed, the values would be implausibly large, so the axis scaling should be checked and corrected.","section":"Figure 5"}],"recommendation":"reject","confidential_remarks":"The circularity of the Fidelity metric is fundamental: because Z-REx’s selection rule and the evaluation metric are the same, the comparison cannot support the paper’s central claim even after correcting the table typos. The paper might be suitable for resubmission after a substantially redesigned evaluation with external validation or human studies, but as it stands it does not meet the bar for a journal publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read: Z-REx is a perturbation-based explainer for GNN link prediction, applied to Zillow's real-estate recommender. The new bit is a domain heuristic—co-clicked city edges—that shrinks the search space for structural perturbation. The method is simple, the problem is real, and the runtime story is plausible. But the headline claim about 61% better Fidelity doesn't hold up under inspection.\n\nThe paper deserves credit for a few things. The co-clicked city step is a sensible domain-specific trick, and the authors are upfront about its limits (Section 5.4 explicitly says the edge-perturbation step needs rethinking for other domains). The case study is a nice concrete illustration, and the appendix includes honest runtime data showing sublinear scaling from 3 to 30 days.\n\nThe soft spot is load-bearing: Fidelity is defined as the drop in nDCG or Δsim after removing the items Z-REx ranks highest. Eq. 3 and Eq. 4 are exactly the selection criteria for features and edges. So the metric measures how well the explainer does at the objective it was optimized against. Comparing against GNNExplainer and SubgraphX, which optimize different objectives, is apples-to-oranges. That is not a minor quibble; it means the central quantitative claim—\"superior explanation accuracy\"—is unsupported. On top of that, Table 2's label (\"nDCG (% decr.)\") contradicts the text (\"preserves 92%\"), and the numbers don't add up (92−9=83, not the claimed 85). No error bars, no significance tests, and the proprietary dataset plus missing code make independent verification impossible.\n\nIs the method worthless? No. As a practical tool for one recommender system, it's plausible and likely useful. But as a research claim about explanation quality, the paper overreaches. The evaluation needs a proper baseline that also optimizes the same perturbation objective, a corrected table, and ideally a human study or a sanity check on whether the identified features and edges actually match user-perceived relevance.\n\nWho's it for? Practitioners at the workshop level. A reader interested in XAI evaluation pitfalls would get value from this as a cautionary example. I'd send it to review if I were the editor—the core idea isn't dead, but the evidence needs to be rebuilt. A major revision could make it a serviceable contribution.","headline":"Plausible domain-specific perturbation explainer for GNN link prediction, but the paper's headline fidelity claim is circular as written and needs a rebuilt evaluation.","tokens_in":18092,"tokens_out":3536,"would_cite":false,"duration_ms":450616,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces Z-REx, a perturbation-based explainer that identifies the features and graph connections a GNN recommender relies on, and reports it outperforms three prior explainers on fidelity.","keywords":["graph neural network explainability","link prediction","recommendation systems","feature perturbation","structural perturbation","fidelity metric","heterogeneous graphs","real estate"],"falsifier":"Train Z-REx on a synthetic heterogeneous graph with planted influential subgraphs and features, then check whether the edges and features Z-REx selects match the planted ones; if the match rate is no better than chance, or if removing randomly chosen edges of the same size produces a comparable drop in nDCG, the interpretability claim would be refuted.","tokens_in":17415,"feed_emoji":"🏠","tokens_out":11498,"duration_ms":79560,"temperature":0.7,"pith_summary":"The paper tries to close a gap in explaining graph-neural-network recommenders: existing GNN explainers are built for node or graph classification, but recommending an item to a user is a link-prediction task. Z-REx explains a single user-to-city recommendation by perturbing the model in two steps: it zeroes out city features one at a time and measures the drop in ranking quality, then removes graph edges and measures the change in embedding similarity. Domain knowledge about co-clicked cities keeps the edge search tractable on a large, sparse real-world real-estate graph. The paper reports that Z-REx's explanations degrade recommendation quality far more than those of PaGE-Link, GNNExplainer, and SubgraphX, and takes that as evidence that Z-REx captures what the model truly uses.","feed_headline":"Beats three explainers with a two-step search for what matters","feed_subtitle":"It finds the features and links a housing recommender truly relies on, beating three prior explainers.","key_machinery":"The central machinery is a two-stage perturbation search. Stage one, feature perturbation, zeroes out each city feature one at a time and records the change in nDCG@K (Eq. 3); features whose removal lowers ranking quality become the influential feature set. Stage two, structural perturbation, collapses the heterogeneous graph into a user-city graph, adds edges between co-clicked cities, extracts a k-hop subgraph around the target user, removes each edge in turn, and records the change in cosine similarity between user and target-city embeddings (Eq. 4). The co-clicked-city insight—cities that share a common predecessor user are proxies for shared preference—is the domain-knowledge constraint that keeps the search space sub-linear and makes whole-graph, instance-level explanation feasible at the scale of tens of millions of edges.","core_discovery":"Z-REx's central claim is that a faithful explanation for a GNN link prediction is the smallest set of node features and graph edges whose perturbation most changes the model's ranking behavior. For a given user and recommended city, it ranks features by zeroing each one out and computing the change in nDCG@K (Eq. 3), and ranks edges by removing each from a k-hop user-city subgraph and computing the change in cosine similarity between user and city embeddings (Eq. 4); the two ranked lists together are the explanation. The paper's ground-truth comparison treats observed user-city interactions as positive and sampled non-interactions as negative, and evaluates fidelity by how much the identified features and edges reduce nDCG and similarity when removed. Z-REx is designed for whole heterogeneous graphs, avoiding path-based or ego-graph limitations, and its use of co-clicked city edges is what makes the edge search scalable. The paper reports gains of 29% over PaGE-Link, 85% over GNNExplainer, and 70% over SubgraphX in the Fidelity metric.","pith_inferences":["The paper leaves implicit that its feature-importance lists could be validated against a full feature-ablation study of the recommender, which would test whether the explanation identifies causal drivers or merely high-leverage perturbations.","A testable extension is to apply the same two-stage perturbation logic to edge-classification or community-detection tasks by swapping in a task-appropriate ranking metric.","Because the Fidelity metric rewards explanations that maximally degrade recommendations, Z-REx may favor rare high-impact edges over the everyday connections users would recognize; a user study on whether the explanations feel intuitive would settle this.","Since the paper treats observed interactions as ground-truth positive evidence, distinguishing weak negatives from dislikes could change which edges are considered important, and measuring that sensitivity is a direct follow-up."],"forward_implications":["For a given user-to-city recommendation, Z-REx outputs an ordered list of important city features and an ordered list of important graph edges, which can be rendered as a small subgraph of co-clicked cities for a user-facing explanation.","Because the explanation search is restricted to co-clicked edges, the method remains tractable on month-long interaction graphs with tens of millions of edges, and the paper reports near-constant per-user inference time between 3-day and 30-day graphs.","If the Fidelity gains are real, Z-REx gives system administrators a tool for auditing which features and connections drive a GNN recommender's ranking decisions.","The same two-step feature-then-structure perturbation recipe transfers to other link-prediction recommenders, provided a domain-specific edge type is chosen to replace co-clicked cities, as the paper's generalizability section argues."],"supporting_citations":[{"why":"This is the mask-optimization GNN explainer that serves as the main general-purpose baseline; Z-REx claims a much larger nDCG drop when its selected features and edges are removed.","marker":"[1]"},{"why":"This is the Monte-Carlo subgraph-search explainer that also supplies the original Fidelity metric definition, which the paper adapts to nDCG for link prediction.","marker":"[3]"},{"why":"This is the closest prior work, a path-based explainer for heterogeneous link prediction; its ego-graph limitation is the gap Z-REx says it closes.","marker":"[4]"},{"why":"This supplies the domain-knowledge premise that location is the most important factor in home buying, justifying the user-city graph construction and the co-clicked-city heuristic.","marker":"[29]"}],"fun_headline_variants":["Z-REx: GNN explainer for real estate, 61% better fidelity","Why this house? Z-REx explains GNN real estate recs","Beats three GNN explainers on fidelity by 61%","Z-REx: perturbation-based GNN explanations for housing recs","Z-REx: human-readable GNN explanations for real estate"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the Fidelity metric—the drop in nDCG@K and cosine similarity when Z-REx's selected features and edges are removed—measures how much the model actually relies on those features and edges, and that observed positive versus negative edges are a valid ground truth for explanation quality.","fun_headline_variants_meta":{"raw":{"variants":["Z-REx: GNN explainer for real estate, 61% better fidelity","Why this house? Z-REx explains GNN real estate recs","Beats three GNN explainers on fidelity by 61%","Z-REx: perturbation-based GNN explanations for housing recs","Z-REx: human-readable GNN explanations for real estate"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00085,"raw_usage":{"total_tokens":3724,"prompt_tokens":1002,"completion_tokens":2722,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":618,"completion_tokens_details":{"reasoning_tokens":2627}},"tokens_in":618,"tokens_out":2722,"duration_ms":16844,"temperature":1.0,"reasoning_tokens":2627,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T10:32:42.081476+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train Z-REx on a synthetic heterogeneous graph with planted influential subgraphs and features, then check whether the edges and features Z-REx selects match the planted ones; if the match rate is no better than chance, or if removing randomly chosen edges of the same size produces a comparable drop in nDCG, the interpretability claim would be refuted.","supporting_citations":[{"cited_title":"Gnnexplainer: Generating explanations for graph neural networks,","cited_arxiv_id":null,"evidence_quote":"This is the mask-optimization GNN explainer that serves as the main general-purpose baseline; Z-REx claims a much larger nDCG drop when its selected features and edges are removed."},{"cited_title":"On explainability of graph neural networks via subgraph explorations,","cited_arxiv_id":null,"evidence_quote":"This is the Monte-Carlo subgraph-search explainer that also supplies the original Fidelity metric definition, which the paper adapts to nDCG for link prediction."},{"cited_title":"Page-link: Path-based graph neural network explanation for heterogeneous link prediction,","cited_arxiv_id":null,"evidence_quote":"This is the closest prior work, a path-based explainer for heterogeneous link prediction; its ego-graph limitation is the gap Z-REx says it closes."},{"cited_title":"Taxonomic recommendations of real estate properties with textual attribute information,","cited_arxiv_id":null,"evidence_quote":"This supplies the domain-knowledge premise that location is the most important factor in home buying, justifying the user-city graph construction and the co-clicked-city heuristic."}],"review_version":1}