{"id":"55af7089-37c0-4444-9222-94f7c3f33797","arxiv_id":"2501.07598","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"AutoGNR combines non-recursive, per-hop and per-type aggregation with differentiable architecture search to automatically select useful node type combinations for heterogeneous graph classification.","lead":"AutoGNR is a graph neural network that aggregates information from each hop and each node type separately, and uses automated search to pick which type combinations to keep. It reports accuracy gains over existing heterogeneous graph models on citation, movie, and large-scale benchmarks, offering a way to avoid hand-designed meta-paths.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Abstract's 'consistently outperforms' is contradicted by Table 3: AutoGNR is numerically below RpHGNN in at least four normal-scale cells, and no significance tests are reported to support the claim.","rationale":"The reader's conditional verdict is appropriate, but the most load-bearing weakness is not the task-dependent search-space constraint identified in the reader's weakest_assumption. In AutoGNR's non-recursive design, every aggregation path starts from the anchor node and only uses raw features of nodes at each hop; paths among other node types that do not involve the anchor are never part of the anchor's computation. Thus the \"drop paths among other types\" constraint is mostly a restatement of the anchor-centric architecture rather than a risky extra assumption that could hide useful signal. The direct problem is the abstract's \"consistently outperforms\" statement. Table 3 shows at least four normal-scale cells where RpHGNN has a higher mean, and the reported standard deviations overlap; without paired tests one cannot tell whether the claimed superiority is real or noise. Large-scale results in Table 6 are uniformly in AutoGNR's favor, so the method may still be valuable, but the normal-scale evidence does not justify the unqualified claim. A paired-test re-analysis is the single check that would settle it. Since this concern is an overstatement rather than a demonstrated fatal flaw, the conditional verdict stands.","tokens_in":19399,"tokens_out":10082,"duration_ms":98350,"concrete_test":"Obtain the per-run scores behind Table 3 (the paper says 50 runs per cell) and perform paired significance tests comparing AutoGNR against the best baseline in each of the 18 normal-scale cells, e.g., paired t-test or Wilcoxon signed-rank with Holm-Bonferroni correction. Report the number of cells where AutoGNR is significantly better, and effect sizes for the four cells where it is numerically lower (ACM 100% Macro/Micro, IMDB 25%/100% Macro). If the significant-better count is much below 18, or the numerically lower cells are not within noise, the abstract claim should be revised from \"consistently outperforms\" to \"competitive\" or \"generally outperforms\".","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is that the paper's headline empirical claim—\"AutoGNR consistently outperforms state-of-the-art methods on both normal and large scale real-world HIN datasets\" (abstract)—is not actually supported by the normal-scale results in Table 3. In four of the eighteen normal-scale cells, AutoGNR is numerically worse than RpHGNN: ACM 100% training Macro-F1 (92.99±0.93 vs. 93.02±0.58), ACM 100% Micro-F1 (92.96±0.86 vs. 92.97±0.59), IMDB 25% Macro-F1 (62.52±1.77 vs. 63.69±1.56), and IMDB 100% Macro-F1 (71.23±0.97 vs. 71.37±1.79). The paper reports only means and standard deviations, with no paired significance tests or effect sizes. Since the central contribution is empirical superiority, the abstract's \"consistently outperforms\" wording overstates the evidence; the conclusion's more cautious \"generally outperforms\" is the most that Table 3 supports. This is a load-bearing concern because the method's practical advantage on normal-scale HINs is part of the headline claim, not a secondary detail.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AutoGNR, a framework for heterogeneous information network learning that replaces recursive message passing with a non-recursive, per-hop aggregation mechanism and uses differentiable neural architecture search to select, for each hop, which combinations of node types to aggregate. The search space includes single types, type combinations, and a zero option for dropping a hop, and is constrained to paths related to the target node type. The method is evaluated on node classification across three normal-scale datasets (ACM, DBLP, IMDB) and two large-scale datasets (PubMed, DBLP2), with runtime comparisons and an ablation study. The central claims are that non-recursive aggregation reduces noise from uncorrelated intermediate types, that the tailored search space is both comprehensive and tractable, and that AutoGNR consistently outperforms state-of-the-art baselines.","tokens_in":19704,"tokens_out":8483,"duration_ms":76868,"significance":"If the empirical claims hold, AutoGNR is a useful contribution: it combines non-recursive aggregation with NAS in a way that avoids manual meta-path design, and the search space explicitly includes combinations of node types rather than only single types. The experimental coverage is substantial—five datasets, five-fold cross-validation with 50 runs per setting, large-scale graphs including DBLP2 with about two million nodes, and runtime comparisons. The paper also provides a useful analysis of the architectures selected by the search. The main weakness is that the headline claim of consistent superiority is not fully supported by the normal-scale results, where the strongest baseline RpHGNN is numerically better in several cells, and no significance tests are provided.","major_comments":[{"comment":"The abstract's claim that AutoGNR 'consistently outperforms state-of-the-art methods' is not supported by the normal-scale results in Table 3. In four of the eighteen normal-scale cells AutoGNR is numerically below RpHGNN: ACM 100% Macro-F1 (92.99 vs. 93.02), ACM 100% Micro-F1 (92.96 vs. 92.97), IMDB 25% Macro-F1 (62.52 vs. 63.69), and IMDB 100% Macro-F1 (71.23 vs. 71.37). Since only means and standard deviations are reported, with no paired significance tests or effect sizes, the evidence does not establish consistent superiority over the strongest baseline on normal-scale HINs. The more cautious 'generally outperforms' wording used in Section 5.4 is the most that Table 3 supports; I ask the authors to either add appropriate statistical tests (e.g., paired tests over the 50 runs) or revise the abstract and contribution statements accordingly.","section":"Abstract; Table 3"},{"comment":"The task-dependent search-space constraint is a central component of the method, but its description is internally inconsistent. The text says that for author classification one can 'drop the unnecessary message passing paths related to the representations of other node types, i.e., paths from A to P, from A to C, and between C and P,' yet Fig. 1(b) and Table 4 include P at 1-hop and C at 2-hop for the author anchor. Please clarify what exactly is dropped and justify why paths that do not return to the target type can be excluded without loss of task-relevant information; currently this is asserted rather than validated, and it is the main place where the method could miss useful cross-type structure.","section":"Section 4.3"}],"minor_comments":[{"comment":"There is a typo in the introduction: 'facilatate' should be 'facilitate'.","section":"Section 1"},{"comment":"The notation '|C|max = 2|T|' appears to be a formatting error; the number of all possible node type combinations should be 2^{|T|}, not 2|T|.","section":"Section 4.3"},{"comment":"The experimental protocol says the model is run 10 times with 10 random seeds, while Table 3 and Figures 3-4 report results averaged over 50 runs; please clarify that the 50 runs come from 5-fold cross-validation times 10 seeds, or otherwise reconcile the numbers.","section":"Section 5.3"},{"comment":"The text describes SeHGNN and RpHGNN as having 'non-recursive updates' in the context of strong performance, but Table 2 marks both as facing the problem of unnecessary feature mixing; please reconcile this apparent inconsistency.","section":"Section 5.4; Table 2"},{"comment":"In the scalability analysis, 'GGAT outperformed GCN' should read 'GAT outperformed GCN'.","section":"Section 5.7"},{"comment":"The row 'AutoGNR (First Order)' has exactly the same numbers as AutoGNR on ACM and IMDB; please clarify whether this is expected or a reporting error.","section":"Table 7"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for TKDE and the method is clearly presented. The main issue is that the abstract's 'consistently outperforms' claim overstates the normal-scale results, and the absence of significance tests makes it hard to judge whether the small gaps against RpHGNN are meaningful. I would support publication after the authors either provide rigorous statistical comparisons or calibrate the claims to what the tables actually show."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: AutoGNR is a sensible, clearly written NAS method for heterogeneous GNNs. The genuinely new piece is the per-hop search space over subsets of node types plus a task-dependent pruning rule, and the paper shows that learned architectures beat an all-nodes aggregate on ACM and IMDB. That is a real result, and the large-scale numbers on PubMed and DBLP2 are the strongest evidence in the paper.\n\nWhat it does well: non-recursive aggregation is not new, but combining it with differentiable search is reasonably executed. Experiments are extensive: five datasets, multiple training ratios, 50 runs per cell. The ablations are informative, especially the supernet/weight-transfer/retrain comparison and the hop-selection frequency analysis, which actually demonstrate the mechanism works. The paper is honest about the role of K and the efficiency trade-offs.\n\nSoft spots, in proportion: the abstract says \"consistently outperforms state-of-the-art,\" but in Table 3 AutoGNR is numerically below RpHGNN in four normal-scale cells (e.g., IMDB 25% Macro-F1: 62.52 vs 63.69), and no significance tests are reported. The gaps are small and AutoGNR wins most cells and all large-scale cells, so this is a wording problem rather than a collapse. Still, \"generally outperforms\" is the accurate claim. Also, no code or data is released, which matters because several wins over RpHGNN are under one Macro-F1 point. Paired significance tests across the 50 runs would be cheap and should be added.\n\nThe task-dependent search space is a real assumption: it restricts aggregation paths to those that start from or return to the target node type. The authors motivate it well and the empirical analysis supports it on these datasets, but it is a constraint that could hide useful cross-type paths in other HINs. I would like to see this discussed as a limitation or tested by relaxing it.\n\nWho it is for: people working on NAS for heterogeneous graphs, and practitioners who want an automated alternative to manual meta-paths. It deserves a serious referee; the method is coherent, the experiments are broad, and the issues are addressable. My recommendation: send it to review, and ask the authors to soften the abstract, add significance tests or effect sizes, and release code. That would move it from conditional to accept.","headline":"Useful NAS-for-HIN paper with a real but narrow novelty; the abstract's 'consistently outperforms' overstates Table 3 and should be fixed before acceptance.","tokens_in":20216,"tokens_out":2603,"would_cite":true,"duration_ms":25462,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AutoGNR claims that replacing recursive message passing with per-hop, per-type aggregation plus differentiable architecture search consistently improves node classification on heterogeneous information networks.","keywords":["heterogeneous information networks","graph neural networks","non-recursive message passing","neural architecture search","node classification","differentiable search","meta-path","large-scale graphs"],"falsifier":"A concrete test would be a heterogeneous graph where the label of a target node depends mainly on a relationship among non-target types that never touches the target type—for example, predicting a paper's field from venue-venue citation patterns that bypass the paper node itself. If a recursive or unconstrained model can exploit that path and AutoGNR's task-dependent search space cannot, the claimed generality of the method would be refuted.","tokens_in":19212,"feed_emoji":"🕸️","tokens_out":7392,"duration_ms":67507,"temperature":0.7,"pith_summary":"AutoGNR is trying to establish that the dominant recursive message-passing design of heterogeneous graph neural networks is itself the cause of a large part of their error: as information travels hop by hop through intermediate nodes, embeddings from uncorrelated node types get fused in and act as noise. The paper's solution is to aggregate each hop separately, restrict each hop to a learned subset of node types (possibly the empty set, so a hop is dropped), and then combine the hop embeddings once. A differentiable neural architecture search, adapted from the DARTS bi-level formulation, chooses those per-hop type subsets automatically, and the resulting discrete architecture is retrained from scratch. The claim, supported by node-classification experiments on five datasets, is that this non-recursive, task-aware pipeline consistently beats meta-path-based, meta-structure-free, and NAS-based heterogeneous GNN baselines on accuracy and is competitive or better on training time. If correct, the practical consequence is that hand-designed meta-paths and recursive mixing are not just inconvenient but actually less accurate than automatically pruning heterogeneous paths per task.","feed_headline":"AutoGNR outperforms recursive heterogeneous GNNs on five HIN datasets","feed_subtitle":"Separating aggregation by hop and node type removes noise; differentiable search finds the right paths automatically.","key_machinery":"The load-bearing mechanism is the non-recursive heterogeneous aggregation update. In the intra-hop step, $$e^k_u=\\mathrm{AGG}_{\\mathrm{intra}}(x_v : v\\in N^k(u),\\$\\varphi$(v)\\in C^k)$$ averages features of k-hop neighbors of the allowed types, with normalization by degree counts; in the inter-hop step, $$h_u=\\frac{1}{K+1}\\big(\\$\\sigma$(F_{\\mathrm{norm}}(x_u))+\\sum_{k=1}^K \\$\\sigma$(F_{\\mathrm{norm}}(e^k_u))\\big)$$ combines the anchor's own feature with each hop embedding exactly once, never recursively. The second carrying component is the differentiable search over the per-hop type sets: each candidate subset $C^k_c$ gets a softmax weight derived from architecture parameters $\\lambda^k$, and a DARTS-style bi-level objective alternately updates model weights on the training loss and architecture weights on the validation loss; the zero matrix $O$ is a candidate so the search can drop an entire hop. This combination is what lets the model prune uncorrelated heterogeneous paths automatically.","core_discovery":"On the paper's own terms, the central discovery is that unnecessary feature mixing across node types and hops is the bottleneck in heterogeneous network learning, and that it can be removed without losing task-relevant information. AutoGNR defines a non-recursive message-passing update in which the k-th hop embedding of an anchor node is produced independently by aggregating only the neighbors whose type belongs to a candidate set $C^k$, and the final embedding is a single average over the anchor's normalized feature and the $K$ hop embeddings. The search space then contains, for each hop, every non-empty subset of node types plus a zero matrix $O$, and a task-dependent constraint removes paths that involve only node types unrelated to the downstream target type. Differentiable architecture search learns a softmax mixing weight for each candidate subset, and after convergence the discrete architecture with the argmax weight per hop is retrained from scratch. The reported results claim consistent Macro-F1 and Micro-F1 improvements over twelve baselines on ACM, DBLP, IMDB, PubMed, and DBLP2, with the searched per-hop selections overlapping known meta-paths such as APA and MDM while also dropping hops or types that hurt performance.","pith_inferences":["A natural extension the authors do not pursue is to let the maximum hop count $K$ itself be part of the search, since their experiments show accuracy first improves then degrades as $K$ grows; a schedule or regularizer on $K$ might find task-specific depths automatically.","The task-dependent constraint is the method's ceiling: if future benchmarks include tasks where cross-type-only paths carry the signal, the search space would need a relaxation such as allowing paths that return to the target type through other types, or a validation-based check of dropped paths.","The noise-reduction story could be tested more directly than the paper does by measuring representation similarity or mutual information between searched and dropped type subsets, rather than only downstream accuracy."],"forward_implications":["On all five datasets in the paper, replacing recursive mixing with non-recursive per-hop aggregation improves node-classification accuracy over the strongest baselines, which would make non-recursive designs the default starting point for heterogeneous GNNs.","The task-dependent search space shrinks the NAS problem enough that the full search-plus-retrain procedure is competitive with, and on DBLP2 faster than, non-automated models, suggesting automation does not have to cost scalability.","The searched architectures overlap with human-designed meta-paths such as APA and MDM, indicating that automatic search can recover expert knowledge without manual design.","Because the framework is agnostic to the choice of $\\mathrm{AGG}_{\\mathrm{intra}}$ and $\\mathrm{AGG}_{\\mathrm{inter}}$, the same non-recursive search recipe can be carried over to other message-passing GNN backbones."],"supporting_citations":[{"why":"Supplies the differentiable bi-level optimization and one-step unrolled gradient that AutoGNR adapts for architecture search.","marker":"[44]"},{"why":"Provides the HAN meta-path baseline and the ACM/DBLP/IMDB data splits used in the normal-scale experiments.","marker":"[22]"},{"why":"Provides the MAGNN baseline that aggregates intermediate meta-path nodes and is compared against on all normal-scale datasets.","marker":"[23]"},{"why":"Provides the SeHGNN baseline, a non-recursive heterogeneous GNN whose competitive results motivate the paper's design.","marker":"[24]"},{"why":"Provides the RpHGNN baseline, a non-recursive random-projection method that is among the strongest competitors on ACM.","marker":"[32]"},{"why":"Provides the DiffMG differentiable meta-graph search baseline that AutoGNR compares against in both accuracy and convergence.","marker":"[40]"},{"why":"Supplies the PubMed and DBLP2 large-scale datasets and their statistics used for the scalability evaluation.","marker":"[47]"},{"why":"Supports the paper's premise that recursive neighbor aggregation mixes in noise and degrades GNN representations.","marker":"[41]"}],"fun_headline_variants":["Non-recursive message passing cuts noise in heterogeneous GNNs","AutoGNR: differentiable NAS for heterogeneous network learning","Separating hops and node types boosts GNN accuracy on HINs","Automated GNN search without recursive message passing","Heterogeneous GNNs: drop recursion, let search pick paths"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that for any given prediction task, all useful information flows through paths that involve the target node type, so paths connecting only other node types can be removed from the search space without losing task-relevant signal.","fun_headline_variants_meta":{"raw":{"variants":["Non-recursive message passing cuts noise in heterogeneous GNNs","AutoGNR: differentiable NAS for heterogeneous network learning","Separating hops and node types boosts GNN accuracy on HINs","Automated GNN search without recursive message passing","Heterogeneous GNNs: drop recursion, let search pick paths"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000294,"raw_usage":{"total_tokens":1754,"prompt_tokens":1030,"completion_tokens":724,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":646,"completion_tokens_details":{"reasoning_tokens":639}},"tokens_in":646,"tokens_out":724,"duration_ms":7129,"temperature":1.0,"reasoning_tokens":639,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:05:53.245380+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete test would be a heterogeneous graph where the label of a target node depends mainly on a relationship among non-target types that never touches the target type—for example, predicting a paper's field from venue-venue citation patterns that bypass the paper node itself. If a recursive or unconstrained model can exploit that path and AutoGNR's task-dependent search space cannot, the claimed generality of the method would be refuted.","supporting_citations":[{"cited_title":"Heterogeneous graph attention network,","cited_arxiv_id":null,"evidence_quote":"Provides the HAN meta-path baseline and the ACM/DBLP/IMDB data splits used in the normal-scale experiments."},{"cited_title":"Magnn: Metapath ag- gregated graph neural network for heterogeneous graph embed- ding,","cited_arxiv_id":null,"evidence_quote":"Provides the MAGNN baseline that aggregates intermediate meta-path nodes and is compared against on all normal-scale datasets."},{"cited_title":"Simple and efficient heterogeneous graph neural network,","cited_arxiv_id":null,"evidence_quote":"Provides the SeHGNN baseline, a non-recursive heterogeneous GNN whose competitive results motivate the paper's design."},{"cited_title":"Efficient heterogeneous graph learning via random projection,","cited_arxiv_id":null,"evidence_quote":"Provides the RpHGNN baseline, a non-recursive random-projection method that is among the strongest competitors on ACM."},{"cited_title":"Diffmg: Differentiable meta graph search for heterogeneous graph neural networks,","cited_arxiv_id":null,"evidence_quote":"Provides the DiffMG differentiable meta-graph search baseline that AutoGNR compares against in both accuracy and convergence."},{"cited_title":"Measuring and relieving the over-smoothing problem for graph neural networks from the topological view,","cited_arxiv_id":null,"evidence_quote":"Supports the paper's premise that recursive neighbor aggregation mixes in noise and degrades GNN representations."}],"review_version":1}