{"id":"4d6a37af-5a44-4add-a9c6-67028bd94da4","arxiv_id":"2412.11180","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"TINED distills GNNs into MLPs layer-by-layer by injecting feature-transformation parameters and matching Dirichlet energy ratios, outperforming prior distillation methods on seven node-classification benchmarks.","lead":"This paper presents TINED, a method that converts a graph neural network into a fast multi-layer perceptron by copying the GNN's layer parameters and matching its smoothing behavior. It reports accuracy gains over prior GNN-to-MLP distillation methods on seven graph datasets while running about 94 times faster than the GNN teacher.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Baseline hyperparameter tuning is unequal; TINED's reported margins may not survive a matched search.","rationale":"The reader's weakest_assumption is the mismatch between Theorem 4.1 and the actual GraphSAGE GP. While I agree the theorem is imprecise and the bound can be vacuous, it is not the load-bearing support for the central claim. TINED's superiority is an empirical assertion, and the theorem is only a post-hoc justification; Table 3 already shows low empirical approximation error even with the mismatched theory. The more decisive threat to the central claim is the asymmetric hyperparameter tuning: TINED receives an extensive search while the baselines do not. A fair comparison with matched search budgets could overturn the comparative results, directly invalidating the headline finding. The abstract's overclaim that TINED, without graph dependency, outperforms GNNs on Arxiv and Products is also a clear presentation error, but it is easily fixed by rewording and does not challenge the method's core. Because the objection is addressable through re-experimentation, the CONDITIONAL verdict remains appropriate rather than moving to ACCEPT or REJECT.","tokens_in":24266,"tokens_out":7926,"duration_ms":66929,"concrete_test":"Retrain GLNN, GLNN*, NOSMOG, NOSMOG*, KRD, and FFG2M on all seven datasets using the same per-dataset hyperparameter search spaces and the same number of random trials as TINED in Appendix A.7. Record the best validation accuracy for each method and recompute the margins in Tables 1 and 2. If TINED no longer achieves the best accuracy on a majority of datasets in the without-graph category, the central comparative claim is weakened and needs qualification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that TINED outperforms prior GNN-to-MLP distillation methods. In Tables 1 and 2, GLNN and NOSMOG use hyperparameters from their original papers; GLNN* and NOSMOG* only adjust layer count and hidden dimension, while Appendix A.7 grants TINED a much broader per-dataset search (learning rate, weight decay, lambda, beta, eta, zeta, normalization, dropout, batch size, DE-ratio smoothing function). The paper does not report an equivalent search for these baselines. Reported margins are modest (e.g., +3.21% over GLNN on Citeseer, +1.54% over NOSMOG*), so it is plausible that a matched hyperparameter search would shrink or reverse them. If so, the central claim that TINED outperforms existing methods is not established. The theoretical issue in Theorem 4.1 (SAGE's GP is a concatenation, not a single Laplacian product) is real but does not affect the empirical comparison; the empirical claim is the load-bearing one.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TINED, a GNN-to-MLP distillation method that operates layer-wise rather than treating teacher and student as monolithic models. TINED injects the feature-transformation weights of a trained GNN teacher into corresponding FC layers of an MLP student, adds an FC layer to emulate graph propagation, and adds a Dirichlet-energy-ratio loss to match layer-wise smoothing behavior. A variant TINED+ additionally uses graph structure at inference. The authors report a theoretical bound on the graph-propagation approximation error and present experiments on seven benchmark datasets under transductive and production settings, with additional results on two heterophilic datasets, ablations, parameter sensitivity studies, and inference-time measurements. The main empirical claim is that TINED outperforms prior GNN-to-MLP distillation baselines (GLNN, NOSMOG, KRD, FFG2M) and that TINED+ exceeds the teacher GNN on all seven datasets.","tokens_in":24425,"tokens_out":6407,"duration_ms":59858,"significance":"If the comparative results survive a controlled hyperparameter search, the layer-wise distillation idea is a useful and nontrivial contribution to the GNN-to-MLP literature. The paper ships source code, reports means and standard deviations over 10 trials, includes ablation and sensitivity analyses, and extends evaluation to heterophilic and large OGB datasets. The Dirichlet-energy-ratio observation (FT tends to be more conservative than GP) is interesting and gives a principled reason for layer-wise matching. The theoretical bound in Theorem 4.1 is, however, a relatively weak linear-algebra statement and is not clearly connected to the actual GP operations of the experimental teachers. The empirical comparison, not the theorem, is the load-bearing evidence, and that comparison currently suffers from an asymmetric hyperparameter search budget.","major_comments":[{"comment":"The central empirical claim that TINED outperforms existing distillation methods is not fully supported by the current comparison because the search effort is unequal. In Appendix A.7, GLNN and NOSMOG use hyperparameters from their original papers, and GLNN* and NOSMOG* only match TINED's number of layers and hidden dimension, while TINED is selected over a much larger per-dataset search space (learning rate, weight decay, lambda, beta, eta, zeta, normalization, dropout, batch size, and DE-ratio smoothing function). The reported margins are small on several datasets, for example 0.14% over GLNN* on Pubmed in Table 1, and Table 5 shows that varying only eta can move Citeseer accuracy by more than 7 percentage points. A matched hyperparameter search for the baselines, with model selection on validation data for all methods, is needed before the superiority claim can be accepted.","section":"§5.2, Appendix A.7"},{"comment":"The abstract's statement that TINED 'outperforms GNNs' is too broad. In the graph-independent setting of Table 1, TINED is below the GraphSAGE teacher on Arxiv (64.44 vs 70.73) and Products (69.48 vs 77.17); only TINED+ exceeds the teacher on all seven datasets. The claim should be qualified to state which variant is meant, for example 'TINED+ exceeds the teacher when graph structure is available at inference, while TINED matches or improves on the teacher on the smaller datasets without graph access.'","section":"Abstract, §5.2"},{"comment":"The approximation bound in Theorem 4.1 is not connected to the GP operations actually used in the main experiments. For GraphSAGE, Eq. (4) defines GP(l) as a concatenation of the node embedding with an aggregation of neighbor embeddings, so the output is not a square matrix product LH with L in R^{n x n}, and lambda_max(L) in the theorem is not defined for that operation. For GAT, Eq. (16) defines GP through a data-dependent attention matrix, and the student FC layer in Eq. (5) includes a bias and nonlinearity that the theorem does not model. In addition, for the unnormalized Laplacian L = D - A used in Definition 4.2, lambda_max(L) can be large, making the bound vacuous. The theoretical claim should be restricted to GNNs whose propagation is exactly a fixed square Laplacian product, such as GCN in Eq. (15), or explicitly presented as a heuristic motivation.","section":"Theorem 4.1, §4.1, Appendix A.2"}],"minor_comments":[{"comment":"The 'prod' rows in Table 2 are reported without standard deviations, unlike the 'ind' and 'tran' rows; please report the variation of the weighted average across seeds.","section":"Table 2"},{"comment":"The notation hat-nabla in Eq. (6) is not defined; please clarify that it denotes the modified gradient used to update the injected parameters.","section":"Eq. (6)"},{"comment":"Figure 4 aggregates results over multiple teacher GNNs and datasets, but the caption does not state what is being averaged and no error bars are shown; please clarify the aggregation protocol.","section":"Figure 4"},{"comment":"The reference to Hinton et al. (2015) is given as arXiv:2211.02989, which is the identifier of a different paper; please correct the arXiv number.","section":"References"},{"comment":"There is a typo in the conclusion: 'grpahs' should be 'graphs'.","section":"Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The main risk to acceptance is the asymmetric hyperparameter search budget: the paper's central claim rests on comparisons where TINED receives a much broader per-dataset search than GLNN*/NOSMOG*. I would recommend requesting a matched search for the baselines, or at minimum a validation-based selection protocol applied uniformly, before considering acceptance. The abstract's overclaim about outperforming GNNs should also be reconciled with Table 1."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two borrowable ideas: inject teacher FT weights into student FC layers, and match per-layer Dirichlet energy ratios. The FT/FC equivalence is real; the DE-ratio observation (GP smooths hard, FT smooths little or diversifies) is nicely measured and consistent across architectures. Experiments are extensive—seven datasets, two heterophilic, four teacher families, ablations, code released.\n\nBut the headline claim is not supported by the tables. In the without-graph setting, TINED beats GLNN/GLNN* on most datasets, but it is worse than NOSMOG on four of seven: Cora 82.63 vs 83.04, Pubmed 77.09 vs 77.34, Arxiv 64.44 vs 71.65, Products 69.48 vs 78.45. Those last two are not close; 'TINED surpasses all baselines without graph dependency' (Section 5.2) is simply false. The with-graph TINED+ is competitive with NOSMOG, but the abstract's unqualified 'outperforms GNNs and leading distillation methods' needs qualifiers.\n\nSecond soft spot: unequal hyperparameter tuning. TINED gets a wide per-dataset search (lr, weight decay, lambda, beta, eta, zeta, dropout, normalization, batch size, DE smoothing), while GLNN* and NOSMOG* only changed layers/hidden dims. Given the modest margins, a matched search could shift results. The ablation shows both components help, so I doubt the method is hollow, but the comparison needs to be fairer.\n\nTheorem 4.1 is a generic projection bound with spectral-norm error; it doesn't model GraphSAGE's concatenation-based GP, so it's weak as a justification. Also, the references contain at least one duplicated/bogus arXiv ID (Hinton et al. 2015 and Jadon et al. 2022 share '2211.02989').\n\nBottom line: the techniques are worth a serious look and the paper deserves peer review, but it needs a major revision: fix the overclaims, rerun or report matched hyperparameter searches for baselines, and either fix the theory or stop presenting it as a GP approximation guarantee. I'd bring it to our reading group.","headline":"Clever FT-injection and DE-ratio distillation; the paper overclaims its advantage over NOSMOG, which beats TINED on the large OGB datasets.","tokens_in":25019,"tokens_out":8813,"would_cite":true,"duration_ms":67988,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","68R10"],"pacs":[],"model":"deepseek-v4-flash","headline":"A layer-by-layer distillation method copies a trained GNN's feature transformations into an MLP, approximates graph propagation, and matches smoothing ratios; the students beat prior distillations and, with graph access, the teacher.","keywords":["GNN-to-MLP distillation","knowledge distillation","teacher injection","Dirichlet energy","graph neural networks","node classification","inference acceleration","over-smoothing"],"falsifier":"For a frozen GraphSAGE teacher on a chosen dataset, compute the per-layer relative error between the actual GP output and the best linear FC approximation $\\|GP^{(\\ell)}(H) - HW^*\\|_F/\\|H\\|_F$. If this error exceeds $\\lambda_{\\max}(L)$, or if ablating the DE-ratio loss and randomizing injected weights still yields the reported accuracy, the method's stated justification for replacing propagation with an FC layer would be contradicted on those layers.","tokens_in":24015,"feed_emoji":"⚡","tokens_out":7036,"duration_ms":55161,"temperature":0.7,"pith_summary":"TINED is a layer-by-layer distillation method that turns a trained graph neural network into a plain MLP, so node classification can run without repeatedly gathering multi-hop neighbourhoods at inference. The paper's central claim is that a GNN layer's feature transformation is formally identical to a fully-connected layer, so the teacher's trained weights can be copied directly into the student, and that the remaining graph-propagation step can be approximated by an extra FC layer whose smoothing behaviour is matched to the teacher through a Dirichlet-energy ratio. Across seven datasets the resulting students outperform prior GNN-to-MLP distillation methods, and the graph-aware variant TINED+ is reported to exceed the teacher GNN on all seven datasets while running 94 times faster.","feed_headline":"Layer-copied MLP beats its GNN teacher on seven graph datasets","feed_subtitle":"By injecting teacher weights and matching smoothing ratios, the student MLP beats the teacher and runs 94x faster.","key_machinery":"Two mechanisms carry the argument. Teacher injection copies the trained transformation $W^{(\\ell)}, b^{(\\ell)}$ of each FT operation in the teacher GNN into a corresponding FC layer of the student MLP, because both compute $\\sigma(hW + b)$; a gradient modifier $\\eta$ limits how far fine-tuning moves the injected weights. Dirichlet energy distillation defines the DE ratio $R_{\\text{op}} = E(\\text{op}(H))/E(H)$ with $E(H) = \\frac{1}{n}\\operatorname{tr}(H^\\top L H)$, and adds a loss that matches student FC-layer ratios to the teacher's GP and FT ratios, preserving the observation that GP smooths aggressively while FT smooths conservatively. Theorem 4.1 supplies the theoretical support for replacing propagation with a linear layer: for a sparse Laplacian $L$ and full-column-rank $H$, some $W^*$ gives $\\|LH - HW^*\\|_F / \\|H\\|_F \\le \\lambda_{\\max}(L)$.","core_discovery":"The central discovery is that fine-grained, layer-wise knowledge—trained feature-transformation parameters and per-layer smoothing ratios—is more transferable from a GNN to an MLP than output soft labels alone. With a two-layer GraphSAGE teacher, TINED without any graph access reaches 74.43% on Citeseer, improving on GLNN by 3.21 percentage points and on NOSMOG by 1.54, and on the OGB Products dataset it gains 9.42 points over a plain MLP. When graph structure is allowed at inference, TINED+ exceeds the GraphSAGE teacher on all seven datasets and achieves 75.39% on Citeseer. The same recipe works when the teacher is GCN, GAT, or APPNP, and on heterophilic graphs.","pith_inferences":["An implication the paper leaves implicit is that matching per-layer Dirichlet energy ratios could serve as a general regularizer against over-smoothing in deeper MLP students, since it forces each FC layer to keep the same smoothing profile as the teacher's corresponding operations.","The injection principle extends to other architecture pairs: whenever a teacher module is computationally identical to a student module, copying parameters and then fine-tuning with a small learning rate may transfer more knowledge than matching outputs alone.","A testable extension is to replace the single FC approximation of graph propagation with a small set of feature-dependent linear maps when the teacher uses attention; the paper's DE-ratio matching would then have an even closer target.","On graphs where the unnormalized Laplacian has a large largest eigenvalue, the theoretical bound becomes loose; measuring the actual approximation error on such graphs would show whether the practical method still holds where its proof is weakest."],"forward_implications":["If TINED's results hold, a graph-free student MLP can replace a message-passing GNN for node classification without sacrificing accuracy, and TINED+ can even exceed the teacher's accuracy.","Inference time drops from 153 ms for the 2-layer GraphSAGE teacher to about 1.6 ms for the student, a 94x speedup and 733x against the 3-layer teacher.","The ablation results attribute the gains to both components: removing teacher injection or removing DE distillation lowers accuracy on every dataset.","The method transfers across teacher architectures (GraphSAGE, GCN, GAT, APPNP), so layer-wise injection is not tied to one GNN design."],"supporting_citations":[{"why":"GLNN soft-label distillation baseline whose objective TINED extends and whose accuracy TINED compares against.","marker":"(Zhang et al., 2022b)"},{"why":"NOSMOG graph-aware distillation baseline; supplies the with-graph comparison and the production-setting protocol.","marker":"(Tian et al., 2022)"},{"why":"GraphSAGE teacher used for the main experiments and the source of the FT-GP layer decomposition in Eq. (4).","marker":"(Hamilton et al., 2017)"},{"why":"GCN teacher and the canonical FT/GP message-passing formulation that motivates the layer-wise decomposition.","marker":"(Kipf and Welling, 2017)"},{"why":"GAT teacher whose attention-based graph propagation tests whether the method extends beyond linear aggregation.","marker":"(Veličković et al., 2018)"},{"why":"Supplies the Dirichlet energy definition used to build DE ratios and the DED loss.","marker":"(Rusch et al., 2023)"},{"why":"FFG2M baseline that TINED outperforms in the no-graph category.","marker":"(Wu et al., 2023a)"},{"why":"KRD baseline that quantifies GNN knowledge and is compared in the main tables.","marker":"(Wu et al., 2023b)"},{"why":"APPNP teacher used in the different-teacher experiments, representing decoupled FT and GP.","marker":"(Klicpera et al., 2019)"},{"why":"VQGraph comparison in Appendix A.9, showing TINED+ accuracy against a structure-aware tokenizer method.","marker":"(Yang et al., 2024)"}],"fun_headline_variants":["MLP distilled from GNN layers beats teacher on 7 datasets","Teacher injection and energy match make MLP beat GNN","Layer-wise knowledge transfer: MLP outruns and outperforms GNN","TINED: Inject GNN teacher into MLP, get 94x speedup and wins","Fast MLP learned from GNN layers tops teacher on all graphs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The practical success rests on an unproven premise: that a single fully-connected layer can closely approximate the specific graph-propagation operation of the chosen GNN, since the proof's linear-map assumption does not cover the concatenation used by GraphSAGE or the learned attention used by GAT.","fun_headline_variants_meta":{"raw":{"variants":["MLP distilled from GNN layers beats teacher on 7 datasets","Teacher injection and energy match make MLP beat GNN","Layer-wise knowledge transfer: MLP outruns and outperforms GNN","TINED: Inject GNN teacher into MLP, get 94x speedup and wins","Fast MLP learned from GNN layers tops teacher on all graphs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000661,"raw_usage":{"total_tokens":3060,"prompt_tokens":1021,"completion_tokens":2039,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":637,"completion_tokens_details":{"reasoning_tokens":1941}},"tokens_in":637,"tokens_out":2039,"duration_ms":12100,"temperature":1.0,"reasoning_tokens":1941,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:13:15.729585+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"For a frozen GraphSAGE teacher on a chosen dataset, compute the per-layer relative error between the actual GP output and the best linear FC approximation $\\|GP^{(\\ell)}(H) - HW^*\\|_F/\\|H\\|_F$. If this error exceeds $\\lambda_{\\max}(L)$, or if ablating the DE-ratio loss and randomizing injected weights still yields the reported accuracy, the method's stated justification for replacing propagation with an FC layer would be contradicted on those layers.","supporting_citations":[{"cited_title":"Predict then propagate: Graph neural networks meet personalized pagerank","cited_arxiv_id":null,"evidence_quote":"APPNP teacher used in the different-teacher experiments, representing decoupled FT and GP."},{"cited_title":"Vqgraph: Rethinking graph representation space for bridging gnns and mlps","cited_arxiv_id":null,"evidence_quote":"VQGraph comparison in Appendix A.9, showing TINED+ accuracy against a structure-aware tokenizer method."}],"review_version":1}