{"id":"9035b40e-9047-4789-b92b-0656a790b093","arxiv_id":"2412.05649","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Replacing GRU cells in RouteNet-Fermi with LSTMs gives slightly lower delay MAPE on several test networks, but this is a routine extension with weak statistical support.","lead":"A Purdue class project reimplemented RouteNet-Fermi, a graph neural network for predicting network delay, jitter, and packet loss, and replaced its GRU cells with LSTM and simple RNN cells. Across several datasets, the LSTM variant often produced the lowest delay error, though the comparisons lack error bars and the code is not provided in the paper.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'LSTM is best in most delay tasks' claim is not supported by the paper's own tables: GRU Model beats LSTM in FatTree128 (0.45% vs 0.58%) and in Traffic Model On/Off and Modulated; no error bars or repeated runs.","rationale":"The paper is a transparent reimplementation study, and the modular three-cell implementation is a reasonable contribution at the level of a course project. The authors are honest about CPU training and task-specific epochs. My concern, however, targets the headline performance claim rather than the implementation. The reader identified the cross-codebase comparability of 'Paper GRU' as the weakest assumption; I agree that is a problem, but an even more basic issue is that the paper's own internal results do not consistently favor LSTM. In the 11 delay-prediction rows across Tables 1, 3, 4, and 5, GRU Model beats or ties LSTM in FatTree128, FatTree64, On/Off, and Modulated. Without repeated seeds or error bars, the difference between 2.96% and 3.82% could be noise; the difference between 2.82% and 2.84% certainly is. The claim 'most delay prediction tasks' is therefore an overstatement. The correct verdict remains conditional: the paper should be revised to report multi-seed statistics and to narrow the claim to 'LSTM is competitive and often best', or the experiments need to demonstrate a robust majority. This does not change the reader's conditional verdict, so I recommend UNCHANGED, but for a sharper reason than the original weakest-assumption note.","tokens_in":9539,"tokens_out":5929,"duration_ms":54355,"concrete_test":"Run each of RNN, GRU, and LSTM variants on the FatTree, Real Traffic, Scalability, Scheduling, and Traffic Model delay tasks with 5 random seeds, identical hyperparameters as in Section 6.1, and report mean ± standard deviation of MAPE. Count the number of tasks where LSTM's mean is lower than GRU Model's by more than the pooled standard error. If that count is not a clear majority (e.g., at least 7 of 11) or if any reported 'win' in Tables 1 and 5 flips sign, revise the central claim to describe LSTM as competitive rather than best.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the LSTM variant achieves the lowest MAPE in most delay prediction tasks. This fails on the paper's own reported numbers even before considering the 'Paper GRU' baseline. In Table 1, FatTree128: GRU Model 0.45% vs LSTM 0.58%; in Table 5, On/Off: GRU Model 2.82% vs LSTM 2.84%, Modulated: GRU Model 5.29% vs LSTM 5.42%. FatTree64 is a tie at 0.50%, with Paper GRU better at 0.44%. Across Tables 1, 3, 4, and 5, LSTM is best among the three reimplemented cells in roughly 6 of 11 delay rows, barely a majority, and all differences are single-run with no confidence intervals. The more defensible claim is that LSTM is competitive and sometimes better, not systematically superior. Since the headline comparison to 'Paper GRU' (e.g., 2.96% vs 3.35% in Table 4) also mixes codebases and training setups, the 'outperforms original' framing rests on two unsupported layers: cross-codebase comparability and unquantified within-codebase variation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a re-implementation of RouteNet-Fermi, a graph neural network for network performance modeling, with the addition of LSTM and simple RNN cells alongside the original GRU cells. The authors evaluate the three cell variants on delay, jitter, and loss prediction across Fat Tree, scheduling, scalability, real-traffic, and traffic-model datasets, and compare the results against the numbers reported in the original RouteNet-Fermi paper. The central claim is that the LSTM variant achieves the lowest MAPE in most delay prediction tasks, while also providing an analysis of validation loss curves and inference-time scaling. The paper makes its code and datasets publicly available.","tokens_in":9835,"tokens_out":6682,"duration_ms":53259,"significance":"If the empirical claims were statistically well-supported, the paper would be a useful reproducibility and engineering contribution: a modular implementation of recurrent cell variants in RouteNet-Fermi, evaluated on public datasets with identical hyperparameters. The comparison of RNN, GRU, and LSTM cells under controlled conditions could inform architecture selection for GNN-based network modeling. However, the current support is weak because the results are single-run point estimates without confidence intervals, the 'Paper GRU' baseline is taken from the original publication rather than reproduced in the same harness, and one experiment adds an uncontrolled fine-tuning step. The contribution is therefore primarily an open-source code artifact rather than a validated scientific claim.","major_comments":[{"comment":"The central claim that the LSTM variant achieves the lowest MAPE in most delay prediction tasks is not supported by the paper's own tables. For example, Table 1 FatTree128 shows GRU Model at 0.45% versus LSTM at 0.58%; Table 5 On/Off shows GRU Model at 2.82% versus LSTM at 2.84%; Table 5 Modulated shows GRU Model at 5.29% versus LSTM at 5.42%; and Table 1 FatTree64 is a tie at 0.50% with Paper GRU better at 0.44%. Moreover, §7.1 states that 'LSTM cells consistently achiev[ed] the lowest MAPE' for the Fat Tree topology, which is contradicted by Table 1. Because no error bars, confidence intervals, or significance tests are reported, the reported differences cannot be distinguished from noise. Please provide repeated runs with variance estimates and revise the claims to match the evidence.","section":"§7, Tables 1 and 5"},{"comment":"The 'Paper GRU' numbers are cited verbatim from the original RouteNet-Fermi paper rather than produced with the same codebase and training pipeline used for the reimplemented variants. Since the authors' own GRU reimplementation often differs substantially from the Paper GRU baseline (e.g., Table 2: 2.18% versus 5.67% for Real Traffic), this is not a controlled comparison. All conclusions that the authors' models outperform the original rely on this cross-codebase comparability, which is not established. Please retrain or rerun the original GRU implementation in the same harness, or provide a detailed component-by-component comparison of the training configurations to justify the comparison.","section":"§7, Tables 1–5"},{"comment":"The Real Traffic experiment is described as using 'a previous checkpoint, fine-tuning it using 200 samples of the GEANT topology.' This introduces an uncontrolled variable: the source of the checkpoint, the fine-tuning procedure, and the number of fine-tuning steps are not specified. As a result, the reported 1.82% MAPE for the LSTM in Table 2 cannot be attributed to the LSTM architecture alone. Please specify and justify the fine-tuning protocol, or rerun the experiment from scratch.","section":"§4.3, Table 2"},{"comment":"The paper claims that LSTM shows 'particular strength in capturing temporal dependencies critical for jitter prediction,' yet Table 5 reports LSTM jitter MAE values that are an order of magnitude worse than the Paper GRU baseline, e.g., Constant Bitrate: 0.164 versus 0.008; On/Off: 0.272 versus 0.018; and All Multiplexed: 0.503 versus 0.034. Section 8 acknowledges that 'for traffic model jitter prediction task, the training struggled compared to the paper,' but the manuscript does not analyze or explain this failure. This internal inconsistency between the claimed strength and the reported results should be resolved.","section":"§5 and §8, Table 5"},{"comment":"The training configuration states that the number of epochs is 'task specific numbers' but does not report those numbers, nor does it specify random seeds, weight initialization, or early-stopping criteria. Without these details the experiments are not reproducible, and the absence of any seed variation makes it impossible to assess the stability of the single-point metrics in the tables.","section":"§6.1"}],"minor_comments":[{"comment":"The abstract contains a typo: 'We improve the the original architecture' should read 'We improve the original architecture.'","section":"Abstract"},{"comment":"The paragraph contains multiple grammatical and formatting errors, including 'so, their measurable needs grows proportional with the size of the traffic intensity and network' and concatenated words such as 'forlarge-scalenetworks' and 'Overcomethislimitation'; the prose needs careful proofreading.","section":"§3.1"},{"comment":"The MAWI repository is cited as reference [27], but reference [27] is the SNDlib paper, and SNDlib is cited as reference [28], which is the MAWILab paper; these two citations appear to be swapped.","section":"§4.3"},{"comment":"In the LSTM forget-gate equation, f_t = σ(W_f · (h_{t−1}, x_t) + b_f), parentheses are used instead of square brackets for the concatenation, which is inconsistent with the GRU and RNN equations.","section":"§6, LSTM equations"},{"comment":"The validation-loss subfigures do not label which curve corresponds to each model variant; adding legends for RNN, LSTM, and GRU would improve interpretability.","section":"Figures 4–9"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is written as a course project report (Purdue CS53600) and its framing is more like a technical blog post than a journal article, with a limited novelty contribution beyond the open-source implementation. The load-bearing issues are the absence of statistical rigor, the uncontrolled 'Paper GRU' baseline, and the unspecified fine-tuning in §4.3; the stress-test concern about the LSTM claim being unsupported by the paper's own tables is fully confirmed. However, these issues are fixable within the scope of an empirical study: multiple-seed runs, confidence intervals, a controlled baseline reproduction, and a revision of the claims to match the evidence could turn this into a useful reproducibility study. There is no indication of misconduct; the reference mix-up in §4.3 appears unintentional."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a straight reimplementation plus cell swap of RouteNet-Fermi. The genuinely useful part is that it ships code and compares three recurrent cells (RNN, GRU, LSTM) under one harness. That part is fine. The paper's headline claim—LSTM wins most delay tasks—does not survive its own tables: GRU beats LSTM on FatTree128 (0.45% vs 0.58%), On/Off (2.82% vs 2.84%), and Modulated (5.29% vs 5.42%). Across the 11 delay rows in Tables 1, 3, 4, and 5, LSTM is best among the three reimplemented cells only about half the time, and every number is a single run with no error bars. The defensible claim is \"LSTM is competitive, sometimes better,\" not \"LSTM outperforms.\"\n\nThe comparison to \"Paper GRU\" is shakier. Those numbers come from the original RouteNet-Fermi paper, not from the authors' own harness. Different codebase, likely different training setup, possibly different data splits. Section 4.3 adds another confound: the Real Traffic experiment fine-tunes a previous checkpoint on 200 GEANT samples. That uncontrolled variable makes the 1.82% vs 5.67% comparison basically unreadable.\n\nWhat is actually new: nothing conceptual. The RNN/GRU/LSTM equations in Section 6 are textbook; the architecture is the original RouteNet-Fermi with the recurrent cell swapped. That can still be a useful engineering artifact. The GitHub link, the public dataset, and the explicit training parameters make it reproducible in principle, and the authors are honest about CPU training and about the jitter difficulties. Those are real merits.\n\nThe citation pattern is fine and not inflated. The prose has typos (\"the the\") but that is cosmetic.\n\nWho this is for: someone who wants a quick signal on whether LSTM cells are worth trying inside RouteNet-Fermi, or who is reimplementing the model themselves. Not for a reader looking for a new method or a rigorous benchmark.\n\nRecommendation: don't treat this as a peer-reviewed contribution yet. If the authors rerun everything with multiple seeds, report distributions, re-implement the original GRU in the same harness, and fix the checkpoint-fine-tuning confound, the comparison would be worth a workshop paper. As it stands, it's a useful tech report, but not a paper I'd send to a serious referee.","headline":"A useful, honest reimplementation of RouteNet-Fermi with LSTM/RNN swaps, but the headline that LSTM wins most delay tasks is not supported by the paper's own tables and the comparison to the original GRU numbers is not apples-to-apples.","tokens_in":10331,"tokens_out":1833,"would_cite":false,"duration_ms":17352,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LSTM cells cut delay error in most RouteNet-Fermi tests","keywords":["network performance modeling","graph neural networks","RouteNet-Fermi","LSTM","GRU","RNN","delay prediction","packet loss prediction"],"falsifier":"Train the original RouteNet-Fermi GRU implementation on the same datasets with the same hidden size (32), batch size (2,000), optimizer (Adam, learning rate 0.001), and epoch counts, then compare its MAPE with the LSTM's reported 2.96% on scheduling delay; if the re-run GRU matches or beats LSTM, the central claim that LSTM cells improve delay prediction loses its support.","tokens_in":9399,"feed_emoji":"📈","tokens_out":6390,"duration_ms":51055,"temperature":0.7,"pith_summary":"This paper re-implements RouteNet-Fermi, a graph neural network that predicts end-to-end delay, jitter, and packet loss in computer networks, and swaps the original GRU recurrent cells for LSTM and simple RNN cells. The authors report that LSTM cells give the lowest mean absolute percentage error on most delay-prediction tasks, for instance 2.96% versus 3.82% for their GRU re-run and 3.35% for the original paper's GRU on the scheduling dataset. They argue that the choice of recurrent cell matters for how well the message-passing model captures temporal dependencies in queue occupancy and traffic. The practical stake is a more accurate but still fast network performance model that could replace simulation for planning and traffic engineering.","feed_headline":"LSTM cells cut delay error in most RouteNet-Fermi tests","feed_subtitle":"Re-running the GNN network model with LSTM instead of GRU lowers delay MAPE to 2.96% on scheduling tasks.","key_machinery":"The load-bearing mechanism is the recurrent cell inside the flow-level and link-level RNNs of the three-stage message-passing architecture. RouteNet-Fermi iteratively updates hidden states for flows, queues, and links; the cell type determines how sequential queue-occupancy information is compressed into these states. The LSTM cell adds an explicit memory cell with forget, input, and output gates, which the paper argues retains longer-range temporal dependencies in traffic and congestion dynamics, while the simple RNN uses only a tanh layer and suffers gradient-flow problems that show up in a 16.53% MAPE on FatTree128. All three variants were trained with identical hyper-parameters, so the reported differences are attributed to cell architecture.","core_discovery":"On the paper's own terms, the central discovery is that within the RouteNet-Fermi three-stage message-passing architecture, replacing the Gated Recurrent Unit cells with Long Short-Term Memory cells improves delay prediction accuracy across most evaluated network scenarios, while simple RNN cells lag, especially on larger topologies. The evidence is a set of tables comparing four configurations (RNN, LSTM, GRU re-run, and the original paper's reported GRU numbers) on Fat Tree, real-traffic, scalability, scheduling, and traffic-model datasets; LSTM posts the lowest MAPE in most delay rows, such as 1.82% on real traffic versus 5.67% for the paper GRU, and 0.70% on scalability delay versus 1.08%. The paper also reports that LSTM matches or beats GRU on jitter in the scheduling scenario (16.73% versus 17.01%) but that the original paper's GRU retains an edge on packet-loss MAE (0.001978) and on several traffic-model jitter rows.","pith_inferences":["Beyond the paper, a matched re-run of the original RouteNet-Fermi code on the same datasets with the same hyper-parameters would be needed to confirm the LSTM advantage is due to the cell type rather than to differences in implementation or training setup.","Beyond the paper, the LSTM's extra parameters and gates likely cost more compute per step; an equal-parameter or equal-inference-time comparison could change the ranking on larger topologies.","Beyond the paper, one testable extension is to vary flow path lengths in the scalability dataset: if LSTM's memory matters, its advantage should grow with longer paths."],"forward_implications":["If the LSTM advantage holds, network operators can obtain more accurate delay estimates from the same GNN framework without adding simulation cost.","The modular cell swap means future work can test other recurrent or attention-based cells under identical conditions to find the best fit for jitter and loss prediction.","The basic RNN's degradation on FatTree128 (16.53% MAPE) indicates that simple recurrent cells are not reliable for larger topologies.","Packet loss prediction remains best with the original GRU implementation (MAE 0.001978), so no single cell dominates every metric and cell choice should be task-specific."],"supporting_citations":[{"why":"Defines the original RouteNet-Fermi architecture and supplies the 'Paper GRU' baseline numbers compared throughout.","marker":"[3]"},{"why":"Supplies the recurrent neural network and LSTM formulations used to build the alternative cells.","marker":"[10]"},{"why":"Supplies the GRU formulation that the original architecture and the re-run GRU baseline use.","marker":"[11]"},{"why":"Supplies the OMNeT++ simulator that generated the datasets on which all models are trained and tested.","marker":"[20]"},{"why":"Defines the Fat Tree topology parameters and traffic profile used in the FatTree evaluation.","marker":"[22]"},{"why":"Supplies the MAWI traffic traces used to build the real-traffic dataset.","marker":"[27]"},{"why":"Supplies the SNDlib traffic matrices used for the real-traffic experiment.","marker":"[28]"}],"fun_headline_variants":["LSTM cells beat GRU on delay across most RouteNet-Fermi tests","RouteNet-Fermi: LSTM improves delay prediction over GRU","Swap GRU for LSTM to cut delay error in network GNN","LSTM hits 1.82% delay MAPE on real traffic, beating GRU's 5.67%","Recurrent cell choice matters: LSTM wins delay in RouteNet-Fermi"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results assume the 'Paper GRU' numbers quoted from the original paper come from a training setup similar enough to the authors' re-runs that the differences reflect cell architecture and not codebase, hyper-parameter, or compute variations.","fun_headline_variants_meta":{"raw":{"variants":["LSTM cells beat GRU on delay across most RouteNet-Fermi tests","RouteNet-Fermi: LSTM improves delay prediction over GRU","Swap GRU for LSTM to cut delay error in network GNN","LSTM hits 1.82% delay MAPE on real traffic, beating GRU's 5.67%","Recurrent cell choice matters: LSTM wins delay in RouteNet-Fermi"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000289,"raw_usage":{"total_tokens":1673,"prompt_tokens":904,"completion_tokens":769,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":520,"completion_tokens_details":{"reasoning_tokens":659}},"tokens_in":520,"tokens_out":769,"duration_ms":6864,"temperature":1.0,"reasoning_tokens":659,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:29:08.339268+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the original RouteNet-Fermi GRU implementation on the same datasets with the same hidden size (32), batch size (2,000), optimizer (Adam, learning rate 0.001), and epoch counts, then compare its MAPE with the LSTM's reported 2.96% on scheduling delay; if the re-run GRU matches or beats LSTM, the central claim that LSTM cells improve delay prediction loses its support.","supporting_citations":[{"cited_title":"& Cabellos-Aparicio, A","cited_arxiv_id":null,"evidence_quote":"Defines the original RouteNet-Fermi architecture and supplies the 'Paper GRU' baseline numbers compared throughout."},{"cited_title":"Extensions of Recurrent Neural Network Language Model,","cited_arxiv_id":null,"evidence_quote":"Supplies the recurrent neural network and LSTM formulations used to build the alternative cells."},{"cited_title":"Modeling and Prediction of the Internet End-to-End Delay Using Recurrent Neural Networks,","cited_arxiv_id":null,"evidence_quote":"Supplies the GRU formulation that the original architecture and the re-run GRU baseline use."},{"cited_title":"Discrete event simulation system,","cited_arxiv_id":null,"evidence_quote":"Supplies the OMNeT++ simulator that generated the datasets on which all models are trained and tested."},{"cited_title":"Sndlib 1.0—survivable network design library,","cited_arxiv_id":null,"evidence_quote":"Supplies the MAWI traffic traces used to build the real-traffic dataset."},{"cited_title":"MAWILab: Combin- ing Diverse Anomaly Detectors for Automated Anomaly Labeling and Performance Benchmarking,","cited_arxiv_id":null,"evidence_quote":"Supplies the SNDlib traffic matrices used for the real-traffic experiment."}],"review_version":1}