Pith. sign in

REVIEW 5 major objections 5 minor 30 references

RouteNet-Fermi: Network Modeling With GNN (Analysis And Re-implementation)

T0 review · 5 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LSTM cells cut delay error in most RouteNet-Fermi tests

desk verdict A useful, honest reimplementation of RouteNet-Fermi with LSTM/RNN swaps, but the headline that LSTM wins most delay tasks is not supported by the paper's own tables and the comparison to the original GRU numbers is not apples-to-apples. read the letter →

arxiv 2412.05649 v1 pith:OIEYY4SX submitted 2024-12-07 cs.NI

Shourya Verma , Simran Kadadi , Swathi Jayaprakash , Arpan Kumar Mahapatra , Ishaan Jain This is my paper · ORCID
classification cs.NI
keywords networkperformancemodelinggraphneuralnetworksRouteNet-FermiLSTMGRURNNdelaypredictionpacketloss
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper re-implements RouteNet-Fermi, a graph neural network that predicts end-to-end delay, jitter, and packet loss in computer networks, and swaps the original GRU recurrent cells for LSTM and simple RNN cells. The authors report that LSTM cells give the lowest mean absolute percentage error on most delay-prediction tasks, for instance 2.96% versus 3.82% for their GRU re-run and 3.35% for the original paper's GRU on the scheduling dataset. They argue that the choice of recurrent cell matters for how well the message-passing model captures temporal dependencies in queue occupancy and traffic. The practical stake is a more accurate but still fast network performance model that could replace simulation for planning and traffic engineering.

What carries the argument

The load-bearing mechanism is the recurrent cell inside the flow-level and link-level RNNs of the three-stage message-passing architecture. RouteNet-Fermi iteratively updates hidden states for flows, queues, and links; the cell type determines how sequential queue-occupancy information is compressed into these states. The LSTM cell adds an explicit memory cell with forget, input, and output gates, which the paper argues retains longer-range temporal dependencies in traffic and congestion dynamics, while the simple RNN uses only a tanh layer and suffers gradient-flow problems that show up in a 16.53% MAPE on FatTree128. All three variants were trained with identical hyper-parameters, so the reported differences are attributed to cell architecture.

What would settle it

Train the original RouteNet-Fermi GRU implementation on the same datasets with the same hidden size (32), batch size (2,000), optimizer (Adam, learning rate 0.001), and epoch counts, then compare its MAPE with the LSTM's reported 2.96% on scheduling delay; if the re-run GRU matches or beats LSTM, the central claim that LSTM cells improve delay prediction loses its support.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that within the RouteNet-Fermi three-stage message-passing architecture, replacing the Gated Recurrent Unit cells with Long Short-Term Memory cells improves delay prediction accuracy across most evaluated network scenarios, while simple RNN cells lag, especially on larger topologies. The evidence is a set of tables comparing four configurations (RNN, LSTM, GRU re-run, and the original paper's reported GRU numbers) on Fat Tree, real-traffic, scalability, scheduling, and traffic-model datasets; LSTM posts the lowest MAPE in most delay rows, such as 1.82% on real traffic versus 5.67% for the paper GRU, and 0.70% on scalability delay versus 1.08%. The paper also reports that LSTM matches or beats GRU on jitter in the scheduling scenario (16.73% versus 17.01%) but that the original paper's GRU retains an edge on packet-loss MAE (0.001978) and on several traffic-model jitter rows.

Load-bearing premise

The results assume the 'Paper GRU' numbers quoted from the original paper come from a training setup similar enough to the authors' re-runs that the differences reflect cell architecture and not codebase, hyper-parameter, or compute variations.

Editorial extensions

If this is right

  • If the LSTM advantage holds, network operators can obtain more accurate delay estimates from the same GNN framework without adding simulation cost.
  • The modular cell swap means future work can test other recurrent or attention-based cells under identical conditions to find the best fit for jitter and loss prediction.
  • The basic RNN's degradation on FatTree128 (16.53% MAPE) indicates that simple recurrent cells are not reliable for larger topologies.
  • Packet loss prediction remains best with the original GRU implementation (MAE 0.001978), so no single cell dominates every metric and cell choice should be task-specific.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, a matched re-run of the original RouteNet-Fermi code on the same datasets with the same hyper-parameters would be needed to confirm the LSTM advantage is due to the cell type rather than to differences in implementation or training setup.
  • Beyond the paper, the LSTM's extra parameters and gates likely cost more compute per step; an equal-parameter or equal-inference-time comparison could change the ranking on larger topologies.
  • Beyond the paper, one testable extension is to vary flow path lengths in the scalability dataset: if LSTM's memory matters, its advantage should grow with longer paths.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper presents a re-implementation of RouteNet-Fermi, a graph neural network for network performance modeling, with the addition of LSTM and simple RNN cells alongside the original GRU cells. The authors evaluate the three cell variants on delay, jitter, and loss prediction across Fat Tree, scheduling, scalability, real-traffic, and traffic-model datasets, and compare the results against the numbers reported in the original RouteNet-Fermi paper. The central claim is that the LSTM variant achieves the lowest MAPE in most delay prediction tasks, while also providing an analysis of validation loss curves and inference-time scaling. The paper makes its code and datasets publicly available.

Significance. If the empirical claims were statistically well-supported, the paper would be a useful reproducibility and engineering contribution: a modular implementation of recurrent cell variants in RouteNet-Fermi, evaluated on public datasets with identical hyperparameters. The comparison of RNN, GRU, and LSTM cells under controlled conditions could inform architecture selection for GNN-based network modeling. However, the current support is weak because the results are single-run point estimates without confidence intervals, the 'Paper GRU' baseline is taken from the original publication rather than reproduced in the same harness, and one experiment adds an uncontrolled fine-tuning step. The contribution is therefore primarily an open-source code artifact rather than a validated scientific claim.

major comments (5)
  1. [§7, Tables 1 and 5] The central claim that the LSTM variant achieves the lowest MAPE in most delay prediction tasks is not supported by the paper's own tables. For example, Table 1 FatTree128 shows GRU Model at 0.45% versus LSTM at 0.58%; Table 5 On/Off shows GRU Model at 2.82% versus LSTM at 2.84%; Table 5 Modulated shows GRU Model at 5.29% versus LSTM at 5.42%; and Table 1 FatTree64 is a tie at 0.50% with Paper GRU better at 0.44%. Moreover, §7.1 states that 'LSTM cells consistently achiev[ed] the lowest MAPE' for the Fat Tree topology, which is contradicted by Table 1. Because no error bars, confidence intervals, or significance tests are reported, the reported differences cannot be distinguished from noise. Please provide repeated runs with variance estimates and revise the claims to match the evidence.
  2. [§7, Tables 1–5] The 'Paper GRU' numbers are cited verbatim from the original RouteNet-Fermi paper rather than produced with the same codebase and training pipeline used for the reimplemented variants. Since the authors' own GRU reimplementation often differs substantially from the Paper GRU baseline (e.g., Table 2: 2.18% versus 5.67% for Real Traffic), this is not a controlled comparison. All conclusions that the authors' models outperform the original rely on this cross-codebase comparability, which is not established. Please retrain or rerun the original GRU implementation in the same harness, or provide a detailed component-by-component comparison of the training configurations to justify the comparison.
  3. [§4.3, Table 2] The Real Traffic experiment is described as using 'a previous checkpoint, fine-tuning it using 200 samples of the GEANT topology.' This introduces an uncontrolled variable: the source of the checkpoint, the fine-tuning procedure, and the number of fine-tuning steps are not specified. As a result, the reported 1.82% MAPE for the LSTM in Table 2 cannot be attributed to the LSTM architecture alone. Please specify and justify the fine-tuning protocol, or rerun the experiment from scratch.
  4. [§5 and §8, Table 5] The paper claims that LSTM shows 'particular strength in capturing temporal dependencies critical for jitter prediction,' yet Table 5 reports LSTM jitter MAE values that are an order of magnitude worse than the Paper GRU baseline, e.g., Constant Bitrate: 0.164 versus 0.008; On/Off: 0.272 versus 0.018; and All Multiplexed: 0.503 versus 0.034. Section 8 acknowledges that 'for traffic model jitter prediction task, the training struggled compared to the paper,' but the manuscript does not analyze or explain this failure. This internal inconsistency between the claimed strength and the reported results should be resolved.
  5. [§6.1] The training configuration states that the number of epochs is 'task specific numbers' but does not report those numbers, nor does it specify random seeds, weight initialization, or early-stopping criteria. Without these details the experiments are not reproducible, and the absence of any seed variation makes it impossible to assess the stability of the single-point metrics in the tables.
minor comments (5)
  1. [Abstract] The abstract contains a typo: 'We improve the the original architecture' should read 'We improve the original architecture.'
  2. [§3.1] The paragraph contains multiple grammatical and formatting errors, including 'so, their measurable needs grows proportional with the size of the traffic intensity and network' and concatenated words such as 'forlarge-scalenetworks' and 'Overcomethislimitation'; the prose needs careful proofreading.
  3. [§4.3] The MAWI repository is cited as reference [27], but reference [27] is the SNDlib paper, and SNDlib is cited as reference [28], which is the MAWILab paper; these two citations appear to be swapped.
  4. [§6, LSTM equations] In the LSTM forget-gate equation, f_t = σ(W_f · (h_{t−1}, x_t) + b_f), parentheses are used instead of square brackets for the concatenation, which is inconsistent with the GRU and RNN equations.
  5. [Figures 4–9] The validation-loss subfigures do not label which curve corresponds to each model variant; adding legends for RNN, LSTM, and GRU would improve interpretability.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: empirical evaluation against simulator ground truth is self-contained, with no load-bearing step reducing to its own inputs.

full rationale

This paper is an empirical re-implementation study rather than a theoretical derivation. It re-implements the RouteNet-Fermi architecture with RNN, GRU, and LSTM cells and evaluates them on OMNeT++-generated datasets, comparing MAPE and MAE against simulator ground truth and against the original RouteNet-Fermi paper's reported numbers. No parameter is fitted to the evaluation target, and no predicted metric is used as an input to define the model or to construct the loss functions. The 'Paper GRU' column is an external published baseline from a different research group, not a self-citation, so the comparison is not circular even though its fairness can be questioned. The central claim that LSTM is often the most accurate is a measured empirical outcome, not a quantity forced by definition. Possible weaknesses, such as absent error bars, single runs, and cross-codebase comparability of the 'Paper GRU' numbers, are validity and reproducibility concerns, not circularity. No self-definitional equation, no fitted input renamed as a prediction, no imported uniqueness theorem, and no ansatz smuggled through self-citation appear in the manuscript. Therefore the circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central comparison rests on treating OMNeT++ simulations as ground truth and on treating the original paper's reported metrics as a valid baseline. No invented entities. Standard model hyperparameters (hidden size 32, learning rate 0.001) are held constant across variants, so they are not fitted to data.

assumptions (3)
  • domain assumption OMNeT++ simulation outputs for delay, jitter, and loss are accurate ground-truth values for network performance.
    Section 4 describes datasets generated with OMNeT++ v5.5.1 and treats the simulator metrics as the targets the models are trained and evaluated against, without any validation against real network measurements.
  • domain assumption The RouteNet-Fermi results labeled 'Paper GRU' in the tables are directly comparable to the authors' reimplemented models.
    Section 7 compares against 'Paper GRU' numbers taken from Ferriol-Galmés et al. rather than re-run under identical experimental conditions, so differences could stem from training procedure rather than cell type.
  • standard math MAPE and MAE are appropriate error metrics for the delay/jitter and loss tasks respectively.
    Section 6.1 specifies these losses; the paper does not discuss whether relative errors are appropriate for jitter, which can be near zero.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RouteNet-Fermi: Network Modeling With GNN (Analysis And Re-implementation)." pith.science (2026). https://pith.science/paper/OIEYY4SX

@misc{pith2026241205649,
  author       = {Pith},
  title        = {Pith review of: RouteNet-Fermi: Network Modeling With GNN (Analysis And Re-implementation)},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OIEYY4SX}},
  note         = {Machine review of arXiv:2412.05649}
}
read the original abstract

Network performance modeling presents important challenges in modern computer networks due to increasing complexity, scale, and diverse traffic patterns. While traditional approaches like queuing theory and packet-level simulation have served as foundational tools, they face limitations in modeling complex traffic behaviors and scaling to large networks. This project presents an extended implementation of RouteNet-Fermi, a Graph Neural Network (GNN) architecture designed for network performance prediction, with additional recurrent neural network variants. We improve the the original architecture by implementing Long Short-Term Memory (LSTM) cells and Recurrent Neural Network (RNN) cells alongside the existing Gated Recurrent Unit (GRU) cells implementation. This work contributes to the understanding of recurrent neural architectures in GNN-based network modeling and provides a flexible framework for future experimentation with different cell types.

Figures

Figures reproduced from arXiv: 2412.05649 by the authors.

Figure 1
Figure 1. Black-box representation of RouteNet-Fermi [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Schematic representation of RouteNet-Fermi [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Comparison of neural network cell architectures. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (5 more)
Figure 5
Figure 5. Figure 5: Comparison of model validation loss per epoch for [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 4
Figure 4. Figure 4: Comparison of model validation loss per epoch for [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 6
Figure 6. Figure 6: Comparison of delay model validation loss per [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 8
Figure 8. Figure 8: Comparison of model validation loss per epoch for [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]
Figure 9
Figure 9. Figure 9: Comparison of model validation loss per epoch for [PITH_FULL_IMAGE:figures/full_fig_p006_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 27 canonical work pages

  1. [1]

    T. G. Robertazzi, Computer Networks and Systems: Queueing Theory and Performance Evaluation , Cham, Switzerland: Springer, 2000

    work page 2000

  2. [2]

    Large-Scale Network Simu- lation: How Big? How Fast?

    R. M. Fujimoto, K. Perumalla, A. Park, H. Wu, M. H. Ammar, and G. F. Riley, “Large-Scale Network Simu- lation: How Big? How Fast?” in Proceedings of the 11th IEEE/ACM International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommu- nication Systems (MASCOTS) , Oct. 2003, pp. 116–123

    work page 2003

  3. [3]

    & Cabellos-Aparicio, A

    Ferriol-Galmés, M., Paillisse, J., Suárez-Varela, J., Rusek, K., Xiao, S., Shi, X., ... & Cabellos-Aparicio, A. (2023). RouteNet-Fermi: Network modeling with graph neural networks. IEEE/ACM transactions on network- ing, 31(6), 3080-3095

    work page 2023

  4. [4]

    Experience-Driven Networking: A Deep Reinforcement Learning Based Approach,

    Z. Xu et al. , “Experience-Driven Networking: A Deep Reinforcement Learning Based Approach,” in Proceed- ings of the IEEE INFOCOM Conference on Computer Communications, Apr. 2018, pp. 1871–1879

    work page 2018

  5. [5]

    Empirical Performance of Weibull Self-Similar Tele-Traffic Model,

    J. Popoola and R. Ipinyomi, “Empirical Performance of Weibull Self-Similar Tele-Traffic Model,” International Journal of Engineering and Applied Sciences , vol. 4, no. 8, 2017, Art. no. 257389

    work page 2017

  6. [6]

    Discrete Event Simulation System,

    A. Varga, “Discrete Event Simulation System,” inPro- ceedings of the European Simulation Multiconference (ESM), 2001, pp. 1–7

    work page 2001

  7. [7]

    The Ns-3 Network Simulator,

    G. F. Riley and T. R. Henderson, “The Ns-3 Network Simulator,” in Modeling and Tools for Network Simula- tion, Cham, Switzerland: Springer, 2010, pp. 15–34

    work page 2010

  8. [8]

    Mimic-Net: Fast Performance Estimates for Data Center Networks with Machine Learning,

    Q. Zhang, K. K. W. Ng, C. Kazer, S. Yan, J. Sedoc, and V. Liu, “Mimic-Net: Fast Performance Estimates for Data Center Networks with Machine Learning,” in Proceedings of the ACM SIGCOMM Conference , Aug. 2021, pp. 287–304

    work page 2021

Show all 30 references
  1. [9]

    Generating Network Topologies That Obey Power Laws,

    C. R. Palmer and J. G. Steffan, “Generating Network Topologies That Obey Power Laws,” in Proceedings of the IEEE Global Telecommunications Conference (Globe- com), Nov. 2000, pp. 434–438

    2000

  2. [10]

    Extensions of Recurrent Neural Network Language Model,

    T. Mikolov, S. Kombrink, L. Burget, J. Cernocky, and S. Khudanpur, “Extensions of Recurrent Neural Network Language Model,” in Proceedings of the IEEE Interna- tional Conference on Acoustics, Speech, and Signal Pro- cessing (ICASSP), May 2011, pp. 5528–5531

    2011

  3. [11]

    Modeling and Prediction of the Internet End-to-End Delay Using Recurrent Neural Networks,

    S. Belhaj and M. Tagina, “Modeling and Prediction of the Internet End-to-End Delay Using Recurrent Neural Networks,” Journal of Networks , vol. 4, no. 6, pp. 528– 535, Aug. 2009

    2009

  4. [12]

    Graphon Neural Networks and the Transferability of Graph Neural Net- works,

    L. Ruiz, L. Chamon, and A. Ribeiro, “Graphon Neural Networks and the Transferability of Graph Neural Net- works,” in Proceedings of the Advances in Neural Infor- mation Processing Systems (NeurIPS), vol. 33, 2020, pp. 1702–1712

    2020

  5. [13]

    The Graph Neural Network Model,

    F. Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, and G. Monfardini, “The Graph Neural Network Model,” IEEE Transactions on Neural Networks , vol. 20, no. 1, pp. 61–80, Jan. 2016

    2016

  6. [14]

    Relational Inductive Bi- ases, Deep Learning, and Graph Networks,

    P. W. Battaglia et al. , “Relational Inductive Bi- ases, Deep Learning, and Graph Networks,” 2018, arXiv:1806.01261

    2018 arXiv

  7. [15]

    Graph Neural Networks: A Review of Methods and Applications,

    J. Zhou et al. , “Graph Neural Networks: A Review of Methods and Applications,” 2018,arXiv:1812.08434

    2018 arXiv

  8. [16]

    Performance Evaluation of Weighted Fair Queuing System Using Ma- trix Geometric Method,

    A. Al-Sawaai, I. Awan, and R. Fretwell, “Performance Evaluation of Weighted Fair Queuing System Using Ma- trix Geometric Method,” in Proceedings of the Inter- national Conference on Research in Networking , Cham, Switzerland: Springer, 2009, pp. 66–78

    2009

  9. [17]

    Analy- sisofaFiniteCapacityNon-PreemptivePriorityQueue,

    A. S. Kapadia, M. F. Kazmi, and A. C. Mitchell, “Analy- sisofaFiniteCapacityNon-PreemptivePriorityQueue,” Computers & Operations Research , vol. 11, no. 3, pp. 337–343, Jan. 1984

    1984

  10. [18]

    Digital Twin for 5G and Beyond,

    H. X. Nguyen, R. Trestian, D. To, and M. Tatipamula, “Digital Twin for 5G and Beyond,”IEEE Communica- tions Magazine, vol. 59, no. 2, pp. 10–15, Feb. 2021

    2021

  11. [19]

    RouteNet-Erlang: A Graph Neural Network for Network Performance Evaluation,

    M. Ferriol-Galmes et al. , “RouteNet-Erlang: A Graph Neural Network for Network Performance Evaluation,” in Proceedings of the IEEE INFOCOM Conference on Computer Communications, May 2022, pp. 2018–2027

    2022

  12. [20]

    Discrete event simulation system,

    A. Varga, “Discrete event simulation system,” in Euro- pean Simulation Multiconference (ESM), 2001, pp. 1–7

    2001

  13. [21]

    Bnnetsimulator,

    M. F.-G. Albert López et al., “Bnnetsimulator,”

  14. [22]

    Deepqueuenet: Towards scal- able and generalized network performance estimation with packet-level visibility,

    Q. Yang, X. Peng et al., “Deepqueuenet: Towards scal- able and generalized network performance estimation with packet-level visibility,” in Proceedings of the ACM SIGCOMM 2022 Conference, ser. SIGCOMM ’22. New York, NY, USA: Association for Computing Machinery, 2022, p.441–457...

    2022

  15. [23]

    Wavelength converter place- ment in least-load-routing-based optical networks using genetic algorithms,

    X. Hei, J. Zhang et al., “Wavelength converter place- ment in least-load-routing-based optical networks using genetic algorithms,” Journal of Optical Networking, vol. 3, no. 5, pp. 363–378, 2004

    2004

  16. [24]

    Fast emergency paths schema to overcome transient link failures in ospf routing,

    F. Barreto et al., “Fast emergency paths schema to overcome transient link failures in ospf routing,” arXiv preprint arXiv:1204.2465, 2012

    2012 arXiv

  17. [25]

    Performance evalua- tion of integrated otn/dwdm networks with single-stage multiplexing of optical channel data units,

    J. Pedro, J. Santos, and J. Pires, “Performance evalua- tion of integrated otn/dwdm networks with single-stage multiplexing of optical channel data units,” in Inter- national Conference on Transparent Optical Net-works, 2011, pp. 1–4

    2011

  18. [26]

    A nonstationary poisson view of internet traffic,

    T. Karagiannis et al., “A nonstationary poisson view of internet traffic,” in IEEE INFOCOM, vol. 3, 2004, pp. 1558–1569

    2004

  19. [27]

    Sndlib 1.0—survivable network design library,

    S. Orlowski, R. Wessäly et al., “Sndlib 1.0—survivable network design library,” Networks: An International Journal, vol. 55, no. 3, pp. 276–286, 2010. [Online]. Avail- able: http://sndlib.zib.de 7

    2010

  20. [28]

    MAWILab: Combin- ing Diverse Anomaly Detectors for Automated Anomaly Labeling and Performance Benchmarking,

    R. Fontugne, P. Borgnat et al., “MAWILab: Combin- ing Diverse Anomaly Detectors for Automated Anomaly Labeling and Performance Benchmarking,” in ACM CoNEXT ’10, Philadelphia, PA, December 2010, pp. –

    2010

  21. [29]

    Quality-of-service class specific traf- fic matrices in ip/mpls networks,

    S. Schnitter et al., “Quality-of-service class specific traf- fic matrices in ip/mpls networks,” in ACM Internet Mea- surement Conference, 2007, pp. 253–258. A Appendix Here all the results from the tables are plotted. 8

    2007

  22. [2023]

    Available: https://github.com/BNN- UPC/BNNetSimulator

    [Online]. Available: https://github.com/BNN- UPC/BNNetSimulator

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.