SCION Path Performance Toolkit and Benchmark for Advancing Machine Learning in Next-Generation Networks: ScionPathML

Damien Rossi; Lars Herschbach; Sina Keshvadi

REVIEW 4 major objections 6 minor 8 references

SCION Path Performance Toolkit and Benchmark for Advancing Machine Learning in Next-Generation Networks: ScionPathML

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ScionPathML packages SCION path measurements into ML-ready datasets and five benchmark tasks.

desk verdict ScionPathML is a genuine toolkit-and-dataset contribution, but its benchmark claims are weakened by synthetic ground truth in two tasks and missing split/error details. read the letter →

arxiv 2509.07154 v1 pith:WDXBWGPO submitted 2025-09-08 cs.NI

Damien Rossi , Lars Herschbach , Sina Keshvadi This is my paper · ORCID

classification cs.NI

keywords path-awarenetworkingSCIONPathMLmachinelearningnetworkmeasurementdatacollectionbenchmarkmultipathtransport

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ScionPathML is a Python library that wraps SCION's path-discovery, ping, bandwidth, and traceroute tools into an automated pipeline producing time-indexed, machine-learning-ready datasets. The paper's central claim is that this toolkit, together with a four-week dataset from four SCIONLab autonomous systems and five benchmark tasks, makes ML research on path-aware networks reproducible and accessible. If the claim is right, researchers without deep SCION expertise can generate datasets, compare models, and build predictors for path performance, failures, and anomalies. Reported baselines quantify the promise: RTT forecast MAE of 3.88 ms, path-failure F1 of 0.86, anomaly-detection AUC of 0.77, and 99% bottleneck-localization accuracy.

What carries the argument

The load-bearing object is ScionPathML itself, a modular Python library that schedules and wraps four SCION command-line tools — scion showpaths, scion ping, scion-bwtestclient, and scion traceroute — and normalizes their outputs into JSON records with consistent metadata, then converts them to CSV. Around that pipeline sits a benchmark harness that defines five tasks with fixed problem formulations and evaluation metrics. The element that makes the claims testable is the standardized time-indexed record: each measurement row carries a timestamp, source and destination AS, a path identifier, and metric values, so models can be trained and compared per path.

What would settle it

Run the same pipeline on a larger SCIONLab deployment (ten or more autonomous systems) with a 5-minute sampling cadence and re-run the five benchmarks; if failure-prediction F1 falls well below 0.86 or RTT-forecast MAE grows far beyond 3.88 ms, the four-AS dataset is not representative. Alternatively, if a trivial persistence model (predicting the last observed value) matches or beats the 3.88 ms RTT MAE on the public dataset, the claimed predictability of RTT would be put in doubt.

Watch

Extended reading notes

Core claim

The paper reports that a four-week measurement campaign with a 30-minute cadence over four SCIONLab autonomous systems produced a dense time-indexed dataset of RTT, loss, jitter, bandwidth, per-hop latencies, and path-availability fingerprints. On this dataset the paper defines and baselines five tasks: one-step-ahead RTT and bandwidth forecasting (MAE 3.88 ms and 21.47 Mbps), path-failure prediction as binary classification (F1 0.86), unsupervised anomaly detection with synthetically injected anomalies (AUC 0.77), QoE-based path recommendation (21–30% satisfaction), and bottleneck-hop localization (99% accuracy). The paper also claims the measurements show a dynamic control plane, with short path lifetimes and asymmetric path availability between endpoints, and a trade-off in which concurrent multipath transmission raises aggregate throughput but can degrade the latency and reliability of individual paths. These observations are presented as evidence that SCION performance is learnable enough for ML to be useful, while still demanding models that cope with instability.

Load-bearing premise

Everything rests on the assumption that four cloud-hosted SCIONLab nodes running under consistent conditions and sampled every 30 minutes represent SCION broadly enough for the baselines and conclusions to transfer; if real topologies are larger, more heterogeneous, or more volatile, the reported numbers may not hold elsewhere.

Editorial extensions

If this is right

RTT forecasting at 3.88 ms MAE suggests lightweight linear models can support proactive path selection on relatively stable SCION paths.
The failure-prediction F1 of 0.86 implies that a substantial share of SCION outages are preceded by measurable performance degradation, so proactive rerouting is feasible for those cases.
The split between linear models for latency and tree-based models for bandwidth points to a hybrid modeling strategy for multi-metric path prediction.
The low QoE satisfaction rates of 21–30% show that heuristic path ranking is the weak spot of the benchmark, making learned policies the obvious next target.
99% bottleneck-localization accuracy indicates that per-hop latency vectors carry enough signal for automated, hop-level network diagnostics.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A direct test of the representativeness assumption would be deploying the same pipeline on a larger, more heterogeneous SCIONLab topology with shorter sampling intervals; if failure-prediction F1 and RTT MAE drop materially, the reported baselines should be read as optimistic upper bounds.
The path asymmetry the paper observes implies that multipath schedulers should not treat forward and reverse path sets as interchangeable, a design constraint the paper leaves implicit.
The sliding-window formulation can be reused for multi-step and long-horizon forecasting or for sequence models the paper did not train, so the dataset's value likely extends beyond the five reported baselines.
Making the synthetic anomaly-injection procedure public would let other unsupervised detectors be compared on identical ground truth, turning Task 3 into a sharper benchmark.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: ScionPathML is a real engineering contribution—a Python toolkit that wraps SCION's CLI measurement tools into a scheduled pipeline and outputs ML-ready CSV files. The paper ships code and data, and that alone makes it worth engaging with if you care about path-aware networking. What's not yet solid are the benchmark numbers, especially two of the five tasks where the ground truth is synthetic but the presentation reads like validation on real SCION diagnostics.

The gap it fills is genuine. There is no prior SCION-specific ML dataset or benchmark framework, and ScionPathML lowers the entry barrier by automating showpaths, ping, bwtest, traceroute, and multipath variants. The design is described in enough detail to reproduce the pipeline, and the four-AS SCIONLab deployment is a reasonable first testbed. The authors are candid in Section 8 that the scope is limited to four nodes under consistent conditions.

Now the soft spots, in order of how much they matter. First, Tasks 3 and 5. In §7.3.2, ground-truth anomalies are synthetically injected; in §7.5.2, bottleneck positions are created by 'introducing artificial delays at randomly chosen hops.' So the reported AUC of 0.77 and accuracy of 99% measure how well the models detect synthetic injections, not real attacks or congestion. That is not a fatal flaw for a first benchmark—controlled injection is a reasonable labeling strategy—but the paper should state this prominently, and the headline numbers currently overstate diagnostic readiness. Second, all baseline results are single point estimates with no error bars, train/test splits, or hyperparameters. That is a real weakness for a benchmark paper; a reader cannot tell whether the F1 of 0.86 is stable across splits. Third, the arXiv abstract describes a longitudinal measurement study of path stability, diversity, and path discrepancy, but the full text is about the toolkit and baselines; those measurement findings do not appear in the manuscript. That mismatch will confuse readers and needs fixing.

Overall, this is a useful artifact paper for the SCION/MPQUIC community. The toolkit and dataset deserve refereeing; the benchmark claims need to be tempered and better documented. I would send it to peer review with a request for major revision, not desk-reject.

Referee Report

4 major / 6 minor

Summary. The paper introduces ScionPathML, an open-source Python toolkit that automates the collection of SCION path performance measurements (RTT, loss, jitter, bandwidth, per-hop RTTs) and converts them into ML-ready time-series datasets. The authors deploy the toolkit on four SCIONLab ASes over four weeks and present five benchmark tasks with baseline models: RTT/bandwidth forecasting, path failure prediction, malicious path detection, multi-objective path recommendation, and bottleneck localization. The paper claims three contributions: a reusable software kit, the first comprehensive public SCION path performance dataset, and benchmark tasks with baseline results. The central toolkit design is described in detail, but the empirical baseline results are limited by synthetic ground truth in two tasks and by missing statistical details in all tasks.

Significance. If the claims are correct, ScionPathML fills a real gap in the SCION ecosystem by providing a standardized, automated measurement pipeline and public data that could lower the barrier for ML research on path-aware networks. The open-source code and dataset are concrete artifacts that the community can reuse. However, the benchmark's value for tracking progress is currently weakened by the synthetic-label evaluation of Tasks 3 and 5 and by the absence of error bars, train/test splits, and hyperparameter details. The paper explicitly acknowledges the four-AS representativeness limitation, which is a strength in transparency. The contribution is more of an engineering infrastructure step than a validated ML benchmark at this stage.

major comments (4)

[§7.5.2 and §7.3.2] The reported headline results for Task 5 (99% bottleneck localization accuracy) and Task 3 (0.77 anomaly detection AUC) are evaluated against synthetically injected ground truth. The text states that 'Synthetic training data was generated by introducing artificial delays at randomly chosen hops' and that 'Ground truth anomalies were synthetically injected to enable controlled evaluation.' Consequently, the numbers measure the model's ability to detect injected delay spikes and injected anomalies, not real SCION bottlenecks or malicious paths. Since the paper claims 'benchmark tasks and baseline models' for diagnostic and security use cases, these results do not currently support that claim. The authors should either collect and label real incidents, or explicitly reframe Tasks 3 and 5 as controlled synthetic benchmarks and temper the language that implies real-world diagnostic validity.
[§7, Tables 1–6] The baseline results are reported as single point estimates without confidence intervals, standard deviations, number of repeated runs, train/test splits, or hyperparameter settings. For a benchmark intended to foster reproducible research, this is insufficient. For example, Table 1 reports an RTT MAE of 3.878 ms with no indication of variance or the temporal split used; Table 2 reports an F1-score to 12 decimal places, which is misleading absent any precision estimate. The paper should specify the evaluation protocol (e.g., temporal versus random split), the model hyperparameters, and report mean and standard deviation over multiple runs.
[§7.2.2] Task 2's ground-truth definition is underspecified. The paper says the model predicts 'whether a path will become unavailable at T+1' but never defines how unavailability is derived from the SCION measurements (e.g., whether it is based on scion showpaths results, ping failures, or another criterion), how missing observations are handled, or the class balance of the failure label. Without this information, the reported F1 of 0.86 cannot be interpreted, reproduced, or compared with future work.
[§6.3 and §7] The measurement campaign description lacks essential statistics: the number of distinct paths observed per AS pair, the total number of samples in the dataset, the number of path failure events, and the distribution of path availability over time. The 30-minute sampling interval is coarse relative to the path churn and short-lived failures acknowledged in §7.2.3 and §8. The paper should document these dataset characteristics so readers can judge the statistical power behind the baseline results and the generalizability claims.

minor comments (6)

[Abstract (front matter) vs. Full-text Abstract] The abstract appearing at the beginning of the manuscript (starting 'Path-aware networks promise...') does not match the abstract in the full text (starting 'Machine learning (ML) holds great promise...'). The two describe different contributions and the paper should be revised to contain a single, consistent abstract.
[Author affiliation] The affiliation for Lars Herschbach lists 'Goethe University Franckfurt' with a typo; it should be 'Frankfurt'.
[Data Availability] The benchmarking tasks are hosted in a Google Drive folder, which is not a stable archival link. The authors should consider hosting the benchmark files on Zenodo, figshare, or a GitHub release to ensure long-term reproducibility.
[Figures] The text references Figures 1–6, but no figures are included in the provided manuscript. The final version should ensure all figures are present and clearly legible.
[Table 2] The F1-score is reported as 0.860541969596 with excessive precision; round to a reasonable number of significant digits (e.g., 0.86) or report with a confidence interval.
[References] Reference [3] is listed as 'Authors et al.' with a placeholder style; it should be completed with the actual author names and full citation.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: ScionPathML is an empirical toolkit and benchmark; Tasks 3/5 synthetic labels are a validity caveat, not a circular reduction.

full rationale

The paper contains no mathematical derivation whose output is present in its inputs. The central claims are the release of a software toolkit, a measurement dataset, and five benchmark tasks with baseline models; each is an empirical artifact rather than a derived result. In Tasks 3 and 5 the ground truth is indeed synthetic (Section 7.3.1: 'Area Under the ROC Curve (AUC-ROC) using synthetically injected anomalies for controlled evaluation'; Section 7.5.2: 'Synthetic training data was generated by introducing artificial delays at randomly chosen hops'), but this is not circular: the models must still learn to map observed features to the injected labels, the reported metrics (AUC 0.77, accuracy 99%) could have been much worse, and the paper explicitly frames the evaluation as controlled and acknowledges external-validity limits ('A limitation of this research is its limited scope as it uses data obtained from four SCION AS nodes...'). The only near-self-citation is reference [3], which appears in Related Work and is not load-bearing. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported, and no ansatz is smuggled in via citation.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No new physical or network entities are introduced. The load-bearing assumptions are the representativeness of the four-AS SCIONLab deployment, the accuracy of the wrapped SCION tools, the sampling cadence, and the use of synthetic injections as proxies for real anomalies and failures.

free parameters (4)

Synthetic anomaly injection parameters (Task 3) = not specified
Ground truth anomalies for malicious path detection are "synthetically injected" in Section 7.3.2; the magnitude, duration, and frequency of injections control the reported AUC of 0.77 and are not reported.
Synthetic bottleneck delay parameters (Task 5) = not specified
Artificial delays introduced at randomly chosen hops in Section 7.5.2; the sharpness of these delays likely drives the 99% accuracy, and no real-world bottleneck validation is given.
Sliding window length and forecast horizon = N=12, T+1
Tasks 1 and 2 use a 12-timestep window and one-step-ahead horizon (Sections 7.1.1 and 7.2.2); results are not tested across other window sizes or horizons.
QoE profile thresholds = specified in Table 4
The five QoE profiles use user-chosen thresholds for RTT, loss, and bandwidth; satisfaction rates of 21-30% depend directly on these choices and are not varied.

assumptions (4)

domain assumption SCIONLab is representative of real SCION network behavior.
The toolkit and benchmark conclusions are drawn from four SCIONLab AS nodes (Sections 6.1 and 8); the paper itself states that generalization to larger topologies is unclear.
domain assumption The wrapped SCION CLI tools report accurate RTT, loss, bandwidth, and per-hop latency.
ScionPathML treats scion ping, scion-bwtestclient, and scion traceroute outputs as ground truth (Sections 4 and 5.1.2) without independent validation of the measurement tools.
domain assumption A 30-minute sampling interval captures path dynamics relevant to failure prediction.
The measurement campaign runs every 30 minutes (Section 6.3), and Section 7.2.3 admits that short-lived disruptions lasting less than one interval are missed.
ad hoc to paper Synthetically injected anomalies and delays model real malicious events and bottlenecks.
Tasks 3 and 5 generate ground truth by injecting patterns into measured data (Sections 7.3.2 and 7.5.2); the validity of the benchmark scores depends on this equivalence, which is not demonstrated.

how reviews work

0 comments

Cite this review

Pith. "Pith review of SCION Path Performance Toolkit and Benchmark for Advancing Machine Learning in Next-Generation Networks: ScionPathML." pith.science (2026). https://pith.science/paper/WDXBWGPO

@misc{pith2026250907154,
  author       = {Pith},
  title        = {Pith review of: SCION Path Performance Toolkit and Benchmark for Advancing Machine Learning in Next-Generation Networks: ScionPathML},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WDXBWGPO}},
  note         = {Machine review of arXiv:2509.07154}
}

read the original abstract

Path-aware networks promise enhanced performance and resilience through multipath transport, but a lack of empirical data on their real-world dynamics hinders the design of effective protocols. This paper presents a longitudinal measurement study of the SCION architecture on the global SCIONLab testbed, characterizing the path stability, diversity, and performance crucial for protocols like Multipath QUIC (MPQUIC). Our measurements reveal a dynamic environment, with significant control-plane churn and short path lifetimes in parts of the testbed. We identify and characterize path discrepancy, a phenomenon where routing policies create asymmetric path availability between endpoints. Furthermore, we observe a performance trade-off where concurrent multipath transmissions can improve aggregate throughput but may degrade the latency and reliability of individual paths. These findings demonstrate that protocols such as MPQUIC should explicitly account for high churn and path asymmetry, challenging common assumptions in multipath protocol design.

Figures

Figures reproduced from arXiv: 2509.07154 by the authors.

**Figure 3.** Predicted Availability for Paths 7.2.3 Interpretation: Many SCION path failures are preceded by detectable performance declines, such as increased latency and jitter, allowing some advance warning before total loss of connectivity. However, brief interruptions and abrupt failures without any prior indicators are more difficult to anticipate, revealing limitations in current predictive monitoring approaches. 5 [PI… view at source ↗

**Figure 4.** Anomaly Scores by Prediction Outcome (True vs. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗

Figures from the paper (1 more)

**Figure 5.** Figure 5: Profiles QoE satisfaction results. 7.4.3 Interpretation The heuristic baseline achieved relatively low QoE satisfaction rates across all profiles, with performance ranging between 21% and 30%. This reflects the difficulty of meeting strict QoE requirements in dynamic…

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

8 extracted references · 8 canonical work pages

[1]

and Zhang, Tiffany Hyun-Jin.SCION: A Secure Internet Architecture

Perrig, Adrian and Szalachowski, Pawel and Reischuk, Raphael M. and Zhang, Tiffany Hyun-Jin.SCION: A Secure Internet Architecture. Springer, 2017. https:// scion-architecture.net/pdf/SCION-book.pdf

work page 2017
[2]

Bandwidth Tester Application

SCIONLab Documentation Team. Bandwidth Tester Application. SCIONLab Documentation,

work page
[3]

New Directions in Automated Traffic Analysis

Authors et al. Machine Learning Applications in Path- Aware Networks.arXiv preprint arXiv:2008.02695, 2020.https://arxiv.org/pdf/2008.02695

work page Pith review arXiv 2008
[4]

nPrintML: A Toolkit for Network Traffic Analysis with Machine Learning

nPrintML Team. nPrintML: A Toolkit for Network Traffic Analysis with Machine Learning. https:// nprint.github.io/nprintml/, 2023

work page 2023
[5]

RIPE Atlas: A Global Internet Measure- ment Network.https://atlas.ripe.net/, 2023

RIPE NCC. RIPE Atlas: A Global Internet Measure- ment Network.https://atlas.ripe.net/, 2023

work page 2023
[6]

iPerf: The TCP/UDP Bandwidth Measurement Tool

Tirumala, A., Qin, F., Dugan, J., Ferguson, J., Gibbs, K. iPerf: The TCP/UDP Bandwidth Measurement Tool. https://iperf.fr/, 2005

work page 2005
[7]

P4- NetML: Enabling Network Measurement and Machine Learning with Programmable Data Planes

Kumar, S., Dang, H., Lin, Y ., Hu, H. P4- NetML: Enabling Network Measurement and Machine Learning with Programmable Data Planes. https://changgang-zheng.github. io/Home-Page/papers/Zheng_et_al_2023_ In-network_machine_learning.pdf, 2022. 8

work page 2022
[2023]

https://docs.scionlab.org/content/ apps/bwtester.html

work page

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

and Zhang, Tiffany Hyun-Jin.SCION: A Secure Internet Architecture

Perrig, Adrian and Szalachowski, Pawel and Reischuk, Raphael M. and Zhang, Tiffany Hyun-Jin.SCION: A Secure Internet Architecture. Springer, 2017. https:// scion-architecture.net/pdf/SCION-book.pdf

work page 2017

[2] [2]

Bandwidth Tester Application

SCIONLab Documentation Team. Bandwidth Tester Application. SCIONLab Documentation,

work page

[3] [3]

New Directions in Automated Traffic Analysis

Authors et al. Machine Learning Applications in Path- Aware Networks.arXiv preprint arXiv:2008.02695, 2020.https://arxiv.org/pdf/2008.02695

work page Pith review arXiv 2008

[4] [4]

nPrintML: A Toolkit for Network Traffic Analysis with Machine Learning

nPrintML Team. nPrintML: A Toolkit for Network Traffic Analysis with Machine Learning. https:// nprint.github.io/nprintml/, 2023

work page 2023

[5] [5]

RIPE Atlas: A Global Internet Measure- ment Network.https://atlas.ripe.net/, 2023

RIPE NCC. RIPE Atlas: A Global Internet Measure- ment Network.https://atlas.ripe.net/, 2023

work page 2023

[6] [6]

iPerf: The TCP/UDP Bandwidth Measurement Tool

Tirumala, A., Qin, F., Dugan, J., Ferguson, J., Gibbs, K. iPerf: The TCP/UDP Bandwidth Measurement Tool. https://iperf.fr/, 2005

work page 2005

[7] [7]

P4- NetML: Enabling Network Measurement and Machine Learning with Programmable Data Planes

Kumar, S., Dang, H., Lin, Y ., Hu, H. P4- NetML: Enabling Network Measurement and Machine Learning with Programmable Data Planes. https://changgang-zheng.github. io/Home-Page/papers/Zheng_et_al_2023_ In-network_machine_learning.pdf, 2022. 8

work page 2022

[8] [2023]

https://docs.scionlab.org/content/ apps/bwtester.html

work page