{"id":"aee38745-e6a4-4366-8381-32ced48b7e0d","arxiv_id":"2509.07154","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A Python toolkit and initial benchmark package that turns SCION path measurements into ML-ready datasets, with baseline models for forecasting, failure prediction, anomaly detection, path recommendation, and bottleneck localization.","lead":"ScionPathML is an open-source Python toolkit that automates the collection of SCION network measurements, such as latency, loss, and bandwidth, and turns them into machine-learning-ready datasets. The paper also introduces five baseline benchmark tasks with initial results, intended to lower the barrier to applying machine learning to SCION networks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Task 5's 99% bottleneck localization and Task 3's 0.77 anomaly AUC are measured on synthetically injected ground truth, so the headline baselines do not yet validate real SCION diagnostics.","rationale":"The paper's central deliverable is a toolkit plus an initial dataset and benchmark baselines. The toolkit and the real-data tasks (RTT/bandwidth forecasting, failure prediction, QoE recommendation) support that deliverable, and Section 8 honestly limits the four-node topology. However, the two diagnostic benchmarks—malicious path detection (§7.3.2) and bottleneck localization (§7.5.2)—are evaluated only with synthetic labels. The 99% bottleneck accuracy is therefore not evidence about real SCION bottlenecks, and the 0.77 AUC does not measure real malicious path behavior. This is an internal-validity weakness, not just an external-scope limitation, and it is not cured by unqualified wording elsewhere in the paper. I also note that the paper's metadata abstract advertises a longitudinal measurement study with path-discrepancy and MPQUIC findings that do not appear in the body; that presentation flaw reinforces the need for caution, but the synthetic-label problem is the more load-bearing technical issue. This concern supports the existing CONDITIONAL verdict rather than moving it, because the toolkit and the real-data benchmarks remain plausible and useful if the paper is revised to reframe the synthetic tasks or add real validation.","tokens_in":7597,"tokens_out":8151,"duration_ms":78768,"concrete_test":"Take the public traceroute CSVs and the Task 2 path-failure labels, build a real bottleneck test set by selecting degraded or failed time bins and labeling the hop with the largest preceding per-hop RTT increase, then evaluate the same Random Forest trained on the synthetic Task 5 data. If accuracy falls below 90%, the 99% headline is an artifact of synthetic delay injection; if it remains near 99%, the concern does not land.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is internal to the benchmark evaluation, not just the acknowledged four-node topology. In §7.5.2, Task 5 (bottleneck localization) is defined as predicting the hop index where a bottleneck begins, but the ground truth is generated by 'introducing artificial delays at randomly chosen hops.' The reported 99% accuracy therefore measures detection of injected delay spikes, not real SCION bottlenecks. Similarly, §7.3.2 evaluates Task 3 (malicious path detection) against 'synthetically injected anomalies,' so the 0.77 AUC does not establish detection of actual malicious paths. Since two of the five benchmark tasks are the paper's only security/diagnostic benchmarks, the claim of 'benchmark tasks and baseline models' for real diagnostic use cases is not supported by these numbers. Missing error bars and train/test splits compound the problem, but the synthetic labels are a validity issue that cannot be fixed by collecting more topology data alone.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ScionPathML, an open-source Python toolkit that automates the collection of SCION path performance measurements (RTT, loss, jitter, bandwidth, per-hop RTTs) and converts them into ML-ready time-series datasets. The authors deploy the toolkit on four SCIONLab ASes over four weeks and present five benchmark tasks with baseline models: RTT/bandwidth forecasting, path failure prediction, malicious path detection, multi-objective path recommendation, and bottleneck localization. The paper claims three contributions: a reusable software kit, the first comprehensive public SCION path performance dataset, and benchmark tasks with baseline results. The central toolkit design is described in detail, but the empirical baseline results are limited by synthetic ground truth in two tasks and by missing statistical details in all tasks.","tokens_in":7790,"tokens_out":4136,"duration_ms":35997,"significance":"If the claims are correct, ScionPathML fills a real gap in the SCION ecosystem by providing a standardized, automated measurement pipeline and public data that could lower the barrier for ML research on path-aware networks. The open-source code and dataset are concrete artifacts that the community can reuse. However, the benchmark's value for tracking progress is currently weakened by the synthetic-label evaluation of Tasks 3 and 5 and by the absence of error bars, train/test splits, and hyperparameter details. The paper explicitly acknowledges the four-AS representativeness limitation, which is a strength in transparency. The contribution is more of an engineering infrastructure step than a validated ML benchmark at this stage.","major_comments":[{"comment":"The reported headline results for Task 5 (99% bottleneck localization accuracy) and Task 3 (0.77 anomaly detection AUC) are evaluated against synthetically injected ground truth. The text states that 'Synthetic training data was generated by introducing artificial delays at randomly chosen hops' and that 'Ground truth anomalies were synthetically injected to enable controlled evaluation.' Consequently, the numbers measure the model's ability to detect injected delay spikes and injected anomalies, not real SCION bottlenecks or malicious paths. Since the paper claims 'benchmark tasks and baseline models' for diagnostic and security use cases, these results do not currently support that claim. The authors should either collect and label real incidents, or explicitly reframe Tasks 3 and 5 as controlled synthetic benchmarks and temper the language that implies real-world diagnostic validity.","section":"§7.5.2 and §7.3.2"},{"comment":"The baseline results are reported as single point estimates without confidence intervals, standard deviations, number of repeated runs, train/test splits, or hyperparameter settings. For a benchmark intended to foster reproducible research, this is insufficient. For example, Table 1 reports an RTT MAE of 3.878 ms with no indication of variance or the temporal split used; Table 2 reports an F1-score to 12 decimal places, which is misleading absent any precision estimate. The paper should specify the evaluation protocol (e.g., temporal versus random split), the model hyperparameters, and report mean and standard deviation over multiple runs.","section":"§7, Tables 1–6"},{"comment":"Task 2's ground-truth definition is underspecified. The paper says the model predicts 'whether a path will become unavailable at T+1' but never defines how unavailability is derived from the SCION measurements (e.g., whether it is based on scion showpaths results, ping failures, or another criterion), how missing observations are handled, or the class balance of the failure label. Without this information, the reported F1 of 0.86 cannot be interpreted, reproduced, or compared with future work.","section":"§7.2.2"},{"comment":"The measurement campaign description lacks essential statistics: the number of distinct paths observed per AS pair, the total number of samples in the dataset, the number of path failure events, and the distribution of path availability over time. The 30-minute sampling interval is coarse relative to the path churn and short-lived failures acknowledged in §7.2.3 and §8. The paper should document these dataset characteristics so readers can judge the statistical power behind the baseline results and the generalizability claims.","section":"§6.3 and §7"}],"minor_comments":[{"comment":"The abstract appearing at the beginning of the manuscript (starting 'Path-aware networks promise...') does not match the abstract in the full text (starting 'Machine learning (ML) holds great promise...'). The two describe different contributions and the paper should be revised to contain a single, consistent abstract.","section":"Abstract (front matter) vs. Full-text Abstract"},{"comment":"The affiliation for Lars Herschbach lists 'Goethe University Franckfurt' with a typo; it should be 'Frankfurt'.","section":"Author affiliation"},{"comment":"The benchmarking tasks are hosted in a Google Drive folder, which is not a stable archival link. The authors should consider hosting the benchmark files on Zenodo, figshare, or a GitHub release to ensure long-term reproducibility.","section":"Data Availability"},{"comment":"The text references Figures 1–6, but no figures are included in the provided manuscript. The final version should ensure all figures are present and clearly legible.","section":"Figures"},{"comment":"The F1-score is reported as 0.860541969596 with excessive precision; round to a reasonable number of significant digits (e.g., 0.86) or report with a confidence interval.","section":"Table 2"},{"comment":"Reference [3] is listed as 'Authors et al.' with a placeholder style; it should be completed with the actual author names and full citation.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The toolkit and dataset are potentially useful community artifacts, but the benchmark evaluation needs substantial rework before the paper meets the standard of a rigorous ML benchmark paper. The synthetic ground truth in Tasks 3 and 5 is the main validity concern; the missing statistical detail compounds it. The abstract inconsistency suggests the final editing was careless. I would encourage the editor to invite a revision addressing the major comments rather than rejecting, since the infrastructure contribution is real and the empirical issues are fixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: ScionPathML is a real engineering contribution—a Python toolkit that wraps SCION's CLI measurement tools into a scheduled pipeline and outputs ML-ready CSV files. The paper ships code and data, and that alone makes it worth engaging with if you care about path-aware networking. What's not yet solid are the benchmark numbers, especially two of the five tasks where the ground truth is synthetic but the presentation reads like validation on real SCION diagnostics.\n\nThe gap it fills is genuine. There is no prior SCION-specific ML dataset or benchmark framework, and ScionPathML lowers the entry barrier by automating showpaths, ping, bwtest, traceroute, and multipath variants. The design is described in enough detail to reproduce the pipeline, and the four-AS SCIONLab deployment is a reasonable first testbed. The authors are candid in Section 8 that the scope is limited to four nodes under consistent conditions.\n\nNow the soft spots, in order of how much they matter. First, Tasks 3 and 5. In §7.3.2, ground-truth anomalies are synthetically injected; in §7.5.2, bottleneck positions are created by 'introducing artificial delays at randomly chosen hops.' So the reported AUC of 0.77 and accuracy of 99% measure how well the models detect synthetic injections, not real attacks or congestion. That is not a fatal flaw for a first benchmark—controlled injection is a reasonable labeling strategy—but the paper should state this prominently, and the headline numbers currently overstate diagnostic readiness. Second, all baseline results are single point estimates with no error bars, train/test splits, or hyperparameters. That is a real weakness for a benchmark paper; a reader cannot tell whether the F1 of 0.86 is stable across splits. Third, the arXiv abstract describes a longitudinal measurement study of path stability, diversity, and path discrepancy, but the full text is about the toolkit and baselines; those measurement findings do not appear in the manuscript. That mismatch will confuse readers and needs fixing.\n\nOverall, this is a useful artifact paper for the SCION/MPQUIC community. The toolkit and dataset deserve refereeing; the benchmark claims need to be tempered and better documented. I would send it to peer review with a request for major revision, not desk-reject.","headline":"ScionPathML is a genuine toolkit-and-dataset contribution, but its benchmark claims are weakened by synthetic ground truth in two tasks and missing split/error details.","tokens_in":8311,"tokens_out":3324,"would_cite":true,"duration_ms":26144,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ScionPathML packages SCION path measurements into ML-ready datasets and five benchmark tasks.","keywords":["path-aware networking","SCION","ScionPathML","machine learning","network measurement","data collection","benchmark","multipath transport"],"falsifier":"Run the same pipeline on a larger SCIONLab deployment (ten or more autonomous systems) with a 5-minute sampling cadence and re-run the five benchmarks; if failure-prediction F1 falls well below 0.86 or RTT-forecast MAE grows far beyond 3.88 ms, the four-AS dataset is not representative. Alternatively, if a trivial persistence model (predicting the last observed value) matches or beats the 3.88 ms RTT MAE on the public dataset, the claimed predictability of RTT would be put in doubt.","tokens_in":7385,"feed_emoji":"📡","tokens_out":9114,"duration_ms":71354,"temperature":0.7,"pith_summary":"ScionPathML is a Python library that wraps SCION's path-discovery, ping, bandwidth, and traceroute tools into an automated pipeline producing time-indexed, machine-learning-ready datasets. The paper's central claim is that this toolkit, together with a four-week dataset from four SCIONLab autonomous systems and five benchmark tasks, makes ML research on path-aware networks reproducible and accessible. If the claim is right, researchers without deep SCION expertise can generate datasets, compare models, and build predictors for path performance, failures, and anomalies. Reported baselines quantify the promise: RTT forecast MAE of 3.88 ms, path-failure F1 of 0.86, anomaly-detection AUC of 0.77, and 99% bottleneck-localization accuracy.","feed_headline":"SCION path data gets an ML toolkit and five benchmarks","feed_subtitle":"Four-week SCIONLab measurements back baselines for forecasting, failure detection, and bottleneck localization.","key_machinery":"The load-bearing object is ScionPathML itself, a modular Python library that schedules and wraps four SCION command-line tools — scion showpaths, scion ping, scion-bwtestclient, and scion traceroute — and normalizes their outputs into JSON records with consistent metadata, then converts them to CSV. Around that pipeline sits a benchmark harness that defines five tasks with fixed problem formulations and evaluation metrics. The element that makes the claims testable is the standardized time-indexed record: each measurement row carries a timestamp, source and destination AS, a path identifier, and metric values, so models can be trained and compared per path.","core_discovery":"The paper reports that a four-week measurement campaign with a 30-minute cadence over four SCIONLab autonomous systems produced a dense time-indexed dataset of RTT, loss, jitter, bandwidth, per-hop latencies, and path-availability fingerprints. On this dataset the paper defines and baselines five tasks: one-step-ahead RTT and bandwidth forecasting (MAE 3.88 ms and 21.47 Mbps), path-failure prediction as binary classification (F1 0.86), unsupervised anomaly detection with synthetically injected anomalies (AUC 0.77), QoE-based path recommendation (21–30% satisfaction), and bottleneck-hop localization (99% accuracy). The paper also claims the measurements show a dynamic control plane, with short path lifetimes and asymmetric path availability between endpoints, and a trade-off in which concurrent multipath transmission raises aggregate throughput but can degrade the latency and reliability of individual paths. These observations are presented as evidence that SCION performance is learnable enough for ML to be useful, while still demanding models that cope with instability.","pith_inferences":["A direct test of the representativeness assumption would be deploying the same pipeline on a larger, more heterogeneous SCIONLab topology with shorter sampling intervals; if failure-prediction F1 and RTT MAE drop materially, the reported baselines should be read as optimistic upper bounds.","The path asymmetry the paper observes implies that multipath schedulers should not treat forward and reverse path sets as interchangeable, a design constraint the paper leaves implicit.","The sliding-window formulation can be reused for multi-step and long-horizon forecasting or for sequence models the paper did not train, so the dataset's value likely extends beyond the five reported baselines.","Making the synthetic anomaly-injection procedure public would let other unsupervised detectors be compared on identical ground truth, turning Task 3 into a sharper benchmark."],"forward_implications":["RTT forecasting at 3.88 ms MAE suggests lightweight linear models can support proactive path selection on relatively stable SCION paths.","The failure-prediction F1 of 0.86 implies that a substantial share of SCION outages are preceded by measurable performance degradation, so proactive rerouting is feasible for those cases.","The split between linear models for latency and tree-based models for bandwidth points to a hybrid modeling strategy for multi-metric path prediction.","The low QoE satisfaction rates of 21–30% show that heuristic path ranking is the weak spot of the benchmark, making learned policies the obvious next target.","99% bottleneck-localization accuracy indicates that per-hop latency vectors carry enough signal for automated, hop-level network diagnostics."],"supporting_citations":[{"why":"Supplies the SCION architecture and terminology that the toolkit and measurement design build on.","marker":"[1]"},{"why":"Documents the SCIONLab bandwidth tester that ScionPathML wraps for throughput measurements.","marker":"[2]"},{"why":"Supplies prior evidence that machine learning can be applied to path-aware networks, motivating the benchmark tasks.","marker":"[3]"},{"why":"Packet-level ML toolkit whose feature-extraction approach ScionPathML contrasts with its path-level measurement pipeline.","marker":"[4]"},{"why":"Large-scale Internet measurement platform that lacks SCION integration, motivating the new toolkit.","marker":"[5]"},{"why":"Throughput measurement tool that requires manual orchestration and no native multipath support, serving as a baseline for comparison.","marker":"[6]"},{"why":"Programmable data-plane ML work in P4/SDN environments that ScionPathML distinguishes from SCION-specific measurement.","marker":"[7]"}],"fun_headline_variants":["SCION path data set for ML: five benchmarks","SCIONLab data fuels ML for path-aware networks","Benchmarking ML on SCION's unstable paths","Five ML benchmarks from real SCION path data","SCION paths: messy, asymmetric, but ML-ready"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on the assumption that four cloud-hosted SCIONLab nodes running under consistent conditions and sampled every 30 minutes represent SCION broadly enough for the baselines and conclusions to transfer; if real topologies are larger, more heterogeneous, or more volatile, the reported numbers may not hold elsewhere.","fun_headline_variants_meta":{"raw":{"variants":["SCION path data set for ML: five benchmarks","SCIONLab data fuels ML for path-aware networks","Benchmarking ML on SCION's unstable paths","Five ML benchmarks from real SCION path data","SCION paths: messy, asymmetric, but ML-ready"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000275,"raw_usage":{"total_tokens":1625,"prompt_tokens":909,"completion_tokens":716,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":640}},"tokens_in":525,"tokens_out":716,"duration_ms":6395,"temperature":1.0,"reasoning_tokens":640,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:12:33.452588+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same pipeline on a larger SCIONLab deployment (ten or more autonomous systems) with a 5-minute sampling cadence and re-run the five benchmarks; if failure-prediction F1 falls well below 0.86 or RTT-forecast MAE grows far beyond 3.88 ms, the four-AS dataset is not representative. Alternatively, if a trivial persistence model (predicting the last observed value) matches or beats the 3.88 ms RTT MAE on the public dataset, the claimed predictability of RTT would be put in doubt.","supporting_citations":[{"cited_title":"and Zhang, Tiffany Hyun-Jin.SCION: A Secure Internet Architecture","cited_arxiv_id":null,"evidence_quote":"Supplies the SCION architecture and terminology that the toolkit and measurement design build on."},{"cited_title":"Bandwidth Tester Application","cited_arxiv_id":null,"evidence_quote":"Documents the SCIONLab bandwidth tester that ScionPathML wraps for throughput measurements."},{"cited_title":"New Directions in Automated Traffic Analysis","cited_arxiv_id":"2008.02695","evidence_quote":"Supplies prior evidence that machine learning can be applied to path-aware networks, motivating the benchmark tasks."},{"cited_title":"nPrintML: A Toolkit for Network Traffic Analysis with Machine Learning","cited_arxiv_id":null,"evidence_quote":"Packet-level ML toolkit whose feature-extraction approach ScionPathML contrasts with its path-level measurement pipeline."},{"cited_title":"RIPE Atlas: A Global Internet Measure- ment Network.https://atlas.ripe.net/, 2023","cited_arxiv_id":null,"evidence_quote":"Large-scale Internet measurement platform that lacks SCION integration, motivating the new toolkit."},{"cited_title":"iPerf: The TCP/UDP Bandwidth Measurement Tool","cited_arxiv_id":null,"evidence_quote":"Throughput measurement tool that requires manual orchestration and no native multipath support, serving as a baseline for comparison."},{"cited_title":"P4- NetML: Enabling Network Measurement and Machine Learning with Programmable Data Planes","cited_arxiv_id":null,"evidence_quote":"Programmable data-plane ML work in P4/SDN environments that ScionPathML distinguishes from SCION-specific measurement."}],"review_version":2}