UTM Performance Under Stressing Scenarios

Ian Jessen

REVIEW 4 major objections 4 minor 8 references

UTM Performance Under Stressing Scenarios

T0 review · 4 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read UTM replanning stalls when 40 operations hit in 30 seconds

desk verdict A useful new network-level UTM simulator with a plausible race-condition story, but the headline SCD-duration figure is an unvalidated implementation artifact risk until the authors add baseline comparison, statistical rigor, and code/data release. read the letter →

arxiv 2509.08124 v1 pith:5JE7VVFL submitted 2025-09-09 cs.NI cs.SYeess.SY

Ian Jessen This is my paper · ORCID

classification cs.NIcs.SYeess.SY

keywords UnmannedTrafficManagementUTMsimulationstrategicconflictdetectionOpaqueVersionNumberraceconditionairspaceaccessnetworklatencyscalability

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper uses a virtual UTM network simulation to test how the UTM architecture behaves when many operators must replan at once. It finds that the protocol's use of opaque version numbers to keep a shared airspace picture creates race conditions: under a burst of 40 replans in a 30-second window, strategic conflict detection alone can take close to 30 seconds, leaving less than half of the 60-second window an inflight operation has before it must declare a contingency. A second result shows that first-come-first-served airspace access is not simply a matter of who submits first: increasing the network latency between one USS and the shared DSS increases the acceptance rate of the other USS's operation. Together, the results argue that the current UTM synchronization approach may need added prioritization or synchronization primitives to remain safe and equitable at scale.

What carries the argument

The central mechanism is the Discovery and Synchronization Service (DSS) with opaque version numbers (OVNs): a central registry and airspace-picture keeper that lets each UTM Service Supplier prove it ran conflict detection against a current picture. Opaque version numbers mean a USS must restart strategic conflict detection whenever the DSS picture changes mid-check, and that restart loop is what converts high planning tempo into long SCD durations. The simulation also uses interposing network proxies to inject latency between a USS and the DSS, which enables the airspace-access result.

What would settle it

Instrument a real or higher-fidelity UTM deployment (or a validated digital twin) with the same 40-replan 30-second burst and measure the 95th-percentile strategic conflict detection duration. If it stays well below 30 seconds, or if the inflight operation completes replan within the 60-second grace period, the claimed demand point does not hold. Alternatively, run the exact scenario with negotiation or rerouting enabled and unlimited restarts; if SCD durations collapse, the race-condition bottleneck is an artifact of the simplified conflict detection.

Watch

Extended reading notes

Core claim

The paper claims that the UTM network has a demand point, revealed in simulation, where in-flight replanning can no longer be completed inside the allowable time window. The mechanism is the DSS's opaque version number (OVN) scheme: a USS must check the common airspace picture, but if the picture changes before it submits, it must restart. Under time-correlated demand spikes, these restarts cascade and push 95th-percentile SCD durations to roughly 30 seconds—nearly half of the 60-second grace period, before route regeneration, operator approval, or uplink are even counted. The paper also claims that network latency, not just submission order, decides who wins contested airspace under first-c

Load-bearing premise

The results depend on the simulation's USS, DSS, and FIMS implementations faithfully matching real UTM behavior under ASTM F3548-21; the paper provides no validation against a real deployment, and its conflict detection is simplified (no negotiation, at most five restarts).

Editorial extensions

If this is right

SCD latency grows with planning tempo because OVN-based race conditions force repeated restarts, so the bottleneck is architectural, not just computational.
At 40 replans in 30 seconds, the 95th-percentile SCD duration of about 30 seconds consumes nearly half the 60-second conformance grace period, leaving little time for the rest of a real replan.
A synchronization or priority mechanism, such as basic synchronization primitives, would be needed to protect time-critical replanning as UTM scales.
First-come-first-served airspace access is sensitive to USS-to-DSS network latency; operators with faster links gain acceptance at the expense of slower ones.
Degrading one USS's link to the DSS can flip which operation is accepted, so cyberattack or network congestion can change airspace allocation outcomes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the observed race-condition scaling applies beyond this scenario, real UTM deployments with more USS providers than the two modeled here could hit similar SCD bottlenecks at lower per-provider replan rates, because the number of contention points grows with the number of providers.
The latency-sensitivity result implies an incentive for USS operators to invest in low-latency DSS connections; regulators may need latency-based fairness metrics if airspace access is to remain neutral across operators.
A direct testable extension is to run the same burst scenario with SCD negotiation or rerouting enabled, or with more than five restart attempts; if durations fall sharply, the bottleneck lies in the abort-and-retry algorithm rather than the OVN protocol itself.
The 60-second grace period is an end-to-end operational requirement, so the finding implies that UTM performance requirements should be specified as end-to-end replan completion time, not just SCD duration.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

Quick take: this is one of those papers where the tool is the contribution and the numbers are the caution. ANAMLL is the first UTM simulation I've seen that puts the network layer—DSS/OVN synchronization, USS-to-DSS latency, race conditions—at the center, rather than treating comms as a fixed delay. The authors build a real federated simulation with an Execution Engine, interposing proxies, and scripted scenarios, and they use it to probe two questions that matter: can in-flight replanning keep up under a correlated demand spike, and does first-come-first-served airspace access get skewed by network latency. That's a genuine gap in the prior M&S literature, which mostly looks at airspace policy or routing algorithms. Credit where due: the scenario design is clear, the race-condition explanation is mechanically plausible (an OVN check invalidating a USS's SCD after a concurrent submission), and the fairness result—where slowing down USS2's link actually improves USS1's acceptance rate—is a nice, if intuitive, demonstration of latency-as-priority.

Where it gets soft is exactly where the stress test lands. The headline number—p95 SCD duration approaching 30 s in the 40-ops/30-s window, consuming half the 60-s grace period—rests entirely on ANAMLL's own implementation of USS, DSS, and SCD. That implementation is unvalidated: no benchmark against a standards-conformant USS/DSS, no reference deployment, no calibration. The paper even says future work will focus on validation, which is honest but also an admission. On top of that, the statistical reporting is thin. We're told results are aggregated across 'numerous executions' of each parameter set, but there's no number of runs, no error bars, no confidence intervals, so the 30-s figure could be one noisy run's p95. The simplified SCD algorithm (4D conflict detection, no negotiation, max 5 restarts) could change the race-condition dynamics in either direction. And the citation errors—[5] is SkyTrakx, not ASTM F3548-21 (that's [3])—need a mechanical fix.

None of this kills the paper. The central mechanism is a known distributed-systems issue and the simulation is a reasonable first probe. But 'demand point' oversells what's established. This is a hypothesis-generating simulation study, not a measurement of UTM. If I were reviewing it, I'd want validation of the USS/DSS implementations against some baseline, a sensitivity analysis around the SCD algorithm details, and the code and data released so others can reproduce the 30-s result.

Bottom line: bring this to a reading group if you want to discuss simulation validation standards in protocol-oriented systems; cite it if you work on UTM or PSU network design, with the caveat that the quantitative claims are unvalidated. I'd send it to peer review—the topic is timely, the tool is a real contribution, and the findings, if they hold up after revision, are worth knowing. It just isn't there yet.

Referee Report

4 major / 4 minor

Summary. The paper introduces ANAMLL, a virtual system-integration laboratory for UTM, and uses it to study a stressing scenario in which 40 pre-planned operations are replanned after an airspace constraint while an inflight operation must replan within 60 s. Two parameter sweeps are reported: the submission window of the 40 replans (75 to 30 s) and the USS2-to-DSS latency (0 to 1.6 s). The paper reports that SCD 95th-percentile duration approaches 30 s in the 40-ops/30-s-window case, which it interprets as consuming nearly half the inflight replan grace period, and that increasing USS2-DSS latency raises the acceptance rate of a USS1 operation, which it interprets as sensitivity of first-come-first-served airspace access to network latency. The conclusion recommends adding synchronization/prioritization to UTM.

Significance. If the quantitative results are credible, the paper identifies a plausible scalability bottleneck in UTM's OVN-based synchronization: race conditions during time-correlated replanning can inflate SCD duration and threaten inflight contingency timelines. The ANAMLL environment itself, with instrumented proxies and scripted executions, is a useful contribution to M&S for UTM. However, all central quantitative claims rest on an unvalidated implementation, with no comparison to a reference USS/DSS, no conformance evidence, no released code/data, and no error bars. The paper makes falsifiable predictions—the p95 SCD-vs-window curve and the acceptance-rate-vs-latency curve—so it is a starting point, but the results are not yet established as properties of the UTM architecture.

major comments (4)

[§III, Figs. 2 and 3] The central quantitative claims—'SCD durations approached 30 sec' and the acceptance-rate latency curve—are presented without any estimate of uncertainty. The text says results are 'aggregated across numerous executions' but does not report the number of replications, dispersion, confidence intervals, randomization scheme, or how the 95th percentile is pooled across runs. For a p95 metric, this is not a cosmetic omission: a percentile estimate from an unreported number of runs cannot support the paper's definition of a 'demand point.' Please report N, per-parameter-set CIs or box plots, and ideally the full distribution.
[§II-A and §II-C] The paper asserts ANAMLL components are 'compliant with the standards of [5]' but the cited [5] is SkyTrakx, not the ASTM F3548-21 standard cited as [3]. More importantly, no conformance test, reference implementation, or real-world data is used to validate timing behavior of the USS, DSS, or SCD. SCD is a simplified 4D conflict detection with no negotiation and a maximum of 5 restarts, and all components run as containers on a single host. These implementation choices can dominate the 30 s p95 result (scheduling, Python/async overhead, OVN handling, restart cap). As written, the 'demand point' is a property of ANAMLL, not shown to be a property of a standards-conformant UTM. Add a validation or benchmark experiment, or substantially soften the causal claims.
[§IV, Discussion] The interpretation that 'nearly half' of the 60 s grace period is consumed by SCD for the inflight operation is not supported by the metric reported. Fig. 2 plots the 95th percentile of SCD durations across all operations in the scenario, not the duration experienced by the non-conformant inflight operation. The 95th percentile of 40 replans need not correspond to that operation, and the figure says nothing about the sequential cost if the inflight operation must retry. Report the actual SCD duration for the inflight operation, or a conditional quantile under the scenario, and separate the protocol-driven race-condition component from single-host implementation overhead.
[§III, Fig. 3 and §IV] The fairness/equity result is presented as a demonstration of FCFS sensitivity to latency, but the experiment is not isolated: increasing USS2-DSS latency changes the relative arrival times at the DSS, which by construction favors USS1. The text should state explicitly that this is a deterministic consequence of the FCFS mechanism, not an emergent failure mode. The authors should also report whether the effect persists if the latency degradation is applied to both USSs or if submission times are jittered. As reported, the effect is unsurprising and lacks statistical support.

minor comments (4)

[Introduction and §II-A] Reference numbering is inconsistent: 'In [4] Zhao et al.' and 'In [4], Hsieh et al.' refer to two different works; the second should be [5]. Also, 'compliant with the standards of [5]' should likely cite [3] (ASTM F3548-21).
[§II-C] The definition of a '4D conflict' is missing: what spatial or temporal separation threshold is used? Without this, the acceptance/rejection logic is under-specified.
[§III, Figs. 2 and 3] Axis labeling: Fig. 2's x-axis decreases from 75 to 30 with no arrow indicating the direction of increasing stress; Fig. 3's y-axis label and legend could state more clearly that the two curves refer to the two different USSs' operations. Also define how the 95th percentile is computed across 'repeated simulations' (pooled per run? averaged per run?).
[General] Wording: 'USS’' should be 'USSs' or 'USSes'; also 'OPASQUE version numbers' should be 'opaque version numbers' (OVN), but the term is spelled correctly elsewhere.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the simulation results are measured outputs of an independent implementation, not fitted inputs or self-citation derivatives; one minor self-citation is not load-bearing.

full rationale

The paper's derivation chain is a simulation experiment: ANAMLL implements USS/FIMS/DSS components, a scenario injects 40 replans plus an inflight replan, and metrics (SCD p95 duration, acceptance rate) are collected from Execution Engine interactions. No equation in the paper reduces a result to an input; no parameter is fitted to make the 30s p95 appear; the SCD algorithm with max 5 restarts is stated as an implementation choice, not as a target. The second result (latency vs acceptance) is a measured consequence of the first-come-first-served DSS logic combined with OVN-based revalidation; it is not assumed in the metric definitions. The only self-citation is [9] (ACAS sXu, co-authored by I. Jessen), cited in the conclusion as a future DAA integration capability; it is not load-bearing for the UTM performance claims. There is a citation accuracy issue: Section II-A says components are 'compliant with the standards of [5]', but [5] is the SkyTrakx toolkit, not a standard; the ASTM F3548-21 standard is [3]. This is a missing/correct-reference flag, not circularity. The paper also explicitly states in Section V that 'future work... will focus on the validation of the simulation results,' acknowledging the lack of external validation. That is a correctness/validity risk, not a circular-derivation risk. Overall: no step in the claimed chain is equivalent to its input by construction, so circularity score is low.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

ANAMLL is a software environment, not a physical entity, and the paper introduces no new particles, forces, or conserved quantities. The central claims rest on assumptions about the representativeness of the simulation and scenario, not on fitted parameters. No numbers were tuned to match target outputs.

assumptions (4)

domain assumption ANAMLL components faithfully implement ASTM F3548-21 UTM behavior.
Section II-A claims compliance with the standard but provides no validation against real UTM deployments.
domain assumption The stress scenario is representative of future low-altitude airspace demand.
Section II-B describes it as hypothetical but realistic; the quantitative results depend on 40 operations, a 60-second grace period, and other scenario-specific choices.
ad hoc to paper A simple 4D conflict detection with no negotiation is sufficient to model SCD behavior.
Section II-C states that SCD is implemented as simple 4D conflict identification with no negotiation or rerouting; real USS implementations may use different algorithms that change race-condition frequency.
domain assumption The 60-second contingency grace period is an appropriate safety threshold.
Section II-B uses 60 seconds as the allowable window for an inflight operation to replan before declaring a contingency; this threshold is not justified beyond the scenario design.

how reviews work

0 comments

Cite this review

Pith. "Pith review of UTM Performance Under Stressing Scenarios." pith.science (2026). https://pith.science/paper/5JE7VVFL

@misc{pith2026250908124,
  author       = {Pith},
  title        = {Pith review of: UTM Performance Under Stressing Scenarios},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5JE7VVFL}},
  note         = {Machine review of arXiv:2509.08124}
}

read the original abstract

Proliferation of new classes of airspace participants, including uncrewed and advanced aerial mobility vehicles, necessitates the development and deployment of novel airspace management solutions, such as the Unmanned Traffic Management (UTM) system and the Provider of Services to UAM (PSU) Network. The efficacy of such systems has been demonstrated on multiple occasions via real-world deployments in limited test environments, however exploration of system behavior under stressing conditions requires the development of appropriate modeling and simulation (M&S) environments. Autonomy Networks for Advanced Mobility at Lincoln Laboratory (ANAMLL) is a virtual Systems Integration Laboratory (SIL) designed to host federated autonomy networks, such as a UTM or PSU Network, and to enable test and validation at scales not available in real-world deployments. As an example of ANAMLL's utility, we explore the performance of a representative UTM network during a stressing demand scenario. In a close examination of the demand scenario, ANAMLL demonstrates a UTM system demand point at which in-flight replanning can no longer be accomplished within an allowable time window. In a second analysis of the same scenario, ANAMLL demonstrates the impact of network connectivity performance on end-user airspace access.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

8 extracted references · 6 canonical work pages

[5]

SkyTrakx: A Toolkit for Simulation and Verification of Unmanned Air -Traffic Management Systems,

C. Hsieh, H. Sibai, H. Taylor, Y. Ni and S. Mitra, "SkyTrakx: A Toolkit for Simulation and Verification of Unmanned Air -Traffic Management Systems," 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), Indianapolis, IN, USA, 2021, pp. 372 -379, doi: 10.1109/ITSC48978.2021.9564492

arXiv 2021
[3]

Standard Specification for UAS Traffic Management (UTM) UAS Service Supplier (USS) Interoperability ,

ASTM Standard F3548-21, " Standard Specification for UAS Traffic Management (UTM) UAS Service Supplier (USS) Interoperability ,” ASTM International , West Conshohocken, PA, 2015, doi: 10.1520/F3548-21, www.astm.org

work page doi:10.1520/f3548-21 2015
[1]

UTM Concept of Operations Version 2.0 ,

“UTM Concept of Operations Version 2.0 ,” U.S. Department of Transportation, Federal Aviation Administration , Office of NextGen, March 2020. Available: https://www.faa.gov/sites/faa.gov/files/2022- 08/UTM_ConOps_v2.pdf

work page 2020
[2]

UAM Concept of Operations Version 2.0 ,

“UAM Concept of Operations Version 2.0 ,” U.S. Department of Transportation, Federal Aviation Administration , Office of NextGen, April 2023. Available: https://www.faa.gov/sites/faa.gov/files/Urban Air Mobility (UAM) Concept of Operations 2.0_0.pdf

work page 2023
[4]

A Simulation Framework For Fast Design Space Exploration Of Unmanned Air System Traffic Management Policies,

Z. Zhao et al., "A Simulation Framework For Fast Design Space Exploration Of Unmanned Air System Traffic Management Policies," 2019 Integrated Communicat ions, Navigation and Surveillance Conference (ICNS), Herndon, VA, USA, 2019, pp. 1 -10, doi: 10.1109/ICNSURV.2019.8735348

work page arXiv 2019
[6]

Development of an Unmanned T raffic Management Simulation with Robot Operating System and Gazebo,

Justin Nguyen, Peter K. Nguyen and Mujahid Abdulrahim. "Development of an Unmanned T raffic Management Simulation with Robot Operating System and Gazebo," AIAA 2022 -1918. AIAA SCITECH 2022 Forum. January 2022

work page 2022
[8]

Advanced Air Mobility Assessment Framework ,

L. E. Alvarez, T. A. Bonin, and J. C. Jones, “Advanced Air Mobility Assessment Framework ,” Massachusetts Institute of Technology, Lexington, MA, USA, Tech. Rep. AD1231040, Mar. 17, 2022. [Online]. Available: https://apps.dtic.mil/sti/html/trecms/AD1231040/index.html

work page 2022
[9]

ACAS sXu: Robust Decentralized Detect and Avoid for Small Unmanned Aircraft Systems,

L. E. Alvarez, I. Jessen, M. P. Owen, J. Silbermann and P. Wood, "ACAS sXu: Robust Decentralized Detect and Avoid for Small Unmanned Aircraft Systems," 2019 IEEE/AIAA 38th Digital Avionics Systems Conference (DASC), San Diego, CA, USA, 2019, pp. 1 -9, doi: 10.1109/DASC43569.2019.9081631

arXiv 2019

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.

[1] [5]

SkyTrakx: A Toolkit for Simulation and Verification of Unmanned Air -Traffic Management Systems,

C. Hsieh, H. Sibai, H. Taylor, Y. Ni and S. Mitra, "SkyTrakx: A Toolkit for Simulation and Verification of Unmanned Air -Traffic Management Systems," 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), Indianapolis, IN, USA, 2021, pp. 372 -379, doi: 10.1109/ITSC48978.2021.9564492

arXiv 2021

[2] [3]

Standard Specification for UAS Traffic Management (UTM) UAS Service Supplier (USS) Interoperability ,

ASTM Standard F3548-21, " Standard Specification for UAS Traffic Management (UTM) UAS Service Supplier (USS) Interoperability ,” ASTM International , West Conshohocken, PA, 2015, doi: 10.1520/F3548-21, www.astm.org

work page doi:10.1520/f3548-21 2015

[3] [1]

UTM Concept of Operations Version 2.0 ,

“UTM Concept of Operations Version 2.0 ,” U.S. Department of Transportation, Federal Aviation Administration , Office of NextGen, March 2020. Available: https://www.faa.gov/sites/faa.gov/files/2022- 08/UTM_ConOps_v2.pdf

work page 2020

[4] [2]

UAM Concept of Operations Version 2.0 ,

“UAM Concept of Operations Version 2.0 ,” U.S. Department of Transportation, Federal Aviation Administration , Office of NextGen, April 2023. Available: https://www.faa.gov/sites/faa.gov/files/Urban Air Mobility (UAM) Concept of Operations 2.0_0.pdf

work page 2023

[5] [4]

A Simulation Framework For Fast Design Space Exploration Of Unmanned Air System Traffic Management Policies,

Z. Zhao et al., "A Simulation Framework For Fast Design Space Exploration Of Unmanned Air System Traffic Management Policies," 2019 Integrated Communicat ions, Navigation and Surveillance Conference (ICNS), Herndon, VA, USA, 2019, pp. 1 -10, doi: 10.1109/ICNSURV.2019.8735348

work page arXiv 2019

[6] [6]

Development of an Unmanned T raffic Management Simulation with Robot Operating System and Gazebo,

Justin Nguyen, Peter K. Nguyen and Mujahid Abdulrahim. "Development of an Unmanned T raffic Management Simulation with Robot Operating System and Gazebo," AIAA 2022 -1918. AIAA SCITECH 2022 Forum. January 2022

work page 2022

[7] [8]

Advanced Air Mobility Assessment Framework ,

L. E. Alvarez, T. A. Bonin, and J. C. Jones, “Advanced Air Mobility Assessment Framework ,” Massachusetts Institute of Technology, Lexington, MA, USA, Tech. Rep. AD1231040, Mar. 17, 2022. [Online]. Available: https://apps.dtic.mil/sti/html/trecms/AD1231040/index.html

work page 2022

[8] [9]

ACAS sXu: Robust Decentralized Detect and Avoid for Small Unmanned Aircraft Systems,

L. E. Alvarez, I. Jessen, M. P. Owen, J. Silbermann and P. Wood, "ACAS sXu: Robust Decentralized Detect and Avoid for Small Unmanned Aircraft Systems," 2019 IEEE/AIAA 38th Digital Avionics Systems Conference (DASC), San Diego, CA, USA, 2019, pp. 1 -9, doi: 10.1109/DASC43569.2019.9081631

arXiv 2019