REVIEW 3 major objections 5 minor 2 cited by
AirTrafficGen: Configurable Air Traffic Scenario Generation with Large Language Models
T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Large language models, fed a graph encoding of an airspace sector, can generate high-traffic, operationally realistic air traffic control training scenarios, with the strongest models producing zero unwanted conflicts even with 30…
desk verdict A promising LLM-based ATC scenario generator whose headline 'operational realism' claim is undercut by a graph-metric bug that misses close passes one time-step apart. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the grid-based sector graph: fixes and route legs are projected onto nodes spaced at 20 nautical miles, matching the relevance threshold used to decide whether two aircraft are 'relevant traffic.' Aircraft move one node per time step (jets) or one node every two time steps (turboprops), and an interaction is counted when two aircraft occupy the same node at the same time or swap nodes in one time step. Because edges only cross at nodes, these two events capture every pass within the threshold. The prompt then instructs the model to analyze route intersections, place aircraft, and internally roll out trajectories to verify its own scenario, which makes every generated scenario automatically checkable.
What would settle it
Give qualified air traffic controllers a set of scenarios that score perfectly on the paper's metrics (zero mean unique interacting pairs) alongside handcrafted training scenarios, and ask them to rate operational realism and usability. If controller ratings do not favor the high-scoring LLM scenarios, or if controllers identify interactions that the 20 nautical mile proxy missed, the central claim of operational realism is falsified.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that state-of-the-art LLMs can solve the spatio-temporal planning problem of placing aircraft on a discretized route network so that no pair comes within the 20 nautical mile relevance threshold, at traffic volumes where random placement produces dozens of interacting pairs. The best models achieve a mean of zero unique interacting pairs across all tested volumes up to N=30, and also track requested interaction counts closely. With the vertical dimension added, the same models follow fine-grained instructions such as 'three aircraft in trail with a fourth climbing through their levels.' The authors argue this makes AirTrafficGen a viable automated alternative to handcrafted scenario design, restricted only by the fidelity of the discretized abstraction.
Load-bearing premise
Everything rests on the 20 nautical mile, two-speed discretization being a faithful-enough proxy for what real controllers treat as relevant traffic; the paper asserts this has limited impact on fidelity but provides no human or real-traffic validation, and Section 8 defers ATCO trials to future work.
Editorial extensions
If this is right
- If correct, controller training centers can generate large, diverse scenario sets from text specifications instead of hand-authoring each one.
- The automatic verification loop means scenario generation can be coupled to a search or prompt-optimization procedure that keeps only scenarios passing conflict-free checks.
- The approach transfers to any sector geometry that can be encoded as a graph, so validation studies for airspace changes could be run at lower cost.
- The observed iterative refinement behavior suggests a human-in-the-loop editing workflow where a controller points out a flaw and the LLM repairs the scenario.
Reading between the lines
- The same graph-encoding trick could be applied to other safety-critical planning tasks with a natural relevance length scale, such as maritime traffic or drone corridor deconfliction, replacing the 20 nautical mile threshold with the corresponding separation rule.
- A stronger test than the paper's internal metric would compare generated scenarios against recorded traffic or against human controller judgments; if the proxy metric and expert judgment diverge, the discretization would need refinement.
- The near-perfect scores of a few models at N=30 may partly reflect the synthetic sector's simplicity, with only seven routes and seven intersections, leaving performance on operationally dense real sectors untested.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. AirTrafficGen proposes to automate the design of air traffic control training scenarios by having large language models emit structured JSON traffic specifications. The paper encodes sector routes as a graph with nodes spaced 20 nmi apart, discretizes aircraft into fast and slow movers, and prompts LLMs to generate scenarios that are then rolled out in a discrete-time simulator. It benchmarks 15 models on four synthetic-sector tasks: non-interacting scenarios under increasing traffic volume, scenario length, sector complexity, and target interaction counts, using mean unique interacting pairs (MUIP) and mean absolute deviation from target interaction counts (MADIP) relative to a random baseline. The paper also presents illustrative examples of fine-grained control over interaction type, location, and vertical profiles, and one case of iterative refinement from textual feedback. The headline claim is that state-of-the-art models such as GPT-5, o3, and GPT-oss-120b generate high-traffic non-interacting scenarios while maintaining operational realism.
Significance. The empirical core is genuinely useful: benchmarks are automatically verified by trajectory rollout, a random baseline anchors the comparison, ten synthetic sectors are used per condition, and the code is reported to be in the supplement. A cost-versus-skill Pareto analysis adds practical value. If the results survive the continuous-time verification proposed below, the paper would be a solid contribution to LLM benchmarking for spatial-temporal planning and to ATC scenario generation. However, the significance is conditional: the abstract's 'operational realism' and 'fine-grained control' claims go beyond what the current graph-level metrics actually measure, and the claimed equivalence between graph interactions and the 20 nmi relevancy rule is not correct as stated.
major comments (3)
- [§4.2] The claim that graph interactions 'capture every instance where aircraft pass within the 20 nmi threshold' is false under the paper's own continuous-space definition. Nodes are 20 nmi apart and a fast aircraft moves one node per time step. Consider two fast aircraft on routes that cross at node X, with aircraft A at X at time t=1 and aircraft B at X at time t=2. No graph interaction is recorded (they neither share a node at the same time nor swap nodes), but at t=1.5 A is 10 nmi past X and B is 10 nmi before X along its incoming edge; for a perpendicular crossing their separation is sqrt(100+100) ≈ 14.1 nmi, and for acute crossings it is smaller. Hence zero MUIP does not imply that no pair came within 20 nmi, and the benchmark does not by itself establish the 'operational realism' asserted in the abstract and Section 8. The authors should either re-verify all benchmark outputs with a continuous-time roll-out using the Section 2 relevancy metric, or re-scope the claims to the graph-level interaction definition.
- [§7 and Table 5] The fine-grained controllability contribution is supported only by illustrative screenshots, not by quantitative evaluation. The controllability benchmark (Table 5, §6.1) measures only the number of unique interacting pairs (MADIP); it does not verify that interactions have the requested type (cross-path, head-on, catch-up) or location, and it is restricted to the lateral plane. Section 7's type/location/vertical examples are not scored or rolled out in a way that confirms they satisfy the prompt constraints. Since the abstract promises 'fine-grained control over interaction presence, type, and location,' the authors need a quantitative verification protocol (e.g., classify each generated pair by type and check location constraints) or should soften the claim.
- [§6.2, Tables 2–5] All benchmark tables report means (or point values) across ten synthetic sectors with no variance, confidence intervals, or per-sector breakdowns, despite temperature 1.0 sampling and only one generation per sector. Consequently, small differences among mid-tier models in Tables 2–5 are not interpretable, and the word 'reliably' for top models rests on an assumption that the ten-sector sample is representative. Reporting standard errors or at least the distribution of MUIP/MADIP across sectors would make the benchmarking claims more robust. This is not the main technical gap, but it matters for the paper's 'rigorous benchmarking' contribution.
minor comments (5)
- [Table 1] The entry 'gpt-4.1-2025-04-14OpenAI' is missing a space before 'OpenAI'.
- [Figure 5] The axis labels '(1) N. Aircraft', '(2) Scenario Length', '(3) Sector Complexity', and '(4) N. Interactions' are informal; consider writing 'Number of aircraft' and 'Number of interactions' for clarity.
- [Figure 10 caption] The word 'Pareto Fronter' should be 'Pareto Frontier'.
- [§7, Figure 13(c)] The notation 'GXXX' is used to denote ground speed but is not explained in the caption or text; please define it.
- [§2] The 20 nmi relevancy threshold is a stated modeling assumption, but the paper does not discuss sensitivity of the results to this choice; a brief note on how results might change for, say, 15 or 25 nmi would be helpful.
Circularity Check
No significant circularity: benchmark scores come from independent trajectory roll-outs, and the only self-citations are background references that do not carry the central claim.
full rationale
The paper's central empirical claim is that LLMs can generate scenarios that pass an automatically checkable non-interaction or interaction-count criterion. This is verified by rolling out the generated JSON schedules on the graph representation and counting interactions, which is an external computation independent of the model's own output; no parameter is fitted to the benchmark result and no predicted quantity is defined in terms of the LLM output. The graph encoding, the 20 nmi relevancy threshold, and the fast/slow speed abstraction are explicitly stated modeling choices rather than conclusions derived from data. Self-citations such as Hodgkin et al. 2025 (for typical speed classes), Pepper and Thomas 2024 (for epistemic uncertainty), and The Alan Turing Institute's BluebirdDT (for visualization) are background or implementation references; none is invoked as a uniqueness theorem or as proof that the central capability claim holds. The assertion in Section 4.2 that node-occupancy and swap events 'capture every instance where aircraft pass within the 20 nmi threshold' is a discretization faithfulness claim that may be questioned on correctness grounds, but it is not a circular reduction: the discrete interaction predicate is not defined in terms of the paper's conclusion about operational realism. Similarly, the absence of human ATCO validation, acknowledged in Section 8, is a validation gap rather than a circularity. Overall, the derivation chain is self-contained against external benchmarks, with at most a minor non-load-bearing self-citation, so the circularity score is low.
Assumptions & free parameters
free parameters (2)
- Relevancy threshold (20 nmi) =
20 nmi
- Aircraft speed discretization (fast=1 node/step, slow=1 node/2 steps) =
1 and 2 time steps per node
assumptions (4)
- standard math Edges in the graph do not cross without sharing a node, so same-node occupancy or node swaps capture every pair that passes within 20 nmi.
- domain assumption Relevant traffic can be adequately represented by the 20 nmi threshold and by pairwise decomposition of multi-aircraft interactions.
- domain assumption The two-speed aircraft model preserves the operational realism needed for training scenarios.
- domain assumption Synthetic sectors with 7 routes and 7 intersections are representative of real sector complexity.
Cite this review
Pith. "Pith review of AirTrafficGen: Configurable Air Traffic Scenario Generation with Large Language Models." pith.science (2026). https://pith.science/paper/D7E74PGF
@misc{pith2026250802269,
author = {Pith},
title = {Pith review of: AirTrafficGen: Configurable Air Traffic Scenario Generation with Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/D7E74PGF}},
note = {Machine review of arXiv:2508.02269}
}
abstract
The manual design of scenarios for Air Traffic Control (ATC) training is a demanding and time-consuming bottleneck that limits the diversity of simulations available to controllers. To address this, we introduce a novel, end-to-end approach, $\texttt{AirTrafficGen}$, that leverages large language models (LLMs) to automate and control the generation of complex ATC scenarios. Our method uses a purpose-built, graph-based representation to encode sector topology (including airspace geometry, routes, and fixes) into a format LLMs can process. Through rigorous benchmarking, we show that state-of-the-art models like Gemini 2.5 Pro, OpenAI o3, GPT-oss-120b and GPT-5 can generate high-traffic scenarios while maintaining operational realism. Our engineered prompting enables fine-grained control over interaction presence, type, and location. Initial findings suggest these models are also capable of iterative refinement, correcting flawed scenarios based on simple textual feedback. This approach provides a scalable alternative to manual scenario design, addressing the need for a greater volume and variety of ATC training and validation simulations. More broadly, this work showcases the potential of LLMs for complex planning in safety-critical domains.
Figures
Figures from the paper (10 more)
Forward citations
Cited by 2 Pith papers
-
Fine-Tuning Large Language Models for Cooperative Tactical Deconfliction of Small Unmanned Aerial Systems
Fine-tuning Qwen-Math-7B with LoRA and GRPO on BlueSky simulator data improves LLM accuracy and consistency in cooperative sUAS tactical deconfliction, reducing near mid-air collisions.
-
Edge Intelligence in Civil Aviation: Paradigms, Techniques, and Applications
A survey of edge-AI techniques and four cloud–edge–device deployment paradigms, mapped to civil aviation applications and future trends.
Reference graph
Works this paper leans on
-
[3]
Cheng Chang, Siqi Wang, Jiawei Zhang, Jingwei Ge, and Li Li
URL https: //arxiv.org/abs/2503.02911. Cheng Chang, Siqi Wang, Jiawei Zhang, Jingwei Ge, and Li Li. Llmscenario: Large language model driven scenario generation. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 54 (11):6581–6594,
-
[5]
Andreas Demetriou, Henrik Alfsvåg, Sadegh Rahrovani, and Morteza Haghir Chehreghani
URL https://arxiv.org/abs/2501.12948. Andreas Demetriou, Henrik Alfsvåg, Sadegh Rahrovani, and Morteza Haghir Chehreghani. A deep learning framework for generation and analysis of driving scenario trajectories. SN Computer Science, 4(3):251,
-
[8]
Wei Dong, Xianqing Chen, and Qiang Yang
doi: 10.1109/TITS.2023.3259322. Wei Dong, Xianqing Chen, and Qiang Yang. Data-driven scenario generation of renewable energy production based on controllable generative adversarial networks with interpretability. Applied Energy, 308:118387,
arXiv 2023
-
[10]
Lan Feng, Quanyi Li, Zhenghao Peng, Shuhan Tan, and Bolei Zhou
URL https://www.eurocontrol.int/publication/ guidelines-coordination-and-transfer-control-atc . Lan Feng, Quanyi Li, Zhenghao Peng, Shuhan Tan, and Bolei Zhou. Trafficgen: Learning to generate diverse and realistic traffic scenarios. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 3567–3575. IEEE,
work page 2023
-
[11]
ISSN 2227-9091. doi: 10.3390/risks10110199. URL https: //www.mdpi.com/2227-9091/10/11/199. Gemma Team. Gemma 3 technical report,
-
[12]
Qiujing Lu, Xuanhan Wang, Yiwei Jiang, Guangming Zhao, Mingyue Ma, and Shuo Feng
URL https://arxiv.org/abs/2504.02529. Qiujing Lu, Xuanhan Wang, Yiwei Jiang, Guangming Zhao, Mingyue Ma, and Shuo Feng. Mul- timodal large language model driven scenario testing for autonomous vehicles. arXiv preprint arXiv:2409.06450,
-
[15]
doi: 10.2514/1.I011359. Qwen Team. Qwen3 technical report,
-
[18]
doi: 10.1007/s13272-024-00772-4. 11 Set of Allowed Routes Through Sector (with directionality) A B C D E F G Figure 8: One of the synthetic sectors used in the benchmarking. The Alan Turing Institute. Project bluebird: An ai system for air traffic control,
Show all 21 references
-
[19]
URL https: //ojs.aaai.org/index.php/AAAI/article/view/32951
doi: 10.1609/aaai.v39i8.32951. URL https: //ojs.aaai.org/index.php/AAAI/article/view/32951. Jiawei Zhang, Chejian Xu, and Bo Li. Chatscene: Knowledge-enabled safety-critical scenario generation for autonomous vehicles. In Proceedings of the IEEE/CVF Conference on Computer Visi...
-
[20]
Values quoted are the average number of unique interacting pairs of aircraft computed across 10 generated scenarios on 10 synthetic sectors
– 2.58 gpt-4.1-2025-04-14OpenAI [2025a] – 8.0 o3 OpenAI [2025b] ✓ 8.0 gpt-5 OpenAI [2025b] ✓ 10.0 gemini-2.5-pro DeepMind [2025b] ✓ 12.50 Table 2: Benchmarking Results for varying number of aircraft. Values quoted are the average number of unique interacting pairs of aircraft ...
2025
-
[30]
We present two Gemini-2.5-Pro outputs on theN = 30traffic-volume benchmark (Figures 11 and 12). The model is successful in Figure 11 and fails in Figure 12, illustrating both the complexity of the task and, in the success case, the strategy employed by a reasoning-capable mode...
2025
-
[2002]
doi: 10.1109/DASC.2002.1067908. OpenAI. Gpt-4.1. Released April 14, 2025, 2025a. Available via OpenAI API: https://openai. com/index/gpt-4-1/. OpenAI. Openai o3 model, 2025b. https://openai.com. OpenAI. gpt-oss-120b and gpt-oss-20b Model Card. https://cdn.openai.com/pdf/ 419b6...
2002 arXiv
-
[2003]
doi: 10.1109/ DASC.2003.1245861. R.D. Oaks and M. Paglione. Generation of realistic air traffic scenarios using a genetic algorithm. In Proceedings. The 21st Digital Avionics Systems Conference, volume 1, pages 2A1–2A1,
2003 arXiv
-
[2004]
doi: 10.2514/6.2004-6399. T. Stefani, J. M. Christensen, A. A. Girija, et al. Automated scenario generation from operational design domain model for testing ai-based systems in aviation. CEAS Aeronautical Journal, 16: 197–212,
2004 doi
-
[2014]
EUROCONTROL
URL https://www.icao.int/MID/Documents/2014/ PBN%20Workshop-Tunis/14%20Validation%20and%20Implementation_vJUL13.pdf. EUROCONTROL. Guidelines for the coordination and transfer of con- trol in atc,
2014
-
[2018]
doi: https://doi
ISSN 0968-090X. doi: https://doi. org/10.1016/j.trc.2018.07.017. URL https://www.sciencedirect.com/science/article/ pii/S0968090X18302353. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training,
2018 doi
-
[2019]
doi: 10.18653/v1/N19-1423
Association for Computational Linguistics. doi: 10.18653/v1/N19-1423. Wenhao Ding, Chejian Xu, Mansur Arief, Haohong Lin, Bo Li, and Ding Zhao. A survey on safety-critical driving scenario generation—a methodological perspective. IEEE Transactions on Intelligent Transportation...
-
[2022]
doi: 10.3390/machines10111101
ISSN 2075-1702. doi: 10.3390/machines10111101. URL https://www.mdpi.com/2075-1702/10/11/1101. Xuan Cai, Xuesong Bai, Zhiyong Cui, Danmu Xie, Daocheng Fu, Haiyang Yu, and Yilong Ren. Text2scenario: Text-driven scenario generation for autonomous driving test,
-
[2023]
BERT: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of...
2019
-
[2024]
Google DeepMind
doi: 10.1109/TSMC.2024.3392930. Google DeepMind. Gemini 2.5 flash-lite. Released June 17, 2025; model card on Google Cloud, 2025a. Stable version publicly available. Google DeepMind. Gemini 2.5 pro. Released June 17, 2025; Technical report available on arXiv, 2025b. arXiv:2507...
2024
-
[2025]
org/abs/2507.19457
URL https://arxiv. org/abs/2507.19457. Theodoros N Arvanitis, Sean White, Stuart Harrison, Rupert Chaplin, and George Despotou. A method for machine learning generation of realistic synthetic datasets for validating healthcare applications. Health Informatics Journal, 28(2):14...
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.