REVIEW 4 major objections 5 minor 44 references
Training Environment for High Performance Reinforcement Learning
T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read Tunnel embeds nonlinear F-16 flight dynamics in an open-source Gymnasium environment, letting air-combat researchers and mission planners change tasks, sensors, and training methods in days rather than months.
desk verdict A useful idea undermined by a missing artifact: the paper advertises an open-source Gymnasium environment but ships no code, and the trade study is unrepeatable. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central machinery is the Tunnel environment itself: a Gymnasium-compatible wrapper that combines a 13-state nonlinear F-16 flight model (forces, moments, kinematics, position, plus thrust lag) with configurable sensor nodes arranged in body-axis elevation and azimuth. The sensors return distances to the nearest wall, giving the agent a LiDAR-like image; observations can be swapped among sensor returns and a 16-element aircraft-state vector, and rewards can be reshaped, as when the study switched from a centerline-distance penalty to arithmetic target rewards. This plug-and-play design is what carries the argument: it makes observation, action, and task changes cheap enough to iterate in days, and it standardizes the interface with the machine-learning community.
What would settle it
Run a policy trained in Tunnel on the same tunnel task inside a higher-fidelity F-16 simulation that includes sensor noise, actuator limits, and atmospheric effects; if the policy's success rate collapses, the claim that Tunnel's dynamics are operationally relevant would be falsified.
Extended reading notes
Core claim
The paper claims that high-performance air-combat autonomy can be productively explored in a training environment rather than a full simulation, and that Tunnel is such an environment. Tunnel couples a 13-state, six-degree-of-freedom F-16 model with Gymnasium's reset-and-step interface, a configurable sensor array that returns range-to-wall distances, and a default action space of stick, throttle, and rudder. The environment's walls stand in for the tactical constraints of a mission, and the author argues this abstraction is enough to test which observation spaces, algorithms, and training schemes deserve further investment. The demonstration trade study found that a simple behavioral-cloning agent could navigate the tunnel while reinforcement-learning agents could not reliably do so, and that a missionized variant with moving threat zones could be built and evaluated in under a week.
Load-bearing premise
The central assumption is that the simplified 13-state F-16 flight model used in Tunnel is realistic enough that conclusions about agent behavior transfer to real high-performance aircraft; the paper itself concedes the extended mission version is not high fidelity.
Editorial extensions
If this is right
- Researchers can benchmark machine-learning algorithms against a common high-performance-aircraft task without months of simulator-specific training.
- Mission planners can prototype sensor configurations, threat zones, and navigation tasks as primitives, then pass the resulting environment to a reinforcement-learning or imitation-learning pipeline.
- A week-long trade study becomes feasible: the paper shows a single developer can compare PPO with recurrent and MLP policies, imitation learning, and classical autopilot control in one week.
- The missionized tunnel demonstrates that GPS-denied, partially observed, adversarial scenarios can be represented well enough to begin exploring them before moving to higher-fidelity simulators.
Reading between the lines
- Editorial inference: the paper's strongest implied claim is about transfer from Tunnel to real flight, yet the reported experiments stop at the abstract tunnel task; Tunnel's real value is likely as a cheap first filter that identifies which algorithm-sensor combinations deserve expensive high-fidelity evaluation, not as a substitute for it.
- Editorial inference: if the simplicity claim holds, Tunnel could serve as a community benchmark for partial observability and adversarial air combat, letting different research groups compare policies on identical dynamics for the first time.
- Editorial inference: the imitation-learning result suggests a testable extension, collecting expert trajectories from the waypoint autopilot and varying observation noise to see how much sensor realism is needed before behavioral cloning breaks down.
- Editorial inference: because the environment supports on-off threat control, self-play between two Tunnel agents could be used to study how engagement zones and sensor visibility shape adversarial policies.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper introduces Tunnel, an OpenAI Gymnasium environment for reinforcement learning with F-16 nonlinear flight dynamics. The manuscript describes the environment's design features (Section V), reports a week-long trade study comparing PPO-trained RNN and MLP agents and behavioral cloning from a PID expert (Section VI), and presents a "missionized" extension with threat zones and sensor modifications. The abstract and Section V.A claim the environment is open source and written in fewer than 300 lines of code. The paper does not include a repository URL, code listing, or quantitative experimental results; it reports mostly categorical outcomes and anecdotal findings, and it acknowledges in Section IX that the missionized extension is "not high fidelity."
Significance. If the environment were publicly available and the demonstrations reproducible, Tunnel could be a useful community resource: it addresses a real gap between inaccessible proprietary simulators and simpler drone environments, and it is explicitly positioned as a training environment rather than a validated high-fidelity simulator. The author correctly distinguishes training environments from simulation (Section IV.E) and cites related open-source efforts. However, as submitted, the contribution cannot be assessed: there is no code to inspect, no quantitative evaluation of agent performance, and the operational-relevance claims are not supported by the evidence in the trade study.
major comments (4)
- [Abstract; Section V.A; Section VIII.A] The paper repeatedly claims that Tunnel is open source, but no repository URL, commit hash, or code listing is provided anywhere in the manuscript. Since the paper's contribution is a software artifact, this absence prevents the reader from instantiating the F-16 13-state dynamics, sensor geometry, or reward structure, and it makes the Section VI trade study unrepeatable. Even the dynamics are not described beyond a citation to [23]. This is a load-bearing omission, not a presentation issue.
- [Section VI.A; Table 1] The trade study is unreplicated and unquantified. No random seeds, episode counts, hyperparameters, training curves, or success-rate statistics are reported. The statement that "there was no significant difference seen between RNN and MLP agents" is made without a statistical test or error bars, and Table 1 uses only the categorical labels "Yes/Marginal/No." As a result, the paper's central demonstration—that Tunnel enables rapid, meaningful comparison of training methods—is not supported.
- [Section VI.A] The environment configuration and reward function were altered after observing agent failures: the reward was changed "based on the results of the Trackmania effort," and the final observation used the last four timesteps of sensor data with no internal state. Because the final configuration is selected post hoc and no held-out validation or separate test set is described, the reported results are at risk of overfitting to the observed failures and do not establish generalizable performance.
- [Appendix A; Section VI.B] The PID "expert" controller is not fully specified. The equations contain only proportional and derivative terms, despite the name PID, and omit the integral term. The manuscript does not define the sampling interval, the units or body-axis conventions for Nz and Ps, or how these commands map to the action space. Since the behavioral-cloning comparison in Section VI.B depends on this expert, the result cannot be reproduced or checked.
minor comments (5)
- [Section I; Appendix A] The nomenclature list omits Nz, Ps, and NX, all of which are used in the equations and observation descriptions; please define these symbols.
- [References] Several references have incomplete bibliographic data (e.g., [7], [9], [10], [21], [34], and [38] contain placeholders such as "????" or missing years/pages); these should be completed.
- [Section III] The paper states that Tunnel is "written using three files and less than 300 lines of code," but no code is available to verify this claim; please provide the repository or remove the specific count.
- [Section VI.C] The text first says the missionized navigation task "was reliably accomplished by use of an autopilot" and then reports instances of trespass; please clarify the success criterion and the observed failure rate.
- [Section IX] The conclusion that the missionized extension is "not high fidelity" should be reflected in the abstract's claim of "operationally relevant aircraft physics" to avoid overstating operational transfer.
Circularity Check
No circularity: the paper is an empirical software/trade-study report with no derivation chain that reduces to its own inputs.
full rationale
The paper presents a reinforcement learning training environment and an anecdotal trade study; it does not claim to derive predictions from first principles. The F-16 dynamics are taken from an external reference (Heidlauf et al. [23]), the PID baseline is fully specified in the Appendix, and success is defined as reaching the tunnel end rather than as matching any fitted output. Trade-study findings are explicitly labeled as demonstrations, not validated predictions: 'Following is a demonstration of the speed that users can expose the agent to new observations, actions, training methodologies and tasks. It is not meant to be complete or prescriptive.' The comparisons between RNN, MLP, behavioral cloning, and PID are empirical comparisons against explicitly stated baselines. No fitted parameter is renamed as a prediction, no uniqueness theorem is invoked, and the paper does not rely on self-citation for its load-bearing content. The main manuscript-level weakness is that the central 'open source' claim is unverifiable because no repository, URL, or commit hash is provided, and the trade-study results are not quantified with seeds or hyperparameters; however, that is a reproducibility and evidence-quality concern, not circular reasoning. Section IX's concession that the missionized extension is 'not high fidelity' further limits operational-transfer conclusions but does not indicate that any result is assumed into existence.
Assumptions & free parameters
free parameters (4)
- PID expert gains (NZ and Ps) =
NZ: -0.002, -0.2; Ps: -0.001, -0.1
- Tunnel reward schedule =
Arithmetic series 100, 200, ..., 38000
- Sensor configuration =
Final training: 3x3 nodes at -60, 0, 60 degrees; dense 3-degree spacing in later setup
- Tunnel geometry =
Width and height four times wingspan; length 1.5 nautical miles
assumptions (4)
- domain assumption The 13-state, 6-DOF F-16 nonlinear flight dynamics model of Heidlauf et al. [23] accurately represents high-performance aircraft behavior.
- domain assumption Rigid-body, flat-Earth assumptions for the flight dynamics.
- standard math A* search yields sensible paths for the missionized task.
- domain assumption The AFRL PID autopilot is a competent expert for imitation learning.
Cite this review
Pith. "Pith review of Training Environment for High Performance Reinforcement Learning." pith.science (2026). https://pith.science/paper/ZK4E7YRX
@misc{pith2026250501953,
author = {Pith},
title = {Pith review of: Training Environment for High Performance Reinforcement Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/ZK4E7YRX}},
note = {Machine review of arXiv:2505.01953}
}
read the original abstract
This paper presents Tunnel, a simple, open source, reinforcement learning training environment for high performance aircraft. It integrates the F16 3D nonlinear flight dynamics into OpenAI Gymnasium python package. The template includes primitives for boundaries, targets, adversaries and sensing capabilities that may vary depending on operational need. This offers mission planners a means to rapidly respond to evolving environments, sensor capabilities and adversaries for autonomous air combat aircraft. It offers researchers access to operationally relevant aircraft physics. Tunnel code base is accessible to anyone familiar with Gymnasium and/or those with basic python skills. This paper includes a demonstration of a week long trade study that investigated a variety of training methods, observation spaces, and threat presentations. This enables increased collaboration between researchers and mission planners which can translate to a national military advantage. As warfare becomes increasingly reliant upon automation, software agility will correlate with decision advantages. Airmen must have tools to adapt to adversaries in this context. It may take months for researchers to develop skills to customize observation, actions, tasks and training methodologies in air combat simulators. In Tunnel, this can be done in a matter of days.
Reference graph
Works this paper leans on
-
[23]
Verification Challenges in F-16 Ground Collision Avoidance and Other Automated Maneuvers,
Heidlauf, P., Collins, A., Bolender, M., and Bak, S., “Verification Challenges in F-16 Ground Collision Avoidance and Other Automated Maneuvers,”International Workshop on Applied Verification for Continuous and Hybrid Systems, 2018
work page 2018
-
[1]
TheNeedforCollaborative CombatAircraftforDisruptiveAirWarfare,
ColMarkA.Gunzinger,U.R.,USAF(Ret.)withMajGenLawrenceA.Stutzriem,andSweetman,B.,“TheNeedforCollaborative CombatAircraftforDisruptiveAirWarfare,” MitchellInstitute,2024. URL https://mitchellaerospacepower.org/the- need-for-collaborativecombat-aircraft-for-disruptive-air-warfare/
work page 2024
-
[2]
Air Force exercises two Collaborative Combat Aircraft option awards,
of Air Force Public Affairs, S., “Air Force exercises two Collaborative Combat Aircraft option awards,”Official Air Force Website,2024. URL https://www.af.mil/News/Article-Display/Article/3754980/air-force-exercises-two- collaborative-combat-aircraft-option-awards/
-
[3]
Hefron, L. C. R., “Air Combat Evolution,”DARPA, 2016. URL https://www.darpa.mil/program/air-combat- evolution
work page 2016
-
[5]
DARPA calling for AI ‘reinforcements’ to bolster US air combat capability,
Harper, J., “DARPA calling for AI ‘reinforcements’ to bolster US air combat capability,” Defense Scoop, 2022. URL https://defensescoop.com/2022/11/21/darpa-calling-for-ai-reinforcements-to-bolster-us-air- combat-capability/. 11
work page 2022
-
[6]
F-16s arrive to be modified for autonomous testing,
Brewer, C. L., “F-16s arrive to be modified for autonomous testing,”Official Air Force Website, 2016. URLhttps://www.af. mil/News/Article-Display/Article/3728795/f-16s-arrive-to-be-modified-for-autonomous-testing/
-
[7]
Airsim: High-fidelity visual and physical simulation for autonomous vehicles
Shah, S., Dey, D., Lovett, C., and Kapoor, A., “Airsim: High-fidelity visual and physical simulation for autonomous vehicles.” In Field and Service Robotics in 2017, ???? Accessed: 2024-05-15
work page 2017
-
[8]
Learning to fly—a gym environment with pybullet physics for reinforcement learning of multiagent,
Panerati, J., Zheng, H., Zhou, S., James Xu, A. P., and Schoellig, A. P., “Learning to fly—a gym environment with pybullet physics for reinforcement learning of multiagent,”In 2021 IEEE/RSJ International Conference on Intelligent Robots, 2021. URL https://github.com/utiasDSL/gym-pybullet-drones
work page 2021
Show all 44 references
-
[9]
DCS World,
“DCS World,”https://www.digitalcombatsimulator.com/en/, ???? Accessed: 2024-05-20
2024
-
[10]
Flight Simulator X,
“Flight Simulator X,” https://store.steampowered.com/app/314160/Microsoft_Flight_Simulator_X_Steam_ Edition/, ???? Accessed: 2024-05-15
2024
-
[11]
“JSBSim,” https://jsbsim.sourceforge.net/, ???? Accessed: 2024-05-26
2024
-
[12]
NHarfang3D Dog-Fight Sandbox: A Reinforcement Learning Research Platform for the Customized Control Tasks of Fighter Aircrafts,
Özbek, M. M., Yıldırım, S., Aksoy, M., Kernin, E., and Koyuncu, E., “NHarfang3D Dog-Fight Sandbox: A Reinforcement Learning Research Platform for the Customized Control Tasks of Fighter Aircrafts,”arXiv preprint arXiv:2210.07282, 2022. URL https://arxiv.org/abs/2210.07282
-
[13]
Predictive Sampling: Real-time Behaviour Synthesis with MuJoCo ,
Howell, T., Gileadi, N., Tunyasuvunakool, S., Zakka, K., Erez, T., and Tassa, Y., “Predictive Sampling: Real-time Behaviour Synthesis with MuJoCo ,”arXiv preprint arXiv: 2212.00541, 2022. URLhttps://arxiv.org/abs/2212.00541
2022 arXiv
-
[14]
The Arcade Learning Environment: An Evaluation Platform for General Agents,
Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M., “The Arcade Learning Environment: An Evaluation Platform for General Agents,”arXiv preprint arXiv: 1207.4708, 2012. URLhttps://arxiv.org/abs/1207.4708
2012 arXiv
-
[15]
Quantifying Generalization in Reinforcement Learning ,
Cobbe, K., Klimov, O., Hesse, C., Kim, T., and Schulman, J., “Quantifying Generalization in Reinforcement Learning ,”arXiv preprint arXiv: 1812.02341, 2018. URLhttps://arxiv.org/abs/1812.02341
2018 arXiv
-
[16]
Isaacs, R.,Games of Pursuit, RAND corportation, 1951
1951
-
[17]
Rule-based air combat simulation,
Burgin, G. H., “Rule-based air combat simulation,”Titan Systems Inc, 1988
1988
-
[18]
Air-combat strategy using approximate dynamic programming,
McGrew, J. S., How, J. P., Williams, B., and Roy, N., “Air-combat strategy using approximate dynamic programming,”Journal of guidance, control, and dynamics, 2010
2010
-
[19]
SecAF Kendall experiences VISTA of future flight test at Edwards AFB,
Hatch, G., and Kozaitis, M., “SecAF Kendall experiences VISTA of future flight test at Edwards AFB,”Official Air Force Website, 2024
2024
-
[20]
The New VISTA Simulation System Design and Implementation,
Haus, P. J., Konopka, B., and Faught, D., “The New VISTA Simulation System Design and Implementation,”AIAA SciTech, 2023
2023
-
[21]
Revealing the Dark Side of the F-16 - FLCS,
“Revealing the Dark Side of the F-16 - FLCS,”https://www.falcon-bms.com/articles/, ???? Falcon BMS
-
[22]
Fly by Wire Flight Control Systems,
Sutherland, M. J. P., “Fly by Wire Flight Control Systems,”DTIC, 1968. URL https://apps.dtic.mil/sti/pdfs/ AD0679158.pdf
1968
-
[24]
Brandt, S.,Introduction to Aeronautics, Third Edition (AIAA Education Series), AIAA Education Series, 2015
2015
-
[25]
W.,A Dictionary of Aviation, Osprey Publishing, 2016
Wragg, D. W.,A Dictionary of Aviation, Osprey Publishing, 2016
2016
-
[26]
PID/First-Order Control Design for a Bank of F-16 Longitudinal Dynamic Systems,
Lee, J. G., and Kim, Y. C., “PID/First-Order Control Design for a Bank of F-16 Longitudinal Dynamic Systems,”IEEE, 2020
2020
-
[27]
USAF Finishes Flight Tests of F-16 Autonomous Gun System ,
Wolfe, F., “USAF Finishes Flight Tests of F-16 Autonomous Gun System ,”Defense Daily, 2023. URL https://www. defensedaily.com/usaf-finishes-flight-tests-of-f-16-autonomous-gun-system/air-force/
2023
-
[28]
A survey of air combat behavior modeling using machine learning ,
Gorton, P. R., Strand, A., and Brathen, K., “A survey of air combat behavior modeling using machine learning ,”arXiv preprint arXiv:2404.13954, 2024. URLhttps://arxiv.org/abs/2404.13954
2024
-
[29]
Imitation learning for modelling air combat behaviour,
Gorton, P., Asprusten, M., and Bråthen, K., “Imitation learning for modelling air combat behaviour,”Norwegian Defense Research Establishment, 2023. 12
2023
-
[30]
Robust flight navigation out of distribution with liquid neural networks,
Chahine†, M., Hasani†*, R., Kao†, P., Ray†, A., Shubert, R., Lechner, M., Amini, A., and Rus, D., “Robust flight navigation out of distribution with liquid neural networks,”Science Robotics, 2023
2023
-
[31]
Hierarchical Reinforcement Learning for Air-to-Air Combat,
Pope, A. P., Ide, J. S., Micovic, D., Diaz, H., Rosenbluth, D., Ritholtz, L., Twedt, J. C., Walker, T. T., Alcedo, K., and Javorsek, D., “Hierarchical Reinforcement Learning for Air-to-Air Combat,”arXiv preprint arXiv:2105.00990, 2021. URL https://arxiv.org/abs/2105.00990
2021 arXiv
-
[32]
Viper Shield All Digital Electronic Warfare Suite,
“Viper Shield All Digital Electronic Warfare Suite,”https://www.l3harris.com/all-capabilities/viper-shield- alq-254v1-all-digital-electronic-warfare-suite , ???? Accessed: 2024-05-23
2024
-
[33]
Resolving Range and Velocity Ambiguity Effectively and Efficiently with GPU,
Wang, Z., “Resolving Range and Velocity Ambiguity Effectively and Efficiently with GPU,”IEEE, 2021. URL https: //ieeexplore.ieee.org/document/10028451
2021
-
[34]
The Development and Use of an Inertial Navigation System as a DP Position Reference Sensor (IPRS),
Vickery, K., “The Development and Use of an Inertial Navigation System as a DP Position Reference Sensor (IPRS),” Dynamic Positioning Committee, ???? Accessed: 2024-05-30
2024
-
[35]
Great Power Competition: Implications for Defense—Issues for Congress,
O’Rourke, R., “Great Power Competition: Implications for Defense—Issues for Congress,”Congressional Research Service, 2024
2024
-
[36]
Nocturne: a scalable driving benchmark for bringing multi-agent learning one step closer to the real world,
Vinitsky, Eugene, Lichtlé, Nathan, Yang, Xiaomeng, Amos, Brandon, Foerster, and Jakob, “Nocturne: a scalable driving benchmark for bringing multi-agent learning one step closer to the real world,”arXiv preprint arXiv:2206.09889, 2022. URL https://arxiv.org/abs/2206.09889
2022 arXiv
-
[37]
Pentagon secures 500M for first tranche of Replicator systems,
Harper, J., “Pentagon secures 500M for first tranche of Replicator systems,”Defense Scoop, 2024
2024
-
[38]
Viper Shield All Digital Electronic Warfare Suite,
“Viper Shield All Digital Electronic Warfare Suite,”https://www.af.mil/Portals/1/documents/2024SAF/GPC/The_ Case_for_Change, ???? Accessed: 2024-05-23
2024
-
[39]
R.,Introduction to Aircraft Flight Mechanics: Performance, Static Stability, Dynamic Stability, Classic, AIAA Education Series, 2016
Yechout, T. R.,Introduction to Aircraft Flight Mechanics: Performance, Static Stability, Dynamic Stability, Classic, AIAA Education Series, 2016
2016
-
[41]
Automated Learning: An Implementation of The A* Search Algorithm over The Random Base Functions,
Tatari, N., “Automated Learning: An Implementation of The A* Search Algorithm over The Random Base Functions,”arXiv preprint arXiv:2211.05085, 2022. URLhttps://arxiv.org/pdf/2211.05085
2022 arXiv
-
[42]
Gigastep - One Billion Steps per Second Multi-agent Reinforcement Learning,
Lechner, M., “Gigastep - One Billion Steps per Second Multi-agent Reinforcement Learning,”OpenReview, 2023. URL https://openreview.net/pdf?id=UgPAaEugH3
2023
-
[43]
LOYAL WINGMAN, FLOCKING, AND SWARMING: NEW MODELS OF DIS- TRIBUTED AIRPOWER,
Wassmuth, D., and BlairANIEL, D., “LOYAL WINGMAN, FLOCKING, AND SWARMING: NEW MODELS OF DIS- TRIBUTED AIRPOWER,”War on the Rocks, 2018. URL https://warontherocks.com/2018/02/loyal-wingman- flocking-swarming-new-models-distributed-airpower/
2018
-
[44]
DARPA Tiles Together a Vision of Mosaic Warfare,
“DARPA Tiles Together a Vision of Mosaic Warfare,” https://www.darpa.mil/work-with-us/darpa-tiles- together-a-vision-of-mosiac-warfare , ???? Accessed: 2023-03-15. 13
2023
-
[2016]
URL https://arxiv.org/abs/2006.04439
2006 arXiv
-
[2024]
URL https://arxiv.org/html/2401.09286v1
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.