Explainable Reinforcement Learning for assisting Air Traffic Controllers

Anduel Mehmeti; Gabriella Gigante; Salvatore Venticinque

REVIEW 3 major objections 5 minor 12 references

Explainable Reinforcement Learning for assisting Air Traffic Controllers

T0 review · 3 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A DQN agent trained to navigate a simulated aircraft around a no-fly zone produces saliency maps that expose its shifting priorities, making its decisions legible to air traffic controllers.

desk verdict A clearly written but thin feasibility study: the DQN likely avoids a fixed no-fly zone in a toy grid, but the saliency phases are not supported because the NFZ inputs are constant features and the evidence is a single episode. read the letter →

arxiv 2607.22525 v1 pith:H7PVNBKP submitted 2026-07-24 cs.AI

Anduel Mehmeti , Gabriella Gigante , Salvatore Venticinque This is my paper · ORCID

classification cs.AI

keywords explainablereinforcementlearningairtrafficcontrolsaliencymapsDeepQ-Networkno-flyzoneavoidancedecisionsupportsystemfeatureattributiontrustinautomation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper aims to show that a Deep Q-Network agent, trained in a simplified 40x40 nautical-mile airspace with a circular no-fly zone, can learn a safe and efficient route from a start point to a target, and that the agent's decisions can be made transparent to human air traffic controllers. The authors introduce saliency maps — the normalized gradient of the maximum Q-value with respect to the input state — as an explainability layer, and argue that these maps reveal distinct phases of decision-making: an early balanced focus on the no-fly zone and target, a mid-episode peak of safety awareness near the restricted zone, and a final goal-dominated approach. The contribution is a first testbed showing how a value-based RL agent in a safety-critical navigation task can be paired with an interpretability tool that reflects shifting priorities. A sympathetic reader would care because explainability is a precondition for trusting automation in air traffic control and for meeting emerging regulatory expectations.

What carries the argument

The key machinery is the Deep Q-Network with experience replay and a target network, combined with an explainability layer defined by the saliency map Saliency = ∂Q_max/∂s, the gradient of the highest action-value with respect to the state input, normalized to produce per-feature importance charts. This gradient is the mechanism that connects the network's internal sensitivity to the six input features, allowing the authors to infer which parts of the state the agent is attending to at each step. The DQN's separate Q-outputs for the three heading adjustments carry the policy, while the saliency map is the interpretive instrument that turns the black box into a transparent sequence of priorit

What would settle it

Run the trained agent across many episodes and seeds, recording success rate and trajectory variance; if the shown curved path is a rare success, the claim of capability collapses. Additionally, perform a causal check: hold all state features fixed except the no-fly zone coordinates, shift them slightly, and measure whether the selected action changes in the direction the saliency map implies; if action changes do not track saliency, the gradient-based explanations are not faithful.

Watch

Extended reading notes

Core claim

The central claim is that a DQN agent with a six-dimensional state representation (aircraft coordinates, distance to target, heading, no-fly zone center and radius) and three heading adjustments (−5°, 0, +5°) learns to avoid the no-fly zone and reach the target, and that its decision logic is readable through saliency maps. On the authors' terms, the saliency analysis shows that as the episode progresses the relative gradient magnitudes shift: safety-related features (the no-fly zone coordinates) dominate near the restricted area, then the target-distance feature takes over, with lateral X-coordinate adjustments supporting the final alignment. The paper presents this as evidence that the age

Load-bearing premise

The explainability conclusion rests on the assumption that the normalized gradient of the maximum Q-value with respect to the input faithfully reflects which features actually influence the agent's decisions; if gradient magnitude is not a reliable attribution for this network, the saliency phases do not support the claimed interpretation.

Editorial extensions

If this is right

If the agent's behavior generalizes, gradient-based saliency can serve as a cheap, online explanation layer for en-route conflict-avoidance decision support, giving controllers a visual reason for a recommended heading change.
The observed phase structure (safety-first, then goal-focused) offers a template for verifying that an RL agent's priorities are aligned with safety requirements before deployment.
The same DQN-plus-saliency recipe can be extended to richer state spaces, e.g., with other aircraft or weather cells, to flag when the agent is attending to the 'wrong' feature.
For regulators, the approach demonstrates a concrete way to produce ops-level explainability of the kind EASA distinguishes, even if validation in higher-fidelity environments remains open.
If saliency tracks true causal influence, it can be used to detect adversarial inputs that shift attention away from safety-critical features.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper leaves implicit that gradient magnitude is not necessarily causal: a feature can have large gradient without the network's output actually depending on it in a counterfactual sense. A natural next step is to validate saliency by ablation or by intervention (e.g., zeroing the NFZ coordinate and checking that the chosen action changes).
The saliency phases are read from what appears to be a single episode. I infer that the authors intend these as characteristic tendencies, but the episode needs to be shown as representative across seeds and episodes for the phase narrative to be robust.
Because the action space is only heading adjustments, the state includes the NFZ center as a static feature; in a dynamic environment with moving zones, saliency over time could reveal whether the agent tracks the zone or relies on position, a distinction the current static setup cannot expose.
The move reward (closer/farther) is coarse, so the agent's preference for smooth curved paths may emerge from the boundary/target rewards; a saliency analysis over reward components (reward decomposition) would test whether the 'minimal deviation' claim is driven by shaping or by goal distance.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a small, honest pilot study, not a result to lean on. A DQN is trained in a 2D 40x40 grid with one static circular no-fly zone and three heading actions; the paper then shows saliency maps from dQmax/ds and reads temporal phases into them. That much is clearly described, and the authors are upfront that it is ongoing work. The one displayed trajectory does avoid the zone and reach the target, so the basic existence claim is plausible. The related work is relevant, and the EU AI Act/EASA framing is appropriate, even if standard.

The soft spots are real. First, the evidence is one episode. No success rate across episodes, no seed variance, no learning curves, no baseline comparison. Statements like the results confirm complex navigation strategies are far stronger than what a single trajectory can support. Second, and more specific, the stress-test note lands: the NFZ is encoded by its fixed center (20,20) and radius r. Those inputs are constant at every state. A constant feature cannot carry the state-dependent proximity information that would be needed for the claimed NFZ coordinates dominate near the zone phase. The gradient can still be nonzero and even vary with ReLU linear regions, so the saliency peaks may be an artifact. The authors never check by perturbing or ablating those inputs, so the interpretability conclusions in Figures 2-4 are unsupported. Third, Table 1 has a copy-paste error: the No-Fly Zone Penalty row says the aircraft touches the grid boundaries, duplicating the Boundary Penalty description. Minor, but sloppy. Also the intro mentions speed adjustments, but the action space only includes heading changes.

What is genuinely useful: the paper is a clean, minimal example of applying DQN plus gradient saliency in an ATC-like setting. It could be a starting point for a more rigorous study, and it is a fair example of how not to validate saliency-based explanations. The authors are not hiding anything; they call it preliminary and list planned extensions.

Who would benefit: someone new to XAI-for-RL wanting a compact illustration, or a researcher looking for a simple environment to demonstrate attribution-failure modes. I would not cite it as evidence for safety or explainability claims. If it came to me as a new submission, I would send it out for review rather than desk reject - it is coherent and the gaps are addressable - but I would expect referees to ask for multi-seed results, success-rate metrics, and a saliency perturbation/ablation check before any of the explanatory conclusions could stand.

Referee Report

3 major / 5 minor

Summary. The paper presents a simplified 40×40 grid air-traffic-control environment in which a DQN agent navigates a single aircraft from (20,39) to (20,0) while avoiding a circular no-fly zone centered at (20,20) with radius 5. The action space consists of heading-angle adjustments of −5, 0, +5 degrees. The authors train a DQN with an input state of six features (aircraft x,y; distance to target; heading angle; NFZ center coordinates) and then compute saliency maps as the gradient of the maximum Q-value with respect to the input state to interpret which features drive decisions. The main claims are that the agent reliably reaches the target, avoids the no-fly zone, and that the saliency maps reveal three temporal phases—early balanced safety/goal prioritization, a middle NFZ-avoidance phase, and a final goal-dominance phase—thus demonstrating dynamic prioritization of safety and efficiency.

Significance. If the explainability claim were properly supported, the paper would be a useful preliminary demonstration of gradient-based saliency for RL-based decision support in a safety-critical domain, and it would align with current regulatory interest in XAI for aviation. The chosen setting is transparent and easy to reproduce, and the idea of using saliency maps as a first explanation layer for DQN route selection is sensible. The authors also clearly frame the work as preliminary, which is appropriate for a short conference contribution. However, the paper does not supply quantitative performance evidence (no success rates, no seeds, no learning curves) and, more importantly, the saliency interpretation rests on attributing decision-relevant information to features that are constant across the whole task. These gaps currently prevent the stated conclusions from being supported.

major comments (3)

[§3, §3.2, §4 (Eq. Saliency=∂Qmax/∂s; Figs. 2–4)] The NFZ is represented in the state by Φ(x_c,y_c,r) with fixed center (20,20) and radius 5. Those coordinates are constant in every state of every episode, so they cannot encode the aircraft’s current proximity to the NFZ or justify the claimed phase-dependent safety prioritization. A gradient with respect to a constant input can be nonzero and can vary across ReLU linear regions, producing saliency peaks that look like dynamic attention without carrying situational information. To support the 'NFZ coordinates dominate near the zone' reading, the authors must add a validation: for example, perturb the NFZ center in a trained episode while keeping the trajectory fixed and show that saliency changes accordingly, or retrain with state-dependent NFZ features (e.g., distance to the NFZ boundary) and compare. Without such an intervention or ablation, the core explainability conclusion does not
[§4, Figure 1, bullet list] The trajectory evaluation is anecdotal. The text claims 'Safety Compliance,' 'Goal-Oriented Navigation,' and 'Precision and Stability,' but only a single episode is shown, with no success rate, no variance across episodes or random seeds, no learning curves, and no baseline comparison (e.g., a direct heading-to-target policy or a DQN without reward shaping). The reader cannot assess whether Figure 1 is representative of the learned policy or an outlier. Add aggregate metrics such as percentage of episodes reaching the target without NFZ violation, mean and standard deviation of path length or cumulative reward, and at least a few independent training runs.
[§3.1, §3] The environment dynamics and training details are underspecified, undermining reproducibility. The paper does not state how the aircraft position is updated from a heading action (equations of motion, step length in nautical miles), how many steps correspond to the displayed trajectory, how ε is decayed over time, the discount factor γ, the replay buffer size, the target-network update frequency, or the number of training episodes. These are standard DQN hyperparameters and are needed not only to reproduce the experiment but also to judge whether the reported single-episode behavior is stable. Please provide a complete training configuration and the state-transition equations.

minor comments (5)

[Table 1] The reward table contains a copy-paste error: the No-Fly Zone Penalty (−200) is described as 'the aircraft touches the grid boundaries,' which is the same description as the Boundary Penalty. The intended description should be 'the aircraft enters the no-fly zone.' This matters because the reward design is central to the agent’s learned behavior.
[§3.2] The equation 'Saliency=∂Qmax/∂s' should specify whether the gradient is taken with respect to a single state or with respect to all six input features, and how normalization is performed (per-step across features, or per-feature across the episode). Also clarify what 'gradient magnitudes' are normalized to.
[§4, Figures 2–4] The temporal phases are presented as 'steps 0–140,' 'steps 150–200,' and 'steps 200–315,' but the trajectory in Figure 1 is not aligned to a step axis, and the authors do not report the total episode length or the distance to the NFZ at the phase boundaries. Adding a step axis to the trajectory figure and reporting the aircraft's minimum distance to the NFZ in each phase would make the phase interpretation more testable.
[§1, §2, §4] Several language and typographical issues: 'Very Recent researches have emphasized...' should be 'Very recent research has emphasized...'; 'a reward structure incentives movement' should be 'a reward structure incentivizes movement'; 'were the problem of interest' should be 'where the problem of interest'; and 'The final publication is available at Springer via:' in the header is an editorial artifact that should be removed before archiving.
[References] Some references are incomplete or inconsistent (e.g., [2], [3], [6] lack full bibliographic details; [12] lacks the journal name). Please ensure all entries are complete and correctly formatted.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: descriptive train-and-saliency study; claims are empirical observations rather than derivations.

full rationale

The paper's derivation chain is: define environment/reward, train a DQN, evaluate one trajectory, compute saliency gradients, and interpret temporal phases. None of these steps takes a fitted quantity and renames it as a prediction. The trajectory result is an empirical observation of the trained policy, and the saliency maps are deterministic functions of the trained Q-network; they are not used to claim a separate quantity that was not measured. The only self-citation ([9], Nebula et al.) supports the general relevance of DQN in ATM and is not load-bearing for the trajectory or saliency claims. The interpretability conclusions rest on the unvalidated assumption that gradient magnitude indicates feature importance, and the single-episode evaluation limits generalizability, but those are methodological limitations, not constructional circularity. No equation is equivalent to its input by construction, no fitted parameter is relabeled as a prediction, and no load-bearing result is imported from self-citation. Hence score 0.

Assumptions & free parameters 6 free parameters · 3 assumptions · 0 invented entities

The central claims rest on hand-chosen reward weights, unspecified DQN hyperparameters, and the unvalidated assumption that gradient saliency faithfully explains the agent's behavior. There are no new postulated entities or forces; the paper is an empirical demonstration rather than a derivation.

free parameters (6)

Reward coefficients = Move penalty -2; NFZ penalty -200; boundary penalty -50; target rewards 5–200 with ranges depending on exit x-coordinate
Hand-chosen to shape navigation behavior; the reported successful trajectory depends on these specific weights, and no sensitivity analysis is provided.
Epsilon-greedy decay schedule = not reported
Exploration schedule is a free choice affecting the learned policy; omitted from the paper.
Discount factor γ = not reported
Required for DQN; not specified in Section 3.1.
Neural network hyperparameters = Two hidden layers of 128 neurons, ReLU; learning rate, batch size, and optimizer not reported
Capacity and training configuration chosen without stated justification or ablation.
Episode maximum steps = 700
Arbitrary horizon that defines whether an episode is a success; no rationale given.
Grid and NFZ geometry = 40×40 grid, NFZ center (20,20), radius 5
Simplified scenario parameters; results may not transfer to other geometries.

assumptions (3)

domain assumption DQN converges to a good policy in the toy grid under the specified reward function
Training details are incomplete, and the success of a single episode assumes convergence; no learning curves or convergence metrics are shown (§3.1, §4).
domain assumption Gradient of the max Q-value with respect to the input is a valid explanation of feature importance
Section 3.2 defines saliency as ∂Qmax/∂s and uses normalized magnitudes to infer agent priorities, but this premise is never validated against ablations or human judgments.
domain assumption A 2D grid with discrete heading actions is a relevant abstraction of ATC en-route navigation
The paper explicitly calls this a simplified testbed (§1, §3), but the qualitative conclusions about explainability for ATC rely on this abstraction being informative.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Explainable Reinforcement Learning for assisting Air Traffic Controllers." pith.science (2026). https://pith.science/paper/H7PVNBKP

@misc{pith2026260722525,
  author       = {Pith},
  title        = {Pith review of: Explainable Reinforcement Learning for assisting Air Traffic Controllers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H7PVNBKP}},
  note         = {Machine review of arXiv:2607.22525}
}

read the original abstract

To effectively integrate AI into high-stakes, critical environments such as healthcare, autonomous driving, and aviation--and to advance toward higher levels of automation and seamless human-AI collaboration--building trust in AI-driven solutions is essential. Trust, in turn, is closely linked to the explainability of AI systems. The rapid advancements in AI across various domains have underscored the challenges of establishing trust, raising increasing interest in AI explainability even more when applied to deep learning. In this context, the present work aims to explore the application of explainability techniques to Reinforcement Learning (RL) algorithms, specifically within the safety-critical domain of Air Traffic Control (ATC). Using a simplified ATC environment as an initial testbed, an intelligent agent is trained with a reinforcement learning algorithm to make decisions on alternative flight routes that avoid no-fly zones. As a preliminary explainability approach, a saliency map is employed, providing insights into the input features that most significantly influence the agent's decision-making process.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 1 canonical work pages

[1]

Applied Sciences12(3), 1295 (2022)

Degas, A., Islam, M.R., Hurter, C., Barua, S., Rahman, H., Poudel, M., Ruscio, D., Ahmed, M.U., Begum, S., Rahman, M.A., et al.: A survey on artificial intelligence (ai) and explainable ai in air traffic management: Current trends and development with future research trajectory. Applied Sciences12(3), 1295 (2022)

2022
[2]

arXiv preprint arXiv:2011.07553 (2020)

Ding, Z., Hernandez-Leal, P., Ding, G.W., Li, C., Huang, R.: Cdt: Cascading decision trees for explainable reinforcement learning. arXiv preprint arXiv:2011.07553 (2020)

arXiv 2011
[3]

In: Proceedings of the Thirty-First International Conference on Automated Planning and Scheduling (ICAPS 2021), pp

Ghosh, S., Laguna, S., Lim, S.H., Wynter, L., Poonawala, H.: A deep ensemble method for multi-agent reinforcement learning: A case study on air traffic control. In: Proceedings of the Thirty-First International Conference on Automated Planning and Scheduling (ICAPS 2021), pp. 468–476 (2021)

2021
[4]

In: Proceedings of the International Conference on Machine Learning (ICML 2018), pp

Greydanus, S., Koul, A., Dodge, J., Fern, A.: Visualizing and understanding atari agents. In: Proceedings of the International Conference on Machine Learning (ICML 2018), pp. 1792– 1801 (2018)

2018
[5]

EASA Regulations (2023)

Guillaume Soudain, F.T.: Easa concept paper: guidance for level 1 & 2 machine learning applications issue 02. EASA Regulations (2023)

2023
[6]

In: 2021 IEEE International Intelligent Transportation Sys- tems Conference (ITSC), pp

Guo, W., Brittain, M., Wei, P.: Safety enhancement for deep reinforcement learning in au- tonomous separation assurance. In: 2021 IEEE International Intelligent Transportation Sys- tems Conference (ITSC), pp. 348–354 (2021)

2021
[7]

In: 2022 IEEE/AIAA 41st Digital Avionics Systems Conference (DASC), pp

Guo, W., Wei, P.: Explainable deep reinforcement learning for aircraft separation assurance. In: 2022 IEEE/AIAA 41st Digital Avionics Systems Conference (DASC), pp. 1–10 (2022)

2022
[8]

Knowledge-Based Systems214, 106,685 (2021)

Heuillet, A., Couthouis, F., Díaz-Rodríguez, N.: Explainability in deep rein- forcement learning. Knowledge-Based Systems214, 106,685 (2021). DOI https://doi.org/10.1016/j.knosys.2020.106685

arXiv 2021

Show all 12 references

[9]

Information14(4), 216 (2023)

Nebula, F., Palumbo, R., Gigante, G., V ozella, A.: Digital assistant for arrival scheduling with conflict prevention capabilities. Information14(4), 216 (2023)

2023
[10]

Intelligent Systems in Accounting, Finance and Management25(2), 63–72 (2018)

Preece, A.: Asking ‘why’ in ai: Explainability of intelligent systems – perspectives and chal- lenges. Intelligent Systems in Accounting, Finance and Management25(2), 63–72 (2018). DOI https://doi.org/10.1002/isaf.1422

2018 doi
[11]

Knowledge-Based Systems263, 110,273 (2023)

Saeed, W., Omlin, C.: Explainable ai (xai): A systematic meta-survey of current challenges and future opportunities. Knowledge-Based Systems263, 110,273 (2023)

2023
[12]

Expert Systems with Applications260, 125,389 (2025)

Wang, L., Yang, H., Lin, Y ., Yin, S., Wu, Y .: Enhancing air traffic control: A transparent deep reinforcement learning framework for autonomous conflict resolution. Expert Systems with Applications260, 125,389 (2025)

2025

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.

[1] [1]

Applied Sciences12(3), 1295 (2022)

Degas, A., Islam, M.R., Hurter, C., Barua, S., Rahman, H., Poudel, M., Ruscio, D., Ahmed, M.U., Begum, S., Rahman, M.A., et al.: A survey on artificial intelligence (ai) and explainable ai in air traffic management: Current trends and development with future research trajectory. Applied Sciences12(3), 1295 (2022)

2022

[2] [2]

arXiv preprint arXiv:2011.07553 (2020)

Ding, Z., Hernandez-Leal, P., Ding, G.W., Li, C., Huang, R.: Cdt: Cascading decision trees for explainable reinforcement learning. arXiv preprint arXiv:2011.07553 (2020)

arXiv 2011

[3] [3]

In: Proceedings of the Thirty-First International Conference on Automated Planning and Scheduling (ICAPS 2021), pp

Ghosh, S., Laguna, S., Lim, S.H., Wynter, L., Poonawala, H.: A deep ensemble method for multi-agent reinforcement learning: A case study on air traffic control. In: Proceedings of the Thirty-First International Conference on Automated Planning and Scheduling (ICAPS 2021), pp. 468–476 (2021)

2021

[4] [4]

In: Proceedings of the International Conference on Machine Learning (ICML 2018), pp

Greydanus, S., Koul, A., Dodge, J., Fern, A.: Visualizing and understanding atari agents. In: Proceedings of the International Conference on Machine Learning (ICML 2018), pp. 1792– 1801 (2018)

2018

[5] [5]

EASA Regulations (2023)

Guillaume Soudain, F.T.: Easa concept paper: guidance for level 1 & 2 machine learning applications issue 02. EASA Regulations (2023)

2023

[6] [6]

In: 2021 IEEE International Intelligent Transportation Sys- tems Conference (ITSC), pp

Guo, W., Brittain, M., Wei, P.: Safety enhancement for deep reinforcement learning in au- tonomous separation assurance. In: 2021 IEEE International Intelligent Transportation Sys- tems Conference (ITSC), pp. 348–354 (2021)

2021

[7] [7]

In: 2022 IEEE/AIAA 41st Digital Avionics Systems Conference (DASC), pp

Guo, W., Wei, P.: Explainable deep reinforcement learning for aircraft separation assurance. In: 2022 IEEE/AIAA 41st Digital Avionics Systems Conference (DASC), pp. 1–10 (2022)

2022

[8] [8]

Knowledge-Based Systems214, 106,685 (2021)

Heuillet, A., Couthouis, F., Díaz-Rodríguez, N.: Explainability in deep rein- forcement learning. Knowledge-Based Systems214, 106,685 (2021). DOI https://doi.org/10.1016/j.knosys.2020.106685

arXiv 2021