Pith. sign in

REVIEW 4 major objections 7 minor 9 references

Light Aircraft Game : Basic Implementation and training results analysis

T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read In an aircraft-combat RL benchmark, HASAC wins unarmed fights while HAPPO wins missile duels.

desk verdict The new LAG environment has some merit, but the paper's central HASAC-vs-HAPPO claim is contradicted by its own Table 1, making the empirical results unusable as written. read the letter →

arxiv 2506.14164 v1 pith:5LFS7YIG submitted 2025-06-17 cs.LG cs.MA

Hanzhong Cao This is my paper
classification cs.LGcs.MA
keywords multi-agentreinforcementlearninglightaircraftgameHAPPOHASACaerialcombathierarchicalcontrolon-policyvsoff-policyrewarddesign
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces Light Aircraft Game (LAG), a lightweight, Gym-based multi-agent combat environment, and uses it to compare two heterogeneous-agent reinforcement learning algorithms: HAPPO, an on-policy trust-region method, and HASAC, an off-policy maximum-entropy method. In the reported experiments, HASAC maintains stable, positive rewards in weaponless coordination tasks, while HAPPO reaches far higher peak rewards in missile combat. The paper argues this reversal reflects a general trade-off: off-policy methods give stable learning for basic interactions, while on-policy methods are better suited to complex, expressive behaviors once their critic is stabilized. If this is right, algorithm choice in multi-agent systems should depend on task dynamics, not just on aggregate benchmark scores.

What carries the argument

The central object is the Light Aircraft Game (LAG) environment, a lightweight Gym-wrapped aircraft combat simulator. Its load-bearing design is hierarchical control: a pre-trained low-level controller maps high-level targets (direction, altitude, velocity) to continuous actuation (aileron, elevator, rudder, throttle), so that combat tasks only need to learn tactical decisions. The reward structure combines AltitudeReward (safety penalties), PostureReward (positional advantage), and EventDrivenReward (sparse ±200 events), which creates different learning pressures in the No Weapon and ShootMissile modes. The evidence for the paper's claim is Table 1, which reports evaluation rewards for HAPPO and HASAC across three protocols and three timesteps, and the reversal of their relative ranking between the two combat modes.

What would settle it

Run both algorithms on the No Weapon and ShootMissile tasks with multiple seeds and identical training budgets; if HAPPO's average No Weapon reward is not consistently below HASAC's, or if HASAC's peak ShootMissile reward matches or exceeds HAPPO's when trained longer, the claimed reversal would be falsified.

Watch

Extended reading notes

Core claim

The paper's central claim is that neither algorithm dominates in the LAG environment: HASAC achieves positive evaluation rewards around 30 in the No Weapon setting across HierarchySelfplay, SelfPlay, and vsBaseline, while HAPPO's rewards remain negative or collapse; in the ShootMissile setting, HAPPO's rewards surge from 385.27 to over 1090.17 in HierarchyVsBaseline, exceeding HASAC's peak of 918.70 in HierarchySelfplay. The author interprets this reversal as evidence that off-policy, entropy-regularized updates stabilize coordination in low-dimensional positional tasks, while the clipped surrogate objective of HAPPO supports the exploratory, high-variance policies needed for missile engagement. Training diagnostics show HAPPO's critic loss is highly unstable in the No Weapon setting, and the author links these critic fluctuations temporally to drops in evaluation reward, suggesting the instability is a cause rather than a side effect.

Load-bearing premise

The comparison assumes that the single-run evaluation rewards in Table 1 are stable measures of algorithm performance, even though the algorithms were evaluated at different timesteps and no seeds or variance are reported.

Editorial extensions

If this is right

  • If the reported reversal is accurate, multi-agent evaluations in LAG should report results separately for each combat mode rather than a single aggregate score, since the two algorithms dominate in different regimes.
  • The paper's diagnosis of HAPPO's critic instability implies that stabilizing value-function learning, for example with better value targets or auxiliary losses, could raise HAPPO's performance in No Weapon settings.
  • Because the low-level controller is trained once in SingleControl and then shared, improvements in that foundational task are expected to transfer to both No Weapon and ShootMissile combat.
  • HASAC's steady positive rewards make it a natural default baseline for coordination-heavy tasks, while HAPPO's high peaks make it the candidate for tasks with sparse, event-driven rewards.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper shows single runs at different timesteps for each algorithm; a multi-seed replication with matched training budgets would test whether the claimed reversal is robust or an artifact of sampling noise.
  • The stability-versus-expressiveness trade-off could be probed in other hierarchical MARL benchmarks that share the low-level/high-level split, to see whether the LAG result generalizes.
  • The ShootMissile task uses a Beta-prior to make missile firing learnable; ablating that prior would clarify whether HAPPO's advantage comes from the on-policy objective or from the shaped reward signal.
  • A combined training scheme, using HASAC to learn stable coordination and then fine-tuning with HAPPO, is a concrete design idea implied by the results but not tested in the paper.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper introduces Light Aircraft Game (LAG), a lightweight multi-agent aircraft-combat Gym environment with hierarchical control, and reports training results of HAPPO and HASAC across No Weapon and ShootMissile tasks. The authors claim that HASAC is better for simpler coordination tasks without weapons, while HAPPO performs better in dynamic missile combat. The manuscript contains a description of the environment and reward design, brief summaries of the two algorithms, a single table of evaluation rewards, and a discussion of training diagnostics for HAPPO.

Significance. If the empirical results were reliable, the paper would provide a useful data point for comparing on-policy and off-policy MARL algorithms on a custom aerial-combat benchmark, and the environment could be a useful community resource. However, the paper's headline conclusions are contradicted by its own Table 1, the experimental section is empty, no code or hyperparameters are provided, and no error bars or seeds are reported. As submitted, the paper does not establish its claims, so its significance is currently limited to a description of the environment concept.

major comments (4)
  1. [§5.2.1, Table 1] The No Weapon conclusion is directly contradicted by Table 1. The text says 'HASAC consistently outperforms HAPPO across all protocols and timesteps' and 'HAPPO's returns remain negative throughout training (from –54.13 to –66.92)' for HierarchySelfplay, but the HAPPO row of Table 1 shows +31.61 at 1.5M timesteps, which is positive and higher than HASAC's values (30.14, 30.25, 30.20) at the reported HASAC timesteps. This is not a statistical nuance: the reported evidence supports the opposite conclusion for that row. Either Table 1 or §5.2.1 must be corrected, and until then the paper's primary claim is unsupported.
  2. [§5.2.2, Table 1] The ShootMissile analysis is also inconsistent with the table. The text states that HASAC 'remains below HAPPO's peak' in the missile setting, but in HierarchySelfplay HASAC reaches 918.70 at 610k, exceeding HAPPO's peak of 910.50 at 1.02M and HAPPO's final value of 782.46. The text further attributes 'a surge from 385.27 to over 1 090.17' to 'HierarchySelfplay & vsBaseline', but Table 1 shows that 1 090.17 appears only in the HierarchyVsBaseline column, while HierarchySelfplay ends at 782.46. The claimed reversal in favor of HAPPO is therefore not established by the reported numbers.
  3. [§4 (Experiments)] Section 4 is empty: it contains only the heading '4. Experiments' and no content, and Section 5.1 ('Full Results') also has no text. The paper reports no hyperparameters, network architectures, learning rates, batch sizes, training budgets, seeds, number of trials, evaluation episodes, hardware, or environment version. Table 1 compares HAPPO at 510k/1.02M/1.5M timesteps with HASAC at 130k/370k/610k timesteps, so the algorithms are not evaluated at comparable training progress, and no variance or confidence intervals are reported. These omissions are load-bearing because every comparative claim in Sections 5 and 6 depends on the numbers in Table 1.
  4. [§2.2, §2.3, §3.2] The paper gives no details or validation for the hand-designed reward and prior. Section 2.3 lists altitude/posture/event rewards with constants such as ±200 but no sensitivity analysis or comparison against alternative reward designs, and Section 2.2 mentions a Beta-distribution prior for missile firing without specifying the prior parameters, the update rule, or an ablation. Since LAG is a custom environment, there is no external benchmark to establish that these rewards define a meaningful task; the reported differences could be artifacts of the reward scheme rather than of HAPPO versus HASAC.
minor comments (7)
  1. [Abstract] The 'Github Repo' text is not a clickable link; if code is released, provide the full URL in a footnote or reference so the reader can access it.
  2. [Table 1] The column header 'Algorithm env timestep' is ambiguous. Please clarify the unit (environment steps), whether the rows are checkpoints from a single training run or separate runs, and whether each number is a mean over evaluation episodes.
  3. [§5.2] The protocol naming is inconsistent: the text refers to 'vsBaseline' and 'HierarchySelfplay', while Table 1's last columns are 'HierarchyVsBaseline'; additionally, ShootMissile has no SelfPlay column even though the NoWeapon setting has one. Please use one consistent set of protocol names.
  4. [§5.3.1] The citation to Jiang et al. (2017), which is about financial portfolio management, seems unrelated to the claim about gradient instability in MARL; please use a more relevant reference or remove it.
  5. [Figure captions] There are several typos and formatting problems, for example 'average episode rewrad' in Figure 5, inconsistent spacing in 'No Weapon' and 'NoWeapon', and Figure 2/3 references that are not immediately tied to the captions. A careful proofreading pass is needed.
  6. [§5.4] The claim that critic instability 'appears to coincide temporally with major drops in the evaluation reward' is not backed by any quantitative analysis; either add a quantitative measure or soften the causal language.
  7. [§2.2, Table 1] The relationship between the task suite (SingleControl, SingleCombat, DualCombat) and the columns of Table 1 is not explained; in particular, it is unclear whether 'HierarchySelfplay' and 'HierarchyVsBaseline' refer to SingleCombat or DualCombat tasks.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper's claims are direct empirical comparisons of two externally published algorithms on a custom environment, with no fitted parameter renamed as a prediction and no load-bearing self-citation.

full rationale

The paper derives no formal result from its inputs. The central claims in §5.2 and §6 are summaries of Table 1: HAPPO and HASAC evaluation rewards under No Weapon and ShootMissile protocols. The algorithms HAPPO and HASAC are cited from Kuba et al. (2022) and Liu et al. (2025), not from the author's own prior work, and the paper does not fit any parameter to the reported rewards and then present the same quantity as a prediction. The environment's reward design (§2.3) is a description, not a derivation; the conclusion that HASAC is stable in simple tasks and HAPPO is expressive in missile combat is a direct reading of the tabulated numbers. Even if Table 1 contains internal inconsistencies (e.g., HAPPO's NoWeapon HierarchySelfplay value of 31.61 at 1.5M steps conflicting with §5.2.1's claim that HAPPO remains negative), that is a correctness or reproducibility concern, not circularity: the comparison is not constructed so that the outcome is true by definition. No self-citation chain, uniqueness theorem, or ansatz-smuggling is present. The absence of error bars and the unequal timestep schedules weaken the empirical support but do not make the argument circular. The empty Section 4 and missing code, seeds, and hyperparameters prevent verification, but that is a completeness issue unrelated to circularity. Therefore the appropriate circularity score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim depends on hand-chosen reward constants and unreported hyperparameters. The algorithms' theoretical guarantees are borrowed from prior work. The custom environment is an unvalidated domain assumption rather than an independent benchmark.

free parameters (4)
  • Reward magnitudes = +200 for kill, -200 for crash/destroyed
    Hand-chosen constants in EventDrivenReward that directly shape the learned behavior.
  • Altitude penalty scaling = [-1,0] ranges
    Ad hoc penalty intervals for altitude violations; exact thresholds not given.
  • Beta prior parameters for missile firing = not reported
    A conjugate prior used to seed missile firing decisions; values not disclosed, affecting policy initialization.
  • Algorithm hyperparameters = not reported
    Learning rates, batch sizes, etc. are absent, so the comparison is uncontrolled.
assumptions (4)
  • standard math HAPPO guarantees monotonic joint policy improvement under trust region updates
    Assumed from Kuba et al. 2022; not re-derived.
  • standard math HASAC converges to a quantal response equilibrium and enjoys monotonic improvement
    Assumed from Liu et al. 2025; not re-derived.
  • domain assumption Aircraft and missile dynamics are adequately modeled by the custom simulators
    The paper relies on its own physics models without validation against real flight data.
  • domain assumption The composite reward function accurately encodes tactical desirability
    No justification or ablation shows that the reward weights (e.g., -200/+200) yield desired behavior.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Light Aircraft Game : Basic Implementation and training results analysis." pith.science (2026). https://pith.science/paper/5LFS7YIG

@misc{pith2026250614164,
  author       = {Pith},
  title        = {Pith review of: Light Aircraft Game : Basic Implementation and training results analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5LFS7YIG}},
  note         = {Machine review of arXiv:2506.14164}
}
read the original abstract

This paper investigates multi-agent reinforcement learning (MARL) in a partially observable, cooperative-competitive combat environment known as LAG. We describe the environment's setup, including agent actions, hierarchical controls, and reward design across different combat modes such as No Weapon and ShootMissile. Two representative algorithms are evaluated: HAPPO, an on-policy hierarchical variant of PPO, and HASAC, an off-policy method based on soft actor-critic. We analyze their training stability, reward progression, and inter-agent coordination capabilities. Experimental results show that HASAC performs well in simpler coordination tasks without weapons, while HAPPO demonstrates stronger adaptability in more dynamic and expressive scenarios involving missile combat. These findings provide insights into the trade-offs between on-policy and off-policy methods in multi-agent settings.

Figures

Figures reproduced from arXiv: 2506.14164 by the authors.

Figure 1
Figure 1. imp weights of agent of happo algorithms under NoWeapon, HierarchySelfplay experiment setting [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. average step rewards of critic of happo algorithms under NoWeapon, HierarchySelfplay experiment setting loss curve (see [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. value loss of critic of happo algorithms under NoWeapon, HierarchySelfplay experiment setting [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: critic grad norm of agent of happo algorithms under NoWeapon, HierarchySelfplay experiment setting [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 9
Figure 9. Figure 9: dist entropy of actors of happo algorithms under NoWeapon, HierarchySelfplay experiment setting [PITH_FULL_IMAGE:figures/full_fig_p006_9.png]
Figure 6
Figure 6. Figure 6: max episode reward in evaluation of happo algorithms under NoWeapon, HierarchySelfplay experiment setting [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

9 extracted references · 9 linked inside Pith

  1. [6]

    Liu, J., Zhong, Y ., Hu, S., Fu, H., Fu, Q., Chang, X., and Yang, Y

    URL https:// arxiv.org/abs/2109.11251. Liu, J., Zhong, Y ., Hu, S., Fu, H., Fu, Q., Chang, X., and Yang, Y . Maximum entropy heterogeneous-agent re- inforcement learning,

    arXiv

  2. [9]

    URL https://arxiv.org/abs/ 1902.04043

    arXiv 1902

  3. [2017]

    URL https://arxiv.org/abs/ 1706.10059. Kuba, J. G., Chen, R., Wen, M., Wen, Y ., Sun, F., Wang, J., and Yang, Y . Trust region policy optimisation in multi- agent reinforcement learning,

    arXiv

  4. [2019]

    Jiang, Z., Xu, D., and Liang, J

    URL https://arxiv.org/abs/ 1812.05905. Jiang, Z., Xu, D., and Liang, J. A deep reinforcement learn- ing framework for the financial portfolio management problem,

    arXiv

  5. [2020]

    Ellis, B., Cook, J., Moalla, S., Samvelyan, M., Sun, M., Mahajan, A., Foerster, J

    URL https://arxiv.org/abs/ 2011.09533. Ellis, B., Cook, J., Moalla, S., Samvelyan, M., Sun, M., Mahajan, A., Foerster, J. N., and Whiteson, S. Smacv2: An improved benchmark for cooperative multi-agent re- inforcement learning,

    arXiv 2011

  6. [2021]

    Samvelyan, M., Rashid, T., de Witt, C

    URLhttps://arxiv.org/abs/2003.06709. Samvelyan, M., Rashid, T., de Witt, C. S., Farquhar, G., Nardelli, N., Rudner, T. G. J., Hung, C.-M., Torr, P. H. S., Foerster, J., and Whiteson, S. The starcraft multi-agent challenge,

    arXiv 2003

  7. [2022]

    de Witt, C

    URL https:// arxiv.org/abs/2206.08686. de Witt, C. S., Gupta, T., Makoviichuk, D., Makoviychuk, V ., Torr, P. H. S., Sun, M., and Whiteson, S. Is inde- pendent learning all you need in the starcraft multi-agent challenge?,

    arXiv

  8. [2023]

    org/abs/2212.07489

    URL https://arxiv. org/abs/2212.07489. Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V ., Zhu, H., Gupta, A., Abbeel, P., and Levine, S. Soft actor-critic algorithms and ap- plications,

    arXiv

Show all 9 references
  1. [2025]

    org/abs/2306.10715

    URL https://arxiv. org/abs/2306.10715. Peng, B., Rashid, T., de Witt, C. A. S., Kamienny, P.-A., Torr, P. H. S., B¨ohmer, W., and Whiteson, S. Facmac: Factored multi-agent centralised policy gradients,

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.