Pith. sign in

REVIEW 3 major objections 5 minor 73 references

Semantic segmentation masks can replace RGB observations in a 3D RL agent, cutting memory-buffer footprint to as little as 1.4% of RGB while keeping scores comparable; adding the mask as a fourth channel improves frag counts.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 22:37 UTC pith:CWM5HVYT

load-bearing objection Plausible memory-saving idea for RL via segmentation masks, but the headline 98.6% compression claim is offline arithmetic, not an end-to-end buffer measurement, and performance claims rest on single seeds. the 3 major comments →

arxiv 2511.11703 v2 pith:CWM5HVYT submitted 2025-11-12 cs.LG cs.AIcs.RO

Enhancing Reinforcement Learning in 3D Environments through Semantic Segmentation: A Case Study in ViZDoom

Jin Huang This is my paper
classification cs.LG cs.AIcs.RO
keywords semantic segmentationreinforcement learningViZDoommemory efficiencyrun-length encodinginput representationPPO3D environments
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This MSc thesis sets out to establish that a one-channel semantic segmentation mask can largely take the place of three-channel RGB images in a visual reinforcement learning agent, and that giving the agent both RGB and semantics is better than RGB alone. In ViZDoom deathmatch scenarios, an agent trained on segmentation masks alone scores about as well as the RGB baseline while needing one third of the memory per observation; when masks are losslessly compressed with run-length encoding, the paper estimates memory use falls below 2% of RGB. Adding the mask as a fourth RGB channel increases average frag counts by roughly 25 percent on the trained map and keeps that edge on unseen maps. The paper also argues that prior attempts to use semantic segmentation in Doom were held back by unnecessary down-sampling and arbitrary colour mapping, and that using masks in their original one-channel form avoids those pitfalls.

Core claim

The central claim is that semantic segmentation masks are a better observation representation for RL than raw RGB in 3D environments: replacing RGB with SS-only preserves performance while enabling a 66.6% reduction in raw observation memory (and up to 98.6% with vectorised run-length encoding), while concatenating the mask to RGB as an extra channel 'significantly enhances' performance. Evaluated in three ViZDoom deathmatch maps with PPO agents, the SS-only agents' frag counts were comparable to the RGB baseline on the trained map (15.0 vs 15.2 with real-time segmentation), and the RGB+SS agent reached 19.2-19.3 versus 15.2 for RGB. The memory figures are computed from stored episodes, with

What carries the argument

The semantic segmentation mask: a one-channel, low-bit-depth image whose pixels are class labels (floor, wall, items, players, etc.) instead of colours. Replacing RGB with this mask ('SS-only') produces the memory compression, and adding it as a fourth stacked channel ('RGB+SS') injects semantic information into the existing visual input. The compression claim rests on a vectorised run-length encoding variant that stores each run's symbol, start position, and length in separate arrays with different bit depths, making the encoding O(n), parallelisable, and — in the paper's estimate — effectively negligible overhead; reconstruction is presented as simple fill operations that could run on the

Load-bearing premise

The 98.6% memory saving is computed on static arrays; it assumes that RLE-compressed semantic masks can actually be stored in a rollout or replay buffer and decoded on the GPU with negligible overhead and no change to training dynamics.

What would settle it

Train a PPO agent whose rollout buffer stores RLE-compressed SS masks (decoded on the GPU before the forward pass), using the paper's hyperparameters, and measure peak memory usage, steps per second, and frag scores. If peak memory is not below roughly 3% of the RGB baseline, or if the score drops more than a few percent from the uncompressed SS-only run, the central compression claim fails. A simpler check: instrument the existing training loop and report actual buffer memory instead of the size of stored episodes.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the memory numbers hold in practice, off-policy RL with experience replay could store tens of times more experience in the same memory, directly addressing the common bottleneck where the replay buffer fills before useful learning happens.
  • SS-only's small footprint makes frame-stacking practical, and the paper's SS(4) variant shows that the temporal information improves navigation on a complex unseen map.
  • RGB+SS works as a general 'add a semantic channel' recipe for 3D visual RL, with the performance edge persisting even when wall and floor textures change.
  • Lossless compression of categorical masks is unusually effective because same-class pixels form contiguous regions; the paper shows raw RGB actually grows when RLE is applied to it, so the benefit is specific to semantic inputs.
  • A one-channel categorical input matching the score of a three-channel continuous input challenges the usual assumption that more visual channels (e.g., RGB over grayscale) always help in ViZDoom-style environments.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The 98.6% figure is an array-level calculation, not a measurement from a training loop; until an agent is trained with RLE-compressed buffers and memory, decode time, and performance are measured, the headline saving should be read as an upper bound.
  • If compressed SS-only buffers work as claimed, the same trick applies to any categorical observation space — semantic maps, occupancy grids, or discrete state encodings — not just Doom.
  • The RGB+SS result suggests a cheaper alternative to architectural changes for scene understanding in visual RL: a small semantic segmentation head plus channel concatenation may capture much of the benefit.
  • A direct next test would be an off-policy agent (e.g., DQN-style) with a compressed SS-only replay buffer; the memory-saving claim matters most there, and sample efficiency per byte would be the metric to watch.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes two observation representations for PPO agents in ViZDoom deathmatches: SS-only, a one-channel semantic-segmentation mask, and RGB+SS, an RGB frame augmented with a semantic channel. Using ground-truth labels from ViZDoom for training and a DeepLabV3 model for real-time inference, the author reports that SS-only reduces memory-buffer consumption by 66.6%, and by up to 98.6% with vectorized run-length encoding, while retaining performance comparable to RGB. RGB+SS is reported to significantly improve frag scores in all evaluated maps, including unseen maps and texture-altered variants. The paper also contributes a density-based heatmap visualization of agent movement and a comparison with a prior ViZDoom segmentation study. All experiments use PPO with a CNN feature extractor; evaluation is based on 400 game-play episodes per agent per map, with built-in ZCajun bots as a non-neural baseline.

Significance. The 66.6% figure is a direct consequence of using one uint8 channel instead of three, so that part of the memory claim is sound and easy to verify. The RLE-based reduction to less than 2% of RGB storage is, if integrated into an actual buffer, a potentially useful contribution for memory-bound off-policy RL and for frame-stacking variants. The RGB+SS results provide a concrete case study of semantic augmentation in a 3D FPS environment, with code, pre-trained models, and recordings released. The paper is transparent about the use of perfect segmentation during training, which cleanly separates segmentation error from policy effects. However, the headline 98.6% saving is only demonstrated on static arrays, not in a training loop, and the performance comparisons rest on single training runs per representation. The contribution is therefore promising but not yet fully established at the level claimed.

major comments (3)
  1. [§4.3, Table 4.2, Appendix A.2] The headline 'up to 98.6%' memory saving is computed on static recorded frame arrays, not on an RL rollout or replay buffer. Section 3.6.2 describes the rollout buffer as storing raw observations; no agent is trained with RLE-compressed SS observations. Table 4.2 compares uint8 frame arrays (11 GB for RGB), whereas SB3 PPO buffers store float32 observations, and variable-length RLE requires per-sample offsets/indexing plus a gather/expand on each minibatch. Appendix A.2 provides only a naive Python reconstruction loop, so the 'negligible overhead' claim is not demonstrated. The end-to-end buffer memory saving and decode cost should be measured, or the claim should be restricted to static storage.
  2. [§4.1, Figure 4.1] All performance comparisons rest on a single training run per representation (after learning-rate selection). The 400 evaluation episodes reduce evaluation noise, but PPO's stochasticity across seeds is unquantified; differences such as RGB 15.2 vs SS-only 15.0 on Map 1, or the 9% drop from GT to RT SS-only on Map 2, are within unknown training variance. The word 'significantly' in the abstract is therefore not supported. Please run multiple seeds or explicitly reframe as a case study and remove 'significant'.
  3. [Abstract, §4.1.3, Chapter 5] The claim that SS-only 'maintains RL performance comparable to RGB' is not true across all evaluated scenarios. On Map 2 (high combat intensity), real-time SS-only scores 13.9 vs RGB's 17.7, matching the ZCajun baseline; on Map 3 real-time SS-only scores 7.8 vs RGB's 8.6. The conclusion itself acknowledges 'slightly worse in unseen maps.' The abstract should be qualified to avoid overclaiming.
minor comments (5)
  1. [Table of Contents] The table of contents contains a malformed entry, '25subsection.3.3.2', and several page-formatting glitches.
  2. [§3.7.2] The text says hyperparameters are kept consistent except for learning rates, but SS(4) agents use a wider MLP and a doubled batch size; this should be reconciled with the workflow in §3.1, step 7.
  3. [Table 4.1] NaN entries (e.g., ShellBox on Map 2, Shotgun on Map 3) are unexplained. State whether these correspond to zero samples of that class in the evaluation episodes.
  4. [§4.3] The statement that GPU decoding 'overhead is also negligible' is asserted without any measured decode throughput or latency. Add a benchmark or explicitly label it as an assumption.
  5. [§3.6.1.1] The phrase '1.5625 units per tic2' appears to contain a typo; clarify the relation of the 3-unit movement threshold to the stated Doom movement parameters.

Circularity Check

0 steps flagged

No material circularity: the performance and compression claims are empirical measurements against baselines; the sole self-citation is code reuse and is not load-bearing.

full rationale

The paper's central claims are empirical rather than derived from fitted constants. PPO agents using RGB, SS-only, and RGB+SS inputs are evaluated by direct frag-score comparisons against an RGB baseline and built-in bots; the reported performance differences are measured outcomes, not predictions forced by construction. The 66.6% memory figure is the arithmetic consequence of defining SS-only as a one-channel mask versus three-channel RGB, and the paper states it as such ('cutting the number of colour channels to 1/3'). The 98.6% figure is a measured run-length-encoding compression ratio over recorded game frames (Table 4.2), not a parameter fitted to achieve a desired result. There is no self-citation chain supplying a theorem or fitted value used as evidence: the only self-reference is [24], the author's undergraduate project, cited for reused source code and a data-flow figure, and it does not support any measured outcome. The main weakness is a validation gap, not circularity: Section 4.3 and Table 4.2 compute compressed sizes of static frame arrays and assert that GPU decoding 'can happen directly' with negligible overhead, while Appendix A.2 provides only a naive Python reconstruction loop and no agent is trained with a compressed rollout or replay buffer. This makes the end-to-end 98.6% memory-saving claim under-supported, but that is a correctness/robustness concern, not a definitional equivalence. The one minor non-load-bearing self-citation places the paper just above the zero point, but no circular step is present.

Axiom & Free-Parameter Ledger

6 free parameters · 6 axioms · 0 invented entities

The paper's central claims depend on a small number of hand-chosen experimental parameters (learning rates, reward-shaping weights, RLE bit-width) and on domain assumptions about ViZDoom's labels, the semantic class taxonomy, and the representativeness of the segmentation training set. No new physical or theoretical entities are introduced.

free parameters (6)
  • Per-representation learning rate = tuned per representation (exact values not all listed)
    Performance comparison across input representations relies on individually tuned learning rates; e.g., SS-only and RGB+SS were reported to have different optimal learning rates than RGB (Section 3.7.2).
  • Movement penalty displacement threshold = 3 units
    Chosen based on Doom movement physics at frame-skip 4 (Section 3.6.1.1); affects reward shaping and learned behavior.
  • Reward shaping coefficients = frag +1, death -1, health/ammo ±0.02 per point, movement ±0.00005/-0.0025
    Taken from reference works [32,49] and augmented with a movement penalty; they shape the task and agent behavior, but are not fitted in this paper.
  • SS model training schedule = lr 1e-4 to 1e-5 at 70k steps, 20 epochs
    Chosen by hand; affects real-time segmentation quality and therefore downstream RL performance.
  • SS(4) network width and batch size = MLP width 128, batch size 64
    Adjusted for the SS(4) variant to stabilise training; this confounds direct comparison with the other 4-channel representation (RGB+SS).
  • RLE-EX bitmasking storage width = 4 bits per element
    Ad hoc compression detail used in Table 4.2 to reach the 98.6% figure; not a standard RL buffer storage format.
axioms (6)
  • domain assumption ViZDoom labels buffer provides pixel-level ground-truth object IDs usable as perfect semantic segmentation
    Used as ground truth for training and controlled evaluation (Section 3.4).
  • ad hoc to paper The 13 semantic classes, with unknown objects mapped to Floor/Ceiling, are sufficient for the deathmatch task
    Adopted from [49]; if this taxonomy loses decision-relevant information, the SS-only and RGB+SS results are specific to this class definition (Section 3.4.2).
  • domain assumption PPO with the specified CNN and MLP is a stable and suitable RL algorithm for ViZDoom deathmatches
    The paper relies on PPO's stability and prior success in ViZDoom [49] rather than demonstrating suitability from first principles (Sections 1.1.3, 2.3.7).
  • domain assumption ZCajun bot scores are a meaningful non-neural baseline
    Used as the non-neural baseline despite the bots being undocumented and possibly not representative of modern opponents (Sections 4.1, 4.1.1).
  • standard math Run-length encoding is lossless and exact for discrete SS masks
    RLE on discrete arrays is lossless; the code in Appendix A.1 and reconstruction in A.2 are standard, assuming no dtype issues.
  • domain assumption The random train/validation split of frames from one cherry-picked agent represents the gameplay distribution of all evaluated agents
    The SS model is trained only on frames collected by an SS-only agent on Map 1, then evaluated on other agents and maps; the distribution shift is acknowledged but not corrected (Sections 3.9, 4.2).

pith-pipeline@v1.3.0-alltime-deepseek · 27422 in / 13485 out tokens · 144045 ms · 2026-08-03T22:37:41.276106+00:00 · methodology

0 comments
read the original abstract

Reinforcement learning (RL) in 3D environments with high-dimensional sensory input poses two major challenges: (1) the high memory consumption induced by memory buffers required to stabilise learning, and (2) the complexity of learning in partially observable Markov Decision Processes (POMDPs). This project addresses these challenges by proposing two novel input representations: SS-only and RGB+SS, both employing semantic segmentation on RGB colour images. Experiments were conducted in deathmatches of ViZDoom, utilizing perfect segmentation results for controlled evaluation. Our results showed that SS-only was able to reduce the memory consumption of memory buffers by at least 66.6%, and up to 98.6% when a vectorisable lossless compression technique with minimal overhead such as run-length encoding is applied. Meanwhile, RGB+SS significantly enhances RL agents' performance with the additional semantic information provided. Furthermore, we explored density-based heatmapping as a tool to visualise RL agents' movement patterns and evaluate their suitability for data collection. A brief comparison with a previous approach highlights how our method overcame common pitfalls in applying semantic segmentation in 3D environments like ViZDoom.

Figures

Figures reproduced from arXiv: 2511.11703 by Jin Huang.

Figure 2.1
Figure 2.1. Figure 2.1: From left to right: an unprocessed RGB game frame from Doom; perfect [PITH_FULL_IMAGE:figures/full_fig_p016_2_1.png] view at source ↗
Figure 2.2
Figure 2.2. Figure 2.2: Residual connection, source: figure 2 of [15] [PITH_FULL_IMAGE:figures/full_fig_p017_2_2.png] view at source ↗
Figure 2.3
Figure 2.3. Figure 2.3: A comparison of feature extraction on a star pattern (a) with: a 3 [PITH_FULL_IMAGE:figures/full_fig_p018_2_3.png] view at source ↗
Figure 2.4
Figure 2.4. Figure 2.4: Atrous spatial pyramid pooling, source: figure 4 of [4] [PITH_FULL_IMAGE:figures/full_fig_p019_2_4.png] view at source ↗
Figure 2.5
Figure 2.5. Figure 2.5: In reinforcement learning, an agent would receive observation of the environ [PITH_FULL_IMAGE:figures/full_fig_p020_2_5.png] view at source ↗
Figure 2.6
Figure 2.6. Figure 2.6: With a 2D view (top), the agent does not have access to some important [PITH_FULL_IMAGE:figures/full_fig_p021_2_6.png] view at source ↗
Figure 3.1
Figure 3.1. Figure 3.1: A positional heatmap of the RL agent used to gather data for training SS [PITH_FULL_IMAGE:figures/full_fig_p029_3_1.png] view at source ↗
Figure 3.2
Figure 3.2. Figure 3.2: Selected maps and alternative textures. In addition to these three maps, two additional variants of Map 1 have been tested, as illustrated in figure B.1 of the appendix. The first one would alter the wall textures (MFLR8 1 and BRONZE3) and another would additionally add moss to the floor texture (FLOOR0 7), as shown in part (b) of figure 3.23 . 1https://github.com/lkiel/rl-doom 2Slight modifications are … view at source ↗
Figure 3.3
Figure 3.3. Figure 3.3: A data-flow diagram of RL in ViZDoom, modified from figure 2 of [24] [PITH_FULL_IMAGE:figures/full_fig_p035_3_3.png] view at source ↗
Figure 3.4
Figure 3.4. Figure 3.4: The feature extractor CNN for our RL agents. [PITH_FULL_IMAGE:figures/full_fig_p036_3_4.png] view at source ↗
Figure 3.5
Figure 3.5. Figure 3.5: A comparison of semantic segmentation results with different backbones. [PITH_FULL_IMAGE:figures/full_fig_p039_3_5.png] view at source ↗
Figure 4.1
Figure 4.1. Figure 4.1: Box-and-whisker plots for evaluation performance of RL agents on each map, [PITH_FULL_IMAGE:figures/full_fig_p045_4_1.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

73 extracted references · 19 linked inside Pith

  1. [1]

    M. G. Bellemare, Y . Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253–279, June 2013

    2013

  2. [2]

    Find length of sequences of identical values in a numpy array (run length encoding), Nov 1954

    Thomas Browne. Find length of sequences of identical values in a numpy array (run length encoding), Nov 1954. Available at https://stackoverflow.com/ a/32681075

    arXiv 1954

  3. [3]

    Fast filter transform for image processing

    Peter J Burt. Fast filter transform for image processing. Computer Graphics and Image Processing, 16(1):20–51, 1981

    1981

  4. [4]

    Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. CoRR, abs/1606.00915, 2016

    Pith/arXiv arXiv 2016

  5. [5]

    Rethinking atrous convolution for semantic image segmentation

    Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. CoRR, abs/1706.05587, 2017

    Pith/arXiv arXiv 2017

  6. [6]

    Encoder-decoder with atrous separable convolution for semantic image segmentation

    Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss, editors,Computer Vision – ECCV 2018, pages 833–851, Cham, 2018. Springer International Publishing

    2018

  7. [7]

    Xception: Deep learning with depthwise separable convolutions

    Franc ¸ois Chollet. Xception: Deep learning with depthwise separable convolutions. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1800–1807, 2017

    2017

  8. [8]

    Deep recurrent q-learning of behav- ioral intervention delivery by a robot from demonstration data

    Madison Clark-Turner and Momotaz Begum. Deep recurrent q-learning of behav- ioral intervention delivery by a robot from demonstration data. 08 2017. 41 Bibliography 42

    2017

  9. [9]

    A compre- hensive survey and performance analysis of activation functions in deep learning

    Shiv Ram Dubey, Satish Kumar Singh, and Bidyut Baran Chaudhuri. A compre- hensive survey and performance analysis of activation functions in deep learning. CoRR, abs/2109.14545, 2021

    Pith/arXiv arXiv 2021

  10. [10]

    Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position

    Kunihiko Fukushima. Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics, 36:193–202, 1980

    1980

  11. [11]

    Ben J. Geisler. An empirical study of machine learning algorithms applied to modeling player behavior in a ”first person shooter” video game. 2002

    2002

  12. [12]

    Pogamut 3 Can Assist Developers in Building AI (Not Only) for Their Videogame Agents, pages 1–15

    Jakub Gemrot, Rudolf Kadlec, Michal B ´ıda, Ondˇrej Burkert, Radek P´ıbil, Jan Havl´ıˇcek, Luk´aˇs Zemˇc´ak, Juraj ˇSimloviˇc, Radim Vansa, Michal ˇStolba, Tom´aˇs Plch, and Cyril Brom. Pogamut 3 Can Assist Developers in Building AI (Not Only) for Their Videogame Agents, pages 1–15. Springer Berlin Heidelberg, Berlin, Heidelberg, 2009

    2009

  13. [13]

    Harris, K

    Charles R. Harris, K. Jarrod Millman, St ´efan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fern´andez del R´ıo, Mark Wiebe, Pearu Peterson, Pierre G´erard-Marchant, Kevin ...

    2020

  14. [14]

    Hausknecht and Peter Stone

    Matthew J. Hausknecht and Peter Stone. Deep recurrent q-learning for partially observable mdps. CoRR, abs/1507.06527, 2015

    Pith/arXiv arXiv 2015

  15. [15]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CoRR, abs/1512.03385, 2015

    Pith/arXiv arXiv 2015

  16. [16]

    Zdoom, Mar 1998

    Marisa Heit, Christoph Oelckers, and Braden Obrzut. Zdoom, Mar 1998. Available at https://zdoom.org

    1998

  17. [17]

    Deep reinforcement learning that matters

    Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. CoRR, abs/1709.06560, 2017

    Pith/arXiv arXiv 2017

  18. [18]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. CoRR, abs/2006.11239, 2020. Bibliography 43

    Pith/arXiv arXiv 2006

  19. [19]

    Long short-term memory

    Sepp Hochreiter and J ¨urgen Schmidhuber. Long short-term memory. Neural computation, 9:1735–80, 12 1997

    1997

  20. [20]

    Virtual-to-real: Learning to control in visual semantic segmentation

    Zhang-Wei Hong, Yu-Ming Chen, Shih-Yang Su, Tzu-Yun Shann, Yi-Hsiang Chang, Hsuan-Kung Yang, Brian Hsi-Lin Ho, Chih-Chieh Tu, Yueh-Chuan Chang, Tsu-Ching Hsiao, Hsin-Wei Hsiao, Sih-Pin Lai, and Chun-Yi Lee. Virtual-to-real: Learning to control in visual semantic segmentation. CoRR, abs/1802.00285, 2018

    Pith/arXiv arXiv 2018

  21. [21]

    Multilayer feedforward networks are universal approximators

    Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural Networks, 2(5):359–366, 1989

    1989

  22. [22]

    Searching for mobilenetv3

    Andrew Howard, Mark Sandler, Bo Chen, Weijun Wang, Liang-Chieh Chen, Mingxing Tan, Grace Chu, Vijay Vasudevan, Yukun Zhu, Ruoming Pang, Hartwig Adam, and Quoc Le. Searching for mobilenetv3. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 1314–1324, 2019

    2019

  23. [23]

    Freedoom, Apr 2003

    Simon Howard and Mike Swanson. Freedoom, Apr 2003. Available at https: //freedoom.github.io

    2003

  24. [24]

    Doom-playing ai via deep reinforcement learning

    Hugo Huang. Doom-playing ai via deep reinforcement learning. 2023. Available at https://pats.cs.cf.ac.uk/archive?Y=2023

    2023

  25. [25]

    David A. Huffman. A method for the construction of minimum-redundancy codes. Proceedings of the IRE, 40(9):1098–1101, 1952

    1952

  26. [26]

    Batch normalization: Accelerating deep network training by reducing internal covariate shift

    Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. CoRR, abs/1502.03167, 2015

    Pith/arXiv arXiv 2015

  27. [27]

    Cybernetic predicting devices

    Alekse˘ı Grigorevich Ivakhnenko and Valentin Grigorevich Lapa. Cybernetic predicting devices. 1966

    1966

  28. [28]

    Vizdoom ai via deep reinforcement learning

    Aaron James. Vizdoom ai via deep reinforcement learning. 2022. Available at https://pats.cs.cf.ac.uk/archive?Y=2022

    2022

  29. [29]

    Analysis of reinforcement learning in autonomous ve- hicles

    Estephanos Jebessa, Kidus Olana, Kidus Getachew, Stuart Isteefanos, and Tauheed Khan Mohd. Analysis of reinforcement learning in autonomous ve- hicles. In 2022 IEEE 12th Annual Computing and Communication Workshop and Conference (CCWC), pages 0087–0091, 2022. Bibliography 44

    2022

  30. [30]

    Slade: It’s a doom editor, May 2024

    Simon Judd. Slade: It’s a doom editor, May 2024. Available at https://github. com/sirjuddington/SLADE

    2024

  31. [31]

    ViZDoom: A Doom-based AI research platform for visual reinforce- ment learning

    Michał Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, and Wojciech Ja´skowski. ViZDoom: A Doom-based AI research platform for visual reinforce- ment learning. In IEEE Conference on Computational Intelligence and Games, pages 341–348, Santorini, Greece, Sep 2016. IEEE. The Best Paper Award

    2016

  32. [32]

    Playing doom with deep reinforcement learning-part 3: Boosting performance with reward shaping..., Nov 2021

    Leandro Kieliger. Playing doom with deep reinforcement learning-part 3: Boosting performance with reward shaping..., Nov 2021

    2021

  33. [33]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017

    2017

  34. [34]

    Actor-critic algorithms

    Vijay Konda and John Tsitsiklis. Actor-critic algorithms. In S. Solla, T. Leen, and K. M ¨uller, editors, Advances in Neural Information Processing Systems , volume 12. MIT Press, 1999

    1999

  35. [35]

    Playing FPS games with deep reinforcement learning

    Guillaume Lample and Devendra Singh Chaplot. Playing FPS games with deep reinforcement learning. CoRR, abs/1609.05521, 2016

    Pith/arXiv arXiv 2016

  36. [36]

    Lecun, L

    Y . Lecun, L. Bottou, Y . Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998

    1998

  37. [37]

    Self-improving reactive agents based on reinforcement learning, planning and teaching

    Long-Ji Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine Learning, 8(3):293–321, May 1992

    1992

  38. [38]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollar, and Larry Zitnick. Microsoft coco: Common objects in context. In ECCV. European Conference on Computer Vision, September 2014

    2014

  39. [39]

    The doom movement bible, Oct 2023

    Linguica. The doom movement bible, Oct 2023

    2023

  40. [40]

    Henriques, and Andrea Vedaldi

    Aravindh Mahendran, Hakan Bilen, Jo ˜ao F. Henriques, and Andrea Vedaldi. Researchdoom and cocodoom: Learning computer vision with games. CoRR, abs/1610.02431, 2016

    Pith/arXiv arXiv 2016

  41. [41]

    Llm for unity, Jan 2024

    Antonis Makropoulos. Llm for unity, Jan 2024. Available at https://github. com/undreamai/LLMUnity. Bibliography 45

    2024

  42. [42]

    Riedmiller

    V olodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. CoRR, abs/1312.5602, 2013

    Pith/arXiv arXiv 2013

  43. [43]

    Exploiting seman- tic segmentation to boost reinforcement learning in video game environments

    Javier Montalvo, ´Alvaro Garc´ıa-Mart´ın, and Jes ´us Besc ´os. Exploiting seman- tic segmentation to boost reinforcement learning in video game environments. Multimedia Tools and Applications, 82(7):10961–10979, March 2023

    2023

  44. [44]

    Ng, Daishi Harada, and Stuart J

    Andrew Y . Ng, Daishi Harada, and Stuart J. Russell. Policy invariance under reward transformations: Theory and application to reward shaping. InProceedings of the Sixteenth International Conference on Machine Learning, ICML ’99, page 278–287, San Francisco, CA, USA, 1999. Morgan Kaufmann Publishers Inc

    1999

  45. [45]

    Nvidia ace, Jun 2024

    NVIDIA. Nvidia ace, Jun 2024. Available at https://github.com/NVIDIA/ ACE

    2024

  46. [46]

    Pytorch: An imperative style, high-performance deep learn- ing library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gre- gory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-per...

    2019

  47. [47]

    Edge detection based on kernel density estimation

    Osvaldo Pereira, Esley Torres, Yasel Garc´es Su´arez, and Roberto Rodr´ıguez. Edge detection based on kernel density estimation. CoRR, abs/1411.1297, 2014

    Pith/arXiv arXiv 2014

  48. [48]

    Generalization, mayhems and limits in recurrent proximal policy optimization, 2022

    Marco Pleines, Matthias Pallasch, Frank Zimmer, and Mike Preuss. Generalization, mayhems and limits in recurrent proximal policy optimization, 2022

    2022

  49. [49]

    Utilising Semantic Segmentation to Improve the Perfor- mance of Reinforcement Learning Based Agents in Doom Video Game

    Alexandros Pournaras. Utilising Semantic Segmentation to Improve the Perfor- mance of Reinforcement Learning Based Agents in Doom Video Game. 2021. Available at https://ikee.lib.auth.gr/record/336332

    2021

  50. [50]

    Ppo vs recurrentppo (aka ppo lstm) on environments with masked velocity (sb3 contrib), May 2022

    Antonin Raffin. Ppo vs recurrentppo (aka ppo lstm) on environments with masked velocity (sb3 contrib), May 2022

    2022

  51. [51]

    Stable-baselines3: Reliable reinforcement learning implementations

    Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernes- tus, and Noah Dormann. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1–8, 2021. Bibliography 46

    2021

  52. [52]

    Robinson and C

    A.H. Robinson and C. Cherry. Results of a prototype television bandwidth compression scheme. Proceedings of the IEEE, 55(3):356–364, 1967

    1967

  53. [53]

    U-net: Convolutional networks for biomedical image segmentation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. CoRR, abs/1505.04597, 2015

    Pith/arXiv arXiv 2015

  54. [54]

    Prioritized experience replay, 2016

    Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay, 2016

    2016

  55. [55]

    Jordan, and Pieter Abbeel

    John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, and Pieter Abbeel. Trust region policy optimization. CoRR, abs/1502.05477, 2015

    Pith/arXiv arXiv 2015

  56. [56]

    High-dimensional continuous control using generalized advantage estimation, 2018

    John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation, 2018

    2018

  57. [57]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017

    Pith/arXiv arXiv 2017

  58. [58]

    A survey of deep reinforcement learning in video games

    Kun Shao, Zhentao Tang, Yuanheng Zhu, Nannan Li, and Dongbin Zhao. A survey of deep reinforcement learning in video games. CoRR, abs/1912.10944, 2019

    Pith/arXiv arXiv 1912

  59. [59]

    Fully convolutional networks for semantic segmentation

    Evan Shelhamer, Jonathan Long, and Trevor Darrell. Fully convolutional networks for semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(4):640–651, 2017

    2017

  60. [60]

    A 3 ×3 isotropic gradient operator for image processing

    Irwin Sobel and Gary Feldman. A 3 ×3 isotropic gradient operator for image processing. Pattern Classification and Scene Analysis, pages 271–272, 01 1973

    1973

  61. [61]

    Sutton and A.G

    R.S. Sutton and A.G. Barto. Reinforcement learning: An introduction. IEEE Transactions on Neural Networks, 9(5):1054–1054, 1998

    1998

  62. [62]

    Deep reinforcement learning for robotics: A survey of real-world successes, 2024

    Chen Tang, Ben Abbatematteo, Jiaheng Hu, Rohan Chandra, Roberto Mart ´ın- Mart´ın, and Peter Stone. Deep reinforcement learning for robotics: A survey of real-world successes, 2024

    2024

  63. [63]

    Thomas and Emma Brunskill

    Philip S. Thomas and Emma Brunskill. Policy gradient methods for reinforcement learning with function approximation and action-dependent baselines. CoRR, abs/1706.06643, 2017. Bibliography 47

    Pith/arXiv arXiv 2017

  64. [64]

    Terry, Ariel Kwiatkowski, John U

    Mark Towers, Jordan K. Terry, Ariel Kwiatkowski, John U. Balis, Gianluca de Cola, Tristan Deleu, Manuel Goul ˜ao, Andreas Kallinteris, Arjun KG, Markus Krimmel, Rodrigo Perez-Vicente, Andrea Pierr´e, Sander Schulhoff, Jun Jet Tai, Andrew Tan Jin Shen, and Omar G. Younis. Gymnasium, March 2023

    2023

  65. [65]

    Tsitsiklis and B

    J.N. Tsitsiklis and B. Van Roy. An analysis of temporal-difference learning with function approximation. IEEE Transactions on Automatic Control, 42(5):674–690, 1997

    1997

  66. [66]

    J.W. Tukey. Exploratory Data Analysis: Limited Preliminary Ed. Number v. 2. Addison-Wesley Publishing Company, 1970

    1970

  67. [67]

    Diffusion models are real-time game engines, 2024

    Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter. Diffusion models are real-time game engines, 2024

    2024

  68. [68]

    Learning From Delayed Rewards

    Christopher Watkins. Learning From Delayed Rewards. PhD thesis, University of Cambridge, 01 1989

    1989

  69. [69]

    A technique for high-performance data compression

    Welch. A technique for high-performance data compression. Computer, 17(6):8– 19, 1984

    1984

  70. [70]

    Image threshold- ing segmentation based on weighted parzen-window and linear programming techniques

    Fusong Xiong, Zhiqiang Zhang, Yun Ling, and Jian Zhang. Image threshold- ing segmentation based on weighted parzen-window and linear programming techniques. Scientific Reports, 12(1):13635, Aug 2022

    2022

  71. [71]

    Kevin Zhou, Hoang Ngan Le, Khoa Luu, Hien V Nguyen, and Nicholas Ayache

    S. Kevin Zhou, Hoang Ngan Le, Khoa Luu, Hien V Nguyen, and Nicholas Ayache. Deep reinforcement learning in medical imaging: A literature review. Medical Image Analysis, 73:102193, 2021

    2021

  72. [72]

    Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B

    Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul F. Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. CoRR, abs/1909.08593, 2019

    Pith/arXiv arXiv 1909

  73. [73]

    "" run length encoding. Partial credit to R rle function. Multi datatype arrays catered for including non Numpy returns: tuple (runlengths, startpositions, values)

    K.J ˚Astr¨om. Optimal control of markov processes with incomplete state infor- mation. Journal of Mathematical Analysis and Applications , 10(1):174–205, 1965. Appendix A Additional Code This part contains the additional source code and pseudocode that couldn’t fit into the main report within the page limitations. 48 Appendix A. Additional Code 49 A.1 Vec...

    1965