Pith. sign in

REVIEW 3 major objections 5 minor 46 references

A sub-$400 camera-and-keyboard platform measures a 1160% Sim-to-Real collapse for RL game agents while proving real-world training still works.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 12:43 UTC pith:NVB2BHDZ

load-bearing objection Clean empirical gap numbers on a cheap physical Breakout loop; the “universal AIoT benchmark” framing is the soft part, not the measurements. the 3 major comments →

arxiv 2607.10309 v1 pith:NVB2BHDZ submitted 2026-07-11 cs.AI

Measure the Sim-to-Real Gap: Designing an Affordable Real-World Benchmark Platform for Reinforcement Learning in AIoT Systems

Rongping Zhou , Omid Tavallaie , Shuaijun Chen , Albert Y. Zomaya This is my paper
classification cs.AI
keywords reinforcement learningSim-to-Real gapAIoTreal-world benchmarkdeep Q-networkedge computingAtari Breakouthuman-normalized score
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Most reinforcement-learning research for Autonomous Internet of Things systems stays inside simulators because real trials are expensive and sometimes dangerous. Without a shared physical testbed, it is hard to know how large the gap between simulation and reality actually is, or whether an algorithm that works in simulation still works when sensors and actuators are real. This paper builds an inexpensive physical AIoT loop in which an edge computer watches a game screen with a camera and issues keystrokes through a hardware-emulated keyboard. On that loop a simulation-trained deep Q-network agent that scores far above human level inside the simulator collapses to near-random performance once deployed, a 1160% drop relative to human skill. The same platform, however, lets the identical algorithm learn directly from real camera and keyboard experience and reach roughly 38% of human performance after ten million steps. The result is a concrete, low-cost, safety-safe benchmark that makes both the gap and the residual feasibility of real-world RL measurable for AIoT researchers.

Core claim

A camera-plus-emulated-keyboard Breakout platform costing under 400 dollars reveals that a simulation-trained DQN agent loses essentially all of its super-human performance when the observation and action streams become physical, yet the same algorithm can still learn from scratch on the physical loop and reach about 38% of human-normalized score after ten million steps. That pair of numbers quantifies both a severe Sim-to-Real gap and residual algorithmic robustness for AIoT-style RL.

What carries the argument

The real-world AIoT benchmark loop: a fixed-crop camera observation, a Teensy-based hardware keyboard that holds each key for 66 ms, and a serial reward channel, all running at real-time frame rates with no step-wait. This closed physical loop supplies the states, actions and rewards that let the Sim-to-Real gap and algorithmic robustness be measured as differences in human-normalized score.

Load-bearing premise

The claim that a single-camera, fixed-crop, 66-millisecond key-press video-game loop is a good enough stand-in for the sensor-actuator-edge interactions found in general AIoT systems such as buildings, traffic or power grids.

What would settle it

Train and evaluate the same DQN agent on a second, independently built copy of the platform whose camera crop, lighting or key-hold timing differ by only a few pixels or milliseconds; if the reported 1160% gap and 0.38 HNS numbers disappear or reverse, the measured gap is an artifact of the particular hardware instance rather than a transferable AIoT property.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The manuscript introduces a low-cost (<USD 400 hardware) real-world platform intended as a Sim-to-Real benchmark for RL in AIoT. An edge-device agent observes a host computer’s screen via USB camera and issues actions through a hardware-emulated keyboard (Teensy), while rewards and termination signals travel over a serial link. Three configurations are compared: pure simulation (ALE/Gymnasium), camera-input only, and the full real-world loop. Using Breakout and standard DQN for 10 M steps, the authors report that a simulation-trained agent reaches an IQM human-normalized score (HNS) of ~11.6 in simulation yet collapses to ~0.01 when deployed on the physical platform (a 1160 % degradation relative to human level), while direct real-world training reaches ~0.38 HNS. Additional ablations show brittleness to a 5-pixel crop shift. The platform, code, and evaluation protocol are released open-source.

Significance. Affordable, safety-constrained physical platforms for RL remain scarce outside robotics; the open-source release, sub-USD-400 bill of materials, and careful reporting (100-episode tests, 95 % CIs on mean/median/IQM/optimality gap, learning curves) are genuine contributions that lower the barrier to real-world experimentation. The measured 11.6 o0.01 HNS collapse and the 0.38 HNS real-world training ceiling are concrete, reproducible numbers for this setup. If the community adopts the platform even as a vision-based edge-control testbed, it can support systematic study of domain shift, latency, and sample efficiency. The broader claim that the platform constitutes a “universal” or “substantial foundation” for general AIoT (smart buildings, traffic, power grids) is more aspirational and currently rests on an architectural analogy rather than multi-domain evidence.

major comments (3)
  1. [Abstract, §§I–III, VII] Abstract, §§I–III and VII assert that the platform supplies a “universal” / “substantial foundation” for Sim-to-Real evaluation of RL in AIoT systems (smart buildings, traffic, power grids, etc.). The only empirical support is a single-camera, fixed-crop, 66 ms key-press Breakout loop (Algs. 2–3, Figs. 3–6). No non-game task, no continuous-control or multi-agent AIoT scenario, and no quantitative comparison to existing AIoT simulators or physical plants is provided. The dominant failure modes (visual domain shift in Fig. 12, total collapse under a 5-pixel crop shift in Figs. 13–16) are classic vision-based RL brittleness, not generic AIoT phenomena. Either the generality claim must be substantially scoped to “vision-based edge control with discrete actuation” or additional evidence (at least one non-game task or a mapping study) is required; otherwise the carefully measured gap remains v
  2. [§V, §V-G] §V reports results exclusively for Breakout under a single algorithm (DQN) and a 10 M-step budget (far below the 200 M-frame ALE standard). While the authors correctly note computational constraints, a platform intended as a general benchmark needs at least a second game or a second algorithm (e.g., PPO/A3C, already mentioned in §VIII) to demonstrate that the measured gap and real-world trainability are not artifacts of one environment–algorithm pair. Without that, the claim that the platform “validates” Sim-to-Real evaluation for AIoT (§V-G) is overstated.
  3. [§IV-A, Fig. 4, Algs. 1–3] The observation pipeline relies on a manually chosen, fixed crop of the camera FOV (Fig. 4, Alg. 1) and a hand-tuned 66 ms key-press duration. These free parameters are acknowledged but never ablated systematically beyond the 5-pixel shift. Because the entire Sim-to-Real gap is attributed to “real-world factors,” the contribution of these design choices versus irreducible physical noise/latency must be quantified; otherwise it is unclear how much of the 1160 % degradation is platform-specific rather than fundamental.
minor comments (5)
  1. [Abstract, §V-E] The 1160 % figure is arithmetic (HNS 11.6 – 0.01 ≈ 11.6 relative to human baseline 1.0) but is repeatedly phrased as “performance degradation relative to the human-level performance,” which can be misread as a 11.6-fold drop from human level rather than from the agent’s own sim performance. Clarify the reference point in the abstract and §V-E.
  2. [Figs. 7–11, 14–16] Figs. 7–11 and 14–16 use non-standard Unicode axis labels that render poorly; replace with ordinary text or vector fonts for production.
  3. [Table I] Table I’s “˜✓” notation for IoT status of Atari/robots is unexplained in the caption; define the symbols explicitly.
  4. [§II-C] §II-C notes that the Physical Atari platform [26] appeared after the present work was completed; a short side-by-side comparison of design goals (robotic joystick vs. AIoT keyboard/serial) would help readers situate the contribution.
  5. [Throughout] Minor typos: “HV AC” → “HVAC”, “A VWC” → “AVWC”, “tokamak” capitalization consistency, and “Sim-to-real” vs. “Sim-to-Real” throughout.

Circularity Check

0 steps flagged

No circularity: headline gap and robustness numbers are direct empirical measurements of trained agents against an external ALE human baseline, not quantities forced by definition, fit, or self-citation.

full rationale

The paper’s central results (simulation-trained DQN HNS ≈ 11.6 collapsing to ≈ 0.01 on the physical platform, yielding the stated 1160 % degradation; real-world training reaching ≈ 0.38 HNS after 10 M steps) are obtained by running the identical DQN algorithm under three concrete system configurations (simulation, camera-only real-world input, full camera-plus-emulated-keyboard real-world) and converting raw episode scores to Human Normalized Score via the standard external formula HNS = (Agent − Random)/(Human − Random) taken from the ALE literature. No parameter is fitted to a subset of the target data and then re-presented as a prediction; no uniqueness theorem or ansatz is imported from the authors’ prior work; and the platform itself is an engineering construction whose performance numbers are independently measurable by any group that rebuilds the hardware. The derivation chain therefore contains no self-definitional loops, fitted-input-as-prediction steps, or load-bearing self-citations. The only potential concern is the unvalidated claim that the Breakout loop is a universal AIoT proxy, but that is an external-validity issue, not circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 1 invented entities

The central empirical claims rest on standard RL machinery plus a small set of engineering choices that define the physical loop. No new physical constants or free parameters are fitted to produce the gap number; the gap is measured after those choices are fixed. The main modeling assumption is that the chosen loop is representative of broader AIoT.

free parameters (3)
  • key-press duration = 66 ms
    Fixed at 66 ms from human keystroke statistics and 4-frame action repeat at 60 FPS; directly sets the real-world control frequency.
  • training budget = 10 M steps
    10 million frames chosen for wall-clock feasibility on the physical platform rather than the ALE-standard 200 M; all performance numbers are reported at this budget.
  • camera crop region = manual crop matching game window
    Manually aligned rectangle that extracts the game window from the raw camera image; a 5-pixel shift later destroys the policy, showing sensitivity to this choice.
axioms (4)
  • domain assumption Atari games under the ALE interface are Markov decision processes whose rewards equal score differences.
    Inherited from Bellemare et al. and used without re-derivation for both simulation and real-world reward signals.
  • domain assumption Average-human scores published for ALE Breakout constitute a valid external baseline for human-normalized score.
    Used to convert raw scores into HNS and thereby into the 1160% gap figure.
  • ad hoc to paper A single USB camera plus a hardware-emulated keyboard with fixed press duration sufficiently captures the sensor-actuator-edge interaction of general AIoT systems for benchmarking purposes.
    Stated as the design basis in Sections I and IV; if false, the measured gap does not generalize beyond video-game score maximization.
  • domain assumption Standard DQN with experience replay, target network, and epsilon-greedy exploration is an adequate algorithm for demonstrating feasibility of real-world training.
    Only algorithm evaluated; future-work section notes that more sample-efficient methods remain to be tested.
invented entities (1)
  • real-world input system / real-world system configurations independent evidence
    purpose: Intermediate and full physical loops that isolate camera noise versus full sensing-plus-actuation effects.
    Defined operationally by the hardware and algorithms in Section IV; they are constructed artifacts rather than postulated unobservables, and the GitHub release supplies an independent handle.

pith-pipeline@v1.1.0-grok45 · 33015 in / 2996 out tokens · 41852 ms · 2026-07-14T12:43:47.573704+00:00 · methodology

0 comments
read the original abstract

Reinforcement learning (RL) is commonly employed to enhance the performance of autonomous systems, including the Autonomous Internet of Things (AIoT). However, the trial-and-error nature of RL, when conducted in real-world environments, is costly and hazardous in some scenarios. Consequently, the majority of RL research is conducted in simulation. This reliance introduces challenges related to the Sim-to-Real transferability. Evaluating the Sim-to-Real algorithmic robustness and the Sim-to-Real gap is a critical prerequisite for research aimed at improving RL performance in the real world. Therefore, industries such as robotics have developed concurrent simulation and physical platforms to facilitate this research. However, a universal Sim-to-Real benchmark platform for AIoT does not currently exist. To address these concerns, we developed a real-world AIoT platform for studying RL in AIoT. On this platform, an agent deployed on an edge device plays video games on a separate host computer via a hardware-emulated keyboard, guided by vision input. This platform uses commercially available components costing less than USD 400, together with two computers. Because the system's objective is game score maximization, it inherently mitigates safety risks associated with real-world RL deployments. Experimental results show the simulation-trained agent suffers a 1160% performance degradation relative to the human-level performance after real-world deployment, indicating a significant Sim-to-Real gap. Direct real-world training using the deep Q-network (DQN) algorithm achieves approximately 38% of human-level performance after 10 million training steps, demonstrating the feasibility of RL under real-world conditions. These results suggest that the proposed Sim-to-Real benchmark platform provides a substantial foundation for qualitative and quantitative evaluations of RL in real-world AIoT systems.

Figures

Figures reproduced from arXiv: 2607.10309 by Albert Y. Zomaya, Omid Tavallaie, Rongping Zhou, Shuaijun Chen.

Figure 1
Figure 1. Figure 1: The agent-environment interaction in an AIoT system. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: This configuration is hereafter referred to as the real [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 3
Figure 3. Figure 3: A PC with a camera input playing video games. (a) A [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Field of view of the camera. (a) The camera input’s [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: A PC with a camera and an actuator playing video [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: A PC with a camera and an emulated keyboard playing [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Training performance of three agents across three [PITH_FULL_IMAGE:figures/full_fig_p012_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: The performance of three agents across three systems. [PITH_FULL_IMAGE:figures/full_fig_p012_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: The performance of two agents. These two agents are [PITH_FULL_IMAGE:figures/full_fig_p013_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: The performance of two agents in the real-world input [PITH_FULL_IMAGE:figures/full_fig_p013_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: The performance of two agents in the real-world [PITH_FULL_IMAGE:figures/full_fig_p014_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: The difference between the cropped field of view [PITH_FULL_IMAGE:figures/full_fig_p014_12.png] view at source ↗
Figure 14
Figure 14. Figure 14: The performance of the agent in the real-world input [PITH_FULL_IMAGE:figures/full_fig_p015_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: The cropped area is moved up 5 pixels in the real [PITH_FULL_IMAGE:figures/full_fig_p015_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: The performance of the agent in the real-world [PITH_FULL_IMAGE:figures/full_fig_p015_16.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

46 extracted references · 7 canonical work pages

  1. [1]

    R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed., Cambridge, MA, USA: The MIT Press, 2018

    2018

  2. [2]

    Goodfellow, Y

    I. Goodfellow, Y . Bengio and A. Courville, Deep Learning, Cambridge, MA, USA: The MIT Press, 2016. [Online]. Available: https://www. deeplearningbook.org

    2016

  3. [3]

    Learning dexterous in-hand manipulation,

    M. Andrychowicz et al., “Learning dexterous in-hand manipulation,” The International Journal of Robotics Research, vol. 39, no. 1, pp. 3-20, Jan. 2020, doi: 10.1177/0278364919887447

    doi:10.1177/0278364919887447 2020

  4. [4]

    Scalable deep reinforcement learning for vision- Based robotic manipulation,

    D. Kalashnikov et al., “Scalable deep reinforcement learning for vision- Based robotic manipulation,” in Proceedings of The 2nd Conference on Robot Learning, Proceedings of Machine Learning Research, 2018. [On- line]. Available: https://proceedings.mlr.press/v87/kalashnikov18a.html

    2018

  5. [5]

    A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play,

    D. Silver et al., “A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play,” Science, vol. 362, no. 6419, pp. 1140-1144, 2018, doi: 10.1126/science.aar6404

    doi:10.1126/science.aar6404 2018

  6. [6]

    Training language models to follow instructions with human feedback,

    L. Ouyang et al., “Training language models to follow instructions with human feedback,” in Advances in Neural Information Process- ing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds. vol. 35, 2022, pp. 27730-27744. [On- line]. Available: https://proceedings.neurips.cc/paper files/paper/2022/ file/b1efde53be364a73914f58805...

    2022

  7. [7]

    Magnetic control of tokamak plasmas through deep reinforcement learning,

    J. Degrave et al., “Magnetic control of tokamak plasmas through deep reinforcement learning,” Nature, vol. 602, no. 7897, pp. 414-419, Feb. 2022, doi: 10.1038/s41586-021-04301-9

    doi:10.1038/s41586-021-04301-9 2022

  8. [8]

    That ‘internet of things’ thing,

    K. Ashton, “That ‘internet of things’ thing,” RFID journal, vol. 22, no. 7, pp. 97-114, Jun. 2009. [Online]. Available: https://www.rfidjournal.com/ expert-views/that-internet-of-things-thing/73881/

    2009

  9. [9]

    Autonomous IoT in a Few Words,

    “Autonomous IoT in a Few Words,” Accessed: Oct. 11, 2024. [Online]. Available: https://hypervsn.com/blog/autonomous-iot-in-a-few-words. html

    2024

  10. [10]

    Edge Computing: Vision and Challenges,

    W. Shi, J. Cao, Q. Zhang, Y . Li, and L. Xu, “Edge Computing: Vision and Challenges,” IEEE Internet of Things Journal, vol. 3, no. 5, pp. 637- 646, 2016, doi: 10.1109/JIOT.2016.2579198

    doi:10.1109/jiot.2016.2579198 2016

  11. [11]

    Internet of Things: A Survey on Enabling Technologies, Protocols, and Applications,

    A. Al-Fuqaha, M. Guizani, M. Mohammadi, M. Aledhari, and M. Ayyash, “Internet of Things: A Survey on Enabling Technologies, Protocols, and Applications,” IEEE Communications Surveys & Tu- torials, vol. 17, no. 4, pp. 2347-2376, Fourth quarter 2015, doi: 10.1109/COMST.2015.2444095

    doi:10.1109/comst.2015.2444095 2015

  12. [12]

    Deep Reinforcement Learning for Autonomous Internet of Things: Model, Ap- plications and Challenges,

    L. Lei, Y . Tan, K. Zheng, S. Liu, K. Zhang, and X. Shen, “Deep Reinforcement Learning for Autonomous Internet of Things: Model, Ap- plications and Challenges,” IEEE Communications Surveys & Tutorials, vol. 22, no. 3, pp. 1722-1760, 2020, doi: 10.1109/COMST.2020.2988367

    work page doi:10.1109/comst.2020.2988367 2020

  13. [13]

    Human-level control through deep reinforcement learning,

    V . Mnih et al., “Human-level control through deep reinforcement learning,” Nature, vol. 518, no. 7540, pp. 529-533, Feb. 2015, doi: 10.1038/nature14236. 17

    doi:10.1038/nature14236 2015

  14. [14]

    Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents,

    M. C. Machado, M. G. Bellemare, E. Talvitie, J. Veness, M. Hausknecht, and M. Bowling, “Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents,” Journal of Artificial Intelligence Research, vol. 61, pp. 523-562, 2018

    2018

  15. [15]

    Data center cooling using model-predictive con- trol,

    N. Lazic et al., “Data center cooling using model-predictive con- trol,” in Advances in Neural Information Processing Systems, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, Eds. Montr ´eal, Canada, vol. 31, 2018. [On- line]. Available: https://proceedings.neurips.cc/paper files/paper/2018/ file/059fdcd96baeb75112f09fa1...

    2018

  16. [16]

    Exploring Deep Reinforcement Learn- ing for Holistic Smart Building Control,

    X. Ding, A. Cerpa, and W. Du, “Exploring Deep Reinforcement Learn- ing for Holistic Smart Building Control,” ACM Trans. Sen. Netw., vol. 20, no. 3, May 2024, Art. no. 70, doi: 10.1145/3656043

    work page doi:10.1145/3656043 2024

  17. [17]

    Broad Reinforcement Learning for Supporting Fast Autonomous IoT,

    X. Wei, J. Zhao, L. Zhou, and Y . Qian, “Broad Reinforcement Learning for Supporting Fast Autonomous IoT,” IEEE Internet of Things Journal, vol. 7, no. 8, pp. 7010-7020, Aug. 2020, doi: 10.1109/JIOT.2020.2980198

    work page doi:10.1109/jiot.2020.2980198 2020

  18. [18]

    River Flow Path Control With Reinforcement Learning,

    D. Liu et al., “River Flow Path Control With Reinforcement Learning,” in Proceedings of 2021 IEEE International Conference on Autonomous Systems (ICAS), Montreal, QC, Canada, 11-13 Aug. 2021, pp. 1-5, doi: 10.1109/ICAS49788.2021.9551113

    work page doi:10.1109/icas49788.2021.9551113 2021

  19. [19]

    A Reinforcement Learning Empowered Cooperative Control Approach for IIoT-Based Virtually Coupled Train Sets,

    H. Wang et al., “A Reinforcement Learning Empowered Cooperative Control Approach for IIoT-Based Virtually Coupled Train Sets,” IEEE Transactions on Industrial Informatics, vol. 17, no. 7, pp. 4935-4945, 2021, doi: 10.1109/TII.2020.3024946

    work page doi:10.1109/tii.2020.3024946 2021

  20. [20]

    Deep Reinforcement Learning for Smart Home Energy Management,

    L. Yu et al., “Deep Reinforcement Learning for Smart Home Energy Management,” IEEE Internet of Things Journal, vol. 7, no. 4, pp. 2751- 2762, 2020, doi: 10.1109/JIOT.2019.2957289

    work page doi:10.1109/jiot.2019.2957289 2020

  21. [21]

    Deep Reinforcement Learning for Internet of Things: A Comprehensive Survey,

    W. Chen, X. Qiu, T. Cai, H. N. Dai, Z. Zheng, and Y . Zhang, “Deep Reinforcement Learning for Internet of Things: A Comprehensive Survey,” IEEE Communications Surveys & Tutorials, vol. 23, no. 3, pp. 1659-1692, 2021, doi: 10.1109/COMST.2021.3073036

    work page doi:10.1109/comst.2021.3073036 2021

  22. [22]

    OpenAI Gym,

    G. Brockman et al., “OpenAI Gym,” 2016, arXiv: 1606.01540

    Pith/arXiv arXiv 2016

  23. [23]

    Gymnasium: A Standard Interface for Reinforcement Learning Environments,

    M. Towers et al., “Gymnasium: A Standard Interface for Reinforcement Learning Environments,” arXiv: 2407.17032

    Pith/arXiv arXiv

  24. [24]

    The arcade learning environment: An evaluation platform for general agents,

    M. G. Bellemare, Y . Naddaf, J. Veness, and M. Bowling, “The arcade learning environment: An evaluation platform for general agents,” Journal of Artificial Intelligence Research, vol. 47, pp. 253-279, 2013

    2013

  25. [25]

    CALE: Continuous Arcade Learning Environment,

    J. Farebrother and P. S. Castro, “CALE: Continuous Arcade Learning Environment,” in The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. [Online]. Available: https://openreview.net/forum?id=vlUK2h1Nvw

    2024

  26. [26]

    Physical Atari: A Robust and Accessible Platform for Real-time Rein- forcement Learning on Robots,

    K. Javed, J. Modayil, G. Kennickell, R. S. Sutton, and J. Carmack, “Physical Atari: A Robust and Accessible Platform for Real-time Rein- forcement Learning on Robots,” 2026, arXiv: 2606.19357

    Pith/arXiv arXiv 2026

  27. [27]

    MuJoCo: A physics engine for model-based control,

    E. Todorov, T. Erez, and Y . Tassa, “MuJoCo: A physics engine for model-based control,” 7-12 Oct. 2012 2012, pp. 5026-5033, doi: 10.1109/IROS.2012.6386109

    doi:10.1109/iros.2012.6386109 2012

  28. [28]

    Open-sourcing MuJoCo

    T. Yuval and T. Saran. “Open-sourcing MuJoCo.” Accessed: Oct. 12, 2024. [Online]. Available: https://deepmind.google/discover/blog/ open-sourcing-mujoco/

    2024

  29. [29]

    Zakka, Y

    K. Zakka, Y . Tassa and MuJoCo Menagerie Contributors, MuJoCo Menagerie: A collection of high-quality simulation models for MuJoCo. (2022). DeepMind. Accessed: Apr. 19, 2025. [Online]. Available: https: //github.com/google-deepmind/mujoco menagerie

    2022

  30. [30]

    Continuous control with deep reinforcement learning,

    T. P. Lillicrap et al., “Continuous control with deep reinforcement learning,” 2019, arXiv: 1509.02971

    Pith/arXiv arXiv 2019

  31. [31]

    PyBullet, a Python module for physics simu- lation for games, robotics and machine learning

    E. Coumans and Y . Bai. “PyBullet, a Python module for physics simu- lation for games, robotics and machine learning.” (2016-2021). Accessed: Jun. 1, 2025. [Online]. Available: https://pybullet.org

    2016

  32. [32]

    Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments,

    M. Mittal et al., “Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments,” IEEE Robotics and Automation Letters, vol. 8, no. 6, pp. 3740-3747, 2023, doi: 10.1109/LRA.2023.3270034

    doi:10.1109/lra.2023.3270034 2023

  33. [33]

    Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning,

    T. Yu et al., “Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning,” 2021, arXiv: 1910.10897

    Pith/arXiv arXiv 2021

  34. [34]

    RLBench: The Robot Learning Benchmark & Learning Environment,

    S. James, Z. Ma, D. R. Arrojo, and A. J. Davison, “RLBench: The Robot Learning Benchmark & Learning Environment,” IEEE Robotics and Automation Letters, vol. 5, no. 2, pp. 3019-3026, 2020, doi: 10.1109/LRA.2020.2974707

    doi:10.1109/lra.2020.2974707 2020

  35. [35]

    Over- coming the Sim-to-Real Gap: Leveraging Simulation to Learn to Explore for Real-World RL,

    A. Wagenmaker, K. Huang, L. Ke, K. Jamieson, and A. Gupta, “Over- coming the Sim-to-Real Gap: Leveraging Simulation to Learn to Explore for Real-World RL,” in Advances in Neural Information Processing Systems, A. Globerson et al., Eds., vol. 37, 2024, pp. 78715-78765. [On- line]. Available: https://proceedings.neurips.cc/paper files/paper/2024/ file/8fa06...

    2024

  36. [36]

    Flappy Bird,

    “Flappy Bird,” Wikipedia, Accessed: Oct. 11, 2024. [Online]. Available: https://en.wikipedia.org/wiki/Flappy Bird

    2024

  37. [37]

    A3C Keras FlappyBird,

    S. Singh, “A3C Keras FlappyBird,” GitHub Repository, Accessed: Oct. 11, 2024. [Online]. Available: https://github.com/sourabhv/FlapPyBird

    2024

  38. [38]

    Breakout,

    “Breakout,” Farama Foundation, Accessed: Jun. 21, 2025. [Online]. https://ale.farama.org/environments/breakout/

    2025

  39. [39]

    Teensy 4.1 Development Board

    P. Stoffregen, “Teensy 4.1 Development Board” (2020) Accessed: May 18, 2025. [Online]. Available: https://www.pjrc.com/store/teensy41.html

    2020

  40. [40]

    etherkey

    F. Mauracher, “etherkey.” (2023) Accessed: Apr. 18, 2025. [Online]. Available: https://github.com/Flowm/etherkey

    2023

  41. [41]

    Ob- servations on Typing from 136 Million Keystrokes,

    V . Dhakal, A. M. Feit, P. O. Kristensson, and A. Oulasvirta, “Ob- servations on Typing from 136 Million Keystrokes,” in Proceed- ings of the 2018 CHI Conference on Human Factors in Com- puting Systems, Montreal QC, Canada, 2018. [Online]. Available: https://doi.org/10.1145/3173574.3174220

    doi:10.1145/3173574.3174220 2018

  42. [42]

    Deep Reinforcement Learning at the Edge of the Statistical Precipice,

    R. Agarwal, M. Schwarzer, P. S. Castro, A. C. Courville, and M. Bellemare, “Deep Reinforcement Learning at the Edge of the Statistical Precipice,” presented at the Advances in Neural Information Processing Systems, 2021. [Online]. Available: https://proceedings.neurips.cc/paper files/paper/2021/file/f514cec81cb148559cf475e7426eed5e-Paper.pdf

    2021

  43. [43]

    real world program,

    R. Zhou, “real world program,” GitHub, Accessed: Apr. 29, 2026. [On- line]. Available: https://github.com/RongpingZhou/real world program

    2026

  44. [44]

    Stable-Baselines3: Reliable Reinforcement Learning Imple- mentations,

    A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann, “Stable-Baselines3: Reliable Reinforcement Learning Imple- mentations,” Journal of Machine Learning Research, vol. 22, no. 268, pp. 1-8, 2021. [Online]. Available: https://jmlr.org/papers/v22/20-1364.html

    2021

  45. [45]

    Asynchronous Methods for Deep Reinforcement Learn- ing,

    V . Mnih et al., “Asynchronous Methods for Deep Reinforcement Learn- ing,” in Proceedings of The 33rd International Conference on Machine Learning, Proceedings of Machine Learning Research, 2016. [Online]. Available: https://proceedings.mlr.press/v48/mniha16.html

    2016

  46. [46]

    Proximal Policy Optimization Algorithms,

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal Policy Optimization Algorithms,” 2017, arXiv: 1707.06347

    Pith/arXiv arXiv 2017