Measure the Sim-to-Real Gap: Designing an Affordable Real-World Benchmark Platform for Reinforcement Learning in AIoT Systems

Albert Y. Zomaya; Omid Tavallaie; Rongping Zhou; Shuaijun Chen

Measure the Sim-to-Real Gap: Designing an Affordable Real-World Benchmark Platform for Reinforcement Learning in AIoT Systems

T0 review · 3 major / 5 minor · reviewed 2026-07-14 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A sub-$400 camera-and-keyboard platform measures a 1160% Sim-to-Real collapse for RL game agents while proving real-world training still works.

desk verdict Clean empirical gap numbers on a cheap physical Breakout loop; the “universal AIoT benchmark” framing is the soft part, not the measurements. read the letter →

arxiv 2607.10309 v2 pith:NVB2BHDZ submitted 2026-07-11 cs.AI

Rongping Zhou , Omid Tavallaie , Shuaijun Chen , Albert Y. Zomaya This is my paper · ORCID

classification cs.AI

keywords reinforcementlearningSim-to-RealgapAIoTreal-worldbenchmarkdeepQ-networkedgecomputingAtariBreakouthuman-normalizedscore

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Most reinforcement-learning research for Autonomous Internet of Things systems stays inside simulators because real trials are expensive and sometimes dangerous. Without a shared physical testbed, it is hard to know how large the gap between simulation and reality actually is, or whether an algorithm that works in simulation still works when sensors and actuators are real. This paper builds an inexpensive physical AIoT loop in which an edge computer watches a game screen with a camera and issues keystrokes through a hardware-emulated keyboard. On that loop a simulation-trained deep Q-network agent that scores far above human level inside the simulator collapses to near-random performance once deployed, a 1160% drop relative to human skill. The same platform, however, lets the identical algorithm learn directly from real camera and keyboard experience and reach roughly 38% of human performance after ten million steps. The result is a concrete, low-cost, safety-safe benchmark that makes both the gap and the residual feasibility of real-world RL measurable for AIoT researchers.

What carries the argument

The real-world AIoT benchmark loop: a fixed-crop camera observation, a Teensy-based hardware keyboard that holds each key for 66 ms, and a serial reward channel, all running at real-time frame rates with no step-wait. This closed physical loop supplies the states, actions and rewards that let the Sim-to-Real gap and algorithmic robustness be measured as differences in human-normalized score.

What would settle it

Train and evaluate the same DQN agent on a second, independently built copy of the platform whose camera crop, lighting or key-hold timing differ by only a few pixels or milliseconds; if the reported 1160% gap and 0.38 HNS numbers disappear or reverse, the measured gap is an artifact of the particular hardware instance rather than a transferable AIoT property.

Watch

Extended reading notes

Core claim

A camera-plus-emulated-keyboard Breakout platform costing under 400 dollars reveals that a simulation-trained DQN agent loses essentially all of its super-human performance when the observation and action streams become physical, yet the same algorithm can still learn from scratch on the physical loop and reach about 38% of human-normalized score after ten million steps. That pair of numbers quantifies both a severe Sim-to-Real gap and residual algorithmic robustness for AIoT-style RL.

Load-bearing premise

The claim that a single-camera, fixed-crop, 66-millisecond key-press video-game loop is a good enough stand-in for the sensor-actuator-edge interactions found in general AIoT systems such as buildings, traffic or power grids.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The useful core is concrete: a sub-$400 camera + Teensy-emulated-keyboard setup, three matched systems (sim, camera-only, full real), and careful numbers. After 10 M DQN steps the sim agent sits at ~11.6 IQM HNS in simulation and collapses to ~0.01 on the physical loop—the 1160% figure is just that arithmetic against the human baseline. Direct real-world training reaches ~0.38 HNS. They report 100-episode tests with CIs, learning curves, wall-clock (0.3 vs 8–10 days), and a 5-pixel crop-shift ablation that kills performance. Code and build notes are public. That is real engineering evidence, not a rehash of MuJoCo-to-robot papers or the later Physical Atari box.

What is new is the deliberately motor-free edge loop aimed at AIoT-style sensing/actuation/timing, plus the first quantitative HNS gap on that hardware. For groups that cannot buy robots or buildings, this is a practical, safe way to study vision noise and control latency.

The soft spot is proportion, not fraud. Only Breakout, only DQN, 10 M frames, manual crop. More importantly, the abstract and conclusion sell a “substantial foundation” / near-universal AIoT benchmark. The support is the minimal sensor–edge–actuator diagram; no non-game task, no comparison to building/traffic/grid systems. The failures they measure (domain shift, crop brittleness) are classic vision RL, not generic AIoT dynamics. That overclaim should be dialed back; it does not erase the platform or the gap numbers for this setup.

Math and baselines are standard ALE HNS; related work is fair. No circular fitting. This is for people who need a cheap physical RL testbed or who write sim-to-real methods and want a reproducible stress case. It deserves a serious referee as a platform paper—expect them to demand multi-game/algorithm checks and a narrower claim. I would engage with the hardware and the measured collapse; I would not treat the AIoT-universality sentence as established.

Referee Report

3 major / 5 minor

Summary. The manuscript introduces a low-cost (<USD 400 hardware) real-world platform intended as a Sim-to-Real benchmark for RL in AIoT. An edge-device agent observes a host computer’s screen via USB camera and issues actions through a hardware-emulated keyboard (Teensy), while rewards and termination signals travel over a serial link. Three configurations are compared: pure simulation (ALE/Gymnasium), camera-input only, and the full real-world loop. Using Breakout and standard DQN for 10 M steps, the authors report that a simulation-trained agent reaches an IQM human-normalized score (HNS) of ~11.6 in simulation yet collapses to ~0.01 when deployed on the physical platform (a 1160 % degradation relative to human level), while direct real-world training reaches ~0.38 HNS. Additional ablations show brittleness to a 5-pixel crop shift. The platform, code, and evaluation protocol are released open-source.

Significance. Affordable, safety-constrained physical platforms for RL remain scarce outside robotics; the open-source release, sub-USD-400 bill of materials, and careful reporting (100-episode tests, 95 % CIs on mean/median/IQM/optimality gap, learning curves) are genuine contributions that lower the barrier to real-world experimentation. The measured 11.6 o0.01 HNS collapse and the 0.38 HNS real-world training ceiling are concrete, reproducible numbers for this setup. If the community adopts the platform even as a vision-based edge-control testbed, it can support systematic study of domain shift, latency, and sample efficiency. The broader claim that the platform constitutes a “universal” or “substantial foundation” for general AIoT (smart buildings, traffic, power grids) is more aspirational and currently rests on an architectural analogy rather than multi-domain evidence.

major comments (3)

[Abstract, §§I–III, VII] Abstract, §§I–III and VII assert that the platform supplies a “universal” / “substantial foundation” for Sim-to-Real evaluation of RL in AIoT systems (smart buildings, traffic, power grids, etc.). The only empirical support is a single-camera, fixed-crop, 66 ms key-press Breakout loop (Algs. 2–3, Figs. 3–6). No non-game task, no continuous-control or multi-agent AIoT scenario, and no quantitative comparison to existing AIoT simulators or physical plants is provided. The dominant failure modes (visual domain shift in Fig. 12, total collapse under a 5-pixel crop shift in Figs. 13–16) are classic vision-based RL brittleness, not generic AIoT phenomena. Either the generality claim must be substantially scoped to “vision-based edge control with discrete actuation” or additional evidence (at least one non-game task or a mapping study) is required; otherwise the carefully measured gap remains v
[§V, §V-G] §V reports results exclusively for Breakout under a single algorithm (DQN) and a 10 M-step budget (far below the 200 M-frame ALE standard). While the authors correctly note computational constraints, a platform intended as a general benchmark needs at least a second game or a second algorithm (e.g., PPO/A3C, already mentioned in §VIII) to demonstrate that the measured gap and real-world trainability are not artifacts of one environment–algorithm pair. Without that, the claim that the platform “validates” Sim-to-Real evaluation for AIoT (§V-G) is overstated.
[§IV-A, Fig. 4, Algs. 1–3] The observation pipeline relies on a manually chosen, fixed crop of the camera FOV (Fig. 4, Alg. 1) and a hand-tuned 66 ms key-press duration. These free parameters are acknowledged but never ablated systematically beyond the 5-pixel shift. Because the entire Sim-to-Real gap is attributed to “real-world factors,” the contribution of these design choices versus irreducible physical noise/latency must be quantified; otherwise it is unclear how much of the 1160 % degradation is platform-specific rather than fundamental.

minor comments (5)

[Abstract, §V-E] The 1160 % figure is arithmetic (HNS 11.6 – 0.01 ≈ 11.6 relative to human baseline 1.0) but is repeatedly phrased as “performance degradation relative to the human-level performance,” which can be misread as a 11.6-fold drop from human level rather than from the agent’s own sim performance. Clarify the reference point in the abstract and §V-E.
[Figs. 7–11, 14–16] Figs. 7–11 and 14–16 use non-standard Unicode axis labels that render poorly; replace with ordinary text or vector fonts for production.
[Table I] Table I’s “˜✓” notation for IoT status of Atari/robots is unexplained in the caption; define the symbols explicitly.
[§II-C] §II-C notes that the Physical Atari platform [26] appeared after the present work was completed; a short side-by-side comparison of design goals (robotic joystick vs. AIoT keyboard/serial) would help readers situate the contribution.
[Throughout] Minor typos: “HV AC” → “HVAC”, “A VWC” → “AVWC”, “tokamak” capitalization consistency, and “Sim-to-real” vs. “Sim-to-Real” throughout.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: headline gap and robustness numbers are direct empirical measurements of trained agents against an external ALE human baseline, not quantities forced by definition, fit, or self-citation.

full rationale

The paper’s central results (simulation-trained DQN HNS ≈ 11.6 collapsing to ≈ 0.01 on the physical platform, yielding the stated 1160 % degradation; real-world training reaching ≈ 0.38 HNS after 10 M steps) are obtained by running the identical DQN algorithm under three concrete system configurations (simulation, camera-only real-world input, full camera-plus-emulated-keyboard real-world) and converting raw episode scores to Human Normalized Score via the standard external formula HNS = (Agent − Random)/(Human − Random) taken from the ALE literature. No parameter is fitted to a subset of the target data and then re-presented as a prediction; no uniqueness theorem or ansatz is imported from the authors’ prior work; and the platform itself is an engineering construction whose performance numbers are independently measurable by any group that rebuilds the hardware. The derivation chain therefore contains no self-definitional loops, fitted-input-as-prediction steps, or load-bearing self-citations. The only potential concern is the unvalidated claim that the Breakout loop is a universal AIoT proxy, but that is an external-validity issue, not circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

The central empirical claims rest on standard RL machinery plus a small set of engineering choices that define the physical loop. No new physical constants or free parameters are fitted to produce the gap number; the gap is measured after those choices are fixed. The main modeling assumption is that the chosen loop is representative of broader AIoT.

free parameters (3)

key-press duration = 66 ms
Fixed at 66 ms from human keystroke statistics and 4-frame action repeat at 60 FPS; directly sets the real-world control frequency.
training budget = 10 M steps
10 million frames chosen for wall-clock feasibility on the physical platform rather than the ALE-standard 200 M; all performance numbers are reported at this budget.
camera crop region = manual crop matching game window
Manually aligned rectangle that extracts the game window from the raw camera image; a 5-pixel shift later destroys the policy, showing sensitivity to this choice.

assumptions (4)

domain assumption Atari games under the ALE interface are Markov decision processes whose rewards equal score differences.
Inherited from Bellemare et al. and used without re-derivation for both simulation and real-world reward signals.
domain assumption Average-human scores published for ALE Breakout constitute a valid external baseline for human-normalized score.
Used to convert raw scores into HNS and thereby into the 1160% gap figure.
ad hoc to paper A single USB camera plus a hardware-emulated keyboard with fixed press duration sufficiently captures the sensor-actuator-edge interaction of general AIoT systems for benchmarking purposes.
Stated as the design basis in Sections I and IV; if false, the measured gap does not generalize beyond video-game score maximization.
domain assumption Standard DQN with experience replay, target network, and epsilon-greedy exploration is an adequate algorithm for demonstrating feasibility of real-world training.
Only algorithm evaluated; future-work section notes that more sample-efficient methods remain to be tested.

invented entities (1)

real-world input system / real-world system configurations independent evidence
purpose: Intermediate and full physical loops that isolate camera noise versus full sensing-plus-actuation effects.
Defined operationally by the hardware and algorithms in Section IV; they are constructed artifacts rather than postulated unobservables, and the GitHub release supplies an independent handle.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Measure the Sim-to-Real Gap: Designing an Affordable Real-World Benchmark Platform for Reinforcement Learning in AIoT Systems." pith.science (2026). https://pith.science/paper/NVB2BHDZ

@misc{pith2026260710309,
  author       = {Pith},
  title        = {Pith review of: Measure the Sim-to-Real Gap: Designing an Affordable Real-World Benchmark Platform for Reinforcement Learning in AIoT Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NVB2BHDZ}},
  note         = {Machine review of arXiv:2607.10309}
}

read the original abstract

Reinforcement learning (RL) is commonly employed to enhance the performance of autonomous systems, including the Autonomous Internet of Things (AIoT). However, the trial-and-error nature of RL, when conducted in real-world environments, is costly and hazardous in some scenarios. Consequently, the majority of RL research is conducted in simulation. This reliance introduces challenges related to the Sim-to-Real transferability. Evaluating the Sim-to-Real algorithmic robustness and the Sim-to-Real gap is a critical prerequisite for research aimed at improving RL performance in the real world. Therefore, industries such as robotics have developed concurrent simulation and physical platforms to facilitate this research. However, a universal Sim-to-Real benchmark platform for AIoT does not currently exist. To address these concerns, we developed a real-world AIoT platform for studying RL in AIoT. On this platform, an agent deployed on an edge device plays video games on a separate host computer via a hardware-emulated keyboard, guided by vision input. This platform uses commercially available components costing less than USD 400, together with two computers. Because the system's objective is game score maximization, it inherently mitigates safety risks associated with real-world RL deployments. Experimental results show the simulation-trained agent suffers a 1160% performance degradation relative to the human-level performance after real-world deployment, indicating a significant Sim-to-Real gap. Direct real-world training using the deep Q-network (DQN) algorithm achieves approximately 49% of human-level performance after 10 million training steps, demonstrating the feasibility of RL under real-world conditions. These results suggest that the proposed Sim-to-Real benchmark platform provides a substantial foundation for qualitative and quantitative evaluations of RL in real-world AIoT systems.