Switch4EAI: Leveraging Console Game Platform for Benchmarking Robotic Athletics

Donghoon Baek; Jeonghwan Kim; Sehoon Ha; Seungeun Rho; Tianyu Li; Wontaek Kim

REVIEW 3 major objections 4 minor 15 references

Switch4EAI: Leveraging Console Game Platform for Benchmarking Robotic Athletics

T0 review · 3 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper shows that a Nintendo Switch and Just Dance can serve as a low-cost, human-comparable benchmark for whole-body robot control, with a Unitree G1 scoring 5,707 against a human's 9,361 on three routines.

desk verdict A clever and genuinely new benchmark idea, but the headline human-robot score is built on an unvalidated hand-controller metric; worth refereeing as a demo/position, not as a quantitative benchmark yet. read the letter →

arxiv 2508.13444 v1 pith:IVTEYT7K submitted 2025-08-19 cs.RO

Tianyu Li , Jeonghwan Kim , Wontaek Kim , Donghoon Baek , Seungeun Rho , Sehoon Ha This is my paper

classification cs.RO

keywords embodiedAIbenchmarkingwhole-bodycontrolhumanoidrobotsJustDanceNintendoSwitchmotionretargetingrobotathletics

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a consumer game console can be repurposed as an evaluation instrument for whole-body robot control. The authors build a cheap, reconfigurable pipeline that feeds in-game dance choreography from Just Dance on a Nintendo Switch to a humanoid robot, then uses the game's built-in scoring as the metric. On a Unitree G1, the robot followed three easy routines and averaged 5,707 points against a human's 9,361. If the approach holds, the same platform could provide standardized, frequently refreshed, and human-comparable benchmarks for embodied AI without requiring custom test rigs. The reported score comes from Joy-Con controllers attached to the robot's hands, so the metric may primarily reflect hand motion rather than full-body fidelity.

What carries the argument

The pipeline is the key mechanism: Streaming Module (HDMI capture of the Switch output) feeds the MoCap Module (ROMP, which reconstructs 3D SMPL poses from monocular RGB video); the Motion Retargeting Module (GMR) maps those poses from the human shape to the robot's morphology; the robot's whole-body control policy (GMT) executes the retargeted motion; and Joy-Con controllers in the robot's hands let the game itself score the attempt. The load-bearing step is the closed loop through the game's own scoring system, which turns off-the-shelf entertainment hardware into a quantitative benchmark without custom sensing or instrumentation.

What would settle it

Run the robot through a dance routine with Joy-Cons on its hands while deliberately degrading only its leg and torso tracking performance (for example, freezing the lower-body policy); if the in-game score remains roughly unchanged, the score is not measuring whole-body performance and the benchmark's central metric fails.

Watch

Extended reading notes

Core claim

The central claim is that commercial motion-sensing console games can serve as physically grounded benchmarks for robotic athletics. The authors assemble a pipeline that captures the Switch's screen, reconstructs the dancer's 3D pose from monocular video using ROMP with the SMPL body model, retargets that motion to the robot using GMR, and executes it with the open-source GMT whole-body controller while a pair of Joy-Con controllers attached to the robot's hands feeds the robot's movements back into the game. The in-game score then becomes a quantitative athletic metric. In a feasibility validation on a Unitree G1 humanoid across three 2-star Just Dance 2020 routines, the robot averaged 5,70

Load-bearing premise

The method relies on Just Dance's own score being a valid measure of whole-body athletic performance, even though the Joy-Cons are strapped to the robot's hands and the game may primarily sense hand motion.

Editorial extensions

If this is right

Whole-body robot control policies can be evaluated directly in a commercial game environment, with the game's generated score serving as a quantitative baseline.
Because the scoring system was designed for human players, the same routines yield a direct human-versus-robot athletic comparison on a single scale.
The platform costs around $400 USD, is portable, and receives frequent content updates, so new dance routines become new benchmark tasks without additional data collection.
The same capture-reconstruct-retarget-score loop is morphology-agnostic in principle and could be applied to other Switch motion games to test balance, endurance, or sports-like skills.
The reported robot score of 5,707 versus the human 9,361 is a feasibility baseline, not a ceiling; improved controllers or hardware execution should raise it.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

My inference: because Just Dance generates its score from the Joy-Cons in the robot's hands, the result as reported primarily certifies arm and hand coordination; adding Joy-Cons to the legs or waist, or combining the game score with camera-based pose error, would let the same platform grade torso and leg fidelity too.
My inference: the pipeline implies that game content itself can act as an ever-growing benchmark suite, so robot progress could be tracked over time by having a robot replay older and newer songs and comparing per-song scores as a leaderboard.
My inference: the in-game score could be used backwards, as a real-world reward signal during policy fine-tuning, turning the game into both the evaluation and the training environment for whole-body control.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this paper has a good idea and a weak demonstration. The idea—use a commercial motion-sensing console game as a cheap, standardized whole-body control benchmark with built-in human comparison—is new as far as I know, and it has real appeal: the Switch costs ~$400, games keep updating, and the same scoring works for humans and different robot morphologies. That's a concrete contribution to benchmarking, and the authors are right that existing sim and mocap benchmarks don't do direct human comparison.

The demo itself is straightforward and honest: HDMI capture, ROMP for 3D pose, GMR for retargeting, GMT for whole-body control, Joy-Cons strapped to the hands. The robot plays three 2-star Just Dance 2020 routines and gets a mean score of 5,707 versus 9,361 for a human. The integration works. That's a legitimate feasibility result.

The soft spot is exactly what the stress-test note says: the in-game score comes from Joy-Con controllers in the robot's hands. Just Dance scoring is based on handheld controller motion, so the score mostly reflects arm/hand timing and orientation, not torso, legs, or whole-body coordination. The paper offers no evidence that the score correlates with full-body motion fidelity. So the quantitative baseline doesn't do what the paper claims—it doesn't measure 'robotic athletic performance' in the whole-body sense. The human comparison is also thin: one mean per condition, no error bars, no trial counts, no description of the human protocol, and code withheld until acceptance. That last point is annoying but common; still, it makes the headline number impossible to audit.

Minor issues: an empty citation in the related work and a few typos, but those don't affect the argument.

On balance: the benchmark concept is worth pursuing and worth publishing as a demo or position. The authors should either reframe the paper around feasibility and drop the quantitative athletic claim, or add validation: compare the Joy-Con score against mocap-based whole-body error, report variability across trials and songs, and release the code and data. As is, the central claim is not yet supported.

I'd send this to peer review rather than desk reject it—a good reviewer can push the authors to either validate or rescope. I wouldn't rely on the score as a benchmark until that's done.

Referee Report

3 major / 4 minor

Summary. The paper proposes Switch4EAI, a pipeline that uses the Nintendo Switch game Just Dance as a benchmark for whole-body robot athletic performance. The pipeline captures the Switch video stream, reconstructs the on-screen dancer's 3D pose with ROMP, retargets the motion to a Unitree G1 humanoid with GMR, and executes it with the open-source GMT whole-body controller. A pair of Joy-Con controllers is attached to the robot's hands, and the in-game score is used to compare the robot with a human player. The authors validate the pipeline on three 'easy' Just Dance routines, reporting a mean robot score of 5,707 versus 9,361 for a human, and argue that commercial game platforms offer a low-cost, dynamic, and directly comparable benchmark for embodied AI.

Significance. If the proposed metric is shown to capture whole-body athletic performance, this would be a valuable contribution: it offers a standardized, low-cost, real-hardware evaluation with a built-in human baseline, addressing a real gap in the field where athletic benchmarks are mostly simulation-based and rarely compared with human performance. The paper's strengths are its use of open-source components (ROMP, GMR, GMT), a real-hardware demonstration on a humanoid, and the idea of using an external, commercially maintained game as a source of benchmark tasks. However, the significance currently hinges on an unvalidated assumption that the Just Dance score, generated by hand-held Joy-Cons, is a valid proxy for whole-body athletic fidelity.

major comments (3)

[§Pipeline and Validation (in-game scoring)] The central human-robot comparison (robot mean 5,707 vs. human 9,361) rests entirely on the Just Dance in-game score. The score is produced from Joy-Con controllers attached to the robot's hands, and the manuscript offers no evidence that this score reflects whole-body motion fidelity, torso/leg involvement, or global coordination. Just Dance's motion scoring is known to rely primarily on Joy-Con accelerometer/gyroscope input, so the reported gap may reflect only arm/hand tracking rather than athletic whole-body control. Since the framework is explicitly proposed as a whole-body evaluation, this is load-bearing. I request a validation experiment: compare the in-game score against a quantitative whole-body motion-similarity metric (e.g., from the reconstructed SMPL or retargeted robot motion), or an ablation where Joy-Cons are held still while performing deliberate whole-body movements, t
[§Pipeline and Validation (quantitative baseline)] The manuscript reports only a single mean score per condition over three routines. There are no per-song scores, no number of trials, no variance or error bars, and no description of the human baseline protocol. For a benchmarking claim, these omissions are not cosmetic: a reader cannot tell whether the robot-human gap is statistically meaningful, whether any trial failed (e.g., a fall yielding a zero score), or whether the human baseline is representative (one subject? many? experience level?). Please report full per-song and per-trial data, trial counts, standard deviations, and a precise protocol for the human comparison, including how the human was instructed and how many human subjects or attempts were used.
[§Pipeline and Validation (reproducibility)] The statement 'We will open source our code after this paper gets accepted' is in tension with the paper's stated goal of providing a 'standardized, dynamic, and physically grounded evaluation framework.' Without releasing the streaming, mocap, retargeting, and controller-integration code—or at least a sufficiently detailed protocol—the proposed benchmark cannot be reproduced or extended by other groups. This is central to the benchmarking contribution, not a peripheral implementation detail. Please make the code available with the revision and include exact game settings (song selection, difficulty, Joy-Con calibration procedure, and any filters applied to the reconstructed pose).

minor comments (4)

[Throughout] There are numerous typographical errors: 'W e', 'T o', 'Conlusion', 'V alidation', 'Switch4EAI(Switch-for-Embodied-AI)' missing a space, and 'future work to for benchmarking' in the abstract. Please proofread carefully.
[References] Reference [10] is incomplete (no venue, no full author list), 'gymnastic movement []' has an empty citation, and reference [15] is an unversioned GitHub URL. Please provide stable identifiers or versioned releases.
[Figure 1 caption] The caption says Joy-Cons are 'attached to its hands,' which reinforces the concern in major comment 1. Consider describing this as a deliberate design choice and discussing its implications for what the score measures.
[Introduction] The claim that these platforms are 'designed to accommodate players of varying sizes' is unsupported; a standard Joy-Con does not measure body dimensions. Either provide evidence or soften this claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported robot and human scores are external game outputs, not quantities derived from or fitted to the paper's own pipeline.

full rationale

The paper's derivation chain is an experimental pipeline, not a formal derivation. Switch4EAI captures HDMI video, reconstructs SMPL poses with ROMP, retargets them with GMR, executes them with the GMT controller on a Unitree G1, and records the proprietary Just Dance score from Joy-Con controllers attached to the robot's hands. Each module is an external open-source component, and the reported robot mean (5,707) and human mean (9,361) are direct game outputs, not quantities fitted to or derived from the pipeline's own assumptions. The same game supplies both the task and the score, but that is a design property of the benchmark and does not make the measurement equivalent to an input: the score is generated by Nintendo's closed algorithm from controller motion and was not constructed from the paper's models. The only co-authored reference ([4], on whole-body teleoperation) appears as background related work and is not load-bearing. The manuscript's own limitations—code only to be released after acceptance, and the statement that the result is not an upper bound—affect reproducibility and interpretation, not circularity. The main substantive weakness, that Just Dance's score may chiefly reflect hand/controller motion rather than whole-body fidelity, is a construct-validity concern about the benchmark, not a circularity: it does not reduce the reported comparison to the pipeline's inputs by construction. Therefore no circular step can be exhibited, and the score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No new free parameters are fit; the pipeline relies on several pretrained external models and a game-specific scoring proxy. The last axiom is the most fragile because it determines whether the benchmark measures athleticism or just arm motion.

assumptions (4)

domain assumption ROMP reconstructs accurate 3D SMPL poses from monocular video of Just Dance
The MoCap Module relies on ROMP; accuracy of downstream retargeting depends on this. No validation against game-internal pose data is provided.
domain assumption GMR retargets SMPL poses to the Unitree G1 morphology without significant distortion
Motion Retargeting Module uses GMR; poor retargeting would break execution. The paper does not quantify retarget error.
domain assumption The GMT whole-body controller tracks the retargeted motions in real time on the G1
Validation uses GMT as an open-source policy; real-time tracking is required for game scoring. No tracking error metrics are reported.
ad hoc to paper The Just Dance score from Joy-Cons is a valid proxy for whole-body athletic performance
Just Dance scoring is based on hand-held controller motion, not full-body pose. The paper presents no evidence tying scores to whole-body motion quality, yet uses the score as the quantitative benchmark.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Switch4EAI: Leveraging Console Game Platform for Benchmarking Robotic Athletics." pith.science (2026). https://pith.science/paper/IVTEYT7K

@misc{pith2026250813444,
  author       = {Pith},
  title        = {Pith review of: Switch4EAI: Leveraging Console Game Platform for Benchmarking Robotic Athletics},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IVTEYT7K}},
  note         = {Machine review of arXiv:2508.13444}
}

read the original abstract

Recent advances in whole-body robot control have enabled humanoid and legged robots to execute increasingly agile and coordinated movements. However, standardized benchmarks for evaluating robotic athletic performance in real-world settings and in direct comparison to humans remain scarce. We present Switch4EAI(Switch-for-Embodied-AI), a low-cost and easily deployable pipeline that leverages motion-sensing console games to evaluate whole-body robot control policies. Using Just Dance on the Nintendo Switch as a representative example, our system captures, reconstructs, and retargets in-game choreography for robotic execution. We validate the system on a Unitree G1 humanoid with an open-source whole-body controller, establishing a quantitative baseline for the robot's performance against a human player. In the paper, we discuss these results, which demonstrate the feasibility of using commercial games platform as physically grounded benchmarks and motivate future work to for benchmarking embodied AI.

Figures

Figures reproduced from arXiv: 2508.13444 by the authors.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

15 extracted references · 11 canonical work pages

[1]

Khazoom, S

C. Khazoom, S. Hong, M. Chignoli, E. Stanger-Jones, and S. Kim. T ailoring solution accuracy for fast whole-body model predictive control of legged robots.IEEE Robotics and Automation Letters, 2024

work page 2024
[2]

Li and P

H. Li and P . M. W ensing. Cafe-mpc: A cascaded-fidelity model predictive control framework with tuning-free whole-body control.IEEE Transactions on Robotics, 2024

work page 2024
[3]

J. Li, Z. Duan, J. Ma, and Q. Nguyen. Gait-net-augmented implicit kino-dynamic mpc for dynamic variable-frequency humanoid locomotion over discrete terrains.arXiv preprint arXiv:2502.02934, 2025

work page Pith review arXiv 2025
[4]

D. Baek, A. Purushottam, J. J. Choi, and J. Ramos. Whole-body bilateral teleoperation with multi-stage object parameter estimation for wheeled humanoid locomanipulation.arXiv preprint arXiv:2508.09846, 2025

work page Pith review arXiv 2025
[5]

W . Xie, J. Han, J. Zheng, H. Li, X. Liu, J. Shi, W . Zhang, C. Bai, and X. Li. Kungfubot: Physics-based humanoid whole-body control for learning highly-dynamic skills.arXiv preprint arXiv:2506.12851, 2025

arXiv 2025
[6]

Z. Chen, M. Ji, X. Cheng, X. Peng, X. B. Peng, and X. W ang. Gmt: General motion tracking for humanoid whole-body control.arXiv:2506.14770, 2025

arXiv 2025
[7]

J. Li, X. Cheng, T . Huang, S. Y ang, R.-Z. Qiu, and X. W ang. Amo: Adaptive motion optimization for hyper-dexterous humanoid whole-body control.arXiv preprint arXiv:2505.03738, 2025

arXiv 2025
[8]

Y . Ze, Z. Chen, J. P . Ara´ujo, Z. ang Cao, X. B. Peng, J. Wu, and C. K. Liu. T wist: Teleoperated whole-body imitation system.arXiv preprint arXiv:2505.02833, 2025

arXiv 2025

Show all 15 references

[9]

Mahmood, N

N. Mahmood, N. Ghorbani, N. F . Troje, G. Pons-Moll, and M. J. Black. AMASS: Archive of motion capture as surface shapes. InInternational Conference on Computer V ision, pages 5442–5451, Oct. 2019

2019
[10]

F . G. Harvey, M. Y urick, D. Nowrouzezahrai, and C. Pal. Robust motion in-betweening. 39(4), 2020

2020
[11]

Nintendo Switch, 2017

Nintendo. Nintendo Switch, 2017. URL https://www.nintendo.com/switch/. Video game console

2017
[12]

Just Dance, 2009

Ubisoft. Just Dance, 2009. URL https://www.ubisoft.com/en-us/game/just-dance/. Video game

2009
[13]

Y . Sun, Q. Bao, W . Liu, Y . Fu, B. Michael J., and T . Mei. Monocular, One-stage, Regression of Multiple 3D People. InICCV, 2021

2021
[14]

Loper, N

M. Loper, N. Mahmood, J. Romero, G. Pons-Moll, and M. J. Black. SMPL: A skinned multi-person linear model. ACM Trans. Graphics (Proc. SIGGRAPH Asia), 34(6):248:1–248:16, Oct. 2015

2015
[15]

Y . Ze, J. P . Ara´ujo, J. Wu, and C. K. Liu. Gmr: General motion retargeting, 2025. URL https://github.com/YanjieZe/GMR. GitHub repository. 3

2025

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

Khazoom, S

C. Khazoom, S. Hong, M. Chignoli, E. Stanger-Jones, and S. Kim. T ailoring solution accuracy for fast whole-body model predictive control of legged robots.IEEE Robotics and Automation Letters, 2024

work page 2024

[2] [2]

Li and P

H. Li and P . M. W ensing. Cafe-mpc: A cascaded-fidelity model predictive control framework with tuning-free whole-body control.IEEE Transactions on Robotics, 2024

work page 2024

[3] [3]

J. Li, Z. Duan, J. Ma, and Q. Nguyen. Gait-net-augmented implicit kino-dynamic mpc for dynamic variable-frequency humanoid locomotion over discrete terrains.arXiv preprint arXiv:2502.02934, 2025

work page Pith review arXiv 2025

[4] [4]

D. Baek, A. Purushottam, J. J. Choi, and J. Ramos. Whole-body bilateral teleoperation with multi-stage object parameter estimation for wheeled humanoid locomanipulation.arXiv preprint arXiv:2508.09846, 2025

work page Pith review arXiv 2025

[5] [5]

W . Xie, J. Han, J. Zheng, H. Li, X. Liu, J. Shi, W . Zhang, C. Bai, and X. Li. Kungfubot: Physics-based humanoid whole-body control for learning highly-dynamic skills.arXiv preprint arXiv:2506.12851, 2025

arXiv 2025

[6] [6]

Z. Chen, M. Ji, X. Cheng, X. Peng, X. B. Peng, and X. W ang. Gmt: General motion tracking for humanoid whole-body control.arXiv:2506.14770, 2025

arXiv 2025

[7] [7]

J. Li, X. Cheng, T . Huang, S. Y ang, R.-Z. Qiu, and X. W ang. Amo: Adaptive motion optimization for hyper-dexterous humanoid whole-body control.arXiv preprint arXiv:2505.03738, 2025

arXiv 2025

[8] [8]

Y . Ze, Z. Chen, J. P . Ara´ujo, Z. ang Cao, X. B. Peng, J. Wu, and C. K. Liu. T wist: Teleoperated whole-body imitation system.arXiv preprint arXiv:2505.02833, 2025

arXiv 2025