{"id":"d1c6cf2e-b848-489a-a5eb-2ad72c44b5cd","arxiv_id":"2507.21638","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Assistax provides fast JAX-based assistive robotics environments with trainable humanoid partners, and shows current RL baselines have a coordination gap when facing unseen human preferences.","lead":"Assistax is a new GPU-accelerated benchmark for teaching robots to help simulated humans in five care tasks, using JAX to make training fast. It adds an ad-hoc teamwork pipeline where a robot must cooperate with unseen simulated human partners, and reports that current RL algorithms struggle with this.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline coordination gap is measured only under an adversarial 5-agent/625-agent preference split, so without a uniform-split control it supports an OOD preference-generalization finding, not a general AHT coordination gap.","rationale":"I read the paper in good faith. The engineering contribution is credible: a JAX/MJX-based suite with released code, tuned MARL baselines, a 630-policy partner population, and a reproducible AHT pipeline. The runtime claims are appropriately hedged for wall-clock training, and the open-loop speedups are clearly labeled as an upper bound. The most load-bearing weakness is not the absence of real human validation, which the Limitations section honestly acknowledges, but the mismatch between the abstract's general claim of a 'coordination gap' and the only supporting experiment: an intentionally extreme preference-based split with no quantitative uniform-split control. The authors admit the gap shrinks under uniform sampling, and their preference-only analysis indicates the gap is driven by preference returns. This makes the headline result closer to an OOD generalization finding than a robust AHT coordination finding. The reader's weakest_assumption focused on MARL policies as proxies for human partners, which is a related external-validity concern but not the most direct threat to the internal empirical claim. My proposed check would settle whether the gap is an artifact of the split, and the paper's own text suggests it would be. Since the reader already issued a conditional verdict and my concern is the same class of overclaim, I do not recommend changing the verdict; the fix is to add the uniform-split control and narrow the abstract's wording accordingly.","tokens_in":19480,"tokens_out":4092,"duration_ms":54071,"concrete_test":"Rerun the PPO AHT experiment of Figure 5 with a random uniform 5-agent training subset sampled from the full 630-agent population, keeping the same test set of withheld agents, the same preference-reward settings, and the same PPOAHT hyperparameters in Table 9. Report the train/test gap for task returns and preference-only returns, and also break down test returns by preference axis (speed, force, touch penalty). If the uniform-split gap is substantially smaller than the adversarial-split gap (for example, less than half on the normalized preference-only returns), then the headline gap should be reframed as an out-of-distribution preference generalization result rather than a general AHT coordination gap.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim is that the AHT pipeline 'reveals a coordination gap for existing RL algorithms on unseen partners with novel preference combinations' (Abstract). In Section 4.2, however, the headline result in Figure 5 is produced by training on only 5 agents selected to have the highest speed-preference weighting and testing on 625 agents drawn from the full population. The text explicitly notes that the gap 'drops significantly when including more agents in the training set or when sampling train and test agents uniformly from the population,' but no quantitative result for the uniform-split control is shown. Without that control, the measured gap conflates two distinct phenomena: (1) an AHT-specific difficulty in coordinating with an unseen partner policy, and (2) a standard out-of-distribution preference generalization failure, since the robot was trained on a narrow high-speed-preference slice and tested across the full preference distribution. The authors' own Figure 10 and Appendix B.3 attribute the gap 'almost entirely' to preference returns, which supports the OOD interpretation. The abstract-level claim is therefore stronger than the evidence presented; the uniform-split comparison is load-bearing for whether Assistax demonstrates a general coordination gap or a specific OOD preference-generalization gap.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"Assistax introduces a JAX/MJX-based benchmark suite of five assistive robotics tasks with a torque-controlled robot and a trainable humanoid partner. The paper contributes MARL baselines (IPPO, MAPPO, MASAC with feed-forward and recurrent variants), a population of 630 pre-trained humanoid policies per task, and an AHT evaluation protocol in which a robot is trained on a subset of partner policies and evaluated on withheld partners. Headline results are large speedups (up to 412x open-loop and roughly 25x wall-clock), competitive MARL learning curves, and a 'coordination gap' under an intentionally extreme preference-based train/test split. The AHT claim is that existing RL algorithms perform substantially worse on unseen partners with novel preference combinations.","tokens_in":19678,"tokens_out":6350,"duration_ms":72805,"significance":"If the coordination-gap result survives a matched train/test distribution control, Assistax would be a valuable high-throughput benchmark: it is one of the few 3D continuous AHT testbeds, it ships pre-trained partner policies via Hugging Face, it reports bootstrap CIs and extensive hyperparameter sweeps, and its JAX implementation makes principled repeatable evaluation practical. As it stands, the headline empirical claim needs re-scoping or an additional control before it can be regarded as evidence of a general AHT coordination gap rather than an out-of-distribution preference-generalization effect.","major_comments":[{"comment":"The central claim that existing RL algorithms exhibit a coordination gap is supported only by an extreme 5-agent/625-agent preference split, with the 5 training agents selected for the highest speed-preference weighting. The text concedes that the gap 'drops significantly when including more agents in the training set or when sampling train and test agents uniformly from the population,' but no quantitative results, curves, or CIs for a uniform-split control are provided anywhere in the paper. Given that Figure 10 and Appendix B.3 attribute the gap 'almost entirely' to preference returns, and given that preferences are provided in the observations (Section 3.2), the natural reading is that the experiment demonstrates out-of-distribution preference-generalization failure, not a general AHT coordination deficit. Please add the uniform-split control as a primary result, report the gap magnitude under both splits, and either temper the abstract/conclusion claim or present the extreme-split result explicitly as a stress test.","section":"Section 4.2, Figures 5 and 10, Appendix B.3"},{"comment":"The AHT evaluation in the main text is PPO-only; SAC appears in Appendix B.2 at 6 million timesteps rather than the 40 million used for PPO and omits the Bed Bath task. Because the abstract says 'existing RL algorithms' (plural), the paper should either include the SAC AHT results at a comparable training budget in the main text, or explicitly scope the coordination-gap claim to the tested PPO baseline.","section":"Section 4.2, Figures 5 and 9"},{"comment":"The paper acknowledges in Limitations that 'MARL policies do not capture the full complexity of human behaviour,' yet the benchmark's motivation and the term 'humanoid partner' may lead readers to infer conclusions about human-robot coordination. This is not a fatal flaw for a simulation benchmark, but the contribution should be phrased consistently as a benchmark for coordination with simulated MARL policies; the external-validity caveat should appear in the introduction or abstract rather than only in the limitations paragraph.","section":"Limitations and Section 3.4"}],"minor_comments":[{"comment":"The population description says 630 humanoid agents per task but then says 610 preference combinations are sampled and 7 disability combinations are used; the relationship between 610, 7, and 630 should be stated explicitly.","section":"Section 3.4"},{"comment":"The text says 168 hyperparameter combinations are searched for each baseline-environment pair, but Appendix C describes a single sweep and says 'We run one very extensive sweep across our MARL baselines'; clarify whether the sweep is per baseline-environment pair or per baseline.","section":"Section 4.1 and Appendix C"},{"comment":"The caption says 'across all five tasks' but shows only four tasks; either add Bed Bath or correct the caption.","section":"Figure 9"},{"comment":"There are several typos, including 'hyperparemeter', 'evaluete', and 'metalearning paradigm'; the nomenclature for the tooth brushing task is also inconsistent ('Teethbrushing' in Table 8 versus 'Tooth Brushing' elsewhere).","section":"Appendix C and Section 4.2"},{"comment":"The 8.3-hour Assistive Gym runtime comparison is described as 'indicative rather than controlled'; please provide at least the hardware/software configuration and the exact command or script used for the comparison, since this number is likely to be quoted.","section":"Section 4.3"},{"comment":"The wall-clock estimate extrapolates from 5 IPPO updates to 610 updates; it would be more reliable to report measured total wall-clock time for at least one full run.","section":"Appendix A.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for RLJ as a benchmark contribution. The main risk is overclaiming the coordination-gap result; the uniform-split control is easy to add and should be required. I would not reject on the basis of the current OOD-split issue because the authors already hint at the control and the benchmark infrastructure itself is valuable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe useful thing in this paper is the testbed, not the headline result. Assistax is the first benchmark I know of that combines JAX/MJX hardware acceleration, five continuous-control assistive tasks with a trainable humanoid partner, and an AHT evaluation pipeline with a population of pre-trained partners. That combination is genuinely new relative to Assistive Gym, Human-Robot Gym, JaxRobotarium, and JaxMARL. The engineering looks solid: tuned MARL baselines (IPPO, MAPPO, MASAC), bootstrap confidence intervals, hyperparameter sweeps with a fixed budget per algorithm, and code on GitHub. The runtime numbers are useful even if indicative, and the Limitations section honestly flags fidelity trade-offs and the fact that MARL policies are not full models of human behavior.\n\nThe soft spot is the AHT claim. The abstract says the pipeline 'reveals a coordination gap for existing RL algorithms on unseen partners with novel preference combinations.' The headline result comes from an extreme split: training on five agents selected for the highest speed-preference weighting and testing on 625 agents sampled from the full population. The paper's own text says the gap drops significantly with a larger training set or uniform sampling, but no quantitative uniform-split control is shown. Without that control, the measured gap is indistinguishable from ordinary out-of-distribution preference generalization: the robot saw a narrow preference slice at train time and the full distribution at test time. Figure 10, which attributes the gap almost entirely to preference returns, supports this reading. The abstract-level claim is therefore stronger than the evidence. The fix is straightforward: reframe the finding as an OOD preference-generalization gap and add the uniform-split and larger-training-set numbers prominently.\n\nMinor issues: the 412x speedup is open-loop only, and the 25x wall-clock number is uncontrolled (the authors flag this). The supplement omits the Hugging Face link and a commit hash; those should be added for reproducibility.\n\nOverall, this is a serious, well-engineered benchmark paper. The central contribution—the suite and pipeline—holds up. The AHT interpretation needs adjustment. I would send it to peer review, but require the reframing and the missing control before acceptance. If I worked in assistive MARL, I would cite it for the benchmark itself, not for the coordination-gap result.\n\nBest,","headline":"Solid benchmark engineering and a genuinely useful testbed; the AHT coordination-gap claim is oversold and needs a uniform-split control before it can stand.","tokens_in":20297,"tokens_out":3640,"would_cite":true,"duration_ms":35356,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Standard RL robot policies lose coordination with unseen humanoid partners with novel preferences; a new GPU-accelerated benchmark, Assistax, surfaces this gap in continuous 3D assistive tasks.","keywords":["Multi-Agent Reinforcement Learning","Ad-Hoc Teamwork","Zero-Shot Coordination","Assistive Robotics","GPU-accelerated simulation","JAX","MuJoCo MJX","benchmark"],"falsifier":"Re-run the AHT evaluation with human-driven partners instead of held-out MARL policies: teleoperate the humanoid (or use behavioural clones of human demonstrations) across the same preference conditions, and check whether a robot trained on the five speed-extreme partners shows a comparable, preference-concentrated return drop. If it does not, the coordination gap is an artefact of the MARL proxy rather than a property of human-robot coordination; a cheaper complementary check is to confirm the paper's own claim that the gap disappears when train and test preference distributions are matched, which would show the effect lives entirely in the distribution shift.","tokens_in":1966,"feed_emoji":"🤖","tokens_out":2703,"duration_ms":161902,"temperature":0.7,"pith_summary":"Assistax is a benchmark for a specific open problem: a caregiving robot must complete a physical assistance task with a person it has never coordinated with before. The paper argues that this should be studied as ad-hoc teamwork in continuous 3D environments, and that past benchmarks were either too simple (grid worlds like Overcooked) or too slow (CPU physics simulators) to do so. Its central claims are that five assistive tasks — scratching, tooth brushing, feeding, bed bathing, and arm repositioning — can be simulated and trained entirely on GPUs via JAX and MuJoCo MJX (up to 412× faster simulation than the CPU-based Assistive Gym), and that when a robot is trained on a small, deliberately skewed set of pre-trained humanoid partners and evaluated on 625 held-out partners, standard PPO and SAC algorithms exhibit a coordination gap concentrated in adapting to unfamiliar preference combinations. If these claims hold, the benchmark gives RL researchers a high-throughput testbed for embodied ad-hoc teamwork, with hundreds of pre-trained humanoid policies released so that results can be reproduced without retraining partners.","feed_headline":"412× faster assistive-robot benchmark exposes a coordination gap","feed_subtitle":"GPU-fast assistive-robotics suite finds PPO and SAC agents lose coordination with unfamiliar humanoid partners.","key_machinery":"The mechanism that carries the argument is the preference reward system combined with the hardware-accelerated training loop. Each task's shared reward is augmented by a preference reward built from three axes — preferred contact-force range, preferred end-effector speed range, and a penalty for new contacts after the initial touch — normalised into a fixed budget of 40–50% of the maximum per-step reward so different preference combinations are comparable. Sampling 610 preference combinations and 7 disability settings during population training yields the 630 humanoid policies; the AHT evaluation then fixes a pre-trained humanoid, trains the robot on a small training population $\\Pi_{\\text{train}}$ under the objective $\\pi^{i*}(\\Pi_{\\text{train}}) = \\arg\\max_{\\pi^i} \\mathbb{E}_{\\pi^{-i} \\sim U(\\Pi_{\\text{train}})}[\\sum_t \\gamma^t R]$, and measures the expected return $M_{\\Pi_{\\text{test}}}$ against withheld partners. The whole loop — simulator, rewards, observations, and policy updates — runs in JAX with MuJoCo MJX collocated on the GPU, which is what makes a 630-policy population and 16-seed evaluations affordable.","core_discovery":"On its own terms, the paper's central claim is that the first ad-hoc-teamwork benchmark for continuous 3D assistive robotics with two distinct embodied agents is now possible, and that building it exposes a measurable coordination gap in existing RL algorithms. The authors co-train robot and humanoid agents with multi-agent RL on five tasks, then construct a population of 630 humanoid policies per task whose diversity comes from emergent co-training conventions, sampled preference combinations, and disability settings. Discarding the robot policies, they train a fresh robot policy (PPO or SAC) against a small, deliberately out-of-distribution training subset — five partners with the highest speed-preference weighting — and score it against the 625 withheld partners. Returns drop substantially on the withheld set, and the paper's preference-only analysis shows the drop is almost entirely explainable by the robot's failure to adapt to novel preference combinations, while the gap shrinks significantly when training and test populations are drawn from similar distributions.","pith_inferences":["Because the evaluation proxy is MARL checkpoints, the measured gap is plausibly a lower bound on the difficulty of real human-robot ad-hoc teamwork, since real humans vary more than this population; an obvious next step, not run in the paper, is the same protocol with teleoperated or demonstration-driven human partners.","The paper's diagnosis that the gap is almost entirely preference adaptation suggests a targeted remedy the authors do not test: since preferences are already passed in the observations, explicitly preference-conditioned or meta-learned robot policies trained across the preference distribution may close much of the gap.","The rigid-body, primitive-geometry fidelity trade-off means the benchmark measures coordination capability, not physical realism; tasks with deformable contact such as washing with cloth remain out of reach until the simulator scales better with collision counts.","Population diversity is induced but not optimised, and the paper's own cross-play matrices show some mutually incompatible partner strategies while the authors point to best-response diversity training as future work; actively diversifying the partner population could make the AHT test set harder and would likely change the reported gap."],"forward_implications":["A 40-million-step IPPO training run takes about 20 minutes on one A100 GPU with 1024 parallel environments, versus roughly 8.3 hours in the CPU-based Assistive Gym, so assistive-robotics RL research becomes as fast to iterate on as grid-world benchmarks.","Standard RL baselines as configured here (PPO and SAC with the paper's tuned hyperparameters) are not sufficient for zero-shot coordination with out-of-distribution human partners in continuous assistive tasks.","The coordination gap is concentrated in the preference-reward component rather than the task component, so the bottleneck is adapting to how the partner wants the task done, not the physics of the task itself.","The gap shrinks markedly when training and test partner populations are more similar, so benchmark users must report both the training-population composition and the test gap to get meaningful AHT numbers.","Releasing the 630 pre-trained humanoid policies lets other researchers evaluate new AHT algorithms against identical withheld partners, making results reproducible without rerunning population training."],"supporting_citations":[{"why":"Source of the five assistive task designs, the co-trained humanoid, and the CPU baseline for the speed comparisons.","marker":"Erickson et al., 2019"},{"why":"The physics engine whose JAX reimplementation (MJX) carries all state transitions in the benchmark.","marker":"Todorov et al., 2012"},{"why":"Supplies the Brax humanoid model used as the simulated human partner.","marker":"Freeman et al., 2021"},{"why":"Provides the JaxMARL implementations of IPPO, MAPPO, and MASAC used as baselines and for population training.","marker":"Rutherford et al., 2024b"},{"why":"Defines ad-hoc teamwork and the meta-learning paradigm the AHT evaluation follows.","marker":"Mirsky et al., 2022"},{"why":"The original ad-hoc teamwork formulation that motivates the coordination objective.","marker":"Stone et al., 2010"},{"why":"Formalises the robust AHT training objective the paper instantiates.","marker":"Rahman et al., 2024"},{"why":"The PPO algorithm used for the PPO AHT baseline.","marker":"Schulman et al., 2017"},{"why":"The SAC algorithm used for the SAC AHT baseline.","marker":"Haarnoja et al., 2018"}],"fun_headline_variants":["412× faster assistive RL benchmark stumbles on new partners","Assistive robot benchmark: faster sims, but hidden coordination gap","GPU-accelerated assistive benchmark exposes partner blind spot","Assistax: assistive robots struggle with unfamiliar partners"],"cache_read_input_tokens":22400,"weakest_assumption_plain":"The load-bearing premise is that the 630-policy MARL population, with diversity induced by co-training, preference sampling, and disability settings, is a meaningful proxy for real human partners; the authors flag in their own limitations that MARL policies do not capture the full complexity of human behaviour, so if the population is unrepresentative the measured coordination gap describes coordination with MARL checkpoints, not with people.","fun_headline_variants_meta":{"raw":{"variants":["412× faster assistive RL benchmark stumbles on new partners","Assistive robot benchmark: faster sims, but hidden coordination gap","GPU-accelerated assistive benchmark exposes partner blind spot","Assistax: assistive robots struggle with unfamiliar partners"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000572,"raw_usage":{"total_tokens":2773,"prompt_tokens":1083,"completion_tokens":1690,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":699,"completion_tokens_details":{"reasoning_tokens":1619}},"tokens_in":699,"tokens_out":1690,"duration_ms":15267,"temperature":1.0,"reasoning_tokens":1619,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T12:31:07.250479+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the AHT evaluation with human-driven partners instead of held-out MARL policies: teleoperate the humanoid (or use behavioural clones of human demonstrations) across the same preference conditions, and check whether a robot trained on the five speed-extreme partners shows a comparable, preference-concentrated return drop. If it does not, the coordination gap is an artefact of the MARL proxy rather than a property of human-robot coordination; a cheaper complementary check is to confirm the paper's own claim that the gap disappears when train and test preference distributions are matched, which would show the effect lives entirely in the distribution shift.","supporting_citations":[],"review_version":1}