{"id":"3e8b0e23-b24b-4b4b-b24a-ab816d416860","arxiv_id":"2607.06052","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":3,"one_line_summary":"A force-aware humanoid benchmark pairs synchronized human motion-force data with simulation-based force replay to evaluate whole-body control policies under realistic physical disturbances.","lead":"ThorArena is a benchmark that evaluates humanoid robot control policies by replaying real-world interaction forces in simulation alongside motion-tracking tasks. It reveals performance gaps that standard force-free benchmarks miss, which matters for deploying humanoid robots in contact-rich settings.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"Thor2 trained on the same motion-force dataset used for benchmark evaluation, creating a train-on-test confound: the 'hidden performance differences' may partly reflect Thor2's familiarity with the exact force profiles rather than genuine force robustness.","rationale":"The reader correctly identified several issues (metric parameter sensitivity, no code release, self-benchmarking bias, no error bars, sim-to-real gap) and chose sim-to-real validity as the weakest assumption. The sim-to-real concern is legitimate but is explicitly acknowledged as future work and does not undermine the paper's in-simulation finding. The more load-bearing concern is the train-on-test confound: Thor2 is the authors' own policy trained on the same motion-force data that the benchmark replays, and it wins on all six tasks. This directly threatens the central claim because the observed 'hidden performance differences' under force could be inflated by Thor2's familiarity with the force profiles. The reader noted the self-benchmarking issue (point 3) but treated it as 'mild' rather than load-bearing. I elevate it because it specifically confounds the force-aware vs no-force comparison that is the paper's core contribution. The verdict remains CONDITIONAL — the benchmark infrastructure and data collection are valuable contributions regardless, but the headline finding needs a clean train/eval separation or cross-policy fine-tuning control to be fully convincing. If the authors can show the force-aware gap persists on held-out force profiles, the claim strengthens considerably.","tokens_in":9453,"tokens_out":1604,"duration_ms":101492,"concrete_test":"Re-run the benchmark evaluation using force profiles from a completely separate data collection session (different operators, different object instances, same six tasks) that Thor2 was never trained on. If Thor2's FATS advantage over the other policies shrinks substantially (e.g., the gap narrows from ~8-11 points to <3 points), the current 'hidden differences' are largely a train-on-test artifact. If the gap persists, the central claim is robust. Alternatively, fine-tune TWIST2 or GMT on the same motion-force dataset and re-evaluate; if the fine-tuned policy's force/no-force gap matches Thor2's, the benchmark measures training data exposure rather than intrinsic force robustness.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that force-aware evaluation reveals substantial performance differences hidden under no-force evaluation. The strongest evidence for this claim is Thor2's dominance under external forces (FATS 81.71 vs 70.04–73.78 for others, Table I) combined with the narrower spread under no-force (84.31 vs 78.43–80.96, Table III). However, the contributions section states: 'The paired motion–force data have been used to train the force-aware humanoid control policy Thor2.' The benchmark's force-replay protocol (§III.C) then replays these same recorded force sequences during evaluation. The paper never explicitly states that Thor2 was trained on a disjoint split from the evaluation set. Even if a split exists, the force profiles within each task are drawn from the same distribution (same task type, same sensor, same collection protocol), so Thor2 has a systematic familiarity advantage with the force characteristics that the other three policies lack. This matters because the central claim is about force-aware evaluation revealing genuine robustness differences. If Thor2's advantage under force is partly due to having trained on similar or identical force profiles, the 'hidden differences' are inflated. The no-force comparison does not fully control for this: under no-force, all policies see the same reference motions (which some may also have trained on), but the force component is unique to Thor2's training. The asymmetry — only Thor2 trained with force data — means the force/no-force delta is confounded with 'trained on force data' vs 'not trained on force data.' The paper would need to either (a) evaluate Thor2 on force profiles from a held-out distribution it never saw, or (b) fine-tune one of the other policies on the same force data to isolate the effect of force-aware training from force-aware evaluation.","agreement_with_reader":"partial"},"referee_report":{"model":"glm-5.2","summary":"The paper presents ThorArena, a benchmark for evaluating force-aware humanoid physical interaction. It contributes three components: (1) a real-world dataset of synchronized whole-body human motion and two-hand interaction forces across six contact-rich tasks (60 sequences per task, 360 total), (2) the Force-Aware Tracking Score (FATS) and complementary diagnostic metrics (robustness ratio, power overhead, survival rate), and (3) a simulation-based force-replay protocol with a unified policy-adapter interface that applies recorded interaction forces to simulated humanoid hand bodies during policy rollout. Four whole-body control policies (Thor2, TWIST2, GMT, SONIC) are evaluated under matched force and no-force conditions, showing wider performance spreads under external forces.","tokens_in":9582,"tokens_out":2261,"duration_ms":110567,"significance":"The synchronized motion-force dataset fills a genuine gap in the humanoid benchmarking literature, where existing resources provide kinematic trajectories but not paired interaction forces. The force-replay protocol and policy-adapter interface are practical design choices that lower the barrier to evaluating diverse policies under consistent disturbance conditions. The FATS metric provides a reasonable composite score that stratifies performance by force regime. The no-force vs. force comparison (Tables I–III) does demonstrate that survival rates and tracking spreads change under external forces, supporting the paper's motivation. The dataset is stated to be released, which adds reproducibility value.","major_comments":[{"comment":"§I, Contributions bullet 1; §IV, Tables I–III: The paper states that the paired motion-force data 'have been used to train the force-aware humanoid control policy Thor2,' but never specifies whether the evaluation sequences are disjoint from Thor2's training set. Thor2 ranks first on all six subtasks under external forces (Table I: FATS 81.71 vs. 70.04–73.78). The central claim that force-aware evaluation reveals 'substantial performance differences' is load-bearing on this comparison. If Thor2 trained on the same (or distributionally identical) force profiles that are replayed during evaluation, its advantage may reflect familiarity with the force characteristics rather than genuine force robustness. The paper should either (a) explicitly state the train/evaluation split and confirm no sequence overlap, or (b) include at least one policy not trained on this dataset as a control, or (c)退","section":null},{"comment":"§III.B, Eq. (2)–(3): The FATS weights (w_low, w_mid, w_high) = (0.2, 0.3, 0.5) and σ = 0.15 m are hand-chosen without justification or sensitivity analysis. Since FATS is the primary metric and the relative ranking of policies could shift under different weight choices (e.g., if a policy degrades more in the mid-force regime than the high-force regime), a sensitivity analysis over at least 2–3 alternative weight configurations would strengthen the claim that the reported differences are robust to metric design choices rather than artifacts of the chosen parameters.","section":null},{"comment":"§III.C, Force-replay protocol: The protocol applies recorded human sensor-frame forces to simulated robot hand bodies, but real contact dynamics during humanoid task execution differ — the robot's own motion changes contact location, timing, and force profile. The paper acknowledges this gap as future work (§V), but the central claim of revealing 'substantial performance differences' depends on whether these simulated force perturbations reflect real-world conditions. At minimum, the paper should discuss the expected direction and magnitude of the sim-to-real discrepancy (e.g., does open-loop force replay over- or under-estimate disturbance severity?) and ideally report results with varied force coefficients (currently only 0 and 1 are used) to characterize how sensitive the policy rankings are to force magnitude.","section":null}],"minor_comments":[{"comment":"§III.B, Eq. (4): The robustness ratio ρ = E_low/E_high is defined as RMSE_low/RMSE_high, but the text later says 'ρ → 1 indicates slower degradation.' This is correct but could confuse readers since a ratio of low/high error approaching 1 could also mean uniformly poor tracking. The paper partially addresses this in §IV.A ('its high ratio partly reflects a larger low-force baseline error'), but this caveat should appear at the metric definition, not only in the results discussion.","section":null},{"comment":"Table II: The 'Low KP' and 'High KP' rows report values of 27.8/37.5/37.8/43.9 and 33.8/40.7/42.4/54.1 respectively. The units (mm) are stated in the text but not in the table header. Adding units to the table would improve readability.","section":null},{"comment":"§III.A: The paper states 60 raw sequences per task but does not report the duration of each sequence, the sampling rate of the force sensors, or the synchronization accuracy between motion capture and force measurements. These details are important for reproducibility of the dataset.","section":null},{"comment":"Fig. 4: The radar plots are small and the axis labels are difficult to read. Consider enlarging or providing a tabular companion.","section":null},{"comment":"§IV.A: The paper mentions 'about 25% lower' for Thor2's keypoint error vs. the second best, but does not specify which policy is the second best in each regime. Specifying this would make the comparison more precise.","section":null},{"comment":"§II: The related work section does not discuss any existing force-aware or contact-rich benchmarks in manipulation (e.g., benchmarks that measure interaction forces in grasping or pushing). Even if these are from a different subfield, briefly positioning ThorArena relative to force-aware manipulation benchmarks would strengthen the novelty claim.","section":null}],"recommendation":"major_revision","confidential_remarks":"The train-on-test concern is the most serious issue. The authors are from the same group that developed Thor2 and collected the dataset, so they are in the best position to clarify the split. If a clean split exists and can be documented, the paper could be strengthened substantially with minimal additional work. If no split exists, the Thor2 results should be reframed as illustrative rather than as benchmark rankings. I would also encourage the authors to consider whether the benchmark would be more credible if Thor2 were excluded from the headline comparison and presented separately as a policy trained on the dataset, with the benchmark rankings focused on the three external policies."},"author_rebuttal":{"model":"glm-5.2","summary":"We thank the referee for the careful and constructive review. The referee raises three major points: (1) potential train/evaluation overlap for Thor2, (2) lack of sensitivity analysis for FATS hyperparameters, and (3) the sim-to-real gap in open-loop force replay. We agree with all three points and will address each in revision. Specifically, we will add an explicit train/evaluation split statement and include TWIST2, GMT, and SONIC as controls not trained on our dataset; add a sensitivity analysis over alternative FATS weight configurations; and add a discussion of expected sim-to-real discrepancy direction plus results with intermediate force coefficients. We cannot fully resolve the sim-to-real gap within this revision (physical robot validation is future work), which we note as a standing limitation.","responses":[{"response":"The referee is correct that this is a critical clarification. In the revised manuscript, we will explicitly state the train/evaluation split: the 360 collected sequences are partitioned such that the evaluation set used in ThorArena is disjoint from the data used to train Thor2. We will add this statement to both the Contributions section (§I) and the Experimental Setup (§IV). Furthermore, we note that TWIST2, GMT, and SONIC serve as the requested controls: none of these three policies were trained on our dataset, yet they are evaluated under the same force-replay protocol. The performance spread among these three policies (FATS 70.04–73.78 under external forces) already demonstrates that force-aware evaluation reveals differences independent of training-set familiarity. We will make this control structure explicit in the revision. That said, we acknowledge that Thor2's advantage over the other policies cannot be fully disentangled from its force-aware training: Thor2 was designed with force awareness in mind, so its superior performance under forces is expected and not necessarily a confound. The benchmark's purpose is to reveal such differences, not to claim that all policies should be force-naive. We will clarify this framing.","revision_made":"yes","referee_comment":"Train/evaluation overlap for Thor2: the paper does not specify whether evaluation sequences are disjoint from Thor2's training set, and Thor2 ranks first on all subtasks. If Thor2 trained on the same force profiles, its advantage may reflect familiarity rather than genuine robustness."},{"response":"We agree. The weight choice was motivated by the intuition that the high-force regime is most informative for force-aware evaluation, but this rationale was not stated in the manuscript and the robustness of rankings to this choice was not verified. In the revision, we will: (1) add a brief justification for the weight selection, (2) include a sensitivity analysis with at least three alternative configurations (e.g., uniform weights (1/3, 1/3, 1/3), low-force-emphasizing (0.5, 0.3, 0.2), and mid-force-emphasizing (0.2, 0.5, 0.3)), and (3) report whether policy rankings change under these alternatives. Based on our preliminary assessment, the ranking of Thor2 as top performer is robust because it achieves the lowest keypoint error in all three force regimes individually (Table II: Low KP 27.8, High KP 33.8, both best). However, the relative ordering of TWIST2, GMT, and SONIC may shift, and we will report this transparently. We will also add a brief note on σ sensitivity.","revision_made":"yes","referee_comment":"FATS weights (0.2, 0.3, 0.5) and σ = 0.15 m are hand-chosen without justification or sensitivity analysis. A sensitivity analysis over alternative weight configurations would strengthen the claim that reported differences are robust to metric design choices."},{"response":"This is a fair and important point. We will address it in two ways. First, we will add a discussion of the expected sim-to-real discrepancy direction: open-loop force replay likely over-estimates disturbance severity in some respects (because a real robot's compliant motion would reduce peak contact forces through give) and under-estimates in others (because real contact involves friction, deformation, and intermittent contact that a fixed-force application does not capture). We will frame this as a known limitation of open-loop replay and note that closed-loop force adaptation is future work, as already acknowledged in §V. Second, we will add experiments with intermediate force coefficients (e.g., 0.25, 0.5, 0.75) to characterize how policy rankings and FATS scores vary with force magnitude. This will show whether the reported performance differences are specific to full-force replay or hold across a range of disturbance intensities. We cannot, within this revision, provide physical robot validation to directly measure the sim-to-real gap; this remains a standing limitation that we will state explicitly.","revision_made":"partial","referee_comment":"Open-loop force replay does not account for how robot motion changes contact dynamics. The paper should discuss expected direction/magnitude of sim-to-real discrepancy and report results with varied force coefficients (currently only 0 and 1)."}],"tokens_in":9328,"tokens_out":1180,"duration_ms":167875,"standing_objections":["We cannot provide direct sim-to-real validation within this revision. Physical robot experiments to quantify the discrepancy between open-loop force replay and real contact dynamics are planned as future work (§V) but are beyond the scope of the current submission. We will state this limitation explicitly in the revised manuscript."]},"desk_editor":{"model":"glm-5.2","letter":"The main thing to know: ThorArena collects synchronized motion-force demonstrations across six contact-rich tasks and uses them to build a simulation-based benchmark that replays recorded hand forces during policy rollouts. That is a genuine gap fill — existing humanoid datasets (AMASS, PHUMA, Humoto) carry motion but no synchronized interaction forces, and existing benchmarks evaluate tracking or task success without force perturbation. The force-replay protocol and the FATS metric are practical, well-defined tools. The core empirical finding — that force-aware evaluation separates policies which look similar under no-force evaluation — is supported by the data. The push chair task is especially convincing: survival drops to 0.73–0.81 for three of four policies under force, from near 1.0 without. That is a real signal. The diagnostic metrics (robustness ratio, power overhead) are useful and the paper is honest about their limitations — it explicitly notes that TWIST2's high robustness ratio partly reflects a larger low-force baseline error. Good. Now the soft spots. The biggest one is the train-on-test confound the stress-test flags. The paper states that the motion-force data was used to train Thor2, and then Thor2 is evaluated on force profiles from the same dataset. The paper never states that Thor2's training split is disjoint from the evaluation split. Even if it is, the force profiles come from the same task types, same sensors, same collection protocol, so Thor2 has a systematic familiarity advantage the other three policies lack. This means Thor2's dominance under force (FATS 81.71 vs 70–74) is confounded — we cannot tell how much is genuine force robustness versus having trained on similar force data. The paper needs to either evaluate Thor2 on held-out force distributions or fine-tune another policy on the same force data to isolate the effect. This is the central issue. The secondary concerns are real but minor by comparison: FATS weights (0.2, 0.3, 0.5) and sigma=0.15m are hand-chosen with no sensitivity analysis; no error bars or significance tests; no code or data released despite the dataset being described as released; sim-to-real validity of force replay is untested. The FATS parameter choices probably do not change rankings given the magnitude of the gaps, but showing that would strengthen the paper. The sim-to-real gap is acknowledged as future work and is acceptable for a benchmark paper at this stage. This is a solid benchmark contribution with a real confound that needs addressing before publication. The force-replay protocol and metric design are worth building on regardless. It deserves a serious referee who can push the authors on the train-test split question and the missing sensitivity analysis.","headline":"Force-aware humanoid benchmark with a real train-on-test confound: Thor2 wins all tasks after training on the same force data used for evaluation.","tokens_in":10536,"tokens_out":631,"would_cite":false,"duration_ms":90752,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"Force Replay Exposes Hidden Weaknesses in Humanoid Robot Control","keywords":["humanoid robotics","force-aware evaluation","whole-body control","motion-force dataset","benchmark","contact-rich interaction","simulation-based evaluation"],"falsifier":"If a policy that scores poorly on ThorArena's force-replay benchmark performs well on a real humanoid robot executing the same contact-rich tasks, or if a policy that scores well fails in the real world, the benchmark's predictive validity for real-world deployment would be undermined.","tokens_in":9445,"feed_emoji":"🤖","tokens_out":1464,"duration_ms":99280,"temperature":0.7,"pith_summary":"This paper argues that conventional evaluation of humanoid robot control policies — which only checks how well a robot tracks motion in free space — systematically hides critical weaknesses that emerge when the robot must also withstand physical interaction forces. The authors build ThorArena, a benchmark that pairs synchronized human motion and hand-force recordings across six contact-rich tasks (table wiping, object lifting/lowering, chair pushing/pulling, cooperative carrying), then replays those recorded forces onto a simulated humanoid's hands while a control policy executes the corresponding motion. The central metric is the Force-Aware Tracking Score (FATS), which weights tracking error more heavily in high-force regimes and combines it with episode survival. The paper's key empirical claim: four representative whole-body control policies that look nearly identical under no-force evaluation (survival rates near 1.0, FATS scores within a 6-point band) diverge sharply when forces are replayed — survival on the push-chair task drops to 0.73–0.81 for three of the four policies, and FATS spreads widen by over 30 points on that task. The benchmark thus demonstrates that force-aware evaluation is necessary, not optional, for assessing whether a humanoid policy is ready for contact-rich deployment.","feed_headline":"Force Replay Exposes Hidden Weaknesses in Humanoid Robot Control","feed_subtitle":"Policies that look identical under standard motion-only tests diverge sharply when real interaction forces are applied — some survival rates","key_machinery":"Force-Aware Tracking Score (FATS): a composite metric that stratifies timesteps into low-, medium-, and high-force regimes using the 33rd and 66th percentiles of total applied hand force, computes keypoint RMSE in each regime, weights them (0.2, 0.3, 0.5) to emphasize high-force behavior, and combines the weighted error with an episode survival factor via an exponential scoring function. The force-replay protocol transforms recorded sensor-frame force vectors into the simulated hand-local frame and applies them as external forces on the robot's hand bodies during policy rollout, with a scalar coefficient set to 1 for force evaluation and 0 for baseline comparison.","core_discovery":"The paper's central finding is that interaction forces act as a stress test that separates policies which appear equivalent under kinematic-only evaluation. Under no-force conditions, all four tested policies (Thor2, TWIST2, GMT, SONIC) achieve survival rates above 0.99 and FATS scores between 78.4 and 84.3 — a narrow range suggesting comparable capability. When recorded human hand forces are replayed in simulation, the same policies exhibit qualitatively different failure modes: Thor2 maintains near-perfect survival (1.0) and the lowest tracking error; SONIC preserves balance through compliant behavior but accumulates the largest upper-body tracking errors; TWIST2 and GMT suffer substantial","pith_inferences":["If the simulated force-replay protocol does not faithfully reproduce real contact dynamics — because the robot's own motion changes contact location, timing, and force profile — then the performance differences revealed by ThorArena may be artifacts of the replay method rather than genuine policy weaknesses. The paper's own acknowledgment that physical-robot validation is future work means the ben","The six-task set, while representative, covers only hand-mediated interactions with relatively static objects. Humanoid deployment scenarios likely involve richer contact patterns (whole-arm contacts, unexpected collisions, multi-contact scenarios) that the current force-replay protocol cannot capture, suggesting the benchmark may under-estimate the range of force-induced failure modes.","The FATS weighting scheme (0.2, 0.3, 0.5 for low/mid/high force) implicitly assumes that high-force regimes are the most informative for policy evaluation. This is plausible but untested — it is possible that medium-force regimes, where policies transition from nominal to stressed behavior, are actually more diagnostic of control quality."],"forward_implications":["If the force-replay protocol is valid, then any humanoid control policy evaluated only under no-force conditions may be certified as deployment-ready while hiding force-induced instability that would cause failures in real contact-rich tasks.","The FATS framework could be extended to other embodied AI domains where external perturbations matter — for instance, evaluating quadruped locomotion over uneven terrain with replayed ground reaction forces, or assessing bimanual manipulation under varying payload forces.","The finding that different policies exhibit distinct trade-offs under force (tracking accuracy vs. survival vs. control effort) suggests that force-aware evaluation could drive a new axis of policy design: not just motion fidelity, but force-robustness profiles tailored to expected interaction types.","The paired motion-force dataset (360 sequences across six tasks) could enable training policies that explicitly anticipate interaction forces rather than merely tolerating them, shifting from reactive robustness to predictive force-aware control."],"fun_headline_variants":["Humanoid Policies That Look Equal Diverge Under Real Interaction Forces","Force Replay Benchmark Separates Humanoid Controllers Hidden by Motion-Only Tests","Contact Forces Expose Policy Failures Invisible to Standard Evaluation","Survival Rates Collapse When Humanoid Robots Face Replayed Human Forces","Force-Aware Benchmark Reveals Qualitatively Different Failure Modes in Humanoid Policies"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The benchmark assumes that replaying recorded human hand forces as fixed external disturbances in simulation is a valid proxy for the contact forces a real humanoid would experience during task execution. In reality, the robot's own motion changes where, when, and how hard it contacts objects, so the replayed forces may not match what the robot would actually encounter.","fun_headline_variants_meta":{"raw":{"variants":["Humanoid Policies That Look Equal Diverge Under Real Interaction Forces","Force Replay Benchmark Separates Humanoid Controllers Hidden by Motion-Only Tests","Contact Forces Expose Policy Failures Invisible to Standard Evaluation","Survival Rates Collapse When Humanoid Robots Face Replayed Human Forces","Force-Aware Benchmark Reveals Qualitatively Different Failure Modes in Humanoid Policies"]},"model":"glm-5.2","effort":"high","cost_usd":0.0,"raw_usage":{"total_tokens":708,"prompt_tokens":616,"completion_tokens":92,"prompt_tokens_details":null},"tokens_in":616,"tokens_out":92,"duration_ms":24055,"temperature":1.0,"reasoning_tokens":null,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-08T17:59:32.699131+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"If a policy that scores poorly on ThorArena's force-replay benchmark performs well on a real humanoid robot executing the same contact-rich tasks, or if a policy that scores well fails in the real world, the benchmark's predictive validity for real-world deployment would be undermined.","supporting_citations":[],"review_version":1}