{"id":"6c044fd6-cd4d-43fb-b9c3-7a5d5878ebe4","arxiv_id":"2508.14120","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"SimGenHOI generates physically plausible humanoid-object interaction sequences by predicting sparse key actions with a diffusion transformer and tracking them with a contact-aware reinforcement learning policy in simulation.","lead":"This paper combines a text-driven diffusion model with a physics-based reinforcement learning controller to generate realistic whole-body humanoid-object interactions, such as lifting and moving boxes. It reports higher tracking success in simulation than prior methods, and uses a mutual fine-tuning loop in which the generator and controller improve each other.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Mutual fine-tuning closes the loop between generator and policy inside Isaac Gym, so the reported physical-realism gains may reflect distribution alignment rather than independent correctness; a held-out policy or second simulator is needed.","rationale":"The reader's weakest-assumption analysis correctly identifies the mutual fine-tuning loop as the most load-bearing point: the generator and policy are co-adapted on each other's outputs, and every physical-realism metric is evaluated in the same simulator used for training. This creates a real risk that the reported improvements measure distribution alignment between a specific generator and a specific policy, not an independent property of physical realism. The paper's ablation (Table 4) does not break this loop, and the absence of code, error bars, or significance tests makes the modest gap (41.67% vs 37.50%) hard to interpret. A held-out policy evaluation would settle whether the fine-tuning gain transfers to a policy that was not part of the loop. Since the paper's contribution is otherwise plausible and the concern is addressable with additional experiments, the conditional verdict remains appropriate rather than moving to reject or accept.","tokens_in":13317,"tokens_out":2708,"duration_ms":29644,"concrete_test":"Train a policy checkpoint on the original FullBodyManipulation data without ever seeing generated motions or mutual fine-tuning; then evaluate the final mutual-fine-tuned generator's motions with this held-out policy in Isaac Gym using the same protocol as Table 4. If Succ_tgt drops materially (e.g., below the 37.50% no-fine-tuning ablation) while the final policy achieves 41.67%, the apparent gain is policy-specific co-adaptation rather than general physical plausibility. If the held-out policy achieves comparable success, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—physically realistic HOI with significantly higher tracking success—rests on the mutual fine-tuning loop described in the 'Fine-Tuning' subsection under 'HOI In Physics Simulation'. In that loop, actions successfully tracked by the contact-aware policy are collected as 'physically plausible' samples to fine-tune the diffusion generator, and the policy is then fine-tuned on the improved generator's outputs. The success criterion is defined by the same policy's interaction early termination and Succ_tgt (object within 0.5 m), both evaluated in Isaac Gym, the same simulator used to train the policy. There is no independent physical anchor: the only external reference is the original FullBodyManipulation mocap, and every physical-consistency metric (FS, H_feet, P_hand) is computed in the same simulator. In this closed loop, the generator can improve tracking success by learning motions that are easy for the current policy (e.g., slower, less contact-rich, or matching the policy's inductive biases), and the policy can conversely specialize to the generator's outputs. Table 4 shows only a 41.67% vs 37.50% success difference with and without fine-tuning, so the reported gain is modest and could be explained by distribution narrowing rather than by increased physical realism. The paper provides no diversity, coverage, or out-of-distribution analysis to rule out this co-adaptation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"SimGenHOI proposes a unified framework that couples a diffusion-transformer-based generative model, which outputs sparse \"key actions\" for humanoid-object interaction (HOI), with a contact-aware whole-body reinforcement-learning policy that tracks these motions in the Isaac Gym simulator. The key-action representation is interpolated into dense trajectories, and an alternating \"mutual fine-tuning\" procedure refines the generator on successful policy rollouts and then fine-tunes the policy on the improved generator outputs. The paper reports generation-quality metrics (e.g., MPJPE, foot sliding, hand penetration) that outperform several adapted baselines on the FullBodyManipulation dataset, and tracking results on seven reference motions. An ablation study examines the contributions of key actions, contact guidance, and fine-tuning. The authors claim that the framework generates realistic, diverse, and physically plausible long-horizon HOI and achieves significantly higher tracking success in simulation.","tokens_in":13584,"tokens_out":6074,"duration_ms":57645,"significance":"If the empirical claims hold, the framework would be a useful step toward integrating generative motion synthesis with physics-based control for humanoid-object interaction, particularly its key-action abstraction for long-horizon generation and its effort to close the loop between generation and control. The paper includes quantitative comparisons with several baselines and an ablation study, and it would release code. However, the validation is largely internal to the same simulator used for training, and the mutual fine-tuning loop introduces a circularity that weakens the physical-realism claim. The paper also lacks error bars, diversity metrics, and a well-defined long-horizon evaluation, which limits the strength of its central claims.","major_comments":[{"comment":"The mutual fine-tuning loop is circular with respect to the claim of improved physical realism. The generator is fine-tuned on actions successfully tracked by the same contact-aware policy, and the policy is fine-tuned on the generator's outputs; the success criterion (Succ_tgt, object within 0.5 m, and interaction early termination) is defined by that same policy and evaluated in the same Isaac Gym simulator used for training. The reported gain from fine-tuning in Table 4 is modest (41.67% vs. 37.50%), and without a diversity or coverage analysis to show that the generator does not simply narrow its output distribution to motions the current policy finds easy, the improvement could reflect distribution alignment rather than an increase in independent physical realism. I recommend an external validation: evaluate the same generated motions with a different tracking policy or in a different physics engine, and report the variance of generated joint/object trajectories to rule out distribution collapse.","section":"Methods, 'HOI In Physics Simulation' > 'Fine-Tuning'; Table 4"},{"comment":"The central claim of \"significantly higher tracking success\" is not supported by inferential statistics. All tables report single point estimates with no error bars, confidence intervals, or multiple seeds. For instance, in Table 4 the difference between the full system (41.67%) and \"w.o. finetune\" (37.50%) corresponds to about 6 of 144 sequences, and the difference with \"w.o. key action\" (34.72%) is about 10 sequences, which may well be within run-to-run noise. Please provide means and standard deviations over at least three random seeds for the policy training and, where feasible, for the diffusion model, and apply an appropriate significance test to the comparisons that underpin the headline claims.","section":"Tables 1-4; Experiments"},{"comment":"The tracking evaluation is based on only 7 manually selected reference motions, which invites selection bias and limits the generality of the tracking comparison. Moreover, Table 3 does not clearly support the claim of superior tracking: OmniGrasp on OMOMO achieves the same 7/7 success in Succ_cont and Succ_tgt and has lower Eobj_pos (40.84 vs. 64.92) and lower Eobj_rot (0.13 vs. 0.72), while SimGen-HOI shows better Eobj_acc and Eobj_vel. The paper should either use a larger and pre-specified test set, report the criteria for selecting the 7 motions, and discuss the mixed results honestly, or temper the claim that the method outperforms prior approaches in tracking.","section":"Experiments, 'Contact-Aware Interaction Tracking' and Table 3"},{"comment":"The paper claims that the framework \"enables long-horizon manipulation tasks\" and generates \"diverse\" humanoid-object interactions, but no experiment quantifies long-horizon performance (e.g., number of autoregressive steps, success over extended sequences, comparison of windowed vs. one-shot generation) or diversity (e.g., pairwise distances between samples generated from the same conditions). These are load-bearing claims in the abstract and conclusion, and they need dedicated evaluation. Without such results, the long-horizon and diversity claims are unsupported.","section":"Abstract and Conclusion"}],"minor_comments":[{"comment":"The text says \"Please see the appendix E for details of the key joints,\" but the manuscript contains no appendix. The list of key joints is necessary to interpret Eq. (8), which states that the tracking reward is applied only to selected key joints. Please either include the appendix or specify the joints in the main text.","section":"'Reward Design'"},{"comment":"The key-action extraction depends on a manually defined importance weight vector w and a stopping threshold; the paper does not state how these are chosen or whether the results are sensitive to them. A brief sensitivity analysis (e.g., varying the threshold or weights and reporting the effect on generation metrics) would strengthen the reproducibility of the approach.","section":"Methods, 'Key Actions Extract'"},{"comment":"The title of Table 2 reads \"Evaluation of motions tracked by our control policy,\" but the rows include OMOMO GT, CHOIS, and OURS200, which appear to be generation-only or baseline inputs. The caption and text should clarify which pipeline (generation only, generation plus tracking, or baseline plus tracking) is used for each row, and the metrics should be explicitly described as computed in the Isaac Gym simulator.","section":"Table 2"},{"comment":"The baseline comparison in Table 3 uses GRIP to add hand details to OMOMO and CHOIS before tracking with PHC-X, while the proposed method assigns default hand joint angles and lets the policy explore contact. This asymmetry should be discussed as a deliberate design choice, and its potential to favor the proposed method (since GRIP may not be well suited to large objects or to integration with PHC-X) should be acknowledged.","section":"Table 3"},{"comment":"There are several typographical and notation issues: \"seqequence\" in the Methods section, the meaning of \"Lin-OMOMO\" versus \"Pred-OMOMO\" is not explained, and the formatting of C_F1, C%, and related metrics is inconsistent. A careful proofread would improve readability.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper presents a promising framework, but the evidence for its main claims is currently weaker than the abstract suggests. The circularity of the mutual fine-tuning evaluation is the most serious concern; if the authors can provide an independent evaluation of physical realism (e.g., with a different tracking policy or a second simulator) and address the statistical rigor and evaluation-set issues, the paper would be substantially stronger. The comparison with OmniGrasp in Table 3 is not obviously favorable to the proposed method on all metrics, so the authors should carefully revise the claims. I recommend major revision rather than rejection because the core ideas are sound and the specific weaknesses appear addressable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [colleague],\n\nYou asked about SimGenHOI. The short version: it's a solid systems paper with a new combination of ideas, but the central claim of 'physically realistic' is undercut by a closed-loop fine-tuning procedure with no independent check, and the success rates are not as high as the abstract suggests.\n\nWhat's new: they adapt sparse key-action extraction (from AWE) to humanoid-object interaction, generate key actions with a DiT diffusion model conditioned on text, object geometry, and waypoints, then track them with a contact-aware RL policy. The mutual fine-tuning loop—alternating between fine-tuning the generator on policy-tracked motions and fine-tuning the policy on generated motions—is the most novel piece.\n\nWhat's good: the paper runs a comprehensive comparison against several baselines (InterDiff, MDM, OMOMO, CHOIS) including a tracking stage, and includes an ablation isolating key actions, contact guidance, and fine-tuning. Physical metrics like foot sliding and penetration improve substantially after tracking, which is real evidence the policy corrects artifacts. The writing is clear and the method is specified well enough to re-implement.\n\nWhere it's soft: the stress-test note is on target. The fine-tuning loop collects 'physically plausible' actions from policy rollouts in Isaac Gym, trains the generator on them, then retrains the policy on the generator's outputs. The success criterion (Succ_tgt within 0.5 m, interaction early termination) is defined in that same simulator. There's no independent physical anchor, so the fine-tuning gain (37.5% vs 41.7% success) could be distribution narrowing—the generator learning motions that are easy for this particular policy—rather than improved realism. The paper doesn't provide diversity or coverage metrics to rule this out. Also, all numbers are single-seed with no error bars, and the 7 reference motions in Table 3 are manually selected. The baseline comparisons may be slightly unfair: CHOIS and OMOMO need GRIP-refined hand grasps and are tracked by a different policy, while SimGen-HOI skips hand details. That doesn't invalidate the paper, but it makes 'significantly higher' hard to judge.\n\nNet: this is a reasonable contribution to humanoid-object interaction generation and control. It's not a breakthrough, but a useful recipe. The right audience works on physics-based character control and HOI synthesis. I'd send it to peer review, but only with a strong request for code, multiple seeds, and an analysis that addresses the co-adaptation risk—a held-out policy, a second simulator, or an in-the-loop diversity measure.","headline":"A well-assembled pipeline for whole-body humanoid-object interaction that deserves peer review, but the mutual fine-tuning loop is a real co-adaptation risk and the reported numbers are modest.","tokens_in":14156,"tokens_out":4158,"would_cite":false,"duration_ms":39323,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By generating humanoid-object interactions as sparse key actions and tracking them with a contact-aware reinforcement-learning policy, SimGenHOI produces motions that are controllable, physically executable in simulation, and more robust…","keywords":["humanoid-object interaction","motion generation","diffusion model","reinforcement learning","whole-body control","contact-aware policy","key action extraction","long-horizon manipulation"],"falsifier":"Run the trained pipeline, without further training, in a different physics engine or on a real humanoid, and compare tracking success, foot sliding, and hand penetration; if the strong metrics collapse, the gains come from simulator-specific alignment rather than physical realism. Alternatively, hold out motion-capture sequences from fine-tuning and test whether policy success improves on those after the mutual loop; if only generator-sampled motions improve, the loop has not improved physical plausibility.","tokens_in":13119,"feed_emoji":"🤖","tokens_out":11366,"duration_ms":103525,"temperature":0.7,"pith_summary":"SimGenHOI sets out to resolve the usual trade-off in humanoid-object interaction synthesis: generative models produce diverse and controllable motion but with artifacts such as hand penetration and foot sliding, while physics-based policies are physically consistent but narrow in behavior. The paper argues that both goals can be met by splitting the problem: a diffusion model generates a sparse set of key actions—critical poses of the hands, feet, and body—conditioned on text, object geometry, sparse waypoints, and the starting pose, and a contact-aware whole-body reinforcement-learning policy then tracks those actions inside a physics simulator, correcting penetration and sliding. A mutual fine-tuning loop closes the gap between the two: successful simulated executions are fed back to refine the generator, and the improved generator's motions are used to further train the policy. If the claim holds, users could specify a long-horizon manipulation in text and get a motion that a simulated humanoid can actually execute, not just a video-like pose sequence. The reported experiments show higher tracking success and reduced contact artifacts compared with adapted prior baselines.","feed_headline":"Key actions plus contact-aware RL make humanoid-object motion executable","feed_subtitle":"Text- and waypoint-controlled whole-body motions that hold up in a physics simulator, not just as rendered poses.","key_machinery":"The load-bearing mechanism has three coupled parts. First, key-action extraction: from a dense motion sequence, the algorithm recursively inserts the frame with the largest weighted reconstruction error as a new key frame, using per-joint weights that emphasize hands and feet, until every interpolated subsegment falls below an error threshold; this converts dense clips into sparse, interpolatable control points. Second, contact guidance: the diffusion model also outputs a contact probability for hands and feet, and this signal is given to the policy as a reference cue with a reward term that encourages predicted and actual contact to match, so the policy knows when contact should occur and can be rewarded for it. Third, mutual fine-tuning: successful policy tracks from the simulator are collected as training samples for the generator, and newly generated motions are used to refine the policy, aligning the distribution of generated reference motions with motions the policy can reliably execute. The whole loop is what carries the claim that generated interaction is physically plausible.","core_discovery":"The paper's claim is that physical executability and generative diversity are not opposed if generation is organized around key actions rather than dense frames. Its HOI generator, a Transformer-based diffusion model, predicts a short sequence of key actions together with object poses and hand/feet contact probabilities; intervening frames are recovered by interpolation. These reference trajectories, including a contact-guidance signal that tells the policy when to grasp and release, are handed to a whole-body policy trained by reinforcement learning in simulation. The policy tracks key joints and object pose while the contact guidance lets it discover grasping behavior through exploration. The authors further show that iteratively fine-tuning the generator on motions the policy successfully tracked, and fine-tuning the policy on the improved generator's outputs, raises tracking success. In their experiments on a human-object manipulation motion-capture dataset, the full system reaches contact and target success on all seven tested reference manipulations while reducing hand penetration and foot sliding relative to direct generation and to tracking baselines.","pith_inferences":["A testable extension follows from the weakest premise: if the mutual fine-tuning loop is genuinely improving physical realism, tracking success on held-out motion-capture sequences never used in fine-tuning should improve too; if only self-generated motions become easier to track, the loop has aligned the generator to the policy rather than to physics.","The key-action extraction threshold and per-joint weights form a fidelity/complexity dial. One could adapt them per task—allocating more key frames to hand and foot phases during a grasp, fewer during transportation—to trade generation cost against policy trackability.","Because the policy is trained entirely in simulation, transferring this pipeline to a real humanoid would likely require domain randomization and system identification; the paper's contact-guidance reward structure gives a natural place to add force and torque feedback.","The interpolation between key actions effectively imposes a piecewise-linear prior on motion, and that prior likely contributes to the reported reduction in foot sliding; this could be isolated by comparing key-action interpolation against dense prediction with the same policy."],"forward_implications":["Text- and waypoint-conditioned long-horizon manipulation (lift, move, put down, push, and return an object) becomes a single pipeline: generate sparse key actions autoregressively, interpolate, track with the policy.","Sparse key actions reduce the number of denoising passes needed for long sequences, because each key action covers a longer temporal span than a dense frame.","The contact-guidance signal lets the policy learn grasping and releasing timing by exploration instead of relying on detailed reference hand poses, widening the range of manipulable objects.","The mutual fine-tuning loop improves tracking success; in the ablation, removing it lowers success from 41.67% to 37.50%, so the loop is doing real work rather than being decorative.","Passing generated motion through the physics simulator substantially improves physical metrics such as foot height, foot sliding, and hand penetration, suggesting that physics filtering alone removes many kinematic artifacts."],"supporting_citations":[{"why":"Supplies the FullBodyManipulation dataset used for training and evaluation and the OMOMO baseline that SimGenHOI extends.","marker":"(Li, Wu, and Liu 2023)"},{"why":"Provides the idea of predicting un-uniform key actions from a current state, which the key-action extraction adapts to motion sequences.","marker":"(Shi et al. 2023)"},{"why":"Supplies the Transformer-based diffusion architecture used as the HOI generator's denoising network.","marker":"(Peebles and Xie 2022)"},{"why":"Defines the DDPM noising and reverse process and the clean-data L1 objective that trains the generator.","marker":"(Ho, Jain, and Abbeel 2020)"},{"why":"Provides the PULSE pre-trained humanoid motion representation that the contact-aware whole-body policy builds upon.","marker":"(Luo et al. 2023a)"},{"why":"Defines the tracking metrics and the OmniGrasp baseline used for comparison in simulation.","marker":"(Luo et al. 2024)"},{"why":"Supplies the SMPL-X body model with articulated hands used as the simulated humanoid.","marker":"(Pavlakos et al. 2019)"},{"why":"Supplies the CLIP text embedding that conditions generation on natural-language prompts.","marker":"(Radford et al. 2021)"}],"fun_headline_variants":["Key actions plus contact-aware RL yield executable whole-body HOI","SimGenHOI: key-action diffusion plus RL for physically real HOI","Whole-body object interactions that survive physics: key actions + RL","Generative key actions, RL contact control: executable humanoid-object moves","Key actions break the sim-reality gap for humanoid-object interaction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The key assumption is that the motions the policy successfully tracks in the simulator are actually physically plausible, because the same simulator is used to train the policy and to evaluate the whole system, so the mutual fine-tuning loop is never validated against an independent source of physical ground truth.","fun_headline_variants_meta":{"raw":{"variants":["Key actions plus contact-aware RL yield executable whole-body HOI","SimGenHOI: key-action diffusion plus RL for physically real HOI","Whole-body object interactions that survive physics: key actions + RL","Generative key actions, RL contact control: executable humanoid-object moves","Key actions break the sim-reality gap for humanoid-object interaction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000217,"raw_usage":{"total_tokens":1466,"prompt_tokens":1008,"completion_tokens":458,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":624,"completion_tokens_details":{"reasoning_tokens":366}},"tokens_in":624,"tokens_out":458,"duration_ms":5109,"temperature":1.0,"reasoning_tokens":366,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:16:48.403247+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the trained pipeline, without further training, in a different physics engine or on a real humanoid, and compare tracking success, foot sliding, and hand penetration; if the strong metrics collapse, the gains come from simulator-specific alignment rather than physical realism. Alternatively, hold out motion-capture sequences from fine-tuning and test whether policy success improves on those after the mutual loop; if only generator-sampled motions improve, the loop has not improved physical plausibility.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Transformer-based diffusion architecture used as the HOI generator's denoising network."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the tracking metrics and the OmniGrasp baseline used for comparison in simulation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the SMPL-X body model with articulated hands used as the simulated humanoid."}],"review_version":2}