{"id":"84c52bcc-4c39-400f-b957-ae41e181590f","arxiv_id":"2504.16843","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A latent-diffusion-image-to-keyframe pipeline enables whole-body trajectory optimization to solve long-horizon humanoid loco-manipulation tasks in simulation, outperforming contact-only guidance.","lead":"The paper uses text-to-image latent diffusion models to generate pictures of people interacting with objects, then converts the pictured hand and foot contacts and body poses into keyframes that steer a whole-body trajectory optimizer for a simulated humanoid robot. It shows that this image-derived guidance lets the optimizer solve two long-horizon loco-manipulation tasks with fewer collisions than contact-only or geometry-unaware baselines.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Contact transfer relies on unvalidated assumed camera intrinsics for LDM-generated images; without sensitivity or accuracy analysis, the downstream TO advantage could depend on this tuning.","rationale":"The reader identified the depth-and-intrinsics-based contact transfer as the weakest assumption, and this stress-test agrees. The method's novelty and usefulness depend on transferring 2D hand/feet locations from generated images into accurate 3D contacts in the simulated scene. That transfer is only as reliable as the lifted point clouds used for semantic-geometric matching. The paper provides no direct accuracy check of the assumed intrinsics or the metric-depth estimates, and the 'empirical trial and error' description indicates the setting was chosen to make the point cloud look reasonable rather than to match a known camera model. Since the same rigid transform also feeds the retargeting module, an error here propagates into the keyframes and the warm start, not only into contact positions. The reported collision-penetration comparisons are suggestive but cannot disentangle geometric transfer accuracy from the robustness of the trajectory optimizer. A sensitivity study on intrinsics and a direct contact-error evaluation against ground truth would settle whether this concern lands. Given this unvalidated but load-bearing step, the reader's CONDITIONAL verdict remains the appropriate outcome; no change is needed.","tokens_in":11259,"tokens_out":4274,"duration_ms":46440,"concrete_test":"For a held-out set of LDM-generated images (e.g., 30 per scenario), fix the estimated depth and re-run contact transfer with three to five plausible intrinsic settings around the chosen focal lengths and principal-point offsets, then compare the resulting 3D contact locations against ground-truth contacts defined in MuJoCo or from manual annotation of the generated images. Report mean contact error and downstream TO penetration; if contact error varies by more than roughly 5 cm or the TO success reverses, the empirical intrinsics choice is a critical uncontrolled parameter.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Sec. III-A.2: the entire contact-transfer and retargeting chain depends on lifting LDM-generated RGB images to metric 3D point clouds using an assumed pinhole camera, with focal lengths set to the image resolution and principal-point offsets to half the focal lengths via 'empirical trial and error', and with depth from a zero-shot metric model. These lifted point clouds are then used to solve a rigid transform (SVD+ICP) that maps 2D hand/feet detections to 3D contacts, and the same transform also determines the base orientation used in retargeting. No quantitative validation is reported that the assumed intrinsics and estimated depth yield correct metric geometry, and no sensitivity analysis is given; the only downstream metric reported is collision penetration. If the lifted generated-object point cloud is systematically distorted in scale, aspect ratio, or perspective, the geometric-overlap criterion can select a rigid transform that aligns the wrong object pose, biasing both the contact locations and the keyframe configurations. This is load-bearing because the central claim that LDM images reliably transfer contacts and configurations into simulation rests entirely on this unvalidated lifting step.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an end-to-end pipeline that uses latent diffusion models (LDMs) to generate RGB images of human-object interaction scenes, then extracts 2D hand/feet contacts and human poses from those images, lifts them to 3D using segmentation, zero-shot metric depth estimation, and assumed camera intrinsics, and retargets them to a Unitree G1 humanoid in MuJoCo. The resulting contact locations and keyframe configurations are fed into a whole-body trajectory optimization (TO) formulation. The method is evaluated on two long-horizon scenarios (basket retrieval with box stepping, and trolley pushing), with comparisons against contact-only and geometry-unaware baselines and ablations on the contact transfer and keyframe components. The central claim is that LDM-derived contacts and configurations make otherwise intractable local TO problems solvable and reduce collision penetration.","tokens_in":11397,"tokens_out":4966,"duration_ms":47694,"significance":"If the claims hold, the paper makes a useful systems contribution by showing a practical way to inject semantic long-horizon knowledge from generative image models into contact-implicit whole-body TO, without task-specific contact heuristics. The pipeline is clearly described, the use of geometric overlap to refine semantic correspondences is well motivated, and the paper includes ablations for both the contact transfer stage and the keyframe utilization. I also see no circularity: the keyframes and contacts are computed before and independently of the TO, and the evaluation metrics are downstream. However, the evaluation is not yet at the standard needed to support the strong claims: only two scenarios are tested, the plots appear to show single runs without error bars or seed counts, and the 3D lifting step for LDM-generated images is unvalidated. These issues are load-bearing and fixable with additional experiments and analysis.","major_comments":[{"comment":"The contact-transfer stage rests on two unvalidated assumptions: that zero-shot metric depth from Metric3D v2 yields correct metric point clouds for LDM-generated RGB images, and that the hand-set pinhole intrinsics (focal lengths equal to the image resolution and principal-point offsets equal to half the focal lengths, chosen by 'empirical trial and error') are adequate. The rigid transform computed by SVD and ICP from these lifted point clouds is used both to map hand and feet detections to 3D contacts and, in Sec. III-A.3, to define the base orientation for retargeting. If the assumed intrinsics or the estimated depth introduce scale, aspect-ratio, or perspective distortion, the geometric-overlap criterion can select a biased rigid transform, corrupting both the contacts and the keyframe configurations. The paper reports no sensitivity analysis for the intrinsics, no comparison against ground-truth depth for the simulated objects, and no direct accuracy metric for the lifted geometry, so the central claim that LDM images reliably transfer contacts and configurations into simulation is currently supported only by downstream penetration plots. Please add ablations over the intrinsics and depth sources, and report quantitative contact/keyframe errors.","section":"Sec. III-A.2"},{"comment":"The quantitative claims that the pipeline 'clearly' reduces penetration and improves TO convergence are made from Figs. 5 and 6, which appear to show single runs with no error bars, seed counts, or variance information. The pipeline is stochastic at multiple stages: LDM image generation, VLM/SAM detection, the sampling-based semantic correspondence search, and possibly the 3D pose estimator, and TO is a local solver sensitive to initialization. Without repeated runs, statements about success rate and comparisons between keyframe settings are not statistically supported. Please report the number of seeds or runs, per-run outcomes, and error bars or box plots, and define the success criterion (e.g., TO convergence, collision-free trajectory, or task completion).","section":"Sec. V"},{"comment":"The 'naive' baseline that provides contacts obtained directly from the semantic-aware foundation model is not fully specified. It is unclear how those 2D semantic correspondences are converted to 3D contact locations, which depth source and camera intrinsics are used for the simulated and generated images, and whether the same human-pose extraction and retargeting steps are applied. Without this specification, the comparison in Fig. 5 may conflate the contribution of the geometric refinement with differences in 3D lifting or keyframe usage. Please specify the baseline precisely, ideally by using the same 3D lifting code with only the semantic-correspondence refinement removed.","section":"Sec. V-A"},{"comment":"The keyframe ablation is hard to interpret as reported. The success of the full keyframe set is not uniform: several subtitles in Fig. 6 show other settings with lower or equal penetration, and some bars are marked 'Unsolved' without a definition of what counts as unsolved. The discussion attributes the anomalies to 'some details in the trajectory optimization solver' without further analysis. Please report convergence status and success rates per condition, define the unsolved criterion, and analyze the cases where the full keyframe set performs worse rather than attributing them to unspecified solver details.","section":"Sec. V-C"}],"minor_comments":[{"comment":"The caption lists '(c) S2 w/ collision penalties' and '(d) S2 w/ collision penalties' identically; one of these should presumably read 'w/o collision penalties' to match the text in Sec. V-B.","section":"Fig. 5"},{"comment":"There is a typo in 'A minimal set of collision constraints are are added'; please remove the duplicated 'are'.","section":"Sec. V-C.2"},{"comment":"The notation 'call' in the first row is unclear; it appears to denote the center of mass of the supported mass, but the acronym and the subscript are not defined in the table or the surrounding text.","section":"Table I"},{"comment":"The homotopy parameter alpha in Eq. (13) is introduced without explaining how it is initialized, how it is driven to zero, and whether it is a decision variable or a fixed schedule; please clarify.","section":"Sec. IV-D"},{"comment":"The legend labels 'n f b bf fw bwbfw' are cryptic; please spell out each abbreviation in the caption so the reader can map the bars to the settings described in Sec. V-C.","section":"Fig. 6"},{"comment":"The reference for the Flux LDM lists the year as '2014, 2024-07-01'; the 2014 date appears to be an error and should be corrected.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The core idea is a reasonable and potentially publishable systems contribution, and I see no circularity in the evaluation. The main risk is that the unvalidated 3D lifting step and the absence of repeated-seed statistics make the quantitative claims fragile. If the authors can supply the requested ablations and error bars, and specify the baselines precisely, I would be willing to support publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is a real systems result, not a repackaging. It's the first pipeline I know of that uses LDM-generated RGB images to extract both contact sequences and whole-body keyframes for humanoid loco-manipulation, and feeds them into a centroidal-dynamics TO. The two MuJoCo scenarios (basket retrieval with a step, trolley push) are genuinely long-horizon and the ablation structure is sensible: geometry-refined contact transfer beats the raw semantic correspondence, and keyframe costs plus warm starts reduce penetration and improve TO convergence. Credit where it's due: the integration is careful and the comparison against a contact-only baseline directly supports their claim that configuration keyframes matter.\n\nThe soft spot is exactly the one you flagged. Section III-A.2 assumes a pinhole camera for generated images, with focal lengths set to image resolution and principal point offsets to half focal lengths, chosen by 'empirical trial and error.' Depth comes from a zero-shot metric model. The whole contact-transfer and retargeting chain sits on this. If the lifted generated-object point cloud is wrong in scale or perspective, the SVD+ICP alignment can lock onto the wrong object pose, biasing both contacts and the base orientation used for retargeting. The paper reports no accuracy check on the lifted geometry, no sensitivity analysis, and no seeds or error bars. That last point is minor-to-moderate on its own, but combined with only two scenarios it makes 'physically consistent' in the abstract too strong. The evidence says: lower penetration than baselines in simulation, not physical consistency.\n\nI also want to be fair: the stress-test note could have overreached if the paper had validated the intrinsics or shown robustness, but it didn't. So the concern stands. Crucially, the weakness is fixable—they could run a sensitivity sweep, add synthetic images with known geometry, and report contact-transfer accuracy separately from downstream TO penetration.\n\nMy take: conditionally solid, deserves a serious referee, but the review should ask for more scenarios, seed counts, a sensitivity analysis of the assumed intrinsics, and either code release or much more detail on the external-model outputs. I'd bring it to reading group and would cite it if I were working on LDM-guided planning for humanoids.","headline":"A genuine systems contribution with a load-bearing geometric-transfer step that is under-validated; deserves review but needs sensitivity analysis and more scenarios.","tokens_in":11987,"tokens_out":1760,"would_cite":true,"duration_ms":15833,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that latent-diffusion-generated images can supply the contact points and robot poses that let whole-body trajectory optimization solve long-horizon humanoid loco-manipulation in simulation.","keywords":["humanoid loco-manipulation","latent diffusion models","trajectory optimization","contact planning","whole-body control","image-guided retargeting","long-horizon planning","simulation validation"],"falsifier":"Take a set of generated images paired with a known 3D scene, for example by rendering a simulated scene from a known camera pose and generating an image of the same object, run the contact-transfer stage, and measure the distance from each extracted contact to the true object surface. If the median distance is comparable to the object dimensions rather than a few centimeters, the contacts cannot guide the optimizer, and the central claim is falsified.","tokens_in":10999,"feed_emoji":"🤖","tokens_out":5830,"duration_ms":50525,"temperature":0.7,"pith_summary":"This paper tries to show that a text-to-image latent diffusion model can act as an automatic demonstration generator for humanoid robots. Instead of hand-coding contact sequences or collecting 3D interaction datasets, the method generates RGB images of a person performing a task, extracts from those images where the hands and feet touch objects and what the body pose is, and feeds that information into a whole-body trajectory optimizer. The paper argues that this is enough to produce physically consistent, collision-free trajectories for long-horizon tasks in simulation, including tasks that local trajectory optimization cannot solve when given only semantic contact labels. A sympathetic reader cares because it points to a way of planning complicated loco-manipulation without expensive teleoperation or task-specific heuristics.","feed_headline":"Generated images guide humanoid robots through long tasks","feed_subtitle":"Contact points and poses pulled from diffusion-made RGB images let trajectory optimization finish loco-manipulation tasks.","key_machinery":"The load-bearing mechanism is a three-stage image-to-optimization pipeline. First, a latent diffusion model generates full-body RGB images of a person carrying out each subtask from a text prompt. Second, a contact-transfer stage lifts 2D hand and foot locations from those images into 3D contact points by matching the generated objects to the simulated scene semantically and then refining the matches with point-cloud geometry and iterative closest point; depth comes from a metric-depth estimator and the missing camera intrinsics are set empirically to the image resolution and half-focal-length principal point offsets. Third, a retargeting stage uses inverse kinematics to map the estimated human pose to the robot's degrees of freedom. The extracted contacts and robot configurations enter the trajectory optimization as keyframe costs and warm starts, which is what turns a local whole-body optimizer into a long-horizon planner.","core_discovery":"The central claim is that the plan for a long-horizon loco-manipulation task can be read off from latent-diffusion-generated RGB images: contact locations and robot configurations extracted from those images serve as keyframes that guide a whole-body trajectory optimization with centroidal dynamics and full kinematics. In the two simulated scenarios tested (stepping onto a box to retrieve a basket from a shelf, and loading a box onto a trolley and pushing it), the full pipeline produces trajectories with little or no penetration, whereas a baseline that feeds only semantic contact labels either fails to converge or produces large penetrations. The paper also demonstrates through ablations that both the geometry-refined contact transfer and the keyframe information (base pose, foot position, and subtask-like warm start) contribute to the optimizer's success.","pith_inferences":["Editorial inference: the critical transfer point is the assumed camera intrinsics for the generated image; if this empirical setting does not hold outside the tested simulator, the extracted 3D contacts would be systematically misplaced, so real-world deployment would likely need a calibration or distortion-correction step.","Editorial inference: the method suggests a general recipe in which generative image models provide rough demonstrations that seed trajectory optimization; the same pattern could apply to other platforms or tasks where demonstrations are hard to obtain, such as quadruped manipulation or dexterous hand tasks.","Editorial inference: because the geometry-overlap criterion is object-agnostic, the contact-transfer stage could be reused with any semantic matcher and depth estimator; a natural test is to swap those components and see whether end-to-end success tracks the contact-transfer error."],"forward_implications":["If the claim is right, a high-level text description and RGB-D images of the objects are enough to plan long-horizon humanoid loco-manipulation, without task-specific heuristics or 3D interaction datasets.","The same generated keyframes let whole-body trajectory optimization solve tasks that local trajectory optimization cannot solve from semantic contact labels alone.","Geometry-aware refinement makes the contact transfer robust enough that only a minimal set of collision constraints is needed for collision-free motion.","Including base pose, foot position, and warm-start keyframes improves both the convergence rate and the penetration scores of the optimizer across different object masses and initial robot yaws.","The approach is demonstrated on two scenarios, so the claimed mechanism transfers across different object types and long-horizon task structures."],"supporting_citations":[{"why":"supplies the latent diffusion model that generates the full-body RGB demonstration images from text prompts","marker":"[19]"},{"why":"provides open-vocabulary object detection that seeds the object masks in both generated and simulated images","marker":"[20]"},{"why":"renders per-pixel segmentation masks that tighten the object bounding boxes for point-cloud extraction","marker":"[21]"},{"why":"supplies the physics simulation and ground-truth depth and camera intrinsics for the simulated scenes","marker":"[22]"},{"why":"estimates metric depth for the generated images where no true depth exists","marker":"[23]"},{"why":"offers raw semantic correspondences between generated and simulated images that the geometry refinement then corrects","marker":"[24]"},{"why":"extracts the 3D human joint positions and orientations used to define retargeting constraints","marker":"[25]"},{"why":"provides the joint-matching strategy used to map human joints onto the robot's degrees of freedom","marker":"[26]"},{"why":"contributes the centroidal dynamics and whole-body kinematics formulation underlying the trajectory optimization","marker":"[27]"}],"fun_headline_variants":["Diffusion images steer humanoid loco-manipulation","AI-painted scenes chart robot paths","Latent diffusion maps humanoid manipulation","Image-derived plans keep robots consistent","Painted scenarios direct humanoid motion"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole pipeline rests on the assumption that the 3D contact points lifted from the generated image reliably land on the real object's surface; if the estimated depth or the assumed camera intrinsics are wrong, the contacts and keyframes are wrong and the optimizer is led into poor local minima.","fun_headline_variants_meta":{"raw":{"variants":["Diffusion images steer humanoid loco-manipulation","AI-painted scenes chart robot paths","Latent diffusion maps humanoid manipulation","Image-derived plans keep robots consistent","Painted scenarios direct humanoid motion"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000193,"raw_usage":{"total_tokens":1281,"prompt_tokens":804,"completion_tokens":477,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":420,"completion_tokens_details":{"reasoning_tokens":413}},"tokens_in":420,"tokens_out":477,"duration_ms":4624,"temperature":1.0,"reasoning_tokens":413,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:53:54.054733+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of generated images paired with a known 3D scene, for example by rendering a simulated scene from a known camera pose and generating an image of the same object, run the contact-transfer stage, and measure the distance from each extracted contact to the true object surface. If the median distance is comparable to the object dimensions rather than a few centimeters, the contacts cannot guide the optimizer, and the central claim is falsified.","supporting_citations":[{"cited_title":"[Online]","cited_arxiv_id":null,"evidence_quote":"supplies the latent diffusion model that generates the full-body RGB demonstration images from text prompts"},{"cited_title":"Telling left from right: Identifying geometry-aware semantic correspondence,","cited_arxiv_id":null,"evidence_quote":"offers raw semantic correspondences between generated and simulated images that the geometry refinement then corrects"},{"cited_title":"Wham: Reconstructing world-grounded humans with accurate 3d motion,","cited_arxiv_id":null,"evidence_quote":"extracts the 3D human joint positions and orientations used to define retargeting constraints"},{"cited_title":"Whole-body motion planning with centroidal dynamics and full kinematics,","cited_arxiv_id":null,"evidence_quote":"contributes the centroidal dynamics and whole-body kinematics formulation underlying the trajectory optimization"}],"review_version":1}