{"id":"39156700-a638-4cfe-91a1-acdaa7d7836a","arxiv_id":"2505.01182","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"TSTMotion is a training-free framework that uses LLM-generated skeleton guidance and two gradient losses to make blank-background motion diffusion models produce scene-aware, text-driven motions.","lead":"This paper introduces a training-free pipeline that combines an LLM with a pre-trained text-to-motion diffusion model so it can generate motions inside a 3D scene. A smart generalist might read it because it promises scene-aware motion generation without the expensive motion-capture datasets that trained baselines require.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LLM Motion Planner's 3D keyframe generation from 2D scene auxiliaries is untested and is the linchpin of the training-free claim; a wrong plan cannot be corrected by the checker.","rationale":"The reader's weakest_assumption identifies the LLM Motion Planner's geometric inference as the untested linchpin; my analysis agrees and sharpens the specific failure mode: the representational gap between the 2D auxiliary maps and the required 3D keyframe coordinates, plus the inability of the Motion Checker to correct a wrong plan. This concern is load-bearing because the entire training-free framework depends on the planner's output—if the guidance is wrong, Eq. 6 aligns the generated motion to the wrong location, and the checker's restart cannot fix it if the LLM repeats the same error. The proposed test directly isolates the planner's accuracy and the checker's role, settling whether the concern lands. I do not see a fatal mathematical error, so I keep the reader's CONDITIONAL verdict unchanged, but the condition is now more explicit: the planner must be validated in isolation and the checker's masking effect quantified.","tokens_in":9076,"tokens_out":10017,"duration_ms":104222,"concrete_test":"Run the Scene Compiler + Motion Planner on 50 randomly selected HUMANISE test prompts using the exact prompt templates. For each prompt, compare the LLM's predicted keyframe joint positions (e.g., pelvis at the goal frame) to the ground-truth motion's corresponding joint positions after aligning coordinate frames. If the median Euclidean error exceeds 0.5 meters, or if the LLM selects the wrong target object (e.g., TV instead of couch) for any prompt, the guidance is unreliable. Additionally, run the full pipeline with the Motion Checker disabled on these same 50 prompts; if the resulting motions do not reach the correct object or have unnatural contact, the checker is merely masking planner failures.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that the Motion Planner (Sec. II-B) convert a 2D road map and a height map of the target object into a sparse set of 3D joint keyframes (s[Mmask]) that correctly localize the interaction and encode the required action. This is an unvalidated representational leap: the LLM sees only a top-down occupancy grid and a height field, yet must output absolute 3D coordinates in the scene's frame. The road map does not explicitly provide the floor plane or scale, and the height map covers only the target, so the LLM must implicitly infer the coordinate origin, floor height, and whether free cells are reachable. If the LLM's keyframe positions are offset (e.g., pelvis coordinates that intersect the target or float in the air), Eq. 6's gradient will faithfully pull the MDM's motion to those wrong coordinates. The Motion Checker (Sec. II-D) can only restart the same LLM; it cannot repair the geometry. The paper reports no isolated accuracy of the Motion Planner, no success rate of the checker, and no failure case analysis. Without this evidence, the entire framework rests on an untested assumption about LLM spatial reasoning, and the 'superior performance' claim is not yet justified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TSTMotion, a training-free framework for scene-aware text-to-motion generation. It composes a Scene Compiler that converts a 3D scene into a road map and a height map, an LLM-based Motion Planner that produces sparse 3D skeleton keyframes as motion guidance, a pre-trained motion diffusion model modified at inference time by two gradients (Eq. 6 aligns the predicted clean motion to the guidance; Eq. 8 penalizes penetration into the scene mesh), and an LLM-based Motion Checker that restarts the pipeline when the output fails scene/text criteria. Experiments on HUMANISE and AffordMotion, plus qualitative results on PROX, are used to claim superior performance over training-based baselines and generalization to outdoor scenes.","tokens_in":9347,"tokens_out":5857,"duration_ms":60265,"significance":"If fully validated, the framework would be practically attractive because it avoids scene-motion training data and model weight updates, while leveraging pre-trained blank-background motion diffusion models. The paper has clear strengths: it releases code/project page, decomposes the problem into interpretable foundation-model components, and provides ablations showing each component contributes on the chosen metrics. The core idea of conditioning a pre-trained MDM via posterior-style gradients on an LLM-generated skeleton guidance is coherent and worth pursuing. However, the current validation is insufficient in two load-bearing respects: the central LLM planning step is not evaluated in isolation, and several headline metrics largely restate the optimization objectives rather than providing independent evidence of scene-awareness.","major_comments":[{"comment":"The Motion Planner is the linchpin of the training-free claim: it must convert a 2D road map, a height map of the target, and a text prompt into absolute 3D joint keyframes that correctly localize the target and encode the interaction. The manuscript provides no isolated evaluation of this conversion—no success rate for the planner, no Motion Checker acceptance statistics, and no failure-case analysis. Since Eq. (6) faithfully pulls the generated motion toward s[Mmask] and the Motion Checker (Sec. II-D) can only restart the same planner, a plan with a mislocated target or an unreachable pelvis position cannot be corrected downstream. I therefore cannot yet accept the claim of superior performance over training-based methods; the paper should add an oracle-style ablation (e.g., replacing the LLM plan with ground-truth contact/goal keyframes) and report planner/checker success rates.","section":"Section II-B and Fig.2"},{"comment":"The construction of the dense guidance tensor s[Mmask] is underspecified. The LLM output in Fig.2 is a small set of sparse keyframes (keyframe_1, keyframe_20, keyframe_40) with only a 'pelvis' joint, whereas Eq. (5) requires a skeleton sequence in R^{N x J x 3} over all N frames and masked joints. No interpolation, forward-kinematic initialization, or temporal filling procedure is described. Without this step, the alignment loss in Eq. (5) cannot be evaluated, so the main mechanism of the method is not fully reproducible from the text. Please specify the exact conversion from the LLM JSON to s[Mmask].","section":"Section II-B and Eq. (5)"},{"comment":"Three of the headline metrics are near-verbatim restatements of the two inference-time losses. Body-to-goal distance and contact score measure the same proximity to the target that Eq. (6) is explicitly minimizing, and non-collision score measures the same penetration avoidance as Eq. (8). Consequently, the improvements in Tables I and III are partly guaranteed by construction and do not constitute independent evidence that the generated motions are scene-appropriate. The independent evidence (FID, R-Precision, Multimodal Distance) is reported only for the AffordMotion benchmark; the paper should also report those metrics on HUMANISE, or otherwise separate the optimization objective from the evaluation.","section":"Section III-A and Section II-C"},{"comment":"The claim that 'only one iteration is necessary' for the Motion Checker is not supported by any experiment varying the iteration count; the improvement from w/o Motion Checker to Ours in Table III is also small (e.g., contact 0.89 vs 0.91). Since the checker is presented as a core component and a restart mechanism, the paper should either show a saturation curve over iterations or soften the claim.","section":"Section II-D and Table III"}],"minor_comments":[{"comment":"The gradient is written as ∇_{x_k} but the loss is evaluated on x̂_k^0; please clarify whether the update is w.r.t. x_k through the denoiser map or directly on x̂_k^0, and state the gradient-stop convention.","section":"Eq. (6)"},{"comment":"The text says 'As shown in Table I' when discussing ablation variants; the results are actually in Table III.","section":"Section III-D"},{"comment":"Only single values for λ=2 and η=0.5 are given; since these hyperparameters control the two training-free modifications, a sensitivity analysis would strengthen the reproducibility of the claims.","section":"Section III-B"},{"comment":"Quality and action scores are described as perceptual studies, but no details on participant number, instructions, or consistency are provided, which makes the significance tests in Table I hard to interpret.","section":"Section III-A"},{"comment":"There are minor typos: 'Mode,l' in the Section II overview paragraph and 'Grountruth' in Table II should be corrected.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"I recommend major revision. The two principal concerns are substantive but fixable within the scope of the manuscript: the LLM Motion Planner needs an isolated validation or oracle ablation, and the evaluation needs metrics that are not direct restatements of the optimization losses. The missing interpolation step from sparse keyframes to dense s[Mmask] is also a reproducibility gap that should be easy to address. If the authors provide this evidence, I would be willing to support acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe thing to know: TSTMotion is a real training-free scene-aware text-to-motion system. It steers a pretrained blank-background motion diffusion model using LLM-generated skeleton keyframes plus two gradient modifications. The idea is timely and the assembly is non-obvious. But the empirical case is weaker than the text claims, mainly because the evaluation metrics are close to the optimization losses.\n\nWhat is new and good: prior scene-aware methods train on paired text-motion-scene data; this paper instead uses a Scene Compiler to render a 3D scene into a road map and a height map, a Motion Planner (GPT-4) that outputs sparse 3D joint keyframes, and an Aligned MDM that applies Eq. 6 (align to keyframes) and Eq. 8 (penalize SDF penetration). That division of labor is clean. The ablations show each component helps on the reported metrics, and the qualitative results show diversity — cartwheels, chicken walks — that training-based HUMANISE models cannot do. For a subfield bottlenecked by data collection, this is a useful direction.\n\nSoft spots, in rough order of size. First, the main quantitative metrics are partly circular. Contact score, non-collision score, and body-to-goal distance are near-verbatim proxies for Lscene and Lalign. The Motion Checker only restarts from the same LLM when alignment fails, so high scores are partly guaranteed by construction. Second, and more load-bearing: the Motion Planner's mapping from a 2D road map plus height map to absolute 3D keyframe coordinates is never validated in isolation. If GPT-4 places the pelvis a meter off, Eq. 6 will faithfully pull the generated motion to the wrong place. The checker can filter but not repair. The paper reports no Planner accuracy, no checker success rate, and no failure cases. That is the assumption the whole training-free claim rests on. Third, minor: Table I reports no error bars for three of five metrics, the prompt templates are not released despite the code claim, and only one LLM and one MDM are tested, so generalizability is asserted more than shown.\n\nThere is no fatal math error; the framework is coherent. But \"superior performance\" should be read as \"promising and plausible,\" not established. The citation pattern is fine — they compare against the right baselines.\n\nWho this is for: anyone working on human-scene interaction, embodied AI, or training-free conditioning of diffusion models. A serious referee should demand isolated Motion Planner evaluation, at least one non-aligned metric, error bars, failure analysis, and released prompts/code. I would engage with it.\n\nRecommendation: send to peer review.","headline":"A genuinely training-free scene-aware text-to-motion pipeline worth engaging with, but the quantitative evidence is partly circular and the LLM Motion Planner — the linchpin — is never validated in isolation.","tokens_in":9892,"tokens_out":2676,"would_cite":true,"duration_ms":26708,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Text-to-motion can be made scene-aware with zero training by steering a diffusion model with LLM-planned keyframes.","keywords":["training-free","scene-aware text-to-motion","motion diffusion models","LLM motion planning","signed distance field","foundation models","human-scene interaction","gradient conditioning"],"falsifier":"Take a fixed 3D scene, run the Scene Compiler, and ask the Motion Planner to output keyframe pelvis positions for 'sitting on the couch'; compare those positions to the couch's true location. If the planner's sitting keyframes land more than 0.5 meters from the couch in a majority of repeated trials, the downstream alignment gradient would faithfully produce motions to the wrong place, contradicting the central claim.","tokens_in":8872,"feed_emoji":"🕺","tokens_out":8717,"duration_ms":71450,"temperature":0.7,"pith_summary":"The paper introduces TSTMotion, the first training-free framework for scene-aware text-to-motion generation. It claims that a pre-trained blank-background text-to-motion diffusion model can be made scene-aware at inference time by conditioning it on a motion guidance -- a masked skeleton sequence of keyframe joint positions -- produced by an LLM from a scene-derived roadmap and height map. Two training-free gradient modifications in the reverse diffusion process enforce alignment with this guidance and penalize penetration into the scene mesh, so no scene-aware motion dataset and no model weight updates are needed. If this works, scene-aware motion generation becomes cheap and generalizes to new indoor and outdoor scenes, and the paper reports superior or competitive results against training-based baselines on HUMANISE, AffordMotion, and PROX.","feed_headline":"Scene-aware text-to-motion with zero training beats trained models","feed_subtitle":"An LLM plans keyframes; two gradient edits steer the diffusion model into the scene.","key_machinery":"The central object is the motion guidance $s[M_{\\text{mask}}]$: a sequence of 3D joint positions over a few keyframes, masked to the joints that matter for the specified text–scene interaction. The mechanism that carries the argument is the pairing of this guidance with two training-free gradient modifications applied during the reverse diffusion of the motion model: an alignment loss (Eq. 5–6) between the diffuser's predicted clean motion and the guidance, and a contact/penetration penalty (Eq. 7–8) built from a signed distance field of the scene mesh over the skinned SMPL body. The guidance itself comes from an LLM Motion Planner that reads the road map and the height map, so the whole pipeline depends on the LLM's capacity to turn 2D layout into plausible 3D joint placement.","core_discovery":"TSTMotion's central claim is that scene awareness can be injected into blank-background motion diffusion models purely at inference time. The framework compiles a 3D scene into a 2D road map of walkable areas and a height map of the target object, asks an LLM (the Motion Planner) to output a sparse skeleton guidance $s[M_{\\text{mask}}]$ -- joint positions for the joints and frames relevant to the interaction -- and then steers the diffusion sampler with two gradients: one that aligns the model's predicted clean motion (mapped by forward kinematics) to $s[M_{\\text{mask}}]$ (Eq. 6), and one that pushes the SMPL mesh of the body out of the scene's signed distance field (Eq. 8). The authors argue that foundation models already encode enough knowledge about scenes, skeletons, and motion semantics to supply this guidance, so the pipeline avoids specialized data and training. They support the claim with quantitative comparisons on HUMANISE and AffordMotion and qualitative results in PROX and unseen outdoor scenes.","pith_inferences":["The Motion Planner is the likely bottleneck: if the LLM mislocates the target or chooses the wrong contact joints, the alignment gradient faithfully generates motion to the wrong place, and the Motion Checker can only filter or restart, not correct the error.","A testable extension is to sample multiple planner proposals per scene and let the Motion Checker select the best one, which could raise success rates on complex interactions without any training.","Since the alignment gradient is added at every denoising step with fixed strength, a schedule that anneals $\\lambda$ over the reverse process might improve stability for long motions.","The Scene Compiler's road map + height map encoding might be reusable as a general way to feed 3D scenes to any LLM-based planner, not just this motion pipeline."],"forward_implications":["Scene-aware text-to-motion generation no longer requires collecting paired text–motion–scene datasets, which lowers the cost for embodied AI, games, and film.","Any pre-trained blank-background motion generator can be repurposed for a new 3D scene by recomputing the spatial auxiliary and re-running the planner, so action diversity scales with the LLM's knowledge rather than dataset annotations.","The gradient-alignment trick (Eq. 6) offers a general recipe for conditioning a motion diffusion model on partial skeleton observations at inference time, applicable beyond scene interaction.","The SDF-based penetration penalty (Eq. 8) is a plug-and-play collision-avoidance module that can be grafted onto other skinned motion generators."],"supporting_citations":[{"why":"Supplies the HUMANISE benchmark, the baseline method, and the perceptual quality/action score protocol used for evaluation.","marker":"[3]"},{"why":"Supplies the AffordMotion test set, the training-based AffordMotion baseline, and the additional text-motion metrics (R-Precision, FID, etc.).","marker":"[6]"},{"why":"Provides the diffusion posterior sampling view that justifies modifying the reverse diffusion via gradients.","marker":"[14]"},{"why":"Provides the SMPL body model used to skin the skeleton for computing the scene penetration penalty.","marker":"[15]"},{"why":"Provides the RAM image tagging model used in the Scene Compiler to recognize objects in rendered images.","marker":"[17]"},{"why":"Provides the OpenIns3D 3D segmenter used to assign object categories and occupancy in the Scene Compiler.","marker":"[18]"},{"why":"Serves as the LLM (GPT-4) deployed for the Motion Planner and Motion Checker.","marker":"[19]"},{"why":"Provides the pre-trained blank-background motion diffusion model (OmniControl) that the framework conditions.","marker":"[20]"}],"fun_headline_variants":["Training-free scene-aware text-to-motion via LLM and gradients","No-training scene-aware motion: LLM guides diffusion through 3D","Scene-aware text-to-motion without scene-ground truth","LLM plans keyframes, gradients steer motion into any scene","Zero-training scene-aware motion from blank-background models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that an LLM, supplied only with a 2D road map and a height map of one target object, can produce a geometrically plausible 3D skeleton keyframe path that correctly locates the object and encodes the requested interaction.","fun_headline_variants_meta":{"raw":{"variants":["Training-free scene-aware text-to-motion via LLM and gradients","No-training scene-aware motion: LLM guides diffusion through 3D","Scene-aware text-to-motion without scene-ground truth","LLM plans keyframes, gradients steer motion into any scene","Zero-training scene-aware motion from blank-background models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000658,"raw_usage":{"total_tokens":3018,"prompt_tokens":959,"completion_tokens":2059,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":575,"completion_tokens_details":{"reasoning_tokens":1974}},"tokens_in":575,"tokens_out":2059,"duration_ms":15482,"temperature":1.0,"reasoning_tokens":1974,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:23:53.986975+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fixed 3D scene, run the Scene Compiler, and ask the Motion Planner to output keyframe pelvis positions for 'sitting on the couch'; compare those positions to the couch's true location. If the planner's sitting keyframes land more than 0.5 meters from the couch in a majority of repeated trials, the downstream alignment gradient would faithfully produce motions to the wrong place, contradicting the central claim.","supporting_citations":[{"cited_title":"Humanise: Language-conditioned human motion generation in 3d scenes,","cited_arxiv_id":null,"evidence_quote":"Supplies the HUMANISE benchmark, the baseline method, and the perceptual quality/action score protocol used for evaluation."},{"cited_title":"Move as You Say, Interact as You Can: Language-guided Human Motion Generation with Scene Affordance","cited_arxiv_id":"2403.18036","evidence_quote":"Supplies the AffordMotion test set, the training-based AffordMotion baseline, and the additional text-motion metrics (R-Precision, FID, etc.)."},{"cited_title":"Ex- pressive body capture: 3D hands, face, and body from a single image,","cited_arxiv_id":null,"evidence_quote":"Provides the SMPL body model used to skin the skeleton for computing the scene penetration penalty."},{"cited_title":"Recognize anything: A strong image tagging model,","cited_arxiv_id":null,"evidence_quote":"Provides the RAM image tagging model used in the Scene Compiler to recognize objects in rendered images."}],"review_version":1}