{"id":"26e20e4a-6bda-4b5e-886a-4cb5b8d088b9","arxiv_id":"2505.24315","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"InteractAnything combines LLM-based relation reasoning, inpainting-based object affordance parsing, and force-closure-inspired optimization to synthesize zero-shot 3D human-object interactions from text and an object mesh.","lead":"InteractAnything generates 3D scenes of a person interacting with any supplied object from a single text description, such as 'a person pulls the chair', without training on interaction datasets. It chains a large language model, a 2D diffusion inpainting model, and a pose optimizer to locate contact points and refine the human pose.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 3D contact claim rests on an unvalidated affordance map: Eq. (4) equates 2D projection proximity with 3D contact, so hallucinated or front-facing body parts can corrupt L_inter and L_fc; a BEHAVE ground-truth contact comparison would settle this.","rationale":"The paper's central claim is that the optimization enforces realistic 3D contact. Tracing the contact signal: L_inter (Eq. 7) minimizes chamfer distance between body-part vertices and object vertices, weighted by the affordance map M computed in Sec. 3.3. The affordance map is the only source of 'where to contact', so if it is wrong, the entire contact claim fails. Eq. (4) defines contact probability as a decreasing function of 2D distance between OpenPose keypoints and the object mask in each rendered view; this cannot distinguish a body part located in front of the object from one actually touching it. Since the LLM initializes the human in front of the object (Sec. 3.2), this ambiguity is systematic rather than random. The adaptive inpainting masks (Sec. 3.3) improve the chance of generating the intended body part but not its 3D placement. The paper does not validate P(p) against any ground-truth contact label, and the only evaluations (CLIP and GPT-4V) measure text alignment and visual preference, not physical contact. The proposed test directly targets the load-bearing premise: comparing predicted P(p) to ground-truth contact on BEHAVE objects would either confirm the heuristic or refute it. I agree with the reader's weakest_assumption; the verdict should remain CONDITIONAL pending this validation.","tokens_in":13924,"tokens_out":8093,"duration_ms":105195,"concrete_test":"On BEHAVE objects (backpack, chair-wood, keyboard, suitcase, basketball), for the action prompts used in the paper (grasp, pull, sit, lift), extract ground-truth contact vertices from the dataset's registered human and object meshes at interaction frames using a distance threshold of 2 cm. Take the predicted affordance map P(p) from Sec. 3.3, threshold it to match the ground-truth contact surface area, and compute the F1 score over object vertices. Average over the 20 object-action pairs. If the mean F1 is below 0.4, the 2D-proximity affordance is not reliably predicting 3D contact and the optimization targets incorrect regions; if it is above 0.7, the premise is validated. This directly tests the load-bearing step.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of realistic 3D contact depends entirely on the affordance map M computed in Sec. 3.3. L_inter (Eq. 7) is a chamfer distance between body-part vertices and object vertices, weighted by M; L_fc (Eq. 9) is also computed on object vertices selected by M. If M is wrong, the optimizer is pulled toward incorrect object regions, and the generated contact cannot be physically correct. The map is built from Eq. (4), f_afford(p) = exp(-||d_i(p)||), where d_i(p) is the 2D distance from OpenPose keypoints to the object's projection mask in view i. This equates image-plane proximity with 3D contact, which is ambiguous: a body part rendered in front of the object projects onto the object mask and yields high affordance even with no physical contact in 3D. Because the LLM initializes the human directly in front of the object (Sec. 3.2) and the same placement repeats across the n_p views, the false-positive region is consistent and averaging does not remove it. The adaptive inpainting masks (Sec. 3.3) improve the probability that the intended body part appears, but they cannot correct hallucinated placement or scale of that part. The paper acknowledges reliance on 2D priors (Sec. 5) but never validates that the resulting P(p) corresponds to true contact: there is no comparison with ground-truth contact maps, no sensitivity analysis, and no ablation with a random or ground-truth affordance map. Thus the load-bearing premise of the contact claim is unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes InteractAnything, a zero-shot pipeline for synthesizing 3D human-object interactions (HOI) from a text prompt and an arbitrary object mesh. The method uses an LLM to infer initial human-object relations and choose options for object rotation, translation, scale, and state; a pre-trained 2D diffusion inpainting model together with OpenPose keypoints to produce a per-vertex object affordance map via a 2D distance-based probability function; multi-view SDS to initialize the human pose; and a final coarse-to-fine optimization that includes contact, normal, penetration, scale, ground, and a SMPL-H-adapted force-closure loss. The authors report CLIP similarity scores and GPT-4V selection percentages showing improvements over Magic3D, DreamFusion, DreamFusion*, and DreamHOI, along with ablations of the LLM initialization and fine-grained optimization. The central claim is that the method generates text-aligned, contact-aware, open-set 3D HOIs without training on HOI datasets.","tokens_in":14365,"tokens_out":3694,"duration_ms":47755,"significance":"If the central claims held, the paper would make a useful contribution: a zero-shot, training-free pipeline for open-set 3D HOI synthesis is a timely and practically relevant goal, and the idea of combining LLM commonsense with diffusionpriors is attractive. The paper also has concrete strengths: it explicitly avoids dataset-specific training, proposes a novel affordance-parsing mechanism, and includes ablations suggesting that the LLM initialization and the fine-grained optimization both contribute to the final quality. The release of a project page is a plus. However, the evaluation is thin and the load-bearing contact-validation claim is not tested: the affordance map is built from 2D projection proximity, which is not equivalent to 3D contact, and no ground-truth or simulator-based contact metric is used. The quantitative results also lack variance, trial counts, and independent validation of the GPT-4V preference scores. These gaps prevent the paper from being acceptable in its current form.","major_comments":[{"comment":"The entire contact claim relies on the affordance map P(p) computed from f_afford(p)=exp(-||d_i(p)||), where d_i(p) is a 2D distance in the image plane. Image-plane proximity is not the same as 3D contact: a body part rendered in front of the object but physically separated from it can still project onto the object mask and receive high affordance. Because the LLM-guided initialization places the human in a consistent relative position and the same placement is reused across the n_p views, this false-positive bias is not removed by averaging in Eq. (5). The map directly weights L_inter (Eq. 7) and selects vertices for L_fc (Eq. 9), so any systematic error in P(p) propagates into the optimized pose. The paper does not validate the map against ground-truth contact, does not include a random-map or oracle-map ablation, and its own limitation section (Sec. 5) only acknowledges a general dependence on 2D priors, not this specific projection-to-3D ambiguity. I request a quantitative validation using BEHAVE ground-truth contact vertices (e.g., precision/recall of predicted contact regions) or an equivalent simulator-based contact test, plus an ablation that replaces P(p) with a uniform or randomized affordance map to show that the proposed map itself is informative.","section":"Sec. 3.3, Eq. (4)-(5), and Sec. 3.5, Eqs. (7)-(9)"},{"comment":"The quantitative evaluation is not reported with sufficient rigor. Table 1 gives CLIP similarity scores as single means with no standard deviations, no per-prompt breakdown, and no statistical test; with only 5 prompts and 10 objects, these differences could easily be within noise. Tables 2 and 3 report values such as 45.6 and 52.1 that appear to be GPT-4V selection percentages, but the paper does not state whether these are percentages, how many trials were run, what the variance across trials was, or how the multiple tests mentioned in Sec. 4.2 were aggregated. GPT-4V is also the same model family that provides the LLM initialization and feedback used during generation, so an evaluation that uses GPT-4V to select the 'most reasonable' result may systematically favor outputs that align with the same prior. I ask for error bars and trial counts, a clearer definition of the metric, and either a human-study comparison or an independently validated metric (e.g., contact-distance against a ground-truth dataset) to rule out this shared-source bias.","section":"Sec. 4.2, Tables 1-3"},{"comment":"The force-closure-like loss L_fc is not sufficiently specified to be reproducible or to support the physical-plausibility claim. The symbol f_v(i,j) is described only as 'the force vector applied at vertex j from interacted human vertex i,' but no formula, source, or derivation is given. The sum inside the square and the dependence on normals n(j) are also unexplained. The paper states 'inspired by differentiable force closure [32]' but replaces the base model from MANO to SMPL-H, and it is unclear how the force vectors are computed in practice. Since the abstract and conclusion claim 'realistic 3D contact' and the loss is a central component of the final optimization, the authors should provide a precise definition of f_v, state how contacts are sampled, and ideally evaluate the resulting grasps with a physics-based quality metric or simulator; otherwise, the physical-plausibility claim remains unsubstantiated.","section":"Sec. 3.5, Eq. (9)"},{"comment":"The ablation shows that removing the LLM-guided initialization drops the GPT-4V selection score, but it does not characterize the sensitivity of the method to the LLM's option choices or to the specific LLM model used. Since the initialization determines the inpainting masks (Eqs. 2-3), the affordance map, and the final pose, a different LLM or a different option set could materially change the results. I suggest adding an analysis of how the generated HOI varies when the LLM is replaced by another model or when the option choices are perturbed, so that the contribution of the LLM is more than a single point comparison.","section":"Sec. 3.2 and Sec. 4.4, Table 3"}],"minor_comments":[{"comment":"The notation in these equations is under-specified: V^H, P^h, r_o^{-1}, s_f, s_p, c_i, t_o^{-1}, and the projection transform J are not all defined in the text or in the caption of Figure 2. Please define each symbol and its dimensionality.","section":"Sec. 3.3, Eqs. (2)-(3)"},{"comment":"The 'spatial constraint' that prevents penetration is mentioned but not given an equation or a precise description. Since it is part of the pose-initialization optimization, a clear definition or citation would help reproducibility.","section":"Sec. 3.4"},{"comment":"The evaluation-set description says '5 prompts and 10 objects categories' but then lists 5 BEHAVE objects and 5 generative objects; please clarify the total number of prompts and the number of distinct object meshes used for each comparison.","section":"Sec. 4.1"},{"comment":"Please state explicitly whether the numbers in these tables are percentages, and if so, the total number of trials and the standard error; also indicate what 'multiple tests' means in terms of independent GPT-4V queries.","section":"Sec. 4.2, Tables 2-3"},{"comment":"The notation s'·L_g is unclear: s is defined as the object state in [0,1], but the prime is not defined. Please clarify the intended operation or remove the prime.","section":"Sec. 3.5, Eq. (6) and Eq. (8)"},{"comment":"The limitation paragraph acknowledges the need for simulator-based evaluation and multi-view consistent generation, but these caveats should be connected to the specific affordance-map and force-closure issues raised above; as written, the section reads as a general future-work list rather than a caveat on the current evidence.","section":"Sec. 5"}],"recommendation":"major_revision","confidential_remarks":"The core idea is timely and the method is clearly presented, but the missing validation of the 3D contact map and the weak statistical underpinning of the quantitative claims are the main barriers to acceptance. A revision that adds ground-truth contact evaluation (e.g., BEHAVE-based precision/recall), a random-affordance ablation, and full trial-level statistics for the GPT-4V scores would address my concerns. I would not reject the paper; the approach is defensible and the issues are fixable within the scope of the manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a real integration effort with a sensible decomposition of zero-shot HOI synthesis, but the central contact claim rests on an affordance map that is only indirectly validated. I would send it to reviewers, and I would push hard for quantitative contact evaluation before acceptance.\n\nWhat's actually new: the pipeline is not one new component but a plausible combination—LLM selects discrete relation options and body-part labels; adaptive inpainting masks are generated from those labels; a 2D distance-to-object-mask function is projected to 3D and weights contact; SMPL-H force closure then drives fine optimization. The specific LLM-guided adaptive masking for inpainting, together with the weighted chamfer and force-closure contact optimization, is not in the cited prior work. The ablation in Table 3 does support the claim that both LLM initialization and the refinement stage matter.\n\nWhat's solid: the problem framing is right, and leveraging pre-trained priors without HOI training is attractive. The qualitative results look better than the baselines, and the GPT-4V scores are at least consistent with that. The paper is honest that it relies on 2D priors.\n\nWhere it's soft: the evaluation is too thin for the claim of \"precise, physically plausible\" contact. CLIP means come without error bars. GPT-4V selection is the main metric, and the same model family guides generation and initialization; not fully circular, but it is shared-source evaluation that needs independent validation. There are no objective contact or penetration numbers and no code or data to reproduce. The deeper issue is Eq. (4): image-plane proximity to the object mask is treated as 3D contact probability. A body part rendered in front of the object scores high affordance even with no physical contact. The paper's own limitation section acknowledges the dependence on 2D priors, but never validates against ground-truth contact. BEHAVE objects are already in the test set, so a BEHAVE-based contact comparison, or at least a sensitivity test with a random or ground-truth affordance map, would settle this.\n\nMinor but worth noting: some equations are sketchy, e.g., the normals term without clear vertex indexing and the mixed use of delta in Eq. (9). Understandable, but needs cleanup.\n\nBottom line: a serious attempt, not a throwaway. The load-bearing assumption is not absurd, but it is unverified, and the paper's metric choices obscure that. I would want to see a revision with contact evaluation and more transparent statistics before treating the results as established.","headline":"A genuinely integrated zero-shot HOI pipeline worth serious review, but the load-bearing contact claim needs objective validation before acceptance.","tokens_in":14857,"tokens_out":2110,"would_cite":false,"duration_ms":27104,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"InteractAnything claims that a text prompt plus any 3D object mesh is enough to synthesize a natural, contact-accurate human-object interaction without training on interaction datasets.","keywords":["zero-shot 3D human-object interaction synthesis","object affordance parsing","LLM-guided initialization","2D diffusion inpainting","multi-view score distillation sampling","force closure","contact-aware pose optimization","open-set 3D objects"],"falsifier":"A concrete test: take object-action pairs with known ground-truth interaction meshes, render each object alone from the views used by the parser, run the affordance parser, and compare the predicted contact-probability peak regions against the vertices actually touched in the ground-truth mesh; if the overlap is no better than chance on a held-out set, the central affordance claim fails. A complementary test is to measure contact error, such as vertex-to-surface distance and penetration depth, between the synthesized human mesh and object mesh: systematic floating hands or deep interpenetration would falsify the force-closure refinement claim.","tokens_in":1942,"feed_emoji":"🖐️","tokens_out":3471,"duration_ms":113703,"temperature":0.7,"pith_summary":"InteractAnything sets out to prove that a text prompt and a single 3D object mesh are enough to synthesize a natural, contact-accurate 3D human-object interaction, even for object categories the system has never seen, and without training on human-object interaction datasets. The method answers three sub-questions in sequence: which body part relates to the object and from which side, answered by a large language model selecting from fixed options; where on the object the body should touch, answered by a pre-trained 2D inpainting diffusion model whose outputs are converted into a 3D contact probability map; and what exact pose realizes the interaction, answered by score-distillation pose synthesis followed by force-closure-based contact optimization. The authors argue that prior text-to-3D and HOI methods fail on open-set objects because they either lack relation reasoning, cannot localize affordances on arbitrary geometry, or cannot enforce detailed contact such as hand grasping. If the claim holds, 3D interaction synthesis becomes a zero-shot operation usable for any object mesh, with consequences for AR/VR, simulation, and animation.","feed_headline":"Text plus any object mesh yields 3D human-object interaction","feed_subtitle":"LLM guidance, 2D affordance parsing, and force-closure refinement give contact-accurate poses for unseen objects.","key_machinery":"The carrying mechanism is the open-set object affordance parser coupled with the force-closure refinement. For a given action, the parser renders the object from several views, creates inpainting masks from LLM-selected body-part labels and inferred human-object placement, and asks a pre-trained 2D diffusion inpainting model to draw the human interacting with the object; a 2D pose detector then supplies keypoints, and the distance from each keypoint to the object mask defines the per-view contact probability $f^{(i)}_{\\mathrm{afford}}(p)=e^{-\\|d_i(p)\\|}$, which is averaged over views and re-projected onto the object surface to give a 3D affordance map $M$. That map weights the chamfer contact loss and selects the object vertices used in the finer force-closure loss $L_{fc}=\\sum_{j\\in[o]}(\\sum_{i\\in[h]} f_v(i,j)\\cdot n(j))^2$, where $f_v(i,j)$ is the force applied by interacted human vertex $i$ at object vertex $j$ and $n(j)$ is the object normal. The same affordance map is what makes the method open-set: it comes from 2D diffusion knowledge rather than from 3D training assets.","core_discovery":"On its own terms, the paper's central claim is that zero-shot 3D human-object interaction synthesis can be decomposed into three stages, each delegated to a pre-trained model, and that this decomposition yields fine-grained, text-aligned interactions for open-set objects without any HOI-specific training. The LLM stage turns a short instruction such as 'a person grasps the chair' into concrete initial values: object rotation, translation, scale, on-ground state, and the semantic body parts involved. The affordance stage renders the object into multiple views, writes LLM-guided full-body and body-part inpainting masks, lets a pre-trained 2D diffusion inpainter hallucinate the interacting human, detects 2D body keypoints, and converts per-view distance to the object mask into a contact probability that is aggregated and re-projected onto the 3D mesh. The pose stage uses multi-view score distillation sampling with a spatial constraint that keeps the human outside the object volume, then refines body and hand parameters with a force-closure loss, normal alignment, chamfer contact loss, penetration penalty, scale loss, and an optional ground loss. The final output is a posed SMPL-H human mesh, a parametric body-and-hand model, and an object mesh with corrected pose, scale, and placement, with contact concentrated where the affordance map says it should be.","pith_inferences":["The paper leaves implicit that a confidence gate on the affordance map could prevent the worst failure mode: when the inpainter draws a partial or misplaced body part, the contact map is wrong and the force-closure step optimizes toward that wrong region.","One extension is to replace SMPL-H with other articulated agents: the affordance parsing and force-closure losses are body-model-agnostic, while only the body prior and the pose estimator would need swapping.","A testable extension follows from the 2D origin of the contact map: as multi-view-consistent 2D diffusion models improve, the affordance map should improve automatically with no change to the optimization stages.","The LLM option-selection design reduces hallucination by constraining answers, but it also caps expressiveness: interactions whose relative position, scale, or body part is not among the predefined options cannot be initialized correctly."],"forward_implications":["The same object mesh can host different interactions: 'sits on the chair surface', 'pulls the chair back', and 'lifts the chair bottom' place contact on different regions of the same chair.","Open-set objects such as a motorcycle, a car, a baby doll, or a humanoid robot receive plausible interactions without any object-specific retraining.","Hand-level grasping detail emerges from the force-closure refinement, and the paper's ablation shows this detail is absent when the fine-grained optimization is removed.","LLM-guided initialization is load-bearing: ablating it drops the method below a strong baseline, because a randomly initialized human-object relation cannot be repaired by later optimization.","The generated interactions can be placed into existing 3D scenes with a ground loss, supporting applications such as sitting on a table or opening a door."],"supporting_citations":[{"why":"Introduces score distillation sampling, the gradient machinery used to align the initial human pose with the text prompt and object geometry.","marker":"[43]"},{"why":"Supplies the pre-trained 2D inpainting diffusion model whose outputs are parsed into contact probabilities for open-set objects.","marker":"[42]"},{"why":"Provides the 2D body-keypoint detector that converts inpainting results into the distance-based contact probability function.","marker":"[5]"},{"why":"Contributes the differentiable force-closure estimator that the paper adapts to SMPL-H for fine-grained hand contact optimization.","marker":"[32]"},{"why":"Provides the large language model queried for human-object relation reasoning and option-based initialization.","marker":"[39]"},{"why":"Supplies the multi-view score distillation sampling loss used in text-object driven human pose synthesis.","marker":"[52]"},{"why":"The closest prior baseline; its failure to ensure precise contact motivates the contact-accurate refinement stage.","marker":"[67]"},{"why":"The prior text-driven HOI generation baseline whose weak interaction precision the paper targets.","marker":"[9]"},{"why":"Supplies the dynamic mask inpainting procedure that refines the initial full-body mask using intermediate diffusion results.","marker":"[25]"}],"fun_headline_variants":["Zero-shot 3D HOI from text and any mesh","LLM plus affordance parsing yields open-set interactions","Text-driven 3D poses with contact-aware object handling","Synthesize human-object interaction without training data","Open-set HOI via LLM reasoning and 2D affordance cues"],"cache_read_input_tokens":16896,"weakest_assumption_plain":"The load-bearing premise is that the images produced by the 2D inpainting model, after the language-model-guided masking, show a sufficiently correct human-object interaction that 2D body keypoints and a distance-based contact map give a trustworthy 3D contact probability distribution; if the inpainter draws a missing or misplaced body part, the contact map is wrong and all later optimization pushes the human toward the wrong region.","fun_headline_variants_meta":{"raw":{"variants":["Zero-shot 3D HOI from text and any mesh","LLM plus affordance parsing yields open-set interactions","Text-driven 3D poses with contact-aware object handling","Synthesize human-object interaction without training data","Open-set HOI via LLM reasoning and 2D affordance cues"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000231,"raw_usage":{"total_tokens":1557,"prompt_tokens":1088,"completion_tokens":469,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":704,"completion_tokens_details":{"reasoning_tokens":386}},"tokens_in":704,"tokens_out":469,"duration_ms":5640,"temperature":1.0,"reasoning_tokens":386,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:26:34.482641+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete test: take object-action pairs with known ground-truth interaction meshes, render each object alone from the views used by the parser, run the affordance parser, and compare the predicted contact-probability peak regions against the vertices actually touched in the ground-truth mesh; if the overlap is no better than chance on a held-out set, the central affordance claim fails. A complementary test is to measure contact error, such as vertex-to-surface distance and penetration depth, between the synthesized human mesh and object mesh: systematic floating hands or deep interpenetration would falsify the force-closure refinement claim.","supporting_citations":[{"cited_title":"Dreamfusion: Text-to-3d using 2d diffusion","cited_arxiv_id":null,"evidence_quote":"Introduces score distillation sampling, the gradient machinery used to align the initial human pose with the text prompt and object geometry."},{"cited_title":"Realtime multi-person 2d pose estimation using part affin- ity fields","cited_arxiv_id":null,"evidence_quote":"Provides the 2D body-keypoint detector that converts inpainting results into the distance-based contact probability function."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes the differentiable force-closure estimator that the paper adapts to SMPL-H for fine-grained hand contact optimization."},{"cited_title":"Gpt-3.5.https://platform.openai.com/ docs/models/gpt-3-5, 2023","cited_arxiv_id":null,"evidence_quote":"Provides the large language model queried for human-object relation reasoning and option-based initialization."},{"cited_title":"In- terfusion: Text-driven generation of 3d human-object inter- action","cited_arxiv_id":null,"evidence_quote":"The prior text-driven HOI generation baseline whose weak interaction precision the paper targets."},{"cited_title":"Beyond the contact: Discovering comprehensive affordance for 3d objects from pre-trained 2d diffusion models","cited_arxiv_id":null,"evidence_quote":"Supplies the dynamic mask inpainting procedure that refines the initial full-body mask using intermediate diffusion results."}],"review_version":1}