{"id":"3716a6e2-3e6e-4e6d-93e9-b2fc60782b9c","arxiv_id":"2507.16813","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A diffusion-transformer method that composes a foreground object into a human image with MLLM-chosen interaction regions, pose keypoint supervision, and appearance/background consistency losses, plus a new paired dataset.","lead":"HOComp is a system for putting a product or object into a photo of a person so the person appears to naturally hold, carry, or use it while both the person and the object keep their original look. It uses a vision-language model to pick where and how the interaction happens, then fine-tunes a diffusion model with pose and appearance losses; the authors also introduce a dataset and benchmark for this task.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unvalidated FLUX.1 Fill inpaintings define both the training masks (Sec 3.4) and the SSIM(BG) evaluation; if they alter pose, shadow, or cloth, the background-consistency claim is inflated.","rationale":"The reader's weakest assumption—that FLUX.1 Fill inpaintings may be unfaithful to real pre-interaction states—is also the most load-bearing uncertainty for the background-consistency half of the paper's central claim. I would phrase the mechanism slightly differently: Lbackground's target x_GT is the original composited image, so the concern is best understood as unvalidated masks and an unvalidated input distribution rather than the loss literally copying inpainting artifacts. Either way, the paper provides no evidence that Ib and M_unchanged represent the true 'before' state, and the same kind of derived masks feed the SSIM(BG) evaluation. The proposed check directly tests this fidelity. If it passes, the conditional can be lifted; if it fails, the SSIM(BG) and DCAP ablation results need to be reinterpreted. The user study and HOI-Score results independently support interaction quality, so this concern does not reject the paper—it keeps the verdict CONDITIONAL and makes code/data release a concrete requirement for verification.","tokens_in":22821,"tokens_out":16202,"duration_ms":194954,"concrete_test":"On 100 randomly selected real IHOC samples, run DWPose on the original composited image IGT and on the FLUX.1 Fill background Ib. Restrict to the unchanged-region mask M_unchanged, excluding the foreground object, and compute the fraction of pose keypoints that move more than 10 px (at 512x512) between IGT and Ib, plus mean SSIM/LPIPS in M_unchanged. If >5% of keypoints shift or if the background SSIM in M_unchanged is substantially below the reported 96.57, the training and evaluation masks are not faithful; then re-run Table 1 SSIM(BG) on HOIBench with masks recomputed to cover all interaction-induced changes and check whether HOComp's 5.86-point margin over AnyDoor survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim has two halves: interaction quality and appearance/background consistency. The background half rests on Lbackground (Eq. 6) and SSIM(BG)=96.57. Both are defined relative to masks produced by FLUX.1 Fill inpainting. Sec 3.4 and Appendix B.1 create the 'background human image' by removing the interacting object with FLUX.1 Fill, and M_unchanged is the inverse of the manual inpaint mask. The paper never validates that these inpaintings correspond to true pre-interaction states; Appendix B.1 even says FLUX.1 Fill is used to 'reconstruct plausible human poses without interactions,' which explicitly allows pose changes. If the inpainter alters the person's pose, clothing, or lighting, or if the manual mask omits interaction-induced changes such as shadows or cloth deformation, then M_unchanged does not delimit 'unchanged' pixels: Lbackground supervises the wrong set of pixels, and SSIM(BG), computed over the complement of a similarly derived interaction region, rewards agreement with that same synthetic definition. Table 2 shows Lb is the dominant term for SSIM(BG) (96.57 -> 58.49 without it), so the reported background-consistency advantage over AnyDoor (90.71) is not independent of the unvalidated inpainting pipeline. This is load-bearing because a core contribution claim—'faithful reproduction of the background human'—would be unsupported if the inpainting ground truth is unfaithful.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HOComp, a diffusion-transformer-based method for compositing a foreground object onto a human-centric background image while aiming for natural human-object interaction and appearance consistency. The method uses an MLLM (GPT-4o) to generate the interaction prompt, object box, and interaction region; a pose-guided loss within the interaction region; shape-aware attention modulation; a multi-view appearance loss; and a background consistency loss. The authors introduce the IHOC dataset (built from HICO-DET real images and FLUX.1 synthetic images, with backgrounds produced by FLUX.1 Fill inpainting) and the HOIBench benchmark, and report results on FID, CLIP-Score, HOI-Score, DINO-Score, SSIM(BG), and a user study, claiming state-of-the-art performance.","tokens_in":23110,"tokens_out":5619,"duration_ms":59275,"significance":"If the claims hold, HOComp would be a useful advance in interaction-aware composition, a task that existing composition methods handle poorly. The MLLM-driven coarse-to-fine pose guidance is a sensible way to inject interaction semantics, and the combination of attention modulation, multi-view supervision, and background consistency is principled. The paper includes component ablations, a user study, and an extensive appendix. However, the central empirical claims rest on two currently fragile pillars: the background ground truth is produced by an inpainting model whose fidelity is never validated, and the main quantitative benchmark is the authors' own, with hyperparameters selected on that same benchmark and no independent validation split. The dataset and benchmark are not released, which further limits verification. The significance is therefore conditional on addressing the validity and independence concerns.","major_comments":[{"comment":"The background human images and unchanged region masks are produced by FLUX.1 Fill inpainting with manually annotated masks, and Appendix B.1 states that the inpainter 'reconstruct[s] plausible human poses without interactions,' explicitly allowing pose changes. The paper never validates that these inpaintings correspond to true pre-interaction states, nor that the manual masks cover all interaction-induced changes such as shadows, reflections, or cloth deformation. Because both the background consistency loss Lbackground (Eq. 6) and the SSIM(BG) metric are computed relative to these same synthetic masks, the reported background-consistency advantage (SSIM(BG)=96.57 vs. AnyDoor 90.71; the large ablation drop to 58.49 without Lb) may largely reward agreement with FLUX.1 Fill's reconstruction rather than faithfulness to the actual pre-interaction scene. The authors should validate inpainting fidelity using real before/after pairs or a human study, and should report background consistency on manually verified unchanged regions.","section":"Sec. 3.4, Appendix B.1, Eq. (6), SSIM(BG)"},{"comment":"The loss coefficients (alpha_1, alpha_2, alpha_3), the attention modulation strength alpha, and the guidance scale are all selected by sweeping HOIBench, which is the same benchmark on which the final quantitative claims are made. There is no held-out validation split, and baselines are not given the same tuning opportunity. This selection-on-the-test-benchmark protocol inflates the reported FID, CLIP, HOI, DINO, and SSIM(BG) numbers and makes the comparison with the nine (and additional five) baselines difficult to interpret. The authors should fix hyperparameters on a validation split, report the selection procedure, or give all baselines the same tuning budget; in all cases they should report variance or error bars across the 600 evaluation instances.","section":"Sec. 4 (Benchmark), Appendix Tables 4 and 8"},{"comment":"The shape-aware attention modulation requires a shape prior Mshape, and the text says this mask is 'extracted from the ground-truth image.' At inference there is no ground-truth image, and the paper does not specify how Mshape is obtained at test time. If Mshape is predicted from the input foreground object, the segmentation module must be described and evaluated; if the ground-truth mask is used, the method is not applicable at inference. This gap is load-bearing for the DINO-Score claim (78.21 with SAAM vs. 66.52 without) and for the entire method's practical validity.","section":"Sec. 3.3, Eq. (3), Fig. 2"},{"comment":"The comparison protocol states that 'all methods with public training code are retrained or fine-tuned on our dataset.' This is a non-standard protocol for zero-shot baselines such as AnyDoor and Paint by Example, and the paper does not report the amount of data, number of steps, or hyperparameter tuning applied to each baseline. Fine-tuning all baselines on IHOC can either advantage or disadvantage them relative to their original design, and without a controlled protocol the quantitative superiority claim is hard to interpret. Additionally, the evaluation details for HOI-Score (e.g., how UPT detections are matched to ground-truth interactions) and FID (number of generated images and reference set) are not given.","section":"Sec. 4.1, Table 1 (referred to as Table 10)"}],"minor_comments":[{"comment":"The main text repeatedly refers to 'Table 10' when the main paper contains only Table 1 and Table 2; the referenced table appears to be Table 10 in Appendix H. Please fix the cross-references to the correct table numbers.","section":"Sec. 4.1 and user study"},{"comment":"The HICO-DET citation is inconsistent: it is reference [5] in the main text (Sec. 3.4) but reference [21] in Sec. 4 and Appendix B.1. Use the same reference consistently.","section":"Sec. 4 and Appendix B.1"},{"comment":"The sentence 'Interaction regions are computed by extracting the minimal bounding box of the interaction area within the unchanged region mask' is ambiguous; the interaction area should be the complement of the unchanged region mask, not a subset of it.","section":"Sec. 3.4, Stage 4"},{"comment":"If no pose keypoints fall inside the interaction region Br, the denominator n in Lp is zero; please state how this case is handled.","section":"Eq. (1)"},{"comment":"The user study reports average ranks but provides no significance tests, confidence intervals, or inter-rater agreement; the statement that the results 'significantly outperform' other methods is not supported by the reported statistics.","section":"User study"},{"comment":"The IHOC dataset and HOIBench benchmark are not released; a link or a release plan should be included so that the results can be reproduced and the benchmark can be used by the community.","section":"Dataset and code release"},{"comment":"Figure 2 is dense and the inference-time path for obtaining the shape mask Mshape is not clearly drawn; the caption should explicitly indicate how Mshape is obtained during inference versus training.","section":"Fig. 2"},{"comment":"There is a typo: 'lefting' should be 'lifting.'","section":"Sec. 3.2"},{"comment":"The coefficient sensitivity analysis in Table 4 changes one coefficient at a time while keeping others fixed, but the paper does not discuss how the reported best configuration was selected or whether the selected values generalize beyond HOIBench.","section":"Appendix D"}],"recommendation":"major_revision","confidential_remarks":"The paper's central empirical claim rests on its own unreleased benchmark and on FLUX.1 Fill inpaintings as ground truth for background consistency. Before considering acceptance, the authors should be asked to validate the inpainting fidelity, move hyperparameter selection to a validation split, clarify the inference-time shape mask, and release the dataset and benchmark. The repeated erroneous cross-references to 'Table 10' in the main text also suggest the manuscript was assembled hastily and needs a careful revision pass."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look if you care about image composition. The genuinely new thing is the task: take a foreground object and a human-centric background, automatically decide where and how the person interacts with it, then generate the composite while preserving object identity and background details. The IHOC dataset (11,700 paired samples, 117 interaction types) is the first paired pre/post-interaction resource for this setup, and the paper runs extensive experiments: ablations, a user study, comparisons against nine methods plus five more in the appendix. The method itself is mostly assembled from known pieces — DINOv2 identity features, AnyDoor-style detail encoder, attention modulation, Zero123+ multi-view loss, CLIP loss — but the integration is coherent, and the pose loss localized to the interaction region is a reasonable idea.\n\nThe soft spot that matters: the background-consistency claim is built on masks from FLUX.1 Fill. Stage 4 of dataset construction removes the object from the composited image using inpainting, and the \"unchanged region mask\" is the complement of that inpainted area. The paper never validates that these inpaintings correspond to actual pre-interaction states; the appendix even says FLUX.1 Fill is used to \"reconstruct plausible human poses without interactions,\" which explicitly allows pose changes. If the inpainter alters pose, clothing, lighting, or misses shadows and reflections, then Lb supervises the wrong pixels and SSIM(BG) rewards agreement with that same synthetic definition. Table 2 shows removing Lb drops SSIM(BG) from 96.57 to 58.49, so the reported background advantage is largely driven by that loss term. That is load-bearing for the \"faithful reproduction of the background human\" contribution.\n\nThere is also a real circularity burden in the evaluation: interaction types, training data, and the UPT HOI metric all trace to HICO-DET, and hyperparameters/guidance scale were selected on HOIBench. No code, data, or evaluation scripts are released, and there are no error bars or significance tests. The user study is a genuinely independent signal and it strongly favors HOComp, which mitigates the circularity but does not remove the audit problem. The authors openly state the MLLM region accuracy is 91.33% on their benchmark, which is honest.\n\nFor a peer review: yes, send it out. The task framing and dataset are useful to the composition and HOI communities regardless of the exact numbers. The main asks should be: release the dataset and benchmark, validate the inpainted backgrounds against real pre-interaction states (even a small human evaluation would help), and report variance across seeds. The central claim is plausible, but not independently auditable as submitted.","headline":"Solid new task and benchmark for interaction-aware composition, with a real audit gap: the background-consistency numbers are defined relative to the same FLUX.1 Fill inpaintings used to build the training masks.","tokens_in":723,"tokens_out":978,"would_cite":true,"duration_ms":31287,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"HOComp composes a foreground object onto a human photo with a natural interaction while keeping object and person appearance unchanged, and reports the best scores on its HOIBench benchmark.","keywords":["human-object composition","image composition","diffusion transformer","multimodal LLM","pose guidance","appearance preservation","human-object interaction dataset","HOIBench benchmark"],"falsifier":"Collect real photographs of the same person immediately before and during an interaction, run FLUX.1 Fill on the during-photo with the object masked out, and compare the inpainted before-image with the real before-image: if pose keypoints, SSIM over the supposed unchanged region, or per-pixel colors differ systematically beyond typical inpainting error, then the training targets and the SSIM(BG) metric encode artifacts of the inpainting model rather than true pre-interaction states.","tokens_in":22574,"feed_emoji":"🖼️","tokens_out":12145,"duration_ms":110332,"temperature":0.7,"pith_summary":"HOComp is a diffusion-transformer framework for composing a foreground object, such as a perfume bottle, onto a human-centric background image so that the person actually interacts with it, while both the person's identity and the object's shape and texture remain unchanged. The paper argues that existing composition methods fail at this task because they treat the human as part of the background and only harmonize the inserted object, producing implausible gestures or altering the person. HOComp instead separates the problem into two mechanisms: a coarse-to-fine pose controller that uses a multimodal LLM to decide where and how the interaction happens and supervises pose keypoints only in that region, and an appearance-preservation module combining attention modulation, multi-view comparison, and a background-consistency loss. To train and test it, the paper introduces IHOC, a dataset of 11,700 paired before/during interaction images covering 117 interaction types, and reports that HOComp outperforms nine existing methods on FID, CLIP-Score, HOI-Score, DINO-Score, and SSIM(BG). If these results hold, the method offers a route to automatic, identity-preserving product-in-photo composition and downstream video generation.","feed_headline":"Model composes humans and objects with natural interaction","feed_subtitle":"HOComp auto-picks the interaction and region, then preserves object shape and background identity, beating nine methods.","key_machinery":"The load-bearing machinery is the pair of modules MRPG and DCAP acting on a fine-tuned FLUX.1 diffusion transformer. MRPG is the coarse-to-fine interaction controller: at the coarse level, the multimodal LLM outputs the text prompt, object bounding box, and interaction region $B_r$, which is converted to a mask and fed as a condition; at the fine level, a pose estimator extracts keypoints from the predicted and ground-truth images and the localized loss $L_p$ supervises only the keypoints inside $B_r$. DCAP is the three-part appearance keeper: shape-aware attention modulation applies a residual adjustment scaled by the object shape prior $M_{shape}$ to the foreground-relevant attention maps, the multi-view appearance loss uses a multi-view generator and CLIP features to enforce semantic texture agreement, and the background consistency loss $L_{background}$ computes L2 distance between predicted and ground-truth pixels where the unchanged-region mask $M_{unchanged}$ is active. The full training objective sums the denoising loss with these three terms, and the IHOC dataset supplies the paired images and masks needed to supervise them.","core_discovery":"On its own terms, the paper's central claim is that interaction-aware human-object composition is best solved by explicitly decoupling what interaction and where it occurs from how the person and object look. Concretely, HOComp's MRPG module asks a multimodal LLM to produce a text prompt, an object box, and an interaction region, then adds a pose-guided loss $L_p$ that measures the distance between predicted and ground-truth pose keypoints only inside that region, forcing the generated pose to reflect the interaction without changing the body elsewhere. The DCAP module then preserves appearance with three complementary signals: a shape-aware attention modulation that nudges foreground-related attention maps toward the object's mask, a multi-view appearance loss that compares CLIP features of synthesized views of the generated object with views of the input object, and a background consistency loss over the unchanged-region mask. The paper reports that on its new HOIBench benchmark, HOComp achieves an FID of 9.27, HOI-Score of 87.39, DINO-Score of 78.21, and SSIM(BG) of 96.57, all the best among the nine compared methods, and ranks first in the user study on image quality, interaction harmonization, and appearance preservation.","pith_inferences":["Beyond the paper: the large HOI-Score gap between HOComp (87.39) and GPT-4o (75.22) suggests that the pose-guided loss, not just the MLLM's text and box proposals, is doing much of the interaction work; a clean test would be to feed the same model oracle interaction regions from the ground-truth masks and see whether the score changes.","Beyond the paper: because the benchmark's ground truth for background preservation is itself defined by the same inpainting model used to create the training data, the SSIM(BG) metric may reward models that reproduce the inpainter's style rather than true photographic consistency.","Beyond the paper: the same coarse-to-fine pattern could be applied to multi-object scenes, with one interaction region per object, and to video by adding a temporal smoothness term on the pose keypoints; neither is tested in the paper."],"forward_implications":["If the reported scores hold, HOComp is the first method for this task that can take only a human image and an object image and automatically decide both where and how to place the object, removing the manual target-region and text-prompt specification required by prior methods.","The method's success suggests that explicitly separating the interaction constraint (pose within a region) from the appearance constraints (object shape and texture, background identity) is enough to avoid the two failure modes of prior composition methods: unnatural gestures and identity drift.","Because HOComp can be plugged into an image-to-video model, the same interaction-aware composition could generate product-demonstration videos in which a person naturally holds, lifts, or uses the product.","The IHOC dataset, with paired before/after images, interaction regions, and unchanged-region masks across 117 interaction types, provides a training and evaluation resource that did not previously exist for this task."],"supporting_citations":[{"why":"Supplies the FLUX.1-dev diffusion transformer that HOComp fine-tunes as its generation backbone.","marker":"[3]"},{"why":"FLUX.1 Fill removes the object from composited images to produce the background-human training targets and the unchanged-region masks.","marker":"[4]"},{"why":"HICO-DET supplies the 117 human-object interaction types around which the IHOC dataset is organized.","marker":"[5]"},{"why":"AnyDoor supplies the lightweight detail encoder that extracts the high-frequency texture map used to condition the diffusion transformer.","marker":"[9]"},{"why":"GPT-4o is the multimodal LLM that outputs the text prompt, object box, and interaction region in MRPG.","marker":"[48]"},{"why":"The DINOv2-based ID encoder provides the foreground identity features conditioned into the model.","marker":"[49]"},{"why":"SAM segments the foreground object from each composited image to create the input object images used in training.","marker":"[56]"},{"why":"Zero123+ synthesizes the multi-view images on which the multi-view appearance loss is computed.","marker":"[61]"},{"why":"DWPose estimates the pose keypoints used in the pose-guided loss and in the interaction evaluation.","marker":"[89]"},{"why":"The UPT HOI detector computes the HOI-Score used to measure interaction accuracy.","marker":"[94]"}],"fun_headline_variants":["Model auto-picks interaction region for object insertion","Composition that respects human pose and object shape","MLLM guides interaction-aware object composition","Object insertion with consistent appearance and interaction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The weakest assumption is that the synthetic before-interaction images, made by inpainting the object out of each composited training image, show what the person and scene truly looked like before the interaction, with every interaction-induced change such as shadows, reflections, or cloth deformation captured by the unchanged-region mask.","fun_headline_variants_meta":{"raw":{"variants":["Model auto-picks interaction region for object insertion","Composition that respects human pose and object shape","MLLM guides interaction-aware object composition","Object insertion with consistent appearance and interaction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000771,"raw_usage":{"total_tokens":3467,"prompt_tokens":1047,"completion_tokens":2420,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":663,"completion_tokens_details":{"reasoning_tokens":2364}},"tokens_in":663,"tokens_out":2420,"duration_ms":21192,"temperature":1.0,"reasoning_tokens":2364,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:00:46.811147+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect real photographs of the same person immediately before and during an interaction, run FLUX.1 Fill on the during-photo with the object masked out, and compare the inpainted before-image with the real before-image: if pose keypoints, SSIM over the supposed unchanged region, or per-pixel colors differ systematically beyond typical inpainting error, then the training targets and the SSIM(BG) metric encode artifacts of the inpainting model rather than true pre-interaction states.","supporting_citations":[{"cited_title":"Efficient two-stage detection of human- object interactions with a novel unary-pairwise transformer","cited_arxiv_id":null,"evidence_quote":"The UPT HOI detector computes the HOI-Score used to measure interaction accuracy."},{"cited_title":"ChatGPT (model 4o)","cited_arxiv_id":null,"evidence_quote":"GPT-4o is the multimodal LLM that outputs the text prompt, object box, and interaction region in MRPG."},{"cited_title":"Dinov2: Learning robust visual features without supervision","cited_arxiv_id":null,"evidence_quote":"The DINOv2-based ID encoder provides the foreground identity features conditioned into the model."},{"cited_title":"Effective whole-body pose estimation with two-stages distillation","cited_arxiv_id":null,"evidence_quote":"DWPose estimates the pose keypoints used in the pose-guided loss and in the interaction evaluation."}],"review_version":1}