{"id":"00edef65-4c75-49ad-9bde-f9b974bef2d3","arxiv_id":"2506.12847","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A two-stage inpainting-based video diffusion transformer that reuses pretrained attention to reenact hand-object interactions with novel objects, reporting SOTA performance on Re-HOLD and a new in-the-wild dataset.","lead":"This paper presents iDiT-HOI, a two-stage video diffusion transformer that swaps the object in a hand-object interaction video for a new reference object while preserving the hand motion. It is a practical method for e-commerce and education video production that claims to generalize to unseen objects without adding new network parameters.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (1) pastes novel reference tokens into masked regions, but training only ever shows self-reconstruction; the generalization claim rests on an untested extrapolation that pretrained attention binds arbitrary object tokens to hand context.","rationale":"The paper's novelty and efficiency claims both hinge on Eq. (1) being sufficient: no added parameters, no extra control signals, and generalization to unseen objects. The reader's weakest_assumption pinpoints exactly this step, and I agree that it is the most load-bearing concern. The training distribution contains only self-reconstruction pairs, where the reference object is the same video's object, so cross-object reenactment is a genuine extrapolation. The existing ablations in Table 3 show that the module helps in-domain, but they do not test extrapolation to novel geometry or appearance. A possible objection is that the method is simply image-conditioned inpainting and should trivially transfer; however, the hand must adapt to the new object's shape, and the masked-region token blending imposes no explicit geometric or contact constraint. The evidence for the extrapolation is a 10-video, self-collected HOI-ITW set, with no code or data release, and the reported metrics do not directly measure physical hand-object contact plausibility. A targeted out-of-distribution evaluation would settle whether the concern lands. This reinforces, rather than changes, the reader's CONDITIONAL verdict.","tokens_in":14733,"tokens_out":8043,"duration_ms":92577,"concrete_test":"Assemble an independent cross-reenactment set of 50+ videos whose object categories are provably absent from the 19k training clips (e.g., transparent containers, deformable plush toys, thin-handled tools, reflective metal), with hand/object masks. Run the fine-tuned iDiT-HOI pipeline (or, if code is not released, ask the authors to run it on this set) and compare identity fidelity (DINO/CLIP similarity between the generated object region and the reference image) and hand-object contact quality against in-domain categories. If unseen-dissimilar categories show a sharp drop while seen-similar categories remain strong, Eq. (1) is not sufficient for novel objects and the headline generalization claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3, Eq. (1): X_cond = (1-X_M)*X_tilde + X_M*X_ref^*. In training, X_ref^* is the VAE latent of the very source video's object warped to the mask; the DiT never experiences a mismatch between reference identity and the hand pose/contact that must be generated. At inference, X_ref^* is an unseen object with different geometry, scale, texture, inserted into an adaptive soft mask. The claim of reusing the pretrained model's context perception without new parameters presupposes that self-attention in the fully fine-tuned DiT binds these inserted tokens to the hand region and propagates them as content. Nothing in the objective enforces such binding for out-of-distribution objects; the network could instead treat masked-region tokens as a local prior that only works when the reference resembles training objects. Because the base models are fully fine-tuned, the 'pretrained context perception' is also partly overwritten, so the extrapolation is even less guaranteed. The only direct evidence is a 10-video, self-collected HOI-ITW cross-reenactment set; the paper does not establish that those objects are far from the 19k training clips, and subject consistency/motion smoothness do not measure hand-object contact quality. Thus the central generalization claim rests on this untested sufficiency of Eq. (1).","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents iDiT-HOI, a two-stage video diffusion transformer framework for hand-object interaction (HOI) reenactment. The core contribution is an inpainting-based token processing unit (Inp-TPU) that blends masked source-video latent tokens with reference-object latent tokens via Eq. (1): X_cond = (1 - X_M) * X_tilde + X_M * X_ref^*. The first stage generates a key frame by inserting the reference object into the hand region; the second stage generates the remaining frames conditioned on that key frame. The model is fully fine-tuned from pretrained Wan-14B and FLUX.1-dev models, and is trained in self-reconstruction mode on 19,000 video clips. At inference, the reference object can be replaced, enabling cross-reenactment. The authors evaluate on the Re-HOLD dataset and a self-collected HOI-ITW dataset, reporting quantitative metrics, ablations, and a user study, and claim state-of-the-art performance and strong generalization to unseen objects.","tokens_in":14990,"tokens_out":5976,"duration_ms":64533,"significance":"The proposed method is practically appealing: it avoids adding new network modules or parameters, uses a two-stage key-frame-then-video design for long-video generation, and builds on strong pretrained DiT backbones. If the generalization claims hold, the method could make in-the-wild object-swapping HOI reenactment practical. The authors provide machine-verifiable tables and ablations, but no released code or dataset, and the reported superiority is not consistently supported by the presented numbers. The central generalization claim rests on an extrapolation from self-reconstruction training to cross-object inference, with only a 10-video in-the-wild test set as direct evidence. Strengths of the paper include the formulation of a parameter-free token blending mechanism and a two-stage pipeline that naturally extends to long videos. The empirical support, however, is weaker than the claims and needs careful reanalysis.","major_comments":[{"comment":"The claim that the method \"achieves top performance in 4 out of 6 metrics and ranks second in 1\" is not supported by Table 1. In cross-reenactment, Ours* has subj. cons. 0.948, which is below Re-HOLD* (0.958) and AnimateAnyone (0.950); mot. smth. 0.995 ties with Re-HOLD* and V ACE*; PSNR 33.74 is below V ACE* (35.86). Only FID (12.07) is clearly best. In self-reenactment, mot. smth. is best (0.996) and subj. cons. ties with Re-HOLD* (0.958). A strict count gives at most 2 clear firsts and 1 clear second, plus a few ties. Similarly, the statement in §4.3.2 that the method achieves top performance in 7 out of 9 metrics is contradicted by Table 2, where on HOI-ITW temporal consistency V ACE (3.575) exceeds Ours (3.55). The reported counts should be corrected and any tie-breaking rule stated explicitly.","section":"§4.3.1 and Table 1"},{"comment":"The central generalization claim is that Eq. (1), which linearly blends masked source tokens with reference-object tokens, lets the pretrained DiT propagate the novel object's identity into the masked hand region. In training, X_ref^* is the VAE latent of the source object from the same video; the model never sees a mismatch between the reference object identity and the target hand pose at training time. At inference, cross-reenactment inserts an entirely different object. The paper asserts that this works because the model \"reuses established context perception capabilities,\" but the model is fully fine-tuned, so the pretrained capabilities are substantially overwritten, and no training signal enforces that inserted arbitrary object tokens are bound to the hand region. The only direct evidence is the 10-video HOI-ITW cross-reenactment set, with no analysis of how far those objects are from the training distribution, no attention-map analysis, and no contact-quality metric. This is a load-bearing assumption that should be tested explicitly or the generalization claim should be softened.","section":"§3.3 and Eq. (1)"},{"comment":"The user study uses only 10 participants, reports no inter-rater agreement, no error bars, and no statistical significance tests. The text claims that the method \"consistently outperforms competitors across all three evaluated dimensions,\" but Table 2 shows that on HOI-ITW temporal consistency V ACE (3.575) is higher than Ours (3.55). Moreover, subject consistency and motion smoothness do not measure hand-object contact quality, which is the central challenge of HOI reenactment. The evaluation section should either add sufficient statistics and a contact-aware metric or temper the claims to match what the evidence supports.","section":"§4.2 and Table 2"},{"comment":"The HOI-ITW dataset is not released, and the statement that \"we ensure that the data or similar data has not been seen by the trained models\" is an unverifiable assertion. Because the training dataset is a private collection of 19,000 clips, a reader cannot check whether the test objects, scenes, or actions are truly out-of-distribution. For a paper whose main claim is generalization to unseen scenarios, the evaluation should either release the test data or provide a measurable distance between training and test distributions (for example, object-class overlap statistics or retrieval-based nearest-neighbor analysis).","section":"§4.1, Dataset"},{"comment":"The claim of \"reusing pretrained context perception without adding new trainable parameters\" is only literally true in the sense that no new modules are introduced. The implementation section states that the models are trained in a full fine-tuning manner, meaning every existing parameter is updated. This is a different and weaker form of reuse than zero-shot prompting or frozen-backbone conditioning. The phrasing in the abstract and introduction should be revised to acknowledge that all pretrained parameters are fine-tuned, so the efficiency comparison with Re-HOLD and V ACE, which duplicate modules, should be stated relative to parameter count rather than to preservation of the original pretrained weights.","section":"§4.1 and §3.3"}],"minor_comments":[{"comment":"The abstract and conclusion use absolute superlatives such as \"outperforms existing methods\" and \"state-of-the-art performance\" without the caveats that the quantitative table only partially supports; consider making these statements conditional on the corrected metric counts.","section":"Abstract and Conclusion"},{"comment":"The method labels in the captions are missing spaces (\"Re-HOLDOursVA C E\"); this should be fixed for readability.","section":"Figure 2 and Figure 3 captions"},{"comment":"The text uses both \"V ASE\" (reference [45]) and \"V ACE\" (reference [25]); these are different methods and the names should be clearly distinguished to avoid confusion.","section":"§2.2"},{"comment":"The footnote says \"* denotes that metrics are computed on a fixed length of 81 frames,\" but only some methods are marked with the asterisk; the table would be clearer if all compared methods used the same evaluation protocol or if the marking were explained in the caption.","section":"Table 1"},{"comment":"The text reports a \"27.08% drop in FID\" when object information fusion is removed. Since a lower FID is better, this is actually an improvement in FID for the full model; the phrase should be clarified as an improvement or a degradation from the ablated model to the full model.","section":"§4.4"},{"comment":"The sentence \"achieving only 1.1, 1.7, and 1.1 in 3 aspects of the user study\" should include the table reference and the exact values from Table 2 (1.125, 1.7, 1.1) for consistency.","section":"§4.3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper's central idea is interesting and the writing is generally clear, but the empirical claims are overstated relative to the reported numbers. The authors should be asked to re-tabulate the metrics with explicit tie handling, to strengthen the generalization evidence with more than 10 unverified in-the-wild videos, and to either release the HOI-ITW data or provide distributional analysis. If the claims are corrected to match the evidence, the paper could become a solid contribution to the HOI reenactment literature. The lack of error bars and significance tests in the user study is a recurring weakness in this field, but it is still load-bearing given the paper's central claim of superior performance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a serious engineering paper with a plausible mechanism, but the prose overstates the results. The core idea—mask-weighted blending of reference object tokens into the masked region of a video DiT, with no new architectural parameters—is simple and could be useful. The two-stage key-frame-then-video design is a sensible way to keep identity and motion coherent. Training on 19k clips with self-reconstruction is reasonable, and the qualitative results look competitive.\n\nThe main soft spot is the generalization claim. In training the reference object always comes from the same video being reconstructed; at inference you insert a genuinely novel object. That is an extrapolation, and the evidence for it is thin: 10 self-collected HOI-ITW videos, no release, and no demonstration that those objects are far from the training distribution. The stress-test note worries that attention may not bind arbitrary tokens; that is a legitimate concern, but not a fatal one—the model does see many objects during fine-tuning, so the mechanism may generalize more than the training setup naively suggests. The paper just does not do enough to show it.\n\nAlso, Table 1 does not support 'top in 4 out of 6' if ties are counted properly: Ours* loses PSNR and subj. cons. on cross-reenactment, ties on two others, and leads only on FID and self mot. smth. No error bars anywhere, the user study is 10 people, and the 'no new parameters' claim is architectural—the base models are fully fine-tuned, so it is not a parameter-free method. These are presentation issues, not load-bearing flaws.\n\nThe method is honest about its limitations (two-stage cost, no 6D pose control). I would send this to peer review; it deserves referee time. A good reviewer would push for error bars, more cross-object evidence, and toning down the SOTA claims. I would cite it if I worked on HOI reenactment.","headline":"A solid engineering paper with a plausible inpainting-based token blending mechanism, but the generalization claims outrun the evidence and the SOTA numbers are overstated.","tokens_in":15594,"tokens_out":2508,"would_cite":true,"duration_ms":26537,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A mask-weighted token blend lets a pretrained video diffusion transformer reenact hand-object interactions, including with objects it has never seen.","keywords":["hand-object interaction","video reenactment","video diffusion transformer","inpainting","token processing","object swapping","in-the-wild generalization","key frame generation"],"falsifier":"Take the trained model and attempt a cross-reenactment where the reference object has a shape or material far outside its training distribution (for example, a transparent glass, a torus, or a highly reflective object), then check whether the attention maps in the diffusion transformer's self-attention layers bind the inserted reference tokens to the hand region. If the object's tokens are ignored, blended into the background, or copied without adapting to the hand pose, the claimed generalization fails.","tokens_in":14473,"feed_emoji":"🤚","tokens_out":4644,"duration_ms":43813,"temperature":0.7,"pith_summary":"The paper tries to establish that in-the-wild hand-object interaction reenactment — replacing the object in a video while keeping the hand motion — can be done by reusing a pretrained video diffusion transformer as-is, without adding any trainable parameters. The key move is to treat the replacement as an inpainting problem: the masked video tokens are blended with tokens from the reference object image, so the model's existing attention sees the new object as content to propagate rather than as a foreign conditioning signal. The authors support the claim with self-reenactment and cross-reenactment experiments on the Re-HOLD dataset and on a new e-commerce in-the-wild dataset, reporting that their method ranks first on 7 of 9 metrics and that it exceeds the previous state of the art especially for unseen objects. If the claim holds, it makes object swapping in hand videos practical for real-world deployment without per-video tuning or duplicated networks.","feed_headline":"Token blending lets a video model swap objects in hand scenes","feed_subtitle":"A two-stage diffusion transformer reuses pretrained attention to reenact unseen objects with no extra parameters.","key_machinery":"The carrying mechanism is the unified inpainting-based token processing unit (Inp-TPU). It temporally extends the reference object image, spatially aligns it to the masked region, encodes both the masked video and the aligned reference into latent tokens, downsamples the mask, and blends the tokens with $X_{\\text{cond}} = (1 - X_M)\\cdot \\tilde{X} + X_M \\cdot X_{\\text{ref}}^*$. This lets the pretrained DiT's attention parameters operate directly on the reference tokens, reusing established context perception without adding parameters. A two-stage design — an image DiT for the key frame and a video DiT for subsequent frames — plus a soft adaptive mask whose aspect ratio matches the target object provides shape control.","core_discovery":"Given a masked source video and a single reference object image, iDiT-HOI produces a reenacted video by first generating a key frame that inserts the object into the hand region, then generating subsequent frames conditioned on that key frame, both steps using the same Inp-TPU token process. The central discovery is that the conditional tokens can simply be computed as $X_{\\text{cond}} = (1 - X_M)\\cdot \\tilde{X} + X_M \\cdot X_{\\text{ref}}^*$, a mask-weighted sum of the source video tokens and the spatially aligned reference tokens, and that this is enough for a fully fine-tuned pretrained diffusion transformer to keep the new object's identity while maintaining temporal coherence. The method is trained in a self-supervised reconstruction mode, yet it is claimed to generalize to genuinely new objects and unfamiliar in-the-wild videos, outperforming prior methods in most evaluated metrics.","pith_inferences":["The approach suggests a general recipe for reference-conditioned video editing: any VAE-latent DiT with an inpainting mask could in principle inject arbitrary visual references through token blending, not just hand-held objects.","Because the model is trained only on self-reconstruction, its cross-object performance depends on attention robustness to out-of-distribution tokens; a stress test with radically different object geometries (transparent, non-rigid, highly textured) would clarify how far the generalization actually extends.","The same Inp-TPU could be applied to other body parts or to object-in-scene editing, such as replacing furniture or tools in a video, provided the mask and reference alignment are available.","If the reported advantage over the concurrent dual-stream baseline in the in-the-wild setting is real, it indicates that parameter duplication is not necessary for identity preservation and that masked-domain attention reuse is a stronger inductive bias than a separate reference stream."],"forward_implications":["Object swapping in hand-object videos can be done without per-video fine-tuning or duplicated reference networks, lowering deployment cost.","Because subsequent frames are generated video-to-video, the last frame of one clip can seed the next, so long videos follow naturally from the same pipeline.","A pretrained image-to-video diffusion model can be repurposed for reference-image-controlled editing solely by choosing how tokens are blended at the input.","The method works from object images rather than text or layout control, making it more precise for tasks like e-commerce product placement."],"supporting_citations":[{"why":"Supplies the pretrained video diffusion transformer whose attention parameters the method reuses for subsequent-frame generation.","marker":"[56]"},{"why":"Supplies the pretrained image diffusion transformer used for key frame generation.","marker":"[29]"},{"why":"Provides the self-reconstruction strategy that reconstructs both the reference image and the key frame to improve fidelity.","marker":"[41]"},{"why":"Serves as the prior layout-instructed HOI reenactment baseline that the method compares against and aims to surpass.","marker":"[11]"},{"why":"Provides the earlier video-level HOI inpainting framework whose object-centric limitations motivate the new unified token process.","marker":"[68]"},{"why":"Supplies the concurrent dual-stream baseline whose parameter duplication and reference fidelity the method is designed to beat.","marker":"[25]"}],"fun_headline_variants":["Token blending reenacts hand-object scenes with no new params","Inpaint key frame, then video: object swap without retraining","Masked token sum lets video model handle unseen objects","Reusing pretrained attention for hand-object reenactment","Single object image reenacts hand scenes via token blend"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that blending masked-video tokens with reference-object tokens is enough to make the pretrained video-diffusion model treat a never-seen object as content to propagate into the hand region, rather than as out-of-distribution noise; the model is only ever trained on self-reconstruction with the same object, so replacing it with a genuinely new object is an extrapolation.","fun_headline_variants_meta":{"raw":{"variants":["Token blending reenacts hand-object scenes with no new params","Inpaint key frame, then video: object swap without retraining","Masked token sum lets video model handle unseen objects","Reusing pretrained attention for hand-object reenactment","Single object image reenacts hand scenes via token blend"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00019,"raw_usage":{"total_tokens":1354,"prompt_tokens":975,"completion_tokens":379,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":591,"completion_tokens_details":{"reasoning_tokens":294}},"tokens_in":591,"tokens_out":379,"duration_ms":4462,"temperature":1.0,"reasoning_tokens":294,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:36:15.456264+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the trained model and attempt a cross-reenactment where the reference object has a shape or material far outside its training distribution (for example, a transparent glass, a torus, or a highly reflective object), then check whether the attention maps in the diffusion transformer's self-attention layers bind the inserted reference tokens to the hand region. If the object's tokens are ignored, blended into the background, or copied without adapting to the hand pose, the claimed generalization fails.","supporting_citations":[{"cited_title":"Wan: Open and advanced large-scale video gen- erative models, 2025","cited_arxiv_id":null,"evidence_quote":"Supplies the pretrained video diffusion transformer whose attention parameters the method reuses for subsequent-frame generation."},{"cited_title":"Flux.https://github.com/ black-forest-labs/flux, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the pretrained image diffusion transformer used for key frame generation."},{"cited_title":"Re-hold: Video hand object interaction reen- actment via adaptive layout-instructed diffusion model","cited_arxiv_id":null,"evidence_quote":"Serves as the prior layout-instructed HOI reenactment baseline that the method compares against and aims to surpass."},{"cited_title":"Hoi-swap: Swapping objects in videos with hand-object in- teraction awareness.NeurIPS, 2024","cited_arxiv_id":null,"evidence_quote":"Provides the earlier video-level HOI inpainting framework whose object-centric limitations motivate the new unified token process."}],"review_version":1}