{"id":"2b8f5276-811e-4267-8a01-e295852e6d45","arxiv_id":"2607.17097","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"HarmoHOI jointly generates synchronized multi-view hand-object interaction videos and globally aligned 3D point tracks from a single reference image and target camera poses.","lead":"HarmoHOI is a diffusion system that turns one reference image and target camera poses into synchronized multi-view videos of a hand using an object, along with 3D point tracks of the motion. It matters because multi-view-consistent HOI generation with usable 3D motion is a bottleneck for animation production and robot-learning data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Metric-scale 3D alignment depends on an unstated inference-time metric depth source; without it the 'metric-scale' claim reduces to up-to-scale alignment.","rationale":"The reader identified inference-time metric depth as the weakest assumption; I independently agree. The manuscript uses metric scale as a headline claim and in the GloMAD description, but the data representation section assumes metric d_ref and scale s as inputs, while the inference algorithm does not specify where these come from. This is a specification gap rather than an internal contradiction, and it is load-bearing because if the output scale is arbitrary or learned from a small set of objects, the method reduces to up-to-scale globally aligned tracks, weakening the claimed metric-scale property and downstream applicability. The core architecture and ablations are otherwise coherent, and the paper has a held-out split and component-wise ablation support. The concern can be resolved by stating the inference depth source and validating scale recovery, so the conditional verdict remains appropriate; no rejection is warranted. The concrete test above is the minimal experiment that would settle whether the scale is genuinely recovered from the stated inputs or secretly depends on unstated metric depth information.","tokens_in":19120,"tokens_out":6151,"duration_ms":58860,"concrete_test":"On the held-out TACO split, run the full inference twice with the same reference image and camera poses, but replace d_ref by lambda*d_ref for lambda=0.5 and lambda=2.0 (and, in a third run, by an affine-transformed relative depth). Measure the scale of the recovered 3D tracks (e.g., median object extent or median distance of hand keypoints to camera center). If the output scale does not change by lambda (or changes with the affine transform), the observed scale is not determined by the stated inputs; if the output scale is constant, it comes from a learned prior that is untested on unseen object categories. Report also the predicted global scale s-hat against the ground-truth scale s on 100 held-out samples; a median relative error above ~10% would mean the 'metric-scale' claim is not supported by the experiments as described.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of metric-scale, globally aligned 3D point tracks rests on a scale source that is never specified at inference. Sec. 3.3 and Algorithm 1 take as inputs a reference depth d_ref and a depth scale s; the pseudo-video unprojection pipeline (Alg. 1 lines 5-7) requires these to turn normalized color-mapped depth into a metric 3D reconstruction. But the inference procedure in Algorithm 2 lists only an 'averager depth scale s-bar' and a reference image; it never says where a metric d_ref comes from. For a single RGB image and relative camera poses, metric scale is unidentifiable. The two learnable scale tokens (Sec. 3.3; Supp. C) can at most regress a plausible scale from learned object priors, not recover a true meter scale. If the deployed reference depth is an affine/relative estimate (e.g., from Depth Anything 3, used in the stage-1 curriculum), the output's 'metric-scale' property is inherited from an unvalidated external depth estimator. The TACO experiments may accidentally sidestep this because the test split comes from a motion-capture dataset where metric depth/scale is available, but the paper does not state that it is used at inference. Thus the quantitative 3D results do not establish metric-scale alignment for the stated input (reference image + camera poses).","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HarmoHOI, a unified diffusion framework for multi-view hand-object interaction (HOI) synthesis. From a single reference image, multi-view target camera poses, and a text prompt, the method jointly generates synchronized multi-view RGB videos and 3D point tracks claimed to be metric-scale and globally aligned. The architecture couples a Mixture of Multi-view Diffusion Transformer (M2DiT), which co-models RGB videos and motion pseudo-videos derived from point tracks, with a Global Motion Aligning Diffusion (GloMAD) module that refines coarse point tracks. A hybrid-data progressive curriculum uses single-view in-the-wild data, synthetic multi-view data, and lab-captured TACO data. Experiments report improved video quality, motion plausibility, and geometric consistency over several baselines on held-out TACO data, with ablations for each component.","tokens_in":19485,"tokens_out":4072,"duration_ms":41940,"significance":"If the claims hold, HarmoHOI is a substantial contribution: it is the first framework to jointly generate multi-view HOI videos and 3D point tracks in a single diffusion process, reuses video foundation model priors through pseudo-video representation, and explicitly targets metric-scale global alignment. The curriculum strategy is also a plausible way to mitigate the scarcity of paired multi-view video and 3D motion data. However, the central 'metric-scale globally aligned' claim depends on an inference-time depth/scale input that the paper never specifies, and the quantitative evidence is based solely on point estimates without error bars or significance tests. The conceptual framework is promising, but the manuscript currently overstates what is demonstrated.","major_comments":[{"comment":"The metric-scale claim is not supported by the stated inference inputs. Algorithm 1 consumes a reference depth d_ref and depth scale s to build motion pseudo-videos (lines 5-7), and Algorithm 2 lists only an 'averager depth scale s-bar' and the reference image. The task definition in Sec. 3 says input is a single reference image, camera poses, and text. A single RGB image is metrically ambiguous; the learnable scale tokens can at best regress a plausible scale from object priors, not recover true metric scale. The paper must state where d_ref and s come from at inference, and validate metric-scale alignment using that source (e.g., comparing outputs obtained with DA3-estimated depth vs. TACO ground-truth depth). If the method only produces up-to-scale geometry, the 'metric-scale' wording and the matching claims in the abstract and Sec. 4 must be revised.","section":"Sec. 3.3, Algorithm 1, Algorithm 2"},{"comment":"All quantitative comparisons are single point estimates with no error bars, confidence intervals, or significance tests. This is load-bearing for the 'state-of-the-art' claim, especially where margins are small (Table 1: CLIP-V 83.18 vs 83.01; Table 2: RPE 14.8 vs 16.7, non-contact rate 0.12 vs 0.15). Please report standard deviations over multiple evaluation seeds or test splits, state the number of test clips, and provide paired significance tests where appropriate. Table 3 also omits RPE for the row without GloMAD, making that ablation incomplete.","section":"Sec. 4.3, Tables 1-3"},{"comment":"The baseline comparison protocol is underspecified. The text says baselines were fine-tuned on 'our data' but does not state whether the same TACO train/test split, reference images, and camera poses were used, nor the fine-tuning hyperparameters. SV4D 2.0 receives a source video and multi-view reference images, so its input is strictly richer; the paper should explain how this is accounted for. Without this detail, the reported gains over prior methods are hard to interpret.","section":"Sec. 4.3, Baselines"}],"minor_comments":[{"comment":"'averager depth scale s-bar' should be 'average depth scale s-bar'. Also, s-bar is never defined; please define it and explain how it is computed at inference.","section":"Algorithm 2"},{"comment":"Table 2 has the typo 'Plaussibility' for 'Plausibility'. In addition, the column header 'Mat. Pix.' in Table 1 should match the metric name 'Matching Pixels' used in Sec. 4.1.","section":"Sec. 4.1 / Table 2"},{"comment":"Depth Anything 3 is used to produce pseudo-depth labels for HOIGen1M in Stage 1 and is also listed as a 3D motion baseline in Table 2. This creates a possible circularity; the paper should acknowledge this and discuss whether the comparison is affected.","section":"Sec. 4.1 / Table 2"},{"comment":"The name 'Mixture of Multi-view Diffusion Transformer' suggests a mixture-of-experts design, but the described architecture is a dual-branch transformer with inter-view attention. Please clarify the naming to avoid confusion.","section":"Sec. 3.3"},{"comment":"The figure uses abbreviations such as 'Bach Size', 'B', 't', 'V', 'hw', 'd' without a full legend. Please spell out 'Batch Size' and define all tensor-dimension symbols in the caption or text.","section":"Fig. 2"}],"recommendation":"major_revision","confidential_remarks":"The key risk is the unstated inference-time metric depth/scale source. If the authors can demonstrate or clearly specify a metric-depth input (e.g., from a calibrated depth estimator) and validate the metric-scale property on held-out data, the paper could become acceptable. The evaluation also needs error bars and a clearer baseline protocol. The central idea is novel and largely sound; the issues are fixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know before reading. First, this is a technically serious paper: the pseudo-video point-track representation and the closed-loop M2DiT/GloMAD design are genuinely new, and the ablations support each piece. Second, the 'metric-scale globally aligned' claim is under-specified at inference. The paper never states where reference depth d_ref comes from at test time; Algorithm 2 lists only a learned average scale, not a metric depth input. If d_ref is up-to-scale from a monocular estimator, the output is at best up-to-scale aligned with a hallucinated scale — the metric claim collapses. That is the load-bearing soft spot, and it is fixable: either state that a metric depth source is required, or reword the claim.\n\nWhat is actually new: representing 3D point tracks as color-mapped depth pseudo-videos in the video VAE latent space, the dual-branch M2DiT with inter-view geometric attention and bidirectional 2D-3D modulation, and the GloMAD refinement loop feeding globally aligned points back into 2D denoising. These go beyond single-view co-generation (SViMo, VideoJAM) and deliver synchronized multi-view video plus 3D motion from one image. The held-out TACO split and consistent gains over fine-tuned baselines give reasonable evidence the components work; the qualitative results show better cross-view alignment than per-view reconstruction. The three-stage curriculum is a sensible answer to multi-view HOI data scarcity.\n\nSoft spots, in order. The scale-source gap above: specify how d_ref and the scale tokens are obtained at inference, and whether TACO evaluation uses dataset metric depth. No error bars or significance tests anywhere; minor for a system paper, but worth adding for headline numbers. Depth Anything 3 is both stage-1 pseudo-label generator and Table 2 baseline — a mild overlap to acknowledge. The comparison set is fine-tuned on the authors' data, which is right, but it makes the SOTA phrasing conditional on that setup. No code release is mentioned; for a system paper, that slows verification. Citation pattern looks fair; self-citations are same-group prior work, which is fine.\n\nFor whom: geometry-aware video generation, HOI synthesis, multi-view diffusion. It deserves a serious referee. I'd send it to review, with the expectation that the authors address the scale-source question and add variance reporting. If they do, it's a solid addition.","headline":"Genuinely new multi-view HOI video+motion co-generation with a load-bearing scale-source gap at inference.","tokens_in":19952,"tokens_out":5777,"would_cite":true,"duration_ms":50695,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"HarmoHOI is a unified diffusion framework that jointly generates synchronized multi-view hand-object interaction videos and globally aligned metric-scale 3D point tracks from a single reference image and target camera poses.","keywords":["multi-view video generation","hand-object interaction synthesis","3D point tracks","diffusion transformer","geometry-aware generation","pseudo-video representation","global motion alignment","hybrid curriculum learning"],"falsifier":"Take a reference image with known ground-truth metric depth, run the pipeline, and compare the output 3D point tracks against motion-capture ground truth; if the tracks are off by a non-constant scale or more than a small rigid misalignment, the central claim fails. A simpler probe: scale the input depth by 0.5 and check whether the output tracks scale correspondingly—if the model ignores or hallucinates the scale, metric alignment is not actually happening.","tokens_in":19045,"feed_emoji":"🖐️","tokens_out":7029,"duration_ms":58795,"temperature":0.7,"pith_summary":"The paper's central claim is that consistent multi-view video of hand-object interaction is best produced by generating appearance and 3D motion together, not by synchronizing 2D views or reconstructing 3D afterward. To establish this, the authors build HarmoHOI, a diffusion framework that denoises RGB videos and 3D point tracks in one pass, representing tracks as color-coded pseudo-videos so a pretrained video model can process them. A second diffusion module, GloMAD, turns up-to-scale coarse tracks into metric-scale, globally aligned trajectories, and the two modules exchange signals during denoising. A staged curriculum trains first on single-view video, then on synchronized multi-view data, to preserve generic visual priors while injecting geometric consistency. If the central claim holds, animators and embodied agents would get synchronized viewpoints plus usable 3D motion from one image, with geometric consistency built in rather than patched on.","feed_headline":"One image yields synchronized multi-view hand videos and 3D motion","feed_subtitle":"Synchronized viewpoints plus metric-scale 3D tracks come from one reference image, serving animation and embodied AI.","key_machinery":"The load-bearing objects are two. First, the motion pseudo-video: 3D point tracks are depth-normalized, reverse-scaled, color-mapped into RGB-like frames, and VAE-encoded exactly like video frames, so the geometric signal shares a latent space with appearance. Second, the Global Motion Aligning Diffusion (GloMAD): a sparse-convolution point-cloud network with inter-view geometric attention that takes the coarse up-to-scale tracks from M2DiT and outputs globally aligned metric-scale trajectories. The two are wired into a closed loop: during denoising, GloMAD's refined tracks are projected back into pseudo-video and injected into M2DiT's next step, while M2DiT's estimated global scale token ma","core_discovery":"On its own terms, HarmoHOI claims that representing 3D point tracks as 'motion pseudo-videos' erases the domain gap between geometry and the latent space of video foundation models, so the same diffusion transformer can co-model 2D appearance and 3D motion. Co-denoising the two branches lets each constrain the other; inter-view geometric attention enforces epipolar consistency at every denoising step. GloMAD then refines the coarse tracks into metric-scale, globally aligned 3D trajectories, closing a loop by projecting refined tracks back into pseudo-video form to guide the next denoising step. With only a reference image, target cameras, and a text prompt, the framework outputs V synchroniz","pith_inferences":["An extension left implicit: the pseudo-video representation is a general recipe; any dynamic 3D signal (object trajectories, human skeleton motion, scene flow) could be color-mapped and injected into a video DiT the same way, potentially turning other video generators into geometry-aware ones.","The method's practical reach depends on resolving where metric reference depth comes from; a monocular depth network plus a scale calibration step could make the pipeline true RGB-only, a natural next step.","The closed-loop design suggests a broader principle: rather than treating geometry as a conditioning side channel, feed a generator's own geometric output back as input at each denoising step—an idea that could transfer to other multi-view or 4D generation tasks."],"forward_implications":["A user supplies one reference image, target camera poses, and a text prompt; the model returns V synchronized videos and K metric-scale 3D point tracks in one pass.","Multi-view geometric consistency is enforced during generation, not through post-hoc reconstruction or ICP alignment.","Because point tracks are encoded as pseudo-videos, a pretrained video diffusion backbone can be adapted to generate 3D motion without training a separate 3D generator from scratch.","The three-stage curriculum (single-view warm-up, multi-view appearance, then multi-view geometry) allows scarce multi-view HOI data to be supplemented by abundant single-view video.","If the scale assumption is met, the 3D tracks have metric meaning, so they can be used directly for animation, simulation, or embodied-agent training."],"fun_headline_variants":["Single image to multi-view hand videos with 3D tracks","Co-generating 2D hands and 3D motion from one image","One reference image drives synchronized multi-view synthesis","Harmonizing appearance and 3D motion for hand interactions","Multi-view hand videos with metric 3D trajectories from one image"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The method assumes that metric-scale depth of the reference image is available at inference, but the paper does not state how this depth is obtained; if only relative depth is available, the 'metric-scale' and 'globally aligned' parts of the claim lose their meaning.","fun_headline_variants_meta":{"raw":{"variants":["Single image to multi-view hand videos with 3D tracks","Co-generating 2D hands and 3D motion from one image","One reference image drives synchronized multi-view synthesis","Harmonizing appearance and 3D motion for hand interactions","Multi-view hand videos with metric 3D trajectories from one image"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000515,"raw_usage":{"total_tokens":2376,"prompt_tokens":822,"completion_tokens":1554,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":566,"completion_tokens_details":{"reasoning_tokens":1468}},"tokens_in":566,"tokens_out":1554,"duration_ms":9346,"temperature":1.0,"reasoning_tokens":1468,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T19:01:25.923777+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a reference image with known ground-truth metric depth, run the pipeline, and compare the output 3D point tracks against motion-capture ground truth; if the tracks are off by a non-constant scale or more than a small rigid misalignment, the central claim fails. A simpler probe: scale the input depth by 0.5 and check whether the output tracks scale correspondingly—if the model ignores or hallucinates the scale, metric alignment is not actually happening.","supporting_citations":[],"review_version":1}