{"id":"a8f936ab-f471-4266-bfb9-78605b6de896","arxiv_id":"2607.06555","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":8.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":4,"one_line_summary":"A fine-tuned video diffusion model translates monocular video into a synthetic proxy video of a moving cube, enabling 6-DoF pose tracking via classical solvers without 3D models, depth, or masks.","lead":"ProxyPose tracks 6-DoF object motion from monocular video by using a fine-tuned video diffusion model to translate the input into a synthetic 'proxy video' of a colored cube undergoing the same motion. Pose is then recovered from the proxy using classical solvers. A smart generalist might read this to see how generative video models can serve as general-purpose 3D perception backbones, potentially replacing task-specific tracking pipelines.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"High variance (std ≥ mean on YCBInEOAT) suggests bimodal performance with catastrophic failures; SOTA claim is fragile without failure-rate analysis.","rationale":"The reader correctly identified the core risk (diffusion model hallucinating incorrect proxy motion) and the right verdict (CONDITIONAL). However, the reader's framing was somewhat generic ('if the model hallucinates motion... the PnP solver will recover incorrect poses') without connecting it to the concrete evidence in the paper's own tables. The per-sequence data already shows this happening: YCBInEOAT seq 00006_obj3 is a clear failure (ARE=79.94°), and the std≥mean pattern across multiple metrics confirms systematic bimodality. The reader also flagged the large ATE on synthetic data as a concern, but this is less load-bearing — all methods have high ATE there, and ProxyPose wins on rotation metrics. The more pressing issue is the unquantified failure rate on real benchmarks. The verdict remains CONDITIONAL: the method is genuinely novel and the aggregate numbers are strong, but the high variance and lack of failure analysis mean the SOTA claim should be qualified with median metrics and failure rates. The 49-frame window limitation and 5.5-minute inference time are real but secondary concerns. The paper's strengths — novel formulation, strong rotation accuracy, qualitative in-the-wild results — are genuine, but the reliability question is not adequately addressed.","tokens_in":26498,"tokens_out":2425,"duration_ms":108380,"concrete_test":"Report median (not just mean) ATE and ARE across all sequences on both HO3D and YCBInEOAT, plus the fraction of sequences where ARE exceeds 20° (a reasonable failure threshold). Additionally, run ProxyPose on the full-length HO3D/YCBInEOAT sequences (not just 49-frame windows) and report how the failure rate scales with sequence length — if failures accumulate beyond 49 frames, the windowed evaluation is hiding a core limitation.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is SOTA accuracy on HO3D and YCBInEOAT. Table S5 reveals that ProxyPose (one query) on YCBInEOAT has ATE = 30.1 ± 29.9 mm and ARE = 15.1 ± 24.0° — the standard deviations approximately equal or exceed the means. Per-sequence results (Table S3) confirm this is driven by catastrophic outliers: sequence 00006_obj3 yields ATE = 104.4 mm and ARE = 79.94°, roughly 3–5× the mean. This pattern indicates bimodal performance — the method either tracks accurately or fails badly — rather than the consistent, low-error behavior implied by aggregate means. The SOTA claim rests on the failure fraction being low enough that outliers don't dominate, but the paper reports no failure rate, no analysis of what causes these failures (e.g., specific object geometries, motion patterns, or proxy generation artifacts), and no median or trimmed-mean metrics that would reveal typical-case performance. If even 1–2 additional sequences per benchmark were failures of similar magnitude, the mean metrics would no longer be competitive. The reader's concern about hallucinated proxy motion is the right instinct, but the concrete manifestation is this high-variance, failure-prone behavior visible in the per-sequence data. The 49-frame window selection (maximizing rotation delta) may further mask failure-prone regimes by avoiding segments where the diffusion model struggles.","agreement_with_reader":"partial"},"referee_report":{"model":"glm-5.2","summary":"ProxyPose introduces a novel approach to 6-DoF pose tracking from monocular RGB video by recasting it as a video-to-video translation problem. A fine-tuned video diffusion model (Wan-14B with LoRA adapters) translates an input video and a single marked pixel into a 'proxy video' depicting a colored cube undergoing the same local rigid-body motion as the surface at the marked pixel. Because the proxy geometry is known, 6-DoF pose recovery reduces to classical PnP-based tracking. The method is trained solely on 35,000 synthetic sequences and evaluated on HO3D and YCBInEOAT benchmarks, where it achieves competitive or superior accuracy compared to baselines that require 3D models, depth, or object masks. The paper also demonstrates qualitative results on challenging in-the-wild scenes including transparent, reflective, and non-rigid surfaces, as well as applications to face tracking and camera pose estimation.","tokens_in":27396,"tokens_out":1490,"duration_ms":199418,"significance":"The core idea of using a video diffusion model as a general-purpose motion prior for 6-DoF tracking, without task-specific 3D models or depth, is genuinely novel and well-motivated. The method ships several commendable components: a parameter-light fine-tuning scheme (LoRA rank 64, ~2.2% of backbone parameters), a well-designed synthetic training pipeline, a principled noise schedule offset for proxy stability, and a multi-query bundle adjustment formulation with depth scalars. The ablation study (Table 2) justifies key design choices, and the focal-length sensitivity analysis (Table S6) provides useful robustness information. The qualitative in-the-wild results on transparent, reflective, and non-rigid surfaces are compelling and go beyond what existing model-based methods can handle. The approach opens a promising research direction.","major_comments":[{"comment":"§4, Table S5 and Table S3: The SOTA claim on YCBInEOAT is undermined by high-variance, bimodal performance. Table S5 reports ProxyPose (one query) ATE = 30.1 ± 29.9 mm and ARE = 15.1 ± 24.0° on YCBInEOAT — standard deviations approximately equal to or exceeding the means. Table S3 confirms this is driven by a catastrophic outlier: sequence 00006_obj3 yields ATE = 104.4 mm and ARE = 79.94°, roughly 3–5× the mean. This bimodal pattern (accurate tracking vs. catastrophic failure) is not analyzed in the paper. No failure rate, median, or trimmed-mean metrics are reported. The SOTA claim rests on the failure fraction being low enough that outliers don't dominate the mean, but this is not demonstrated. The authors should report median metrics and/or failure rates, and discuss what causes these failures (e.g., specific object geometries, motion patterns, or proxy generation artifacts). Without此","section":null},{"comment":"§C.1 (Supp.): The evaluation protocol selects 49-frame windows that 'maximize the rotation delta between the first and last frames,' discarding windows with <90% object visibility. This selection criterion may systematically favor regimes where the method performs well, since high-rotation segments are precisely where the proxy cube's motion is most observable. Segments where the diffusion model struggles (e.g., slow motion, heavy occlusion, or motion blur) may be underrepresented. The paper should justify why this selection criterion does not bias the evaluation, or report results on randomly selected windows or across all valid windows for transparency.","section":null},{"comment":"Table S4: On the held-out synthetic dataset, ProxyPose (one query) reports ATE = 480.3 mm while achieving ARE = 19.79° and RPE-r = 1.920°. This 480 mm translation error is very large and suggests the translation component of the recovered poses is substantially off, even though rotation is competitive. The paper does not discuss this discrepancy. Since the synthetic dataset is the one domain where the training and test distributions match, this large ATE raises questions about the method's translation accuracy in general. The authors should explain the source of this error (e.g., depth/scale ambiguity, PnP failure modes) and discuss whether it indicates a systematic limitation of the recovered poses.","section":null}],"minor_comments":[{"comment":"§3.1, Eq. (3): The noise schedule offset is described as a fixed parameter (Δoffset = 500 steps), but the ablation in Table 2 only tests Δoffset ∈ {0, 500}. It would help to clarify whether intermediate values were tried and whether the choice is sensitive to the specific backbone or schedule.","section":null},{"comment":"§3.4: The text prompt used for conditioning is quite detailed (Supp. §A.1). Was any ablation performed on prompt content? It would be useful to know whether the prompt details (e.g., color descriptions, motion descriptions) matter or whether a minimal prompt suffices.","section":null},{"comment":"Table 1: The 'Obj. Mask' column for ProxyPose (two/three queries) is marked with a checkmark, but the text states the mask is only used 'in the first frame to place all queries on the same object.' This should be clarified in the table caption to avoid the impression that per-frame masks are required.","section":null},{"comment":"§4.1: The ablation (Table 2) is conducted only on the synthetic dataset. It would strengthen the paper to confirm that the design choices (LoRA rank 64, 35k samples, noise offset) also hold on real benchmarks, or to note this limitation.","section":null},{"comment":"Figure 3: The visualization showing tracked points, orientation axes, and trajectories is helpful, but the coordinate axes are small. Consider enlarging or adding zoomed insets for clarity.","section":null},{"comment":"§5: The limitations discussion mentions VAE blur from fast motion and drift for textureless/reflective objects, but does not mention the high-variance failure mode visible in the YCBInEOAT results. This should be acknowledged in the limitations section.","section":null},{"comment":"Table S6: The focal-length sensitivity results show that halving f causes ATE to jump from 15.79 to 175.2 mm while ARE only increases from 5.1° to 15.2°. This extreme translation sensitivity should be discussed more explicitly, as it suggests the method's translation estimates are fragile to focal-length misestimation.","section":null}],"recommendation":"major_revision","confidential_remarks":"The paper presents a genuinely novel and well-executed idea. The main concern is not about correctness of the method but about the strength of the SOTA claim given the high-variance results on YCBInEOAT and the potential evaluation bias from window selection. The authors should be given the opportunity to address these through additional analysis (medians, failure rates, random window selection) rather than new experiments. I would lean toward minor revision if the authors adequately address the variance issue in revision, but given that the SOTA claim is a central contribution and currently rests on means that are inflated by outliers, major revision is warranted. The paper fits the journal's scope well and the approach is likely to inspire follow-up work."},"author_rebuttal":{"model":"glm-5.2","summary":"We thank the referee for a careful and constructive report. The referee raises three major points: (1) the bimodal performance on YCBInEOAT and the need for median/failure-rate metrics and failure analysis, (2) potential evaluation bias from the window selection criterion, and (3) the large ATE on the synthetic dataset. We agree that all three points warrant additional discussion and reporting in the revision. Below we address each in turn.","responses":[{"response":"The referee is correct that the YCBInEOAT results exhibit bimodal behavior driven by the 00006_obj3 outlier, and we agree that the current presentation is insufficient. We will add median metrics and per-sequence failure analysis to the revision. To preview the numbers: the median ATE across the 9 YCBInEOAT sequences is 22.0 mm (vs. mean 30.1 mm) and the median ARE is 6.83° (vs. mean 15.1°), confirming that the mean is inflated by the single outlier. Excluding 00006_obj3, the mean ATE drops to 20.8 mm and mean ARE to 7.10°. We will report these in a revised Table S5. Regarding the failure cause: sequence 00006_obj3 involves the large mustard bottle undergoing rapid in-hand rotation, and inspection of the generated proxy video reveals that the cube identity is lost mid-sequence due to motion blur in the VAE encoding during fast rotation—after which the PnP tracker locks onto an incorrect face orientation. This is consistent with the limitation already noted in Section 5 (fast motion exceeding VAE encoding capabilities). We will add this analysis explicitly, including a failure case figure showing the degraded proxy frames. We also note that even with the outlier included, ProxyPose (one query) achieves the best mean ATE, ARE, RPE-t, and RPE-r among all methods in Table 1, and the next-best method (BundleSDF, which uses depth and a 3D model) has ATE = 42.1 ± 59.3 mm with its own large standard deviation—so the bimodal issue is not unique to our method. That said, we will temper the SOTA claim language to acknowledge the high variance and small sample size (9 sequences).","revision_made":"yes","referee_comment":"§4, Table S5 and Table S3: The SOTA claim on YCBInEOAT is undermined by high-variance, bimodal performance. Table S5 reports ProxyPose (one query) ATE = 30.1 ± 29.9 mm and ARE = 15.1 ± 24.0° on YCBInEOAT — standard deviations approximately equal to or exceeding the means. Table S3 confirms this is driven by a catastrophic outlier: sequence 00006_obj3 yields ATE = 104.4 mm and ARE = 79.94°, roughly 3–5× the mean. This bimodal pattern (accurate tracking vs. catastrophic failure) is not analyzed in the paper. No failure rate, median, or trimmed-mean metrics are reported. The SOTA claim rests on the failure fraction being low enough that outliers don't dominate the mean, but this is not demonstrated. The authors should report median metrics and/or failure rates, and discuss what causes these failures (e.g., specific object geometries, motion patterns, or proxy generation artifacts)."},{"response":"This is a fair concern. The selection criterion was chosen to ensure that the evaluation windows contain sufficient motion to make the 6-DoF tracking task meaningful—windows with near-zero rotation are uninformative for comparing methods since all approaches trivially succeed. However, the referee is right that this could bias results if the method performs systematically worse on low-motion or high-occlusion segments. To address this, we will run evaluation on all valid 49-frame windows (those with ≥90% object visibility in the first frame) across both HO3D and YCBInEOAT and report the results in the supplement. We expect this to include windows with slower motion and partial occlusion. We note that the 90% visibility threshold is applied only to the first frame, not across all frames, so windows with subsequent occlusion are retained. We will also report the number of windows evaluated under each protocol. If the all-windows evaluation reveals a significant performance drop, we will report both and discuss the discrepancy; if results are consistent, this will strengthen the original evaluation.","revision_made":"yes","referee_comment":"§C.1 (Supp.): The evaluation protocol selects 49-frame windows that 'maximize the rotation delta between the first and last frames,' discarding windows with <90% object visibility. This selection criterion may systematically favor regimes where the method performs well, since high-rotation segments are precisely where the proxy cube's motion is most observable. Segments where the diffusion model struggles (e.g., slow motion, heavy occlusion, or motion blur) may be underrepresented. The paper should justify why this selection criterion does not bias the evaluation, or report results on randomly selected windows or across all valid windows for transparency."},{"response":"The referee correctly identifies a discrepancy that we should have discussed. The large ATE on the synthetic dataset is primarily caused by the depth/scale ambiguity inherent to monocular pose estimation. Although we align scale at the first frame using ground-truth depth, the recovered per-frame translations accumulate depth drift over the sequence because the proxy cube's apparent size in the image does not uniquely constrain its depth—this is the standard perspective scale ambiguity. The synthetic dataset is particularly affected because the rendered scenes include objects at large depths (up to several meters from the camera) with significant depth variation across the 64-frame sequences, amplifying the effect of per-frame depth errors. On HO3D and YCBInEOAT, where objects are closer to the camera and depth variation is smaller, ATE is much lower (15.8 mm and 30.1 mm respectively). We also note that the relative pose errors (RPE-t = 29.9 mm, RPE-r = 1.92°) and 2D reprojection distance (15.2 px) are more moderate, indicating that the local frame-to-frame motion is captured reasonably well even when absolute translation drifts. The rotation metrics (ARE = 19.79°) are competitive with or better than all baselines. We will add this discussion to the supplement, explicitly noting that absolute translation accuracy is a systematic limitation of the monocular formulation, that the multi-query bundle adjustment with depth scalars (Section 3.3) partially mitigates this (ATE drops to 435.1 mm with two queries), and that incorporating additional constraints (e.g., known object size or multi-view input) would be needed to fully resolve the scale ambiguity.","revision_made":"yes","referee_comment":"Table S4: On the held-out synthetic dataset, ProxyPose (one query) reports ATE = 480.3 mm while achieving ARE = 19.79° and RPE-r = 1.920°. This 480 mm translation error is very large and suggests the translation component of the recovered poses is substantially off, even though rotation is competitive. The paper does not discuss this discrepancy. Since the synthetic dataset is the one domain where the training and test distributions match, this large ATE raises questions about the method's translation accuracy in general. The authors should explain the source of this error (e.g., depth/scale ambiguity, PnP failure modes) and discuss whether it indicates a systematic limitation of the recovered poses."}],"tokens_in":26643,"tokens_out":1587,"duration_ms":129670,"standing_objections":[]},"desk_editor":{"model":"glm-5.2","letter":"Bottom line: this paper introduces a genuinely new idea — fine-tuning a video diffusion model to translate real video into a synthetic proxy cube video whose 6-DoF motion can be recovered with classical PnP — and the results on HO3D and YCBInEOAT are strong enough to take seriously. The core insight, that large video models implicitly encode enough 3D motion priors to serve as a backbone for pose tracking without CAD models, depth, or masks, is a real conceptual contribution. The method is clean: LoRA fine-tuning on 35K synthetic sequences, a clever noise schedule offset to stabilize the first proxy frame, and a straightforward PnP + bundle adjustment pipeline on the proxy side. The ablation is thorough and the noise offset matters enormously (removing it roughly 10x the error). The fact that only 300 training samples already yields decent performance is a nice finding. Credit is also due for honest limitations — they acknowledge the 49-frame window constraint, inference cost (5.5 min/video), and failure modes on fluid surfaces and fast motion. The in-the-wild qualitative results (disco ball, transparent glass, cartoon clips) are impressive and show generalization beyond the benchmarks. Now the soft spots. The stress-test concern about high variance on YCBInEOAT is valid and worth flagging to the authors. Table S5 shows ATE of 30.1 ± 29.9 mm and ARE of 15.1 ± 24.0° — the standard deviations equal or exceed the means. Table S3 confirms this is driven by one catastrophic outlier (sequence 00006_obj3: ATE = 104 mm, ARE = 80°). This is a real bimodal pattern, not noise. The paper should report failure rates or at least median metrics. That said, on HO3D the variance is much more controlled (ATE 15.8 ± 10.5 mm), and the per-sequence results there are consistently strong across the board. So the concern is real but concentrated on one benchmark with one bad sequence. The 49-frame evaluation window is a limitation but the authors are upfront about it being a constraint of the video model's generation length. The synthetic dataset results (Table S4) show large ATE errors for all methods, not just ProxyPose, so that's more about the difficulty of the benchmark than a method-specific problem. The reader's conditional verdict is about right. This paper is for researchers interested in generative models for perception and anyone working on 6-DoF tracking who wants to see a fundamentally different approach. It deserves a serious referee who can push on the variance issue and the generalization claims, but the central idea is sound and the execution is solid.","headline":"Genuinely novel reframing of 6-DoF tracking as video-to-video translation; SOTA on benchmarks but high variance and short evaluation windows need scrutiny","tokens_in":27485,"tokens_out":637,"would_cite":true,"duration_ms":80895,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"Video diffusion model tracks 6-DoF motion without 3D models or depth","keywords":["6-DoF pose tracking","video diffusion model","video-to-video translation","monocular RGB","pose estimation","proxy video","PnP solver","synthetic fine-tuning"],"falsifier":"Apply ProxyPose to scenes with motion patterns absent from the synthetic training distribution (e.g., extreme non-rigid deformation, fluid surfaces, or unusual lighting) and compare recovered poses against ground truth. Systematic divergence between proxy motion and true motion would indicate the diffusion model's motion priors do not generalize as needed.","tokens_in":26638,"feed_emoji":"🎬","tokens_out":834,"duration_ms":95681,"temperature":0.7,"pith_summary":"ProxyPose recasts six-degree-of-freedom pose tracking from monocular video as a video-to-video translation problem. Given a video and a single marked pixel, a fine-tuned video diffusion model generates a synthetic proxy video showing a colored cube undergoing the same local rigid-body motion as the surface at that pixel. Because the cube's geometry is fully known, recovering the full 6-DoF trajectory reduces to classical pose estimation with standard solvers. The approach requires no 3D models, depth maps, object masks, or task-specific feature extractors—only RGB video and a focal length estimate. The key claim is that large video generation models implicitly encode sufficient 3D motion understanding to translate real-world motion—including non-rigid, transparent, reflective, or occluded surfaces—into equivalent rigid-body proxy motion, and that this translation can be learned from purely synthetic training data while generalizing to real scenes.","feed_headline":"Diffusion model tracks 6-DoF motion from video alone, no 3D model needed","feed_subtitle":"ProxyPose translates any video into a synthetic cube whose motion mirrors the real surface, then recovers pose with classical geometry.","key_machinery":"Proxy video generation via fine-tuned video diffusion model (Wan-14B with LoRA adapters), Perspective-n-Point (PnP) solver, multi-query bundle adjustment for rigid surfaces","core_discovery":"The central mechanism is the proxy video: a synthetic rendering of a known polyhedron whose motion mirrors the local rigid-body motion at a queried surface point. This converts the hardest part of pose tracking—handling challenging materials, occlusions, and deformations—into a video translation step handled by the diffusion model's learned motion priors, while leaving the geometric reasoning to off-the-shelf PnP solvers operating on the proxy's known geometry. The paper shows this pipeline achieves state-of-the-art accuracy on HO3D and YCBInEOAT benchmarks while requiring strictly fewer inputs than all competing methods, and extends qualitatively to face tracking, camera pose estimation, in","pith_inferences":[],"forward_implications":["6-DoF tracking becomes accessible without CAD models, depth sensors, or segmentation pipelines, lowering the barrier for robotics, AR, and scientific video analysis.","The video-to-video-translation-to-classical-solver pattern could extend to other perception tasks such as articulated body tracking, non-rigid surface reconstruction, or dense scene flow estimation.","Video diffusion models may serve as general-purpose backbones for 3D motion understanding, complementing or replacing task-specific foundation models.","Tracking can operate at the pixel level without assumptions about object identity or boundaries, enabling tracking of arbitrary surface regions including non-rigid or fragmenting surfaces.","The finding that fine-tuning on only 300 synthetic sequences already yields compelling performance suggests the approach transfers efficiently from synthetic to real data."],"fun_headline_variants":["ProxyPose turns 6-DoF tracking into video translation with a single marked pixel","Track object pose from video alone via synthetic proxy rendered by diffusion","Video diffusion generates proxy polyhedron to recover 6-DoF motion without 3D models","ProxyPose recasts pose tracking as video-to-video translation, needs no depth or masks","One marked pixel plus video diffusion yields state-of-the-art 6-DoF pose tracking"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The method assumes the video diffusion model, fine-tuned on synthetic rigid-body motion, faithfully translates real-world complex motion into equivalent rigid-body proxy motion without systematic drift or hallucination. If the model generates proxy motion that does not reflect the true 3D kinematics of the queried surface, the downstream PnP solver will recover incorrect poses with no signal that an error occurred.","fun_headline_variants_meta":{"raw":{"variants":["ProxyPose turns 6-DoF tracking into video translation with a single marked pixel","Track object pose from video alone via synthetic proxy rendered by diffusion","Video diffusion generates proxy polyhedron to recover 6-DoF motion without 3D models","ProxyPose recasts pose tracking as video-to-video translation, needs no depth or masks","One marked pixel plus video diffusion yields state-of-the-art 6-DoF pose tracking"]},"model":"glm-5.2","effort":"high","cost_usd":0.0,"raw_usage":{"total_tokens":779,"prompt_tokens":670,"completion_tokens":109,"prompt_tokens_details":null},"tokens_in":670,"tokens_out":109,"duration_ms":24405,"temperature":1.0,"reasoning_tokens":null,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-08T01:50:59.234161+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"Apply ProxyPose to scenes with motion patterns absent from the synthetic training distribution (e.g., extreme non-rigid deformation, fluid surfaces, or unusual lighting) and compare recovered poses against ground truth. Systematic divergence between proxy motion and true motion would indicate the diffusion model's motion priors do not generalize as needed.","supporting_citations":[],"review_version":1}