{"id":"a15bb217-563f-45ac-beda-7bb36e496868","arxiv_id":"2507.01603","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A training-free diffusion-guidance framework that couples scale alignment across windows and geometric multi-view constraints inside the denoising loop yields more scale- and geometry-consistent depth for long videos.","lead":"DepthSync improves long-video depth estimation by synchronizing depth scales across sliding windows and aligning 3D geometry during diffusion denoising. It is a training-free plug-in that makes existing diffusion-based depth estimators more consistent over hundreds of frames.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Geometry guidance's PnP poses are computed from the depth being optimized, creating a circularity that may inflate MFC and pose-evaluation gains; the paper does not test against externally-fixed poses.","rationale":"The paper's central claim—that DepthSync improves depth accuracy and consistency for long videos—is empirically well supported on the four datasets, and the MFC improvement measured with ground-truth poses is a strong check against a fully degenerate self-consistent solution. Nevertheless, the geometry guidance contains a potential circularity: PnP poses are estimated from the same depth maps being optimized, and those poses then define the reprojection and tracking losses (Supp. A.2.1, Eqs. 7-9). If the initial depth has systematic errors, PnP will fold those errors into the pose, and the guidance will reinforce a geometry that is internally consistent but not necessarily truer to the scene. The pose evaluation in Table 2 is also derived from the optimized depth with the same tracking pipeline, so it cannot fully disentangle genuine geometric improvement from self-consistency. The reader's weakest assumption identified the PnP accuracy assumption; I am more specific about the feedback loop and the missing ablation of the pose source. Hyperparameter sensitivity is secondary because the same fixed hyperparameters produce gains across four datasets, which is independent evidence of transfer, though a sensitivity analysis would still be welcome. The proposed concrete test, freezing poses from the baseline depth, directly breaks the loop and would settle whether the geometry guidance's benefit is due to external correspondences or to the self-referential pose estimation. Because this is a testable risk rather than a demonstrated flaw, the conditional acceptance remains appropriate.","tokens_in":17925,"tokens_out":15316,"duration_ms":167576,"concrete_test":"On the ScanNet ablation subset, run geometry guidance with PnP poses computed once from the baseline DepthCrafter depth (frozen during all 80 optimization iterations), instead of re-estimating from the current depth. Compare AbsRel, δ1, and MFC to the reported 'Ours' numbers (main Table 3 and Supp. Table 8). If the improvement over 'Geometry Guidance Only' largely disappears, the self-referential pose loop is responsible for the gains; if it persists, the external 2D tracks (CoTracker3) and normals alone drive the improvement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3 (Eq. 5) and Supp. A.2 define geometry guidance using camera poses estimated by PnP from the current predicted depth and off-the-shelf 2D tracks (Eqs. 7-9). Because PnP solves for a pose that best explains the 2D observations given the predicted 3D structure, any depth errors are absorbed into the pose. The reprojection and tracking losses then pull the depth toward the 3D structure implied by that self-derived pose. This is a feedback loop: the depth is optimized to be consistent with poses that were estimated from itself. The MFC metric uses ground-truth poses (Sec. 4.1), so the reported MFC improvements are evidence against a purely degenerate solution. However, the pose-based evaluation in Table 2 derives poses from the optimized depth using the same tracking correspondences used in guidance, so those numbers are partly self-confirming. The paper never ablates the pose source: it does not compare against guidance with poses frozen from the baseline depth or from an independent RGB-only SLAM. Without such an ablation, it is unclear whether the geometric gains reflect genuine alignment to true scene structure or merely self-consistency of the depth with its own PnP poses. This is load-bearing because the geometry guidance is the component claimed to produce 'geometry-consistent' depth.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents DepthSync, a training-free, inference-time guidance framework that wraps the DepthCrafter diffusion video-depth model. It keeps the sliding-window inference scheme and applies two guidance terms inside the last denoising steps: scale guidance, which least-squares aligns the depth scale of overlapping windows against the previous window, and geometry guidance, which uses off-the-shelf 2D tracks, PnP-derived camera poses, surface normals, and smoothness losses to regularize intra-window depth geometry. Experiments on ScanNet, GMU Kitchen, KITTI, and Bonn are reported for video lengths of 150–590 frames, with AbsRel, delta1, MFC, and derived-pose metrics. An ablation and a post-optimization comparison are included, along with supplementary ablations of loss terms, guidance start step, and inference cost.","tokens_in":18260,"tokens_out":7406,"duration_ms":78781,"significance":"If the reported gains are robust, DepthSync is a useful contribution: it improves cross-window scale consistency and intra-window geometry without retraining, and the scale-guidance mechanism is simple and effective across datasets. The paper includes a thoughtful comparison to post-optimization, and the supplementary ablations cover the main loss terms. The main weaknesses are the circular reliance of geometry guidance on poses estimated from the depth being optimized, the absence of error bars or sensitivity analysis for a method with many hand-set hyperparameters, and the limited-length post-optimization comparison. The MFC metric provides an independent (ground-truth-pose) signal, which strengthens the evidence, but the pose-based evaluation in Table 2 is not fully conclusive for the reasons above.","major_comments":[{"comment":"The geometry guidance uses camera poses estimated by PnP from the current predicted depth (Supp. Eq. 7-9) and then optimizes that same depth with reprojection and tracking losses. This is a feedback loop: the optimization target is defined by the very quantity being optimized, so the losses enforce self-consistency rather than accuracy to true scene geometry. The pose-based evaluation in Table 2 is similarly self-confirming, because it derives poses from the optimized depth using the same off-the-shelf tracks and PnP pipeline. MFC (Sec. 4.1) is an independent check because it uses ground-truth poses, and the MFC gains are encouraging; however, the paper should add an ablation in which the poses are fixed from an external source (e.g., ground-truth ScanNet poses or an RGB-only SLAM estimate) or frozen from the baseline DepthCrafter depth, and show that the geometry guidance still improves accuracy and MFC. Without this, the central claim of 'geometry-consistent depth' is not fully established.","section":"Sec. 3.3 / Supp. A.2"},{"comment":"No error bars or statistical significance tests are reported for any result; every number is a point estimate on a fixed test split. The ablations in Table 3 and Supplement D are run on small subsets (first 35 and first 25 ScanNet test scenes), and the guidance hyperparameters (learning rates 0.01/0.02, 80 iterations per step, scale-guidance learning rates and 1000 iterations, guidance start at the second-to-last step, and loss weights alpha_d=35, alpha_t=2, alpha_n=0.1, alpha_s=1) are hand-fixed with no sensitivity analysis. Since the method relies on iterative optimization that could be sensitive to these choices and to the subset used for tuning, the paper should report variance across videos and a sensitivity study over at least the guidance start step, learning rates, and loss weights. This is necessary to support the claim of consistent, dataset-independent improvements.","section":"Sec. 4.2 / Table 1 / Supp. A.1"},{"comment":"The comparison against post-optimization is performed only at a short video length (150 frames, 2 sliding windows) on a subset of ScanNet. This is precisely the regime where cross-window scale drift is least severe, as the paper itself demonstrates in Table 1 and Fig. 3. To substantiate the claim that injecting guidance into the denoising loop is superior to applying the same constraints post-hoc, the paper should include a post-optimization comparison at longer video lengths (e.g., 270 or 450 frames) and on at least one additional dataset. The current table does not rule out the possibility that post-optimization narrows the gap or even matches the guidance approach as video length grows.","section":"Sec. 4.3 / Table 3"}],"minor_comments":[{"comment":"The reference 'as shown in Fig. Y' is an unresolved placeholder; it should point to a specific figure number (e.g., Fig. 1 or Fig. 5).","section":"Sec. 3.3"},{"comment":"The base learning rates for scale guidance are reported as '6e4' and '1e6'; these values appear to be missing minus signs (likely 6e-4 and 1e-6 or similar). Please correct, since these exact numbers are needed to reproduce the method.","section":"Supp. A.1"},{"comment":"The baseline AbsRel of 0.174 differs from the DepthCrafter ScanNet row in Table 1 (0.141 at length 150) because the ablation uses a different subset; this should be stated in the table caption or in the text.","section":"Table 3"},{"comment":"The citation placeholder '[?]' for 2DGS should be replaced with the proper reference.","section":"Supp. Table 4"},{"comment":"The notation p_j ∼ K T_{i→j} d_i(p_i) K^{-1} p_i is projective and omits the depth division; define the relation precisely to avoid ambiguity.","section":"Supp. Eq. (7)"},{"comment":"'solvePnPRansac inferface' is a typo for 'interface'.","section":"Supp. A.2.1"},{"comment":"The order of the two guidance terms is clarified only in Supp. A.1 (geometry first, then scale); consider stating this order in the main text to avoid confusion.","section":"Sec. 4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is potentially publishable after revision. The main risk is not novelty but the robustness of the evidence. If the authors can provide the proposed ablation with externally fixed poses and report error bars and a sensitivity analysis, the geometry-consistency claim would be substantially stronger. I do not see a need to check for undisclosed related work beyond the references cited."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"DepthSync is a training-free guidance framework that targets the two big failure modes of diffusion-based video depth on long videos: scale drift across sliding windows and within-window geometric inconsistency. The components aren't new — universal guidance, least-squares scale alignment, and PnP-based multi-view losses are established — but the combination and the placement inside the denoising loop is a legitimate new application. The empirical work is solid: consistent gains over DepthCrafter (its base) on four datasets across increasing video lengths, with the largest improvements exactly where the problem is hardest. The ablations show both guidance terms earn their keep, and the comparison against post-optimization makes a fair case that in-loop guidance helps more than the same constraints applied after denoising. Credit where due: this is a clean, reproducible-by-description method and a useful baseline for the long-video depth problem.\n\nThe soft spot is the geometry guidance's self-referential loop. The poses are estimated via PnP from the current predicted depth, then used to pull that same depth toward consistency with those poses. The MFC metric uses ground-truth poses and it improves, so the method is not merely collapsing to a degenerate solution. But the pose evaluation in Table 2 re-derives poses from the optimized depth using the same tracking correspondences used in guidance, so those pose improvements are partly self-confirming. The paper never ablates the pose source — for instance, frozen poses from the baseline depth or an independent RGB-only SLAM — which is the experiment that would separate \"depth is more geometrically accurate\" from \"depth is more self-consistent with its own PnP poses.\" That question is load-bearing for the claim of improved geometry, and it is fixable with one additional experiment.\n\nSmaller issues: no code, no error bars, and hyperparameters are tuned on the first 35 ScanNet scenes used for the ablation. The gains are consistent enough across four datasets that I don't suspect fragility, but reporting variance would help.\n\nOverall: a serious, useful contribution that deserves peer review. I'd ask for the pose-source ablation before trusting the geometry numbers, but the scale-guidance part is already convincing. Send it to review, and push for that one experiment.","headline":"A genuinely useful training-free guidance for long-video depth scale consistency, but the geometry-guidance PnP self-loop needs one more ablation before I'd fully trust the pose gains.","tokens_in":18728,"tokens_out":3345,"would_cite":true,"duration_ms":35143,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DepthSync adds two guidance signals to the denoising loop, cross-window scale synchronization and intra-window geometric alignment, improving depth accuracy and consistency on long videos without retraining.","keywords":["video depth estimation","diffusion guidance","long video consistency","sliding window scale synchronization","geometry alignment","training-free inference","3D reconstruction","multi-frame consistency"],"falsifier":"A concrete test: on a long static video with heavy overlap occlusion or texture-poor regions, degrade the off-the-shelf tracks in half of the windows and run the full pipeline; if the PnP-derived poses are the load-bearing assumption, MFC and AbsRel should worsen relative to the base model because geometry guidance would be optimizing against a wrong target. The paper's own choice to withhold geometry guidance on the dynamic Bonn dataset marks this as the point where the assumption is most likely to break.","tokens_in":17758,"feed_emoji":"🎥","tokens_out":12022,"duration_ms":125390,"temperature":0.7,"pith_summary":"DepthSync aims to show that the two persistent failure modes of diffusion-based video depth estimation on long videos, depth scale drift between overlapping windows and 3D geometric misalignment within a window, can be corrected without retraining by steering the denoising process itself. The method adds two guidance signals to the last steps of denoising: scale guidance, which fits a least-squares scale and shift between overlapping windows and pushes the current window's depth toward the aligned result, and geometry guidance, which derives camera poses from the predicted depth via PnP and optimizes the depth against reprojection, tracking, surface-normal, and smoothness losses. The two terms work together, and on ScanNet, GMU Kitchen, KITTI, and Bonn they consistently improve relative depth error, threshold accuracy, and multi-frame consistency over the base model and other baselines, with the gap widening as videos get longer. This matters because long-video depth consistency is what determines whether predicted depths can be used for 3D reconstruction and spatial reasoning, and the remedy requires no training data or model modification.","feed_headline":"In-loop guidance halts depth drift across long video windows","feed_subtitle":"Training-free guidance syncs depth scale across windows and aligns 3D geometry on four benchmarks.","key_machinery":"The load-bearing machinery has two parts. Scale guidance: at each guided denoising step, the current window's predicted clean latent is decoded to depth, a least-squares scale $s$ and shift $t$ are fit to align the overlapping frames with the stored depth of the previous window, and the aligned depth $d_{\\mathrm{aligned}} = s \\cdot \\hat{d}_{\\mathrm{cur}} + t$ is re-encoded into latent space to serve as a pseudo-label for an MSE loss whose gradients update the noise prediction (forward guidance). Geometry guidance: off-the-shelf 2D tracking produces pixel correspondences, Perspective-n-Point (PnP) recovers camera poses from the current predicted depth, and the decoded depth is optimized by backward guidance against a weighted sum of a depth reprojection loss (warp one depth frame to a neighbor and compare), a tracking loss (3D distance between tracked points projected by the depths), a surface-normal alignment loss, and an edge-aware smoothness loss, with the depth reprojection term weighted highest. The two guidance terms act in the last two of five denoising steps, geometry first, scale second, with hand-set learning rates, iteration counts, and loss weights documented in the supplementary material.","core_discovery":"The central claim is that multi-window depth scale drift in diffusion-based video depth prediction is best corrected by synchronizing scales at every guided denoising step rather than only at window initialization, and that within-window geometry should be enforced by explicit 3D constraints derived from the predicted depths themselves. Concretely, scale guidance decodes the current predicted clean latent to depth, fits a least-squares scale $s$ and shift $t$ over the overlapping frames against the previous window's stored depth, and re-encodes the aligned full-window depth as a pseudo-label for an MSE loss on the latent, so the denoiser is pushed toward the synchronized scale throughout the last steps of sampling. Geometry guidance instead estimates camera poses from off-the-shelf 2D tracks via PnP and optimizes the decoded depth by backward guidance against a weighted sum of depth reprojection, tracking, surface-normal, and edge-aware smoothness losses. The paper reports that the two terms together improve AbsRel, $\\delta_1$, and multi-frame consistency (MFC) over all baselines on ScanNet, GMU Kitchen, KITTI, and Bonn; that each term used alone beats post-hoc optimization with the same losses; and that on the highly dynamic Bonn dataset, where only scale guidance is applied, the roughly 16% $\\delta_1$ gain over DepthCrafter at 590 frames comes from cross-window scale synchronization alone.","pith_inferences":["Editorial extension: because scale guidance only needs overlapping windows and a latent depth representation, it should transfer to any diffusion-based video depth model that encodes depth into latents; a testable prediction is that attaching it to a different base model reproduces similar AbsRel and MFC gains.","The paper withholds geometry guidance on the highly dynamic Bonn split; a natural follow-up is motion-masked geometry guidance that excludes tracked points on moving objects from the PnP fit, with the prediction that dynamic-scene MFC improves as well.","All guidance hyperparameters are fixed across datasets in the supplementary material, so the reported margins may be a floor; grid-searching the loss weights per dataset would likely widen the gap over DepthCrafter and would also reveal which of the four geometry losses carries the improvement.","The case study showing that per-window alignment largely restores accuracy implies that cheap post-hoc stitching of window scales is a strong baseline; the paper's comparison shows in-loop guidance beats that baseline, so the remaining gap quantifies the value of coupling the regularization to the diffusion trajectory."],"forward_implications":["Under the paper's global-alignment evaluation protocol, DepthSync's guided depths stay accurate at video lengths where the base models degrade, so long videos can be processed window-by-window without losing a common scale.","Poses derived from the guided depths by PnP outperform dedicated pose-estimation baselines (RelPose++, PoseDiffusion, RayDiffusion) on ScanNet under ATE and RPE metrics, so the same depth output directly supports monocular odometry.","Applying the regularization inside the denoising loop beats applying the identical losses after inference, indicating that the diffusion prior actively steers the geometric optimization rather than merely providing a starting point.","A lightweight variant that keeps scale guidance and drops geometry guidance runs in about 2% of the full method's time (18.1 s versus 961 s per 90 frames) while still improving over the base model, making cross-window scale synchronization the cheap and effective fix for long-video drift."],"supporting_citations":[{"why":"Provides the universal guidance formalism, including forward and backward guidance, that DepthSync adapts for its two guidance mechanisms.","marker":"[1]"},{"why":"Supplies the image-to-video diffusion prior that the depth estimation models targeted by DepthSync are built on.","marker":"[4]"},{"why":"Provides the PnP-RANSAC solver used to derive camera poses from predicted depth and 2D tracks in the geometry guidance.","marker":"[13]"},{"why":"Is the source of the edge-aware smoothness loss used as a local constraint in the geometry guidance.","marker":"[19]"},{"why":"Is the base diffusion video depth model that DepthSync guides, and the primary baseline for all comparisons.","marker":"[22]"},{"why":"Supplies the off-the-shelf tracking network whose pixel correspondences feed PnP pose estimation and the tracking loss.","marker":"[25]"},{"why":"Is the alternative window-initialization baseline (frame-wise noise levels) that DepthSync is compared against.","marker":"[42]"},{"why":"Provides the DDIM clean-sample formula used to decode latents into depth for both guidance terms.","marker":"[43]"},{"why":"Provides the surface-normal predictions that define the normal alignment loss in the geometry guidance.","marker":"[55]"}],"fun_headline_variants":["Training-free guidance syncs depth scale and geometry across video windows","In-loop scale and geometry guidance halts depth drift in long videos","DepthSync: scale and geometry consistency for long video depth, no training","Pushing denoising with scale and 3D geometry beats depth drift"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The geometry guidance presumes that camera poses recovered by PnP from the currently predicted depth and the off-the-shelf 2D tracks are accurate enough to define a useful optimization target, and that the hand-fixed guidance weights, learning rates, and iteration counts carry over to any new dataset without per-dataset tuning.","fun_headline_variants_meta":{"raw":{"variants":["Training-free guidance syncs depth scale and geometry across video windows","In-loop scale and geometry guidance halts depth drift in long videos","DepthSync: scale and geometry consistency for long video depth, no training","Pushing denoising with scale and 3D geometry beats depth drift"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00073,"raw_usage":{"total_tokens":3299,"prompt_tokens":1006,"completion_tokens":2293,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":622,"completion_tokens_details":{"reasoning_tokens":2216}},"tokens_in":622,"tokens_out":2293,"duration_ms":19093,"temperature":1.0,"reasoning_tokens":2216,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:46:59.244550+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete test: on a long static video with heavy overlap occlusion or texture-poor regions, degrade the off-the-shelf tracks in half of the windows and run the full pipeline; if the PnP-derived poses are the load-bearing assumption, MFC and AbsRel should worsen relative to the base model because geometry guidance would be optimizing against a wrong target. The paper's own choice to withhold geometry guidance on the dynamic Bonn dataset marks this as the point where the assumption is most likely to break.","supporting_citations":[{"cited_title":"Universal guidance for diffusion models","cited_arxiv_id":null,"evidence_quote":"Provides the universal guidance formalism, including forward and backward guidance, that DepthSync adapts for its two guidance mechanisms."},{"cited_title":"Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography.Communications of the ACM, 24(6):381–395, 1981","cited_arxiv_id":null,"evidence_quote":"Provides the PnP-RANSAC solver used to derive camera poses from predicted depth and 2D tracks in the geometry guidance."},{"cited_title":"Pm-huber: Patchmatch with huber regularization for stereo matching","cited_arxiv_id":null,"evidence_quote":"Is the source of the edge-aware smoothness loss used as a local constraint in the geometry guidance."},{"cited_title":"Depthcrafter: Generating consistent long depth sequences for open-world videos","cited_arxiv_id":null,"evidence_quote":"Is the base diffusion video depth model that DepthSync guides, and the primary baseline for all comparisons."},{"cited_title":"Learning temporally consistent video depth from video diffusion priors","cited_arxiv_id":null,"evidence_quote":"Is the alternative window-initialization baseline (frame-wise noise levels) that DepthSync is compared against."},{"cited_title":"Denois- ing diffusion implicit models","cited_arxiv_id":null,"evidence_quote":"Provides the DDIM clean-sample formula used to decode latents into depth for both guidance terms."},{"cited_title":"Stablenormal: Reducing diffusion variance for stable and sharp normal.ACM Transactions on Graphics (TOG), 43(6):1–18, 2024","cited_arxiv_id":null,"evidence_quote":"Provides the surface-normal predictions that define the normal alignment loss in the geometry guidance."}],"review_version":1}