{"id":"58ad85aa-4f47-4a40-b499-23e5c11e7a15","arxiv_id":"2504.14687","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Seurat predicts per-point depth changes over time in monocular video from 2D point trajectories alone, trained on synthetic data and evaluated zero-shot on real benchmarks.","lead":"Seurat estimates how objects move closer or farther in a monocular video by analyzing the 2D motion of tracked points, without using stereo cameras. Trained only on synthetic videos, it transfers to real footage and produces smoother depth over time than per-frame depth estimators.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Under camera zoom, 2D trajectories are identical to true depth motion, so Seurat's mapping cannot distinguish the two; the paper's general claim requires a constant-focal-length assumption that is never stated or tested.","rationale":"The reader's weakest assumption is the train/inference mismatch between ground-truth trajectories in training and off-the-shelf tracker outputs at inference. That is a real engineering gap, and Sec. 8 of the supplementary material admits it explicitly. However, a more fundamental concern is that the mapping from 2D trajectories to depth ratios is not identifiable under changing camera intrinsics. Camera zoom and object/camera translation along the optical axis produce the same 2D trajectory scaling but opposite ground-truth depth interpretations. Since Seurat has no focal-length input and the paper never tests changing intrinsics, the general claim that relative depth can be inferred from 2D trajectories alone is overbroad. This is not a disagreement with the empirical results: on TAPVid-3D, where cameras have fixed intrinsics, the method performs well, and the reported gains are credible. The concern is about the scope of the central claim and the unstated assumption that makes it valid. The proposed synthetic zoom experiment would settle whether the model has actually learned to separate zoom from depth change or has simply memorized a fixed-intrinsics prior. If the test confirms the confound, the paper should be revised to state the constant-intrinsics assumption and to discuss zoom as a failure mode. Since the reader's verdict is already CONDITIONAL and this concern does not overturn the benchmark results, the verdict should remain unchanged.","tokens_in":17412,"tokens_out":14453,"duration_ms":150146,"concrete_test":"Render a static Kubric scene with focal length ramping from f to 2f while all 3D points remain stationary; track a dense grid with CoTracker, run Seurat's depth-ratio head, and check whether predicted log depth ratios deviate from 0 by more than a small threshold, with a dolly-motion control.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that relative depth changes can be inferred from 2D point trajectories alone. This requires that image-plane expansion be uniquely attributable to depth change. In a pinhole camera, a point's pixel coordinate is u = f X/Z, so a 2D trajectory (u_t, v_t) is compatible with both (i) changing depth Z_t and (ii) changing focal length f_t with a static scene. The same 2D tracks can therefore correspond to very different depth histories. Seurat's input consists only of trajectories and visibility masks; there is no camera-intrinsics or zoom input. In the theoretical derivation of Sec. 3.2, the focal length f cancels in the density ratio only because it is assumed constant over time, but this assumption is not stated, and no experiment with changing intrinsics is reported. Training on Kubric with fixed intrinsic parameters and evaluating on TAPVid-3D videos with fixed cameras means the confound is never exercised: the model can learn a prior that all scale changes in the 2D trajectories are caused by depth changes. Under optical zoom or digital crop, the predicted depth ratios would therefore be systematically wrong while the same trajectories would be correctly interpreted in a non-zoom setting. This gap directly threatens the generalization claim in the abstract, not just the numerical results on the benchmark.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Seurat, a method that predicts per-point depth changes over time from 2D point trajectories and visibility masks. The input trajectories come from off-the-shelf trackers such as CoTracker or LocoTrack. A two-branch transformer architecture processes a dense supporting grid of trajectories and the query trajectories separately, with cross-attention from the supporting branch into the query branch, and predicts log depth ratios within sliding windows. The window-wise predictions are accumulated and then rescaled with a monocular depth estimator such as ZoeDepth or DepthPro to produce metric depth estimates. The method is trained on synthetic Kubric MOVi-F data with ground-truth trajectories and is evaluated zero-shot on the TAPVid-3D benchmark across Aria, DriveTrack, and PStudio, reporting 3D-AJ, APD, and temporal coherence. The paper reports consistent gains over baselines that combine point trackers with per-frame or video depth estimators, plus ablations supporting the two-branch design, sliding windows, and window-wise loss.","tokens_in":17600,"tokens_out":7570,"duration_ms":68942,"significance":"If the claims hold, the paper makes a useful contribution: it shows that temporal depth changes can be extracted from 2D trajectory patterns alone, without stereo, multi-view, or strong depth backbones at the ratio-prediction stage, and it demonstrates zero-shot transfer from a single synthetic dataset to real-world benchmarks. The evaluation is not circular: the ratio model is trained on synthetic ground-truth depth and tested on external real-world data, and the method works with two different trackers, which mitigates the concern that results depend on a particular tracker from the same group. The ablations are informative, including the handcrafted density baseline, the texture-input ablation, and the Gaussian-smoothing control. The main weaknesses are an unstated and untested constant-focal-length assumption, a train/inference mismatch in trajectory quality that is not analyzed, and a missing key hyperparameter for the sliding window.","major_comments":[{"comment":"The derivation cancels the focal length f between times t and t0, which is valid only if the focal length is constant over time. Under optical zoom or digital crop, a static scene produces projected 2D trajectories that are formally indistinguishable from trajectories caused by depth motion, so the mapping from 2D trajectories to depth ratios is non-identifiable without an additional assumption or an input that encodes the changing intrinsics. The paper never states the constant-focal-length assumption, and no experiment with time-varying intrinsics is reported; both training (Sec. 4.2, Kubric with fixed camera intrinsics) and evaluation (TAPVid-3D, fixed-camera videos) avoid exercising this confound. I ask the authors to state the assumption explicitly and to either add an experiment with changing focal length (for example, synthetic sequences with smoothly varying f, or digitally cropped videos) or restrict the abstract and introduction claims to fixed-intrinsics videos.","section":"Sec. 3.2, Eqs. (1)-(4)"},{"comment":"The model is trained on ground-truth trajectory positions and ground-truth occlusion masks, as stated in Sec. 8, while at inference it receives trajectories from CoTracker or LocoTrack, which contain tracking noise, drift, and failures. The reported benchmark results show that the pipeline works in practice with both trackers, but the sensitivity of the learned depth-ratio mapping to tracker error is never quantified. I request an analysis that corrupts ground-truth trajectories with Gaussian noise, drift, and incorrect occlusion labels during evaluation, reporting how APD, 3D-AJ, and TC degrade as a function of perturbation level. This would directly test the most fragile premise of the method.","section":"Sec. 4.2 and supplementary Sec. 8"},{"comment":"The sliding-window stride S is a free parameter used in training, in the accumulation formula Eq. (9), and in inference, but the implementation details specify only the window size W = 8 and never state the value of S. Without S, the exact experimental setup and the ratio-accumulation procedure cannot be reproduced. Report the stride (and any overlap schedule) used in all experiments.","section":"Sec. 4.2 and Eq. (9)"}],"minor_comments":[{"comment":"The cosine factor in Eq. (3) appears to be squared by mistake: from Eq. (2), the density ratio is (d_t0/d_t)^2 * (cosθ_t/cosθ_t0), not (d_t0/d_t)^2 * (cosθ_t/cosθ_t0)^2. The subsequent Eq. (4) is consistent with the corrected form, so this seems to be a typographical error rather than a substantive one.","section":"Sec. 3.2, Eq. (3)"},{"comment":"Table 7's caption contains the typo 'meidan scaling' (should be 'median scaling'), and Table 9's header cites 'DepthPro [3]' where the reference should be [4].","section":"Table 7 caption and Table 9 header"},{"comment":"The description of ablation (IV), 'ratio depth with respect to the query point,' is unclear: since query points are trajectories, please specify the anchor time and how this loss differs from the proposed window-wise log-ratio loss.","section":"Sec. 4.4, Table 3 (left)"},{"comment":"Several numeric entries in Table 6 have irregular spacing (for example, '4.8 8 .6' and '7.7 14 .1'), which makes the table hard to read; reformat the entries.","section":"Table 6"}],"recommendation":"major_revision","confidential_remarks":"The zoom ambiguity is a genuine scope issue, but it is fixable by restating the constant-intrinsics assumption and adding an experiment; I do not see circularity in the evaluation, and the synthetic-to-real setup is a strength. The missing stride value and the lack of a code release are reproducibility concerns that should be addressed in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is a genuinely new way to get temporally coherent depth from monocular video: feed 2D point trajectories into a transformer and predict per-point depth ratios. It is not a rehash of existing video depth estimators. Second, the empirical results are real. On TAPVid-3D, Seurat beats per-frame MDE-plus-tracking baselines by a large margin in APD and especially in temporal coherence, and it also wins against video-depth baselines like DepthCrafter and ChronoDepth. The ablations are careful, and the handcrafted-baseline comparison in Table 4 is a nice touch that shows the learned model beats a direct implementation of the density formula.\n\nThe theoretical motivation—density changes of tracked points encode depth changes—is intuitive, and the derivation in Sec. 3.2 is simple but sufficient. The decoupled supporting-grid branch is a sensible architectural choice, and the sliding-window log-ratio loss is well motivated. The paper is also honest about some limitations in Sec. 9, though it misses the biggest one.\n\nThe stress-test concern lands. The mapping from 2D trajectories to depth ratios is only valid for constant focal length. In Eq. (1)–(4), f cancels only because it is assumed time-invariant, but that assumption is never stated. Under zoom or a digital crop, the same 2D tracks can arise from a depth change or a focal-length change, and Seurat would systematically misinterpret the latter. Training on Kubric (fixed intrinsics) and evaluating on TAPVid-3D (no zoom) means the confound is never exercised. The benchmark numbers stand, but the abstract's claim of \"generalizing effectively\" overstates the scope. This is fixable: state the assumption, add a synthetic zoom test, or test on a real video with known changing intrinsics.\n\nSecond soft spot is the train/inference gap. The model is trained on ground-truth trajectories and occlusion masks, but deployed on noisy CoTracker/LocoTrack outputs. The paper shows the method works with both trackers, but never quantifies sensitivity to tracking noise. A simple experiment adding controlled noise to ground-truth trajectories would bound the effect. Minor issues: the sliding-window stride S is never specified, no code is linked, and all numbers are single-run without error bars.\n\nNone of this is fatal. The core idea is sound and the experiments are convincing. Who is this for: anyone working on video depth, point tracking, or dynamic 3D reconstruction. It deserves a serious referee—the novelty and results justify the time—but the authors should be required to state the focal-length assumption and provide the missing implementation details. I would accept with revisions, not a desk reject.","headline":"Genuinely fresh idea—depth ratios from 2D trajectories—with strong TAPVid-3D results, but the unstated constant-focal-length assumption and the synthetic-to-real tracker gap need explicit handling before the generalization claim is safe.","tokens_in":18201,"tokens_out":2061,"would_cite":true,"duration_ms":19559,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Trajectory patterns alone can reveal depth changes over time.","keywords":["monocular video depth","point tracking","depth ratio","temporal coherence","zero-shot transfer","transformer","sliding window","TAPVid-3D"],"falsifier":"Feed the trained model TAPVid-3D trajectories that have been perturbed with increasing levels of Gaussian noise or with realistic tracker failure modes, and plot 3D-AJ and APD against noise amplitude: if even small perturbations that are well within typical tracker error cause large depth-ratio errors, the synthetic-to-real transfer is riding on trajectory quality rather than on learned geometry.","tokens_in":17134,"feed_emoji":"🎥","tokens_out":7808,"duration_ms":66731,"temperature":0.7,"pith_summary":"This paper argues that depth changes over time in a monocular video can be read off the 2D trajectories of tracked points: as a surface recedes or approaches the camera, its projected points spread out or bunch together, and that density change encodes the relative depth change. The authors build a transformer-based model that takes a set of 2D trajectories and their occlusion flags, predicts a depth ratio for each point relative to a reference frame, and then fuses these ratios with an off-the-shelf monocular depth estimator to recover metric depth. Trained only on synthetic videos, the model transfers zero-shot to real egocentric, driving, and studio videos, and on TAPVid-3D it reports smoother and more accurate depth changes than per-frame or video-depth baselines. The contribution matters because it points to a depth signal that uses no stereo, no multi-view geometry, no extra sensors, and no pretrained image backbone.","feed_headline":"2D point tracks alone can reveal object depth changes","feed_subtitle":"Synthetic-trained transformers lift 2D tracks to smooth 3D depth changes, beating per-frame depth baselines.","key_machinery":"The load-bearing object is the depth ratio along a trajectory, $r_{i,t}=d_{i,t}/d_{i,t_0}$, which the model predicts as a log ratio relative to the first frame of a sliding window. The architecture is a two-branch transformer: a supporting branch encodes a uniform grid of 24 by 24 trajectories with alternating temporal and spatial attention, capturing global scene motion, and a query branch processes the query trajectories with cross-attention into the supporting branch, so that biased query-point distributions do not contaminate the scene-motion estimate. The supervised target is an L1 loss on log depth ratios, with the theoretical density-depth identity $\\rho^{\\mathrm{image}}_{t_0}/\\rho^{\\mathrm{image}}_t = (d_{t_0}/d_t)^2(\\cos\\theta_t/\\cos\\theta_{t_0})^2$ as the motivating signal. At inference, ratios from overlapping windows of length 8 are accumulated by exponentiation and stitching, and per-visible-subsequence median scaling against a monocular depth estimator converts them to metric depths; iterative refinement and an auxiliary occlusion-position head stabilize the predictions.","core_discovery":"The central claim is that relative depth over time is recoverable from motion alone. Under a pinhole projection, the density of projected points from a small locally rigid patch scales inversely with the square of the depth, so the ratio of densities at two times determines the depth ratio up to an unknown surface-orientation factor; the paper argues that a transformer can learn this mapping implicitly without estimating surface normals explicitly. Concretely, the model predicts the log depth ratio $\\ell^{w}_{i,t} = \\log(d^{w}_{i,t}/d^{w}_{i,0})$ within short sliding windows, accumulates the ratios across windows, and rescales them to metric depth by matching medians with a monocular depth estimator per visible subsequence. Evaluated on TAPVid-3D with CoTracker and LocoTrack trajectories, the paper reports substantially better position accuracy and temporal coherence than unprojection baselines and video-depth estimators.","pith_inferences":["If trajectory geometry alone carries depth-change information, a promising next step is to train the same kind of model without depth supervision, for instance by enforcing consistency with rigid-flow or epipolar constraints between predicted depths.","The texture-patch ablation result suggests that appearance can hurt transfer; a purely geometric, appearance-free input may generalize better across rendering styles, so swapping RGB patches out entirely is likely more robust than adding them.","The sliding-window accumulation in Eq. (9) concatenates log-ratio errors, so very long videos may drift; adding overlap consistency or a global temporal smoothing term would be a natural robustness extension.","Within a frame, the model has limited spatial relative-depth ability by design, so it functions as a temporal coherence module rather than a full depth estimator; end-to-end training with a monocular branch could remove the need for the median-scaling fusion step."],"forward_implications":["Monocular video depth becomes obtainable without stereo rigs, multi-view setups, inertial sensors, or a pretrained image backbone.","Long and dynamic sequences, where per-frame depth estimators flicker, get temporally smooth depth because changes are predicted inside short windows and stitched across the whole video.","The depth-ratio output is metric-free, so it can be combined with any monocular metric depth estimator through per-subsequence median scaling.","Because the model is trained on synthetic data alone, the same recipe could be applied to new motion domains without collecting real depth labels."],"supporting_citations":[{"why":"Supplies the CoTracker trajectories used as input at inference and the alternating temporal/spatial attention design for the encoder.","marker":"[24]"},{"why":"LocoTrack, the other off-the-shelf point tracker whose trajectories the model consumes at inference.","marker":"[9]"},{"why":"Provides the iterative-refinement convention and the point-tracking transformer layer structure the model follows.","marker":"[8]"},{"why":"ZoeDepth, one of the monocular metric depth estimators used in the median-scaling fusion step.","marker":"[3]"},{"why":"DepthPro, the other metric depth estimator used in the fusion step and in baseline comparisons.","marker":"[4]"},{"why":"Kubric synthetic dataset generator used to create the 90,000 training samples.","marker":"[18]"},{"why":"TAPVid-3D benchmark, the dataset and protocol on which all depth, position, and temporal-coherence metrics are reported.","marker":"[27]"}],"fun_headline_variants":["Motion cues alone reveal depth from point tracks","Transformers lift 2D tracks to smooth 3D depth","Zero-shot depth from synthetic-trained point tracks","Track points, infer depth: no stereo needed"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the depth-ratio mapping learned from clean, ground-truth synthetic trajectories still holds when the input comes from real, noisy off-the-shelf trackers; the paper does not measure how quickly predictions degrade as tracker errors grow.","fun_headline_variants_meta":{"raw":{"variants":["Motion cues alone reveal depth from point tracks","Transformers lift 2D tracks to smooth 3D depth","Zero-shot depth from synthetic-trained point tracks","Track points, infer depth: no stereo needed"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000237,"raw_usage":{"total_tokens":1472,"prompt_tokens":872,"completion_tokens":600,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":488,"completion_tokens_details":{"reasoning_tokens":539}},"tokens_in":488,"tokens_out":600,"duration_ms":5692,"temperature":1.0,"reasoning_tokens":539,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:42:49.863602+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Feed the trained model TAPVid-3D trajectories that have been perturbed with increasing levels of Gaussian noise or with realistic tracker failure modes, and plot 3D-AJ and APD against noise amplitude: if even small perturbations that are well within typical tracker error cause large depth-ratio errors, the synthetic-to-real transfer is riding on trajectory quality rather than on learned geometry.","supporting_citations":[{"cited_title":"Flowtrack: Revisiting optical flow for long- range dense tracking","cited_arxiv_id":null,"evidence_quote":"Provides the iterative-refinement convention and the point-tracking transformer layer structure the model follows."},{"cited_title":"Kubric: A scalable dataset generator","cited_arxiv_id":null,"evidence_quote":"Kubric synthetic dataset generator used to create the 90,000 training samples."}],"review_version":1}