{"id":"d43b99c9-5467-4531-aac1-0cf37d34deec","arxiv_id":"2411.11361","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"An autoregressive transformer that predicts depth maps in low-to-high resolution steps with recursively refined depth bins achieves state-of-the-art monocular depth accuracy on KITTI and NYU Depth v2.","lead":"This paper proposes DAR, a depth estimator that predicts depth maps autoregressively, from low to high resolution and from coarse to fine depth bins, and reports state-of-the-art results on KITTI and NYU Depth v2 with models up to 2.0B parameters. A generalist reader would care because it suggests autoregressive large models, the architecture behind GPT-style systems, can serve as competitive dense-prediction engines, not just text generators.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The MTBin recursion's one-bin error tolerance is never quantified; if the previous prediction is off by more than one bin, Eq. 9 cannot represent the true depth, and the claimed granularity-AR gains may not be driving the reported SOTA.","rationale":"The reader's weakest assumption is exactly the load-bearing concern I would raise: MTBin's error tolerance is limited to one bin on each side, and the paper provides no measurement of how often ground truth falls outside that expanded range. The central claim includes the assertion that the two autoregressive objectives plus the specific designs (patch-wise causal mask, MTBin, Bins Injection) deliver the SOTA results. If the granularity AR recursion excludes a non-negligible fraction of pixels, then the reported gains cannot be attributed to the coarse-to-fine mechanism; they might come from the pretrained ViT-L encoder or the resolution AR path. I see no internal inconsistency, but the missing coverage analysis is a testable empirical premise, not a mere presentational issue. A conditional verdict is appropriate: the paper should either provide coverage statistics or release code/checkpoints so the community can check them. The reader's other concerns (encoder symbol, matched-size margins, scaling-law support, first-AR claim) are secondary and do not affect the central mechanism as directly. My proposed test directly settles whether the coverage assumption holds in practice.","tokens_in":15299,"tokens_out":7612,"duration_ms":78898,"concrete_test":"Obtain or reimplement DAR-Small (no public code is provided). On the NYU Depth v2 validation set, at each autoregressive step k=2..5, find for each valid pixel the bin index t of \\tilde{D}_{k-1}(x) under the current bin edges, then check whether D_gt(x) lies within [b^{t-1}_{k-1}, b^{t+2}_{k-1}]. Report the per-step coverage fraction and the RMSE computed only on the pixels that fail coverage at the final step. If the coverage at step 5 is below roughly 99%, or if uncovered pixels have substantially higher RMSE than the overall RMSE, then the one-bin error tolerance is insufficient and the claim that MTBin's granularity AR drives the reported improvement is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"At each granularity step, MTBin defines the next bin centers from the previous prediction \\tilde{D}_{k-1}(x): if it lies in bin t, the new support is [b^{t-1}_{k-1}, b^{t+2}_{k-1}] (Eqs. 6-7). Eq. 9 then predicts \\tilde{D}_k as a convex combination of centers inside that interval. Any pixel whose ground truth falls outside this three-bin window is unrepresentable at that step: the softmax in Eq. 9 cannot concentrate at the true depth, and the scale-invariant loss (Eq. 13) has a nonzero floor. The recursion can only slide the window by at most one original bin per subsequent step, so an initial error of several bins requires multiple steps to reacquire the truth; with K=5, large errors may never be recovered. The paper never reports how often ground truth leaves the expanded interval, nor the contribution of uncovered pixels to the final RMSE. Because the model uses its own previous prediction (not teacher forcing) to set the bins, an early misprediction can exclude the true depth permanently. If a non-negligible fraction of pixels falls outside the support, the reported SOTA numbers may not be produced by the coarse-to-fine granularity AR mechanism the paper claims; they could instead reflect the pretrained encoder and the resolution AR objective. This is the most load-bearing unverified assumption in the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DAR, an autoregressive model for monocular depth estimation that predicts depth maps in a sequence of low-to-high resolutions (resolution AR) while simultaneously discretizing the depth range into progressively finer bins via a new Multiway Tree Bins (MTBin) strategy and injecting bin information into the latent token maps. Using a pretrained ViT encoder and a decoder-only DAR Transformer, the authors report state-of-the-art results on KITTI and NYU Depth v2, zero-shot generalization to SUN RGB-D, and scaling behavior up to 2.0B parameters.","tokens_in":15392,"tokens_out":11954,"duration_ms":111382,"significance":"If the claims hold, the paper provides evidence that autoregressive prediction is an effective and scalable paradigm for dense continuous prediction, and the resolution-granularity factorization is a useful new design. The strengths include benchmarking on held-out test splits against external baselines, a held-out zero-shot dataset, and an ablation study that shows each proposed component improves the baseline. The main weakness is that the core granularity mechanism rests on an unverified coverage assumption about the MTBin recursion, which the paper does not quantify.","major_comments":[{"comment":"The MTBin recursion expands the depth support to only the adjacent bins of the previous prediction, so any ground-truth value outside [b^{t-1}_{k-1}, b^{t+2}_{k-1}] is unrepresentable by the linear combination in Eq. (9) at that step; the paper never reports how often this occurs at each autoregressive step, nor how the loss in Eq. (13) treats such pixels. Because the bin selection is based on the model's own previous prediction rather than teacher forcing, an early error of more than one bin width can permanently exclude the true depth from the support and create an irreducible loss floor. Please provide coverage statistics on the training and test sets, and describe the training strategy (e.g., clamping, teacher forcing, or probabilistic bin selection) that mitigates this issue.","section":"Sec. 3.4, Eqs. (6)-(9)"},{"comment":"The operation that selects the bin index t from the continuous predicted depth \\tilde{D}_{k-1}(x) is a hard threshold (Eq. 5), and the subsequent bin boundaries in Eq. (7) are defined through this discrete index; the paper does not explain how gradients flow through this selection during end-to-end training. If a straight-through estimator or a soft approximation is used, it should be stated; otherwise it is unclear that the granularity objective can be optimized as described.","section":"Sec. 3.4, Eq. (5)"},{"comment":"The claim of beating Depth Anything \"with a similar model size\" is not tightly controlled: DAR-Small has 440M parameters versus Depth Anything's 343M, and DAR-Large has 2.0B, so part of the improvement may be attributable to the additional parameters and the multi-step compute budget. Please report a parameter-matched comparison (e.g., a DAR variant with roughly 343M parameters) or give wall-clock inference time and FLOPs for the compared models to support the claim that the autoregressive paradigm itself is the source of the gains.","section":"Sec. 4.2, Tables 2-3"}],"minor_comments":[{"comment":"The claim of being the \"first autoregressive model for MDE\" is stronger than needed given that Ord2Seq, cited later, is already an autoregressive ordinal-regression sequence predictor; consider rephrasing to \"first autoregressive model for MDE with the proposed resolution and granularity objectives.\"","section":"Sec. 1"},{"comment":"The parameters α and β of the scale-invariant loss are said to follow [29], but the original BTS loss uses different notational conventions; please clarify the exact relationship.","section":"Eq. (13) and Sec. 4.1"},{"comment":"The patch-wise causal mask is only described informally; a formal definition of the mask matrix over the concatenated token maps of all resolutions would make the method reproducible.","section":"Sec. 3.3"},{"comment":"The diagram is dense and the tensor shapes are not annotated; adding shape annotations at each stage would improve readability.","section":"Fig. 3"},{"comment":"Specify whether Depth Anything is used in its zero-shot pretrained form or after fine-tuning on NYU Depth v2, since the caption says all models are \"trained on NYU Depth V2.\"","section":"Table 4"},{"comment":"The parameter counts in the ablation rows decrease when adding MTBin and BI (420M to 363M); please explain what components are included in each row so the attribution of gains is unambiguous.","section":"Table 5"}],"recommendation":"major_revision","confidential_remarks":"The paper's empirical claims are substantial and internally consistent, but the main mechanism (MTBin) has a missing coverage analysis that could affect the validity of the granularity AR contribution. The non-differentiable bin selection and the unmatched parameter comparison with Depth Anything also need to be addressed. If the authors can provide coverage statistics and a controlled comparison, the paper may become acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Just read the DAR paper. It's a good one. The core idea—coupling resolution autoregression (VAR-style next-scale prediction) with granularity autoregression (coarse-to-fine ordinal regression with recursive bins)—is clean and the engineering is solid. The numbers are real: they beat Depth Anything on NYU and KITTI, and the zero-shot SUN RGB-D result is competitive. The ablation in Table 5 is honest: each component (MTBin, Bins Injection, scale-up) lifts the baseline. That's the right way to build a paper.\n\nThe problems are in the claims, not the experiments. First, the introduction calls DAR 'the first autoregressive model for MDE,' but their own reference [56] (Ord2Seq, same first author) is an autoregressive network for depth ordinal regression. That's a straightforward overclaim and easy to fix.\n\nSecond, the scaling-law claim is thin. Three model sizes on one dataset with no compute axis or fitted curve is not 'scaling laws akin to LLMs.' It's a performance trend. Minor, but worth rephrasing.\n\nThird—and this is the one I'd push on in review—the MTBin error-tolerance design is never quantified. Equations 5-7 say the next range is the previous bin expanded one bin on each side. If the coarse prediction is off by more than a bin, the true depth is outside the support and Eq. 9 cannot represent it. The recursion can shift the window by only about one bin width per step, so with K=5 a large initial error may never be recovered. The paper doesn't report how often the ground truth falls outside the expanded interval, or how the loss handles those pixels. The ablation suggests the mechanism works for most pixels, but 'most' needs a number. A simple diagnostic would settle it.\n\nAlso worth noting: the zero-shot generalization claim is partly inherited from the pretrained Depth Anything encoder, and the abstract's '5% improvement' is measured at 2B parameters against a 343M baseline; at matched size it's about 3%. Both are presentation issues, not fatal.\n\nNo code or checkpoints, which hurts reproducibility, but that's common in this area.\n\nBottom line: this paper deserves a serious referee. Send it out. Ask for a coverage analysis of MTBin and a rewrite of the 'first' claim. If those land, it's a solid CVPR-level contribution.","headline":"Solid, well-executed depth-estimation paper with real SOTA numbers, but the 'first autoregressive MDE' claim is overbroad and the MTBin error-tolerance assumption needs a diagnostic.","tokens_in":16179,"tokens_out":6495,"would_cite":true,"duration_ms":57074,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper advances the claim that monocular depth estimation can be recast as two coupled autoregressive objectives—over resolution and over granularity—and that the resulting DAR model beats prior state of the art on KITTI and NYU Depth…","keywords":["autoregressive depth estimation","monocular depth estimation","resolution autoregressive objective","granularity autoregressive objective","Multiway Tree Bins","patch-wise causal mask","ordinal regression","zero-shot generalization"],"falsifier":"Run DAR's training loop on KITTI or NYU and record, at each autoregressive step, the fraction of valid pixels whose ground-truth depth falls inside the expanded bin range $[b^{t-1}, b^{t+2}]$ of MTBin. If that coverage fraction drops substantially below 1 at later steps—or if removing the one-bin expansion and splitting only the predicted bin produces nearly the same final RMSE—then the error-tolerance mechanism is not doing the work the paper's argument requires, and the coarse-to-fine recursion can lock onto wrong intervals.","tokens_in":14867,"feed_emoji":"📏","tokens_out":7924,"duration_ms":67961,"temperature":0.7,"pith_summary":"Monocular depth estimation predicts a per-pixel depth map from a single RGB image, and is usually framed as one-shot dense regression. This paper argues that the task can instead be posed as two coupled autoregressive prediction problems: generating depth maps from low to high resolution, and refining depth values from coarse to fine bins. The resulting model, DAR, uses a decoder-only Transformer with a patch-wise causal mask for the resolution axis and a Multiway Tree Bins strategy with Bins Injection for the granularity axis. On KITTI and NYU Depth V2 the paper reports state-of-the-art results, and scaling the model to 2.0B parameters brings KITTI RMSE to 1.799, about 5% below the 1.896 of the previous best method. The paper also reports zero-shot generalization to an unseen dataset without large-scale pretraining, and argues that autoregressive prediction is therefore an effective and scalable paradigm for monocular depth estimation.","feed_headline":"Autoregressive depth model beats SOTA by 5% on KITTI","feed_subtitle":"DAR predicts depth from low resolution to high, and coarse bins to fine, with a 2B model reaching KITTI RMSE 1.799.","key_machinery":"The two load-bearing mechanisms are the patch-wise causal attention mask and the Multiway Tree Bins (MTBin) strategy. The mask lets the DAR Transformer treat each resolution level's token map as a single merged patch-wise token, so that a token at step $k$ can attend to its own map and all prefix maps but not to future maps; this is what carries the low-to-high resolution autoregressive objective. MTBin takes the previous step's predicted depth for a pixel, expands the bin range to include the adjacent bins on either side as error tolerance, and splits that range into new sub-bins that become the depth candidates for the next finer prediction; this is what carries the coarse-to-fine granularity autoregressive objective. A Bins Injection module, a ConvGRU that takes convolved bin candidates as context, couples the two processes by feeding granularity information into the resolution autoregressive token maps.","core_discovery":"The central claim is that the two ordering properties inherent to a depth map—spatial resolution and value granularity—are sufficient to turn monocular depth estimation into an autoregressive sequence problem. DAR is presented as the first autoregressive model for this task. It predicts depth maps at increasing resolutions, conditioning each higher-resolution map on all previous maps through a patch-wise causal mask, while simultaneously refining each pixel's depth range through MTBin, which re-centers the bin range on the previous coarse prediction and expands it by one bin on each side for error tolerance. The final depth at each step is a linear combination of softmax probabilities and bin centers, and the bin information is injected back into the latent token map through a ConvGRU module. The paper reports that this design outperforms prior state of the art on both NYU Depth V2 and KITTI, and that performance keeps improving as the model is scaled from 440M to 1B to 2B parameters, which it takes as evidence that autoregressive modeling is an effective and scalable paradigm for monocular depth estimation.","pith_inferences":["One implication the paper leaves implicit is that the one-bin error tolerance in MTBin sets a ceiling on how many refinement steps are useful: after enough steps, a non-trivial fraction of pixels will fall outside the expanded range, and their loss becomes irreducible. A testable extension would be to measure that coverage fraction per step and adapt the number of steps accordingly.","The resolution autoregressive objective can be read as a learned replacement for the skip connections and feature fusion used in encoder-decoder depth models; if so, the same mask-plus-upsampling pattern might transfer to other dense prediction tasks such as surface-normal or optical-flow estimation.","The paper's own limitations section notes that the progressive multi-step paradigm can blur boundaries and that the autoregressive Transformer is parameter-heavy; a natural follow-up is to test distillation or lighter AR transformer variants that preserve the reported gains at lower cost."],"forward_implications":["If the reported numbers hold, autoregressive prediction becomes a competitive alternative to encoder-decoder and diffusion backbones for monocular depth estimation, using the same architectural family that powers large language models.","Scaling behavior on model size suggests depth quality continues to improve up to at least 2.0B parameters, so larger DAR models would be a natural next test rather than a plateau.","Because the method reports strong zero-shot transfer after training on only one dataset, it may reduce the need for massive unlabeled pretraining in depth estimation.","The granularity objective with MTBin offers a new way to interleave discrete bin prediction with continuous regression, which could transfer to other continuous per-pixel prediction tasks.","The design points toward a practical route for adding metric depth prediction to existing autoregressive foundation models that currently lack dense visual output capabilities."],"supporting_citations":[{"why":"Introduces next-scale autoregressive image generation, the template for DAR's low-to-high resolution objective.","marker":"[53]"},{"why":"Treats ordinal regression as label-sequence prediction, the ancestry of the granularity autoregressive objective.","marker":"[56]"},{"why":"The state-of-the-art baseline and backbone on KITTI and NYU that DAR compares against and outperforms.","marker":"[63]"},{"why":"Established discretizing depth into ordered bins and formulating depth estimation as ordinal regression.","marker":"[14]"},{"why":"Adaptive binning approach for monocular depth that MTBin refines into a recursive coarse-to-fine scheme.","marker":"[4]"},{"why":"Scaling laws for neural language models used to motivate scaling DAR from 440M to 2.0B parameters.","marker":"[24]"},{"why":"Decoder-only Transformer architecture that DAR's Transformer is built from.","marker":"[42]"},{"why":"Residual quantization autoregressive image generation, a prior example of progressive multi-scale autoregressive visual prediction.","marker":"[28]"}],"fun_headline_variants":["Autoregressive scaling: DAR hits 1.799 RMSE on KITTI","DAR: Autoregressive depth model beats prior SOTA by 5%","From coarse to fine: autoregressive depth estimation scales to 2B","First autoregressive MDE: DAR sets new SOTA on KITTI and NYU","Autoregressive depth: 2B model achieves 1.799 RMSE on KITTI"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The granularity autoregressive objective assumes that each pixel's previous-step prediction lands within one bin width of the true depth, because MTBin only expands the search range to the adjacent bins before splitting; if the coarse prediction is off by more than that, the true depth falls outside every candidate bin center and the model's loss at that pixel is irreducibly high.","fun_headline_variants_meta":{"raw":{"variants":["Autoregressive scaling: DAR hits 1.799 RMSE on KITTI","DAR: Autoregressive depth model beats prior SOTA by 5%","From coarse to fine: autoregressive depth estimation scales to 2B","First autoregressive MDE: DAR sets new SOTA on KITTI and NYU","Autoregressive depth: 2B model achieves 1.799 RMSE on KITTI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000243,"raw_usage":{"total_tokens":1560,"prompt_tokens":1009,"completion_tokens":551,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":625,"completion_tokens_details":{"reasoning_tokens":439}},"tokens_in":625,"tokens_out":551,"duration_ms":5032,"temperature":1.0,"reasoning_tokens":439,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:38:20.918451+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run DAR's training loop on KITTI or NYU and record, at each autoregressive step, the fraction of valid pixels whose ground-truth depth falls inside the expanded bin range $[b^{t-1}, b^{t+2}]$ of MTBin. If that coverage fraction drops substantially below 1 at later steps—or if removing the one-bin expansion and splitting only the predicted bin produces nearly the same final RMSE—then the error-tolerance mechanism is not doing the work the paper's argument requires, and the coarse-to-fine recursion can lock onto wrong intervals.","supporting_citations":[{"cited_title":"Ord2Seq: Regarding ordinal regression as label sequence prediction","cited_arxiv_id":null,"evidence_quote":"Treats ordinal regression as label-sequence prediction, the ancestry of the granularity autoregressive objective."},{"cited_title":"Depth anything: Unleashing the power of large-scale unlabeled data","cited_arxiv_id":null,"evidence_quote":"The state-of-the-art baseline and backbone on KITTI and NYU that DAR compares against and outperforms."},{"cited_title":"Deep ordinal regression net- work for monocular depth estimation","cited_arxiv_id":null,"evidence_quote":"Established discretizing depth into ordered bins and formulating depth estimation as ordinal regression."},{"cited_title":"AdaBins: Depth estimation using adaptive bins","cited_arxiv_id":null,"evidence_quote":"Adaptive binning approach for monocular depth that MTBin refines into a recursive coarse-to-fine scheme."},{"cited_title":"Language models are unsu- pervised multitask learners","cited_arxiv_id":null,"evidence_quote":"Decoder-only Transformer architecture that DAR's Transformer is built from."},{"cited_title":"Autoregressive image generation using residual quantization","cited_arxiv_id":null,"evidence_quote":"Residual quantization autoregressive image generation, a prior example of progressive multi-scale autoregressive visual prediction."}],"review_version":1}