{"id":"40387b14-a885-4089-9346-35df48ec1157","arxiv_id":"2508.04928","paper_version":5,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Appending a few trainable tokens to each encoder layer of a frozen monocular depth estimator aligns fisheye image embeddings with perspective embeddings, enabling zero-shot fisheye depth estimation.","lead":"Researchers added small trainable 'calibration tokens' to the frozen internal layers of pretrained depth-estimation models, letting those models handle fisheye camera images without retraining. The tokens are trained on synthetic distortions of ordinary perspective photos, then tested zero-shot on real fisheye indoor and driving scenes, where they beat the unadapted models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Synthetic fisheye training cannot cover FOV>180, so zero-shot generalization to KITTI-360 is not established by the reported training distribution.","rationale":"The reader's weakest_assumption is that the synthetic distortion distribution is representative of real fisheye cameras, including KITTI-360's FOV exceeding 180 degrees. My reading of the method and supplement identifies the same load-bearing premise, and I make it more precise: the synthetic pipeline cannot even generate FOV>180 samples from perspective images, so the training distribution cannot cover a key part of the test distribution. This is an internal limitation of the training setup, not merely an external generalization risk. The reader already issued a CONDITIONAL verdict with MODERATE confidence, and this concern is exactly the kind of condition that justifies that verdict. I do not see grounds to reject the method outright, because the approach is plausible and shows consistent improvements on ScanNet++ (FOV <= 180) and central-region improvements on KITTI-360. I also do not see grounds to accept without additional evidence. Therefore the verdict should remain unchanged: CONDITIONAL, pending a demonstration that the synthetic distortion range covers the test cameras' projection geometry or that FOV>180 coverage is unnecessary.","tokens_in":18158,"tokens_out":4014,"duration_ms":49584,"concrete_test":"Estimate the actual projection function of KITTI-360 fisheye cameras (e.g., fit Kannala-Brandt coefficients to the provided calibration) and compute the maximum ray angle theta present. Then re-train the same calibration tokens using synthetic training images that include rays with theta > 90 degrees (e.g., rendered from 3D scenes or derived from equirectangular source images) and compare KITTI-360 RMSE and delta1 against the current tokens. If including FOV>180 training samples changes results substantially, the original zero-shot claim is unsupported; if results are unchanged, the concern is moot.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that tokens trained on synthetic fisheye distortions generalize to real fisheye cameras without retraining. The training distribution is generated by applying Kannala-Brandt distortion (Eq. 1) with coefficients in [-1.0,-0.01] to perspective images (Supp. B.3). A perspective image contains only rays with angle theta < 90 degrees to the optical axis. To represent a fisheye camera with FOV > 180 degrees, as KITTI-360 has (Sec. 4.1), the projection must include rays with theta > 90 degrees, which cannot be sampled from a perspective source image. Therefore, the tokens never see the distortion geometry characteristic of the most extreme regions of a >180-degree fisheye camera. The reported KITTI-360 improvements may be driven by central regions with mild distortion and by scale/alignment effects rather than by learned handling of extreme FOV. This does not invalidate the method for FOV <= 180 fisheye cameras, but it directly undermines the claim that a single token set generalizes to arbitrary fisheye cameras and that camera intrinsics are not needed at inference.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Calibration Tokens, a set of trainable tokens appended to the encoder layers of frozen Transformer-based foundational monocular depth estimators (FMDEs), to adapt perspective-trained models to fisheye images. The training is self-supervised: perspective images are synthetically distorted with Kannala-Brandt parameters, the FMDE predicts depth for both the perspective and distorted views, and a log-L1 loss is computed in the perspective reference frame after undistorting the fisheye depth prediction. No real fisheye images or ground-truth depth are used for training. The method is evaluated on ScanNet++ (indoor) and KITTI-360 (outdoor) with MiDaS, DepthAnything, and UniDepth, reporting RMSE and δ1 improvements over no-token baselines and comparison to DepthAnyCamera and FoV A-Depth, while claiming a single token set works for both indoor and outdoor scenes.","tokens_in":18332,"tokens_out":4525,"duration_ms":59110,"significance":"If validated, the central idea is attractive and practical: a small set of trainable tokens (<0.1% parameter increase, <1% latency increase) can adapt a frozen FMDE to fisheye cameras without finetuning, while remaining backward-compatible with perspective images and not requiring camera intrinsics at inference. The paper provides code, evaluates on multiple FMDEs, and includes ablations of design choices. The main risks are evaluative rather than architectural: the alignment protocol for relative-depth models is unspecified, and the synthetic distortion distribution may not cover the extreme rays of >180-degree fisheye cameras. These issues are load-bearing for the paper's strongest claims but appear addressable with additional analysis.","major_comments":[{"comment":"MiDaS and DepthAnything produce affine-invariant relative depth, but Table 1 reports RMSE against ground truth without specifying any scale/shift alignment. If raw predictions are compared directly, RMSE is not meaningful; if alignment is used, the procedure must be described. Please state exactly what alignment (per-image least-squares, global, none) was applied, and report scale-invariant metrics such as δ1 consistently. This is necessary to interpret the reported improvements.","section":"§4, Table 1, Supp. B.4"},{"comment":"Synthetic training data is generated by distorting perspective images, so every ray in the generated fisheye image satisfies θ < 90° relative to the optical axis. KITTI-360's fisheye cameras have FOV exceeding 180°, which includes rays with θ > 90°. Therefore the token set never encounters the geometry of the outermost fisheye regions during training, and the reported KITTI-360 improvements do not by themselves establish zero-shot generalization to >180° fisheye cameras. Please report results restricted to θ ≤ 90° versus θ > 90° (or a comparable FOV breakdown), and temper the generalization claim accordingly.","section":"§3.2, Eq. (1), Supp. B.3"},{"comment":"For DepthAnything on KITTI-360, RMSE improves from 2.214 to 2.043 but δ1 drops from 0.839 to 0.810. The text states that Calibration Tokens 'consistently improve accuracy' and that both MiDaS and DepthAnything improve on RMSE, but the δ1 decrease is not discussed. This is a direct inconsistency in the stated claim. Please address whether this is an alignment artifact or a real trade-off, and avoid claiming consistent improvement across all metrics.","section":"Table 1, KITTI-360 row"},{"comment":"The ablation study validates the design choices—LogL1 loss and layer-wise tokens—directly on the same real test sets (ScanNet++ and KITTI-360) that are used for final reporting, without a held-out validation split. This selection-on-test-set inflates the apparent advantage of the chosen configuration. Please add a validation split or explicitly acknowledge this as a limitation, and report the ablated variants on that split.","section":"§4.3, Table 4"}],"minor_comments":[{"comment":"The formula for φ is written as arctan((y − cx)/(x − cy)); the roles of x, y and cx, cy appear swapped. Please correct the notation.","section":"Eq. (2)"},{"comment":"The Discussion uses 'FMEs' instead of 'FMDEs' in the first sentence.","section":"§5"},{"comment":"The text says the authors 'measure relative depth estimation performance to mitigate the gap introduced by fisheye images,' but the reported metrics RMSE and δ1 are defined in absolute/linear depth space. Please clarify how relative depth is handled in the evaluation.","section":"Supp. B.4"},{"comment":"The rows for DepthAnyCamera trained on the 'Mix 200K' set show very large RMSE values, but no training details (optimizer, iterations, resolution) are given. Since this baseline is the main comparison, please include training hyperparameters or clarify whether the same protocol was used.","section":"Table 1, DepthAnyCamera rows"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is sound and the experimental setup is ambitious, but the evaluation protocol needs tightening before the central claims can be accepted. The missing alignment specification for relative-depth models and the lack of a θ>90° analysis for KITTI-360 are the two points I would treat as gatekeeping. The selection-on-test-set ablation is also concerning but could be addressed by a validation split or explicit caveat. I do not see grounds for rejection: the method is well-motivated, the overhead numbers are credible, and the authors provide code."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Calibration tokens are a simple and surprisingly cheap way to push perspective-trained depth foundation models to fisheye images: append a few trainable tokens per encoder layer, keep the backbone frozen, and train with a self-supervised consistency loss between the fisheye output and the model's own perspective output. On ScanNet++ and KITTI-360, it delivers consistent RMSE improvements over each frozen baseline, adds less than 1% memory and latency, and the code is public. That is a genuinely practical result, and the paper deserves a serious referee.\n\nThe new idea is the specific combination: layer-wise tokens as a calibration mechanism, plus a loss that applies the distortion in the input but undoes it in the output so you can supervise against the model's own high-quality perspective estimates. The paper is honest about the self-referential nature of the supervision, and the supplement's ablation showing supervised and self-supervised versions perform nearly identically is a nice check.\n\nNow the soft spots, in proportion. The biggest one is the FOV coverage problem. Training distortions are generated by applying Kannala-Brandt coefficients to perspective images. A perspective image only contains rays within its own field of view, typically <90 degrees half-angle, so the synthesized fisheye images never contain rays with theta > 90 degrees. KITTI-360's fisheye cameras exceed 180 degrees FOV, meaning a substantial part of the image has rays outside the training distribution. The reported KITTI-360 improvements may stem from central regions and scale alignment rather than from learned handling of extreme FOV. The paper's claim that the tokens 'demonstrate robustness across various distortion models' is stronger than the training setup supports. This doesn't kill the method for ordinary fisheye cameras, but it does mean the zero-shot generalization to >180-degree systems is not established.\n\nSecond, the ablations for logL1 and layer-wise tokens are run on the test benchmarks. That is model selection on the test set. A held-out validation set would make the design choices credible.\n\nThird, the evaluation protocol for relative depth models (MiDaS, DepthAnything) is under-specified. They report RMSE and delta1 but don't state how the inverse depth or relative predictions are aligned to metric ground truth. Without that, the metric comparisons are hard to trust.\n\nFourth, the finetuning baseline collapses (e.g., MiDaS from 0.506 to 2.178 RMSE) and the paper doesn't explain why. Either the comparison isn't fair or there's something worth discussing.\n\nOn novelty: the mechanism is closely related to visual prompt tuning and register tokens, but the paper only partially cites that lineage. The contribution is the application to fisheye depth plus the distortion-consistency loss, not the token mechanism itself.\n\nWho should read it: people deploying depth foundation models on wide-angle cameras, and anyone working on lightweight adaptation of frozen transformers. It deserves peer review. A fair referee would ask for a validation split, a clear alignment protocol, and an explicit discussion of the FOV ceiling imposed by the synthetic data.","headline":"Calibration tokens are a cheap, practical fix for fisheye depth on frozen perspective models, but the paper's zero-shot generalization to >180-degree cameras is not supported by a training set that never sees rays beyond the perspective FOV.","tokens_in":18902,"tokens_out":3624,"would_cite":true,"duration_ms":40839,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A few trainable tokens let a frozen depth model estimate depth on fisheye images, without retraining, finetuning, or undistorting the input.","keywords":["monocular depth estimation","fisheye cameras","calibration tokens","vision transformers","self-supervised learning","covariate shift","Kannala-Brandt distortion","foundational models"],"falsifier":"Train the tokens with only mild synthetic distortions and test on KITTI-360's >180-degree fisheye images; if adding tokens then fails to improve over the frozen baseline's RMSE, the claim that the synthetic distortion range covers real fisheye cameras is falsified.","tokens_in":17975,"feed_emoji":"📷","tokens_out":7338,"duration_ms":80648,"temperature":0.7,"pith_summary":"Foundational monocular depth estimators, trained on tens of millions of perspective images, produce poor depth maps on fisheye images because lens distortion shifts the distribution of their internal representations. The paper claims this shift can be corrected in latent space: appending a small set of trainable Calibration Tokens to each transformer encoder layer modulates fisheye embeddings back toward the perspective-image distribution, so the frozen model estimates depth accurately on fisheye images. The tokens are trained self-supervised, using only perspective images distorted by a polynomial fisheye model, with the loss computed after warping the predicted fisheye depth back into the perspective frame. With eight tokens per layer, the method adds less than one percent memory and latency, improves over prior fisheye depth baselines indoors and outdoors, and leaves perspective performance untouched since the tokens can simply be removed.","feed_headline":"Calibration tokens cut fisheye depth error by up to 17%","feed_subtitle":"No retraining, no undistortion, no intrinsics: one frozen depth model handles fisheye images.","key_machinery":"Layer-wise Calibration Tokens $\\Phi \\in \\mathbb{R}^{L \\times M \\times F}$: a distinct set of $M$ trainable token vectors, $F$-dimensional, appended to the patch-token sequence at each of the $L$ encoder layers of a frozen vision-transformer FMDE and discarded from the output. The attention mechanism lets these tokens modulate fisheye latent embeddings toward the perspective distribution. The training objective warps the fisheye depth prediction back to the perspective frame with the inverse synthetic distortion $T^{-1}$ and minimizes $\\log(|\\tilde{d} - T^{-1} \\circ \\hat{d}| + 1)$, so supervision comes from the FMDE's own high-fidelity perspective depth estimates.","core_discovery":"Calibration Tokens are a set of $M$ trainable vectors appended to the patch-token sequence at every encoder layer of a vision-transformer FMDE, where they modulate the latent embeddings through attention. The paper's central discovery is that this latent-space recalibration is enough: the frozen FMDE, already capable of high-fidelity depth estimation on perspective images, also produces accurate depth on fisheye images once its embeddings are aligned to the perspective embedding distribution. The alignment is learned entirely from perspective images by synthesizing fisheye distortions with the Kannala-Brandt polynomial model, then minimizing the log-$L_1$ difference between the FMDE's perspe","pith_inferences":["Beyond the paper: the same latent-alignment mechanism could in principle adapt FMDEs to other camera-induced covariate shifts, such as severe barrel or pincushion distortion, rolling shutter, or synthetic non-lens degradations, as long as they can be generated from perspective images; the paper tests only fisheye distortion.","Because supervision is the FMDE's own perspective output, the method's ceiling is the FMDE's perspective accuracy; in scenes where the FMDE misestimates perspective depth, Calibration Tokens cannot recover that error.","The zero-shot transfer to KITTI-360's field of view exceeding 180 degrees hints that the tokens learn something more general than the exact synthetic coefficient range used in training; a direct way to test that would be sweeping the synthetic distortion range and measuring the error curve, which the paper does not report."],"forward_implications":["One frozen FMDE can serve both perspective and fisheye cameras: append Calibration Tokens for fisheye input, omit them for perspective input, with no change to the backbone.","No camera calibration parameters are needed at inference; the tokens, trained over a range of synthetic distortions, absorb the calibration information.","Training scale is not limited by scarce real fisheye data, since any calibrated perspective image dataset can be synthetically distorted for token training.","Because only tokens are trained, adapting to a newer FMDE is much cheaper than full-model finetuning, and the same token mechanism is portable to any transformer-based FMDE.","The method consistently improves over the frozen baseline across backbones (12% RMSE reduction for MiDaS, 17% for DepthAnything indoors, 13% for UniDepth indoors) while adding under 0.05% memory and under 1 ms of latency."],"supporting_citations":[{"why":"MiDaS: the foundational perspective-trained depth estimator that supplies the baseline depth estimates used as self-supervision, and one of the backbones evaluated.","marker":"[50]"},{"why":"DepthAnything: the large-scale FMDE backbone on which the tokens produce the largest indoor RMSE improvement.","marker":"[74]"},{"why":"UniDepth v2: the FMDE backbone used for the main quantitative comparison and for the ablations.","marker":"[46]"},{"why":"Kannala-Brandt polynomial distortion model: the formula used to synthesize fisheye training images from perspective images.","marker":"[26]"},{"why":"AugUndo: the invertible-augmentation idea that motivates computing the self-supervised loss in the perspective frame after inverse warping.","marker":"[69]"},{"why":"Log-L1 loss formulation adopted for training stability, especially in distorted border regions.","marker":"[44]"},{"why":"DepthAnyCamera: the state-of-the-art fisheye depth baseline whose performance the method matches or surpasses without using camera intrinsics at test time.","marker":"[24]"},{"why":"FoV A-Depth: the equirectangular-projection baseline compared against on indoor and outdoor fisheye benchmarks.","marker":"[37]"},{"why":"ScanNet++: real indoor fisheye test set with ground-truth depth, used as the primary indoor benchmark.","marker":"[77]"},{"why":"KITTI-360: real outdoor fisheye test set with field of view exceeding 180 degrees, used as the primary outdoor benchmark.","marker":"[36]"}],"fun_headline_variants":["Calibration tokens align depth models to fisheye without retraining","Frozen depth nets see fisheye clearly with calibration tokens","Latent-space tokens fix fisheye depth, no new images needed","Self-supervised tokens unlock fisheye depth from perspective models","Token-based calibration adapts depth estimators to fisheye cameras"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The synthetic fisheye distortions, created by randomly distorting perspective images with Kannala-Brandt polynomial coefficients in the range [-1.0, -0.01], are representative enough of real fisheye cameras that a few tokens trained on synthetic images transfer zero-shot to real fisheye images, including cameras with fields of view beyond 180 degrees.","fun_headline_variants_meta":{"raw":{"variants":["Calibration tokens align depth models to fisheye without retraining","Frozen depth nets see fisheye clearly with calibration tokens","Latent-space tokens fix fisheye depth, no new images needed","Self-supervised tokens unlock fisheye depth from perspective models","Token-based calibration adapts depth estimators to fisheye cameras"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00028,"raw_usage":{"total_tokens":1525,"prompt_tokens":796,"completion_tokens":729,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":540,"completion_tokens_details":{"reasoning_tokens":640}},"tokens_in":540,"tokens_out":729,"duration_ms":7758,"temperature":1.0,"reasoning_tokens":640,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T23:40:22.851051+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the tokens with only mild synthetic distortions and test on KITTI-360's >180-degree fisheye images; if adding tokens then fails to improve over the frozen baseline's RMSE, the claim that the synthetic distortion range covers real fisheye cameras is falsified.","supporting_citations":[{"cited_title":"Towards Robust Monocu- lar Depth Estimation: Mixing Datasets for Zero-Shot Cross- Dataset Transfer","cited_arxiv_id":null,"evidence_quote":"MiDaS: the foundational perspective-trained depth estimator that supplies the baseline depth estimates used as self-supervision, and one of the backbones evaluated."},{"cited_title":"Depth anything: Unleashing the power of large-scale unlabeled data","cited_arxiv_id":null,"evidence_quote":"DepthAnything: the large-scale FMDE backbone on which the tokens produce the largest indoor RMSE improvement."},{"cited_title":"A generic camera calibration method for fish-eye lenses","cited_arxiv_id":null,"evidence_quote":"Kannala-Brandt polynomial distortion model: the formula used to synthesize fisheye training images from perspective images."},{"cited_title":"Augundo: Scaling up augmentations for monocular depth completion and estima- tion","cited_arxiv_id":null,"evidence_quote":"AugUndo: the invertible-augmentation idea that motivates computing the self-supervised loss in the perspective frame after inverse warping."},{"cited_title":"Excavating the potential capacity of self- supervised monocular depth estimation","cited_arxiv_id":null,"evidence_quote":"Log-L1 loss formulation adopted for training stability, especially in distorted border regions."},{"cited_title":"Depth Any Camera: Zero-Shot Metric Depth Estimation from Any Camera","cited_arxiv_id":"2501.02464","evidence_quote":"DepthAnyCamera: the state-of-the-art fisheye depth baseline whose performance the method matches or surpasses without using camera intrinsics at test time."},{"cited_title":"Fova-depth: Field-of-view agnostic depth es- timation for cross-dataset generalization","cited_arxiv_id":null,"evidence_quote":"FoV A-Depth: the equirectangular-projection baseline compared against on indoor and outdoor fisheye benchmarks."},{"cited_title":"Scannet++: A high-fidelity dataset of 3d in- door scenes","cited_arxiv_id":null,"evidence_quote":"ScanNet++: real indoor fisheye test set with ground-truth depth, used as the primary indoor benchmark."},{"cited_title":"Kitti-360: A novel dataset and benchmarks for urban scene understanding in 2d and 3d","cited_arxiv_id":null,"evidence_quote":"KITTI-360: real outdoor fisheye test set with field of view exceeding 180 degrees, used as the primary outdoor benchmark."}],"review_version":1}