{"id":"f884842b-0dc1-4b8a-94a2-c8cf7e1c72da","arxiv_id":"2607.12993","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"X-Lens fuses arbitrary calibrated fisheye and pinhole views into real-time metric depth at 41 FPS with a 0.04B-parameter model and a new 266K-frame synthetic dataset.","lead":"X-Lens is a small neural network that estimates metric depth in real time from a mix of fisheye and pinhole cameras, using just 0.04B parameters and running at up to 41 FPS. It was trained on OmniScene, a new 266K-frame synthetic six-camera dataset the authors built.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Heterogeneous-camera superiority is established only on an in-distribution synthetic benchmark against zero-shot baselines; the 25.4% AbsRel margin needs a matched-training or real-mixed-rig test before it can be attributed to the architecture.","rationale":"I agree with the reader's conditional assessment. The architecture is coherent, ablations in Table 4 are internally consistent, and real homogeneous results (KITTI360 fisheye, OmniOcc pinhole) provide some evidence of transfer. However, the central quantitative claim (25.4% AbsRel reduction in heterogeneous setting) rests entirely on OmniScene-Full, where X-Lens had access to the same synthetic distribution, camera intrinsics, and rig geometry during Stages 2–3 while baselines did not. The supplement's split protocol (Sec. D) confirms scene-disjointness but not distribution-disjointness. The paper's own Limitations section concedes that unseen fisheye/FOV models still cause a sim-to-real gap, and no real mixed-lens benchmark is reported. Because the test scenes share the exact rig and renderer, the benchmark cannot distinguish 'the geometry modules generalize across heterogeneous cameras' from 'the model memorized the OmniScene camera distribution.' A matched baseline trained on the same data is the minimal experiment that would separate these; if the gap persists, the claim is substantially strengthened. The missing dataset link also hampers independent reproduction, but the data-access confound is the more load-bearing issue. Therefore the reader's CONDITIONAL verdict is appropriate and no adjustment is needed.","tokens_in":23972,"tokens_out":7534,"duration_ms":159421,"concrete_test":"Fine-tune the strongest applicable baseline (e.g., MapAnything or UniDAC) on the same OmniScene Stage-3 training data used for X-Lens (91 scenes, four fisheye + two pinhole rig, same view-count sampling), then re-evaluate on the held-out 12-scene OmniScene-Full split of Table 3. If the 25.4% AbsRel advantage shrinks or disappears, the headline claim is explained by training-distribution access; if it persists under matched training, the geometry modules are the likely cause.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is the comparison underlying the central claim. In Table 3, X-Lens is trained on OmniScene (Stage 2 fisheye adaptation on OmniScene; Stage 3 heterogeneous joint training on OmniScene and KITTI360), while the compared baselines (UniDAC, DepthAnyCamera, MapAnything) are applied zero-shot, with monocular baselines applied per view. The OmniScene-Full test split is scene-disjoint but shares the same Kujiale/UE assets, Kannala-Brandt 180° fisheye model, pinhole intrinsics, six-camera rig, and occupancy-waypoint trajectory sampler as the training split (Sec. 4; Sec. D). Thus the 25.4% AbsRel improvement (0.1368→0.1021) and the 68.1% Scale AbsRel improvement may largely measure train-test distribution overlap rather than the calibration-token/Jacobian-bias design. The internal ablations (Table 4) show the modules help within this distribution, but they do not control for data access. The paper's own Limitations section concedes a sim-to-real gap for unseen fisheye/FOV configurations, and no real mixed fisheye+pinhole evaluation is reported (KITTI360 is fisheye-only; OmniOcc is pinhole-only). So the headline 'superior heterogeneous-camera metric depth accuracy' is unsupported outside the synthetic training distribution and versus unmatched baselines.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents X-Lens, a 0.04B-parameter feed-forward model for metric depth estimation from a variable number of calibrated fisheye and pinhole views. The model combines a DINOv2/DPT backbone with three main components: layer- and camera-type-specific calibration tokens, a Jacobian-parameterized distortion bias injected into cross-view attention, and a confidence-weighted 'Scale Attention' pooled metric-scale head. Training is staged: (1) pinhole multi-view pretraining on 13 public datasets, (2) fisheye adaptation using only calibration tokens on the new OmniScene synthetic dataset, and (3) joint heterogeneous fine-tuning on OmniScene, KITTI360, and pinhole data. OmniScene contains about 266K six-view frames (four 180° fisheye plus two pinhole cameras) rendered from 103 Kujiale/UE scenes. Experiments cover fisheye-only (KITTI360, OmniScene-Single/Quad), pinhole-only (ETH3D, ScanNet++V2, OmniOcc), and heterogeneous (OmniScene-Full) settings. The headline claim is that X-Lens reduces AbsRel by 25.4% on OmniScene-Full over the strongest baseline (UniDAC) while using 88.9% fewer parameters and running at 22 FPS. The paper also reports competitive fisheye results and 41 FPS on KITTI360, plus a small downstream robotic-manipulation study.","tokens_in":24393,"tokens_out":4414,"duration_ms":44261,"significance":"If the heterogeneous-camera claims hold beyond the exact training distribution, X-Lens would be a practically valuable contribution: a compact, real-time model with explicit metric scale, no auxiliary reconstruction heads, and native support for mixed fisheye/pinhole rigs. The method is presented clearly, the loss in Eq. (7) directly supervises the metric scale through the |m-hat - m|/m term, so there is no evident circularity, and the staged training design is well motivated. The ablation study in Table 4 is thorough and internally consistent, showing that each stage and each proposed module contributes within the evaluated distribution. The release of code and the OmniScene dataset is a useful resource. However, the central 'superior heterogeneous-camera metric depth accuracy' claim is currently supported only by an in-distribution synthetic benchmark against zero-shot baselines, and the paper's own limitations section concedes a sim-to-real gap for unseen fisheye/FOV configurations. The significance of the work therefore depends on additional matched-training or real-mixed-rig evidence.","major_comments":[{"comment":"The central heterogeneous-camera claim is measured only on OmniScene-Full. The test scenes are scene-disjoint, but the test split uses the same rendering assets (Kujiale/UE), the same Kannala-Brandt 180° fisheye model, the same pinhole intrinsics, the same six-camera rig, and the same occupancy-waypoint trajectory sampler as the training split. The compared baselines (UniDAC, DepthAnyCamera, MapAnything) are applied zero-shot, with monocular baselines applied per view without cross-view fusion, while X-Lens is trained on OmniScene in Stages 2 and 3. The 25.4% AbsRel reduction (0.1368→0.1021) and 68.1% Scale AbsRel reduction may therefore largely reflect train-test distribution overlap and evaluation asymmetry rather than the calibration-token/Jacobian-bias design. The internal ablations in Table 4 show the modules help within this distribution, but they do not control for data access. Pl","section":"§5.2, Table 3; §4; §D"},{"comment":"On real pinhole ETH3D, X-Lens achieves AbsRel 0.0445 and δ1 0.9723, compared with MapAnything at 0.0228/0.9992 and VGGT-Omega at 0.0055/0.9994. This is a large gap on a real-world benchmark and contradicts the abstract's 'competitive performance on conventional pinhole-only settings.' The paper reports only Scale AbsRel as the headline there (0.1217, which is best), but the dense-depth metrics are substantially worse than strong pinhole baselines. This result should be honestly contextualized; as written, it undermines the claim that X-Lens preserves pinhole performance and raises concerns about zero-shot generalization beyond the synthetic training distribution.","section":"§5.1, Table 2 (ETH3D)"},{"comment":"The Limitations section explicitly states that for 'unseen fisheye lens models with extreme FOV configurations that deviate drastically from the training distribution, a noticeable sim-to-real gap remains,' and the model strictly requires ground-truth calibration. There is no real-world mixed fisheye+pinhole evaluation in the paper: KITTI360 is fisheye-only, OmniOcc is pinhole-only, and OmniScene-Full is synthetic. Since the paper's central claim is 'superior heterogeneous-camera metric depth accuracy,' the absence of any real mixed-rig test means the headline claim is unsupported outside the synthetic training distribution. At minimum, add a small real mixed-rig evaluation or explicitly reframe the headline as an in-domain synthetic result.","section":"§7 (Limitations)"}],"minor_comments":[{"comment":"The comparison with MapAnything claims '96.7% fewer parameters,' but MapAnything is a general feed-forward reconstruction model solving a broader task (joint depth, point maps, poses) than X-Lens. Parameter-count comparisons across models with different output scopes should be accompanied by a task-scope qualifier.","section":"§5.2, Table 3"},{"comment":"The FPS numbers are measured on a single H100 without specifying batch size, memory, or whether the reported FPS includes preprocessing/calibration-token injection. Please provide a reproducible speed protocol.","section":"§5, Implementation Details"},{"comment":"The 'Stage-2 only' row for pinhole OmniOcc is identical to 'Stage-1 only' (0.1052/0.0717/0.2801). This is expected since Stage 2 uses only fisheye data, but it should be stated explicitly to avoid confusion.","section":"§6, Table 4"},{"comment":"The wording 'noticeable sim-to-real gap remains, causing a slight degradation in performance' is internally inconsistent: a 'noticeable gap' is hard to reconcile with 'slight degradation.' Please align the language with the actual evidence.","section":"§7, Limitations"}],"recommendation":"major_revision","confidential_remarks":"The paper is well written and the method is internally coherent, but the central heterogeneous-camera claim is currently supported only by an in-distribution synthetic benchmark against zero-shot baselines. The missing real mixed-rig evaluation and the weak real-pinhole ETH3D dense-depth numbers are the main obstacles. A major revision that adds matched-training baselines or a real mixed-fisheye/pinhole experiment would make the claim credible. I do not see a load-bearing mathematical error; the issue is the scope and fairness of the evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I read through X-Lens with your stress-test note in hand, and I think you've put your finger on the main issue. The engineering is real — 0.04B parameters, 22-41 FPS, a clean factored output, and a large new synthetic dataset with 266K six-view frames. The layer-wise calibration tokens and Jacobian distortion bias are new, and the ablations show they help within the training distribution. No mathematical circularity either: metric scale is supervised directly in Eq. (7), and the loss decomposition separates shape from scale cleanly.\n\nThe soft spot is exactly where you say it is. The headline 'superior heterogeneous-camera metric depth accuracy' is supported only by OmniScene-Full: synthetic, rendered with the same six-camera rig, the same Kannala-Brandt fisheye model, and the same trajectory sampler used for training. The baselines (UniDAC, DepthAnyCamera, MapAnything) are all zero-shot. So the 25.4% AbsRel gap mostly measures train-test overlap, not cross-lens architecture. The internal ablations are consistent but they don't control for data access — they show the modules help inside this distribution, not that the method generalizes to a real mixed rig.\n\nThe paper is honest about the sim-to-real gap in its limitations section, and it never claims to have tested a real mixed fisheye-plus-pinhole rig, because it hasn't. KITTI360 is fisheye-only; OmniOcc is pinhole-only. On real pinhole ETH3D, X-Lens sits at AbsRel 0.0445 versus VGGT-Omega's 0.0055, so calling the pinhole performance 'competitive' is about scale prediction and speed, not dense accuracy. That is fine, but it should be stated that way.\n\nOne more practical issue: OmniScene is advertised as newly released, but I don't see a working dataset link in the paper — only a project page and a GitHub repo. If the dataset is the main reproducible contribution, that needs to be sorted out.\n\nMinor thing: the appendix robot-policy experiment feels like a separate paper and is too preliminary to be evidence, but it doesn't hurt the geometry story.\n\nBottom line: this deserves a serious referee. I'd ask for a real mixed-rig evaluation or at minimum a matched-training comparison on OmniScene before the central claim is accepted. The architecture and the dataset are worth a revised version, and I'd be happy to see it go through the normal review process.","headline":"Solid engineering and a big synthetic dataset, but the heterogeneous-camera claim is built on in-distribution synthetic numbers and zero-shot baselines.","tokens_in":24898,"tokens_out":3117,"would_cite":true,"duration_ms":31729,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"X-Lens establishes that a 0.04B-parameter network can estimate metric depth from mixed fisheye and pinhole views, cutting AbsRel by 25.4% over the strongest baseline on a six-view heterogeneous benchmark while running at 22 FPS.","keywords":["metric depth estimation","heterogeneous cameras","fisheye","pinhole","multi-view depth","real-time inference","synthetic dataset","geometry-aware attention"],"falsifier":"Take a real mixed fisheye-pinhole rig (roughly four 180-degree fisheye and two pinhole cameras) with LiDAR ground truth and intrinsics that differ from OmniScene's, and evaluate X-Lens zero-shot without any fine-tuning. If AbsRel degrades to near or above the level of strong monocular any-camera baselines, or if removing the Jacobian distortion bias makes no measurable difference, the claim that the geometry modules drive heterogeneous accuracy would be falsified.","tokens_in":1480,"feed_emoji":"🎯","tokens_out":1583,"duration_ms":174955,"temperature":0.7,"pith_summary":"The paper tries to prove that real-time metric depth estimation does not require giant reconstruction-oriented models or panorama stitching: a compact feed-forward network can natively fuse calibrated fisheye and pinhole views and still beat much larger systems on heterogeneous-camera accuracy. To do this, X-Lens reasons entirely in ray space, uses learnable calibration tokens to adapt each lens's distortion, and injects a Jacobian-based bias into cross-view attention so the model matches features using local projection geometry. The authors also release OmniScene, a large synthetic six-view dataset, and report superior accuracy on it plus competitive results on conventional fisheye-only and pinhole-only benchmarks. If the central claim holds, mixed-rig perception for robots, vehicles, and embodied systems becomes feasible at a fraction of the usual compute cost.","feed_headline":"0.04B model beats 1.2B rivals on mixed fisheye-pinhole depth","feed_subtitle":"Relative depth error falls 25.4% while parameter count drops 88.9% and six-view runs at 22 FPS","key_machinery":"The central object is the generic unprojection map G, which maps each pixel to a unit ray while absorbing per-camera calibration and camera type; every attention and positional operation then runs in this shared ray space rather than in pixel or pinhole-camera coordinates. Two mechanisms carry the heterogeneous-camera argument: (1) calibration tokens, learnable per network layer and per camera type, injected only for fisheye views and kept view-local in cross-attention, so lens distortion is absorbed without contaminating cross-view fusion; (2) a Jacobian distortion bias, computed from per-patch ray fields and their finite-difference Jacobians, added to cross-view attention logits to bias ma","core_discovery":"X-Lens is a feed-forward network that takes a variable number of calibrated fisheye and pinhole views and predicts, per view, a dense normalized depth map, a confidence map, and a single global metric scalar; multiplying the scalar by the normalized depth yields metric depth. All internal geometry is expressed in ray space through a generic unprojection map that absorbs both pinhole intrinsics and fisheye distortion, avoiding panoramic resampling. Two modules carry the heterogeneous alignment: multi-view calibration tokens, injected layer-wise only for fisheye views, provide view-local lens correction, and a Jacobian-parameterized distortion bias, added to cross-view attention logits, favors","pith_inferences":["A natural next test, left implicit by the paper, is zero-shot evaluation on a real mixed fisheye-pinhole rig with LiDAR ground truth; if transfer holds, X-Lens could serve as a cheap geometric encoder for robot policies, a direction the appendix begins to explore.","The Jacobian-bias idea is generic: any multi-camera transformer that knows pixel-to-ray mappings could inject similar local projection priors, potentially improving tasks such as occupancy prediction or 3D detection on heterogeneous rigs.","The heaviest dependency is the synthetic-to-real bridge; the paper's own limitation section admits a sim-to-real gap for extreme FOV, so the headline gain may shrink outside OmniScene-like intrinsics.","A testable extension is to ablate the calibration tokens and Jacobian bias on a real out-of-domain rig; if their contribution vanishes there, the gain may reflect training-distribution overlap rather than geometric conditioning."],"forward_implications":["If the central claim is correct, real-time metric depth on mixed fisheye-pinhole rigs is achievable with a 0.04B-parameter model, eliminating the need for panorama stitching or separate networks per camera type.","The 25.4% AbsRel improvement over the strongest baseline on OmniScene-Full, with 88.9% fewer parameters, suggests the geometry-aware modules add measurable accuracy rather than relying on scale.","The three-stage training recipe (pinhole pre-train, fisheye token adaptation, heterogeneous joint fine-tune) provides a transferable schedule for other heterogeneous sensor-fusion problems.","Because all reasoning is in ray space, the same architecture can ingest pinhole, fisheye, and 360-degree cameras without architectural changes, as the paper states.","The released OmniScene dataset, with scene-disjoint splits, provides a benchmark and training resource for future heterogeneous-camera depth research."],"fun_headline_variants":["25% better depth error with 89% fewer params on mixed cameras","Fisheye and pinhole depth in one 0.04B model at 41 FPS","Compact model beats heavy baselines for heterogeneous depth","X-Lens: 0.04B params, 25% lower AbsRel on mixed views","Real-time metric depth from fisheye and pinhole with 0.04B"],"cache_read_input_tokens":26112,"weakest_assumption_plain":"The headline heterogeneous-camera advantage rests on OmniScene-Full, a synthetic benchmark rendered with the same six-camera rig geometry, asset pipeline, and trajectory sampler that generated the model's training data, so the result may reflect in-distribution familiarity rather than genuine cross-lens generalization.","fun_headline_variants_meta":{"raw":{"variants":["25% better depth error with 89% fewer params on mixed cameras","Fisheye and pinhole depth in one 0.04B model at 41 FPS","Compact model beats heavy baselines for heterogeneous depth","X-Lens: 0.04B params, 25% lower AbsRel on mixed views","Real-time metric depth from fisheye and pinhole with 0.04B"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000417,"raw_usage":{"total_tokens":2005,"prompt_tokens":777,"completion_tokens":1228,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":521,"completion_tokens_details":{"reasoning_tokens":1129}},"tokens_in":521,"tokens_out":1228,"duration_ms":9637,"temperature":1.0,"reasoning_tokens":1129,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T06:08:22.377902+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a real mixed fisheye-pinhole rig (roughly four 180-degree fisheye and two pinhole cameras) with LiDAR ground truth and intrinsics that differ from OmniScene's, and evaluate X-Lens zero-shot without any fine-tuning. If AbsRel degrades to near or above the level of strong monocular any-camera baselines, or if removing the Jacobian distortion bias makes no measurable difference, the claim that the geometry modules drive heterogeneous accuracy would be falsified.","supporting_citations":[],"review_version":2}