Fisheye3R: Adapting Unified 3D Feed-Forward Foundation Models to Fisheye Lenses

Alex Wong; Dongxu Zhao; Eric Turner; Erin Hong; Ruxiao Duan; Yunwen Zhou

REVIEW 3 major objections 2 minor 1 cited by

Fisheye3R adapts multi-view 3D foundation models to fisheye inputs without fisheye ground truth and without hurting perspective performance.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-13 16:04 UTC pith:67VI44YE

load-bearing objection Abstract promises a useful fisheye adaptation for 3D foundation models without GT or perspective regression, but the supplied full text is a different paper, so the claims cannot be audited. the 3 major comments →

arxiv 2603.28896 v2 pith:67VI44YE submitted 2026-03-30 cs.CV

Fisheye3R: Adapting Unified 3D Feed-Forward Foundation Models to Fisheye Lenses

Ruxiao Duan , Erin Hong , Dongxu Zhao , Eric Turner , Alex Wong , Yunwen Zhou This is my paper

classification cs.CV

keywords fisheye camerasmulti-view 3D reconstructionfeed-forward foundation modelsradial distortionmodel adaptationself-supervised learningcamera pose estimationdepth estimation

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Feed-forward multi-view 3D reconstruction models trained on ordinary perspective photos fail when the camera uses a fisheye lens, because the nonlinear projection rearranges pixels in ways those models never saw. Collecting enough labeled fisheye data to retrain them from scratch is impractical. Fisheye3R is an adaptation framework that teaches the same models to accept high radial distortion natively, while preserving their original accuracy on perspective images. It does this with flexible training recipes: one that needs only unlabeled perspective images (self-supervised) and another that needs no fisheye training data at all (supervised). Across three existing foundation models the adapted systems improve camera pose, depth, point maps, and field-of-view estimates on fisheye imagery, giving a practical path to wide-angle 3D reconstruction without a large new labeled corpus.

Core claim

The authors claim that the performance drop of perspective-trained multi-view 3D foundation models on fisheye images is driven mainly by the changed spatial arrangement of pixels under nonlinear projection, and that this can be corrected by an adaptation framework (Fisheye3R) that never requires real fisheye images with ground truth—using either self-supervision on unlabeled perspective data or supervised adaptation without fisheye training data—while leaving perspective performance intact and improving pose, depth, point-map, and field-of-view estimates on fisheye inputs.

What carries the argument

Fisheye3R: a modular adaptation framework plus flexible learning schemes (self-supervised on unlabeled perspective images only, and supervised without any fisheye training data) that retarget existing multi-view 3D foundation models so they accept high radial distortion natively.

Load-bearing premise

The main reason perspective-trained 3D models fail on fisheye is the nonlinear remapping of pixels, and that gap can be closed by adaptation that never trains on real fisheye images with ground truth.

What would settle it

Take the same three foundation models, apply Fisheye3R, and evaluate on held-out real fisheye multi-view sequences with accurate pose and depth ground truth: if camera pose, depth, point-map, and FoV errors do not improve over the unadapted baselines (or if perspective performance drops), the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Existing perspective-trained multi-view 3D models can be deployed on fisheye cameras without collecting large labeled fisheye datasets.
Self-supervised adaptation from unlabeled perspective images alone is enough to recover useful fisheye performance.
Camera pose, depth, point maps, and field-of-view estimates on wide-FOV imagery improve consistently across multiple foundation architectures.
Production pipelines can keep one adapted model that handles both perspective and high-distortion inputs without separate branches.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same pixel-arrangement adaptation idea may transfer to other non-pinhole projections (catadioptric, ultra-wide rectilinear) if the dominant error remains geometric rather than photometric.
If residual failures appear under extreme distortion, they likely come from multi-view consistency or calibration error rather than pure spatial layout, suggesting a next diagnostic split of error sources.
Pairing Fisheye3R with lightweight online calibration could make consumer AR/VR headsets with fisheye optics reuse existing 3D foundation weights with minimal extra data.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Abstract promises a useful fisheye adaptation for 3D foundation models without GT or perspective regression, but the supplied full text is a different paper, so the claims cannot be audited.

read the letter

The one thing you need to know: we only have a clean abstract for Fisheye3R. The body that came with the cache is an unrelated MPC trade-execution paper (2603.28898). So this is not a verdict on the method; it is a note on what the abstract actually claims and what is missing.

What is new on the face of it is practical and well-scoped. Feed-forward multi-view 3D models (VGGT, π³, MapAnything) trained on perspective data degrade under strong radial distortion. Fisheye3R is framed as an adaptation layer that lets those models take fisheye natively, with two learning schemes that avoid the usual data bottleneck: self-supervised adaptation from unlabeled perspective images only, and supervised adaptation that does not require fisheye training data. The abstract also claims no regression on perspective inputs and consistent gains on pose, depth, point maps, and FoV. Code is promised. That combination—if real—is exactly what robotics and XR people need when they already have a strong perspective foundation model and a fisheye sensor.

The soft spot is not a flaw in the argument; it is evidentiary absence. We cannot check the projection handling, the synthetic or pseudo-label protocol, the non-regression numbers on perspective, ablations, or whether residual photometry/calibration gaps dominate. The abstract’s load-bearing assumption—that the main failure is nonlinear pixel rearrangement and that it can be fixed without real fisheye GT—is plausible but untested here. Circularity risk (train/test contamination, synthetic fisheye that is too easy) is the usual one for this genre and cannot be ruled out or confirmed.

Who it is for: people shipping multi-view 3D on wide-FoV cameras who want to keep existing foundation models. A serious editor should send the real manuscript to referees once the correct PDF is attached; the problem is real, the framing is honest, and three-model evaluation plus open code is enough to deserve referee time. Until then I would not cite it or put it in reading group. Get the actual paper and re-read.

Referee Report

3 major / 2 minor

Summary. Based solely on the abstract of arXiv:2603.28896, the paper claims that feed-forward multi-view 3D foundation models trained on perspective imagery degrade on fisheye inputs because nonlinear projection rearranges pixel geometry. It proposes Fisheye3R, an adaptation framework that natively handles high radial distortion without regressing on perspective images, using flexible schemes that allow self-supervised adaptation from unlabeled perspective data and supervised adaptation without any fisheye training data. Experiments across VGGT, π³, and MapAnything are said to improve camera pose, depth, point maps, and field-of-view estimation on fisheye imagery, with code released. The body text supplied under this review, however, is an unrelated manuscript on model predictive control for trade execution (arXiv:2603.28898), so the method, equations, data protocol, and results of Fisheye3R cannot be audited from the materials provided.

Significance. If the abstract claims hold under proper evaluation, the work would be practically significant for robotics, AR/XR, and autonomous systems that rely on wide-FoV cameras: it would let existing large-scale 3D foundation models be reused on fisheye inputs without collecting scarce fisheye ground truth, while preserving perspective performance. The dual self-supervised / no-fisheye-GT supervised adaptation design and multi-model evaluation would be a useful systems contribution. Those strengths cannot be credited or stress-tested here because the full Fisheye3R manuscript is not available in this review package.

major comments (3)

Manuscript identity mismatch: the CACHEABLE full text is “Model Predictive Control For Trade Execution” (arXiv:2603.28898, q-fin.TR), not Fisheye3R (arXiv:2603.28896, cs.CV). Sections, equations, figures, tables, and experimental protocols for Fisheye3R are absent. No load-bearing claim about adaptation, non-regression on perspective images, or gains on pose/depth/point-map/FoV can be verified. This is a blocking evidentiary failure for peer review.
Abstract-only central claim cannot be checked: the abstract asserts that nonlinear projection-induced pixel rearrangement is the dominant failure mode and that self-supervised adaptation on unlabeled perspective images (and supervised adaptation without fisheye training data) suffice to fix it without real fisheye GT. Without the method body, synthetic-data protocol, ablations, baselines, error bars, or train/test leakage controls, this assumption remains untested. A correct resubmission must supply those sections so the claim can be audited.
Non-regression and multi-model consistency are load-bearing but unshown: the abstract promises no performance regression on perspective images and consistent gains across VGGT, π³, and MapAnything on four metrics. Tables and held-out protocols are not present in the supplied text, so the central empirical contribution cannot be assessed.

minor comments (2)

Only the Fisheye3R abstract is available; figure/table numbering, notation, and related-work placement cannot be reviewed until the correct PDF is provided.
Code URL is given in the abstract (https://github.com/android-xr/fisheye3r); once the correct manuscript is attached, reproducibility of the adaptation schemes should be checked against that release.

Circularity Check

0 steps flagged

No significant circularity: Fisheye3R is an empirical adaptation paper; available abstract states no derivation that reduces by construction to its inputs.

full rationale

Only the Fisheye3R abstract is available for arXiv:2603.28896; the cached full manuscript text is a different paper (MPC for trade execution, arXiv:2603.28898) and cannot be used as the derivation chain. From the abstract alone, the work proposes an adaptation framework (self-supervised on unlabeled perspective images; supervised without fisheye training data) and reports empirical gains on pose, depth, point map, and FoV for VGGT, π³, and MapAnything. There is no claimed first-principles derivation, uniqueness theorem, fitted constant renamed as prediction, or self-definitional identity. Residual risks (e.g., synthetic fisheye protocol, train/test contamination) are methodological and not circularity under the stated criteria. Score 0 with empty steps is the honest finding.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 1 invented entities

Abstract-only review. Load-bearing premises are domain assumptions about why fisheye fails and that adaptation without real fisheye GT is sufficient. No free parameters or invented physical entities are extractable from the abstract; method-specific losses, schedules, and distortion models would appear in the missing body.

axioms (3)

domain assumption Performance degradation of perspective-trained multi-view 3D foundation models on fisheye is primarily caused by nonlinear projection changing spatial pixel arrangements.
Stated as the causal diagnosis in the abstract; if other factors dominate, the adaptation target may be wrong.
ad hoc to paper Self-supervised adaptation on unlabeled perspective images (and supervised adaptation without fisheye training data) can transfer enough geometry to improve fisheye inference.
Core methodological premise of Fisheye3R’s flexible learning schemes; not a standard theorem.
domain assumption Native fisheye accommodation can be achieved without performance regression on perspective images.
Claimed design constraint; assumes multi-domain capacity of the adapted models.

invented entities (1)

Fisheye3R adaptation framework no independent evidence
purpose: Extend existing multi-view 3D foundation models to accept high radial-distortion fisheye inputs while preserving perspective performance.
Named contribution of the paper; independent evidence would be public code plus third-party replications on held-out fisheye benchmarks.

pith-pipeline@v1.1.0-grok45 · 7568 in / 2666 out tokens · 25342 ms · 2026-07-13T16:04:06.990496+00:00 · methodology

0 comments

read the original abstract

Feed-forward foundation models for multi-view 3-dimensional (3D) reconstruction have been trained on large-scale datasets of perspective images; when tested on wide field-of-view images, e.g., from a fisheye camera, their performance degrades. This degradation arises from changes in spatial arrangements of pixels induced by the non-linear projection model that maps 3D points onto the 2D image plane. While one may surmise that training on fisheye images would resolve this problem, there are far fewer fisheye images with ground truth than perspective images, which limits generalization. To enable inference on imagery exhibiting high radial distortion, we propose \textit{Fisheye3R}, a novel adaptation framework that extends these multi-view 3D reconstruction foundation models to natively accommodate fisheye inputs without performance regression on perspective images. To address the scarcity of fisheye images and ground truth, we introduce flexible learning schemes that support self-supervised adaptation using only unlabeled perspective images and supervised adaptation without any fisheye training data. Extensive experiments across three foundation models, including VGGT, $\pi^3$, and MapAnything, demonstrate that our approach consistently improves camera pose, depth, point map, and field-of-view estimation on fisheye images. Code is available at https://github.com/android-xr/fisheye3r.

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

RayTun3R: Online Camera Adaptation in 3D Foundation Models
cs.CV 2026-07 accept novelty 6.5

A 10.7k-parameter residual PE/RoPE adapter, fit online on a short fisheye clip, recovers 2–12× better rotation from frozen pinhole 3D foundation models while keeping single-pass inference.