REVIEW 4 cited by
Improving 2D Feature Representations by 3D-Aware Fine-Tuning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Current visual foundation models are trained purely on unstructured 2D data, limiting their understanding of 3D structure of objects and scenes. In this work, we show that fine-tuning on 3D-aware data improves the quality of emerging semantic features. We design a method to lift semantic 2D features into an efficient 3D Gaussian representation, which allows us to re-render them for arbitrary views. Using the rendered 3D-aware features, we design a fine-tuning strategy to transfer such 3D awareness into a 2D foundation model. We demonstrate that models fine-tuned in that way produce features that readily improve downstream task performance in semantic segmentation and depth estimation through simple linear probing. Notably, though fined-tuned on a single indoor dataset, the improvement is transferable to a variety of indoor datasets and out-of-domain datasets. We hope our study encourages the community to consider injecting 3D awareness when training 2D foundation models. Project page: https://ywyue.github.io/FiT3D.
Forward citations
Cited by 4 Pith papers
-
JOG3R: Towards 3D-Consistent Video Generators
Jointly training a video diffusion model with a 3D point map reconstruction head improves the 3D consistency of generated videos and yields usable camera pose estimates on static scenes.
-
LoRA3D: Low-Rank Self-Calibration of 3D Geometric Foundation Models
LoRA3D specializes pretrained 3D foundation models to target scenes via confidence-calibrated pseudo-labels from multi-view robust optimization and LoRA fine-tuning, improving performance by up to 88%.
-
Multiview Equivariance Improves 3D Correspondence Understanding with Minimal Feature Finetuning
Finetuning ViT features with SmoothAP on Objaverse multiview correspondences improves 3D correspondence tasks, with meaningful gains even from a single object and a single iteration.
-
Maybe you don't need a U-Net: convolutional feature upsampling for materials micrograph segmentation
A lightweight CNN upsampler, distilled from FeatUp features, makes frozen DINOv2 patch features sharp enough for interactive segmentation of micrographs with sparse labels, and its workflow beats fine-tuning a U-Net i...
Discussion (0). Continue with ORCID to comment.