REVIEW 2 major objections 3 minor 32 references
Image-to-Point Cloud Registration Made Easy with Rectified Flow-based LiDAR Upsampling
T0 review · 2 major / 3 minor · reviewed 2026-08-02 · deepseek-v4-flash
Pith's one-line read The paper argues that image-to-point-cloud registration can be reduced to image-to-image matching: generate a dense intensity image from a single sparse LiDAR scan with a rectified flow model, match it to the camera image with an off-the-sh
desk verdict A genuinely new I2P recipe — generate a dense intensity image from a sparse scan and let off-the-shelf matchers do the rest — but the depth-lookup for PnP is under-validated and the headline numbers don't reconcile cleanly with the per-sequence tables. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing component is Conditional Rectified Flow for LiDAR image upsampling. Rectified flow learns a velocity field that transports Gaussian noise to a target image along straight-line paths, so one Euler step already produces a usable dense image; the conditioning input is the sparse LiDAR intensity image. The model is pre-trained on an image-inpainting task with random point, line, and block masks (learning to fill in sparse patterns generically) and fine-tuned on dense intensity images from a few LiDAR models. The other half of the pipeline is standard: ALIKED keypoints with LightGlue matching on the generated image, followed by PnP-RANSAC with real LiDAR depth looked up within a
What would settle it
Run the pipeline on a dense-vegetation sequence with a 16-beam spinning LiDAR (large vertical scan gaps) and report the inlier ratio of PnP correspondences at the 10 px threshold. The paper's premise says appearance completion plus 3-pixel depth lookup keeps enough true inliers; if the mean inlier count falls below roughly 5 per frame or the 10°/5 m registration recall falls below 50%, the central claim of generalizing to unstructured, sparse-scan conditions is falsified.
Extended reading notes
Core claim
The discovery is that a rectified-flow generator, pre-trained on ordinary grayscale images via self-supervised inpainting and fine-tuned on a small collection of dense LiDAR intensity images, can synthesize enough of a LiDAR's reflected-intensity texture that a generic feature matcher finds reliable correspondences with a camera image. The authors deliberately complete only appearance, not depth: the 3D location of each matched keypoint is taken from the nearest real LiDAR point within a three-pixel radius. This works because sparse scans still constrain where structures are, and the generated texture fills in the missing surface appearance. The empirical result is a mean 4.89-degree/1.63-me
Load-bearing premise
The whole pose estimate rests on the assumption that each matched keypoint's true 3D location is well approximated by the nearest LiDAR point within a three-pixel search radius; when the generated intensity texture sells a keypoint that has no real depth nearby, PnP-RANSAC is fed biased 2D-3D pairs and accuracy collapses (the paper's own foliage failure case).
Editorial extensions
If this is right
- Registration no longer needs paired image-point cloud data or ground-truth sensor poses for training; only a small amount of dense LiDAR intensity images is required for fine-tuning.
- A new LiDAR model can be adapted by collecting a few dense intensity scans and fine-tuning, leaving the feature matcher untouched.
- One-step generation keeps the total per-frame cost near 0.68 s, making the pipeline plausible as a low-rate matching module alongside odometry.
- Because the pipeline is agnostic to the feature matcher, improvements in image matching should transfer directly to I2P accuracy.
- The same image-based formulation could be extended to omnidirectional projection models.
Reading between the lines
- This suggests a general recipe: any sensor that yields a sparse projected image (radar, sonar, thermal) might be registered to RGB images by learning to densify the intensity/reflectivity channel, reusing mature image matchers.
- The failure mode in vegetation points to a testable boundary: when the generated texture is driven by foliage rather than rigid surfaces, the depth-lookup assumption breaks; supplementing the pipeline with a learned depth prior or multi-frame accumulation would directly attack the reported weakness.
- Since 1-step and 10-step generation differ little in pose accuracy, the rectified flow is close to a deterministic mapping; distilling it into a single feed-forward network could remove the ODE integration entirely and push runtime below 0.1 s.
- The explicit choice to avoid depth completion leaves a clear counterfactual: if depth were also upsampled, keypoints in texture-rich but depth-sparse regions would stop being discarded, potentially improving recall in unstructured scenes.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a pipeline for image-to-point cloud registration that treats LiDAR as an imaging sensor: a single sparse LiDAR scan is projected into a sparse intensity image, upsampled to a dense intensity image by a Conditional Rectified Flow model, matched to a camera image using pre-trained ALIKED+LightGlue, and the 6-DoF relative pose is estimated via PnP-RANSAC. The upsampling model is pre-trained self-supervised on image inpainting and fine-tuned on small in-house LiDAR datasets, avoiding paired image-point cloud training data. Experiments on three R3LIVE sequences report a mean error of 4.89 deg / 1.63 m and about 0.68 s per frame for one generation step, and the paper claims the method outperforms prior I2P baselines.
Significance. If the results hold, the paper offers a conceptually simple and practical alternative to learning-based I2P methods: it avoids paired supervision and ground-truth sensor poses, reuses powerful off-the-shelf image matchers, and runs at a practical speed. The self-supervised pretraining strategy for adapting a generative model to LiDAR intensity images is a thoughtful contribution, and the ablations on generation steps and pretraining are informative. However, the central claim rests on an unvalidated depth-lookup step whose failure is visible in the paper's own experiments, and the 'outperforming existing methods' statement is stronger than the reported numbers support.
major comments (2)
- [III-D, Table VI, Section V-B] The depth-recovery premise for PnP is not validated. The method assigns each LiDAR keypoint the 3D position of the nearest LiDAR point within a 3-pixel radius because only intensity is upsampled. The paper never measures how accurately this approximation represents the true surface seen by the generated intensity texture. Table VI shows that RRE changes from 8.72 to 4.89 degrees across radii 0-5, indicating high sensitivity. Section V-B and Fig. 8 show that in hku park, unstable generated intensity in foliage leads to false matches and degraded pose, exactly the scenario where the depth lookup is most likely to return a point from a different surface. The authors should evaluate the geometric fidelity of the lifted 3D keypoints against the dense accumulated point cloud (e.g., report per-keypoint depth error), and discuss why the depth-radii choice is robust. Without such evidence, the co
- [Abstract, Section V-B, Table I] The abstract's claim that the method 'outperforms existing methods' is contradicted by the authors' own Table I on hku park. At the 45 deg / 10 m threshold, the proposed method achieves RR=91.36 while 2D3D-MATR achieves 96.57; at 10 deg / 5 m, the proposed method achieves RR=75.37 versus 77.92 for 2D3D-MATR. The method is better than the baselines on hkust campus and hku campus for most metrics, but it is not uniformly better. The claim should be qualified to the structured sequences or reported as an average, and the superiority on hku park should be withdrawn. This is a load-bearing point because the central contribution statement in the abstract and Section V must be accurate.
minor comments (3)
- [Table I] The 'none/none' threshold row is confusing: it reports RR=100.00 for all methods, which is tautological if no threshold is applied. Please clarify what this row represents, or remove it if it only means that all trials produce a finite estimate.
- [Table IV / Abstract] The mean RRE/RTE of 4.89 deg / 1.63 m appears to correspond to the 1-step row of Table IV, but it is not stated how this average is computed over the three sequences. Please specify whether it is a per-pair mean, per-sequence mean, or something else, and report standard deviations for the mean metrics in the abstract.
- [Section IV-C.2] The fine-tuning data description should explicitly state that no images or scans from the R3LIVE evaluation sequences were used in fine-tuning, to rule out data leakage. The current wording 'we collected dense LiDAR intensity images' is suggestive but not fully explicit.
Circularity Check
No equation-level circularity; the 6-DoF estimate is not a refit of its inputs, and the two self-citations are supporting rather than load-bearing.
full rationale
The derivation chain is self-contained. Conditional Rectified Flow is trained with the standard velocity loss L(θ)=E[||X_tgt−X_src−V_θ(X_t,C,t)||^2] on masked RGB images and dense LiDAR intensity images, and at inference the ODE is integrated from Gaussian noise conditioned on the sparse intensity image. The dense intensity image contributes only 2D keypoints; 3D points fed to PnP-RANSAC are taken from the original sparse LiDAR within a 3-pixel depth-search radius (Section III-D), so the estimated pose is not defined in terms of the target pose and is not fitted to R3LIVE ground truth. Evaluation uses external pre-trained matchers (ALIKED/LightGlue) and GLIM-derived poses as ground truth. The self-citations [19] (RGB-to-dense-LiDAR-intensity matching) and [32] (GLIM) are supporting: [19] is corroborated by the paper's own correspondence-quality experiments (Table II, Fig. 9), and [32] is an independent SLAM system applied uniformly to all methods. The depth-search radius is a hyperparameter tuned on the evaluation set, and the hku park failure (Section V-B, Fig. 8) exposes a real correctness risk, but these are not reductions of the prediction to its inputs by construction. Overall, no significant circularity.
Assumptions & free parameters
free parameters (5)
- depth search radius =
3 px
- feature matching threshold =
low (unspecified)
- generation steps =
1 (headline config)
- virtual camera projection parameters =
unspecified
- pretraining mask distributions =
randomly sampled (ranges unstated)
assumptions (5)
- domain assumption A single LiDAR scan projected to intensity, completed by conditioned rectified flow, yields an image whose texture aligns with the camera's grayscale view of the same geometry.
- domain assumption Off-the-shelf LightGlue/ALIKED (trained on RGB pairs) match camera grayscale images to generated LiDAR intensity images.
- ad hoc to paper A keypoint's 3D position is recoverable from the nearest LiDAR depth within a 3-pixel radius of its pixel in the upsampled image.
- standard math Rectified-flow ODE transport along straight-line interpolants is an adequate generative model for single-scan LiDAR intensity completion.
- domain assumption Image-inpainting pretraining on GSV-Cities transfers to LiDAR scan-pattern completion after small fine-tuning.
Cite this review
Pith. "Pith review of Image-to-Point Cloud Registration Made Easy with Rectified Flow-based LiDAR Upsampling." pith.science (2026). https://pith.science/paper/DAY2LW3S
@misc{pith2026260714639,
author = {Pith},
title = {Pith review of: Image-to-Point Cloud Registration Made Easy with Rectified Flow-based LiDAR Upsampling},
year = {2026},
howpublished = {\url{https://pith.science/paper/DAY2LW3S}},
note = {Machine review of arXiv:2607.14639}
}
read the original abstract
Image-to-Point Cloud Registration (I2P) is essential for integrating camera and LiDAR in perception and autonomous systems, yet the modality gap between images and point clouds makes it difficult to achieve both high accuracy and strong generalization. In this paper, we propose a simple yet effective I2P method that treats LiDAR as an imaging sensor: from a single sparse LiDAR scan, we generate a dense LiDAR intensity image using Conditional Rectified Flow, match it with a camera image using a pre-trained feature matcher, and estimate the 6-DoF relative pose via PnP-RANSAC. The proposed model is pre-trained through a self-supervised image completion task and fine-tuned on a small amount of LiDAR data (neither image-point cloud pairs nor ground-truth sensor poses are required), enabling it to scale to diverse LiDAR and camera configurations. Experiments on the R3LIVE dataset show that the proposed method achieves a mean error of 4.89{\deg} / 1.63 m, outperforming existing methods, while completing a single registration in approximately 0.68 s.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[1]
2d3d-matchnet: Learning to match keypoints across 2d image and 3d point cloud,
M. Feng, S. Hu, M. H. Ang, and G. H. Lee, “2d3d-matchnet: Learning to match keypoints across 2d image and 3d point cloud,” in2019 International Conference on Robotics and Automation. IEEE, 2019, pp. 4790–4796
2019
-
[2]
CorrI2P: Deep image-to-point cloud registration via dense correspondence,
S. Ren, Y . Zeng, J. Hou, and X. Chen, “CorrI2P: Deep image-to-point cloud registration via dense correspondence,”IEEE Transactions on Circuits and Systems for Video Technology, vol. 33, no. 3, pp. 1198– 1208, 2022. (a) 1 step (b) 5 steps (c) 10 steps Fig. 11. Generated dense LiDAR intensity images with (a) 1, (b) 5, and (c) 10 generation steps. TABLE V A...
2022
-
[3]
Cofii2p: Coarse-to-fine correspondences-based image to point cloud registration,
S. Kang, Y . Liao, J. Li, F. Liang, Y . Li, X. Zou, F. Li, X. Chen, Z. Dong, and B. Yang, “Cofii2p: Coarse-to-fine correspondences-based image to point cloud registration,”IEEE Robotics and Automation Letters, vol. 9, no. 11, pp. 10 264–10 271, 2024
2024
-
[4]
2d3d-matr: 2d-3d matching transformer for detection-free registration between im- ages and point clouds,
M. Li, Z. Qin, Z. Gao, R. Yi, C. Zhu, Y . Guo, and K. Xu, “2d3d-matr: 2d-3d matching transformer for detection-free registration between im- ages and point clouds,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 14 128–14 138
2023
-
[5]
DeepI2P: Image-to-point cloud registration via deep classification,
J. Li and G. H. Lee, “DeepI2P: Image-to-point cloud registration via deep classification,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 15 960–15 969
2021
-
[6]
LCCNet: LiDAR and camera self-calibration using cost volume network,
X. Lv, B. Wang, Z. Dou, D. Ye, and S. Wang, “LCCNet: LiDAR and camera self-calibration using cost volume network,” inProceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 2894–2901
2021
-
[7]
H. Wang, Y . Liu, B. Wang, Y . Sun, Z. Dong, W. Wang, and B. Yang, “Freereg: Image-to-point cloud registration leveraging pre- trained diffusion models and monocular depth estimators,”arXiv preprint arXiv:2310.03420, 2023
arXiv 2023
-
[8]
Su- perglue: Learning feature matching with graph neural networks,
P.-E. Sarlin, D. DeTone, T. Malisiewicz, and A. Rabinovich, “Su- perglue: Learning feature matching with graph neural networks,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 4938–4947
2020
Show all 32 references
-
[9]
LoFTR: Detector- free local feature matching with transformers,
J. Sun, Z. Shen, Y . Wang, H. Bao, and X. Zhou, “LoFTR: Detector- free local feature matching with transformers,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 8922–8931
2021
-
[10]
Lightglue: Local feature matching at light speed,
P. Lindenberger, P.-E. Sarlin, and M. Pollefeys, “Lightglue: Local feature matching at light speed,” inProceedings of the IEEE/CVF international conference on computer vision, 2023, pp. 17 627–17 638
2023
-
[11]
Xoftr: Cross-modal feature matching transformer,
¨O. Tuzcuo ˘glu, A. K ¨oksal, B. Sofu, S. Kalkan, and A. A. Alatan, “Xoftr: Cross-modal feature matching transformer,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2024, pp. 4275–4286
2024
-
[12]
Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow,
X. Liu, C. Gong, and Q. Liu, “Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow,” inInternational Conference on Learning Representations, 2023
2023
-
[13]
Denoising diffusion probabilistic models,
J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,”Advances in neural information processing systems, vol. 33, pp. 6840–6851, 2020
2020
-
[14]
Denoising diffusion implicit models,
J. Song, C. Meng, and S. Ermon, “Denoising diffusion implicit models,”arXiv preprint arXiv:2010.02502, 2020
2010 arXiv
-
[15]
Distinctive image features from scale-invariant key- points,
D. G. Lowe, “Distinctive image features from scale-invariant key- points,”International journal of computer vision, vol. 60, no. 2, pp. 91–110, 2004
2004
-
[16]
Surf: Speeded up robust features,
H. Bay, T. Tuytelaars, and L. Van Gool, “Surf: Speeded up robust features,” inEuropean conference on computer vision. Springer, 2006, pp. 404–417
2006
-
[17]
ORB: An effi- cient alternative to SIFT or SURF,
E. Rublee, V . Rabaud, K. Konolige, and G. Bradski, “ORB: An effi- cient alternative to SIFT or SURF,” in2011 International conference on computer vision. IEEE, 2011, pp. 2564–2571. TABLE VI EFFECT OF DEPTH SEARCH RADIUS(DSR). DSR (pixels) RRE ( ◦)↓RTE (m)↓IR (%)↑ 0 8.72 2.296...
2011
-
[18]
ORB-SLAM: A versatile and accurate monocular SLAM system,
R. Mur-Artal, J. M. M. Montiel, and J. D. Tardos, “ORB-SLAM: A versatile and accurate monocular SLAM system,”IEEE transactions on robotics, vol. 31, no. 5, pp. 1147–1163, 2015
2015
-
[19]
General, single-shot, target-less, and automatic lidar-camera extrinsic calibration toolbox,
K. Koide, S. Oishi, M. Yokozuka, and A. Banno, “General, single-shot, target-less, and automatic lidar-camera extrinsic calibration toolbox,” arXiv preprint arXiv:2302.05094, 2023
2023 arXiv
-
[20]
Dust3r: Geometric 3d vision made easy,
S. Wang, V . Leroy, Y . Cabon, B. Chidlovskii, and J. Revaud, “Dust3r: Geometric 3d vision made easy,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2024, pp. 20 697–20 709
2024
-
[21]
Vggt: Visual geometry grounded transformer,
J. Wang, M. Chen, N. Karaev, A. Vedaldi, C. Rupprecht, and D. Novotny, “Vggt: Visual geometry grounded transformer,” inPro- ceedings of the Computer Vision and Pattern Recognition Conference, 2025, pp. 5294–5306
2025
-
[22]
Mapany- thing: Universal feed-forward metric 3d reconstruction,
N. Keetha, N. M ¨uller, J. Sch ¨onberger, L. Porzi, Y . Zhang, T. Fischer, A. Knapitsch, D. Zauss, E. Weber, N. Antuneset al., “Mapany- thing: Universal feed-forward metric 3d reconstruction,”arXiv preprint arXiv:2509.13414, 2025
2025 arXiv
-
[23]
Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography,
M. A. Fischler and R. C. Bolles, “Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography,”Communications of the ACM, vol. 24, no. 6, pp. 381–395, 1981
1981
-
[24]
EP n P: An accurate O (n) solution to the P n P problem,
V . Lepetit, F. Moreno-Noguer, and P. Fua, “EP n P: An accurate O (n) solution to the P n P problem,”International journal of computer vision, vol. 81, no. 2, pp. 155–166, 2009
2009
-
[25]
Lidar data synthesis with denoising diffusion probabilistic models,
K. Nakashima and R. Kurazume, “Lidar data synthesis with denoising diffusion probabilistic models,” in2024 IEEE International Confer- ence on Robotics and Automation. IEEE, 2024, pp. 14 724–14 731
2024
-
[26]
Fast LiDAR data generation with rectified flows,
K. Nakashima, X. Liu, T. Miyawaki, Y . Iwashita, and R. Kurazume, “Fast LiDAR data generation with rectified flows,” in2025 IEEE International Conference on Robotics and Automation. IEEE, 2025, pp. 10 057–10 063
2025
-
[27]
Aliked: A lighter keypoint and descriptor extraction network via deformable transformation,
X. Zhao, X. Wu, W. Chen, P. C. Chen, Q. Xu, and Z. Li, “Aliked: A lighter keypoint and descriptor extraction network via deformable transformation,”IEEE Transactions on Instrumentation and Measure- ment, vol. 72, pp. 1–16, 2023
2023
-
[28]
Image super-resolution via iterative refinement,
C. Saharia, J. Ho, W. Chan, T. Salimans, D. J. Fleet, and M. Norouzi, “Image super-resolution via iterative refinement,”IEEE transactions on pattern analysis and machine intelligence, vol. 45, no. 4, pp. 4713– 4726, 2022
2022
-
[29]
U-net: Convolutional networks for biomedical image segmentation,
O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” inInternational Confer- ence on Medical image computing and computer-assisted intervention. Springer, 2015, pp. 234–241
2015
-
[30]
Gsv-cities: Toward appropriate supervised visual place recognition,
A. Ali-Bey, B. Chaib-Draa, and P. Giguere, “Gsv-cities: Toward appropriate supervised visual place recognition,”Neurocomputing, vol. 513, pp. 194–203, 2022
2022
-
[31]
CT- ICP: Real-time elastic LiDAR odometry with loop closure,
P. Dellenbach, J.-E. Deschaud, B. Jacquet, and F. Goulette, “CT- ICP: Real-time elastic LiDAR odometry with loop closure,” in2022 International Conference on Robotics and Automation. IEEE, 2022, pp. 5580–5586
2022
-
[32]
GLIM: 3D range- inertial localization and mapping with GPU-accelerated scan matching factors,
K. Koide, M. Yokozuka, S. Oishi, and A. Banno, “GLIM: 3D range- inertial localization and mapping with GPU-accelerated scan matching factors,”Robotics and Autonomous Systems, vol. 179, p. 104750, 2024
2024
Reviewed August 2, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.