REVIEW 3 major objections 5 minor 1 cited by
Trajectory-based Road Autolabeling with Lidar-Camera Fusion in Winter Conditions
T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read Roads can be autolabeled without manual annotations by fusing lidar trajectory points with camera image features, and this fusion outperforms both single-sensor trajectory methods and a supervised model on winter roads.
desk verdict First credible lidar-camera fusion for trajectory-based road autolabeling, with reproducible code/data and a solid winter-dataset eval; main caveat is manual lidar-camera calibration and single-dataset evaluation. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the wheel trajectory anchored in each lidar scan ring. Future vehicle poses are matched to lidar points on each ring, left and right wheel positions are estimated from heading and track width, and those points are projected into the image to define trajectory pixels. The lidar autolabel combines an exponential height label with a sum of thresholded upward gradients relative to the wheel reference; the camera autolabel uses DINOv2 patch-feature cosine similarity to a trajectory prototype. The fused label is the mean of the two continuous labels, refined to a discrete mask by a fully connected CRF.
What would settle it
Take the released dataset, perturb the manually refined extrinsic rotation by one to three degrees around pitch and yaw, regenerate autolabels, and retrain the image-only model; if IoU drops sharply or the trajectory pixels visibly leave the road, the result depends on calibration tuning that the paper does not benchmark.
Extended reading notes
Core claim
On its own terms the paper discovers that lidar and camera give complementary trajectory-based road labels, and that their simple mean is more accurate than either alone. The lidar stream labels points by height and by cumulative thresholded gradients relative to the vehicle's wheel trajectory in each scan ring; the camera stream labels patches by cosine similarity to a DINOv2 road prototype computed from projected trajectory pixels. Averaging the two streams and applying CRF post-processing yields 90.2 IoU autolabels, which transfer to an image-only prediction model that scores 90.9 IoU on the winter test set, exceeding the supervised baseline's 89.7 IoU.
Load-bearing premise
The load-bearing assumption is that the manually refined lidar-camera rotation is accurate enough that trajectory pixels and wheel reference points land on the road, since any systematic misalignment would corrupt both autolabel streams before training.
Editorial extensions
If this is right
- An image-only road segmenter can be trained from unlabeled winter driving data to 90.9 IoU, outperforming the same model trained on 200 manually labeled images.
- Because the trained predictor runs in under 3 ms per frame, the autolabeling approach can feed real-time deployment rather than just offline dataset creation.
- The fusion degrades gracefully: cases where lidar sees vibration noise as high gradient are corrected by the camera stream, and cases where the camera stream overextends onto visually similar road edges are corrected by the lidar stream.
- The public dataset and source code let other groups regenerate autolabels and train their own segmenters without hand labeling.
Reading between the lines
- Beyond the paper, the same two-stream fusion could be run online: the camera road prototype could be updated from the vehicle's current trajectory, letting the model re-autolabel as conditions change rather than only at dataset build time.
- The manual calibration step is the most fragile point, so a natural extension is an automatic extrinsic calibration check; if the method is robust to small rotation errors, deployment is much simpler than the paper's procedure suggests.
- The paper mentions accumulating consecutive scans as future work; doing so could fix the oncoming-lane failure case by giving the lidar stream multiple looks at a second road segment separated by a snowbank.
- The 90.9-versus-89.7 margin over the supervised baseline is measured on one winter dataset; testing on other weathers and terrains would show whether fusion's advantage is specific to snow-covered roads with low visual contrast.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a trajectory-based road-autolabeling method that fuses lidar and camera data without manual labels. From GNSS/INS poses, the method finds the vehicle center and wheel positions on each lidar scan ring (Section 3.1), derives a lidar autolabel from height and gradient features referenced to the trajectory points (Section 3.2), and derives a camera autolabel from DINOv2 patch similarity to the mean trajectory prototype (Section 3.3). The two continuous labels are averaged and refined with CRF post-processing. A Deeplabv3 model trained on the resulting autolabels is evaluated on a new 800-image winter test set covering suburban and countryside scenes. The final autolabel reaches 90.2 IoU (Table 1), and the trained image-only model reaches 90.9 IoU (Table 2), outperforming a supervised baseline trained on 200 manual labels (89.7 IoU) and clearly exceeding Seo et al. (75.2), SAM2 (67.3), and lidar boundary detection (70.2) on the same dataset.
Significance. If the results are reproducible, the contribution is practically valuable: the method generates large amounts of road labels in a difficult domain (snowy winter roads) without manual labeling, and the code and dataset are released publicly. The evaluation protocol is clean in that the manually labeled test set is never used to set autolabel parameters or to select the prediction model, and the ablation in Table 1 isolates the contribution of each autolabel component. The trained prediction model is image-only and runs in under 3 ms, so the method is compatible with real-time deployment. The load-bearing weaknesses are the unquantified manual extrinsic calibration, on which both autolabel streams depend (Section 3.4), and the absence of error bars on the single test evaluation, which matters because the margin over the supervised baseline is only 1.2 IoU. These issues are fixable with a sensitivity analysis and repeated runs, but they must be addressed before the clear-margin claim is fully supported.
major comments (3)
- [§3.4, Data] Section 3.4 states that the lidar-camera extrinsic rotation was manually refined to optimize alignment in the validation set. Because the projected wheel points define both the lidar reference points used in Eqs. (1)-(4) and the camera trajectory pixels used to build the road prototype in Eqs. (5)-(6), a small rotational error systematically corrupts both autolabeling streams simultaneously; for example, a one-degree rotation at typical mid-range depths shifts the projected polygon by tens of pixels. Since the refinement targets the validation set, the validation IoU used to select the sigma parameters is optimistically biased, and the test set, recorded with the same rig on the same road types, does not independently certify the calibration. I therefore ask for (i) an automatic or target-based calibration check with reported reprojection error, and (ii) a sensitivity study that perturbs the extrinsic rotation by, e.g., plus or minus 0.25, 0.5, and 1 degree, recomputes both autolabels, and reports the resulting ablation IoU of Table 1 and the test IoU of Table 2.
- [§4, Table 2] Table 2 reports a single evaluation run on 800 test images with no error bars or repeated training runs. The claimed margin over the supervised baseline is 1.2 IoU overall (90.9 vs 89.7) and only 0.2 IoU on the suburb scene (90.7 vs 90.5), a difference that could plausibly arise from sampling noise given that the hyperparameters sigma_C, sigma_H, and sigma_G were selected on the 200-image validation set. I request paired bootstrap confidence intervals or results over several training seeds for OURS and Supervised, and I recommend that the abstract and conclusions temper the clear-margin phrasing when it refers to the supervised baseline; the large margins over the other baselines are unaffected by this concern.
- [§4, Table 2 and Section 2.5] The paper's central claim is that lidar-camera fusion improves over camera-only trajectory-based learning, but Table 2 does not include a camera-only trajectory-based model trained and evaluated on this dataset. The closest such method, TADAP [11], is the authors' own prior work and its DINOv2 prototype mechanism is exactly the camera stream of the proposed method. Since the ablation in Table 1 measures autolabel quality rather than trained prediction quality, I ask for an additional trained-model comparison: a Deeplabv3 model trained on the camera-only autolabels (or a direct TADAP evaluation) alongside OURS, so that the prediction-level gain can be attributed to the fusion rather than to the shared camera stream.
minor comments (5)
- [Abstract and Section 1] There are a few orthographic errors, most notably publically available in the Abstract and Section 1, which should be publicly available; a copyedit should catch the remaining typos.
- [§4, Table 1] The paragraph introducing Table 1 does not state on which split the ablation IoUs are computed; please clarify whether Table 1 reports validation or test IoU and how the CRF parameters were set.
- [§3.1] The filter saying that the distance between consecutive center points must be more than 1 m is ambiguous; clarify whether this is a minimum or maximum spacing threshold and how it interacts with the 5 m sampling of the training set.
- [§3.5] The description of the Supervised baseline as trained on 200 manually labeled validation images via cross-validation is underspecified; state the number of folds and the model-selection procedure used.
- [Section 2.5 and contribution list] The claim that the method is the only trajectory-based method with public source code is difficult to verify and could be rephrased as a statement about the authors' release rather than a literature-level claim.
Circularity Check
No significant circularity: the autolabel pipeline is trajectory-driven and manual labels are used only for evaluation and validation-based parameter selection.
full rationale
The derivation chain is: recorded vehicle poses define trajectory points in the lidar scan (Section 3.1); these points seed both the lidar-based height/gradient labels (Section 3.2) and the camera-based DINOv2 similarity labels via the trajectory-pixel polygon (Section 3.3); the fused autolabels train a Deeplabv3 model, which is then evaluated on a disjoint manually labeled test set (Section 3.4-4). Each stage uses independent inputs. The manual labels are never used to generate autolabels or to train the prediction model; they are used only for validation-based selection of the sigma hyperparameters and for final test evaluation, which is standard model selection, not circularity. The camera prototype is defined as the mean of trajectory-pixel features and the label is a cosine-similarity to that prototype; this is the intended operational definition of trajectory-based labeling rather than a prediction that reduces to its input, and the resulting labels are independently checked against manual annotations in Table 1. The lidar autolabels similarly use height and gradient relative to trajectory reference points, which is a heuristic from the traversed path, not from the manual ground truth. The only self-citation, [11] for the DINOv2-based camera component, is corroborated by the paper's own ablation (camera-only IoU of 83.5 in Table 1), so it is not load-bearing. The manual extrinsic refinement in Section 3.4 is a legitimate sensitivity concern, but it is a fixed sensor calibration rather than a fitted parameter that predicts the test labels, and the test set is disjoint, so it does not constitute circularity.
Assumptions & free parameters
free parameters (6)
- sigma_C (camera similarity sensitivity) =
0.6
- sigma_H (height label sensitivity) =
0.1
- sigma_G (gradient label sensitivity) =
0.02
- Trajectory scan matching thresholds =
1 m / 1 m / 1 m / 2 m / 10 px
- Height-label radial distance cutoff =
5 m
- DINOv2 trajectory patch threshold =
200 patches
assumptions (4)
- domain assumption The area between the vehicle's left and right wheel tracks, derived from GNSS/INS poses, is positive road label.
- domain assumption The road surface is lower than its surroundings and road boundaries produce upward height gradients.
- domain assumption Lidar scans are not badly degraded, for example by a water layer on asphalt.
- domain assumption DINOv2 features encode visual similarity that separates road from background in winter scenes.
Cite this review
Pith. "Pith review of Trajectory-based Road Autolabeling with Lidar-Camera Fusion in Winter Conditions." pith.science (2026). https://pith.science/paper/J7WTEZZB
@misc{pith2026241202370,
author = {Pith},
title = {Pith review of: Trajectory-based Road Autolabeling with Lidar-Camera Fusion in Winter Conditions},
year = {2026},
howpublished = {\url{https://pith.science/paper/J7WTEZZB}},
note = {Machine review of arXiv:2412.02370}
}
read the original abstract
Robust road segmentation in all road conditions is required for safe autonomous driving and advanced driver assistance systems. Supervised deep learning methods provide accurate road segmentation in the domain of their training data but cannot be trusted in out-of-distribution scenarios. Including the whole distribution in the trainset is challenging as each sample must be labeled by hand. Trajectory-based self-supervised methods offer a potential solution as they can learn from the traversed route without manual labels. However, existing trajectory-based methods use learning schemes that rely only on the camera or only on the lidar. In this paper, trajectory-based learning is implemented jointly with lidar and camera for increased performance. Our method outperforms recent standalone camera- and lidar-based methods when evaluated with a challenging winter driving dataset including countryside and suburb driving scenes. The source code is available at https://github.com/eerik98/lidar-camera-road-autolabeling.git
Figures
Figures from the paper (4 more)
Forward citations
Cited by 1 Pith paper
-
Label Correction for Road Segmentation Using Road-side Cameras
A single manual road label per roadside camera is transferred to thousands of winter frames via Fourier-Mellin registration, and models trained on this data segment roads better on roadside and dashcam views.
Reference graph
Works this paper leans on
-
[11]
Tadap: Trajectory-aided drivable area auto-labeling with pretrained self- supervised features in winter driving conditions,
E. Alamikkotervo, R. Ojala, A. Sepp ¨anen, and K. Tammi, “Tadap: Trajectory-aided drivable area auto-labeling with pretrained self- supervised features in winter driving conditions,” IEEE Transactions on Intelligent V ehicles, 2024
2024
-
[1]
Vision meets robotics: The kitti dataset,
A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, “Vision meets robotics: The kitti dataset,” The International Journal of Robotics Research , vol. 32, no. 11, pp. 1231–1237, 2013
2013
-
[2]
nuscenes: A multimodal dataset for autonomous driving,
H. Caesar, V . Bankiti, A. H. Lang, S. V ora, V . E. Liong, Q. Xu, A. Krish- nan, Y . Pan, G. Baldan, and O. Beijbom, “nuscenes: A multimodal dataset for autonomous driving,” in Proceedings of the IEEE /CVF conference on computer vision and pattern recognition , pp. 11621–11631, 2020
work page 2020
-
[3]
Bdd100k: A diverse driving dataset for heterogeneous multi- task learning,
F. Yu, H. Chen, X. Wang, W. Xian, Y . Chen, F. Liu, V . Madhavan, and T. Darrell, “Bdd100k: A diverse driving dataset for heterogeneous multi- task learning,” in Proceedings of the IEEE /CVF conference on computer vision and pattern recognition, pp. 2636–2645, 2020
work page 2020
-
[4]
The cityscapes dataset for semantic urban scene understanding,
M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, “The cityscapes dataset for semantic urban scene understanding,” in Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 3213–3223, 2016
work page 2016
-
[5]
F. A. Shaik, A. Reddy, N. R. Billa, K. Chaudhary, S. Manchanda, and G. Varma, “Idd-aw: A benchmark for safe and robust segmentation of drive scenes in unstructured traffic and adverse weather,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 4614–4623, 2024
work page 2024
-
[6]
Acdc: The adverse conditions dataset with correspondences for semantic driving scene understanding,
C. Sakaridis, D. Dai, and L. Van Gool, “Acdc: The adverse conditions dataset with correspondences for semantic driving scene understanding,” in Proceedings of the IEEE /CVF International Conference on Computer Vision, pp. 10765–10775, 2021
work page 2021
-
[7]
Self-supervised traversability prediction by learning to reconstruct safe terrain,
R. Schmid, D. Atha, F. Sch ¨oller, S. Dey, S. Fakoorian, K. Otsu, B. Ridge, M. Bjelonic, L. Wellhausen, M. Hutter, et al., “Self-supervised traversability prediction by learning to reconstruct safe terrain,” in 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 12419–12425, IEEE, 2022
work page 2022
Show all 28 references
-
[8]
Learning o ff-road terrain traversability with self-supervisions only,
J. Seo, S. Sim, and I. Shim, “Learning o ff-road terrain traversability with self-supervisions only,” IEEE Robotics and Automation Letters , vol. 8, no. 8, pp. 4617–4624, 2023. 5 Table 1: Ablation study of the proposed autolabeling components for our dataset. Intersection over...
2023
-
[9]
V-strong: Visual self-supervised traversability learning for o ff-road navigation,
S. Jung, J. Lee, X. Meng, B. Boots, and A. Lambert, “V-strong: Visual self-supervised traversability learning for o ff-road navigation,” in 2024 IEEE International Conference on Robotics and Automation (ICRA) , pp. 1766–1773, IEEE, 2024
2024
-
[10]
Scate: A scalable frame- work for self-supervised traversability estimation in unstructured envi- ronments,
J. Seo, T. Kim, K. Kwak, J. Min, and I. Shim, “Scate: A scalable frame- work for self-supervised traversability estimation in unstructured envi- ronments,” IEEE Robotics and Automation Letters, vol. 8, no. 2, pp. 888– 895, 2023
2023
-
[12]
A co-point mapping-based approach to drivable area detection for self-driving cars,
Z. Liu, S. Yu, and N. Zheng, “A co-point mapping-based approach to drivable area detection for self-driving cars,” Engineering, vol. 4, no. 4, pp. 479–490, 2018
2018
-
[13]
Probabilistic traversability map generation using 3d-lidar and camera,
J. Sock, J. Kim, J. Min, and K. Kwak, “Probabilistic traversability map generation using 3d-lidar and camera,” in 2016 IEEE international con- ference on robotics and automation (ICRA) , pp. 5631–5637, IEEE, 2016
2016
-
[14]
Roadrunner–learning traversability estimation for au- tonomous off-road driving,
J. Frey, S. Khattak, M. Patel, D. Atha, J. Nubert, C. Padgett, M. Hut- ter, and P. Spieler, “Roadrunner–learning traversability estimation for au- tonomous off-road driving,” arXiv preprint arXiv:2402.19341, 2024
2024 arXiv
-
[15]
Learning-on- the-drive: Self-supervised adaptation of visual offroad traversability mod- els,
E. Chen, C. Ho, M. Maulimov, C. Wang, and S. Scherer, “Learning-on- the-drive: Self-supervised adaptation of visual offroad traversability mod- els,” arXiv preprint arXiv:2306.15226, 2023
2023 arXiv
-
[16]
Self-supervised drivable area and road anomaly segmentation using rgb-d data for robotic wheelchairs,
H. Wang, Y . Sun, and M. Liu, “Self-supervised drivable area and road anomaly segmentation using rgb-d data for robotic wheelchairs,” IEEE Robotics and Automation Letters, vol. 4, no. 4, pp. 4386–4393, 2019
2019
-
[17]
Self-supervised learning of the driv- able area for autonomous vehicles,
J. Mayr, C. Unger, and F. Tombari, “Self-supervised learning of the driv- able area for autonomous vehicles,” in2018 IEEE/RSJ International Con- ference on Intelligent Robots and Systems (IROS) , pp. 362–369, IEEE, 2018
2018
-
[18]
Self-supervised driv- able area segmentation using lidar’s depth information for autonomous driving,
F. Ma, Y . Liu, S. Wang, J. Wu, W. Qi, and M. Liu, “Self-supervised driv- able area segmentation using lidar’s depth information for autonomous driving,” in 2023 IEEE /RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 41–48, IEEE, 2023
2023
-
[19]
Camera–lidar sen- sor fusion for drivable area detection in winter weather using convolu- tional neural networks,
N. A. Rawashdeh, J. P. Bos, and N. J. Abu-Alrub, “Camera–lidar sen- sor fusion for drivable area detection in winter weather using convolu- tional neural networks,” Optical Engineering, vol. 62, no. 3, pp. 031202– 031202, 2023
2023
-
[20]
Emerging properties in self-supervised vision transformers,
M. Caron, H. Touvron, I. Misra, H. J ´egou, J. Mairal, P. Bojanowski, and A. Joulin, “Emerging properties in self-supervised vision transformers,” in Proceedings of the IEEE /CVF international conference on computer vision, pp. 9650–9660, 2021
2021
-
[21]
Dinov2: Learning robust visual features without supervision,
M. Oquab, T. Darcet, T. Moutakanni, H. V o, M. Szafraniec, V . Khali- dov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, et al. , “Dinov2: Learning robust visual features without supervision,” arXiv preprint arXiv:2304.07193, 2023
2023 arXiv
-
[22]
Segment anything,
A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y . Lo,et al., “Segment anything,” in Proceedings of the IEEE /CVF International Conference on Computer Vision, pp. 4015–4026, 2023
2023
-
[23]
Sam 2: Segment anything in images and videos,
N. Ravi, V . Gabeur, Y .-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. R¨adle, C. Rolland, L. Gustafson, et al., “Sam 2: Segment anything in images and videos,” arXiv preprint arXiv:2408.00714, 2024
2024 arXiv
-
[24]
A 3d lidar data-based dedicated road boundary detection algorithm for autonomous vehicles,
P. Sun, X. Zhao, Z. Xu, R. Wang, and H. Min, “A 3d lidar data-based dedicated road boundary detection algorithm for autonomous vehicles,” IEEE Access, vol. 7, pp. 29623–29638, 2019
2019
-
[25]
Speed and accuracy tradeo ff for lidar data based road boundary detection,
G. Wang, J. Wu, R. He, and B. Tian, “Speed and accuracy tradeo ff for lidar data based road boundary detection,” IEEE/CAA Journal of Auto- matica Sinica, vol. 8, no. 6, pp. 1210–1220, 2020
2020
-
[26]
Road-segmentation-based curb detection method for self-driving via a 3d-lidar sensor,
Y . Zhang, J. Wang, X. Wang, and J. M. Dolan, “Road-segmentation-based curb detection method for self-driving via a 3d-lidar sensor,”IEEE trans- actions on intelligent transportation systems , vol. 19, no. 12, pp. 3981– 3991, 2018
2018
-
[27]
Efficient inference in fully connected crfs with gaussian edge potentials,
P. Kr ¨ahenb¨uhl and V . Koltun, “Efficient inference in fully connected crfs with gaussian edge potentials,” Advances in neural information process- ing systems, vol. 24, 2011
2011
-
[28]
Rethinking atrous convolution for semantic image segmen- tation,
L.-C. Chen, “Rethinking atrous convolution for semantic image segmen- tation,” arXiv preprint arXiv:1706.05587, 2017. 7
2017 arXiv
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.