Pith. sign in

REVIEW 4 major objections 7 minor 82 references

Multimodal and Multiview Deep Fusion for Autonomous Marine Navigation

T0 review · 4 major / 7 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A cross-attention transformer fuses cameras, infrared, and LiDAR into a real-time bird's-eye view for marine navigation.

desk verdict Worth reading for the real maritime BEV dataset and the honest transfer of CVT to a new domain, but Table IV alone cannot support the accuracy and robustness claims without a documented split and baselines. read the letter →

arxiv 2505.01615 v1 pith:43ZEJB5N submitted 2025-05-02 cs.CV cs.AI

Dimitrios Dagdilelis , Panagiotis Grigoriadis , Roberto Galeazzi This is my paper · ORCID
classification cs.CVcs.AI
keywords sensorfusionbird's-eyeviewsegmentationcross-attentiontransformerautonomousmarinenavigationLiDARlong-waveinfraredtemporalsemantic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a cross-attention transformer can fuse multiview RGB cameras, long-wave infrared cameras, and sparse LiDAR into a single real-time bird's-eye-view (BEV) map of a vessel's surroundings, and that this map is accurate enough to support autonomous marine navigation. The authors extend a camera-only BEV transformer by treating LiDAR point clouds as pseudo-images and adding temporal fusion, with X-band radar and electronic nautical charts used to build ground-truth maps for training. On data collected during six hours of daytime sea trials on a tugboat, the fused model improves mean intersection-over-union on all five classes (boat, buoy, water, shoreline, land) compared with RGB-only baselines. The point is that maritime perception can move from isolated target detection to complete scene understanding, matching what autonomous driving already does.

What carries the argument

The load-bearing mechanism is a cross-attention transformer that treats map-view segmentation as sequence-to-sequence translation. A set of 625 learnable BEV queries, each positionally encoded by a pseudo-camera looking down at the map plane, attends over a sequence of features coming from all camera and pseudo-camera views. Each input feature is augmented with a view-aware ray direction computed from camera intrinsics and extrinsics and with a temporal encoding, so the model can learn correspondences between image pixels and BEV cells without explicit depth prediction or precise calibration. LiDAR is made compatible with this pipeline by rasterising the point cloud onto a BEV grid and storing per-cell statistics of the z-coordinates, producing a dense pseudo-image. A temporal module aligns BEV features across time instances using ego-motion and fuses them with 3D convolutions.

What would settle it

Run the trained model on a labelled harbour scene containing a bridge or a high quay wall, and compare predicted BEV positions with surveyed ground truth: if elevated structures are consistently misplaced or smeared along the camera-ray direction, the planar assumption is violated in exactly the way the method cannot absorb.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that deep, cross-modal fusion through a transformer produces a calibration-robust semantic BEV segmentation of the marine environment. The model takes as input four RGB views, four LWIR views, and two LiDAR point clouds aggregated over three time instances (5 s apart), rasterizes the LiDAR into dense pseudo-images, encodes each modality with its own backbone, and uses learnable BEV queries with view-aware directional embeddings to attend over the combined feature sequence. The decoder outputs a 200x200 map at 3 m per pixel covering 600 by 600 metres centred on the ship, with five classes. Experiments report a monotonic improvement in mean IoU as modalities are added: boat IoU rises from 11% (RGB only) to 17% (temporal RGB plus LWIR plus LiDAR), and water IoU from 70% to 91% once temporal alignment is included. The authors interpret these results as evidence that BEV perception has a functional place in autonomous ship navigation.

Load-bearing premise

Everything rests on the planar-world assumption that all objects and surfaces lie on the zero-height water plane; if a scene contains elevated structures or terrain, both the ground-truth labels and the learned image-to-map projection become systematically wrong.

Editorial extensions

If this is right

  • If the reported gains hold, a vessel can obtain a 600-metre semantic surround map from sensors already common on ships, without per-voyage recalibration.
  • The monotonic IoU improvements across modality ablations support adding LWIR and LiDAR to RGB as a robustness gain for water and land segmentation under poor visibility.
  • Temporal alignment raises static-class accuracy (water IoU from 70% to 91%), meaning that a short memory of past frames materially stabilises the map.
  • Because the architecture treats LiDAR as just another view, the same fusion design can accept future sparse modalities, such as Doppler radar, as extra pseudo-views.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next test the paper does not run is whether the model's calibration robustness holds under deliberate synthetic rotation or translation of one camera; the learned embeddings should be probed against known pose errors.
  • The ground truth is built from ENC and radar, the same sources a deployed system may rely on, so an independent surveyed dataset would be needed to rule out label leakage inflating the reported IoU.
  • If the planar assumption is relaxed, the same query mechanism could in principle operate on a 2.5D or voxel grid, extending the method to ports with cranes, bridges, and quay structures.
  • Attention saliency could double as a sensor-diagnostic tool: a persistent mismatch between attended regions and expected target locations would flag a drifting camera or radar.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes a cross-attention transformer for bird's-eye-view semantic map segmentation in autonomous marine navigation. Inputs are multi-view RGB and long-wave infrared (LWIR) images together with LiDAR point clouds, processed as pseudo-images; training labels are generated from electronic nautical charts, X-band radar, GNSS/attitude data, and manual verification of moving targets. Temporal aggregation is performed with ego-motion alignment and 3D convolutions. The method is evaluated on six hours of real-world data collected aboard a tugboat, and Table IV reports per-class and mean IoU for five variants (RGB-only, adding LWIR, adding LiDAR, and temporal variants). The paper claims accurate, robust, real-time BEV perception and states that no prior work has validated multimodal BEV perception for autonomous waterborne navigation.

Significance. If the evaluation were solid, this would be a useful contribution to maritime perception: it extends the CVT architecture to fuse RGB, LWIR, and LiDAR in a BEV segmentation framework, introduces a real-world maritime dataset with automatically generated labels from ENC/radar, and provides ablations over modalities and temporal aggregation. The real-sea data collection and the attention-map visualizations are assets. However, the significance is currently limited by the experimental protocol: the absence of a documented train/test split, external baselines, error bars, and inference-time measurements makes it impossible to substantiate the headline claims of accurate, robust, real-time performance. The work is presented as the first of its kind for marine navigation, but the evidence in Table IV does not yet support that position.

major comments (4)
  1. [Section V-A, Table IV] The evaluation never specifies a train/test split. Section V-A reports 120 epochs of training and then states 'We repeat the evaluation over a multitude of experiments,' but it does not say which part of the 6-hour sequence was held out for the numbers in Table IV, how the temporal sequence was partitioned, or whether the reported rows were used for model selection. Without a documented held-out split, the IoU values in Table IV are consistent with memorization of training data, and the 3-6 point improvements attributed to LWIR, LiDAR, and temporal fusion cannot be distinguished from overfitting. Please specify the split, the number of test samples, and the validation protocol used to choose hyperparameters.
  2. [Section V-A, Table IV] Table IV contains a single number per method and class, with no error bars, no multiple seeds, and no statistical test. Additionally, no external baseline (e.g., CVT, LSS, BEVFormer, or a simple LiDAR-only BEV baseline) is reported, so the absolute IoU values cannot be interpreted relative to the state of the art. The Boat class, which is the most safety-relevant moving-target class, achieves only 11-17% IoU across all variants, yet the abstract and Section VI claim 'accurate and robust' scene representation; this discrepancy needs explicit discussion.
  3. [Abstract, Section VI] The abstract and conclusion claim real-time prediction, but no inference time or latency is reported anywhere in Section V. Given that Section VII acknowledges that cross-attention complexity scales quadratically with feature resolution, timesteps, and views, the real-time claim requires a per-sample runtime measurement on the deployed hardware, or the claim should be removed.
  4. [Section II, Section V-A] The paper claims the method 'remains robust to sensor-calibration errors' (Section II), but no experiment perturbs calibration parameters or compares performance with and without the geometric augmentation described in Section V-A. Random crop/rotation augmentation is a training-time regularization technique, not an evaluation of calibration robustness. Either add a calibration-error ablation or soften the claim to match the evidence.
minor comments (7)
  1. [Throughout] There are numerous typos and LaTeX artifacts: the author affiliation says 'Engineernig,' the text contains 'Sections section II-section III,' Section V-A says 'salience heta-maps,' and the caption of Fig. 2 contains an unresolved '??'. These should be cleaned up.
  2. [Section IV-A] The planar-world assumption is acknowledged but not validated. Since it underlies both ground-truth rasterization and camera-to-BEV projection, a brief justification (e.g., elevation statistics of the harbor area) or a sensitivity analysis would increase confidence in the label generation.
  3. [Section IV-D, Section VII] The limitations section mentions that a 'W-Radar modality was excluded due to its sparse sampling,' but Table III lists no W-Radar sensor. Please clarify which sensor is meant and why it appears only in the limitations.
  4. [Section IV-B, Eq. (2)] The set-builder notation for Z_{ij} contains two vertical bars and is malformed; it should be rewritten with a single condition. The notation for the point cloud also uses 'i' ambiguously as both a point index and an image index.
  5. [Section IV-B, Section V-A] The text says LWIR images are 'converted to RGB,' but the implementation section says they are single-channel and replicated to three channels. Please make the wording consistent.
  6. [Section V] The dataset is not released and no sample counts or class statistics are given. Reporting these would aid reproducibility and interpretation of the per-class IoU values.
  7. [Section V-B, Figs. 13-14] The attention-map visualizations are qualitative. A quantitative evaluation of the attention mechanism (e.g., correlation with ground-truth object locations) would strengthen the explainability claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity identified: supervision comes from external ENC/XBR/manual annotation, and inference inputs (RGB/LWIR/LiDAR) are distinct from the label-generation sources.

full rationale

The paper's derivation chain is a standard supervised map-view segmentation pipeline. Ground-truth BEV maps are generated from external chart, radar, and navigation data, not from the model's own outputs: Section IV-A states 'We fuse ENC, with XBR and own-ship geo-location and attitude data, to generate BEV ground truth maps' and 'The derived XBR-over-water maps, are manually annotated, using camera images as cross-referencing data.' The model inputs at inference are RGB, LWIR, and LiDAR (Section IV-B), while radar and ENC enter only in supervision. No equation defines a predicted quantity in terms of itself: the loss is the standard focal loss between ground-truth y and prediction y-hat, and the cross-attention queries are learnable parameters rather than fitted values drawn from the evaluation targets. No self-citation chain is load-bearing; the architectural influences cited (CVT [13], BEVFormer [11], transformer [66]) are external works with independent content. The absence of a documented train/test split, baselines, and error bars in Section V is a serious evaluation-reporting gap, but it is a rigor issue rather than circularity: even if the model memorized the training sequence, that would not make the derivation circular. The planar-world assumption in Section IV-A ('the assumption of all BEV features residing on the same plane is valid in our operation environment') is an acknowledged modeling assumption that conditions label validity, not a tautology. Accordingly, no circular steps are identified and the circularity score is 0.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

This paper is an empirical deep-learning study, so there are no physically fitted constants; the relevant free parameters are architectural and dataset choices. The main assumptions are about the validity of the planar BEV space, the quality of the automatically generated and manually curated labels, and the representativeness of the six-hour dataset.

free parameters (5)
  • BEV map resolution = 3 m/pixel over 600m x 600m
    Output grid resolution chosen by hand; affects all IoU numbers and the difficulty of the segmentation task.
  • Number of BEV queries = 25x25 = 625
    Capacity of the transformer decoder, chosen without reported sensitivity analysis.
  • Latent feature dimension d_m and d_e = 128
    Embedding sizes for features and queries, chosen by hand.
  • Temporal window and sampling interval = t=3, 5 s apart
    Amount of history fused; no ablation on different t is reported.
  • Pseudo-image grid size d_S
    The LiDAR rasterization resolution is never given in the paper, making the method hard to reproduce.
assumptions (5)
  • domain assumption All relevant world features lie on the same horizontal plane (z=0) when projected to BEV.
    Section IV-A states this assumption is 'valid in our operation environment' but it is load-bearing for the ENC/XBR label generation and for camera-to-BEV projection.
  • domain assumption The six-hour daytime recording on tugboat Balder is representative of the maritime navigation conditions the system targets.
    The abstract claims adverse weather and complex environments, but the dataset contains only one daytime session in Limfjorden, with no reported weather or traffic variation.
  • domain assumption The manually reviewed XBR-over-water maps correctly identify all moving targets after cross-referencing with camera images.
    Moving-target labels are produced by manual annotation of radar blobs; any missed or falsely confirmed targets are baked into the ground truth.
  • domain assumption Cross-attention transformers can learn implicit depth from perspective camera features without explicit depth supervision.
    The approach follows CVT [13] and assumes the network can infer 3D structure from the positional ray encodings; if this fails, the BEV projection of camera features collapses.
  • domain assumption Camera intrinsic and extrinsic parameters are known well enough for positional ray encodings and temporal alignment.
    The method claims calibration robustness from augmentations, but the geometry is initialized from calibration matrices and ego-motion; large errors are only partially mitigated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multimodal and Multiview Deep Fusion for Autonomous Marine Navigation." pith.science (2026). https://pith.science/paper/43ZEJB5N

@misc{pith2026250501615,
  author       = {Pith},
  title        = {Pith review of: Multimodal and Multiview Deep Fusion for Autonomous Marine Navigation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/43ZEJB5N}},
  note         = {Machine review of arXiv:2505.01615}
}
read the original abstract

We propose a cross attention transformer based method for multimodal sensor fusion to build a birds eye view of a vessels surroundings supporting safer autonomous marine navigation. The model deeply fuses multiview RGB and long wave infrared images with sparse LiDAR point clouds. Training also integrates X band radar and electronic chart data to inform predictions. The resulting view provides a detailed reliable scene representation improving navigational accuracy and robustness. Real world sea trials confirm the methods effectiveness even in adverse weather and complex maritime settings.

Figures

Figures reproduced from arXiv: 2505.01615 by the authors.

Figure 1
Figure 1. High level overview of the proposed approach. Sensor streams are processed by modality-specific feature extractors. Modality features are processed [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Outlook from a virtual camera in a top down view pose, based on [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Tugboat Balder used for the data collection at Limfjorden, DK. Annotated are: a) XBR b) RGB & LWIR camera platform c) LiDAR We exploit the complementary nature of the available data sources to construct multi-class BEV ground truth maps y ∈ B as illustrated in fig. 4. ENC are integral to navigation systems and software used on ships and various marine vehicles, playing a critical role in facilitating safe sea travel… view at source ↗
Figures from the paper (10 more)
Figure 5
Figure 5. Figure 5: Creating a ground truth BEV map y using XBR, ENC, satellite￾compass, and GNSS. geodetic model. The process involves an initial conversion from pixel coordinates to local Cartesian coordinates, followed by a transformation to geodetic coordinates using the specified loc…
Figure 6
Figure 6. Figure 6: Camera view corresponding to fig. 5. The semantic map [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Softmaxed BEV map predictions yˆ. Each BEV map in the figure’s tiles, predicts the presence of one of the 5 classes of interest, within 600x600m area around own-ship, where pixel intensity encodes the class’s likelihood. Let also S represent the processed pseudo-image …
Figure 8
Figure 8. Figure 8: Individual modality field of views annotated by maker type and color, [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: PCA (n=1) of a pseudo-camera lidar point cloud [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 10
Figure 10. Figure 10: Pre-processing G of LiDAR point cloud data. The points are projected on a BEV grid defined by the view a BEV point of view and the z = 0 ground place. Each cell contains statistics of the z-coordinate values of the points that fall within the cell (see also fig. 9). W…
Figure 11
Figure 11. Figure 11: Our proposed fusion framework: We present a cross-modal and cross-view fusion architecture for the task of BEV segmentation based on camera [PITH_FULL_IMAGE:figures/full_fig_p009_11.png]
Figure 12
Figure 12. Figure 12: View-aware cross attention. Pixel positions are lifted to unit [PITH_FULL_IMAGE:figures/full_fig_p009_12.png]
Figure 13
Figure 13. Figure 13: Overlaying attention map on the input data. The figure illustrates the correlation between BEV and camera view representations. Image position on [PITH_FULL_IMAGE:figures/full_fig_p011_13.png]
Figure 14
Figure 14. Figure 14: Attention maps Wprobe calculated from the two cross-attention modules, that operate on different input feature resolutions (fig. 11). The produced score map demonstrate the ability of the model to focus on salient regions that correspond to a BEV region (here to the r…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

82 extracted references · 50 canonical work pages

  1. [1]

    Vision-RADAR fusion for Robotics BEV Detections: A Survey,

    A. Singh, “Vision-RADAR fusion for Robotics BEV Detections: A Survey,” 2 2023

    2023

  2. [2]

    Real-time 360 degrees view for the operator of milliampere 2,

    M. T. Paasche, O. K. Helgesen, and E. F. Brekke, “Real-time 360 degrees view for the operator of milliampere 2,” Journal of Physics: Conference Series, vol. 2618, no. 1, 2023

    2023

  3. [3]

    Object detection based on multi- sensor proposal fusion in maritime environment,

    F. Farahnakian, M. H. Haghbayan, J. Poikonen, M. Laurinen, P. Nevalainen, and J. Heikkonen, “Object detection based on multi- sensor proposal fusion in maritime environment,” Proceedings - 17th Ieee International Conference on Machine Learning and Applications, Icmla 2018, p. 8614183, 2018

    work page 2018

  4. [4]

    Ship detection based on lidar and visual information fusion,

    R. Ma, Y . Yin, and K. Bao, “Ship detection based on lidar and visual information fusion,” 2022 Conference on Lasers and Electro-optics, Cleo 2022 - Proceedings , p. JW3B.12, 2022

    work page 2022

  5. [5]

    Lidar-based simultaneous multi-object tracking and static mapping in nearshore scenario,

    Z. Yao, X. Chen, N. Xu, N. Gao, and M. Ge, “Lidar-based simultaneous multi-object tracking and static mapping in nearshore scenario,” Ocean Engineering , vol. 272, p. 113939, 2023. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0029801823003232

    work page 2023

  6. [6]

    Comparison of multi-sensor fusion methods for maritime target object tracking,

    J. Han, Y . Cho, J. Kim, and P. Lee, “Comparison of multi-sensor fusion methods for maritime target object tracking,” Journal of Institute of Control, Robotics and Systems , vol. 25, no. 6, pp. 551–556, 2019

    work page 2019

  7. [7]

    An efficient multi-sensor fusion approach for object detection in maritime environments,

    M. H. Haghbayan, F. Farahnakian, J. Poikonen, M. Laurinen, P. Nevalainen, J. Plosila, and J. Heikkonen, “An efficient multi-sensor fusion approach for object detection in maritime environments,” Ieee Conference on Intelligent Transportation Systems, Proceedings, Itsc, vol. 2018-, pp. 2163–2170, 2018

    work page 2018

  8. [8]

    Multimodal perception for obstacle detection for flying boats-unmanned surface vehicle (usv),

    R. Douguet, D. Heller, and J. Laurent, “Multimodal perception for obstacle detection for flying boats-unmanned surface vehicle (usv),” in OCEANS 2023-Limerick. IEEE, 2023, pp. 1–8

    work page 2023

Show all 82 references
  1. [9]

    Sensor combinations in heterogeneous multi-sensor fusion for maritime target tracking,

    O. K. Helgesen, E. F. Brekke, H. H. Helgesen, and O. Engelhardtsen, “Sensor combinations in heterogeneous multi-sensor fusion for maritime target tracking,” Fusion 2019 - 22nd International Conference on Information Fusion, p. 9011297, 2019

    2019

  2. [10]

    Stereo camera-based free space estimation for docking in urban waters,

    T. A. Nygard, N. Dalhaug, R. Mester, E. Brekke, and A. Stahl, “Stereo camera-based free space estimation for docking in urban waters,” Modeling Identification and Control , 2024. 12 RGB0 Wa Wa + data Wb Wb + data RGB1 RGB2 RGB3 LWIR0 LWIR1 LWIR2 LWIR3 LiDAR Fig. 14. Attention ...

    2024

  3. [11]

    Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers,

    Z. Li, W. Wang, H. Li, E. Xie, C. Sima, T. Lu, Y . Qiao, and J. Dai, “Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers,” in European conference on computer vision. Springer, 2022, pp. 1–18

    2022

  4. [12]

    Petr: Position embedding trans- formation for multi-view 3d object detection,

    Y . Liu, T. Wang, X. Zhang, and J. Sun, “Petr: Position embedding trans- formation for multi-view 3d object detection,” in European Conference on Computer Vision . Springer, 2022, pp. 531–548

    2022

  5. [13]

    Cross-view Transformers for real-time Map-view Semantic Segmentation,

    B. Zhou and P. Kr ¨ahenb¨uhl, “Cross-view Transformers for real-time Map-view Semantic Segmentation,” 5 2022

    2022

  6. [14]

    Cross-view Semantic Segmentation for Sensing Surroundings,

    B. Pan, J. Sun, H. Y . T. Leung, A. Andonian, and B. Zhou, “Cross-view Semantic Segmentation for Sensing Surroundings,” 6 2019

    2019

  7. [15]

    Orthographic Feature Trans- form for Monocular 3D Object Detection,

    T. Roddick, A. Kendall, and R. Cipolla, “Orthographic Feature Trans- form for Monocular 3D Object Detection,” 11 2018

    2018

  8. [16]

    Delving into the Devils of Bird’s-eye-view Perception: A Review, Evaluation and Recipe,

    H. Li, C. Sima, J. Dai, W. Wang, L. Lu, H. Wang, J. Zeng, Z. Li, J. Yang, H. Deng, H. Tian, E. Xie, J. Xie, L. Chen, T. Li, Y . Li, Y . Gao, X. Jia, S. Liu, J. Shi, D. Lin, and Y . Qiao, “Delving into the Devils of Bird’s-eye-view Perception: A Review, Evaluation and Recipe,” ...

    2023

  9. [17]

    Fcos3d: Fully convolutional one- stage monocular 3d object detection,

    T. Wang, X. Zhu, J. Pang, and D. Lin, “Fcos3d: Fully convolutional one- stage monocular 3d object detection,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 913–922

    2021

  10. [18]

    Smoke: Single-stage monocular 3d object detection via keypoint estimation,

    Z. Liu, Z. Wu, and R. T ´oth, “Smoke: Single-stage monocular 3d object detection via keypoint estimation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, 2020, pp. 996–997

    2020

  11. [19]

    Bevdet: High- performance multi-camera 3d object detection in bird-eye-view,

    J. Huang, G. Huang, Z. Zhu, Y . Ye, and D. Du, “Bevdet: High- performance multi-camera 3d object detection in bird-eye-view,” arXiv preprint arXiv:2112.11790, 2021

    2021 arXiv

  12. [20]

    Bevfusion: Multi-task multi-sensor fusion with unified bird’s-eye view representation,

    Z. Liu, H. Tang, A. Amini, X. Yang, H. Mao, D. L. Rus, and S. Han, “Bevfusion: Multi-task multi-sensor fusion with unified bird’s-eye view representation,” in 2023 IEEE international conference on robotics and automation (ICRA). IEEE, 2023, pp. 2774–2781

    2023

  13. [21]

    A sim2real deep learning ap- proach for the transformation of images from multiple vehicle-mounted cameras to a semantically segmented image in bird’s eye view,

    L. Reiher, B. Lampe, and L. Eckstein, “A sim2real deep learning ap- proach for the transformation of images from multiple vehicle-mounted cameras to a semantically segmented image in bird’s eye view,” in 2020 IEEE 23rd International Conference on Intelligent Transportation Sys...

    2020

  14. [22]

    Fiery: Future instance prediction in bird’s- eye view from surround monocular cameras,

    A. Hu, Z. Murez, N. Mohan, S. Dudas, J. Hawke, V . Badrinarayanan, R. Cipolla, and A. Kendall, “Fiery: Future instance prediction in bird’s- eye view from surround monocular cameras,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 15 273–15 282

    2021

  15. [23]

    Hdmapnet: An online hd map construction and evaluation framework,

    Q. Li, Y . Wang, Y . Wang, and H. Zhao, “Hdmapnet: An online hd map construction and evaluation framework,” in 2022 International Conference on Robotics and Automation (ICRA) . IEEE, 2022, pp. 4628–4634

    2022

  16. [24]

    Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d,

    J. Philion and S. Fidler, “Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d,” in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIV 16 . Springer, 2020, pp. 194–210

    2020

  17. [25]

    St-p3: End-to-end vision-based autonomous driving via spatial-temporal feature learning,

    S. Hu, L. Chen, P. Wu, H. Li, J. Yan, and D. Tao, “St-p3: End-to-end vision-based autonomous driving via spatial-temporal feature learning,” in European Conference on Computer Vision. Springer, 2022, pp. 533– 13 549

    2022

  18. [26]

    Pv-rcnn: Point-voxel feature set abstraction for 3d object detection,

    S. Shi, C. Guo, L. Jiang, Z. Wang, J. Shi, X. Wang, and H. Li, “Pv-rcnn: Point-voxel feature set abstraction for 3d object detection,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 10 529–10 538

    2020

  19. [27]

    Center-based 3d object detection and tracking,

    T. Yin, X. Zhou, and P. Krahenbuhl, “Center-based 3d object detection and tracking,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2021, pp. 11 784–11 793

    2021

  20. [28]

    Embracing single stride 3d object detector with sparse transformer,

    L. Fan, Z. Pang, T. Zhang, Y .-X. Wang, H. Zhao, F. Wang, N. Wang, and Z. Zhang, “Embracing single stride 3d object detector with sparse transformer,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2022, pp. 8458–8468

    2022

  21. [29]

    Afdetv2: Rethinking the necessity of the second stage for object detection from point clouds,

    Y . Hu, Z. Ding, R. Ge, W. Shao, L. Huang, K. Li, and Q. Liu, “Afdetv2: Rethinking the necessity of the second stage for object detection from point clouds,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 1, 2022, pp. 969–979

    2022

  22. [30]

    Pyramid r-cnn: Towards better performance and adaptability for 3d object detection,

    J. Mao, M. Niu, H. Bai, X. Liang, H. Xu, and C. Xu, “Pyramid r-cnn: Towards better performance and adaptability for 3d object detection,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 2723–2732

    2021

  23. [31]

    Mppnet: Multi-frame feature intertwining with proxy points for 3d temporal ob- ject detection,

    X. Chen, S. Shi, B. Zhu, K. C. Cheung, H. Xu, and H. Li, “Mppnet: Multi-frame feature intertwining with proxy points for 3d temporal ob- ject detection,” in European Conference on Computer Vision. Springer, 2022, pp. 680–697

    2022

  24. [32]

    V oxelnet: End-to-end learning for point cloud based 3d object detection,

    Y . Zhou and O. Tuzel, “V oxelnet: End-to-end learning for point cloud based 3d object detection,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2018, pp. 4490–4499

    2018

  25. [33]

    Pointpillars: Fast encoders for object detection from point clouds,

    A. H. Lang, S. V ora, H. Caesar, L. Zhou, J. Yang, and O. Beijbom, “Pointpillars: Fast encoders for object detection from point clouds,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 12 697–12 705

    2019

  26. [34]

    Pointpainting: Se- quential fusion for 3d object detection,

    S. V ora, A. H. Lang, B. Helou, and O. Beijbom, “Pointpainting: Se- quential fusion for 3d object detection,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 4604– 4612

    2020

  27. [35]

    Multimodal virtual point 3d detection,

    T. Yin, X. Zhou, and P. Kr ¨ahenb¨uhl, “Multimodal virtual point 3d detection,” Advances in Neural Information Processing Systems, vol. 34, pp. 16 494–16 507, 2021

    2021

  28. [36]

    Autoalign: Pixel-instance feature aggregation for multi-modal 3d object detection,

    Z. Chen, Z. Li, S. Zhang, L. Fang, Q. Jiang, F. Zhao, B. Zhou, and H. Zhao, “Autoalign: Pixel-instance feature aggregation for multi-modal 3d object detection,” arXiv preprint arXiv:2201.06493 , 2022

    2022 arXiv

  29. [37]

    Deepfusion: Lidar-camera deep fusion for multi-modal 3d object detection,

    Y . Li, A. W. Yu, T. Meng, B. Caine, J. Ngiam, D. Peng, J. Shen, Y . Lu, D. Zhou, Q. V . Le et al. , “Deepfusion: Lidar-camera deep fusion for multi-modal 3d object detection,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 17 1...

    2022

  30. [38]

    BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird’s-Eye View Representation,

    Z. Liu, H. Tang, A. Amini, X. Yang, H. Mao, D. Rus, and S. Han, “BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird’s-Eye View Representation,” 5 2022

    2022

  31. [39]

    Rethinking imagenet pre-training,

    K. He, R. Girshick, and P. Doll ´ar, “Rethinking imagenet pre-training,” in Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 4918–4927

    2019

  32. [40]

    Second: Sparsely embedded convolutional detection,

    Y . Yan, Y . Mao, and B. Li, “Second: Sparsely embedded convolutional detection,” Sensors, vol. 18, no. 10, p. 3337, 2018

    2018

  33. [41]

    Structure aware single-stage 3d object detection from point cloud,

    C. He, H. Zeng, J. Huang, X.-S. Hua, and L. Zhang, “Structure aware single-stage 3d object detection from point cloud,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2020, pp. 11 873–11 882

    2020

  34. [42]

    V oxel transformer for 3d object detection,

    J. Mao, Y . Xue, M. Niu, H. Bai, J. Feng, X. Liang, H. Xu, and C. Xu, “V oxel transformer for 3d object detection,” in Proceedings of the IEEE/CVF international conference on computer vision , 2021, pp. 3164–3173

    2021

  35. [43]

    Multi-View 3D Object Detection Network for Autonomous Driving,

    X. Chen, H. Ma, J. Wan, B. Li, and T. Xia, “Multi-View 3D Object Detection Network for Autonomous Driving,” Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017 , vol. 2017-January, pp. 6526–6534, 11 2016

    2017

  36. [44]

    Pixor: Real-time 3d object detection from point clouds,

    B. Yang, W. Luo, and R. Urtasun, “Pixor: Real-time 3d object detection from point clouds,” in Proceedings of the IEEE conference on Computer Vision and Pattern Recognition , 2018, pp. 7652–7660

    2018

  37. [45]

    Hdnet: Exploiting hd maps for 3d object detection,

    B. Yang, M. Liang, and R. Urtasun, “Hdnet: Exploiting hd maps for 3d object detection,” in Conference on Robot Learning . PMLR, 2018, pp. 146–155

    2018

  38. [46]

    Rt3d: Real- time 3-d vehicle detection in lidar point cloud for autonomous driving,

    Y . Zeng, Y . Hu, S. Liu, J. Ye, Y . Han, X. Li, and N. Sun, “Rt3d: Real- time 3-d vehicle detection in lidar point cloud for autonomous driving,” IEEE Robotics and Automation Letters , vol. 3, no. 4, pp. 3434–3440, 2018

    2018

  39. [47]

    Yolo3d: End-to-end real-time 3d oriented object bounding box detection from lidar point cloud,

    W. Ali, S. Abdelkarim, M. Zidan, M. Zahran, and A. El Sallab, “Yolo3d: End-to-end real-time 3d oriented object bounding box detection from lidar point cloud,” in Proceedings of the European conference on computer vision (ECCV) workshops , 2018, pp. 0–0

    2018

  40. [48]

    Complex-yolo: An euler-region-proposal for real-time 3d object detection on point clouds,

    M. Simony, S. Milzy, K. Amendey, and H.-M. Gross, “Complex-yolo: An euler-region-proposal for real-time 3d object detection on point clouds,” in Proceedings of the European conference on computer vision (ECCV) workshops, 2018, pp. 0–0

    2018

  41. [49]

    Pointaugmenting: Cross-modal augmentation for 3d object detection,

    C. Wang, C. Ma, M. Zhu, and X. Yang, “Pointaugmenting: Cross-modal augmentation for 3d object detection,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2021, pp. 11 794–11 803

    2021

  42. [50]

    Bevfusion: A simple and robust lidar-camera fusion framework,

    T. Liang, H. Xie, K. Yu, Z. Xia, Z. Lin, Y . Wang, T. Tang, B. Wang, and Z. Tang, “Bevfusion: A simple and robust lidar-camera fusion framework,” 2022

    2022

  43. [51]

    Unifying voxel-based representation with transformer for 3d object detection,

    Y . Li, Y . Chen, X. Qi, Z. Li, J. Sun, and J. Jia, “Unifying voxel-based representation with transformer for 3d object detection,” 6 2022

    2022

  44. [52]

    Inverse perspective mapping simplifies optical flow computation and obstacle detection,

    H. A. Mallot, H. H. B ¨ulthoff, J. J. Little, and S. Bohrer, “Inverse perspective mapping simplifies optical flow computation and obstacle detection,” Biological cybernetics, vol. 64, no. 3, pp. 177–185, 1991

    1991

  45. [53]

    3D-LaneNet: End-to-End 3D Multiple Lane Detection

    N. Garnett, R. Cohen, T. Pe’er, R. Lahav, and D. Levi, “3D-LaneNet: End-to-End 3D Multiple Lane Detection.” [Online]. Available: https://sites.google.com/view/danlevi/3dlanes

  46. [54]

    Deep learning based vehicle position and orientation estimation via inverse perspective mapping image,

    Y . Kim and D. Kum, “Deep learning based vehicle position and orientation estimation via inverse perspective mapping image,” IEEE Intelligent Vehicles Symposium, Proceedings , vol. 2019-June, pp. 317– 323, 6 2019

    2019

  47. [55]

    Driving among Flatmobiles: Bird-Eye-View occupancy grids from a monocular camera for holistic trajectory planning

    A. Loukkal, Y . Grandvalet, T. Drummond, and Y . Li, “Driving among Flatmobiles: Bird-Eye-View occupancy grids from a monocular camera for holistic trajectory planning.”

  48. [56]

    Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D,

    J. Philion and S. Fidler, “Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D,” 8 2020

    2020

  49. [57]

    Robust free area mapping for autonomous harbour navigation,

    M. K. Plenge-Feidenhans’l, “Robust free area mapping for autonomous harbour navigation,” Ph.D. dissertation, 2023

    2023

  50. [58]

    Pseudo-lidar from visual depth estimation: Bridging the gap in 3d object detection for autonomous driving,

    Y . Wang, W. Chao, D. Garg, B. Hariharan, M. Campbell, and K. Q. Weinberger, “Pseudo-lidar from visual depth estimation: Bridging the gap in 3d object detection for autonomous driving,” CoRR, vol. abs/1812.07179, 2018

    2018 arXiv

  51. [59]

    Categorical depth distribution network for monocular 3d object detection,

    C. Reading, A. Harakeh, J. Chae, and S. L. Waslander, “Categorical depth distribution network for monocular 3d object detection,” CoRR, vol. abs/2103.01100, 2021

    2021 arXiv

  52. [60]

    Bevdepth: Acquisition of reliable depth for multi-view 3d object detec- tion,

    Y . Li, Z. Ge, G. Yu, J. Yang, Z. Wang, Y . Shi, J. Sun, and Z. Li, “Bevdepth: Acquisition of reliable depth for multi-view 3d object detec- tion,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 37, no. 2, 2023, pp. 1477–1485

    2023

  53. [61]

    Liga-stereo: Learning lidar geometry aware representations for stereo-based 3d detector,

    X. Guo, S. Shi, X. Wang, and H. Li, “Liga-stereo: Learning lidar geometry aware representations for stereo-based 3d detector,” 2021 Ieee/cvf International Conference on Computer Vision (iccv) , pp. 3133– 3143, 2021

    2021

  54. [62]

    Is pseudo lidar needed for monocular 3d object detection,

    D. Park, R. Ambrus, V . Guizilini, J. Li, and A. Gaidon, “Is pseudo lidar needed for monocular 3d object detection,” in 2021 IEEE CVF International Conference on Computer Vision ICCV , 2021, pp. 3122– 3132

    2021

  55. [63]

    Dsgn: Deep stereo geometry network for 3d object detection,

    Y . Chen, S. Liu, X. Shen, and J. Jia, “Dsgn: Deep stereo geometry network for 3d object detection,” 2020 Ieee/cvf Conference on Computer Vision and Pattern Recognition (cvpr) , pp. 12 533–12 542, 2020

    2020

  56. [64]

    Translating images into maps,

    A. Saha, O. Mendez, C. Russell, and R. Bowden, “Translating images into maps,” in 2022 International conference on robotics and automation (ICRA). IEEE, 2022, pp. 9200–9206

    2022

  57. [65]

    Cross-view semantic segmentation for sensing surroundings,

    B. Pan, J. Sun, H. Y . T. Leung, A. Andonian, and B. Zhou, “Cross-view semantic segmentation for sensing surroundings,” IEEE Robotics and Automation Letters, vol. 5, no. 3, pp. 4867–4873, 2020

    2020

  58. [66]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems , vol. 30, 2017

    2017

  59. [67]

    Simple- BEV: What Really Matters for Multi-Sensor BEV Perception?

    A. W. Harley, Z. Fang, J. Li, R. Ambrus, and K. Fragkiadaki, “Simple- BEV: What Really Matters for Multi-Sensor BEV Perception?” 6 2022

    2022

  60. [68]

    HDMapNet: An Online HD Map Construction and Evaluation Framework,

    Q. Li, Y . Wang, Y . Wang, and H. Zhao, “HDMapNet: An Online HD Map Construction and Evaluation Framework,” 7 2021

    2021

  61. [69]

    PETR: Position Embedding Transformation for Multi-View 3D Object Detection,

    Y . Liu, T. Wang, X. Zhang, and J. Sun, “PETR: Position Embedding Transformation for Multi-View 3D Object Detection,” 3 2022

    2022

  62. [70]

    Focal loss for dense object detection,

    T.-Y . Lin, P. Goyal, R. Girshick, K. He, and P. Doll ´ar, “Focal loss for dense object detection,” in Proceedings of the IEEE international conference on computer vision , 2017, pp. 2980–2988

    2017

  63. [71]

    Birdnet: a 3d object detection framework from 14 lidar information,

    J. Beltr ´an, C. Guindel, F. M. Moreno, D. Cruzado, F. Garcia, and A. De La Escalera, “Birdnet: a 3d object detection framework from 14 lidar information,” in 2018 21st International Conference on Intelligent Transportation Systems (ITSC) . IEEE, 2018, pp. 3517–3523

    2018

  64. [72]

    U-net: Convolutional networks for biomedical image segmentation,

    O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” 2015

    2015

  65. [73]

    Fishing net: Future inference of semantic heatmaps in grids,

    N. Hendy, C. Sloan, F. Tian, P. Duan, N. Charchut, Y . Xie, C. Wang, and J. Philbin, “Fishing net: Future inference of semantic heatmaps in grids,” 2020

    2020

  66. [74]

    Are Transformers universal approximators of sequence-to-sequence functions?

    C. Yun, S. Bhojanapalli, A. S. Rawat, S. J. Reddi, and S. Kumar, “Are Transformers universal approximators of sequence-to-sequence functions?” 8th International Conference on Learning Representations, ICLR 2020, 12 2019

    2020

  67. [75]

    Unifusion: unified multi- view fusion transformer for spatial-temporal representation in bird’s-eye- view,

    Z. Qin, J. Chen, C. Chen, X. Chen, and X. Li, “Unifusion: unified multi- view fusion transformer for spatial-temporal representation in bird’s-eye- view,” 2022

    2022

  68. [76]

    BEVerse: Unified Perception and Prediction in Birds-Eye-View for Vision-Centric Autonomous Driving,

    Y . Zhang, Z. Zhu, W. Zheng, J. Huang, G. Huang, J. Zhou, and J. Lu, “BEVerse: Unified Perception and Prediction in Birds-Eye-View for Vision-Centric Autonomous Driving,” 5 2022

    2022

  69. [77]

    Efficientnet: Rethinking model scaling for con- volutional neural networks,

    M. Tan and Q. Le, “Efficientnet: Rethinking model scaling for con- volutional neural networks,” in International conference on machine learning. PMLR, 2019, pp. 6105–6114

    2019

  70. [78]

    Bevformer v2: Adapting modern image backbones to bird’s-eye-view recognition via perspective supervision,

    C. Yang, Y . Chen, H. Tian, C. Tao, X. Zhu, Z. Zhang, G. Huang, H. Li, Y . Qiao, L. Lu et al. , “Bevformer v2: Adapting modern image backbones to bird’s-eye-view recognition via perspective supervision,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern ...

    2023

  71. [79]

    Simple- bev: What really matters for multi-sensor bev perception?

    A. W. Harley, Z. Fang, J. Li, R. Ambrus, and K. Fragkiadaki, “Simple- bev: What really matters for multi-sensor bev perception?” in 2023 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2023, pp. 2759–2765

    2023

  72. [80]

    Decoupled weight decay regularization,

    I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” arXiv preprint arXiv:1711.05101 , 2017

    2017 arXiv

  73. [81]

    Super-convergence: Very fast training of neural networks using large learning rates,

    L. N. Smith and N. Topin, “Super-convergence: Very fast training of neural networks using large learning rates,” in Artificial intelligence and machine learning for multi-domain operations applications , vol. 11006. SPIE, 2019, pp. 369–386

    2019

  74. [82]

    Deformable DETR: Deformable Transformers for End-to-End Object Detection,

    X. Zhu, W. Su, L. Lu, B. Li, X. Wang, and J. Dai, “Deformable DETR: Deformable Transformers for End-to-End Object Detection,” 10 2020

    2020

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.