Pith. sign in

REVIEW 3 major objections 5 minor 30 references

HiLO: High-Level Object Fusion for Autonomous Driving using Transformers

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read A compact transformer that fuses camera and radar object lists beats Kalman-filter fusion by up to 25.9 points in F1-score on real driving data.

desk verdict Useful transformer-for-high-level-fusion idea and a big new dataset, but the headline F1 gap is computed on a test set filtered with its own annotations, so the quantitative claim is not yet established. read the letter →

arxiv 2506.02554 v1 pith:6CP7YQRZ submitted 2025-06-03 cs.RO cs.AIcs.CVcs.LG

classification cs.ROcs.AIcs.CVcs.LG
keywords high-levelsensorfusiontransformerautonomousdrivingobjectKalmanfiltercross-domainevaluationcamera-radarreal-worlddataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces HiLO, a transformer-based method that fuses object lists from camera and radar sensors—rather than raw data or learned features—into a unified set of tracked objects for an autonomous vehicle. The paper argues that classical high-level fusion with Kalman filters depends on hand-tuned motion and noise models that miss important effects, and it shows that a small encoder-decoder transformer can capture those effects. On a new dataset of about 1.85 million real urban and highway frames collected with near-production sensors, HiLO raises F1-score by up to 25.9 percentage points and mean IoU by up to 6.1 percentage points over adapted Kalman-filter baselines. The model runs in roughly 3.4 ms on a CPU and is small enough for near-production use, but it degrades under cross-domain shifts (for example, trained on highway and tested on urban), which the authors flag as a target for domain adaptation.

What carries the argument

The load-bearing mechanism is a DETR-inspired encoder-decoder transformer. The encoder runs self-attention (called multi-modal attention) over all sensor detections, whose (x,y) coordinates are augmented with sinusoidal positional embeddings; the decoder uses a fixed set of 20 learnable query vectors to attend to the encoded features and directly output fused bounding boxes, class logits, and a "no-object" class for empty queries. Training uses Hungarian matching between predictions and ground-truth annotations, with a loss that combines weighted cross-entropy, L1 box loss, generalized IoU, and an orientation loss. This replaces the Kalman filter's explicit geometric inflation, Mahalanobis gating, and hand-tuned covariance updates with a learned association and fusion step.

What would settle it

Recompute F1 and mIoU on the full, unfiltered test set and on the subset of samples excluded by the confidence-threshold and single-sided-lidar rules; if HiLO's advantage over AKFA shrinks or reverses on those excluded samples, the headline gains depend on the filtering choices.

Watch

Extended reading notes

Core claim

The paper's central claim is that high-level object fusion—joining object lists from multiple sensors into one detection set—can be solved with a compact DETR-derived transformer, and that this learned approach substantially outperforms classical Kalman-filter fusion on real-world data. On its new dataset, HiLO reaches 69.1% F1 on highway and 44.3% on urban test splits, against 46.7% and 29.4% for the best Kalman baseline (AKFA), while the mean IoU improves by up to 6.1 percentage points. The paper attributes the gains to attention-based association across sensors and joint estimation of position, extent, class, and orientation, which it argues replace the hand-tuned geometric association and covariance modeling that classical filters rely on.

Load-bearing premise

The evaluation assumes that filtering out low-confidence detections and omitting samples with single-sided lidar coverage does not remove systematically harder cases; if those cases are harder, the reported F1 and mIoU improvements may be inflated.

Editorial extensions

If this is right

  • If the results hold, high-level fusion built from camera and radar object lists can approach feature-level fusion performance at a fraction of the compute, making it viable for near-production vehicles.
  • The classical Adapted Kalman Filter is itself improved by the paper's covariance-inflation modification (AKFA), which raises F1 by several points, suggesting that tuning measurement covariance per sensor is a low-cost step for existing systems.
  • Domain-specific training outperforms combined-domain training, so deploying on a new environment will likely require domain adaptation or more model capacity rather than simply more data.
  • Cross-domain evaluation, common in other machine-learning fields, proves informative here: the transformer loses 7.3 and 21.4 F1 points across the urban-highway shift, yet still beats the Kalman baselines, indicating that learned fusion retains an advantage even under distribution shift.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The filtering step (removing detections below a 5th-percentile confidence threshold and dropping single-sided-lidar samples) may selectively eliminate the hardest association cases, which could favor the learned method; rerunning on the fully unfiltered test stream would test whether the headline gains persist.
  • Because HiLO consumes object lists rather than raw sensor data, the same architecture could be retrained for other sensor stacks (for example, lidar plus camera, or infrastructure-provided detections), without changing the model family.
  • The large cross-domain drop from urban to highway suggests that a relatively small domain-adaptation wrapper, as the authors hint, might recover most of the gap with far less data than joint training.
  • Temporal information is currently unused; feeding the model a short history of fused objects, which the paper lists as future work, could address occlusions and intermittent detections and may close much of the remaining gap to feature-level fusion.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes HiLO, a DETR-style transformer for high-level fusion of camera and radar object tracks, and AKFA, an Adapted Kalman Filter variant with an additional per-sensor measurement covariance. It introduces a large self-collected real-world dataset of roughly 1.85 million samples from urban and highway domains, and reports inter-domain and cross-domain results for AKF, AKFA, and HiLO. The headline claim is an F1-score improvement of up to 25.9 percentage points and an mIoU improvement of up to 6.1 percentage points over the Kalman-filter baselines. The manuscript also reports runtime and memory measurements and states that code, data, and models are publicly available.

Significance. If the empirical comparison is valid, HiLO would be a practically relevant high-level fusion approach with a lightweight transformer architecture and reasonable CPU runtime. The paper has clear strengths: a new large-scale real-world dataset, a cross-domain evaluation protocol, a strong classical baseline with dedicated tuning, public code/data/model release, and explicit runtime figures. The central empirical claim, however, rests on an annotation-guided filtering procedure applied to the test set and on single-run evaluations without uncertainty estimates; these issues need to be addressed before the claimed improvements can be taken as established.

major comments (3)
  1. [IV-C] The filtering rule is annotation-defined. The confidence threshold is set at the 5th percentile of camera and radar detections that overlap with annotations, and detections are then removed if no close annotation exists. This removes exactly the low-confidence false positives (which by definition have no matching annotation) while retaining low-confidence true positives, which mechanically inflates precision and F1 for every method. Since HiLO is trained on the filtered data while AKF/AKFA only consume the filtered sensor tracks as measurements, the magnitude of the effect can differ across methods. The headline 25.9 pp F1 gain and the mIoU gains in Table I are computed on this filtered test set. Please report results on the unfiltered test set, or use a filter that does not consult annotations, and quantify how many samples and detections are removed by each filtering step, including the single-sided-lidar exclusion.
  2. [V, Table I] No variance measures, confidence intervals, or repeated evaluations are reported. Hyperparameter selection uses a stochastic Bayesian search (10 runs per hyperparameter) and HiLO training is stochastic, yet every number in Table I appears to come from a single run. Please report means and standard deviations over multiple training seeds or bootstrap confidence intervals, and state whether small cross-domain gaps such as the urban-to-highway F1 difference between HiLO (47.7) and AKFA (45.9) are statistically meaningful.
  3. [V-C] The conclusion that cross-evaluated HiLO 'still outperforms the classical approaches' is only weakly supported for the urban-to-highway direction, where the F1 margin over AKFA is 1.8 percentage points and the mIoU margin is 1.7 percentage points; without uncertainty estimates, this specific claim is not established. Please either provide significance testing for all pairwise comparisons or soften this conclusion for the cross-domain cases with small margins.
minor comments (5)
  1. [Table I] The table header uses 'AFKA' for the method that is otherwise called 'AKFA'; the abbreviation should be consistent.
  2. [V-C and III] There are typos: 'precent points' in Section V-C, 'classical approch' in Section III, and 'attetion' in the Related Work should be 'attention'.
  3. [IV-B] In Equation (3), the Kalman gain uses the notation P_hat{y}'_{j-1,j,I(k)} with a prime that is not defined; please clarify whether this is a typo for the predicted covariance P_hat{y}.
  4. [V-A] The sentence 'HiLO achieves the highest results across all metrics, indicating classical approaches may neglect important effects in their modeling' goes beyond the evidence; experiments show a performance difference but do not identify the modeling effects responsible.
  5. [VI] The acknowledged limitation of automatic annotation quality is helpful, but it also underscores the need for the unfiltered evaluation requested in the major comments; please make the connection explicit.

Circularity Check

1 steps flagged · score 4.0 of 10

Headline F1/mIoU gaps are computed on a test set cleaned by the test annotations themselves (low-confidence detections 'without close annotation' are deleted), so the reported numbers are partially defined by the evaluation labels; the model derivation and the baselines themselves remain independent.

  1. self definitional [Section IV-C (Data Collection and Filtering); values reported in Abstract, Table I, and Section VI Conclusion]
    "To mitigate the impact of annotation errors on fusion training and evaluation, filtering was applied. A confidence score threshold was set at the 5th percentile of camera and radar detections overlapping with annotations, removing these detections from the dataset if no close annotation existed. This threshold established an upper bound for the classical approach's confidence score."

    The filter's removal rule ('no close annotation existed') is exactly the detection-to-annotation correspondence that the evaluation uses to define a false positive (IoU matching against annotations, threshold 0.5, Section V Metrics). The test input is therefore selected by reference to the same labels that define F1 and mIoU: low-confidence detections that would otherwise be counted as false positives are deleted before any fusion method can emit them, and the AKF/AKFA confidence threshold is capped by this same annotation-derived bound. Thus the headline improvements (25.9 pp F1, 6.1 pp mIoU) are measured on an annotation-selected subset, entangling the reported performance with the evaluation labels rather than purely with fusion quality.

full rationale

The central derivation chain is otherwise self-contained and independent. HiLO is a standard DETR-style encoder-decoder (external components: DETR [14], Hungarian matching [22], gIoU [23]), trained on 75/15/10 splits grouped by recording date, with hyperparameters selected on the validation set and best-validation checkpoint reported on the test set. The AKF baseline is external (Aeberhard [8]); the AKFA extension's per-sensor additional covariance is tuned on the validation set, which is ordinary baseline tuning, not a fitted parameter renamed as a prediction. All reported test metrics are computed from held-out forward passes, so no number in Table I equals a fitted parameter by construction. The only self-citation ([1] Diehl et al.) supports a generic statement about planning/situation prediction and is not load-bearing for the fusion claim. No uniqueness theorem is imported from the authors' prior work, and no ansatz is smuggled in via citation. The one flagged concern is the annotation-defined input filter in Section IV-C: its removal criterion is the metric's own false-positive criterion, and the claimed 25.9/6.1 percentage-point gaps are produced on the resulting label-selected subset. The direction of any resulting bias is not established - the cleaning could plausibly help the classical baselines more than HiLO - but the absolute values and the classical confidence bound are entangled with the evaluation labels, so the strong claim that 'classical approaches may neglect important effects in their modeling' (Section V-A) rests on a partially self-defined comparison. This warrants score 4: a real, quotable self-definitional step in the evaluation, while the core model-versus-baseline comparison retains independent empirical content.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central comparison rests on standard Kalman and DETR mathematics, with the main auxiliary choices being validation-tuned hyperparameters and an ad hoc data-filtering rule. No new physical entities or first-principles derivations are introduced.

free parameters (6)
  • AKFA per-sensor measurement covariance = Tuned on validation set, value not reported
    Added in Eq. (3) to inflate the innovation covariance; optimized per sensor type with Bayesian search.
  • AKF/AKFA existence score thresholds = Tuned per sensor, values not reported
    Thresholds on sensor existence scores and a separate threshold on the fused output are tuned on validation.
  • Process noise covariance Q = Tuned, values not reported
    Diagonal Q for the constant-velocity motion model is optimized for the Kalman filter variants.
  • Association parameters alpha and epsilon = Tuned, values not reported
    Alpha sets the chi-square gate threshold; epsilon is the auction algorithm slack, both tuned on validation.
  • HiLO loss and matching weights = Tuned, values not reported
    Learning rate, matching costs, loss weights, and class weight technique (ENS) chosen via Bayesian search on validation.
  • Data filtering confidence threshold = 5th percentile of detections overlapping annotations
    Chosen from the data to remove detections without close annotation; this preprocessing affects all methods.
assumptions (4)
  • domain assumption The automatic offline annotation tool provides accurate ground truth.
    All metrics are computed against these annotations; the paper itself lists annotation quality as a limitation (Section VI).
  • domain assumption The tracked object lists from the camera and radar ECUs are valid and sufficient inputs for high-level fusion.
    The comparison assumes both methods operate on the same provided sensor tracks, which may contain errors the fusion must handle.
  • standard math Standard Kalman filter and DETR mathematics apply without modification.
    The paper builds on Aeberhard's AKF [8] and Carion et al.'s DETR [14] and does not re-derive their assumptions.
  • ad hoc to paper The data exclusion rules do not systematically favor the learning-based method.
    The 5th-percentile confidence filter and single-sided lidar sample exclusion are introduced specifically for this dataset (Section IV-C) and could alter relative performance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HiLO: High-Level Object Fusion for Autonomous Driving using Transformers." pith.science (2026). https://pith.science/paper/6CP7YQRZ

@misc{pith2026250602554,
  author       = {Pith},
  title        = {Pith review of: HiLO: High-Level Object Fusion for Autonomous Driving using Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6CP7YQRZ}},
  note         = {Machine review of arXiv:2506.02554}
}
abstract

The fusion of sensor data is essential for a robust perception of the environment in autonomous driving. Learning-based fusion approaches mainly use feature-level fusion to achieve high performance, but their complexity and hardware requirements limit their applicability in near-production vehicles. High-level fusion methods offer robustness with lower computational requirements. Traditional methods, such as the Kalman filter, dominate this area. This paper modifies the Adapted Kalman Filter (AKF) and proposes a novel transformer-based high-level object fusion method called HiLO. Experimental results demonstrate improvements of $25.9$ percentage points in $\textrm{F}_1$ score and $6.1$ percentage points in mean IoU. Evaluation on a new large-scale real-world dataset demonstrates the effectiveness of the proposed approaches. Their generalizability is further validated by cross-domain evaluation between urban and highway scenarios. Code, data, and models are available at https://github.com/rst-tu-dortmund/HiLO .

Figures

Figures reproduced from arXiv: 2506.02554 by the authors.

Figure 1
Figure 1. Overview of the cross-domain evaluation of high [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of the HiLO architecture. The x and y coordinates of all detections in Ss,tA are augmented with additional features using sinusoidal em￾beddings at two frequencies in the position encoder to enhance the positional accuracy of the model estimates [21]. HiLO employs an encoder-decoder architecture, with a multi￾layer perceptron (MLP) transforming the input into a high￾dimensional feature space: Fobj = fMLP(ca… view at source ↗
Figure 3
Figure 3. Comparison of the object positions and class distri [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: BEV on qualitative results of the fusion approaches [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

30 extracted references · 27 canonical work pages

  1. [1]

    ”Uncertainty-aware model-based offline reinforcement learning for automated driving.” RA-L, 2023

    Diehl, C., et al. ”Uncertainty-aware model-based offline reinforcement learning for automated driving.” RA-L, 2023

  2. [2]

    ”Multi-modality 3D object detection in autonomous driving: A review.” Neurocomputing, 2023

    Tang, Y ., et al. ”Multi-modality 3D object detection in autonomous driving: A review.” Neurocomputing, 2023

  3. [3]

    ”Multi-sensor data fusion for real-time multi-object tracking.” Processes 11.2: 501, 2023

    Senel, N., et al. ”Multi-sensor data fusion for real-time multi-object tracking.” Processes 11.2: 501, 2023

  4. [4]

    ”Recent Advances in Multi-modal 3D Scene Under- standing: A Comprehensive Survey and Evaluation.” arXiv preprint arXiv:2310.15676, 2023

    Lei, Y ., et al. ”Recent Advances in Multi-modal 3D Scene Under- standing: A Comprehensive Survey and Evaluation.” arXiv preprint arXiv:2310.15676, 2023

  5. [5]

    ”Transformer-based sensor fusion for autonomous driving: A survey.” ICCV , 2023

    Singh, A. ”Transformer-based sensor fusion for autonomous driving: A survey.” ICCV , 2023

  6. [6]

    ”3d multi-object tracking: A baseline and new evaluation metrics.” IROS, 2020

    Weng, X., et al. ”3d multi-object tracking: A baseline and new evaluation metrics.” IROS, 2020

  7. [7]

    ”Simpletrack: Understanding and rethinking 3d multi- object tracking.” ECCV , 2022

    Pang, Z., et al. ”Simpletrack: Understanding and rethinking 3d multi- object tracking.” ECCV , 2022

  8. [8]

    ”Object-level fusion for surround environment percep- tion in automated driving applications”

    Aeberhard, M. ”Object-level fusion for surround environment percep- tion in automated driving applications”. VDI Verlag, 2017

Show all 30 references
  1. [9]

    Nilsson, S., and A. Klekamp. ”A comparison of architectures for track fusion.” ITSC, 2015

  2. [10]

    ”Vehicle detection, tracking and behavior analysis in urban driving environments using road context.” ICRA, 2018

    Verma, S., et al. ”Vehicle detection, tracking and behavior analysis in urban driving environments using road context.” ICRA, 2018

  3. [11]

    Shrivastava

    Andert, E., and A. Shrivastava. ”Accurate cooperative sensor fusion by parameterized covariance generation for sensing and localization pipelines in cavs.” ITSC, 2022

  4. [12]

    ”nuscenes: A multimodal dataset for autonomous driving.” CVPR, 2020

    Caesar, H., et al. ”nuscenes: A multimodal dataset for autonomous driving.” CVPR, 2020

  5. [13]

    ”Multi-modal sensor fusion and object tracking for autonomous racing.” IEEE Trans

    Karle, P., et al. ”Multi-modal sensor fusion and object tracking for autonomous racing.” IEEE Trans. on Intelligent Vehicles, 2023

  6. [14]

    ”End-to-end object detection with transformers.” ECCV , 2020

    Carion, N., et al. ”End-to-end object detection with transformers.” ECCV , 2020

  7. [15]

    ”Bevfusion: A simple and robust lidar-camera fusion framework.” NeurIPS, 2022

    Liang, T., et al. ”Bevfusion: A simple and robust lidar-camera fusion framework.” NeurIPS, 2022

  8. [16]

    ”A comparative review on multi-modal sensors fusion based on deep learning.” Signal Processing, 2023

    Tang, Q., et al. ”A comparative review on multi-modal sensors fusion based on deep learning.” Signal Processing, 2023

  9. [17]

    ”Deformable detr: Deformable transformers for end-to- end object detection.” arXiv preprint arXiv:2010.04159, 2020

    Zhu, X., et al. ”Deformable detr: Deformable transformers for end-to- end object detection.” arXiv preprint arXiv:2010.04159, 2020

  10. [18]

    ”UniTraj: A Unified Framework for Scalable Vehicle Trajectory Prediction.” arXiv preprint arXiv:2403.15098, 2024

    Feng, L., et al. ”UniTraj: A Unified Framework for Scalable Vehicle Trajectory Prediction.” arXiv preprint arXiv:2403.15098, 2024

  11. [19]

    ”Transformer-based models for automatic iden- tification of argument relations: A cross-domain evaluation.” IEEE Intelligent Systems, 2021

    Ruiz-Dolz, R., et al. ”Transformer-based models for automatic iden- tification of argument relations: A cross-domain evaluation.” IEEE Intelligent Systems, 2021

  12. [20]

    ”Deep transfer learning for intelligent vehicle perception: A survey.” Green Energy and Intelligent Transportation, 2023

    Liu, X., et al. ”Deep transfer learning for intelligent vehicle perception: A survey.” Green Energy and Intelligent Transportation, 2023

  13. [21]

    ”IR-MCL: Implicit Representation-Based Online Global Localization.” RA-L, 2023

    Kuang, H., et al. ”IR-MCL: Implicit Representation-Based Online Global Localization.” RA-L, 2023

  14. [22]

    ”The Hungarian method for the assignment problem.” NRL, 1955

    Kuhn, H.W. ”The Hungarian method for the assignment problem.” NRL, 1955

  15. [23]

    ”Generalized Intersection Over Union: A Metric and a Loss for Bounding Box Regression.” CVPR, 2019

    Seyed Hamid, R., et al. ”Generalized Intersection Over Union: A Metric and a Loss for Bounding Box Regression.” CVPR, 2019

  16. [24]

    S., et al

    Aurelio, Y . S., et al. ”Learning from imbalanced data sets with weighted cross-entropy function.” Neural Processing Letters 50, 2019

  17. [25]

    Blackman and R

    S. Blackman and R. Popoli, ”Design and Analysis of Modern Tracking Systems”. Norwood, MA: Artech House, 1999

  18. [26]

    Bertsekas, D. P. ”Auction Algorithms.” Encyclopedia of Optimization 1, 2009

  19. [27]

    ”A comparative analysis of object detection metrics with a companion open-source toolkit.” Electronics 10.3, 2021

    Padilla, R., et al. ”A comparative analysis of object detection metrics with a companion open-source toolkit.” Electronics 10.3, 2021

  20. [28]

    H., and G

    Victoria, A. H., and G. Maragatham. ”Automatic tuning of hyperpa- rameters using Bayesian optimization.” Evolving Systems 12.1, 2021

  21. [29]

    ”Class-balanced loss based on effective number of samples.” CVPR, 2019

    Cui, Y ., et al. ”Class-balanced loss based on effective number of samples.” CVPR, 2019

  22. [30]

    ”Ms3d: Leveraging multiple detectors for unsupervised domain adaptation in 3d object detection.” ITSC, 2023

    Tsai, D., et al. ”Ms3d: Leveraging multiple detectors for unsupervised domain adaptation in 3d object detection.” ITSC, 2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.