REVIEW 3 major objections 5 minor 30 references
HiLO: High-Level Object Fusion for Autonomous Driving using Transformers
T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read A compact transformer that fuses camera and radar object lists beats Kalman-filter fusion by up to 25.9 points in F1-score on real driving data.
desk verdict Useful transformer-for-high-level-fusion idea and a big new dataset, but the headline F1 gap is computed on a test set filtered with its own annotations, so the quantitative claim is not yet established. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a DETR-inspired encoder-decoder transformer. The encoder runs self-attention (called multi-modal attention) over all sensor detections, whose (x,y) coordinates are augmented with sinusoidal positional embeddings; the decoder uses a fixed set of 20 learnable query vectors to attend to the encoded features and directly output fused bounding boxes, class logits, and a "no-object" class for empty queries. Training uses Hungarian matching between predictions and ground-truth annotations, with a loss that combines weighted cross-entropy, L1 box loss, generalized IoU, and an orientation loss. This replaces the Kalman filter's explicit geometric inflation, Mahalanobis gating, and hand-tuned covariance updates with a learned association and fusion step.
What would settle it
Recompute F1 and mIoU on the full, unfiltered test set and on the subset of samples excluded by the confidence-threshold and single-sided-lidar rules; if HiLO's advantage over AKFA shrinks or reverses on those excluded samples, the headline gains depend on the filtering choices.
Extended reading notes
Core claim
The paper's central claim is that high-level object fusion—joining object lists from multiple sensors into one detection set—can be solved with a compact DETR-derived transformer, and that this learned approach substantially outperforms classical Kalman-filter fusion on real-world data. On its new dataset, HiLO reaches 69.1% F1 on highway and 44.3% on urban test splits, against 46.7% and 29.4% for the best Kalman baseline (AKFA), while the mean IoU improves by up to 6.1 percentage points. The paper attributes the gains to attention-based association across sensors and joint estimation of position, extent, class, and orientation, which it argues replace the hand-tuned geometric association and covariance modeling that classical filters rely on.
Load-bearing premise
The evaluation assumes that filtering out low-confidence detections and omitting samples with single-sided lidar coverage does not remove systematically harder cases; if those cases are harder, the reported F1 and mIoU improvements may be inflated.
Editorial extensions
If this is right
- If the results hold, high-level fusion built from camera and radar object lists can approach feature-level fusion performance at a fraction of the compute, making it viable for near-production vehicles.
- The classical Adapted Kalman Filter is itself improved by the paper's covariance-inflation modification (AKFA), which raises F1 by several points, suggesting that tuning measurement covariance per sensor is a low-cost step for existing systems.
- Domain-specific training outperforms combined-domain training, so deploying on a new environment will likely require domain adaptation or more model capacity rather than simply more data.
- Cross-domain evaluation, common in other machine-learning fields, proves informative here: the transformer loses 7.3 and 21.4 F1 points across the urban-highway shift, yet still beats the Kalman baselines, indicating that learned fusion retains an advantage even under distribution shift.
Reading between the lines
- The filtering step (removing detections below a 5th-percentile confidence threshold and dropping single-sided-lidar samples) may selectively eliminate the hardest association cases, which could favor the learned method; rerunning on the fully unfiltered test stream would test whether the headline gains persist.
- Because HiLO consumes object lists rather than raw sensor data, the same architecture could be retrained for other sensor stacks (for example, lidar plus camera, or infrastructure-provided detections), without changing the model family.
- The large cross-domain drop from urban to highway suggests that a relatively small domain-adaptation wrapper, as the authors hint, might recover most of the gap with far less data than joint training.
- Temporal information is currently unused; feeding the model a short history of fused objects, which the paper lists as future work, could address occlusions and intermittent detections and may close much of the remaining gap to feature-level fusion.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes HiLO, a DETR-style transformer for high-level fusion of camera and radar object tracks, and AKFA, an Adapted Kalman Filter variant with an additional per-sensor measurement covariance. It introduces a large self-collected real-world dataset of roughly 1.85 million samples from urban and highway domains, and reports inter-domain and cross-domain results for AKF, AKFA, and HiLO. The headline claim is an F1-score improvement of up to 25.9 percentage points and an mIoU improvement of up to 6.1 percentage points over the Kalman-filter baselines. The manuscript also reports runtime and memory measurements and states that code, data, and models are publicly available.
Significance. If the empirical comparison is valid, HiLO would be a practically relevant high-level fusion approach with a lightweight transformer architecture and reasonable CPU runtime. The paper has clear strengths: a new large-scale real-world dataset, a cross-domain evaluation protocol, a strong classical baseline with dedicated tuning, public code/data/model release, and explicit runtime figures. The central empirical claim, however, rests on an annotation-guided filtering procedure applied to the test set and on single-run evaluations without uncertainty estimates; these issues need to be addressed before the claimed improvements can be taken as established.
major comments (3)
- [IV-C] The filtering rule is annotation-defined. The confidence threshold is set at the 5th percentile of camera and radar detections that overlap with annotations, and detections are then removed if no close annotation exists. This removes exactly the low-confidence false positives (which by definition have no matching annotation) while retaining low-confidence true positives, which mechanically inflates precision and F1 for every method. Since HiLO is trained on the filtered data while AKF/AKFA only consume the filtered sensor tracks as measurements, the magnitude of the effect can differ across methods. The headline 25.9 pp F1 gain and the mIoU gains in Table I are computed on this filtered test set. Please report results on the unfiltered test set, or use a filter that does not consult annotations, and quantify how many samples and detections are removed by each filtering step, including the single-sided-lidar exclusion.
- [V, Table I] No variance measures, confidence intervals, or repeated evaluations are reported. Hyperparameter selection uses a stochastic Bayesian search (10 runs per hyperparameter) and HiLO training is stochastic, yet every number in Table I appears to come from a single run. Please report means and standard deviations over multiple training seeds or bootstrap confidence intervals, and state whether small cross-domain gaps such as the urban-to-highway F1 difference between HiLO (47.7) and AKFA (45.9) are statistically meaningful.
- [V-C] The conclusion that cross-evaluated HiLO 'still outperforms the classical approaches' is only weakly supported for the urban-to-highway direction, where the F1 margin over AKFA is 1.8 percentage points and the mIoU margin is 1.7 percentage points; without uncertainty estimates, this specific claim is not established. Please either provide significance testing for all pairwise comparisons or soften this conclusion for the cross-domain cases with small margins.
minor comments (5)
- [Table I] The table header uses 'AFKA' for the method that is otherwise called 'AKFA'; the abbreviation should be consistent.
- [V-C and III] There are typos: 'precent points' in Section V-C, 'classical approch' in Section III, and 'attetion' in the Related Work should be 'attention'.
- [IV-B] In Equation (3), the Kalman gain uses the notation P_hat{y}'_{j-1,j,I(k)} with a prime that is not defined; please clarify whether this is a typo for the predicted covariance P_hat{y}.
- [V-A] The sentence 'HiLO achieves the highest results across all metrics, indicating classical approaches may neglect important effects in their modeling' goes beyond the evidence; experiments show a performance difference but do not identify the modeling effects responsible.
- [VI] The acknowledged limitation of automatic annotation quality is helpful, but it also underscores the need for the unfiltered evaluation requested in the major comments; please make the connection explicit.
Circularity Check
Headline F1/mIoU gaps are computed on a test set cleaned by the test annotations themselves (low-confidence detections 'without close annotation' are deleted), so the reported numbers are partially defined by the evaluation labels; the model derivation and the baselines themselves remain independent.
-
self definitional
[Section IV-C (Data Collection and Filtering); values reported in Abstract, Table I, and Section VI Conclusion]
"To mitigate the impact of annotation errors on fusion training and evaluation, filtering was applied. A confidence score threshold was set at the 5th percentile of camera and radar detections overlapping with annotations, removing these detections from the dataset if no close annotation existed. This threshold established an upper bound for the classical approach's confidence score."
The filter's removal rule ('no close annotation existed') is exactly the detection-to-annotation correspondence that the evaluation uses to define a false positive (IoU matching against annotations, threshold 0.5, Section V Metrics). The test input is therefore selected by reference to the same labels that define F1 and mIoU: low-confidence detections that would otherwise be counted as false positives are deleted before any fusion method can emit them, and the AKF/AKFA confidence threshold is capped by this same annotation-derived bound. Thus the headline improvements (25.9 pp F1, 6.1 pp mIoU) are measured on an annotation-selected subset, entangling the reported performance with the evaluation labels rather than purely with fusion quality.
full rationale
The central derivation chain is otherwise self-contained and independent. HiLO is a standard DETR-style encoder-decoder (external components: DETR [14], Hungarian matching [22], gIoU [23]), trained on 75/15/10 splits grouped by recording date, with hyperparameters selected on the validation set and best-validation checkpoint reported on the test set. The AKF baseline is external (Aeberhard [8]); the AKFA extension's per-sensor additional covariance is tuned on the validation set, which is ordinary baseline tuning, not a fitted parameter renamed as a prediction. All reported test metrics are computed from held-out forward passes, so no number in Table I equals a fitted parameter by construction. The only self-citation ([1] Diehl et al.) supports a generic statement about planning/situation prediction and is not load-bearing for the fusion claim. No uniqueness theorem is imported from the authors' prior work, and no ansatz is smuggled in via citation. The one flagged concern is the annotation-defined input filter in Section IV-C: its removal criterion is the metric's own false-positive criterion, and the claimed 25.9/6.1 percentage-point gaps are produced on the resulting label-selected subset. The direction of any resulting bias is not established - the cleaning could plausibly help the classical baselines more than HiLO - but the absolute values and the classical confidence bound are entangled with the evaluation labels, so the strong claim that 'classical approaches may neglect important effects in their modeling' (Section V-A) rests on a partially self-defined comparison. This warrants score 4: a real, quotable self-definitional step in the evaluation, while the core model-versus-baseline comparison retains independent empirical content.
Assumptions & free parameters
free parameters (6)
- AKFA per-sensor measurement covariance =
Tuned on validation set, value not reported
- AKF/AKFA existence score thresholds =
Tuned per sensor, values not reported
- Process noise covariance Q =
Tuned, values not reported
- Association parameters alpha and epsilon =
Tuned, values not reported
- HiLO loss and matching weights =
Tuned, values not reported
- Data filtering confidence threshold =
5th percentile of detections overlapping annotations
assumptions (4)
- domain assumption The automatic offline annotation tool provides accurate ground truth.
- domain assumption The tracked object lists from the camera and radar ECUs are valid and sufficient inputs for high-level fusion.
- standard math Standard Kalman filter and DETR mathematics apply without modification.
- ad hoc to paper The data exclusion rules do not systematically favor the learning-based method.
Cite this review
Pith. "Pith review of HiLO: High-Level Object Fusion for Autonomous Driving using Transformers." pith.science (2026). https://pith.science/paper/6CP7YQRZ
@misc{pith2026250602554,
author = {Pith},
title = {Pith review of: HiLO: High-Level Object Fusion for Autonomous Driving using Transformers},
year = {2026},
howpublished = {\url{https://pith.science/paper/6CP7YQRZ}},
note = {Machine review of arXiv:2506.02554}
}
abstract
The fusion of sensor data is essential for a robust perception of the environment in autonomous driving. Learning-based fusion approaches mainly use feature-level fusion to achieve high performance, but their complexity and hardware requirements limit their applicability in near-production vehicles. High-level fusion methods offer robustness with lower computational requirements. Traditional methods, such as the Kalman filter, dominate this area. This paper modifies the Adapted Kalman Filter (AKF) and proposes a novel transformer-based high-level object fusion method called HiLO. Experimental results demonstrate improvements of $25.9$ percentage points in $\textrm{F}_1$ score and $6.1$ percentage points in mean IoU. Evaluation on a new large-scale real-world dataset demonstrates the effectiveness of the proposed approaches. Their generalizability is further validated by cross-domain evaluation between urban and highway scenarios. Code, data, and models are available at https://github.com/rst-tu-dortmund/HiLO .
Figures
Reference graph
Works this paper leans on
-
[1]
”Uncertainty-aware model-based offline reinforcement learning for automated driving.” RA-L, 2023
Diehl, C., et al. ”Uncertainty-aware model-based offline reinforcement learning for automated driving.” RA-L, 2023
work page 2023
-
[2]
”Multi-modality 3D object detection in autonomous driving: A review.” Neurocomputing, 2023
Tang, Y ., et al. ”Multi-modality 3D object detection in autonomous driving: A review.” Neurocomputing, 2023
work page 2023
-
[3]
”Multi-sensor data fusion for real-time multi-object tracking.” Processes 11.2: 501, 2023
Senel, N., et al. ”Multi-sensor data fusion for real-time multi-object tracking.” Processes 11.2: 501, 2023
work page 2023
-
[4]
Lei, Y ., et al. ”Recent Advances in Multi-modal 3D Scene Under- standing: A Comprehensive Survey and Evaluation.” arXiv preprint arXiv:2310.15676, 2023
arXiv 2023
-
[5]
”Transformer-based sensor fusion for autonomous driving: A survey.” ICCV , 2023
Singh, A. ”Transformer-based sensor fusion for autonomous driving: A survey.” ICCV , 2023
work page 2023
-
[6]
”3d multi-object tracking: A baseline and new evaluation metrics.” IROS, 2020
Weng, X., et al. ”3d multi-object tracking: A baseline and new evaluation metrics.” IROS, 2020
work page 2020
-
[7]
”Simpletrack: Understanding and rethinking 3d multi- object tracking.” ECCV , 2022
Pang, Z., et al. ”Simpletrack: Understanding and rethinking 3d multi- object tracking.” ECCV , 2022
work page 2022
-
[8]
”Object-level fusion for surround environment percep- tion in automated driving applications”
Aeberhard, M. ”Object-level fusion for surround environment percep- tion in automated driving applications”. VDI Verlag, 2017
work page 2017
Show all 30 references
-
[9]
Nilsson, S., and A. Klekamp. ”A comparison of architectures for track fusion.” ITSC, 2015
2015
-
[10]
”Vehicle detection, tracking and behavior analysis in urban driving environments using road context.” ICRA, 2018
Verma, S., et al. ”Vehicle detection, tracking and behavior analysis in urban driving environments using road context.” ICRA, 2018
2018
-
[11]
Shrivastava
Andert, E., and A. Shrivastava. ”Accurate cooperative sensor fusion by parameterized covariance generation for sensing and localization pipelines in cavs.” ITSC, 2022
2022
-
[12]
”nuscenes: A multimodal dataset for autonomous driving.” CVPR, 2020
Caesar, H., et al. ”nuscenes: A multimodal dataset for autonomous driving.” CVPR, 2020
2020
-
[13]
”Multi-modal sensor fusion and object tracking for autonomous racing.” IEEE Trans
Karle, P., et al. ”Multi-modal sensor fusion and object tracking for autonomous racing.” IEEE Trans. on Intelligent Vehicles, 2023
2023
-
[14]
”End-to-end object detection with transformers.” ECCV , 2020
Carion, N., et al. ”End-to-end object detection with transformers.” ECCV , 2020
2020
-
[15]
”Bevfusion: A simple and robust lidar-camera fusion framework.” NeurIPS, 2022
Liang, T., et al. ”Bevfusion: A simple and robust lidar-camera fusion framework.” NeurIPS, 2022
2022
-
[16]
”A comparative review on multi-modal sensors fusion based on deep learning.” Signal Processing, 2023
Tang, Q., et al. ”A comparative review on multi-modal sensors fusion based on deep learning.” Signal Processing, 2023
2023
-
[17]
”Deformable detr: Deformable transformers for end-to- end object detection.” arXiv preprint arXiv:2010.04159, 2020
Zhu, X., et al. ”Deformable detr: Deformable transformers for end-to- end object detection.” arXiv preprint arXiv:2010.04159, 2020
2010 arXiv
-
[18]
”UniTraj: A Unified Framework for Scalable Vehicle Trajectory Prediction.” arXiv preprint arXiv:2403.15098, 2024
Feng, L., et al. ”UniTraj: A Unified Framework for Scalable Vehicle Trajectory Prediction.” arXiv preprint arXiv:2403.15098, 2024
2024 arXiv
-
[19]
”Transformer-based models for automatic iden- tification of argument relations: A cross-domain evaluation.” IEEE Intelligent Systems, 2021
Ruiz-Dolz, R., et al. ”Transformer-based models for automatic iden- tification of argument relations: A cross-domain evaluation.” IEEE Intelligent Systems, 2021
2021
-
[20]
”Deep transfer learning for intelligent vehicle perception: A survey.” Green Energy and Intelligent Transportation, 2023
Liu, X., et al. ”Deep transfer learning for intelligent vehicle perception: A survey.” Green Energy and Intelligent Transportation, 2023
2023
-
[21]
”IR-MCL: Implicit Representation-Based Online Global Localization.” RA-L, 2023
Kuang, H., et al. ”IR-MCL: Implicit Representation-Based Online Global Localization.” RA-L, 2023
2023
-
[22]
”The Hungarian method for the assignment problem.” NRL, 1955
Kuhn, H.W. ”The Hungarian method for the assignment problem.” NRL, 1955
1955
-
[23]
”Generalized Intersection Over Union: A Metric and a Loss for Bounding Box Regression.” CVPR, 2019
Seyed Hamid, R., et al. ”Generalized Intersection Over Union: A Metric and a Loss for Bounding Box Regression.” CVPR, 2019
2019
-
[24]
S., et al
Aurelio, Y . S., et al. ”Learning from imbalanced data sets with weighted cross-entropy function.” Neural Processing Letters 50, 2019
2019
-
[25]
Blackman and R
S. Blackman and R. Popoli, ”Design and Analysis of Modern Tracking Systems”. Norwood, MA: Artech House, 1999
1999
-
[26]
Bertsekas, D. P. ”Auction Algorithms.” Encyclopedia of Optimization 1, 2009
2009
-
[27]
”A comparative analysis of object detection metrics with a companion open-source toolkit.” Electronics 10.3, 2021
Padilla, R., et al. ”A comparative analysis of object detection metrics with a companion open-source toolkit.” Electronics 10.3, 2021
2021
-
[28]
H., and G
Victoria, A. H., and G. Maragatham. ”Automatic tuning of hyperpa- rameters using Bayesian optimization.” Evolving Systems 12.1, 2021
2021
-
[29]
”Class-balanced loss based on effective number of samples.” CVPR, 2019
Cui, Y ., et al. ”Class-balanced loss based on effective number of samples.” CVPR, 2019
2019
-
[30]
”Ms3d: Leveraging multiple detectors for unsupervised domain adaptation in 3d object detection.” ITSC, 2023
Tsai, D., et al. ”Ms3d: Leveraging multiple detectors for unsupervised domain adaptation in 3d object detection.” ITSC, 2023
2023
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.