pith. sign in

arxiv: 2606.19267 · v1 · pith:RC4RMIUJnew · submitted 2026-06-17 · 💻 cs.RO · cs.SY· eess.SY

A Mixed-Reality Testbed for Autonomous Vehicles

H. M. Sabbir Ahmad , Ehsan Sabouni , Emrullah Celik , Zean Wan , Damola Ajeyemi , Christos G. Cassandras , Wenchao Li This is my paper

Pith reviewed 2026-06-26 21:03 UTC · model grok-4.3

classification 💻 cs.RO cs.SYeess.SY
keywords mixed-realityhardware-in-the-loopautonomous vehiclescontrol barrier functionsconnected autonomous vehiclestestbedsimulationmulti-agent systems
0
0 comments X

The pith

A mixed-reality hardware-in-the-loop testbed integrates physical mobile robots with high-fidelity virtual simulations to validate autonomous vehicle algorithms.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to show that blending physical robots equipped with multimodal sensors into photorealistic virtual environments creates a practical way to test connected autonomous vehicle systems in complex, safety-critical scenarios. This matters because full physical testing carries high risk and cost while pure simulation often fails to capture real hardware behavior. The testbed adds wireless connectivity for multi-agent work and includes a framework that combines perception, planning, and an online learning controller based on control barrier functions to enforce safety. Experiments confirm the setup can move algorithms from virtual testing to hardware deployment while handling large numbers of agents through the mix of real and simulated entities.

Core claim

The authors establish a mixed-reality hardware-in-the-loop testbed that seamlessly combines a physical testbed of mobile robots with multimodal sensors operating in high-fidelity simulation environments, supports vehicular connectivity, accommodates large numbers of agents through physical and virtual combinations, and includes a safety-guaranteed framework that integrates perception, planning, and a novel online learning-based controller using Control Barrier Functions for connected autonomous vehicles.

What carries the argument

The mixed-reality HIL testbed that places physical robots with sensors into photorealistic virtual environments, together with the Control Barrier Function-based online learning controller that enforces safety during perception, planning, and control.

If this is right

  • Validation of perception, planning, and control algorithms occurs in diverse safety-critical driving scenarios created in the virtual environment.
  • Research on multi-agent systems including connected autonomous vehicles proceeds with wireless communication and scalable agent counts.
  • The testbed supports experiments that demonstrate bridging simulation to real-world hardware deployment.
  • Safety guarantees hold through the combination of the physical robots and the CBF-based controller in mixed settings.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The hybrid setup could shorten AV development cycles by letting teams iterate on edge cases without full physical risk.
  • If latency stays low, the same physical-virtual scaling might apply to testing other autonomous systems such as delivery robots.
  • Standard test protocols for regulatory review of vehicle behaviors could draw on this mixed-reality pattern.
  • Multi-lab collaborative experiments become feasible when one site supplies the physical robots and another supplies additional virtual agents.

Load-bearing premise

The physical-virtual integration introduces no significant latency, synchronization errors, or communication delays that would undermine the safety guarantees of the control barrier functions.

What would settle it

An experiment in which a safety-critical maneuver produces a control barrier function violation traceable to measured integration latency or desynchronization between the physical robot and the virtual environment.

Figures

Figures reproduced from arXiv: 2606.19267 by Christos G. Cassandras, Damola Ajeyemi, Ehsan Sabouni, Emrullah Celik, H. M. Sabbir Ahmad, Wenchao Li, Zean Wan.

Figure 1
Figure 1. Figure 1: Overview of the mixed-reality HIL testbed using the CARLA simulator and physical mobile robots. The testbed [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Data flow of the mixed-reality testbed. virtual vehicle and digital twin runs an individual ROS node on the same PC that hosts the RSU and the CARLA server. Both the physical robots and the CARLA agents continuously publish their state information—such as position, velocity, and attitude—to the RSU. Based on the coordination policy, the RSU disseminates relevant information about surrounding agents to each… view at source ↗
Figure 3
Figure 3. Figure 3: End-to-end framework comprising of multimodal sensing, sensor fusion, state estimation and CBF-based controller. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Online self-supervised algorithm training result in [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Plot of training loss with robot data for 5 seeds. With [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗
Figure 5
Figure 5. Figure 5: Training and validation results for Sensor Fusion. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: Success rate under different weather conditions [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗
read the original abstract

We propose a mixed-reality, hardware-in-the-loop (HIL) testbed for autonomous vehicles that seamlessly integrates a physical testbed of mobile robots with a high-fidelity simulation environment. The virtual simulation enables the creation of diverse, safety-critical driving scenarios to validate state-of-the-art perception, planning, and control algorithms, while augmenting simulations with physical robots equipped with multimodal sensors in photorealistic virtual environments further facilitating rigorous validation. Our testbed also features vehicular connectivity using wireless communication and can accommodate a large number of agents through the combination of physical robots and virtual simulated agents, supporting research on multi-agent systems including Connected and Autonomous Vehicles (CAVs). Finally, we present a safety-guaranteed framework combining perception, planning and a novel online learning-based controller using Control Barrier Functions (CBFs) for CAVs. Experiments using the proposed framework are used to validate and demonstrate the key functionalities and the overall utility of the testbed to bridge the gap between simulation and real-world hardware deployment.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 0 minor

Summary. The manuscript proposes a mixed-reality hardware-in-the-loop (HIL) testbed for autonomous vehicles that integrates a physical testbed of mobile robots equipped with multimodal sensors into a high-fidelity simulation environment. The virtual component enables diverse safety-critical scenarios, while physical robots in photorealistic settings support rigorous validation; the testbed also incorporates wireless V2X connectivity and scales to many agents via mixed physical-virtual setups for multi-agent CAV research. A safety-guaranteed framework is presented that combines perception, planning, and a novel online learning-based controller using Control Barrier Functions (CBFs). Experiments are stated to validate the testbed's key functionalities and utility in bridging simulation and real-world deployment.

Significance. If the HIL integration proves robust and the CBF safety guarantees hold under deployed conditions, the testbed would offer a useful platform for scalable validation of perception-planning-control pipelines in multi-agent CAV settings, particularly by allowing controlled introduction of physical hardware into otherwise simulated scenarios. The explicit combination of physical robots and virtual agents is a constructive feature for studying connectivity and coordination.

major comments (2)
  1. [Abstract] Abstract: The central claim of a 'safety-guaranteed framework' using CBFs is load-bearing, yet the manuscript contains no analysis, extension (e.g., input-to-state safety), or measurements addressing variable latency, clock skew, or rendering-to-sensor mismatch arising from wireless links, photorealistic rendering, and mixed physical-virtual agents. Without such handling, the theoretical CBF certificates (η(x) ≥ 0 implying u ∈ K(x)) do not necessarily transfer to the HIL closed loop.
  2. [Abstract] Abstract (validation paragraph): The statement that 'Experiments using the proposed framework are used to validate...' is load-bearing for demonstrating utility, but the manuscript provides no quantitative results, timing data, safety-violation metrics, or error analysis from those experiments, preventing assessment of whether unmodeled HIL effects remain within robustness margins.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive and detailed feedback on our manuscript. We address the two major comments point by point below, acknowledging where the current version falls short and outlining the revisions we will make.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The central claim of a 'safety-guaranteed framework' using CBFs is load-bearing, yet the manuscript contains no analysis, extension (e.g., input-to-state safety), or measurements addressing variable latency, clock skew, or rendering-to-sensor mismatch arising from wireless links, photorealistic rendering, and mixed physical-virtual agents. Without such handling, the theoretical CBF certificates (η(x) ≥ 0 implying u ∈ K(x)) do not necessarily transfer to the HIL closed loop.

    Authors: We agree that the manuscript does not provide explicit analysis or measurements of HIL-induced effects such as variable latency, clock skew, or rendering-to-sensor mismatch on the CBF safety guarantees. The framework is presented with theoretical CBF certificates, but these HIL-specific factors are not addressed. In the revised manuscript we will add a dedicated discussion subsection on these issues, including potential robustness margins, conservative design choices, and any available preliminary measurements from the testbed hardware. revision: yes

  2. Referee: [Abstract] Abstract (validation paragraph): The statement that 'Experiments using the proposed framework are used to validate...' is load-bearing for demonstrating utility, but the manuscript provides no quantitative results, timing data, safety-violation metrics, or error analysis from those experiments, preventing assessment of whether unmodeled HIL effects remain within robustness margins.

    Authors: The experiments section demonstrates the testbed functionalities and framework through a combination of qualitative demonstrations and basic quantitative validation. However, we acknowledge that detailed timing data, safety-violation counts, and error analysis specifically quantifying HIL effects are not reported. In the revision we will expand the experimental results with additional quantitative metrics, timing measurements, and safety-related statistics drawn from the existing experiment logs. revision: yes

Circularity Check

0 steps flagged

No circularity in testbed proposal or CBF framework

full rationale

The paper describes an experimental mixed-reality HIL testbed that integrates physical robots with simulation and presents a CBF-based controller framework validated through experiments. No derivation chain reduces any claim to self-definition, fitted parameters renamed as predictions, or load-bearing self-citations. The safety framework combines perception, planning, and CBFs as an independent construction whose utility is demonstrated externally via hardware experiments rather than by tautological redefinition of its inputs.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The claim rests on standard domain assumptions in control theory and robotics with no free parameters, invented entities, or ad-hoc axioms detailed in the abstract.

axioms (1)
  • domain assumption Control Barrier Functions provide safety guarantees when combined with perception and planning for CAVs
    Invoked in the safety-guaranteed framework description without further justification in the abstract.

pith-pipeline@v0.9.1-grok · 5733 in / 1380 out tokens · 38927 ms · 2026-06-26T21:03:26.623352+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

19 extracted references

  1. [1]

    Carla: An open urban driving simulator,

    A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V . Koltun, “Carla: An open urban driving simulator,” inProceedings of the 1st Annual Conference on Robot Learning. PMLR, 2017, pp. 1–16

    2017

  2. [2]

    A survey on small-scale testbeds for connected and auto- mated vehicles and robot swarms,

    A. Mokhtarian, J. Xu, P. Scheffe, M. Kloock, S. Schäfer, H. Bang, V .-A. Le, S. Ulhas, J. Betz, S. Wilson, S. Berman, L. Paull, A. Prorok, and B. Alrifaee, “A survey on small-scale testbeds for connected and auto- mated vehicles and robot swarms,”arXiv preprint arXiv:2408.14199, 2024

    arXiv 2024

  3. [3]

    A scaled smart city for experimental validation of connected and automated vehicles,

    A. Stager, L. Bhan, A. Malikopoulos, and L. Zhao, “A scaled smart city for experimental validation of connected and automated vehicles,” IF AC-PapersOnLine, vol. 51, no. 9, pp. 130–135, 2018

    2018

  4. [4]

    A research and educational robotic testbed for real- time control of emerging mobility systems: From theory to scaled experiments,

    B. Chalaki, L. E. Beaver, A. M. I. Mahbub, H. Bang, and A. A. Malikopoulos, “A research and educational robotic testbed for real- time control of emerging mobility systems: From theory to scaled experiments,”arXiv preprint arXiv:2109.05370, 2021

    arXiv 2021

  5. [5]

    Duckietown: an open, inexpensive and flexible platform for autonomy education and research,

    L. Paull, J. Tani, H. Ahn, J. Alonso-Mora, L. Carlone, M. Cap, C. De Wagter, A. Dosovitskiy, R. Dubé, V . Espinozaet al., “Duckietown: an open, inexpensive and flexible platform for autonomy education and research,”2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 1497–1504, 2017

    2017

  6. [6]

    The robotarium: A remotely accessible swarm robotics research testbed,

    D. Pickem, P. Glotfelter, L. Wang, M. Mote, A. D. Ames, E. Feron, and M. Egerstedt, “The robotarium: A remotely accessible swarm robotics research testbed,”2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 1699–1706, 2017

    2017

  7. [7]

    A scaled experiment platform to study interactions between humans and cavs,

    P. Scheffe and B. Alrifaee, “A scaled experiment platform to study interactions between humans and cavs,” in2018 IEEE Intelligent V ehicles Symposium (IV). IEEE, 2018, pp. 1939–1944

    2018

  8. [8]

    A fleet of miniature cars for experiments in cooperative driving,

    N. Hyldmar, Y . He, and A. Prorok, “A fleet of miniature cars for experiments in cooperative driving,” in2019 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2019, pp. 3238–3244

    2019

  9. [9]

    Development of sensors testbed for autonomous vehicles,

    J. M. Vargas, S. Alsweiss, M. Jernigan, A. B. Amin, M. Brinkmann, J. Santos, and R. Razdan, “Development of sensors testbed for autonomous vehicles,” in2019 IEEE 9th Annual Computing and Communication Workshop and Conference (CCWC). IEEE, 2019, pp. 0574–0579

    2019

  10. [10]

    Real-is-sim: Bridging the sim-to-real gap with a dynamic digital twin for real-world robot policy evaluation,

    J. Abou-Chakra, L. Sun, K. Rana, B. May, K. Schmeckpeper, M. V . Minniti, and L. Herlant, “Real-is-sim: Bridging the sim-to-real gap with a dynamic digital twin for real-world robot policy evaluation,” arXiv preprint arXiv:2504.03597, 2025

    arXiv 2025

  11. [11]

    RoadRunner: Design 3D Scenes for Automated Driv- ing Simulation,

    I. MathWorks, “RoadRunner: Design 3D Scenes for Automated Driv- ing Simulation,” https://www.mathworks.com/products/roadrunner.html, 2025, accessed: Sep. 11, 2025

    2025

  12. [12]

    LIMO PRO: ROS / Gazebo Mobile Robot Platform with NVIDIA Orin Nano, T-mini Pro LiDAR, Orbbec Dabai Depth Camera,

    A. Robotics, “LIMO PRO: ROS / Gazebo Mobile Robot Platform with NVIDIA Orin Nano, T-mini Pro LiDAR, Orbbec Dabai Depth Camera,” https://global.agilex.ai/products/limo-pro, 2025, accessed: Sep. 11, 2025

    2025

  13. [13]

    Multi-view 3d object detection network for autonomous driving

    X. Chen, H. Ma, J. Wan, B. Li, and T. Xia, “Multi-view 3d object detection network for autonomous driving.”

  14. [14]

    W. Xiao, C. G. Cassandras, and C. Belta,Safe autonomy with control barrier functions: theory and applications. Springer, 2023

    2023

  15. [15]

    Control lyapunov functions and hybrid zero dynamics,

    A. D. Ames, K. Galloway, and J. W. Grizzle, “Control lyapunov functions and hybrid zero dynamics,” in2012 IEEE 51st IEEE Conference on Decision and Control (CDC), 2012, pp. 6837–6842

    2012

  16. [16]

    OptNet: Differentiable optimization as a layer in neural networks,

    B. Amos and J. Z. Kolter, “OptNet: Differentiable optimization as a layer in neural networks,” inProceedings of the 34th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 70. PMLR, 06–11 Aug 2017, pp. 136–145

    2017

  17. [17]

    Selective sensor fusion for neural visual-inertial odometry,

    C. Chen, S. Rosa, Y . Miao, C. X. Lu, W. Wu, A. Markham, and N. Trigoni, “Selective sensor fusion for neural visual-inertial odometry,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 10 542–10 551

    2019

  18. [18]

    Hmarl-cbf: Hierarchical multi-agent reinforcement learning with control barrier functions for safety-critical autonomous systems,

    H. M. S. Ahmad, E. Sabouni, A. Wasilkoff, P. Budhraja, Z. Guo, S. Zhang, C. Fan, C. G. Cassandras, and W. Li, “Hmarl-cbf: Hierarchical multi-agent reinforcement learning with control barrier functions for safety-critical autonomous systems,” inAdvances in Neural Information Processing Systems, 2025

    2025

  19. [19]

    Congested traffic states in empirical observations and microscopic simulations,

    M. Treiber, A. Hennecke, and D. Helbing, “Congested traffic states in empirical observations and microscopic simulations,”Physical review E, vol. 62, no. 2, p. 1805, 2000

    2000