pith. sign in

arxiv: 2604.11708 · v1 · submitted 2026-04-13 · 💻 cs.RO · cs.SE· cs.SY· eess.SY

ACT: Automated CPS Testing for Open-Source Robotic Platforms

Aditya A. Krishnan , Donghoon Kim , Hokeun Kim This is my paper

Pith reviewed 2026-05-10 15:00 UTC · model grok-4.3

classification 💻 cs.RO cs.SEcs.SYeess.SY
keywords cyber-physical systemsautomated testingopen-source softwarerobotic platformscontinuous integrationGitHubhardware-in-the-loopCPS testing
0
0 comments X

The pith

The ACT framework automates continuous testing of open-source CPS software by integrating physical robotic platforms into GitHub CI pipelines.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Open-source cyber-physical systems software often lacks testing on actual robotic hardware, so critical errors remain hidden until real-world use. Multiple independent contributors make this worse because no single team can run full platform tests on every change. The paper proposes Automated CPS Testing (ACT) to run software automatically on real robotic platforms and tie the results into standard open-source tools such as GitHub. The authors built a working prototype and tested it on an educational robot to show the approach is feasible. If it works as intended, open-source robotic projects could catch hardware-specific bugs much earlier in the development cycle.

Core claim

We propose Automated CPS Testing (ACT) that performs automated, continuous testing of open-source software with its robotic platforms, integrated with the open-source infrastructure such as GitHub. We implement an ACT prototype and conduct a case study on an open-source CPS with an educational robotic platform to demonstrate its capabilities.

What carries the argument

The ACT prototype that connects physical robotic hardware execution directly to GitHub-based continuous integration for automated, ongoing validation.

If this is right

  • Code changes can be checked for hardware compatibility automatically on every commit rather than only during manual testing sessions.
  • Errors that appear only when software interacts with real sensors or actuators become visible during routine development.
  • Open-source maintainers gain a way to enforce platform testing without needing every contributor to own robotic hardware.
  • The same pattern could extend to other open-source CPS projects that currently rely on simulation alone.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If ACT reduces undetected errors, similar automation layers might be adopted for non-robotic CPS domains such as smart-grid or medical-device software.
  • Widespread use could shift contributor effort away from ad-hoc hardware debugging toward feature development.
  • Combining ACT with simulation environments might lower hardware wear while preserving the ability to catch platform-specific issues.

Load-bearing premise

Automated integration of physical robotic platforms into GitHub CI pipelines will reliably detect important hardware-related errors without excessive false positives or unsustainable maintenance effort.

What would settle it

Running the ACT prototype on the educational robot case study and observing either that it misses a known hardware-interaction bug or that it generates frequent false alarms requiring constant manual intervention.

Figures

Figures reproduced from arXiv: 2604.11708 by Aditya A. Krishnan, Donghoon Kim, Hokeun Kim.

Figure 1
Figure 1. Figure 1: Proposed ACT platform for automated testing [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 4
Figure 4. Figure 4: The IMU readings on the robot’s display (left) and [PITH_FULL_IMAGE:figures/full_fig_p003_4.png] view at source ↗
Figure 3
Figure 3. Figure 3: Deviation (in absolute percentage) between actual [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Setup for the bump experiment; a servo motor con [PITH_FULL_IMAGE:figures/full_fig_p003_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Wheel-motion tracking for RPM estimation using [PITH_FULL_IMAGE:figures/full_fig_p004_6.png] view at source ↗
read the original abstract

Open-source software for cyber-physical systems (CPS) often lacks robust testing involving robotic platforms, resulting in critical errors that remain undetected. This is especially challenging when multiple modules of CPS software are developed by various open-source contributors. To address this gap, we propose Automated CPS Testing (ACT) that performs automated, continuous testing of open-source software with its robotic platforms, integrated with the open-source infrastructure such as GitHub. We implement an ACT prototype and conduct a case study on an open-source CPS with an educational robotic platform to demonstrate its capabilities.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper proposes Automated CPS Testing (ACT), a framework for integrating physical robotic platforms into GitHub-style continuous integration pipelines to enable automated, continuous testing of open-source cyber-physical systems software. It describes the implementation of an ACT prototype and presents a case study using an educational robotic platform to illustrate detection of errors missed by conventional open-source processes.

Significance. If the central claim holds, the work would be significant for open-source robotics and CPS development by addressing the gap in hardware-in-the-loop testing within contributor-driven projects. The explicit integration with existing open-source infrastructure such as GitHub is a practical strength that could lower barriers to adoption if the automation and reliability issues are resolved.

major comments (2)
  1. [Abstract and Case Study] Abstract and Case Study section: the claim that ACT addresses 'critical errors that remain undetected' lacks any quantitative support, including error detection rates, false-positive counts, comparison baselines against standard CI, or concrete examples of bugs found in the educational-platform case study. This is load-bearing for the central contribution.
  2. [Prototype implementation and Case Study] Prototype implementation and Case Study sections: the description of automated continuous testing with physical hardware does not address calibration, power cycling, sensor resets, or recovery from mechanical/environmental faults. These are required for the physical layer to operate without routine human intervention, directly undermining the 'automated, continuous' claim.
minor comments (1)
  1. [Abstract] The abstract would be clearer if it named the specific educational robotic platform and briefly indicated the scale of the case study (e.g., number of test runs or modules exercised).

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments on our manuscript. We address each major comment below and indicate the revisions we will make.

read point-by-point responses

  1. Referee: [Abstract and Case Study] Abstract and Case Study section: the claim that ACT addresses 'critical errors that remain undetected' lacks any quantitative support, including error detection rates, false-positive counts, comparison baselines against standard CI, or concrete examples of bugs found in the educational-platform case study. This is load-bearing for the central contribution.

    Authors: We acknowledge that the current presentation of the case study would be strengthened by quantitative evidence. The case study illustrates the detection of errors in the educational robotic platform that were not caught by standard open-source processes. In the revised manuscript, we will include concrete examples of specific bugs identified during the ACT testing process, along with available metrics such as the number of tests executed and issues detected. We will also add a comparison to baseline CI approaches where feasible. This revision will better substantiate the central claim. revision: yes

  2. Referee: [Prototype implementation and Case Study] Prototype implementation and Case Study sections: the description of automated continuous testing with physical hardware does not address calibration, power cycling, sensor resets, or recovery from mechanical/environmental faults. These are required for the physical layer to operate without routine human intervention, directly undermining the 'automated, continuous' claim.

    Authors: This is a valid point regarding the practical challenges of physical hardware in continuous testing. Our prototype focuses on the integration with GitHub and the orchestration of tests on the robotic platform. To address this, we will revise the relevant sections to describe the mechanisms implemented for calibration, power cycling, sensor resets, and fault recovery in the ACT prototype. If certain aspects require human intervention, we will explicitly discuss them as current limitations and outline plans for full automation in future work. revision: yes

Circularity Check

0 steps flagged

No circularity: system proposal with prototype and case study

full rationale

The paper proposes a testing framework (ACT) and describes its implementation plus an educational-platform case study. No equations, fitted parameters, derivations, or load-bearing self-citations appear in the provided text. The central claim is an engineering integration proposal whose validity rests on external demonstration rather than any internal reduction to prior fitted values or self-referential definitions. This matches the default non-circular case for non-mathematical system papers.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 1 invented entities

The central claim rests on the assumption that physical robotic testing can be automated and integrated into open-source workflows without prohibitive cost or complexity; no free parameters or invented physical entities are introduced.

invented entities (1)
  • ACT framework no independent evidence
    purpose: Automated continuous testing of open-source CPS with physical robots
    Newly proposed system name and architecture; no independent evidence provided beyond the case study mention.

pith-pipeline@v0.9.0 · 5392 in / 1128 out tokens · 37882 ms · 2026-05-10T15:00:13.257112+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

31 extracted references · 31 canonical work pages

  1. [1]

    Sara Abbaspour Asadollah, Rafia Inam, and Hans Hansson. 2015. A survey on testing for cyber physical system. InIFIP International Conference on Testing Software and Systems. Springer, 194–207

    work page 2015

  2. [2]

    Norhafizan Ahmad, Raja Ariffin Raja Ghazilla, Nazirah M Khairi, and Vi- jayabaskar Kasi. 2013. Reviews on various inertial measurement unit (IMU) sensor applications.International Journal of Signal Processing Systems1, 2 (2013), 256–262

    work page 2013

  3. [3]

    Autoware Foundation. 2020. Autoware: Open Software for Autonomous Driving. https://github.com/autowarefoundation/autoware. https://www.autoware.org Version 1.0.0

    work page 2020

  4. [4]

    Christian Birchler, Sajad Khatiri, Pooja Rani, Timo Kehrer, and Sebastiano Panichella. 2025. A roadmap for simulation-based testing of autonomous cyber- physical systems: Challenges and future direction.ACM Transactions on Software Engineering and Methodology34, 5 (2025), 1–9

    work page 2025

  5. [5]

    Lionel Briand, Shiva Nejati, Mehrdad Sabetzadeh, and Domenico Bianculli. 2016. Testing the untestable: model testing of complex software-intensive systems. In Proceedings of the 38th international conference on software engineering companion. 789–792

    work page 2016

  6. [6]

    Moughlbay, Angel P

    Enric Cervera, Philippe Martinet, Raul Marin, Amine A. Moughlbay, Angel P. del Pobil, Jaime Alemany, Roger Esteller, and Gustavo Casañ. 2016. The Robot Programming Network.Journal of Intelligent & Robotic Systems81, 1 (2016), 77–95. doi:10.1007/s10846-015-0201-7

    work page doi:10.1007/s10846-015-0201-7 2016

  7. [7]

    Accessed on November 9, 2025

    Pololu Corporation. Accessed on November 9, 2025. Pololu 3pi+ 2040 Robot. https://www.pololu.com/category/300/3pi-plus-2040-robot

    work page 2025

  8. [8]

    Accessed on November 9, 2025

    Pololu Corporation. Accessed on November 9, 2025. Pololu 3pi+ 2040 Robot – 30:1 Micro Metal Gearmotor MP 6V with Extended Motor Shaft. https://www. pololu.com/product/2378

    work page 2025

  9. [9]

    Alexandre Decan, Tom Mens, Pooya Rostami Mazrae, and Mehdi Golzadeh. 2022. On the use of GitHub actions in software development repositories. In2022 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 235–245

    work page 2022

  10. [10]

    Accessed on November 9, 2025

    GitHub Docs. Accessed on November 9, 2025. Self-hosted runners. https: //docs.github.com/en/actions/concepts/runners/self-hosted-runners

    work page 2025

  11. [11]

    Lee, and Marten Lohstroh

    Abhi Gundrala, Hokeun Kim, Aditya Krishnan, Edward A. Lee, and Marten Lohstroh. Accessed on November 9, 2025. Embedded Systems Labs. https: //www.lf-lang.org/embedded-lab/index.html

    work page 2025

  12. [12]

    Jens Kober and Jan Peters. 2009. Learning motor primitives for robotics. In2009 IEEE International Conference on Robotics and Automation. IEEE, 2112–2118

    work page 2009

  13. [13]

    Pavneet Singh Kochhar, Ferdian Thung, David Lo, and Julia Lawall. 2014. An empirical study on the adequacy of testing in open source projects. In2014 21st Asia-Pacific Software Engineering Conference, Vol. 1. IEEE, 215–222

    work page 2014

  14. [14]

    Prajwal Koirala and Cody Fleming. 2024. F1tenth Autonomous Racing With Of- fline Reinforcement Learning Methods. In2024 IEEE 27th International Conference on Intelligent Transportation Systems (ITSC). IEEE, 3922–3929

    work page 2024

  15. [15]

    Ela Liberman-Pincu, Shanee Honig, and Tal Oron-Gilad. 2023. Why are You Blinking at Me? Exploring Users’ Understanding of Robotic Status Indicators. In International Conference on Computer-Human Interaction Research and Applica- tions. Springer, 282–294

    work page 2023

  16. [16]

    Jun-Wei Lin, Navid Salehnamadi, and Sam Malek. 2020. Test automation in open- source Android apps: A large-scale empirical study. InProceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering. 1078– 1089

    work page 2020

  17. [17]

    Steven Macenski, Tully Foote, Brian Gerkey, Chris Lalancette, and William Woodall. 2022. Robot Operating System 2: Design, architecture, and uses in the wild.Science Robotics7, 66 (2022), eabm6074. doi:10.1126/scirobotics.abm6074

    work page doi:10.1126/scirobotics.abm6074 2022

  18. [18]

    Claudio Mandrioli, Max Nyberg Carlsson, and Martina Maggio. 2024. Testing Abstractions for Cyber-Physical Control Systems-RCR Report.ACM Transactions on Software Engineering and Methodology(2024)

    work page 2024

  19. [19]

    James McLurkin, Adam McMullen, Nick Robbins, Golnaz Habibi, Aaron Becker, Alvin Chou, Hao Li, Meagan John, Nnena Okeke, Joshua Rykowski, et al. 2014. A robot system design for low-cost multi-robot manipulation. In2014 IEEE/RSJ international conference on intelligent robots and systems. IEEE, 912–918

    work page 2014

  20. [20]

    Ravina Mithe, Supriya Indalkar, and Nilam Divekar. 2013. Optical character recognition.International journal of recent technology and engineering (IJRTE)2, 1 (2013), 72–75

    work page 2013

  21. [21]

    Armin Mokhtarian, Jianye Xu, Patrick Scheffe, Maximilian Kloock, Simon Schäfer, Heeseung Bang, Viet-Anh Le, Sangeet Ulhas, Johannes Betz, Sean Wilson, et al

    work page

  22. [22]

    A survey on small-scale testbeds for connected and automated vehicles and robot swarms: A guide for creating a new testbed.IEEE Robotics & Automation Magazine(2024)

    work page 2024

  23. [23]

    National Institute of Standards and Technology. 2016. Cyber-Physical Sys- tems/Internet of Things Testbed. Available at: https://www.nist.gov/programs- projects/cyber-physical-systemsinternet-things-testbed. Last reviewed on March 9, 2016. Accessed on November 8, 2025

    work page 2016

  24. [24]

    Accessed on November 9, 2025

    Lingua Franca Organization. Accessed on November 9, 2025. Lingua Franca: Build predictable concurrent, time-sensitive, and distributed systems. https: //www.lf-lang.org/

    work page 2025

  25. [25]

    Owain Parry, Gregory M Kapfhammer, Michael Hilton, and Phil McMinn. 2021. A survey of flaky tests.ACM Transactions on Software Engineering and Methodology (TOSEM)31, 1 (2021), 1–74

    work page 2021

  26. [26]

    Zahra Sadri-Moshkenani, Justin Bradley, and Gregg Rothermel. 2022. Survey on test case generation, selection and prioritization for cyber-physical systems. Software Testing, Verification and Reliability32, 1 (2022), e1794

    work page 2022

  27. [27]

    Aviral Shrivastava, Mohammadreza Mehrabian, Mohammad Khayatian, Patricia Derler, Hugo Andrade, Kevin Stanton, Ya-Shian Li-Baboud, Edward Griffor, Marc Weiss, and John Eidson. 2017. A testbed to verify the timing behavior of cyber- physical systems. InProceedings of the 54th Annual Design Automation Conference

    work page 2017

  28. [28]

    Charanjot Singh, Nikita Seth Gaba, Manjot Kaur, and Bhavleen Kaur. 2019. Com- parison of different CI/CD tools integrated with cloud platform. In2019 9th International Conference on Cloud Computing, Data Science & Engineering (Con- fluence). IEEE, 7–12

    work page 2019

  29. [29]

    Janos Sztipanovits, Xenofon Koutsoukos, Gabor Karsai, Nicholas Kottenstette, Panos Antsaklis, Vijay Gupta, Bill Goodwine, John Baras, and Shige Wang. 2011. Toward a science of cyber–physical system integration.Proc. IEEE100, 1 (2011), 29–44

    work page 2011

  30. [30]

    Fiorella Zampetti, Vittoria Nardone, and Massimiliano Di Penta. 2022. Problems and solutions in applying continuous integration and delivery to 20 open-source cyber-physical systems. InProceedings of the 19th International Conference on Mining Software Repositories. 646–657

    work page 2022

  31. [31]

    Fiorella Zampetti, Damian Tamburri, Sebastiano Panichella, Annibale Panichella, Gerardo Canfora, and Massimiliano Di Penta. 2023. Continuous integration and delivery practices for cyber-physical systems: An interview-based study.ACM Transactions on Software Engineering and Methodology32, 3 (2023), 1–44

    work page 2023