REVIEW 2 major objections 8 minor 1 cited by
An Empirical Study of Interaction Bugs in ROS-based Software
T0 review · 2 major / 8 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read This paper claims that interaction bugs in ROS-based robotic systems can be organized into three categories with 21 atomic root causes, and that reusable fix patterns cover 60.33% of the 121 studied bugs.
desk verdict First useful taxonomy of ROS interaction bugs; trust the categories more than the percentages. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the three-level taxonomy built by iterative open coding: whole iBugs are first sorted into intra-system, hardware, or environmental interactions; each type is then decomposed into subcategories, and finally into 21 atomic root causes (examples include namespace mishandling, naming mismatch, improper synchronization, executor misuse, definition violation, improper resource management, incorrect hardware interfaces, hardware heterogeneity, improper replanning, and improper collision detection). The taxonomy carries the entire argument because every percentage, fix pattern, and detection suggestion is computed by attaching each of the 121 bugs to one atomic category. To it is joined a fix-pattern table mapping root causes to common repair actions, which is what yields the 60.33% coverage figure.
What would settle it
Re-run the collection on the same ten projects without the keyword pre-filter: label all closed bug-labeled issues as interaction-related or not by three independent coders, then compare the category proportions. If the intra-system share moves well away from 68.60%, or if fewer than 60.33% of the newly identified iBugs fit the existing fix patterns, the claimed distribution and pattern coverage are artifacts of the selection filter.
Extended reading notes
Core claim
On the paper's own terms, the discovery is a working taxonomy of interaction bugs in ROS: three interaction categories, 21 atomic root-cause categories, and a root-cause-to-fix-pattern mapping. The empirical distribution is lopsided: 68.60% of iBugs are intra-system (83/121), 14.88% hardware (18/121), and 16.53% environmental (20/121). Within intra-system bugs, interaction topology error is the most frequent single cause at 19.01% of all iBugs; within hardware bugs, improper resource management accounts for 50%; within environmental bugs, improper collision detection accounts for 45%. The paper further claims that 22 reusable fix patterns cover 76.19% of the atomic root causes and 60.33% of the studied iBugs, with the remaining bugs fixed case-by-case. It also traces impacts of 86 iBugs, finding that execution-stage failures, especially software crashes, dominate.
Load-bearing premise
The study's percentages assume that searching issue text with keywords like 'topic', 'callback', and 'hardware,' plus manual confirmation, recovers a representative sample of interaction bugs, and that the ten projects chosen by star and issue counts represent ROS applications broadly.
Editorial extensions
If this is right
- If the taxonomy is right, the highest-yield targets for prevention are interaction topology errors, executor misuse, and interface misuse, which together account for most intra-system iBugs.
- The finding that improper resource management underlies half of hardware iBugs implies that lifecycle checks and resource-conflict checks belong in any hardware-facing test suite.
- Because 60.33% of sampled iBugs share reusable fix patterns, tool builders have a concrete inventory to turn into static detectors or automated repair templates.
- Since the majority of traceable impacts land in the execution stage (66.28% of the 86), runtime monitoring and execution-level testing should be a priority for reliability engineering.
- The taxonomy's environmental category, most often improper collision detection, suggests that simulation-based testing with varied obstacle layouts would exercise the least-understood iBug class.
Reading between the lines
- Editorial extension: the paper's 60.33% fix-pattern coverage is computed over a keyword-filtered sample; a keyword-independent scan of all bug-labeled patches in the same ten projects would show whether the coverage statistic survives selection bias.
- Editorial extension: if the taxonomy transfers to other robotics middlewares, it could serve as a shared vocabulary for cross-system interaction failures beyond ROS, but the paper does not test that transfer.
- Editorial extension: a direct testable outgrowth would be to map each of the 22 fix patterns to a static check or unit test and measure detection rate and false positives on the 121 bugs; the paper stops at suggesting directions.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This manuscript reports an empirical study of interaction bugs (iBugs) in ROS-based robotic systems. The authors select ten actively maintained projects from the ROS index, identify 364 candidate iBugs through keyword search and manual confirmation, randomly sample 121 for detailed analysis, and manually label each bug's interaction type, root cause, fix pattern, and impact using open coding. They propose a taxonomy with three major types (intra-system, hardware, and environmental) and 21 atomic root causes, and report findings on prevalence (e.g., 68.60% intra-system, 19.01% interaction topology errors), on fix patterns covering 60.33% of the studied bugs, and on impacts concentrated in the execution stage. A public dataset is provided.
Significance. If the taxonomy is valid, this is the first systematic empirical study of iBugs in ROS and fills a clear gap between existing work on dependency bugs, misconfigurations, and general ROS bugs. The paper's strengths include a publicly available dataset, a detailed taxonomy with concrete bug examples, and a discussion of detection and avoidance implications that is connected to existing tools such as ROSDiscover, RoboFuzz, and ROSInfer. Because the prevalence and coverage percentages depend on the completeness and neutrality of the keyword-based collection, the distributional findings require additional validation before they can be used as ground truth for future detectors. The qualitative taxonomy itself is likely to survive such validation, but the reported percentages may shift.
major comments (2)
- [3.2] The key collection step filters by keywords ('topic', 'callback', 'hardware', 'behaviour', 'interact') that are not category-neutral: 'topic' and 'callback' preferentially surface intra-system interaction reports, and 'hardware' directly selects hardware-related reports. No precision or recall is reported for this filter. The random sample of 121 is therefore a sample of the keyword-filtered 364 candidates, not of the full iBug population; Findings 1-7 (e.g., 68.60% intra-system, 19.01% topology, 14.88% hardware, 60.33% fix-pattern coverage) are estimates of the filtered population unless keyword recall is category-independent, which is neither argued nor tested. Please add a recall validation, for example by manually examining a random sample of closed bug-labeled issues that do not match the keywords, or by re-running the collection with an expanded keyword set and reporting the effect on the main percentages.
- [3.3] The open-coding procedure does not report any inter-rater reliability measure. The paper states that three authors labeled subsets and resolved conflicts by discussion, and that saturation was reached after three iterations, but no agreement statistics or per-iteration category-change counts are given. Since several atomic categories contain very few instances (e.g., [B.3] has 3, [C.4] has 3), the stability of the fine-grained taxonomy is hard to assess from the manuscript alone. Please report at least one agreement measure (e.g., Cohen's kappa or percentage agreement per iteration) or provide the labeling records as part of the dataset.
minor comments (8)
- [Abstract] The abstract contains a typo: 'Several findingsa are derived' should read 'Several findings are derived'.
- [4.1.2] In the Hardware heterogeneity example, 'bacause' is a typo for 'because'.
- [Figure 3] The label '[A.2.2] Improper synchonization' misspells 'synchronization'.
- [5.1] The phrase 'The complex setup and usage of ROS executors affects a lot of iBugs' is informal; consider rephrasing to 'affects many iBugs'.
- [3.2] The manuscript says keywords are 'iteratively refined' but does not give the final keyword set or the refinement history, which limits reproducibility; please state the exact final list and the criteria used to stop refinement.
- [3.2] The statement that the 121 bugs are a 'randomly sampled' one-third of 364 should include the sampling method or a seed, and the sampled issue identifiers should be listed in the dataset for reproducibility.
- [References] References [28] and [29] are duplicates (both are Kolak et al., 'It Takes a Village to Build a Robot'); one should be removed and the citations renumbered.
- [3.1] The paper should state explicitly whether the study covers ROS 1, ROS 2, or both; the selection is from the Humble distribution index, which is ROS 2, but earlier text refers to ROS generally without this scope restriction.
Circularity Check
No significant circularity: the paper is a descriptive empirical study whose findings are manually labeled summaries, not predictions derived from fitted inputs.
full rationale
This is a descriptive empirical study rather than a derivational chain, so there is no equation-level reduction of a claimed result to its own inputs. The iBug definition in Section 2.2 and the three-way classification are presented before collection, but the taxonomy of root causes, fixes, and impacts is induced from 121 manually examined bugs through an open-coding procedure (Section 3.3), and the percentages in Findings 1-8 are arithmetic summaries of the resulting labels rather than predictions obtained from a fitted model. The keyword search in Section 3.2 operationalizes the iBug definition and may raise a legitimate external-validity or sampling-bias concern, since no precision or recall is reported and keywords such as 'hardware' are not category-neutral; however, the taxonomy categories are not used as the search criteria to derive the taxonomy, and the paper makes no statistical claim that would collapse the findings into the filter. No fitted parameters, uniqueness theorems, or load-bearing self-citations appear; the only self-reference is the dataset link [1], which is not used to justify any central premise. The skeptical concern about keyword recall is a correctness/validity threat, not a circularity, and should be weighed in a threats-to-validity assessment rather than as a circularity score.
Assumptions & free parameters
assumptions (4)
- domain assumption Closed, bug-labeled issues and pull requests correspond to genuine software bugs.
- domain assumption The selected ten projects, filtered by more than 100 stars and 400 issues, are representative of ROS-based applications.
- domain assumption Keyword-based search with iterative refinement retrieves the majority of interaction bugs.
- domain assumption Three-author open coding with consensus yields a reliable taxonomy.
Cite this review
Pith. "Pith review of An Empirical Study of Interaction Bugs in ROS-based Software." pith.science (2026). https://pith.science/paper/EMYJJBJV
@misc{pith2026250710235,
author = {Pith},
title = {Pith review of: An Empirical Study of Interaction Bugs in ROS-based Software},
year = {2026},
howpublished = {\url{https://pith.science/paper/EMYJJBJV}},
note = {Machine review of arXiv:2507.10235}
}
read the original abstract
Modern robotic systems integrate multiple independent software and hardware components, each responsible for distinct functionalities such as perception, decision-making, and execution. These components interact extensively to accomplish complex end-to-end tasks. As a result, the overall system reliability depends not only on the correctness of individual components, but also on the correctness of their interactions. Failures often manifest at the boundaries between components, yet interaction-related reliability issues in robotics--referred to here as interaction bugs (iBugs)--remain underexplored. This work presents an empirical study of iBugs within robotic systems built using the Robot Operating System (ROS), a widely adopted open-source robotics framework. A total of 121 iBugs were analyzed across ten actively maintained and representative ROS projects. The identified iBugs are categorized into three major types: intra-system iBugs, hardware iBugs, and environmental iBugs, covering a broad range of interaction scenarios in robotics. The analysis includes an examination of root causes, fixing strategies, and the impact of these bugs. Several findingsa are derived that shed light on the nature of iBugs and suggest directions for improving their prevention and detection. These insights aim to inform the design of more robust and safer robotic systems.
Figures
Figures from the paper (2 more)
Forward citations
Cited by 1 Pith paper
-
IcFuzz: Fuzzing Isaac Sim with Semantic Stage Guidance and Multi-level Mutation
IcFuzz, an LLM-guided fuzzer for Isaac Sim, reaches about double the code coverage of prior baselines and has found 11 bugs, 9 confirmed or fixed.
Reference graph
Works this paper leans on
-
[1]
Understanding Interaction Bugs in Robotic Systems
2025. Dataset of “Understanding Interaction Bugs in Robotic Systems” . Re- trieved March 13, 2025 from https://anonymous.4open.science/r/Understanding- Interaction-Bugs-in-Robotic-Systems-FC8E/README.md
work page 2025
-
[2]
AndreasAZiegler. 2019. Navigation2 Issue 911: Robot continues moving when pressing “cancel” button during navigation. Retrieved October 31, 2024 from https://github.com/ros-navigation/navigation2/issues/911
work page 2019
-
[3]
Sunil Kumar Reddy Anumandla and JG Tejani. 2023. Robotic Automation in Rubber Processing: Improving Safety and Productivity. Asian Journal of Applied Science and Engineering 12, 1 (2023), 7–15
work page 2023
-
[4]
Ramez Awad, Georg Heppner, Arne Roennau, and Mirko Bordignon. 2016. ROS engineering workbench based on semantically enriched app models for improved reusability. In 21st IEEE International Conference on Emerging Technologies and Factory Automation, ETFA 2016, 2016. IEEE, 1–9. https://doi.org/10.1109/ETFA. 2016.7733581
-
[5]
Jia-Ju Bai, Haoxuan Song, and Shimin Hu. 2024. Multi-Dimensional and Message- Guided Fuzzing for Robotic Programs in Robot Operating System. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, ASPLOS 2024 . ACM, 763–778. https: //doi.org/10.1145/3620665.3640425
arXiv 2024
-
[6]
Geoffrey Biggs, Jacob Perron, and Shane Loretz. 2020. ROS 2 Design: Actions . Retrieved October 31, 2024 from https://design.ros2.org/articles/actions.html# goal-states
work page 2020
-
[7]
Sara Bouraine, Abdelhak Bougouffa, and Ouahiba Azouaoui. 2022. Particle swarm optimization for solving a scan-matching problem based on the normal distributions transform. Evol. Intell. 15, 1 (2022), 683–694. https://doi.org/10. 1007/S12065-020-00545-Y
work page 2022
-
[8]
Alex Brooks, Tobias Kaupp, Alexei Makarenko, Stefan B. Williams, and Anders Orebäck. 2005. Orca: A Component Model and Repository. In Software Engineer- ing for Experimental Robotics, Workshop on Principles and Practice of Software Development in Robotics, PPSDR@ICRA 2005 (Springer Tracts in Advanced Robotics, Vol. 30). 231–251. https://doi.org/10.1007/97...
Show all 45 references
-
[9]
Williams, and Anders Orebäck
Alex Brooks, Tobias Kaupp, Alexei Makarenko, Stefan B. Williams, and Anders Orebäck. 2005. Towards component-based robotics. In2005 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2005 . IEEE, 163–168. https://doi. org/10.1109/IROS.2005.1545523
2005
-
[10]
Davide Brugali and Patrizia Scandurra. 2009. Component-based robotic en- gineering (Part I) [Tutorial]. IEEE Robotics Autom. Mag. 16, 4 (2009), 84–96. https://doi.org/10.1109/MRA.2009.934837
2009
-
[11]
Davide Brugali and Azamat Shakhimardanov. 2010. Component-Based Robotic Engineering (Part II). IEEE Robotics Autom. Mag. 17, 1 (2010), 100–112. https: //doi.org/10.1109/MRA.2010.935798
2010
-
[12]
Schmerl, Alcides Fonseca, and Christopher Steven Timperley
Paulo Canelas, Bradley R. Schmerl, Alcides Fonseca, and Christopher Steven Timperley. 2024. Understanding Misconfigurations in ROS: An Empirical Study and Current Approaches. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA...
2024
-
[13]
Kaitlyn Cottrell, Dibyendu Brinto Bose, Hossain Shahriar, and Akond Rahman
-
[14]
Tobias Dürschmid, Christopher Steven Timperley, David Garlan, and Claire Le Goues. 2024. ROSInfer: Statically Inferring Behavioral Component Models for ROS-based Robotics Systems. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, ICSE 2024 ....
2024
-
[15]
Pablo Estefo, Jocelyn Simmonds, Romain Robbes, and Johan Fabry. 2019. The Robot Operating System: Package reuse and community dynamics. J. Syst. Softw. 151 (2019), 226–242. https://doi.org/10.1016/J.JSS.2019.02.024
2019 doi
-
[16]
Muhammad Ashraf Faheem, Nabeel Zafar, Parkash Kumar, Md Mehedi Hassan Melon, Nayem Uddin Prince, and Mohd Abdullah Al Mamun. 2024. AI AND ROBOTIC: ABOUT THE TRANSFORMATION OF CONSTRUCTION INDUSTRY AUTOMATION AS WELL AS LABOR PRODUCTIVITY. Remittances Review 9, S3 (July 2024) (...
2024
-
[18]
Market Research Future. 2024. Robotics Market Research Report . Retrieved October 30, 2024 from https://www.marketresearchfuture.com/reports/robotics- market-4732
2024
-
[19]
Cipriano Galindo, Juan-Antonio Fernández-Madrigal, Javier González, and Alessandro Saffiotti. 2008. Robot task planning using semantic maps. Robotics Auton. Syst. 56, 11 (2008), 955–966. https://doi.org/10.1016/J.ROBOT.2008.08.007
2008 doi
-
[20]
Sergio García, Daniel Strüber, Davide Brugali, Thorsten Berger, and Patrizio Pelliccione. 2020. Robotics software engineering: a perspective from the service robotics domain. In ESEC/FSE ’20: 28th ACM Joint European Software Engineering Conference and Symposium on the Foundati...
2020
-
[21]
Gerkey, Richard T
Brian P. Gerkey, Richard T. Vaughan, Kasper Støy, Andrew Howard, Gaurav S. Sukhatme, and Maja J. Mataric. 2001. Most valuable player: a robot device server for distributed control. In IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2001: Expanding the...
2001
-
[22]
Uwe Jahn, Carsten Wolff, and Peter Schulz. 2019. Concepts of a Modular System Architecture for Distributed Robotic Systems. Comput. 8, 1 (2019), 25. https: //doi.org/10.3390/COMPUTERS8010025
2019 doi
-
[23]
Mohd Javaid, Abid Haleem, Ravi Pratap Singh, and Rajiv Suman. 2021. Substantial capabilities of robotics in enhancing industry 4.0 implementation. Cognitive Robotics 1 (2021), 58–75
2021
-
[24]
JnxF and gavanderhoorn. 2021. ros2_fuzz. Retrieved October 23, 2024 from https://github.com/rosin-project/ros2_fuzz
2021
-
[25]
Sayali Kate, Michael Chinn, Hongjun Choi, Xiangyu Zhang, and Sebastian G. Elbaum. 2021. PHYSFRAME: type checking physical frames of reference for robotic systems. In ESEC/FSE ’21: 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Softw...
2021
-
[26]
Seulbae Kim and Taesoo Kim. 2022. RoboFuzz: fuzzing robotic systems over robot operating system (ROS) for finding correctness bugs. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE ...
2022
-
[27]
Yeseung Kim, Dohyun Kim, Jieun Choi, Jisang Park, Nayoung Oh, and Daehyung Park. 2024. A survey on integration of large language models with intelligent robots. Intell. Serv. Robotics 17, 5 (2024), 1091–1107. https://doi.org/10.1007/ S11370-024-00550-5
2024
-
[29]
Sophia Kolak, Afsoon Afzal, Claire Le Goues, Michael Hilton, and Christo- pher Steven Timperley. 2020. It Takes a Village to Build a Robot: An Empir- ical Study of The ROS Ecosystem. In IEEE International Conference on Soft- ware Maintenance and Evolution, ICSME 2020, 2020 . I...
2020
-
[30]
Matt Luckcuck, Marie Farrell, Angelo Ferrando, Rafael C Cardoso, Louise A Dennis, and Michael Fisher. 2022. A compositional approach to verifying modular Conference’17, July 2017, Washington, DC, USA Zhixiang Chen, Zhuangbin Chen*, Xingjie Cai, Wei Li, and Zibin Zheng robotic ...
2022 arXiv
-
[31]
Lewis, Bradley R
Ivano Malavolta, Grace A. Lewis, Bradley R. Schmerl, Patricia Lago, and David Garlan. 2020. How do you architect your robots?: state of the practice and guidelines for ROS-based systems. In ICSE-SEIP 2020: 42nd International Con- ference on Software Engineering, Software Engin...
2020
-
[32]
Giorgio Metta, Paul Fitzpatrick, and Lorenzo Natale. 2006. YARP: yet another robot platform. International Journal of Advanced Robotic Systems 3, 1 (2006), 8
2006
-
[33]
Aditya Patil. 2024. Recovery Behaviours used in Navigation Stack . Retrieved March 7, 2025 from https://medium.com/@patiladitya1309/recovery-behaviours- used-in-navigation-stack-ros-5df991f408c7
2024
-
[34]
Morgan Quigley, Ken Conley, Brian Gerkey, Josh Faust, Tully Foote, Jeremy Leibs, Rob Wheeler, Andrew Y Ng, et al. 2009. ROS: an open-source Robot Operating System. In ICRA workshop on open source software , Vol. 3. Kobe, Japan, 5
2009
-
[35]
Open Robotics. 2019. Executors. Retrieved March 2, 2025 from https://docs.ros. org/en/humble/Concepts/Intermediate/About-Executors.html#executors
2019
-
[36]
Open Robotics. 2025. ROS Index. Retrieved January 11, 2025 from https://index. ros.org/repos/
2025
-
[37]
ros visualization. 2016. rqt. Retrieved February 27, 2025 from https://github. com/ros-visualization/rqt
2016
-
[38]
Katherine Scott and Tully Foote. 2024. 2023 ROS Metrics Report . Retrieved October 26, 2024 from https://discourse.ros.org/t/2023-ros-metrics-report/35837
2024
-
[39]
Carolyn B. Seaman. 1999. Qualitative Methods in Empirical Studies of Software Engineering. IEEE Trans. Software Eng. 25, 4 (1999), 557–572. https://doi.org/10. 1109/32.799955
1999
-
[40]
Yuheng Shen, Jianzhong Liu, Yiru Xu, Hao Sun, Mingzhe Wang, Nan Guan, Heyuan Shi, and Yu Jiang. 2024. Enhancing ROS System Fuzzing through Callback Tracing. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2024. ACM, 76–87....
2024
-
[41]
Xuezhi Song, Yi Li, Zhen Dong, Shuning Liu, Junming Cao, and Xin Peng. 2023. An Empirical Study on Fault Diagnosis in Robotic Systems. In IEEE International Conference on Software Maintenance and Evolution, ICSME 2023, 2023 . IEEE, 207–
2023
-
[42]
Lilia Tang, Chaitanya Bhandari, Yongle Zhang, Anna Karanika, Shuyang Ji, In- dranil Gupta, and Tianyin Xu. 2023. Fail through the Cracks: Cross-System Interaction Failures in Modern Cloud Systems. In Proceedings of the Eighteenth European Conference on Computer Systems, EuroSy...
2023
-
[43]
Schmerl, David Garlan, and Claire Le Goues
Christopher Steven Timperley, Tobias Dürschmid, Bradley R. Schmerl, David Garlan, and Claire Le Goues. 2022. ROSDiscover: Statically Detecting Run-Time Architecture Misconfigurations in Robotics Systems. In 19th IEEE International Conference on Software Architecture, ICSA 2022...
2022
-
[44]
Christopher Steven Timperley, Gijs van der Hoorn, André Santos, Harshavardhan Deshpande, and Andrzej Wasowski. 2024. ROBUST: 221 bugs in the Robot Operating System. Empir. Softw. Eng. 29, 3 (2024), 57. https://doi.org/10.1007/ S10664-024-10440-0
2024
-
[45]
Kai-Tao Xie, Jia-Ju Bai, Yong-Hao Zou, and Yu-Ping Wang. 2022. ROZZ: Property- based Fuzzing for Robotic Programs in ROS. In 2022 International Conference on Robotics and Automation, ICRA 2022, P 2022 . IEEE, 6786–6792. https://doi.org/10. 1109/ICRA46639.2022.9811701
2022
-
[219]
https://doi.org/10.1109/ICSME58846.2023.00030
2023
-
[2021]
In IEEE 45th Annual Computers, Software, and Applications Conference, COMPSAC 2021, 2021
An Empirical Study of Vulnerabilities in Robotics. In IEEE 45th Annual Computers, Software, and Applications Conference, COMPSAC 2021, 2021 . IEEE, 735–744. https://doi.org/10.1109/COMPSAC51774.2021.00105
2021
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.