Pith. sign in

REVIEW 4 major objections 5 minor 22 references

Work in Progress: Middleware-Transparent Callback Enforcement in Commoditized Component-Oriented Real-time Systems

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Pith's one-line read Giving each ROS 2 callback its own OS thread removes the middleware layer from real-time scheduling models.

desk verdict A solid WIP that makes per-callback OS scheduling in ROS 2 concrete, but its headline 'ignore the middleware' claim rests on an unvalidated timing assumption. read the letter →

arxiv 2505.06546 v1 pith:R5Y5PWV2 submitted 2025-05-10 cs.OS cs.RO

classification cs.OScs.RO
keywords ROS2real-timeschedulingmiddlewaretransparencycallbackOSthreadnestedDAGIsolatedExecutor
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that nested scheduling in ROS 2, where the executor layer queues and dispatches callbacks onto shared OS threads, is an artificial implementation constraint rather than a necessity. If each callback is permanently bound to its own OS thread, the executor becomes invisible: OS scheduling policy, priority, and affinity apply directly to individual callbacks, and the system reduces to a DAG of callback vertices. The authors build CallbackIsolatedExecutor, a ROS 2 executor that realizes this one-to-one mapping, and measure that its overhead in user-kernel switches, context switches, and memory stays below the MultiThreadedExecutor and within a fixed ratio of the SingleThreadedExecutor. If the claim holds, future ROS 2 real-time scheduling research can ignore the executor layer entirely and reuse classical real-time and DAG scheduling theory on the callback graph.

What carries the argument

CallbackIsolatedExecutor, a ROS 2 executor variant in which every callback is permanently mapped to a dedicated OS thread, with one callback per CallbackGroup and no wait-set shared across threads. This mapping lets Linux scheduling policies (SCHED_DEADLINE, SCHED_FIFO, and CFS with nice values) and CPU affinity be applied per callback, which is what allows the middleware layer to drop out of the scheduling model.

What would settle it

Run a publish-subscribe pair in which the subscriber callback's worst-case execution time exceeds the publisher's period: if queued invocations are observed and end-to-end latency deviates from a per-callback DAG model that permits only one outstanding invocation per vertex, the middleware-transparent assumption fails for that workload.

Watch

Extended reading notes

Core claim

The central discovery is that the middleware layer in ROS 2 can be made transparent to real-time scheduling by establishing a persistent one-to-one correspondence between callbacks and OS threads. Under this design, each callback lives in its own CallbackGroup and owns a dedicated thread, so scheduler, priority, and affinity are set per callback at the OS level. With the additional condition that a callback does not become ready for its next invocation before its previous invocation finishes, the executor's queueing behavior no longer affects execution order, and the scheduling problem becomes classical DAG scheduling on callback vertices. The paper implements CallbackIsolatedExecutor and reports that user-kernel switches and context switches remain below MultiThreadedExecutor for all tested callback counts (1 to 24) and within roughly 1.4x (inter-process) and 5x (intra-process) of SingleThreadedExecutor, with memory usage only slightly higher.

Load-bearing premise

The central claim collapses if a callback can be released again while a previous invocation is still running; the paper assumes each callback's minimum inter-arrival time exceeds its worst-case turnaround time, and asserts without workload measurements that many practical ROS 2 applications satisfy this.

Editorial extensions

If this is right

  • Scheduling analysis of ROS 2 applications reduces to DAG scheduling on callback vertices, so existing real-time and DAG scheduling algorithms apply without executor-specific analysis.
  • CallbackIsolatedExecutor incurs lower context-switch and user-kernel-switch overhead than MultiThreadedExecutor regardless of the number of callbacks.
  • Overhead relative to SingleThreadedExecutor stays within a bounded ratio (1.4x inter-process, 5x intra-process), so the cost of isolation does not grow with callback count.
  • Designers must keep callbacks non-reentrant: each callback's minimum inter-arrival time must exceed its worst-case turnaround time, otherwise the middleware cannot be ignored.
  • CPU time not consumed by real-time DAGs on SCHED_DEADLINE or SCHED_FIFO remains available to CFS tasks, so real-time scheduling and high utilization can coexist.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The fixed overhead ratios were measured on a synthetic publish-subscribe pair; production robot workloads with mixed message sizes and shared-memory intra-process communication could shift those ratios, so the bounded-ratio claim still awaits testing on large node graphs.
  • The non-reentrancy condition could be enforced automatically rather than assumed: a framework could detect when a callback is released while still running and either queue, split, or reject the release, widening the class of applications that can safely ignore the executor.
  • If the approach generalizes, the distinction between executor-level scheduling policies and OS-level policies may become obsolete for ROS 2, and a direct comparison with event-pushing executor designs would be the natural next test of where queueing overhead remains.
  • The latency benefit may be larger than the paper measures: eliminating shared wait-set reconstruction and lock contention is described qualitatively as an advantage but not isolated as an end-to-end latency number in the experiments.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes CallbackIsolatedExecutor, a ROS 2 executor that establishes a persistent dedicated OS thread per callback (through one-callback CallbackGroups) so that Linux scheduling policy, priority, and affinity can be configured at the thread level and the middleware executor can be ignored in scheduling analysis. Under the assumption that a callback is never released again before its previous invocation completes, the ROS 2 scheduling problem is reformulated as DAG scheduling over callbacks, bypassing nested scheduling. Overhead measurements (user-kernel switches, context switches, memory) compare the new executor with ROS 2 SingleThreadedExecutor and MultiThreadedExecutor for inter-process and intra-process publish-subscribe with varying callback counts.

Significance. If the central condition holds, the paper removes a genuine obstacle: executor-specific response-time analyses would no longer be needed, and existing fixed-priority or DAG scheduling theory could be applied directly to ROS 2 callback graphs. The implementation is open source, the overhead data are direct measurements with no fitted parameters, and the design is clearly described. The main value is in enabling future ROS 2 real-time scheduling research to model only callback graphs. However, the paper's headline claims are conditional on an unvalidated non-reentrancy/timing assumption and on overhead comparisons that are not statistically substantiated; the result is a plausible design statement rather than a fully demonstrated middleware-transparency result.

major comments (4)
  1. [Section II and Section III] The central 'ignore the middleware' claim rests on the condition stated in Section II that a callback never gets ready for the next period until the previous execution finishes (minimum inter-arrival time exceeding worst-case turnaround time), and Section III justifies this by asserting that many practical ROS 2 applications do not implement reentrant callbacks. The second statement does not establish the first. A non-reentrant callback is code that is not written for concurrent invocations; a new timer tick or message can still arrive while an invocation is running, causing the executor or the underlying DDS queue to hold it until the thread becomes free. In that case the next invocation's release time and jitter are determined by the middleware queue, not solely by the thread's OS scheduling parameters. Moreover, worst-case turnaround time depends on OS scheduling, preemption, and affinity, so the condition can fail in exactly the overloaded or stressed regimes for which per-callback priority assignment is intended. The experiments in Section IV use a 10 ms timer with short callbacks and therefore trivially satisfy the condition; they provide no evidence about real ROS 2 workloads. The central claim is thus conditional, and the condition is not validated by any measurement, workload study, or quantitative argument.
  2. [Abstract and Section IV] The abstract states that the costs of CallbackIsolatedExecutor 'remain lower than those of the MultiThreadedExecutor, regardless of the number of callbacks,' but Fig. 5(a) shows that for intra-process publish-subscribe, CallbackIsolatedExecutor's user-kernel mode switches exceed MultiThreadedExecutor's at most callback counts; the text in Section IV itself acknowledges that MultiThreadedExecutor has a 'slight advantage' in this condition. Since the abstract is the published claim, it needs correction to reflect that the advantage holds for context-switch count and memory use, but not for user-kernel mode switches under intra-process communication, or the claim is contradicted by the paper's own data.
  3. [Section IV] The overhead ratio claim is not statistically substantiated. Section IV reports that CallbackIsolatedExecutor stays within a fixed ratio (1.4x for inter-process and 5x for intra-process) of SingleThreadedExecutor, but no experimental repetitions, confidence intervals, or standard deviations are reported for any of the three measurements. Without information on the number of trials and run-to-run variance, the reader cannot assess whether the ratio is stable across runs or whether the apparent trends in Fig. 5 are meaningful. The fixed-ratio statement is a central quantitative claim and needs at least a statement of the number of trials and variance.
  4. [Section V] The conclusion broadens the claim by saying that 'the real-time community no longer needs to account for the existence of the Executor in ROS 2 scheduling in practical cases' and that 'real-world ROS 2 systems rarely have dozens of callbacks per node,' but no evidence is provided for this scalability claim. The experiments only go up to 24 callbacks, and one thread per callback may incur significant context-switch and wait-set contention in larger systems. The future-work statement that the approach will be evaluated on Autoware further indicates that the central practical claim remains to be demonstrated; the conclusion should be scoped accordingly.
minor comments (5)
  1. [Figure 5] The figure would be more informative with error bars or shaded confidence intervals, and the legend entries such as 'single(separate)' should be expanded or defined in the caption for readability.
  2. [Footnote] The open-source URL in the footnote contains a space ('callback isolated executor'); it should be a proper link or code-formatted path with an underscore, otherwise the URL is not usable.
  3. [Section III] The text says that 'in practice, CallbackGroups, not callbacks, are mapped to threads.' This is an important implementation detail that should be stated more prominently, since it means the one-to-one correspondence is actually between CallbackGroups and threads, and the one-callback-per-group constraint is what makes it per-callback.
  4. [Section II] The term 'user-kernel switches' is used throughout; it would be clearer to define it explicitly (e.g., as system calls or user-kernel mode transitions) so that readers can interpret the measurement methodology.
  5. [Section III] The constraint that each CallbackGroup contains one callback may require changes to existing third-party ROS 2 nodes; the paper would benefit from a brief discussion of the migration burden for black-box nodes.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's central claims are a design proposal backed by direct measurements, with no fitted parameter or self-citation chain doing load-bearing work.

full rationale

The paper's central claim is that a persistent one-to-one mapping between callbacks and OS threads lets the middleware executor layer be ignored. This is presented as an explicit conditional design statement: 'If a callback never gets ready for the next period until the previous execution finishes ... the middleware layer can be ignored in scheduling.' That condition is an assumption about the workload, not an equation fitted to the target result; it may be hard to validate, but the reasoning is not circular. The overhead evaluation (user-kernel switches, context switches, memory) consists of direct measurements of three executors, and the claimed 1.4x/5x ratios are observed quantities rather than predictions derived from fitted inputs. The only self-citation is [20] (Yano and Azumi, both authors of this paper) used to motivate DAG models with per-DAG deadlines; it is background context and does not bear the weight of the middleware-transparency claim. No uniqueness theorem is imported from the authors' prior work, no ansatz is smuggled in via citation, and no known result is renamed. The paper's weakness is that its non-reentrancy argument ('Many practical ROS 2 applications do not implement reentrant callbacks') does not establish the inter-arrival > turnaround condition under scheduling stress; that is a correctness/validation gap, not circularity. Under the stated assumptions the middleware layer is irrelevant by construction, which is a modeling consequence rather than a circular derivation.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters or invented entities appear in this paper. The argument rests on the three domain assumptions listed, especially the non-reentrancy condition that makes the middleware layer transparent.

assumptions (3)
  • domain assumption Each CallbackGroup contains exactly one callback.
    Section III, first bullet. The implemented mapping is CallbackGroup to thread; if a group holds multiple callbacks, per-callback OS scheduling parameters are not achievable.
  • domain assumption Callbacks are not reentrant: minimum inter-arrival time exceeds worst-case turnaround time.
    Sections II and III. This is the load-bearing condition under which the middleware executor can be ignored; if violated, internal queueing remains and the middleware layer cannot be treated as transparent.
  • domain assumption Linux EDF, FIFO, and CFS schedulers act in priority order and CFS reclaims unused real-time bandwidth.
    Section III. The proposed formulation assumes this standard Linux scheduler behavior as the enforcement mechanism for per-callback scheduling parameters.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Work in Progress: Middleware-Transparent Callback Enforcement in Commoditized Component-Oriented Real-time Systems." pith.science (2026). https://pith.science/paper/R5Y5PWV2

@misc{pith2026250506546,
  author       = {Pith},
  title        = {Pith review of: Work in Progress: Middleware-Transparent Callback Enforcement in Commoditized Component-Oriented Real-time Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/R5Y5PWV2}},
  note         = {Machine review of arXiv:2505.06546}
}
read the original abstract

Real-time scheduling in commoditized component-oriented real-time systems, such as ROS 2 systems on Linux, has been studied under nested scheduling: OS thread scheduling and middleware layer scheduling (e.g., ROS 2 Executor). However, by establishing a persistent one-to-one correspondence between callbacks and OS threads, we can ignore the middleware layer and directly apply OS scheduling parameters (e.g., scheduling policy, priority, and affinity) to individual callbacks. We propose a middleware model that enables this idea and implements CallbackIsolatedExecutor as a novel ROS 2 Executor. We demonstrate that the costs (user-kernel switches, context switches, and memory usage) of CallbackIsolatedExecutor remain lower than those of the MultiThreadedExecutor, regardless of the number of callbacks. Additionally, the cost of CallbackIsolatedExecutor relative to SingleThreadedExecutor stays within a fixed ratio (1.4x for inter-process and 5x for intra-process communication). Future ROS 2 real-time scheduling research can avoid nested scheduling, ignoring the existence of the middleware layer.

Figures

Figures reproduced from arXiv: 2505.06546 by the authors.

Figure 1
Figure 1. Component-oriented real-time system (e.g., ROS 2). [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Middleware design of existing Executors and [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Proposed formulation of ROS 2 scheduling enforce [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Comparison of CallbackIsolatedExecutor overhead with existing Executors (see [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 22 canonical work pages

  1. [20]

    Work-in-progress: Multi-deadline DAG scheduling model for autonomous driving systems,

    A. Yano and T. Azumi, “Work-in-progress: Multi-deadline DAG scheduling model for autonomous driving systems,” in Proc. of IEEE Real-Time Systems Symposium (RTSS) , 2024

  2. [1]

    Open source robotics foundation (OSRF). ROS2

    “Open source robotics foundation (OSRF). ROS2.” [Online]. Available: https://github.com/ros2

  3. [2]

    Data-flow availability: Achieving timing assurance in autonomous systems,

    A. Li and N. Zhang, “Data-flow availability: Achieving timing assurance in autonomous systems,” in Proc. of 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI) , 2024

  4. [3]

    End-to-end timing analysis in ROS2,

    H. Teper, M. G ¨unzel, N. Ueter, G. von der Br ¨uggen, and J.-J. Chen, “End-to-end timing analysis in ROS2,” in Proc. of IEEE Real-Time Systems Symposium (RTSS) , 2022

  5. [4]

    Timing-aware ROS 2 architecture and system optimization,

    H. Teper, T. Betz, G. V on Der Br ¨uggen, K.-H. Chen, J. Betz, and J.-J. Chen, “Timing-aware ROS 2 architecture and system optimization,” in Proc. of IEEE 29th International Conference on Embedded and Real- Time Computing Systems and Applications (RTCSA) , 2023

  6. [5]

    End-to-end timing analysis and optimization of multi- executor ROS 2 systems,

    H. Teper, T. Betz, M. G ¨unzel, D. Ebner, G. von der Br ¨uggen, J. Betz, and J.-J. Chen, “End-to-end timing analysis and optimization of multi- executor ROS 2 systems,” in Proc. of IEEE 30th Real-Time and Embedded Technology and Applications Symposium (RTAS) , 2024

  7. [6]

    Automatic latency management for ROS 2: Benefits, challenges, and open problems,

    T. Blass, A. Hamann, R. Lange, D. Ziegenbein, and B. B. Brandenburg, “Automatic latency management for ROS 2: Benefits, challenges, and open problems,” in Proc. of IEEE 27th Real-Time and Embedded Technology and Applications Symposium (RTAS) , 2021

  8. [7]

    On-device CPU scheduling for robot sys- tems,

    A. Partap, S. Grayson, M. Huzaifa, S. Adve, B. Godfrey, S. Gupta, K. Hauser, and R. Mittal, “On-device CPU scheduling for robot sys- tems,” in Proc. of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , 2022. *https://github.com/sykwer/callback isolated executor

Show all 22 references
  1. [8]

    Response-time analysis of ROS 2 processing chains under reservation-based schedul- ing,

    D. Casini, T. Blaß, I. L ¨utkebohle, and B. Brandenburg, “Response-time analysis of ROS 2 processing chains under reservation-based schedul- ing,” in Proc. of 31st Euromicro Conference on Real-Time Systems (ECRTS), 2019

  2. [9]

    Response time analysis and priority assignment of processing chains on ROS2 executors,

    Y . Tang, Z. Feng, N. Guan, X. Jiang, M. Lv, Q. Deng, and W. Yi, “Response time analysis and priority assignment of processing chains on ROS2 executors,” in Proc. of IEEE Real-Time Systems Symposium (RTSS), 2020

  3. [10]

    PiCAS: New design of priority-driven chain-aware scheduling for ROS2,

    H. Choi, Y . Xiang, and H. Kim, “PiCAS: New design of priority-driven chain-aware scheduling for ROS2,” in Proc. of IEEE 27th Real-Time and Embedded Technology and Applications Symposium (RTAS) , 2021

  4. [11]

    A ROS 2 response-time analysis exploiting starvation freedom and execution-time variance,

    T. Blaß, D. Casini, S. Bozhko, and B. B. Brandenburg, “A ROS 2 response-time analysis exploiting starvation freedom and execution-time variance,” in Proc. of IEEE Real-Time Systems Symposium (RTSS) , 2021

  5. [12]

    Priority-driven real-time scheduling in ROS 2: Potential and challenges,

    H. Choi, D. Enright, H. Sobhani, Y . Xiang, and H. Kim, “Priority-driven real-time scheduling in ROS 2: Potential and challenges,” in Proc. of 1st International Workshop on Real-time And intelliGent Edge computing (RAGE), 2022

  6. [13]

    Real-time scheduling and analysis of processing chains on multi-threaded executor in ROS 2,

    X. Jiang, D. Ji, N. Guan, R. Li, Y . Tang, and Y . Wang, “Real-time scheduling and analysis of processing chains on multi-threaded executor in ROS 2,” in Proc. of IEEE Real-Time Systems Symposium (RTSS) , 2022

  7. [14]

    A new algorithm for real-time scheduling and resource mapping for robot operating systems (ROS),

    K. Chaaban, “A new algorithm for real-time scheduling and resource mapping for robot operating systems (ROS),” Applied Sciences, vol. 13, no. 3, p. 1532, 2023

  8. [15]

    Real-time performance analysis of processing systems on ROS 2 executors,

    Y . Tang, N. Guan, X. Jiang, X. Luo, and W. Yi, “Real-time performance analysis of processing systems on ROS 2 executors,” in Proc. of IEEE 29th Real-Time and Embedded Technology and Applications Symposium (RTAS), 2023

  9. [16]

    Timing analysis and priority-driven enhancements of ROS 2 multi-threaded executors,

    H. Sobhani, H. Choi, and H. Kim, “Timing analysis and priority-driven enhancements of ROS 2 multi-threaded executors,” in Proc. of IEEE 29th Real-Time and Embedded Technology and Applications Symposium (RTAS), 2023

  10. [17]

    Dynamic priority scheduling of multithreaded ROS 2 executor with shared resources,

    A. Al Arafat, K. Wilson, K. Yang, and Z. Guo, “Dynamic priority scheduling of multithreaded ROS 2 executor with shared resources,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems , vol. 43, no. 11, pp. 3732–3743, 2024

  11. [18]

    Thread carefully: Preventing starvation in the ROS 2 multithreaded executor,

    H. Teper, D. Kuhse, M. G ¨unzel, G. von der Br ¨uggen, F. Howar, and J.-J. Chen, “Thread carefully: Preventing starvation in the ROS 2 multithreaded executor,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, vol. 43, no. 11, pp. 3588–3599, 2024

  12. [19]

    Bridging the gap between ROS 2 and classical real-time scheduling for periodic tasks,

    H. Teper, O. Bell, M. G ¨unzel, C. Gill, and J.-J. Chen, “Bridging the gap between ROS 2 and classical real-time scheduling for periodic tasks,” 2024. [Online]. Available: https://arxiv.org/abs/2408.03696

  13. [21]

    An open approach to autonomous vehicles,

    S. Kato, E. Takeuchi, Y . Ishiguro, Y . Ninomiya, K. Takeda, and T. Hamada, “An open approach to autonomous vehicles,” IEEE Micro, vol. 35, no. 6, pp. 60–68, 2015

  14. [22]

    Autoware on board: Enabling autonomous vehicles with embedded systems,

    S. Kato, S. Tokunaga, Y . Maruyama, S. Maeda, M. Hirabayashi, Y . Kit- sukawa, A. Monrroy, T. Ando, Y . Fujii, and T. Azumi, “Autoware on board: Enabling autonomous vehicles with embedded systems,” in Proc. of ACM/IEEE International Conference on Cyber-Physical Systems (ICCPS), 2018

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.