{"id":"5bf27229-5b64-48ca-8323-ac0bdc4c529b","arxiv_id":"2505.06546","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The paper proposes CallbackIsolatedExecutor, a ROS 2 executor with one OS thread per callback, and measures its user-kernel switches, context switches, and memory overhead against standard executors.","lead":"This paper presents a ROS 2 executor that gives every callback its own operating system thread, so each callback can be scheduled directly by the Linux scheduler. The goal is to let real-time scheduling research ignore the middleware scheduler and work directly on callback graphs.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The load-bearing condition is that a callback is never released before a previous invocation finishes; the paper's appeal to non-reentrant code does not establish this, so the claim that the middleware can be ignored remains unvalidated.","rationale":"The reader identified the non-reentrancy assumption as the weakest link, and I agree. My stress test sharpens the point: the paper conflates 'non-reentrant callback code' with 'non-overlapping releases.' In ROS 2, a callback can be non-reentrant in code while still being released again before the previous invocation completes; the executor simply queues the next invocation and runs it after the current one returns. Under that condition, the one-to-one callback-thread mapping alone does not make the middleware transparent, because the middleware queue, not the OS scheduler, decides when the queued invocation starts. This is a real, load-bearing gap for the central claim. The microbenchmark cannot settle it because it uses regular 10 ms timers and short callbacks. At the same time, the paper candidly states the assumption and frames the formulation as a design constraint, so the result is a plausible conditional contribution rather than a refuted one. I would keep the CONDITIONAL verdict; the next step is an empirical test of the assumption on realistic workloads.","tokens_in":6520,"tokens_out":6806,"duration_ms":74862,"concrete_test":"Instrument the open-source CallbackIsolatedExecutor to log, for each callback, event-arrival timestamps and callback start/end timestamps (e.g., via rclcpp event hooks plus futex/wait tracing), and run a representative ROS 2 workload—such as a 10 ms timer driving a 30-50 ms subscription callback, or an Autoware module—and count occurrences where a new event arrives before the prior invocation finishes and the pending-queue length exceeds one. If any such occurrence is observed, the ignore-the-middleware model underpredicts response time relative to the actual executor.","verdict_should_be":"UNCHANGED","load_bearing_attack":"To ignore the middleware layer, Section II requires that 'a callback never gets ready for the next period until the previous execution finishes (i.e., its minimum inter-arrival time exceeds its worst-case turnaround time).' This is a timing property of the workload plus the scheduling policy, not a property of callback code. The paper justifies the constraint in Section III by saying that 'many practical ROS 2 applications do not implement reentrant callbacks,' but a non-reentrant callback only means the code is not written for concurrent invocations; it can still receive a new message or timer tick while the previous invocation is running, in which case the executor queues the next invocation and executes it back-to-back after the current one returns. When that happens, the thread's runnable state is not solely determined by its OS scheduling parameters: the middleware's pending queue determines when the next invocation starts and can add arbitrary release jitter. Because worst-case turnaround time itself depends on the OS scheduler (preemption by higher-priority threads, affinity restrictions), the condition can fail in exactly the stressed scheduling regimes the paper targets. No measurement or workload study is provided to show that real ROS 2 applications satisfy the inter-arrival > turnaround condition; the experiments use a 10 ms timer with short callbacks and do not stress this boundary. The central claim is therefore a conditional design statement, not a demonstrated property of commodity ROS 2 systems.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CallbackIsolatedExecutor, a ROS 2 executor that establishes a persistent dedicated OS thread per callback (through one-callback CallbackGroups) so that Linux scheduling policy, priority, and affinity can be configured at the thread level and the middleware executor can be ignored in scheduling analysis. Under the assumption that a callback is never released again before its previous invocation completes, the ROS 2 scheduling problem is reformulated as DAG scheduling over callbacks, bypassing nested scheduling. Overhead measurements (user-kernel switches, context switches, memory) compare the new executor with ROS 2 SingleThreadedExecutor and MultiThreadedExecutor for inter-process and intra-process publish-subscribe with varying callback counts.","tokens_in":6752,"tokens_out":4759,"duration_ms":48329,"significance":"If the central condition holds, the paper removes a genuine obstacle: executor-specific response-time analyses would no longer be needed, and existing fixed-priority or DAG scheduling theory could be applied directly to ROS 2 callback graphs. The implementation is open source, the overhead data are direct measurements with no fitted parameters, and the design is clearly described. The main value is in enabling future ROS 2 real-time scheduling research to model only callback graphs. However, the paper's headline claims are conditional on an unvalidated non-reentrancy/timing assumption and on overhead comparisons that are not statistically substantiated; the result is a plausible design statement rather than a fully demonstrated middleware-transparency result.","major_comments":[{"comment":"The central 'ignore the middleware' claim rests on the condition stated in Section II that a callback never gets ready for the next period until the previous execution finishes (minimum inter-arrival time exceeding worst-case turnaround time), and Section III justifies this by asserting that many practical ROS 2 applications do not implement reentrant callbacks. The second statement does not establish the first. A non-reentrant callback is code that is not written for concurrent invocations; a new timer tick or message can still arrive while an invocation is running, causing the executor or the underlying DDS queue to hold it until the thread becomes free. In that case the next invocation's release time and jitter are determined by the middleware queue, not solely by the thread's OS scheduling parameters. Moreover, worst-case turnaround time depends on OS scheduling, preemption, and affinity, so the condition can fail in exactly the overloaded or stressed regimes for which per-callback priority assignment is intended. The experiments in Section IV use a 10 ms timer with short callbacks and therefore trivially satisfy the condition; they provide no evidence about real ROS 2 workloads. The central claim is thus conditional, and the condition is not validated by any measurement, workload study, or quantitative argument.","section":"Section II and Section III"},{"comment":"The abstract states that the costs of CallbackIsolatedExecutor 'remain lower than those of the MultiThreadedExecutor, regardless of the number of callbacks,' but Fig. 5(a) shows that for intra-process publish-subscribe, CallbackIsolatedExecutor's user-kernel mode switches exceed MultiThreadedExecutor's at most callback counts; the text in Section IV itself acknowledges that MultiThreadedExecutor has a 'slight advantage' in this condition. Since the abstract is the published claim, it needs correction to reflect that the advantage holds for context-switch count and memory use, but not for user-kernel mode switches under intra-process communication, or the claim is contradicted by the paper's own data.","section":"Abstract and Section IV"},{"comment":"The overhead ratio claim is not statistically substantiated. Section IV reports that CallbackIsolatedExecutor stays within a fixed ratio (1.4x for inter-process and 5x for intra-process) of SingleThreadedExecutor, but no experimental repetitions, confidence intervals, or standard deviations are reported for any of the three measurements. Without information on the number of trials and run-to-run variance, the reader cannot assess whether the ratio is stable across runs or whether the apparent trends in Fig. 5 are meaningful. The fixed-ratio statement is a central quantitative claim and needs at least a statement of the number of trials and variance.","section":"Section IV"},{"comment":"The conclusion broadens the claim by saying that 'the real-time community no longer needs to account for the existence of the Executor in ROS 2 scheduling in practical cases' and that 'real-world ROS 2 systems rarely have dozens of callbacks per node,' but no evidence is provided for this scalability claim. The experiments only go up to 24 callbacks, and one thread per callback may incur significant context-switch and wait-set contention in larger systems. The future-work statement that the approach will be evaluated on Autoware further indicates that the central practical claim remains to be demonstrated; the conclusion should be scoped accordingly.","section":"Section V"}],"minor_comments":[{"comment":"The figure would be more informative with error bars or shaded confidence intervals, and the legend entries such as 'single(separate)' should be expanded or defined in the caption for readability.","section":"Figure 5"},{"comment":"The open-source URL in the footnote contains a space ('callback isolated executor'); it should be a proper link or code-formatted path with an underscore, otherwise the URL is not usable.","section":"Footnote"},{"comment":"The text says that 'in practice, CallbackGroups, not callbacks, are mapped to threads.' This is an important implementation detail that should be stated more prominently, since it means the one-to-one correspondence is actually between CallbackGroups and threads, and the one-callback-per-group constraint is what makes it per-callback.","section":"Section III"},{"comment":"The term 'user-kernel switches' is used throughout; it would be clearer to define it explicitly (e.g., as system calls or user-kernel mode transitions) so that readers can interpret the measurement methodology.","section":"Section II"},{"comment":"The constraint that each CallbackGroup contains one callback may require changes to existing third-party ROS 2 nodes; the paper would benefit from a brief discussion of the migration burden for black-box nodes.","section":"Section III"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this as a WIP, not a full theory paper. The new thing is the persistent one-to-one mapping between callbacks and OS threads, implemented as CallbackIsolatedExecutor. That is a concrete, sensible way to make per-callback scheduler configuration possible, and it directly removes the executor from the scheduling model, at least in the cases where the mapping holds. The overhead study is the real contribution: direct measurements of user-kernel switches, context switches, and memory for 1-24 callbacks, across inter- and intra-process setups. Those numbers support the main cost claim: the isolated executor is roughly in line with the SingleThreadedExecutor and beats MultiThreadedExecutor in most settings.\n\nThe paper does have a few honest soft spots. The abstract says costs 'remain lower than those of MultiThreadedExecutor,' but the body admits that user-kernel switches in the same-process case are slightly worse. That overstatement should be fixed. There are no error bars or repeated-run statistics, which matters because the differences at higher callback counts are not huge. And the evaluation is a microbenchmark with 10ms timers and short callbacks; it does not stress the one condition the whole transparency argument depends on.\n\nThat condition is the real issue. The paper requires that a callback never become ready for its next invocation before the previous invocation finishes. The authors justify it by saying many ROS 2 applications do not implement reentrant callbacks. But non-reentrant code can still receive a new message while the previous invocation is running; the executor then queues it and runs it back-to-back. That queueing is the middleware layer the paper wants to ignore. The condition is a workload/scheduling property, not a code property, and no measurements or workload study show it holds in practice. The paper does suggest splitting long callbacks, but that is not demonstrated either. So the 'ignore the executor' claim is valid only under an unverified assumption. The DAG scheduling formulation is fine, but it inherits that condition.\n\nCitation pattern looks okay. The self-citation [20] is a relevant DAG model from one of the authors; it is not used to inflate the contribution. The related work on EventsExecutor is appropriately cited.\n\nWho should read this: anyone working on ROS 2 real-time scheduling. It is a credible WIP with open-source code. A serious referee should not desk-reject it; the reviewer should push the authors to either validate the non-reentrancy condition with real workloads or state the contribution as conditional on that assumption. I would accept it for a WIP track with that caveat.","headline":"A solid WIP that makes per-callback OS scheduling in ROS 2 concrete, but its headline 'ignore the middleware' claim rests on an unvalidated timing assumption.","tokens_in":7292,"tokens_out":2671,"would_cite":true,"duration_ms":23759,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Giving each ROS 2 callback its own OS thread removes the middleware layer from real-time scheduling models.","keywords":["ROS 2","real-time scheduling","middleware transparency","callback","OS thread","nested scheduling","DAG scheduling","CallbackIsolatedExecutor"],"falsifier":"Run a publish-subscribe pair in which the subscriber callback's worst-case execution time exceeds the publisher's period: if queued invocations are observed and end-to-end latency deviates from a per-callback DAG model that permits only one outstanding invocation per vertex, the middleware-transparent assumption fails for that workload.","tokens_in":6318,"feed_emoji":"🤖","tokens_out":5503,"duration_ms":48129,"temperature":0.7,"pith_summary":"This paper argues that nested scheduling in ROS 2, where the executor layer queues and dispatches callbacks onto shared OS threads, is an artificial implementation constraint rather than a necessity. If each callback is permanently bound to its own OS thread, the executor becomes invisible: OS scheduling policy, priority, and affinity apply directly to individual callbacks, and the system reduces to a DAG of callback vertices. The authors build CallbackIsolatedExecutor, a ROS 2 executor that realizes this one-to-one mapping, and measure that its overhead in user-kernel switches, context switches, and memory stays below the MultiThreadedExecutor and within a fixed ratio of the SingleThreadedExecutor. If the claim holds, future ROS 2 real-time scheduling research can ignore the executor layer entirely and reuse classical real-time and DAG scheduling theory on the callback graph.","feed_headline":"One thread per callback lets ROS 2 skip the executor layer","feed_subtitle":"A new executor maps callbacks to threads one-to-one, so classical DAG scheduling theory applies directly.","key_machinery":"CallbackIsolatedExecutor, a ROS 2 executor variant in which every callback is permanently mapped to a dedicated OS thread, with one callback per CallbackGroup and no wait-set shared across threads. This mapping lets Linux scheduling policies (SCHED_DEADLINE, SCHED_FIFO, and CFS with nice values) and CPU affinity be applied per callback, which is what allows the middleware layer to drop out of the scheduling model.","core_discovery":"The central discovery is that the middleware layer in ROS 2 can be made transparent to real-time scheduling by establishing a persistent one-to-one correspondence between callbacks and OS threads. Under this design, each callback lives in its own CallbackGroup and owns a dedicated thread, so scheduler, priority, and affinity are set per callback at the OS level. With the additional condition that a callback does not become ready for its next invocation before its previous invocation finishes, the executor's queueing behavior no longer affects execution order, and the scheduling problem becomes classical DAG scheduling on callback vertices. The paper implements CallbackIsolatedExecutor and reports that user-kernel switches and context switches remain below MultiThreadedExecutor for all tested callback counts (1 to 24) and within roughly 1.4x (inter-process) and 5x (intra-process) of SingleThreadedExecutor, with memory usage only slightly higher.","pith_inferences":["The fixed overhead ratios were measured on a synthetic publish-subscribe pair; production robot workloads with mixed message sizes and shared-memory intra-process communication could shift those ratios, so the bounded-ratio claim still awaits testing on large node graphs.","The non-reentrancy condition could be enforced automatically rather than assumed: a framework could detect when a callback is released while still running and either queue, split, or reject the release, widening the class of applications that can safely ignore the executor.","If the approach generalizes, the distinction between executor-level scheduling policies and OS-level policies may become obsolete for ROS 2, and a direct comparison with event-pushing executor designs would be the natural next test of where queueing overhead remains.","The latency benefit may be larger than the paper measures: eliminating shared wait-set reconstruction and lock contention is described qualitatively as an advantage but not isolated as an end-to-end latency number in the experiments."],"forward_implications":["Scheduling analysis of ROS 2 applications reduces to DAG scheduling on callback vertices, so existing real-time and DAG scheduling algorithms apply without executor-specific analysis.","CallbackIsolatedExecutor incurs lower context-switch and user-kernel-switch overhead than MultiThreadedExecutor regardless of the number of callbacks.","Overhead relative to SingleThreadedExecutor stays within a bounded ratio (1.4x inter-process, 5x intra-process), so the cost of isolation does not grow with callback count.","Designers must keep callbacks non-reentrant: each callback's minimum inter-arrival time must exceed its worst-case turnaround time, otherwise the middleware cannot be ignored.","CPU time not consumed by real-time DAGs on SCHED_DEADLINE or SCHED_FIFO remains available to CFS tasks, so real-time scheduling and high utilization can coexist."],"supporting_citations":[{"why":"Defines end-to-end timing analysis of ROS 2 processing chains under nested scheduling, the baseline complexity this design aims to remove.","marker":"[3]"},{"why":"Extends the nested-scheduling model to multi-executor systems, showing why the executor layer is treated as part of scheduling today.","marker":"[5]"},{"why":"Models ROS 2 systems as DAGs for latency management, the graph formulation the proposed scheduler inherits.","marker":"[6]"},{"why":"Models robot systems as DAGs for on-device CPU scheduling, part of the graph-based modeling this design targets.","marker":"[7]"},{"why":"An earlier attempt to bridge ROS 2 and classical real-time scheduling whose limited functionality motivates CallbackIsolatedExecutor.","marker":"[19]"},{"why":"Supplies the multi-deadline DAG model the paper adopts when applying per-DAG deadlines to complex applications.","marker":"[20]"}],"fun_headline_variants":["One callback per thread: ROS 2 skips the executor for scheduling","Middleware-transparent callbacks: direct OS scheduling in ROS 2","CallbackIsolatedExecutor: thread-per-callback cuts scheduling layers","ROS 2 real-time: map callbacks to threads, ignore the executor","Thread-per-callback design makes ROS 2 middleware transparent"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim collapses if a callback can be released again while a previous invocation is still running; the paper assumes each callback's minimum inter-arrival time exceeds its worst-case turnaround time, and asserts without workload measurements that many practical ROS 2 applications satisfy this.","fun_headline_variants_meta":{"raw":{"variants":["One callback per thread: ROS 2 skips the executor for scheduling","Middleware-transparent callbacks: direct OS scheduling in ROS 2","CallbackIsolatedExecutor: thread-per-callback cuts scheduling layers","ROS 2 real-time: map callbacks to threads, ignore the executor","Thread-per-callback design makes ROS 2 middleware transparent"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000212,"raw_usage":{"total_tokens":1408,"prompt_tokens":924,"completion_tokens":484,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":540,"completion_tokens_details":{"reasoning_tokens":393}},"tokens_in":540,"tokens_out":484,"duration_ms":4575,"temperature":1.0,"reasoning_tokens":393,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:38:52.685031+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a publish-subscribe pair in which the subscriber callback's worst-case execution time exceeds the publisher's period: if queued invocations are observed and end-to-end latency deviates from a per-callback DAG model that permits only one outstanding invocation per vertex, the middleware-transparent assumption fails for that workload.","supporting_citations":[{"cited_title":"End-to-end timing analysis in ROS2,","cited_arxiv_id":null,"evidence_quote":"Defines end-to-end timing analysis of ROS 2 processing chains under nested scheduling, the baseline complexity this design aims to remove."},{"cited_title":"End-to-end timing analysis and optimization of multi- executor ROS 2 systems,","cited_arxiv_id":null,"evidence_quote":"Extends the nested-scheduling model to multi-executor systems, showing why the executor layer is treated as part of scheduling today."},{"cited_title":"Automatic latency management for ROS 2: Benefits, challenges, and open problems,","cited_arxiv_id":null,"evidence_quote":"Models ROS 2 systems as DAGs for latency management, the graph formulation the proposed scheduler inherits."},{"cited_title":"On-device CPU scheduling for robot sys- tems,","cited_arxiv_id":null,"evidence_quote":"Models robot systems as DAGs for on-device CPU scheduling, part of the graph-based modeling this design targets."},{"cited_title":"Bridging the Gap between ROS~2 and Classical Real-Time Scheduling for Periodic Tasks","cited_arxiv_id":"2408.03696","evidence_quote":"An earlier attempt to bridge ROS 2 and classical real-time scheduling whose limited functionality motivates CallbackIsolatedExecutor."},{"cited_title":"Work-in-progress: Multi-deadline DAG scheduling model for autonomous driving systems,","cited_arxiv_id":null,"evidence_quote":"Supplies the multi-deadline DAG model the paper adopts when applying per-DAG deadlines to complex applications."}],"review_version":1}