{"id":"24cf4c7e-0fcf-4612-8adb-08843c92adeb","arxiv_id":"2501.18942","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Autoware and Apollo differ in module design, and Apollo's shared-memory middleware is faster but more memory-hungry than Autoware's serialized DDS middleware.","lead":"This paper compares the two leading open-source autonomous driving platforms, Autoware and Apollo, and runs middleware benchmarks between them. It finds Apollo's CyberRT has lower transmission latency but higher memory use, while Autoware's ROS 2 based FastDDS is slower on large data.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The middleware benchmark confounds transport choice with middleware design: FastDDS is forced onto TCP while CyberRT uses shared memory, so the headline latency gap in Table II does not isolate DDS versus CyberRT.","rationale":"The reader's weakest assumption identifies the representativeness of the standalone middleware benchmark, and I agree that this is the central soft spot. My concern is more specific: the experimental setup in Section IV-B configures FastDDS exclusively over TCP while CyberRT uses shared memory, so the benchmark conflates transport protocol with middleware architecture. This is load-bearing because the paper's headline claim is precisely about which middleware is faster for large-scale data; if FastDDS were tested with its own shared-memory transport, the magnitude and possibly the direction of the gap could change. The reader also noted the lack of error bars and trial counts, which compounds the issue but is secondary. I do not think the paper should be rejected; the architectural description and source-level enumeration are useful, and the qualitative tradeoff (shared memory lower latency, higher memory) is plausible. However, the experimental support for the quantitative claim is weaker than the conclusion suggests, so the conditional verdict is appropriate. I marked agreement as 'partial' because the reader emphasized representativeness generally, whereas I would emphasize the specific transport confound and the missing end-to-end latency definition as the decisive test.","tokens_in":7819,"tokens_out":2776,"duration_ms":29186,"concrete_test":"Rerun the Section IV-B experiment under three conditions on the same host: (1) FastDDS with TCP as configured in the paper, (2) FastDDS with its shared-memory transport (and separately with UDP), and (3) CyberRT. For each condition, measure end-to-end latency from the publisher's send call to the subscriber's receive callback for the same message sizes and frequencies, with at least 30 trials and report mean, median, and standard deviation. If FastDDS over shared memory or UDP closes the gap to CyberRT, the headline claim must be qualified; if CyberRT still wins by an order of magnitude, the conclusion survives but with a corrected transport-controlled interpretation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central quantitative conclusion, stated in Section V, is that CyberRT achieves lower latency for large-scale data transmission than FastDDS. That conclusion rests entirely on the Section IV-B benchmark, which compares FastDDS configured with TCP, explicitly 'to minimize message loss,' against CyberRT's shared-memory transport. This is a transport confound: TCP on loopback adds serialization, copying, and acknowledgment overhead that shared memory avoids. FastDDS also supports shared-memory and UDP transports, and Autoware's ROS2 stack can use those transports; the benchmark therefore measures a particular transport configuration, not the architectural difference between DDS and CyberRT. Table II reports a 22,000x latency gap for high-end sensor data (38075 us vs 1.72 us), but this magnitude is not interpretable as a DDS-versus-CyberRT difference. In addition, no error bars, trial counts, or end-to-end latency definition are provided; a CyberRT latency of 0.11 us for the functional-module case is so close to a single shared-memory write that it raises the question of whether the measurement captured subscriber-side receipt or only publisher-side enqueue. If the latter, the latency comparison is not apples-to-apples.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents a head-to-head comparison of two open-source autonomous driving software platforms, Autoware and Apollo. The authors qualitatively analyze the middleware, localization, perception, planning, and control modules of each platform, then quantitatively compare module subcomponents and benchmark the middleware layers (FastDDS in Autoware versus CyberRT in Apollo) in terms of latency, CPU usage, memory consumption, and message loss rate. The central quantitative claim, stated in Section V, is that CyberRT achieves lower latency for large-scale data transmission than FastDDS at the cost of higher memory usage, based on the Section IV-B benchmark. The paper also offers qualitative observations about architectural differences in planning and control design.","tokens_in":7988,"tokens_out":3573,"duration_ms":32608,"significance":"If the quantitative comparison were methodologically sound, the paper would provide a useful practical reference for researchers and engineers selecting between Autoware and Apollo. The qualitative module analysis and the enumeration of subcomponents in Table I are a convenient synthesis of the two platforms' architectures, and the paper does not rely on circular reasoning or fitted parameters. However, the central quantitative contribution rests on a thin benchmark that confounds transport choice with middleware design, and it lacks the statistical reporting needed to support the strong conclusions. The paper's value currently lies more in its qualitative survey aspect than in its performance measurements.","major_comments":[{"comment":"The latency comparison confounds transport choice with middleware architecture. FastDDS is configured with TCP explicitly to minimize message loss, while CyberRT uses shared memory. TCP on loopback adds serialization, copying, and acknowledgment overhead that shared memory avoids; FastDDS also supports shared-memory and UDP transports, and Autoware's ROS2 stack can be configured to use them. Therefore the reported latency gap (e.g., 38075 us versus 1.72 us for high-end sensor data) cannot be attributed to DDS versus CyberRT as architectural alternatives. The authors should either benchmark FastDDS with at least one non-TCP transport, or explicitly scope the conclusion to the specific TCP configuration tested.","section":"Section IV-B, Table II"},{"comment":"No trial counts, error bars, standard deviations, or definition of the measured latency endpoint are provided. The reported CyberRT latency of 0.11 us for the functional-module case is close to the cost of a single shared-memory read/write on the test hardware, which raises the question of whether the measurement captures only publisher-side enqueue rather than subscriber-side receipt. The authors should specify the measurement methodology, including which timestamps mark start and end, the number of warm-up and measured iterations, and the variability across runs, so that the comparison is reproducible and apples-to-apples.","section":"Section IV-B, Fig. 9 and Table II"},{"comment":"The subcomponent counts are presented as quantitative evidence of functional coverage, but no explicit counting criteria are given. It is unclear what qualifies as a subcomponent, how overlapping or deprecated components are handled, whether the counts reflect maintained code or documentation entries, and how the line is drawn between, for instance, Autoware's 17 LiDAR perception subcomponents and Apollo's 9. Without a defined and auditable rubric, Table I cannot be independently verified, which weakens the paper's claim of a 'systematic' quantitative comparison.","section":"Section IV-A, Table I"},{"comment":"The benchmark tests isolated middleware with synthetic messages, not the actual Autoware and Apollo pipelines. The conclusion in Section V extrapolates from this microbenchmark to 'large-scale data transmission' in the full-stack platforms, but the message sizes, frequencies, number of subscribers, and transport configurations in the benchmark are not tied to the real workloads of the two platforms. The authors should either include application-level or realistic sensor-data workloads, or temper the conclusion to a microbenchmark observation that does not generalize to full-stack performance.","section":"Section IV-B, Section V"}],"minor_comments":[{"comment":"The heading 'Localizaiton' should be spelled 'Localization'.","section":"Section III-B"},{"comment":"The phrase 'comparing the the subcomponents' contains a duplicated article and should be corrected.","section":"Section IV-A"},{"comment":"The word 'funeded' should be 'funded'.","section":"Acknowledgment"},{"comment":"Reference [2] lists the first author as 'AndreiGavrilov' with a missing space; this appears to be a formatting error.","section":"References"},{"comment":"The statement that DDS 'creates TCP or UDP sockets' is an oversimplification, since DDS implementations also support shared-memory transports; this oversimplification is directly related to the confound identified in the benchmark.","section":"Section III-A"},{"comment":"The phrase 'Data is acquired' should be 'Data are acquired' for grammatical agreement, or rephrased.","section":"Fig. 6 caption"},{"comment":"The paper would benefit from a statement on code and data availability for the middleware benchmark, as none is currently provided.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The main risk is the transport confound in the middleware benchmark: if the authors cannot rerun the experiments with properly matched transports or cannot provide the measurement details requested, the quantitative claims should be substantially downgraded and the paper reframed primarily as a qualitative comparison. The topic is within scope for a robotics venue, but the current experimental rigor is not at the level required to support the strong latency and memory conclusions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know up front. First, the paper's actually useful parts are the architectural side-by-side and the Table I subcomponent counts; that is real work and not in the cited surveys. Second, the headline quantitative result—CyberRT far lower latency than FastDDS—is undermined by a transport confound, and the benchmark as reported should not be taken at face value.\n\nThe module review is solid. It is organized by localization, perception, planning, and control, and it cleanly lays out differences like Apollo's RTK-GNSS and ESKF, Autoware's NDT-based LiDAR, scenario-based planning differences, and controller choices. Enumerating subcomponents from source and docs gives engineers a concrete sense of functional coverage. That contribution stands on its own.\n\nThe middleware experiment is the weak spot. FastDDS is run over TCP \"to minimize message loss,\" while CyberRT uses shared memory. TCP on loopback carries serialization, copying, and ACK overhead that shared memory avoids, so the 22,000x gap in the high-end sensor row is mostly a transport comparison, not a DDS-versus-CyberRT architecture comparison. FastDDS also supports shared-memory and UDP transports; Autoware's ROS2 stack can use them, so the chosen configuration understates what FastDDS can do in practice. Additionally, there are no error bars, no trial counts, and no definition of latency is given. 0.11 microseconds is close to a single shared-memory write and could be capturing publisher-side enqueue rather than end-to-end subscriber receipt. The authors should be asked to rerun with multiple transport configurations, report distributions, and specify the measurement point.\n\nThe memory tradeoff direction is plausible—shared memory allocation should use more RSS—but the size of the memory difference (roughly 6–100 MB) is a minor point and does not rescue the latency claim. The qualitative direction of the memory finding is probably right, but the exact numbers are not reliable.\n\nBottom line: this is a useful practical reference for engineers choosing between Autoware and Apollo, not a scientific result that resolves a question. It deserves a serious referee: the substance is there, but the experimental section needs major revision and a reproducible protocol. I would accept it for review and then hold it to a higher standard on the benchmark.","headline":"The architecture comparison and subcomponent counts are genuinely useful; the middleware benchmark, however, compares TCP against shared memory rather than DDS against CyberRT, so the headline latency gap should not be trusted as reported.","tokens_in":8538,"tokens_out":1632,"would_cite":false,"duration_ms":15065,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Apollo's shared-memory middleware outperforms Autoware's DDS on latency but uses more memory.","keywords":["Autoware","Apollo","autonomous driving software platform","middleware","CyberRT","FastDDS","shared memory","latency"],"falsifier":"Run the same message-size, frequency, and subscriber sweeps with FastDDS configured exactly as Autoware's default launch files configure it (including any shared-memory transport), and measure end-to-end latency of a complete perception-to-planning pipeline on identical hardware; if the CyberRT-versus-FastDDS latency gap shrinks by orders of magnitude or reverses, the paper's central trade-off does not generalize.","tokens_in":7583,"feed_emoji":"🚗","tokens_out":9070,"duration_ms":70184,"temperature":0.7,"pith_summary":"This paper provides a head-to-head, quantitative basis for choosing between the two leading open-source autonomous-driving stacks, Autoware and Apollo. It compares how each platform implements the core modules—localization, perception, planning, and control—by counting subcomponents from source code, then runs a controlled middleware benchmark pitting FastDDS (Autoware's DDS) against CyberRT (Apollo's shared-memory middleware). The headline result is that CyberRT is far faster at transferring large sensor messages (0.11 vs 153 µs in the functional-module test, 1.72 vs 38075 µs for high-end sensor data) while using more memory, with no message loss on either side. The claim matters because middleware latency is a real-time bottleneck for autonomous driving, and platform selection has long-term integration consequences.","feed_headline":"Apollo's shared memory beats Autoware's DDS on latency","feed_subtitle":"Head-to-head benchmark: CyberRT delivers data at 0.11 µs vs 153 µs for FastDDS, at the cost of extra RAM.","key_machinery":"The central object is the middleware layer: FastDDS, the Data Distribution Service implementation used in Autoware, versus CyberRT, the proprietary communication middleware used in Apollo. CyberRT's shared-memory transport with zero-copy reads, which eliminates serialization, is the mechanism that produces low latency; FastDDS's socket-based serialization and fragmentation is the mechanism that produces higher latency at large data sizes and lower memory use.","core_discovery":"Autoware and Apollo take structurally different middleware approaches: Autoware runs on ROS2 with DDS, which serializes and fragments messages before transmission, while Apollo uses CyberRT, which writes sensor data directly into shared memory and reads it from there without serialization. In the paper's measurements, CyberRT's latency stays below about 2 µs even for high-end sensor data, while FastDDS rises to tens of milliseconds; memory usage is correspondingly higher for CyberRT, and neither middleware loses messages. The paper also claims the two platforms differ meaningfully in how they implement localization, perception, planning, and control, so the choice of platform is not just a benchmark score but a set of architectural trade-offs.","pith_inferences":["If the microbenchmark transfers to real stacks, Autoware's DDS serialization becomes a scalability ceiling for sensor-heavy configurations, which would push adopters toward Apollo or toward a shared-memory transport inside ROS2.","A full-stack benchmark would need to include application-level compute times, because middleware latency gains can be swamped by perception and planning execution; the paper measures only the middleware layer.","The memory cost of CyberRT (tens of megabytes in these tests) is likely to grow with the number of concurrent subscribers and message sizes, so the trade-off may widen in multi-sensor fusion systems.","Autoware's ROS2 base gives it access to a broader ecosystem of tools and packages, and a developer might accept DDS overhead for that interoperability; the paper does not quantify ecosystem value."],"forward_implications":["Platform selectors can expect Apollo and CyberRT to pass large sensor messages far faster than Autoware and FastDDS in similar configurations, at the cost of higher memory use.","Autoware's DDS serialization is the main source of latency growth with message size, so latency-sensitive Autoware deployments should target the middleware layer rather than the algorithms when optimizing.","Neither middleware loses messages in the tested conditions, so message integrity is not a differentiator at these scales.","The module-level comparison shows the two platforms are not interchangeable: Autoware has more localization subcomponents, Apollo has more camera-perception and planning scenario coverage, and their controller designs differ.","Apollo's sequential-stage planning architecture can avoid computing trajectories for inactive scenarios, which the paper contrasts with Autoware's parallel scenario execution."],"supporting_citations":[{"why":"This systematic analysis of DDS implementations supplies the basis for characterizing FastDDS in the benchmark.","marker":"[6]"},{"why":"The FastDDS documentation is used to configure the middleware and define the TCP settings in the experiments.","marker":"[10]"},{"why":"Documents the memory-centric communication mechanism and the serialization bottleneck that motivate the middleware comparison.","marker":"[23]"},{"why":"Describes CyberRT's shared-memory, low-latency communication design, which the paper credits for Apollo's latency advantage.","marker":"[24]"},{"why":"Explains the ROS2/DDS middleware relationship, grounding Autoware's use of DDS.","marker":"[26]"},{"why":"Provides the earlier qualitative Autoware-versus-Apollo comparison that this paper extends with quantitative measurements.","marker":"[29]"},{"why":"The Apollo source code is used to enumerate the subcomponents of Apollo's core modules.","marker":"[4]"},{"why":"The Autoware source code is used to enumerate the subcomponents of Autoware's core modules.","marker":"[17]"}],"fun_headline_variants":["Apollo's shared memory is 1400x faster than Autoware's DDS","CyberRT vs FastDDS: 0.11 µs vs 153 µs latency gap","Open-source AV platforms: Apollo leads on middleware speed, not memory","Pick your AV platform: Apollo's CyberRT wins latency, loses RAM"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that a standalone microbenchmark of FastDDS versus CyberRT on synthetic messages faithfully represents how each platform performs inside a full autonomous-driving stack, even though real pipelines use different communication settings, message sizes, and shared-memory access patterns.","fun_headline_variants_meta":{"raw":{"variants":["Apollo's shared memory is 1400x faster than Autoware's DDS","CyberRT vs FastDDS: 0.11 µs vs 153 µs latency gap","Open-source AV platforms: Apollo leads on middleware speed, not memory","Pick your AV platform: Apollo's CyberRT wins latency, loses RAM"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000747,"raw_usage":{"total_tokens":3295,"prompt_tokens":877,"completion_tokens":2418,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":493,"completion_tokens_details":{"reasoning_tokens":2330}},"tokens_in":493,"tokens_out":2418,"duration_ms":15634,"temperature":1.0,"reasoning_tokens":2330,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T21:51:51.793410+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same message-size, frequency, and subscriber sweeps with FastDDS configured exactly as Autoware's default launch files configure it (including any shared-memory transport), and measure end-to-end latency of a complete perception-to-planning pipeline on identical hardware; if the CyberRT-versus-FastDDS latency gap shrinks by orders of magnitude or reverses, the paper's central trade-off does not generalize.","supporting_citations":[{"cited_title":"Systematic analysis of dds implementations","cited_arxiv_id":null,"evidence_quote":"This systematic analysis of DDS implementations supplies the basis for characterizing FastDDS in the benchmark."},{"cited_title":"Fast dds documentation","cited_arxiv_id":null,"evidence_quote":"The FastDDS documentation is used to configure the middleware and define the TCP settings in the experiments."},{"cited_title":"Memory-centric communication mechanism for real-time autonomous navigation applications","cited_arxiv_id":null,"evidence_quote":"Documents the memory-centric communication mechanism and the serialization bottleneck that motivate the middleware comparison."},{"cited_title":"A robotic communication middleware combining high performance and high reliability","cited_arxiv_id":null,"evidence_quote":"Describes CyberRT's shared-memory, low-latency communication design, which the paper credits for Apollo's latency advantage."},{"cited_title":"Exploring the performance of ros2","cited_arxiv_id":null,"evidence_quote":"Explains the ROS2/DDS middleware relationship, grounding Autoware's use of DDS."},{"cited_title":"Performance of open autonomous vehicle platforms: Autoware and apollo","cited_arxiv_id":null,"evidence_quote":"Provides the earlier qualitative Autoware-versus-Apollo comparison that this paper extends with quantitative measurements."},{"cited_title":"Apollo github","cited_arxiv_id":null,"evidence_quote":"The Apollo source code is used to enumerate the subcomponents of Apollo's core modules."},{"cited_title":"Autoware github","cited_arxiv_id":null,"evidence_quote":"The Autoware source code is used to enumerate the subcomponents of Autoware's core modules."}],"review_version":1}