{"id":"e3a152ef-4f35-4f9b-aa57-088c89ea2184","arxiv_id":"2412.18104","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A systematic set of Linux kernel fixes for workqueue, ASID, jiffies, and device management reduces cross-core interference on isolated real-time cores, with up to 4x lower worst-case jitter than RT-Linux in the authors' measurements.","lead":"Engineers from Huawei and Peking University report six years of patches that reduce cross-core timing interference in the Linux kernel, targeting task scheduling, resource sharing, and concurrency. Measured against RT-Linux on an isolated-core testbed, their combined system cuts worst-case jitter by 4x and improves schedulability by up to 3.5x.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline 8.7x jitter and 11.5x schedulability improvements compare openEuler (which includes PREEMPT_RT and many non-author changes) against vanilla Linux 5.10, so they do not isolate the paper's 34 fixes; only the 4.0x jitter and 3.5x schedulability improvements over RT-Linux are attributable…","rationale":"The reader's CONDITIONAL verdict and the identified weakest assumption are sensible: Table II's maximum-latency values are empirical sample maxima, not verified worst-case release-jitter bounds, so the schedulability results in §V-C inherit uncertainty from those samples. However, my stress-test pass finds a more load-bearing problem that the reader's weakest-assumption statement only partially captures. The abstract and conclusion attribute the 8.7x jitter reduction and the 11.5x schedulability improvement to the paper's fixes, but the experiment underlying those numbers compares openEuler against vanilla Linux 5.10. Since openEuler includes PREEMPT_RT and many other changes beyond the authors' patches, the measured difference between these two systems cannot be attributed to the 34 fixes. The controlled comparison that does isolate the fixes, openEuler versus RT-Linux, shows 4.0x jitter and 3.5x schedulability improvements. The paper therefore overstates its contribution in the headline numbers, even though the underlying direction and the existence of upstream-merged fixes are credible. I do not recommend changing the verdict to REJECT: the fixes are real, many are publicly inspectable, and the paper is an experience report rather than a formal evaluation. CONDITIONAL remains appropriate, with the condition being that the authors must either provide a proper A/B comparison isolating their patches or revise the headline claims to match the RT-Linux-controlled comparison. The concrete test I propose is a direct cherry-pick experiment, which is feasible because the upstream patch identifiers are listed in the references.","tokens_in":21457,"tokens_out":4144,"duration_ms":41746,"concrete_test":"Build two kernels from the same source base: Linux 5.10.201 with PREEMPT_RT (matching RT-Linux), and the identical tree with the authors' upstream patches from references [35]-[45] cherry-picked on top. Run the same interference workload from Table IV on the 72 non-isolated cores while running cyclictest with a 50 us wake-up period on the 24 isolated cores, for a run of at least one hour per configuration, and record the maximum latency. If the patched kernel's maximum latency is approximately 4x lower than the unpatched RT-Linux kernel, the authors' fixes are validated at the 4.0x level and the 8.7x figure should be explicitly attributed to the combined RT-Linux-plus-fixes stack. If the difference is much smaller, the paper's headline improvement is an artifact of comparing openEuler against vanilla Linux rather than isolating the authors' patches.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central quantitative claim is that \"our enhancements reduce the worst-case jitter by a factor of 8.7, resulting in a maximum 11.5x improvement over system schedulability.\" The experiments behind these numbers compare openEuler, a full commercial distribution that integrates both PREEMPT_RT and the authors' kernel modifications, against Vanilla Linux, an unmodified Linux 5.10 kernel. These two systems differ in far more than the 34 fixes: openEuler includes a real-time patch series, a newer kernel base, and numerous distribution-level changes. The comparison that actually isolates the authors' contributions is openEuler versus RT-Linux, since openEuler is described as RT-Linux plus the authors' modifications. That comparison yields a 4.0x jitter reduction and up to 3.5x schedulability improvement, not the 8.7x and 11.5x values used in the abstract and conclusions. The 8.7x and 11.5x figures therefore attribute to the authors' fixes effects that may be partly or fully caused by PREEMPT_RT, kernel-version improvements, or other openEuler changes. This is not a purely cosmetic issue: the schedulability analysis in §V-C uses the Table II maximum-latency values as release jitter, so any overstatement in the jitter reduction propagates directly into the 11.5x schedulability number. The reader's concern about sample maxima being used as deterministic jitter bounds is valid, but it is secondary: even if the measured maxima were true worst-case bounds, the experiment still would not support the headline attribution without an A/B test that controls for non-author differences between the compared systems.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports six years of industrial experience in identifying and mitigating cross-core performance interference in Linux. It groups the problems into three categories (task management, resource management, and concurrency management), presents representative bugs and their fixes (several claimed to be merged upstream), and distills lessons for future OS design. The evaluation measures worst-case latency with cyclictest and oslat, runs SchedCAT-based schedulability analyses, and benchmarks cFS and ROS2 end-to-end. The headline claims are an 8.7x reduction in worst-case jitter and up to an 11.5x improvement in schedulability, with smaller but still substantial improvements over RT-Linux.","tokens_in":21764,"tokens_out":5754,"duration_ms":52342,"significance":"If the quantitative claims survive scrutiny, this is a valuable industrial experience report with concrete, inspectable kernel patches and a plausible demonstration that cross-core OS interference can be systematically reduced. Strengths of the paper include that several fixes are backed by upstream commit references, the measurements are internally consistent, the evaluation compares against external baselines rather than fitting parameters to its own output, and the schedulability analysis uses standard tools (SchedCAT, Audsley RTA, ILP-based blocking-time analysis). The main weaknesses are that the headline numbers conflate the authors' patches with PREEMPT_RT and other distribution-level changes, and that the schedulability analysis treats finite-run sample maxima as deterministic worst-case jitter bounds. These issues are load-bearing for the paper's central quantitative contribution, so the significance is currently overstated.","major_comments":[{"comment":"The abstract and conclusions claim an 8.7x jitter reduction and up to an 11.5x schedulability improvement, but these numbers compare openEuler against Vanilla Linux 5.10. The paper states in §V that openEuler integrates both RT-Linux (PREEMPT_RT) and the authors' kernel modifications, so the comparison with Vanilla Linux does not isolate the authors' contribution; it conflates their fixes with the PREEMPT_RT patch set and any other openEuler changes. The comparison that does isolate the contribution is openEuler versus RT-Linux, which yields a 4.0x jitter reduction and up to a 3.5x schedulability improvement. The headline claims should be reframed around the RT-Linux comparison, or the evaluation should add a configuration that applies only the authors' patches on top of RT-Linux to separate the two effects.","section":"§V Experimental Setup; Abstract"},{"comment":"The schedulability analysis in §V-C uses the maximum cyclictest latency observed over finite runs (12 us for openEuler, 48 us for RT-Linux, 104 us for Vanilla Linux) as a deterministic release-jitter bound for every task. This is a sample maximum, not a verified worst-case bound; calling the model 'conservative' is therefore incorrect, since the true worst-case jitter could be larger than the measured maximum. The reported schedulability improvements, including the 11.5x figure, are directly sensitive to this input value. The paper needs either a sensitivity analysis across a range of jitter values or a statistically justified bound (e.g., many repeated trials with confidence intervals, or an extreme-value analysis) before the schedulability claims can be considered robust.","section":"§V-C, Table II"},{"comment":"The paper claims that 34 cross-core interference bugs were fixed and that 40 patches were incorporated into the Linux mainline, but only about 12 commits are cited in the reference list, and some of those (e.g., the perf top fixes in [37] and [38]) are not obviously cross-core interference fixes. Without a complete mapping from each of the 34 bugs to its upstream commit, workaround, or private patch, the central claim that 'our enhancements' produce the measured improvements cannot be independently verified. Please add a table listing all 34 bugs, their categories, the affected subsystems, and the corresponding upstream commit or workaround status.","section":"§III; References [35]-[47]"}],"minor_comments":[{"comment":"The phrase 'Compared to the version without our improvements' is ambiguous; the actual comparisons are against Vanilla Linux and RT-Linux, which should be stated explicitly.","section":"Abstract"},{"comment":"The sentence 'the worst-case latency of 50 us reflects the ground truth' is confusingly phrased; it should say that the measurement at the 50 us wake-up period is the value reported in Table II.","section":"§V-B, Figure 6"},{"comment":"The phrase 'subtitle cross-core interference bugs' should read 'subtle cross-core interference bugs'.","section":"§IV-E"},{"comment":"The word 'robostic' should be 'robotic'.","section":"§V-D"},{"comment":"The unit 'us' should be typeset as 'µs' or 'μs' consistently in text and figures.","section":"Throughout"},{"comment":"Several reference URLs contain line breaks or are malformed (e.g., [27], [28], [29]) and should be cleaned up and formatted as proper hyperlinks.","section":"References"},{"comment":"The phrase 'we despairingly found' is informal for a journal paper; 'we found' would be more appropriate.","section":"§IV-A"},{"comment":"The abbreviation 'RTLinux' in the figure legend should be 'RT-Linux' to match the text, and the scenario abbreviations (SN, CD, MB, WM) should be expanded in the caption or footnote for readability.","section":"Figure 10"}],"recommendation":"major_revision","confidential_remarks":"This is an experience-oriented industrial paper whose scientific contribution rests on the quantitative interference and schedulability improvements. The two load-bearing issues are the conflation of the authors' patches with PREEMPT_RT in the headline numbers and the use of sample maxima as worst-case jitter bounds. Both are fixable in revision (reframe claims, add a sensitivity analysis, and provide a complete patch list), so I do not recommend rejection, but the current version substantially overstates the measured contribution. The patch-completeness issue is also important for the journal's reproducibility standards."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a real industrial data point, not a toy. The authors fixed 34 cross-core interference bugs in Linux over six years; 40 patches are in mainline, and the ASID partitioning and workqueue activation restriction are concrete kernel techniques you can inspect upstream. The task/resource/concurrency categorization is useful, and the lessons section reads like actual experience rather than filler. If you work on OS interference or isolated cores, you should know this paper.\n\nWhat it does well: the fixes are concrete, the evaluation compares against vanilla Linux and RT-Linux under explicit interference workloads, and the internal comparison (openEuler vs RT-Linux) supports a 4.0x worst-case jitter reduction and up to 3.5x schedulability improvement. Feeding measured jitter into SchedCAT is a legitimate way to show schedulability benefit. The cFS and ROS2 end-to-end results are a plus, especially the ROS2 packet-loss case.\n\nSoft spots, in proportion. The abstract's headline 8.7x jitter and 11.5x schedulability numbers compare openEuler (RT-Linux plus the authors' patches plus a newer distro base) against vanilla Linux 5.10. That comparison does not isolate the 34 fixes, and the stress-test note is right: the attributable improvements are the 4.0x and 3.5x over RT-Linux. The authors should re-state the headline claims or provide an A/B test that controls for PREEMPT_RT and other distro differences. Second, the schedulability analysis treats the maximum cyclictest latency from finite runs (12 us, etc.) as a deterministic release-jitter bound. That is a sample maximum, not a verified worst-case bound; a longer run could find a higher value and shrink the schedulability ratios. This is real but secondary to the attribution problem. Third, there are no error bars or repeated-run statistics, and no artifact beyond the public patches, so exact replication is not possible. The ASID reserved subspace size is a parameter but not a fitted one; that is fine, though the paper could say how they chose it.\n\nWho this is for: real-time systems researchers, Linux kernel isolation developers, and anyone building predictable systems on Linux. The central direction holds up; the magnitudes need re-scoping. I would send this to review with a request to fix the baseline attribution before acceptance, not desk-reject it.","headline":"Genuine industry report with upstream-merged kernel fixes; the 8.7x/11.5x headline overstates attribution, but the RT-Linux comparisons and public patch set justify serious review.","tokens_in":22342,"tokens_out":2053,"would_cite":true,"duration_ms":19699,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"After six years and 34 kernel fixes, Linux's worst-case cross-core jitter drops 8.7x and schedulability improves up to 11.5x.","keywords":["cross-core interference","Linux kernel","real-time systems","core isolation","worst-case latency","schedulability","workqueue","ASID isolation"],"falsifier":"On the same ARM hardware described in the paper, run the paper's interference workload suite while recording cyclictest on 24 isolated cores for at least 24 hours; if the observed worst-case jitter substantially exceeds 12 us, or if the cFS and ROS2 worst-case latencies do not reproduce the reported reductions, the central claim would need revision. A direct experiment would also intentionally exhaust ASIDs on non-isolated cores and check whether an isolated core's context-switch time remains flat.","tokens_in":21235,"feed_emoji":"🐧","tokens_out":7492,"duration_ms":65798,"temperature":0.7,"pith_summary":"Cross-core interference—slowdowns on one core caused by kernel activity on another—is a serious threat to real-time systems, but the paper argues that the operating system itself is a major and under-studied source of that interference. Drawing on six years of industrial experience, the authors identify 34 interference bugs in Linux across task management, resource management, and concurrency management, and they have merged 40 patches into the Linux mainline. Their integrated version, openEuler, reduces worst-case task-activation jitter by a factor of 8.7 compared with vanilla Linux and achieves up to 11.5x better system schedulability. End-to-end latency in cFS and ROS2 drops by 1.6x to 2.1x compared with RT-Linux under the tested workloads. The paper's practical claim is that systematic, category-based auditing of kernel code can eliminate most OS-induced cross-core interference in production systems.","feed_headline":"Cross-core jitter cut 8.7x in Linux after 34 kernel fixes","feed_subtitle":"Six years of upstream Linux patches slash worst-case latency on isolated cores and lift schedulability up to 11.5x.","key_machinery":"The load-bearing mechanism is a three-category taxonomy of cross-core interference bugs—task management, resource management, and concurrency management—together with an isolation-aware partition model in which isolated cores run only real-time tasks and non-isolated cores cannot directly interact with them. The argument is carried by the specific fixes: restricting worker-thread activation to the same partition, checking isolation status in core-selection paths, partitioning the ASID space so non-isolated cores never exhaust or lock the ASIDs of isolated cores, flushing per-core queues only when pending work exists, and shrinking seqlock-held critical sections in jiffies updates. These fixes close concrete IPI, TLB-flush, and lock-contention paths that were previously invisible to Linux's fragmented isolation mechanisms.","core_discovery":"The central discovery is that the Linux kernel's own subsystems are a principal and removable source of cross-core interference on isolated cores. Through case studies the authors show concrete mechanisms: workqueue operations wake worker threads on isolated cores via IPIs; task migration and irqbalance ignore isolated-core status; ASID exhaustion triggered by non-isolated cores forces TLB flushes and lock contention on isolated cores; NIC uninstall flushes per-core backlog queues everywhere; jiffies synchronization holds seqlocks in long critical sections; and vmstat aggregation runs work on all cores. For each class the paper reports a fix—activation restriction, isolation-aware core selection, partitioned ASID space, on-demand flush, and compressed critical sections—resulting in measured worst-case cyclictest latency of 12 us under heavy interference compared with 104 us for vanilla Linux and 48 us for RT-Linux. The paper further argues that the same three-way taxonomy and lessons generalize to future OS design.","pith_inferences":["Beyond the paper's own results, the same task/resource/concurrency taxonomy could be applied to other monolithic kernels or to hypervisor-based systems, where hidden shared resources are likely to produce analogous interference.","The ASID-partitioning fix generalizes to other shared hardware identifiers, such as cache tags or per-device buffers, and could become a design guideline for hardware vendors.","The paper's jitter numbers come from finite measurement runs; turning the 12 us value into a certified bound would require either longer stress campaigns or a formal analysis of the patched kernel paths.","A testable extension is to replay the paper's interference suite on recent mainline kernels to detect regressions as new subsystems are added."],"forward_implications":["Under the paper's interference workload, worst-case cyclictest latency on isolated cores drops from 104 us to 12 us, and worst-case jitter falls by a factor of 8.7 relative to vanilla Linux.","EDF and fixed-priority schedulability improve by up to 11.5x over vanilla Linux under the same jitter model, and locking-protocol schedulability reaches up to 8x improvement over vanilla Linux in the read-write-lock scenario.","End-to-end worst-case latency in cFS and ROS2 drops by 1.6x to 2.1x relative to RT-Linux, including elimination of the observed 255 ms ROS2 packet-loss event.","Because 40 patches are already merged into Linux mainline, downstream distributions can inherit the interference fixes without proprietary kernel forks.","The three-category taxonomy gives kernel developers a concrete checklist for locating and fixing similar interference bugs in other subsystems."],"supporting_citations":[{"why":"Measurement tool that produces the task-activation jitter values used as worst-case latency and release-jitter inputs.","marker":"[29]"},{"why":"Schedulability analysis toolkit used to compute schedulability for EDF, fixed-priority, and locking-protocol scenarios.","marker":"[67]"},{"why":"Task-set synthesis method used to generate the 500 random task sets for schedulability experiments.","marker":"[68]"},{"why":"Classic response-time analysis used for fixed-priority schedulability evaluation.","marker":"[69]"},{"why":"Source of the spin-based reader-writer lock analyzed in the locking-protocol schedulability tests.","marker":"[70]"},{"why":"Core Flight System used as the satellite-software end-to-end evaluation workload.","marker":"[30]"},{"why":"Robot Operating System 2 used as the robotics end-to-end evaluation workload.","marker":"[31]"},{"why":"ROS2 performance evaluation framework used to measure maximum communication latency in the four scenarios.","marker":"[77]"}],"fun_headline_variants":["Linux kernel's own interference fixed: jitter down 8.7x","Cross-core jitter in Linux reduced 8.7x after 34 kernel fixes","From 104us to 12us: Linux kernel changes reduce worst-case jitter","Linux patches boost schedulability 11.5x while cutting jitter 8.7x"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The schedulability improvements assume that the maximum cyclictest latency measured over finite runs (12 us) is a reliable upper bound on task release jitter; if the true worst-case jitter is larger, the reported 11.5x schedulability gains shrink.","fun_headline_variants_meta":{"raw":{"variants":["Linux kernel's own interference fixed: jitter down 8.7x","Cross-core jitter in Linux reduced 8.7x after 34 kernel fixes","From 104us to 12us: Linux kernel changes reduce worst-case jitter","Linux patches boost schedulability 11.5x while cutting jitter 8.7x"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000573,"raw_usage":{"total_tokens":2724,"prompt_tokens":977,"completion_tokens":1747,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":593,"completion_tokens_details":{"reasoning_tokens":1656}},"tokens_in":593,"tokens_out":1747,"duration_ms":15376,"temperature":1.0,"reasoning_tokens":1656,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:01:45.495439+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On the same ARM hardware described in the paper, run the paper's interference workload suite while recording cyclictest on 24 isolated cores for at least 24 hours; if the observed worst-case jitter substantially exceeds 12 us, or if the cFS and ROS2 worst-case latencies do not reproduce the reported reductions, the central claim would need revision. A direct experiment would also intentionally exhaust ASIDs on non-isolated cores and check whether an isolated core's context-switch time remains flat.","supporting_citations":[{"cited_title":"Cyclictest, https://wiki.linuxfoundation.org/realt ime/documentation/howto/tools/cyclictest/start,","cited_arxiv_id":null,"evidence_quote":"Measurement tool that produces the task-activation jitter values used as worst-case latency and release-jitter inputs."},{"cited_title":"The schedulability test collection and toolkit","cited_arxiv_id":null,"evidence_quote":"Schedulability analysis toolkit used to compute schedulability for EDF, fixed-priority, and locking-protocol scenarios."},{"cited_title":"Techniques for the synthesis of multiprocessor tasksets,","cited_arxiv_id":null,"evidence_quote":"Task-set synthesis method used to generate the 500 random task sets for schedulability experiments."},{"cited_title":"Applying new scheduling theory to static priority pre-emptive scheduling,","cited_arxiv_id":null,"evidence_quote":"Classic response-time analysis used for fixed-priority schedulability evaluation."},{"cited_title":"Spin-based reader-writer syn- chronization for multiprocessor real-time systems,","cited_arxiv_id":null,"evidence_quote":"Source of the spin-based reader-writer lock analyzed in the locking-protocol schedulability tests."},{"cited_title":"core flight system (cfs),","cited_arxiv_id":null,"evidence_quote":"Core Flight System used as the satellite-software end-to-end evaluation workload."},{"cited_title":"Irobot-ros/ros2-performance: Framework to evaluate peformance of ros 2","cited_arxiv_id":null,"evidence_quote":"ROS2 performance evaluation framework used to measure maximum communication latency in the four scenarios."}],"review_version":1}