Pith. sign in

REVIEW 3 major objections 2 minor 17 references

Controller rankings in edge scheduling reverse with load intensity, as a reinforcement learning method loses nearly 29 percentage points to a rule-based heuristic under heavier workloads.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-27 08:12 UTC pith:H5IAYDGB

load-bearing objection The paper delivers a practical open benchmark for CMMS schedulers with clear evidence that rankings reverse under heavier load, but the tested configs are too narrow to support broad claims about configuration dependence. the 3 major comments →

arxiv 2606.12343 v1 pith:H5IAYDGB submitted 2026-06-10 cs.DC

Fair Comparison of Scheduling Algorithms on Heterogeneous Edge Clusters: A Continuous Adaptive Benchmark

Zihang Wang , Boris Sedlak , Juan Luis Herrera , Schahram Dustdar This is my paper
classification cs.DC
keywords scheduling algorithmsedge computingbenchmark platformheterogeneous clustersreinforcement learningservice level objectivesworkload drivercontinuous multi-mode scheduling
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces a benchmark platform to enable fair comparisons of scheduling algorithms for Continuous Multi-Mode Scheduling on heterogeneous edge clusters. It evaluates six controllers across five cluster setups and two load regimes, finding that performance rankings depend strongly on the configuration. A deep reinforcement learning controller that performs well under light loads is outperformed by a simpler rule-based approach by almost 29 points when loads intensify, while also incurring about 500 times the operational overhead per decision. The platform separates raw SLO compliance from steady-state compliance to reveal switching costs that aggregate scores would hide. This matters because prior evaluations used inconsistent setups and single workloads, making it hard to know which algorithm to choose in practice.

Core claim

We present an open source benchmark with a unified controller interface, a closed-loop workload driver for multiple patterns, and dual-metric SLO scoring that distinguishes overall compliance from compliance during stable operation. Across 424 episodes with six controllers, five cluster configurations, and two load regimes, controller rankings prove strongly configuration-dependent: the deep reinforcement-learning winner under light workloads loses to a rule-based heuristic by nearly 29 percentage points once load intensifies, at roughly 500 times the per-decision operational overhead.

What carries the argument

The open source benchmark platform with unified interface, closed-loop workload driver, and dual-metric SLO scoring for Continuous Multi-Mode Scheduling problems.

Load-bearing premise

The five cluster configurations and two load regimes used in the benchmark sufficiently represent real heterogeneous edge deployments for the observed ranking reversals to apply more broadly.

What would settle it

Running the same six controllers on a substantially different set of heterogeneous edge cluster configurations or with continuously varying workloads and finding no significant ranking reversals would falsify the claim of strong configuration dependence.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Controller selection must account for expected workload intensity rather than assuming one algorithm dominates.
  • Per-decision operational overhead becomes a critical factor when comparing complex controllers like deep reinforcement learning to simpler heuristics.
  • Using separate raw and steady-state SLO metrics prevents conflating switching costs with overall performance.
  • Unified evaluation stacks and multiple workload patterns are necessary for reliable algorithm comparisons.
  • Rankings observed in light load regimes do not predict behavior under intensified loads.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Deployments in real edge environments may benefit from load-aware or hybrid scheduling that switches between controller types.
  • The benchmark platform could serve as a standard testbed for developing new Continuous Multi-Mode Scheduling algorithms.
  • High overhead of advanced controllers may limit their practicality in resource-constrained edge settings despite potential benefits in specific scenarios.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 2 minor

Summary. The paper introduces an open-source benchmark platform for Continuous Multi-Mode Scheduling (CMMS) on heterogeneous edge clusters. It provides a unified controller interface, closed-loop workload driver across multiple patterns, and dual-metric SLO scoring (raw compliance vs. steady-state compliance). Experiments evaluate six controllers over five cluster configurations and two load regimes in 424 episodes, showing that rankings are strongly configuration-dependent: a DRL controller wins under light load but loses to a rule-based heuristic by nearly 29 percentage points under intensified load, at ~500× per-decision overhead. The dual metrics also expose switching costs hidden by aggregate scores.

Significance. If the tested configurations prove representative, the results demonstrate that single-workload or single-configuration evaluations can produce misleading rankings in edge scheduling, a finding with direct implications for how future CMMS research should be conducted. The open-source platform, closed-loop driver, and separation of raw vs. steady-state SLOs are concrete strengths that could improve reproducibility and practical relevance in the distributed systems community. The scale (424 episodes) and explicit overhead reporting further add value.

major comments (3)
  1. [Abstract and §4] Abstract and §4 (Experiments): the central claim that rankings reverse by 'nearly 29 percentage points' under intensified load is presented without error bars, confidence intervals, or statistical significance tests across the 424 episodes; this is load-bearing for the configuration-dependence conclusion because scheduling outcomes are stochastic.
  2. [§3.1] §3.1 (Cluster and Workload Models): the five synthetic cluster topologies and two load regimes are used to support the generalization that rankings are 'strongly configuration-dependent,' yet no sensitivity analysis is reported for additional axes such as interconnect bandwidth distributions, node failure rates, or arrival processes drawn from production traces; this directly affects whether the observed reversals transfer beyond the closed test set.
  3. [§4.4] §4.4 (Overhead Measurement): the 500× per-decision operational overhead for the DRL controller is stated as a key practical drawback, but the measurement protocol (hardware platform, timing granularity, inclusion/exclusion of training vs. inference) is not specified in sufficient detail to allow independent reproduction or comparison.
minor comments (2)
  1. The abstract states the platform is 'open source' but does not include a repository URL or DOI; this should be added in the introduction or a footnote to enable immediate access.
  2. [§2] Notation for the two SLO variants (raw vs. steady-state) is introduced in the abstract but could be formalized earlier with explicit equations in §2 to improve clarity for readers unfamiliar with the distinction.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for their thorough review and constructive comments. We address each major comment below, proposing revisions to improve the manuscript's statistical rigor, discussion of limitations, and reproducibility.

read point-by-point responses

  1. Referee: [Abstract and §4] Abstract and §4 (Experiments): the central claim that rankings reverse by 'nearly 29 percentage points' under intensified load is presented without error bars, confidence intervals, or statistical significance tests across the 424 episodes; this is load-bearing for the configuration-dependence conclusion because scheduling outcomes are stochastic.

    Authors: We agree that including statistical measures is essential for substantiating the configuration-dependence claim given the stochastic nature of scheduling. In the revised version, we will add error bars (standard deviation or confidence intervals) to the key figures and tables in §4, and perform appropriate statistical tests (e.g., paired t-tests) across the 424 episodes to assess the significance of the observed ranking reversals. This will also be reflected in the abstract. revision: yes

  2. Referee: [§3.1] §3.1 (Cluster and Workload Models): the five synthetic cluster topologies and two load regimes are used to support the generalization that rankings are 'strongly configuration-dependent,' yet no sensitivity analysis is reported for additional axes such as interconnect bandwidth distributions, node failure rates, or arrival processes drawn from production traces; this directly affects whether the observed reversals transfer beyond the closed test set.

    Authors: The selected configurations were designed to cover a range of heterogeneity levels and load intensities to illustrate the dependence. However, we acknowledge the value of sensitivity analysis on additional parameters. In the revision, we will expand §3.1 and the discussion section to include a more detailed justification of the chosen parameters and explicitly note the lack of sensitivity analysis on interconnect bandwidth, node failures, and production traces as a limitation, suggesting it as future work. A full sensitivity study would require new experiments and is beyond the current scope. revision: partial

  3. Referee: [§4.4] §4.4 (Overhead Measurement): the 500× per-decision operational overhead for the DRL controller is stated as a key practical drawback, but the measurement protocol (hardware platform, timing granularity, inclusion/exclusion of training vs. inference) is not specified in sufficient detail to allow independent reproduction or comparison.

    Authors: We agree that the overhead measurement protocol requires more detail for reproducibility. In the revised §4.4, we will specify: the hardware platform used (CPU/GPU specifications), the timing method (e.g., Python time.perf_counter for wall-clock time per decision), the granularity (per-decision inference time), and confirm that only runtime inference overhead is measured, excluding any training or initialization phases. revision: yes

Circularity Check

0 steps flagged

No circularity: purely empirical benchmark with direct observations

full rationale

The paper conducts an empirical evaluation of six controllers across five cluster configurations and two load regimes (424 episodes total), reporting observed ranking reversals and overhead differences directly from the runs. No derivation, prediction, or first-principles result is claimed that reduces to a fitted parameter, self-citation, or ansatz defined inside the work; the central claim is a set of measured outcomes under the stated experimental conditions, making the analysis self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

No free parameters, axioms beyond standard experimental design, or invented entities are required; the contribution is the benchmark harness and the empirical comparison it enables.

pith-pipeline@v0.9.1-grok · 5780 in / 1151 out tokens · 17003 ms · 2026-06-27T08:12:31.344149+00:00 · methodology

0 comments
read the original abstract

Modern Artificial Intelligence (AI) workloads deployed across the heterogeneous tiers of an edge--cloud continuum must satisfy multi-dimensional Service Level Objectives (SLOs) over latency, throughput, and output quality. For each incoming task, the scheduler picks both a target node and a processing mode (e.g., full or reduced inference precision). We call this class of problems \emph{Continuous Multi-Mode Scheduling} (CMMS). Comparing CMMS algorithms fairly is difficult because prior studies typically evaluate each controller in its own stack, under a single workload, and without reporting per-decision overhead. To close these gaps, we present an open source benchmark platform that features (i) a unified controller interface, (ii) a closed-loop workload driver covering multiple workload patterns, and (iii) dual-metric SLO scoring that reports raw SLO (overall compliance) and steady-state SLO (compliance during stable operation) separately. Running six controllers across five cluster configurations and two load regimes (424 episodes), we find that controller rankings are strongly configuration-dependent: a deep reinforcement-learning winner under light workloads loses to a rule-based heuristic by nearly 29 percentage points once load intensifies, at roughly 500$\times$ the per-decision operational overhead. We further show that separating raw from steady-state SLOs exposes switching costs that a single aggregate score would otherwise conflate.

Figures

Figures reproduced from arXiv: 2606.12343 by Boris Sedlak, Juan Luis Herrera, Schahram Dustdar, Zihang Wang.

Figure 1
Figure 1. Figure 1: Three-layer platform architecture. The fixed middle layer comprises [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Per-step SLO compliance on the 8-node cluster under two workload [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Raw SLO (solid) and Steady-State SLO (hatched) on 8-node LIGHT [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

17 extracted references

  1. [1]

    Equilibrium in the computing continuum through active inference,

    B. Sedlak, V . C. Pujol, P. K. Donta, and S. Dustdar, “Equilibrium in the computing continuum through active inference,”Future Generation Computer Systems, vol. 160, pp. 92–108, 2024

    2024

  2. [2]

    Benchmarking dynamic slo compliance in distributed computing continuum systems,

    A. Lapkovskis, B. Sedlak, S. Magnússon, S. Dustdar, and P. K. Donta, “Benchmarking dynamic slo compliance in distributed computing continuum systems,” in2025 IEEE International Conference on Edge Computing and Communications (EDGE), 2025, pp. 93–102

    2025

  3. [3]

    Live video analytics at scale with approximation and delay-tolerance,

    H. Zhang, G. Ananthanarayanan, P. Bodik, M. Philipose, P. Bahl, and M. J. Freedman, “Live video analytics at scale with approximation and delay-tolerance,” inProceedings of the 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI), 2017, pp. 377–392

    2017

  4. [4]

    Serving DNNs like clockwork: Performance predictability from the bottom up,

    A. Gujarati, R. Karimi, S. Alzayat, W. Hao, A. Kaufmann, Y . Vigfusson, and J. Mace, “Serving DNNs like clockwork: Performance predictability from the bottom up,” inProceedings of the 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2020, pp. 443–462

    2020

  5. [5]

    Learning scheduling algorithms for data processing clusters,

    H. Mao, M. Schwarzkopf, S. B. Venkatakrishnan, Z. Meng, and M. Alizadeh, “Learning scheduling algorithms for data processing clusters,” inProceedings of the ACM SIGCOMM Conference, 2019, pp. 270–288

    2019

  6. [6]

    Active inference-based adaptive routing for heterogeneous edge AI services,

    Z. Wang, B. Sedlak, and S. Dustdar, “Active inference-based adaptive routing for heterogeneous edge AI services,” inService-Oriented Com- puting – ICSOC 2025 Workshops (SOC4AI). Springer, 2025, to appear

    2025

  7. [7]

    Adaptive stream processing on edge devices through active inference,

    B. Sedlak, V . Casamayor Pujol, A. Morichetta, P. K. Donta, and S. Dustdar, “Adaptive stream processing on edge devices through active inference,”Evolving Systems, 2025

    2025

  8. [8]

    Chameleon: Scalable adaptation of video analytics,

    J. Jiang, G. Ananthanarayanan, P. Bodik, S. Sen, and I. Stoica, “Chameleon: Scalable adaptation of video analytics,” inProceedings of the 2018 ACM SIGCOMM Conference, 2018, pp. 253–266

    2018

  9. [9]

    Neural adaptive video streaming with Pensieve,

    H. Mao, R. Netravali, and M. Alizadeh, “Neural adaptive video streaming with Pensieve,” inProceedings of the ACM SIGCOMM Conference, 2017, pp. 197–210

    2017

  10. [10]

    Joint configuration adaptation and bandwidth allocation for edge-based real- time video analytics,

    C. Wang, S. Zhang, Y . Chen, Z. Qian, J. Wu, and M. Xiao, “Joint configuration adaptation and bandwidth allocation for edge-based real- time video analytics,” inProceedings of IEEE INFOCOM, 2020, pp. 257–266

    2020

  11. [11]

    MLPerf inference benchmark,

    V . J. Reddi, C. Cheng, D. Kanter, P. Mattson, G. Schmuelling, C.-J. Wu, B. Anderson, M. Breughe, M. Charlebois, W. Chouet al., “MLPerf inference benchmark,” inProceedings of the ACM/IEEE International Symposium on Computer Architecture (ISCA), 2020, pp. 446–459

    2020

  12. [12]

    MLPerf training benchmark,

    P. Mattson, C. Cheng, C. Coleman, G. Diamos, P. Micikevicius, D. Patterson, H. Tang, G.-Y . Wei, P. Bailis, V . Bittorfet al., “MLPerf training benchmark,” inProceedings of the 3rd Conference on Machine Learning and Systems (MLSys), 2020

    2020

  13. [13]

    An open-source benchmark suite for microservices and their hardware-software implications for cloud and edge systems,

    Y . Gan, Y . Zhang, D. Cheng, A. Shetty, P. Rathi, N. Katarki, A. Bruno, J. Hu, B. Ritchken, B. Jacksonet al., “An open-source benchmark suite for microservices and their hardware-software implications for cloud and edge systems,” inProceedings of the 24th ACM International Conference on Architectural Support for Programming Languages and Operating System...

    2019

  14. [14]

    BUNGEE: An elasticity benchmark for self-adaptive IaaS cloud environments,

    N. R. Herbst, S. Kounev, A. Weber, and H. Groenda, “BUNGEE: An elasticity benchmark for self-adaptive IaaS cloud environments,” inProceedings of the 10th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS). IEEE Press, 2015, pp. 46–56

    2015

  15. [15]

    Ultralytics yolov8,

    G. Jocher, A. Chaurasia, and J. Qiu, “Ultralytics yolov8,” https://github. com/ultralytics/ultralytics, 2023

    2023

  16. [16]

    Deep reinforcement learning with double Q-learning,

    H. van Hasselt, A. Guez, and D. Silver, “Deep reinforcement learning with double Q-learning,” inProceedings of the AAAI Conference on Artificial Intelligence, 2016, pp. 2094–2100

    2016

  17. [17]

    T. Parr, G. Pezzulo, and K. J. Friston,Active Inference: The Free Energy Principle in Mind, Brain, and Behavior. MIT Press, 2022. 6

    2022