Pith. sign in

REVIEW 1 major objections 35 references

Terastal uses offline layer variants and online scheduling to cut deadline misses by 30-40% in multi-DNN workloads on heterogeneous accelerators with 2.24% accuracy loss.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-27 21:18 UTC pith:FG5DIZRB

load-bearing objection Terastal's layer-variant idea helps close latency gaps for multi-DNN scheduling on heterogeneous accelerators, but the abstract supplies no experimental details so the claimed gains are hard to evaluate. the 1 major comments →

arxiv 2606.06818 v1 pith:FG5DIZRB submitted 2026-06-05 cs.DC cs.ARcs.LG

Terastal: Layer-Variant-based Scheduling for Real-Time Multi-DNN Workloads on Heterogeneous Accelerators

Sing-Yao Wu , Fengshuo Song , Eli Bozorgzadeh This is my paper
classification cs.DC cs.ARcs.LG
keywords heterogeneous acceleratorsmulti-DNN schedulingreal-time systemslayer variantsdeadline miss ratesoft real-timeDNN inferenceaccelerator mapping
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents Terastal as a framework that first designs customized versions of DNN layers offline to shrink their execution time differences across accelerator types, then uses those variants in an online scheduler that chooses both the accelerator and the variant for each layer. This targets the problem that skewed multi-DNN workloads create large latency gaps, limiting how flexibly tasks can be mapped and causing more deadline misses under soft real-time constraints. A reader would care because many AI systems now run several models simultaneously on mixed hardware such as CPUs plus GPUs, and missing deadlines breaks responsiveness even when average latency looks acceptable. The approach keeps accuracy loss small by constraining the variant search to stay within a tolerance while still improving scheduling headroom.

Core claim

Terastal combines offline heterogeneity-aware virtual budget assignment and layer-variant design with online scheduling to jointly optimize accelerator mapping and variant selection under timing and accuracy constraints, reducing the deadline miss rate per model by 40.58 percent, 30.53 percent, and 36.27 percent compared with FCFS, EDF, and DREAM respectively while incurring only 2.24 percent average normalized accuracy loss across models with variants.

What carries the argument

Layer variants: customized implementations of individual DNN layers that reduce latency gaps on non-preferred accelerators while bounding accuracy loss.

Load-bearing premise

Customized layer variants can be designed offline to reduce latency gaps on non-preferred accelerators while keeping accuracy loss small enough that the online scheduler can still satisfy timing constraints under skewed workloads.

What would settle it

A set of models and workloads for which no offline-designed layer variants exist that shrink cross-accelerator latency differences enough to let the scheduler meet its timing targets without accuracy loss above the 2.24 percent average reported.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Scheduling flexibility increases because latency differences no longer force rigid accelerator assignments.
  • Deadline miss rates drop across FCFS, EDF, and DREAM baselines while accuracy remains close to the original models.
  • Heterogeneous accelerators can be utilized more evenly without violating soft real-time guarantees.
  • The joint offline design and online selection process balances latency, accuracy, and timing in one framework.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If layer variants prove reusable across similar model families, the offline design cost could be amortized over many deployments.
  • The same variant idea might extend to energy or power constraints if the design step also targets those metrics.
  • Automatic generation of variants through search or fine-tuning could replace manual customization in future versions.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 0 minor

Summary. The paper introduces Terastal, a soft real-time framework for multi-DNN workloads on heterogeneous accelerators. It proposes layer variants—customized layer implementations that reduce latency gaps on non-preferred accelerators—combined with offline heterogeneity-aware virtual budget assignment and layer-variant design, plus online scheduling to jointly optimize accelerator mapping and variant selection under timing and accuracy constraints. The central empirical claim is that Terastal reduces per-model deadline miss rates by 40.58%, 30.53%, and 36.27% versus FCFS, EDF, and DREAM while incurring only 2.24% average normalized accuracy loss.

Significance. If the reported gains hold under realistic workloads and the layer-variant construction proves generalizable, the work could meaningfully improve schedulability in heterogeneous accelerator systems for soft real-time DNN inference. The offline/online split is a pragmatic way to handle the accuracy–latency trade-off, and the focus on deadline-miss reduction rather than average latency is appropriate for the target domain.

major comments (1)
  1. [Abstract] Abstract: the quantitative claims (40.58%, 30.53%, 36.27% miss-rate reductions and 2.24% accuracy loss) are presented without any description of experimental setup, DNN models used, accelerator configurations, workload traces or arrival patterns, number of runs, or statistical measures. Because these numbers constitute the primary evidence for the central claim, their unverifiability from the provided text is load-bearing.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the constructive feedback and positive evaluation of the work's potential impact. The sole major comment concerns the abstract's lack of experimental context for the reported quantitative results. We address this directly below.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the quantitative claims (40.58%, 30.53%, 36.27% miss-rate reductions and 2.24% accuracy loss) are presented without any description of experimental setup, DNN models used, accelerator configurations, workload traces or arrival patterns, number of runs, or statistical measures. Because these numbers constitute the primary evidence for the central claim, their unverifiability from the provided text is load-bearing.

    Authors: We agree that the abstract, in its current form, does not provide sufficient context for the primary empirical claims, which limits immediate verifiability. While the full experimental details (models, accelerators, workloads, run counts, and statistical reporting) appear in Sections 5 and 6 of the manuscript, we acknowledge that the abstract should be more self-contained. In the revised version we will expand the abstract with a concise clause summarizing the key experimental parameters (DNN models evaluated, accelerator platform, workload characteristics, and averaging over multiple runs) while preserving the existing quantitative results and overall length. revision: yes

Circularity Check

0 steps flagged

No significant circularity

full rationale

The paper describes an empirical scheduling framework whose central claims are measured deadline-miss reductions and accuracy loss obtained from experiments on heterogeneous accelerators. No equations, fitted parameters, self-citations, or derivation steps appear in the supplied text that would reduce any reported quantity to an input by construction; the results are presented as direct experimental outcomes rather than predictions derived from the same data or prior self-authored theorems.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract alone supplies no information on free parameters, background axioms, or new postulated entities; full text would be needed to audit these.

pith-pipeline@v0.9.1-grok · 5698 in / 1105 out tokens · 18383 ms · 2026-06-27T21:18:18.211500+00:00 · methodology

0 comments
read the original abstract

Heterogeneous DNN accelerators improve soft real-time multi-DNN execution by mapping each layer to its preferred accelerator to reduce latency. However, under skewed workloads, large layer-latency differences across accelerators limit scheduling flexibility and increase deadline misses. To address this challenge, we introduce layer variants, customized layer implementations that reduce latency gaps on non-preferred accelerators. We then present Terastal, a soft real-time framework for layer-variant design and scheduling on heterogeneous DNN accelerators. Terastal combines offline heterogeneity-aware virtual budget assignment and layer-variant design, and online scheduling to jointly optimize accelerator mapping and variant selection under timing and accuracy constraints. Experimental results show that Terastal reduces deadline miss rate per model by 40.58%, 30.53%, and 36.27% compared with FCFS, EDF, and DREAM, respectively, while incurring only 2.24% average normalized accuracy loss across models with variants.

Figures

Figures reproduced from arXiv: 2606.06818 by Eli Bozorgzadeh, Fengshuo Song, Sing-Yao Wu.

Figure 1
Figure 1. Figure 1: Example of converting a WS-preferred convolution into a layer variant [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Terastal Framework Overview memory, allowing consecutive layers of the same inference to execute on different accelerators. We assume layer-granularity scheduling, where each layer is modeled as a non-preemptive job. This is consistent with prior DNN accelerator scheduling work [1], [4] and avoids additional hardware and software support required for intra-layer preemption [13]. Scheduling decisions are th… view at source ↗
Figure 3
Figure 3. Figure 3: Per-layer latency of VGG11 on WS and OS accelerators (top) and [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Average per-model deadline miss rate comparison in different [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

35 extracted references · 2 canonical work pages · 2 internal anchors

  1. [1]

    Dream: A dynamic scheduler for dynamic real-time multi-model ml workloads,

    S. Kim, H. Kwon, J. Song, J. Jo, Y .-H. Chen, L. Lai, and V . Chandra, “Dream: A dynamic scheduler for dynamic real-time multi-model ml workloads,” inProceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, V olume 4, 2023, pp. 73–86

    2023

  2. [2]

    NVDLA deep learning accelerator,

    NVIDIA, “NVDLA deep learning accelerator,” http://nvdla.org, 2017, accessed: 2025-09-29

    2017

  3. [3]

    Shidiannao: Shifting vision processing closer to the sensor,

    Z. Du, R. Fasthuber, T. Chen, P. Ienne, L. Li, T. Luo, X. Feng, Y . Chen, and O. Temam, “Shidiannao: Shifting vision processing closer to the sensor,” inProceedings of the 42nd annual international symposium on computer architecture, 2015, pp. 92–104

    2015

  4. [4]

    Heterogeneous dataflow accelerators for multi-dnn workloads,

    H. Kwon, L. Lai, M. Pellauer, T. Krishna, Y .-H. Chen, and V . Chan- dra, “Heterogeneous dataflow accelerators for multi-dnn workloads,” in 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA). IEEE, 2021, pp. 71–83

    2021

  5. [5]

    Serving multi-dnn workloads on fpgas: A coordinated architecture, scheduling, and mapping perspective,

    S. Zeng, G. Dai, N. Zhang, X. Yang, H. Zhang, Z. Zhu, H. Yang, and Y . Wang, “Serving multi-dnn workloads on fpgas: A coordinated architecture, scheduling, and mapping perspective,”IEEE Transactions on Computers, vol. 72, no. 5, pp. 1314–1328, 2022

    2022

  6. [6]

    Scar: Schedul- ing multi-model ai workloads on heterogeneous multi-chiplet module accelerators,

    M. Odema, L. Chen, H. Kwon, and M. A. Al Faruque, “Scar: Schedul- ing multi-model ai workloads on heterogeneous multi-chiplet module accelerators,” in2024 57th IEEE/ACM International Symposium on Microarchitecture (MICRO). IEEE, 2024, pp. 565–579

    2024

  7. [7]

    Meeting end-to-end deadlines through distributed local deadline assignments,

    S. Hong, T. Chantem, and X. S. Hu, “Meeting end-to-end deadlines through distributed local deadline assignments,” in2011 IEEE 32nd Real-Time Systems Symposium, 2011, pp. 183–192

    2011

  8. [8]

    Sgprs: Seamless gpu partitioning real- time scheduler for periodic deep learning workloads,

    A. F. Babaei and T. Chantem, “Sgprs: Seamless gpu partitioning real- time scheduler for periodic deep learning workloads,” in2024 Design, Automation & Test in Europe Conference & Exhibition (DATE). IEEE, 2024, pp. 1–2

    2024

  9. [9]

    Daris: An oversubscribed spatio-temporal scheduler for real-time dnn inference on gpus,

    ——, “Daris: An oversubscribed spatio-temporal scheduler for real-time dnn inference on gpus,” in2025 62nd ACM/IEEE Design Automation Conference (DAC). IEEE, 2025, pp. 1–7

    2025

  10. [10]

    Magma: An optimization framework for mapping multiple dnns on multiple accelerator cores,

    S.-C. Kao and T. Krishna, “Magma: An optimization framework for mapping multiple dnns on multiple accelerator cores,” in2022 IEEE International Symposium on High-Performance Computer Architecture (HPCA). IEEE, 2022, pp. 814–830

    2022

  11. [11]

    Performance implications of multi-chiplet neural processing units on autonomous driving perception,

    M. Odema, L. Chen, H. Kwon, and M. A. Al Faruque, “Performance implications of multi-chiplet neural processing units on autonomous driving perception,” in2025 Design, Automation & Test in Europe Conference (DATE). IEEE, 2025, pp. 1–7

    2025

  12. [12]

    Taichi: Efficient execution for multi-dnns using graph-based scheduling,

    X. Zhou, H. Lu, T. Wang, Z. Wan, J. Chen, J. Yu, and K. Wang, “Taichi: Efficient execution for multi-dnns using graph-based scheduling,” in 2025 Design, Automation & Test in Europe Conference (DATE). IEEE, 2025, pp. 1–7

    2025

  13. [13]

    Prema: A predictive multi-task scheduling algorithm for preemptible neural processing units,

    Y . Choi and M. Rhu, “Prema: A predictive multi-task scheduling algorithm for preemptible neural processing units,” in2020 IEEE International Symposium on High Performance Computer Architecture (HPCA). IEEE, 2020, pp. 220–233

    2020

  14. [14]

    Planaria: Dynamic architecture fission for spatial multi-tenant acceleration of deep neural networks,

    S. Ghodrati, B. H. Ahn, J. K. Kim, S. Kinzer, B. R. Yatham, N. Alla, H. Sharma, M. Alian, E. Ebrahimi, N. S. Kimet al., “Planaria: Dynamic architecture fission for spatial multi-tenant acceleration of deep neural networks,” in2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO). IEEE, 2020, pp. 681–697

    2020

  15. [15]

    Moca: Memory-centric, adaptive execution for multi-tenant deep neural networks,

    S. Kim, H. Genc, V . V . Nikiforov, K. Asanovi ´c, B. Nikoli ´c, and Y . S. Shao, “Moca: Memory-centric, adaptive execution for multi-tenant deep neural networks,” in2023 IEEE International Symposium on High- Performance Computer Architecture (HPCA). IEEE, 2023, pp. 828– 841

    2023

  16. [16]

    A deep reinforcement learning based online scheduling policy for deep neural network multi-tenant multi-accelerator systems,

    F. G. Blanco, E. Russo, M. Palesi, D. Patti, G. Ascia, and V . Catania, “A deep reinforcement learning based online scheduling policy for deep neural network multi-tenant multi-accelerator systems,” inProceedings of the 61st ACM/IEEE Design Automation Conference, 2024, pp. 1–6

    2024

  17. [17]

    Blastnet: Exploiting duo-blocks for cross-processor real-time dnn inference,

    N. Ling, X. Huang, Z. Zhao, N. Guan, Z. Yan, and G. Xing, “Blastnet: Exploiting duo-blocks for cross-processor real-time dnn inference,” in Proceedings of the 20th ACM Conference on Embedded Networked Sensor Systems, 2022, pp. 91–105

    2022

  18. [18]

    Early-exit deep neural network-a comprehensive survey,

    H. Rahmath P, V . Srivastava, K. Chaurasia, R. G. Pacheco, and R. S. Couto, “Early-exit deep neural network-a comprehensive survey,”ACM Computing Surveys, vol. 57, no. 3, pp. 1–37, 2024

    2024

  19. [19]

    Scheduling real-time deep learning services as imprecise computations,

    S. Yao, Y . Hao, Y . Zhao, H. Shao, D. Liu, S. Liu, T. Wang, J. Li, and T. Abdelzaher, “Scheduling real-time deep learning services as imprecise computations,” in2020 IEEE 26th international conference on embedded and real-time computing systems and applications (RTCSA). IEEE, 2020, pp. 1–10

    2020

  20. [20]

    Anytime-lidar: Deadline-aware 3d object detection,

    A. Soyyigit, S. Yao, and H. Yun, “Anytime-lidar: Deadline-aware 3d object detection,” in2022 IEEE 28th International Conference on Em- bedded and Real-Time Computing Systems and Applications (RTCSA). IEEE, 2022, pp. 31–40

    2022

  21. [21]

    DeeperLab: Single-Shot Image Parser

    T.-J. Yang, M. D. Collins, Y . Zhu, J.-J. Hwang, T. Liu, X. Zhang, V . Sze, G. Papandreou, and L.-C. Chen, “Deeperlab: Single-shot image parser,” arXiv preprint arXiv:1902.05093, 2019

    work page internal anchor Pith review Pith/arXiv arXiv 1902

  22. [22]

    Maestro: A data-centric approach to understand reuse, performance, and hardware cost of dnn mappings,

    H. Kwon, P. Chatarasi, V . Sarkar, T. Krishna, M. Pellauer, and A. Parashar, “Maestro: A data-centric approach to understand reuse, performance, and hardware cost of dnn mappings,”IEEE micro, vol. 40, no. 3, pp. 20–29, 2020

    2020

  23. [23]

    Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search,

    B. Wu, X. Dai, P. Zhang, Y . Wang, F. Sun, Y . Wu, Y . Tian, P. Vajda, Y . Jia, and K. Keutzer, “Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 10 734–10 742

    2019

  24. [24]

    3d hand shape and pose estimation from a single rgb image,

    L. Ge, Z. Ren, Y . Li, Z. Xue, Y . Wang, J. Cai, and J. Yuan, “3d hand shape and pose estimation from a single rgb image,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 10 833–10 842

    2019

  25. [25]

    Sparse-to-dense: Depth prediction from sparse depth samples and a single image,

    F. Ma and S. Karaman, “Sparse-to-dense: Depth prediction from sparse depth samples and a single image,” in2018 IEEE international confer- ence on robotics and automation (ICRA). IEEE, 2018, pp. 4796–4803

    2018

  26. [26]

    Mobilenetv2: Inverted residuals and linear bottlenecks,

    M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, “Mobilenetv2: Inverted residuals and linear bottlenecks,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 4510–4520

    2018

  27. [27]

    Planercnn: 3d plane detection and reconstruction from a single image,

    C. Liu, K. Kim, J. Gu, Y . Furukawa, and J. Kautz, “Planercnn: 3d plane detection and reconstruction from a single image,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 4450–4459

    2019

  28. [28]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778

    2016

  29. [29]

    Very Deep Convolutional Networks for Large-Scale Image Recognition

    K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,”arXiv preprint arXiv:1409.1556, 2014

    work page internal anchor Pith review Pith/arXiv arXiv 2014

  30. [30]

    Rethinking the inception architecture for computer vision,

    C. Szegedy, V . Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 2818–2826

    2016

  31. [31]

    Swin transformer: Hierarchical vision transformer using shifted windows,

    Z. Liu, Y . Lin, Y . Cao, H. Hu, Y . Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” inProceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 10 012–10 022

    2021

  32. [32]

    Xrbench: An extended reality (xr) machine learning benchmark suite for the metaverse,

    H. Kwon, K. Nair, J. Seo, J. Yik, D. Mohapatra, D. Zhan, J. Song, P. Capak, P. Zhang, P. Vajdaet al., “Xrbench: An extended reality (xr) machine learning benchmark suite for the metaverse,”Proceedings of Machine Learning and Systems, vol. 5, pp. 1–20, 2023

    2023

  33. [33]

    Imagenet large scale visual recognition challenge,

    O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernsteinet al., “Imagenet large scale visual recognition challenge,”International journal of computer vision, vol. 115, no. 3, pp. 211–252, 2015

    2015

  34. [34]

    The pascal visual object classes challenge: A retrospective,

    M. Everingham, S. A. Eslami, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes challenge: A retrospective,”International journal of computer vision, vol. 111, no. 1, pp. 98–136, 2015

    2015

  35. [35]

    Vision meets robotics: The kitti dataset,

    A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, “Vision meets robotics: The kitti dataset,”The international journal of robotics research, vol. 32, no. 11, pp. 1231–1237, 2013

    2013