REVIEW 4 major objections 5 minor 29 references
RankMap: Priority-Aware Multi-DNN Manager for Heterogeneous Embedded Devices
T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read RankMap, a priority-aware multi-DNN manager for heterogeneous embedded devices, claims up to 3.6x higher average throughput and 57.5x better prioritization than prior methods while preventing DNN starvation through Monte Carlo Tree Search…
desk verdict Priority-aware MCTS scheduler worth engaging, but the lack of estimator validation on the search trajectory and unreported threshold th are real gaps that need addressing. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the reward-driven search loop: each candidate mapping $M$ is scored by $O(M)^T \times p$, where $O(M)$ is the estimator's per-DNN throughput vector and $p$ is the priority vector, and any mapping with a predicted throughput below a threshold $th$ is disqualified. The estimator itself is a multi-task attention-based CNN of about $3.7$ million parameters: per-layer features are encoded as 22-dimensional vectors, compressed by a VQ-VAE into 16-dimensional codebook embeddings, and fed through shared residual and self-attention layers with one decoder per DNN. MCTS uses the estimator's feedback to stochastically prune the mapping decision tree, cutting a search space of roughly $4\times10^{10}$ mappings for one example workload down to a roughly 30-second search. The threshold $th$ is what turns throughput prediction into starvation avoidance, by excluding all mappings that would leave any DNN below the cutoff.
What would settle it
Take the exact Orange Pi 5 setup and workload mixes from Section V, run RankMap with a freshly trained estimator, and compare the measured per-DNN throughput of the returned mappings against the estimator's predictions; if any returned mapping realizes potential throughput $P=0$ for a DNN, or if average normalized throughput drops to near the baseline, the paper's central claims of starvation prevention and $3.6\times$ throughput would be contradicted.
Extended reading notes
Core claim
The central claim is that priority-aware fine-grained DNN partitioning, searched with MCTS whose reward is the priority-weighted throughput vector from an attention-based CNN estimator, dominates coarse-grained and greedy managers on heterogeneous embedded hardware. In experiments on an Orange Pi 5 with six random mixes each of 3, 4, and 5 concurrent DNNs, RankMap in dynamic-priority mode achieves up to $3.6\times$ higher normalized average throughput than the GPU-only baseline and up to $1.6$–$2.2\times$ over the strongest prior manager, while static-priority mode keeps the high-priority DNN's potential throughput up to $57.5\times$ higher than baselines. Potential throughput $P$ is defined as a DNN's current throughput divided by its isolated GPU throughput, so $P=0$ means starvation. Across 72 sampled runs neither RankMap mode produced a single starved DNN, whereas the compared managers starved between 5 and 19 DNNs. The paper also reports that dynamic priorities track achieved potential throughput with an average Pearson correlation of $0.85$ for 3-DNN mixes, declining as the platform saturates.
Load-bearing premise
The throughput estimator, trained on 10,000 randomly generated mappings, must stay accurate on the high-reward mappings that MCTS actually selects; a systematic overprediction there would make the chosen mapping deliver less throughput and less starvation safety than reported.
Editorial extensions
If this is right
- If RankMap is correct, multi-DNN scheduling on heterogeneous edge devices can be treated as a learned search problem instead of a hand-tuned heuristic rule.
- High-priority DNNs can be given near-isolated performance under heavy co-tenant load while the whole workload keeps above-baseline average throughput.
- A hard disqualification threshold on predicted per-DNN throughput can prevent starvation without forcing a throughput-first objective.
- The multi-task estimator trained on 10,000 random mappings can predict throughput for mixes of up to five concurrent DNNs drawn from a 23-DNN pool, making deployment faster than evolutionary approaches that re-profile per generation.
- Static user-defined priorities and dynamic computational-profile priorities are complementary modes: dynamic mode maximizes average throughput, static mode maximizes protection of one critical DNN.
Reading between the lines
- The $57.5\times$ prioritization gain is measured on the high-priority DNN's potential throughput, not on end-to-end SLA metrics such as tail latency; a producer deploying RankMap should verify latency tail behavior under real query arrival patterns.
- Since the paper evaluates only computer-vision DNNs, an open extension is whether the layer-vector representation transfers to transformer-style LLM workloads at the edge.
- The threshold $th$ is treated as an input rather than a tuned parameter; a natural testable extension is to expose $th$ as an SLA knob and study the throughput-versus-starvation trade-off curve.
- The estimator's accuracy is validated on random mappings, but MCTS selects high-reward regions of the space; a targeted check is whether measured throughput on RankMap's chosen mappings matches the estimator's predictions as closely as on the random training distribution.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. RankMap is a run-time manager for executing multiple concurrent DNNs on heterogeneous embedded devices. The paper proposes representing a multi-DNN mapping as a tensor of per-layer descriptors, compressing the descriptors with a VQ-VAE, and training a multi-task attention-based CNN to predict each DNN's throughput for a given mapping. A Monte Carlo Tree Search (MCTS) then explores the mapping space, using the estimator's output to score mappings with a priority-weighted reward and disqualifying mappings in which any predicted throughput falls below a threshold. Experiments on an Orange Pi 5 compare RankMap in static and dynamic priority modes against Baseline, MOSAIC, ODMDEF, GA, and OmniBoost on 3-, 4-, and 5-DNN mixes, reporting normalized throughput, potential throughput of the highest-priority DNN, starvation counts, priority correlation, and a dynamic workload scenario. The paper claims up to 3.6x higher average throughput, up to 57.5x better prioritization, and no DNN starvation.
Significance. If the claims hold, RankMap is a useful contribution: it tackles a practical problem, combines a learned estimator with stochastic search in a way that is plausible for the stated scale, and it is evaluated on real hardware against several published baselines. The inclusion of a dynamic workload scenario and a starvation analysis is a strength. However, the evidence as presented is incomplete in load-bearing places: estimator accuracy is not measured on the mappings RankMap actually selects, the starvation-disqualification threshold is never reported, and the empirical comparisons have no variance information. The significance is therefore conditional on additional validation rather than being fully established by the current manuscript.
major comments (4)
- [Section V (Dataset and estimator training)] Estimator accuracy is the load-bearing component of MCTS, yet the only reported accuracy is an L2 loss of about 0.08 on a 10% split that the paper explicitly says was used 'for feedback during training'. There is no independent held-out test set, and no predicted-versus-measured throughput comparison for the high-reward mappings selected by MCTS in Figures 5-8. Since 10K random mappings are a negligible fraction of the roughly 4e10 mapping space for a single 4-DNN mix, and since MCTS deliberately explores high-reward regions, the measured x3.6 throughput and no-starvation results cannot be attributed to estimator-guided search unless estimator error on the search trajectory is quantified. Please report per-DNN predicted versus measured throughput for the mappings actually deployed, and evaluate on workloads whose DNNs are held out from the 23-DNN training pool.
- [Section IV-E (Reward function)] The disqualification threshold th in the MCTS objective is never assigned a value in the experiments, and no sensitivity analysis is provided. Because th determines which mappings are retained and rewards which mappings are chosen, the no-starvation result in Section V-B depends entirely on this hand-chosen parameter. If th was selected using the evaluation workloads, then the no-starvation claim is circular. Please report the exact th values used for all experiments, describe how they were chosen, and show how the throughput, prioritization, and starvation metrics vary with th.
- [Section V-A (Throughput and prioritization comparison)] All reported results are single measurements from one hardware board with no repeated runs, error bars, or confidence intervals. The headline ratios (x3.6 and x57.5) are computed from only six mixes per workload size, so run-to-run variability and measurement noise could change the relative ordering of methods. Please provide multiple independent runs for at least the key mixes, or an explicit quantitative statement of measurement variance on the Orange Pi 5.
- [Sections V-A and V-C (Priority metrics)] The paper motivates priority awareness through SLA requirements, but the evaluation does not use an SLA or deadline-based metric. The 'prioritization' claims are supported by two different proxies: the potential throughput of the single highest-priority DNN (Section V-A) and the Pearson correlation between P and priority vector p (Section V-C). Neither proxy directly measures whether SLA constraints are satisfied. The abstract's 'satisfying priority constraints x57.5 more efficiently' overstates what is demonstrated. Please either report an SLA/deadline-based metric or revise the claim to describe the actual proxy being measured.
minor comments (5)
- [Equation (1)] The definition of the 22-dimensional layer vector is hard to parse; please clarify which components make up ifm, ofm, w, and ps and how the total dimension of 22 is obtained.
- [Section IV-E] The expression 3^(8+20+18+18) is ambiguous; write it with parentheses and a brief explanation that the exponent is the number of valid partition points summed over the four DNNs.
- [Section V-B and Figure 8] Starvation is defined as potential throughput P=0 in the text and Figure 7, but Figure 8 shows a starved marker and an axis extending below zero; please reconcile the definition and the plot annotation.
- [Section V-D] The run-time comparison is only qualitative; please provide actual response-time numbers for all managers as a small table or in the text.
- [Section V] The 10% split used for feedback during training is referred to as a way to monitor training, not as a test set; please state explicitly that no independent test set is used, or add one.
Circularity Check
No significant circularity: the reported throughput, prioritization, and starvation results are hardware-measured, and the learned estimator is only a search heuristic, not the evaluation oracle.
full rationale
The paper's central claims (x3.6 higher average throughput, x57.5 prioritization improvement, no starvation) are established by executing the selected mappings on the Orange Pi 5 and measuring actual throughput, not by reading values off the learned estimator. The estimator is used inside MCTS to prune the search space, but the final reported T and P are measured on hardware, so the derivation is not circular by construction. The estimator's L2 loss of about 0.08 is reported on a 10% split that is explicitly 'reserved for feedback during training,' which weakens the generalization evidence, but that is a validation-strength concern, not a circularity: the headline numbers do not come from the estimator. The disqualification threshold th is not reported in the experiments, and if it had been tuned on the evaluation workloads that would be a fitted-input concern, but there is no textual evidence that this happened; the paper presents th as a hyperparameter of the reward function. The self-citations (e.g., OmniBoost [6], MapFormer [2], and the authors' balancing-throughput work [4]) are used as baselines and related work, not as load-bearing justifications for RankMap's own correctness. No equation in the paper reduces the claimed result to its inputs: the MCTS objective is a constrained weighted maximization, and the measured outcomes are independent of the estimator's internal predictions. Overall, the derivation chain is self-contained with respect to circularity, though estimator generalization and unreported threshold values are legitimate empirical risks.
Assumptions & free parameters
free parameters (2)
- Disqualification threshold th
- MCTS computational budget
assumptions (4)
- domain assumption The 22-dimensional layer representation (Equation 1) captures all information needed to predict per-DNN throughput on the target hardware.
- domain assumption The 10K randomly generated mappings are representative of the entire mapping solution space, including the high-reward regions MCTS explores.
- domain assumption The Orange Pi 5 board is representative of heterogeneous embedded devices.
- domain assumption The ARM Compute Library and OpenCL execution are deterministic, and throughput (inferences per second) is a valid objective for all DNNs in the workload.
Cite this review
Pith. "Pith review of RankMap: Priority-Aware Multi-DNN Manager for Heterogeneous Embedded Devices." pith.science (2026). https://pith.science/paper/3JK2IUO7
@misc{pith2026241117867,
author = {Pith},
title = {Pith review of: RankMap: Priority-Aware Multi-DNN Manager for Heterogeneous Embedded Devices},
year = {2026},
howpublished = {\url{https://pith.science/paper/3JK2IUO7}},
note = {Machine review of arXiv:2411.17867}
}
read the original abstract
Modern edge data centers simultaneously handle multiple Deep Neural Networks (DNNs), leading to significant challenges in workload management. Thus, current management systems must leverage the architectural heterogeneity of new embedded systems to efficiently handle multi-DNN workloads. This paper introduces RankMap, a priority-aware manager specifically designed for multi-DNN tasks on heterogeneous embedded devices. RankMap addresses the extensive solution space of multi-DNN mapping through stochastic space exploration combined with a performance estimator. Experimental results show that RankMap achieves x3.6 higher average throughput compared to existing methods, while preventing DNN starvation under heavy workloads and improving the prioritization of specified DNNs by x57.5.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Machine learning at facebook: Understanding inference at the edge,
C.-J. Wu et al., “Machine learning at facebook: Understanding inference at the edge,” in 2019 IEEE international symposium on high performance computer architecture (HPCA). IEEE, 2019, pp. 331–344
work page 2019
-
[2]
A. Karatzas and I. Anagnostopoulos, “Mapformer: Attention-based multi- dnn manager for throughout & power co-optimization on embedded devices,” in 2024 IEEE/ACM International Conference on Computer- Aided Design (ICCAD) . IEEE/ACM, 2024, pp. 1–9
work page 2024
-
[3]
C. Hsieh et al., “Surf: Self-aware unified runtime framework for parallel programs on heterogeneous mobile architectures,” in 2019 IFIP/IEEE 27th International Conference on Very Large Scale Integration (VLSI- SoC). IEEE, 2019
work page 2019
-
[4]
Balancing throughput and fair execution of multi-dnn workloads on heterogeneous embedded devices,
A. Karatzas and I. Anagnostopoulos, “Balancing throughput and fair execution of multi-dnn workloads on heterogeneous embedded devices,” IEEE Transactions on Emerging Topics in Computing , 2024
work page 2024
-
[5]
Scheduling of deep learning applications onto heteroge- neous processors in an embedded device,
D. Kang et al., “Scheduling of deep learning applications onto heteroge- neous processors in an embedded device,” IEEE Access, 2020
work page 2020
-
[6]
Omniboost: Boosting throughput of heterogeneous embedded devices under multi-dnn workload,
A. Karatzas and I. Anagnostopoulos, “Omniboost: Boosting throughput of heterogeneous embedded devices under multi-dnn workload,” in 2023 60th ACM/IEEE Design Automation Conference (DAC) . IEEE, 2023
work page 2023
-
[7]
Cd-msa: Cooperative and deadline- aware scheduling for efficient multi-tenancy on dnn accelerators,
C. Wang, Y . Bai, and D. Sun, “Cd-msa: Cooperative and deadline- aware scheduling for efficient multi-tenancy on dnn accelerators,” IEEE Transactions on Parallel and Distributed Systems , 2023
work page 2023
-
[8]
Less is more: Optimizing function calling for llm execution on edge devices,
V . Paramanayakam, A. Karatzas, I. Anagnostopoulos, and D. Stamoulis, “Less is more: Optimizing function calling for llm execution on edge devices,” arXiv preprint arXiv:2411.15399 , 2024
arXiv 2024
Show all 29 references
-
[9]
Deep learning with edge computing: A review,
J. Chen and X. Ran, “Deep learning with edge computing: A review,” Proceedings of the IEEE , vol. 107, no. 8, pp. 1655–1674, 2019
2019
-
[10]
Pythia: An edge first agent for state prediction in high-dimensional environments,
A. Karatzas and I. Anagnostopoulos, “Pythia: An edge first agent for state prediction in high-dimensional environments,” IEEE Embedded Systems Letters, 2024
2024
-
[11]
Deepmon: Mobile gpu-based deep learning framework for continuous vision applications,
L. N. Huynh, Y . Lee, and R. K. Balan, “Deepmon: Mobile gpu-based deep learning framework for continuous vision applications,” in Proceedings of the 15th Annual International Conference on Mobile Systems, Appli- cations, and Services , 2017, pp. 82–95
2017
-
[12]
Vila: On pre-training for visual language models,
J. Lin, H. Yin, W. Ping, Y . Lu, P. Molchanov, A. Tao, H. Mao, J. Kautz, M. Shoeybi, and S. Han, “Vila: On pre-training for visual language models,” 2023
2023
-
[13]
A. S. Developers. (2023) Arm NN: The easy way to deploy edge ML. Arm Community. [On- line]. Available: https://community.arm.com/arm-community-blogs/b/ tools-software-ides-blog/posts/arm-nn-the-easy-way-to-deploy-edge-ml
2023
-
[14]
High-throughput cnn inference on embedded arm big. little multicore processors,
S. Wang et al. , “High-throughput cnn inference on embedded arm big. little multicore processors,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems , 2019
2019
-
[15]
Arm-co-up: Arm co operative u tilization of p rocessors,
E. Aghapour, D. Sapra, A. Pimentel, and A. Pathania, “Arm-co-up: Arm co operative u tilization of p rocessors,” ACM Transactions on Design Automation of Electronic Systems , 2024
2024
-
[16]
Band: coordinated multi-dnn inference on heteroge- neous mobile processors,
J. S. Jeong et al. , “Band: coordinated multi-dnn inference on heteroge- neous mobile processors,” in Proceedings of the 20th Annual Interna- tional Conference on Mobile Systems, Applications and Services , 2022
2022
-
[17]
Heterogeneous dataflow accelerators for multi-dnn workloads,
H. Kwon et al. , “Heterogeneous dataflow accelerators for multi-dnn workloads,” in 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA). IEEE, 2021, pp. 71–83
2021
-
[18]
Multi-objective hardware-mapping co-optimisation for multi-dnn workloads on chiplet-based accelerators,
A. Das, E. Russo, and M. Palesi, “Multi-objective hardware-mapping co-optimisation for multi-dnn workloads on chiplet-based accelerators,” IEEE Transactions on Computers , 2024
2024
-
[19]
Targeting dnn inference via efficient utilization of heterogeneous precision dnn accelerators,
O. Spantidi et al. , “Targeting dnn inference via efficient utilization of heterogeneous precision dnn accelerators,” IEEE Transactions on Emerging Topics in Computing , 2022
2022
-
[20]
Pipeline parallelism for dnn inference with practical performance guarantees,
A. Archer, M. Fahrbach, K. Liu, and P. Prabhu, “Pipeline parallelism for dnn inference with practical performance guarantees,” arXiv preprint arXiv:2311.03703, 2023
2023 arXiv
-
[21]
Mosaic: Heterogeneity-, communication-, and constraint- aware model slicing and execution for accurate and efficient inference,
M. Han et al., “Mosaic: Heterogeneity-, communication-, and constraint- aware model slicing and execution for accurate and efficient inference,” in 2019 28th International Conference on Parallel Architectures and Compilation Techniques (PACT). IEEE, 2019
2019
-
[22]
Odmdef: on-device multi-dnn execution framework utilizing adaptive layer-allocation on general purpose cores and acceler- ators,
C. Lim and M. Kim, “Odmdef: on-device multi-dnn execution framework utilizing adaptive layer-allocation on general purpose cores and acceler- ators,” IEEE Access, vol. 9, pp. 85 403–85 417, 2021
2021
-
[23]
Neural discrete representation learning,
A. Van Den Oord, O. Vinyals et al. , “Neural discrete representation learning,” Advances in neural information processing systems , 2017
2017
-
[24]
Distributed representations of words and phrases and their compositionality,
T. Mikolov et al. , “Distributed representations of words and phrases and their compositionality,” Advances in neural information processing systems, vol. 26, 2013
2013
-
[25]
Hifi-codec: Group-residual vector quantization for high fidelity audio codec,
D. Yang, S. Liu et al. , “Hifi-codec: Group-residual vector quantization for high fidelity audio codec,” arXiv preprint arXiv:2305.02765 , 2023
2023 arXiv
-
[26]
Attention is all you need,
A. Vaswani, N. Shazeer et al. , “Attention is all you need,” Advances in neural information processing systems , vol. 30, 2017
2017
-
[27]
Efficient attention: Attention with linear complexities,
Z. Shen, M. Zhang et al. , “Efficient attention: Attention with linear complexities,” in Proceedings of the IEEE/CVF winter conference on applications of computer vision , 2021, pp. 3531–3539
2021
-
[28]
Monte carlo tree search: A review of recent modifications and applications,
M. ´Swiechowski et al. , “Monte carlo tree search: A review of recent modifications and applications,” Artificial Intelligence Review , 2022
2022
-
[29]
(2017) Arm compute library
ARM. (2017) Arm compute library. [Online]. [Online]. Available: https://www.arm.com/technologies/compute-library
2017
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.