{"id":"9dbd363b-5730-4c70-bacd-91c27b2fc3e0","arxiv_id":"2411.17867","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A priority-aware multi-DNN scheduler using Monte Carlo tree search and a learned throughput estimator reports higher throughput and no starvation on one embedded board.","lead":"RankMap is a software manager that schedules several neural networks running at once on a small embedded computer, splitting each network across CPU and GPU cores. It reports up to 3.6 times higher throughput than a common baseline and never lets any network stall, which matters for edge data centers serving many users.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Estimator accuracy is never measured on the MCTS-selected mappings that RankMap actually deploys, and the 10% validation split is used for training feedback; if the estimator misgeneralizes to high-reward regions, the reported throughput and starvation gains are not attributable to the claimed…","rationale":"The paper’s measured throughput numbers are hardware measurements, so for the 72 reported workload samples the x3.6 and no-starvation observations are real as far as they go. The load-bearing question is whether the claimed mechanism—MCTS guided by the learned estimator plus a starvation threshold—is what produced them. The estimator is trained on random mappings, MCTS searches a massively larger space, and no accuracy check is reported for the selected mappings; the 10% split called “for feedback during training” is not a held-out test. This is the same weakest assumption the reader identified, so I agree with the conditional verdict rather than strengthening it. It is addressable: a focused evaluation of estimator error on the actual chosen mappings, plus disclosure of th and seeds, would settle whether the gains are attributable to the proposed method. I do not see an internal inconsistency that would justify rejection; the concern is missing evidence, not demonstrated failure.","tokens_in":12483,"tokens_out":6071,"duration_ms":58477,"concrete_test":"Run a held-out evaluation of the estimator on the exact final mappings produced by MCTS for the 18 workload mixes in Figs. 5–8: record predicted and measured per-DNN throughput for each chosen mapping, compute per-DNN relative error, and compare with the error on random validation mappings. If the mean absolute relative error on selected mappings is comparable to the random-validation error, the estimator-guided search is supported; if it is substantially larger, the search is exploiting estimator bias and the central throughput/starvation claims are not established. The authors should also report th used for each workload and release code/seeds so the test is reproducible.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central mechanism is that MCTS uses a learned estimator to identify high-reward mappings; if the estimator is wrong on the search trajectory, RankMap reduces to random search, so the measured gains are unexplained. The estimator is trained on 10K random mappings drawn from a pool of 23 DNNs, while the solution space for a single 4-DNN workload is already ~4e10 (Sec. IV-E). Random sampling therefore covers a negligible fraction of mappings, and the high-throughput mappings selected by MCTS live in an under-sampled tail. The paper reports only L2 loss ~0.08 on a 10% split that is explicitly reserved “for feedback during training” (Sec. V), not independent held-out accuracy, and it never reports predicted-vs-measured throughput for the specific mappings chosen in Figs. 5–8. Without that comparison, the x3.6 throughput and no-starvation results could be due to fortuitous exploration rather than estimator-guided search. The disqualification threshold th (Sec. IV-E) is also never reported in the experiments, so it is unknown whether “no starvation” is enforced by the estimator or only asserted post hoc; a threshold tuned on the evaluation workloads would be circular.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"RankMap is a run-time manager for executing multiple concurrent DNNs on heterogeneous embedded devices. The paper proposes representing a multi-DNN mapping as a tensor of per-layer descriptors, compressing the descriptors with a VQ-VAE, and training a multi-task attention-based CNN to predict each DNN's throughput for a given mapping. A Monte Carlo Tree Search (MCTS) then explores the mapping space, using the estimator's output to score mappings with a priority-weighted reward and disqualifying mappings in which any predicted throughput falls below a threshold. Experiments on an Orange Pi 5 compare RankMap in static and dynamic priority modes against Baseline, MOSAIC, ODMDEF, GA, and OmniBoost on 3-, 4-, and 5-DNN mixes, reporting normalized throughput, potential throughput of the highest-priority DNN, starvation counts, priority correlation, and a dynamic workload scenario. The paper claims up to 3.6x higher average throughput, up to 57.5x better prioritization, and no DNN starvation.","tokens_in":12751,"tokens_out":5028,"duration_ms":49420,"significance":"If the claims hold, RankMap is a useful contribution: it tackles a practical problem, combines a learned estimator with stochastic search in a way that is plausible for the stated scale, and it is evaluated on real hardware against several published baselines. The inclusion of a dynamic workload scenario and a starvation analysis is a strength. However, the evidence as presented is incomplete in load-bearing places: estimator accuracy is not measured on the mappings RankMap actually selects, the starvation-disqualification threshold is never reported, and the empirical comparisons have no variance information. The significance is therefore conditional on additional validation rather than being fully established by the current manuscript.","major_comments":[{"comment":"Estimator accuracy is the load-bearing component of MCTS, yet the only reported accuracy is an L2 loss of about 0.08 on a 10% split that the paper explicitly says was used 'for feedback during training'. There is no independent held-out test set, and no predicted-versus-measured throughput comparison for the high-reward mappings selected by MCTS in Figures 5-8. Since 10K random mappings are a negligible fraction of the roughly 4e10 mapping space for a single 4-DNN mix, and since MCTS deliberately explores high-reward regions, the measured x3.6 throughput and no-starvation results cannot be attributed to estimator-guided search unless estimator error on the search trajectory is quantified. Please report per-DNN predicted versus measured throughput for the mappings actually deployed, and evaluate on workloads whose DNNs are held out from the 23-DNN training pool.","section":"Section V (Dataset and estimator training)"},{"comment":"The disqualification threshold th in the MCTS objective is never assigned a value in the experiments, and no sensitivity analysis is provided. Because th determines which mappings are retained and rewards which mappings are chosen, the no-starvation result in Section V-B depends entirely on this hand-chosen parameter. If th was selected using the evaluation workloads, then the no-starvation claim is circular. Please report the exact th values used for all experiments, describe how they were chosen, and show how the throughput, prioritization, and starvation metrics vary with th.","section":"Section IV-E (Reward function)"},{"comment":"All reported results are single measurements from one hardware board with no repeated runs, error bars, or confidence intervals. The headline ratios (x3.6 and x57.5) are computed from only six mixes per workload size, so run-to-run variability and measurement noise could change the relative ordering of methods. Please provide multiple independent runs for at least the key mixes, or an explicit quantitative statement of measurement variance on the Orange Pi 5.","section":"Section V-A (Throughput and prioritization comparison)"},{"comment":"The paper motivates priority awareness through SLA requirements, but the evaluation does not use an SLA or deadline-based metric. The 'prioritization' claims are supported by two different proxies: the potential throughput of the single highest-priority DNN (Section V-A) and the Pearson correlation between P and priority vector p (Section V-C). Neither proxy directly measures whether SLA constraints are satisfied. The abstract's 'satisfying priority constraints x57.5 more efficiently' overstates what is demonstrated. Please either report an SLA/deadline-based metric or revise the claim to describe the actual proxy being measured.","section":"Sections V-A and V-C (Priority metrics)"}],"minor_comments":[{"comment":"The definition of the 22-dimensional layer vector is hard to parse; please clarify which components make up ifm, ofm, w, and ps and how the total dimension of 22 is obtained.","section":"Equation (1)"},{"comment":"The expression 3^(8+20+18+18) is ambiguous; write it with parentheses and a brief explanation that the exponent is the number of valid partition points summed over the four DNNs.","section":"Section IV-E"},{"comment":"Starvation is defined as potential throughput P=0 in the text and Figure 7, but Figure 8 shows a starved marker and an axis extending below zero; please reconcile the definition and the plot annotation.","section":"Section V-B and Figure 8"},{"comment":"The run-time comparison is only qualitative; please provide actual response-time numbers for all managers as a small table or in the text.","section":"Section V-D"},{"comment":"The 10% split used for feedback during training is referred to as a way to monitor training, not as a test set; please state explicitly that no independent test set is used, or add one.","section":"Section V"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"RankMap is a genuine attempt at a real problem: multi-DNN scheduling with priorities on heterogeneous embedded devices. The combination of MCTS with a learned multi-task throughput estimator and a priority-weighted reward is new relative to the baselines they compare against, and the hardware results on the Orange Pi 5 are believable, not hand-waved. The dynamic workload trace in Figure 8 is a nice illustration of the throughput-vs-starvation trade-off.\n\nThat said, the evidence has several addressable gaps. No error bars or multiple runs appear anywhere; all comparisons are point estimates. Only one hardware platform is used. The estimator is trained on workloads drawn from the same 23-DNN pool used to construct the test mixes, and the 10% validation split is explicitly described as 'feedback during training' rather than an independent held-out set. The disqualification threshold th is never reported, so the 'no starvation' results could be partly a consequence of threshold tuning. The stress-test concern is the most serious: the paper never reports estimator accuracy on the specific mappings MCTS selects. Since MCTS is guided by the estimator, if the estimator mispredicts in the high-reward region, the measured gains are not attributable to the claimed mechanism. The authors report only L2 loss around 0.08 on that training-feedback split. This is a load-bearing gap for the central claim.\n\nNone of this sinks the paper. The measured throughput and starvation numbers are real hardware measurements, so there's no direct circularity in the results themselves. The gaps are fixable: report multiple runs with variance, test on a second board, hold out evaluation DNN pools disjoint from training, report th, and measure predicted-vs-actual throughput on the final selected mappings. A revised version with those additions would be a solid contribution.\n\nThis is a paper for the embedded systems and edge computing community. I'd bring it to a reading group as a useful example of learned schedulers, and I'd send it to reviewers if it were submitted; it deserves serious referee time. My own verdict would be conditional accept with requests for the above evidence.","headline":"Priority-aware MCTS scheduler worth engaging, but the lack of estimator validation on the search trajectory and unreported threshold th are real gaps that need addressing.","tokens_in":13245,"tokens_out":2110,"would_cite":false,"duration_ms":19222,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RankMap, a priority-aware multi-DNN manager for heterogeneous embedded devices, claims up to 3.6x higher average throughput and 57.5x better prioritization than prior methods while preventing DNN starvation through Monte Carlo Tree Search…","keywords":["multi-DNN scheduling","heterogeneous embedded systems","DNN partitioning","Monte Carlo Tree Search","throughput estimation","priority-aware scheduling","starvation prevention","edge inference"],"falsifier":"Take the exact Orange Pi 5 setup and workload mixes from Section V, run RankMap with a freshly trained estimator, and compare the measured per-DNN throughput of the returned mappings against the estimator's predictions; if any returned mapping realizes potential throughput $P=0$ for a DNN, or if average normalized throughput drops to near the baseline, the paper's central claims of starvation prevention and $3.6\\times$ throughput would be contradicted.","tokens_in":12311,"feed_emoji":"⚡","tokens_out":8095,"duration_ms":66595,"temperature":0.7,"pith_summary":"RankMap is a runtime manager for embedded devices that run several deep neural networks at the same time. It splits each DNN into fine-grained pipeline stages and maps those stages across the big CPU cluster, the LITTLE CPU cluster, and the GPU, using Monte Carlo Tree Search guided by a learned per-DNN throughput estimator. The paper reports that this approach raises average system throughput by up to $3.6\\times$ over previous managers, satisfies priority constraints up to $57.5\\times$ better for a critical DNN, and prevents DNN starvation across all tested workloads. If these results hold, edge data centers can serve mixed-priority DNN workloads at higher speed without sacrificing low-priority applications, which is the practical case the paper sets out to make.","feed_headline":"RankMap raises edge DNN throughput 3.6x and stops starvation","feed_subtitle":"A learned search maps partitioned DNNs across CPU clusters and GPU, keeping priority apps fed under heavy load.","key_machinery":"The load-bearing mechanism is the reward-driven search loop: each candidate mapping $M$ is scored by $O(M)^T \\times p$, where $O(M)$ is the estimator's per-DNN throughput vector and $p$ is the priority vector, and any mapping with a predicted throughput below a threshold $th$ is disqualified. The estimator itself is a multi-task attention-based CNN of about $3.7$ million parameters: per-layer features are encoded as 22-dimensional vectors, compressed by a VQ-VAE into 16-dimensional codebook embeddings, and fed through shared residual and self-attention layers with one decoder per DNN. MCTS uses the estimator's feedback to stochastically prune the mapping decision tree, cutting a search space of roughly $4\\times10^{10}$ mappings for one example workload down to a roughly 30-second search. The threshold $th$ is what turns throughput prediction into starvation avoidance, by excluding all mappings that would leave any DNN below the cutoff.","core_discovery":"The central claim is that priority-aware fine-grained DNN partitioning, searched with MCTS whose reward is the priority-weighted throughput vector from an attention-based CNN estimator, dominates coarse-grained and greedy managers on heterogeneous embedded hardware. In experiments on an Orange Pi 5 with six random mixes each of 3, 4, and 5 concurrent DNNs, RankMap in dynamic-priority mode achieves up to $3.6\\times$ higher normalized average throughput than the GPU-only baseline and up to $1.6$–$2.2\\times$ over the strongest prior manager, while static-priority mode keeps the high-priority DNN's potential throughput up to $57.5\\times$ higher than baselines. Potential throughput $P$ is defined as a DNN's current throughput divided by its isolated GPU throughput, so $P=0$ means starvation. Across 72 sampled runs neither RankMap mode produced a single starved DNN, whereas the compared managers starved between 5 and 19 DNNs. The paper also reports that dynamic priorities track achieved potential throughput with an average Pearson correlation of $0.85$ for 3-DNN mixes, declining as the platform saturates.","pith_inferences":["The $57.5\\times$ prioritization gain is measured on the high-priority DNN's potential throughput, not on end-to-end SLA metrics such as tail latency; a producer deploying RankMap should verify latency tail behavior under real query arrival patterns.","Since the paper evaluates only computer-vision DNNs, an open extension is whether the layer-vector representation transfers to transformer-style LLM workloads at the edge.","The threshold $th$ is treated as an input rather than a tuned parameter; a natural testable extension is to expose $th$ as an SLA knob and study the throughput-versus-starvation trade-off curve.","The estimator's accuracy is validated on random mappings, but MCTS selects high-reward regions of the space; a targeted check is whether measured throughput on RankMap's chosen mappings matches the estimator's predictions as closely as on the random training distribution."],"forward_implications":["If RankMap is correct, multi-DNN scheduling on heterogeneous edge devices can be treated as a learned search problem instead of a hand-tuned heuristic rule.","High-priority DNNs can be given near-isolated performance under heavy co-tenant load while the whole workload keeps above-baseline average throughput.","A hard disqualification threshold on predicted per-DNN throughput can prevent starvation without forcing a throughput-first objective.","The multi-task estimator trained on 10,000 random mappings can predict throughput for mixes of up to five concurrent DNNs drawn from a 23-DNN pool, making deployment faster than evolutionary approaches that re-profile per generation.","Static user-defined priorities and dynamic computational-profile priorities are complementary modes: dynamic mode maximizes average throughput, static mode maximizes protection of one critical DNN."],"supporting_citations":[{"why":"Supplies the VQ-VAE quantization mechanism that compresses per-layer representations into codebook embeddings for the estimator.","marker":"[23]"},{"why":"Supplies the self-attention module used in the estimator backbone to handle statistically independent channel dimensions.","marker":"[26]"},{"why":"Supplies the Monte Carlo Tree Search algorithm that prunes the mapping space using estimator feedback.","marker":"[28]"},{"why":"Provides the baseline linear-regression slicing framework whose throughput and lack of priority support RankMap compares against.","marker":"[21]"},{"why":"Provides the baseline on-device multi-DNN execution framework used for comparison in throughput and starvation experiments.","marker":"[22]"},{"why":"Provides the evolutionary-algorithm scheduler baseline that RankMap claims to outperform in speed and scalability.","marker":"[5]"},{"why":"Provides the prior CNN-based throughput optimizer baseline; RankMap's comparison highlights improvements in prioritization and starvation prevention.","marker":"[6]"}],"fun_headline_variants":["RankMap: 3.6x edge DNN throughput, zero starvation","RankMap: 3.6x throughput, 57.5x priority, no starvation","Priority-aware DNN manager: 3.6x throughput, 0 starved","RankMap cuts DNN starvation to zero, triples throughput","Heterogeneous edge DNN manager yields 3.6x throughput, no starvation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The throughput estimator, trained on 10,000 randomly generated mappings, must stay accurate on the high-reward mappings that MCTS actually selects; a systematic overprediction there would make the chosen mapping deliver less throughput and less starvation safety than reported.","fun_headline_variants_meta":{"raw":{"variants":["RankMap: 3.6x edge DNN throughput, zero starvation","RankMap: 3.6x throughput, 57.5x priority, no starvation","Priority-aware DNN manager: 3.6x throughput, 0 starved","RankMap cuts DNN starvation to zero, triples throughput","Heterogeneous edge DNN manager yields 3.6x throughput, no starvation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000684,"raw_usage":{"total_tokens":3076,"prompt_tokens":893,"completion_tokens":2183,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":509,"completion_tokens_details":{"reasoning_tokens":2077}},"tokens_in":509,"tokens_out":2183,"duration_ms":14844,"temperature":1.0,"reasoning_tokens":2077,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:44:34.240702+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the exact Orange Pi 5 setup and workload mixes from Section V, run RankMap with a freshly trained estimator, and compare the measured per-DNN throughput of the returned mappings against the estimator's predictions; if any returned mapping realizes potential throughput $P=0$ for a DNN, or if average normalized throughput drops to near the baseline, the paper's central claims of starvation prevention and $3.6\\times$ throughput would be contradicted.","supporting_citations":[{"cited_title":"Monte carlo tree search: A review of recent modifications and applications,","cited_arxiv_id":null,"evidence_quote":"Supplies the Monte Carlo Tree Search algorithm that prunes the mapping space using estimator feedback."},{"cited_title":"Neural discrete representation learning,","cited_arxiv_id":null,"evidence_quote":"Supplies the VQ-VAE quantization mechanism that compresses per-layer representations into codebook embeddings for the estimator."},{"cited_title":"Attention is all you need,","cited_arxiv_id":null,"evidence_quote":"Supplies the self-attention module used in the estimator backbone to handle statistically independent channel dimensions."},{"cited_title":"Mosaic: Heterogeneity-, communication-, and constraint- aware model slicing and execution for accurate and efficient inference,","cited_arxiv_id":null,"evidence_quote":"Provides the baseline linear-regression slicing framework whose throughput and lack of priority support RankMap compares against."},{"cited_title":"Odmdef: on-device multi-dnn execution framework utilizing adaptive layer-allocation on general purpose cores and acceler- ators,","cited_arxiv_id":null,"evidence_quote":"Provides the baseline on-device multi-DNN execution framework used for comparison in throughput and starvation experiments."},{"cited_title":"Scheduling of deep learning applications onto heteroge- neous processors in an embedded device,","cited_arxiv_id":null,"evidence_quote":"Provides the evolutionary-algorithm scheduler baseline that RankMap claims to outperform in speed and scalability."},{"cited_title":"Omniboost: Boosting throughput of heterogeneous embedded devices under multi-dnn workload,","cited_arxiv_id":null,"evidence_quote":"Provides the prior CNN-based throughput optimizer baseline; RankMap's comparison highlights improvements in prioritization and starvation prevention."}],"review_version":1}