REVIEW 3 major objections 5 minor 33 references
Agentic CPU-GPU Scheduling for Heterogeneous AI Workloads
T0 review · 3 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash
Pith's one-line read A large-language-model scheduler paired with a runtime monitor can choose the same CPU/GPU tool placements as exhaustive search in all 13 heterogeneous scheduling scenarios tested.
desk verdict The LLM+monitor scheduling idea is genuinely interesting and the paper is transparent about its limits, but the headline performance comparison excludes search overhead and the scenarios are small; the mapping-accuracy result is more solid than the latency claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the agentic scheduler: a closed loop between an offline profiler, an LLM scheduler, and a runtime monitor. The monitor's feedback mechanisms carry the argument: sliding-window running averages correct optimistic static latencies under utilization contention; symmetric reprobes measure a tool on an alternative device with the rest of the mapping fixed; swap reprobes test two-tool exchanges when a single move would exceed the VRAM budget; and exploration hints trigger diversity when mappings stall. The problem itself is formalized as a dependency DAG where each tool executes in one of three modes—immediate GPU execution (gpu_now, counts against VRAM), queued GPU execution
What would settle it
Re-run scenario S13 with a stopwatch on the whole closed loop, summing wall-clock time of every execution—failed OOM attempts, reprobes, swap reprobes, and adopted mapping runs—until the first 4/4 oracle mapping appears; if that cumulative total exceeds the single-run end-to-end latency of the best static heuristic on the same hardware, then the reported speedup is an artifact of free exploration.
Extended reading notes
Core claim
The paper's central claim is that pairing an LLM decision layer with an algorithmic runtime monitor—a combination the authors call the agentic scheduler—can recover the brute-force-optimal device mapping in every one of the 13 four-tool workflows they construct, with zero offline training. The monitor does not choose placements; it expands what the LLM can see, supplying running averages under GPU contention, single-tool reprobes of unobserved placements, and bounded two-tool swap reprobes that reveal rearrangements invisible to any local move. With profile cards, the LLM often emits the oracle mapping immediately; without them, it converges to the oracle within one to three rounds. The pape
Load-bearing premise
The load-bearing assumption is that the monitor's probing executions are not counted in the end-to-end latency being optimized; if probe cost were included, the agentic scheduler could be slower in total time-to-decision than one-shot baselines.
Editorial extensions
If this is right
- GPU-first placement is measurably suboptimal: profiling 19 tools yields four scheduling zones, so deployments that default every GPU-capable tool to the GPU will routinely misplace tools and pay 1.5–2.7x end-to-end latency penalties under contention.
- Static profiles are not enough once multiple tools share the GPU; reliable scheduling requires runtime measurement of contention, or a scheduler that is willing to offload strategic tools.
- A queued-GPU mode materially helps memory-constrained workflows: deferring one strongly GPU-preferred tool can free enough VRAM to keep another tool resident, a two-tool trade-off that a greedy memory-only policy misses.
- An LLM can match bandit-style exploration on mapping quality while needing fewer rounds, because it folds measured what-if evidence into a prompt instead of treating every complete mapping as an independent arm to explore.
- The separation of observation (monitor) and decision (LLM) is the load-bearing design principle, and the paper presents it as a general pattern for combinatorial systems optimization problems.
Reading between the lines
- In our reading, the reported end-to-end times are for the best mapping only; the probe, reprobe, and swap executions used to discover it are excluded. If those discovery costs were included in total time-to-decision, the agentic scheduler might not beat one-shot classical baselines.
- The paper explicitly disclaims a formal convergence guarantee; the 13/13 oracle matches are empirical convergence within 1–3 rounds over 20-round runs, not a proof that any starting point reaches the optimum.
- The monitor's probes scale roughly quadratically with the number of tools per mapping change; extrapolating beyond four-tool scenarios will likely require probe sampling or a learned exploration policy.
- The observation/decision split could transfer to other combinatorial system-tuning problems—compiler pass ordering, database knobs, cache policies—where an LLM reasons over targeted what-if measurements instead of training a policy.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies device scheduling for agentic AI tool workflows. It profiles 19 AI tools on CPU and GPU, categorizes them into scheduling zones, and shows that a blanket GPU-first policy is often suboptimal. It formulates scheduling as a DAG placement problem with three execution modes (cpu, gpu_now, gpu_queue) under a VRAM budget, and proposes an agentic scheduler that combines an LLM policy with an algorithmic runtime monitor providing running averages, single-tool reprobes, swap reprobes, and exploration hints. The evaluation uses 13 four-tool scenarios spanning precedence constraints, GPU contention, and VRAM constraints. The paper claims that the agentic scheduler reaches the brute-force oracle mapping in all 13 scenarios, matches UCB1 on mapping accuracy, and outperforms HEFT, StarPU, and all-GPU, with zero offline training.
Significance. If the headline claims were supported by a fair evaluation, this would be a useful contribution: it would demonstrate that a general-purpose LLM, paired with measured runtime feedback, can make optimal placement decisions in small heterogeneous workflows without offline training. The paper is also unusually transparent: Appendix B discloses six prompt iterations, Appendix D acknowledges selection artifacts, and the scenario tables give enough detail for reproduction. However, the two central quantitative claims are not yet established. The performance comparison against one-shot baselines excludes the cost of the agent's own probing and repeated measurements, and the 13/13 oracle-match result was obtained after iterative prompt engineering on the same 13 scenarios. The underlying idea remains plausible, but the evaluation as reported is not sufficient to support the abstract's unqualified claims.
major comments (3)
- [§4.1, §4.5, Appendix D, Fig. 5] The reported performance comparison is asymmetric. For adaptive strategies (UCB1, StarPU, LLM), Fig. 5 and Table 7 use the best-found mapping's latency, measured only on rounds emitting that mapping; search/probe cost is excluded. Algorithm 1 (Lines 3–14) executes full-workflow reprobes and swap reprobes, and S13's C↔B swap probe alone is a full workflow execution. HEFT and all-GPU are one-shot. If total time-to-decision (all probes, repeated medians, LLM calls) were counted, the agentic scheduler would incur many full workflow executions before producing a mapping, while HEFT's cost is one profile lookup. Thus the abstract's unqualified 'outperforms HEFT, StarPU, and all-GPU' claim is not supported. Report total wall-clock cost including probes and LLM inference, or label the claim as final-mapping quality with search cost excluded and compare adaptive baselines under the same accountin
- [Appendix B, §3.3] The 13/13 oracle-match result is not an independent test: the prompt was iteratively engineered on the same 13 scenarios. Appendix B states that the prompt evolved through six versions, with versions 4/5 used for fan-out scenarios and version 6 for three-way scenarios. The 1× and 7× ratio cutoffs and the 5% stability threshold are hand-picked from Table 1 and observed scenario outcomes. This is a form of test-set tuning, so 'zero offline training' is only narrowly true. A held-out scenario family or an explicit development/test split is needed before claiming generalization.
- [§4.1, Appendix D] Fig. 5 reports the best of 20 rounds for adaptive strategies. Taking the minimum of 20 noisy measurements biases the comparison; the apparent ratios slightly below 1.0 are acknowledged in Appendix D as artifacts. Table 7 partially mitigates this by reporting steady-state means, but it still selects rounds that emit the best-found mapping. Report per-round distributions or time-to-first-arrival of the best mapping for all adaptive baselines, rather than a single best-of-20 value.
minor comments (5)
- [§4.1] The text says each round executes the workflow multiple times and reports the median, but it never states how many repetitions per round. Please report this number and whether it is constant across scenarios and baselines.
- [Table 5, Appendix C] Table 5 lists S13 as Match 4/4 while the note says the best-found mapping differs from oracle in tool D, and Appendix C shows D oscillates between cpu and gpu_queue after R3. Clarify whether 'Match' is the best single round (R3) or some steady-state aggregation, and reconcile the table note with the 4/4 entry.
- [§3.3, Fig. 4] The 5% stability threshold is used in the cold-start example but is not defined in Algorithm 1 or in the monitor specification. Specify which component applies this threshold and how it is computed from running averages.
- [§5] Section 5 correctly states there is no formal convergence guarantee. This limitation should be connected to the small number of scenarios; all 13 scenarios use four-tool DAGs, so the 13/13 oracle claim is not evidence for larger workflows. A statement of scope would make the contribution clearer.
- [§4.2, Table 4] The text says the LLM reaches 4/4 on S1–S4 from the profiling cards in the first scheduling round. A per-round trace or a stated first-4/4 round for S1–S4 would make this consistent with the S11–S13 reporting style.
Circularity Check
No significant circularity: the central claim is an empirical search result against an independently measured oracle, not a derivation reduced to its inputs.
full rationale
The paper's central claim—reaching the brute-force optimal mapping in all 13 scenarios—is an empirical search result, not a derivation. The oracle is independently defined in Section 4.1 as "the feasible mapping with the lowest measured median end-to-end latency under the same execution environment," and the scheduler's mapping is chosen from its own monitor measurements; the oracle is not used to construct prompts or to label training data. The monitor's probes are actual executions (Algorithm 1), and the LLM selects among the measured alternatives; this is feedback-driven search, not fitting an equation to the target. The hand-set offload thresholds (ratio 1/7 and the 5% flip threshold) are heuristics described in Section 3.3 as derived from isolated profiling, not shown to be fitted to oracle labels; even if they were overfit, that would be a generalization concern, not circularity. The self-admitted absence of a convergence guarantee (Section 5) and the best-of-20 reporting (Appendix D) are methodological limitations; they do not make the 'reaches oracle' claim definitionally identical to its inputs. There are no load-bearing self-citations or imported uniqueness theorems: all cited baselines, tools, and datasets are external. No specific reduction of the form 'Eq. X = Eq. Y by construction' or 'fitted parameter renamed as prediction' can be quoted, so the correct finding is no significant circularity.
Assumptions & free parameters
free parameters (5)
- Offload ratio upper cutoff =
7
- Offload ratio lower cutoff =
1
- What-if stability threshold =
5%
- Sliding-window length
- Exploration-hint trigger conditions
assumptions (6)
- standard math Precedence-constrained scheduling with resource capacities is NP-hard
- domain assumption Medians of 30 isolated runs provide stable per-tool latency and memory profiles
- domain assumption Concurrent GPU execution can be adequately captured by measured running averages and bounded reprobes
- domain assumption The specific LLM (DeepSeek-R1-Distill-Qwen-32B-AWQ) with the engineered prompt correctly reasons over the provided scheduling evidence
- domain assumption The brute-force oracle (lowest measured median E2E among feasible mappings) is the correct ground truth, and measurement noise is small relative to mapping differences
- domain assumption gpu_queue tasks occupy no VRAM while waiting and pay a fixed load-in cost when admitted
Cite this review
Pith. "Pith review of Agentic CPU-GPU Scheduling for Heterogeneous AI Workloads." pith.science (2026). https://pith.science/paper/KISKAND2
@misc{pith2026260722242,
author = {Pith},
title = {Pith review of: Agentic CPU-GPU Scheduling for Heterogeneous AI Workloads},
year = {2026},
howpublished = {\url{https://pith.science/paper/KISKAND2}},
note = {Machine review of arXiv:2607.22242}
}
read the original abstract
Agentic AI systems compose heterogeneous tool workloads on shared GPU/CPU infrastructure, yet existing frameworks assign all GPU-capable tools to the GPU by default. We profile 19 AI tools across GPU and CPU and find that 11 are GPU-preferred, 4 are ambiguous, 1 is CPU-preferred due to PCIe transfer dominance, and 3 are device-neutral, establishing that blanket GPU-first scheduling is suboptimal. We formulate device scheduling as assigning each tool to one of three options: immediate GPU execution, queued GPU execution, or CPU offload, under a VRAM budget, and identify two runtime factors that cause end-to-end latency to diverge from static profiles: GPU utilization contention and VRAM capacity contention. We present an agentic scheduler that pairs an LLM agent with an algorithmic runtime monitor, where the monitor expands what the LLM can observe via running averages, symmetric reprobing, swap reprobing, and exploration hints, without ever prescribing which mapping to adopt. Across 13 scenarios spanning serial execution, parallel contention, and memory-constrained execution, the agentic scheduler reaches the brute-force optimal mapping in all 13 scenarios, matching the best classical baseline on mapping accuracy while avoiding bandit-style exploration over complete mappings, and outperforming HEFT, StarPU, and the all-GPU policy while requiring zero offline training.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[1]
Ravichandra Addanki, Shaileshh Bojja Venkatakrishnan, Shreyan Gupta, Hongzi Mao, and Mohammad Alizadeh. 2019. Placeto: Learning generalizable device placement algorithms for distributed machine learning. InAdvances in Neural Information Processing Systems, Vol. 32
2019
-
[2]
Peter Auer, Nicolò Cesa-Bianchi, and Paul Fischer. 2002. Finite-time analysis of the multiarmed bandit problem.Machine Learning47, 2 (2002), 235–256
2002
-
[3]
Cédric Augonnet, Samuel Thibault, Raymond Namyst, and Pierre-André Wacre- nier. 2009. StarPU: A unified platform for task scheduling on heterogeneous multicore architectures. InProceedings of the 15th International Euro-Par Confer- ence. Springer, 863–874
2009
-
[4]
CrewAI, Inc. 2024. CrewAI: Framework for orchestrating role-playing, au- tonomous AI agents. https://github.com/crewAIInc/crewAI
2024
-
[5]
Chris Cummins, Volker Seber, Hugh Leather, et al. 2023. Large language models for compiler optimization.arXiv preprint arXiv:2309.07062(2023)
arXiv 2023
-
[6]
DeepSeek-AI. 2025. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning.arXiv preprint arXiv:2501.12948(2025)
arXiv 2025
-
[7]
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Ima- geNet: A large-scale hierarchical image database. InIEEE Conference on Computer Vision and Pattern Recognition. 248–255
2009
-
[8]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding.arXiv preprint arXiv:1810.04805(2019)
arXiv 2019
Show all 33 references
-
[9]
Aaron Grattafiori, Abhimanyu Dubey, et al. 2024. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783(2024). 10
2024 arXiv
-
[10]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 770–778
2016
-
[11]
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with GPUs.IEEE Transactions on Big Data7, 3 (2019), 535–547
2019
-
[12]
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Princ...
2023
-
[13]
LangChain, Inc. 2024. LangGraph: Build stateful, multi-actor applications with LLMs. https://github.com/langchain-ai/langgraph
2024
-
[14]
Jianping Lao, Yibo Wang, Yufei Li, Jianping Wang, Yunjia Zhang, et al . 2024. GPTuner: A manual-reading database tuning system via GPT-guided Bayesian optimization. InProceedings of the VLDB Endowment, Vol. 17
2024
-
[15]
Minghao Li, Tengchao Lv, Jingye Chen, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang, Zhoujun Li, and Furu Wei. 2023. TrOCR: Transformer-based optical char- acter recognition with pre-trained models.Proceedings of the AAAI Conference on Artificial Intelligence37, 11 (2023), 13...
2023
-
[16]
Peilong Li, Yan Luo, Ning Zhang, and Yu Cao. 2015. HeteroSpark: A GPU- accelerated heterogeneous spark cluster for machine learning.arXiv preprint arXiv:1507.03339(2015)
2015 arXiv
-
[17]
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. AWQ: Activation-aware weight quantization for LLM compression and acceleration. Proceedings of Machine Learning and Systems6 (2024), 87–100
2024
-
[18]
Azalia Mirhoseini, Anna Goldie, Hieu Pham, Benoit Steiner, Quoc V Le, and Jeff Dean. 2018. A hierarchical model for device placement. InInternational Conference on Learning Representations
2018
-
[19]
Azalia Mirhoseini, Hieu Pham, Quoc V Le, Benoit Steiner, Rasmus Larsen, Yuefeng Zhou, Sanjay Kumar, Mohammad Norouzi, Samy Bengio, and Jeff Dean. 2017. Device placement optimization with reinforcement learning. InProceedings of the 34th International Conference on Machine Lear...
2017
-
[20]
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset.arXiv preprint arXiv:1611.09268(2016)
2016 arXiv
-
[21]
Aditya Paliwal, Felix Gimeno, Vinod Nair, Yujia Li, Miles Luber, Petros Alexopou- los, and Jure Leskovec. 2020. REGAL: Transfer learning for fast optimization of computation graphs. InProceedings of the 3rd MLSys Conference
2020
-
[22]
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Lib- rispeech: An ASR corpus based on public domain audio books.IEEE International Conference on Acoustics, Speech and Signal Processing(2015), 5206–5210
2015
-
[23]
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. InInternational Conference on Machine Learnin...
2021
-
[24]
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. InInternational Conference on Machine Learning. 28492–28518
2023
-
[25]
Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. 3982–3992
2019
-
[26]
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Dis- tilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.arXiv preprint arXiv:1910.01108(2019)
2019 arXiv
-
[27]
Richard Socher, Alex Peres, Christopher D Potts, and Christopher D Manning
-
[28]
Haluk Topcuoglu, Salim Hariri, and Min-You Wu. 2002. Performance-effective and low-complexity task scheduling for heterogeneous computing.IEEE Transactions on Parallel and Distributed Systems13, 3 (2002), 260–274
2002
-
[29]
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou
-
[30]
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. InProceedings of the 2018 Conference of the North American Chapter of the Association for Com- putational Linguistics. 1112–1122
2018
-
[31]
LLM only
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. 2024. AutoGen: Enabling next-gen LLM applications via multi-agent conversation. InICLR 2024 Workshop on LLM Agents. 11 Supplementary Material A S13...
2024
-
[2013]
InProceedings of the 2013 Conference on Empirical Methods in Natural Language Processing
Recursive deep models for semantic compositionality over a sentiment treebank. InProceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. 1631–1642
2013
-
[2020]
InAdvances in Neural Information Processing Systems, Vol
MiniLM: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. InAdvances in Neural Information Processing Systems, Vol. 33. 5776–5788
Reviewed August 1, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.