REVIEW 6 major objections 6 minor 30 references
BanditWare: A Contextual Bandit-based Framework for Hardware Prediction
T0 review · 6 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read A decaying contextual epsilon-greedy bandit with per-hardware linear models can recommend the right hardware within about 25 rounds.
desk verdict Honest engineering paper whose central data-efficiency claim only holds in a synthetic regime; real-data results hover near random, but the tolerance mechanism and clear writing make it worth a serious referee. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is Algorithm 1, named the ``Decaying Contextual $\epsilon$-Greedy Strategy with Tolerant Selection.'' For each hardware option $H_i$, the algorithm maintains a linear model of runtime whose coefficients are refit by ordinary least squares on that hardware's observed runs. At each round it explores uniformly with probability $\epsilon$ and otherwise exploits: it computes the estimated runtime on every hardware, identifies the predicted fastest option, and then chooses the most resource-efficient hardware whose estimated runtime is within $(1+t_r)\hat{R}_{\text{fastest}}+t_s$. The exploration rate starts at $\epsilon_0=1$ and decays by a factor $\alpha=0.99$ after each round, and all three experiments use these settings.
What would settle it
Run Algorithm 1 on a workflow whose runtime is deliberately nonlinear in its input features (for example, matrix multiplication that crosses memory-bandwidth or cache thresholds), and check whether per-hardware held-out $R^2$ and recommendation accuracy stay near random over 100 rounds; if they do, the linear per-hardware model is the failure point.
Extended reading notes
Core claim
On its own terms, the paper claims that a decaying contextual $\epsilon$-greedy bandit, with one linear model $\hat{R}(H_i,x)=w_i^Tx+b_i$ per hardware option refit by least squares on the runs observed so far, is sufficient for online hardware recommendation. In the Cycles experiment, matching the full-model RMSE took about 20 samples instead of 1316, about 98.5% fewer data points. In the BP3D experiment, the framework approached the full-data RMSE after roughly 25 rounds, with an average RMSE 17.90% worse than the full 1316-sample baseline at round 25. In matrix multiplication with matrix sizes at least 5000, bandit recommendation accuracy reached nearly 0.8 versus 0.2 for random choice among five hardware options. The paper notes that when the hardware options have nearly identical runtimes, full-fit accuracy can fall to random level, as it does on BP3D, and it attributes this to hardware homogeneity rather than to the algorithm.
Load-bearing premise
The load-bearing premise is that each hardware setting's runtime is well approximated by a straight-line function of the workflow's input features, refit on the few dozen runs collected so far.
Editorial extensions
If this is right
- Hardware recommendations for scientific workflows can be produced online with no large offline training set; a few dozen executed runs are enough for the per-hardware linear models to stabilize.
- For workflows whose runtime varies smoothly and linearly with input features, the bandit approaches the accuracy of a regression trained on the full historical dataset.
- Tolerance parameters turn the recommender into a resource-efficiency tool, exchanging a small, user-set slowdown for less resource-intensive hardware when the fastest choice is not much better than cheaper ones.
- When available hardware runs the workflow at nearly the same speed, the approach will not separate the options and behaves like random selection.
Reading between the lines
- Not tested here: a short, deliberately chosen probe set at startup could shrink the 20--25 round warm-up, since the paper's uniform random exploration is a simple way to gather initial data but not necessarily the fastest.
- The tolerance rule can be read as an implicit multi-objective trade between runtime and resource use; making that trade explicit, for instance with a Pareto-front or cost-aware objective, could yield regret guarantees rather than only empirical convergence.
- Because the weakest component is the per-hardware linear model, replacing the least-squares refit with local or kernelized regression could extend the same bandit loop to nonlinear runtimes, and the RMSE-over-rounds plots would immediately show whether the extension helps.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes BanditWare, an online hardware recommendation system based on a decaying contextual epsilon-greedy bandit with per-hardware linear models. The algorithm estimates runtime on each hardware configuration as R(Hi,x)=w_i^T x + b_i, selects a hardware using a tolerant greedy rule with occasional random exploration, observes the actual runtime, refits ordinary least squares models per hardware, and decays the exploration rate. Evaluations are reported on three workloads: Cycles on four synthetic hardware settings, BurnPro3D on three NDP Kubernetes configurations, and a tiled matrix multiplication application on five configurations. The headline claims are that BanditWare matches the full-data RMSE with about 20 samples in the Cycles experiment (98.5% fewer data points), that on BP3D it converges close to the full-data baseline within 25 rounds, and that on matrix multiplication it reaches near-0.8 accuracy on a truncated subset with size at least 5000, versus 0.2 random accuracy.
Significance. If the central data-efficiency claim were supported, BanditWare would be a useful lightweight alternative to offline ML for hardware recommendation in platforms like NDP that lack large historical traces. The paper is clearly written and transparent about several limitations, including the homogeneity of the BP3D hardware and the difficulty of recommending hardware for short runtimes. It also describes Algorithm 1 concretely enough to be reimplemented. However, the evidence for the headline claim is concentrated in a constructed linear regime: the synthetic Cycles experiment is engineered to satisfy the linear assumption, and on the unselected real workloads the learned accuracy is at or near random (BP3D full-fit accuracy 34.2%, matmul full-dataset accuracy about 0.3 versus 0.2 random). The paper is better described as a proof-of-concept under favorable assumptions than as a demonstrated solution for real heterogeneous hardware.
major comments (6)
- [§3.2, Eq. (1); §4.2] The central modeling assumption is the per-hardware linear runtime model R(Hi,x)=w_i^T x + b_i. The paper's own BP3D results show this assumption is not adequate on the real workload: with all features, the full-data model's top-1 hardware accuracy is 34.2%, which the authors state equals random choice, and 100 linear models trained on 25 samples have mean R2 of 12.83% (Section 4.2, Figure 5). Since hardware selection accuracy is the objective, the system is not learning accurate recommendations on this unselected workload; the claim that "Algorithm 1 is able to learn a good model" (Section 4.2) is therefore not supported by the evidence. The only experiment where the linear assumption clearly holds is the synthetic Cycles setting of Section 4.1, where num_tasks is the sole feature and the hardware trade-offs are constructed.
- [§4.1, Fig. 4a] The headline data-efficiency result—"same error rate as using 1316 data points but with only 20 samples (98.5% fewer)"—is not generalizable as stated. The Cycles experiment uses four synthetic hardware settings whose runtimes are exactly linear in num_tasks, so a one-feature linear model per hardware needs only a handful of points; this is a favorable regime constructed to satisfy Eq. (1), not a demonstration on a real workload. Moreover, the "1316 data points" baseline appears to belong to the BP3D dataset in Section 4.2, not to the Cycles dataset of 80 runs described in Section 4.1; the paper should clarify this discrepancy and report per-hardware sample counts, since Algorithm 1 stores data separately per arm.
- [§4.2, Fig. 7] The text states that the average RMSE "converges to match the RMSE of all 1316 samples" after about 25 rounds, but the immediately following numbers give an RMSE of 20182.91±12290.82 at round 25 versus a full-fit baseline of 12257.43, i.e., 17.90% worse on average, with error bars as large as the baseline value. "Match" and "17.90% worse" are inconsistent, and the large variance means the convergence claim is not established. The conclusion's phrasing ("performs only 17.90% worse than the theoretically best possible") is more honest but should replace the convergence language in Section 4.2.
- [§4.3, Figs. 9-10] The near-0.8 accuracy on matrix multiplication is obtained only after truncating to size>=5000; on the complete dataset the accuracy is about 0.3, only slightly above the 0.2 random baseline (Figure 9a). The truncation is post-hoc and selects the regime where runtimes are long and the linear size feature is most informative. This does not support the general claim that BanditWare learns accurate hardware recommendations online. The full-dataset result should be reported as the primary outcome, with the truncated subset clearly labeled as a separate secondary analysis justified by an a priori criterion.
- [§4 (all experiments)] The paper never defines the "accuracy" metric used throughout Section 4. It appears to be the fraction of rounds in which the bandit selects the hardware chosen by the full-data model (or the true fastest hardware), but the definition is not stated. Since accuracy is the main reported performance measure in Figures 4, 7, 9-12, the missing definition makes the results non-reproducible and should be added, together with a description of how ties and tolerance-based selections are counted.
- [§4] The evaluation compares BanditWare only to offline linear regression baselines and random accuracy; there is no comparison to standard online bandit algorithms such as LinUCB or Thompson sampling with linear rewards. Because the paper's contribution is a bandit framework, this omission leaves open whether the decaying contextual epsilon-greedy strategy is competitive with existing methods. Adding at least one standard bandit baseline would substantially strengthen the central claim.
minor comments (6)
- [Algorithm 1, step 12] With alpha=0.99 and epsilon0=1, the exploration probability after 100 rounds is about 0.37, so the system continues to explore heavily throughout the reported horizon; the authors should comment on this choice and report sensitivity to alpha and epsilon0.
- [§4.1, Figure 3] The x-axis label "Hardware, mode" and the caption text appear garbled, and the legend mixing "Predicted" and "Actual" is confusing; please clean up the figure and caption.
- [§4.2, Figure 6] The x-axis label is repeated ("area area area"), and the figure does not show error bars or confidence regions despite the reported variation across simulations.
- [Algorithm 1, tolerant selection] The phrase "most resource efficiency" is not formally defined; the paper should specify the resource-efficiency objective (e.g., fewest CPUs, least memory, or a weighted combination) used in the experiments.
- [General] The paper does not state whether code and data are available; given the emphasis on reproducibility, a data/code availability statement would help.
- [References] Reference [24] is cited as prior work showing that BP3D runtime can be linearly modeled, but the specific result is not located; please give the relevant section or page.
Circularity Check
No significant circularity: the online-versus-batch linear-model comparisons are independently evaluated, though the strongest data-efficiency demo is run in a constructed linear regime and one BP3D linearity premise relies on same-author prior work.
full rationale
Algorithm 1 is an explicitly stated decaying contextual epsilon-greedy bandit with per-hardware linear models refit by least squares; its evaluation compares models trained on progressively collected data against full-data batch linear fits. That comparison is a genuine statistical question and is not equivalent to the algorithm's inputs by construction. Experiment 1 is a synthetic, single-feature (num_tasks) regime that the paper itself qualifies with "when the runtime can be predicted as a linear combination of input variables and the hardware configurations present a meaningful trade-off," so the roughly 20-sample result is a sanity check for the linear model class rather than independent evidence of general data efficiency; this is a scope limitation, not circularity. The only circularity-adjacent passage is Section 4.2's assertion "We know from prior work [24] that the BP3D workflow runtime can be accurately modeled as a linear combination of the input features," where reference [24] is a same-author citation and the paper's own Figure 5 reports average R2 of 12.83% for 25-sample all-feature linear models. Because the linear assumption is stated openly in Section 3.2 as a modeling assumption, and the online-versus-batch RMSE comparison is computed from the paper's own data, this self-citation is not load-bearing for the framework's central derivation; it is a support-quality weakness, not a circular reduction. Overall, no equation or fitted parameter is renamed as a prediction, and no result is forced by definition or by a self-citation chain.
Assumptions & free parameters
free parameters (5)
- decay factor alpha =
0.99
- initial exploration rate epsilon0 =
1
- tolerance_ratio =
0 (implicit) or 0.05 in Figure 12
- tolerance_seconds =
0 (implicit) or 20 in Figure 11
- linear coefficients w_i and bias b_i per hardware =
fitted by least squares at each round
assumptions (3)
- domain assumption Runtime on each hardware is linear in the workflow feature vector.
- domain assumption Past runs on a hardware configuration are sufficient to predict future runs on the same configuration.
- domain assumption Execution time is the only performance objective unless tolerance parameters are set.
Cite this review
Pith. "Pith review of BanditWare: A Contextual Bandit-based Framework for Hardware Prediction." pith.science (2026). https://pith.science/paper/FHMYO3EA
@misc{pith2026250613730,
author = {Pith},
title = {Pith review of: BanditWare: A Contextual Bandit-based Framework for Hardware Prediction},
year = {2026},
howpublished = {\url{https://pith.science/paper/FHMYO3EA}},
note = {Machine review of arXiv:2506.13730}
}
read the original abstract
Distributed computing systems are essential for meeting the demands of modern applications, yet transitioning from single-system to distributed environments presents significant challenges. Misallocating resources in shared systems can lead to resource contention, system instability, degraded performance, priority inversion, inefficient utilization, increased latency, and environmental impact. We present BanditWare, an online recommendation system that dynamically selects the most suitable hardware for applications using a contextual multi-armed bandit algorithm. BanditWare balances exploration and exploitation, gradually refining its hardware recommendations based on observed application performance while continuing to explore potentially better options. Unlike traditional statistical and machine learning approaches that rely heavily on large historical datasets, BanditWare operates online, learning and adapting in real-time as new workloads arrive. We evaluated BanditWare on three workflow applications: Cycles (an agricultural science scientific workflow) BurnPro3D (a web-based platform for fire science) and a matrix multiplication application. Designed for seamless integration with the National Data Platform (NDP), BanditWare enables users of all experience levels to optimize resource allocation efficiently.
Figures
Figures from the paper (9 more)
Reference graph
Works this paper leans on
-
[1]
Intelligent resource prediction for hpc and scientific workflows,
B. Shealy, “Intelligent resource prediction for hpc and scientific workflows,” 2021
work page 2021
-
[2]
Cloud computing virtualization of resources allocation for distributed systems,
H. Shukur, S. Zeebaree, R. Zebari, D. Zeebaree, O. Ahmed, and A. Salih, “Cloud computing virtualization of resources allocation for distributed systems,”Journal of Applied Science and Technology Trends, vol. 1, no. 2, pp. 98–105, 2020
work page 2020
-
[3]
J. S. Vetter, R. Brightwell, M. Gokhale, P. McCormick, R. Ross, J. Shalf, K. Antypas, D. Donofrio, T. Humble, C. Schuman, B. Van Essen, S. Yoo, A. Aiken, D. Bernholdt, S. Byna, K. Cameron, F. Cappello, B. Chapman, A. Chien, M. Hall, R. Hartman-Baker, Z. Lan, M. Lang, J. Leidel, S. Li, R. Lucas, J. Mellor-Crummey, P. Peltz Jr., T. Peterka, M. Strout, and J...
-
[4]
A comprehensive evaluation of novel ai accelerators for deep learning workloads,
M. Emani, Z. Xie, S. Raskar, V. Sastry, W. Arnold, B. Wilson, R. Thakur, V. Vish- wanath, Z. Liu, M. E. Papka, C. O. Bohorquez, R. Weisner, K. Li, Y. Sheng, Y. Du, J. Zhang, A. Tsyplikhin, G. Khaira, J. Fowers, R. Sivakumar, V. Godsoe, A. Ma- cias, C. Tekur, and M. Boyd, “A comprehensive evaluation of novel ai accelerators for deep learning workloads,” in...
work page 2022
- [5]
-
[6]
Empowering agroecosystem modeling with htc scientific workflows: The cycles model use case,
R. F. Da Silva, R. Mayani, Y. Shi, A. R. Kemanian, M. Rynge, and E. Deelman, “Empowering agroecosystem modeling with htc scientific workflows: The cycles model use case,” in2019 IEEE International Conference on Big Data (Big Data). IEEE, 2019, pp. 4545–4552. Title Suppressed Due to Excessive Length 17
work page 2019
-
[7]
BurnPro3D: A Platform for Prescribed Fire Planning and Optimization,
“BurnPro3D: A Platform for Prescribed Fire Planning and Optimization,” 2023. [Online]. Available: https://burnpro3d.sdsc.edu/index.html
work page 2023
-
[8]
The pacific research platform: Making high-speed networking a reality for the scientist,
L. Smarr, C. Crittenden, T. DeFanti, J. Graham, D. Mishin, R. Moore, P. Papadopoulos, and F. Würthwein, “The pacific research platform: Making high-speed networking a reality for the scientist,” inProceedings of the Practice and Experience on Advanced Research Computing: Seamless Creativity, ser. PEARC ’18. New York, NY, USA: Association for Computing Mac...
arXiv 2018
Show all 30 references
-
[9]
Montage: a grid portal and software toolkit for science-grade astronomical image mosaicking,
J. C. Jacob, D. S. Katz, G. B. Berriman, J. C. Good, A. Laity, E. Deelman, C. Kesselman, G. Singh, M.-H. Su, T. Princeet al., “Montage: a grid portal and software toolkit for science-grade astronomical image mosaicking,”International Journal of Computational Science and Engine...
2009
-
[10]
Imagenet: A large- scale hierarchical image database,
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large- scale hierarchical image database,” in2009 IEEE Conference on Computer Vision and Pattern Recognition, 2009, pp. 248–255
2009
-
[11]
Gpt-4 technical report,
J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkatet al., “Gpt-4 technical report,”arXiv preprint arXiv:2303.08774, 2023
2023 arXiv
-
[12]
The workload on parallel supercomputers: modeling the characteristics of rigid jobs,
U. Lublin and D. G. Feitelson, “The workload on parallel supercomputers: modeling the characteristics of rigid jobs,”Journal of Parallel and Distributed Computing, vol. 63, no. 11, pp. 1105–1122, 2003. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S074...
2003
-
[13]
Workload modeling for performance evaluation,
D. G. Feitelson, “Workload modeling for performance evaluation,” inPerformance Evaluation of Complex Systems: Techniques and Tools, M. C. Calzarossa and S. Tucci, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2002, pp. 114–141
2002
-
[14]
A case study of data management challenges presented in large-scale machine learning workflows,
C. S. Lee, V. Hewes, G. Cerati, J. Kowalkowski, A. Aurisano, A. Agrawal, A. Choudhary, and W.-K. Liao, “A case study of data management challenges presented in large-scale machine learning workflows,” in2023 IEEE/ACM 23rd International Symposium on Cluster, Cloud and Internet ...
2023
-
[15]
Analysis of the jobs resource utilization on a production system,
J. Emeras, C. Ruiz, J.-M. Vincent, and O. Richard, “Analysis of the jobs resource utilization on a production system,” inJob Scheduling Strategies for Parallel Pro- cessing: 17th International Workshop, JSSPP 2013, Boston, MA, USA, May 24, 2013 Revised Selected Papers 17. Spri...
2013
-
[16]
Evalix: Classification and pre- diction of job resource consumption on hpc platforms,
J. Emeras, S. Varrette, M. Guzek, and P. Bouvry, “Evalix: Classification and pre- diction of job resource consumption on hpc platforms,” inJob Scheduling Strategies for Parallel Processing, N. Desai and W. Cirne, Eds. Cham: Springer International Publishing, 2017, pp. 102–122
2017
-
[17]
Managing the topology of heterogeneous cluster nodes with hardware locality (hwloc),
B. Goglin, “Managing the topology of heterogeneous cluster nodes with hardware locality (hwloc),” in2014 International Conference on High Performance Comput- ing & Simulation (HPCS). IEEE, 2014, pp. 74–81
2014
-
[18]
Predicting the performance-cost trade-off of applications across multi- ple systems,
A. Nassereldine, S. Diab, M. Baydoun, K. Leach, M. Alt, D. Milojicic, and I. El Hajj, “Predicting the performance-cost trade-off of applications across multi- ple systems,” in2023 IEEE/ACM 23rd International Symposium on Cluster, Cloud and Internet Computing (CCGrid), 2023, pp...
2023
-
[19]
On the use of machine learning to predict the time and resources consumed by applications,
A. Matsunaga and J. A. Fortes, “On the use of machine learning to predict the time and resources consumed by applications,” in2010 10th IEEE/ACM International Conference on Cluster, Cloud and Grid Computing. IEEE, 2010, pp. 495–504
2010
-
[20]
The use of machine learning algorithms in recommender systems: A systematic review,
I. Portugal, P. Alencar, and D. Cowan, “The use of machine learning algorithms in recommender systems: A systematic review,”Expert Systems with Applications, vol. 97, pp. 205–227, 2018. 18 T. Coleman et al
2018
-
[21]
Reinforcement learning based recommender systems: A survey,
M. M. Afsar, T. Crump, and B. Far, “Reinforcement learning based recommender systems: A survey,”ACM Computing Surveys, vol. 55, no. 7, pp. 1–38, 2022
2022
-
[22]
A similarity metric designed to speed up, using hardware, the recommender systems k-nearest neighbors algorithm,
J. Bobadilla, F. Ortega, A. Hernando, and G. G. de Rivera, “A similarity metric designed to speed up, using hardware, the recommender systems k-nearest neighbors algorithm,”Knowledge-Based Systems, vol. 51, pp. 27–34, 2013. [Online]. Available: https://www.sciencedirect.com/sc...
2013
-
[23]
A literature review of recommendation systems,
K. Bhareti, S. Perera, S. Jamal, M. H. Pallege, V. Akash, and S. Wiieweera, “A literature review of recommendation systems,” in2020 IEEE International Con- ference for Innovation in Technology (INOCON), 2020, pp. 1–7
2020
-
[24]
Towards an integrated performance framework for fire science and management workflows,
H. Ahmed, R. Shende, I. Perez, D. Crawl, S. Purawat, and I. Altintas, “Towards an integrated performance framework for fire science and management workflows,”
-
[25]
Introduction to multi-armed bandits,
A. Slivkinset al., “Introduction to multi-armed bandits,”Foundations and Trends®in Machine Learning, vol. 12, no. 1-2, pp. 1–286, 2019
2019
-
[26]
Marginal productivity index policies for dynamic priority allocation in restless bandit models
P. Jacko, “Marginal productivity index policies for dynamic priority allocation in restless bandit models.”
-
[27]
Survey on applications of multi-armed and contextual bandits,
D. Bouneffouf, I. Rish, and C. Aggarwal, “Survey on applications of multi-armed and contextual bandits,” in2020 IEEE Congress on Evolutionary Computation (CEC), 2020, pp. 1–8
2020
-
[28]
A survey on contextual multi-armed bandits,
L. Zhou, “A survey on contextual multi-armed bandits,”arXiv preprint arXiv:1508.03326, 2015
2015 arXiv
-
[29]
Empowering agroecosystem modeling with htc scientific workflows: The cycles model use case,
R. F. d. Silva, R. Mayani, Y. Shi, A. R. Kemanian, M. Rynge, and E. Deelman, “Empowering agroecosystem modeling with htc scientific workflows: The cycles model use case,” in2019 IEEE International Conference on Big Data (Big Data), 2019, pp. 4545–4552
2019
-
[2024]
Available: https://arxiv.org/abs/2407.21231
[Online]. Available: https://arxiv.org/abs/2407.21231
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.