{"id":"bb86eeb0-efa7-4ed3-95f9-d5080964d53e","arxiv_id":"2506.13730","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"BanditWare uses a decaying epsilon-greedy contextual bandit with linear runtime models to recommend hardware for scientific workflows, learning online with far fewer samples than offline ML approaches.","lead":"BanditWare is an online system that uses a contextual multi-armed bandit algorithm to recommend hardware settings for scientific workflows, learning from each run instead of requiring large historical datasets. It was tested on three applications, including an agricultural model, a fire science platform, and matrix multiplication, and is aimed at simplifying hardware choices on the National Data Platform.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central data-efficiency claim is only demonstrated in a favorable regime: full real-dataset experiments hover near random, and the 0.8 accuracy comes from a post-hoc truncated subset.","rationale":"The reader correctly flags the linear-model assumption, and the paper's real-data sections actually test it: BP3D all-feature linear fits have mean R2≈0.13 and full-data top-1 accuracy equal to random, while matmul on the full dataset reaches only 0.3 accuracy versus 0.2 random. The near-0.8 accuracy appears only after truncating to size≥5000, a post-hoc favorable regime. This is not an internal inconsistency—the paper is honest about these limitations—but it means the headline claim of efficient, accurate online hardware recommendation is not yet verified by the presented evidence. A full-dataset matmul run provides a decisive check: if accuracy remains near random, the truncated subset cannot carry the central claim. I would not reject the framework outright; the synthetic Cycles experiment and the truncated matmul result show the method can work when its assumptions hold. However, the current evidence is insufficient for the general claim, so the verdict should move from CONDITIONAL to UNVERDICTED pending the full-dataset evaluation.","tokens_in":11021,"tokens_out":8898,"duration_ms":101756,"concrete_test":"Run Algorithm 1 on the complete 2520-run matmul dataset (all sizes, not truncated) with size as the only feature, and at round 50 report top-1 hardware recommendation accuracy and average runtime regret relative to the full-data best-hardware oracle and to random choice. If top-1 accuracy remains near 0.3, the subset result cannot support the central few-round learning claim.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"Algorithm 1's success depends on the §3.2 linear runtime model R(H_i,x)=w_i^T x+b_i and on hardware configurations having learnable, meaningful differences. The paper's own real-data evaluations show this condition fails in the full datasets. For BP3D, 100 linear models trained on 25 samples with all features have mean R2≈0.13, and the full-data model's top-1 hardware accuracy is 34.2%, equal to random (Section 4.2). For matmul, BanditWare on the complete dataset reaches only about 0.3 accuracy versus 0.2 random; the reported near-0.8 accuracy is obtained only after truncating to size≥5000 (Section 4.3). The synthetic Cycles experiment is constructed with linear hardware trade-offs (num_tasks as single feature), so it cannot independently validate the general claim. The paper is transparent about BP3D homogeneity and short-run limitations, but these acknowledgements concede that the central claim—learning accurate hardware recommendations online within a few dozen rounds—has not been demonstrated on an unselected real workload. The load-bearing evidence is therefore a favorable regime, not a robust result.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes BanditWare, an online hardware recommendation system based on a decaying contextual epsilon-greedy bandit with per-hardware linear models. The algorithm estimates runtime on each hardware configuration as R(Hi,x)=w_i^T x + b_i, selects a hardware using a tolerant greedy rule with occasional random exploration, observes the actual runtime, refits ordinary least squares models per hardware, and decays the exploration rate. Evaluations are reported on three workloads: Cycles on four synthetic hardware settings, BurnPro3D on three NDP Kubernetes configurations, and a tiled matrix multiplication application on five configurations. The headline claims are that BanditWare matches the full-data RMSE with about 20 samples in the Cycles experiment (98.5% fewer data points), that on BP3D it converges close to the full-data baseline within 25 rounds, and that on matrix multiplication it reaches near-0.8 accuracy on a truncated subset with size at least 5000, versus 0.2 random accuracy.","tokens_in":11147,"tokens_out":7262,"duration_ms":72299,"significance":"If the central data-efficiency claim were supported, BanditWare would be a useful lightweight alternative to offline ML for hardware recommendation in platforms like NDP that lack large historical traces. The paper is clearly written and transparent about several limitations, including the homogeneity of the BP3D hardware and the difficulty of recommending hardware for short runtimes. It also describes Algorithm 1 concretely enough to be reimplemented. However, the evidence for the headline claim is concentrated in a constructed linear regime: the synthetic Cycles experiment is engineered to satisfy the linear assumption, and on the unselected real workloads the learned accuracy is at or near random (BP3D full-fit accuracy 34.2%, matmul full-dataset accuracy about 0.3 versus 0.2 random). The paper is better described as a proof-of-concept under favorable assumptions than as a demonstrated solution for real heterogeneous hardware.","major_comments":[{"comment":"The central modeling assumption is the per-hardware linear runtime model R(Hi,x)=w_i^T x + b_i. The paper's own BP3D results show this assumption is not adequate on the real workload: with all features, the full-data model's top-1 hardware accuracy is 34.2%, which the authors state equals random choice, and 100 linear models trained on 25 samples have mean R2 of 12.83% (Section 4.2, Figure 5). Since hardware selection accuracy is the objective, the system is not learning accurate recommendations on this unselected workload; the claim that \"Algorithm 1 is able to learn a good model\" (Section 4.2) is therefore not supported by the evidence. The only experiment where the linear assumption clearly holds is the synthetic Cycles setting of Section 4.1, where num_tasks is the sole feature and the hardware trade-offs are constructed.","section":"§3.2, Eq. (1); §4.2"},{"comment":"The headline data-efficiency result—\"same error rate as using 1316 data points but with only 20 samples (98.5% fewer)\"—is not generalizable as stated. The Cycles experiment uses four synthetic hardware settings whose runtimes are exactly linear in num_tasks, so a one-feature linear model per hardware needs only a handful of points; this is a favorable regime constructed to satisfy Eq. (1), not a demonstration on a real workload. Moreover, the \"1316 data points\" baseline appears to belong to the BP3D dataset in Section 4.2, not to the Cycles dataset of 80 runs described in Section 4.1; the paper should clarify this discrepancy and report per-hardware sample counts, since Algorithm 1 stores data separately per arm.","section":"§4.1, Fig. 4a"},{"comment":"The text states that the average RMSE \"converges to match the RMSE of all 1316 samples\" after about 25 rounds, but the immediately following numbers give an RMSE of 20182.91±12290.82 at round 25 versus a full-fit baseline of 12257.43, i.e., 17.90% worse on average, with error bars as large as the baseline value. \"Match\" and \"17.90% worse\" are inconsistent, and the large variance means the convergence claim is not established. The conclusion's phrasing (\"performs only 17.90% worse than the theoretically best possible\") is more honest but should replace the convergence language in Section 4.2.","section":"§4.2, Fig. 7"},{"comment":"The near-0.8 accuracy on matrix multiplication is obtained only after truncating to size>=5000; on the complete dataset the accuracy is about 0.3, only slightly above the 0.2 random baseline (Figure 9a). The truncation is post-hoc and selects the regime where runtimes are long and the linear size feature is most informative. This does not support the general claim that BanditWare learns accurate hardware recommendations online. The full-dataset result should be reported as the primary outcome, with the truncated subset clearly labeled as a separate secondary analysis justified by an a priori criterion.","section":"§4.3, Figs. 9-10"},{"comment":"The paper never defines the \"accuracy\" metric used throughout Section 4. It appears to be the fraction of rounds in which the bandit selects the hardware chosen by the full-data model (or the true fastest hardware), but the definition is not stated. Since accuracy is the main reported performance measure in Figures 4, 7, 9-12, the missing definition makes the results non-reproducible and should be added, together with a description of how ties and tolerance-based selections are counted.","section":"§4 (all experiments)"},{"comment":"The evaluation compares BanditWare only to offline linear regression baselines and random accuracy; there is no comparison to standard online bandit algorithms such as LinUCB or Thompson sampling with linear rewards. Because the paper's contribution is a bandit framework, this omission leaves open whether the decaying contextual epsilon-greedy strategy is competitive with existing methods. Adding at least one standard bandit baseline would substantially strengthen the central claim.","section":"§4"}],"minor_comments":[{"comment":"With alpha=0.99 and epsilon0=1, the exploration probability after 100 rounds is about 0.37, so the system continues to explore heavily throughout the reported horizon; the authors should comment on this choice and report sensitivity to alpha and epsilon0.","section":"Algorithm 1, step 12"},{"comment":"The x-axis label \"Hardware, mode\" and the caption text appear garbled, and the legend mixing \"Predicted\" and \"Actual\" is confusing; please clean up the figure and caption.","section":"§4.1, Figure 3"},{"comment":"The x-axis label is repeated (\"area area area\"), and the figure does not show error bars or confidence regions despite the reported variation across simulations.","section":"§4.2, Figure 6"},{"comment":"The phrase \"most resource efficiency\" is not formally defined; the paper should specify the resource-efficiency objective (e.g., fewest CPUs, least memory, or a weighted combination) used in the experiments.","section":"Algorithm 1, tolerant selection"},{"comment":"The paper does not state whether code and data are available; given the emphasis on reproducibility, a data/code availability statement would help.","section":"General"},{"comment":"Reference [24] is cited as prior work showing that BP3D runtime can be linearly modeled, but the specific result is not located; please give the relevant section or page.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the venue's scope as an applied systems/CI contribution, but the contribution is incremental: the algorithm is a standard decaying epsilon-greedy contextual bandit with per-arm linear regression. The main value is the application to NDP workflows and the transparent reporting of limitations. I would not recommend rejection because the authors could address the load-bearing concerns with a sharper framing, a consistent baseline definition, and additional experiments on more heterogeneous hardware; however, the current wording overclaims on the strength of the synthetic and truncated results."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: BanditWare is a clean, honest engineering paper. The new bit is the specific combination — a decaying contextual epsilon-greedy bandit with per-hardware linear models and a tolerance knob — applied to NDP workflow hardware recommendation. The tolerance mechanism is a genuinely practical idea: trade a little runtime for better resource efficiency, and it gets a fair test here.\n\nThe headline claim, though, only holds in a favorable regime. The 98.5% fewer data points result comes from Experiment 1 on synthetic hardware, where runtime is linear in one feature (num_tasks) by construction. That is not a weakness in itself — the paper says so — but it means the result does not transfer to real systems. On the real BP3D data, the full-data accuracy is 34.2%, which is random, and the paper admits the hardware settings are too similar. On matmul, the full dataset gives about 0.3 accuracy versus 0.2 random; the near-0.8 accuracy appears only after truncating to size ≥ 5000 and sometimes adding tolerance. These are post-hoc selections, not pre-registered analyses.\n\nThe linear model is load-bearing but fragile. The paper claims low sample complexity because it refits separate OLS models per arm; that is really just linear regression, not a sophisticated bandit. There is no comparison against LinUCB or even a plain UCB, so we do not know if the decaying epsilon-greedy choice matters. The paper also does not ship code or data, which makes the results hard to check.\n\nWhat the paper does well: it is transparent about the BP3D homogeneity, it explains the tolerance trade-off clearly, and it does not oversell the synthetic result. The writing is direct and the related work is adequate. The authors seem to know where the weak spots are.\n\nWho this is for: people building lightweight online recommenders for heterogeneous clusters. It is not a theoretical contribution, and it is not yet a validated systems contribution either.\n\nRecommendation: this deserves a serious referee. The problem is real, the idea is reasonable, and the authors are honest. The revision needs a larger real-hardware evaluation with pre-registered metrics, a LinUCB baseline, and published code and data. With those, the paper could be solid. Without them, the central claim stays unproven.","headline":"Honest engineering paper whose central data-efficiency claim only holds in a synthetic regime; real-data results hover near random, but the tolerance mechanism and clear writing make it worth a serious referee.","tokens_in":11783,"tokens_out":1643,"would_cite":false,"duration_ms":19741,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A decaying contextual epsilon-greedy bandit with per-hardware linear models can recommend the right hardware within about 25 rounds.","keywords":["contextual multi-armed bandit","decaying epsilon-greedy","hardware recommendation","runtime prediction","scientific workflows","online learning","resource allocation","linear regression"],"falsifier":"Run Algorithm 1 on a workflow whose runtime is deliberately nonlinear in its input features (for example, matrix multiplication that crosses memory-bandwidth or cache thresholds), and check whether per-hardware held-out $R^2$ and recommendation accuracy stay near random over 100 rounds; if they do, the linear per-hardware model is the failure point.","tokens_in":10725,"feed_emoji":"🎰","tokens_out":9477,"duration_ms":92611,"temperature":0.7,"pith_summary":"BanditWare is an online recommendation system that picks which hardware setting should run a scientific workflow, using a decaying contextual epsilon-greedy bandit—a decision rule that balances random exploration against exploiting what it has learned—instead of a large pre-collected training set. The paper's central claim is that this lightweight algorithm can learn useful per-hardware runtime models within a few dozen rounds: on a synthetic-hardware experiment it reaches the RMSE of a 1316-sample regression with about 20 samples, and on the BP3D workflow it converges toward the full-data fit after roughly 25 rounds. The authors argue this matters because real distributed platforms host heterogeneous hardware while historical run data is scarce, so a system that improves with every executed run can help non-expert users allocate resources efficiently. The same algorithm can trade a user-controlled runtime slowdown for more resource-efficient hardware through tolerance parameters.","feed_headline":"Bandit framework learns to pick hardware in ~25 rounds","feed_subtitle":"It matches full-data runtime accuracy with a tiny fraction of samples and adapts as workloads arrive.","key_machinery":"The load-bearing object is Algorithm 1, named the ``Decaying Contextual $\\epsilon$-Greedy Strategy with Tolerant Selection.'' For each hardware option $H_i$, the algorithm maintains a linear model of runtime whose coefficients are refit by ordinary least squares on that hardware's observed runs. At each round it explores uniformly with probability $\\epsilon$ and otherwise exploits: it computes the estimated runtime on every hardware, identifies the predicted fastest option, and then chooses the most resource-efficient hardware whose estimated runtime is within $(1+t_r)\\hat{R}_{\\text{fastest}}+t_s$. The exploration rate starts at $\\epsilon_0=1$ and decays by a factor $\\alpha=0.99$ after each round, and all three experiments use these settings.","core_discovery":"On its own terms, the paper claims that a decaying contextual $\\epsilon$-greedy bandit, with one linear model $\\hat{R}(H_i,x)=w_i^Tx+b_i$ per hardware option refit by least squares on the runs observed so far, is sufficient for online hardware recommendation. In the Cycles experiment, matching the full-model RMSE took about 20 samples instead of 1316, about 98.5% fewer data points. In the BP3D experiment, the framework approached the full-data RMSE after roughly 25 rounds, with an average RMSE 17.90% worse than the full 1316-sample baseline at round 25. In matrix multiplication with matrix sizes at least 5000, bandit recommendation accuracy reached nearly 0.8 versus 0.2 for random choice among five hardware options. The paper notes that when the hardware options have nearly identical runtimes, full-fit accuracy can fall to random level, as it does on BP3D, and it attributes this to hardware homogeneity rather than to the algorithm.","pith_inferences":["Not tested here: a short, deliberately chosen probe set at startup could shrink the 20--25 round warm-up, since the paper's uniform random exploration is a simple way to gather initial data but not necessarily the fastest.","The tolerance rule can be read as an implicit multi-objective trade between runtime and resource use; making that trade explicit, for instance with a Pareto-front or cost-aware objective, could yield regret guarantees rather than only empirical convergence.","Because the weakest component is the per-hardware linear model, replacing the least-squares refit with local or kernelized regression could extend the same bandit loop to nonlinear runtimes, and the RMSE-over-rounds plots would immediately show whether the extension helps."],"forward_implications":["Hardware recommendations for scientific workflows can be produced online with no large offline training set; a few dozen executed runs are enough for the per-hardware linear models to stabilize.","For workflows whose runtime varies smoothly and linearly with input features, the bandit approaches the accuracy of a regression trained on the full historical dataset.","Tolerance parameters turn the recommender into a resource-efficiency tool, exchanging a small, user-set slowdown for less resource-intensive hardware when the fastest choice is not much better than cheaper ones.","When available hardware runs the workflow at nearly the same speed, the approach will not separate the options and behaves like random selection."],"supporting_citations":[{"why":"Supplies the BP3D simulation dataset and the prior result that its runtime is linearly predictable from input features, the basis for Experiment 2.","marker":"[24]"},{"why":"Provides the Cycles workflow runs used in Experiment 1 to test the algorithm on synthetic hardware settings.","marker":"[29]"},{"why":"Formalizes multi-armed bandit learning over time under uncertainty, which Algorithm 1 instantiates in contextual form.","marker":"[25]"},{"why":"Defines the epsilon-greedy exploration-exploitation strategy that the decaying contextual variant extends.","marker":"[27]"},{"why":"Surveys contextual multi-armed bandit algorithms, grounding the choice of a contextual model for hardware-aware recommendations.","marker":"[28]"}],"fun_headline_variants":["Bandit matches full-data hardware picks with 98.5% less data","Online bandit learns hardware choice in ~25 rounds","Contextual bandit recommends hardware from ~25 runs","Hardware prediction cut to ~25 rounds by adaptive bandit"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that each hardware setting's runtime is well approximated by a straight-line function of the workflow's input features, refit on the few dozen runs collected so far.","fun_headline_variants_meta":{"raw":{"variants":["Bandit matches full-data hardware picks with 98.5% less data","Online bandit learns hardware choice in ~25 rounds","Contextual bandit recommends hardware from ~25 runs","Hardware prediction cut to ~25 rounds by adaptive bandit"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000398,"raw_usage":{"total_tokens":2080,"prompt_tokens":938,"completion_tokens":1142,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":554,"completion_tokens_details":{"reasoning_tokens":1072}},"tokens_in":554,"tokens_out":1142,"duration_ms":11401,"temperature":1.0,"reasoning_tokens":1072,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:25:45.033708+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Algorithm 1 on a workflow whose runtime is deliberately nonlinear in its input features (for example, matrix multiplication that crosses memory-bandwidth or cache thresholds), and check whether per-hardware held-out $R^2$ and recommendation accuracy stay near random over 100 rounds; if they do, the linear per-hardware model is the failure point.","supporting_citations":[{"cited_title":"Towards an integrated performance framework for fire science and management workflows,","cited_arxiv_id":null,"evidence_quote":"Supplies the BP3D simulation dataset and the prior result that its runtime is linearly predictable from input features, the basis for Experiment 2."},{"cited_title":"Empowering agroecosystem modeling with htc scientific workflows: The cycles model use case,","cited_arxiv_id":null,"evidence_quote":"Provides the Cycles workflow runs used in Experiment 1 to test the algorithm on synthetic hardware settings."},{"cited_title":"Introduction to multi-armed bandits,","cited_arxiv_id":null,"evidence_quote":"Formalizes multi-armed bandit learning over time under uncertainty, which Algorithm 1 instantiates in contextual form."},{"cited_title":"Survey on applications of multi-armed and contextual bandits,","cited_arxiv_id":null,"evidence_quote":"Defines the epsilon-greedy exploration-exploitation strategy that the decaying contextual variant extends."}],"review_version":1}