{"id":"9f48e8d8-1e65-438d-9bcf-8ddc5b8ebb18","arxiv_id":"2501.01057","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A lightweight UCB-based autotuner (LASP) tuned four HPC applications on a Jetson Nano, finding configurations within 12% of the oracle and improving over defaults by 6-14%.","lead":"LASP uses a multi-armed bandit algorithm on low-cost edge devices to automatically pick performance settings for HPC applications, then transfers those settings to full-scale clusters. If it works, it could cut the expensive tuning time normally spent on supercomputers, at the cost of some accuracy.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LF-to-HF transfer evidence is only a 25% bound, while the 12% near-oracle claim is measured on the Jetson surrogate, so the edge-to-HPC promise is unproven.","rationale":"The reader's weakest assumption is exactly the load-bearing point. LASP's advertised advantage is offloading expensive tuning to low-cost edge runs; if the selected configuration is far from optimal on the real HPC target, the method loses its raison d'etre. The paper's own Fig. 2 quantifies the LF-to-HF gap as at most 25% for the top-20 set, while the default-configuration gains reported in Fig. 8 are 6-14%, so the transfer error can erase the entire benefit. Moreover, the 'within 12%' statement in Section V-D refers to distance from the oracle computed on the Jetson: Section V-A explicitly says all autotuning results are on the Jetson Nano. Thus the central number does not by itself substantiate the edge-then-HPC workflow. A single deployment experiment on the i7-14700 comparing LASP's chosen configuration to the high-fidelity oracle would settle this. Other issues, such as the MinMax normalization in Algorithm 1 requiring global extrema unknown to an online algorithm, best-run regret selection, and comparisons only against default settings, are real but secondary; the low-fidelity/high-fidelity transfer question is the one on which the central claim actually stands or falls. Because the concern matches the reader's own identified weak assumption and the proposed experiment can move the verdict either way, I would keep the CONDITIONAL verdict unchanged rather than escalate it.","tokens_in":14574,"tokens_out":4080,"duration_ms":41266,"concrete_test":"For each application, run LASP on the Jetson Nano at the stated low-fidelity setting for T=1000 iterations, take its returned xopt and the application's default configuration, and execute both on the Intel i7-14700 at the same high-fidelity setting used for the Fig. 2 oracle. Compute distance-to-oracle and performance gain over default on the Intel target, repeating over 100 independent LASP runs. If the median high-fidelity distance of xopt exceeds 12%, or if xopt is not better than the default on the Intel platform, the 'within 12%' claim and the edge-to-HPC workflow are not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's motivating workflow is to tune parameters on a Jetson Nano at low fidelity and deploy them on an HPC platform at high fidelity (Fig. 1). The headline 'within 12% of optimal' claim (Section V-D) is computed from runs on the Jetson edge device: Section V-A states 'All the autotuning results and shown in the subsequent section are done on the Jetson Nano device.' So the 12% figure is distance from the low-fidelity oracle, not from the high-fidelity oracle on the target HPC platform. The only evidence that tuned parameters transfer is Fig. 2 (Section II.C), which shows that the top-20 low-fidelity configurations, when run at high fidelity on the Intel i7-14700, land within 25% of the high-fidelity oracle. A 25% transfer error is larger than the reported default-configuration gains (6-14% in Fig. 8), so the configuration LASP selects could be no better than the default on the actual HPC target. The central value proposition, tuning on edge and executing on HPC, therefore rests on a weak 25% bound rather than the advertised 12% near-oracle performance.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LASP, a multi-armed bandit (UCB) algorithm for tuning application-level parameters of HPC proxy applications on edge devices. The intended workflow is to tune at low fidelity on an NVIDIA Jetson Nano and transfer the resulting configuration to high-fidelity execution on an HPC platform (Fig. 1). The paper reports exhaustive-search-based oracle comparisons, a distance-from-oracle of within 12% for Hypre, gains of 6-14% over default settings for four applications, and a lower CPU/memory footprint than the BLISS autotuner. The evaluation covers Lulesh, Kripke, Clomp, and Hypre, and includes a sensitivity study with synthetic measurement noise.","tokens_in":14795,"tokens_out":7168,"duration_ms":72006,"significance":"If the reported behavior were produced by a genuinely online algorithm, LASP would be a useful contribution: the problem is real, the experiments span four distinct proxy applications, and the resource-footprint comparison with BLISS is a worthwhile datapoint. The paper is also candid in Section IV-B about scalability limitations. However, the significance currently depends on closing three gaps: the normalization step in Algorithm 1 requires global information that an online algorithm does not have; the low-fidelity-to-high-fidelity transfer evidence is only a 25% bound, which is weaker than the reported default-configuration gains; and tuning quality is not compared with any non-default autotuning baseline. With those gaps closed, the paper could be a solid systems contribution, but in its current form the central claims are not supported.","major_comments":[{"comment":"Algorithm 1 (line 2) normalizes execution time τ and power ρ by global minima and maxima over the entire configuration space. In an online bandit setting, τ_min, τ_max, ρ_min, and ρ_max are unavailable until every configuration has been evaluated; if they are obtained by a prior exhaustive scan, LASP already knows the oracle (the configuration attaining the minimum execution time), and its exploration is a post-hoc ranking rather than an online search. If they are instead updated online, the reward of every previously sampled arm changes whenever an extremum is updated, and the UCB means and counts in Eq. (2) are not defined consistently. Because the 12% near-oracle result in Section V-D is computed with this normalization, the central claim of a lightweight online autotuner is not supported as stated. The authors should either give an online-feasible normalization (e.g., fixed known bounds or streaming min-max updates with a re-derivation of the UCB update), or rerun the evaluation with such a scheme.","section":"Section IV, Algorithm 1 line 2"},{"comment":"The claim that 'LASP can reach within 12% of the optimal configuration' is measured on the Jetson Nano (Section V-A), i.e., distance to the low-fidelity oracle. The only evidence for transfer to the target HPC platform is Fig. 2(a), which shows that top-20 low-fidelity configurations land within 25% of the high-fidelity oracle on an Intel i7-14700. Since the default-configuration gains in Fig. 8 are 6-14%, a 25% transfer error can place the selected configuration at or below default performance on the HPC target. The paper needs to report the distance from the high-fidelity oracle for the exact configuration returned by LASP, and ideally report high-fidelity default-vs-LASP performance gains on the target system.","section":"Section V-D and Section II.C"},{"comment":"The regret bound cited after Eq. (7) is the standard UCB bound for rewards in [0,1], and Section III explicitly assumes rewards are constrained to [0,1]. However, the reward in Eq. (5) is a weighted sum of reciprocals of normalized execution time and power. A normalized metric can approach zero for a near-optimal configuration, making the reciprocal unbounded and generally larger than 1. Therefore the bound in Eq. (7) cannot be applied to LASP's reward as defined, and the regret curves in Section V-E evaluate a different quantity. The paper must either transform the reward to a bounded [0,1] range while preserving the optimum, or provide an appropriate regret analysis for the actual unbounded reward.","section":"Section III, Eq. (7), and Eq. (5)"},{"comment":"The only comparison against another autotuner is the CPU/memory footprint comparison with BLISS in Fig. 10. The text states that LASP 'did not do better in terms of efficiently finding the optimal parameters' but gives no quantitative comparison of tuning quality, number of evaluations to reach the optimum, or final distance from the oracle for BLISS or any other baseline. Without such a comparison, the contribution of LASP relative to existing Bayesian optimization or Hyperband approaches is not established; performance gains over a fixed default configuration alone do not position the method against the state of the art.","section":"Section V-D and Fig. 10"},{"comment":"The regret analysis is reported for the 'best-run (one time least regret run)'. Selecting the single best run among multiple random seeds does not provide an expectation or high-probability statement, and it biases the reported regret in a favorable direction. The paper should report mean regret with confidence intervals over multiple runs, as is done for the distance-from-oracle measurement in Fig. 9, rather than a single cherry-picked trajectory.","section":"Section V-E, Fig. 11"},{"comment":"The sensitivity study adds 5%, 10%, and 15% random noise to measured data. This is observation noise, not a changing environment: the reward distributions themselves are stationary throughout the run. No experiment changes the workload, input size, or reward distribution over time, so the abstract's claim that LASP 'adapts seamlessly to changing environments' is not supported. The authors should test adaptation under a non-stationary reward process, for example by switching problem size or input type midway through the run, and show that LASP recovers the new optimum.","section":"Section V-F"}],"minor_comments":[{"comment":"The text says LASP is 'demonstrated in Section 10', but the manuscript has no Section 10; this is presumably a reference to a figure or another section and should be corrected.","section":"Section IV-A"},{"comment":"There are textual errors in this section: 'performance gain performance gain' is duplicated, and 'we ranLASP' is missing a space. These should be fixed in revision.","section":"Section V-D"},{"comment":"The heatmaps use color intensity without a colorbar or numeric scale, so selection frequencies cannot be read quantitatively; adding colorbars would make the convergence claims more verifiable.","section":"Figures 6 and 7"},{"comment":"The overlap count in Fig. 2(b) is reported without stating the total number of configurations considered or the expected overlap under random selection, which makes it difficult to judge whether the observed overlap is substantially better than chance.","section":"Section II.C, Fig. 2(b)"}],"recommendation":"major_revision","confidential_remarks":"The paper has a coherent idea and an honest limitation section, but the current evaluation does not support the online, near-oracle, transferable-tuner narrative. The most serious issue is the global MinMax normalization in Algorithm 1, which as written requires full knowledge of the configuration space before tuning begins. This is fixable in principle by redesigning the normalization and re-running the experiments, so I do not recommend rejection, but the revision needs to address the transfer measurement and baseline comparisons as well. If the authors can provide an online-feasible algorithm, direct high-fidelity transfer results, and at least one tuning-quality baseline, the contribution would be within scope for a systems/performance journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's my take on arXiv:2501.01057. LASP is a UCB-based autotuner for HPC applications on edge devices, with the twist that tuned parameters are meant to transfer to HPC clusters. The paper's actual contribution is the edge-device study and the low-fidelity/high-fidelity transfer analysis. The algorithm itself is textbook UCB (Eq. 2); the novelty is the application setting and the resource-constrained evaluation.\n\nWhat the paper does well: the experiments cover four real HPC proxy apps on a Jetson Nano in two power modes, with a resource-footprint comparison against BLISS showing LASP is genuinely lighter. The authors are transparent about limitations (scalability, heterogeneous devices) and include a noise-injection sensitivity study. The reported gains over default configurations (6-14%) are modest but real.\n\nThe soft spots are serious, though. The MinMax normalization in Algorithm 1 needs global min/max execution time and power over the whole configuration space. An online algorithm cannot know those without exhaustive search; the paper doesn't say how LASP gets them in practice. That undercuts the \"online\" claim.\n\nMore importantly, the headline 'within 12% of optimal' claim (Section V-D) is measured on the Jetson, i.e., against the low-fidelity oracle. The only transfer evidence is Fig. 2, which shows the top-20 low-fidelity configurations land within 25% of the high-fidelity oracle on the Intel platform. Since the default gains are 6-14%, a 25% transfer error means the configuration LASP selects could be no better than default on the actual HPC target. The paper's motivating workflow—tune on edge, deploy on cluster—therefore rests on a weak bound. The stress-test note is correct.\n\nAlso, the regret analysis uses the best run per application, not the mean, which flatters convergence. The dynamic-adaptivity claim is only tested with synthetic noise, not with actual workload shifts. And the comparison with BLISS is only about resource usage, not tuning quality; the authors admit LASP doesn't find better configurations, which is fair but limits the claim of effectiveness.\n\nWho is this for? Researchers working on edge-based autotuning might find the resource-utilization data useful. But the paper needs major revision to address the normalization issue and, above all, to provide direct transfer evidence: evaluate a configuration selected by LASP on the high-fidelity platform and show it beats default there. Without that, the central promise is unproven.\n\nI'd send it to peer review with a request for major revision; the idea is sound and the experimental platform is real, but the transfer claim needs to be substantiated, not just asserted.\n\nRecommended.","headline":"LASP is a lightweight UCB autotuner for edge devices with a sound practical evaluation, but its edge-to-HPC transfer claim rests on a 25% bound that is weaker than the reported default gains, so the paper needs major revision.","tokens_in":15379,"tokens_out":2889,"would_cite":false,"duration_ms":25357,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LASP, a multi-armed bandit autotuner that runs on resource-constrained edge devices, claims to reach within 12% of the optimal HPC configuration while using far less CPU and memory than Bayesian optimization baselines.","keywords":["HPC parameter autotuning","edge devices","multi-armed bandit","upper confidence bound","HPC applications","performance modeling","low-fidelity transfer","LASP"],"falsifier":"Run LASP on the edge device at low fidelity, take the configuration it returns, and measure its execution time at high fidelity on the target HPC cluster against a true exhaustive-search oracle on that cluster; the core workflow is falsified if the transferred configuration lands more than 25% away from that cluster-side oracle, which is the gap the paper already reports for the top-20 low-fidelity set.","tokens_in":14351,"feed_emoji":"⚙️","tokens_out":7680,"duration_ms":58957,"temperature":0.7,"pith_summary":"The paper sets out to show that HPC parameter autotuning can be moved off expensive supercomputers and onto cheap, resource-constrained edge devices. Its proposed algorithm, LASP, is a multi-armed bandit that treats each parameter configuration as an arm and uses an upper confidence bound rule to balance exploring new configurations with exploiting good ones, updating in real time from execution-time and power measurements. The authors claim that LASP reaches within 12% of the optimal configuration even for the large Hypre search space when optimizing execution time, and delivers 6-14% performance gains over default configurations across four HPC applications. If true, this gives users a lightweight, portable tuning step that adapts to noisy measurements and changing workloads, and whose results can be transferred from low-fidelity edge runs to high-fidelity HPC execution.","feed_headline":"Bandit tuner finds near-optimal HPC configs on edge devices","feed_subtitle":"LASP runs on a Jetson Nano, cuts tuning cost, and transfers parameters to HPC clusters.","key_machinery":"The mechanism carrying the argument is the upper confidence bound (UCB) arm-selection rule, Eq. (2): $UCB(x,t) = R_x + \\sqrt{2 \\ln t / N_x}$, where $R_x$ is the weighted reward of configuration $x$ and $N_x$ is the number of times it has been tried. UCB is the 'optimism under uncertainty' device that forces LASP to keep sampling under-explored configurations while shifting weight to arms that have looked good so far. Around it the paper builds a reward model, Eq. (5), that maps measured execution time and power consumption into a scalar reward via inverse MinMax normalization and user weights $\\alpha$ and $\\beta$, and a low-fidelity-to-high-fidelity surrogate workflow in which edge-device runs at reduced mesh or zone sizes stand in for the final HPC execution.","core_discovery":"LASP's central claim is that a UCB-based multi-armed bandit can carry out HPC configuration autotuning on an edge device with low overhead. Each unique application-parameter combination is an arm; the reward for an arm is the inverse of a MinMax-normalized weighted sum of execution time and power consumption, with user-set weights $\\alpha$ and $\\beta$; and the algorithm selects the arm maximizing $R_x + \\sqrt{2 \\ln t / N_x}$. After $T$ iterations it returns the most frequently chosen configuration. Experiments on Lulesh, Kripke, Clomp, and Hypre on a Jetson Nano show that this procedure converges to within 12% of the oracle for the largest search space (Hypre) when time-optimized, and improves over default settings by 6-14% across applications. The paper also reports that the top-20 low-fidelity configurations land within 25% of the high-fidelity oracle, and that LASP keeps working when 5-15% random noise is added to measurements, which is offered as evidence of adaptability to volatile edge environments.","pith_inferences":["Whether the 12%-of-oracle bound holds on the high-fidelity cluster, not just on the edge device, is not measured in the paper; a direct test would compare cluster-side distance to oracle for LASP's returned configuration.","The paper uses the standard UCB regret bound without deriving a new one; the novelty is the application to edge-device HPC autotuning, not a new regret guarantee, so theoretical gains would need a separate analysis for non-stationary rewards.","Since power consumption saturates on the Jetson Nano under compute-heavy loads, the time objective is the more discriminative signal; on devices where power varies more, the power-focused mode might behave differently than reported here.","A natural extension is to test LASP against non-stochastic bandit methods (e.g., EXP3 or Hyperband-style best-arm identification) in the same edge setting, since the paper only compares to a Bayesian-optimization baseline."],"forward_implications":["Autotuning can be offloaded from expensive HPC node hours to cheap edge devices, with tuning cost paid once and the found configuration reused on the cluster.","Users can express a time-versus-power trade-off through $\\alpha$ and $\\beta$, and LASP will steer to different optima accordingly, as shown in the performance-gain analysis.","Because LASP receives online feedback, it can re-tune when the workload, input size, or device state changes, and it tolerates measurement noise up to 15% in the paper's sensitivity study.","The resource-footprint comparison with BLISS indicates LASP can run alongside other edge workloads, making autotuning feasible in co-located, power-capped settings (5W and 10W modes).","The within-12%-of-oracle result on Hypre suggests the approach scales to very large configuration spaces (92,160 configs) where exhaustive search would be impractical."],"supporting_citations":[{"why":"Supplies the UCB selection rule that LASP's algorithm is built on.","marker":"[40]"},{"why":"Provides the stochastic bandit formulation and the logarithmic regret bound used to argue efficiency.","marker":"[25]"},{"why":"The BLISS baseline, a Bayesian-optimization autotuner whose tuning quality and resource footprint LASP is compared against.","marker":"[16]"},{"why":"One of the four HPC applications, providing the Kripke search space used in evaluation.","marker":"[38]"},{"why":"Provides Hypre, the large search space that yields the within-12% claim.","marker":"[46]"},{"why":"One of the four HPC applications, used to evaluate OpenMP parameter tuning.","marker":"[47]"},{"why":"Provides Lulesh, one of the four HPC applications used in the performance-gain analysis.","marker":"[48]"}],"fun_headline_variants":["UCB bandit autotunes HPC params on edge devices","LASP: lightweight bandit tuner for HPC on edge devices","Bandit learning tunes HPC apps on edge, 6-14% faster than default","Edge autotuning with bandits: near-oracle results on Hypre search","Multi-armed bandit autotuner for edge HPC: low overhead, high accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that parameter configurations that win at low fidelity on the edge device remain near-optimal at high fidelity on the actual HPC platform where the application will run; if that transfer breaks, the whole tuning-on-edge workflow produces configurations that are not optimal for the real target.","fun_headline_variants_meta":{"raw":{"variants":["UCB bandit autotunes HPC params on edge devices","LASP: lightweight bandit tuner for HPC on edge devices","Bandit learning tunes HPC apps on edge, 6-14% faster than default","Edge autotuning with bandits: near-oracle results on Hypre search","Multi-armed bandit autotuner for edge HPC: low overhead, high accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001121,"raw_usage":{"total_tokens":4658,"prompt_tokens":935,"completion_tokens":3723,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":551,"completion_tokens_details":{"reasoning_tokens":3618}},"tokens_in":551,"tokens_out":3723,"duration_ms":28393,"temperature":1.0,"reasoning_tokens":3618,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:35:27.721506+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run LASP on the edge device at low fidelity, take the configuration it returns, and measure its execution time at high fidelity on the target HPC cluster against a true exhaustive-search oracle on that cluster; the core workflow is falsified if the transferred configuration lands more than 25% away from that cluster-side oracle, which is the gap the paper already reports for the top-20 low-fidelity set.","supporting_citations":[{"cited_title":"Using confidence bounds for exploitation-exploration trade- offs,","cited_arxiv_id":null,"evidence_quote":"Supplies the UCB selection rule that LASP's algorithm is built on."},{"cited_title":"Bliss: auto-tuning complex applications using a pool of diverse lightweight learning models,","cited_arxiv_id":null,"evidence_quote":"The BLISS baseline, a Bayesian-optimization autotuner whose tuning quality and resource footprint LASP is compared against."},{"cited_title":"Kripke-a massively parallel transport mini-app,","cited_arxiv_id":null,"evidence_quote":"One of the four HPC applications, providing the Kripke search space used in evaluation."},{"cited_title":"hypre: A library of high performance preconditioners,","cited_arxiv_id":null,"evidence_quote":"Provides Hypre, the large search space that yields the within-12% claim."},{"cited_title":"Clomp: Ac- curately characterizing openmp application overheads,","cited_arxiv_id":null,"evidence_quote":"One of the four HPC applications, used to evaluate OpenMP parameter tuning."},{"cited_title":"Lulesh 2.0 updates and changes,","cited_arxiv_id":null,"evidence_quote":"Provides Lulesh, one of the four HPC applications used in the performance-gain analysis."}],"review_version":1}