REVIEW 4 major objections 5 minor 20 references
HPCAdvisor: A Tool for Assisting Users in Selecting HPC Resources in the Cloud
T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read HPCAdvisor is an open-source tool that fully automates the lifecycle of collecting HPC benchmarking data in the cloud and reduces it to a Pareto front over execution time and cost.
desk verdict HPCAdvisor is a genuine open-source engineering contribution for cloud HPC resource selection, but the paper is a tool report whose single-run Pareto advice needs a variability caveat or study. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is the HPCAdvisor tool itself, a Python program that wraps a scenario-sweep loop over VM type, node count, processes per node, and application input. The key interface is a contract with the user: a pair of bash functions, hpcadvisor_setup and hpcadvisor_run, the latter of which can emit key-value metrics via 'HPCADVISORVAR' lines captured into the dataset. The tool's back end currently uses Azure Batch to create and resize node pools per VM type, executing setup once per VM type and then running compute tasks per scenario. Advice is produced by computing the Pareto front of the executed scenarios with respect to execution time and cost, so the user is offered a set of non-dominated configurations rather than a single 'best' answer.
What would settle it
Run a scenario that HPCAdvisor places on the Pareto front several times under the same conditions (e.g., the reported OpenFOAM case at 16 hb120rs_v3 nodes, 34 s, $0.544) and measure the spread of execution times. If the spread rivals the gaps between consecutive Pareto-front points, then the front can misorder options and the advice depends on the luck of a single run.
Extended reading notes
Core claim
The paper's central claim is that a practical tool can automate the entire data-collection cycle for HPC resource selection in the cloud. Given two user inputs—a YAML specification of cloud subscription, region, VM types, node counts, and application input parameters, plus a bash script defining application setup and run—HPCAdvisor provisions the cloud resources, executes the full parameter sweep, extracts metrics through an 'HPCADVISORVAR' key-value protocol, and records everything in a dataset. From that dataset it automatically draws execution-time-vs-nodes, time-vs-cost, speedup, and efficiency plots, and it distills the advice into a Pareto front: the set of scenarios that are not dominated by any other in both execution time and cost. The paper demonstrates the tool with OpenFOAM and LAMMPS examples, including a LAMMPS run at up to 1,920 cores on InfiniBand-connected VMs, and reports that the resulting advice lists scenarios such as 16-node hb120rs_v3 at 34 seconds and $0.544 for OpenFOAM.
Load-bearing premise
The advice assumes a scenario's execution time is stable enough to be represented by a single run, without needing repeated trials or variance estimates.
Editorial extensions
If this is right
- Users can obtain a cost-performance shortlist for their own application input in hours of automated cloud time, without hand-building a benchmarking environment.
- The tool doubles as a parameter-sweep engine for HPC applications, giving it use beyond single resource-selection queries.
- Because the back end is replaceable and the configuration is YAML-plus-bash, the same front end can be pointed at other orchestrators such as Slurm.
- The planned 'smart sampling' module aims to prune scenarios that cannot enter the Pareto front, lowering the monetary cost of collecting advice.
- With enough accumulated execution data, the vision is to give users a Pareto-front recommendation for a new input with little or no new execution.
Reading between the lines
- A natural extension the paper gestures at but does not develop: the same Pareto-front logic could drive an active-learning loop that proposes a small set of scenarios to run next, treating the front itself as the approximation target.
- The replaceable back end means the tool's claims are not tied to Azure; porting the scenario-sweep loop to another provider or an on-prem Slurm cluster would test whether the automation generalizes.
- For users with a fixed budget, the front could be re-ranked by a utility function (e.g., willingness to pay per saved second), which the paper leaves to the user's own judgment.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents HPCAdvisor, an open-source tool for assisting users in selecting HPC cloud resources (VM type, number of nodes, processes per node) given a specific application input. The tool automatically provisions a cloud environment (using Azure Batch as the current backend), generates and executes a combinatorial set of scenarios, collects execution-time and cost data, produces plots (execution time vs. nodes, execution time vs. cost, speedup, efficiency), and returns advice in the form of a Pareto front of non-dominated configurations. The paper describes the design and implementation, including user input files, environment deployment, data collection, and example outputs for OpenFOAM and LAMMPS with up to 1,920 cores. It also outlines planned optimizations for smart sampling and scenario reduction.
Significance. If the tool works as presented, it addresses a real practical gap: HPC users without deep benchmarking expertise often struggle to choose cloud resources that balance performance and cost. HPCAdvisor's contribution is primarily engineering and design, not algorithmic novelty. The nominal strengths are its end-to-end automation, use of standard bash scripts for application setup/run, and open-source availability, which makes it extensible. The example outputs and plots demonstrate that the tool can produce plausible resource-selection advice for real HPC applications. However, the paper does not provide a controlled evaluation: there is no comparison against alternative selection methods, no quantitative assessment of the quality of the Pareto-front advice, and no discussion of the stability of the results under cloud performance variability. In addition, several concrete errors in the manuscript's pseudocode and examples undermine its reproducibility as written.
major comments (4)
- [Section III-C, Algorithm 1] The indentation of Algorithm 1 places the create/execute/store/update steps (lines 8-11) inside the 'if previousVMType != task.vmtype' block. As written, for each VM type only the first scenario in the task list would be executed; subsequent scenarios with the same VM type would fall into the empty else branch and be skipped. This contradicts the paper's claim that HPCAdvisor executes all combinations of VM types, node counts, and inputs, and it is also inconsistent with the example outputs in Section III-D, which show multiple node counts per VM type. Please correct the pseudocode indentation or clarify the intended control flow.
- [Section III-A, Listing 1] The 'appinputs' block in the main configuration file contains two entries with the same YAML key 'mesh' but different values. Standard YAML mappings require unique keys; most parsers will either reject the file or silently keep only the last value. The paper states that this configuration yields 3 x 6 x 2 = 36 scenarios, but with a duplicate key only 18 scenarios would be produced. Please show the correct YAML syntax for specifying multiple values of the same input parameter (e.g., a list of mappings or a list of values).
- [Section III-C/E, Figures 2-5 and Listings 3-4] The Pareto-front advice is computed from a single execution per scenario (Algorithm 1, line 9), with no repeated runs, confidence intervals, or variance analysis. HPC cloud VMs, especially multi-node InfiniBand clusters, are subject to run-to-run performance noise from network contention, CPU steal, and neighbor interference. If a single measured execution time is an outlier, the resulting Pareto front can omit a genuinely attractive configuration or include a dominated one. Since the paper explicitly states in Section III-F that the aim is to generate a Pareto front rather than exact execution times, the stability of the front under measurement noise is load-bearing for the tool's advice. The paper should at minimum acknowledge this limitation and ideally provide a sensitivity analysis or repeated runs for a subset of scenarios.
- [Section III-A, Listing 2] In the LAMMPS example script, line 37 echoes 'HPCADVISORVAR APPEXECTIME=$LAMMPSCLOCKTIME', but the variable assigned on line 34 is APPEXECTIME, not LAMMPSCLOCKTIME. As written, the HPCADVISORVAR output would contain an empty execution time, which would corrupt the advice data for the LAMMPS example. Please fix the variable name to match (either echo $APPEXECTIME or rename the variable) and verify that the example output in Listing 4 is reproducible.
minor comments (5)
- [Table I] The configuration file uses 'ppr: 100' (described as a percentage), but Table I lists PPN as 'Processes per node' and Listing 2 uses the variable $PPN. The relationship between the user-supplied percentage and the environment variable is not explained; please clarify the naming and semantics.
- [Section III-A, Listing 2] The long 'source /cvmfs/software.eessi.io/versions/2023.06/init/bash' command is line-wrapped in the manuscript. Please ensure it appears on a single line so that the example is directly executable.
- [Section III-D, Figure 3] In Figure 3, the x-axis is labeled 'Execution time (seconds)' and the y-axis 'Cost (USD)', but the caption says 'Execution Time vs. Cost'. The figure is visually clear, but the axis labels should be consistent with the caption style used in other figures (e.g., 'Execution time (seconds)' on the x-axis is fine, just verify the orientation).
- [Section II] The related work section does not mention how existing resource-selection tools (e.g., Oikonomos and A2Cloud-RF) handle the uncertainty of cloud performance measurements; a brief comparison of their validation methodologies with HPCAdvisor's would strengthen the positioning.
- [Section III-F] The paper uses the term 'Design of Experiments' without a citation; consider adding a reference for readers unfamiliar with the concept.
Circularity Check
No circularity: HPCAdvisor's Pareto-front advice is computed directly from measured scenario executions, with no fitted parameters or predictive model whose outputs are fed back into its inputs.
full rationale
The paper makes no derived prediction claim. HPCAdvisor's central functionality is automated data collection: the tool deploys cloud resources, executes every scenario combination (VM type, node count, processes per node, application input), stores the measured results, and then computes a Pareto front from the recorded execution time and cost values (Algorithm 1, Section III-C; advice examples in Listings 3-4). There is no fitted model, no calibrated parameter, and no equation that maps inputs to outputs other than the direct measurement of the scenario itself. The Pareto front is defined as the set of non-dominated measured points, so the advice is the measured data organized, not a quantity derived from a separate input. The single self-citation [1] is cited only as background on HPC cloud cost/performance research in the introduction and is not load-bearing for the tool's design or results. The planned optimizations in Section III-F (machine learning, regression, scenario discarding) are explicitly described as future work and are not used in the current implementation or in the validation examples. The concern that single executions per scenario may be affected by cloud performance variability is an empirical robustness issue, not an instance of circular reasoning, because the reported values are measurements rather than predictions forced by construction. Accordingly, no circular step can be exhibited, and the appropriate finding is no significant circularity.
Assumptions & free parameters
assumptions (3)
- domain assumption Azure Batch and the underlying cloud APIs behave consistently, and VM cost can be estimated from VM types.
- domain assumption A single execution per scenario is representative; execution time does not vary enough to change the Pareto front.
- domain assumption User-provided bash scripts correctly set up and run the application and emit metrics through HPCADVISORVAR lines.
Cite this review
Pith. "Pith review of HPCAdvisor: A Tool for Assisting Users in Selecting HPC Resources in the Cloud." pith.science (2026). https://pith.science/paper/XKNKVBZV
@misc{pith2026241115448,
author = {Pith},
title = {Pith review of: HPCAdvisor: A Tool for Assisting Users in Selecting HPC Resources in the Cloud},
year = {2026},
howpublished = {\url{https://pith.science/paper/XKNKVBZV}},
note = {Machine review of arXiv:2411.15448}
}
read the original abstract
Cloud platforms are increasingly being used to run HPC workloads. Major cloud providers offer a wide variety of virtual machine (VM) types, enabling users to find the optimal balance between performance and cost. However, this extensive selection of VM types can also present challenges, as users must decide not only which VM types to use but also how many nodes are required for a given workload. Although benchmarking data is available for well-known applications from major cloud providers, the choice of resources is also influenced by the specifics of the user's application input. This paper presents the vision and current implementation of HPCAdvisor, a tool designed to assist users in defining their HPC clusters in the cloud. It considers the application's input and utilizes a major cloud provider as a use case for its back-end component.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
HPC cloud for scientific and business applications: taxon- omy, vision, and research challenges,
M. A. Netto, R. N. Calheiros, E. R. Rodrigues, R. L. Cunha, and R. Buyya, “HPC cloud for scientific and business applications: taxon- omy, vision, and research challenges,”ACM Computing Surveys (CSUR), vol. 51, no. 1, pp. 1–29, 2018
work page 2018
-
[2]
Evaluating HPC job run time predictions using application input parameters,
K. Lamar, A. Goponenko, O. Aaziz, B. A. Allan, J. M. Brandt, and D. Dechev, “Evaluating HPC job run time predictions using application input parameters,” in Proceedings of the 17th ACM International Con- ference on Distributed and Event-based Systems (DEBS’23) , 2023, pp. 127–138
work page 2023
-
[3]
Performance prediction of parallel applications: a sys- tematic literature review,
J. Flores-Contreras, H. A. Duran-Limon, A. Chavoya, and S. H. Almanza-Ruiz, “Performance prediction of parallel applications: a sys- tematic literature review,” The Journal of Supercomputing , vol. 77, pp. 4014–4055, 2021
work page 2021
-
[4]
A novel two-step job runtime estimation method based on input parameters in HPC system,
Q. Wang, J. Li, S. Wang, and G. Wu, “A novel two-step job runtime estimation method based on input parameters in HPC system,” in 2019 IEEE 4th International Conference on Cloud Computing and Big Data Analysis (ICCCBDA). IEEE, 2019, pp. 311–316
work page 2019
-
[5]
Backfilling using system- generated predictions rather than user runtime estimates,
D. Tsafrir, Y . Etsion, and D. G. Feitelson, “Backfilling using system- generated predictions rather than user runtime estimates,” IEEE Trans- actions on Parallel and Distributed Systems, vol. 18, no. 6, pp. 789–803, 2007
work page 2007
-
[6]
Cross-platform performance prediction of parallel applications using partial execution,
L. T. Yang, X. Ma, and F. Mueller, “Cross-platform performance prediction of parallel applications using partial execution,” in SC’05: Proceedings of the 2005 ACM/IEEE Conference on Supercomputing . IEEE, 2005, pp. 40–40
work page 2005
-
[7]
Prediction services for distributed computing,
W. Smith, “Prediction services for distributed computing,” in 2007 IEEE International Parallel and Distributed Processing Symposium . IEEE, 2007, pp. 1–10
work page 2007
-
[8]
Predicting cloud performance for HPC applications before deployment,
G. Mariani, A. Anghel, R. Jongerius, and G. Dittmann, “Predicting cloud performance for HPC applications before deployment,” Future Generation Computer Systems , vol. 87, pp. 618–628, 2018
work page 2018
Show all 20 references
-
[9]
Oikonomos: An opportunistic, deep-learning, resource-recommendation system for cloud HPC,
J.-H. Betting, D. Liakopoulos, M. Engelen, and C. Strydis, “Oikonomos: An opportunistic, deep-learning, resource-recommendation system for cloud HPC,” in 2023 IEEE 34th International Conference on Application-specific Systems, Architectures and Processors (ASAP) . IEEE, 2023, ...
2023
-
[10]
On the user–scheduler dialogue: studies of user-provided runtime estimates and utility functions,
C. B. Lee and A. Snavely, “On the user–scheduler dialogue: studies of user-provided runtime estimates and utility functions,” The International Journal of High Performance Computing Applications , vol. 20, no. 4, pp. 495–506, 2006
2006
-
[11]
Are user runtime estimates inherently inaccurate?
C. Bailey Lee, Y . Schwartzman, J. Hardy, and A. Snavely, “Are user runtime estimates inherently inaccurate?” in Job Scheduling Strategies for Parallel Processing: 10th International Workshop, JSSPP 2004, New York, NY, USA, June 13, 2004. Revised Selected Papers 10 . Springer,...
2004
-
[12]
Exploring job running path to predict runtime on multiple production supercomputers,
W. Yang, X. Liao, D. Dong, and J. Yu, “Exploring job running path to predict runtime on multiple production supercomputers,” Journal of Parallel and Distributed Computing , vol. 175, pp. 109–120, 2023
2023
-
[13]
Selecting efficient cloud resources for HPC workloads,
J. R. Brunetta and E. Borin, “Selecting efficient cloud resources for HPC workloads,” in Proceedings of the 12th IEEE/ACM International Conference on Utility and Cloud Computing , 2019, pp. 155–164
2019
-
[14]
A2cloud-rf: A random forest based statistical framework to guide resource selection for high-performance scientific computing on the cloud,
D. Samuel, S. Khan, C. J. Balos, Z. Abuelhaj, A. D. Dutoi, C. Kari, D. Mueller, and V . K. Pallipuram, “A2cloud-rf: A random forest based statistical framework to guide resource selection for high-performance scientific computing on the cloud,” Concurrency and Computation: Pra...
2020
-
[15]
Enabling continuous testing of hpc systems using reframe,
V . Karakasis, T. Manitaras, V . H. Rusu, R. Sarmiento-P ´erez, C. Big- namini, M. Kraushaar, A. Jocksch, S. Omlin, G. Peretti-Pezzi, J. P. Augusto et al. , “Enabling continuous testing of hpc systems using reframe,” in Tools and Techniques for High Performance Computing: Sele...
2019
-
[16]
Ramble: Reproducible And Measurable Benchmarks in a Layered Environment,
Ramble Development Team, “Ramble: Reproducible And Measurable Benchmarks in a Layered Environment,” 2024. [Online]. Available: https://ramble.readthedocs.io/en/latest/
2024
-
[17]
Pavilion 2: A framework for running and managing tests on HPC systems,
Pavilion 2 Development Team, “Pavilion 2: A framework for running and managing tests on HPC systems,” 2024. [Online]. Available: https://pavilion2.readthedocs.io/en/latest/
2024
-
[18]
The spack package manager: bringing order to HPC software chaos,
T. Gamblin, M. LeGendre, M. R. Collette, G. L. Lee, A. Moody, B. R. De Supinski, and S. Futral, “The spack package manager: bringing order to HPC software chaos,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis , ...
2015
-
[19]
Easybuild: Building software with ease,
K. Hoste, J. Timmerman, A. Georges, and S. De Weirdt, “Easybuild: Building software with ease,” in 2012 SC Companion: High Perfor- mance Computing, Networking Storage and Analysis . IEEE, 2012, pp. 572–582
2012
-
[20]
Eessi: A cross-platform ready-to-use optimised scientific software stack,
B. Dr ¨oge, V . Holanda Rusu, K. Hoste, C. van Leeuwen, A. O’Cais, and T. R ¨oblitz, “Eessi: A cross-platform ready-to-use optimised scientific software stack,” Software: Practice and Experience , vol. 53, no. 1, pp. 176–210, 2023
2023
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.