{"id":"5ca7879c-0b17-445f-9e25-479128e5d3a8","arxiv_id":"2411.15448","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"HPCAdvisor is an open-source tool that automatically deploys cloud HPC clusters, runs user-defined benchmark scenarios, and recommends resource choices using a Pareto front of execution time versus cost.","lead":"This paper describes HPCAdvisor, an open-source tool that automatically creates cloud HPC clusters, runs benchmark scenarios, and collects timing and cost data. It helps users choose VM types and node counts by presenting the best trade-offs between speed and cost as a Pareto front.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Pareto-front advice is built from single runs per scenario; without repeated executions or variance bounds, the recommended resource set may be unstable under cloud performance noise.","rationale":"The reader's weakest assumption identifies the same single-run measurement issue. I agree that this is the most load-bearing concern because the tool's value proposition is the advice itself; if the Pareto front is sensitive to noise, a user may select a suboptimal resource. The concern is not fatal to the paper's description of the tool's architecture or automation, but it does support the CONDITIONAL verdict: the implementation is plausible, yet the advisory output lacks demonstrated reliability. No additional technical flaws were found in the described pipeline; the single-run reproducibility issue is the key gap that a revision should address.","tokens_in":10541,"tokens_out":2922,"duration_ms":28837,"concrete_test":"Rerun a representative scenario grid (e.g., the LAMMPS configuration from Listing 3) at least 5 times per scenario on different days and times, recompute the Pareto front for each repetition, and measure the Jaccard similarity of the non-dominated sets across repetitions. If the same configuration appears non-dominated in fewer than ~80% of repetitions, the single-run Pareto advice is not stable enough to support the claimed advisory value.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The advice in Section III-E is a Pareto front over execution time and cost, where each scenario's execution time comes from one measured run (Algorithm 1: execute compute task, store task data). Figures 2-5 and Listings 3-4 show no replicates, confidence intervals, or outlier handling. HPC cloud VMs, especially multi-node InfiniBand clusters, can show run-to-run variability from network contention, CPU steal, or neighbor interference. If a single measured time is an outlier, the Pareto front can omit a genuinely attractive configuration or include a dominated one. Since the paper's central claim is that HPCAdvisor exposes 'the best solutions in the search space considering execution time and cost,' measurement noise directly threatens the correctness of that advice. The paper itself frames the goal as generating a Pareto front rather than exact times (Section III-F), so the stability of the front under noise is load-bearing. This gap is not addressed anywhere in the validation or discussion.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents HPCAdvisor, an open-source tool for assisting users in selecting HPC cloud resources (VM type, number of nodes, processes per node) given a specific application input. The tool automatically provisions a cloud environment (using Azure Batch as the current backend), generates and executes a combinatorial set of scenarios, collects execution-time and cost data, produces plots (execution time vs. nodes, execution time vs. cost, speedup, efficiency), and returns advice in the form of a Pareto front of non-dominated configurations. The paper describes the design and implementation, including user input files, environment deployment, data collection, and example outputs for OpenFOAM and LAMMPS with up to 1,920 cores. It also outlines planned optimizations for smart sampling and scenario reduction.","tokens_in":10667,"tokens_out":5552,"duration_ms":46679,"significance":"If the tool works as presented, it addresses a real practical gap: HPC users without deep benchmarking expertise often struggle to choose cloud resources that balance performance and cost. HPCAdvisor's contribution is primarily engineering and design, not algorithmic novelty. The nominal strengths are its end-to-end automation, use of standard bash scripts for application setup/run, and open-source availability, which makes it extensible. The example outputs and plots demonstrate that the tool can produce plausible resource-selection advice for real HPC applications. However, the paper does not provide a controlled evaluation: there is no comparison against alternative selection methods, no quantitative assessment of the quality of the Pareto-front advice, and no discussion of the stability of the results under cloud performance variability. In addition, several concrete errors in the manuscript's pseudocode and examples undermine its reproducibility as written.","major_comments":[{"comment":"The indentation of Algorithm 1 places the create/execute/store/update steps (lines 8-11) inside the 'if previousVMType != task.vmtype' block. As written, for each VM type only the first scenario in the task list would be executed; subsequent scenarios with the same VM type would fall into the empty else branch and be skipped. This contradicts the paper's claim that HPCAdvisor executes all combinations of VM types, node counts, and inputs, and it is also inconsistent with the example outputs in Section III-D, which show multiple node counts per VM type. Please correct the pseudocode indentation or clarify the intended control flow.","section":"Section III-C, Algorithm 1"},{"comment":"The 'appinputs' block in the main configuration file contains two entries with the same YAML key 'mesh' but different values. Standard YAML mappings require unique keys; most parsers will either reject the file or silently keep only the last value. The paper states that this configuration yields 3 x 6 x 2 = 36 scenarios, but with a duplicate key only 18 scenarios would be produced. Please show the correct YAML syntax for specifying multiple values of the same input parameter (e.g., a list of mappings or a list of values).","section":"Section III-A, Listing 1"},{"comment":"The Pareto-front advice is computed from a single execution per scenario (Algorithm 1, line 9), with no repeated runs, confidence intervals, or variance analysis. HPC cloud VMs, especially multi-node InfiniBand clusters, are subject to run-to-run performance noise from network contention, CPU steal, and neighbor interference. If a single measured execution time is an outlier, the resulting Pareto front can omit a genuinely attractive configuration or include a dominated one. Since the paper explicitly states in Section III-F that the aim is to generate a Pareto front rather than exact execution times, the stability of the front under measurement noise is load-bearing for the tool's advice. The paper should at minimum acknowledge this limitation and ideally provide a sensitivity analysis or repeated runs for a subset of scenarios.","section":"Section III-C/E, Figures 2-5 and Listings 3-4"},{"comment":"In the LAMMPS example script, line 37 echoes 'HPCADVISORVAR APPEXECTIME=$LAMMPSCLOCKTIME', but the variable assigned on line 34 is APPEXECTIME, not LAMMPSCLOCKTIME. As written, the HPCADVISORVAR output would contain an empty execution time, which would corrupt the advice data for the LAMMPS example. Please fix the variable name to match (either echo $APPEXECTIME or rename the variable) and verify that the example output in Listing 4 is reproducible.","section":"Section III-A, Listing 2"}],"minor_comments":[{"comment":"The configuration file uses 'ppr: 100' (described as a percentage), but Table I lists PPN as 'Processes per node' and Listing 2 uses the variable $PPN. The relationship between the user-supplied percentage and the environment variable is not explained; please clarify the naming and semantics.","section":"Table I"},{"comment":"The long 'source /cvmfs/software.eessi.io/versions/2023.06/init/bash' command is line-wrapped in the manuscript. Please ensure it appears on a single line so that the example is directly executable.","section":"Section III-A, Listing 2"},{"comment":"In Figure 3, the x-axis is labeled 'Execution time (seconds)' and the y-axis 'Cost (USD)', but the caption says 'Execution Time vs. Cost'. The figure is visually clear, but the axis labels should be consistent with the caption style used in other figures (e.g., 'Execution time (seconds)' on the x-axis is fine, just verify the orientation).","section":"Section III-D, Figure 3"},{"comment":"The related work section does not mention how existing resource-selection tools (e.g., Oikonomos and A2Cloud-RF) handle the uncertainty of cloud performance measurements; a brief comparison of their validation methodologies with HPCAdvisor's would strengthen the positioning.","section":"Section II"},{"comment":"The paper uses the term 'Design of Experiments' without a citation; consider adding a reference for readers unfamiliar with the concept.","section":"Section III-F"}],"recommendation":"major_revision","confidential_remarks":"This is a tool/vision paper, and the bar for acceptance likely depends on the venue's tolerance for engineering contributions with informal evaluation. The central claim is modest and the example outputs give some evidence that the tool works, but the manuscript currently contains several concrete technical errors (Algorithm 1 indentation, duplicate YAML keys, and a variable-name mismatch in Listing 2) that would prevent a reader from reproducing the claimed behavior. These are fixable, but they go beyond mere typos and affect the core parameter-sweep and advice-generation description. After these issues are resolved, the larger missing piece is an evaluation of the reliability of the Pareto-front advice under cloud performance variability; the paper should at least discuss this limitation explicitly and provide some evidence of stability, or qualify the claims about the advice accordingly. I would not recommend rejection if the authors are willing to make these revisions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: HPCAdvisor is a legitimate engineering contribution. It wraps Azure Batch provisioning, parameter sweeps, plotting, and Pareto-front advice into one open-source package with a simple YAML-plus-bash interface, and the examples (OpenFOAM, LAMMPS, up to 1,920 cores) show it actually runs end to end. That combination is new for this niche, even though each piece exists in ReFrame, Ramble, Pavilion2, or Oikonomos.\n\nWhat the paper does well: the design is described concretely (config files, environment variables, task-processing algorithm), the related work is fair, and the tool is available on GitHub with examples for WRF, NAMD, OpenFOAM, LAMMPS, and GROMACS. The paper is honest that this is the current implementation and that optimizations are ongoing. No circularity problem—the Pareto front is computed from measured times and costs, not from a fitted model.\n\nWhere it is soft: there is no controlled evaluation. The evidence is example outputs and screenshots. No repeated runs, no confidence intervals, no comparison against an alternative selection method or manual benchmarking effort. This is not fatal for a tool report, but it does hit the central advice feature. Each scenario's execution time is a single point, and cloud VMs are noisy (network contention, CPU steal, neighbor interference). The Pareto front can change if a point is an outlier. The paper says it aims to generate a Pareto front rather than exact times, but the front is still a point estimate from one sample; the user-facing advice could mislead. A short variance study or even a caveat in the advice output would help. Also minor: Listing 1 reuses the key `mesh:` twice, which is invalid YAML, and the cost figures exclude storage and licenses, which is fine as long as users know.\n\nWho gets value: cloud HPC users, especially small teams without benchmarking expertise, and tool developers looking for a starting point. It is a systems/experience paper, not a research breakthrough. It deserves a serious referee—a workshop or tool track would be a good fit. My recommendation: send it to peer review with the expectation of major revision. The revision should add at least a variability analysis on a few scenarios, a comparison or a clear statement that the tool currently gives best-available point estimates, and a fix for the YAML example.","headline":"HPCAdvisor is a genuine open-source engineering contribution for cloud HPC resource selection, but the paper is a tool report whose single-run Pareto advice needs a variability caveat or study.","tokens_in":11196,"tokens_out":2759,"would_cite":true,"duration_ms":25758,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"HPCAdvisor is an open-source tool that fully automates the lifecycle of collecting HPC benchmarking data in the cloud and reduces it to a Pareto front over execution time and cost.","keywords":["HPC","cloud computing","resource selection","Pareto front","performance benchmarking","cost analysis","parameter sweep","Azure Batch"],"falsifier":"Run a scenario that HPCAdvisor places on the Pareto front several times under the same conditions (e.g., the reported OpenFOAM case at 16 hb120rs_v3 nodes, 34 s, $0.544) and measure the spread of execution times. If the spread rivals the gaps between consecutive Pareto-front points, then the front can misorder options and the advice depends on the luck of a single run.","tokens_in":10321,"feed_emoji":"⚙️","tokens_out":5367,"duration_ms":42340,"temperature":0.7,"pith_summary":"Selecting cloud resources for an HPC workload means choosing among many VM types, node counts, and process counts, and the best choice depends on the user's specific application input. HPCAdvisor attempts to settle that choice empirically rather than by guesswork: it automatically provisions a cloud environment, runs every combination the user specifies, collects and organizes the timing data, and then presents the user a Pareto front of the non-dominated options. The paper establishes that this end-to-end automation is feasible, showing working outputs from OpenFOAM and LAMMPS with up to 1,920 cores. The value, if the tool works as described, is that nonexpert users can obtain cost-performance advice for their own input data without building a benchmarking harness.","feed_headline":"Automates cloud HPC benchmarking: returns a cost-time Pareto front","feed_subtitle":"Give it your app, VM list, node counts, and inputs; it executes all combinations and shows the Pareto-optimal options.","key_machinery":"The load-bearing machinery is the HPCAdvisor tool itself, a Python program that wraps a scenario-sweep loop over VM type, node count, processes per node, and application input. The key interface is a contract with the user: a pair of bash functions, hpcadvisor_setup and hpcadvisor_run, the latter of which can emit key-value metrics via 'HPCADVISORVAR' lines captured into the dataset. The tool's back end currently uses Azure Batch to create and resize node pools per VM type, executing setup once per VM type and then running compute tasks per scenario. Advice is produced by computing the Pareto front of the executed scenarios with respect to execution time and cost, so the user is offered a set of non-dominated configurations rather than a single 'best' answer.","core_discovery":"The paper's central claim is that a practical tool can automate the entire data-collection cycle for HPC resource selection in the cloud. Given two user inputs—a YAML specification of cloud subscription, region, VM types, node counts, and application input parameters, plus a bash script defining application setup and run—HPCAdvisor provisions the cloud resources, executes the full parameter sweep, extracts metrics through an 'HPCADVISORVAR' key-value protocol, and records everything in a dataset. From that dataset it automatically draws execution-time-vs-nodes, time-vs-cost, speedup, and efficiency plots, and it distills the advice into a Pareto front: the set of scenarios that are not dominated by any other in both execution time and cost. The paper demonstrates the tool with OpenFOAM and LAMMPS examples, including a LAMMPS run at up to 1,920 cores on InfiniBand-connected VMs, and reports that the resulting advice lists scenarios such as 16-node hb120rs_v3 at 34 seconds and $0.544 for OpenFOAM.","pith_inferences":["A natural extension the paper gestures at but does not develop: the same Pareto-front logic could drive an active-learning loop that proposes a small set of scenarios to run next, treating the front itself as the approximation target.","The replaceable back end means the tool's claims are not tied to Azure; porting the scenario-sweep loop to another provider or an on-prem Slurm cluster would test whether the automation generalizes.","For users with a fixed budget, the front could be re-ranked by a utility function (e.g., willingness to pay per saved second), which the paper leaves to the user's own judgment."],"forward_implications":["Users can obtain a cost-performance shortlist for their own application input in hours of automated cloud time, without hand-building a benchmarking environment.","The tool doubles as a parameter-sweep engine for HPC applications, giving it use beyond single resource-selection queries.","Because the back end is replaceable and the configuration is YAML-plus-bash, the same front end can be pointed at other orchestrators such as Slurm.","The planned 'smart sampling' module aims to prune scenarios that cannot enter the Pareto front, lowering the monetary cost of collecting advice.","With enough accumulated execution data, the vision is to give users a Pareto-front recommendation for a new input with little or no new execution."],"supporting_citations":[{"why":"Establishes that cloud HPC research has focused on performance evaluation and cost analysis, motivating the tool's time/cost criterion.","marker":"[1]"},{"why":"Shows that only a few application input parameters strongly influence run time, the premise for making the tool accept the user's actual inputs.","marker":"[2]"},{"why":"Provides the observation of fixed per-step execution time after warm-up, which the tool's optimization and approximation plans rely on.","marker":"[6]"},{"why":"Argues for using the real application input rather than scaled-down datasets, which is why HPCAdvisor executes the user's actual input in every scenario.","marker":"[13]"}],"fun_headline_variants":["Automates cloud HPC benchmarking to build a cost-time Pareto front","HPCAdvisor runs your app, plots Pareto front for VM and node choice","Tool auto-tests VM types and node counts, returns best options","From YAML and bash script, HPCAdvisor crunches cloud HPC options","Find optimal cloud HPC setups with automated benchmarking"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The advice assumes a scenario's execution time is stable enough to be represented by a single run, without needing repeated trials or variance estimates.","fun_headline_variants_meta":{"raw":{"variants":["Automates cloud HPC benchmarking to build a cost-time Pareto front","HPCAdvisor runs your app, plots Pareto front for VM and node choice","Tool auto-tests VM types and node counts, returns best options","From YAML and bash script, HPCAdvisor crunches cloud HPC options","Find optimal cloud HPC setups with automated benchmarking"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000242,"raw_usage":{"total_tokens":1505,"prompt_tokens":907,"completion_tokens":598,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":504}},"tokens_in":523,"tokens_out":598,"duration_ms":6004,"temperature":1.0,"reasoning_tokens":504,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:16:39.347736+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a scenario that HPCAdvisor places on the Pareto front several times under the same conditions (e.g., the reported OpenFOAM case at 16 hb120rs_v3 nodes, 34 s, $0.544) and measure the spread of execution times. If the spread rivals the gaps between consecutive Pareto-front points, then the front can misorder options and the advice depends on the luck of a single run.","supporting_citations":[{"cited_title":"HPC cloud for scientific and business applications: taxon- omy, vision, and research challenges,","cited_arxiv_id":null,"evidence_quote":"Establishes that cloud HPC research has focused on performance evaluation and cost analysis, motivating the tool's time/cost criterion."},{"cited_title":"Evaluating HPC job run time predictions using application input parameters,","cited_arxiv_id":null,"evidence_quote":"Shows that only a few application input parameters strongly influence run time, the premise for making the tool accept the user's actual inputs."},{"cited_title":"Cross-platform performance prediction of parallel applications using partial execution,","cited_arxiv_id":null,"evidence_quote":"Provides the observation of fixed per-step execution time after warm-up, which the tool's optimization and approximation plans rely on."},{"cited_title":"Selecting efficient cloud resources for HPC workloads,","cited_arxiv_id":null,"evidence_quote":"Argues for using the real application input rather than scaled-down datasets, which is why HPCAdvisor executes the user's actual input in every scenario."}],"review_version":1}