Efficient and Reuseable Cloud Configuration Search Using Discovery Spaces

Alessandro Pomponio; Burkhard Ringlein; Christian Pinto; Christoph Hagleitner; Michael Johnston; Srikumar Venugopal; Vassilis Vassiliadis

REVIEW 4 major objections 6 minor 37 references

Efficient and Reuseable Cloud Configuration Search Using Discovery Spaces

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A Discovery Space data model plus representative sub-space comparison lets cloud configuration searches reuse and transfer data, cutting search cost by over 90%.

desk verdict The Discovery Space abstraction and RSSC transfer method are genuinely useful, but the 'over 90% speed-up' headline only holds against brute-force enumeration; against the paper's own optimizer baselines it drops to roughly 50-55%. read the letter →

arxiv 2506.21467 v1 pith:OC2HU7EU submitted 2025-06-26 cs.DC

Michael Johnston , Burkhard Ringlein , Christoph Hagleitner , Alessandro Pomponio , Vassilis Vassiliadis , Christian Pinto , Srikumar Venugopal This is my paper · ORCID

classification cs.DC

keywords configurationsearchcloudresourceoptimizationdataabstractionknowledgetransfersurrogatemodelblack-boxLLMinferencebiganalytics

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the high cost of cloud configuration search can be cut dramatically by treating search data as a shareable, structured resource rather than as a private by-product of each optimizer run. It proposes Discovery Space, a data model that wraps a workload's configuration space, its measurable properties, and all acquired samples into one object with a common schema, so multiple optimizers can read and add to the same store without interfering. On top of that model it builds representative sub-space comparison (RSSC), which selects a few representative configurations from a well-studied source space, measures their counterparts in a related target space, and uses a linear regression to decide whether knowledge transfers and to predict the unmeasured points. In the paper's evaluation, reusing samples across runs saves up to 94% of optimization cost, and RSSC transfers knowledge across similar workloads and GPU setups with up to 92% savings. A sympathetic reader would care because configuration spaces with millions of options make exhaustive measurement impossible, so any safe, automatically checked reuse directly reduces time and money.

What carries the argument

The load-bearing object is the Discovery Space data model, defined as a tensor product of a configuration probability space and an action space, implemented as a class with four elements: the probability-space definition, the action-space definition, a shared sample store, and time-stamped sampling records. Its work is to give every configuration search a common schema so samples are reconcilable, actionability is encoded in the space itself, and operations are stateless. The second mechanism is representative sub-space comparison (RSSC): cluster source samples by the property to transfer, take one centroid per cluster, translate those configurations to the target space, measure them, fit a linear regression between source and target values, and only if $r>0.7$ and the slope p-value is below 0.01 use that regression as a surrogate to fill in the rest of the target space. The same linear fit is therefore doing double duty as a transferability test and as the predictive model.

What would settle it

Run RSSC on a pair of spaces whose true relationship is known to be strongly nonlinear, for example the SI-TRANS case (A100 PCIE to A100 SXM4) where the paper reports $r=0.29$ and a failed transfer; the method's central claim would be falsified if, across many such pairs, the go/no-go criteria frequently pass while the predicted top-5 and rank resolution remain poor, or if they frequently fail while a nonlinear surrogate would have transferred well.

Watch

Extended reading notes

Core claim

The central discovery is that a configuration search problem can be represented as a Discovery Space $D = (P,\Omega)\otimes A$, where $(\Omega, P)$ is the configuration probability space and $A$ the action space of measurable experiments, and that this representation makes data reuse safe: a shared sample store with one schema means any sample written by one operation can be read by another as long as the configuration is generated through that space's sample method. The paper further claims that when two spaces differ by a small change (a different GPU model, a different LLM, a different attachment type), a handful of representative points, chosen by clustering the source space and translated by a parameter mapping, is enough to decide transferability. The transfer criteria are a correlation $r > 0.7$ and a slope p-value below 1% for the linear regression between source and target representative values; when these pass, the same regression serves as a surrogate model that predicts every unmeasured target configuration. The paper reports that in cases where the criteria pass, the transferred model identifies the best or near-best configuration, 60--100% of the true top-5, and a rank resolution of 1--3, with measured sampling savings up to 92%.

Load-bearing premise

The method assumes that a linear relationship fitted on a small set of representative points (4 to 33 samples in the experiments) holds across the whole target configuration space, so the same regression can serve both as the transfer test and as the predictor for all unmeasured configurations.

Editorial extensions

If this is right

With passive incremental sampling, repeated independent optimizer runs on the same Discovery Space reuse earlier measurements, saving 20% of runtime by the 10th run on the largest tested spaces and up to 94% on the smallest.
When the RSSC transfer criteria pass, the linear surrogate predicts the best or near-best configuration, retrieves 60--100% of the true top-5 configurations in the top-5 predictions, and requires far fewer samples than running an optimizer on the target space.
Because no single optimizer dominates across workloads (BOHB wins for MULTIINF, Bayesian optimization for SINGLEINF, Ax for TPCDS), a workload-agnostic data model that lets multiple optimizers share one sample store is a practical necessity, not just a convenience.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper's own SI-TRANS result is the productive failure case: it shows RSSC can flag when transfer is unsafe, but the criteria are tuned to linear relationships, so strongly monotonic but nonlinear mappings might be rejected even though a rank-preserving transfer could still help.
The same Discovery Space schema could be used to detect configuration-space drift over time, because sampling records are time-stamped; one could compare an old space to a re-measured sub-space and use RSSC's criteria to decide whether cached predictions are still valid.
RSSC's representative points could be chosen adversarially rather than by global clustering; the paper's top5 baseline already shows local selection can give false positives, so a natural extension is a point-selection rule that minimizes worst-case prediction error rather than cluster representativeness.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The one thing you should know: the abstract's 'configuration search speed-ups of over 90%' is not supported by the evaluation as stated. Table VI defines the savings as time saved over brute-force enumeration of all configurations. But no one runs a brute-force search. The paper's own optimizer baselines in Table V show optimizers sampling well under 2% of each space, and Section V-D says optimizers can sample up to 2x more points than RSSC (9 vs. 4 for FT-TRANS). Against that realistic baseline, the savings are around 50-55%, not 92%. The headline overstates the practical benefit. That said, the paper does real work. The TRACE characteristics and Discovery Space data model are a clean, sensible abstraction for workload-agnostic, shareable configuration search. They resemble Vizier/Optuna studies and trials, but the common-context schema and explicit reconciliation rules are a useful contribution. RSSC - clustering to pick representative points, a go/no-go linear-transfer criterion, and a surrogate built on the same fit - is a reasonable, simple transfer method, and its application to cross-space configuration transfer appears new in the cited literature. The evaluation covers four real workloads (TPC-DS, two LLM inference setups, fine-tuning) with exhaustively characterized spaces, and the SI-TRANS negative result is reported honestly, which I credit. The soft spots beyond the headline are fixable but real. First, the same linear regression is used both to decide transferability and as the surrogate predicting all unmeasured points. The paper does not state whether the representative points are excluded from the reported best% and top5% metrics. If they are not excluded, those metrics are inflated by construction. That needs to be clarified. Second, Table VI has no error bars or variance across clustering runs; the transfer criteria thresholds (r > 0.7, p < 0.01) are also arbitrary, though that is a minor point. Third, no code or data are released, which makes the transfer results hard to verify. The equal-time cost model is stated explicitly, but it is a simplification that could overstate savings when measurements vary in duration. The linear-transfer assumption is a genuine risk, but the go/no-go criterion partially mitigates it - SI-TRANS fails the test and no transfer is attempted. That is the right behavior. For whom? Cloud systems researchers working on configuration tuning, performance modeling, and transfer learning. It deserves a serious referee, but the authors need to rework the savings claim, release artifacts, add uncertainty estimates, and state whether representative points are excluded. I would send it to review, expecting major revisions.

Referee Report

4 major / 6 minor

Summary. The paper proposes Discovery Space, a data-model abstraction for configuration search that formalizes configuration spaces, action spaces, sample stores, and sampling records, with the goal of enabling workload-agnostic search, transparent data sharing, and knowledge reuse. It also introduces representative sub-space comparison (RSSC), a method that selects a small set of representative configurations from a source space, tests a linear transferability criterion against a related target space, and, if the criterion passes, fits a linear surrogate to predict the remaining target configurations. The evaluation covers four workloads (TPC-DS, two LLM inference tasks, and LLM fine-tuning) and three transfer scenarios, claiming configuration search speed-ups of over 90% in the abstract and reporting savings of 86-92% in Table VI.

Significance. The paper's main strength is the Discovery Space abstraction, which is clearly described and useful in principle: it decouples optimization algorithms from workload-specific experiments and provides a structured way to share measurements across runs. The incremental-sampling evaluation (Fig. 7) gives concrete evidence that reusing prior samples can reduce optimization cost, and the honest reporting of the SI-TRANS negative case is commendable. However, the headline quantitative claim is not supported as stated: the 'over 90% speed-up' is measured against brute-force enumeration, not against an optimizer baseline, and the authors themselves note that their optimizer baselines sample up to 2x more points than RSSC. The transfer evaluation also lacks statistical rigor (no error bars, unclear whether fit points are held out). If the claims are re-framed and the evaluation is tightened, the work could be a useful systems contribution, but as it stands the central claim overstates the benefit.

major comments (4)

[Abstract and Section V-D, Table VI] The abstract's claim of 'configuration search speed-ups of over 90%' is not supported by the evaluation as reported. The %savings in Table VI is defined in the caption as 'time-saved over a brute force evaluation of all configurations,' not over an existing optimizer baseline. In Section V-D the authors state that for FT-TRANS and MI-TRANS, optimizer runs 'can sample up to 2x more points than RSSC ... e.g. 9 versus 4 for FT-TRANS.' Against that baseline, the savings are at most (9-4)/9 ≈ 55% for FT-TRANS and roughly 50% for MI-TRANS, rather than 86-92%. Please re-state the speed-up claims relative to the paper's own optimizer baselines, or explicitly qualify that the 90%+ figure refers only to avoiding brute-force enumeration.
[Section V-B2 and Table VI] The evaluation does not state whether the representative sub-space points used to fit the linear surrogate are excluded from the reported best%, top5%, and rank-resolution metrics. Because the surrogate is fit on those points, including them in the quality metrics can artificially inflate the results (for example, FT-TRANS reports best%=100% with 8 representative points out of 56). Please clarify whether the metrics are computed on held-out points only, and if not, recompute them on the non-representative configurations.
[Section IV-3 and IV-4] The transfer criteria (r>0.7, p<0.01) and the surrogate model are the same linear regression fit on the representative points. This means the go/no-go transfer decision is not validated independently of the predictor. The paper shows one negative case (SI-TRANS) and two positive cases, but does not demonstrate that the thresholds are predictive of held-out prediction quality in general. Given that only 4-33 points are used for the fit, the risk of overfitting is non-negligible. Please provide a validation of the transfer criteria against held-out data, or at least discuss this limitation explicitly.
[Table VI and Section V-D] The transfer evaluation reports single values without any measure of variance or number of repetitions for the clustering-based RSSC. The optimizer baselines are run 10 times, but Table VI appears to report results from a single RSSC execution per test case. For a claim of this magnitude, confidence intervals or repeated runs are needed. Additionally, the three transfer test spaces are small (48-288 configurations), and only two of the three transfers succeed, so the 'over 90%' claim rests on a very narrow empirical base.

minor comments (6)

[Title] The title contains a typo: 'Reuseable' should be 'Reusable'.
[Section V-B1] The sentence 'We used three three optimization methods' contains a duplicated 'three'; please remove the duplicate.
[Section VI] The statement 'there have been no application of these frameworks to configuration search' is contradicted by the same section's discussion of CherryPick [5] and Morphling [7], which apply Bayesian optimization to configuration search; reword to 'no general-purpose framework has been widely applied' or similar.
[Figure 1] The x-axis labels appear to be instance types (e.g., c4.gm16Gi), but the caption describes 'increasing number of CPU cores and GPU memory allocation'; please clarify how the x-axis encodes these parameters.
[Section III-B1] The notation D=(P,Ω)⊗A is introduced, but the action space A and the Discovery Space A in Figure 5 share the same symbol; consider using a different symbol for the action space in the formal definition to avoid confusion.
[General reproducibility] No code or data are released, which limits the reproducibility of the reported results; please consider making the implementation and benchmark data available or provide a detailed appendix.

Circularity Check

1 steps flagged · score 6.0 of 10

Reported best% metric in RSSC evaluation reduces to the best measured representative point, making the transfer 'prediction' partly in-sample.

fitted input called prediction [Section V-B2 (metrics definition) and Table VI (MI-TRANS row)]
"we evaluated for the various scenarios tested: (1) the performance percentile of the best configuration of the predictive model (best%); ... Table VI: MI-TRANS cum latency (ms) clustering 4 1.0 0.00014 ✓ 80.4% 60% 1 92%"

The best% metric is reported as the performance percentile of the best configuration 'of the predictive model'. For MI-TRANS cum latency, the linear surrogate has r=1.0 and rank resolution 1, so an accurate out-of-sample predictor should place the true best at or near the top, giving best% near 100. Instead the reported value is 80.4%, which is exactly the expected maximum percentile of 4 uniformly sampled points (4/5 = 80%). This indicates that best% is the best among the 4 measured representative points—i.e., the data used to fit the regression—not a prediction for unmeasured configurations. The paper never states that the representative fit points are excluded from best%/top5%, and the numbers show they are not. The prediction-quality metric therefore reduces to the fit input.

full rationale

The Discovery Space data model and the incremental-sampling evaluation are self-contained and do not depend on self-citation; those contributions are not circular. The RSSC transfer method is an empirical fit, and using the same linear relationship as both the transfer criterion and the surrogate is an ansatz rather than a formal derivation. However, the central evaluation of RSSC's prediction quality is partially circular: Table VI's best% values are, by the paper's own numbers, the best among the measured representative points rather than the best configuration predicted for unmeasured points. The MI-TRANS cum latency case (r=1.0, rank resolution 1, yet best%=80.4%, the expected maximum of 4 samples) demonstrates the metric scores the fit points themselves. Thus the headline transfer-quality claims are partly training-set statistics. The 'over 90% speed-up' figure is computed against brute-force enumeration rather than the paper's own optimizer baselines, which is a baseline-choice concern but not a circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 1 invented entities

The central results depend on the linear-transfer assumption (ad hoc), the unproven TRACE sufficiency claim, the equal-measurement-time simplification, and unreleased ground-truth data. The free parameters are the hand-set transfer thresholds, the fitted linear coefficients, and the cluster-based representative-set size.

free parameters (3)

Transfer criteria thresholds (r, p) = r > 0.7; p < 0.01
Hand-set in Section IV-3 as the go/no-go rule for knowledge transfer. No sensitivity analysis is reported, and these thresholds control when the 92% savings are claimed.
Linear surrogate model coefficients (slope, intercept) = Not reported; r values in Table VI
Fit to representative source and target samples in Section IV-4 and used to predict all remaining target configurations. Prediction quality and reported savings depend on these fitted values.
Representative sub-space size (cluster count) = 4 to 33 points across test cases
Selected via silhouette clustering in Section IV-2 rather than hand-set, but it determines the size of the linear fit and the number of target measurements required; its effect is not analyzed.

assumptions (5)

ad hoc to paper A linear relationship fitted on representative points generalizes to the entire target space.
Section IV-3 and IV-4: the transfer criteria and the surrogate model are both linear, and SI-TRANS demonstrates failure when the relationship is weak.
domain assumption The five TRACE characteristics are necessary and sufficient for transparent, safe sharing and reuse.
Section III-A asserts this without proof or formal analysis; it is a design claim, not a demonstrated property.
domain assumption All measurements take equal time, so normalized cost can be expressed as new samples divided by total samples.
Section V-B1 states this for simplicity; it is unrealistic across workloads such as TPC-DS queries and LLM fine-tuning and directly drives the reported time-savings percentages.
domain assumption Exhaustive characterization of each configuration space gives the true best configurations and CDFs.
Section V-A1 says spaces were exhaustively characterized, but raw measurements are not released and measurement noise or repeatability is not reported.
domain assumption Silhouette clustering on the property to be transferred yields points representative of the global source-target relationship.
Section IV-2: this is the basis for selecting the representative sub-space, with no proof that cluster centroids are optimal for transfer.

invented entities (1)

Discovery Space data-model (D = (P, Omega) tensor A) independent evidence
purpose: Formal abstraction for representing configuration search spaces, samples, and applicable measurements, enabling shared and reusable search data.
Implemented and evaluated on four workloads with a SQL store and Ray Tune wrapper, so it is not an unfalsifiable postulate. Its claimed workload-agnosticism and safety rest on a small, unreleased evaluation.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Efficient and Reuseable Cloud Configuration Search Using Discovery Spaces." pith.science (2026). https://pith.science/paper/OC2HU7EU

@misc{pith2026250621467,
  author       = {Pith},
  title        = {Pith review of: Efficient and Reuseable Cloud Configuration Search Using Discovery Spaces},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OC2HU7EU}},
  note         = {Machine review of arXiv:2506.21467}
}

read the original abstract

Finding the optimal set of cloud resources to deploy a given workload at minimal cost while meeting a defined service level agreement is an active area of research. Combining tens of parameters applicable across a large selection of compute, storage, and services offered by cloud providers with similar numbers of application-specific parameters leads to configuration spaces with millions of deployment options. In this paper, we propose Discovery Space, an abstraction that formalizes the description of workload configuration problems, and exhibits a set of characteristics required for structured, robust and distributed investigations of large search spaces. We describe a concrete implementation of the Discovery Space abstraction and show that it is generalizable across a diverse set of workloads such as Large Language Model inference and Big Data Analytics. We demonstrate that our approach enables safe, transparent sharing of data between executions of best-of-breed optimizers increasing the efficiency of optimal configuration detection in large search spaces. We also demonstrate how Discovery Spaces enable transfer and reuse of knowledge across similar search spaces, enabling configuration search speed-ups of over 90%.

Figures

Figures reproduced from arXiv: 2506.21467 by the authors.

**Figure 3.** A view of a data object representing a Discovery Space. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

**Figure 4.** A view of data sharing between two Discovery Spaces. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗

Figures from the paper (3 more)

**Figure 5.** Figure 5: The representative sub-space comparison knowledge transfer method. For simplicity this figure shows a change to the configuration space, however it could be to the action space or to both. Workload Agnostic and Multiple Optimization Algorithms: In order to be used in a…

**Figure 6.** Figure 6: Comparison of optimizer performance on the TP-OPT [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

**Figure 7.** Figure 7: Comparison of the impact of re-using existing con [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

37 extracted references · 23 canonical work pages

[1]

OPTIMUSCLOUD: Heterogeneous Con- figuration Optimization for Distributed Databases in the Cloud,

A. Mahgoub et al., “OPTIMUSCLOUD: Heterogeneous Con- figuration Optimization for Distributed Databases in the Cloud,” presented at the 2020 USENIX Annual Technical Conference (USENIX ATC 20), 2020, pp. 189–203

work page 2020
[2]

BestConfig: Tapping the performance potential of systems via automatic configuration tuning,

Y . Zhu et al., “BestConfig: Tapping the performance potential of systems via automatic configuration tuning,” inProceedings of the 2017 Symposium on Cloud Computing (SoCC 17), Sep. 24, 2017, pp. 338–350.DOI: 10.1145/3127479.3128605

arXiv 2017
[3]

Wei et al.,Finetuned Language Models Are Zero-Shot Learners, arXiv:2109.01652 [cs], Feb

J. Wei et al.,Finetuned Language Models Are Zero-Shot Learners, arXiv:2109.01652 [cs], Feb. 2022

arXiv 2022
[4]

Search-based Methods for Multi-Cloud Configuration,

M. Lazuka et al., “Search-based Methods for Multi-Cloud Configuration,” in2022 IEEE 15th International Conference on Cloud Computing (CLOUD), ISSN: 2159-6190, Jul. 2022, pp. 438–448.DOI: 10.1109/CLOUD55607.2022.00067

arXiv 2022
[5]

CherryPick: Adaptively unearthing the best cloud configurations for big data analytics,

O. Alipourfard et al., “CherryPick: Adaptively unearthing the best cloud configurations for big data analytics,” inNSDI, vol. 2, 2017, pp. 4–2

work page 2017
[6]

Forecasting a storm: Divining optimal configurations using genetic algorithms and supervised learning,

M. Trotter, T. Wood, and J. Hwang, “Forecasting a storm: Divining optimal configurations using genetic algorithms and supervised learning,” in2019 IEEE International Conference on Autonomic Computing (ICAC), Jun. 2019, pp. 136–146. DOI: 10.1109/ICAC.2019.00025

arXiv 2019
[7]

Morphling: Fast, near-optimal auto- configuration for cloud-native model serving,

L. Wang et al., “Morphling: Fast, near-optimal auto- configuration for cloud-native model serving,” inProceedings of the ACM Symposium on Cloud Computing, ser. SoCC ’21, New York, NY , USA: Association for Computing Machinery, 2021, pp. 639–653.DOI: 10.1145/3472883.3486987

work page arXiv 2021
[8]

Selecting the best VM across multiple public clouds: A data-driven performance modeling approach,

N. J. Yadwadkar et al., “Selecting the best VM across multiple public clouds: A data-driven performance modeling approach,” inProceedings of the 2017 Symposium on Cloud Computing, ser. SoCC ’17, New York, NY , USA: Association for Computing Machinery, Sep. 24, 2017, pp. 452–465.DOI: 10.1145/3127479.3131614

work page arXiv 2017

Show all 37 references

[9]

Open Source Vizier: Distributed Infrastructure and API for Reliable and Flexible Blackbox Optimization,

X. Song et al., “Open Source Vizier: Distributed Infrastructure and API for Reliable and Flexible Blackbox Optimization,” 2022, Publisher: arXiv Version Number: 2.DOI: 10.48550/ ARXIV .2207.13676

2022 doi arXiv
[10]

Optuna: A Next-generation Hyperparameter Optimization Framework,

T. Akiba et al., “Optuna: A Next-generation Hyperparameter Optimization Framework,” en, inProceedings of the 25th ACM SIGKDD International Conference on Knowledge Dis- covery & Data Mining, Anchorage AK USA: ACM, Jul. 2019, pp. 2623–2631.DOI: 10.1145/3292500.3330701

2019
[11]

BOAH: A Tool Suite for Multi-Fidelity Bayesian Optimization & Analysis of Hyperparameters

M. Lindauer et al. “BOAH: A Tool Suite for Multi-Fidelity Bayesian Optimization & Analysis of Hyperparameters.” arXiv: 1908.06756[cs, stat], Accessed: Sep. 4, 2023. [Online]. Available: http://arxiv.org/abs/1908.06756 preprint

1908 arXiv
[12]

Google Vizier: A Service for Black-Box Optimization,

D. Golovin et al., “Google Vizier: A Service for Black-Box Optimization,” inProceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ser. KDD ’17, New York, NY , USA: Association for Computing Machinery, Aug. 13, 2017, pp. 1487–1495.D...

2017
[13]

Liaw et al.,Tune: A Research Platform for Distributed Model Selection and Training, arXiv:1807.05118 [cs, stat], Jul

R. Liaw et al.,Tune: A Research Platform for Distributed Model Selection and Training, arXiv:1807.05118 [cs, stat], Jul. 2018.DOI: 10.48550/arXiv.1807.05118

2018 doi arXiv
[14]

A benchmark suite and performance analy- sis of user-space provenance collectors,

S. Grayson et al., “A benchmark suite and performance analy- sis of user-space provenance collectors,” inProceedings of the 2nd ACM Conference on Reproducibility and Replicability, ser. ACM REP ’24, New York, NY , USA: Association for Computing Machinery, Jul. 2024, pp. 85–95....

2024
[15]

23, 2023

Various,Spark on Kubernetes Benchmark utility, https://github.com/aws-samples/emr-on-eks-benchmark accessed on 2024-01-16, AWS Samples, Nov. 23, 2023

2024
[16]

Spark: Cluster Computing with Working Sets,

M. Zaharia et al., “Spark: Cluster Computing with Working Sets,” Jun. 2010

2010
[17]

HuggingFace,Text generation inference server: Large lan- guage model text generation inference, 2023

2023
[18]

Scikit-optimize Sequential model-based optimization in Python

scikit-optimize community. “Scikit-optimize Sequential model-based optimization in Python.” https://scikit- optimize.github.io/stable/, accessed 2023-06-02

2023
[19]

Adaptive experimentation platform

Meta Platforms, Inc. “Adaptive experimentation platform.” https://ax.dev accessed on 2023-06-02

2023
[20]

Do the best cloud configurations grow on trees?: An experimental evaluation of black box algorithms for optimizing cloud workloads,

M. Bilal et al., “Do the best cloud configurations grow on trees?: An experimental evaluation of black box algorithms for optimizing cloud workloads,” en,Proceedings of the VLDB Endowment, vol. 13, no. 12, pp. 2563–2575, Aug. 2020.DOI: 10.14778/3407790.3407845

2020
[21]

Selecta: Heteroge- neous cloud storage configuration for data analytics,

A. Klimovic, H. Litz, and C. Kozyrakis, “Selecta: Heteroge- neous cloud storage configuration for data analytics,” in2018 {USENIX} Annual Technical Conference ({USENIX}{ATC} 18), 2018, pp. 759–773

2018
[22]

Ernest: Efficient performance pre- diction for large-scale advanced analytics,

S. Venkataraman et al., “Ernest: Efficient performance pre- diction for large-scale advanced analytics,” in13 th USENIX Symposium on Networked Systems Design and Implementation (NSDI 16), 2016, pp. 363–378

2016
[23]

Arrow: Low-Level Augmented Bayesian Op- timization for Finding the Best Cloud VM,

C.-J. Hsu et al., “Arrow: Low-Level Augmented Bayesian Op- timization for Finding the Best Cloud VM,” in2018 IEEE 38th International Conference on Distributed Computing Systems (ICDCS), ISSN: 2575-8411, Jul. 2018, pp. 660–670.DOI: 10. 1109/ICDCS.2018.00070

2018
[24]

Resource Configuration Tuning for Stream Data Processing Systems via Bayesian Optimization,

S. Huang et al., “Resource Configuration Tuning for Stream Data Processing Systems via Bayesian Optimization,”Intel- ligent Computing, vol. 2022, pp. 2022/9820424, Jan. 2022. DOI: 10.34133/2022/9820424

2022 doi
[25]

ConEx: Efficient Exploration of Big-Data System Configurations for Better Performance,

R. Krishna et al., “ConEx: Efficient Exploration of Big-Data System Configurations for Better Performance,”IEEE Trans- actions on Software Engineering, vol. 48, no. 3, pp. 893–909, Mar. 2022.DOI: 10.1109/TSE.2020.3007560

2022
[26]

Spark-based Cloud Data Analytics using Multi-Objective Optimization,

F. Song et al., “Spark-based Cloud Data Analytics using Multi-Objective Optimization,”2021 IEEE 37th International Conference on Data Engineering (ICDE), pp. 396–407, Apr. 2021, Conference Name: 2021 IEEE 37th International Con- ference on Data Engineering (ICDE) ISBN: 9781728...

2021
[27]

A Survey on Automatic Parameter Tuning for Big Data Processing Systems,

H. Herodotou, Y . Chen, and J. Lu, “A Survey on Automatic Parameter Tuning for Big Data Processing Systems,” en,ACM Computing Surveys, vol. 53, no. 2, pp. 1–37, Mar. 2021.DOI: 10.1145/3381027

2021 doi
[28]

MArk: Exploiting Cloud Services for Cost- Effective, SLO-Aware Machine Learning Inference Serving,

C. Zhang et al., “MArk: Exploiting Cloud Services for Cost- Effective, SLO-Aware Machine Learning Inference Serving,” 2019

2019
[29]

INFaaS: Automated Model-less Inference Serving,

F. Romero et al., “INFaaS: Automated Model-less Inference Serving,” en, in2021 USENIX Annual Technical Conference (USENIX ATC 21), 2021, pp. 397–411

2021
[31]

xCloudServing: Automated ML Serving Across Clouds,

M. Lazuka et al., “xCloudServing: Automated ML Serving Across Clouds,” in2023 IEEE 16th International Conference on Cloud Computing (CLOUD), Jul. 2023, pp. 1–12.DOI: 10.1109/CLOUD60044.2023.00011

2023
[32]

Algorithms for Hyper-Parameter Optimiza- tion,

J. Bergstra et al., “Algorithms for Hyper-Parameter Optimiza- tion,” inAdvances in Neural Information Processing Systems, vol. 24, Curran Associates, Inc., 2011

2011
[33]

BOHB: Robust and Efficient Hyperparameter Optimization at Scale,

S. Falkner, A. Klein, and F. Hutter, “BOHB: Robust and Efficient Hyperparameter Optimization at Scale,” en, inPro- ceedings of the 35th International Conference on Machine Learning, ISSN: 2640-3498, PMLR, Jul. 2018, pp. 1437– 1446

2018
[34]

Elastic Machine Learning Algorithms in Amazon SageMaker,

E. Liberty et al., “Elastic Machine Learning Algorithms in Amazon SageMaker,” en, inProceedings of the 2020 ACM SIGMOD International Conference on Management of Data, Portland OR USA: ACM, Jun. 2020, pp. 731–737.DOI: 10. 1145/3318464.3386126

2020
[35]

Ray: A Distributed Framework for Emerging {AI} Applications,

P. Moritz et al., “Ray: A Distributed Framework for Emerging {AI} Applications,” presented at the 13th USENIX Sym- posium on Operating Systems Design and Implementation (OSDI 18), 2018, pp. 561–577

2018
[36]

Mimir: Finding Cost-efficient Storage Configurations in the Public Cloud,

H. Park, G. R. Ganger, and G. Amvrosiadis, “Mimir: Finding Cost-efficient Storage Configurations in the Public Cloud,” inProceedings of the 16th ACM International Conference on Systems and Storage, ser. SYSTOR ’23, New York, NY , USA: Association for Computing Machinery, Jun. ...

2023
[37]

Titan: A scheduler for foundation model fine- tuning workloads,

W. Gao et al., “Titan: A scheduler for foundation model fine- tuning workloads,” inProceedings of the 13 th Symposium on Cloud Computing, ser. SoCC ’22, New York, NY , USA: Association for Computing Machinery, 2022, pp. 348–354. DOI: 10.1145/3542929.3563460

2022
[38]

BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization,

M. Balandat et al., “BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization,” inAdvances in Neural Information Processing Systems, vol. 33, Curran Associates, Inc., 2020, pp. 21 524–21 538

2020

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

OPTIMUSCLOUD: Heterogeneous Con- figuration Optimization for Distributed Databases in the Cloud,

A. Mahgoub et al., “OPTIMUSCLOUD: Heterogeneous Con- figuration Optimization for Distributed Databases in the Cloud,” presented at the 2020 USENIX Annual Technical Conference (USENIX ATC 20), 2020, pp. 189–203

work page 2020

[2] [2]

BestConfig: Tapping the performance potential of systems via automatic configuration tuning,

Y . Zhu et al., “BestConfig: Tapping the performance potential of systems via automatic configuration tuning,” inProceedings of the 2017 Symposium on Cloud Computing (SoCC 17), Sep. 24, 2017, pp. 338–350.DOI: 10.1145/3127479.3128605

arXiv 2017

[3] [3]

Wei et al.,Finetuned Language Models Are Zero-Shot Learners, arXiv:2109.01652 [cs], Feb

J. Wei et al.,Finetuned Language Models Are Zero-Shot Learners, arXiv:2109.01652 [cs], Feb. 2022

arXiv 2022

[4] [4]

Search-based Methods for Multi-Cloud Configuration,

M. Lazuka et al., “Search-based Methods for Multi-Cloud Configuration,” in2022 IEEE 15th International Conference on Cloud Computing (CLOUD), ISSN: 2159-6190, Jul. 2022, pp. 438–448.DOI: 10.1109/CLOUD55607.2022.00067

arXiv 2022

[5] [5]

CherryPick: Adaptively unearthing the best cloud configurations for big data analytics,

O. Alipourfard et al., “CherryPick: Adaptively unearthing the best cloud configurations for big data analytics,” inNSDI, vol. 2, 2017, pp. 4–2

work page 2017

[6] [6]

Forecasting a storm: Divining optimal configurations using genetic algorithms and supervised learning,

M. Trotter, T. Wood, and J. Hwang, “Forecasting a storm: Divining optimal configurations using genetic algorithms and supervised learning,” in2019 IEEE International Conference on Autonomic Computing (ICAC), Jun. 2019, pp. 136–146. DOI: 10.1109/ICAC.2019.00025

arXiv 2019

[7] [7]

Morphling: Fast, near-optimal auto- configuration for cloud-native model serving,

L. Wang et al., “Morphling: Fast, near-optimal auto- configuration for cloud-native model serving,” inProceedings of the ACM Symposium on Cloud Computing, ser. SoCC ’21, New York, NY , USA: Association for Computing Machinery, 2021, pp. 639–653.DOI: 10.1145/3472883.3486987

work page arXiv 2021

[8] [8]

Selecting the best VM across multiple public clouds: A data-driven performance modeling approach,

N. J. Yadwadkar et al., “Selecting the best VM across multiple public clouds: A data-driven performance modeling approach,” inProceedings of the 2017 Symposium on Cloud Computing, ser. SoCC ’17, New York, NY , USA: Association for Computing Machinery, Sep. 24, 2017, pp. 452–465.DOI: 10.1145/3127479.3131614

work page arXiv 2017