Pith. sign in

REVIEW 4 major objections 7 minor 58 references

Intelligent Pooling: Proactive Resource Provisioning in Large-scale Cloud Service

T0 review · 4 major / 7 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A live pool sized by a linear program and a fast hybrid forecast can cut Spark cluster idle time by 43% at a 99% hit rate.

desk verdict Solid deployed systems paper; the 43% number is real but under-specified, and the FCFS approximation means the optimizer's Pareto curve is biased. read the letter →

arxiv 2411.11326 v1 pith:EQUUGIY7 submitted 2024-11-18 cs.DB

Deepak Ravikumar , Alex Yeo , Yiwen Zhu , Aditya Lakra , Harsha Nagulapalli , Santhosh Kumar Ravindran , Steve Suh , Niharika Dutta
show 6 more authors
Andrew Fogarty Yoonjae Park Sumeet Khushalani Arijit Tarafdar Kunal Parekh Subru Krishnan
This is my paper · ORCID
classification cs.DB
keywords proactiveresourceprovisioningSparkclusterpoolingtime-seriesforecastingsingularspectrumanalysislinearprogrammingsampleaverageapproximationcloudCOGSoptimizationlivepoolsizing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a cloud service can avoid most of the cost of pre-provisioning Spark clusters by predicting aggregate demand and re-solving a small linear program every few minutes. The system, Intelligent Pooling, keeps a live pool of ready clusters, predicts the cluster request rate with a fast hybrid forecaster, and chooses the pool size that minimizes a weighted sum of idle cluster time and customer wait time. Evaluated on production traces, it achieves up to 43% lower cluster idle time than static pooling while still serving 99% of requests instantly. If the claim holds, managed Spark services can offer near-zero startup latency without paying for peak-sized pools, and the same recipe could apply to any resource with a long setup time.

What carries the argument

The load-bearing object is a cumulative curve-gap linear program. With $D(t)$ cumulative demand, $N(t)$ target pool size, and $\tau$ the cluster creation latency, the identities $A(t)=D(t)+N(t)$ and $A'(t)=A(t-\tau)$ translate the pool mechanism into a pure area trade-off: idle time is the positive gap between ready clusters and demand, and wait time is the negative gap. The LP minimizes $\alpha\sum_t\Delta^+(t)+\beta\sum_t\Delta^-(t)$ over the pool-size trajectory, with constraints for stability, capacity bounds, and maximum new requests, and is solved by sample average approximation on forecast demand. The second mechanism is SSA+, which wraps a fast SSA forecaster with a small error-prediction network trained by the same asymmetric loss, giving deep-model-style overshoot control at a fraction of the training cost. Together they convert raw request-rate telemetry into a pool-size recommendation in seconds, allowing continuous re-optimization every five minutes.

What would settle it

Take a recorded production request trace, simulate the true drain behavior (on-demand requests served immediately, re-hydrated clusters given to later requests), and compare the simulated idle and wait totals with the LP's FCFS-computed areas; if the true idle-time reduction over a static pool at a 99% hit rate falls short of 43%, the FCFS approximation is the cause.

Watch

Extended reading notes

Core claim

Intelligent Pooling maintains a constant-size live pool of pre-configured Spark clusters; each customer request takes one cluster and triggers a re-hydration request, so the cumulative re-hydration curve $A(t)$ is the demand curve $D(t)$ shifted up by the current pool size $N(t)$, and the cumulative ready curve $A'(t)$ is $A(t)$ shifted right by the cluster creation latency $\tau$. Idle cluster time is the area where $A'(t) > D(t)$ and customer wait time is the area where $A'(t) < D(t)$. Treating those positive and negative gaps as LP variables $\Delta^+(t)$ and $\Delta^-(t)$, the optimizer chooses $N(t)$ over time to minimize $\alpha\sum_t \Delta^+(t) + \beta\sum_t \Delta^-(t)$, subject to stability, min/max and max-new-request constraints; this is a sample average approximation over the observed or predicted demand. Demand is forecast by SSA+, a hybrid that combines a Singular Spectrum Analysis forecaster with a roughly 30-parameter two-layer neural network that learns the overshoot needed to meet wait-time targets, trained with the same asymmetric loss. On production data, the 2-step pipeline (forecast demand, then optimize) dominates the end-to-end approach, and targeting a 99% pool hit rate the system reports up to 43% idle-time reduction over static pooling, with production COGS savings exceeding 60% in some regions.

Load-bearing premise

The whole optimization assumes clusters are handed to requests in arrival order, but the paper admits that once the pool drains, on-demand requests are served before re-hydrated clusters, so the modeled idle and wait areas, and therefore the optimal pool size and the 43% savings figure, could be miscalibrated.

Editorial extensions

If this is right

  • Static peak-sized pools become unnecessary: a dynamic pool that follows forecast demand keeps the same 99% hit rate while cutting idle clusters by up to 43%.
  • The 2-step pipeline (predict demand, then run the LP) beats training a model directly on historical optimal pool sizes, so forecast-then-optimize is the preferred architecture.
  • Shorter stability intervals in the optimizer improve the Pareto frontier between idle time and wait time, so frequent re-optimization (every few minutes) is worth its compute cost.
  • The same LP applies to session pools as well as cluster pools, extending near-zero startup latency to interactive notebook scenarios.
  • At Fabric scale the savings translate to tens of millions of dollars per year in avoided COGS, with some production regions reporting over 60% cost reduction.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the paper acknowledges that first-come-first-serve is violated when the pool drains (on-demand requests jump the queue), the reported idle-time savings are likely optimistic; a priority-aware LP would be a direct test of how much the reordering matters.
  • The asymmetric loss that penalizes wait time more than idle time is a general pattern for any capacity problem with an SLA, so the SSA+ design could transfer to serverless function pre-warming or GPU instance pooling.
  • The curve-gap LP depends only on aggregate request count and startup latency, not on per-user identity, so it should generalize across regions and workload mixes without retraining per customer.
  • An untested but natural extension is to use the LP's dual prices to value a reduction in $\tau$, giving a concrete dollar target for engineering work on faster cluster creation.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper describes Intelligent Pooling, a production system for proactive provisioning of Spark clusters in Microsoft Fabric. The system has two components: an LP/SAA optimizer that chooses a time-varying pool size to trade off cluster idle time against customer wait time (Eqs. 1-11), and a hybrid ML predictor (SSA+, combining singular spectrum analysis with a shallow neural net) that forecasts demand with low training latency. The authors evaluate the optimizer on historical production data, compare several forecasting models, present wait-time/idle-time Pareto curves, report a 43% reduction in cluster idle time compared to static pooling at a 99% pool hit rate, and report large COGS savings from deployment across production regions.

Significance. If the reported results are reliable, the paper makes a useful practical contribution: it shows how a simple LP with a lightweight forecaster can replace static pooling in a large cloud service, and the deployment is a genuine strength. The LP formulation is transparent and the hybrid SSA+ model is an interesting engineering choice for low-latency retraining. However, the headline quantitative claims are not yet pinned down: the main reported savings number lacks a precisely defined baseline and uncertainty quantification, the FCFS approximation in the optimization model is acknowledged to be violated in exactly the draining regime that matters for the 99% hit-rate target, and the COGS savings in Table 2 and Section 7.5 are order-of-magnitude estimates without a stated cost model. These issues are fixable, but they are load-bearing for the paper's central claims.

major comments (4)
  1. [Section 4.1, footnote 1; Eqs. (4)-(7)] The FCFS approximation is not a harmless ordering detail. Equations (1)-(3) define A'(t) as a shift-right of A(t), and the LP constraints (4)-(7) force Delta+(t) and Delta-(t) to be mutually exclusive, so at any instant the model counts either idle time or wait time, never both. Footnote 1 admits that in production, once the pool drains, on-demand requests are sent to the normal creation path and re-hydrated clusters that become ready later are held for subsequent requests, not used to serve already-waiting customers. Under this actual policy, a re-hydrated cluster can sit idle while earlier on-demand requests are still waiting for their own tau-later clusters; idle and wait coexist. The SAA objective (8) therefore mis-states the cost of a given pool trajectory, and the selected pool size at the 99% hit-rate target, together with the 43% idle-time reduction, is not established. The authors should re-run the optimization with a simulator that implements the actual scheduling policy, or provide production measurements showing that the FCFS approximation changes the resulting Pareto curve negligibly.
  2. [Section 7.3 and abstract] The headline claim of 'up to 43% reduction in cluster idle time compared to static pooling' is not reproducible as stated. The paper never defines the static pooling policy used as the baseline in this comparison. The 'no-intelligence' baseline in Eq. (17) is one candidate (y_hat = gamma * max(y_train)), but gamma is not given and the text does not state that this is the comparator for the 43% figure. The claim also appears without error bars, confidence intervals, or a statement of how many regions/datasets were used and how much the savings varied across them. The authors should specify the baseline pool-size trajectory, the data window, the evaluation protocol, and the variance across regions.
  3. [Table 2 and Section 7.5] The COGS savings are reported as order-of-magnitude ranges ('$>20M', '$>15M', 'on track to save tens of millions', '18% to 64%') without a cost model, a definition of the comparison baseline, or a statement of whether the numbers are measured or extrapolated. For example, Table 2 gives different savings targets for SSA+ and mWDN, but the text does not explain how a dollar figure is obtained from an idle-time reduction, what cluster sizes and regional prices are assumed, or how the 18%-to-64% deployment-range is computed. Without this information, the financial claims cannot be audited. The authors should provide the cost model and the exact mapping from idle-time/wait-time metrics to COGS, or explicitly label these as rough internal estimates.
  4. [Section 7.4 and Table 1] The claim that deployed SSA+ 'reaches similar performance as mWDN' is not supported by the reported numbers. In Table 1, mWDN has the best average MAE (4.59) while SSA+ has 4.91, and SSA+ is not the best model in any of the six rows. The paper also says Table 1 reports RMSE and MAE but only MAE is shown. If the similarity claim is based on the Pareto curves in Figure 5 rather than Table 1, the authors should say so and quantify the difference; otherwise the statement should be softened or removed.
minor comments (7)
  1. [Section 7.3 and Figure 5] Figure 5 is central to the comparison of 2-step and E2E pipelines, but the axes, units, and the exact meaning of each curve are not fully described in the caption or text; please add labels and a legend so the Pareto comparison can be read directly.
  2. [Table 1] The text states that performance is reported in terms of RMSE and MAE, but Table 1 contains only an MAE column; either add RMSE or correct the text.
  3. [Section 7.1] The dataset description says 'East US' in Section 7.1 but Table 1 uses 'East US 2'; please make the region naming consistent and specify whether these are the same region.
  4. [Figure 6] The numerical labels on the bars in Figure 6 are hard to read and poorly formatted (e.g., '10061 54985110 2614'); please improve the figure so the training-time values are legible.
  5. [Section 7.5] There is a typo: 'we run the the pipeline' should be 'we run the pipeline'. Also, the sentence about 'low the end-to-end latency' is missing a verb or article.
  6. [Section 2] The citation '[47?]' appears with a question mark in the background section; please replace it with the correct reference.
  7. [Section 6] The self-tuning mechanism fits a piecewise-linear relation alpha' = f(t_wait) on the previous 10 data points, but no experiments or convergence analysis are shown; a sentence on how this was validated in production would be useful.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claims are empirically evaluated against held-out production data and a static baseline.

full rationale

The paper's derivation chain is an engineering feedback loop rather than a circular one. The SAA optimizer (Section 4) takes demand as input and outputs a pool size trajectory by minimizing a weighted sum of idle and wait areas; the ML predictor (Section 5) is trained on historical request rates with an 80-20 train-test split; the hyper-parameter tuning (Section 6) adjusts alpha' based on observed wait times. The reported 43% idle-time reduction at 99% hit rate is an empirical outcome of comparing the dynamic pooling pipeline against a static pooling baseline on production data, not a quantity that is assumed by the model or that follows from the model by construction. The LP equations (1)-(8) define the cost accounting, but the savings figure is a measured result of applying that accounting to actual demand traces. The two self-citations ([39] and [51]) appear in related-work or motivation contexts and are not load-bearing for the central contribution. The FCFS approximation admitted in footnote 1 is a modeling simplification that could bias the quantitative estimates, but it is a correctness risk rather than a circularity; it does not make the output equivalent to the input. No step reduces by construction to its own inputs, and no fitted parameter is renamed as a prediction in a way that forces the headline result.

Assumptions & free parameters 6 free parameters · 3 assumptions · 0 invented entities

No new physical entities are postulated. The system introduces 'live pools', 'SSA+', and smoothing filters, but these are engineering constructs with clear purposes and no independent evidence requirement beyond the paper's evaluation.

free parameters (6)
  • alpha (alpha-prime) = auto-tuned in [0,1] via piecewise-linear fit to target wait time
    Weights idle time versus wait time in the LP objective (Eqs. 8 and 16); central cost-performance knob.
  • beta = eliminated by setting beta = 1 - alpha-prime
    Originally a second penalty weight in Eq. 8; later removed by reparameterization.
  • STABLENESS = 5 minutes in experiments, 10 minutes in production
    Forces pool size to hold constant for a block of time (Eq. 11); chosen by hand to limit churn.
  • SMOOTHING_FACTOR (SF) = set to tau (cluster creation latency) in production
    Window for the max filter in Eq. 18; chosen manually to fatten spikes.
  • ML training hyperparameters = window 150, epochs 15, batch 768, horizon 1200, lr 0.001, series 1800
    Common for the deep models; tuned on validation set.
  • MAX_NEW_REQUEST, MIN_POOL_SIZE, MAX_POOL_SIZE = not specified, set from regional capacity
    Constraints in Eqs. 9-10; values depend on Azure region capacity.
assumptions (3)
  • domain assumption First-come-first-serve queueing discipline with fixed cluster creation latency tau (Eqs. 1-3)
    Footnote 1 admits the FCFS mapping is violated when the pool drains because on-demand requests are served out of order; the LP objective (Eq. 8) relies on this mapping.
  • domain assumption Demand in the evaluation window is representative of production demand
    The SAA optimizer is run on 14 days of East US Synapse data (Sec 7.1); the paper generalizes to all Fabric regions.
  • domain assumption The weighted sum of idle and wait time is a valid proxy for COGS and customer experience
    The objective in Eq. 8 is used to derive pool sizes and savings; the conversion to dollars in Table 2 is not detailed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Intelligent Pooling: Proactive Resource Provisioning in Large-scale Cloud Service." pith.science (2026). https://pith.science/paper/EQUUGIY7

@misc{pith2026241111326,
  author       = {Pith},
  title        = {Pith review of: Intelligent Pooling: Proactive Resource Provisioning in Large-scale Cloud Service},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EQUUGIY7}},
  note         = {Machine review of arXiv:2411.11326}
}
read the original abstract

The proliferation of big data and analytic workloads has driven the need for cloud compute and cluster-based job processing. With Apache Spark, users can process terabytes of data at ease with hundreds of parallel executors. At Microsoft, we aim at providing a fast and succinct interface for users to run Spark applications, such as through creating simple notebook "sessions" by abstracting the underlying complexity of the cloud. Providing low latency access to Spark clusters and sessions is a challenging problem due to the large overheads of cluster creation and session startup. In this paper, we introduce Intelligent Pooling, a system for proactively provisioning compute resources to combat the aforementioned overheads. To reduce the COGS (cost-of-goods-sold), our system (1) predicts usage patterns using an innovative hybrid Machine Learning (ML) model with low latency and high accuracy; and (2) optimizes the pool size dynamically to meet customer demand while reducing extraneous COGS. The proposed system auto-tunes its hyper-parameters to balance between performance and operational cost with minimal to no engineering input. Evaluated using large-scale production data, Intelligent Pooling achieves up to 43% reduction in cluster idle time compared to static pooling when targeting 99% pool hit rate. Currently deployed in production, Intelligent Pooling is on track to save tens of million dollars in COGS per year as compared to traditional pre-provisioned pools.

Figures

Figures reproduced from arXiv: 2411.11326 by the authors.

Figure 1
Figure 1. Benefits of Intelligent Pooling Generic Job Service Work Item Service Kusto ML Predictor SAA Optimizer Pooling Size Optimization Hyper-parameter Tuning Create Create Fetch Recommendation Intelligent Pooling Worker Pooling Worker Target Pool Size Live Pool Cosmos DB Create/Delete Clusters Request Cluster Service Create/Delete Clusters [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Architecture resources in a given pool, and upon receiving a client request, a pre-provisioned resource can be used instantly. To maintain the target number of resources in the pool, we send a new request, referred to as a re-hydration request, to Generic Job Service to add a new cluster or session back to the pool whenever a pooled re￾source is consumed or fails (due to exceeding a pre-defined lifespan or unexpecte… view at source ↗
Figure 3
Figure 3. Cumulative cluster creation number 𝐷(𝑡), cumu￾lative re-hydration requests 𝐴(𝑡), number of clusters ready 𝐴 ′ (𝑡) at time 𝑡, and pool size at time 0, 𝑁 (0), the total wait time (red area) for obtaining a running cluster and idle time (grey area) for clusters in the pool. For instance, at time 𝑡 = 0, a pool is created with 𝑁 (0) = 4 clusters, and whenever a user request for a new cluster is received, (corresponding t… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Pool size increases advance demand. and adjust for the parameters accordingly such that we can always maintain the optimal balance as desired. To achieve this, we eliminate the 𝛽 hyper-parameter from Equa￾tion 8 and rewrite the objective function as: min 𝛼 ′ · ∑︂ 𝑡 Δ +…
Figure 5
Figure 5. Figure 5: Wait time vs idle time for mWDN, SSA and baseline [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Training time vs input data size. 09-01 00 09-01 03 09-01 06 09-01 09 09-01 12 09-01 15 09-01 18 09-01 21 09-02 00 0 5 10 15 Job Count raw filtered [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

58 extracted references · 57 canonical work pages

  1. [1]

    Zeeshan Ahmed, Saeed Amizadeh, Mikhail Bilenko, Rogan Carr, Wei-Sheng Chin, Yael Dekel, Xavier Dupre, Vadim Eksarevskiy, Senja Filipi, Tom Finley, et al

    work page

  2. [2]

    Amazon. 2022. Amazon A WS. Retrieved July 2, 2022 from https://aws.amazon. com

    work page 2022

  3. [3]

    Apache. 2022. Apache Flink. Retrieved July 2, 2022 from https://flink.apache.org

    work page 2022

  4. [4]

    Apache. 2022. Apache Storm. Retrieved July 2, 2022 from https://storm.apache.org

    work page 2022

  5. [5]

    AWS. 2022. Azure Synapse. Retrieved July 2, 2022 from https://aws.amazon.com/ emr/features/spark/

    work page 2022

  6. [6]

    Amazon AWS. 2022. GluonTS-Probabilistic Time Series Modeling in Python . Re- trieved July 2, 2022 from https://ts.gluon.ai/stable/

    work page 2022

  7. [7]

    Francisco J Baldan, Sergio Ramirez-Gallego, Christoph Bergmeir, Francisco Her- rera, and Jose M Benitez. 2016. A forecasting methodology for workload fore- casting in cloud systems. IEEE Transactions on Cloud Computing 6, 4 (2016), 929–941

    work page 2016

  8. [8]

    Luciano Baresi and Giovanni Quattrocchi. 2018. Towards vertically scalable spark applications. In European Conference on Parallel Processing . Springer, 106–118

    work page 2018

Show all 58 references
  1. [9]

    JV Bibal Benifa and D Dejey. 2019. Rlpas: Reinforcement learning-based proactive auto-scaler for resource provisioning in cloud environment. Mobile Networks and Applications 24, 4 (2019), 1348–1363

    2019

  2. [10]

    Anshuman Biswas, Shikharesh Majumdar, Biswajit Nandy, and Ali El-Haraki

  3. [11]

    Raouia Bouabdallah, Soufiene Lajmi, and Khaled Ghedira. 2016. Use of reactive and proactive elasticity to adjust resources provisioning in the cloud provider. In 2016 IEEE 18th International Conference on High Performance Computing and Communications; IEEE 14th International C...

    2016

  4. [12]

    Yuxing Chen, Jiaheng Lu, Chen Chen, Mohammad Hoque, and Sasu Tarkoma

  5. [13]

    DataBricks. 2022. Best practices: pools for Databricks . Retrieved July 2, 2022 from https://docs.databricks.com/clusters/instance-pools/pool-best-practices.html

    2022

  6. [14]

    Jeffrey Dean and Luiz André Barroso. 2013. The tail at scale. Commun. ACM 56, 2 (2013), 74–80

    2013

  7. [15]

    In Proceed- ings of the 28th ACM International Conference on Information and Knowledge Management

    Cost-effective resource provisioning for Spark workloads. In Proceed- ings of the 28th ACM International Conference on Information and Knowledge Management. 2477–2480

  8. [16]

    Facebook. 2022. Prophet: Forecasting at scale . Retrieved July 2, 2022 from https://facebook.github.io/prophet/

    2022

  9. [17]

    Avrilia Floratou, Ashvin Agrawal, Bill Graham, Sriram Rao, and Karthik Ra- masamy. 2017. Dhalion: self-regulating stream processing in heron. Proceedings of the VLDB Endowment 10, 12 (2017), 1825–1836

    2017

  10. [18]

    Nicole DeHoratius, Adam J Mersereau, and Linus Schrage. 2008. Retail inventory management when records are inaccurate. Manufacturing & Service Operations Management 10, 2 (2008), 257–277

    2008

  11. [19]

    Google. 2022. Google Cloud Platform. Retrieved July 2, 2022 from https://cloud. google.com

    2022

  12. [20]

    Google. 2022. Serverless Spark. Retrieved July 2, 2022 from https://cloud.google. com/dataproc-serverless/docs

    2022

  13. [21]

    Nina Golyandina and Anton Korobeynikov. 2014. Basic singular spectrum analysis and forecasting with R. Computational Statistics & Data Analysis 71 (2014), 934–954

    2014

  14. [22]

    Nikolas Roman Herbst, Nikolaus Huber, Samuel Kounev, and Erich Amrehn

  15. [23]

    Hassan Ismail Fawaz, Benjamin Lucas, Germain Forestier, Charlotte Pelletier, Daniel F Schmidt, Jonathan Weber, Geoffrey I Webb, Lhassane Idoumghar, Pierre- Alain Muller, and François Petitjean. 2020. Inceptiontime: Finding alexnet for time series classification. Data Mining an...

    2020

  16. [24]

    Google. 2022. Spark through Vertex AI . Retrieved July 2, 2022 from https: //cloud.google.com/vertex-ai-workbench

    2022

  17. [25]

    Anton J Kleywegt, Alexander Shapiro, and Tito Homem-de Mello. 2002. The sample average approximation method for stochastic discrete optimization.SIAM Journal on Optimization 12, 2 (2002), 479–502

    2002

  18. [26]

    Yi-Hsuan Lee, Kuo-Chan Huang, Cheng-Hsien Wu, Yen-Hsuan Kuo, and Kuan- Chou Lai. 2017. A Framework for Proactive Resource Provisioning in IaaS Clouds. Applied Sciences 7, 8 (2017), 777

    2017

  19. [27]

    Jinzhao Liu, Yaoxue Zhang, Yuezhi Zhou, Di Zhang, and Hao Liu. 2014. Aggres- sive resource provisioning for ensuring QoS in virtualized environments. IEEE transactions on cloud computing 3, 2 (2014), 119–131

    2014

  20. [28]

    Reihaneh Khorsand, Mostafa Ghobaei-Arani, and Mohammadreza Ramezanpour

  21. [29]

    Software: Practice and Experience 49, 11 (2019), 1618–1642

    A self-learning fuzzy approach for proactive resource provisioning in cloud environment. Software: Practice and Experience 49, 11 (2019), 1618–1642

    2019

  22. [30]

    Microsoft. 2022. Azure Data Explorer - Kusto . Retrieved July 2, 2022 from https://docs.microsoft.com/en-us/azure/data-explorer/kusto/query/

    2022

  23. [31]

    Microsoft. 2022. Azure Fabric. Retrieved July 24, 2023 from https://learn.microsoft. com/en-us/fabric/data-engineering/spark-compute

    2022

  24. [32]

    Microsoft. 2022. Azure HDInsight. Retrieved July 2, 2022 from https://docs. microsoft.com/en-us/azure/hdinsight/spark/apache-spark-overview

    2022

  25. [33]

    Tania Lorido-Botran, Jose Miguel-Alonso, and Jose A Lozano. 2014. A review of auto-scaling techniques for elastic applications in cloud environments. Journal of grid computing 12, 4 (2014), 559–592

    2014

  26. [34]

    Qinghua Lu, Liming Zhu, Xiwei Xu, Len Bass, Shanshan Li, Weishan Zhang, and Ning Wang. 2014. Mechanisms and architectures for tail-tolerant system operations in cloud. In6th{USENIX} Workshop on Hot Topics in Cloud Computing (HotCloud 14)

    2014

  27. [35]

    Microsoft. 2022. Microsoft Azure. Retrieved July 2, 2022 from https://azure. microsoft.com

    2022

  28. [36]

    Microsoft. 2022. NimbusMLbu. Retrieved July 2, 2022 from https://docs.microsoft. com/en-us/nimbusml/overview

    2022

  29. [37]

    Peter P Nghiem and Silvia M Figueira. 2016. Towards efficient resource provi- sioning in MapReduce. J. Parallel and Distrib. Comput. 95 (2016), 29–41

    2016

  30. [38]

    Microsoft. 2022. Azure Synapse. Retrieved July 2, 2022 from https://docs.microsoft. com/en-us/azure/synapse-analytics/spark/apache-spark-overview

    2022

  31. [39]

    Microsoft. 2022. Introduction to Cosmos DB . Retrieved July 2, 2022 from https: //docs.microsoft.com/en-us/azure/cosmos-db/introduction

    2022

  32. [40]

    Olga Poppe, Qun Guo, Willis Lang, Pankaj Arora, Morgan Oslake, Shize Xu, and Ajay Kalhan. 2022. Moneyball: proactive auto-scaling in Microsoft Azure SQL database serverless. PVLDB 15, 6 (2022), 1279–1287

    2022

  33. [41]

    Sabidur Rahman, Tanjila Ahmed, Minh Huynh, Massimo Tornatore, and Biswanath Mukherjee. 2018. Auto-scaling VNFs using machine learning to improve QoS and reduce cost. In 2018 IEEE International Conference on Commu- nications (ICC). IEEE, 1–6

    2018

  34. [42]

    Jianfei Ruan, Qinghua Zheng, and Bo Dong. 2015. Optimal resource provision- ing approach based on cost modeling for spark applications in public clouds. In Proceedings of the Doctoral Symposium of the 16th International Middleware Conference. 1–4

    2015

  35. [43]

    Yoori Oh, Jieun Choi, Eunjung Song, Moonji Kim, and Yoonhee Kim. 2016. A SLA-based Spark cluster scaling method in cloud environment. In 2016 18th Asia-Pacific Network Operations and Management Symposium (APNOMS) . IEEE, 1–4

    2016

  36. [44]

    Olga Poppe, Tayo Amuneke, Dalitso Banda, Aritra De, Ari Green, Manon Knoertzer, Ehi Nosakhare, Karthik Rajendran, Deepak Shankargouda, Meina Wang, Alan Au, Carlo Curino, Qun Guo, Alekh Jindal, Ajay Kalhan, Morgan Oslake, Sonia Parchani, Vijay Ramani, Raj Sellappan, Saikat Sen,...

  37. [45]

    Wikipedia. 2023. Pareto front. https://en.wikipedia.org/wiki/Pareto_front

    2023

  38. [46]

    Timothy Wood, KK Ramakrishnan, Prashant Shenoy, and Jacobus Van der Merwe

  39. [47]

    Matei Zaharia, Mosharaf Chowdhury, Michael J Franklin, Scott Shenker, and Ion Stoica. 2010. Spark: Cluster computing with working sets. In 2nd USENIX Workshop on Hot Topics in Cloud Computing (HotCloud 10)

    2010

  40. [48]

    Matei Zaharia, Reynold S Xin, Patrick Wendell, Tathagata Das, Michael Armbrust, Ankur Dave, Xiangrui Meng, Josh Rosen, Shivaram Venkataraman, Michael J Franklin, et al. 2016. Apache spark: a unified engine for big data processing. Commun. ACM 59, 11 (2016), 56–65

    2016

  41. [49]

    Kundjanasith Thonglek, Kohei Ichikawa, Chatchawal Sangkeettrakarn, and Api- vadee Piyatumrong. 2021. Auto-scaling system in apache spark cluster using model-based deep reinforcement learning. In Heuristics for Optimization and Learning. Springer, 347–360

    2021

  42. [50]

    Jingyuan Wang, Ze Wang, Jianfeng Li, and Junjie Wu. 2018. Multilevel wavelet decomposition network for interpretable time series analysis. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2437–2446

    2018

  43. [51]

    Yiwen Zhu, Subru Krishnan, Konstantinos Karanasos, Isha Tarte, Conor Power, Abhishek Modi, Manoj Kumar, Deli Zhang, Kartheek Muthyala, Nick Jurgens, et al. 2021. KEA: Tuning an Exabyte-Scale Data Infrastructure. In Proceedings of the 2021 International Conference on Management...

    2021

  44. [56]

    George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. 2021. A transformer-based framework for multivariate time series representation learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining . 2114–2124

    2021

  45. [57]

    Xiaoxi Zhang, Chuan Wu, Zongpeng Li, and Francis CM Lau. 2017. Proactive VNF provisioning with multi-timescale cloud resources: Fusing online learning and online optimization. In IEEE INFOCOM 2017-IEEE Conference on Computer Communications. IEEE, 1–9

    2017

  46. [2011]

    ACM Sigplan Notices 46, 7 (2011), 121–132

    CloudNet: dynamic pooling of cloud resources by live WAN migration of virtual machines. ACM Sigplan Notices 46, 7 (2011), 121–132

    2011

  47. [2013]

    In Proceedings of the 4th ACM/SPEC International Conference on Performance Engineering

    Self-adaptive workload classification and forecasting for proactive resource provisioning. In Proceedings of the 4th ACM/SPEC International Conference on Performance Engineering. 187–198

  48. [2014]

    In 2014 IEEE 6th International Conference on Cloud Computing Technology and Science

    Automatic resource provisioning: a machine learning based proactive ap- proach. In 2014 IEEE 6th International Conference on Cloud Computing Technology and Science. IEEE, 168–173

    2014

  49. [2019]

    Machine learning at Microsoft with ML. NET. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining . 2448–2458

  50. [2020]

    In PVLDB

    Seagull: An Infrastructure for Load Prediction and Optimized Resource Allocation. In PVLDB. VLDB Endowment, 154–162

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.