{"id":"5c3b6c2b-6c53-4b0a-a77c-de719c52536f","arxiv_id":"2607.24773","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"On long-lived Azure VM workloads, XGBoost and CatBoost with bootstrap conformal prediction produce the best-ranked upper-bound CPU utilization intervals for right-sizing recommendations.","lead":"Using public Microsoft Azure VM traces, this paper wraps seven machine-learning forecasters in bootstrapped conformal prediction intervals and reports that XGBoost and CatBoost give the most accurate upper-bound CPU forecasts for cloud right-sizing. If the ranking holds, operators get a practical recipe for uncertainty-aware instance sizing that could reduce over-provisioning in large data centers.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Bootstrap CP intervals rely on uncorrelated residuals, likely violated for non-stationary vCPU series; without diagnostics, the claim of reliable intervals is unsupported.","rationale":"The reader's weakest assumption correctly targets the uncorrelated-residual premise of the bootstrap conformal intervals. This is the most load-bearing concern because the paper's primary claim is the production of reliable prediction intervals for provisioning recommendations. If the intervals are invalid, the entire RSR framework loses its uncertainty guarantee, even if point forecasts are accurate. The concern is not merely a missing baseline or metric; it strikes at the core methodology. The concrete test would directly assess whether the intervals achieve nominal coverage on held-out data, which is what 'reliable' must mean in this context. I find no other issue more fundamental: the lack of numerical tables and baselines is secondary because even those would not establish interval validity. Therefore, I agree with the reader that the paper should be conditional on providing autocorrelation diagnostics or coverage validation. The verdict remains CONDITIONAL, so no adjustment is needed.","tokens_in":12555,"tokens_out":2992,"duration_ms":32658,"concrete_test":"Select a random sample of 100 long-lived VMs (≥29 days) from AzurePublicDatasetV1/V2. For each VM, train XGBoost and CatBoost with the paper's 75/15 splits, generate 90% bootstrap PIs via skforecast, then: (1) run Ljung-Box test (lag 20) on the one-step-ahead residuals from the training/validation period; (2) compute empirical coverage on the held-out 15% test set. If the median Ljung-Box p-value is <0.05 or if the empirical coverage is below 85%, the interval reliability claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that CP-augmented models produce reliable prediction intervals for VM workload forecasting (Sec. VI). The intervals are built via skforecast's bootstrap method, which, as the paper states in Sec. III-D, 'requires only that the residuals (errors) be uncorrelated.' However, the data are 5-minute vCPU series that the paper itself describes as non-stationary, seasonal, and bursty (Sec. IV-C), with known sources like Cryptojacking and DDoS. Recursive/DirRec multi-step forecasting (Sec. III-C2) propagates errors, making multi-step residuals even more autocorrelated. The paper reports no autocorrelation test (e.g., Ljung-Box) and no empirical coverage calibration. Without these, the bootstrap intervals do not carry the distribution-free conformal guarantee; they are only valid under the bootstrap assumption. If residuals are autocorrelated, intervals will undercover, directly undermining the reliability claim used for right-sizing decisions (Sec. I-A), where an upper-bound miss can cause SLA violations. The Limitations section (V-C) acknowledges unpredictability but omits this specific validity threat. This is the load-bearing risk: the entire contribution rests on the correctness of intervals that are never validated against actual coverage.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a forecasting pipeline for Azure VM vCPU utilization, combining seven ML regressors with bootstrap conformal prediction (via skforecast) to produce prediction intervals, and evaluates four backtesting strategies on long-lived VM candidates from the Azure Public Datasets V1/V2. The central claim is that the resulting intervals are reliable enough to support right-sizing recommendations, and that XGBoost/CatBoost are the top-performing models across backtesting strategies. The experimental section presents rankings as bar-chart frequencies and discusses efficiency trade-offs, while a limitations section acknowledges workload unpredictability and missing service-level metadata.","tokens_in":12889,"tokens_out":3838,"duration_ms":43109,"significance":"If established, the paper would provide a useful practical evaluation of conformal prediction for cloud workload forecasting and a model-ranking benchmark over publicly available Azure traces. Strengths include the use of public datasets, a reproducible GitHub repository, a broad set of regression models, and four backtesting strategies. However, the central reliability claim is not yet supported by the evidence: the bootstrap CP intervals are not validated for the non-stationary, autocorrelated time series studied, and the empirical results lack quantitative forecast-error and coverage numbers, baselines, and a precisely specified VM subset. The paper is therefore of interest to the cloud operations community, but the current draft requires substantial additional analysis before its conclusions can be accepted.","major_comments":[{"comment":"The validity of the proposed conformal prediction intervals is not established. The paper states that skforecast's bootstrapping 'requires only that the residuals (errors) be uncorrelated' (§III-D), but the vCPU series are explicitly described as non-stationary, seasonal, and bursty (§IV-C), and the DirRec/recursive forecasting strategy propagates errors across steps (§III-C2). No autocorrelation test (e.g., Ljung-Box) is reported, and no empirical coverage evaluation (e.g., PICP per horizon) is provided. The only PICP-based ranking is mentioned as 'Likely' with no numeric support (§V-A). Without these diagnostics, the bootstrap intervals do not carry the conformal guarantee claimed, and the RSR upper-bound use case (where a miss can cause SLA violations) is not protected. This is a load-bearing issue for the paper's central claim and should be addressed either by switching to a split-co","section":"§III-D, §III-C2, §IV-C"},{"comment":"The empirical evaluation is presented only as absolute-frequency bar charts for Top-1 and Top-3 model counts. No numeric MAE or PICP values, no error bars, and no baseline comparisons (e.g., naive, SARIMA, or prior-work methods) are reported. The statement in §IV-C that 'we need models that are capable of making skillful forecasts compared to naive models and tuned SARIMA models' is never followed by such a comparison. Consequently, the claims of 'promising forecasting results' and the model rankings are not quantitatively substantiated. The paper should include tables with per-model and per-backtesting-strategy error and coverage metrics, and at least one baseline comparison.","section":"§V, Fig. 9"},{"comment":"The VM subset used for the ranking experiments is unspecified. The caption of Fig. 9 refers to 'limited VM candidates due to expensive computation,' but the exact number of VMs, the selection criteria, the distribution across workload classes (Delay-insensitive/Interactive/Unknown), and the train/test split used for the ranking are not given. Without this information, the generalizability of the 'global top-tier models' claim cannot be assessed. Please state the sample size and the sampling procedure explicitly.","section":"§V-A, Fig. 9"},{"comment":"There is a discrepancy in the data split. Section IV says 'we allocate the first 75% of each VM workload time series to training and validation/calibration. The remaining 15% ... is reserved for testing,' which leaves 10% of the data unaccounted for. For medium-scale forecasting, the ratio is stated as 85% training and 15% testing, but no equivalent allocation is given for the large-scale case. Since conformal prediction requires a proper calibration set, the missing 10% may be critical. Please clarify the exact split and the role of the calibration data in the interval construction.","section":"§IV"}],"minor_comments":[{"comment":"Equation (1) is a tautological definition rather than a measurement equation. Consider removing or reformulating it to avoid confusion.","section":"§III-A, Eq. (1)"},{"comment":"The survey counts in Fig. 3 are presented without a methodology or a reference to the underlying survey. Please either cite the survey source or describe how the 63 publications were compiled.","section":"§II, Fig. 3"},{"comment":"Fig. 7 shows an 'Interactive-V2' trace with 5,422 observations (about 18 days), which appears inconsistent with the stated long-lifetime criterion of '≥29 consecutive days.' Please clarify whether this subplot is an example from a different filtering stage.","section":"§IV-A, Fig. 7"},{"comment":"The sentence beginning 'Likely, the results of ranked ML-powered conformal forecasts...' is phrased as a conjecture. If PICP-based rankings were computed, present them as definitive results with values.","section":"§V-A"},{"comment":"The reference list has inconsistent formatting (e.g., incomplete entries, missing journal names, and inconsistent access-date conventions). A thorough cleanup is needed.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is an empirical, application-oriented study with a potentially useful comparison of ML models for cloud workload forecasting. The main risk is not that the claims are circular, but that the central reliability claim rests on bootstrap CP intervals whose validity assumptions are unverified. The missing quantitative results and unspecified VM subset further weaken the paper. I think this can be fixed within the scope of a revision, provided the authors add coverage diagnostics, autocorrelation tests, baseline comparisons, and detailed experimental metadata. If the authors cannot obtain the needed data or if the coverage results fail, the paper may need to be re-scoped as a point-forecast ranking study without the CP reliability claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a useful practice-oriented benchmark: applying bootstrap conformal prediction to seven regressors with four backtesting strategies on public Azure VM workloads is a sensible configuration, and the paper is honest about many of its limitations. It gives credit where due: public datasets, a reproducibility link, sane preprocessing (Savitzky-Golay, median imputation, outlier clipping), and a clear ranking of XGBoost/CatBoost as top performers for long-lived VMs, with an efficiency trade-off plot. Practitioners running cloud workload forecasts could get value from the model-vs-backtesting comparison.\n\nBut the central claim - that these are \"reliable prediction intervals\" - does not hold up as reported. The skforecast bootstrap method the paper relies on (Sec. III-D) explicitly assumes uncorrelated residuals, and the paper itself describes the vCPU series as non-stationary, seasonal, and bursty (Sec. IV-C). With recursive/DirRec forecasting, error autocorrelation is essentially guaranteed. No Ljung-Box or similar test is reported, and no empirical coverage validation is shown. The PICP-based rankings are therefore built on intervals whose validity is untested. The stress-test note is right, and the Limitations section (V-C) misses this specific threat.\n\nThe reporting is also too thin in several places. The train/test split in Sec. IV sums to 90%, leaving 10% unaccounted for. There are no numeric MAE/PICP tables, no error bars, no comparison against naive or SARIMA baselines even though such baselines are mentioned in the problem statement. The number of VMs used for the experiments is left as \"limited VM candidates\" without a concrete count. Hyperparameters, filter widths, and lag windows are not specified in the paper. These are all fixable, but as is they make it hard to reproduce or verify the rankings.\n\nWho is this for? Cloud operators choosing models for VM right-sizing, and researchers interested in applied conformal prediction for time series - the latter more as a cautionary example of what not to skip. I would not cite it in my own work as a valid demonstration of conformal coverage, but it could be a useful empirical datapoint after the validity gaps are addressed.\n\nPeer review: yes, send it out. A serious referee can push for the missing diagnostics and tables. The core experiment is worthwhile, but the paper currently overclaims the reliability of its intervals.","headline":"An applied benchmark with a real validity problem: the reliability claim for the conformal intervals rests on an untested residual-independence assumption, and the reporting is too thin to carry the conclusions.","tokens_in":660,"tokens_out":702,"would_cite":false,"duration_ms":39807,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that bootstrapped conformal prediction converts 5-minute vCPU utilization series into reliable mid- and long-term prediction intervals that can drive VM right-sizing recommendations.","keywords":["conformal prediction","prediction intervals","cloud workload forecasting","right-sizing","virtual machines","time series","backtesting","vCPU utilization"],"falsifier":"Compute the Ljung-Box or similar autocorrelation test on the one-step-ahead residuals of a random sample of the long-lived VM series; if significant autocorrelation is found, or if empirical coverage of the reported intervals on a held-out test set falls materially below the nominal 1-alpha level, the central 'reliable prediction intervals' claim is falsified.","tokens_in":12471,"feed_emoji":"☁️","tokens_out":4469,"duration_ms":45321,"temperature":0.7,"pith_summary":"This paper tries to establish that conformal prediction, applied to bootstrapped one-step-ahead residuals, produces trustworthy prediction intervals for mid- and long-term vCPU utilization forecasts of cloud VMs, and that those intervals can drive right-sizing recommendations. The authors build a forecasting pipeline on high-resolution 5-minute vCPU traces, compare seven machine-learning regressors under four backtesting strategies, and rank the models for long-lived VM workloads. A sympathetic reader would care because right-sizing decisions today rely on point forecasts or conservative historical maxima; if the intervals hold their stated coverage, operators can provision against an upper bound with a statistical guarantee, reducing both over-provisioning and SLA risk.","feed_headline":"Conformal prediction yields reliable intervals for VM right-sizing","feed_subtitle":"Tree-based models rank top for long-lived workloads, giving operators upper bounds for provisioning.","key_machinery":"The load-bearing mechanism is bootstrap conformal prediction: after training a point forecaster, one-step-ahead residuals are resampled and added to the model's predictions to synthesize a forecast distribution, from which the alpha/2 and 1-alpha/2 empirical quantiles define the interval. This avoids normality assumptions on residuals and requires only that residuals be uncorrelated. The forecasting core uses a direct-recursive multi-step strategy (DirRec) inside the forecaster, with four backtesting regimes for evaluation.","core_discovery":"On the paper's own terms, the central discovery is that bootstrapped residual conformal prediction turns recursive multi-step forecasts of vCPU utilization into prediction intervals whose upper bound is accurate enough for provisioning decisions, and that XGBoost and CatBoost are the consistently top-ranked regressors for long-lived VM workload profiles across all backtesting strategies. The ranking holds for upper-bound (PI) evaluation; for target-value MAE, linear regression is reported as the top model. The paper stops short of claiming the intervals are optimal, presenting them instead as a practical, modular route to uncertainty-aware right-sizing.","pith_inferences":["The paper does not test residual autocorrelation, so a natural extension is to diagnose whether the uncorrelated-residual premise actually holds on these workloads; if it fails, coverage guarantees weaken.","The ranking is based on limited VM candidates and focused on long-lived VMs; the same model hierarchy may not hold for short-lived or highly bursty workloads.","Because the study is univariate and vCPU-only, the approach could likely be extended to memory and multivariate forecasting, where memory footprint is currently absent from public cloud traces."],"forward_implications":["If the intervals are valid, cloud operators can replace conservative historical-max upper bounds with statistically calibrated upper bounds from conformal intervals, lowering over-provisioning.","XGBoost and CatBoost emerge as the practical choices for large-scale long-horizon vCPU forecasting because they balance accuracy and runtime.","Refitting strategies, especially fixed-origin and rolling-origin backtesting, improve top-1 model placements, indicating that updating models on recent data sharpens the forecasts.","The modular pipeline can be applied to other VM metrics or datasets without redesign."],"fun_headline_variants":["Conformal prediction sharpens VM right-sizing upper bounds","Tree models rank top for VM utilization interval forecasting","RSR: XGBoost and CatBoost best for long-lived VM provisioning PIs","Bootstrapped conformal prediction yields reliable VM provisioning intervals","Uncertainty-aware VM sizing: tree models lead for upper-bound intervals"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The entire interval-reliability claim rests on the assumption that the one-step-ahead forecast residuals are uncorrelated, yet the vCPU time series are described as non-stationary, seasonal, and bursty, and the paper reports no test of residual autocorrelation.","fun_headline_variants_meta":{"raw":{"variants":["Conformal prediction sharpens VM right-sizing upper bounds","Tree models rank top for VM utilization interval forecasting","RSR: XGBoost and CatBoost best for long-lived VM provisioning PIs","Bootstrapped conformal prediction yields reliable VM provisioning intervals","Uncertainty-aware VM sizing: tree models lead for upper-bound intervals"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001185,"raw_usage":{"total_tokens":4747,"prompt_tokens":779,"completion_tokens":3968,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":3880}},"tokens_in":523,"tokens_out":3968,"duration_ms":30303,"temperature":1.0,"reasoning_tokens":3880,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T11:27:58.784737+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the Ljung-Box or similar autocorrelation test on the one-step-ahead residuals of a random sample of the long-lived VM series; if significant autocorrelation is found, or if empirical coverage of the reported intervals on a held-out test set falls materially below the nominal 1-alpha level, the central 'reliable prediction intervals' claim is falsified.","supporting_citations":[],"review_version":1}