{"id":"ea1b7046-b89e-416c-ae81-7bf16afe6681","arxiv_id":"1908.04491","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"uPredict times CPU, memory, and disk probes inside a VM before each application run and uses per-application regression or neural network models to predict execution time from the probe readings, reporting average errors of 9.8% to 17% on a contended private cloud and below 4% on public clouds.","lead":"Cloud users cannot see their neighbors, but uPredict predicts how slow a virtual machine will run by timing three tiny probes before each job. The framework could give ordinary tenants cheap, in-situ performance forecasts for load balancing and planning, though the reported accuracy likely looks better than deployment reality.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported prediction errors are inflated by a temporally interleaved train/test split: each test point's four nearest temporal neighbors are training data under the same contention regime, so the headline error rates do not estimate real out-of-sample error.","rationale":"The paper is a good-faith, clearly written study with a sensible user-level profiling idea and substantial experimental effort. The private-cloud trend validation and the load-balancing case study suggest the probes carry real signal. The load-bearing weakness is not the idea itself but the evidence for the headline accuracy numbers: the every-fifth-point split makes test points near-duplicates of training data under the same two-hour contention regime, and the NN hyperparameter optimization on the training set adds further optimism. This is an evaluation-validity problem, not an internal inconsistency, so the appropriate disposition is conditional rather than rejection. The reader's stated weakest assumption was contention stationarity (Section 4.3); I see that as a genuine limitation but secondary to the train/test leakage, which affects every reported error bar. A time-separated split with a persistence baseline is the single check that would settle whether uPredict actually predicts future behavior. If the errors survive that check, the paper's central claim is substantially supported; if they do not, the framework needs re-profiling or a different evaluation before its quantitative claims can be accepted.","tokens_in":22715,"tokens_out":6961,"duration_ms":74562,"concrete_test":"Re-run the evaluation with a temporal block split for each benchmark/VM: train on the first 80% of the time-ordered tuples and test on the remaining 20% (or hold out one or more complete 2-hour background-VM regimes so no test point is temporally adjacent to training points). Report mean and 95th-percentile errors for all seven models, and add a persistence baseline that predicts each test application's execution time as the most recent training point's execution time, using no micro-benchmark features. If uPredict's errors are not materially below the persistence baseline, or if public-cloud errors rise well above 4% under the time-separated split, the quantitative claims are not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The quantitative support for the central claim t_app = f(c_cpu, c_mem, c_disk) (Eq. 1) is undermined by the evaluation split in Section 5.1. The paper trains on the first four of every five consecutive data points and tests on the fifth. Since each iteration costs only ~9 s of profiling plus one application run, and the private-cloud background VMs change only every 2 hours, the five points in a group are almost always collected under the same contention regime; the test point is temporally sandwiched between training points. The models can interpolate or memorize local conditions, so the reported 9.8-17% (private) and <4% (public) errors estimate interpolation error, not prediction for a future contention state. On low-contention public clouds, where execution times fluctuate by at most 25%, a persistence baseline (predict the most recent observed execution time) would plausibly match the sub-4% error without using any micro-benchmark feature. Section 4.2.3 worsens this by tuning NN hyperparameters on the training set itself, a risk the authors acknowledge. The stationarity limitation of Section 4.3 is real but separate; it would matter even with a clean split. The private-cloud trend in Figure 2 is encouraging qualitative evidence, but it does not quantify out-of-sample accuracy.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes uPredict, a user-level profiling framework for predicting the execution time of single-VM applications in multi-tenant clouds. Three custom micro-benchmarks probe CPU, memory, and disk contention; the resulting counter values (c_cpu, c_mem, c_disk) are used as features in polynomial regression, SVR, and neural-network models to predict application execution time through the learned function in Eq. (1). The framework is evaluated on a private OpenStack cloud and on AWS and GCE using 17 benchmarks from PARSEC, NPB, and CloudSuite, with reported average prediction errors of 9.8% to 17% on the private cloud and below 4% on the public clouds. A load-balancing case study reports reductions in execution and turnaround times relative to queue-based balancing. The central claim is that an ordinary cloud user, without hypervisor or PMU access, can forecast in-situ single-VM application performance using only user-level probes.","tokens_in":22862,"tokens_out":4531,"duration_ms":43074,"significance":"If the quantitative claims hold, this is a valuable contribution: it addresses a practical gap by predicting in-situ performance from the user's perspective, uses only three user-level micro-benchmarks, and is validated over a large experimental campaign (roughly 70 days on the private cloud and 10 to 20 days on each public cloud). The qualitative trend-following evidence in Figure 2 provides initial support for the micro-benchmark probe hypothesis. The load-balancing case study is a useful demonstration of a concrete application. However, the reported accuracy numbers are currently the main quantitative support for the framework, and the evaluation protocol used to obtain them does not establish out-of-sample prediction error for future contention states. The core claims are therefore defensible but need re-analysis with a proper temporal evaluation protocol.","major_comments":[{"comment":"The train/test split is temporally interleaved and does not measure out-of-sample prediction error for a future contention state. The paper states that 'for every 5 consecutive data points' the first 4 are training and the last is testing. Since the private-cloud background VMs change only every 2 hours (Section 5.1), and each iteration requires only about 9 seconds of profiling plus one application run, the four training neighbors of a given test point are, in most cases, collected under the same contention regime. The model can therefore interpolate or memorize local counter-to-time mappings rather than predict a new contention state. The reported 9.8%–17% private-cloud and sub-4% public-cloud errors thus estimate interpolation error. Please re-evaluate with a chronological split (e.g., train on the first 80% of each benchmark's time series and test on the last 20%) and report the resulting errors. Also report a persistence baseline that predicts the most recently observed execution time; this is especially important for the public clouds, where the paper reports execution-time fluctuations of at most 25%.","section":"Section 5.1, data split"},{"comment":"NN hyperparameter optimization is performed on the training set itself, as the authors acknowledge. Because the model structure is selected on the same data used to fit the model, the reported neural-network errors are optimistically biased: the reported 9.8% private-cloud NN error and the corresponding public-cloud numbers may not reflect accuracy on unseen data. Please perform structure selection on a held-out validation set or with nested cross-validation, and report how the errors change when the structure is selected without access to the test portion.","section":"Section 4.2.3"},{"comment":"The framework's central assumption is that the contention measured by the roughly 9-second probe sequence immediately before an application run equals the contention the application will experience during its entire execution. This stationarity assumption is load-bearing for Eq. (1), since the probe counters are the only features used for prediction. The paper itself notes that accuracy degrades when contention changes during execution, such as during background VM startup or shutdown. The private-cloud evaluation appears to include such transitions (Figure 2 shows low-contention periods at the beginning of each 2-hour interval). Please quantify prediction error conditional on whether the contention regime was stable between profiling and execution, and state clearly the conditions under which the reported accuracy is expected to hold.","section":"Section 4.3"}],"minor_comments":[{"comment":"The abstract and conclusion state that the load-balancing scheme reduces execution and turnaround times by 19% and 10%, respectively, but Table 2 (high-load) shows uPredict-based turnaround time (1066 s) is worse than queue-based turnaround time (987 s). The claimed 10% turnaround improvement is supported only by Table 3 (low-load). Please qualify the claim to specify the load condition.","section":"Section 5.6 and abstract"},{"comment":"There are several typos that should be corrected: 'Cannel' in the Figure 2 caption should be 'Canneal'; 'addtion' in Section 4.2.3 should be 'addition'; 'empolyed' in Section 5.1 should be 'employed'; and Section 5.3 contains 'the the prediction errors'.","section":"Throughout"},{"comment":"Equation (2) is garbled in the text ('/summationdisplay.1') and should be typeset as a standard summation so that the SVR form is readable.","section":"Equation (2)"},{"comment":"The paper states that fixed-structure NN models have an overall average error of 60% but does not show these results in any figure or table. Since this result motivates the hyperparameter-optimization step, it would be helpful to report it in a table or appendix.","section":"Section 5.3"},{"comment":"The sensitivity study of profiling length uses only 5 applications and does not include NAS or CloudSuite workloads. Please state whether these 5 applications are representative of the broader set and whether the 0.4-second and 3-second conclusions are expected to generalize.","section":"Section 5.5"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: the core idea is reasonable and the authors put in serious experimental work, but the headline accuracy numbers shouldn't be trusted as out-of-sample performance. The split takes every fifth point as test while training on the previous four, and since contention changes on a two-hour timescale, those points are almost the same regime. The models are basically interpolating. The NN hyperparameter search on the training set makes it worse, and the authors admit that.\n\nWhat's actually new: a user-level framework that uses three simple micro-benchmarks (CPU register increments, strided memory reads, disk reads) as contention probes, then learns per-application, per-VM models mapping probe counters to execution time. That's a clean, deployable idea for ordinary tenants without hypervisor or PMU access. The probe designs inherit from prior work, but the in-situ prediction framing is a real step beyond PARIS and Scheuner-Leitner, which target average performance across instance types.\n\nWhat's done well: the experiments are extensive (17 benchmarks, three clouds, about 70 days on the private cloud). Figure 2 is genuinely encouraging: predicted trends follow measured ones across 24 hours of controlled background VM changes. The paper is also honest about limitations: memory probe misses cache contention, fixed input sizes, and the stationarity assumption in Section 4.3. That's more transparent than most systems papers.\n\nThe soft spots are real but not fatal. The main one is the split. The errors of 9.8-17% (private) and <4% (public) estimate interpolation, not prediction. On public clouds, where execution times fluctuate by at most 25%, a simple persistence baseline (last observed time) might do just as well; there's no such baseline. The load-balancing results also lack variance numbers. Fixing this doesn't require a rethink, just a cleaner evaluation: time-separated training/test windows, a persistence baseline, and ideally released data. The qualitative Figure 2 suggests the probes do track contention; I'd expect out-of-sample errors to be higher, but still useful.\n\nWho it's for: researchers working on cloud performance prediction or contention-aware scheduling from the tenant side. It's worth a serious referee, but I'd expect major revisions. If I were handling it, I'd insist on the temporal split before trusting the numbers.","headline":"A sensible user-level probe-based prediction idea backed by large experiments, but the reported error rates are inflated by a temporally leaky train/test split.","tokens_in":23532,"tokens_out":2286,"would_cite":true,"duration_ms":22865,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A cloud user can forecast single-VM app runtimes under contention using only three user-level probes.","keywords":["cloud performance prediction","resource contention","micro-benchmarks","user-level profiling","single-VM applications","multi-tenant clouds","neural network models","load balancing"],"falsifier":"Run a benchmark under a controlled private cloud, profile contention, start the application, and then sharply change the co-located workload, such as launching or killing a CPU-bound VM, partway through the run. If uPredict's predicted execution time stays close to the stationary-contention value while the measured time shifts, that directly demonstrates the failure mode; repeating this across several benchmarks would show how often the stationary assumption breaks.","tokens_in":22368,"feed_emoji":"☁️","tokens_out":13277,"duration_ms":115609,"temperature":0.7,"pith_summary":"uPredict claims that an ordinary cloud user, with no access to the hypervisor or to hardware performance counters, can predict how long a single-VM application will run under the resource contention currently present in a multi-tenant cloud. The key move is to run three short micro-benchmarks inside the user's own VM—one stressing the virtual CPUs, one pressing off-chip memory bandwidth, one hammering uncached disk reads—and to use the progress counters from those probes as features in a per-application, per-VM model $t_{app} = f(c_{cpu}, c_{mem}, c_{disk})$. Training data come from repeatedly measuring the probes and the application's execution time while co-located workloads vary. The paper reports average prediction errors of 9.8% to 17% on a heavily contended private cloud and below 4% on two public clouds. If correct, this gives ordinary users a practical way to forecast performance and to make scheduling and load-balancing decisions without the cloud provider's cooperation.","feed_headline":"Three user probes forecast app runtimes within 4% on public clouds","feed_subtitle":"No hypervisor access: micro-benchmark counters feed learned models that forecast single-VM runtimes.","key_machinery":"The object that carries the argument is the triple of micro-benchmark counters. The CPU probe runs one in-register increment loop per virtual CPU so its progress is unaffected by memory; the memory probe sweeps a 2GB array with 128-byte strides so every access goes off-chip rather than to cache; the disk probe reads a 256MB file with the OS file cache disabled. Each probe runs for a fixed window, about three seconds in most experiments, and returns the number of completed operations, which is meant to reflect how much of each resource the VM can actually get under current contention. These numbers become the input features of the model $f$ in $t_{app} = f(c_{cpu}, c_{mem}, c_{disk})$, trained per application and per VM; the model then converts a fresh set of probe readings into a predicted execution time. The neural-network version also relies on automated hyperparameter search over network depth and width, since a fixed large structure can be many times worse than an optimized one.","core_discovery":"The central claim is that the impact of invisible multi-tenant contention on a single-VM application is captured, for that application and VM, by three user-level probe counters measured just before the run: the number of in-register increments completed by a CPU stress loop, the number of off-core memory accesses completed while striding through a 2GB array with 128-byte strides, and the number of uncached disk reads completed on a 256MB file. With enough training tuples of the form $(c_{cpu}, c_{mem}, c_{disk}, t_{app})$ collected while co-located workloads change, a regression or neural-network model learns the function in Equation (1), and at prediction time only the three counters need to be measured. In the paper's evaluation, the neural-network models achieve average errors around 9.8% even under high contention in a private cloud, and 3.8% and 3.4% on two public clouds, while simpler polynomial models land near 17% in the high-contention setting and roughly match the neural networks on lightly contended public clouds. The authors also show the predicted values track the pattern of measured run times as background VMs appear and disappear, which they take as evidence that the micro-benchmarks do assess the severity of contention experienced by user applications.","pith_inferences":["The same counter-to-performance learning could be extended to variable input sizes by adding an input-size feature to the model; the paper fixes input size and lists input variation as future work.","The stationarity premise suggests that for long-running jobs uPredict would need periodic re-profiling mid-run, an extension the paper acknowledges but does not implement.","A fourth micro-benchmark targeting network bandwidth would likely extend the framework to network-intensive single-VM applications, which the current probe set does not cover.","Because the models are retrained per application and VM, they could be refreshed online as co-location patterns drift over days, turning the framework into a continuously adapting predictor; the paper only tests static training sets."],"forward_implications":["Ordinary cloud users can obtain runtime performance forecasts for single-VM applications without any cooperation from the provider or access to hypervisor-level counters.","In lightly contended public clouds, the cheapest models, such as 2-degree polynomial regression, are nearly as accurate as neural networks, so low-overhead prediction is sufficient in that regime.","Under heavy contention, the accuracy gap between a cheap polynomial model and an optimized neural network is about 7 to 8 percentage points on average, and the gap is largest for memory-intensive applications whose access patterns differ from the memory probe's.","Most large prediction errors occur when background VMs start or stop and the contention changes between profiling and execution; with stable contention the framework's predictions follow the measured pattern.","A load balancer that routes requests by predicted completion time instead of queue length can reduce average application execution and turnaround times by up to 19% and 10%, with profiling overhead included in turnaround time."],"supporting_citations":[{"why":"Shows that micro-benchmark profiling from the user side can estimate cloud application performance, the approach uPredict adapts to in-situ contention.","marker":"[50]"},{"why":"User-level model for selecting among VM types from offline profiling; the baseline uPredict extends by accounting for runtime resource contention.","marker":"[62]"},{"why":"Establishes that contention in CPUs, memory, storage and network drives performance degradation in datacenters, motivating the three probes.","marker":"[15]"},{"why":"Demonstrates that a probe's slowed progress under injected pressure reveals application sensitivity to contention, the principle behind the counter values.","marker":"[39]"},{"why":"Uses machine learning to predict application interference from features, the learning pattern uPredict applies from user-level features.","marker":"[40]"},{"why":"Relies on hardware PMU readings to measure instantaneous contention, contrasting with uPredict's user-level alternative.","marker":"[63]"},{"why":"Provides the interference-generation workloads used to create controlled background contention in the private-cloud validation.","marker":"[14]"},{"why":"Supplies a set of evaluated parallel benchmark applications used to train and test the predictive models.","marker":"[10]"},{"why":"Supplies another set of evaluated parallel benchmark applications for the model evaluation.","marker":"[5]"},{"why":"Supplies scale-out cloud benchmark applications included in the evaluation.","marker":"[18]"}],"fun_headline_variants":["Three user probes predict VM app performance within 4% on public clouds","No hypervisor needed: three probes forecast VM app runtimes within 4%","User-level micro-benchmarks forecast single-VM app performance with 4% error","Three probe counters predict cloud app speed within 4% without hypervisor access","Predict VM app runtimes in multi-tenant clouds with just three user probes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The contention measured by the micro-benchmarks in the roughly nine seconds before an application starts is the same contention the application will experience for its entire run; if co-located workloads change during the run, the probe readings are stale and the prediction is unreliable.","fun_headline_variants_meta":{"raw":{"variants":["Three user probes predict VM app performance within 4% on public clouds","No hypervisor needed: three probes forecast VM app runtimes within 4%","User-level micro-benchmarks forecast single-VM app performance with 4% error","Three probe counters predict cloud app speed within 4% without hypervisor access","Predict VM app runtimes in multi-tenant clouds with just three user probes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000946,"raw_usage":{"total_tokens":4089,"prompt_tokens":1047,"completion_tokens":3042,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":663,"completion_tokens_details":{"reasoning_tokens":2939}},"tokens_in":663,"tokens_out":3042,"duration_ms":22206,"temperature":1.0,"reasoning_tokens":2939,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:42:19.761221+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a benchmark under a controlled private cloud, profile contention, start the application, and then sharply change the co-located workload, such as launching or killing a CPU-bound VM, partway through the run. If uPredict's predicted execution time stays close to the stationary-contention value while the measured time shifts, that directly demonstrates the failure mode; repeating this across several benchmarks would show how often the stationary assumption breaks.","supporting_citations":[{"cited_title":"Scheuner and P","cited_arxiv_id":null,"evidence_quote":"Shows that micro-benchmark profiling from the user side can estimate cloud application performance, the approach uPredict adapts to in-situ contention."},{"cited_title":"Y adwadkar, Bharath Hariharan, Joseph E","cited_arxiv_id":null,"evidence_quote":"User-level model for selecting among VM types from offline profiling; the baseline uPredict extends by accounting for runtime resource contention."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes that contention in CPUs, memory, storage and network drives performance degradation in datacenters, motivating the three probes."},{"cited_title":"Mishra, J","cited_arxiv_id":null,"evidence_quote":"Uses machine learning to predict application interference from features, the learning pattern uPredict applies from user-level features."},{"cited_title":"Wyatt, II, Stephen Herbein, Todd Gamblin, Ad am Moody, Dong H","cited_arxiv_id":null,"evidence_quote":"Relies on hardware PMU readings to measure instantaneous contention, contrasting with uPredict's user-level alternative."},{"cited_title":"Delimitrou and C","cited_arxiv_id":null,"evidence_quote":"Provides the interference-generation workloads used to create controlled background contention in the private-cloud validation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies a set of evaluated parallel benchmark applications used to train and test the predictive models."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies another set of evaluated parallel benchmark applications for the model evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies scale-out cloud benchmark applications included in the evaluation."}],"review_version":1}