{"id":"53422847-283a-451e-9381-7c8f494eda9c","arxiv_id":"2504.20740","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"PolarisProfiler groups historical workloads by runtime behavior and classifies new workloads by static metadata, estimating resource usage with reported normalized error below 50% for 93% of tested Alibaba workloads.","lead":"This paper formalizes and tests a profiling method that assigns incoming computing workloads to groups built from past resource-usage traces, using only static metadata such as job name and user to make the assignment. If it works, orchestrators could estimate a workload's resource needs instantly, without running it first.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Same-trace random split and post-hoc choice of the 5th-percentile predictor make the reported 93% accuracy an in-sample fit, not demonstrated real-time prediction for new workloads.","rationale":"The formal model in Section 2 is generic, and the implementation choices (HDBSCAN, XGBoost, SHAP) are standard; there is no specific technical objection to the pipeline itself. The load-bearing issue is in the evaluation of the headline number. A random split within a two-month trace tests interpolation over sampled workloads, not extrapolation to future submissions, and the post-hoc selection of the 5th quantile makes the number optimistic even for that split. The Google analysis in Section 4 has the same structure and does not repair this. The ACQUIRES metric and feedback loop are also not validated against alternatives, but the central claim would be settled by the temporal and baseline check described above. Since the paper already presents the approach as preliminary and Section 6 acknowledges the stationarity limitation, the appropriate verdict remains conditional: the framework is promising, but the stated accuracy should not be treated as established until the proposed check is run.","tokens_in":27858,"tokens_out":5744,"duration_ms":60436,"concrete_test":"On the Alibaba data, split strictly by submission time: cluster and train the classifier only on workloads starting in the first month, then evaluate only on workloads starting in the second month (or a later contiguous window not used in any step, including quantile selection). On the same temporal test set, compute the RMSE_perc CDF for a no-profile baseline that predicts the global 5th quantile and, separately, the per-workload-type median from the training month. If the fraction with RMSE_perc below 50 drops materially, or the baseline matches the 93% figure, the reported accuracy does not support real-time generalization.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim (Abstract, Section 3.5) is that metadata-only classification gives normalized error below 50% for about 93% of Alibaba workloads, enabling accurate real-time estimates. For that claim to support real-time profiling, profiles and the prediction rule must generalize to workloads not drawn from the exact trace used to build them. The evaluation does not establish this. Section 3.5 takes 10,000 'unseen' workloads from the same two-month Alibaba trace used for clustering and classifier training (Sections 3.1 and 3.4); there is no temporal split, so the test set is same-distribution. Moreover, the prediction rule is selected on the test set: 'After experimenting with different settings, the approach that gave us the best results is to use the 5th quantile' (Section 3.5). No comparison is made to a trivial profile-free predictor such as a global 5th quantile or per-workload-type median. Because the reported metric is a normalized error around a single low quantile of heavy-tailed resource-usage distributions, a large fraction of workloads can fall below the 50% threshold even if the metadata classifier contributes nothing. Section 6 lists long-time evolution as future work, confirming that distribution shift is untested. The headline accuracy is therefore a statement about fit to the observed trace, not about future workloads.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PolarisProfiler, a metadata-based workload profiling approach for computing-continuum resource management. The approach first clusters historical workloads using runtime telemetry (HDBSCAN on the Alibaba case study, DBSCAN with autoencoder projection on Google data), then trains an XGBoost classifier that maps a priori static metadata (user, job name, task name, group, workload type) to the resulting profile groups. For a new workload, the assigned profile's 5th-percentile statistics are used as predictions of CPU, GPU, memory usage, and duration. The central empirical claim is that this yields an overall normalized RMSE below 50% for about 93% of classified workloads on the Alibaba dataset, with comparable results on Google cluster data. The paper also introduces the ACQUIRES clustering-quality metric and a feedback-loop mechanism, and it makes the implementation publicly available.","tokens_in":28152,"tokens_out":7569,"duration_ms":68506,"significance":"If the empirical claim were established, the approach would be a useful contribution: it is non-invasive, operates in real time, uses only metadata available at submission time, and is evaluated on two large public traces with released code. The formalization of the pipeline components provides a clear vocabulary for reasoning about profile generation, classification, and feedback. The extension to Google cluster data with different clustering mechanisms supports the claim of general applicability. However, the significance is conditional on the evaluation convincingly demonstrating out-of-sample generalization of the prediction, which the current design and reporting do not establish.","major_comments":[{"comment":"The headline claim that 'our methodology yields an overall error rate below 50% for the 93% of classified workloads' is presented as evidence of accurate real-time estimation for newly submitted workloads. However, the 10,000 'unseen' test workloads are drawn from the same two-month Alibaba trace used for clustering (Section 3.3) and classifier training (Section 3.4); there is no temporal or distributional split. A classifier and profile set built on that trace cannot be distinguished by this experiment from a system that merely memorizes the trace's empirical distribution. Section 6 lists long-time evolution as future work, confirming that distribution shift is untested. To support the real-time claim, the evaluation should include a temporal split (e.g., train on an earlier window and test on a later window) or an explicit demonstration of robustness to distribution shift.","section":"Section 3.5; Abstract"},{"comment":"The prediction summary statistic is selected on the test set: 'After experimenting with different settings, the approach that gave us the best results is to use the 5th quantile for the prediction.' This is post-hoc model selection on the evaluation data and inflates the reported accuracy. In addition, no trivial baseline is reported: a global 5th percentile of historical feature values, a per-workload-type median, or even a constant prediction would likely also yield a large fraction of workloads below the 50% normalized-RMSE threshold for heavy-tailed resource-usage distributions. Without such baselines, the 93% figure does not demonstrate that the metadata-based profile assignment improves prediction. Please report baselines and select the quantile on a validation split.","section":"Section 3.5"},{"comment":"The definition of the 'overall' normalized RMSE is missing. The text names RMSE_perc and reports CDFs and box plots for the four features 'altogether,' but no formula is given for how the four per-feature normalized errors are aggregated into a single value, nor is the normalization for each feature defined. This makes the headline 93% value non-reproducible. Moreover, the box plots in Figures 14 and 15 show only 16 profile labels (0,1,3,7,8,9,10,12,13,14,16,19,20,21,23,25), whereas Table 3 reports classification for 26 profiles; the paper should state how the 10,000 test workloads are distributed across profiles and how the denominator for the 93% claim is computed, especially given the exclusion of the outliers group and of profiles with near-zero F1-scores in Table 3.","section":"Section 3.5; Figures 14-15"},{"comment":"The ACQUIRES metric is used to select HDBSCAN over OPTICS in the Profile Generator grid search, but it introduces a free parameter C_optimal (the 'ideal' number of clusters) and equal component weights that are not justified. For HDBSCAN, which does not take a cluster count as input, the paper does not specify how C_optimal is obtained, and no sensitivity analysis is provided for the equal-weight assumption. Since the choice of clustering algorithm and hyperparameters shapes the profile groups and hence the downstream classifier, this is a load-bearing step of the evaluation; the authors should either justify C_optimal and the weights, or show that the final prediction results are robust to their variation.","section":"Section 2.3.1; Section 3.3.2"}],"minor_comments":[{"comment":"The text says 'Figure?? represents the ECDF' and Section 2.3 contains 'Figure ?? highlights this perspective'; these missing cross-references should be fixed.","section":"Section 3.5"},{"comment":"The phrase 'groups together various types of workload workload' contains a duplicated word and should be corrected.","section":"Section 2.1"},{"comment":"The sentence listing the predicted features says 'We do so for duration, CPU usage, and GPU usage' but memory usage is also reported in Figures 13c and 14c; clarify whether memory is predicted with the same 5th-quantile rule.","section":"Section 3.5"},{"comment":"The Google dataset results are described as 'similar performance' but no aggregate percentage of workloads below the 50% RMSE threshold is reported; report the same summary statistic for comparability.","section":"Section 4"},{"comment":"The freshness condition in the UT(t) equation uses 'min(CS(C_i,t)) < tau_f' but CS is not defined; define it or use the previously defined F(C_i,t).","section":"Section 2.3"},{"comment":"The phrase 'Height-weight-weight' appears twice and is likely a typo for a computational graph description.","section":"Section 5.3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript draws heavily on the authors' own prior work (PolarisProfiler, [44]) and several companion papers; the incremental contribution of the formalization and the two-dataset evaluation should be clearly delineated. The paper would also benefit from an explicit statement that the 'formal study' is a formal specification rather than a set of verified theorems, to avoid mismatched expectations from the title."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this is a follow-up to the authors' earlier PolarisProfiler work, not a from-scratch idea. The genuinely new pieces are the formal model, the ACQUIRES clustering-quality metric, the feedback loop, and the extended evaluation on Alibaba plus Google cluster data. That is real incremental work, and much of the execution is careful. They check Hopkins statistics, grid-search HDBSCAN and OPTICS, report per-profile classifier scores, and use SHAP for interpretability. On the Alibaba trace, metadata-only classification into runtime-defined profile groups reaches about 95% label accuracy and the clusters look homogeneous. Those results are plausible and worth taking seriously.\n\nThe formal model is light—it formalizes clustering and classification objectives, profile statistics, and update triggers in a straightforward way—and ACQUIRES is a simple equally weighted composite. Fine as a heuristic, not deep.\n\nThe soft spots are concentrated in the central accuracy claim. The 10,000 \"unseen\" workloads are randomly drawn from the same two-month Alibaba trace used to build the clusters and train the classifier. There is no temporal split, so the test set is same-distribution. The prediction rule itself was selected on that test set: \"After experimenting with different settings, the approach that gave us the best results is to use the 5th quantile.\" And the 50% normalized-RMSE threshold is arbitrary. Together these make the headline \"error below 50% for 93% of workloads\" a statement about fitting the observed trace, not demonstrated accuracy on future workloads. The paper honestly lists long-time evolution as future work, which confirms the gap.\n\nWhat is missing is a trivial baseline. A global 5th quantile or per-workload-type median would show how much the metadata classifier actually adds. Given the heavy-tailed distributions and the choice to predict a low quantile, a constant predictor can score well on a large fraction of workloads, so the 93% number is uninterpretable without that comparison. Also, the headline excludes out-of-group workloads and the figures show only 16 of the 26 profiles, and the released code link is an unstable GitHub edit URL. The Google extension is decent evidence that the pipeline transfers to another dataset, but it inherits the same evaluation issues. The self-citation to their own prior PolarisProfiler paper is heavy but legitimate; that is the work being extended.\n\nWho this is for: researchers working on metadata-based profiling or pre-scheduling resource estimates will get value from the formalization and the two-dataset case study. It deserves a serious referee—the method is sensible and the study is transparent about most of its components—but the revision needs baselines, a time-based split, and a justified rather than test-selected prediction rule before the accuracy claim can be published as stated. Send it to review, and expect the reviewers to push hard on the evaluation design.","headline":"A careful extension of the authors' own PolarisProfiler line—real but incremental—whose headline 93% accuracy is a same-trace fit, not demonstrated real-time prediction, until baselines and a temporal split are added.","tokens_in":28702,"tokens_out":3689,"would_cite":false,"duration_ms":38045,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper formalizes and tests a metadata-only profiler that assigns runtime profiles to workloads at submission time.","keywords":["workload profiling","static metadata","resource management","computing continuum","HDBSCAN clustering","XGBoost classification","Alibaba PAI traces","Google cluster data"],"falsifier":"Compute the same RMSE prediction test on workloads submitted after the training trace (a later month or a different cluster); if the share of workloads with error below 50% falls substantially below the reported 93%, the stationarity assumption underlying the real-time accuracy claim fails.","tokens_in":27631,"feed_emoji":"⚙️","tokens_out":5226,"duration_ms":50663,"temperature":0.7,"pith_summary":"This paper claims that workload runtime behavior can be estimated in real time from static, a priori metadata alone — user, job name, workload type, task type, and group — without sandbox profiling or runtime observation. The proposed PolarisProfiler first clusters historical execution traces by runtime similarity, then learns a mapping from metadata to the resulting profile groups, and finally uses each group's quantile statistics to predict resource usage and duration for a new submission. The empirical case studies on Alibaba machine-learning traces and Google cluster data support the claim: on 10,000 unseen Alibaba workloads, the classifier assigns profiles with 95.19% accuracy and the combined runtime estimates stay below 50% error for about 93% of workloads. If correct, this gives infrastructure orchestrators in the computing continuum a fast, non-invasive way to make scheduling and provisioning decisions while still meeting SLOs.","feed_headline":"Metadata alone predicts job runtime for 93% of workloads","feed_subtitle":"Profiles learned from past traces let schedulers estimate CPU, memory, GPU, and duration before a job starts.","key_machinery":"The central machinery is a two-stage model with a feedback loop. A Profile Generator clusters completed workloads by runtime features using density-based clustering (HDBSCAN on Alibaba, DBSCAN with autoencoder projection on Google), producing profile groups with per-feature statistical summaries such as percentiles and means. A Profile Classifier, implemented as XGBoost, maps static metadata features to profile group labels. A Feedback Loop monitors violations, outliers, and profile freshness, and triggers re-clustering when quality degrades. The formalization represents each workload as $w_i = (r(w_i), m(w_i))$, defines the clustering objective as minimizing intra-cluster distance, the classifier as a function $f: M \\to C$, and introduces the named quality metric ACQUIRES (Algorithm's Cluster QualIty-REcall Score), which combines outlier reduction, cluster-count correctness, and the mean silhouette score.","core_discovery":"The central claim is that a workload's runtime behavior can be profiled before it executes, using only metadata that is available when the workload is submitted. PolarisProfiler groups completed workloads into profile groups based on runtime similarity, trains a classifier on the static metadata of the workloads inside each group, and then assigns every new workload to a profile group from its metadata alone. The profile group provides quantile-based estimates of CPU usage, GPU usage, memory usage, and duration. On the Alibaba PAI trace, the resulting predictions have normalized RMSE below 50% for roughly 93% of 10,000 unseen workloads, and the same architecture produces strong classification and prediction results on Google cluster data, supporting the claim that the approach generalizes beyond a single dataset or workload type.","pith_inferences":["Editorial inference: the metadata-to-profile mapping could also support admission control and overcommitment decisions, since it yields not just a point estimate but a distribution of expected resource usage.","Editorial inference: a stronger test would apply the trained classifier to workloads from a later period or a different data center; the paper's 10,000 test workloads come from the same two-month Alibaba trace, so distribution shift is untested.","Editorial inference: the SHAP analysis identifies specific users, groups, and parameter-server tasks as decision drivers, suggesting that user and group identity encode recurring job templates; this could be exploited for better estimates or could become a robustness liability if users change over time.","Editorial inference: the 5th-quantile prediction rule is a simple summary; the paper's own skewness analysis implies that per-profile distribution-aware estimators could reduce the high-error tail."],"forward_implications":["An orchestrator can assign a newly submitted workload a runtime profile at submission time, without sandbox probing or waiting for that specific job's execution history.","Profiles carry quantile summaries, so resource provisioning can be based on expected variance rather than a single point estimate.","The feedback loop gives the system a mechanism to stay representative: when violations or outliers exceed thresholds, re-clustering refreshes the profile groups.","Because the formalization leaves the clusterer and classifier open, the same framework can be instantiated with different algorithms on different infrastructure traces.","The Google cluster result suggests the approach is not specific to machine-learning workloads, extending to general batch jobs with latency classes and resource requests."],"supporting_citations":[{"why":"Supplies the Alibaba PAI trace with the metadata features (user, job, workload, task, group) used for clustering and classification.","marker":"[70]"},{"why":"Supplies the Google cluster data used to test generalizability of the approach.","marker":"[66]"},{"why":"Provides the HDBSCAN algorithm used as the profile generator on the Alibaba dataset.","marker":"[10]"},{"why":"Provides the accelerated hierarchical density-based clustering implementation used in the evaluation.","marker":"[41]"},{"why":"The earlier PolarisProfiler work that this paper formalizes and extends.","marker":"[44]"},{"why":"The Hopkins statistic used to verify that the runtime data is clusterable before profile generation.","marker":"[5]"},{"why":"The SHAP explainability method used to interpret which metadata features drive the classifier's decisions.","marker":"[36]"},{"why":"The silhouette score used as the internal cohesion component of the ACQUIRES quality metric.","marker":"[60]"}],"fun_headline_variants":["Metadata-only profiling forecasts runtime before execution","Pre-run metadata profiles job resource needs precisely","Static metadata yields runtime estimates on the fly","93% of unseen workloads get runtime fits from metadata","Profiles from metadata slash resource guessing in continuum"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The historical runtime traces used to build profile groups faithfully represent the runtime behavior of the workloads that will arrive later.","fun_headline_variants_meta":{"raw":{"variants":["Metadata-only profiling forecasts runtime before execution","Pre-run metadata profiles job resource needs precisely","Static metadata yields runtime estimates on the fly","93% of unseen workloads get runtime fits from metadata","Profiles from metadata slash resource guessing in continuum"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000248,"raw_usage":{"total_tokens":1552,"prompt_tokens":956,"completion_tokens":596,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":572,"completion_tokens_details":{"reasoning_tokens":528}},"tokens_in":572,"tokens_out":596,"duration_ms":6503,"temperature":1.0,"reasoning_tokens":528,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:21:45.445429+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the same RMSE prediction test on workloads submitted after the training trace (a later month or a different cluster); if the share of workloads with error below 50% falls substantially below the reported 93%, the stationarity assumption underlying the real-time accuracy claim fails.","supporting_citations":[{"cited_title":"In 19th{USENIX} Symposium on Networked Systems Design and Implementation ({NSDI} 22) (2022)","cited_arxiv_id":null,"evidence_quote":"Supplies the Alibaba PAI trace with the metadata features (user, job, workload, task, group) used for clustering and classification."},{"cited_title":"Large-scale cluster management at google with borg","cited_arxiv_id":null,"evidence_quote":"Supplies the Google cluster data used to test generalizability of the approach."},{"cited_title":"J., Moulavi, D., and Sander, J","cited_arxiv_id":null,"evidence_quote":"Provides the HDBSCAN algorithm used as the profile generator on the Alibaba dataset."},{"cited_title":"Accelerated hierarchical density based clustering","cited_arxiv_id":null,"evidence_quote":"Provides the accelerated hierarchical density-based clustering implementation used in the evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The earlier PolarisProfiler work that this paper formalizes and extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The Hopkins statistic used to verify that the runtime data is clusterable before profile generation."},{"cited_title":"M., Erion, G., Chen, H., DeGrave, A., Prutkin, J","cited_arxiv_id":null,"evidence_quote":"The SHAP explainability method used to interpret which metadata features drive the classifier's decisions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The silhouette score used as the internal cohesion component of the ACQUIRES quality metric."}],"review_version":1}