{"id":"dfe12fd0-0a09-4c31-8820-6c3a388dab7d","arxiv_id":"2411.11325","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":8,"one_line_summary":"A three-stage profile-based SKU recommender for new cloud resources, claiming over 60% slack reduction via synthetic evaluation and a preference-learning loop.","lead":"Lorentz recommends initial cloud database sizes from customer profile data instead of workload traces, and reports cutting wasted capacity by more than 60% in tests. The system matters because many new cloud services have no usage history, and poor sizing either wastes money or causes performance throttling.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The >60% slack-reduction claim rests on a synthetic test set whose upscaling injects the same profile features the provisioners consume; on unmodified production data the paper reports no improvement over defaults, so the headline result is unsupported for real deployments.","rationale":"I read the paper in good faith: Lorentz addresses a real cold-start provisioning problem, and the three-stage design is coherent and clearly described. The Stage 1 rightsizing evaluation on real production telemetry is legitimate and shows a 34% slack reduction with eliminated throttling relative to user selections. However, the paper's headline claim of 'over 60% slack reduction' depends entirely on the Stage 2 provisioner evaluation, and that evaluation is circular. The synthetic upscaling procedure in Section 5.2 constructs workload labels from the same profile features (ResourceGroup, CloudCustomerGuid, VerticalName) that the provisioners use as inputs. A model trained and evaluated on such data can achieve high accuracy by learning the injected mapping, which tells us nothing about whether real profile tags are predictive of capacity needs. The paper's own real-data result undermines the headline: on unmodified production data, the provisioners match default baselines, and the maximum possible slack reduction is 33%. The concluding sentence of Section 5.2 acknowledges the missing verification that profile data is informative. The Pith Reader's weakest assumption correctly identifies the same load-bearing concern, so I agree with the REJECT verdict. My proposed check, permuting the synthetic scale factors to break the profile-to-label link, would directly test whether the reported advantage is an artifact of the synthetic procedure.","tokens_in":21277,"tokens_out":3914,"duration_ms":42096,"concrete_test":"Re-run the Section 5.2 synthetic evaluation exactly, but after computing the profile hierarchy, randomly permute the assigned scale factors across feature values so that the upscaling multiplier chi_w is independent of ResourceGroup, CloudCustomerGuid, and VerticalName while preserving the same marginal distribution of workload scales. Then retrain the hierarchical and target-encoding provisioners on this permuted synthetic data and recompute the slack/throttling Pareto curves in Figures 10-11. If the 66%/54% slack reductions shrink to near baseline, the original result is an artifact of the injected profile-to-label dependency; if they persist, the provisioners generalize beyond the injected signal and the concern is mitigated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Lorentz's central quantitative claim is that it reduces wasted capacity by over 60% without increasing throttling. Section 5.2 shows that on the unmodified production test set, the provisioners achieve 'similar average slack to default-value baselines' and that the maximum possible slack reduction is 33%. The >60% figure comes instead from 'Synthetic workload upscaling,' where each workload is multiplied by 2^chi_w and chi_w is computed by assigning scale factors to exactly three profile features: ResourceGroup, CloudCustomerGuid, and VerticalName. These are the same profile features that the hierarchical and target-encoding provisioners are trained on. The synthetic labels are therefore generated as a deterministic function of the input features, so the evaluation demonstrates only that the models can recover an injected dependency, not that real profile tags carry information about capacity demand. The paper itself flags this in the final paragraph of Section 5.2: 'it's important that any VM profile data used as inputs is verified to be informative of their corresponding workloads.' That verification is not provided. The abstract and conclusion nevertheless state that production-data evaluation demonstrates >60% slack reduction, which is inconsistent with the reported 33% upper bound on real data. If profile data is not actually informative of demand, the headline result disappears.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Lorentz, a three-stage system for recommending initial cloud compute SKUs without workload traces: (1) rightsizing existing workloads from utilization telemetry to produce training labels, (2) two profile-data-based provisioners (a hierarchical bucket model and a target-encoding tree model), and (3) an online personalizer that adjusts recommendations using customer satisfaction signals. The authors evaluate on production data from Azure PostgreSQL DB and on synthetic workloads, claiming that Lorentz reduces slack by over 60% without increasing throttling and that the personalizer converges to true customer preferences. The central claim is shown to rest on the synthetic upscaling experiment, while the unmodified production-data evaluation reports only a 33% maximum possible slack reduction and no improvement over default baselines.","tokens_in":21571,"tokens_out":3356,"duration_ms":32667,"significance":"If the headline result were valid, Lorentz would be a practically valuable contribution: it addresses a real provisioning problem, requires no workload traces, provides explainable recommendations, and includes a feedback loop for personalization. The production rightsizing analysis (Stage 1) and the synthetic personalization convergence study are useful engineering contributions, and the paper is generally well written. However, the central quantitative claim is not supported by the evidence presented. On unmodified production data the provisioners do not beat defaults, and the >60% figure comes from a synthetic experiment whose label-generation procedure injects the same profile features that the provisioners consume. Because the main claimed advantage over existing approaches depends exactly on profile data being informative of workload demand, and because that informativeness is not established, the contribution as stated is not credible.","major_comments":[{"comment":"The paper's headline result—'reducing slack by >60%' (Abstract and Section 7)—is obtained from the synthetic upscaling experiment, not from unmodified production data. In that experiment, each workload is multiplied by 2^{χ_w}, where χ_w is computed by assigning scale factors to the three profile features ResourceGroup, CloudCustomerGuid, and VerticalName; these are exactly the features used by the hierarchical and target-encoding provisioners. The evaluation thus demonstrates largely that the models can recover a dependency injected by the label-generation process. The paper itself acknowledges in the final paragraph of Section 5.2 that 'it's important that any VM profile data used as inputs is verified to be informative of their corresponding workloads,' but no such verification is provided. On the unmodified production test set, Section 5.2 reports that the provisioners 'achieve similar average slack to default-value baselines at most throttling ratios, with a maximum possible slack reduction of 33%.' Therefore the central quantitative claim is unsupported for real deployments.","section":"§5.2 (Synthetic workload upscaling; Provisioner evaluation)"},{"comment":"The abstract and conclusion state that evaluation on production data demonstrates >60% slack reduction or elimination of >60% of wasted COGS. This is inconsistent with Section 5.2, which reports a maximum possible slack reduction of 33% on the real test set and cost reductions of 27% (hierarchical) and 8% (target encoding) relative to user selection. The >60% number appears only in the synthetic workload experiment. The claims in the abstract and conclusion must be corrected to attribute the result to synthetic data, or the synthetic experiment must be shown to be representative of production conditions; as written, the paper overstates what was measured.","section":"Abstract; §7 (Conclusion)"},{"comment":"The rightsizing of censored workloads assumes that the desired capacity is at least 2^K times the currently selected capacity for every throttled workload, with K = 1 in the experiments. This is an assumption, not derived from data; the text says K can be calibrated using telemetry of formerly throttled servers whose capacities were scaled up, but no such calibration evidence is presented. If this assumption is inaccurate, the Stage 1 labels are systematically biased, and that bias propagates through Stage 2 and inflates apparent slack improvements. Please provide the calibration analysis or a sensitivity study over K.","section":"§3.2, Eq. (9); Table 2"}],"minor_comments":[{"comment":"The statement that users select the ideal capacity only 43% of the time is defined relative to Lorentz's own rightsized capacities ('relative to Lorentz's rightsized capacities (see Section 3.2)'), not against an independent ground truth. The abstract and introduction phrase this as an objective fact about user behavior; the paper should make the relativity explicit in those places.","section":"§2.2; §1"},{"comment":"The Pareto comparison to user selections is implemented through default-value baselines, justified by the observation that 89% of users choose the default or the next larger vCore value. This is a reasonable approximation, but the slack-reduction percentages in the text are relative to these default baselines rather than to the actual user-selected SKU distribution on the upscaled test set; the comparison should be stated more precisely.","section":"§5.2, Figure 10"},{"comment":"There are several typos in this subsection: 'geenrated' should be 'generated', 'labled' should be 'labeled', and in the caption of Figure 7 'defied decay rate' should be 'defined decay rate'.","section":"§5.3.1"},{"comment":"The Stage 1 evaluation uses absolute slack (slack ratio multiplied by capacity), while the rightsizing optimizer in Eq. (7) operates on slack ratios. The text notes this is deliberate for business cost alignment, but the discrepancy between the optimized objective and the reported metric should be discussed explicitly, since the 34% slack reduction may partly reflect the weighting rather than the rightsizing criterion.","section":"§5.1, Figure 9"}],"recommendation":"reject","confidential_remarks":"The central issue is not presentation but the validity of the main empirical claim: the >60% slack reduction is a synthetic-data artifact, and the production-data evaluation shows no improvement over defaults. This cannot be repaired by a local revision, because the paper's core thesis—that profile data alone can predict capacity needs—is exactly what remains unverified. The authors might consider resubmitting as an experience/vision paper with an honest statement of the production-data upper bound and a clear research agenda for validating profile informativeness, but in its current form the manuscript does not support its stated conclusions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a coherent system paper for a genuinely hard problem—recommending an initial cloud SKU without any workload trace—but the headline result is not supported by the evidence as written. The framework is worth knowing; the >60% claim should not be repeated.\n\nWhat's new: Lorentz combines rightsized telemetry labels, hierarchy-based and target-encoding provisioners, and a propagation-style personalizer for the cold-start setting. That combination is genuinely different from Doppler and Autopilot, which both require traces. Stage 1 rightsizing is sensible and evaluated on real telemetry: it eliminates throttling and cuts slack by 34% relative to user choices. The personalization simulation is clearly described and shows convergence, though only on synthetic signals.\n\nThe problem is Section 5.2. On the unmodified production test set, the provisioners only match default baselines, and the paper reports a maximum possible slack reduction of 33%. The >60% figure comes from synthetic workloads upscaled by 2^chi, where chi is assigned using exactly the profile features the provisioners consume—ResourceGroup, CloudCustomerGuid, and VerticalName. That makes the synthetic labels a deterministic function of the input features, so the evaluation shows the models can recover an injected dependency, not that real profile tags carry capacity-demand information. The paper itself flags this in the final paragraph of Section 5.2: profile data \"must be verified to be informative of their corresponding workloads.\" That verification is never provided. Yet the abstract and conclusion state production data demonstrates >60% slack reduction, which is inconsistent with the 33% ceiling on real data. This is a load-bearing flaw, not a quibble.\n\nMinor: the Stage 3 evaluation is entirely simulated with hand-set ground truth, so it demonstrates algorithm behavior, not evidence about real CRIs. The CRI dataset has only 5 price-sensitive tickets out of 4,400, so the signal is extremely sparse, and the propagation decay parameters are under-specified. These are limitations, not fatal flaws.\n\nWho benefits: systems folks working on cloud provisioning, and anyone who wants a clear example of how synthetic upscaling can create circular validation. It's a useful cautionary read. The framework is salvageable—evaluate on real cold-start provisioning outcomes, or at least report the real-data result honestly and demote the synthetic claim. I'd send it to a serious referee, but I would not accept it as is, and I wouldn't cite the headline result.","headline":"Lorentz is a coherent cold-start SKU recommendation framework with a real problem and a sensible Stage 1, but the >60% slack reduction headline is supported only by a circular synthetic evaluation and contradicted by the paper's own real-data results.","tokens_in":731,"tokens_out":798,"would_cite":false,"duration_ms":27761,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Lorentz claims that cloud capacity for new databases can be chosen near-optimally from customer profile tags alone, cutting wasted capacity by over 60% without extra throttling, by rightsizing existing workloads and continuously…","keywords":["SKU recommendation","cloud provisioning","capacity rightsizing","customer profile data","personalization","slack reduction","throttling","target encoding"],"falsifier":"Run Lorentz end-to-end on a fresh cloud service whose profile tags have been randomly permuted across customers, or on the real unmodified production workloads without the synthetic upscaling; if slack reduction against static defaults collapses toward zero, the claim that profile data alone carries the capacity signal is refuted. The paper already reports that on unmodified production data the provisioners only match default baselines, which is the same experiment at a smaller scale.","tokens_in":21056,"feed_emoji":"☁️","tokens_out":7674,"duration_ms":72672,"temperature":0.7,"pith_summary":"Lorentz is a three-stage system for recommending the initial compute capacity, or SKU, of a cloud database before any workload has run on it. The paper's central claim is that customer profile tags—industry, segment, subscription, resource group—carry enough signal about future capacity needs to beat both user choices and static defaults: on Azure PostgreSQL DB, Lorentz cuts wasted capacity by more than 60% without increasing throttling. The system first rightsizes existing databases from their usage telemetry to create reliable training labels, then learns a mapping from profile features to rightsized capacity, and finally personalizes each recommendation by learning the customer's price-versus-performance preference from satisfaction signals. If the claim holds, new cloud resources can be provisioned near-optimally at creation time, saving cost and avoiding throttling without requiring workload traces.","feed_headline":"Profile data alone cuts wasted cloud capacity by 60%","feed_subtitle":"It sizes cloud databases before any workload runs, using profile tags and customer feedback.","key_machinery":"The load-bearing mechanism is the three-stage pipeline plus one identity: the personalized recommendation is a multiplicative shift in log-capacity space, $c^{**} = b^{\\lambda} c^{*}$, so the learned parameter $\\lambda$ is interpretable as 'how many powers of $b$ to raise or lower the Stage 2 prediction.' Stage 1's rightsizing optimizer selects the SKU whose slack is closest to a target $s^{*}$ subject to a throttling bound $\\tau$, treating censored workloads specially by requiring at least $2K \\times$ current capacity. Stage 2's hierarchical provisioner learns a feature hierarchy from profile data via an entropy-based method and recommends the $p$-th percentile of rightsized capacities from the coarsest bucket with enough samples; the target-encoding alternative replaces each categorical tag with the mean label of that tag's bucket and feeds the encodings to a gradient-boosted tree model. Stage 3's personalizer stores a $\\lambda$-profile per customer, subscription, resource group, and server offering, and updates it by weighted message propagation with decay factors across stratifications, resource groups, and subscriptions.","core_discovery":"On its own terms, the paper establishes that SKU selection for newly created cloud resources can be treated as a supervised learning problem whose labels come from a rightsizing pass over existing users rather than from the users' own, often wrong, choices. Stage 1 computes a rightsized capacity for each existing database by minimizing slack subject to a zero-throttling constraint, using a censored-workload rule that scales throttled machines up by at least $2K$. Stage 2 offers two provisioners—a hierarchical bucket model that recommends the $p$-th percentile of rightsized capacities among similar customers, and a target-encoding model that feeds encoded profile tags into a tree-based regressor. Stage 3 adjusts the Stage 2 recommendation by a per-customer score $\\lambda$ through $c^{**} = b^{\\lambda} c^{*}$, learned by propagating sparse satisfaction signals across resource groups and subscriptions. The paper reports that the provisioners reduce mean slack by 66% (hierarchical) and 54% (target encoding) at comparable throttling on synthetically upscaled workloads, and that the personalizer converges in simulation to within half a capacity step of the true preference.","pith_inferences":["The gap between the synthetic results (66% slack reduction) and unmodified production results (matching defaults) suggests the realized benefit on a new service will be proportional to how much real profile-tag diversity exists; a service with uniform low-usage workloads cannot expect the headline gain.","Because $\\lambda$ multiplies capacity in powers of two, the reported convergence error of 0.5 corresponds to recommending an adjacent SKU size; the personalizer's guarantees are therefore discrete-grid guarantees, not continuous capacity accuracy.","Randomizing or anonymizing profile tags would make a clean stress test: if slack reduction survives, the model is exploiting some other signal, and if it collapses, the tags are doing the work.","The same pipeline could be applied to any cloud resource with billing and profile tags, and adding trace data when it becomes available would tighten within-bucket SKU variability, bridging offline initial provisioning and later autoscaling."],"forward_implications":["New database instances receive a recommended SKU at creation time, before any workload trace exists, which removes the default-minimum bias that leads many users to pick the smallest option.","Across the evaluated synthetic workload diversity, following the recommendation instead of user selections or defaults reduces wasted capacity by over 60% at the same throttling level.","The hierarchical provisioner retains near-full accuracy when trained on only 10% of the data, so the approach is deployable where labeled examples are scarce.","The personalization loop converges to within half a SKU step of true preference in simulation even with sparse and noisy feedback signals.","Because the system keeps the similar-customer buckets and the learned $\\lambda$ profile, it can present the user with the matching customers and the current preference score as an explanation for each recommendation."],"supporting_citations":[{"why":"Establishes the prior automated SKU recommendation method that requires workload traces, the contrast that Lorentz removes.","marker":"[2]"},{"why":"Provides the autoscaling approach that Stage 1 builds on for adjusting existing workloads before using them as labels.","marker":"[29]"},{"why":"Supplies the entropy-based hierarchy learning used to order profile features in the hierarchical provisioner and personalization structure.","marker":"[37]"},{"why":"Supplies the gradient-boosted tree regressor that the target-encoding provisioner uses on encoded profile features.","marker":"[17]"},{"why":"Motivates the message-propagation rule that spreads sparse customer satisfaction signals across subscriptions and resource groups.","marker":"[25]"},{"why":"Supports the choice of tree-based models over deep learning for tabular profile data, which the target-encoding provisioner depends on.","marker":"[13]"}],"fun_headline_variants":["Profile-based SKU recommendations slash cloud waste by 60%","No workload traces? Profile data still cuts cloud slack 60%","Learn cloud sizes from profiles, not workload traces","Profile tags and feedback cut cloud capacity waste 60%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire headline reduction depends on customer profile tags (industry, segment, resource group) actually predicting how much compute a new database will need, and on the synthetic test in Section 5.2—which makes workloads larger exactly for customers with certain profile tags—being a realistic stand-in for real workload diversity.","fun_headline_variants_meta":{"raw":{"variants":["Profile-based SKU recommendations slash cloud waste by 60%","No workload traces? Profile data still cuts cloud slack 60%","Learn cloud sizes from profiles, not workload traces","Profile tags and feedback cut cloud capacity waste 60%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000279,"raw_usage":{"total_tokens":1657,"prompt_tokens":946,"completion_tokens":711,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":562,"completion_tokens_details":{"reasoning_tokens":643}},"tokens_in":562,"tokens_out":711,"duration_ms":7173,"temperature":1.0,"reasoning_tokens":643,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:39:44.180990+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Lorentz end-to-end on a fresh cloud service whose profile tags have been randomly permuted across customers, or on the real unmodified production workloads without the synthetic upscaling; if slack reduction against static defaults collapses toward zero, the claim that profile data alone carries the capacity signal is refuted. The paper already reports that on unmodified production data the provisioners only match default baselines, which is the same experiment at a smaller scale.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the autoscaling approach that Stage 1 builds on for adjusting existing workloads before using them as labels."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the entropy-based hierarchy learning used to order profile features in the hierarchical provisioner and personalization structure."},{"cited_title":"Lightgbm: A highly efficient gradient boosting decision tree","cited_arxiv_id":null,"evidence_quote":"Supplies the gradient-boosted tree regressor that the target-encoding provisioner uses on encoded profile features."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates the message-propagation rule that spreads sparse customer satisfaction signals across subscriptions and resource groups."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supports the choice of tree-based models over deep learning for tabular profile data, which the target-encoding provisioner depends on."}],"review_version":1}