{"id":"23a1f63e-e984-4951-b136-4d855db2caf7","arxiv_id":"2506.21467","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A config-search data model and a linear representative-point transfer method can reuse measurements across similar cloud workloads, cutting sampled configurations by up to 92% when a linear relationship holds.","lead":"This paper introduces Discovery Space, a shared data model for cloud configuration experiments, and a method that reuses measurements from one workload or hardware setup to speed up searches on similar setups. It reports up to 92% savings in search effort when a simple linear relationship connects the old and new setups.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'over 90% speed-up' is measured against brute-force enumeration, not against an optimizer; using the paper's own optimizer baselines, savings drop to roughly 50–55%, undercutting the headline claim.","rationale":"I read the paper's central claim as the quantitative 'over 90% configuration search speed-up' from RSSC. The most load-bearing condition for that claim is not only that the linear surrogate is accurate, but that the speed-up is measured against a meaningful alternative. The paper defines %savings against brute-force enumeration (Table VI caption), which is an unrealistic baseline for the configuration-search problem the paper itself motivates: optimizers halt after sampling a tiny fraction of the space (Table V), so the marginal benefit of RSSC over an optimizer is much smaller. Using the paper's own optimizer baselines from §V-D (optimizers take ~2× the samples of RSSC), the savings drop to roughly 50–55%. This concern does not require disputing the linearity assumption; even a perfect transfer would not yield 90% savings over a competent optimizer. The reader's weakest-assumption focuses on the linear-global-fit risk, which is a valid secondary concern, but the savings metric is more directly tied to the headline claim. I therefore recommend keeping the verdict CONDITIONAL, with a request to report savings against optimizer baselines and to release artifacts for reproducibility.","tokens_in":16275,"tokens_out":6078,"duration_ms":67343,"concrete_test":"Recompute the %savings for FT-TRANS and MI-TRANS using the median number of samples taken by the best-performing optimizer (BO/BOHB/Ax) on the same target space as the denominator, instead of the full space size; if the savings drop below 60%, the abstract's 'over 90%' claim should be qualified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"RSSC's reported savings (Table VI: 86–92%) are defined in the caption as 'time-saved over a brute force evaluation of all configurations.' For FT-TRANS, this is (56−8)/56 ≈ 86% or (56−5)/56 ≈ 92%; for MI-TRANS, (48−4)/48 = 92%. But the relevant alternative to configuration search is not exhaustive enumeration: the paper's own optimizer baselines in §V-D sample up to 2× more points than RSSC (e.g., 9 vs. 4 for FT-TRANS). Against that baseline, the savings are at most (9−4)/9 ≈ 55%, and for MI-TRANS plausibly around 50%. The abstract's unqualified 'configuration search speed-ups of over 90%' therefore overstates the benefit relative to existing search methods, even if the linear transfer model is perfectly accurate. The central quantitative claim is thus not supported by the evaluation as reported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Discovery Space, a data-model abstraction for configuration search that formalizes configuration spaces, action spaces, sample stores, and sampling records, with the goal of enabling workload-agnostic search, transparent data sharing, and knowledge reuse. It also introduces representative sub-space comparison (RSSC), a method that selects a small set of representative configurations from a source space, tests a linear transferability criterion against a related target space, and, if the criterion passes, fits a linear surrogate to predict the remaining target configurations. The evaluation covers four workloads (TPC-DS, two LLM inference tasks, and LLM fine-tuning) and three transfer scenarios, claiming configuration search speed-ups of over 90% in the abstract and reporting savings of 86-92% in Table VI.","tokens_in":16491,"tokens_out":5457,"duration_ms":53707,"significance":"The paper's main strength is the Discovery Space abstraction, which is clearly described and useful in principle: it decouples optimization algorithms from workload-specific experiments and provides a structured way to share measurements across runs. The incremental-sampling evaluation (Fig. 7) gives concrete evidence that reusing prior samples can reduce optimization cost, and the honest reporting of the SI-TRANS negative case is commendable. However, the headline quantitative claim is not supported as stated: the 'over 90% speed-up' is measured against brute-force enumeration, not against an optimizer baseline, and the authors themselves note that their optimizer baselines sample up to 2x more points than RSSC. The transfer evaluation also lacks statistical rigor (no error bars, unclear whether fit points are held out). If the claims are re-framed and the evaluation is tightened, the work could be a useful systems contribution, but as it stands the central claim overstates the benefit.","major_comments":[{"comment":"The abstract's claim of 'configuration search speed-ups of over 90%' is not supported by the evaluation as reported. The %savings in Table VI is defined in the caption as 'time-saved over a brute force evaluation of all configurations,' not over an existing optimizer baseline. In Section V-D the authors state that for FT-TRANS and MI-TRANS, optimizer runs 'can sample up to 2x more points than RSSC ... e.g. 9 versus 4 for FT-TRANS.' Against that baseline, the savings are at most (9-4)/9 ≈ 55% for FT-TRANS and roughly 50% for MI-TRANS, rather than 86-92%. Please re-state the speed-up claims relative to the paper's own optimizer baselines, or explicitly qualify that the 90%+ figure refers only to avoiding brute-force enumeration.","section":"Abstract and Section V-D, Table VI"},{"comment":"The evaluation does not state whether the representative sub-space points used to fit the linear surrogate are excluded from the reported best%, top5%, and rank-resolution metrics. Because the surrogate is fit on those points, including them in the quality metrics can artificially inflate the results (for example, FT-TRANS reports best%=100% with 8 representative points out of 56). Please clarify whether the metrics are computed on held-out points only, and if not, recompute them on the non-representative configurations.","section":"Section V-B2 and Table VI"},{"comment":"The transfer criteria (r>0.7, p<0.01) and the surrogate model are the same linear regression fit on the representative points. This means the go/no-go transfer decision is not validated independently of the predictor. The paper shows one negative case (SI-TRANS) and two positive cases, but does not demonstrate that the thresholds are predictive of held-out prediction quality in general. Given that only 4-33 points are used for the fit, the risk of overfitting is non-negligible. Please provide a validation of the transfer criteria against held-out data, or at least discuss this limitation explicitly.","section":"Section IV-3 and IV-4"},{"comment":"The transfer evaluation reports single values without any measure of variance or number of repetitions for the clustering-based RSSC. The optimizer baselines are run 10 times, but Table VI appears to report results from a single RSSC execution per test case. For a claim of this magnitude, confidence intervals or repeated runs are needed. Additionally, the three transfer test spaces are small (48-288 configurations), and only two of the three transfers succeed, so the 'over 90%' claim rests on a very narrow empirical base.","section":"Table VI and Section V-D"}],"minor_comments":[{"comment":"The title contains a typo: 'Reuseable' should be 'Reusable'.","section":"Title"},{"comment":"The sentence 'We used three three optimization methods' contains a duplicated 'three'; please remove the duplicate.","section":"Section V-B1"},{"comment":"The statement 'there have been no application of these frameworks to configuration search' is contradicted by the same section's discussion of CherryPick [5] and Morphling [7], which apply Bayesian optimization to configuration search; reword to 'no general-purpose framework has been widely applied' or similar.","section":"Section VI"},{"comment":"The x-axis labels appear to be instance types (e.g., c4.gm16Gi), but the caption describes 'increasing number of CPU cores and GPU memory allocation'; please clarify how the x-axis encodes these parameters.","section":"Figure 1"},{"comment":"The notation D=(P,Ω)⊗A is introduced, but the action space A and the Discovery Space A in Figure 5 share the same symbol; consider using a different symbol for the action space in the formal definition to avoid confusion.","section":"Section III-B1"},{"comment":"No code or data are released, which limits the reproducibility of the reported results; please consider making the implementation and benchmark data available or provide a detailed appendix.","section":"General reproducibility"}],"recommendation":"major_revision","confidential_remarks":"The paper's central abstraction is interesting, but the empirical evaluation does not support the headline speed-up claim as stated. The authors should re-benchmark against optimizer baselines, clarify whether fit points are excluded from quality metrics, and provide error bars. The lack of released code/data is also a concern for a systems paper. I recommend major revision to align claims with evidence and strengthen the evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing you should know: the abstract's 'configuration search speed-ups of over 90%' is not supported by the evaluation as stated. Table VI defines the savings as time saved over brute-force enumeration of all configurations. But no one runs a brute-force search. The paper's own optimizer baselines in Table V show optimizers sampling well under 2% of each space, and Section V-D says optimizers can sample up to 2x more points than RSSC (9 vs. 4 for FT-TRANS). Against that realistic baseline, the savings are around 50-55%, not 92%. The headline overstates the practical benefit. That said, the paper does real work. The TRACE characteristics and Discovery Space data model are a clean, sensible abstraction for workload-agnostic, shareable configuration search. They resemble Vizier/Optuna studies and trials, but the common-context schema and explicit reconciliation rules are a useful contribution. RSSC - clustering to pick representative points, a go/no-go linear-transfer criterion, and a surrogate built on the same fit - is a reasonable, simple transfer method, and its application to cross-space configuration transfer appears new in the cited literature. The evaluation covers four real workloads (TPC-DS, two LLM inference setups, fine-tuning) with exhaustively characterized spaces, and the SI-TRANS negative result is reported honestly, which I credit. The soft spots beyond the headline are fixable but real. First, the same linear regression is used both to decide transferability and as the surrogate predicting all unmeasured points. The paper does not state whether the representative points are excluded from the reported best% and top5% metrics. If they are not excluded, those metrics are inflated by construction. That needs to be clarified. Second, Table VI has no error bars or variance across clustering runs; the transfer criteria thresholds (r > 0.7, p < 0.01) are also arbitrary, though that is a minor point. Third, no code or data are released, which makes the transfer results hard to verify. The equal-time cost model is stated explicitly, but it is a simplification that could overstate savings when measurements vary in duration. The linear-transfer assumption is a genuine risk, but the go/no-go criterion partially mitigates it - SI-TRANS fails the test and no transfer is attempted. That is the right behavior. For whom? Cloud systems researchers working on configuration tuning, performance modeling, and transfer learning. It deserves a serious referee, but the authors need to rework the savings claim, release artifacts, add uncertainty estimates, and state whether representative points are excluded. I would send it to review, expecting major revisions.","headline":"The Discovery Space abstraction and RSSC transfer method are genuinely useful, but the 'over 90% speed-up' headline only holds against brute-force enumeration; against the paper's own optimizer baselines it drops to roughly 50-55%.","tokens_in":834,"tokens_out":831,"would_cite":true,"duration_ms":35338,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A Discovery Space data model plus representative sub-space comparison lets cloud configuration searches reuse and transfer data, cutting search cost by over 90%.","keywords":["configuration search","cloud resource optimization","data abstraction","knowledge transfer","surrogate model","black-box optimization","LLM inference","big data analytics"],"falsifier":"Run RSSC on a pair of spaces whose true relationship is known to be strongly nonlinear, for example the SI-TRANS case (A100 PCIE to A100 SXM4) where the paper reports $r=0.29$ and a failed transfer; the method's central claim would be falsified if, across many such pairs, the go/no-go criteria frequently pass while the predicted top-5 and rank resolution remain poor, or if they frequently fail while a nonlinear surrogate would have transferred well.","tokens_in":16074,"feed_emoji":"♻️","tokens_out":6990,"duration_ms":75433,"temperature":0.7,"pith_summary":"This paper claims that the high cost of cloud configuration search can be cut dramatically by treating search data as a shareable, structured resource rather than as a private by-product of each optimizer run. It proposes Discovery Space, a data model that wraps a workload's configuration space, its measurable properties, and all acquired samples into one object with a common schema, so multiple optimizers can read and add to the same store without interfering. On top of that model it builds representative sub-space comparison (RSSC), which selects a few representative configurations from a well-studied source space, measures their counterparts in a related target space, and uses a linear regression to decide whether knowledge transfers and to predict the unmeasured points. In the paper's evaluation, reusing samples across runs saves up to 94% of optimization cost, and RSSC transfers knowledge across similar workloads and GPU setups with up to 92% savings. A sympathetic reader would care because configuration spaces with millions of options make exhaustive measurement impossible, so any safe, automatically checked reuse directly reduces time and money.","feed_headline":"Reuse one workload's tuning data to cut cloud search cost by 92%","feed_subtitle":"A shared data model plus a handful of representative points lets one search transfer its findings to a similar space.","key_machinery":"The load-bearing object is the Discovery Space data model, defined as a tensor product of a configuration probability space and an action space, implemented as a class with four elements: the probability-space definition, the action-space definition, a shared sample store, and time-stamped sampling records. Its work is to give every configuration search a common schema so samples are reconcilable, actionability is encoded in the space itself, and operations are stateless. The second mechanism is representative sub-space comparison (RSSC): cluster source samples by the property to transfer, take one centroid per cluster, translate those configurations to the target space, measure them, fit a linear regression between source and target values, and only if $r>0.7$ and the slope p-value is below 0.01 use that regression as a surrogate to fill in the rest of the target space. The same linear fit is therefore doing double duty as a transferability test and as the predictive model.","core_discovery":"The central discovery is that a configuration search problem can be represented as a Discovery Space $D = (P,\\Omega)\\otimes A$, where $(\\Omega, P)$ is the configuration probability space and $A$ the action space of measurable experiments, and that this representation makes data reuse safe: a shared sample store with one schema means any sample written by one operation can be read by another as long as the configuration is generated through that space's sample method. The paper further claims that when two spaces differ by a small change (a different GPU model, a different LLM, a different attachment type), a handful of representative points, chosen by clustering the source space and translated by a parameter mapping, is enough to decide transferability. The transfer criteria are a correlation $r > 0.7$ and a slope p-value below 1% for the linear regression between source and target representative values; when these pass, the same regression serves as a surrogate model that predicts every unmeasured target configuration. The paper reports that in cases where the criteria pass, the transferred model identifies the best or near-best configuration, 60--100% of the true top-5, and a rank resolution of 1--3, with measured sampling savings up to 92%.","pith_inferences":["The paper's own SI-TRANS result is the productive failure case: it shows RSSC can flag when transfer is unsafe, but the criteria are tuned to linear relationships, so strongly monotonic but nonlinear mappings might be rejected even though a rank-preserving transfer could still help.","The same Discovery Space schema could be used to detect configuration-space drift over time, because sampling records are time-stamped; one could compare an old space to a re-measured sub-space and use RSSC's criteria to decide whether cached predictions are still valid.","RSSC's representative points could be chosen adversarially rather than by global clustering; the paper's top5 baseline already shows local selection can give false positives, so a natural extension is a point-selection rule that minimizes worst-case prediction error rather than cluster representativeness."],"forward_implications":["With passive incremental sampling, repeated independent optimizer runs on the same Discovery Space reuse earlier measurements, saving 20% of runtime by the 10th run on the largest tested spaces and up to 94% on the smallest.","When the RSSC transfer criteria pass, the linear surrogate predicts the best or near-best configuration, retrieves 60--100% of the true top-5 configurations in the top-5 predictions, and requires far fewer samples than running an optimizer on the target space.","Because no single optimizer dominates across workloads (BOHB wins for MULTIINF, Bayesian optimization for SINGLEINF, Ax for TPCDS), a workload-agnostic data model that lets multiple optimizers share one sample store is a practical necessity, not just a convenience."],"supporting_citations":[{"why":"Represents the class of workload-specific configuration search systems whose data are not shared between runs, the baseline against which the paper's sharing objective is measured.","marker":"[1]"},{"why":"Uses performance models to guide configuration search; a representative prior method that the Discovery Space abstraction generalizes beyond.","marker":"[5]"},{"why":"Targets near-optimal auto-configuration for model serving and is compared as a workload-specific approach.","marker":"[7]"},{"why":"Provides a study/trial data abstraction for black-box optimization that the paper contrasts with its shared sample store.","marker":"[9]"},{"why":"Another black-box optimizer with a study abstraction, used as a comparison point for data reuse and multiple algorithms.","marker":"[10]"},{"why":"Supplies one of the three black-box optimizers (BOHB) used in the optimizer comparison.","marker":"[11]"},{"why":"The distributed execution platform on which all three optimizer baselines run in the evaluation.","marker":"[13]"},{"why":"Empirical demonstration that the best black-box optimizer depends on the workload, which motivates the multiple-algorithm objective.","marker":"[20]"}],"fun_headline_variants":["Reuse cloud tuning data to cut search cost by 92%","Discovery Spaces transfer config search knowledge, saving 92%","Safe data sharing speeds cloud config search up to 92%","Transferable search spaces: 92% faster cloud tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a linear relationship fitted on a small set of representative points (4 to 33 samples in the experiments) holds across the whole target configuration space, so the same regression can serve both as the transfer test and as the predictor for all unmeasured configurations.","fun_headline_variants_meta":{"raw":{"variants":["Reuse cloud tuning data to cut search cost by 92%","Discovery Spaces transfer config search knowledge, saving 92%","Safe data sharing speeds cloud config search up to 92%","Transferable search spaces: 92% faster cloud tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000205,"raw_usage":{"total_tokens":1403,"prompt_tokens":968,"completion_tokens":435,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":584,"completion_tokens_details":{"reasoning_tokens":365}},"tokens_in":584,"tokens_out":435,"duration_ms":5399,"temperature":1.0,"reasoning_tokens":365,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:25:10.950319+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run RSSC on a pair of spaces whose true relationship is known to be strongly nonlinear, for example the SI-TRANS case (A100 PCIE to A100 SXM4) where the paper reports $r=0.29$ and a failed transfer; the method's central claim would be falsified if, across many such pairs, the go/no-go criteria frequently pass while the predicted top-5 and rank resolution remain poor, or if they frequently fail while a nonlinear surrogate would have transferred well.","supporting_citations":[{"cited_title":"OPTIMUSCLOUD: Heterogeneous Con- figuration Optimization for Distributed Databases in the Cloud,","cited_arxiv_id":null,"evidence_quote":"Represents the class of workload-specific configuration search systems whose data are not shared between runs, the baseline against which the paper's sharing objective is measured."},{"cited_title":"CherryPick: Adaptively unearthing the best cloud configurations for big data analytics,","cited_arxiv_id":null,"evidence_quote":"Uses performance models to guide configuration search; a representative prior method that the Discovery Space abstraction generalizes beyond."},{"cited_title":"Morphling: Fast, near-optimal auto- configuration for cloud-native model serving,","cited_arxiv_id":null,"evidence_quote":"Targets near-optimal auto-configuration for model serving and is compared as a workload-specific approach."},{"cited_title":"Open Source Vizier: Distributed Infrastructure and API for Reliable and Flexible Blackbox Optimization","cited_arxiv_id":"2207.13676","evidence_quote":"Provides a study/trial data abstraction for black-box optimization that the paper contrasts with its shared sample store."},{"cited_title":"BOAH: A Tool Suite for Multi-Fidelity Bayesian Optimization & Analysis of Hyperparameters","cited_arxiv_id":"1908.06756","evidence_quote":"Supplies one of the three black-box optimizers (BOHB) used in the optimizer comparison."}],"review_version":1}