{"id":"603a8053-b1c3-40c5-97cf-f1c8044a1a62","arxiv_id":"2504.18714","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":12,"one_line_summary":"Bayesian Optimization of memory tiering knobs yields up to 2x performance gains over default configurations, although the best settings rarely transfer across inputs or hardware.","lead":"This paper tunes the internal settings of memory tiering systems using Bayesian Optimization, a guided search method. The results show that better settings speed up memory-heavy workloads by up to 2x, which matters as data centers adopt slower CXL and NVM memory tiers.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline 2x claim depends on a modified HeMem with a new write-sampling knob and four behavior-changing fixes; stock-HeMem applicability is unverified.","rationale":"Good-faith reading: the paper asks how much performance can be gained by using application behavior to set tiering parameters. The empirical answer rests on comparisons inside a modified HeMem. The modifications are disclosed, which is commendable, but they change what 'default' and 'existing system' mean. Fix 3 is the sharpest issue: stock HeMem deliberately ignores allocations under 1 GiB, and lowering this threshold to 128 MiB makes previously ignored workloads subject to tiering. That is a system change, not a knob value. Fix 4 adds a new degree of freedom; tuning a newly added knob does not demonstrate that existing knobs were the bottleneck. Fix 1 may disproportionately benefit the high sampling frequencies that BO tends to select. Thus the 2x-over-default claim is not yet established for the publicly available HeMem. This matches the reader's weakest assumption. I do not see an internal inconsistency in the BO methodology itself, and the transfer experiments honestly bound the generality of tuned configurations. No verdict change is needed: the paper should remain conditional on artifact release and on a stock-HeMem validation. I would not reject outright, because the qualitative analysis of why defaults migrate useless pages is valuable and partially supported even without the exact 2x number.","tokens_in":19160,"tokens_out":6442,"duration_ms":68577,"concrete_test":"Check out the unmodified upstream HeMem release and rerun the Section 4.2 tuning pipeline on GUPS, Silo, and Btree with the same 100-iteration SMAC budget and at least 3 independent repetitions, keeping the original 1 GiB minimum allocation and the original unified read/write sampling behavior. If the best-tuned configuration no longer beats the stock default by roughly 2x on these workloads, the headline claim should be restricted to the authors' modified HeMem variant rather than to existing tiering systems.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that tuning the parameter knobs of existing tiering systems yields up to 2x improvement over default configurations. For HeMem, the experiments in Section 4.2 run against a variant that the authors modified in Section 3.2. Fix 3 changes the minimum allocation size from 1 GiB to 128 MiB, so workloads whose largest allocations fall in that range are tiered by the modified HeMem but were ignored by stock HeMem; the Figure 2 default is therefore the modified default, not the stock default. Fix 1 changes PEBS interrupt generation, which selectively lowers the overhead of high sampling rates that several best configurations use (Table 5), potentially inflating the tuned-over-default gap. Fix 4 adds a new WRITE_SAMPLING_PERIOD knob that does not exist in stock HeMem, and the Btree analysis in Table 5 explicitly credits this knob for the speedup; part of the gain is not obtained by tuning knobs that exist in the released system. Consequently, the reported 2x-over-default comparison has not been established for the publicly available HeMem, and the same reservation applies to the 1.56x-over-Memtris comparison, which uses the same modified HeMem.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes using Bayesian Optimization (SMAC) to tune the configuration knobs of existing memory tiering systems (HeMem and HMSDK). It reports a grid-search case study showing that two HeMem thresholds cause large performance differences, then applies BO to select knob values per workload. The evaluation reports up to 2.09x improvement over HeMem's default configuration and 1.56x over Memtis, along with analyses of why specific knob changes help (e.g., eliminating useless migrations, adjusting write sampling). It also tests generalization across inputs, thread counts, memory ratios, and machines, and reports a transfer experiment showing that configurations often do not transfer across inputs.","tokens_in":19437,"tokens_out":6570,"duration_ms":57545,"significance":"If the quantitative claims hold, the paper demonstrates a simple, practical mechanism for improving tiered-memory performance without redesigning the tiering engine, and it provides a useful negative result about the limits of configuration transfer. The per-workload analysis of migration patterns is a strength and gives concrete guidance for future adaptive tiering designs. However, the central quantitative claim rests on a modified HeMem implementation whose behavior differs from the stock system, and the experiments lack repeated runs, so the magnitude of the claimed gains is not yet established. The paper is transparent about the transfer failures, which is a point in its favor, but the headline numbers should be interpreted as upper-bound, per-workload fitted improvements rather than expected production gains.","major_comments":[{"comment":"The four modifications to HeMem described in §3.2 change the system being evaluated. Fix 3 reduces the minimum allocation size from 1 GiB to 128 MiB, so workloads whose largest allocations fall in the 128 MiB–1 GiB range are tiered by the modified HeMem but would be ignored by the stock system; the default baseline in Figure 2 is therefore the modified default, not the stock default. Fix 4 introduces a new knob, WRITE_SAMPLING_PERIOD, that does not exist in stock HeMem, and Table 5 credits that knob for the Btree speedup. Consequently, the headline \"2x over default\" claim (Abstract, §7) has not been established for the publicly available HeMem. Please either repeat the key comparisons on stock HeMem, or scope the claim explicitly and provide an ablation showing the contribution of each modification to the default-vs-best gap.","section":"§3.2 (Deployment Issues), §4.2 (Figure 2), Table 5"},{"comment":"The reported speedups appear to come from a single evaluation per configuration. The best configuration is selected by BO as the minimum of a noisy objective, and the improvement is measured against one run of the default; without repeated runs or confidence intervals, the magnitude of the improvement (e.g., 2.09x in Figure 2) can be inflated by measurement noise. Please run the default and the final best configuration multiple times (at least 3–5) for the main workloads and report mean and standard deviation, and also report the BO runs (e.g., seeds or multiple optimizer trials) to show the stability of the found configuration.","section":"§4.1 (Experimental Setup), Figures 2, 6, 7, 9, 10, 11, 13"},{"comment":"The comparison to Memtis is asymmetric. HeMem-Best is a per-workload configuration selected by the optimizer after 100 iterations of evaluating that same workload, whereas Memtis is evaluated only with its default configuration and, as far as the paper states, without any tuning of its parameters. The \"1.56x over state-of-the-art tiering system\" claim in the Abstract thus measures tuned HeMem against untuned Memtis, not parameter tuning generally. Either tune Memtis's own knobs with the same BO pipeline and report that result, or explicitly narrow the claim to \"tuned HeMem beats untuned Memtis\" and discuss whether Memtis would also benefit from tuning.","section":"§4.6 (Figure 13), Abstract"},{"comment":"The transfer experiments in §4.3 show that the best configuration for one input often performs worse than the default on another input (Figure 7). This is an honest and valuable result, but it also implies that the \"up to 2x\" gain is a fitted maximum for a workload whose full access behavior is known in advance, hence not the expected gain in an online or production setting. The conclusion should explicitly state that the reported improvements are upper-bound estimates for the offline per-workload tuning scenario, and should qualify the \"2x\" headline accordingly.","section":"§4.3 (Figure 7), §7 (Conclusion)"}],"minor_comments":[{"comment":"The execution-time heatmaps in Figure 1 do not indicate whether each cell is a single run or aggregated over multiple runs; please clarify.","section":"§2 (Figure 1)"},{"comment":"The claim that removing PERF_SAMPLE_WEIGHT does not affect sampling accuracy is not validated; please provide evidence, such as a comparison of sampled access counts before and after the change.","section":"§3.2 (item 1)"},{"comment":"The spelling of \"HeMem\" is inconsistent (e.g., \"Hemem\" appears in the Abstract); please standardize throughout.","section":"Abstract and Introduction"},{"comment":"The far-memory bandwidth column lists two values (e.g., 7.45/2.25) without defining the two measures; please specify whether these are read/write bandwidths.","section":"Table 3"},{"comment":"The thread-count labels on the x-axis are not defined in the caption; please clarify what \"2 4 6 8\" and \"12\" refer to.","section":"§4.4.1 (Figure 9a)"},{"comment":"The memory-ratio labels (\"2-1\", \"1-1\", \"1-2\", \"1-4\", \"1-8\", \"1-16\") are not explained in the caption; please define the ordering, e.g., fast:slow tier size ratio.","section":"§4.4.2 (Figure 9b)"},{"comment":"The paper does not state whether the modified HeMem code and tuning pipeline will be released; a reproducibility statement or artifact pointer would strengthen the work.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper's core qualitative finding—that tiering knobs matter and BO can find good configurations—is plausible and useful. The load-bearing issue is that the quantitative claims are based on a modified HeMem implementation with an added knob and behavior-changing fixes, so the numbers should not be attributed to the stock system without additional experiments. Adding repeated runs and an ablation of the HeMem modifications would address the main review concerns. The paper also does not mention artifact availability, which is increasingly expected for systems work."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. First, this is the first paper I've seen that treats memory-tiering parameters as tunable knobs and runs Bayesian optimization over them, and the workload-level analysis of why default configurations fail is genuinely useful. Second, the headline 2x improvement is real in their setup, but their setup is not stock HeMem, and that changes how much weight the headline should carry.\n\nWhat's new: applying BO to HeMem and HMSDK is a new application of an established method, and the empirical findings are informative. The grid-search case study shows large performance swings from just two knobs. The discovery of a hidden knob (cooling_pages) that matters for Silo is a concrete insight. The analysis of why defaults fail — unnecessary migrations for streaming workloads, write-heavy initialization phases, high sampling overhead — is well supported by the migration traces and access heatmaps. The transfer experiments are honest: tuned configurations often fail to generalize across inputs or hardware, and the authors say so plainly. The HMSDK results show the issue is not specific to HeMem.\n\nThe soft spots are real but proportionate. Section 3.2 documents four changes to HeMem: a minimum allocation size change from 1GiB to 128MiB, a PEBS interrupt fix, a race condition fix, and a new WRITE_SAMPLING_PERIOD knob. The minimum allocation change means workloads with allocations between 128MiB and 1GiB are now tiered by the modified system but would be ignored by stock HeMem. The PEBS fix reduces sampling overhead, which preferentially benefits the high sampling rates that many best configurations use. And the new write-sampling knob is explicitly credited for the Btree speedup in Table 5. So the comparison against 'default' is against the modified default, not the publicly available HeMem. That doesn't kill the paper's qualitative case, but it does mean the 2x number is not established for the stock system. There are also no error bars or repeated runs shown, and no code or data release, which makes it hard to verify any of the magnitudes.\n\nWho should read it: systems researchers working on tiered memory and adaptive auto-tuning. The paper deserves a serious referee; the direction is sound and the analysis is useful. I'd push for artifact evaluation and for the authors to either release the modified HeMem or re-run the headline comparison against stock, and to add variance information.","headline":"Useful first application of Bayesian optimization to memory-tiering knob tuning, with honest transfer experiments, but the headline 2x claim rests on a modified HeMem baseline and needs statistical and artifact backing.","tokens_in":19963,"tokens_out":2361,"would_cite":true,"duration_ms":22838,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Existing memory tiering systems can run up to 2x faster when a Bayesian optimizer tunes their configuration knobs.","keywords":["memory tiering","Bayesian optimization","parameter tuning","page migration","hot and cold pages","HeMem","HMSDK","CXL memory"],"falsifier":"Run the published HeMem code unmodified on the same eight workloads with the paper's best-per-configuration knob values; if the tuned values do not beat the stock default on most workloads, the claim that tuning existing systems yields up to 2x is contingent on the authors' modifications. A quicker check: measure whether the stock HeMem default, with only the write_sampling_period knob added, already accounts for most of the performance gap.","tokens_in":18935,"feed_emoji":"⚙️","tokens_out":4979,"duration_ms":46624,"temperature":0.7,"pith_summary":"The paper argues that existing memory tiering systems leave performance on the table because they rely on static, pre-configured thresholds that do not adapt to workload behavior or hardware. It proposes treating a tiering system as a black box and using Bayesian optimization to search its parameter space, finding that a handful of tuned knob values can improve performance by up to 2x over default configurations and 1.56x over Memtis, a state-of-the-art dynamic system. The paper also shows that the best settings depend on workload, input, thread count, and fast-to-slow memory ratio, and that hidden implementation parameters such as HeMem's cooling_pages can matter as much as the documented ones. If true, this suggests that tiering systems should be designed to adapt all their parameters, not just one or two thresholds.","feed_headline":"Tuning memory-tier knobs doubles performance over defaults","feed_subtitle":"A Bayesian optimizer finds workload-specific settings that beat both defaults and state-of-the-art tiering.","key_machinery":"The central object is the configuration space of a tiering system's parameters, searched by Sequential Model-based Algorithm Configuration (SMAC), a Bayesian optimization framework whose random-forest surrogate predicts execution time for candidate knob settings and whose acquisition function balances exploration and exploitation. The argument is carried by HeMem's knobs, including read_hot_threshold, write_hot_threshold, cooling_threshold, sampling_period, migration_period, and the hidden cooling_pages, plus a newly exposed write_sampling_period; the optimizer recompiles HeMem with each suggested setting. The same pipeline tunes HMSDK's DAMON-based monitoring parameters. The key mechanism is that different knob combinations produce different page migration patterns, so the optimizer's search over knobs effectively searches over data-placement policies.","core_discovery":"On its own terms, the paper establishes that the parameters (knobs) of existing tiering systems like HeMem and HMSDK encode enough flexibility to realize workload-specific data placement and migration policies. A Bayesian optimizer (SMAC) given a budget of 100 runs learns which knob values minimize execution time for a given workload and hardware, converging sample-efficiently. Across eight diverse workloads, the best found configuration outperforms the default by 1.07–2.09x; the strongest cases come from workloads with streaming access patterns where the default wastes bandwidth on migrations. The paper also demonstrates that tuning benefits generalize across machines, inputs, thread counts, and memory ratios, though gains shrink when tiers have similar bandwidth (NUMA-emulated CXL).","pith_inferences":["The paper's single-run, offline tuning could be combined with phase detection: re-running the optimizer at each program phase would likely improve workloads like Btree that mix write-heavy and read-heavy phases, a direction the authors acknowledge they did not pursue.","The tuned configurations and their measured access patterns form a dataset on which a cheaper predictor could be trained, letting a deployment skip the expensive search once patterns are recognized.","The DAMON limitation for GUPS suggests that region-based access monitoring cannot be fixed by parameter tuning; systems with scattered hot pages need finer-grained sampling, independent of knob values.","The cost-benefit view of migrations implies that bandwidth-aware migration throttling, rather than static migration_period, could capture many of the tuning gains automatically."],"forward_implications":["Existing tiering systems can be improved without redesigning their policies: a search over their existing knobs yields 1.07–2.09x speedups over their default configurations on the workloads tested.","The best parameter values are not portable: they change with workload, input dataset, thread count, and fast-to-slow memory ratio, so a single global configuration cannot capture the gains.","Hidden or undocumented parameters, such as HeMem's cooling_pages, can be as important as documented thresholds, so tuning should expose and search the full implementation parameter space.","Dynamically adapting only one threshold, as Memtis does, is insufficient; the evaluation shows a tuned HeMem configuration outperforms Memtis on every workload in the set.","On hardware where tiers have similar bandwidth, such as the NUMA-emulated CXL setup, tuning gains are small because migration costs and benefits are nearly balanced."],"supporting_citations":[{"why":"HeMem: supplies the main tiering system whose knobs are tuned and whose default configurations are the baseline for the 2x improvement claim.","marker":"[35]"},{"why":"Memtis: the state-of-the-art dynamic-threshold tiering system that tuned HeMem configurations are compared against.","marker":"[24]"},{"why":"SMAC: the Bayesian optimization framework used as the tuner; its random-forest surrogate and acquisition function drive the configuration search.","marker":"[18]"},{"why":"HMSDK: the second tiering system evaluated, showing that the tuning approach generalizes beyond HeMem.","marker":"[23]"},{"why":"DAMON: the access-monitoring framework HMSDK relies on, whose region-based sampling limits what tuning can achieve for scattered hot pages.","marker":"[32]"},{"why":"CHopt: the ideal data placement algorithm the paper invokes as the target behavior for tuned migration patterns.","marker":"[49]"},{"why":"LlamaTune: prior work establishing SMAC-style Bayesian optimization for system knobs, used to justify the optimizer's sample efficiency.","marker":"[22]"}],"fun_headline_variants":["Bayesian tuning doubles memory-tier performance","Auto-tuning memory knobs yields 2x tiered memory gains","Workload-aware parameter search boosts tiered memory 2x","Bayesian optimization finds faster memory tiering configs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline 2x improvement is measured on a modified version of HeMem that includes four bug fixes and a new write-sampling knob; if those changes, rather than the tuned parameter values, are what separate the default from the tuned configuration, the result does not transfer to the publicly available HeMem.","fun_headline_variants_meta":{"raw":{"variants":["Bayesian tuning doubles memory-tier performance","Auto-tuning memory knobs yields 2x tiered memory gains","Workload-aware parameter search boosts tiered memory 2x","Bayesian optimization finds faster memory tiering configs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000784,"raw_usage":{"total_tokens":3435,"prompt_tokens":892,"completion_tokens":2543,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":508,"completion_tokens_details":{"reasoning_tokens":2478}},"tokens_in":508,"tokens_out":2543,"duration_ms":19333,"temperature":1.0,"reasoning_tokens":2478,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:11:12.807438+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the published HeMem code unmodified on the same eight workloads with the paper's best-per-configuration knob values; if the tuned values do not beat the stock default on most workloads, the claim that tuning existing systems yields up to 2x is contingent on the authors' modifications. A quicker check: measure whether the stock HeMem default, with only the write_sampling_period knob added, already accounts for most of the performance gap.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"HeMem: supplies the main tiering system whose knobs are tuned and whose default configurations are the baseline for the 2x improvement claim."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Memtis: the state-of-the-art dynamic-threshold tiering system that tuned HeMem configurations are compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"SMAC: the Bayesian optimization framework used as the tuner; its random-forest surrogate and acquisition function drive the configuration search."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"HMSDK: the second tiering system evaluated, showing that the tuning approach generalizes beyond HeMem."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DAMON: the access-monitoring framework HMSDK relies on, whose region-based sampling limits what tuning can achieve for scattered hot pages."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CHopt: the ideal data placement algorithm the paper invokes as the target behavior for tuned migration patterns."}],"review_version":1}