{"id":"16e3dc39-7816-4baa-9d6d-5abefddc44d9","arxiv_id":"2508.06978","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Offloading MoE expert weights to SSDs can raise per-token generation energy by roughly an order of magnitude versus HBM, and prefetching does not fix it.","lead":"This paper analyzes the energy cost of storing mixture-of-experts model weights on SSD instead of fast DRAM memory during text generation. It finds a large energy penalty, up to about 12 times, and suggests flash memory would need about ten times better read efficiency to compete.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Abstract-only analysis: 12x energy penalty hinges on assuming expert-weight reads dominate per-token energy; without including compute/idle energy, the amplification ratio is unsupported.","rationale":"The reader's verdict is UNVERDICTED because only the abstract was reviewed. My stress-test focuses on the same weakest assumption: the dominance of expert-weight read energy in per-token energy. The abstract's 12x number is a ratio of storage-read energies, but the claim about total per-token energy requires that other energy terms be negligible or scale identically. This is a concrete, checkable condition. However, since the full text is not available, I cannot verify whether the paper already accounts for E_other in a way that preserves the 12x ratio. The reader's UNVERDICTED verdict already reflects this lack of verification. My concern does not move the verdict; it reinforces the need for the full derivation and an empirical or equation-based check. Therefore verdict_should_be is UNCHANGED, and agreement_with_reader is agree because we identified the same load-bearing point.","tokens_in":645,"tokens_out":1898,"duration_ms":21828,"concrete_test":"Recalculate the per-token decode energy as E_total = E_weight_read(storage) + E_other, with E_weight_read taken from the paper's model and E_other from published per-token energy measurements of MoE decode on comparable hardware (e.g., total GPU energy per token). Determine the minimum E_other for which the SSD/HBM total ratio drops below 2x. If realistic E_other exceeds the SSD weight-read energy, the headline 12x penalty is not representative of end-to-end inference. Also, re-derive the device per-bit energies from a specific SSD datasheet including controller and interface energy, not just NAND core energy, and compare the ratio to 12x.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim, that SSD offloading raises per-token decode energy up to ~12x and that prefetching cannot avoid it, rests on an implicit energy model in which per-token energy is dominated by reading the activated expert weights from wherever they reside. The abstract states that SSD reads 'dominate the total inference energy budget,' but no supporting breakdown is available. If compute, activation traffic, static power, or other HBM accesses contribute a non-negligible constant E_other per token, then the true ratio is (E_SSD_read + E_other)/(E_HBM_read + E_other), which shrinks as E_other grows. For example, if E_other equals several times the SSD read energy, the 12x ratio collapses toward a small overhead. Since the full text is unavailable, the dominance assumption is the most load-bearing unverified step. Additionally, the per-bit read energies for SSD/DRAM/HBM are quoted from prior datasheets; real workload energy depends on page size, controller, interface, over-provisioning, and access pattern. No ad hominem is intended; this is a question about the model's completeness.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper analyzes the energy cost of offloading Mixture-of-Experts (MoE) expert weights from HBM to SSDs during LLM decode. Based on a comparison of SSD, DDR, and HBM per-bit read energies for models such as DeepSeek-R1, it claims that SSD offloading increases per-token generation energy by up to roughly 12x compared with HBM, that SSD reads dominate the total inference energy budget, that prefetching cannot mitigate this energy penalty, and that an order-of-magnitude improvement in Flash read energy would be needed for SSDs to become energy-viable.","tokens_in":940,"tokens_out":1684,"duration_ms":20052,"significance":"If the quantitative claims hold, the paper would be a valuable corrective to a growing line of systems research that treats SSD offloading of MoE weights as a latency problem only. The 12x figure and the 'prefetching cannot help' argument would give architects a concrete energy-based criterion for when Flash offloading is worthwhile. The work also makes a falsifiable prediction about Flash energy scaling. At present, however, the abstract alone does not provide the energy model, workload parameters, device constants, or validation needed to assess even the sign, let alone the magnitude, of the claimed effect.","major_comments":[{"comment":"The central claim that SSD offloading raises per-token decode energy by up to ~12x rests on an implicit model in which expert-weight reads dominate the per-token energy budget. The abstract says SSD reads 'dominate the total inference energy budget' but provides no breakdown. If compute, activation traffic, static/idle power, or other HBM traffic contributes a per-token term E_other, the actual ratio is (E_SSD_read + E_other)/(E_HBM_read + E_other), which shrinks as E_other grows. The paper must specify the workload (batch size, decode length, model size, active/expert reads per token) and the non-weight energy contributions, and show that E_other is indeed negligible or small. Without this, the 12x figure is not supported.","section":"Abstract (energy-model claim)"},{"comment":"The 12x ratio depends directly on per-bit read energies for SSD, DDR, and HBM. The abstract cites no sources and gives no parameter values, no page/block read energy, no controller or interface overhead, no over-provisioning effects, and no DRAM refresh or HBM TSV effects. Since these constants are the multiplicative basis of the result, the paper should include a sensitivity analysis over the plausible ranges of these values and validate at least one workload against measurements or a published energy model.","section":"Abstract (device constants)"},{"comment":"The claim that prefetching cannot mitigate the energy penalty is stated categorically. Prefetching may hide latency, but it also changes the energy profile: it can read larger SSD pages/blocks than strictly necessary, may require buffering in SRAM/DRAM, and may activate idle SSD power for longer. Conversely, selective offloading, compression, or larger batch sizes could amortize or reduce SSD energy per token. The paper should define the prefetching policy it assumes and show that under that policy the 12x ratio remains representative across reasonable system configurations.","section":"Abstract (prefetching claim)"}],"minor_comments":[{"comment":"The abstract would benefit from a clear statement of the methodology (analytic model, simulation, or measurement) and from error bars or ranges around the ~12x figure. The phrase 'current SSDs' should also be pinned to specific device classes or datasheets.","section":"Abstract (methodology)"},{"comment":"No references are visible in the abstract. The paper should situate the energy analysis against prior SSD-offloading systems and existing LLM inference energy models, and should state what new quantity is being contributed beyond earlier energy-per-bit comparisons.","section":"Abstract (references)"}],"recommendation":"uncertain","confidential_remarks":"This review is based only on the abstract because no full text was provided. The central claim is plausible but entirely unverified from the available material: the energy model, device parameters, workload definition, and sensitivity analysis are missing. I cannot recommend acceptance or rejection without seeing the full text. An 'uncertain' verdict is appropriate; the editor should obtain the full manuscript before a decision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I only have the abstract, so this letter is conditional in a specific way: the central quantitative claim, ~12x per-token energy penalty from SSD offloading, is a derived number, and the derivation is not visible. What is genuinely useful is the question. As MoE models scale, SSD offloading is being proposed mainly as a capacity play, and most papers focus on latency hiding. Energy is the right second axis to check. The abstract's conclusion that prefetching fixes latency but not energy is simple, correct if the energy premise holds, and worth saying.\n\nThe real soft spot is the dominance assumption. The abstract states SSD reads 'dominate the total inference energy budget,' but that is effectively the conclusion, not a premise. For the 12x ratio to hold, per-token energy from reading the activated expert weights has to be the overwhelming term, relative to compute, activation traffic, other HBM traffic, and static power. Otherwise the ratio compresses toward 1. The stress-test note makes that point correctly. I can't tell from the abstract whether the full paper handles this with a breakdown and sensitivity analysis. The numbers also come from datasheet per-bit energies, so real workload behavior (page size, controller, access pattern) matters. These are normal modeling concerns, not fatal objections, but they are load-bearing.\n\nOne thing in the paper's favor: the claim is falsifiable and the model is simple enough to check. If the full text reports a complete energy breakdown, includes a compute/other-energy term, and does sensitivity over device parameters, this is exactly the kind of paper that belongs in front of a systems reviewer. If the breakdown omits anything beyond the weight reads, the 12x is not supported.\n\nI'd cite this as a cautionary check for anyone considering SSD offload for energy reasons, but the 12x number with a caveat. I would send it to peer review, not desk reject: the question is timely, the analysis is reproducible in principle, and the stakes for system design are real. I'd ask the reviewer to specifically verify that the dominance assumption is tested, not asserted.","headline":"A plausible but unverifiable energy-accounting argument against SSD offloading for MoE decode; the 12x number lives or dies with the model's treatment of non-weight energy.","tokens_in":1332,"tokens_out":1652,"would_cite":true,"duration_ms":17230,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that offloading MoE expert weights from HBM to SSDs raises per-token decode energy by up to roughly 12x, and that prefetching cannot undo the penalty.","keywords":["Mixture-of-Experts","LLM inference","decode stage","SSD offloading","HBM","memory energy","prefetching","flash read energy"],"falsifier":"On a system that can serve the same MoE model with experts in HBM and then in an SSD, with the same token stream and batch size, measure wall-clock time and energy per token; if the SSD-versus-HBM energy ratio is far below roughly 12x, or if prefetching changes it substantially, the central dominance claim fails. A second check is to replace the paper's per-bit read energies with measured values for the actual NAND, DDR, and HBM parts used.","tokens_in":610,"feed_emoji":"⚡","tokens_out":4742,"duration_ms":45794,"temperature":0.7,"pith_summary":"Large mixture-of-experts (MoE) language models are growing too big for the fast on-chip memory (HBM) that normally feeds them, so a popular idea is to push expert weights out to cheaper SSDs. This paper argues that the idea is self-defeating during text generation: reading expert weights from an SSD costs far more energy per bit than reading them from HBM, and since decode must pull a fresh set of expert weights for every token, the SSD's lower cost-per-capacity is paid for with up to roughly 12 times more energy per generated token. Prefetching, the usual fix, only hides latency; it does not reduce the number of bits read or the energy per bit. The authors therefore conclude that current SSDs should not be used to offload MoE weights when energy matters, and that flash read energy would need to improve by about an order of magnitude before MoE sparsity could make SSD offloading energy-competitive.","feed_headline":"SSD offload raises MoE decode energy up to 12x","feed_subtitle":"Prefetching hides the latency, not the power cost: reading experts from flash dominates the inference energy budget.","key_machinery":"The load-bearing quantity is the per-bit read energy of the storage technology—the energy consumed to deliver one bit of weight data to the compute unit—compared across HBM, DDR, and flash SSD. The argument couples this with the MoE decode property: every token must read the weights of its active experts afresh, so per-token energy is roughly (number of active expert weights read) times (per-bit read energy). Prefetching enters only as a latency-hiding mechanism and does not alter either factor, which is why the paper finds it cannot mitigate the energy penalty.","core_discovery":"The paper's central claim is that for the decode phase of MoE LLM inference, the energy cost of moving expert weights into compute is decided by the storage medium they are read from, and that current SSDs are the worst of the three options compared. Since each generated token requires reading the weights of the experts selected for that token, per-token energy scales with the per-bit read energy of wherever those weights live. Using published per-bit read energies for HBM, DDR (CPU memory), and flash SSD, and modeling systems at the scale of models like DeepSeek-R1, the paper estimates that offloading to current SSDs raises per-token energy by up to roughly 12x versus the HBM baseline and m","pith_inferences":["If the per-bit read-energy gap persists across process generations, other forms of offloading that move active weights farther from compute will hit the same wall; the governing criterion is energy per useful bit, not capacity per dollar.","The analysis suggests algorithmic ways to reduce the number of weight bits touched per token—for example, expert routing that reuses weights across tokens or compresses experts—can be as valuable as faster storage, since the energy penalty scales with bits read.","A testable extension: models with different numbers of active experts per token should show roughly proportional per-token energy penalties under SSD offload; confirming that would validate the mechanism.","Because the paper's future scenario hinges on flash read energy improving about 10x, a natural engineering target is a low-energy bulk-read mode for SSDs used when weights are streamed rather than randomly accessed."],"forward_implications":["SSD offloading of MoE weights should be rejected for energy-sensitive decode workloads; the roughly 12x per-token energy penalty dominates the inference energy budget.","Prefetching, though it hides latency, cannot rescue the design; energy, not speed, becomes the binding constraint.","Keeping expert weights in HBM remains the energy-favorable configuration on current hardware.","Flash read energy must drop by roughly an order of magnitude before MoE sparsity can make SSD offloading energy-viable.","Energy models for future MoE systems should treat weight-read energy as a primary scaling term rather than focusing only on compute."],"supporting_citations":[],"fun_headline_variants":["SSD offload: 12x energy spike for MoE token generation","MoE on SSD burns 12x more energy per token","Flash reads make MoE decode 12x less efficient","Prefetching hides latency, not the 12x SSD energy cost","SSD offloading: energy trap for MoE inference"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The conclusion depends on per-token decode energy being dominated by the energy of reading expert weights from their storage medium, and on the per-bit read energies quoted for SSD, DDR, and HBM holding for real workloads; if that dominance or those numbers differ, the roughly 12x penalty changes.","fun_headline_variants_meta":{"raw":{"variants":["SSD offload: 12x energy spike for MoE token generation","MoE on SSD burns 12x more energy per token","Flash reads make MoE decode 12x less efficient","Prefetching hides latency, not the 12x SSD energy cost","SSD offloading: energy trap for MoE inference"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00058,"raw_usage":{"total_tokens":2571,"prompt_tokens":749,"completion_tokens":1822,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":493,"completion_tokens_details":{"reasoning_tokens":1733}},"tokens_in":493,"tokens_out":1822,"duration_ms":12848,"temperature":1.0,"reasoning_tokens":1733,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T22:23:09.797816+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a system that can serve the same MoE model with experts in HBM and then in an SSD, with the same token stream and batch size, measure wall-clock time and energy per token; if the SSD-versus-HBM energy ratio is far below roughly 12x, or if prefetching changes it substantially, the central dominance claim fails. A second check is to replace the paper's per-bit read energies with measured values for the actual NAND, DDR, and HBM parts used.","supporting_citations":[],"review_version":1}