REVIEW 3 major objections 2 minor
SSD Offloading for LLM Mixture-of-Experts Weights Considered Harmful in Energy Efficiency
T0 review · 3 major / 2 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read This paper claims that offloading MoE expert weights from HBM to SSDs raises per-token decode energy by up to roughly 12x, and that prefetching cannot undo the penalty.
desk verdict A plausible but unverifiable energy-accounting argument against SSD offloading for MoE decode; the 12x number lives or dies with the model's treatment of non-weight energy. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing quantity is the per-bit read energy of the storage technology—the energy consumed to deliver one bit of weight data to the compute unit—compared across HBM, DDR, and flash SSD. The argument couples this with the MoE decode property: every token must read the weights of its active experts afresh, so per-token energy is roughly (number of active expert weights read) times (per-bit read energy). Prefetching enters only as a latency-hiding mechanism and does not alter either factor, which is why the paper finds it cannot mitigate the energy penalty.
What would settle it
On a system that can serve the same MoE model with experts in HBM and then in an SSD, with the same token stream and batch size, measure wall-clock time and energy per token; if the SSD-versus-HBM energy ratio is far below roughly 12x, or if prefetching changes it substantially, the central dominance claim fails. A second check is to replace the paper's per-bit read energies with measured values for the actual NAND, DDR, and HBM parts used.
Extended reading notes
Core claim
The paper's central claim is that for the decode phase of MoE LLM inference, the energy cost of moving expert weights into compute is decided by the storage medium they are read from, and that current SSDs are the worst of the three options compared. Since each generated token requires reading the weights of the experts selected for that token, per-token energy scales with the per-bit read energy of wherever those weights live. Using published per-bit read energies for HBM, DDR (CPU memory), and flash SSD, and modeling systems at the scale of models like DeepSeek-R1, the paper estimates that offloading to current SSDs raises per-token energy by up to roughly 12x versus the HBM baseline and m
Load-bearing premise
The conclusion depends on per-token decode energy being dominated by the energy of reading expert weights from their storage medium, and on the per-bit read energies quoted for SSD, DDR, and HBM holding for real workloads; if that dominance or those numbers differ, the roughly 12x penalty changes.
Editorial extensions
If this is right
- SSD offloading of MoE weights should be rejected for energy-sensitive decode workloads; the roughly 12x per-token energy penalty dominates the inference energy budget.
- Prefetching, though it hides latency, cannot rescue the design; energy, not speed, becomes the binding constraint.
- Keeping expert weights in HBM remains the energy-favorable configuration on current hardware.
- Flash read energy must drop by roughly an order of magnitude before MoE sparsity can make SSD offloading energy-viable.
- Energy models for future MoE systems should treat weight-read energy as a primary scaling term rather than focusing only on compute.
Reading between the lines
- If the per-bit read-energy gap persists across process generations, other forms of offloading that move active weights farther from compute will hit the same wall; the governing criterion is energy per useful bit, not capacity per dollar.
- The analysis suggests algorithmic ways to reduce the number of weight bits touched per token—for example, expert routing that reuses weights across tokens or compresses experts—can be as valuable as faster storage, since the energy penalty scales with bits read.
- A testable extension: models with different numbers of active experts per token should show roughly proportional per-token energy penalties under SSD offload; confirming that would validate the mechanism.
- Because the paper's future scenario hinges on flash read energy improving about 10x, a natural engineering target is a low-energy bulk-read mode for SSDs used when weights are streamed rather than randomly accessed.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper analyzes the energy cost of offloading Mixture-of-Experts (MoE) expert weights from HBM to SSDs during LLM decode. Based on a comparison of SSD, DDR, and HBM per-bit read energies for models such as DeepSeek-R1, it claims that SSD offloading increases per-token generation energy by up to roughly 12x compared with HBM, that SSD reads dominate the total inference energy budget, that prefetching cannot mitigate this energy penalty, and that an order-of-magnitude improvement in Flash read energy would be needed for SSDs to become energy-viable.
Significance. If the quantitative claims hold, the paper would be a valuable corrective to a growing line of systems research that treats SSD offloading of MoE weights as a latency problem only. The 12x figure and the 'prefetching cannot help' argument would give architects a concrete energy-based criterion for when Flash offloading is worthwhile. The work also makes a falsifiable prediction about Flash energy scaling. At present, however, the abstract alone does not provide the energy model, workload parameters, device constants, or validation needed to assess even the sign, let alone the magnitude, of the claimed effect.
major comments (3)
- [Abstract (energy-model claim)] The central claim that SSD offloading raises per-token decode energy by up to ~12x rests on an implicit model in which expert-weight reads dominate the per-token energy budget. The abstract says SSD reads 'dominate the total inference energy budget' but provides no breakdown. If compute, activation traffic, static/idle power, or other HBM traffic contributes a per-token term E_other, the actual ratio is (E_SSD_read + E_other)/(E_HBM_read + E_other), which shrinks as E_other grows. The paper must specify the workload (batch size, decode length, model size, active/expert reads per token) and the non-weight energy contributions, and show that E_other is indeed negligible or small. Without this, the 12x figure is not supported.
- [Abstract (device constants)] The 12x ratio depends directly on per-bit read energies for SSD, DDR, and HBM. The abstract cites no sources and gives no parameter values, no page/block read energy, no controller or interface overhead, no over-provisioning effects, and no DRAM refresh or HBM TSV effects. Since these constants are the multiplicative basis of the result, the paper should include a sensitivity analysis over the plausible ranges of these values and validate at least one workload against measurements or a published energy model.
- [Abstract (prefetching claim)] The claim that prefetching cannot mitigate the energy penalty is stated categorically. Prefetching may hide latency, but it also changes the energy profile: it can read larger SSD pages/blocks than strictly necessary, may require buffering in SRAM/DRAM, and may activate idle SSD power for longer. Conversely, selective offloading, compression, or larger batch sizes could amortize or reduce SSD energy per token. The paper should define the prefetching policy it assumes and show that under that policy the 12x ratio remains representative across reasonable system configurations.
minor comments (2)
- [Abstract (methodology)] The abstract would benefit from a clear statement of the methodology (analytic model, simulation, or measurement) and from error bars or ranges around the ~12x figure. The phrase 'current SSDs' should also be pinned to specific device classes or datasheets.
- [Abstract (references)] No references are visible in the abstract. The paper should situate the energy analysis against prior SSD-offloading systems and existing LLM inference energy models, and should state what new quantity is being contributed beyond earlier energy-per-bit comparisons.
Circularity Check
No circularity detected in abstract-only analysis
full rationale
The abstract presents a quantitative energy comparison of LLM MoE decode under three storage scenarios (SSD, DDR, HBM). The claimed ~12x penalty is derived from standard per-bit read-energy constants and a decode workload where reading activated expert weights is the dominant energy term. This is a forward calculation from stated input parameters, not a derivation that assumes its conclusion. No fitted parameters are relabeled as predictions, no self-citation is load-bearing, and no mathematical identity makes the conclusion equivalent to the premises. The abstract itself does not contain the detailed equations, but nothing in the visible text suggests circularity. The main risks (dominance of weight-read energy, accuracy of datasheet numbers) are empirical/correctness issues, not circular reasoning. Therefore, the circularity score is 0.
Assumptions & free parameters
assumptions (3)
- domain assumption Per-token decode energy is dominated by reading expert weights from the storage medium.
- domain assumption The energy-per-bit values used for SSD, DDR, and HBM reflect current device measurements or published specifications.
- domain assumption MoE inference reads only a subset of experts per token (sparsity), which is the basis for the future viability projection.
Cite this review
Pith. "Pith review of SSD Offloading for LLM Mixture-of-Experts Weights Considered Harmful in Energy Efficiency." pith.science (2026). https://pith.science/paper/HVY5QBTS
@misc{pith2026250806978,
author = {Pith},
title = {Pith review of: SSD Offloading for LLM Mixture-of-Experts Weights Considered Harmful in Energy Efficiency},
year = {2026},
howpublished = {\url{https://pith.science/paper/HVY5QBTS}},
note = {Machine review of arXiv:2508.06978}
}
read the original abstract
Large Language Models (LLMs) applying Mixture-of-Experts (MoE) scale to trillions of parameters but require vast memory, motivating a line of research to offload expert weights from fast-but-small DRAM (HBM) to denser Flash SSDs. While SSDs provide cost-effective capacity, their read energy per bit is substantially higher than that of DRAM. This paper quantitatively analyzes the energy implications of offloading MoE expert weights to SSDs during the critical decode stage of LLM inference. Our analysis, comparing SSD, CPU memory (DDR), and HBM storage scenarios for models like DeepSeek-R1, reveals that offloading MoE weights to current SSDs drastically increases per-token-generation energy consumption (e.g., by up to ~12x compared to the HBM baseline), dominating the total inference energy budget. Although techniques like prefetching effectively hide access latency, they cannot mitigate this fundamental energy penalty. We further explore future technological scaling, finding that the inherent sparsity of MoE models could potentially make SSDs energy-viable if Flash read energy improves significantly, roughly by an order of magnitude.
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.