{"id":"ea5084a9-36b5-4276-bb6a-f857b4fdbc31","arxiv_id":"2504.14893","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"H2M2 uses an asymmetric HBM+LPDDR memory with compute on both sides and a dynamic kernel-mapping algorithm to speed up LLM generation, beating an LPDDR-only system by up to 2.94x but lagging an 8-HBM system by 1.57x.","lead":"This paper proposes H2M2, an asymmetric memory design pairing high-bandwidth HBM with large-capacity LPDDR, each with its own accelerator, to run large language model inference. It reports 1.46x to 2.94x speedups over an LPDDR-only baseline and argues the design is cheaper and more energy-efficient than many high-bandwidth GPUs.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Near-optimal mapping claim depends on an undisclosed hyperparameter in Algorithm 1; without disclosure and held-out validation, the reported 5% gap to Oracle may be fitted rather than predictive.","rationale":"The reader's weakest assumption and my load-bearing concern coincide: Algorithm 1's mapping decision uses an undisclosed hyperparameter that may have been calibrated on the same workloads used for evaluation. This is the most serious threat to the paper's headline 'near-optimal mapping' claim, because the Oracle comparison in Table 3 and Figures 12, 13, and 15 is exactly the evidence used to establish that claim. The LPDDR-only speedup is more robust, as it primarily reflects the hardware asymmetry and cycle simulation, so I do not think the whole paper needs rejection. However, the manuscript's central algorithmic contribution cannot be fully trusted until the hyperparameter is disclosed and validated on untouched workloads. The paper also provides no released artifacts, which strengthens the need for this check. The dynamic sequence length study and sensitivity analysis are useful, but they do not address the calibration question. I therefore leave the verdict unchanged at CONDITIONAL, with the condition being disclosure and independent validation of the mapping model's hyperparameter.","tokens_in":23157,"tokens_out":2982,"duration_ms":28513,"concrete_test":"Ask the authors to disclose the hyperparameter value(s) used in Algorithm 1 and the calibration procedure, then evaluate H2M2 on at least two held-out LLMs not in the current set, e.g., Falcon-40B/180B and Mistral-7B, across a comparable batch/sequence grid without re-tuning the hyperparameter. If the H2M2/Oracle gap remains under 5% on these held-out workloads with a hyperparameter derived from first principles (e.g., FLOPs per byte moved per sublayer rather than fitted), the concern is resolved. As a cheaper supplementary check, perturb the hyperparameter by +/-20% for the GPT3-175B case and observe whether the Table 3 mapping overhead changes materially; if it does, the near-optimal result depends critically on exact calibration.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Algorithm 1 (Section 4.3.2) selects the kernel-memory assignment by minimizing the peak execution time on the HBM and LPDDR sides. The peak execution model is described as dividing total arithmetic operations by peak accelerator throughput, then multiplying by 'a hyperparameter that reflects the arithmetic intensity' (Section 4.3.2). No value, per-sublayer treatment, or calibration procedure is provided. Because the same three LLMs used for the headline results are the only workloads evaluated, the reported closeness of H2M2 to the Oracle mapping (Table 3: mapping overheads of 2.56%, 3.76%, and 0.60%; Figures 12, 13, and 15) could be the product of fitting this hidden parameter rather than of an independently predictive model. A single scalar cannot properly represent arithmetic intensity, which differs across attention, qkv-linear, and fc and changes with batch size and sequence length. Without specifying the hyperparameter and its origin, the 'near-optimal mapping' contribution is not independently established. The speedup over LPDDR-only is less vulnerable to this concern, since that comparison is dominated by the measured memory behavior in the cycle-level simulator, but the paper's dynamic mapping contribution and its Oracle validation rest directly on the hidden parameter. The simulator and data are not released, so the claim cannot be independently checked from the manuscript alone.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes H2M2, an asymmetric memory architecture for LLM generation-phase inference in which an HBM-attached accelerator and a large LPDDR-attached accelerator cooperate on a heterogeneous memory board. The main contributions are a head-aware kernel-memory mapping granularity, a runtime mapping algorithm (Algorithm 1) that greedily allocates sublayers to HBM in the order attention, qkv-linear, fc, and a hardware memory abstraction based on flat page tables and MMUs to handle KV-cache growth and mapping changes. Using a cycle-level simulator, the authors report speedups of 1.46x, 1.55x, and 2.94x over an LPDDR-only capacity-centric baseline for GPT3-175B, Chinchilla-70B, and Llama2-70B, with mapping overheads of 2.56%, 3.76%, and 0.60% relative to an oracle mapping.","tokens_in":23391,"tokens_out":10303,"duration_ms":96026,"significance":"If the results hold, this is a valuable systems contribution: it demonstrates a concrete way to combine bandwidth-centric HBM and capacity-centric LPDDR with accelerators on both sides, and it provides quantitative evidence for the importance of kernel-to-memory mapping granularity and dynamic mapping in LLM inference. The energy comparison and sensitivity study strengthen the practical case. The main speedup over the LPDDR-only baseline is fairly robust because even the suboptimal FlexGen-style mapping already achieves 1.30x in Figure 7. However, the paper's headline near-optimal mapping claim is not yet supported with sufficient rigor: it depends on an undisclosed arithmetic-intensity hyperparameter and on a priority order derived from the same three workloads used for evaluation, and no artifact is released. The central architectural idea is promising, but the mapping contribution needs disclosure and independent validation before the quantitative claims can be accepted.","major_comments":[{"comment":"The performance model for mapping decisions is not fully specified. The text states that the ideal execution time is the total arithmetic operations divided by peak throughput, multiplied by a hyperparameter that reflects the arithmetic intensity. No value, per-sublayer treatment, or calibration procedure is provided. Because arithmetic intensity differs between attention, qkv-linear, and fc and varies with batch size and sequence length, a single scalar cannot represent it. Since the same three workloads are used both to motivate the priority order and to report the mapping overheads, the reported closeness to Oracle (2.56%, 3.76%, and 0.60% in Table 3) may be a fitted outcome rather than an independent prediction. Please disclose the hyperparameter, specify how it is obtained for each sublayer, and validate the mapping algorithm on held-out workloads or hardware configurations.","section":"Section 4.3.2, Algorithm 1 and Table 3"},{"comment":"Algorithm 1 is too underspecified to reproduce. The three bullets under Find n express feasibility and a min-max objective verbally, but no equations define the capacity constraints, the peak execution time, or the role of the arithmetic-intensity hyperparameter. The paper calls the problem a simple linear problem but gives no linear program. This also makes the claimed 0.05ms solve time impossible to verify.","section":"Section 4.3.2, Algorithm 1"},{"comment":"The evaluation relies on a custom cycle-level simulator, but the manuscript only says it was cross-validated by profiling the open-source multi-core NPU simulator and DRAM simulator without reporting any validation data. The simulator and workload configurations are not released, so the absolute speedup numbers cannot be independently checked. Please provide a validation section or an artifact, or clearly state the limitations.","section":"Section 5.1"},{"comment":"The FlexGen comparison is not reproducible as presented. The authors say they modified Eq. (1) to suit asymmetric memory systems, but the modified optimization problem is never shown. Without the modified constraints and objective, the claim that FlexGen's model is structurally incapable of finding the best mapping cannot be verified.","section":"Section 3.2, Figure 7"}],"minor_comments":[{"comment":"H2D2 should be H2M2 in the sentence reporting the 1.48x speedup.","section":"Section 5.3"},{"comment":"There are typos in this section: Lllama2 should be Llama2, and excedding should be exceeding.","section":"Section 5.2.3"},{"comment":"trasnlated should be translated.","section":"Section 5.5"},{"comment":"The caption says batch size 32, but the x-axis includes B16, B32, and B64; please fix the inconsistency.","section":"Figure 7 caption"},{"comment":"Several labels in Figure 9 are garbled, including underu,liza,on and J underu,liza,on; these should be cleaned up.","section":"Figure 9"},{"comment":"The statement that attention cannot be batched with multiple requests is imprecise; attention over multiple requests is routinely executed as batched GEMV with per-request KV caches (e.g., vLLM). Please clarify that the point is that attention does not become a single GEMM rather than that it cannot be batched at all.","section":"Section 2.2.3"}],"recommendation":"major_revision","confidential_remarks":"The main risk to the paper's quantitative claims is the undisclosed arithmetic-intensity hyperparameter in the mapping model and the absence of held-out validation. If the authors disclose the parameter and add a validation study, the paper could become a solid systems contribution. The lack of a released simulator is a secondary concern that should be addressed, at least with a detailed validation section."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nTwo things to know about this paper. First, H2M2 is a plausible and well-motivated architecture: put accelerators on both HBM and LPDDR, map LLM kernels to the two sides at head granularity, and manage the mapping dynamically as the KV cache grows. The speedups over an LPDDR-only baseline (1.46x, 1.55x, 2.94x for GPT3-175B, Chinchilla-70B, Llama2-70B) are believable; they come from a cycle-level simulator and are driven by memory behavior, not by the mapping optimizer. Second, the claim that their greedy mapping algorithm gets within 5% of an oracle rests on a hyperparameter in the mapping model that is supposed to reflect arithmetic intensity. The paper never gives its value or calibration procedure, and because the same three models are used for both calibration and evaluation, the close-to-oracle result may be partly fitted.\n\nWhat is actually new: the head-aware mapping granularity analysis is a concrete insight. The paper shows that attention is the most bandwidth-sensitive sublayer, that mapping attention to HBM matters more than qkv-linear or fc, and that the priority order attention > qkv-linear > fc holds across batch/sequence configurations. That is a useful result for anyone designing accelerators or memory systems for LLM serving. The hardware memory abstraction (flat page tables, TLB management, zero-copy remote access) is a clean way to handle KV-cache growth and remapping, and the measured overhead of about 1% seems believable. The dynamic sequence length experiment is a nice test that the mapping policy adapts, and the paper honestly shows that H2M2 loses to an 8-HBM system on raw speed but wins on energy.\n\nSoft spots, in order. The hidden hyperparameter is the big one. A single scalar cannot represent arithmetic intensity across sublayer types and changing batch sizes; the paper should disclose the value, explain how it was chosen, and ideally validate it on a held-out model or configuration. The lack of a reference implementation or simulator makes it hard to check the oracle mapping, the 0.05 ms solver cost, and the energy numbers. The 8-HBM comparison is useful but under-specified: what communication cost, what capacity, what bandwidth? And the energy model only counts memory, a simplification that should be stated more prominently. None of these sink the core idea. The speedup over LPDDR-only is grounded in the simulator, and the asymmetric-memory direction is worth pursuing.\n\nThis paper deserves serious peer review. The contribution is concrete, the evaluation is mostly sound, and the missing details are fixable with disclosure and a bit more validation. I would accept it with major and minor revisions. Who is it for: computer architects working on LLM inference hardware and systems people interested in heterogeneous memory. I would bring it to a reading group; it will generate a good discussion about what counts as a fitted result in simulation papers.","headline":"Solid asymmetric-memory proposal for LLM serving, but the near-optimal mapping claim needs the hidden arithmetic-intensity parameter disclosed before I fully trust it.","tokens_in":23980,"tokens_out":2937,"would_cite":true,"duration_ms":25382,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"H2M2 claims that putting accelerators on both fast HBM and cheap LPDDR memory, plus a runtime kernel-placement rule, runs LLM inference 1.46–2.94x faster than an LPDDR-only system and within 5% of the optimal mapping.","keywords":["large language models","asymmetric memory","HBM","LPDDR","KV cache","kernel-memory mapping","memory abstraction","hardware accelerator"],"falsifier":"Use H2M2's mapping algorithm on a held-out LLM whose layer shape differs from the three evaluated models, for example a mixture-of-experts model or one with very different head and group sizes, without re-tuning the arithmetic-intensity hyperparameter, and compare against the exhaustive-search oracle; if the placement gap exceeds 5 percent on average, the near-optimal mapping claim fails.","tokens_in":22920,"feed_emoji":"⚡","tokens_out":7772,"duration_ms":64143,"temperature":0.7,"pith_summary":"The paper argues that the most cost-effective way to serve large language models is not to move everything into expensive high-bandwidth memory, and not to treat slow memory as a backing store, but to attach a small accelerator to each memory type and run both in parallel. For LLM generation, where attention kernels are bandwidth-bound and the KV cache grows every token, the central problem becomes deciding which kernels live on the HBM side and which on the LPDDR side at each moment. H2M2 solves this with a greedy runtime mapping algorithm and a hardware memory abstraction that lets the KV cache grow and the mapping change without rewriting kernels. On GPT3-175B, Chinchilla-70B, and Llama2-70B, the paper reports 1.46x, 1.55x, and 2.94x speedups over an LPDDR-only system, staying within 5 percent of an oracle mapping that always knows the best placement.","feed_headline":"Asymmetric memory runs LLMs 1.5x–2.9x faster on cheap DRAM","feed_subtitle":"Small accelerators beside each memory bank plus a greedy placement rule keep LLM inference within 5% of perfect.","key_machinery":"The load-bearing object is the asymmetric memory system itself: one HBM3 module (96 GB, 3 TB/s) and one LPDDR5X module (512 GB, 544 GB/s), each with its own accelerator chip, linked by a high-bandwidth interconnect. Three mechanisms sit on top. First, head-aware mapping granularity splits qkv-linear and attention by attention heads and splits fc by column, so both sides compute in parallel with minimal synchronization. Second, Algorithm 1 is a greedy min-max solver that processes sublayers in the priority order attention, qkv-linear, fc and picks the number of heads to place in HBM so that the estimated peak execution times on the two sides are balanced while memory capacity is respected. Third, a hardware memory abstraction with per-side MMUs and flat page tables decouples logical contiguity from physical placement, making KV-cache allocation, head-fusion, and mapping migration cheap.","core_discovery":"The central claim is that an asymmetric memory architecture with computation attached to both the bandwidth-centric and the capacity-centric memory can serve decoder-based LLM inference close to the speed of an oracle that always picks the best kernel-memory mapping, while using far less high-bandwidth memory. The key pattern the paper establishes is that attention is the sublayer that must be favored in HBM: its GEMV kernels have constant arithmetic intensity and are highly bandwidth-bound, and its KV-cache footprint grows with sequence length, whereas qkv-linear and fc become less critical as batch size grows. A second claim is that the best mapping can be found cheaply at runtime by a greedy algorithm that, for each sublayer, chooses how many attention heads go to HBM so as to balance the estimated peak execution times on both sides. A third claim is that hardware page-based memory abstraction keeps the cost of dynamic remapping and KV-cache growth small, with measured overhead under 1.4 percent.","pith_inferences":["The attention-first priority likely transfers to any memory-bound stage of an LLM, such as long-context attention or mixture-of-experts routing, but the paper evaluates only three dense decoder models, so this broader claim is our inference, not the paper's.","If the undisclosed arithmetic-intensity hyperparameter in Algorithm 1 was calibrated on the same three workloads used for evaluation, the reported near-oracle gap is a fitted result; the natural test is a held-out fourth model, which we give as the falsifier.","The hardware memory abstraction is not limited to KV caches: it could also support transparent offloading of weights or activations in other heterogeneous-memory workloads, though the paper does not explore that.","The head-aware split suggests a path to scaling asymmetric memory beyond one HBM plus one LPDDR pair, for example by adding more capacity-centric modules and extending Algorithm 1 to a multi-way min-max problem."],"forward_implications":["LLM serving can keep most weights in large, cheap LPDDR and reserve HBM for the bandwidth-bound attention kernels, reducing the number of HBM devices needed to serve a given model.","Dynamic KV-cache growth and variable sequence lengths can be handled by page-level allocation rather than contiguous pre-reservation, so HBM capacity is used close to fully across sequence lengths.","The greedy mapping algorithm tracks the oracle within 5 percentage points on all three evaluated models, including a 128-iteration scenario with randomly ending requests.","Memory abstraction overhead stays below 1.4 percent, so the runtime gains of asymmetric memory are not offset by translation and migration costs.","Compared with an eight-HBM system, H2M2 uses roughly 0.58x the memory energy per token, making the cost-performance tradeoff favorable for large models."],"supporting_citations":[{"why":"Defines the asymmetric memory configuration (HBM3 plus LPDDR5X with accelerators on both sides) that H2M2 assumes and uses for its capacity and bandwidth parameters.","marker":"[32]"},{"why":"Supplies the LPDDR-only capacity-centric baseline system and the memory energy values used to compare H2M2 against a homogeneous setup.","marker":"[36]"},{"why":"Provides the host-memory offloading linear program whose static, sublayer-blind mapping H2M2 shows is suboptimal in asymmetric memory, motivating the new mapping algorithm.","marker":"[42]"},{"why":"Introduces PagedAttention's software KV-cache virtualization, which H2M2 extends into hardware-level memory abstraction for all tensors.","marker":"[24]"},{"why":"The multi-core NPU simulator whose profiling cross-validates the cycle-level simulator used for evaluation.","marker":"[16]"},{"why":"The DRAM simulator used to cross-validate memory timing behavior in the cycle-level performance model.","marker":"[29]"},{"why":"Supplies DRAM access-latency measurements used as simulator parameters.","marker":"[10]"},{"why":"Defines GPT3-175B, one of the three evaluated LLM workloads and the source of the decoder topology in the analysis.","marker":"[5]"},{"why":"Defines Chinchilla-70B, the second benchmark workload used in the evaluation.","marker":"[14]"},{"why":"Defines Llama2-70B, the third benchmark and the source of the grouped-query attention analysis.","marker":"[46]"}],"fun_headline_variants":["LLMs run 2.9x faster using cheap DRAM with smart placement","Attention to HBM, rest to DRAM: LLM inference speeds up 2.9x","Hardware-managed asymmetric memory: LLMs near-oracle speed, less HBM","H2M2: greedy GPU-side mapping speeds LLM inference on cheap DRAM"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The runtime mapping algorithm's estimate of how long each sublayer takes is scaled by an undisclosed hyperparameter meant to capture arithmetic intensity, and if that parameter was tuned on the same three models used for the speedup measurements, the reported closeness to oracle placement is calibration rather than prediction.","fun_headline_variants_meta":{"raw":{"variants":["LLMs run 2.9x faster using cheap DRAM with smart placement","Attention to HBM, rest to DRAM: LLM inference speeds up 2.9x","Hardware-managed asymmetric memory: LLMs near-oracle speed, less HBM","H2M2: greedy GPU-side mapping speeds LLM inference on cheap DRAM"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000952,"raw_usage":{"total_tokens":4084,"prompt_tokens":992,"completion_tokens":3092,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":608,"completion_tokens_details":{"reasoning_tokens":2999}},"tokens_in":608,"tokens_out":3092,"duration_ms":21616,"temperature":1.0,"reasoning_tokens":2999,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:37:24.585474+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Use H2M2's mapping algorithm on a held-out LLM whose layer shape differs from the three evaluated models, for example a mixture-of-experts model or one with very different head and group sizes, without re-tuning the arithmetic-intensity hyperparameter, and compare against the exhaustive-search oracle; if the placement gap exceeds 5 percent on average, the near-optimal mapping claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the asymmetric memory configuration (HBM3 plus LPDDR5X with accelerators on both sides) that H2M2 assumes and uses for its capacity and bandwidth parameters."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the LPDDR-only capacity-centric baseline system and the memory energy values used to compare H2M2 against a homogeneous setup."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the host-memory offloading linear program whose static, sublayer-blind mapping H2M2 shows is suboptimal in asymmetric memory, motivating the new mapping algorithm."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The multi-core NPU simulator whose profiling cross-validates the cycle-level simulator used for evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The DRAM simulator used to cross-validate memory timing behavior in the cycle-level performance model."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies DRAM access-latency measurements used as simulator parameters."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines GPT3-175B, one of the three evaluated LLM workloads and the source of the decoder topology in the analysis."}],"review_version":1}