{"id":"475a2e1f-56e8-4a5c-867e-096ac4a6af74","arxiv_id":"2607.26633","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A heterogeneous GPU-PNM serving system that routes short prompts to GPUs and long prompts to near-memory sparse-attention devices, with runtime migration for growing contexts, reports large throughput and tail-latency gains over GPU-only serving.","lead":"NELSSA is a GPU-plus-processing-near-memory serving system that sends short LLM requests to GPUs and long-context requests to near-memory accelerators, migrating requests as their context grows. It reports up to 5.5x decode throughput and 15x lower P99 latency on mixed-length workloads, though the largest gains come from projected full-capacity hardware.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"5.5x speedup is a 512 GB hardware projection; measured prototype only matches GPU-only baselines.","rationale":"The reader's conditional verdict is appropriately targeted. The single most load-bearing assumption is the unverified 16x memory-capacity projection behind the abstract's 5.5x. Section 7 explicitly states the prototype has 32 GB per PNM device and that throughput is reported at a projected 512 GB configuration; Section 8.1 concedes that in the directly measured range NELSSA only sustains throughput 'comparable to GPU-only baselines.' Since the 5.5x value is produced by configurations beyond the measured range, the headline claim depends entirely on the assumption that the full-capacity board preserves the prototype's 200 GB/s effective bandwidth and per-step overheads. The paper's own validation (Figure 11, ±5.6%) only bounds the PNM-attn estimation error within the measured volume range; it does not validate capacity scaling. I agree with the reader's weakest_assumption, and would keep the verdict CONDITIONAL/UNCHANGED: the architectural mechanism is plausible and directly measured micro-metrics (memory headroom, batch size, TPOT) support the qualitative benefit, but the quantitative headline needs re-scoping or hardware validation. A secondary concern is that the system-wide mixed-length throughput in Section 8.3 is obtained via emulation into vLLM, not direct end-to-end measurement; this reinforces the need to mark measured vs. projected/emulated results in the abstract.","tokens_in":22553,"tokens_out":12055,"duration_ms":129099,"concrete_test":"Recompute the projected curves in Figure 10 using the measured PNM effective bandwidth at the largest working set the current prototype can hold (4x32 GB aggregate, i.e., 1M-token batch-1), rather than the constant value used in Section 8.1, and with all four PNM devices active. If the recomputed 5.5x configuration drops below, say, 4x, the headline is sensitive to the unverified capacity-scaling assumption; if it remains above 5x, the projection is robust. The decisive follow-up is to rerun on an actual 512 GB module when available.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline 5.5x decode-throughput result is not a measured outcome of the prototype. Section 7 states the prototype has 32 GB per PNM device and that the authors 'accordingly report projected throughput at a 512 GB per-device configuration'; Section 8.1 then concedes that 'within the directly measured range... NELSSA... sustains throughput comparable to the GPU-only baselines.' The 5.5x figure comes from configurations beyond that range (e.g., 1M-token requests at batch sizes exceeding what 4x32 GB can hold), produced by extrapolating the measured latency breakdown to 512 GB while holding per-device effective bandwidth (200 GB/s), PNM-overhead, transfer, and 'others' fixed. This projection is plausible but untested: the full-capacity board is unbuilt, and the validation in Figure 11 bounds only the PNM-attn estimation error (±5.6%) within the measured data-volume range, not the effect of 16x capacity scaling or all-four-devices contention. If the 512 GB modules cannot sustain the same effective random-access bandwidth at larger working sets, or if fixed per-step overheads grow, the abstract's central claim collapses to a lower, unquantified benefit. The paper does disclose the projection transparently, but the abstract and introduction present the 5.5x/15x as headline results without marking measured vs. projected.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"NELSSA is a GPU-PNM heterogeneous serving system for mixed-length LLM decode workloads. It routes requests below a hardware-derived crossover threshold to GPU FlashAttention and longer requests to sparse attention executed on CXL-attached PNM devices, batches all QKV/MLP work on the GPU, and migrates dynamically growing KV caches to the PNM tier in the background without recomputation. The paper contributes an end-to-end prototype (SK hynix CXL-PNM modules with DDR5-6400, ARM Neoverse V2 cores, RoCEv2 RDMA), a decode-latency model for the crossover threshold (Eqs. 1-3), head-wise multi-PNM KV partitioning, and an evaluation against 1-GPU and 2-GPU vLLM baselines and RetroInfer. The abstract claims up to 5.5x decode throughput and 15x P99 TPOT reduction. The directly measured micro-metrics (memory headroom, running batch size, P95/P99 TPOT, migration jitter) support the mechanism; the 5.5x throughput figure, however, is projected to a 512 GB per-device configuration that is not yet built, and the mixed-workload throughput results are in part emulated by injecting measured GPU occupancy traces into unmodified vLLM rather than measured end-to-end.","tokens_in":22947,"tokens_out":11881,"duration_ms":117577,"significance":"If the claims hold, this is a meaningful systems contribution: a credible, full-stack instantiation of length-based execution placement across GPU and PNM tiers, built on real hardware. The strongest parts are the directly measured micro-evidence — GPU memory headroom and running batch size under long-request injection (Fig. 15), migration jitter (Fig. 19), the measured-vs-estimated attention-latency agreement within ±5.6% (Fig. 11), and the accuracy gate at a 4% selection ratio (Fig. 13). The disclosure of the 512 GB projection (Section 7) is transparent, and the migration design is a clean solution to dynamic context growth. The principal risk is that the headline throughput number is not an experimental result: it depends on untested capacity-scaling assumptions, and the paper itself concedes in Section 8.1 that within the directly measured range NELSSA only sustains throughput 'comparable to the GPU-only baselines.' The stress-test concern therefore lands, and the central claim needs re-scoping or additional validation before the paper can be accepted as-is.","major_comments":[{"comment":"The abstract's central 5.5× throughput claim is not an experimental result: it is projected to a 512 GB per-device PNM configuration that the authors state they are 'currently developing' (§7), and §8.1 concedes that within the directly measured range 'NELSSA... sustains throughput comparable to the GPU-only baselines.' Every configuration with large gains (e.g., 5.5× at 1M tokens, batch 8) lies outside the measured range. The projection holds BWeff=200 GB/s, PNM-overhead, Transfer, and Others fixed (§8.1 and Fig. 11 caption); the ±5.6% validation bounds PNM-attn estimation only within the measured data-volume range, not 16× capacity scaling or all-four-device contention at larger working sets. Please report projected throughput under a BWeff sensitivity sweep (e.g., 100–200 GB/s) with per-step overheads scaled, and mark measured vs. projected explicitly wherever 5.5× appears (abstract,","section":"§7, §8.1, Fig. 10"},{"comment":"The routing threshold and the projected throughput share the same analytic model and the same prototype-derived parameters (C, S, BWeff, Tcomm), so the threshold is not an independent external benchmark. In particular, C is never defined beyond 'centroid compression factor' and its measured value is not reported, and the crossover from Eq. (3) is not validated against the measured decode-latency crossover (Fig. 3 is schematic). Since a wrong Tinput would either route short requests to PNM (latency regression) or leave long requests on the GPU (OoM), please (a) report C and Tcomm as measured on the prototype, (b) overlay measured decode latency for GPU and PNM paths vs. L against Eqs. (1)-(2), and (c) show the sensitivity of the Fig. 10 speedups to a ±20–30% perturbation of Tinput and to the selection ratio S.","section":"§4.1, Eqs. (2)-(3)"},{"comment":"The end-to-end mixed-workload throughput results (Figs. 14, 17) are not measured end-to-end: they are produced by injecting measured GPU occupancy traces into an unmodified vLLM engine. The validation of this emulation (4.12% error) is performed at approximately 100K tokens only, and the projected configurations in Fig. 10 (1M tokens, batch ≥ 8) are validated neither directly nor by emulation. This compounds the projection risk identified above. Please add a table classifying each reported number as directly measured / emulated / projected, state the region of applicability of the 4.12% error bound, and demonstrate (or bound) the emulation error in the projected batch-length range rather than asserting it.","section":"§8.3, §8.4"},{"comment":"The abstract claims NELSSA maintains 'the same level of accuracy as the GPU-only baseline,' but §8.2 reports only two RULER tasks (niah_multikey_3, qa_2) and shows no dense-attention baseline accuracy curve; 'stable accuracy' at a selection ratio ≥4% is a weaker claim than parity with the GPU-only dense path. Furthermore, S=4% is a free parameter of Eq. (2), and no sensitivity of the headline throughput to S is reported — Fig. 13's throughput data are shown at unspecified batch/context conditions. Please add a per-task accuracy table vs. the dense baseline over the evaluated 128K–1024K range (with context lengths stated) and a throughput-vs-S curve at the headline configuration.","section":"§8.2, Fig. 13, Abstract"}],"minor_comments":[{"comment":"The sentence 'This section presents the system architecture that makes these mechanisms practically realizable' is duplicated verbatim in consecutive paragraphs.","section":"§5.1"},{"comment":"The measured/projected boundary is not visually marked. Add shading or a divider so the reader can tell which bars are measured, and explain the 'X' OoM markers in the caption.","section":"Fig. 10"},{"comment":"Several axis labels and legends contain garbled glyph/unicode artifacts (e.g., the 'uni0000...' sequences in Fig. 18), rendering part of the figures unreadable. The figures need to be regenerated cleanly.","section":"Figs. 13, 18"},{"comment":"Specify how C is measured for the prototype (algorithm and measured value), and report M_kv per token for Llama3-8B so that the measured-range boundaries stated in §8.1 (e.g., batch 8 at 128K) are reproducible from the hardware specifications.","section":"Eq. (2)"},{"comment":"The communication-overhead percentages are produced by an analytical model, not measured. The text and caption should label this clearly as model-based extrapolation rather than presenting it alongside directly measured results.","section":"Fig. 12"},{"comment":"The burst-size values (BS=1 vs BS=2) and RPS for Fig. 14(a) appear only in the caption; state them in the text, and give the long-request fraction used for Fig. 14(a) explicitly.","section":"§8.3"},{"comment":"The 2×H100 baseline is called 'cost-comparable,' but no cost or power data are given for the PNM node (host CPU, NIC, four DDR5 modules). Either provide cost/power numbers or soften the term to 'cost-comparable by assumption.'","section":"§7, §9"}],"recommendation":"major_revision","confidential_remarks":"The paper is from SK hynix/KAIST, and reference [8] (IEEE CAL 2025), cited in §2.2 and §9.2 as 'prior work' on sparse attention on PNM, shares its first author with this submission; it is not flagged as the authors' own work. I do not read this as misconduct, but the editor may wish to have the self-citation explicitly disclosed. The larger editorial judgment is whether a 5.5× headline that is entirely a projection to unbuilt 512 GB hardware is acceptable in the abstract; my major-revision recommendation asks that the claims be re-scoped to measured vs. projected and that the projection be stress-tested with sensitivity analysis. The directly measured micro-evidence (Figs. 15, 19, 11) is solid and should be preserved; the fix is within the manuscript's scope. I also note the 'to the authors' knowledge... first end-to-end system on real PNM hardware' claim in §9.2 should be softened or supported with a more systematic prior-art check."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing to know: this is a real end-to-end prototype of length-based GPU/PNM placement for mixed-length LLM serving, and that combination is new. They derive a crossover threshold from a simple latency model, route short requests to the GPU and long requests to PNM, and migrate dynamically growing contexts one-way without recomputation. They built it on actual CXL-attached PNM hardware with sparse attention kernels, RDMA, and a host dispatcher, and the directly measured micro-metrics—memory headroom, running batch size, tail latency, migration jitter—support the mechanism. The accuracy check at 4% selection ratio is reasonable, and the vLLM emulation for system-wide throughput is validated to about 4%.\n\nHere's the rub. The headline 5.5x throughput and 15x P99 latency numbers in the abstract are projections to a 512 GB per-device PNM board that does not exist. The prototype has 32 GB per device. Section 7 says this explicitly, and Section 8.1 admits that within the measured range NELSSA sustains throughput 'comparable to the GPU-only baselines.' The 5.5x comes from extrapolating the measured per-step latency breakdown while holding effective bandwidth, transfer costs, and fixed overheads constant across a 16x capacity increase. That's plausible but untested, and the abstract presents it as a measured result. The system-level benefits in the mixed-length scenarios (larger GPU batch, reduced HoL blocking) are real and measured, but the central speedup claim rests on the projection.\n\nTwo smaller issues. First, the motivation leans on the claim that PNM-only routing hurts short requests, but there's no PNM-only baseline in the eval. It would be easy to add. Second, the workload is synthetic—sampled from Mooncake under Poisson, not a real trace replayed end-to-end—though they do inject a calibrated workload into unmodified vLLM, which is a reasonable emulation.\n\nOverall, this is a thoughtful systems paper. The architecture is genuinely interesting, the hardware prototype is real, and the projection is disclosed openly in the body—it's just not marked as projected in the abstract. If the 512 GB board sustains the assumed bandwidth and overheads, this could be an important contribution. If not, the measured contribution is more modest: it expands the serving envelope under OOM conditions and confirms the HoL-mitigation mechanism.\n\nI'd send it to peer review. The right revision is to re-scope the headline claims to the measured prototype or give real sensitivity analysis on the projection, and add a PNM-only baseline. Worth discussing at reading group; I'd cite it once the measured/projected line is drawn.","headline":"A real GPU-PNM serving prototype with a novel length-based placement mechanism, but the headline speedup is a 512 GB projection; the measured system is credible but roughly performance-neutral versus GPU-only outside OOM cases.","tokens_in":23433,"tokens_out":3024,"would_cite":true,"duration_ms":31162,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"NELSSA claims that routing short LLM requests to GPUs and long-context requests to near-memory sparse-attention hardware, with runtime migration for growing contexts, lifts decode throughput by up to 5.5x and cuts P99 latency by up to 15x a","keywords":["LLM serving","mixed-length workloads","Processing-near-Memory (PNM)","length-based request placement","sparse attention","KV cache migration","CXL memory disaggregation","decode throughput"],"falsifier":"Build or measure a full-capacity 512 GB per-device PNM module and rerun the Section 8.1 projection at the batch sizes where the 5.5x advantage is claimed (e.g., large batches at 128K-512K contexts). If the effective random-access bandwidth at those data volumes falls below the constant value the paper assumes, the projected throughput advantage disappears; if it holds, the claim stands. A cheaper check is to measure actual tokens/sec with several long requests resident on the current 32 GB prototype at batch sizes beyond the directly measured range and compare against the GPU-only baseline.","tokens_in":22486,"feed_emoji":"⚡","tokens_out":8567,"duration_ms":76141,"temperature":0.7,"pith_summary":"The paper argues that LLM serving workloads now mix requests from a few hundred to hundreds of thousands of tokens, and GPU-only systems handle this mix poorly because long-context requests exhaust GPU memory and stall short requests. NELSSA is proposed as a division of labour: GPUs run dense attention for short requests plus all the non-attention compute, while Processing-near-Memory (PNM) devices run sparse attention for long requests inside large-capacity memory. The point of division is set by a hardware-derived crossover length rather than by heuristic policy, and a background migration mechanism moves requests that grow during decoding onto the PNM tier without recomputation. If the argument holds, mixed-length serving can keep GPU batches large and tail latency low without sacrificing accuracy, which matters as agentic and reasoning workloads push context lengths upward.","feed_headline":"Length-split hardware serving lifts LLM throughput 5.5x","feed_subtitle":"Short prompts use GPUs; long ones use near-memory chips, keeping GPU batches big and tail latency down up to 15x.","key_machinery":"The load-bearing object is a hardware-derived crossover threshold T_input: the sequence length at which GPU dense attention latency equals distributed PNM sparse attention latency. The paper models GPU latency as L*M_kv/BW_GPU and PNM latency as centroid-search cost plus L*S*M_kv/(N*BW_eff_PNM) plus communication cost; equating them gives L_crossover(N), which grows smaller as more PNM devices are added. Around this threshold, NELSSA builds split-batch hybrid routing (the GPU runs all requests' QKV/MLP together and only branches at attention), seamless background migration with incremental chunked clustering of newly generated tokens, head-wise KV partitioning across PNM modules for load bal","core_discovery":"NELSSA's central claim is that decode attention should be executed on the tier whose memory is best matched to the request's current length, and that the boundary between tiers can be computed from hardware measurements. The paper equates a memory-bandwidth latency model for GPU dense attention with a model for distributed PNM sparse attention; solving the equation yields a crossover sequence length that becomes the routing threshold. Below it, requests run on the GPU; above it, attention runs entirely on PNM, while QKV projection and feed-forward networks are still batched globally on the GPU. Requests that start short and grow are handed off in the background after their KV cache is transf","pith_inferences":["Editorial inference: the crossover-threshold method is generic; any two compute/memory tiers with measurable latency-vs-length curves could be placed on the same kind of curve-crossing argument, so the design could extend to CPU-attached memory or storage-class tiers, not just PNM.","Editorial inference: the projection to full-capacity 512 GB PNM boards is the natural place to test the paper's central numbers; if effective random-access bandwidth falls as KV-cache volume grows, the 5.5x figure would shrink, though the batch-size restoration effect could persist.","Editorial inference: the static crossover threshold could be made adaptive at the orchestrator level—something the paper lists as future work—which would let the system shift traffic between tiers as queueing delays build, turning NELSSA into a closed-loop controller.","Editorial inference: the accuracy operating point (4% selection ratio) is established on specific long-context benchmark tasks; workloads with different token relevance distributions may need their own sparsity calibration, so the exact fixed ratio should not be assumed universal."],"forward_implications":["GPU batches stay large under mixed traffic instead of collapsing to near one when long contexts arrive, so decode throughput tracks short-request density rather than the longest resident context.","Long-context requests no longer block short ones: tail time-per-output-token improves by roughly 6x at P95 and up to 15x at P99 in the paper's mixed-length experiments.","Requests whose context grows mid-generation can migrate to the PNM tier in the background, avoiding both recomputation and PCIe-bound swap recovery.","Because only attention branches by length, the GPU still performs all QKV and feed-forward work in one global batch, so tensor-core utilization is not sacrificed by the split.","The benefit widens as the long-request fraction grows: at high long-request ratios the GPU-only baseline loses throughput steeply while the split system degrades much more slowly."],"fun_headline_variants":["Length-aware GPU+PNM split lifts decode 5.5x","Short to GPU, long to PNM: 15x lower tail latency","Mixed-length LLM serving gets 5.5x via length routing","Crossover length dictates GPU vs PNM for attention","NELSSA: Length-based chip placement speeds LLM serving"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The headline gains depend on a hardware projection—the prototype has 32 GB per PNM device and the paper reports projected throughput at 512 GB per device, holding measured effective bandwidth constant across that range (Sections 7 and 8.1)—so if full-capacity PNM modules do not sustain the same bandwidth at larger KV-cache volumes, the 5.5x claim collapses; within the directly measured range, the paper concedes throughput only comparable to GPU-only baselines.","fun_headline_variants_meta":{"raw":{"variants":["Length-aware GPU+PNM split lifts decode 5.5x","Short to GPU, long to PNM: 15x lower tail latency","Mixed-length LLM serving gets 5.5x via length routing","Crossover length dictates GPU vs PNM for attention","NELSSA: Length-based chip placement speeds LLM serving"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000199,"raw_usage":{"total_tokens":1240,"prompt_tokens":810,"completion_tokens":430,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":554,"completion_tokens_details":{"reasoning_tokens":338}},"tokens_in":554,"tokens_out":430,"duration_ms":5710,"temperature":1.0,"reasoning_tokens":338,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T11:52:54.637418+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build or measure a full-capacity 512 GB per-device PNM module and rerun the Section 8.1 projection at the batch sizes where the 5.5x advantage is claimed (e.g., large batches at 128K-512K contexts). If the effective random-access bandwidth at those data volumes falls below the constant value the paper assumes, the projected throughput advantage disappears; if it holds, the claim stands. A cheaper check is to measure actual tokens/sec with several long requests resident on the current 32 GB prototype at batch sizes beyond the directly measured range and compare against the GPU-only baseline.","supporting_citations":[],"review_version":1}