{"id":"cd7868b0-81b5-4f4e-9528-b707ae2b7083","arxiv_id":"2608.00867","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"In multi-die GPU scale-up systems, placing communication threadblocks and memory near I/O ports and routing traffic to nearby ports speeds up small collectives by up to 1.8x and LLM inference decode by 7% on average.","lead":"This paper finds that in large multi-die GPUs, the physical location of compute, memory, and network ports inside each GPU changes inter-GPU communication latency by up to 2x, and proposes two policies, NUNA-aware routing and placement, that shorten those paths. Applied together, the policies speed up latency-sensitive collectives by up to 1.8x and cut average time-per-output-token in LLM inference by about 7%.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline speedups depend on the assumed 0.9 us NoC / 1 us scale-up latency ratio; no sensitivity analysis is given, and the intro's 1.8x transfer-slowdown figure appears inconsistent with that ratio.","rationale":"Good-faith reading: the paper has a real motivating observation (remote-store latency varies with CU and address on MI210/MI355X/MI300X) and proposes a coherent, low-overhead mechanism stack. That is genuine evidence and a plausible contribution. The concern is not that NUNA is imaginary; it is that the headline magnitudes are fixed by a single assumed latency ratio for a hypothetical next-generation system. No parameter sweep is reported for the most important ratio, and the text contains two internal inconsistencies about the size of the claimed effect. These do not by themselves refute the central claim, but they make the 1.8x and 7% numbers unverified as stated. The proposed sweep would settle whether the conclusion is robust or an artifact of the 0.9/1.0 us assumption. This supports, rather than changes, the reader's CONDITIONAL verdict: the paper needs a sensitivity analysis and artifact release before the headline numbers should be taken as established.","tokens_in":26092,"tokens_out":9563,"duration_ms":83608,"concrete_test":"Re-run the ASTRA-sim evaluation with the scale-up network die-to-die latency swept over {0.25 us, 0.5 us, 1 us, 2 us, 4 us} and link/switch latency scaled proportionally, keeping the NoC parameters fixed; recompute (i) the NAP+NAR speedup for 100 kB All-Gather on 8 and 64 GPUs (Figure 9) and (ii) the decode TPOT geomean over the 12-model suite (Figure 15). If at scale-up latency 2x the NoC latency the collective speedup falls below about 1.3x and the TPOT gain below about 3%, the headline 1.8x/7% numbers are an artifact of the assumed 0.9 us vs 1 us ratio. Also recompute the specific configuration behind the intro's '80% reduction' claim to determine whether the correct speedup is 5x or 1.8x; if it is 5x the abstract understates the result, and if it is 1.8x the intro overstates it.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise is that intra-socket NoC latency is comparable to scale-up link latency on next-generation GPUs, so that NAP/NAR's reduction of the NoC segment translates into large collective speedups. Section 3.1.3 states this as a single point estimate: 'the NoC transfer takes up to 0.9 us, while the scale-up latency is 1 us.' The paper provides no sensitivity analysis around this ratio, and the real-system profiling in Section 3.2 shows non-uniformity exists now but does not establish the future ratio. The central mechanisms only shorten the intra-socket segment; any fixed scale-up latency dilutes the benefit. The paper's own numbers expose the fragility: with a 3x NoC spread (0.3 to 0.9 us) plus 1 us scale-up, the worst/best total transfer ratio is 1.9/1.3 = 1.46x, not the 1.8x 'uncontended GPU-to-GPU transfer slowdown' claimed in Section 1. Relatedly, Section 1 also says NAP+NAR 'reduce collective execution time by up to 80%', which is a 5x speedup and contradicts the abstract's 1.8x; both cannot be correct. The end-to-end 7% TPOT gain is additionally computed under a serial compute/communication model (Section 5.2), so overlap in real inference would shrink it further. The claim is therefore conditional on the latency-ratio assumption and on the internal consistency of the speedup accounting.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces the term non-uniform network access (NUNA) to describe how intra-socket wire distance between compute units, HBM stacks, and I/O ports makes inter-GPU scale-up communication latency depend on spatial placement. It proposes two optimizations: NUNA-aware routing (NAR), which restricts latency-sensitive flits to I/O ports physically closer to the requesting CU via statically partitioned address ranges, and NUNA-aware placement (NAP), which places collective threadblocks and memory pages near those I/O ports. The paper profiles three real AMD GPU systems to show that remote-access latency varies with CU and address location, then evaluates NAR and NAP in the ASTRA-sim 3.0 simulator. The headline claims are up to 1.8x collective speedups for NAP+NAR over the locality-unaware baseline and 7% mean (28% max) end-to-end time-per-output-token improvement for LLM inference.","tokens_in":26419,"tokens_out":5089,"duration_ms":45401,"significance":"If the claimed speedups are validated, the paper makes a timely contribution: it identifies a new locality dimension for scale-up GPU systems and provides a coherent, backward-compatible software/hardware mechanism for exploiting it. Strengths include the real-system profiling of MI210, MI355X, and MI300X, which documents that remote-access latency variation grows with socket size; the isolation study showing that NAR is necessary for NAP to help; and the sensitivity studies for collective size and NAR granularity. The paper does not, however, ship code or validation data, and its central quantitative claims depend on a self-built simulator and on two strong assumptions: the NoC/scale-up latency ratio and a serial compute-communication model. Those assumptions are load-bearing for the 1.8x and 7% headline numbers, so the result is conditional until the assumptions are tested and the internal speedup accounting is reconciled.","major_comments":[{"comment":"The manuscript states two incompatible headline speedups. The abstract claims up to 1.8x faster collectives for NAP+NAR, while the introduction says 'NAP and NAR applied together can reduce collective execution time by up to 80%' (a 5x speedup). Both cannot describe the same comparison against the same baseline; the 80% reduction is hard to reconcile with the 1.8x upper bound reported later. This needs to be corrected and the exact comparison (which configuration, collective, message size, and pod size) stated for each number.","section":"§1 and Abstract"},{"comment":"The motivating 1.8x 'uncontended GPU-to-GPU transfer slowdown' is not supported by the paper's own latency numbers. Section 3.1.3 gives a NoC transfer of up to 0.9 us and a scale-up latency of 1 us; with a 3x NoC spread (0.3 to 0.9 us) plus a fixed 1 us scale-up term, the worst/best total transfer ratio is (1.9)/(1.3) = 1.46x, not 1.8x. Because NAP and NAR act only on the intra-socket segment, any fixed scale-up latency dilutes the benefit, and the claimed 1.8x collective speedup therefore depends on the assumed latency ratio. The paper provides no sensitivity analysis over this ratio. I ask the authors to (i) show exactly how the 1.8x transfer-slowdown and 1.8x collective-speedup numbers are derived, and (ii) add a sensitivity study varying the NoC/scale-up latency ratio (e.g., 0.3x to 3x of the assumed 0.9 us) and report collective speedup as a function of that ratio.","section":"§3.1.3 and §1"},{"comment":"The end-to-end inference results are computed under a serial compute/communication model, as stated in Section 5.2: 'we model compute and communication serially since intra-batch overlap is ongoing research.' This assumption removes any overlap between collective communication and compute, which exposes the full communication time and therefore inflates the measured benefit of faster collectives. Since the paper itself cites prior overlap work (e.g., T3 [80] and Splitwise [78]), a serial model is a worst-case assumption, not a neutral one. The 7% mean TPOT claim is thus conditional on zero overlap. The authors should quantify how the end-to-end speedup changes under partial or full overlap (e.g., by applying an overlap factor to the collective time in their analytical model).","section":"§5.2"},{"comment":"The central collective-speedup results come from ASTRA-sim 3.0, a simulator co-authored by this team (reference [98]), and the paper does not ship the simulator configuration, trace-generation tool, or validation data. Section 3.2's real-system profiling validates the existence of latency variation on current hardware, but it does not validate that the simulated 24x6 mesh with the chosen hop latencies reproduces those measurements. Without a calibration/validation experiment comparing simulated remote-store latency distributions against the measured MI300X values, the quantitative claims (1.8x for NAP+NAR, up to 28% TPOT improvement) cannot be independently checked. I would like the paper to include a validation section, or at minimum release the exact configuration and a comparison of simulated versus measured latency distributions.","section":"§5.1 and §3.2"}],"minor_comments":[{"comment":"There are several typos: 'Specifcially' in the introduction, 'to to limit' in Section 4.1.2, and 'traffics' in Section 6.3.","section":"§1, §4.1.2, §6.3"},{"comment":"Figure 2 reports collective sizes derived from an in-house trace-generation tool, but the methodology is deferred to Section 5.2 and no validation is given for the trace shapes or collective sizes. A short validation sentence (e.g., comparison with known model configurations) would help the reader trust the prefill/decode CDFs.","section":"§2.2, Figure 2"},{"comment":"The simulated system table lists '144 I/O ports (72 per die)' for a 4-compute-die socket, while Section 4.1.1 and Figure 5 describe a socket with four I/O ports. The figure is clearly illustrative, but the relationship between the illustrative four-port model and the simulated 144-port model should be stated explicitly to avoid confusion.","section":"§5.3, Table 2"},{"comment":"The description of the UA latency histogram as 'the Gaussian distribution of individual path latencies in a mesh' is informal; the reported 920 ns average and 1,320 ns maximum are more informative. Reporting percentiles (e.g., p50/p99/max) directly in the text would make the comparison clearer.","section":"§6.2"},{"comment":"The phrase 'exposed communication ratio is approximately 4:1 for prefill and 5:1 for decode' is ambiguous: it is not clear whether this is compute-to-communication or communication-to-compute, and how the ratio is computed. Clarifying the definition would help the Amdahl's law argument.","section":"§6.7"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the scope of a systems/architecture venue, and the idea is worth pursuing. The main risk is that the quantitative claims rely on a self-built simulator and on two assumptions (the NoC/scale-up latency ratio and serial compute/communication overlap) that are not stress-tested. I would ask the editor to require the sensitivity analysis and the internal-consistency fix as conditions for acceptance, rather than treating them as optional."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"NUNA names a real effect that prior NUMA/NUCA work on GPUs has mostly missed: the physical distance between a CU and the I/O port changes inter-GPU communication latency, and the effect grows as sockets get bigger. The real-system profiling across MI210, MI355X, and MI300X is a genuine motivating observation, and the two mechanisms (NAR and NAP) are plausible and reasonably well specified. The related-work comparison is careful, and the microbenchmark sweep across collectives, pod sizes, and isolation studies is more thorough than most simulation papers of this type.\n\nThe soft spots are real, though. The headline numbers don't survive a close reading. The abstract says 'up to 1.8x faster collectives'; Section 1 says NAP+NAR can 'reduce collective execution time by up to 80%.' An 80% reduction is a 5x speedup. Those two upper bounds cannot both be correct for the same metric, and the paper never explains the discrepancy. Second, the claimed 1.8x worst-case GPU-to-GPU transfer slowdown is not derivable from the paper's own latency numbers: 0.9 us NoC plus 1 us scale-up in the worst case versus 0.3 us plus 1 us in the best case gives 1.46x, not 1.8x. Third, the load-bearing assumption that intra-socket NoC latency is comparable to scale-up latency (0.9 vs 1 us) rests on a single point estimate. There is no sensitivity analysis around this ratio, and the end-to-end numbers are produced by an in-house analytical model that serializes compute and communication. The 7% TPOT gain is therefore an upper bound at best, and it shrinks if real inference overlaps compute with communication.\n\nNone of this kills the core idea. The real-system data shows the effect exists today; the simulation shows it could matter in a plausible next-generation system. The paper is honest about some limitations, like oversubscription for large collectives. The issue is that the quantitative claims are presented with more confidence than the evidence supports.\n\nThis paper deserves peer review, not desk rejection. The referee should ask for artifact release, a sensitivity analysis over the NoC/scale-up latency ratio, reconciliation of the 1.8x and 80% claims, and an end-to-end model that acknowledges or bounds overlap effects. I'd bring it to a reading group and I'd probably cite the NUNA concept, but I wouldn't repeat the headline numbers without qualification.","headline":"A useful new framing of intra-socket I/O locality for scale-up GPUs, with real motivating measurements, but headline speedup claims are internally inconsistent and the evaluation is simulation-only.","tokens_in":27017,"tokens_out":4139,"would_cite":true,"duration_ms":35006,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Non-uniform network access inside large multi-die GPUs is a first-order cost in GPU-to-GPU communication, and routing plus placement that respect it make latency-sensitive collectives up to 1.8x faster.","keywords":["non-uniform network access (NUNA)","multi-die GPU","scale-up network","collective communication","latency-sensitive collectives","LLM inference","threadblock placement","memory page placement"],"falsifier":"Run a 1 MB All-Gather on a next-generation multi-die GPU with a rail-optimized single-level Clos, and compare round-robin placement with full-port hashing against the paper's NAP+NAR scheme. If the measured speedup comes nowhere near 1.8x -- for example, because scale-up link latency or switch queueing so dominates the remote write that the choice of CU, HBM stack, and I/O port no longer changes the total time -- the central claim fails. A simpler check is to measure remote-store latency from the fastest versus slowest compute unit to the same remote address: the paper predicts that gap grows with socket size.","tokens_in":25857,"feed_emoji":"⚡","tokens_out":11624,"duration_ms":90839,"temperature":0.7,"pith_summary":"This paper is trying to establish that, in large multi-die GPUs, the physical location of a threadblock, its data, and the I/O port its traffic leaves from changes remote-memory latency by almost 2x, and that this non-uniform network access (NUNA) is a first-order cost in GPU-to-GPU communication. It claims that two software-controllable policies -- NUNA-aware routing (NAR), which sends latency-sensitive flits only to nearby I/O ports, and NUNA-aware placement (NAP), which puts threadblocks and memory pages near those ports -- jointly make latency-sensitive collectives up to 1.8x faster than a locality-unaware baseline. On an end-to-end evaluation across 12 LLM architectures on 2-64 GPUs, the combined scheme cuts time per output token by 7% on average and up to 28%, with prefill time-to-first-token down 6% on average. The practical stake is that LLM inference decode is dominated by small, latency-bound collectives, which is exactly the regime where NUNA gains are largest.","feed_headline":"Geometry-aware routing and placement speed GPU collectives up to 1.8x","feed_subtitle":"Spatial placement inside the socket cuts latency-sensitive collective time, speeding LLM inference by 7 percent.","key_machinery":"The load-bearing object is the NAR domain: a statically partitioned slice of physical address space that constrains a CU's off-chip flits to a subset of I/O ports physically near that CU, while still load-balancing across ports within the subset. NAP's algorithm takes a collective plan (threadblock access patterns over fixed-size chunks), first groups threadblocks and chunks into logical groups, then greedily maps each group to a NAR domain and, within it, to the CU and HBM stack closest to the I/O ports subject to resource budgets such as outstanding-request capacity and bandwidth-delay product. The paper also exposes NAR granularity, the number of domains, as the tuning knob that trades low latency against port load balancing. These mechanisms are implemented with existing threadblock-to-CU affinity masks and page-to-stack placement, plus a modification to address-hashing logic, so they are backward-compatible and disable-able.","core_discovery":"The paper's central claim is that the remote-store path between GPUs has two comparable legs: the on-chip trip from a compute unit to an edge I/O port and through the receiving socket to its memory stack, and the scale-up link between sockets; for the modeled next-generation system the on-chip leg reaches 0.9 $\\mu$s while the scale-up leg is 1 $\\mu$s. Because these legs are comparable, where a threadblock sits, where its buffers live, and which I/O port hashes the traffic determine the latency of a small collective. The paper argues that NAR plus NAP addresses all three choices together: NAR routes flits to a physically close subset of ports, and NAP places communicating threadblocks on nearby CUs and their chunks on nearby HBM stacks, with a greedy two-phase algorithm that resolves resource contention. In simulation, the combined scheme reduces All-Gather, All-Reduce, and All-to-All execution time by up to 80% (up to 1.8x speedup) for small collectives, with a 1.32x geomean communication speedup in prefill and 1.56x in decode across the LLM workloads.","pith_inferences":["The 1.8x figure is tied to the modeled parameter regime: on sockets where the scale-up link dominates the on-chip leg the same policies would yield less, while on still-larger dies the gains could exceed 1.8x.","The same NAR/NAP machinery generalizes to any latency-sensitive off-chip traffic with a known access plan, such as remote atomics or sparse attention token routing, not just the three collectives evaluated.","Because the paper serializes compute and communication, its end-to-end numbers are optimistic about exposure; if future systems overlap compute and collectives well, per-collective speedups would remain but the 7% TPOT gain would shrink.","A natural testable extension is to derive the topology input to NAP from measured per-CU remote-store latencies rather than geometric layout, so the method stays calibrated if real meshes or packaging differ from the modeled 24x6 mesh."],"forward_implications":["Small, latency-bound collectives (up to roughly 1 MB) benefit most, with 1.5-1.9x speedups that decay to about 1.1x by 64 MB as bandwidth saturation hides the on-chip spatial effect.","The scheme cuts decode time-per-output-token by 7% on average and 28% at best, and prefill time-to-first-token by 6% on average and 11% at best, across the 12 evaluated LLM architectures and pod sizes from 2 to 64 GPUs.","Placement without routing is not enough: NAP alone can slow down 10 MB collectives because hashed traffic traverses the whole socket and congests the on-chip network; routing and placement must be designed jointly.","The best NAR granularity depends on threadblock count, so the allocation algorithm should choose the number of domains per workload rather than fixing it in hardware.","Concurrent compute kernels degrade the NUNA-aware collective less than the baseline, with degradation appearing only at roughly twice the GEMM size, because localized I/O traffic avoids mid-die congestion."],"supporting_citations":[{"why":"Supplies the CU-level simulator in which all collective and end-to-end evaluations in the paper are run.","marker":"[98]"},{"why":"Provides the threadblock-level collective plan representation that the placement algorithm takes as input.","marker":"[25]"},{"why":"Supplies the multi-chip-module GPU design point and threadblock-scheduling baseline that NUNA extends and contrasts with.","marker":"[13]"},{"why":"Prior NUMA-aware GPU work with fine-grained I/O port assignment that the paper argues misses latency differences across ports.","marker":"[63]"},{"why":"Source of measured GPU on-chip network latency characteristics used to set the intra-socket transfer delays.","marker":"[44]"},{"why":"Provides modern GPU architecture benchmarking used together with [44] to calibrate on-chip and memory parameters.","marker":"[57]"},{"why":"Provides the locality-aware data and threadblock management baseline that NAP is compared against.","marker":"[47]"},{"why":"Supplies the threadblock-to-CU and data-placement abstraction that the NAP implementation builds on.","marker":"[96]"}],"fun_headline_variants":["Spatial placement and routing give GPU collectives up to 1.8x speedup","NUNA-aware thread and data placement speeds LLM inference 7%","Where you place GPU threads and data on the die speeds collectives 1.8x","Greedy placement and spatial routing cut GPU collective latency up to 1.8x"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that moving data across the GPU's interior to the edge I/O port (up to 0.9 $\\mu$s) costs about as much time as sending it between GPUs over the scale-up link (1 $\\mu$s), so the exact placement of threads, data, and ports inside the socket is a first-order factor; if inter-GPU link latency grows faster than on-chip latency, or if compute hides the on-chip transfer, the speedups shrink.","fun_headline_variants_meta":{"raw":{"variants":["Spatial placement and routing give GPU collectives up to 1.8x speedup","NUNA-aware thread and data placement speeds LLM inference 7%","Where you place GPU threads and data on the die speeds collectives 1.8x","Greedy placement and spatial routing cut GPU collective latency up to 1.8x"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000863,"raw_usage":{"total_tokens":3775,"prompt_tokens":1010,"completion_tokens":2765,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":626,"completion_tokens_details":{"reasoning_tokens":2673}},"tokens_in":626,"tokens_out":2765,"duration_ms":17800,"temperature":1.0,"reasoning_tokens":2673,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:16:00.075373+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a 1 MB All-Gather on a next-generation multi-die GPU with a rail-optimized single-level Clos, and compare round-robin placement with full-port hashing against the paper's NAP+NAR scheme. If the measured speedup comes nowhere near 1.8x -- for example, because scale-up link latency or switch queueing so dominates the remote write that the choice of CU, HBM stack, and I/O port no longer changes the total time -- the central claim fails. A simpler check is to measure remote-store latency from the fastest versus slowest compute unit to the same remote address: the paper predicts that gap grows with socket size.","supporting_citations":[{"cited_title":"ASTRA-sim 3.0: Next-Level Distributed Machine Learning Simulations via High-Fidelity GPU and Infrastructure Modeling","cited_arxiv_id":"2606.10440","evidence_quote":"Supplies the CU-level simulator in which all collective and end-to-end evaluations in the paper are run."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of measured GPU on-chip network latency characteristics used to set the intra-socket transfer delays."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides modern GPU architecture benchmarking used together with [44] to calibrate on-chip and memory parameters."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the locality-aware data and threadblock management baseline that NAP is compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the threadblock-to-CU and data-placement abstraction that the NAP implementation builds on."}],"review_version":2}