{"id":"6dfca161-6910-4246-ae8e-d20689f497d3","arxiv_id":"2412.14335","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Using DMA engines for collectives (ConCCL) lifts concurrent compute-communication speedup on MI300X from 21% to 72% of ideal.","lead":"This paper measures why overlapping GPU computation and communication in ML rarely reaches ideal speedup, and shows that moving communication to the GPU's existing DMA engines closes most of that gap. On an AMD MI300X system, their ConCCL proof-of-concept raises average speedup from 21% of ideal to 72%, up to 1.67x over serial execution.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ConCCL speedup claims are uninterpretable without demonstrating that the DMA-based collectives actually transfer the correct data.","rationale":"The reader's weakest_assumption was that MI300X sDMA engines avoid compute units and L1/L2 caches. That assumption is well-supported by the chip's architecture and by the paper's own design description (Section II-A, VI-A); even if partially wrong, it would weaken the mechanism story but not necessarily the measured speedup. By contrast, the absence of any correctness verification for the DMA-based collectives is a direct threat to the validity of every speedup number in Figure 10. If the transfers do not produce the correct gathered/transposed buffers, then the reported overlap is not real communication, and the 72% figure could simply reflect incomplete work. This is a concrete, easily checked precondition, and it is omitted from the paper. The reader did flag correctness as an addressable weakness in the rationale, but not as the weakest assumption, so agreement is partial. The verdict remains CONDITIONAL: the central claim is plausible and the measurements are systematic, but the paper must supply a correctness check (and ideally artifacts) before the speedup can be accepted as a genuine communication offload result.","tokens_in":16584,"tokens_out":5543,"duration_ms":52312,"concrete_test":"Run the ConCCL all-gather and all-to-all PoCs for all C3 sizes in Table II on the 8-GPU node while the concurrent GEMM is executing; after each run, compare every output buffer on every rank against a reference produced by RCCL or a CPU-based implementation (e.g., all-gather from each rank's initial buffer). If any byte differs, the speedup measurements in Figure 10 are invalid. Even a single failing size would require re-evaluating the central claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"ConCCL's central result (72% of ideal speedup, up to 1.67x) is measured on DMA-engine-based all-gather/all-to-all proof-of-concepts. For these measurements to mean anything, the communication must actually complete correctly. The paper reports timing and throughput comparisons (Section VI-C, Figure 9) and states ConCCL is on par with RCCL for >=128MB, but it never states that transferred data was validated against a reference. This is not a minor omission: the PoCs use a simple direct algorithm with transfers scheduled individually via hsa_amd_memory_async_copy_on_engine, and the paper describes no content validation. If a transfer silently copied the wrong bytes or was dropped, the GEMM would appear faster simply because the communication was not really happening, inflating the speedup. The 21% vs 72% comparison (c3_base vs ConCCL) is only valid under byte-exact communication. No correctness check is described anywhere in the text, so the headline claim currently cannot be distinguished from a benchmarking artifact.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper characterizes concurrent computation and communication (C3) on an 8-GPU AMD MI300X node. It introduces a taxonomy based on GEMM type and collective type, measures isolated compute and memory requirements, and defines ideal speedup as serial execution time divided by the maximum of isolated GEMM and collective times. It reports that baseline C3 achieves only 21% of the ideal speedup, that schedule prioritization and compute-unit partitioning raise this to about 42%, and that offloading all-gather/all-to-all communication to the GPU's sDMA engines with ConCCL proof-of-concepts achieves on average 72% of ideal speedup (up to 1.67x). The paper also proposes runtime heuristics for schedule prioritization and resource partitioning.","tokens_in":16789,"tokens_out":7481,"duration_ms":64719,"significance":"The paper provides a useful, systematic characterization of an important real platform and makes a plausible case that DMA-engine-based collectives can reduce compute/cache interference in C3. The strengths include a concrete taxonomy with model-derived and synthetic scenarios, isolated compute/memory profiling, simple runtime heuristics with a held-out validation (24/30 scenarios, max 1.5% loss), and the use of an external baseline (RCCL) rather than fitting parameters to the target result. The main empirical claims are not circular. However, reproducibility and correctness are currently limited by the absence of a code artifact, the absence of error bars for the 9-run measurements, and the absence of any validation that ConCCL transfers are byte-exact.","major_comments":[{"comment":"No correctness check for ConCCL transfers is described anywhere in the manuscript. The paper reports that ConCCL is 'at par' with RCCL for >=128MB and uses this to justify evaluating C3 with ConCCL, but the speedup numbers in Figure 10 are only meaningful if the DMA collectives actually transfer the correct data. Because the ConCCL PoCs use simple direct transfers via hsa_amd_memory_async_copy_on_engine, a dropped or wrong-byte transfer would make the GEMM appear faster and would inflate the reported 72% of ideal speedup. Please add an explicit validation step (e.g., buffer comparison or checksum after each collective for every size and configuration) and report that all timing runs used validated transfers.","section":"Section VI-B/VI-C/VI-F (Figs. 9-10)"},{"comment":"The paper runs 9 measured executions per configuration but reports only point averages, with no error bars, confidence intervals, or min-max values. The central quantitative claims (21%, 42%, 66%, 72% of ideal speedup; 1.67x) are averages over a small number of runs; without a measure of variance it is impossible to tell whether the differences among c3_base, c3_sp, c3_rp, ConCCL, and ConCCL_rp are larger than run-to-run noise. Please report per-configuration dispersion, such as standard deviations or min-max ranges, for Figures 8 and 10.","section":"Section IV-A1 and all result figures"}],"minor_comments":[{"comment":"The text says 'all-gather kernels need 32 CUs, while all-to-all kernels need 64 CUs', but the Figure 5 captions state 'All-gather default #CUs=64, All-to-all default #CUs=56'. Please clarify which number is the saturation threshold and which is the library default.","section":"Section IV-B1 vs. Figure 5"},{"comment":"No code or artifact is provided; given that the contributions are empirical, an artifact appendix with the ConCCL PoCs and measurement scripts would substantially improve reproducibility.","section":"General"},{"comment":"The c3_rp result is defined as the best-performing CU allocation from a sweep over all powers-of-two allocations, which is an oracle/upper-bound result; the text should make clear that the 41% figure in Figure 8 is not the expected performance of a fixed strategy but an optimized upper bound, even though the heuristic in Section V-C recovers much of the benefit.","section":"Section V-B"},{"comment":"There is a typo: 'as such as be scheduled concurrently' should read 'as such can be scheduled concurrently'.","section":"Section II-C"},{"comment":"The definition of 'latency-bound' communication ('kernel latency at/before this size does not increase commensurate to size') is vague; a quantitative criterion, such as a bandwidth plateau threshold, would make the taxonomy easier to reproduce.","section":"Section III"}],"recommendation":"major_revision","confidential_remarks":"The missing correctness validation is the main blocking issue: without it, the headline ConCCL speedup could be a benchmarking artifact. The rest of the empirical claims would also benefit from error-bar reporting. I believe the manuscript is salvageable with an added validation section and dispersion reporting; the central idea is interesting and the study is otherwise systematic."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Take the headline numbers with a grain of salt until the authors show that ConCCL's DMA transfers actually move the correct bytes. The rest of the paper is a solid, useful characterization of C3 on MI300X.\n\nThe paper does several things well. It sets up a taxonomy of C3 (G-long, C-long, GC-equal) and applies it to a sensible set of GEMM sizes from LLaMA training plus synthetic collectives. The isolated-execution analysis of CU and bandwidth needs is straightforward and the resulting ideal speedup projection is honest. The schedule-prioritization and resource-partitioning results (21% to 42% of ideal speedup) are credible and internally consistent. The ConCCL proof-of-concept is a simple but real demonstration that DMA offload can bypass CU and L1/L2 interference on a shipping AMD part, reaching about 72% of ideal speedup. The heuristics are validated on held-out C3 scenarios, so the main claim is not circular.\n\nThe soft spots are in the measurement and reporting. First and most important, the paper never states that it validates the contents of ConCCL transfers. If the DMA copies silently drop bytes or copy the wrong data, the GEMM would look artificially fast. The speedup comparison between c3_base and ConCCL only makes sense under byte-exact communication. This is not a minor omission; it's a load-bearing check that's currently absent. Second, there are no error bars: only nine runs per configuration, and the paper reports the best CU allocation from a sweep as the result. That selection bias is worth a sentence or two. Third, no code or data are released, so the measurements can't be reproduced. None of these are hard to fix. The correctness check is a few lines of code; the error bars just require reporting standard deviation; and the artifact can be posted. But without the first, the central result is unverifiable.\n\nWho should read this? People who care about GPU communication offload, especially on MI300X, and anyone designing runtimes for compute-communication overlap. It is not a paradigm shift, but it makes a concrete case for DMA engines, and the taxonomy will be cited.\n\nRecommendation: send it to peer review, but require the authors to verify and report transfer correctness, provide error bars and clearly state the CU allocation selection methodology, and make the PoC code and data available. If they do that, the paper will be a useful, defensible contribution.","headline":"A solid C3 characterization that is currently unverifiable on its central claim because the paper never checks that the DMA transfers actually move the right bytes.","tokens_in":17334,"tokens_out":3765,"would_cite":true,"duration_ms":32221,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Offloading GPU communication to DMA engines closes most of the gap between realized and ideal speedup for concurrent ML compute and communication.","keywords":["concurrent computation and communication","GPU DMA engines","collective communication","GEMM interference","schedule prioritization","resource partitioning","MI300X","ML training"],"falsifier":"Run a compute-bound GEMM alone, then run the same GEMM concurrently with a large DMA transfer and with a compute-unit-based collective, measuring both GEMM slowdown and L2 or Infinity Cache traffic with a profiler. If the DMA-concurrent GEMM slows down by more than a negligible amount, or if DMA transfers measurably increase cache access counts, the no-interference premise fails and ConCCL's average 72%-of-ideal figure would not hold on that system.","tokens_in":16397,"feed_emoji":"⚡","tokens_out":6299,"duration_ms":52321,"temperature":0.7,"pith_summary":"This paper tries to establish that concurrent computation and communication (C3), a common pattern in ML training, is far less beneficial on GPUs than it could be, and that most of the lost benefit is avoidable. On an MI300X 8-GPU node, overlapping a matrix-multiplication kernel (GEMM) with an all-gather or all-to-all collective reaches only 21% of the ideal speedup, defined as serial time over the longer of the two isolated kernels. The paper shows that scheduling the communication kernel first and partitioning compute units raises that to 42%, and that offloading communication to the GPU's DMA engines, through proof-of-concept libraries called ConCCL, raises it to 72% on average, up to 1.67x speedup. The reason ConCCL works is that DMA transfers use no compute units and bypass L1/L2 caches, whereas compute-unit-based collectives steal both from the concurrent GEMM. If true, the result makes a concrete case for making GPU DMA engines more capable.","feed_headline":"DMA offload lifts GPU overlap to 72% of ideal speedup","feed_subtitle":"On AMD MI300X, all-gather and all-to-all via DMA engines raise realized speedup from 21% to 72% of ideal.","key_machinery":"The central mechanism is the GPU's SDMA (system direct memory access) engine: a copy engine on the I/O die that, when given a transfer command, moves data between HBM and the intra-node network without occupying compute units and without touching L1/L2 caches. ConCCL breaks each collective operation into individual transfers scheduled on these engines from the CPU through an HSA runtime call. The supporting mechanisms are the 'ideal speedup' yardstick, defined as serial execution time divided by the maximum of isolated GEMM time and isolated collective time, and the C3 taxonomy (G-long, C-long, GC-equal; compute-bound versus memory-bound GEMMs; latency-bound versus bandwidth-bound collectives) that organizes the measured scenarios. Schedule prioritization and compute-unit partitioning are the two software-only optimizations that bring performance from 21% to 42% of ideal before the DMA offload is applied.","core_discovery":"The paper's central discovery is that the gap between realized and ideal concurrent computation-communication speedup on an 8-GPU MI300X node is mostly caused by compute-unit and cache interference, not by memory bandwidth, and that this gap can be substantially closed by offloading collectives to the chip's DMA engines. With ConCCL proof-of-concepts for all-gather and all-to-all, average realized speedup rises from 21% to 72% of ideal, with a maximum of 1.67x over serial execution, and this is achieved without new hardware. The paper supports this with a taxonomy of C3 scenarios, isolated kernel measurements, and two software-only optimizations that recover part of the gap before the DMA offload is added.","pith_inferences":["Going beyond the paper, the remaining gap between 72% and 100% of ideal is likely HBM bandwidth contention, and a runtime could test this by partitioning HBM channels or traffic classes, which the paper explicitly leaves to future work.","Going beyond the paper, the launch and synchronization cost that makes ConCCL 4x slower than conventional collectives below 32MB suggests that on future GPUs a DMA queue readable directly by the GPU could extend the technique to latency-bound C3, not just bandwidth-bound scenarios.","Going beyond the paper, the same taxonomy and one-time slowdown-table method could be applied to other GPU architectures with different numbers of compute units and different cache topologies, making the runtime heuristics portable beyond MI300X.","Going beyond the paper, if DMA engines gained arithmetic capability, all-reduce could also be offloaded, and the measured 72%-of-ideal plateau suggests that this would be the next practical target for closing the C3 performance gap."],"forward_implications":["If GPU DMA engines are improved, concurrent computation and communication on GPUs can approach the ideal speedup defined by the max of isolated compute and communication times.","Communication libraries can match compute-unit-based collectives on large transfers while freeing all compute units for concurrent GEMMs, as ConCCL does at sizes of 128MB and above.","A runtime can improve C3 performance simply by scheduling the lower-workgroup-count kernel first, without requiring special hardware.","Resource partitioning can be guided by a one-time lookup table of kernel slowdowns, which predicted the optimal compute-unit allocation for 24 of 30 studied C3 scenarios.","All-to-all, which needs more compute units, benefits the most from DMA offload, going from 1.05x to 1.43x speedup over serial execution.","The remaining 28% gap between realized and ideal speedup is attributed to HBM bandwidth contention, which the paper leaves to future work.","ConCCL can be applied to the intra-node phase of hierarchical multi-node collectives, extending its benefit to large-scale training without new hardware.","The launch and synchronization cost that makes ConCCL slower than conventional collectives below 32MB suggests that a GPU-side DMA launch path would extend the technique to latency-bound C3, not just bandwidth-bound scenarios."],"supporting_citations":[{"why":"Documents that concurrent kernels can even cause slowdowns, motivating the need to close the compute-communication overlap gap.","marker":"[5]"},{"why":"RCCL is the compute-unit-based collective library that provides the baseline C3 implementations and the comparison point for ConCCL.","marker":"[14]"},{"why":"The HSA runtime call used by ConCCL to schedule individual DMA-engine transfers.","marker":"[18]"},{"why":"Source of the LLaMA-70B and LLaMA-405B training workloads from which the GEMM and collective sizes are derived.","marker":"[4]"},{"why":"NanoFlow, a prior system also applying resource partitioning to overlapping computation and communication, which the paper compares its simpler heuristic against.","marker":"[3]"},{"why":"FSDP, an algorithmic source of C3 used to derive seven of the studied all-gather scenarios.","marker":"[2]"},{"why":"MSCCL++, which has also offloaded large collectives to DMA engines but does so from GPU threads, in contrast to ConCCL's CPU-orchestrated DMA offload.","marker":"[25]"}],"fun_headline_variants":["DMA offload lifts GPU overlap to 72% of ideal speedup","GPU DMA engines raise overlap speedup to 72% of ideal","Offload communication to DMA for 72% ideal GPU overlap","DMA-based collectives hit 72% ideal speedup on MI300X","C3 on GPUs: DMA offload closes speedup gap to 72%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire ConCCL benefit rests on the architectural assumption that the MI300X's DMA engines transfer data without consuming GPU compute cores or L1/L2 cache, leaving all 304 compute units and the caches fully available to the concurrent GEMM; that property is taken from the chip design rather than measured under concurrent load in this paper.","fun_headline_variants_meta":{"raw":{"variants":["DMA offload lifts GPU overlap to 72% of ideal speedup","GPU DMA engines raise overlap speedup to 72% of ideal","Offload communication to DMA for 72% ideal GPU overlap","DMA-based collectives hit 72% ideal speedup on MI300X","C3 on GPUs: DMA offload closes speedup gap to 72%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000736,"raw_usage":{"total_tokens":3316,"prompt_tokens":996,"completion_tokens":2320,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":612,"completion_tokens_details":{"reasoning_tokens":2221}},"tokens_in":612,"tokens_out":2320,"duration_ms":14839,"temperature":1.0,"reasoning_tokens":2221,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:19:05.653390+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a compute-bound GEMM alone, then run the same GEMM concurrently with a large DMA transfer and with a compute-unit-based collective, measuring both GEMM slowdown and L2 or Infinity Cache traffic with a profiler. If the DMA-concurrent GEMM slows down by more than a negligible amount, or if DMA transfers measurably increase cache access counts, the no-interference premise fails and ConCCL's average 72%-of-ideal figure would not hold on that system.","supporting_citations":[{"cited_title":"{ARK}:{GPU-driven} code execution for distributed deep learning,","cited_arxiv_id":null,"evidence_quote":"Documents that concurrent kernels can even cause slowdowns, motivating the need to close the compute-communication overlap gap."},{"cited_title":"ROCm Communication Collectives Library (RCCL)","cited_arxiv_id":null,"evidence_quote":"RCCL is the compute-unit-based collective library that provides the baseline C3 implementations and the comparison point for ConCCL."},{"cited_title":"ROCm: ROCR-Runtime,","cited_arxiv_id":null,"evidence_quote":"The HSA runtime call used by ConCCL to schedule individual DMA-engine transfers."},{"cited_title":"MSCCL++: A GPU-driven communication stack for scalable AI applications","cited_arxiv_id":null,"evidence_quote":"MSCCL++, which has also offloaded large collectives to DMA engines but does so from GPU threads, in contrast to ConCCL's CPU-orchestrated DMA offload."}],"review_version":1}