{"id":"6179ce31-adf4-4439-a559-5c927ebc9892","arxiv_id":"2504.20854","paper_version":1,"verdict":"UNVERDICTED","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"GENIE is a proposed framework that replaces GPUs with CPU-initiated traffic and an adapted ASTRA-sim simulator to test how real network hardware impacts ML workload performance.","lead":"This paper proposes GENIE, a framework that uses CPU-generated network traffic to test how real network hardware affects machine learning training performance, removing the need for expensive GPUs. It offers network operators and researchers a way to detect and diagnose infrastructure issues before committing GPU clusters to a training job.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CPU-to-GPU traffic fidelity is the load-bearing assumption, and the paper defers all validation of it; without such validation, the central claim that GENIE captures real network impact on ML workloads is unsubstantiated.","rationale":"The reader's UNVERDICTED verdict is appropriate. This paper is explicitly a vision and design proposal: no implementation, evaluation, or data are presented, and the central claim is conditional on an unvalidated fidelity assumption that the authors themselves flag as future work. The most load-bearing assumption is indeed that CPU-initiated traffic faithfully emulates GPU-originated GPUDirect RDMA at the network level, including burstiness, flow concurrency, and switch buffer interactions. The paper provides no evidence for this and even lists design choices that could plausibly violate it. My only addition is that the end-to-end claim also depends on ASTRA-sim's modeling of workload dynamics, but this is secondary to the traffic-fidelity issue. This is absence of evidence, not evidence of failure, so the verdict should remain UNVERDICTED rather than moving to ACCEPT or REJECT.","tokens_in":7798,"tokens_out":6279,"duration_ms":69836,"concrete_test":"Build a minimal GENIE prototype on a CPU-only testbed (8–16 nodes) and run the same Chakra workload graph with real NCCL on GPUs and with GENIE on CPUs through the same switch. Instrument switch queue depths and NIC packet captures; compare per-message completion times, queue-occupancy distributions, and end-to-end workload times under three conditions: healthy fabric, background congestion, and a configured NIC fault. If GENIE's traffic lacks the burstiness/concurrency of GPUDirect traffic (e.g., queue-occupancy distributions or degradation factors differ by more than 20%), the fidelity assumption fails; if they match, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that CPU-injected RDMA traffic reproduce the network-relevant properties of GPUDirect RDMA traffic well enough that workload-level slowdowns (buffer pressure, congestion, incast, failures) are preserved. Nothing in the paper establishes this. Section 4 states the requirement ('A key requirement of GENIE is to generate GPU traffic with CPUs') and lists implementation choices (libibverbs, Gloo, perftest), but Section 5 concedes that 'carefully study and validate the fidelity of the GPU emulation' is future work. There are concrete mechanisms by which the proxy can break: GPUDirect RDMA streams from GPU memory through the PCIe switch, so the NIC's DMA pattern, outstanding-descriptor depth, and packet pacing differ from host-memory RDMA; NCCL's collective algorithms use multiple QPs and fine-grained chunk pipelining that a CPU-side generator driven by ASTRA-sim messages may not reproduce without explicit calibration. In addition, GENIE's reported 'workload performance' is ASTRA-sim's simulated time, so even a faithful traffic trace would not by itself validate the workload-level prediction. Figure 1 validates an unrelated AllReduce simulator gap, not GENIE. The claim is therefore an untested design hypothesis, not a demonstrated result.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes GENIE, a framework for testing real network infrastructure for large-scale ML training without using GPUs. The design combines a distributed, modified ASTRA-sim as a 'workload feeder' that walks Chakra execution graphs and emulates compute/memory delays, with a 'traffic generator' that uses CPU-side RDMA libraries (libibverbs, libfabric, Gloo, perftest) to inject inter-node messages over the real network fabric. The paper motivates the need with a measurement showing that a packet-level simulator does not capture an unmodeled NIC degradation during repeated AllReduce calls (Figure 1), and it sketches GENIE's architecture and modular transport support. The authors state in Section 5 that the system has not yet been built and that validation of the CPU-to-GPU traffic fidelity is future work.","tokens_in":7989,"tokens_out":11566,"duration_ms":116261,"significance":"If the core assumption were validated, GENIE would be a valuable tool for network vendors and operators, lowering the cost of testing ML network infrastructure. The paper is clearly written, identifies a real problem, and proposes a modular design that leverages existing artifacts (Chakra, ASTRA-sim, Gloo) rather than building from scratch. It also makes a falsifiable hypothesis: CPU-generated traffic with the same message sizes and logical timing should yield the same network-induced workload delays as GPU-generated traffic. The explicit acknowledgement of the open fidelity question in Section 5 is honest. However, the manuscript presents no implementation, no traffic-fidelity comparison, and no end-to-end validation, so the central claim remains a design hypothesis.","major_comments":[{"comment":"The central claim of the paper, that GENIE 'captures the impact of real hardware network behavior on ML workload performance' (Abstract), rests entirely on the unvalidated assumption that CPU-initiated RDMA traffic reproduces the network-relevant properties of GPUDirect RDMA traffic. Section 4 states this as a key requirement and proposes using libibverbs, libfabric, Gloo, or perftest, but Section 5 explicitly lists 'carefully study and validate the fidelity of the GPU emulation' as future work. No measurement compares CPU-generated traffic with GPU-generated traffic in terms of burstiness, flow concurrency, queue depth, or switch buffer occupancy. Without such a comparison, the framework may miss the microburst and buffer-pressure effects that determine workload-level slowdowns, so the abstract's claim is unsupported.","section":"§4 ('Creating GPU Communication with Traffic Generator') and §5"},{"comment":"The only experimental evidence in the paper, Figure 1, demonstrates that a packet-level simulator fails to capture an unmodeled NIC degradation during repeated AllReduce runs. This is motivational evidence for real-hardware testing, but it does not exercise GENIE in any way; therefore it cannot support the claim that GENIE captures network impact on workloads. The figure's relevance to the proposed design would need to be rebuilt around a comparison of GENIE's predictions against ground-truth GPU training runs.","section":"§1 and Figure 1"},{"comment":"GENIE's workload-level performance is a hybrid of ASTRA-sim's simulated compute/memory durations and real network message completion times, but the paper does not validate this hybrid in the distributed, real-time setting. The graph reader progresses only when ASTRA-sim's collective library reports message completion, so the accuracy of the reported workload time depends on the fidelity of ASTRA-sim's compute/memory model, the correctness of the distributed simulation clock, and the synchronization across nodes. None of these are evaluated or even described in detail. The paper should either restrict its claim to network-level fidelity or provide an end-to-end comparison against a real GPU training run on the same workload.","section":"§4 ('Modeling Workload with ASTRA-sim')"},{"comment":"The paper assumes that ASTRA-sim's collective library produces the same message-level schedule as a real collective runtime such as NCCL. If the simulated chunking, pipeline depth, or collective algorithm differs from the production runtime, the resulting network traffic, and hence the measured network impact, will not match a real training job. No evidence is presented that the collective library's message decomposition matches NCCL's behavior on the target hardware, so this is a second load-bearing fidelity assumption that needs direct validation.","section":"§4 ('Modeling Workload with ASTRA-sim')"}],"minor_comments":[{"comment":"The figure legend and caption are unclear: the y-axis is labeled 'Bus BW (GB/s)', the caption says 'We inject an anomaly', and the text says 'unpredicted NIC degradation'; please clarify what was injected and what the simulated curve represents.","section":"Figure 1"},{"comment":"The phrase 'application agonistic manner' should read 'application-agnostic manner'.","section":"§4 ('Creating GPU Communication with Traffic Generator')"},{"comment":"Reference [23] lists 'OpenFabrics Allance'; this should be 'OpenFabrics Alliance'.","section":"References"},{"comment":"The Abstract uses present tense ('GENIE captures') while Section 5 states that the system has not been built; rephrase the Abstract to describe a design goal or a hypothesis, or provide the validation that would support the present-tense claim.","section":"Abstract and Section 5"},{"comment":"The footnote dismisses intra-node communication, but many distributed training workloads use a mix of NVLink and network traffic; a sentence on how this scope decision affects the realism of the emulated inter-node traffic would help.","section":"Footnote 1"},{"comment":"The claim that perftest 'can generate arbitrary point-to-point messages' is inaccurate: perftest is a fixed-pattern bandwidth/latency benchmark tool, not a library for arbitrary message patterns; please clarify the intended use.","section":"§4 ('Creating GPU Communication with Traffic Generator')"},{"comment":"The workload performance metric is not precisely defined; please state whether it is wall-clock time per iteration, end-to-end training time for a fixed number of steps, or another quantity, and explain how it is read out from the distributed GENIE instances.","section":"§4 ('Modeling Workload with ASTRA-sim')"}],"recommendation":"major_revision","confidential_remarks":"This manuscript reads like a workshop-style position paper: it is short, contains no implementation or evaluation, and explicitly defers both building and validating the system to future work. The underlying idea is plausible and the architecture is clearly described, but the abstract and Section 4 make present-tense claims that the paper does not support. If the journal expects full research papers with empirical validation, this manuscript is below the bar and should be rejected; I have chosen major_revision because the central hypothesis is defensible and the missing validation is, in principle, obtainable. The authors should be asked to either reframe the paper as a design/vision statement with hedged claims, or, preferably, add a proof-of-concept fidelity study comparing CPU-generated and GPU-generated traffic on a small testbed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a clear, honest vision paper for GENIE, a way to test network infrastructure for ML workloads without GPUs by driving CPU RDMA traffic from an ASTRA-sim workload model. It does not yet verify the central claim that GENIE captures real network impact on ML workload performance. The paper itself says validation is future work, so the reader's UNVERDICTED verdict is fair.\n\nWhat is genuinely new: combining Chakra's workload graphs, ASTRA-sim's event-driven simulation, and real CPU-injected RDMA traffic over actual switches is not in the cited literature. The paper also does a good job of motivating the problem: the AllReduce measurement in Figure 1 shows a real simulator gap, and the arguments about proprietary networks and hardware misconfigurations are reasonable. The design is modular and sensible.\n\nThe soft spots are significant but not hidden. The load-bearing assumption is that CPU-originated RDMA traffic can reproduce the network-relevant properties of GPUDirect RDMA traffic—burstiness, QP concurrency, NIC DMA behavior, switch buffer pressure. Nothing in the paper establishes that. Section 4 lists implementation choices but Section 5 explicitly defers fidelity validation to future work. Also, GENIE's reported workload performance is ASTRA-sim's simulated time; even a perfect traffic trace would not by itself validate that the workload-level prediction is correct. So the abstract's claim that GENIE 'captures the impact of real hardware network behavior on ML workload performance' is currently an untested design hypothesis, not a demonstrated result. The authors are transparent about this, which is to their credit.\n\nWho gets value: researchers and practitioners thinking about network validation for ML clusters, especially those who want a low-cost alternative to GPU testbeds. The paper is a useful position statement and a possible starting point for a real system. But as a contribution to a top-tier systems or networking venue, it needs at least a proof-of-concept evaluation of the CPU-to-GPU fidelity question and some end-to-end comparison against a real GPU run.\n\nMy recommendation: do not send this to full peer review as a completed systems paper. Send it to a workshop or a vision track if one exists. If the authors return with even a small-scale validation of the fidelity claim, it would deserve serious referee time.","headline":"Honest vision paper for GPU-free network testing, but the central fidelity claim is deferred and unvalidated.","tokens_in":8563,"tokens_out":2580,"would_cite":false,"duration_ms":25410,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By replacing GPUs with CPU nodes that inject real network traffic while a workload simulator decides when, GENIE aims to make large-scale ML network testing cheap enough to run anywhere.","keywords":["network infrastructure testing","distributed ML training","GPU-free emulation","RDMA traffic generation","collective communication","workload graph simulation","hardware testbed","network fault diagnosis"],"falsifier":"Run one representative distributed training workload (for example, an AllReduce-heavy LLM or an all-to-all MoE layer) on a GPU cluster and on a CPU-only GENIE testbed with the same topology, NICs, and switches; induce the same network anomaly (a degraded link or disabled congestion control) and compare per-collective completion times, switch queue depths, and packet-drop counts. If the CPU emulation misses the microbursts or flow concurrency of GPU traffic, the slowdowns will diverge, and the fidelity premise fails.","tokens_in":7575,"feed_emoji":"🌐","tokens_out":6600,"duration_ms":64589,"temperature":0.7,"pith_summary":"GENIE is a proposed testing framework for the network infrastructure of large-scale ML training clusters that does not require GPUs. Its core claim is that CPU nodes can generate the real RDMA traffic patterns of GPU-to-GPU communication, while a distributed simulation of the workload graph decides when each message is sent, so that genuine hardware behavior—congestion, switch buffer pressure, misconfiguration, or NIC degradation—shows up as measurable slowdowns in a full training workload. This matters because GPU clusters are too costly to reserve for network validation, and simulators cannot model proprietary or faulty hardware. The paper lays out the design and positions fidelity validation as the next step; no end-to-end measurements are presented yet.","feed_headline":"CPU-only testbed exposes how network faults slow ML training","feed_subtitle":"By injecting real RDMA traffic from CPU nodes under a workload simulator's control, GENIE aims to validate networks without GPUs.","key_machinery":"The carrying mechanism is a distributed, network-coupled version of the ASTRA-sim workload simulator serving as the \"workload feeder,\" paired with a \"traffic generator\" acting as its network backend. The workload feeder reads a graph-based representation of the training job (operators as vertices, dependencies as edges), issues compute operations as timed sleeps, and asks the generator to send or receive each collective's constituent messages. The traffic generator translates those requests into real network calls through lower-level transport libraries, so the CPU NIC becomes the stand-in for GPU memory and GPUDirect RDMA. Message completion reported back to the simulator preserves the dependency-driven dynamics, making the real network the arbiter of workload progress.","core_discovery":"The paper's central claim is that GENIE reproduces the interaction between an ML training workload and a real network fabric using only CPU nodes. Each GENIE instance plays the role of one training rank: it walks the operator graph of the workload, sleeps through compute durations, decomposes each collective into point-to-point messages, and injects those messages into the physical network through the NIC using standard transport libraries. Completion of each message is fed back into the simulator, so the dependency chain of the workload—and therefore its critical path—advances only as fast as the real network allows. Because switches cannot tell GENIE traffic from GPU training traffic, the real fabric's buffer behavior, congestion control, and failure modes become part of the experiment. The authors state this as a foundation to be built and validated, not as a measured result.","pith_inferences":["If CPU-emulated traffic reproduces GPU traffic's burstiness, the same testbed could be reused as a cheap fault-injection and regression platform: replay a workload graph while intentionally toggling link rates, buffer sizes, or routing policies, and measure which network changes move the critical path.","The fidelity question is testable directly: compare GENIE's per-collective completion times and switch queue occupancy against a GPU ground truth under induced incast or congestion; divergence would pinpoint exactly which traffic properties (microbursts, flow concurrency, PCIe pacing) the CPU emulation misses.","Because compute is only simulated as delays, GENIE could in principle scale testing to workloads far larger than any available GPU cluster, limited by CPU count and network ports rather than accelerator availability.","The dependency-driven message injection could be applied to debugging in reverse: given a production slowdown, replay the workload graph on the suspect fabric and bisect which network component accounts for the added latency."],"forward_implications":["Operators can validate a physical network fabric—including proprietary transports like Slingshot, Infiniband, or Spectrum-X—before committing GPU time to a training run.","Misconfigurations and hardware degradations that simulators cannot predict, such as a disabled congestion-control setting or a degraded NIC, become visible as workload-level slowdowns rather than only as low-level counters.","Workload-level effects of collective algorithms, topology choices, and switch buffer behaviors can be studied on real hardware at far lower cost than building a GPU cluster.","Because GENIE instances coordinate only through the collectives themselves, no extra synchronization layer is imposed on the testbed, preserving the natural timing of the workload."],"supporting_citations":[{"why":"Supplies the distributed workload simulator that GENIE adapts into its workload feeder.","marker":"[36]"},{"why":"Provides the graph-based workload representation GENIE consumes to issue operators and dependencies.","marker":"[32]"},{"why":"Offers CPU-only point-to-point collective messaging usable as a traffic-generation backend.","marker":"[7]"},{"why":"Gives the low-level RDMA verbs interface for injecting real network calls from the CPU.","marker":"[29]"},{"why":"Provides an alternative transport library for generating network traffic in the testbed.","marker":"[23]"},{"why":"Supplies a CPU-only RDMA benchmarking tool the paper proposes as a traffic generator option.","marker":"[24]"},{"why":"Defines the GPU-requiring single-collective benchmark GENIE aims to go beyond for workload-level testing.","marker":"[20]"},{"why":"Demonstrates the simulator-only approach to RDMA congestion control that GENIE replaces with real hardware.","marker":"[14]"}],"fun_headline_variants":["CPU-only testbed mimics GPU traffic for realistic ML network testing","Genie runs ML workloads on CPU nodes to expose real network delays","No GPUs: CPU emulation of GPU traffic tests network impact on ML","Framework simulates GPU communication with CPUs for network validation","Test ML networking without GPUs using Genie's CPU-driven emulation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"CPU-initiated network messages with the same sizes and timing reproduce the network-level behavior of real GPU-originated traffic closely enough that congestion, buffer pressure, and fault-induced slowdowns still appear in workload performance.","fun_headline_variants_meta":{"raw":{"variants":["CPU-only testbed mimics GPU traffic for realistic ML network testing","Genie runs ML workloads on CPU nodes to expose real network delays","No GPUs: CPU emulation of GPU traffic tests network impact on ML","Framework simulates GPU communication with CPUs for network validation","Test ML networking without GPUs using Genie's CPU-driven emulation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00027,"raw_usage":{"total_tokens":1529,"prompt_tokens":752,"completion_tokens":777,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":368,"completion_tokens_details":{"reasoning_tokens":687}},"tokens_in":368,"tokens_out":777,"duration_ms":7455,"temperature":1.0,"reasoning_tokens":687,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:16:38.297614+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run one representative distributed training workload (for example, an AllReduce-heavy LLM or an all-to-all MoE layer) on a GPU cluster and on a CPU-only GENIE testbed with the same topology, NICs, and switches; induce the same network anomaly (a degraded link or disabled congestion control) and compare per-collective completion times, switch queue depths, and packet-drop counts. If the CPU emulation misses the microbursts or flow concurrency of GPU traffic, the slowdowns will diverge, and the fidelity premise fails.","supporting_citations":[{"cited_title":"ASTRA-sim2.0: Modeling Hierarchical Networks and Disaggregated Systems for Large-model Training at Scale","cited_arxiv_id":null,"evidence_quote":"Supplies the distributed workload simulator that GENIE adapts into its workload feeder."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Offers CPU-only point-to-point collective messaging usable as a traffic-generation backend."},{"cited_title":"libibverbs","cited_arxiv_id":null,"evidence_quote":"Gives the low-level RDMA verbs interface for injecting real network calls from the CPU."},{"cited_title":"libfabric","cited_arxiv_id":null,"evidence_quote":"Provides an alternative transport library for generating network traffic in the testbed."},{"cited_title":"Perftest","cited_arxiv_id":null,"evidence_quote":"Supplies a CPU-only RDMA benchmarking tool the paper proposes as a traffic generator option."},{"cited_title":"NCCL Tests","cited_arxiv_id":null,"evidence_quote":"Defines the GPU-requiring single-collective benchmark GENIE aims to go beyond for workload-level testing."},{"cited_title":"Impact of RoCE Congestion Control Policies on Dis- tributed Training of DNNs","cited_arxiv_id":null,"evidence_quote":"Demonstrates the simulator-only approach to RDMA congestion control that GENIE replaces with real hardware."}],"review_version":1}