{"id":"f7987a1b-347c-4b62-a52e-561c3ad8df0d","arxiv_id":"2505.08936","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"ATLAHS converts real application traces into GOAL format and simulates them across multiple backends with under 5% error on validated AI and HPC workloads.","lead":"ATLAHS is an open-source simulator toolchain that turns real execution traces from AI training, HPC, and storage applications into network simulations. It reports prediction errors under five percent on validated AI and HPC workloads and ships with a public trace library.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"AI accuracy claim rests on unmeasured LogGOPS parameters; no sensitivity analysis supports the '<5% error' bound.","rationale":"ATLAHS is an open-source, well-engineered toolchain with public traces and a useful AstraSim comparison. The HPC validations use Netgauge-measured parameters and are credible. The AI validations, however, fix the network cost model from external, unmeasured parameters, and the paper provides no sensitivity analysis. Since MoE 8x70B has only 5.3% non-overlapped computation, the network dominates and predicted time is largely determined by G and L; a 10% error in G propagates nearly proportionally into communication time. The fact that htsim underpredicts in every AI configuration suggests a small systematic bias, which is consistent with a parameter or overlap-model error. This is a correctness/scope concern, not a dismissal of the tool: a calibration or sensitivity experiment would settle it. The reader's CONDITIONAL verdict remains appropriate because the central accuracy claim is defensible for the tested workloads but is not yet robust to parameter uncertainty or the acknowledged static-DAG simplification.","tokens_in":22843,"tokens_out":7296,"duration_ms":74229,"concrete_test":"Rerun the Figure 8 AI validation for the two most communication-dominated configurations (MoE 8x70B and Llama 70B) with G varied over +/-20% (0.032-0.048 ns/B) and L varied over +/-20% (2960-4440 ns), all else fixed, and record predicted iteration times. If any point leaves the +/-5% band around the measured runtime, the '<5% accuracy' claim is not robust to parameter uncertainty. If all points remain inside the band, the concern is settled for these configurations.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central AI validation claim (error under 5% on LLM/MoE workloads, Fig. 8) depends entirely on the LogGOPS parameter set L=3700 ns, o=200 ns, g=5 ns, O=0, G=0.04 ns/B, S=0 chosen in Section 5.2. These values are 'estimated from the benchmarking works' of other GH200 systems, not measured on the Alps cluster where the traces were collected, and no sensitivity analysis is reported. This is load-bearing because these constants are the only network cost model in ATLAHS LGS and are also applied to ATLAHS htsim. In the most communication-dominated validations (MoE 8x70B with 5.3% non-overlapped computation, Llama 70B with 9.9%), a small error in G or L translates almost directly into a several-percent error in predicted iteration time. The observed htsim errors are systematically negative (up to -4.9%), which is exactly the signature of a biased parameter or overlap model. The static-GOAL-DAG limitation acknowledged in Section 7 (missing CUDA cross-stream dependencies) can bias overlap in the same direction, and no experiment isolates this effect from parameter error. The '<5%' claim is therefore a point estimate on six configurations, not a demonstrated robustness property.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents ATLAHS, an application-centric network simulation toolchain that translates real-world traces from AI (NCCL), HPC (MPI), and distributed storage (block I/O) into GOAL-style DAGs and simulates them through multiple backends (LogGOPSim, htsim, NS-3). It validates the toolchain on six LLM/MoE training configurations on the Alps cluster and fifteen HPC configurations on a CSCS testbed, reporting prediction errors mostly within 5% for both ATLAHS LGS and ATLAHS htsim, while also comparing against AstraSim for two configurations and reporting significantly smaller trace files than Chakra. Case studies illustrate congestion-control effects on storage traffic, differences between message-level and packet-level backends, and job-placement effects in a shared cluster.","tokens_in":23063,"tokens_out":4481,"duration_ms":42016,"significance":"ATLAHS is a potentially valuable open-source infrastructure: it unifies diverse workload formats under GOAL, releases a substantial trace dataset, supports multi-job and multi-tenant scenarios, and demonstrates that a message-level backend can match packet-level accuracy in fully provisioned, symmetric topologies. The HPC validation is broad (15 configurations) and uses measured LogGOPS parameters, and the trace-size and simulation-speed comparisons are concrete and reproducible. If the AI-accuracy claim can be supported against parameter uncertainty, this work would be a solid contribution to network-simulation practice for AI/HPC convergence.","major_comments":[{"comment":"The AI-validation accuracy claim depends entirely on LogGOPS parameters (L=3700 ns, o=200 ns, g=5 ns, O=0, G=0.04 ns/B, S=0) that are estimated from external benchmarking works rather than measured on the Alps cluster where the traces were collected. No sensitivity analysis is reported, so the asserted consistent <5% error for AI workloads is a point estimate on six configurations, not a demonstrated robustness property. Since several workloads are communication-dominated (MoE 8x70B has only 5.3% non-overlapped computation; Llama 70B has 9.9%), modest errors in G or L translate almost directly into per-iteration time errors. Please add a sensitivity analysis over plausible parameter ranges (especially G and L) for both ATLAHS backends, or measure the parameters directly on the validation system, and also isolate the effect of the static-GOAL-DAG simplification acknowledged in §7.","section":"§5.2, Fig. 8"},{"comment":"The text states that prediction error 'remains consistently below 5% across all cases and applications,' but Fig. 10 includes a reported error of -5.2% (for the htsim backend on one of the HPC configurations). This contradicts the stated claim. Please correct the text or the figure, or clarify whether the -5.2% value is a typo or an excluded outlier.","section":"Fig. 10, §5.3"},{"comment":"The distributed-storage component is described as a first-class part of the toolchain and appears in the title and abstract, but it is never validated against measured storage-system runtimes or packet-level ground truth. The storage case study compares two congestion-control algorithms using traces generated under assumptions about Azure Direct Drive from public documentation; the results are relative and have no accuracy check. Please add a validation experiment for the storage path, or explicitly scope the accuracy claims in the abstract and conclusion to AI and HPC workloads.","section":"§3.1.3, §6.1"}],"minor_comments":[{"comment":"The abstract claims 'consistently less than 5% error' without noting that the storage component is unvalidated; please qualify the claim to the validated AI and HPC domains.","section":"Abstract / §5.3"},{"comment":"The sentence 'From our testing, the runtime of complex traces is reduced from 10× to100 × the after the improvements' contains a typo and should be reworded to state the speedup range clearly.","section":"§5.1"},{"comment":"Backend naming is inconsistent ('HTSIM' in Fig. 10 caption vs 'htsim' in the text); also, in Fig. 10 the two sets of red percentages are not explicitly labelled in the visualization, making it hard to map errors to backends.","section":"Fig. 8 / Fig. 10"},{"comment":"The claim that ATLAHS 'consistently outperforms AstraSim' in accuracy is based on only two configurations where AstraSim completed; consider phrasing this as 'in the two successful AstraSim comparisons' to match the evidence.","section":"§5.2"},{"comment":"Reference [82] is incomplete; it ends with 'embedded-checkout=true' and appears to be a truncated Bloomberg URL.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"This is a strong systems paper from an established group, and the open-source release plus trace dataset are genuinely valuable. The main risk is the AI-accuracy claim: because the LogGOPS parameters are not measured on the validation system and no sensitivity analysis is provided, the '<5%' statement is not yet fully supported. The HPC validation is the strongest part of the paper. I also note that the paper's scope includes distributed storage, but that component is unvalidated; the authors should either add validation or soften the abstract/title claims. The -5.2% versus 'below 5%' inconsistency in Fig. 10 should be fixed before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a solid engineering contribution: ATLAHS takes the group's earlier GOAL/LogGOPSim work and builds a real toolchain that turns NCCL, MPI, and block-I/O traces into GOAL DAGs, runs them through message-level or packet-level backends, and merges multi-tenant workloads. The released trace collection and the NCCL-to-GOAL decomposition pipeline are new and valuable, and the htsim speedups (10-100x) are a genuine practical win. Credit where due: the HPC validation is broad (15 configs, 5 apps) and holds under 5% error, and the AI validation, while narrower, still covers meaningful LLM and MoE configurations with mostly small errors.\n\nNow the soft spots, in proportion. The stress-test note is fair: the AI LogGOPS parameters (L=3700, o=200, g=5, G=0.04, etc.) are estimated from other GH200 studies, not measured on the Alps cluster, and there is no sensitivity analysis. Since these constants feed both backends, the sub-5% AI claim is really a point estimate on six configurations, not a demonstrated robustness bound. The systematically negative htsim errors (up to -4.9%) could indeed be a bias in the parameter set or the static-GOAL-DAG overlap model, and the paper does not isolate the two. That said, the reader's circularity burden is correctly low: ground truth is measured runtime, the parameters are externally sourced, and no equation is fitted to the validation points. This is not circular, but it is under-determined.\n\nStorage is honestly labeled as a case study, not a validation, and the Direct Drive model is based on public docs with assumptions. Fine for a toolchain paper, but the abstract's 'consistently less than 5% error' overreaches. The AstraSim comparison is partial — only two configurations ran successfully — but the reported speedups and trace-size ratios are concrete and credible.\n\nBottom line: this is a well-executed, open, reproducible systems paper. The central validation claim is defensible for HPC and for the tested AI cases, but the paper should tone down the generality claim and add a sensitivity analysis, ideally measuring the LogGOPS parameters on the actual system. Worth a serious referee — I'd send it out, with a request for that sensitivity analysis. I'd cite it for the toolchain and trace release, not for the AI accuracy bound.","headline":"A genuinely useful open-source simulator toolchain with solid HPC validation, but the AI <5% claim rests on unmeasured LogGOPS parameters and the storage support is unvalidated, so treat the headline accuracy claim as conditional.","tokens_in":23639,"tokens_out":641,"would_cite":true,"duration_ms":7736,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ATLAHS claims that tracing real AI, HPC, and storage applications into GOAL graphs lets a flexible simulator predict runtimes within 5% error.","keywords":["application-centric network simulation","GOAL format","execution tracing","LLM training","MPI workloads","distributed storage","LogGOPSim","congestion control"],"falsifier":"On the same cluster used for tracing, directly measure the interconnect's latency, overhead, and per-byte transfer time and rerun the Llama 7B 128-GPU validation with those measured values; if the predicted iteration time deviates from the measured runtime by more than 5%, the central accuracy claim depends on the borrowed parameters rather than on the toolchain itself.","tokens_in":22641,"feed_emoji":"📡","tokens_out":8539,"duration_ms":77375,"temperature":0.7,"pith_summary":"ATLAHS is an open-source toolchain that traces real applications from AI, HPC, and distributed storage into GOAL directed acyclic graphs of send, receive, and compute tasks, then replays those graphs on message-level or packet-level network simulators. The paper's central claim is that this application-centric approach predicts the runtimes of real workloads—training iterations for Llama and Mixture-of-Experts models and runs of five HPC codes—with errors consistently below 5% across the validated configurations. The paper also claims that ATLAHS is faster than the current AI-focused simulator AstraSim, produces smaller trace files, and supports multi-job and multi-tenant scenarios by merging GOAL graphs. A sympathetic reader would care because simulators that rely on synthetic microbenchmarks can miss congestion behavior that real traces expose, as demonstrated in the paper's congestion-control case study.","feed_headline":"Simulator predicts real AI and HPC runtimes within 5%","feed_subtitle":"Tracing real apps into one DAG format beats a specialized AI simulator in speed and trace size","key_machinery":"The load-bearing mechanism is the GOAL directed acyclic graph, a format in which every workload is a set of send, receive, and calc (computation) vertices with dependency edges, assigned to compute streams to model concurrency. ATLAHS's tracer front ends convert real executions into these graphs: NCCL collectives are decomposed into point-to-point send/receive schedules according to algorithm, channel, and protocol settings; MPI operations are converted through PMPI tracing; block I/O commands are converted through a bpftrace-based tracer. Dummy zero-cost vertices synchronize parallel streams, and merging graphs from multiple jobs models multi-tenancy. The GOAL graphs are then scheduled by ATLAHS and executed on pluggable backends—LogGOPSim for message-level speed and htsim for packet-level fidelity—through a minimal interface of send, recv, calc, and eventOver. This combination is what lets one unified representation span AI, HPC, and storage.","core_discovery":"The core discovery is that a single, compact intermediate representation—the GOAL format, with only send, receive, and computation vertices and edges expressing dependencies—is expressive enough to capture the communication and computation patterns of LLM training, MPI scientific codes, and distributed storage traffic, and to reproduce measured application runtimes within 5% error. The toolchain obtains these graphs by tracing NCCL through a GPU profiler with added annotations, tracing MPI through the PMPI interface, and tracing block I/O through an eBPF-based tracer; it then decomposes NCCL collectives into point-to-point schedules, merges per-GPU graphs into per-node graphs, and replaces intra-node communication with computation. On the two configurations where AstraSim ran successfully, ATLAHS was both more accurate and faster, and its GOAL trace files were consistently smaller than AstraSim's Chakra traces. The paper further shows that the message-level and packet-level backends agree within 1-2% on fully provisioned symmetric networks, while diverging by over 120% when oversubscription causes packet drops that only the packet-level backend can see.","pith_inferences":["If the 5%-error claim is robust, a natural next step is to use ATLAHS itself as a fast screening tool for congestion-control and topology changes before packet-level simulation, since the two backends agree when the network is well provisioned.","The acknowledged static-DAG limitation suggests the approach would under-predict runtime for dynamically scheduled communication, such as fault-tolerant storage protocols or data-dependent GNN training; a testable extension is to add dynamic vertices to GOAL and compare against real runs.","The lack of a sensitivity analysis for the AI LogGOPS parameters leaves open whether the reported accuracy is stable; a direct measurement of those parameters on the target hardware would settle it.","Combining traces from different domains in one graph (as done for Llama and LULESH) could be extended to model shared infrastructure contention more generally, but the paper only demonstrates network-level contention, not memory or cache interference."],"forward_implications":["Application traces, not synthetic microbenchmarks, become the default workload for evaluating network designs, since they expose issues such as Swift's multi-hop congestion weakness that microbenchmarks hide.","Network architects can safely use the fast message-level backend for fully provisioned symmetric topologies, but must switch to a packet-level backend when oversubscription, packet drops, or queue dynamics matter.","Multi-job and multi-tenant performance questions—such as how job placement affects a shared cluster—can be answered by merging GOAL graphs of different applications.","The storage case study implies that congestion control choice (sender-based vs receiver-based) significantly changes storage request completion under oversubscribed topologies.","Because GOAL trace files are several times smaller than Chakra traces, sharing reproducible workload traces at large scale becomes cheaper."],"supporting_citations":[{"why":"Supplies the LogGOPSim message-level simulator and the scheduling core that ATLAHS extends.","marker":"[39]"},{"why":"Defines the GOAL format, the unified DAG representation at the center of the toolchain.","marker":"[40]"},{"why":"AstraSim 2.0, the baseline AI simulator ATLAHS is compared against for accuracy, runtime, and trace size.","marker":"[84]"},{"why":"Prior validation that the GOAL abstraction is sufficient to model and emulate network communication accurately.","marker":"[72]"},{"why":"GPU-to-GPU interconnect measurements used to estimate the AI LogGOPS parameters.","marker":"[22]"},{"why":"Grace Hopper data-movement benchmarking used to estimate the AI LogGOPS parameters.","marker":"[30]"},{"why":"Provides the measurement method used to obtain the HPC LogGOPS parameters.","marker":"[37]"},{"why":"MPRDMA congestion control used by the htsim backend in validation and case studies.","marker":"[58]"},{"why":"Describes how Chakra traces were produced from PyTorch and Kineto to ensure identical execution patterns in the AstraSim comparison.","marker":"[66]"},{"why":"Storage workload traces used in the distributed-storage congestion-control case study.","marker":"[79]"}],"fun_headline_variants":["One DAG to simulate AI, HPC, and storage within 5% error","ATLAHS: traces real apps, simulates networks with <5% error","Trace once, simulate anywhere: AI, HPC, storage within 5%","Beat AstraSim: faster sims, smaller traces, 5% error bound"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The AI workload predictions rest on network-cost parameters (latency, overhead, per-byte transfer time) borrowed from benchmarks of similar hardware rather than measured on the target cluster, and the paper does not show how much the predictions would change if those parameters were different.","fun_headline_variants_meta":{"raw":{"variants":["One DAG to simulate AI, HPC, and storage within 5% error","ATLAHS: traces real apps, simulates networks with <5% error","Trace once, simulate anywhere: AI, HPC, storage within 5%","Beat AstraSim: faster sims, smaller traces, 5% error bound"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000563,"raw_usage":{"total_tokens":2680,"prompt_tokens":964,"completion_tokens":1716,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":580,"completion_tokens_details":{"reasoning_tokens":1628}},"tokens_in":580,"tokens_out":1716,"duration_ms":10909,"temperature":1.0,"reasoning_tokens":1628,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:44:11.382413+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On the same cluster used for tracing, directly measure the interconnect's latency, overhead, and per-byte transfer time and rerun the Llama 7B 128-GPU validation with those measured values; if the predicted iteration time deviates from the measured runtime by more than 5%, the central accuracy claim depends on the borrowed parameters rather than on the toolchain itself.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the GOAL format, the unified DAG representation at the center of the toolchain."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MPRDMA congestion control used by the htsim backend in validation and case studies."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Describes how Chakra traces were produced from PyTorch and Kineto to ensure identical execution patterns in the AstraSim comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Storage workload traces used in the distributed-storage congestion-control case study."}],"review_version":1}