Pith. sign in

REVIEW 4 major objections 5 minor 42 references

LLMPrism: Black-box Performance Diagnosis for Production LLM Training Platforms

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LLMPrism claims that switch-level network flow records alone can identify LLM training jobs, infer their parallelism strategies, reconstruct per-GPU timelines to under 0.3% error, and drive production diagnosis.

desk verdict A practical, well-deployed systems idea that deserves a serious taker, but the headline timeline-accuracy claim rests on one job and an assumption that ZeRO-style workloads may violate. read the letter →

arxiv 2505.00342 v1 pith:TVOWIR7L submitted 2025-05-01 cs.SE

Zhihan Jiang , Rui Ren , Guangba Yu , Yulun Wu , Wenwei Gu , Yichen Li , Yujie Huang , Cong Feng
show 3 more authors
Zengyin Yang Yongqiang Yang Michael R. Lyu
This is my paper · ORCID
classification cs.SE
keywords LLMtrainingperformancediagnosisnetworkflowanalysisblack-boxmonitoringparallelismidentificationtimelinereconstructionRoCEproductionplatform
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LLMPrism tries to establish that the communication traffic of distributed LLM training is so regular that switch-level network flows, data many data-center operators already collect for network monitoring, can replace intrusive profilers. The system claims that a platform provider with no access to tenant code or job configuration can identify which GPUs belong to each training job, tell data-parallel from pipeline-parallel communication, and reconstruct each GPU's step-by-step training timeline from flow timings and sizes alone. If this holds, cloud operators gain a continuous, near-zero-overhead way to detect training slowdowns, network congestion, and stragglers without tenant cooperation. The paper reports production deployment with 100% accuracy on job and parallelism identification from one-minute flow windows and timeline reconstruction error below 0.3%.

What carries the argument

The load-bearing machinery is a set of three communication regularities plus one change-point detector. The regularities are spatial stability (ranks only talk inside their own pipeline-parallel or data-parallel groups), temporal periodicity (training steps repeat at stable intervals), and a type contrast (pipeline-parallel traffic is point-to-point with uniform small flows; data-parallel traffic is collective with many varied-size flows). LLMPrism combines these with Bayesian Online Change-point Detection, an online, linear-time method that flags moments where the gap pattern between flows changes, to cut each flow stream into steps. A flow-size mode test labels each pair as data-parallel or pipeline-parallel, and a transitive closure over data-parallel pairs corrects noisy mislabels. Timeline reconstruction then treats the data-parallel burst at each step's end as the boundary marker, producing a per-GPU chronological sequence of communication events; diagnosis applies a k-sigma rule, flagging values more than k standard deviations from the mean, to find outliers in step duration, data-parallel-group duration, and per-switch average bandwidth.

What would settle it

Run a job whose optimizer buckets and overlaps gradient all-reduce throughout the backward pass, or that updates parameters asynchronously, while collecting the same switch-level flows, and compare LLMPrism's inferred per-GPU step boundaries against profiler ground truth. If the inferred boundaries shift by more than the claimed 0.3% error for that workload, the central claim is refuted for that workload class.

Watch

Extended reading notes

Core claim

The paper's central discovery is that an LLM training job writes an implicit schedule into its network traffic. Communication partners are fixed for the job's lifetime; the same step pattern repeats periodically; and the two cross-machine communication modes are separable, with pipeline-parallel sends being small and uniform while data-parallel collectives appear as a burst of variable-size flows at the end of each step. LLMPrism converts these regularities into a reconstruction pipeline: cluster GPUs by who talks to whom, merge clusters that share the same physical machines to cover intra-machine traffic invisible to switches, then use change-point detection on inter-flow gaps to mark step boundaries and flow-size counts to label each communication pair. The output is a per-GPU timeline of pipeline-parallel and data-parallel communication events with the gaps read as computation, and diagnosis is done by flagging steps, data-parallel groups, and switches whose durations or bandwidths deviate beyond a fixed sigma threshold. The paper's evidence is that this reconstruction is accurate enough for day-to-day production fault detection.

Load-bearing premise

The central assumption is that every training step ends with a cleanly separated burst of data-parallel collective traffic on each GPU; if a job overlaps that communication with computation or spreads it through the step with gradient bucketing, the step boundaries inferred from flow gaps drift and the reconstructed timeline loses its meaning.

Editorial extensions

If this is right

  • Platform providers can monitor every tenant LLM job without code changes, configuration access, or tenant cooperation, because the data source is switch mirroring already common in data centers.
  • A one-minute flow window is enough to identify jobs and data-parallel/pipeline-parallel strategies, so diagnosis can begin almost immediately after a job starts or a fault is suspected.
  • Reconstructed timelines convert vague 'job is slow' reports into specific pointers: delayed steps, slow data-parallel groups, or degraded switches.
  • Because monitoring runs outside the training process, it adds near-zero overhead, unlike profiler-based monitoring that measurably slows training.
  • Aggregating data-parallel flow bandwidth by switch localizes network congestion to specific switches, not just to affected jobs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper evaluates only synchronous, step-bounded jobs on its own platform. A natural extension is to test whether the same flow signatures can anchor step boundaries for asynchronous optimizers or fully overlapped, bucketed communication, where the end-of-step data-parallel burst is weakened or absent.
  • Because LLMPrism infers job boundaries, parallelism, and step timing from network data alone, it also exposes tenant behaviour that platform providers were previously barred from seeing; this diagnostic power is also a privacy consideration the paper does not discuss.
  • The reconstruction pipeline could be extended to automatic root-cause correlation: aligning reconstructed step slowdowns with per-switch bandwidth drops could separate job-side stalls from network-side congestion, a layer the paper gestures at but does not fully develop.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. LLMPrism is a black-box monitoring and diagnosis system for LLM training platforms. Using only switch-level RoCE network flow data, it (1) recognizes individual training jobs via connected components of communicating GPU pairs merged with physical topology; (2) classifies communication pairs into DP and PP communication using BOCD step segmentation and a DFS-based transitive refinement; (3) reconstructs per-GPU training timelines by applying BOCD to DP flow intervals, treating end-of-step DP collectives as step boundaries; and (4) performs anomaly detection using k-sigma rules on step durations, DP group durations, and switch-level bandwidth. The evaluation on a production platform (2,880 GPUs, five 1,024-GPU jobs with known parallelism, and one 1,024-GPU validation job) reports 100% accuracy for job and parallelism recognition, timeline reconstruction error below 0.3%, and a switch-degradation case study.

Significance. If the claims hold, the paper contributes a practical and non-intrusive methodology for continuous performance diagnosis in multi-tenant LLM training platforms, which is a real gap in current practice. The strengths are the production-scale deployment, use of external ground truth for timeline reconstruction (PyTorch Profiler) and parallelism identification (tenant configurations), and the honest admission of privacy constraints that motivate the black-box setting. The central limitation is a workload assumption: the step-boundary detection requires each training step to end with a detectable DP collective burst. This assumption is not validated for ZeRO-style bucketing or asynchronous optimizers, and the headline 0.3% error rests on a single validation job. The contribution is valuable but needs stronger validation and more precise scoping.

major comments (4)
  1. [§IV-C and §V-C] The training-timeline reconstruction assumes that each training step ends with a segment of DP collective communication on every rank, and this assumption is load-bearing for the claimed 0.3% error. The paper cites DeepSpeed-ZeRO as a supported optimization, but ZeRO's gradient bucketing spreads all-reduce traffic through the backward pass and asynchronous optimizers can decouple the final all-reduce from the step boundary. Under such overlap, the BOCD change-points in DP flow intervals may split or merge training steps, so the reconstructed per-GPU timeline no longer corresponds to real steps. The only reconstruction validation (Sec. V-C) is one author-created 1,024-GPU job; the five jobs used for parallelism identification (Sec. V-B) are not used to validate reconstruction, and no ZeRO or async-overlap workload is reported. Please either validate on such workloads or explicitly scope the applicability claim.
  2. [§V-C] The reconstruction error metric is undefined. The text reports 'reconstruction error remained within 0.3%' without specifying whether this is a per-event timing error, a per-step duration error, or an end-to-end time error, and without reporting the number of GPUs, steps, and ranks used in the comparison. To be reproducible, define the metric relative to the PyTorch Profiler reference and report the distribution across ranks and steps.
  3. [§V-A and Abstract] The abstract claims '100% accuracy in identifying training jobs and their parallelism strategies,' but the job-recognition evaluation (Sec. V-A) is a single snapshot with manual verification and no reported precision/recall or number of error cases. State explicitly whether the 100% claim applies to job recognition or only to parallelism identification, and support it with quantitative results over multiple windows.
  4. [§IV-B and Table I] Table I reports only average accuracy across five jobs; per-job accuracy, confidence intervals, and the per-job effect of the DFS refinement are absent. Because the 100% accuracy for 1-minute flows relies on the transitive refinement, the paper should show that this refinement does not mask systematic misclassification on specific parallelism configurations (e.g., jobs with small DP groups).
minor comments (5)
  1. [§IV-C] 'disccused' should be 'discussed'.
  2. [§VII] 'generazability' should be 'generalizability'.
  3. [§IV-B, Eq. (1)] The BOCD description is imprecise; the threshold on the change-point probability (0.95) is a design choice and should be stated as a posterior probability, not a likelihood.
  4. [§II-B] The reference to 'Meta ROCET [32]' does not match the cited paper's title; please align the name and citation.
  5. [Figure 2] Figure 2 uses 'Iteration K' while the text uses 'training step'; please unify terminology.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: parallelism classification and timeline reconstruction are checked against external ground truth and an independent profiler; the minor self-citations are peripheral context.

full rationale

LLMPrism's derivation chain is self-contained: its three headline capabilities are each validated against references that are external to the algorithm's own inputs. Job recognition (Alg. 1) is a union-find clustering over observed flow endpoints merged by physical topology, and its accuracy is established by manual verification with internal tenants rather than by reusing the algorithm's output as its own ground truth. Parallelism identification (Alg. 2) applies BOCD to inter-flow intervals, classifies a pair as PP when the modal distinct flow-size count per detected step is 1 and DP otherwise, and refines via DP transitivity; the reported 100% accuracy is measured against ground-truth tenant parallelism configurations, an independent label set. Timeline reconstruction (Sec. IV-C) re-applies BOCD to DP-flow intervals and treats inter-event gaps as compute, and the claimed 0.3% error is measured against PyTorch Profiler traces, an independent reference. All thresholds are stated a priori (BOCD change-point likelihood 0.95 in Sec. IV-B; k = 3 in the k-sigma rule in Sec. IV-D), so no parameter fitted to the evaluation data is later presented as a prediction. The self-citations ([15] Prism, [16] FaultProfit, [20] L4) occur in introductory statements about black-box visibility and training-failure diagnosis and are not used to derive any of the system's load-bearing outputs, so they do not constitute load-bearing self-citation. Finally, the concern that ZeRO-style gradient bucketing or asynchronous optimizers could break the assumed pattern that 'each training step concludes with a segment of DP collective communication traffic' is a genuine robustness and correctness risk under communication-computation overlap, but it is an input modeling assumption of the method, not an equivalence between inputs and outputs, and it is not used to define the evaluation metric.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on a small set of domain assumptions about LLM training communication, namely spatial stability, periodic DP bursts at step end, separability of intra-step and inter-step intervals, and flow-size distinguishability between DP and PP. Three thresholds or modeling choices are hand-set. No new physical entities are introduced.

free parameters (3)
  • BOCD change-point likelihood threshold = 0.95
    Hand-set threshold for reporting a change-point in step boundaries. The paper does not justify why 0.95 is chosen or measure sensitivity to it.
  • BOCD prior and hazard parameters = not specified
    BOCD requires priors over run lengths and observation models; only the 0.95 change-point threshold is given. These unspecified modeling choices affect step division.
  • k in k-sigma anomaly rule = 3
    Hand-set outlier threshold for cross-step, cross-group, and switch-level degradation detection. Sensitivity to this value is not analyzed.
assumptions (4)
  • domain assumption Communications during LLM training are strictly confined to GPUs within the same DP or PP group.
    Used in Sec. IV-A job recognition to merge GPUs into clusters. Assumes no cross-job communication and no other traffic, such as checkpointing or monitoring, that would merge clusters.
  • domain assumption Each training step concludes with a detectable segment of DP collective communication traffic.
    Used in Sec. IV-C timeline reconstruction. If this periodic DP burst is overlapped, split, or absent, step boundaries inferred from DP flows become unreliable.
  • domain assumption Intervals between communication flows within the same training step are significantly shorter than intervals between adjacent steps.
    Used in Sec. IV-B for BOCD step division. Workloads with uneven micro-batch schedules or frequent small gaps may violate this assumption.
  • domain assumption PP communication flows are small and consistent in size per step, while DP communication flows are large and split into multiple flow sizes.
    Used in Sec. IV-B to classify communication type from the mode of distinct flow sizes per step. Packet loss, retransmission, and incomplete flow collection are acknowledged as noise sources.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLMPrism: Black-box Performance Diagnosis for Production LLM Training Platforms." pith.science (2026). https://pith.science/paper/TVOWIR7L

@misc{pith2026250500342,
  author       = {Pith},
  title        = {Pith review of: LLMPrism: Black-box Performance Diagnosis for Production LLM Training Platforms},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TVOWIR7L}},
  note         = {Machine review of arXiv:2505.00342}
}
read the original abstract

Large Language Models (LLMs) have brought about revolutionary changes in diverse fields, rendering LLM training of utmost importance for modern enterprises. To meet this demand, multi-tenant large-scale LLM training platforms have been built to offer LLM training services. Nevertheless, due to the complexity and synchronous nature of LLM training process, performance issues occur frequently and can result in substantial resource wastage. The limited visibility from the perspective of platform providers impedes existing profiling methods and poses challenges to the monitoring and diagnosis of the performance of LLM training jobs. For the first time, this paper proposes the utilization of underlying network flow data to reconstruct the training timelines of jobs based on the distinct characteristics in the LLM training procedure. We design LLMPrism, the first black-box performance diagnosis system for LLM training platforms. By progressively recognizing LLM training jobs, identifying their parallelism strategies, and reconstructing the training timelines, LLMPrism achieves non-intrusive, lightweight, and continuous monitoring of LLM training systems. Leveraging this monitoring capability, it further effectively diagnoses potential performance issues. Since Oct. 2024, LLMPrism has been deployed on our large-scale production Platform-X, in which the evaluations and deployment experiences demonstrate that LLMPrism can achieve accurate timeline reconstruction with an error within 0.3% and effectively diagnose various performance issues.

Figures

Figures reproduced from arXiv: 2505.00342 by the authors.

Figure 1
Figure 1. The training parallelism and network monitoring. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The overall framework of LLMPrism. • Performance Overhead. Profiling tools inevitably introduce additional overhead, negatively affecting the original train￾ing efficiency. For instance, XPUTimer [24] incurs a 5% performance degradation in jobs utilizing 3D parallelism. • Visibility Constraints. Due to tenant privacy concerns, users typically refrain from sharing job configurations and profil￾ing data with platform … view at source ↗
Figure 3
Figure 3. LLM training job recognition procedure of LLMPrism [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: An example of reconstructed training timeline. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

42 extracted references · 29 canonical work pages

  1. [1]

    A comprehensive sur- vey on process-oriented automatic text summarization with exploration of llm-based methods,

    H. Jin, Y . Zhang, D. Meng, J. Wang, and J. Tan, “A comprehensive sur- vey on process-oriented automatic text summarization with exploration of llm-based methods,” arXiv preprint arXiv:2403.02901 , 2024

    arXiv 2024

  2. [2]

    Log-based Anomaly Detection based on EVT Theory with feedback

    J. Liu, J. Huang, Y . Huo, Z. Jiang, J. Gu, Z. Chen, C. Feng, M. Yan, and M. R. Lyu, “Scalable and adaptive log-based anomaly detection with expert in the loop,” arXiv preprint arXiv:2306.05032 , 2023

    work page Pith review arXiv 2023

  3. [3]

    Exploring the effectiveness of llms in automated logging statement generation: An empirical study,

    Y . Li, Y . Huo, Z. Jiang, R. Zhong, P. He, Y . Su, L. C. Briand, and M. R. Lyu, “Exploring the effectiveness of llms in automated logging statement generation: An empirical study,” IEEE Transactions on Software Engineering , 2024

    work page 2024

  4. [4]

    Lilac: Log parsing using llms with adaptive parsing cache,

    Z. Jiang, J. Liu, Z. Chen, Y . Li, J. Huang, Y . Huo, P. He, J. Gu, and M. R. Lyu, “Lilac: Log parsing using llms with adaptive parsing cache,” Proceedings of the ACM on Software Engineering , vol. 1, no. FSE, pp. 137–160, 2024

    work page 2024

  5. [5]

    Go static: Contextualized logging statement generation,

    Y . Li, Y . Huo, R. Zhong, Z. Jiang, J. Liu, J. Huang, J. Gu, P. He, and M. R. Lyu, “Go static: Contextualized logging statement generation,” Proceedings of the ACM on Software Engineering , vol. 1, no. FSE, pp. 609–630, 2024

    work page 2024

  6. [6]

    Coca: Generative root cause analysis for distributed systems with code knowl- edge,

    Y . Li, Y . Wu, J. Liu, Z. Jiang, Z. Chen, G. Yu, and M. Lyu, “Coca: Generative root cause analysis for distributed systems with code knowl- edge,” in 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE Computer Society, 2025, pp. 770–770

    work page 2025

  7. [7]

    Scaling laws for neural language models,

    J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling laws for neural language models,” arXiv preprint arXiv:2001.08361 , 2020

    arXiv 2001

  8. [8]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,

    D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi et al., “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,” arXiv preprint arXiv:2501.12948 , 2025

    arXiv 2025

Show all 42 references
  1. [9]

    Uni- cron: Economizing self-healing llm training at scale,

    T. He, X. Li, Z. Wang, K. Qian, J. Xu, W. Yu, and J. Zhou, “Uni- cron: Economizing self-healing llm training at scale,” arXiv preprint arXiv:2401.00134, 2023

    2023 arXiv

  2. [10]

    Fire-flyer ai-hpc: A cost-effective software-hardware co-design for deep learning,

    W. An, X. Bi, G. Chen, S. Chen, C. Deng, H. Ding, K. Dong, Q. Du, W. Gao, K. Guan et al. , “Fire-flyer ai-hpc: A cost-effective software-hardware co-design for deep learning,” in SC24: International Conference for High Performance Computing, Networking, Storage and Analysis. I...

    2024

  3. [11]

    The llama 3 herd of models,

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan et al. , “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783 , 2024

    2024 arXiv

  4. [12]

    Amazon sagemaker,

    “Amazon sagemaker,” 2024. [Online]. Available: https://aws.amazon. com/sagemaker/

    2024

  5. [13]

    Google vertex ai,

    “Google vertex ai,” 2024. [Online]. Avail- able: https://console.cloud.google.com/vertex-ai?hl=en&inv=1&invt= Abkx0g&project=fine-effect-362306

    2024

  6. [14]

    Evolution of aegis: Fault diagnosis for ai model training cloud service in production (experience track)

    J. Dong, K. Qian, P. Zhang, Z. Zheng, L. Chen, F. Feng, Y . Zhu, G. Lu, Z. Ren, X. Li et al. , “Evolution of aegis: Fault diagnosis for ai model training cloud service in production (experience track).”

  7. [15]

    Prism: Revealing hidden functional clusters from massive instances in cloud systems,

    J. Liu, Z. Jiang, J. Gu, J. Huang, Z. Chen, C. Feng, Z. Yang, Y . Yang, and M. R. Lyu, “Prism: Revealing hidden functional clusters from massive instances in cloud systems,” in 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 2023, pp...

    2023

  8. [16]

    Faultprofit: Hierarchical fault profiling of incident tickets in large-scale cloud systems,

    J. Huang, J. Liu, Z. Chen, Z. Jiang, Y . Li, J. Gu, C. Feng, Z. Yang, Y . Yang, and M. R. Lyu, “Faultprofit: Hierarchical fault profiling of incident tickets in large-scale cloud systems,” in Proceedings of the 46th International Conference on Software Engineering: Software En...

    2024

  9. [17]

    Falcon: Pinpointing and mitigating stragglers for large-scale hybrid-parallel training,

    T. Wu, W. Wang, Y . Yu, S. Yang, W. Wu, Q. Duan, G. Yang, J. Wang, L. Qu, and L. Zhang, “Falcon: Pinpointing and mitigating stragglers for large-scale hybrid-parallel training,” arXiv preprint arXiv:2410.12588 , 2024

    2024 arXiv

  10. [18]

    Megascale: Scaling large language model training to more than 10,000 gpus,

    Z. Jiang, H. Lin, Y . Zhong, Q. Huang, Y . Chen, Z. Zhang, Y . Peng, X. Li, C. Xie, S. Nong et al., “Megascale: Scaling large language model training to more than 10,000 gpus,” in 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24) , 2024, pp. 745–760

    2024

  11. [19]

    Revisiting reliabil- ity in large-scale machine learning research clusters,

    A. Kokolis, M. Kuchnik, J. Hoffman, A. Kumar, P. Malani, F. Ma, Z. DeVito, S. Sengupta, K. Saladi, and C.-J. Wu, “Revisiting reliabil- ity in large-scale machine learning research clusters,” arXiv preprint arXiv:2410.21680, 2024

    arXiv

  12. [20]

    L4: Diagnosing large-scale llm training failures via automated log analysis,

    Z. Jiang, J. Huang, Z. Chen, Y . Li, G. Yu, C. Feng, Y . Yang, Z. Yang, and M. R. Lyu, “L4: Diagnosing large-scale llm training failures via automated log analysis,” arXiv preprint arXiv:2503.20263 , 2025

    2025 arXiv

  13. [21]

    Communication-efficient distributed llm training with adaptive traffic control,

    H. Jin, “Communication-efficient distributed llm training with adaptive traffic control,” in 2024 IEEE International Conference on Big Data (BigData). IEEE, 2024, pp. 8685–8687

    2024

  14. [22]

    Optimus: Ac- celerating large-scale multi-modal llm training by bubble exploitation,

    W. Feng, Y . Chen, S. Wang, Y . Peng, H. Lin, and M. Yu, “Optimus: Ac- celerating large-scale multi-modal llm training by bubble exploitation,” arXiv preprint arXiv:2408.03505 , 2024. 7

    2024 arXiv

  15. [23]

    Accelerating large language model training with in-package optical links for scale-out systems,

    A. Patel, D. Biswas, J. Kundu, Y . Ban, N. Pantano, A. Mallik, J. Ryck- aert, and J. Myers, “Accelerating large language model training with in-package optical links for scale-out systems,” in 2024 IEEE Computer Society Annual Symposium on VLSI (ISVLSI) . IEEE, 2024, pp. 118– 123

    2024

  16. [24]

    Xputimer: Anomaly diagnostics for divergent llm training in gpu clusters of thousand-plus scale,

    W. Cui, J. Zhang, H. Zhao, C. Liu, W. Zhang, J. Sha, Q. Chen, B. He, and M. Guo, “Xputimer: Anomaly diagnostics for divergent llm training in gpu clusters of thousand-plus scale,” arXiv preprint arXiv:2502.05413 , 2025

    2025

  17. [25]

    Holistic trace analysis, pytorch,

    “Holistic trace analysis, pytorch,” 2024. [Online]. Available: https: //pytorch.org/tutorials/beginner/hta intro tutorial.html

    2024

  18. [26]

    Meta strobelight,

    “Meta strobelight,” 2025. [Online]. Available: https://engineering.fb.com/2025/01/21/production-engineering/ strobelight-a-profiling-service-built-on-open-source-technology/

    2025

  19. [27]

    [Online]

    “Erspan,” 2024. [Online]. Available: https://netseccloud.com/ understanding-span-ports

    2024

  20. [28]

    Rdma over converged ethernet: A review,

    G. Kaur and M. Bala, “Rdma over converged ethernet: A review,” International Journal of Advances in Engineering & Technology , vol. 6, no. 4, p. 1890, 2013

    2013

  21. [29]

    Megatron-lm: Training multi-billion parameter language models using model parallelism,

    M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catan- zaro, “Megatron-lm: Training multi-billion parameter language models using model parallelism,” arXiv preprint arXiv:1909.08053 , 2019

    1909 arXiv

  22. [30]

    {CASSINI}:{Network- Aware} job scheduling in machine learning clusters,

    S. Rajasekaran, M. Ghobadi, and A. Akella, “ {CASSINI}:{Network- Aware} job scheduling in machine learning clusters,” in 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24), 2024, pp. 1403–1420

    2024

  23. [31]

    Alpa: Automating inter-and {Intra- Operator} parallelism for distributed deep learning,

    L. Zheng, Z. Li, H. Zhang, Y . Zhuang, Z. Chen, Y . Huang, Y . Wang, Y . Xu, D. Zhuo, E. P. Xing et al., “Alpa: Automating inter-and {Intra- Operator} parallelism for distributed deep learning,” in 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22),...

    2022

  24. [32]

    Rdma over ethernet for distributed training at meta scale,

    A. Gangidi, R. Miao, S. Zheng, S. J. Bondu, G. Goes, H. Morsy, R. Puri, M. Riftadi, A. J. Shetty, J. Yang et al. , “Rdma over ethernet for distributed training at meta scale,” in Proceedings of the ACM SIGCOMM 2024 Conference , 2024, pp. 57–70

    2024

  25. [33]

    Opt: Open pre-trained transformer language models, 2022,

    S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V . Lin et al. , “Opt: Open pre-trained transformer language models, 2022,” URL https://arxiv. org/abs/2205.01068, vol. 3, pp. 19–0, 2023

    2022 arXiv

  26. [34]

    Bayesian online prediction of change points,

    D. Agudelo-Espa ˜na, S. Gomez-Gonzalez, S. Bauer, B. Sch ¨olkopf, and J. Peters, “Bayesian online prediction of change points,” in Conference on uncertainty in artificial intelligence . PMLR, 2020, pp. 320–329

    2020

  27. [35]

    Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters,

    J. Rasley, S. Rajbhandari, O. Ruwase, and Y . He, “Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters,” in Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining , 2020, pp. 3505– 3506

    2020

  28. [36]

    Anomaly detection of hadoop log data using moving average and 3-sigma,

    S. Son, M.-S. Gil, Y .-S. Moon, and H.-S. Won, “Anomaly detection of hadoop log data using moving average and 3-sigma,” KIPS Transactions on Software and Data Engineering , vol. 5, no. 6, pp. 283–288, 2016

    2016

  29. [37]

    The llama family,

    “The llama family,” 2025. [Online]. Available: https://huggingface.co/ meta-llama

    2025

  30. [38]

    Torch profiler,

    “Torch profiler,” 2025. [Online]. Available: https://pytorch.org/docs/ stable/profiler.html

    2025

  31. [39]

    ebpf documentary,

    “ebpf documentary,” 2025. [Online]. Available: https://ebpf.io

    2025

  32. [40]

    Analysis of large-scale multi-tenant gpu clusters for dnn training workloads,

    M. Jeon, S. Venkataraman, A. Phanishayee, J. Qian, W. Xiao, and F. Yang, “Analysis of large-scale multi-tenant gpu clusters for dnn training workloads,” in 2019 USENIX Annual Technical Conference (USENIX ATC 19) , 2019, pp. 947–960

    2019

  33. [41]

    Pytorch,

    “Pytorch,” 2024. [Online]. Available: https://pytorch.org

    2024

  34. [42]

    Deepspeed,

    “Deepspeed,” 2024. [Online]. Available: https://www.deepspeed.ai/ 8

    2024

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.