Pith. sign in

REVIEW 3 major objections 5 minor 50 references

Demystifying the MLPerf Benchmark Suite

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fast GPU links can cut multi-GPU training time by up to 42%.

desk verdict A genuinely useful first characterization of MLPerf v0.5, but the headline claim that NVLink is required rests on a confounded five-platform comparison; the descriptive findings are solid, the causal one needs a controlled experiment. read the letter →

arxiv 1908.09207 v1 pith:RISYQCCM submitted 2019-08-24 cs.LG stat.ML

Snehil Verma , Qinzhe Wu , Bagus Hanindhito , Gunjan Jha , Eugene B. John , Ramesh Radhakrishnan , Lizy K. John This is my paper · ORCID
classification cs.LGstat.ML
keywords MLPerfdeeplearningbenchmarkGPUinterconnectNVLinkmulti-GPUscalingdistributedtrainingrooflinemodelmixedprecision
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper characterizes the MLPerf training benchmark suite to find where real training time goes and what hardware actually matters. It argues that MLPerf, as an application-level suite, occupies a different performance envelope from kernel-level benchmarks like DeepBench and from optimization-focused suites like DAWNBench, and that this diversity is what lets it expose system bottlenecks. The central findings are that GPU interconnect topology is a first-order factor in multi-GPU distributed training, that scaling efficiency varies widely across models, and that a scheduler exploiting this variation can save hours of wall-clock time without new hardware. If the paper is right, system builders should invest in dedicated low-latency GPU interconnects and in scaling-aware job schedulers, while benchmark users should not assume any single model represents all of deep learning.

What carries the argument

The argument is carried by a comparative measurement methodology rather than by a single identity. The paper profiles seven MLPerf v0.5 submissions plus DAWNBench and DeepBench workloads, reduces eight collected workload characteristics (PCIe utilization, GPU utilization, CPU utilization, DDR and HBM2 footprints, flop throughput, memory throughput, and epoch count) with PCA to place benchmarks in a workload space, plots them on a V100 roofline model, measures scaling efficiency from 1 to 8 GPUs, collects PCIe and NVLink bus-utilization counters, and compares five 4-GPU servers whose interconnect topologies differ (NVLink, PCIe switch with GPUDirect P2P, and CPU PCIe ports). The central object is the GPU interconnect topology, treated as the varying system attribute that explains the training-time differences, supported by bus-utilization data that show super-linear growth in PCIe and NVLink traffic as GPUs are added.

What would settle it

Run the seven MLPerf workloads on a single server whose GPUs can be switched between NVLink, PCIe-switch, and CPU-PCIe topologies without changing CPUs, memory, or GPU form factor; if the 11-to-42 percent training-time gaps between topologies shrink to near zero, the claim that a dedicated low-latency interconnect is required fails.

Watch

Extended reading notes

Core claim

The paper's central discovery is that on five 4-GPU platforms that differ mainly in how GPUs are interconnected, training time for the same MLPerf workloads is fastest when a dedicated NVLink is present, slower when a PCIe switch enables GPUDirect peer-to-peer, and slowest when GPUs are attached through CPU PCIe ports. The advantage of NVLink over CPU-side PCIe ranges from roughly 11% for image classification to 42% for translation, with the PCIe-switch platform reaching parity on image classification. The paper also finds that scaling efficiency varies sharply: ResNet-50 in TensorFlow reaches 7.04x speedup on 8 GPUs while Neural Collaborative Filtering reaches only 2.32x, and a search over scheduling permutations saves 2.8 hours of total training time on a 4-GPU system and 4.1 hours on 2 GPUs compared with naive sequential scheduling. These results, together with observations that CPU utilization roughly doubles when the GPU count doubles, support the paper's conclusion that interconnect design and scheduling strategy are primary levers for distributed deep learning performance.

Load-bearing premise

The claim that a dedicated low-latency GPU interconnect is required rests on treating the GPU interconnect as the only meaningful difference between five 4-GPU servers, even though those servers also differ in CPU model, memory configuration, GPU form factor, and PCIe switch presence.

Editorial extensions

If this is right

  • Systems that lack a direct GPU-to-GPU interconnect (NVLink or an equivalent P2P-capable PCIe design) will train translation, detection, and classification models 11 to 42 percent slower than NVLink-equipped peers at the same GPU count.
  • The measured scaling spread (7.04x speedup for ResNet-50 TensorFlow versus 2.32x for NCF on 8 GPUs) means a scheduler can cut total wall time by running poorly scaling models on single GPUs and distributing highly scalable ones; the paper's search saves 2.8 hours on 4 GPUs and 4.1 hours on 2 GPUs versus naive sequential scheduling.
  • Doubling the number of GPUs roughly doubles host CPU utilization for all MLPerf submissions, so server designs need CPU headroom and fast CPU-to-GPU paths to avoid making the host the bottleneck.
  • Mixed-precision training with Tensor Cores delivers 1.5x to 3.3x speedups on MLPerf workloads, and enabling XLA kernel fusion cuts ResNet-50 TensorFlow training time by about 40 percent, both while reaching the same quality target.
  • Because MLPerf benchmarks occupy distinct regions of the PCA workload space and no two are very close, a user can pick a four-benchmark subset whose metric ranges still span most of the full suite's coverage, saving evaluation time.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct extension is that interconnect-aware scheduling should matter more at larger scale: the paper's 8-GPU optimal schedule saves only 0.4 hours, but its own data show super-linear growth in PCIe and NVLink utilization with GPU count, suggesting that bigger clusters would see larger scheduling gains.
  • The five-platform comparison leaves CPU model, memory size, and GPU form factor varying alongside interconnect, so the 11-to-42 percent range is best read as an upper-bound estimate of interconnect impact until a single machine is tested under interchangeable topologies.
  • The differing interconnect sensitivity across models implies communication intensity can be ranked per architecture; a cluster scheduler could co-locate communication-heavy jobs such as NCF and Transformer on NVLink nodes and place more memory-bound image-classification jobs on cheaper PCIe nodes.
  • The scheduling search assumes training times add without interference, but in production, contention on shared PCIe and UPI links could erode the reported savings, so the gain is an upper bound until measured under multi-tenant load.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper reports an empirical characterization of the MLPerf v0.5 training benchmark suite, comparing it with DAWNBench and DeepBench via workload-clustering (PCA/dendrogram), roofline analysis, kernel profiling, and system-level utilization measurements. It also evaluates mixed-precision/Tensor Core speedups, compiler optimization (XLA), scaling efficiency across 1–8 GPUs, a scheduling illustration, and a five-platform comparison of GPU interconnect topologies. The headline claims are that a dedicated low-latency GPU interconnect (NVLink) is required for optimal multi-GPU training, that exploiting scaling differences across benchmarks can save hours in scheduling, and that CPU utilization grows with GPU count.

Significance. If the central claims hold, the paper provides useful, quantified guidance for GPU system design and for scheduling multi-GPU training jobs. Its strengths include a detailed kernel taxonomy (Appendix tables), a clear demonstration of intra-suite diversity through PCA and dendrograms, and explicit measurements of CPU/GPU/memory utilization across a variety of real MLPerf workloads. The scheduling example (Figure 6) offers a concrete, reproducible-style illustration of how scaling heterogeneity can be exploited. However, the headline interconnect claim rests on a comparison of platforms that differ in multiple hardware dimensions simultaneously, and the paper does not present a controlled test that isolates interconnect topology; until that is addressed, the main conclusion is not yet established.

major comments (3)
  1. [Section 5.5 / Figure 7 / Table 4] The claim that NVLink (or a dedicated low-latency interconnect) is required for optimal distributed training is supported only by comparing five 4-GPU platforms that differ in GPU form factor (SXM2 vs PCIe FHFL), CPU model/base frequency, memory configuration (12 vs 24 DIMMs, 16 vs 32 GB), number of CPU sockets/UPI domains, and presence of PCIe switches, as shown in Table 4. For example, T640 uses 'PCIe & UPI 3', C4140 (B) uses a PCIe switch, C4140 (K)/(M) use NVLink, and R940 XA uses UPI with four CPUs. These non-interconnect differences can materially affect training time, yet Figure 7 attributes the observed ordering entirely to interconnect topology. The paper must either present a controlled comparison (e.g., the same server model with and without NVLink, or with PCIe P2P enabled/disabled) or explicitly soften the claim to a correlation rather than a causal requirement.
  2. [Section 5.5 (paragraph beginning 'Two of the five servers')] There is an internal inconsistency about which platform has which interconnect. The text states 'Two of the five servers, C4140 (M) and C4104 (K) include the high-speed proprietary NVLink interconnect,' but Table 4 lists C4140 (M) as NVLink and C4140 (B) as PCIe. A later sentence in the same section says 'The C4140 (M) which uses a PCIe topology...', directly contradicting Table 4. This ambiguity makes the grouping behind Figure 7 unverifiable and must be corrected before the results can be interpreted.
  3. [Table 6 / Section 5.4 vs Figure 7] The reported NVLink utilization for ResNet-50 with 4 GPUs is only 2,867 MB/s, roughly 2% of the available NVLink bandwidth, yet Figure 7 shows an 11% training-time improvement for image classification on NVLink systems. If average bandwidth utilization is this low, a bandwidth-driven explanation for the NVLink benefit is not compelling without additional evidence, such as peak/percentile bus utilization, latency sensitivity, or NCCL communication-pattern analysis. The current data do not connect the utilization numbers to the causal claim in Section 5.5.
minor comments (5)
  1. [Section 3.2] The statement that 'hyperparameters like batch size and learning rate were scaled accordingly' is vague; the authors should state exactly which hyperparameters were changed, from what values to what values, and on which platforms, so that time-to-accuracy comparisons are reproducible and interpretable.
  2. [Figure 6] The caption says 'different color shades correspond to the executions of the 7 different MLPerf workloads,' but no legend is provided in the figure. Add a legend or labels so the reader can identify which workload each color represents.
  3. [Table 5] GNMT Py is listed as an MLPerf workload but is omitted from the scalability table; the paper should state the reason (e.g., time constraints or run failure) so the reader knows the scaling results do not cover all benchmarks.
  4. [Section 4.1] There is a typo: 'while while more diversity exists' should read 'while more diversity exists.' Please proofread the text.
  5. [Table 4 / Section 5.5] The system name 'C4104 (K)' in Section 5.5 appears to be a typo for 'C4140 (K)' as used in Table 4. Please unify the nomenclature.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper reports direct measurements, an exhaustive scheduling search over measured runtimes, and descriptive analyses, with no fitted parameters renamed as predictions.

full rationale

This paper is an empirical characterization study. Each claim is supported by directly measured data: training times (Figure 7, Table 4), utilization counters (Table 6), scaling speedups (Table 5), and kernel profiles (Appendix). The 'optimal scheduling' result is an explicit exhaustive search over measured runtimes, not a fitted model prediction: the text states 'we search through all permutations of scheduling 7 MLPerf benchmarks on multiple GPUs.' The PCA, dendrogram, and roofline analyses are descriptive transformations of the same measured counters, not derived predictions. No load-bearing step invokes a self-citation, a uniqueness theorem, or an ansatz from the authors' prior work. The main interpretive claim about NVLink rests on an observational comparison across five platforms that differ in GPU form factor, CPU and memory configuration, and hyperparameter scaling, so the causal attribution is vulnerable to confounds; that is a correctness or validity limitation, not circularity, because the training-time ordering is reported empirically rather than inferred from the interconnect labels by construction. Therefore no circular step is identified and the score is 0.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

The paper introduces no fitted parameters, new entities, or new theoretical constructs. Its conclusions rest on measurement assumptions and on representative-selection assumptions; the largest burden is the confounded topology comparison.

assumptions (6)
  • domain assumption The Google and NVIDIA GPU submissions to MLPerf v0.5 are representative of the MLPerf suite and of deep learning training workloads.
    Section 3.2 selects these submissions for availability; conclusions are generalized to 'MLPerf benchmarks' throughout the paper.
  • ad hoc to paper Rescaling batch size and learning rate to make runs complete preserves the benchmarks' defining characteristics and quality targets.
    Section 3.2 states that hyperparameters were scaled, but the effect on workload behavior is not validated.
  • ad hoc to paper Training-time differences among the five 4-GPU platforms are caused by GPU interconnect topology, not by other hardware differences.
    Section 5.5 and Figure 7 attribute the observed training-time gaps to interconnect, while the platforms also differ in CPU, memory, and GPU form factor.
  • domain assumption Profiling tools nvprof, dstat, and dmon and the Empirical Roofline Toolkit provide accurate measurements of kernels, utilization, and machine ceilings.
    Section 3.3 and Section 4.2 rely on these tools for all quantitative claims about kernels, utilization, and roofline positions.
  • domain assumption The eight chosen workload characteristics and PCA capture the similarity structure relevant for benchmark selection.
    Section 4.1 selects eight characteristics and uses PCA; the first four principal components cover 88 percent of variance, leaving some structure unexplored.
  • domain assumption Workload runtimes are deterministic enough that a schedule search over single measured times reflects real multi-job behavior.
    Section 4.5 constructs optimal schedules from one set of runtime measurements without modeling interference or variance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Demystifying the MLPerf Benchmark Suite." pith.science (2026). https://pith.science/paper/RISYQCCM

@misc{pith2026190809207,
  author       = {Pith},
  title        = {Pith review of: Demystifying the MLPerf Benchmark Suite},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RISYQCCM}},
  note         = {Machine review of arXiv:1908.09207}
}
read the original abstract

MLPerf, an emerging machine learning benchmark suite strives to cover a broad range of applications of machine learning. We present a study on its characteristics and how the MLPerf benchmarks differ from some of the previous deep learning benchmarks like DAWNBench and DeepBench. We find that application benchmarks such as MLPerf (although rich in kernels) exhibit different features compared to kernel benchmarks such as DeepBench. MLPerf benchmark suite contains a diverse set of models which allows unveiling various bottlenecks in the system. Based on our findings, dedicated low latency interconnect between GPUs in multi-GPU systems is required for optimal distributed deep learning training. We also observe variation in scaling efficiency across the MLPerf models. The variation exhibited by the different models highlight the importance of smart scheduling strategies for multi-GPU training. Another observation is that CPU utilization increases with increase in number of GPUs used for training. Corroborating prior work we also observe and quantify improvements possible by compiler optimizations, mixed-precision training and use of Tensor Cores.

Figures

Figures reproduced from arXiv: 1908.09207 by the authors.

Figure 2
Figure 2. Dendrogram of MLPerf, DAWNBench and Deep [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. V100 roofline model marked with MLPerf bench [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Mixed Precision training (supported by Tensor Cor [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figures from the paper (3 more)
Figure 6
Figure 6. Figure 6: Scheduling a mix of MLPerf workloads on 4 GPUs: [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Training time on 4-GPU systems. Time on systems wit [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Utilization of CPU (left) and GPU (right) for singl [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

50 extracted references · 46 canonical work pages

  1. [1]

    TensorFlow: Large-scale machine learning on heterogeneous systems,

    M. Abadi, A. Agarwal, P . Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y . Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Man´ e, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P . Tucker, V . V a...

    work page 2015

  2. [2]

    Fathom: Reference workloads for modern deep learning methods,

    R. Adolf, S. Rama, B. Reagen, G.-Y . Wei, and D. Brooks, “Fathom: Reference workloads for modern deep learning methods,” 2016

    work page 2016

  3. [3]

    Deepbench: Benchmarking deep learning operations on different hardware,

    Baidu, “Deepbench: Benchmarking deep learning operations on different hardware,” 2017. [Online]. Available: https://github.com/baidu-research/DeepBench

    work page 2017

  4. [4]

    An update to deepbench with a focus on deep learning inference,

    Baidu, “An update to deepbench with a focus on deep learning inference,” 2017. [Online]. Available: https:// github.com/baidu-research/DeepBench

    work page 2017

  5. [5]

    Resnet18 + minor modifications (submission at DAWNBench),

    bkj, “Resnet18 + minor modifications (submission at DAWNBench),” https://github.com/bkj/basenet/ tree/49b2b61e5b9420815c64227c5a10233267c1fb14/ examples, 2018

    work page 2018

  6. [6]

    Findings of the 2017 conference on machine translation (wmt17),

    O. Bojar, R. Chatterjee, C. Federmann, Y . Graham, B. Haddow, S. Huang, M. Huck, P . Koehn, Q. Liu, V . Lo- gacheva, C. Monz, M. Negri, M. Post, R. Rubino, L. Spe- cia, and M. Turchi, “Findings of the 2017 conference on machine translation (wmt17),” in WMT, 2017

    work page 2017

  7. [7]

    Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems,

    T. Chen, M. Li, Y . Li, M. Lin, N. Wang, M. Wang, T. Xiao, B. Xu, C. Zhang, and Z. Zhang, “Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems,” CoRR, vol. abs/1512.01274, 2015. [Online]. Available: http://arxiv.org/abs/1512.01274

    arXiv 2015

  8. [8]

    cudnn: Effi- cient primitives for deep learning,

    S. Chetlur, C. Woolley, P . V andermersch, J. Cohen, J. Tran, B. Catanzaro, and E. Shelhamer, “cudnn: Effi- cient primitives for deep learning,” 2014

    work page 2014

Show all 50 references
  1. [9]

    Dawnbench : An end-to-end deep learning benchmark and competition,

    C. A. Coleman, D. Narayanan, D. Kang, T. Zhao, J. Zhang, L. Nardi, P . Bailis, K. Olukotun, C. R´ e, and M. Zaharia, “Dawnbench : An end-to-end deep learning benchmark and competition,” in NIPS ML Systems W ork- shop, 2017

    2017

  2. [10]

    Imagenet: A large-scale hierarchical image database,

    J. Deng, W . Dong, R. Socher, L. Li, Kai Li, and Li Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE Conference on Computer Vision and Pattern Recognition, June 2009, pp. 248–255

    2009

  3. [11]

    White paper fujitsu server primergy & primequest memory performance of xeon scalable processor(skylake-sp) based systems,

    FUJITSU, “White paper fujitsu server primergy & primequest memory performance of xeon scalable processor(skylake-sp) based systems,” https://sp.ts.fu jitsu. com/dmsp/Publications/public/wp-skylake-memory- performance-ww-en.pdf, 2018

    2018

  4. [12]

    MovieLens,

    GroupLens, “MovieLens,” https://grouplens.org/dat asets/ movielens/20m/, 2016

    2016

  5. [13]

    Deep learning with limited numerical pre- cision,

    S. Gupta, A. Agrawal, K. Gopalakrishnan, and P . Narayanan, “Deep learning with limited numerical pre- cision,” 2015. 12

    2015

  6. [14]

    Ai benchmarks remain immature,

    L. Gwennap, “Ai benchmarks remain immature,” Micro- processor Report, January 28, 2019

    2019

  7. [15]

    The movielens datasets: History and context,

    F. M. Harper and J. A. Konstan, “The movielens datasets: History and context,” ACM Trans. Interact. Intell. Syst. , vol. 5, no. 4, pp. 19:1–19:19, Dec. 2015. [Online]. Available: http://doi.acm.org/10.1145/2827872

    2015 doi

  8. [16]

    Mask r-cnn,

    K. He, G. Gkioxari, P . Dollar, and R. Girshick, “Mask r-cnn,” IEEE Transactions on Pattern Analysis and Ma- chine Intelligence, pp. 1–1, 2018

    2018

  9. [17]

    Deep residual learn- ing for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learn- ing for image recognition,” 2015

    2015

  10. [18]

    Identity mappings in deep residual networks,

    K. He, X. Zhang, S. Ren, and J. Sun, “Identity mappings in deep residual networks,” 2016

    2016

  11. [19]

    Neural collaborative filtering,

    X. He, L. Liao, H. Zhang, L. Nie, X. Hu, and T.-S. Chua, “Neural collaborative filtering,” 2017

    2017

  12. [20]

    Quantized neural networks: Training neu- ral networks with low precision weights and activations,

    I. Hubara, M. Courbariaux, D. Soudry, R. El-Y aniv, and Y . Bengio, “Quantized neural networks: Training neu- ral networks with low precision weights and activations,” 2016

    2016

  13. [21]

    Learning multiple layers of features f rom tiny images,

    A. Krizhevsky, “Learning multiple layers of features f rom tiny images,” https://www.cs.toronto.edu/ ∼ kriz/learning- features-2009-TR.pdf, 2009

    2009

  14. [22]

    Microsoft coco: Common objects in context,

    T.-Y . Lin, M. Maire, S. Belongie, L. Bourdev, R. Gir- shick, J. Hays, P . Perona, D. Ramanan, C. L. Zitnick, and P . Dollr, “Microsoft coco: Common objects in context,” 2014

    2014

  15. [23]

    Ssd: Single shot multibox detector,

    W . Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y . Fu, and A. C. Berg, “Ssd: Single shot multibox detector,” 2015

    2015

  16. [24]

    Mlperf design challenges,

    P . Mattson, “Mlperf design challenges,” in FastPath 2019, ISPASS, 2019

    2019

  17. [25]

    Mixed precision training,

    P . Micikevicius, S. Narang, J. Alben, G. Diamos, E. Else n, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. V enkatesh, and H. Wu, “Mixed precision training,” 2017

    2017

  18. [26]

    Performance characteristics of common transports and buses,

    Microway, “Performance characteristics of common transports and buses,” https://www.microway.com/ knowledge-center-articles/performance-characteristi cs- of-common-transports-buses/, 2019

    2019

  19. [27]

    “MLPerf,” https://mlperf.org/, MLPerf, 2018

    2018

  20. [28]

    Nvidia collective communications library (nccl),

    NVIDIA, “Nvidia collective communications library (nccl),” https://developer.nvidia.com/nccl

  21. [29]

    Nvidia tesla v100 gpu accelerator,

    NVIDIA, “Nvidia tesla v100 gpu accelerator,” https:// images.nvidia.com/content/technologies/volta/pdf/tesla- volta-v100-datasheet-letter-fnl-web.pdf, 2018

    2018

  22. [30]

    Automatic mixed precision (amp),

    NVIDIA, “Automatic mixed precision (amp),” https:// developer.nvidia.com/automatic-mixed-precision, 2019

    2019

  23. [31]

    Nvidia system management in- terface program,

    NVIDIA Corporation, “Nvidia system management in- terface program,” https://developer.download.nvidia.c om/ compute/DCGM/docs/nvidia-smi-367.38.pdf, 2016

    2016

  24. [32]

    Automatic differentiation in pytorch,

    A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Y ang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, “Automatic differentiation in pytorch,” in NIPS-W, 2017

    2017

  25. [33]

    Squad : 100,000+ questions for machine comprehension of text,

    P . Rajpurkar, J. Zhang, K. Lopyrev, and P . Liang, “Squad : 100,000+ questions for machine comprehension of text,” 2016

    2016

  26. [34]

    Mastering the game of go with deep neural networks and tree search,

    D. Silver, A. Huang, C. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser, I. Antonoglou, V . Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis, “Mast...

    2016

  27. [35]

    Mastering chess and shogi by self-play with a general re- inforcement learning algorithm,

    D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, T. Lillicrap, K. Simonyan, and D. Hassabis, “Mastering chess and shogi by self-play with a general re- inforcement learning algorithm,” 2017

    2017

  28. [36]

    Mas- tering the game of go without human knowledge,

    D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglo u, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, Y . Chen, T. Lillicrap, F. Hui, L. Sifre, G. van den Driessche, T. Graepel, and D. Hassabis, “Mas- tering the game of go without human knowledge,” Nature, vol. ...

    2017

  29. [37]

    Minigo: A minimalist Go engine modeled after AlphaGo Zero, built on MuGo,

    “Minigo: A minimalist Go engine modeled after AlphaGo Zero, built on MuGo,” https://github.com/tensorflow/ minigo, tensorflow

  30. [38]

    XLA (accelerated linear algebra),

    “XLA (accelerated linear algebra),” https://www. tensorflow.org/xla/jit, tensorflow

  31. [39]

    Iostat: I/o statistics tool,

    The FreeBSD Project, “Iostat: I/o statistics tool,” https://www.freebsd.org/cgi/man.cgi?query=iostat& manpath=FreeBSD+12.0-RELEASE+and+Ports

  32. [40]

    Netstat: Network status and statistics tool,

    The FreeBSD Project, “Netstat: Network status and statistics tool,” https://www.freebsd.org/cgi/man.cgi ? query=netstat&sektion=1&manpath=FreeBSD+12.0- RELEASE+and+Ports

  33. [41]

    Vmstat: Virtual memory statistics tool,

    The FreeBSD Project, “Vmstat: Virtual memory statistics tool,” https://www.freebsd.org/cgi/man.cgi ? query=vmstat&sektion=8&manpath=FreeBSD+12.0- RELEASE+and+Ports

  34. [42]

    Attention is all you need,

    A. V aswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” 2017

    2017

  35. [43]

    Nvidia gpu utilization plugin for dstat ,

    V . Vryniotis, “Nvidia gpu utilization plugin for dstat ,” https://raw.githubusercontent.com/datumbox/dstat/ master/plugins/dstat nvidia gpu.py, 2017. 13

    2017

  36. [44]

    Dstat: V ersatile resource statistics too l,

    D. Wie¨ ers, “Dstat: V ersatile resource statistics too l,” http://dag.wiee.rs/home-made/dstat/

  37. [45]

    Roofline: An insightful visual performance model for multicore architectures,

    S. Williams, A. Waterman, and D. Patterson, “Roofline: An insightful visual performance model for multicore architectures,” Commun. ACM, vol. 52, no. 4, pp. 65–76, Apr. 2009. [Online]. Available: http://doi.acm.org/10. 1145/1498765.1498785

    2009

  38. [46]

    Google’s neural machine translation system: Bridging the gap between human and machine transla- tion,

    Y . Wu, M. Schuster, Z. Chen, Q. V . Le, M. Norouzi, W . Macherey, M. Krikun, Y . Cao, Q. Gao, K. Macherey, J. Klingner, A. Shah, M. Johnson, X. Liu, L. Kaiser, S. Gouws, Y . Kato, T. Kudo, H. Kazawa, K. Stevens, G. Kurian, N. Patil, W . Wang, C. Y oung, J. Smith, J. Riesa, A....

    2016

  39. [47]

    Berkeley cs roofline toolkit,

    C. Y ang, “Berkeley cs roofline toolkit,” https://bitbu cket. org/berkeleylab/cs-roofline-toolkit

  40. [48]

    DrQA (sub- mission at DAWNBench),

    R. Y ang, Facebook-ParlAI, and B. Koonce, “DrQA (sub- mission at DAWNBench),” https://github.com/hitvoice/ DrQA, 2018

    2018

  41. [49]

    Why machine learning needs benchmarks,

    C. Y oung, “Why machine learning needs benchmarks,” Computer Architecture T oday, ACM SIGARCH, 2018

    2018

  42. [50]

    Tbd: Bench- marking and analyzing deep neural network training,

    H. Zhu, M. Akrout, B. Zheng, A. Pelegris, A. Phan- ishayee, B. Schroeder, and G. Pekhimenko, “Tbd: Bench- marking and analyzing deep neural network training,” 2018. Appendix 14 Table 7: The classification of kernels from 7 benchmarks from MLPerf v0.5.0 submission. Classes are s...

    2018

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.