Pith. sign in

REVIEW 4 major objections 4 minor 70 references

AIBench: An Industry Standard Internet Service AI Benchmark Suite

T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read AIBench claims the first end-to-end Internet-service AI benchmark, built around an e-commerce search critical path.

desk verdict AIBench is a substantial and genuinely useful benchmark framework, but the 'industry standard' and production-representativeness claims need better evidence before the end-to-end numbers are taken at face value. read the letter →

arxiv 1908.08998 v2 pith:R3RMVA74 submitted 2019-08-13 cs.CV cs.AIcs.IRcs.PFcs.SE

Wanling Gao , Fei Tang , Lei Wang , Jianfeng Zhan , Chunxin Lan , Chunjie Luo , Yunyou Huang , Chen Zheng
show 17 more authors
Jiahui Dai Zheng Cao Daoyi Zheng Haoning Tang Kunlin Zhan Biao Wang Defei Kong Tong Wu Minghe Yu Chongkang Tan Huan Li Xinhui Tian Yatao Li Junchao Shao Zhenyu Wang Xiaoyu Wang Hainan Ye
This is my paper · ORCID
classification cs.CVcs.AIcs.IRcs.PFcs.SE
keywords AIbenchmarksuiteInternetservicesend-to-ende-commercesearchworkloadcharacterizationlatencyanalysisGPUstallscomponentbenchmarks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper presents AIBench as the first industry-standard benchmark suite for AI inside Internet services. It argues that prior AI benchmarks offer only isolated micro or component benchmarks, so they miss the critical path of a real service, where AI inference and training are interleaved with search, ranking, databases, and user requests. To fill that gap, AIBench provides a modular framework with sixteen component benchmarks and twelve micro benchmarks, plus an end-to-end e-commerce search benchmark built from anonymized data and workloads abstracted from a top e-commerce provider. The paper's measurements indicate that AI components dominate online latency, especially the recommender, and that offline training time concentrates in six kernel types. A sympathetic reader would take the contribution as a public, reproducible substrate for studying AI systems at the scale and shape of an actual Internet service.

What carries the argument

The central mechanism is the AIBench framework's module decomposition and its reuse in the end-to-end application benchmark. The framework separates data input, AI problem domains, online inference, offline training, and deployment tools into loosely coupled modules, letting each component run alone or in composition. The end-to-end benchmark chains the online server's four submodules (search planer, recommender, searcher, ranker) with the offline analyzer's trainer and indexer, so the measured execution path includes both AI inference and the non-AI infrastructure (databases, inverted indexes, serving) that real services interleave with AI. The data generators and popularity-based index sharding are the load-bearing pieces that let the benchmark scale and reproduce workload characteristics without proprietary data.

What would settle it

If an independent deployment of the public AIBench code at the documented 16-node scale produced a latency breakdown where the searcher rather than the recommender dominated, or where the 253 MB model did not sharply raise p99 latency, the paper's critical-path and bottleneck claims would be refuted. A more direct falsifier would be the partner's production telemetry, if released, showing that AI inference is not on the dominant latency path of real e-commerce search.

Watch

Extended reading notes

Core claim

The central claim is that AIBench is the first end-to-end Internet service AI benchmark suite, and that an end-to-end view changes what benchmarking reveals. Built on a framework of loosely coupled modules, the suite implements sixteen component benchmarks covering AI problem domains such as classification, recommendation, object detection, speech recognition, and learning to rank, plus twelve micro benchmarks for primitive operations like convolution and batch normalization. The end-to-end application benchmark models an e-commerce search service: a search planer receives queries, a recommender performs category prediction and personalized recommendation, a searcher retrieves products from sharded indexes, and a ranker scores results with a learning-to-rank network, while an offline analyzer trains models and builds indexes. On a 16-node CPU/GPU cluster, the paper reports that AI-related modules change the online critical path and raise tail latency, with the recommender contributing the largest latency share, and that offline training is dominated by six kernel types, namely convolution, GEMM, batch normalization, ReLU, element-wise operations, and gradient computation, matching the suite's micro benchmarks. The authors would state the discovery as: benchmarking an Internet service with AI in its critical path requires an end-to-end application workload, and AIBench is the first public instance of one.

Load-bearing premise

The benchmark's e-commerce workload faithfully reproduces the critical path, data distribution, and traffic pattern of a real industry-scale Internet service, even though the paper discloses no public fidelity validation and its footnote 2 admits the baseline latencies are far higher than the partner's production numbers.

Editorial extensions

If this is right

  • AIBench gives the systems community a public, configurable e-commerce AI workload whose data generators can be scaled, so latency and throughput numbers can be reproduced and compared across cluster sizes.
  • The latency breakdown, with the recommender at about 75.7 ms average versus searcher and ranker both under 4 ms, identifies online inference and user-database access, not search itself, as the bottleneck to attack in an AI-augmented service.
  • The reported jump in p99 TensorFlow serving latency from about 149 ms to 5335 ms when the model grows from 184 MB to 253 MB quantifies the service-quality versus model-complexity trade-off architects must make.
  • Offline training's six hotspot kernels, identified across the ten component benchmarks, point optimization effort at convolution, GEMM, batch normalization, ReLU, element-wise operations, and gradient computation.
  • The GPU stall analysis singles out memory dependency and execution dependency stalls as the top performance limiters, suggesting data layout, locality, and instruction-level parallelism as concrete optimization levers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the clinical-diagnosis sketch implies the framework is a template for other end-to-end AI services; a natural test is to build a medical-imaging diagnosis benchmark from the same modules and compare which AI components dominate its critical path.
  • Beyond the paper: because the six offline hotspot kernels match the micro benchmarks, an architecture study could simulate just those kernels as a proxy for the full workload, trading some fidelity for speed; the paper itself does not make this claim.
  • Beyond the paper: the fixed popularity split (15/50/50) and the ratio of AI to non-AI modules are configurable; varying them would show whether the recommender's dominance is a property of the workload shape or of the specific implementation.
  • Beyond the paper: the acknowledged gap between reported latencies and the partner's production numbers suggests a testable hypothesis, that the benchmark's bottleneck ranking, not its absolute latencies, is what transfers to a real service; public production aggregates could confirm or refute that transfer.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper presents AIBench, a benchmark suite intended to cover AI workloads in Internet services. It contributes a configurable framework with data input, component-benchmark, micro-benchmark, online-inference, offline-training, and deployment modules; identifies sixteen AI problem domains across search, social network, and e-commerce; and implements sixteen component benchmarks and twelve micro benchmarks. On top of this framework, the authors build an end-to-end e-commerce search application benchmark by composing ten component benchmarks, and they report workload characterizations on a 16-node CPU/GPU cluster: online latency and module-level latency breakdowns, offline training GPU kernel and stall analyses, and an argument that AI components dominate the critical path. The paper claims this is the first industry-standard Internet service AI benchmark suite, developed with seventeen industry partners, with specifications, source code, and performance numbers publicly available.

Significance. If accepted as a faithful reproduction of a production e-commerce critical path, the paper is significant: it provides a public, extensible benchmark artifact, spans sixteen AI problem domains, and reports concrete latency and GPU-stall characterizations that could guide system and architecture optimization. The strengths of the paper are the breadth of the component benchmarks, the explicit composition of a full-service workload, and the public availability of the code and data. The main uncertainty is external validity: the synthetic workload is not validated against production data distributions, query logs, or module-level production latencies, and the paper's own footnote 2 says the baseline latency is much higher than the partner's real numbers. The characterization results may therefore describe this particular deployment rather than a representative industry-scale service, so the 'industry standard' and 'critical path' claims currently outrun the evidence.

major comments (4)
  1. [§4.1, §6.1, footnote 2] The end-to-end benchmark is described as abstracting real-world data sets and workloads from a top e-commerce provider, but the paper provides no fidelity validation: there is no comparison of synthetic and production data distributions, no validation of the JMeter query model against real query logs, and no module-level latency comparison against production. Footnote 2 concedes that the baseline average latency is "quite high" compared with the partner's real numbers. Thus the observed 161/392/956 ms latencies and the conclusion that the recommender dominates the critical path may be properties of this particular implementation (Neo4j graph access, Flask/Nginx serving, TensorFlow Serving configuration, JMeter overhead) rather than of a realistic industry-scale service. The paper should either add a public fidelity study or rephrase the abstract and conclusions to present the workload as a plausible synthetic model rather than an industry-standard representation.
  2. [Abstract, §4.1, §5.2, §6.1] The claim that the benchmark "contains the primary modules in the critical paths of an industry scale application" is not supported by the reported deployment scale. The evaluation uses a 16-node cluster, a 100,000-product database, 32 product attribute fields, and 1,000 simulated users, which is far smaller than the production footprint of a top e-commerce provider. The paper should specify which production modules are included and which are omitted, explain how the 100,000-product scale was chosen, and provide evidence, rather than configuration options alone, that the benchmark scales to materially different cluster sizes as claimed.
  3. [§3.3, §6.2.1] The paper says the twelve micro benchmarks were identified by profiling the component benchmarks, and yet Section 6.2.1 takes the consistency between the six GPU hotspot kernels and the micro benchmarks as validation ("consistent with our micro benchmarks"). This is circular if the micro benchmarks were derived from the same component benchmarks that are later used to validate the hotspot-kernel finding. The authors should clarify the derivation: if the micro benchmarks were selected empirically from the same workloads, the consistency is a design outcome and should be described as such; if they were selected a priori, the procedure should be documented.
  4. [§6.1] The conclusion that AI components significantly change the critical paths and the trade-off discussion rely on the absolute latency of TensorFlow serving in this baseline implementation. Because footnote 2 acknowledges that the baseline implementation lacks many production optimizations, the paper should analyze whether the reported serving latency increase (average 30.78 to 125.71 ms and 99th percentile 149 to 5335 ms for a larger model) would persist under production-grade serving configurations, model quantization, batching, or GPU inference. Without such robustness analysis, the optimization guidance in Section 6 remains specific to this deployment rather than generalizable to industry-scale services.
minor comments (4)
  1. [Throughout] The manuscript contains several typographical and naming errors: "scenairo" in the Introduction, "state-of-the-practise" in Sections 1 and 2, "Pipi busy" for pipe busy in Section 6.2.2, "query planer" for search planer in Section 4.1.1, and "Cifar" in Table 4.
  2. [§6.1, Figure 3] The text refers to Fig. 3(a), 3(b), and 3(c), but the figure appears as a single image; please split or clearly label the sub-figures so that the latency, module-breakdown, and recommender-breakdown plots can be distinguished.
  3. [Abstract, author list] The paper repeatedly states that AIBench was developed with "seventeen industry partners," but the author list contains sixteen industry-affiliated institutions (Alibaba, Baidu, Tencent, 58.com, NetEase, ByteDance, NIM, Zhihu, Lenovo, Paypal, Moqi, MSRA, Huawei, JD.com, CloudTa, and Intellifusion); please reconcile this count.
  4. [§4.1.1] The searcher clusters are described as containing products with high, medium, and low popularity "occupying the proportion of 15%, 50%, and 50%, respectively," which sums to 115%; the intended categorization should be stated unambiguously, for instance by clarifying whether the high-popularity category is a subset of the medium-popularity category or a separate partition.

Circularity Check

2 steps flagged · score 4.0 of 10

Micro-benchmark validation is self-referential, and the AI-critical-path finding is partly by construction, but the benchmark suite itself remains an independent artifact.

  1. self definitional [Section 6.2.1 (Running Time Breakdown), with Section 3.1 and footnote 1]
    "Through statistics, we find that the most time-consuming functions among all the ten component benchmarks have much in common, and they are classified into six categories of kernels: convolution, general matrix multiply (gemm), batch normalization, relu activation, element-wise operation and gradient calculation, which is consistent with our micro benchmarks and further indicates the decision of including them is correct."

    The micro benchmarks were not an independent check: Section 3.1 says 'we profile the most intensive units of computation across those component benchmarks, and implement them as a set of micro benchmarks,' and footnote 1 states the six kernels 'are named based on CUDA functions and they are a subset of our micro benchmarks.' Therefore the hotspot kernels are consistent with the micro benchmarks by construction, because the micro benchmarks were extracted from the same component benchmarks. Using this consistency as evidence that the micro-benchmark selection is correct validates the selector on the very data used to build it.

  2. self definitional [Section 6.1 (Evaluation of Online Server) and Section 4.1.1 (Online Server)]
    "The weights of AI-related components on service performance. AI components change the critical path significantly."

    The end-to-end benchmark was designed with AI modules as mandatory stages on the online serving path: the search planer sends each query to the Recommender, then to the Searcher, then to the Ranker (steps 2-7 of Section 4.1.1). Thus the qualitative conclusion that AI components lie on the critical path is an architectural property of the benchmark rather than an empirical discovery. The measured latencies are empirical, but the 'AI changes the critical path' finding is built into the benchmark's structure.

full rationale

The paper is not a mathematical derivation; it presents a benchmark suite and workload characterizations, so most circularity patterns do not apply. The central artifact, AIBench, is independently implemented and publicly released, and its existence is not circular. The main self-referential loop is the validation of the micro benchmarks: the micro benchmarks were profiled from the same component benchmarks that are later used to confirm the hotspot-kernel finding, making the consistency check true by construction. A second, milder circularity is that AI components were placed in the online critical path by design, and the conclusion that they 'change the critical path' is therefore partly a design property. However, the benchmark's representative-ness and the reported latency numbers are external-validity concerns rather than circularity: the paper admits in footnote 2 that its baseline latency is much higher than the industry partner's production numbers, which weakens generalization but does not make the derivation circular. Overall, the core benchmark artifact has independent content, but two supporting characterization claims reduce to self-definitional construction, warranting a moderate score.

Assumptions & free parameters 8 free parameters · 5 assumptions · 0 invented entities

The central claims rest on hand-set workload parameters and unvalidated representativeness assumptions, not on fitted physical constants or new entities. The free parameters listed above are experimental configuration choices that shape every reported latency and efficiency number. The axioms capture the domain assumptions about representativeness and tooling that the paper does not independently verify. No new scientific entities are postulated.

free parameters (8)
  • Simulated user concurrency = 1000 users
    Section 6.1: the query generator simulates 1000 users; latency results scale with this choice.
  • Warm-up time = 30 seconds
    Section 6.1: fixed warm-up period before measurements are collected.
  • Completed query count = 20000 requests
    Section 6.1: performance numbers are collected when 20,000 queries finish.
  • Think time distribution = Poisson, rate not reported
    Section 6.1: user think times follow a Poisson distribution, but the mean rate is not given, so the load is not fully specified.
  • Product database size = 100000 products
    Section 6.1: the product database contains one hundred thousand products.
  • Product attribute fields = 32 fields
    Section 6.1: each product has 32 attribute fields, affecting index size and search cost.
  • Product popularity split = 15%, 50%, 50%
    Section 4.1.1: high, medium, and low popularity shares are hand-chosen; the values sum to 115%, which is internally inconsistent.
  • Comparison model sizes = 184 MB and 253 MB
    Section 6.1: the larger-model comparison uses a 253 MB model versus a 184 MB baseline; the paper does not explain how the models were constructed.
assumptions (5)
  • domain assumption The sixteen selected AI problem domains are a representative and comprehensive abstraction of AI use in search engines, social networks, and e-commerce.
    The selection in Section 3.2 is based on the authors' and partners' analysis, not on a reproducible survey or external standard.
  • domain assumption The e-commerce benchmark's architecture (search planer, recommender, searcher, ranker) captures the primary critical paths of an industry-scale provider.
    Section 4.1 asserts this design; the paper does not compare against a real production workload or validate module proportions.
  • domain assumption Synthetic data generated from the data schema and public datasets preserves the real-world workload characteristics of the e-commerce provider.
    Sections 3.4 and 4.1 claim anonymized real-world characteristics, but no fidelity analysis or comparison to production logs is presented.
  • domain assumption Profiling tools (perf on CPU, nvprof on GPU) provide accurate measurements of reported cache misses, stalls, and running time breakdowns.
    Section 5.3 uses these tools without cross-validation; the stall taxonomy depends on Nvidia's profiler interpretation.
  • domain assumption The hotspot CUDA functions can be reliably classified into the six kernel categories used for the breakdown.
    Section 6.2.1 groups functions by computation logic manually; the categories are not formally defined, and the learning-to-rank benchmark is assigned a convolution hotspot, which is surprising for a ranking task.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AIBench: An Industry Standard Internet Service AI Benchmark Suite." pith.science (2026). https://pith.science/paper/R3RMVA74

@misc{pith2026190808998,
  author       = {Pith},
  title        = {Pith review of: AIBench: An Industry Standard Internet Service AI Benchmark Suite},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/R3RMVA74}},
  note         = {Machine review of arXiv:1908.08998}
}
read the original abstract

Today's Internet Services are undergoing fundamental changes and shifting to an intelligent computing era where AI is widely employed to augment services. In this context, many innovative AI algorithms, systems, and architectures are proposed, and thus the importance of benchmarking and evaluating them rises. However, modern Internet services adopt a microservice-based architecture and consist of various modules. The diversity of these modules and complexity of execution paths, the massive scale and complex hierarchy of datacenter infrastructure, the confidential issues of data sets and workloads pose great challenges to benchmarking. In this paper, we present the first industry-standard Internet service AI benchmark suite---AIBench with seventeen industry partners, including several top Internet service providers. AIBench provides a highly extensible, configurable, and flexible benchmark framework that contains loosely coupled modules. We identify sixteen prominent AI problem domains like learning to rank, each of which forms an AI component benchmark, from three most important Internet service domains: search engine, social network, and e-commerce, which is by far the most comprehensive AI benchmarking effort. On the basis of the AIBench framework, abstracting the real-world data sets and workloads from one of the top e-commerce providers, we design and implement the first end-to-end Internet service AI benchmark, which contains the primary modules in the critical paths of an industry scale application and is scalable to deploy on different cluster scales. The specifications, source code, and performance numbers are publicly available from the benchmark council web site http://www.benchcouncil.org/AIBench/index.html.

Figures

Figures reproduced from arXiv: 1908.08998 by the authors.

Figure 1
Figure 1. AIBench Framework. To be easily deployed on a large-scale cluster, the framework provides deployment tools that contain two automated deployment templates using Ansible and Kubernetes, respectively. Among them, the Ansible templates support scalable deployment on physical machines or virtual machines, while the kubernetes templates are used to deploy on container clusters. A configuration file needs to be specified … view at source ↗
Figure 2
Figure 2. AIBench Implementation. 3.4 Data Model To cover a diversity of data sets from various applications, we collect 15 representative data sets, including ImageNet [47], Cifar [48], LSUN [49], WMT English-German [50], Cityscapes [51], LibriSpeech [52], Microsoft COCO data set [53], LFW [54], VGGFace2 [55], Robot pushing data set [41], MovieLens data set [56], ShapeNet data set [57], Gigaword data set [58], MNIST data set… view at source ↗
Figure 3
Figure 3. Latency of Online Server. for Image-to-Image, MovieLens (190 MB) for recommendation, Librispeech (59.3 GB) for speech recognition, and Gowalla (107 MB) for learning to rank, respectively. 5.3 Performance Data Collection We use network time protocol (NTP) [68] to perform clock synchronization in all cluster nodes and obtain the latency and tail latency metrics of online server. We use a profiling tool—Perf [69] to co… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: SM Efficiency [PITH_FULL_IMAGE:figures/full_fig_p017_4.png]
Figure 5
Figure 5. Figure 5: Running Time Breakdown of the Ten Component Benchmarks. [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]
Figure 6
Figure 6. Figure 6: Stall Breakdown of Each Kernel. We also identify the functional level stalls, including the hotspot functions illustrated in [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

70 extracted references · 60 canonical work pages

  1. [1]

    Deep learning,

    Y . LeCun, Y . Bengio, and G. Hinton, “Deep learning,”Nature, vol. 521, no. 7553, pp. 436–444, 2015

    2015

  2. [2]

    Perceive your users in depth: Learning universal user representations from multiple e-commerce tasks,

    Y . Ni, D. Ou, S. Liu, X. Li, W. Ou, A. Zeng, and L. Si, “Perceive your users in depth: Learning universal user representations from multiple e-commerce tasks,” inProceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 596–605, ACM, 2018

    work page 2018

  3. [3]

    Applied 18 machine learning at facebook: A datacenter infrastructure perspective,

    K. Hazelwood, S. Bird, D. Brooks, S. Chintala, U. Diril, D. Dzhulgakov, M. Fawzy, B. Jia, Y . Jia, A. Kalro, J. Law, K. Lee, J. Lu, P. Noordhuis, M. Smelyanskiy, L. Xiong, and X. Wang, “Applied 18 machine learning at facebook: A datacenter infrastructure perspective,” in 2018 IEEE International Symposium on High Performance Computer Architecture (HPCA), p...

    work page 2018

  4. [4]

    Tensorflow: Large-scale machine learning on heterogeneous distributed systems,

    M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y . Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mane, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V . Vanhouc...

    arXiv 2016

  5. [5]

    In-datacenter performance analysis of a tensor processing unit,

    N. P. Jouppi, C. Young, N. Patil, D. Patterson, G. Agrawal, R. Bajwa, S. Bates, S. Bhatia, N. Boden, A. Borchers, R. Boyle, P.-l. Cantin, C. Chao, C. Clark, J. Coriell, M. Daley, M. Dau, J. Dean, B. Gelb, T. V . Ghaemmaghami, R. Gottipati, W. Gulland, R. Hagmann, C. Ho, D. Hogberg, J. Hu, R. Hundt, D. Hurt, J. Ibarz, A. Jaffey, A. Jaworski, A. Kaplan, H. ...

    work page 2017

  6. [6]

    Two decades of recommender systems at amazon. com,

    B. Smith and G. Linden, “Two decades of recommender systems at amazon. com,” Ieee internet computing, vol. 21, no. 3, pp. 12–18, 2017

    work page 2017

  7. [7]

    Memory hierarchy for web search,

    G. Ayers, J. H. Ahn, C. Kozyrakis, and P. Ranganathan, “Memory hierarchy for web search,” in2018 IEEE International Symposium on High Performance Computer Architecture (HPCA), pp. 643–656, IEEE, 2018

    work page 2018

  8. [8]

    The datacenter as a computer: An introduction to the design of warehouse-scale machines,

    L. A. Barroso and U. H ¨olzle, “The datacenter as a computer: An introduction to the design of warehouse-scale machines,”Synthesis Lectures on Computer Architecture, vol. 4, no. 1, pp. 1–108, 2009

    work page 2009

Show all 70 references
  1. [9]

    Bigbench: Towards an industry standard benchmark for big data analytics,

    A. Ghazal, M. Hu, T. Rabl, F. Raab, M. Poess, A. Crolotte, and H.-A. Jacobsen, “Bigbench: Towards an industry standard benchmark for big data analytics,” inSIGMOD 2013, 2013

    2013

  2. [10]

    Data motifs: A lens towards fully understanding big data and ai workloads,

    W. Gao, J. Zhan, L. Wang, C. Luo, D. Zheng, F. Tang, B. Xie, C. Zheng, X. Wen, X. He, H. Ye, and R. Ren, “Data motifs: A lens towards fully understanding big data and ai workloads,”Parallel Architectures and Compilation Techniques (PACT), 2018 27th International Conference on, 2018

    2018

  3. [11]

    Cvr: Efficient vectorization of spmv on x86 processors,

    B. Xie, J. Zhan, X. Liu, W. Gao, Z. Jia, X. He, and L. Zhang, “Cvr: Efficient vectorization of spmv on x86 processors,” in2018 IEEE/ACM International Symposium on Code Generation and Optimization (CGO), 2018

    2018

  4. [12]

    The nas parallel benchmarks,

    D. H. Bailey, E. Barszcz, J. T. Barton, D. S. Browning, R. L. Carter, L. Dagum, R. A. Fatoohi, P. O. Frederickson, T. A. Lasinski, R. S. Schreiber, H. Simon, V . Venkatakrishnan, and S. Weeratunga, “The nas parallel benchmarks,”The International Journal of Supercomputing Appli...

    1991

  5. [13]

    “Mlperf.” https://mlperf.org

  6. [14]

    Fathom: reference workloads for modern deep learning methods,

    R. Adolf, S. Rama, B. Reagen, G.-Y . Wei, and D. Brooks, “Fathom: reference workloads for modern deep learning methods,” inWorkload Characterization (IISWC), pp. 1–10, IEEE, 2016

    2016

  7. [15]

    Deepbench

    “Deepbench.” https://svail.github.io/DeepBench/. 19

  8. [16]

    Dnnmark: A deep neural network benchmark suite for gpus,

    S. Dong and D. Kaeli, “Dnnmark: A deep neural network benchmark suite for gpus,” inProceedings of the General Purpose GPUs, pp. 63–72, ACM, 2017

    2017

  9. [17]

    Dawnbench: An end-to-end deep learning benchmark and competition,

    C. Coleman, D. Narayanan, D. Kang, T. Zhao, J. Zhang, L. Nardi, P. Bailis, K. Olukotun, C. R ´e, and M. Zaharia, “Dawnbench: An end-to-end deep learning benchmark and competition,” Training, vol. 100, no. 101, p. 102, 2017

    2017

  10. [18]

    Tbd: Benchmarking and analyzing deep neural network training,

    H. Zhu, M. Akrout, B. Zheng, A. Pelegris, A. Phanishayee, B. Schroeder, and G. Pekhimenko, “Tbd: Benchmarking and analyzing deep neural network training,”arXiv preprint arXiv:1803.06905, 2018

    2018 arXiv

  11. [19]

    HPC AI500: A Benchmark Suite for HPC AI systems,

    Z. Jiang, W. Gao, L. Wang, X. Xiong, Y . Zhang, X. Wen, C. Luo, H. Ye, X. Lu, Y . Zhang, S. Feng, K. Li, W. Xu, and J. Zhan, “HPC AI500: A Benchmark Suite for HPC AI systems,”2018 BenchCouncil International Symposium on Benchmarking, Measuring and Optimizing (Bench18), 2018

    2018

  12. [20]

    AIoT Bench: Towards Comprehensive Benchmarking Mobile and Embedded Device Intelligence,

    C. Luo, F. Zhang, C. Huang, X. Xiong, J. Chen, L. Wang, W. Gao, H. Ye, T. Wu, R. Zhou, and J. Zhan, “AIoT Bench: Towards Comprehensive Benchmarking Mobile and Embedded Device Intelligence,” 2018 BenchCouncil International Symposium on Benchmarking, Measuring and Optimizing (Be...

    2018

  13. [21]

    Edge AIBench: Towards Comprehensive End-to-end Edge Computing Benchmark- ing,

    T. Hao, Y . Huang, X. Wen, W. Gao, F. Zhang, C. Zheng, L. Wang, H. Ye, K. Hwang, Z. Ren, and J. Zhan, “Edge AIBench: Towards Comprehensive End-to-end Edge Computing Benchmark- ing,”2018 BenchCouncil International Symposium on Benchmarking, Measuring and Optimizing (Bench18), 2018

    2018

  14. [22]

    In BenchCouncil International Symposium on Benchmarking, Measuring and Optimizing (Bench18), 2018

    Gao W, Luo C, Wang L, Xiong X, Chen J, Hao T, Jiang Z, Fan F, Du M, Huang Y , Zhang F, Wen X, Zheng C, He X, Dai J, Ye H, Cao Z, Jia Z, Zhan K, Tang H, Zheng D, Xie B, Li W, Wang X, Zhan J.: AIBench: Towards Scalable and Comprehensive Datacenter AI Benchmarking. In BenchCounci...

    2018

  15. [23]

    BigDataBench: A scalable and unified big data and ai benchmark suite,

    W. Gao, J. Zhan, L. Wang, C. Luo, D. Zheng, X. Wen, R. Ren, C. Zheng, X. He, H. Ye et al., “BigDataBench: A scalable and unified big data and ai benchmark suite,” arXiv preprint arXiv:1802.08254, 2018

    2018 arXiv

  16. [24]

    BigDataBench: A big data benchmark suite from internet services,

    L. Wang, J. Zhan, C. Luo, Y . Zhu, Q. Yang, Y . He, W. Gao, Z. Jia, Y . Shi, S. Zhang et al. , “BigDataBench: A big data benchmark suite from internet services,” IEEE International Symposium On High Performance Computer Architecture (HPCA), 2014

    2014

  17. [25]

    Characterizing data analysis workloads in data centers,

    Z. Jia, L. Wang, J. Zhan, L. Zhang, and C. Luo, “Characterizing data analysis workloads in data centers,” in2013 IEEE International Symposium on Workload Characterization (IISWC). IEEE, 2013, pp. 66–76

    2013

  18. [26]

    Frustrated with replicating claims of a shared model? a solution,

    A. Dakkak, C. Li, J. Xiong, and W.-m. Hwu, “Frustrated with replicating claims of a shared model? a solution,”arXiv preprint arXiv:1811.09737, 2019

    2019 arXiv

  19. [27]

    Bdgs: A scalable big data generator suite in big data benchmarking,

    Z. Ming, C. Luo, W. Gao, R. Han, Q. Yang, L. Wang, and J. Zhan, “Bdgs: A scalable big data generator suite in big data benchmarking,”arXiv preprint arXiv:1401.5465, 2014

    2014 arXiv

  20. [28]

    Full resolu- tion image compression with recurrent neural networks,

    G. Toderici, D. Vincent, N. Johnston, S. Jin Hwang, D. Minnen, J. Shor, and M. Covell, “Full resolu- tion image compression with recurrent neural networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5306–5314, 2017

    2017

  21. [29]

    Perspective transformer nets: Learning single-view 3d object reconstruction without 3d supervision,

    X. Yan, J. Yang, E. Yumer, Y . Guo, and H. Lee, “Perspective transformer nets: Learning single-view 3d object reconstruction without 3d supervision,” in Advances in Neural Information Processing Systems, pp. 1696–1704, 2016. 20

    2016

  22. [30]

    Spatial transformer networks,

    M. Jaderberg, K. Simonyan, A. Zisserman, and K. Kavukcuoglu, “Spatial transformer networks,” in Advances in neural information processing systems, pp. 2017–2025, 2015

    2017

  23. [31]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” inProceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016

    2016

  24. [32]

    Wasserstein gan,

    M. Arjovsky, S. Chintala, and L. Bottou, “Wasserstein gan,”arXiv preprint arXiv:1701.07875, 2017

    2017 arXiv

  25. [33]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez,Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” inAdvances in neural information processing systems, pp. 5998–6008, 2017

    2017

  26. [34]

    Show and tell: Lessons learned from the 2015 mscoco image captioning challenge,

    O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, “Show and tell: Lessons learned from the 2015 mscoco image captioning challenge,” IEEE transactions on pattern analysis and machine intelli- gence, vol. 39, no. 4, pp. 652–663, 2017

    2015

  27. [35]

    Unpaired image-to-image translation using cycle- consistent adversarial networks,

    J.-Y . Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle- consistent adversarial networks,” inProceedings of the IEEE international conference on computer vision, pp. 2223–2232, 2017

    2017

  28. [36]

    Deep speech 2: End-to-end speech recognition in english and mandarin,

    D. Amodei, S. Ananthanarayanan, R. Anubhai, J. Bai, E. Battenberg, C. Case, J. Casper, B. Catanzaro, Q. Cheng, G. Chen, J. Chen, J. Chen, Z. Chen, M. Chrzanowski, A. Coates, G. Diamos, K. Ding, N. Du, E. Elsen, J. Engel, W. Fang, L. Fan, C. Fougner, L. Gao, C. Gong, A. Hannun,...

    2016

  29. [37]

    Facenet: A unified embedding for face recognition and clustering,

    F. Schroff, D. Kalenichenko, and J. Philbin, “Facenet: A unified embedding for face recognition and clustering,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 815–823, 2015

    2015

  30. [38]

    Facial expression recognition under a wide range of head poses,

    R.-L. Vieriu, S. Tulyakov, S. Semeniuta, E. Sangineto, and N. Sebe, “Facial expression recognition under a wide range of head poses,” in 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), vol. 1, pp. 1–7, IEEE, 2015

    2015

  31. [39]

    Faster r-cnn: Towards real-time object detection with region proposal networks,

    S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” inAdvances in neural information processing systems, pp. 91–99, 2015

    2015

  32. [40]

    Neural collaborative filtering,

    X. He, L. Liao, H. Zhang, L. Nie, X. Hu, and T.-S. Chua, “Neural collaborative filtering,” in Proceedings of the 26th international conference on world wide web, pp. 173–182, International World Wide Web Conferences Steering Committee, 2017

    2017

  33. [41]

    Unsupervised learning for physical interaction through video prediction,

    C. Finn, I. Goodfellow, and S. Levine, “Unsupervised learning for physical interaction through video prediction,” inAdvances in neural information processing systems, pp. 64–72, 2016

    2016

  34. [42]

    Abstractive text summarization using sequence- to-sequence rnns and beyond,

    R. Nallapati, B. Zhou, C. Gulcehre, and B. Xiang, “Abstractive text summarization using sequence- to-sequence rnns and beyond,”arXiv preprint arXiv:1602.06023, 2016

    2016 arXiv

  35. [43]

    Ranking distillation: Learning compact ranking models with high performance for recommender system,

    J. Tang and K. Wang, “Ranking distillation: Learning compact ranking models with high performance for recommender system,” in ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2018

    2018

  36. [44]

    Cosine normalization: Using cosine similarity instead of dot product in neural networks,

    C. Luo, J. Zhan, L. Wang, and Q. Yang, “Cosine normalization: Using cosine similarity instead of dot product in neural networks,”arXiv preprint arXiv:1702.05870, 2017. 21

    2017 arXiv

  37. [45]

    Batch normalization: Accelerating deep network training by reducing internal covariate shift,

    S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” inInternational Conference on Machine Learning, pp. 448–456, 2015

    2015

  38. [46]

    Dropout: a simple way to prevent neural networks from overfitting.,

    N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: a simple way to prevent neural networks from overfitting.,”Journal of machine learning research, vol. 15, no. 1, pp. 1929–1958, 2014

    1929

  39. [47]

    Imagenet: A large-scale hierarchical image database,

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and F.-F. Li, “Imagenet: A large-scale hierarchical image database,” in Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pp. 248–255, IEEE, 2009

    2009

  40. [48]

    The cifar-10 dataset,

    A. Krizhevsky, V . Nair, and G. Hinton, “The cifar-10 dataset,” online: http://www. cs. toronto. edu/kriz/cifar. html, vol. 55, 2014

    2014

  41. [49]

    Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop,

    F. Yu, A. Seff, Y . Zhang, S. Song, T. Funkhouser, and J. Xiao, “Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop,” arXiv preprint arXiv:1506.03365, 2015

    2015 arXiv

  42. [50]

    https://nlp.stanford.edu/projects/nmt/

  43. [51]

    The cityscapes dataset for semantic urban scene understanding,

    M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, “The cityscapes dataset for semantic urban scene understanding,” inProceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213–3223, 2016

    2016

  44. [52]

    Librispeech: an asr corpus based on public domain audio books,

    V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: an asr corpus based on public domain audio books,” in 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5206–5210, IEEE, 2015

    2015

  45. [53]

    Microsoft coco: Common objects in context,

    T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll ´ar, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in European conference on computer vision, pp. 740– 755, Springer, 2014

    2014

  46. [54]

    Labeled faces in the wild: A database forstudying face recognition in unconstrained environments,

    G. B. Huang, M. Mattar, T. Berg, and E. Learned-Miller, “Labeled faces in the wild: A database forstudying face recognition in unconstrained environments,” in Workshop on faces in’Real- Life’Images: detection, alignment, and recognition, 2008

    2008

  47. [55]

    Vggface2: A dataset for recognising faces across pose and age,

    Q. Cao, L. Shen, W. Xie, O. M. Parkhi, and A. Zisserman, “Vggface2: A dataset for recognising faces across pose and age,” in 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp. 67–74, IEEE, 2018

    2018

  48. [56]

    The movielens datasets: History and context,

    F. M. Harper and J. A. Konstan, “The movielens datasets: History and context,”ACM Transactions on Interactive Intelligent Systems (TiiS), vol. 5, no. 4, p. 19, 2016

    2016

  49. [57]

    Shapenet: An information-rich 3d model repository,

    A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, J. Xiao, L. Yi, and F. Yu, “Shapenet: An information-rich 3d model repository,” arXiv preprint arXiv:1512.03012, 2015

    2015 arXiv

  50. [58]

    A neural attention model for sentence sum- marization,

    A. M. Rush, S. Harvard, S. Chopra, and J. Weston, “A neural attention model for sentence sum- marization,” in ACLWeb. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, 2017

    2015

  51. [59]

    Mnist handwritten digit database,

    Y . LeCun, C. Cortes, and C. Burges, “Mnist handwritten digit database,” AT&T Labs [Online]. Available: http://yann. lecun. com/exdb/mnist, vol. 2, p. 18, 2010

    2010

  52. [60]

    Friendship and mobility: user movement in location-based social networks,

    E. Cho, S. A. Myers, and J. Leskovec, “Friendship and mobility: user movement in location-based social networks,” inProceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 1082–1090, ACM, 2011. 22

    2011

  53. [61]

    Apache jmeter,

    A. JMeter, “Apache jmeter,” Online.(2016). http://jmeter. apache. org/-Visited, pp. 04–25, 2017

    2016

  54. [62]

    Spring boot reference guide,

    P. Webb, D. Syer, J. Long, S. Nicoll, R. Winch, A. Wilkinson, M. Overdijk, C. Dupuis, and S. Deleuze, “Spring boot reference guide,”Part IV . Spring Boot features, vol. 24, 2013

    2013

  55. [63]

    Fasttext.zip: Compressing text classification models,

    A. Joulin, E. Grave, P. Bojanowski, M. Douze, H. J´egou, and T. Mikolov, “Fasttext.zip: Compressing text classification models,” arXiv preprint arXiv:1612.03651, 2016

    2016 arXiv

  56. [64]

    Grinberg, Flask web development: developing web applications with python

    M. Grinberg, Flask web development: developing web applications with python. ” O’Reilly Media, Inc.”, 2018

    2018

  57. [65]

    Nginx: the high-performance web server and reverse proxy,

    W. Reese, “Nginx: the high-performance web server and reverse proxy,”Linux Journal, vol. 2008, no. 173, p. 2, 2008

    2008

  58. [66]

    Tensorflow-serving: Flexible, high-performance ml serving,

    C. Olston, N. Fiedel, K. Gorovoy, J. Harmsen, L. Lao, F. Li, V . Rajashekhar, S. Ramesh, and J. Soyke, “Tensorflow-serving: Flexible, high-performance ml serving,” arXiv preprint arXiv:1712.06139, 2017

    2017 arXiv

  59. [67]

    Gormley and Z

    C. Gormley and Z. Tong, Elasticsearch: the definitive guide: a distributed real-time search and analytics engine. ” O’Reilly Media, Inc.”, 2015

    2015

  60. [68]

    Network time protocol (ntp),

    D. L. Mills, “Network time protocol (ntp),” Network, 1985

    1985

  61. [69]

    The new linux perf tools,

    A. C. De Melo, “The new linux perf tools,” in Slides from Linux Kongress, vol. 18, 2010

    2010

  62. [70]

    Nvidia profiling toolkit

    “Nvidia profiling toolkit.” https://docs.nvidia.com/cuda/profiler-users-guide/index.html. 23

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.