RECORD
Performance and Power: Systematic Evaluation of AI Workloads on Accelerators with CARAML
Hardware Architecture · Submitted 19 Sep 2024
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim CARAML provides a compact, automated, extensible, and reproducible framework for assessing the performance and energy of ML workloads across various novel hardware architectures.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
The rapid advancement of machine learning (ML) technologies has driven the development of specialized hardware accelerators designed to facilitate more efficient model training. This paper introduces the CARAML benchmark suite, which is employed to assess performance and energy consumption during the training of transformer-based large language models and computer vision models on a range of hardware accelerators, including systems from NVIDIA, AMD, and Graphcore. CARAML provides a compact, automated, extensible, and reproducible framework for assessing the performance and energy of ML workloads across various novel hardware architectures. The design and implementation of CARAML, along with a custom power measurement tool called jpwr, are discussed in detail.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | The rapid advancement of machine learning (ML) technologies has driven the development of specialized hardware accelerators designed to facilitate more efficient model training. | background 1.00 | 0.04 | 0.02 | asserted 1.00 |
| s1 | This paper introduces the CARAML benchmark suite, which is employed to assess performance and energy consumption during the training of transformer-based large language models and computer vision models on a range of hardware accelerators, including systems from NVIDIA, AMD, and Graphcore. | contribution 0.99 | 0.85 | 0.08 | asserted 1.00 |
| s2 | CARAML provides a compact, automated, extensible, and reproducible framework for assessing the performance and energy of ML workloads across various novel hardware architectures. | contribution 0.81 | 0.74 | 0.10 | asserted 1.00 |
| s3 | The design and implementation of CARAML, along with a custom power measurement tool called jpwr, are discussed in detail. | method 0.77 | 0.91 | 0.03 | asserted 0.99 |
What was examined, and what was not
- arXiv listing abstract v2
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
The rapid advancement of machine learning (ML) technologies has driven the development of specialized hardware accelerators designed to facilitate more efficient model training. This paper introduces the CARAML benchmark suite, which is employed to assess performance and energy consumption during the training of transformer-based large language models and computer vision models on a range of hardware accelerators, including systems from NVIDIA, AMD, and Graphcore. CARAML provides a compact, automated, extensible, and reproducible framework for assessing the performance and energy of ML workloads across various novel hardware architectures. The design and implementation of CARAML, along with a custom power measurement tool called jpwr, are discussed in detail.
Human review
How the literature uses it
1 arXiv paper cite this in 4 sentences
Papers citing it · 2 sentences without a confident label
Uses its method or tool 1 paper
-
The systems of the JURECA Evaluation Platform (see Section [ref]) were benchmarked using CARAML [ref] , with detailed measurements of energy consumption and training performance (see Fig.2 in [ref] ).
Training LLMs on HPC Systems: Best Practices from the OpenGPT-X Project · Benchmarking and Hyper-parameter optimization · 0.84
What it cites · 4 sentences without a confident label
Uses the method or tool of 2 papers
-
The NLP benchmark task in CARAML utilizes the Megatron-LM framework [ref] , a robust, research-focused software platform developed by NVIDIA using .
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism · Natural Language Processing · 0.99
-
The NLP benchmark task in CARAML utilizes the Megatron-LM framework [ref] , a robust, research-focused software platform developed by NVIDIA using .
Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM · Natural Language Processing · 0.99
Cites as background 2 papers
-
State-of-the-art methods include 3D parallelism [ref] (combining data, tensor, and pipeline parallelism), sequence parallelism [ref] , activation recomputation [ref] , and optimizations like flash attention [ref] .
Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM · Natural Language Processing · 0.90
-
While the transformer architecture has also found its way into image recognition [ref] , convolutional neural networks are extremely mature and widely used in production environments, and represent an important benchmark case.
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale · Computer Vision · 0.86
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.