RECORD
Supporting Very Large Models using Automatic Dataflow Graph Partitioning
Distributed, Parallel, and Cluster Computing · Submitted 24 Jul 2018
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim Our experiments on an 8-GPU machine show that Tofu enables the training of very large CNN and RNN models.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
This paper presents Tofu, a system that partitions very large DNN models across multiple GPU devices to reduce per-GPU memory footprint. Tofu is designed to partition a dataflow graph of fine-grained tensor operators in order to work transparently with a general-purpose deep learning platform like MXNet. In order to automatically partition each operator, we propose to describe the semantics of an operator in a simple language which represents tensors as lambda functions mapping from tensor coordinates to values. To optimally partition different operators in a dataflow graph, Tofu uses a recursive search algorithm that minimizes the total communication cost. Our experiments on an 8-GPU machine show that Tofu enables the training of very large CNN and RNN models. It also achieves 25% - 400% speedup over alternative approaches to train very large models.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | This paper presents Tofu, a system that partitions very large DNN models across multiple GPU devices to reduce per-GPU memory footprint. | contribution 1.00 | 0.82 | 0.28 | asserted 1.00 |
| s1 | Tofu is designed to partition a dataflow graph of fine-grained tensor operators in order to work transparently with a general-purpose deep learning platform like MXNet. | method 0.69 | 0.55 | 0.04 | asserted 1.00 |
| s2 | In order to automatically partition each operator, we propose to describe the semantics of an operator in a simple language which represents tensors as lambda functions mapping from tensor coordinates to values. | method 0.93 | 0.84 | 0.04 | asserted 1.00 |
| s3 | To optimally partition different operators in a dataflow graph, Tofu uses a recursive search algorithm that minimizes the total communication cost. | method 0.98 | 0.88 | 0.07 | asserted 1.00 |
| s4 | Our experiments on an 8-GPU machine show that Tofu enables the training of very large CNN and RNN models. | result 1.00 | 0.96 | 0.31 | asserted 1.00 |
| s5 | It also achieves 25% - 400% speedup over alternative approaches to train very large models. | result 1.00 | 0.92 | 0.31 | asserted 1.00 |
What was examined, and what was not
- arXiv listing abstract v2
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
This paper presents Tofu, a system that partitions very large DNN models across multiple GPU devices to reduce per-GPU memory footprint. Tofu is designed to partition a dataflow graph of fine-grained tensor operators in order to work transparently with a general-purpose deep learning platform like MXNet. In order to automatically partition each operator, we propose to describe the semantics of an operator in a simple language which represents tensors as lambda functions mapping from tensor coordinates to values. To optimally partition different operators in a dataflow graph, Tofu uses a recursive search algorithm that minimizes the total communication cost. Our experiments on an 8-GPU machine show that Tofu enables the training of very large CNN and RNN models. It also achieves 25% - 400% speedup over alternative approaches to train very large models.
Human review
How the literature uses it
3 arXiv papers cite this in 10 sentences
Papers citing it · 6 sentences without a confident label
Uses its method or tool 1 paper
-
To search for the partition for each layer, we apply the dynamic programming-based search and recursive partitioning method proposed in [ref] which partitions on tensors and feature maps, respectively.
HyPar: Towards Hybrid Parallelism for Deep Learning Accelerator Array · Introduction · 1.00
Cites it as background 1 paper
-
Recently, mixed model and data parallelism is explored in deep learning accelerator architectures [ref] multi-GPU training systems [ref] .
HyPar: Towards Hybrid Parallelism for Deep Learning Accelerator Array · Introduction · 0.85
What it cites · 18 sentences without a confident label
Compares with 1 paper
-
Note that, for the case of WResNet, this gives similar result as the one-weird-trick strategy proposed in [ref] , because all the convolution layers are partitioned by the batch dimension and the only fully-connected layer in WResNet occupies <1% of the total time.
One weird trick for parallelizing convolutional neural networks · Comparing different partition algorithms · 0.87
Cites as background 10 papers
-
Existing proposals use greedy or dynamic-programming algorithm to optimize a mostly linear graph of layers [ref] , or perform stochastic searches [ref] for general graphs.
Device Placement Optimization with Reinforcement Learning · How to optimize partitioning for a graph? · 0.89
-
Existing proposals use greedy or dynamic-programming algorithm to optimize a mostly linear graph of layers [ref] , or perform stochastic searches [ref] for general graphs.
Beyond Data and Model Parallelism for Deep Neural Networks · How to optimize partitioning for a graph? · 0.89
-
WResNet [ref] is a widened version of the original residual network model [ref] .
Wide Residual Networks · Experimental setup · 0.91
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.