RECORD
G-DARTS-A: Groups of Channel Parallel Sampling with Attention
Machine Learning · Submitted 16 Oct 2020
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim The searched architecture is more powerful and better adapted to different deployments.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
Differentiable Architecture Search (DARTS) provides a baseline for searching effective network architectures based gradient, but it is accompanied by huge computational overhead in searching and training network architecture. Recently, many novel works have improved DARTS. Particularly, Partially-Connected DARTS(PC-DARTS) proposed the partial channel sampling technique which achieved good results. In this work, we found that the backbone provided by DARTS is prone to overfitting. To mitigate this problem, we propose an approach named Group-DARTS with Attention (G-DARTS-A), using multiple groups of channels for searching. Inspired by the partially sampling strategy of PC-DARTS, we use groups channels to sample the super-network to perform a more efficient search while maintaining the relative integrity of the network information. In order to relieve the competition between channel groups and keep channel balance, we follow the attention mechanism in Squeeze-and-Excitation Network. Each group of channels shares defined weights thence they can provide different suggestion for searching. The searched architecture is more powerful and better adapted to different deployments. Specifically, by only using the attention module on DARTS we achieved an error rate of 2.82%/16.36% on CIFAR10/100 with 0.3GPU-days for search process on CIFAR10. Apply our G-DARTS-A to DARTS/PC-DARTS, an error rate of 2.57%/2.61% on CIFAR10 with 0.5/0.4 GPU-days is achieved.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | Differentiable Architecture Search (DARTS) provides a baseline for searching effective network architectures based gradient, but it is accompanied by huge computational overhead in searching and training network architecture. | background 0.99 | 0.07 | 0.03 | asserted 1.00 |
| s1 | Recently, many novel works have improved DARTS. | background 1.00 | 0.04 | 0.02 | asserted 1.00 |
| s2 | Particularly, Partially-Connected DARTS(PC-DARTS) proposed the partial channel sampling technique which achieved good results. | background 1.00 | 0.03 | 0.03 | asserted 1.00 |
| s3 | In this work, we found that the backbone provided by DARTS is prone to overfitting. | gap 0.42 | 0.92 | 0.06 | asserted 1.00 |
| s4 | To mitigate this problem, we propose an approach named Group-DARTS with Attention (G-DARTS-A), using multiple groups of channels for searching. | contribution 0.91 | 0.81 | 0.19 | asserted 1.00 |
| s5 | Inspired by the partially sampling strategy of PC-DARTS, we use groups channels to sample the super-network to perform a more efficient search while maintaining the relative integrity of the network information. | method 0.94 | 0.81 | 0.08 | asserted 0.98 |
| s6 | In order to relieve the competition between channel groups and keep channel balance, we follow the attention mechanism in Squeeze-and-Excitation Network. | method 0.97 | 0.57 | 0.04 | asserted 0.96 |
| s7 | Each group of channels shares defined weights thence they can provide different suggestion for searching. | method 0.99 | 0.67 | 0.07 | asserted 0.98 |
| s8 | The searched architecture is more powerful and better adapted to different deployments. | interpretation 0.68 | 0.61 | 0.42 | asserted 1.00 |
| s9 | Specifically, by only using the attention module on DARTS we achieved an error rate of 2.82%/16.36% on CIFAR10/100 with 0.3GPU-days for search process on CIFAR10. | result 1.00 | 0.93 | 0.21 | asserted 1.00 |
| s10 | Apply our G-DARTS-A to DARTS/PC-DARTS, an error rate of 2.57%/2.61% on CIFAR10 with 0.5/0.4 GPU-days is achieved. | result 1.00 | 0.94 | 0.30 | asserted 1.00 |
What was examined, and what was not
- arXiv listing abstract v1
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
Differentiable Architecture Search (DARTS) provides a baseline for searching effective network architectures based gradient, but it is accompanied by huge computational overhead in searching and training network architecture. Recently, many novel works have improved DARTS. Particularly, Partially-Connected DARTS(PC-DARTS) proposed the partial channel sampling technique which achieved good results. In this work, we found that the backbone provided by DARTS is prone to overfitting. To mitigate this problem, we propose an approach named Group-DARTS with Attention (G-DARTS-A), using multiple groups of channels for searching. Inspired by the partially sampling strategy of PC-DARTS, we use groups channels to sample the super-network to perform a more efficient search while maintaining the relative integrity of the network information. In order to relieve the competition between channel groups and keep channel balance, we follow the attention mechanism in Squeeze-and-Excitation Network. Each group of channels shares defined weights thence they can provide different suggestion for searching. The searched architecture is more powerful and better adapted to different deployments. Specifically, by only using the attention module on DARTS we achieved an error rate of 2.82%/16.36% on CIFAR10/100 with 0.3GPU-days for search process on CIFAR10. Apply our G-DARTS-A to DARTS/PC-DARTS, an error rate of 2.57%/2.61% on CIFAR10 with 0.5/0.4 GPU-days is achieved.
Human review
How the literature uses it
What this paper does with the arXiv papers it cites
What it cites · 3 sentences without a confident label
Uses the method or tool of 2 papers
-
We use momentum SGD [ref] to optimize the weight $w$, initial learning rate $η_w= 0.025$ (anneal it to zero according to the cosine schedule without restarting), momentum $0.9$ and weight decay $3×10^-4$, while we use Adam [ref] as the optimizer for $α$ with initial learning rate$η_α= 3×10^-4$, momentum $β= (0.5, 0.999)$and weight decay$0.001$, and cutout [ref] status is True.
Adam: A Method for Stochastic Optimization · Implementation Details · 1.00
-
We use momentum SGD [ref] to optimize the weight $w$, initial learning rate $η_w= 0.025$ (anneal it to zero according to the cosine schedule without restarting), momentum $0.9$ and weight decay $3×10^-4$, while we use Adam [ref] as the optimizer for $α$ with initial learning rate$η_α= 3×10^-4$, momentum $β= (0.5, 0.999)$and weight decay$0.001$, and cutout [ref] status is True.
Improved Regularization of Convolutional Neural Networks with Cutout · Implementation Details · 1.00
Cites as background 4 papers
-
After Google [ref] and MIT [ref] released NAS based on reinforcement learning in 2016, through improvements in search space [ref] , search strategies [ref] , and evaluation methods [ref] in recent years, NAS algorithms have gradually reduced the demand for GPUs and become more "civilized", enabling more people to have opportunities to provide new idea which promote the development of automated machine learning.
Neural Architecture Search with Reinforcement Learning · Introduction · 0.97
-
After Google [ref] and MIT [ref] released NAS based on reinforcement learning in 2016, through improvements in search space [ref] , search strategies [ref] , and evaluation methods [ref] in recent years, NAS algorithms have gradually reduced the demand for GPUs and become more "civilized", enabling more people to have opportunities to provide new idea which promote the development of automated machine learning.
Efficient Neural Architecture Search via Parameter Sharing · Introduction · 0.97
-
After Google [ref] and MIT [ref] released NAS based on reinforcement learning in 2016, through improvements in search space [ref] , search strategies [ref] , and evaluation methods [ref] in recent years, NAS algorithms have gradually reduced the demand for GPUs and become more "civilized", enabling more people to have opportunities to provide new idea which promote the development of automated machine learning.
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks · Introduction · 0.97
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.