RECORD
Neural Architecture Transfer 2: A Paradigm for Improving Efficiency in Multi-Objective Neural Architecture Search
Machine Learning · Submitted 3 Jul 2023
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim Experimental results show that NATv2 successfully improves NAT and is highly recommended for investigating high-performance architectures with a minimal number of parameters.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
Deep learning is increasingly impacting various aspects of contemporary society. Artificial neural networks have emerged as the dominant models for solving an expanding range of tasks. The introduction of Neural Architecture Search (NAS) techniques, which enable the automatic design of task-optimal networks, has led to remarkable advances. However, the NAS process is typically associated with long execution times and significant computational resource requirements. Once-For-All (OFA) and its successor, Once-For-All-2 (OFAv2), have been developed to mitigate these challenges. While maintaining exceptional performance and eliminating the need for retraining, they aim to build a single super-network model capable of directly extracting sub-networks satisfying different constraints. Neural Architecture Transfer (NAT) was developed to maximise the effectiveness of extracting sub-networks from a super-network. In this paper, we present NATv2, an extension of NAT that improves multi-objective search algorithms applied to dynamic super-network architectures. NATv2 achieves qualitative improvements in the extractable sub-networks by exploiting the improved super-networks generated by OFAv2 and incorporating new policies for initialisation, pre-processing and updating its networks archive. In addition, a post-processing pipeline based on fine-tuning is introduced. Experimental results show that NATv2 successfully improves NAT and is highly recommended for investigating high-performance architectures with a minimal number of parameters.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | Deep learning is increasingly impacting various aspects of contemporary society. | background 1.00 | 0.03 | 0.01 | asserted 1.00 |
| s1 | Artificial neural networks have emerged as the dominant models for solving an expanding range of tasks. | background 1.00 | 0.04 | 0.01 | asserted 1.00 |
| s2 | The introduction of Neural Architecture Search (NAS) techniques, which enable the automatic design of task-optimal networks, has led to remarkable advances. | background 1.00 | 0.05 | 0.02 | asserted 1.00 |
| s3 | However, the NAS process is typically associated with long execution times and significant computational resource requirements. | background 0.91 | 0.08 | 0.01 | asserted 0.99 |
| s4 | Once-For-All (OFA) and its successor, Once-For-All-2 (OFAv2), have been developed to mitigate these challenges. | background 0.99 | 0.08 | 0.02 | asserted 1.00 |
| s5 | While maintaining exceptional performance and eliminating the need for retraining, they aim to build a single super-network model capable of directly extracting sub-networks satisfying different constraints. | background 0.85 | 0.08 | 0.03 | asserted 0.92 |
| s6 | Neural Architecture Transfer (NAT) was developed to maximise the effectiveness of extracting sub-networks from a super-network. | background 0.98 | 0.13 | 0.03 | asserted 1.00 |
| s7 | In this paper, we present NATv2, an extension of NAT that improves multi-objective search algorithms applied to dynamic super-network architectures. | contribution 1.00 | 0.88 | 0.10 | asserted 1.00 |
| s8 | NATv2 achieves qualitative improvements in the extractable sub-networks by exploiting the improved super-networks generated by OFAv2 and incorporating new policies for initialisation, pre-processing and updating its networks archive. | method 0.65 | 0.84 | 0.17 | asserted 1.00 |
| s9 | In addition, a post-processing pipeline based on fine-tuning is introduced. | method 0.72 | 0.88 | 0.04 | asserted 0.99 |
| s10 | Experimental results show that NATv2 successfully improves NAT and is highly recommended for investigating high-performance architectures with a minimal number of parameters. | result 0.81 | 0.91 | 0.86 | asserted 0.99 |
What was examined, and what was not
- arXiv listing abstract v1
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
Deep learning is increasingly impacting various aspects of contemporary society. Artificial neural networks have emerged as the dominant models for solving an expanding range of tasks. The introduction of Neural Architecture Search (NAS) techniques, which enable the automatic design of task-optimal networks, has led to remarkable advances. However, the NAS process is typically associated with long execution times and significant computational resource requirements. Once-For-All (OFA) and its successor, Once-For-All-2 (OFAv2), have been developed to mitigate these challenges. While maintaining exceptional performance and eliminating the need for retraining, they aim to build a single super-network model capable of directly extracting sub-networks satisfying different constraints. Neural Architecture Transfer (NAT) was developed to maximise the effectiveness of extracting sub-networks from a super-network. In this paper, we present NATv2, an extension of NAT that improves multi-objective search algorithms applied to dynamic super-network architectures. NATv2 achieves qualitative improvements in the extractable sub-networks by exploiting the improved super-networks generated by OFAv2 and incorporating new policies for initialisation, pre-processing and updating its networks archive. In addition, a post-processing pipeline based on fine-tuning is introduced. Experimental results show that NATv2 successfully improves NAT and is highly recommended for investigating high-performance architectures with a minimal number of parameters.
Human review
How the literature uses it
2 arXiv papers cite this in 2 sentences
Papers citing it
Cites it as background 2 papers
-
Methods such as POPNASv3 [ref] , NAT [ref] , and NATv2 [ref] offer Pareto-optimal solutions, addressing scalability and computational challenges inherent in multi-objective optimisation.
POMONAG: Pareto-Optimal Many-Objective Neural Architecture Generator · Related Works · 0.86
-
Similarly, Sarti et al. expand these ideas with Neural Architecture Transfer 2, a paradigm designed to enhance efficiency in multi-objective NAS [ref] .
ZO-DARTS++: An Efficient and Size-Variable Zeroth-Order Neural Architecture Search Algorithm · Flexible Neural Architecture Search Methods · 0.80
What it cites · 4 sentences without a confident label
Builds on 1 paper
-
The authors maintained the same underlying training principle as the original algorithm but adapted it to a network search space which was extended with proven techniques from the field of artificial neural network design, thereby elevating the obtained super-network to higher levels [ref] .
Enhancing Once-For-All: A Study on Parallel Blocks, Skip Connections and Early Exits · Introduction · 0.83
Uses the method or tool of 4 papers
-
On the other hand, the second post-processing method utilizes the Anticipate, Ensemble and Prune (AEP) technique [ref] .
Anticipate, Ensemble and Prune: Improving Convolutional Neural Networks via Aggregated Early Exits · Post-Processing · 0.99
-
To explore different post-processing combinations for single exit networks and early exit networks, the following optimisers were tested: SGD, AdamW [ref] , and Ranger (the combination of the LookAhead [ref] and RAdam [ref] optimisers).
Decoupled Weight Decay Regularization · Experiments Configuration · 1.00
-
To explore different post-processing combinations for single exit networks and early exit networks, the following optimisers were tested: SGD, AdamW [ref] , and Ranger (the combination of the LookAhead [ref] and RAdam [ref] optimisers).
On the Variance of the Adaptive Learning Rate and Beyond · Experiments Configuration · 1.00
Cites as background 2 papers
-
As the name suggests, the objective of this approach is to perform massive computations in a single instance by constructing a super-network, from which sub-networks satisfying different constraints can be readily extracted, while maintaining excellent performance [ref] .
Once-for-All: Train One Network and Specialize it for Efficient Deployment · Introduction · 0.88
-
This new version introduces significant improvements, particularly in the search space [ref] .
Enhancing Once-For-All: A Study on Parallel Blocks, Skip Connections and Early Exits · Once-For-All-2 · 0.89
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.