Neural Architecture Search: Insights from 1000 Papers
Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel pith:QQGR4IDRrecord.jsonopen to challenge →
read the original abstract
In the past decade, advances in deep learning have resulted in breakthroughs in a variety of areas, including computer vision, natural language understanding, speech recognition, and reinforcement learning. Specialized, high-performing neural architectures are crucial to the success of deep learning in these areas. Neural architecture search (NAS), the process of automating the design of neural architectures for a given task, is an inevitable next step in automating machine learning and has already outpaced the best human-designed architectures on many tasks. In the past few years, research in NAS has been progressing rapidly, with over 1000 papers released since 2020 (Deng and Lindauer, 2021). In this survey, we provide an organized and comprehensive guide to neural architecture search. We give a taxonomy of search spaces, algorithms, and speedup techniques, and we discuss resources such as benchmarks, best practices, other surveys, and open-source libraries.
This paper has not been read by Pith yet.
Forward citations
Cited by 14 Pith papers
-
Convergence Theory for Iterative LLM-Based Neural Architecture Search: A Parametric Cross-Entropy Framework with Closed-Form Proxy Reliability
Iterative LLM-NAS is equivalent to a parametric cross-entropy method with proven monotonic quality improvement, geometric convergence of elite probability, and a closed-form proxy reliability rho_S = (6/pi) arcsin(rho...
-
From Regression to Inference: Meta-Learning Predictors for Neural Architecture Search
Meta-learning a Convolutional Neural Process to infer neural architecture performance from context-target splits on synthesized tasks improves top-K ranking and achieves state-of-the-art selection on NAS-Bench-101 and...
-
Zero-Shot Neural Network Evaluation with Sample-Wise Activation Patterns
SWAP-Score evaluates neural networks without training by quantifying sample-wise activation patterns, achieving high correlation with true performance on CIFAR-10 for CNNs and GLUE for Transformers while enabling fast NAS.
-
Delta-Based Neural Architecture Search: LLM Fine-Tuning via Code Diffs
Fine-tuned 7B LLMs generating unified diffs for neural architecture refinement achieve 66-75% valid rates and 64-66% mean first-epoch accuracy, outperforming full-generation baselines by large margins while cutting ou...
-
Self-Directed Task Identification
SDTI lets models identify the correct target variable in datasets in a zero-shot setting using standard neural networks, beating baselines by 14% F1 on synthetic benchmarks.
-
Q-PhotoNAS: Hybrid Quantum Neural Architecture Search Framework on Photonic Devices
Q-PhotoNAS applies genetic algorithm search to jointly optimize classical preprocessing, phase encoding, and photonic circuit structure for hybrid quantum-classical models, reporting 99.44% and 98.78% accuracy on Digi...
-
OPT-BENCH: Evaluating the Iterative Self-Optimization of LLM Agents in Large-Scale Search Spaces
OPT-BENCH and OPT-Agent evaluate LLM self-optimization in large search spaces, showing stronger models improve via feedback but stay constrained by base capacity and below human performance.
-
Evolving Many Worlds: Towards Open-Ended Discovery in Petri Dish NCA via Population-Based Training
PBT-NCA evolves PD-NCAs via a composite novelty-diversity objective to generate sustained emergent lifelike behaviors including waves, spore scattering, and migrating macro-structures at the edge of chaos.
-
Defending against Backdoor Attacks via Module Switching
Module-switching defense disrupts backdoors more effectively than weight averaging with fewer models and remains robust even when some models share the same backdoors.
-
On Efficient Variants of Segment Anything Model: A Survey
A survey that reviews efficient variants of the Segment Anything Model, categorizes acceleration strategies, and provides a unified hardware evaluation on benchmarks.
-
Running hardware-aware neural architecture search on embedded devices under 512MB of RAM
A HW-NAS framework executable on resource-limited embedded devices generates optimized CNNs for low-end MCUs and reports state-of-the-art human-recognition accuracy on the Visual Wake Word dataset.
-
Universal Time-Series Representation Learning: A Survey
A survey that proposes a taxonomy for universal time-series representation learning and reviews existing deep learning studies along with experimental setups.
-
Neural Architecture Search for Generative Adversarial Networks: A Comprehensive Review and Critical Analysis
A literature review that categorizes and compares NAS techniques for GANs, noting benefits of evolutionary and gradient-based methods along with needs for better metrics and diverse datasets.
-
Spiking Neural Network Architecture Search: A Survey
A survey of Spiking Neural Network architecture search techniques viewed through a hardware/software co-design lens.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.