Robust Evolutionary Multi-Objective Network Architecture Search for Reinforcement Learning (EMNAS-RL)

Alexandra Gianzina; Andreas Ebert; Hanno Gottschalk; Nihal Acharya Adde

REVIEW 4 major objections 6 minor 12 references

Robust Evolutionary Multi-Objective Network Architecture Search for Reinforcement Learning (EMNAS-RL)

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims a genetic search over neural-network designs beats a hand-built autonomous-driving agent in simulation.

desk verdict The 4% over a manual baseline rests on an untested 20-epoch ranking shortcut, and the OTL-vs-EMNAS comparison conflates parallelization with transfer learning; the paper is a reasonable engineering starting point, not a demonstrated advance. read the letter →

arxiv 2506.08533 v1 pith:4ZT5NTHV submitted 2025-06-10 cs.LG cs.AImath.OC

Nihal Acharya Adde , Alexandra Gianzina , Hanno Gottschalk , Andreas Ebert This is my paper

classification cs.LGcs.AImath.OC

keywords neuralarchitecturesearchevolutionaryalgorithmmulti-objectiveoptimizationreinforcementlearningautonomousdrivingteacher-studenttransferproximalpolicy

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper applies evolutionary multi-objective neural architecture search (NAS) for the first time to a large-scale reinforcement-learning task in autonomous driving. The search evolves convolutional network designs for a PPO driving policy, ranking candidates by reward, parameter count, and FLOPs, and a teacher-student module transfers the previous generation's best policy to the next generation. The authors report that the fully retrained winning architecture reached a peak cumulative reward of 1190, a 4% improvement over the manually set architecture's 1140, and that the transfer-learning variant improves stability across generations. The value of the claim is that it suggests architecture design for complex RL agents can be automated and can outperform human design.

What carries the argument

The load-bearing mechanism is a multi-objective evolutionary loop built on NSGA-II: candidate architectures are encoded as chromosomes of operations within alternating normal and reduction cells, and each individual is ranked on negative reward, FLOPs, and parameter count. An early-exit population initialization keeps initial architectures below a parameter threshold beta, while the Optimized Transfer Learning module pre-trains each new generation via behavior cloning on 12,000 state-action pairs collected from the previous generation's best network, followed by PPO fine-tuning. To keep the search affordable, architectures are ranked after only 20 PPO epochs instead of the full 300-epoch training run.

What would settle it

Take the architectures that ranked lowest after 20 PPO epochs during the search, fully retrain each for 300 epochs, and compare final cumulative rewards to the winner's 1190; if several low-ranked architectures finish at or above 1190, the 20-epoch ranking is not predictive of full-training performance.

Watch

Extended reading notes

Core claim

The paper's central claim is that multi-objective evolutionary NAS, combined with optimized transfer learning, can automatically discover convolutional network architectures for an RL-based autonomous-driving policy that outperform manually designed architectures while using fewer parameters. In their experiments the best searched architecture, after full 300-epoch retraining, achieved a peak total cumulative reward of 1190 versus 1140 for the manual baseline, and the teacher-student transfer method produced higher median rewards and tighter reward distributions than evolution without transfer. The authors also report that parallel training on four GPUs reduced search runtime by a factor of two to three.

Load-bearing premise

The paper ranks architectures after only 20 PPO epochs and assumes this ordering predicts performance after the full 300-epoch training; if the low-fidelity ranking does not match full-training rankings, the winning architecture is arbitrary and the reported 4% advantage collapses.

Editorial extensions

If this is right

If the central claim holds, network architecture selection for RL policies in autonomous driving can be automated rather than hand-tuned, with the search returning smaller models.
The teacher-student transfer step should make successive generations cheaper and more stable, since each population begins from the previous best policy instead of from scratch.
The four-GPU parallel training scheme makes multi-objective NAS practical at the population and generation scales used, reducing search time by a factor of two to three.
The low-fidelity 20-epoch ranking, if predictive, could be reused to screen architectures for other RL tasks where full training is prohibitively expensive.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

My inference: the single 4% gain comes from one retrained run, not from a distribution of runs, so the more defensible statistical claim in the paper is the improvement in median reward and reduced variability from OTL.
My inference: the teacher-student transfer likely provides the largest benefit in early generations, when populations are far from the previous best policy; the paper does not report per-generation learning curves, so this is directly testable.
My inference: if the 20-epoch ranking transfers to other domains, the same evolutionary search could be applied to vision-based manipulation or navigation tasks, but the fidelity assumption would need to be revalidated for each new simulator.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

Short version: this paper applies a known MOEA-NAS recipe (NSGA-II + early-exit initialization, from EEEANet) to RL for autonomous driving, adds a teacher-student warm start between generations, and reports a 4% reward gain over one manual architecture. The engineering story is clear and the writing is honest, but the headline result is not established by the evidence as presented.

What is actually new: no one seems to have run multi-objective evolutionary NAS inside an RL loop for autonomous driving before, and the OTL transfer scheme—behavior cloning from the previous generation's best policy to all current students—is a sensible way to reuse computation across generations. The paper also reports concrete compute costs (GPU days), which is more than most NAS papers do. For that, it deserves credit.

The soft spots, in order of size. First, the decisive assumption: architectures are ranked after 20 PPO epochs, then the winner is fully retrained for 300 epochs and compared to a manual baseline. The paper states this lower-fidelity choice, cites a prior experiment [12] for it, but never provides a correlation between 20-epoch fitness and 300-epoch reward. Without that, the winning architecture could be the one that got lucky in the first 20 epochs. This is a structural gap, not a nitpick. Second, Table 2 compares plain EMNAS (serial) against OTL with parallelization (OTL(P)). The 2-3x runtime gain is clearly from parallelization, and any stability/median-reward differences are confounded by the parallel training scheme. The claim that OTL itself helps is therefore not cleanly tested. Third, the final 1190-vs-1140 comparison appears to be a single run per architecture, with no repeated-seed statistics. For a 4% effect, you need more than one trial. Fourth, no code, no config files, no data release, and no independent benchmark, so the numbers are not externally checkable.

The citation pattern is fine. The self-citation to the prior experiment is reasonable. The math isn't deep, but nothing circular.

Who should read it: people working on NAS for RL or on autonomous driving at OEMs, who want a starting point for evolutionary architecture search with transfer. It is not a paper that settles anything about whether low-fidelity NAS rankings transfer to full training.

I would send it to a serious referee: the application is timely, and the referee can ask for the missing rank-correlation analysis and repeated runs. Without that, the paper is a workshop-quality engineering report.

Recommendation: engage, but only with those revisions.

Referee Report

4 major / 6 minor

Summary. The manuscript proposes EMNAS-RL, an evolutionary multi-objective network architecture search method for reinforcement learning in an autonomous driving simulator. The search space uses convolutional normal/reduction cells with operations such as depthwise separable and dilated convolutions; the fitness function combines reward, FLOPS, and parameter count; and an early-exit population initialization threshold β restricts the initial population. Two enhancements are introduced: Optimized Transfer Learning (OTL), where the previous generation's best network provides behavior-cloning data for the next generation, and parallel training on four V100 GPUs. Experiments in Table 1 vary β, Table 2 compares EMNAS against OTL under parallelization across six population/generation settings, and the best architecture from Table 2 is retrained for 300 PPO iterations, yielding a reported reward of 1190, about 4% above a manually designed architecture (1140). The paper concludes that EMNAS "consistently outperform[s] manually designed architectures" and that OTL improves stability.

Significance. If the empirical claims were supported, this would be a useful applied contribution: it demonstrates a way to make multi-objective evolutionary NAS tractable for large-scale RL in autonomous driving, and it proposes a transfer-learning mechanism (behavior cloning from the previous generation's champion) plus parallel training that other RL applications could adopt. The multi-objective treatment of reward, FLOPS, and parameters is sensible, and the EEPI threshold is a practical device for constraining an expensive search space. However, the central claims are not yet established: the OTL comparison is confounded with parallelization, the low-fidelity 20-epoch ranking is not validated against 300-epoch performance, and the final manual-baseline comparison lacks statistical detail. The paper currently reads as a promising proof-of-concept rather than a fully supported empirical study.

major comments (4)

[Section 4, Table 2] The comparison between EMNAS and OTL(P) is confounded because EMNAS is reported without parallelization while OTL(P) is reported with four GPUs. Consequently, the 2-3x runtime reduction and the stability improvements (higher medians in most rows) cannot be attributed to the OTL transfer-learning mechanism; they might be due entirely to parallelization. A controlled comparison should include at least EMNAS with parallelization and OTL without parallelization, ideally in a 2x2 experimental design.
[Section 4, final retraining paragraph] The headline result, a 4% reward increase (1190 vs. 1140) over a manually designed architecture, is not supported by the evidence as presented. The manual baseline is not described (architecture, PPO hyperparameters, training length, number of runs), the winning model is retrained once, and no variance or repeated-seed results are reported. It is also unclear whether the reported "peak total cumulative reward" is the same scalar used during the search or a different evaluation metric. A credible comparison requires matched training budgets, multiple seeds, and error bars.
[Section 3, low-fidelity ranking; Section 4, championship retraining] The search ranks architectures after 20 PPO epochs, whereas the selected champion is evaluated after 300 epochs, and the paper provides no evidence that 20-epoch rankings predict 300-epoch performance. Because mutation, selection, and survival decisions all rely on the low-fidelity estimates, a weak rank correlation would make the selected champion effectively arbitrary. The authors state that they use "learning curve extrapolation" and "relative ranking to mitigate estimation bias," but no extrapolation model or correlation analysis is described. They should report a rank-correlation study between low- and full-fidelity scores, or retrain several top- and mid-ranked architectures, to verify that the search objective tracks the final claim.
[Section 4, Table 2 and Figure 6] The claim that "OTL consistently achieves higher median rewards" is contradicted by the 15-population/10-generation row, where OTL(P) has a lower median (321) than EMNAS (334). The further statement that OTL outperforms EMNAS "60% of the time when comparing maximum rewards" is also a weak descriptive summary: in Table 2, OTL(P) has a higher maximum in four of six rows, not a statistically supported advantage. The percentile-based evidence needs repeated trials or confidence intervals before stability claims can be made.

minor comments (6)

[Section 3, Eq. (1)] The fitness expression min{-Reward(x), FLOPS(x), Params(x)} is ambiguous as a multi-objective operation; the later sentence about normalization and equal weighting is not reflected in the equation. Please clarify how NSGA-II combines or sorts these objectives.
[Section 4, Table 1] The threshold β is selected from only two runs per configuration, and the reward differences (e.g., 452 vs. 482) are within plausible stochastic noise. Treating this as a definitive choice of β=5 is under-justified.
[Section 3, OTL description] The hyperparameter decay schedule for learning rate, PPO clip, and entropy is not specified, making the OTL method hard to reproduce. Please define the decay factors and at which generations they are applied.
[Section 3, behavior cloning] The 12,000 state-action pairs are mentioned, but their composition, collection policy, and the BC training procedure (number of epochs, optimizer, loss) are not described; these details are needed to assess the teacher-student mechanism.
[Section 5, Conclusion] The phrase "consistently outperform manually designed architectures" is stronger than the evidence, since only one searched architecture is fully retrained and the manual baseline is not specified. The conclusion should be qualified.
[General] No code or data availability statement is provided; given the use of a proprietary Unity-based simulator, a statement on what artifacts can be shared would improve reproducibility.

Circularity Check

0 steps flagged · score 2.0 of 10

No structural circularity; only a minor self-citation informs the 20-epoch ranking heuristic, while the final retrained comparison is an independent empirical result.

full rationale

The central claim is an empirical comparison: the searched architecture is fully retrained for 300 PPO iterations and reaches reward 1190 versus 1140 for the manual baseline. This outcome is not computed from the search fitness function or from any fitted parameter; it is measured after full training. The evolutionary objectives in Eq. (1), the NSGA-II selection, the EEPI threshold choice, and the OTL behavior-cloning procedure are all operational mechanisms, not definitions that encode the final reward. The only self-referential element is the citation of the authors' prior work [12] to justify ranking architectures after 20 epochs instead of 300. This is a tuning heuristic used during search, and it is not load-bearing for the final 300-epoch retrained comparison in the sense of reducing to the paper's own input. The absence of a correlation analysis between 20-epoch rankings and 300-epoch performance is a genuine validity risk, but it is a structural assumption about benchmark fidelity, not a circular derivation: the final 1190 value is not contained in the 20-epoch fitness values by construction. No uniqueness theorem, ansatz smuggling, or re-labeling of a known result forces the outcome. Therefore no specific circular step is identified, and the paper receives a low score reflecting only the minor self-citation. Central claims remain independently testable by external evaluation.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claim rests on proxy evaluations, transfer-dataset sufficiency, hyperparameter choices, and an underspecified manual baseline, rather than on a mathematical derivation. Six hyperparameters are fixed without sensitivity analysis, and the low-fidelity search protocol is assumed to preserve rankings. No new physical entities are introduced.

free parameters (6)

EEPI threshold beta = 5 (millions of parameters)
Selected from Table 1 because beta=5 gave higher reward and lower parameter/FLOPS than beta=3 on small searches; this is hyperparameter tuning on the reported experiments.
Low-fidelity ranking cutoff = 20 PPO epochs (versus 300)
Used to rank architectures; justified only by the authors' earlier observation [12], not by a validation study within this paper.
Behavior cloning dataset size = 12,000 state-action pairs per generation
No ablation or coverage analysis is provided; the OTL result depends on this fixed dataset being sufficient.
Mutation probability = 0.1
Chosen as a fixed hyperparameter without sensitivity analysis.
Crossover probability = 0.5-0.9 (randomized)
Chosen as a hyperparameter range without sensitivity analysis.
Survival probability = 0.2
Controls how many previous individuals survive; no sensitivity analysis.

assumptions (5)

domain assumption The 20-epoch low-fidelity training preserves the ranking of architectures over full 300-epoch training
The search selects the winning model on this proxy (Section 3 under EMNAS-RL Methodology). If the proxy ranking is wrong, the reported final reward gain is not reliable.
domain assumption Behavior cloning from 12,000 saved state-action pairs of the previous best network transfers a useful and unbiased policy to all students
OTL pre-trains every next-generation network on this dataset (Section 3, Optimized Transfer Learning); dataset coverage and expert quality are not analyzed.
domain assumption The Unity simulator reward function is a faithful measure of driving quality and is comparable across architectures
Reward is the primary objective in Eq. (1), and the manual baseline reward of 1140 is compared against it, but the reward function is not specified in detail.
domain assumption The manually designed baseline architecture was trained under conditions comparable to the searched winner
The final comparison reports 1190 versus 1140 (Section 4) without giving the baseline architecture, hyperparameters, or number of runs.
ad hoc to paper The lower-fidelity configuration (84x84x3 input, 4 stacked cells, 4 blocks per cell, 16 initial channels) preserves relative architecture quality
These reductions are introduced to accelerate the search (Section 3) and are not validated against full-fidelity rankings.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Robust Evolutionary Multi-Objective Network Architecture Search for Reinforcement Learning (EMNAS-RL)." pith.science (2026). https://pith.science/paper/4ZT5NTHV

@misc{pith2026250608533,
  author       = {Pith},
  title        = {Pith review of: Robust Evolutionary Multi-Objective Network Architecture Search for Reinforcement Learning (EMNAS-RL)},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4ZT5NTHV}},
  note         = {Machine review of arXiv:2506.08533}
}

read the original abstract

This paper introduces Evolutionary Multi-Objective Network Architecture Search (EMNAS) for the first time to optimize neural network architectures in large-scale Reinforcement Learning (RL) for Autonomous Driving (AD). EMNAS uses genetic algorithms to automate network design, tailored to enhance rewards and reduce model size without compromising performance. Additionally, parallelization techniques are employed to accelerate the search, and teacher-student methodologies are implemented to ensure scalable optimization. This research underscores the potential of transfer learning as a robust framework for optimizing performance across iterative learning processes by effectively leveraging knowledge from earlier generations to enhance learning efficiency and stability in subsequent generations. Experimental results demonstrate that tailored EMNAS outperforms manually designed models, achieving higher rewards with fewer parameters. The findings of these strategies contribute positively to EMNAS for RL in autonomous driving, advancing the field toward better-performing networks suitable for real-world scenarios.

Figures

Figures reproduced from arXiv: 2506.08533 by the authors.

**Figure 2.** An illustration of the ENAS algorithm highlighting its three main phases: [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Multi-objective Evolutionary Algorithm with EEPI. Illustration of the [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: A flowchart illustrating the primary logic of TL, showcasing its integra [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

**Figure 5.** Figure 5: Evolution of rewards and number of model parameters during the archi [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

**Figure 6.** Figure 6: Reward results presented as box plots for EMNAS (yellow) and OTL [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

**Figure 7.** Figure 7: Normal and reduction cell structure Lastly, as seen in [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

12 extracted references · 10 canonical work pages

[12]

Hyperparameter optimiza- tion for ˆA driving strategies based on ˆA reinforcement learning

Nihal Acharya Adde, Hanno Gottschalk, and Andreas Ebert. Hyperparameter optimiza- tion for ˆA driving strategies based on ˆA reinforcement learning. In Giuseppe Nicosia, Varun Ojha, Sven Giesselbach, M. Panos Pardalos, and Renato Umeton, editors, Ma- chine Learning, Optimization, and Data Science , pages 24–38, Cham, 2025. Springer Nature Switzerland

work page 2025
[1]

Regularized evolution for image classifier architecture search

Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference on artificial intelligence, volume 33, pages 4780–4789, 2019

2019
[2]

Eeea-net: An early exit evolutionary neural architecture search

Chakkrit Termritthikun, Yeshi Jamtsho, Jirarat Ieamsaard, Paisarn Muneesawang, and Ivan Lee. Eeea-net: An early exit evolutionary neural architecture search. Engineering Applications of Artificial Intelligence , 104:104397, 2021

work page 2021
[3]

Nsga-net: NAS using multi-objective genetic algorithm

Zhichao Lu, Ian Whalen, Vishnu Boddeti, Yashesh Dhebar, Kalyanmoy Deb, Erik Good- man, and Wolfgang Banzhaf. Nsga-net: NAS using multi-objective genetic algorithm. In Proceedings of the genetic and evolutionary computation conference , 2019

work page 2019
[4]

Behavioral cloning from observation

Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. pages 4950–4957, 07 2018

work page 2018
[5]

https://unity.com/

Unity3D. https://unity.com/. Accessed: 2023-03-04

work page 2023
[6]

Proxi- mal policy optimization algorithms

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proxi- mal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017

arXiv 2017
[7]

Evolu- tionary Neural Network Architecture Search, pages 247–281

Zeqiong Lv, Xiaotian Song, Yuqi Feng, Yuwei Ou, Yanan Sun, and Mengjie Zhang. Evolu- tionary Neural Network Architecture Search, pages 247–281. Springer Nature Singapore, Singapore, 2024

work page 2024

Show all 12 references

[8]

Yen, and Jiancheng Lv

Yanan Sun, Bing Xue, Mengjie Zhang, Gary G. Yen, and Jiancheng Lv. Automatically designing cnn architectures using the genetic algorithm for image classification. IEEE Transactions on Cybernetics, 50(9):3840–3854, 2020

2020
[9]

Evolving neural networks through augment- ing topologies

Kenneth O Stanley and Risto Miikkulainen. Evolving neural networks through augment- ing topologies. Evolutionary computation, 10(2):99–127, 2002

2002
[10]

A fast and elitist multiobjective genetic algorithm: NSGA-II

Kalyanmoy Deb, Amrit Pratap, Sameer Agarwal, and TAMT Meyarivan. A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE transactions on evolutionary computation, 6(2):182–197, 2002

2002
[11]

Evolutionary computation and conver- gence to a pareto front

David A Van Veldhuizen, Gary B Lamont, et al. Evolutionary computation and conver- gence to a pareto front. In genetic programming conference, pages 221–228, 1998

1998

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [12]

Hyperparameter optimiza- tion for ˆA driving strategies based on ˆA reinforcement learning

Nihal Acharya Adde, Hanno Gottschalk, and Andreas Ebert. Hyperparameter optimiza- tion for ˆA driving strategies based on ˆA reinforcement learning. In Giuseppe Nicosia, Varun Ojha, Sven Giesselbach, M. Panos Pardalos, and Renato Umeton, editors, Ma- chine Learning, Optimization, and Data Science , pages 24–38, Cham, 2025. Springer Nature Switzerland

work page 2025

[2] [1]

Regularized evolution for image classifier architecture search

Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference on artificial intelligence, volume 33, pages 4780–4789, 2019

2019

[3] [2]

Eeea-net: An early exit evolutionary neural architecture search

Chakkrit Termritthikun, Yeshi Jamtsho, Jirarat Ieamsaard, Paisarn Muneesawang, and Ivan Lee. Eeea-net: An early exit evolutionary neural architecture search. Engineering Applications of Artificial Intelligence , 104:104397, 2021

work page 2021

[4] [3]

Nsga-net: NAS using multi-objective genetic algorithm

Zhichao Lu, Ian Whalen, Vishnu Boddeti, Yashesh Dhebar, Kalyanmoy Deb, Erik Good- man, and Wolfgang Banzhaf. Nsga-net: NAS using multi-objective genetic algorithm. In Proceedings of the genetic and evolutionary computation conference , 2019

work page 2019

[5] [4]

Behavioral cloning from observation

Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. pages 4950–4957, 07 2018

work page 2018

[6] [5]

https://unity.com/

Unity3D. https://unity.com/. Accessed: 2023-03-04

work page 2023

[7] [6]

Proxi- mal policy optimization algorithms

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proxi- mal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017

arXiv 2017

[8] [7]

Evolu- tionary Neural Network Architecture Search, pages 247–281

Zeqiong Lv, Xiaotian Song, Yuqi Feng, Yuwei Ou, Yanan Sun, and Mengjie Zhang. Evolu- tionary Neural Network Architecture Search, pages 247–281. Springer Nature Singapore, Singapore, 2024

work page 2024