REVIEW 3 major objections 6 minor 34 references
OrchNAS claims a server-side NAS service can design per-device subnets that meet energy, compute, and memory budgets while improving accuracy.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-01 04:41 UTC pith:R5K3Z2N4
load-bearing objection A genuinely integrated energy-aware federated NAS framework, but the energy numbers are computed from an unvalidated FLOPs proxy and the main theorem is more asserted than proven. the 3 major comments →
OrchNAS: Orchestrated Neural Architecture Search Service for Personalised Federated Edge Intelligence
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The paper's central claim is that energy consumption in federated NAS can be made a first-class constraint rather than an afterthought. Energy per training round is modeled as E_k(a_k) = α_k F(a_k) T_k, so it is proportional to FLOPs and local steps. Architectures are scored as validation accuracy minus an energy penalty; each service greedily removes the operation with the lowest accuracy-loss-per-energy-saved score until its constraints hold; and a non-negative dual variable is raised whenever energy exceeds budget, making the energy constraint a saddle-point problem. Theorem 1 states that, under convexity and bounded energy, the average energy-budget violation over time is asymptotically
What carries the argument
The machinery is a three-level energy-aware design: (1) an evolutionary, energy-scored global architecture search on the server, (2) progressive greedy edge-wise pruning per service using score ΔAcc - βΔE until FLOPs, params, and energy constraints are feasible, and (3) a primal-dual Lagrangian personalisation step whose dual variable η_k is updated by projected gradient ascent on energy violation. The load-bearing identity is the energy model E = αFT, which turns architecture choices into a scalar energy cost, and the saddle-point formulation that converts the energy constraint into a penalty.
Load-bearing premise
The entire energy-aware pipeline treats energy as a linear function of FLOPs and local training steps, ignoring memory access, communication, idle power, and hardware-specific efficiency; if this model is not faithful to real device energy, the energy claims describe a synthetic quantity rather than physical battery consumption.
What would settle it
Run OrchNAS on two devices with identical FLOPs and local steps but different memory bandwidth or idle power, and compare actual battery drain with the model's predicted energy. If measured energy differs substantially while predicted energy is equal, the energy-aware scores and pruning decisions are optimizing a synthetic quantity, not physical energy.
If this is right
- Federated learning services can receive architectures tailored to their current battery, compute, and memory budgets without manual design or on-device search.
- The primal-dual update gives a simple distributed mechanism for enforcing resource constraints: each device only needs its own energy estimate and a scalar dual variable.
- Since architecture search runs centrally on the NAS service, the per-device cost of personalisation is limited to pruning and local fine-tuning, improving scalability.
- A single coefficient λ controls the accuracy-energy trade-off, letting a fleet operator dial in the desired operating point.
- Dynamic device states (budgets that change per round) are handled naturally, as pruning and dual updates re-run each round.
Where Pith is reading between the lines
- The energy model E = αFT treats energy as linear in FLOPs and local steps; memory access, communication, idle power, and hardware-specific efficiency are not modelled, so reported energy savings may be synthetic unless the proxy is validated on real hardware.
- Theorem 1's proof is deferred to standard saddle-point optimisation analysis without stating the required assumptions beyond convexity and Lipschitz continuity; applying the O(ρ) bound to non-convex deep networks is a leap, and the empirical 4.8% violation rate is the more concrete evidence.
- The same primal-dual skeleton could enforce other per-device constraints, such as latency or memory, by swapping the cost function; a direct extension would be to add a latency penalty and test whether the violation bound and accuracy-energy trade-off transfer.
- A hardware-in-the-loop benchmark comparing predicted energy (αFT) against actual battery drain on a few device classes would clarify whether the accuracy-energy Pareto improvements persist physically.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes OrchNAS, a server-side NAS-as-a-service framework for personalized federated learning on heterogeneous edge devices. It defines an energy model E_k(a_k) = α_k F(a_k) T_k, uses evolutionary search with an energy-aware score to select a global architecture, derives per-service subnets by progressive greedy energy-aware pruning, and enforces energy budgets through a primal-dual update. Experiments on five datasets compare against SPIDER, PerFedRLNAS, diffusion-based NAS, Green-EDP, and MCFL, reporting higher accuracy and lower energy, with additional ablations, scalability, and sensitivity analyses. The public code link and breadth of experiments are positive features, but the energy claims rest entirely on an unvalidated surrogate model, and the theoretical guarantee is stated without a matching proof.
Significance. If the energy model were independently validated, the orchestrated NAS-service idea would be a useful practical contribution to personalized federated edge intelligence. The paper is broad: five datasets, several NAS/PFL baselines, component-wise ablations, scalability experiments, and publicly available code. The accuracy improvements over baselines (e.g., +1.1% to +2.2% with lower FLOPs in Table I) are independent of the energy-proxy issue and are the strongest part of the empirical contribution. However, the central energy-efficiency claim is currently an artifact of the paper's own energy definition: all reported Joules, energy reductions, feasibility rates, and violation rates are computed from E=αFT, with α never calibrated against hardware. The theorem in Section IV-D is also not proven for the actual nonconvex, pruning-based algorithm. These are load-bearing limitations that must be addressed before the paper's headline claims can be accepted.
major comments (3)
- [Section IV-D, Theorem 1 and Eq. (24)] The theorem is not derived for the setting actually used. The proof is two sentences citing [16] without identifying a supporting result; F_k for deep networks is nonconvex, the primal update is implemented as greedy pruning rather than a convex minimization, and the energy function is linear, not strongly convex. The claimed bound ε=O(ρ) does not follow from the proof's 'cumulative violation O(√T), divided by T' reasoning, which would give O(1/√T) and contains no ρ. Moreover the dual update Eq. (23) bounds only average violation; it cannot justify the text's repeated claims of 'strict' energy budget satisfaction. The theorem should be replaced with a statement whose assumptions match the algorithm and a complete proof, or removed with the primal-dual step presented as a heuristic with empirical support only.
- [Section III Eq. (3); Section V-B, Tables I-IV] All reported energy values depend on the surrogate E_k(a_k)=α_k F(a_k) T_k, but α_k is never measured or specified and no hardware measurement validates the FLOPs-to-Joules mapping. For edge devices, memory access, communication, idle power, and hardware-specific operator efficiency can dominate; a FLOPs-reducing operation such as depthwise convolution may increase measured energy under memory-bound conditions. Thus the claimed 30% energy reduction, 92.6% feasibility rate, and 4.8% violation rate describe a synthetic quantity, not physical battery consumption. Section V-F explicitly notes missing latency trade-offs but never addresses proxy validity. The authors should either add real-device energy measurements (e.g., on representative edge hardware, reporting correlation/error relative to Eq. (3), and specifying how α_t_k is set) or rescope all energy claims as 'estimated FLOP-based cos
- [Section IV-B/C/D, Eqs. (7), (16), (22)-(23)] The energy-aware score, the pruning score, the primal objective, and the dual update all optimize exactly the same E=αFT that is later reported as 'energy consumption' in Tables I-IV. Consequently the observed energy reductions and constraint-satisfaction rates are consequences of the algorithm minimizing its own objective, not independent evidence of energy efficiency. The accuracy comparisons are independent and valuable, but the headline claim that OrchNAS 'significantly reduces energy consumption' is circular as currently presented. An external validation metric—hardware-measured energy, or at least a separately measured cost model not used in the training loop—is needed before the energy-efficiency claim can be supported.
minor comments (6)
- [Algorithm 2 vs Eq. (10)] Algorithm 2 line 5 uses τ_t_k while Eq. (10) uses T_t_k. Please unify the notation for local training steps.
- [Eq. (10)] Typo: '(a_t_k)' appears where 'α_t_k' is intended.
- [Eqs. (8) and (25)] The aggregation weight p_t_k is used in Eq. (8) but defined only later in Eq. (25). Define it before first use.
- [Section V-A, search space] The search space is described as 3×5×3×3×3=405. If depth is a dimension, the effective space over layers is larger unless each architecture is constrained to one global configuration; please clarify whether the enumerated 405 count is per network or per layer.
- [Section V-B, Table I] The text says energy is 'estimated from the computational cost' but Table headers report units of J. Please either report hardware-measured energy or explicitly label the values as model-estimated 'FLOP-based cost' in the table and text.
- [Algorithm 2] The pruning loop calls Acc_k(a') for every candidate removal; the cost of these forward evaluations is not discussed. A complexity estimate or practical note would help reproducibility of the reported overheads.
Circularity Check
Energy claims are self-definitional: 'energy' is defined as α·FLOPs·T, then optimized, pruned, and reported from that same formula.
specific steps
-
self definitional
[Section III, Eq. (3); Section IV-C, Eq. (10); Section V-B]
"The training energy consumption of service k is modelled as E_k(a_k)=α_k F(a_k) T_k (3) ... energy consumption is estimated from the computational cost of each architecture, accounting for its FLOPs and local training workload under service-specific conditions."
The paper defines the quantity it later reports as an empirical result. The same E from Eq. (3)/(10) is used in the energy-aware score (Eq. 7), the pruning objective (Eq. 13), the primal update (Eq. 22), the dual update (Eq. 23), and all reported energy numbers in Tables I–IV. No independent energy measurement or calibration of α is provided. Reducing E is therefore equivalent by construction to reducing FLOPs or local steps, so the claimed energy reductions and the 4.8% violation rate describe the paper's own synthetic metric, not physical battery consumption.
-
self definitional
[Section IV-C, Algorithm 2, Eqs. (15)–(16); Section V-C, Table III]
"ΔE_t_o = E_t_k(a_t_k) − E_t_k(a_t_k \ o) (15) ... Score_t(o)=ΔAcc_t_o − βΔE_t_o (16) ... OrchNAS ... reduces training energy consumption by 35.8%"
The greedy pruner ranks operations by ΔE, which is computed from E=αFT. Removing any operation with positive FLOPs necessarily decreases E, so the 'energy reduction' reported in Table III is a direct arithmetic consequence of the pruning rule and the paper's own energy definition. It is not an independent measurement of device-level energy consumption, and the feasibility rate is also evaluated against the same αFT-based budget.
full rationale
The accuracy claims are independent: test accuracy is measured on data and is not defined by the optimization objective. The self-citations [31]–[34] are not load-bearing for the central method, and the saddle-point citation [16] is an external standard reference. The circularity is confined to the energy branch of the paper. Energy is defined in Eq. (3)/(10) as α·FLOPs·T; the same quantity is optimized by the global search score, by greedy pruning, and by the primal–dual dual update, and then reported as the experimental 'energy' metric. Since no hardware measurement, α calibration, or external energy benchmark is provided, the claimed 19–35.8% energy reductions and the 4.8% violation rate are consequences of the paper's own definition rather than empirical evidence about physical energy. This makes the energy-related claims partially circular, while the method and accuracy results retain independent content.
Axiom & Free-Parameter Ledger
free parameters (5)
- energy coefficient alpha_k =
not reported
- energy-accuracy trade-off lambda =
0, 0.01, 0.05, 0.1 (sensitivity only)
- pruning trade-off beta =
not reported
- proximal coefficient mu =
not reported
- dual step size rho =
not reported
axioms (4)
- domain assumption Local objective F_k is convex and Lipschitz continuous
- domain assumption One-shot supernet weight sharing gives accurate inherited weights for subnets
- ad hoc to paper Training energy is linear in FLOPs and local steps
- domain assumption Greedy pruning realizes the primal architecture minimization
read the original abstract
We propose OrchNAS, an energy-aware, personalised, federated edge intelligence framework that leverages a Neural Architecture Search Service to automatically design service-adaptive models for heterogeneous edge environments. The framework orchestrates the architecture search process on a server-side NAS service, enabling edge services to derive personalised architectures under device-level energy, computation, and memory constraints. We introduce an energy-aware global architecture search mechanism that learns a compact global representation across heterogeneous services. We develop an energy-efficient architecture selection mechanism that enables each service to derive a personalised subnet that satisfies its resource constraints via a progressive, greedy, energy-aware pruning strategy. We propose an energy-efficient personalised model optimisation scheme that updates service-adaptive parameters while preserving global representations, where a primal-dual optimisation mechanism enforces strict energy budgets during architecture adaptation. Experiments on real-world and benchmark datasets demonstrate the effectiveness of the proposed approach.
Figures
Reference graph
Works this paper leans on
-
[1]
SPI- DER: Searching personalised neural architecture for fed- erated learning,
E. Mushtaq, C. He, J. Ding, and S. Avestimehr, “SPI- DER: Searching personalised neural architecture for fed- erated learning,”arXiv:2112.13939, 2021
Pith/arXiv arXiv 2021
-
[2]
PEACHES: Personalised federated learning with neural architecture search in edge computing,
J. Yan, J. Liu, H. Xu, Z. Wang, and C. Qiao, “PEACHES: Personalised federated learning with neural architecture search in edge computing,”IEEE Transactions on Mobile Computing, pp. 10296–10312, 2024
2024
-
[3]
Personalized Neural Ar- chitecture Search for Federated Learning,
M. Hoang and C. Kingsford, “Personalized Neural Ar- chitecture Search for Federated Learning,” 2021
2021
-
[4]
PerFedRLNAS: One-for-all per- sonalised federated neural architecture search,
D. Yao and B. Li, “PerFedRLNAS: One-for-all per- sonalised federated neural architecture search,” inProc. AAAI Conf. AI, vol. 38, no. 15, pp. 16398–16406, 2024
2024
-
[5]
Heterogeneity-Aware Personalized Federated Neural Architecture Search,
A. Yang and Y . Liu, “Heterogeneity-Aware Personalized Federated Neural Architecture Search,”Entropy, p. 759, 2025
2025
-
[6]
Neural architecture search driven by locally guided diffusion for personalised federated learning,
P. Liao, X. Wang, Y . Jin, W. Du, and H. Hu, “Neural architecture search driven by locally guided diffusion for personalised federated learning,” inProc. IEEE/CVF (ICCV), pp. 4222–4231, 2025
2025
-
[7]
Green-EDP: Aligning personalisation in federated learning and green artificial intelligence throughout the encoder-decoder architecture,
J. A. Ruiz-Mill, N.Rodriguez-Barroso, and M.V Lu- zon, “Green-EDP: Aligning personalisation in federated learning and green artificial intelligence throughout the encoder-decoder architecture,”Progress in AI, pp. 1–18, 2025
2025
-
[8]
A multi-task based clustering per- sonalized federated learning method,
A. Xiong, and et al., “A multi-task based clustering per- sonalized federated learning method,”Big Data Mining and Analytics, pp. 1017–1030, 2024
2024
-
[9]
Neural architec- ture search: A survey,
T. Elsken, J. H. Metzen, and F. Hutter, “Neural architec- ture search: A survey,” Journal of ML Research, pp. 1 21, 2019
2019
-
[10]
https://docs.cloud.google.com/vertex ai/docs/training/neural architecture search/overview
Google Cloud, ”Vertex AI NAS overview, Google Cloud Docs”, 2021. https://docs.cloud.google.com/vertex ai/docs/training/neural architecture search/overview
2021
-
[11]
MobileNetV2: Inverted residuals and linear bottlenecks,
M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.- C. Chen, “MobileNetV2: Inverted residuals and linear bottlenecks,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2018, pp. 4510–4520
2018
-
[12]
MobileNets: Efficient convo- lutional neural networks for mobile vision applications,
A. G. Howard, and et al., “MobileNets: Efficient convo- lutional neural networks for mobile vision applications,” arXiv:1704.04861, 2017
Pith/arXiv arXiv 2017
-
[13]
Random search and repro- ducibility for neural architecture search,
L. Li and A. Talwalkar, “Random search and repro- ducibility for neural architecture search,” in Proc. Conf. UAI, 2019
2019
-
[14]
CIFAR-10 object recog- nition dataset,
A. Krizhevsky and G. Hinton, “CIFAR-10 object recog- nition dataset,”Kaggle, 2009. [Online]. Available:https: //www.kaggle.com/datasets/cifar-10
2009
-
[15]
ImageNet large scale visual recognition challenge,
O. Russakovsky and et al., “ImageNet large scale visual recognition challenge,” International Journal of CV , pp. 211–252, 2015
2015
-
[16]
Distributed optimisation and statistical learning via the alternating direction method of multipliers,
S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, “Distributed optimisation and statistical learning via the alternating direction method of multipliers,” Foundations and Trends in ML, pp. 1–122, 2011
2011
-
[17]
Towards green service composition approach in the cloud,
S. Wang, A. Zhou, R. Bao, and W. Chou, “Towards green service composition approach in the cloud,” IEEE TSC, pp. 532–545, 2018
2018
-
[18]
Dynamic task offloading with minority game for Internet of vehicles in cloud-edge computing,
B. Shen and et al., “Dynamic task offloading with minority game for Internet of vehicles in cloud-edge computing,” in Proc. ICWS, 2020, pp. 372–379
2020
-
[19]
On the continuous processing of health data in edge-fog- cloud computing by using micro/nanoservice composi- tion,
D. D. Sanchez-Gallegos and A. Galaviz-Mosqueda, “On the continuous processing of health data in edge-fog- cloud computing by using micro/nanoservice composi- tion,” in Proc. IEEE ICWS, 2020, pp. 143–150
2020
-
[20]
F2NAS: Flexible federated neural architecture search in green edge computing,
Z. Zhao, C. Qiu, Y . Zhao, X. Wang, and H. Yao, “F2NAS: Flexible federated neural architecture search in green edge computing,” in Proc. IEEE, 2024
2024
-
[21]
Edge intelligence: Empowering intel- ligence to the edge of network,
D. Xu and et al., “Edge intelligence: Empowering intel- ligence to the edge of network,”Proc. IEEE, pp. 1778– 1837, 2021
2021
-
[22]
Learning multiple layers of features from tiny images,
A. Krizhevsky, “Learning multiple layers of features from tiny images,” University of Toronto, 2009
2009
-
[23]
Reading digits in natural images with unsupervised feature learning,
Y . Netzer and et al., “Reading digits in natural images with unsupervised feature learning,” in NIPS Workshop on DL and Unsupervised Feature Learning, 2011
2011
-
[24]
Tiny ImageNet (Stanford CS231N) [Dataset],
J. Deng, W. Dong, R. Socher, et al., “Tiny ImageNet (Stanford CS231N) [Dataset],” 2024. [Online]. Available: https://cstr.cn/32010.11.sjtu.scidata.00000019
2024
-
[25]
ImageNet Ob- ject Localisation Challenge,
A. Howard, E. Park, and W. Kan, “ImageNet Ob- ject Localisation Challenge,” Kaggle, 2018. [Online]. https://kaggle.com/competitions/imagenet object locali- sation challenge
2018
-
[26]
FLaaS: Fed- erated Learning as a Service,
N. Kourtellis, K. Katevas, and D. Perino, “FLaaS: Fed- erated Learning as a Service,” inProc. 1st Workshop on Distributed ML, 2020, pp. 7–13
2020
-
[27]
Communication-Efficient Local Stochastic Gradi- ent Descent for Scalable Deep Learning,
S. Lee, Q. Kang, A. Agrawal, A. Choudhary, and W.-k. Liao, “Communication-Efficient Local Stochastic Gradi- ent Descent for Scalable Deep Learning,” inProc. IEEE (Big Data), 2020, pp. 718–727
2020
-
[28]
Automatic Neural Network Construction Based on Neural Tangent Kernel for IRS-Aided Beamforming,
H. Shi, T. Ji, Z. Wang, S. Jin, and Y . Huang, “Automatic Neural Network Construction Based on Neural Tangent Kernel for IRS-Aided Beamforming,”IEEE Trans. Wire- less Commun., pp. 1155–1168, 2026
2026
-
[29]
Accurate and robust NAS via a flexible supernet,
Y . Feng, Y . Ou, and Y . Sun, “Accurate and robust NAS via a flexible supernet,”IEEE Transactions on NN and Learning Systems, pp. 1–15, 2026
2026
-
[30]
An evolutionary framework for multi-objective neural architecture search,
F. Ming, W. Gong, B. Xue, M. Zhang, and Y . Jin, “An evolutionary framework for multi-objective neural architecture search,”IEEE Transactions on Evolutionary Computation, pp. 1–1, 2025
2025
-
[31]
A Survey on UA V-enabled Edge Computing: Resource Management Perspective,
X. Xia, S. M. M. Fattah, and M. A. Babar, “A Survey on UA V-enabled Edge Computing: Resource Management Perspective,”ACM Computing Surveys, pp. 1–36, 2024
2024
-
[32]
Perera, S
M. Perera, S. Fattah, S. Mistry, and A. Krishna, ”Rein- forcement Learning Controlled Adaptive PSO for Task Offloading in IIoT Edge Computing,” inACM Web Conference 2025, 2025, pp. 1249–1253
2025
-
[33]
Long- Term IaaS Provider Selection Using Short-Term Trial Experience,
S. M. M. Fattah, A. Bouguettaya, and S. Mistry, “Long- Term IaaS Provider Selection Using Short-Term Trial Experience,” inProc. IEEE ICWS, 2019, pp. 304–311
2019
-
[34]
Context-Aware Selection of Machine Learning as a Service (MLaaS) in IoT Environments,
K. Patel, S. Mistry, D. Kanneganti, and A. Krishna, “Context-Aware Selection of Machine Learning as a Service (MLaaS) in IoT Environments,” inProc. WISE, Springer, 2024, pp. 190–205
2024
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.