Pith. sign in

REVIEW 4 major objections 6 minor 53 references

Guidelines for the Quality Assessment of Energy-Aware NAS Benchmarks

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read GPU power API choice skews energy data in a major NAS benchmark

desk verdict A clean empirical catch: EA-HAS-Bench's per-epoch energy is unreliable because nvidia-smi under-samples, and pyNVML restores 0.99 correlation; deserves review, though artifacts and CIs are needed. read the letter →

arxiv 2505.15631 v1 pith:Q47YUO42 submitted 2025-05-21 cs.LG

Nick Kocher , Christian Wassermann , Leona Hennig , Jonas Seng , Holger Hoos , Kristian Kersting , Marius Lindauer , Matthias Müller This is my paper · ORCID
classification cs.LG
keywords energy-awareNASpowermeasurementNvidiaSMIpyNVMLsurrogatebenchmarkingCodeCarbonGPUenergybenchmarkquality
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that energy-aware neural architecture search benchmarks can only be trusted if their power measurements are reliable, span a wide range of GPU usage, and report holistic cost. To test this, the authors re-ran the data collection of EA-HAS-Bench while simultaneously logging an external power meter. They found that per-epoch energy readings taken with Nvidia SMI correlate with the power meter at only 0.64 (Pearson), with 39% of epochs under-sampled, while switching to the underlying NVML library raises the correlation to 0.99. They also show that the RegNet search space uses only a narrow slice of the GPU's power range, and that the popular Code Carbon tool underestimates total node energy unless calibrated against measured idle and busy power bounds.

What carries the argument

The key mechanism is the sampling process behind per-epoch power readings: Nvidia SMI is a command-line utility that wraps the NVML library, and invoking SMI from a separate thread perturbs the timing of the underlying NVML calls, producing epochs with too few samples and thus falsely low average power. The paper's validation pipeline consists of a calibrated external power meter (ZES ZIMMER LMG450) recording the whole node, with RAPL-based CPU and memory power and a constant 'busy' power of 811 W subtracted to isolate GPU power, and then compares SMI, pyNVML, and Code Carbon readings against that ground truth using correlation and Kolmogorov–Smirnov tests.

What would settle it

Run a single RegNet training while logging per-epoch sample counts, SMI-reported energy, pyNVML-reported energy, and power-meter energy simultaneously; if the correlation between sample count and reported energy disappears when NVML is used, and if the per-epoch Pearson correlation of NVML with the power meter stays above 0.98, the paper's central claim holds. A further check would vary CPU load during training and verify whether the 811 W busy-power subtraction keeps the power-meter GPU estimate stable across load levels.

Watch

Extended reading notes

Core claim

The central claim is that the choice of GPU measurement API determines whether per-epoch energy data in EA-HAS-Bench is usable at low fidelity. Using Nvidia SMI on top of its underlying library disturbs the sampling thread, so roughly 39% of epochs receive too few power samples and the reported energy drops into a low-power plateau; those under-sampled epochs destroy the correlation with a calibrated external power meter (Pearson 0.64 per epoch). Replacing SMI with pyNVML, which was used in the original data collection, restores near-perfect agreement (0.99), and even simply filtering out under-sampled epochs raises the correlation to 0.95. The paper further shows that the benchmark's GPU usage range is narrow (146–305 W against a 75–800 W envelope) and that holistic reporting tools such as Code Carbon systematically underestimate node energy, with an offline calibration using idle and busy power bounds reducing the maximum inaccuracy from 10.3% to 6.6%.

Load-bearing premise

The power meter ground truth is obtained by subtracting RAPL-based CPU and memory power and a constant 'busy' power of 811 W, so if the unmeasured off-socket power is not actually constant during training, all correlations and error bounds shift.

Editorial extensions

If this is right

  • Per-epoch energy labels in EA-HAS-Bench should be re-collected with pyNVML or validated by an external meter, otherwise low-fidelity NAS methods will optimize against corrupted energy estimates.
  • A simple sample-ratio check (samples per epoch versus reported energy) can be used as a quality gate for any energy-aware benchmark, since the underlying SMI artifact shows up as a strong correlation (0.57) between these two quantities.
  • The narrow 146–305 W power range means that energy-aware searches on this benchmark may not transfer to low-power edge devices, so future benchmarks need search spaces that exercise a wider fraction of GPU power.
  • Code Carbon's memory-power estimate (750 W) is far above the node's real memory draw (12 W), and the overestimation partially compensates for missing off-socket power; calibrating with idle and busy bounds gives a tighter, evidence-based range of error.
  • Multi-GPU training behaves differently from single-GPU training: power correlates with GPU utilization on a single GPU but only with memory utilization when all four GPUs are used, so benchmark quality should be assessed per deployment scenario.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Other GPU-based NAS or AutoML benchmarks that log power through SMI at default settings may contain the same per-epoch sampling artifact; re-running their data collection with a timestamped sample log would reveal whether reported energy tracks the number of samples taken.
  • The 0.64 Pearson correlation at per-epoch granularity implies that surrogate models trained on EA-HAS-Bench's SMI data may encode the sampling artifact rather than genuine energy differences between architectures, so accuracy–energy trade-offs derived from that benchmark should be treated as provisional.
  • A cheap, testable extension would be to add a sample-count column to every energy benchmark release: any epoch with fewer than a threshold number of power samples could be flagged or corrected, giving downstream users a direct measure of data quality without needing an external power meter.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes three design principles for energy-aware NAS benchmarks (reliable power measurements, wide GPU usage range, and holistic cost reporting) and evaluates EA-HAS-Bench against them. Using a calibrated external power meter on a single four-H100 node, the authors report a per-epoch Pearson correlation of only 0.64 between Nvidia SMI-based energy readings and the meter across 53,850 epochs of 500 sampled RegNet architectures, with 39% of epochs under-sampled. Replacing SMI with pyNVML in a 20-architecture follow-up yields correlation 0.99, leading the paper to attribute the fault to the SMI sampling path. The paper also documents a narrow GPU usage range (146-305 W; maximum usage around 40%) and proposes calibrating Code Carbon's holistic cost estimates using measured idle/busy off-socket power bounds, reducing the reported maximum inaccuracy from 10.3% to 6.6%.

Significance. The central result, if it holds, is important: it invalidates per-epoch energy data in a published benchmark and offers concrete guidance for building energy-aware NAS benchmarks. The study's strengths are the use of an externally calibrated power meter as ground truth, a large-scale 53,850-epoch validation study, and a cause-isolation experiment that replaces SMI with pyNVML. The proposed Code Carbon calibration is a practical contribution. However, the strength of the causal claim depends on assumptions about the constancy of uncaptured off-socket power and on matched comparisons between measurement APIs, which are not fully established in the current manuscript.

major comments (4)
  1. [Section IV, Eq. (5)] The ground-truth GPU power is defined as P_GPU = P_total - P_CPU - P_memory - P_busy, with P_busy = 811 W measured once during a prime-number stress load. All correlation and KS-test claims in Table I inherit this definition. A constant error in P_busy would not affect Pearson or Spearman correlations, but if the uncaptured off-socket power is not constant across the multi-day, 53,850-epoch study (e.g., due to thermal drift, concurrent system activity, or CPU power-state changes), the derived GPU power contains time-varying noise that would attenuate the SMI correlation independently of SMI's sampling defect and could widen the apparent gap to the shorter 20-architecture pyNVML follow-up. The paper's evidence for constancy is indirect. Please report the residual time series P_total - P_CPU - P_memory - P_GPU_NVML over a long run, or repeat the idle/busy calibration at the start, middle, and end of the study; without this, the absolute GPU-power values and the KS-test outcomes rest on an unverified assumption.
  2. [Section VI.A and Table I] The central SMI-versus-NVML comparison is confounded by different sample sizes and architecture draws. The SMI row is computed from the 500-architecture/53,850-epoch single-GPU study, while the NVML and Code Carbon rows come from experiments with 20 architectures each, sampled independently. Because the same architectures are not measured under both APIs, the causal attribution that SMI on top of pyNVML is the cause of the low sampling rate is not fully identified; the correlation gap could partly reflect architecture distribution or run-specific thermal state. Please provide a matched comparison, e.g., compute SMI-based correlations on the same 20 architectures used for the NVML run, or state explicitly why the architecture sample cannot affect the API comparison.
  3. [Section V.B] The SMI (corrected) row in Table I is produced by deleting under-sampled epochs, which is an ex post analysis rather than a data-collection remedy; the text should state its status as such. The sentence 'Correcting for low number of samples turned the correlation around to -0.39' is also ambiguous: it is not clear whether the authors regress out the sample count or simply report the correlation after excluding under-sampled epochs. Since a negative correlation is not expected under the stated ideal, please define the correction precisely and discuss the selection effect.
  4. [Table I and Section VI.B] The main quantitative claims are presented without confidence intervals or per-architecture error bars. The 0.64 versus 0.99 contrast is load-bearing, yet the former is based on 53,850 epochs and the latter on a much smaller follow-up; reporting bootstrap confidence intervals and the distribution of per-architecture correlations would show whether the API effect is robust and would quantify the uncertainty from the 20-architecture experiments.
minor comments (6)
  1. [Section V] The claim that 'RegNets are a variation of ResNets, where residual blocks are replaced by recurrent neural networks, such as LSTMs' does not match the RegNet search space of EA-HAS-Bench; please correct the description and reference.
  2. [Section IV] The term P_idle is used both for the idle-run residual (783 W) and for the Firestarter-load residual (941 W); please rename these to avoid confusion with P_busy.
  3. [Figure 1] The green curve is labeled 'Power meter-SMI' but the caption does not state that it is the difference; please define it in the caption.
  4. [Section VI.C] The sentence about 'the multi-GPU measurements on the righthand side of Figure 5' should specify whether it refers to the utilization panel or the memory-utilization panel.
  5. [Section VI.D] The statement about 'the overestimation of memory consumption by RAPL' is imprecise; Code Carbon's 3 W per 8 GB memory estimate is not a RAPL measurement. Please correct this wording.
  6. [General] The paper does not state whether the raw power-meter and RAPL traces, or the measurement code, will be released; given the reproducibility goals, an availability statement would be valuable.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: central claims rest on external power-meter measurements and physically calibrated constants, not on fitted or self-referential inputs.

full rationale

The paper's central claim is empirical: Nvidia SMI per-epoch power readings correlate poorly (Pearson 0.64) with a calibrated external power meter, while pyNVML-based readings correlate at 0.99. This is a direct measurement comparison against an external instrument, not a derivation from fitted parameters. The only in-house constant, P_busy = 811 W in Equation 5, is obtained from a prime-number stress test on the same node, not by regression on the training data or on the Code Carbon error; the paper states 'initial experiments show that calculating large prime numbers with a P_busy of 811 W is a good lower bound for the busy power consumption.' A constant offset would not by construction produce the per-epoch correlation gap between SMI and NVML, since the reported Pearson and Spearman correlations are insensitive to a fixed additive constant. The Code Carbon calibration uses measured idle and Firestarter bounds plus the prime-number busy value, and the improvement from 10.3% to 8.9%/6.6% is evaluated against the same power meter; however, the bounds are physically measured quantities rather than parameters fitted to minimize the reported error, so this is a calibration experiment, not a fitted input renamed as a prediction. No self-citation is load-bearing, and no uniqueness theorem or ansatz is imported from the authors' prior work. The paper's own acknowledged limitations, such as evaluating on one server type, sampling only 20 architectures in the main experiments, and assuming a constant off-socket load, are external-validity and correctness risks rather than circularity; they are explicitly stated in Section VII. The results are therefore self-contained with respect to the tested hardware and measurement protocol, and there is no step in which an output is equivalent to its input by construction.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on an external power meter as ground truth, adjusted by subtracting RAPL-based CPU/memory power and a single measured busy-power constant (811 W). The Code Carbon calibration introduces two additional measured bounds (783 W idle, 941 W Firestarter). These three physical measurements are empirical inputs, not invented entities.

free parameters (3)
  • P_busy_prime = 811 W
    Measured during a prime-number calculation on the test node; used in Eq. 5 to estimate the uncaptured off-socket power when deriving GPU power from the external power meter, and as the 'Busy estimation' in the Code Carbon calibration. The value is empirical, not fitted to the paper's headline results, but no uncertainty is reported.
  • P_idle = 783 W
    Measured idle power of the node, used as one bound in the Code Carbon calibration (labeled 'Upper bound (Idle)' in Figure 8).
  • P_firestarter = 941 W
    Measured power under the Firestarter stress test, used as the other bound in the Code Carbon calibration (labeled 'Lower bound (Firestarter)' in Figure 8).
assumptions (4)
  • domain assumption RAPL reports accurate CPU and memory power for the PKG and DRAM domains on this server
    Used in Eq. 5 to subtract CPU and memory power from the whole-node power meter reading; the paper cites [42] and [46] for RAPL accuracy but does not validate it on this specific node.
  • domain assumption The uncaptured off-socket power is approximately constant during training runs
    The power meter adjustment subtracts a single P_busy (811 W) for every training epoch (Eqs. 4-5); if the off-socket load varies, the derived GPU power is systematically biased. This is discussed qualitatively in Sections IV and VII.
  • domain assumption The external power meter's 100 ms sampling adequately captures per-epoch GPU energy dynamics
    The ZES ZIMMER LMG450 continuously integrates active power, and the paper treats it as ground truth for per-epoch energy without quantifying truncation error from finite sampling.
  • domain assumption SMI's power.draw returns an average over a 1 s window that is usable as an instantaneous sample at 10 Hz
    The paper relies on Nvidia documentation [40], [41] for this behavior in interpreting SMI samples; this is a property of the measurement tool, not an independent mathematical fact.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Guidelines for the Quality Assessment of Energy-Aware NAS Benchmarks." pith.science (2026). https://pith.science/paper/Q47YUO42

@misc{pith2026250515631,
  author       = {Pith},
  title        = {Pith review of: Guidelines for the Quality Assessment of Energy-Aware NAS Benchmarks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q47YUO42}},
  note         = {Machine review of arXiv:2505.15631}
}
read the original abstract

Neural Architecture Search (NAS) accelerates progress in deep learning through systematic refinement of model architectures. The downside is increasingly large energy consumption during the search process. Surrogate-based benchmarking mitigates the cost of full training by querying a pre-trained surrogate to obtain an estimate for the quality of the model. Specifically, energy-aware benchmarking aims to make it possible for NAS to favourably trade off model energy consumption against accuracy. Towards this end, we propose three design principles for such energy-aware benchmarks: (i) reliable power measurements, (ii) a wide range of GPU usage, and (iii) holistic cost reporting. We analyse EA-HAS-Bench based on these principles and find that the choice of GPU measurement API has a large impact on the quality of results. Using the Nvidia System Management Interface (SMI) on top of its underlying library influences the sampling rate during the initial data collection, returning faulty low-power estimations. This results in poor correlation with accurate measurements obtained from an external power meter. With this study, we bring to attention several key considerations when performing energy-aware surrogate-based benchmarking and derive first guidelines that can help design novel benchmarks. We show a narrow usage range of the four GPUs attached to our device, ranging from 146 W to 305 W in a single-GPU setting, and narrowing down even further when using all four GPUs. To improve holistic energy reporting, we propose calibration experiments over assumptions made in popular tools, such as Code Carbon, thus achieving reductions in the maximum inaccuracy from 10.3 % to 8.9 % without and to 6.6 % with prior estimation of the expected load on the device.

Figures

Figures reproduced from arXiv: 2505.15631 by the authors.

Figure 1
Figure 1. Switching of low- and high-power states measured by SMI queries during training of a neural architecture on one GPU. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. eCDF of the power meter measurements and the SMI [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. GPU power vs the ratio of taken samples and expected [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Top: eCDF of the aggregated single-GPU training [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Top left: GPU power consumption vs GPU utilisation for single-GPU training. Bottom left: GPU power consumption vs GPU memory utilisation for single-GPU training. Top right: GPU power consumption vs GPU utilisation for multi-GPU training. Bottom right: GPU power consump…
Figure 6
Figure 6. Figure 6: Power measurements across time for Code Carbon and [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 8
Figure 8. Figure 8: Boxplot with differences in energy measurements between Code Carbon and the power meter for single-GPU training. [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

53 extracted references · 46 canonical work pages

  1. [1]

    Gradient-based learning applied to document recognition,

    Y . Lecun, L. Bottou, Y . Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,”Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, 1998

    1998

  2. [2]

    Fully convolutional networks for semantic segmentation,

    J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” inProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 3431– 3440

    2015

  3. [3]

    Long short-term memory,

    S. Hochreiter and J. Schmidthuber, “Long short-term memory,”Neural Computation, vol. 9, no. 8, pp. 1735–1780, 1997

    work page 1997

  4. [4]

    ImageNet classification with deep convolutional neural networks,

    A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification with deep convolutional neural networks,” inProceedings of the Ad- vances in Neural Information Processing Systems (NeurIPS), vol. 25, 2012, pp. 1097–1105

    work page 2012

  5. [5]

    Going deeper with convolutions,

    C. Szegedy, W. Liu, Y . Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V . Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” inProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 1–9

    work page 2015

  6. [6]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” inProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770–778

    2016

  7. [7]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” in Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), vol. 30, 2017, pp. 5998–6008

    work page 2017

  8. [8]

    Hutter, L

    F. Hutter, L. Kotthoff, and J. Vanschoren,Automated Machine Learning: Methods, Systems, Challenges. Springer Nature, 2019

    work page 2019

Show all 53 references
  1. [9]

    Neural architecture search: a survey,

    T. Elsken, J. H. Metzen, and F. Hutter, “Neural architecture search: a survey,”Journal of Machine Learning Research, vol. 20, no. 1, p. 1997–2017, 2019

    1997

  2. [10]

    Meta pseudo labels,

    H. Pham, Z. Dai, Q. Xie, and Q. V . Le, “Meta pseudo labels,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 11 552–11 563

    2021

  3. [11]

    Searching for efficient transformers for language modeling,

    D. So, W. Ma ´nke, H. Liu, Z. Dai, N. Shazeer, and Q. V . Le, “Searching for efficient transformers for language modeling,” inProceedings of the Advances in Neural Information Processing Systems (NeurIPS), vol. 34, 2021, pp. 6010–6022

    2021

  4. [12]

    Neural architecture search with reinforcement learning,

    B. Zoph and Q. Le, “Neural architecture search with reinforcement learning,” inProceedings of the International Conference on Learning Representations (ICLR), 2017

    2017

  5. [13]

    Green AI,

    R. Schwartz, J. Dodge, N. A. Smith, and O. Etzioni, “Green AI,” Communications of the ACM, vol. 63, no. 12, p. 54–63, 2020

    2020

  6. [14]

    Towards Green Automated Machine Learning: Status Quo and Future Directions,

    T. Tornede, A. Tornede, J. Hanselle, F. Mohr, M. Wever, and E. H ¨ullermeier, “Towards Green Automated Machine Learning: Status Quo and Future Directions,”Journal of Artificial Intelligence Research, vol. 77, pp. 427–457, 2023

    2023

  7. [15]

    NAS-Bench-201: Extending the Scope of Repro- ducible Neural Architecture Search,

    X. Dong and Y . Yang, “NAS-Bench-201: Extending the Scope of Repro- ducible Neural Architecture Search,” inProceedings of the International Conference on Learning Representations (ICLR), 2020

    2020

  8. [16]

    Hardware-aware neural architecture search: Survey and taxonomy,

    H. Benmeziane, K. El Maghraoui, H. Ouarnoughi, S. Niar, M. Wistuba, and N. Wang, “Hardware-aware neural architecture search: Survey and taxonomy,” inProceedings of the International Joint Conference on Artificial Intelligence (IJCAI), 2021, pp. 4322–4329

    2021

  9. [17]

    Edge intelligence: Paving the last mile of artificial intelligence with edge computing,

    Z. Zhou, X. Chen, E. Li, L. Zeng, K. Luo, and J. Zhang, “Edge intelligence: Paving the last mile of artificial intelligence with edge computing,”Proc. IEEE, vol. 107

  10. [18]

    Learning iot in edge: Deep learning for the internet of things with edge computing,

    H. Li, K. Ota, and M. Dong, “Learning iot in edge: Deep learning for the internet of things with edge computing,”IEEE Netw., vol. 32

  11. [19]

    HW-NAS-Bench: Hardware-Aware Neural Architecture Search Benchmark,

    C. Li, Z. Yu, Y . Fu, Y . Zhang, Y . Zhao, H. You, Q. Yu, Y . Wang, C. Hao, and Y . Lin, “HW-NAS-Bench: Hardware-Aware Neural Architecture Search Benchmark,” inProceedings of the International Conference on Learning Representations (ICLR), 2021

    2021

  12. [20]

    EA-HAS-bench: Energy-aware hyperparameter and architecture search benchmark,

    S. Dou, X. Jiang, C. R. Zhao, and D. Li, “EA-HAS-bench: Energy-aware hyperparameter and architecture search benchmark,” inProceedings of the International Conference on Learning Representations (ICLR), 2023

    2023

  13. [21]

    NVIDIA Management Library,

    NVIDIA Corporation, “NVIDIA Management Library,” 2025. [Online]. Available: https://developer.nvidia.com/management-library-nvml

    2025

  14. [22]

    mlco2/codecarbon: v2.4.1,

    B. C. et Al, “mlco2/codecarbon: v2.4.1,” 2024. [Online]. Available: https://github.com/mlco2/codecarbon

    2024

  15. [23]

    Auto-pytorch tabular: Multi- fidelity metalearning for efficient and robust autodl,

    L. Zimmer, M. Lindauer, and F. Hutter, “Auto-pytorch tabular: Multi- fidelity metalearning for efficient and robust autodl,”IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 3079 – 3090, 2021

    2021

  16. [24]

    Efficient neural architecture search via parameters sharing,

    H. Pham, M. Guan, B. Zoph, Q. Le, and J. Dean, “Efficient neural architecture search via parameters sharing,” inProceedings of the International Conference on Machine Learning (ICML), 2018, pp. 4095– 4104

    2018

  17. [25]

    Efficient multi-objective neural architecture search via lamarckian evolution,

    T. Elsken, J. H. Metzen, and F. Hutter, “Efficient multi-objective neural architecture search via lamarckian evolution,” inProceedings of the International Conference on Learning Representations (ICLR), 2019

    2019

  18. [26]

    Large-scale evolution of image classifiers,

    E. Real, S. Moore, A. Selle, S. Saxena, Y . L. Suematsu, J. Tan, Q. V . Le, and A. Kurakin, “Large-scale evolution of image classifiers,” inProceedings of the International Conference on Machine Learning (ICML), 2017, pp. 2902–2911

    2017

  19. [27]

    DARTS: Differentiable Architecture Search,

    H. Liu, K. Simonyan, and Y . Yang, “DARTS: Differentiable Architecture Search,” inProceedings of the International Conference on Learning Representations (ICLR), 2019

    2019

  20. [28]

    DARTS-: Robustly Stepping out of Performance Collapse Without Indicators,

    X. Chu, X. Wang, B. Zhang, S. Lu, X. Wei, and J. Yan, “DARTS-: Robustly Stepping out of Performance Collapse Without Indicators,” in Proceedings of the International Conference on Learning Representa- tions (ICLR), 2021

    2021

  21. [29]

    Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search,

    X. Chu, T. Zhou, B. Zhang, and J. Li, “Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search,” inProceedings of the European Conference on Computer Vision (ECCV), 2020, pp. 465–480

    2020

  22. [30]

    Progressive differentiable architecture search: Bridging the depth gap between search and evalua- tion,

    X. Chen, L. Xie, J. Wu, and Q. Tian, “Progressive differentiable architecture search: Bridging the depth gap between search and evalua- tion,”Proceedings of the International Conference on Computer Vision (ICCV), pp. 1294–1303, 2019

    2019

  23. [31]

    Understanding and robustifying differentiable architecture search,

    A. Zela, T. Elsken, T. Saikia, Y . Marrakchi, T. Brox, and F. Hutter, “Understanding and robustifying differentiable architecture search,” in Proceedings of the International Conference on Learning Representa- tions (ICLR), 2020

    2020

  24. [32]

    Zero-shot neural architecture search: Challenges, solutions, and opportunities,

    G. Li, D.-T. Hoang, K. Bhardwaj, M. Lin, Z. Wang, and R. Mar- culescu, “Zero-shot neural architecture search: Challenges, solutions, and opportunities,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 12, pp. 7618–7635, 2023

    2023

  25. [33]

    MicroNAS: Zero-Shot Neural Architecture Search for MCUs,

    Y . Qiao, H. Xu, Y . Zhang, and S. Huang, “MicroNAS: Zero-Shot Neural Architecture Search for MCUs,”Proceedings of the Design, Automation & Test in Europe Conference & Exhibition, 2024

    2024

  26. [34]

    An efficient multi-objective evolutionary zero-shot neural architecture search framework for image classification,

    J. Zhang, L. Zhang, Y . Wang, J. Wang, X. Wei, and W. Liu, “An efficient multi-objective evolutionary zero-shot neural architecture search framework for image classification,”International Journal of Neural Systems, vol. 33, no. 05, p. 2350016, 2023

    2023

  27. [35]

    NAS-Bench-101: Towards reproducible neural architecture search,

    C. Ying, A. Klein, E. Christiansen, E. Real, K. Murphy, and F. Hutter, “NAS-Bench-101: Towards reproducible neural architecture search,” inProceedings of the International Conference on Machine Learning (ICML), 2019

    2019

  28. [36]

    NAS-Bench-301 and the Case for Surrogate Benchmarks for Neural Architecture Search,

    J. N. Siems, L. Zimmer, A. Zela, J. Lukasik, M. Keuper, and F. Hutter, “NAS-Bench-301 and the Case for Surrogate Benchmarks for Neural Architecture Search,” inProceedings of the International Conference on Learning Representations (ICLR), 2021

    2021

  29. [37]

    JAHS- Bench-201: A Foundation For Research On Joint Architecture And Hyperparameter Search,

    A. Bansal, D. Stoll, M. Janowski, A. Zela, and F. Hutter, “JAHS- Bench-201: A Foundation For Research On Joint Architecture And Hyperparameter Search,” inProceedings of the Advances in Neural Information Processing Systems (NeurIPS), 2022

    2022

  30. [38]

    EC-NAS: Energy Consumption Aware Tabular Benchmarks for Neural Architecture Search,

    P. Bakhtiarifard, C. Igel, and R. Selvan, “EC-NAS: Energy Consumption Aware Tabular Benchmarks for Neural Architecture Search,” inProceed- ings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024, p. 5660–5664

    2024

  31. [39]

    Auto-NBA: Efficient and Effective Search Over the Joint Space of Networks, Bitwidths, and Accelerators,

    Y . Fu, Y . Zhang, Y . Zhang, D. Cox, and Y . Lin, “Auto-NBA: Efficient and Effective Search Over the Joint Space of Networks, Bitwidths, and Accelerators,” inProceedings of the International Conference on Machine Learning (ICML), 2021, pp. 3505–3517

    2021

  32. [40]

    Accurate and convenient energy measurements for gpus: A detailed study of nvidia gpu’s built-in power sensor,

    Z. Yang, K. Adamek, and W. Armour, “Accurate and convenient energy measurements for gpus: A detailed study of nvidia gpu’s built-in power sensor,” inProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC), 2024, p. 1–17

    2024

  33. [41]

    NVIDIA System Management Interface Documentation,

    NVIDIA Corporation, “NVIDIA System Management Interface Documentation,” 2025. [Online]. Available: https://docs.nvidia.com/ deploy/nvidia-smi/index.html

    2025

  34. [42]

    RAPL: memory power estimation and capping,

    H. David, E. Gorbatov, U. R. Hanebutte, R. Khanna, and C. Le, “RAPL: memory power estimation and capping,” inProceedings of the ACM/IEEE International Symposium on Low Power Electronics and Design, 2010, p. 189–194

    2010

  35. [43]

    Rouvoy, “pyRAPL,” 2023

    R. Rouvoy, “pyRAPL,” 2023. [Online]. Available: https://github.com/ powerapi-ng/pyRAPL/

    2023

  36. [44]

    Power monitoring with papi for extreme scale architectures and dataflow-based program- ming models,

    H. McCraw, J. Ralph, A. Danalis, and J. Dongarra, “Power monitoring with papi for extreme scale architectures and dataflow-based program- ming models,” inProceedings of the International Conference on Cluster Computing, 2014, pp. 385–391

    2014

  37. [45]

    LIKWID: A Lightweight Performance-Oriented Tool Suite for x86 Multicore Environments,

    J. Treibig, G. Hager, and G. Wellein, “LIKWID: A Lightweight Performance-Oriented Tool Suite for x86 Multicore Environments,” in Proceedings of the International Conference on Parallel Processing Workshops, 2010, pp. 207–216

    2010

  38. [46]

    A Validation of DRAM RAPL Power Measurements,

    S. Desrochers, C. Paradis, and V . M. Weaver, “A Validation of DRAM RAPL Power Measurements,” inProceedings of the Second Interna- tional Symposium on Memory Systems, 2016, p. 455–470

    2016

  39. [47]

    Quantifying the carbon emissions of machine learning,

    A. Lacoste, A. Luccioni, V . Schmidt, and T. Dandres, “Quantifying the carbon emissions of machine learning,”CoRR, vol. abs/1910.09700, 2019

    1910 arXiv

  40. [48]

    Energy usage reports: Environmental awareness as part of algorithmic accountability,

    K. Lottick, S. Susai, S. A. Friedler, and J. P. Wilson, “Energy usage reports: Environmental awareness as part of algorithmic accountability,” CoRR, vol. abs/1911.08354, 2019

    1911 arXiv

  41. [49]

    How to estimate carbon footprint when training deep learning models? A guide and review,

    L. B. Heguerte, A. Bugeau, and L. Lannelongue, “How to estimate carbon footprint when training deep learning models? A guide and review,”CoRR, vol. abs/2306.08323, 2023

    2023 arXiv

  42. [50]

    The carbon footprint of machine learning training will plateau, then shrink,

    D. Patterson, J. Gonzalez, U. H ¨olzle, Q. Le, C. Liang, L.-M. Munguia, D. Rothchild, D. R. So, M. Texier, and J. Dean, “The carbon footprint of machine learning training will plateau, then shrink,”Computer, vol. 55, no. 7, pp. 18–28, 2022

    2022

  43. [51]

    Introducing FIRESTARTER: A processor stress test utility,

    D. Hackenberg, R. Oldenburg, D. Molka, and R. Sch ¨one, “Introducing FIRESTARTER: A processor stress test utility,” inProceedings of the International Green Computing Conference, 2013, pp. 1–9

    2013

  44. [52]

    RegNet: Self- Regulated Network for Image Classification,

    J. Xu, Y . Pan, X. Pan, S. Hoi, Z. Yi, and Z. Xu, “RegNet: Self- Regulated Network for Image Classification,”IEEE Transactions on Neural Networks and Learning Systems, vol. 34, no. 11, pp. 9562–9567, 2023

    2023

  45. [53]

    ImageNet Large Scale Visual Recognition Challenge,

    O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, “ImageNet Large Scale Visual Recognition Challenge,” International Journal of Computer Vision, vol. 115, no. 3, pp. 211–252, 2015

    2015

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.