Searching on a Budget: HW-NAS with 10 Latency Probes
Pith reviewed 2026-05-22 21:22 UTC · model grok-4.3
The pith
A controller pre-trained on synthetic devices finds latency-efficient networks on unseen hardware using only ten direct latency measurements.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central claim is that pre-training an architecture controller on a distribution of synthetic devices enables it to design latency-efficient networks for an unseen target device by interacting with that device through only a small number of high-fidelity latency measurements at test time, all without any pre-collected information about the target hardware and while using only training-free accuracy proxies.
What carries the argument
The two-stage controller that performs in-context adaptation on the target device via a few direct latency probes after meta-training on synthetic devices.
If this is right
- The controller can be deployed directly to the target device without any pre-collected latency data or predictors.
- Meta-training scales because only training-free accuracy proxies are used, avoiding full network training overhead.
- The method generalizes to unseen devices as shown on the HW-NATS-Bench benchmark.
- Latency-efficient architectures are identified through in-context adaptation that requires only a few real-world latency evaluations at test time.
- Risk-sensitive applications benefit from reduced exposure to estimation errors that arise in approximate latency models.
Where Pith is reading between the lines
- The same pre-training strategy could be applied to other hardware metrics such as energy or memory to support multi-objective searches on new devices.
- One could investigate whether increasing the diversity or number of synthetic devices during pre-training further improves adaptation speed on real hardware.
- This controller-based approach might integrate with existing NAS toolkits to reduce the data-collection burden when targeting custom or low-volume edge platforms.
Load-bearing premise
The distribution of synthetic devices seen during pre-training is representative enough of real unseen hardware for the controller to generalize and adapt successfully with few measurements.
What would settle it
Deploying the method on a real device whose latency behavior lies clearly outside the synthetic distribution and finding that the resulting architectures show no latency improvement over a non-adaptive random search using the same number of probes would falsify the generalization claim.
read the original abstract
Existing hardware-aware NAS (HW-NAS) methods typically assume access to precise information circa the target device, either via analytical approximations of the post-compilation latency model, or through learned latency predictors. Such approximate approaches risk introducing estimation errors that may prove detrimental in risk-sensitive applications. In this work, we propose a two-stage HW-NAS framework, in which we first learn an architecture controller on a distribution of synthetic devices, and then directly deploy the controller on a target device. At test-time, our network controller deploys directly to the target device without relying on any pre-collected information, and only exploits direct interactions. In particular, the pre-training phase on synthetic devices enables the controller to design an architecture for the target device by interacting with it through a small number of high-fidelity latency measurements. To guarantee accessibility of our method, we only train our controller with training-free accuracy proxies, allowing us to scale the meta-training phase without incurring the overhead of full network training. We benchmark on HW-NATS-Bench, demonstrating that our method generalizes to unseen devices and searches for latency-efficient architectures by in-context adaptation using only a few real-world latency evaluations at test-time.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript proposes a two-stage HW-NAS framework: an architecture controller is meta-trained on a distribution of synthetic devices using only training-free accuracy proxies, then deployed directly to an unseen target device where it performs in-context adaptation using approximately 10 real-world latency measurements to identify latency-efficient architectures. Experiments on HW-NATS-Bench are used to support the claim of generalization to unseen devices without pre-collected information or learned predictors.
Significance. If the empirical generalization holds, the approach would be significant for HW-NAS in risk-sensitive or data-scarce settings, as it replaces approximate latency models with direct device interaction after synthetic pre-training. The training-free proxies enabling scalable meta-training and the minimal probe budget are concrete strengths that could reduce estimation errors compared to predictor-based methods.
major comments (2)
- [§3] §3 (Method, synthetic device generation): The central generalization claim depends on the synthetic device distribution being sufficiently representative of real hardware; the manuscript must specify the parametric model used to generate synthetic devices and provide quantitative evidence (e.g., coverage metrics or distribution distances) that it captures effects such as compiler optimizations and memory hierarchy variations, otherwise the in-context adaptation step has no recovery mechanism.
- [§5] §5 (Experiments, HW-NATS-Bench results): The reported generalization to unseen devices must include ablations varying the number of latency probes (e.g., 5 vs. 10 vs. 20) and direct comparisons against baselines that use pre-collected data or more measurements; without these, the '10 probes' budget claim cannot be assessed as load-bearing for the efficiency advantage.
minor comments (2)
- [Abstract] Abstract: The phrase 'only exploits direct interactions' is repeated without clarifying what constitutes an 'interaction' versus a latency measurement; this notation should be defined once in the method section.
- Figure captions (throughout): Several figures lack axis labels or error bars on latency/accuracy trade-offs, reducing clarity of the in-context adaptation results.
Simulated Author's Rebuttal
We thank the referee for the constructive feedback. We address each major comment below and indicate the changes we will make to the manuscript.
read point-by-point responses
-
Referee: [§3] §3 (Method, synthetic device generation): The central generalization claim depends on the synthetic device distribution being sufficiently representative of real hardware; the manuscript must specify the parametric model used to generate synthetic devices and provide quantitative evidence (e.g., coverage metrics or distribution distances) that it captures effects such as compiler optimizations and memory hierarchy variations, otherwise the in-context adaptation step has no recovery mechanism.
Authors: We agree that a more detailed specification of the synthetic device model and supporting quantitative evidence will strengthen the generalization claim. Section 3.2 currently outlines the parametric generation process (operator-level latency multipliers and memory hierarchy costs sampled from uniform distributions over ranges informed by real-device profiling). In the revision we will expand this section with the exact parameter ranges and add a new figure and table in §5 comparing the synthetic latency distribution to real HW-NATS-Bench devices via KL divergence and mean absolute percentage error on a held-out architecture set. This will explicitly demonstrate coverage of compiler and memory effects. revision: yes
-
Referee: [§5] §5 (Experiments, HW-NATS-Bench results): The reported generalization to unseen devices must include ablations varying the number of latency probes (e.g., 5 vs. 10 vs. 20) and direct comparisons against baselines that use pre-collected data or more measurements; without these, the '10 probes' budget claim cannot be assessed as load-bearing for the efficiency advantage.
Authors: We acknowledge that explicit ablations on probe count and comparisons to data-heavy baselines will help readers evaluate the efficiency claim. While the main text focuses on the 10-probe setting, the supplementary material already contains results for 5, 10, and 20 probes; we will move the relevant table and curves into the main §5. We will also add a direct comparison against a latency-predictor baseline trained on 100 pre-collected measurements from the target device. These additions will be included in the revised manuscript. revision: yes
Circularity Check
No circularity; derivation relies on direct measurements and external pre-training
full rationale
The paper's core method is a two-stage process: pre-train a controller on a distribution of synthetic devices using training-free proxies, then deploy it for in-context adaptation on an unseen target device via a small number of direct high-fidelity latency measurements. No step defines a quantity in terms of itself, renames a fitted parameter as a prediction on the same data, or reduces a central claim to a self-citation chain. The generalization assumption (synthetic distribution covers real devices) is an empirical premise, not a definitional loop. The provided text contains no equations or self-citations that would trigger any of the enumerated circularity patterns.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.