{"id":"3f8f97fc-56fa-405a-b3a1-e0c22a88fe6f","arxiv_id":"2608.13293","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"For NAS-Bench-201, INT4 post-training quantization fully reorganizes the Pareto front, yet an FP32 zero-shot surrogate still achieves higher Pareto-space coverage than an INT4-trained surrogate.","lead":"This paper studies how 4-bit post-training quantization reshapes the accuracy-versus-cost tradeoff landscape for neural networks, then feeds that analysis into a three-stage pipeline that maps surviving networks onto reconfigurable hardware. For NAS-Bench-201, it finds that an ordinary FP32-trained search surrogate covers the quantized Pareto space better than a surrogate trained specifically on INT4 data, which would let edge-AI teams skip a retraining step.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Comparison against the INT4-trained surrogate is unverifiable: no training protocol is given, and Table III labels both arms '(zero-shot)', conflicting with the prose's 'fully trained INT4-specific surrogate'.","rationale":"The reader and I converge on the same weak point: the INT4 surrogate's training setup is unspecified. I add two concretizations: the table header appears to label the INT4 arm '(zero-shot)', which would undermine the comparison entirely, and the reported RS hypervolume gap is not clearly significant. These are reproducibility and correctness issues, not disagreements with community consensus, and they do not overturn the paper's other contributions such as the full-space stability metrics or the three-stage pipeline. The central empirical claim is exactly that FP32 transfer beats dedicated INT4 training, so without knowing how the INT4 comparator was trained and without a significance test, the claim is not established. A condition requiring full training disclosure and proper significance reporting is the right disposition, so I keep the reader's conditional verdict.","tokens_in":7922,"tokens_out":5987,"duration_ms":65317,"concrete_test":"Clarify Table III: specify whether the INT4 surrogate was (a) retrained on INT4 labels with the identical encoder, loss, data split, and hyperparameters as the FP32 surrogate, or (b) the FP32 surrogate applied zero-shot. If (b), rerun the entire comparison with a genuinely retrained INT4 surrogate. For (a), publish the full training recipe and rerun the 50-run RS/MOEA comparison, reporting paired 95% confidence intervals for the hypervolume difference and the percentage of runs where FP32 > INT4. If the RS gap is not significant, restrict the zero-shot superiority conclusion to MOEA or add an explicit effect-size and significance caveat.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing comparison is FP32 zero-shot vs a 'fully trained INT4-specific surrogate' (Sec. IV-C). The manuscript gives no training protocol for that surrogate: encoder architecture, loss, epochs, data split, label generation (raw Brevitas PTQ vs two-epoch-finetuned), or hyperparameters. This matters because Table III is ambiguous: its column header labels both arms '(zero-shot)', directly contradicting the prose. If the INT4 arm was not retrained, the headline result is not a comparison against a dedicated INT4-trained surrogate at all. Even taking the prose at face value, the RS hypervolume gap (0.5740 ± 0.2012 vs 0.5113 ± 0.1846) is not significant under a simple two-sample test based on the reported statistics (t ≈ 1.6, p ≈ 0.11), while the MOEA gap is significant; the paper reports no significance test and no paired confidence interval. The claim may still be true, but the most load-bearing comparison is currently unverifiable from the manuscript.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses post-search quantization in hardware-aware NAS. It proposes a three-stage pipeline: a Stage I Pareto-rank surrogate trained on NAS-Bench-201/CIFAR-10 FP32 data; a Stage II quantization bridge applying INT4 post-training quantization (PTQ) via Brevitas with Pareto re-ranking, filtering, and feedback; and a Stage III design-space exploration over CGRA4ML configurations using an analytical oracle. The empirical core is a full-space study over all 15,625 NAS-Bench-201 architectures under INT4 PTQ, reporting formal stability metrics (front survival, dominance flips, KT correlation, rank sensitivity), and a search comparison claiming that an FP32 zero-shot surrogate outperforms a dedicated INT4-trained surrogate in normalized hypervolume under both random search and MOEA (ratios 1.12 and 1.07).","tokens_in":8177,"tokens_out":6579,"duration_ms":65114,"significance":"The paper has notable strengths: it enumerates the complete 15,625-architecture NAS-Bench-201 space for the stability analysis, uses externally generated Brevitas quantization labels rather than its own surrogate predictions, reports 50-run means for the search comparison, and builds on standard benchmarks and open-source tools. If the headline comparison is substantiated, the result is practically significant: practitioners could retain FP32 surrogates for INT4-aware search even though the efficient frontier membership changes completely. The proposed stability metrics are also potentially useful. However, the main claim currently rests on an undocumented INT4-trained surrogate, a contradiction between the table header and the prose about that surrogate, and hypervolume differences that are not tested for significance. These issues are load-bearing but fixable within the scope of the manuscript.","major_comments":[{"comment":"The headline comparison between the FP32 zero-shot surrogate and the “fully trained INT4-specific surrogate” is not reproducible from the manuscript. No training protocol is given for the INT4 surrogate: architecture, loss, label source, data split, epochs, or hyperparameters are all absent. Table III's column header labels both arms as “(zero-shot)”, which directly contradicts the prose claim that the INT4 arm is a “fully trained INT4-specific surrogate.” Furthermore, no significance test is reported; under a simple two-sample t-test on the reported means and standard deviations at n=50, the RS hypervolume difference (0.5740 ± 0.2012 vs 0.5113 ± 0.1846) is not significant (t ≈ 1.6, p ≈ 0.11), so only the MOEA difference currently supports the central claim. Please provide the full INT4 training protocol and paired significance tests (e.g., bootstrap or paired test over the 50 runs).","section":"IV-C, Table III"},{"comment":"The “formal stability metrics” are never defined. “Pareto Front Survival Rate,” “Dominance Flip Rate,” “KT-Rank Correlation,” and “Pareto Rank Sensitivity” are reported as numbers without equations or algorithmic definitions, and the paper does not state explicitly whether they are computed on the full 15,625-architecture ground-truth set or only on sampled subsets. These metrics are one of the two central contributions, so they must be defined precisely to make the empirical study reproducible.","section":"IV-C, Table III and abstract"},{"comment":"The procedure that generates the ground-truth INT4 labels is ambiguous. Table I lists PTQ Id 6 with a raw top-1 accuracy of 26.22%, while the text says that a two-epoch finetuning restores accuracy to 83.53%. The paper never states which protocol was applied to obtain the 15,625 quantized accuracies used for the stability metrics, Fig. 6, and the INT4 surrogate. Because the two protocols produce very different accuracy distributions, the exact label-generation procedure must be specified.","section":"IV-A and IV-C"},{"comment":"The narrative is internally in tension: the text states that the migrated FP32 Pareto front “aligns closely” with the ground-truth INT4 front and that “the FP32 Pareto structure is largely preserved,” while Table III reports 0% front survival and the text concludes “complete reorganisation of the efficient frontier.” These statements can be reconciled (front membership can change while the curve location remains similar), but the paper should explain this explicitly instead of using both phrasings without clarification.","section":"IV-C, Fig. 6 vs Table III"}],"minor_comments":[{"comment":"Equation (5) includes a penalty term λ·max(0, R×C − MAX_PEs)×10^4, while the text says the valid space is reduced to ≈1,200 combinations by a hard constraint R×C ≤ MAX_PEs. Please clarify whether the constraint is enforced by filtering, by the penalty, or by both, and whether the penalty is ever active in the reported runs.","section":"III-C, Eq. (5)"},{"comment":"The sentence “In order to test different designs for each survivor model and decide on the best architecture, a DSE procedure for CGRA4ML is prepared” appears twice in succession; one occurrence should be removed.","section":"III-C"},{"comment":"Since the valid DSE space is stated to contain only ≈1,200 configurations and an analytical oracle is available, the paper should state whether the evolutionary DSE result was verified against exhaustive enumeration; this would strengthen the claim that the reported configuration is optimal.","section":"IV-D, Table IV"},{"comment":"The “Backend” column entries such as “fx” are not defined, and the meaning of “layer-wise” for the equalization and quantization-type columns should be stated explicitly.","section":"Table I"},{"comment":"The QONNX-to-QKeras translation layer (ref. [14], same first author) is used to bridge Brevitas quantized models to CGRA4ML, but no numerical validation is shown that the translated QKeras models reproduce the Brevitas quantized semantics. A small accuracy or activation-matching check would make the Stage III results more credible.","section":"III-C"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead this one. The full-space INT4 PTQ stability analysis on NAS-Bench-201 is the real contribution: 15,625 architectures, ground-truth quantized labels, and formal metrics like dominance flip rate and Pareto rank sensitivity. That part is reproducible from the benchmark plus Brevitas, and the visual in Fig. 6 plus the KT numbers tell a consistent story: the FP32 Pareto structure is largely preserved under INT4, even though the front membership completely changes. That is a genuinely new empirical finding, and it is useful for anyone doing hardware-aware NAS without folding quantization into search.\n\nThe paper also lays out a reasonable three-stage pipeline (NAS frontend, quantization bridge, CGRA DSE). The CGRA part is less convincing because the analytical oracle is taken on faith and the QONNX-to-QKeras translation layer is the authors' own prior work, but that is not the load-bearing claim.\n\nNow the soft spots, and they are real. The headline comparison — FP32 zero-shot vs 'fully trained INT4-specific surrogate' — is under-specified. No training protocol is given for the INT4 surrogate: architecture, epochs, data split, hyperparameters, nothing. And Table III literally labels both columns '(zero-shot)', which contradicts the prose. If the INT4 arm was not actually retrained, the headline result collapses into a comparison of two zero-shot surrogates, which is a different claim. Even taking the prose at face value, the RS hypervolume gap (0.5740 vs 0.5113) is within one standard deviation and not significant under a simple two-sample test; only the MOEA gap looks solid. The paper reports no significance tests and no paired confidence intervals. These are fixable issues, but they are load-bearing.\n\nThe stability metrics are solid, the transferability analysis is fine, and the authors are honest about the 0% front survival rate. The writing is clear. This is not a flawed paper; it is an under-described experiment. A serious referee should ask for the INT4 surrogate details, a corrected table, significance testing, and ideally artifacts. If those come back, the empirical claim will be citable.\n\nVerdict: give it a proper peer review, not a desk reject. It is a subfield-scale advance with a reproducible core. I would bring it to the reading group to discuss the methodology, but I would not cite it yet.","headline":"Full-space INT4 stability analysis is a real contribution, but the headline surrogate comparison is under-specified and needs a revision round before the empirical claim is citable.","tokens_in":8687,"tokens_out":2271,"would_cite":false,"duration_ms":22340,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"INT4 quantization reorganizes the Pareto front completely, yet the FP32-trained surrogate still wins on Pareto space coverage.","keywords":["neural architecture search","post-training quantization","Pareto front stability","zero-shot surrogate transfer","hardware-aware NAS","coarse-grained reconfigurable array","design space exploration","edge AI"],"falsifier":"Train an INT4 surrogate with the same architecture, epochs, data split, and hyperparameter budget as the FP32 surrogate, then rerun the 50-trial random-search and evolutionary comparisons on the same 15,625-architecture benchmark; if the FP32 zero-shot surrogate's normalized hypervolume advantage disappears or inverts, the paper's central claim fails.","tokens_in":7734,"feed_emoji":"⚙️","tokens_out":6357,"duration_ms":63858,"temperature":0.7,"pith_summary":"Post-training quantization to 4-bit integers reshuffles which neural architectures sit on the efficient frontier, but a search surrogate trained on clean full-precision data transfers to the quantized domain better than a surrogate trained directly on quantized labels. On a standard benchmark of 15,625 cell architectures, the paper reports normalized hypervolume ratios of 1.12 for random search and 1.07 for an evolutionary search in favor of the FP32 zero-shot surrogate, with a 0% survival rate for the original FP32 Pareto front under INT4. The paper also assembles these results into a three-stage pipeline: a hardware-agnostic search frontend, a quantization bridge that re-ranks and filters candidates, and an evolutionary backend that maps surviving quantized architectures onto a reconfigurable array accelerator. The practical upshot is that practitioners may not need to retrain their search surrogates when deployment shifts to quantized hardware, even though the Pareto-optimal architecture set changes entirely.","feed_headline":"FP32 search beats INT4-trained search on quantized Pareto space","feed_subtitle":"Zero-shot full-precision surrogate covers more of the INT4 efficient frontier, despite a 0% front survival rate.","key_machinery":"The load-bearing object is the Pareto rank surrogate: a predictor trained to reproduce the relative multi-objective ordering of architectures rather than to regress each objective separately, using a listwise ranking loss. The paper builds a three-stage pipeline around it: Stage I searches a standard 15,625-architecture cell-based benchmark using accuracy and FLOPs; Stage II applies INT4 post-training quantization, re-ranks the candidates, filters dominated architectures, and triggers a new search when no survivors remain; Stage III runs an evolutionary design-space exploration over a reconfigurable array compiler, scoring hardware configurations by a three-term normalized fitness of latency, processing-element idle ratio, and area. The empirical engine is the full ground truth for all architectures in both FP32 and INT4, which lets the Pareto stability metrics be computed exactly rather than estimated.","core_discovery":"The paper's central empirical finding is that post-training quantization to INT4 does not merely degrade accuracy; it reorganizes the entire Pareto structure of the search space. The survival rate of the FP32 efficient frontier under INT4 is 0%, one in five dominance relationships flips, and the ground-truth rank correlation drops to 0.6655. Despite this reorganisation, a surrogate trained on FP32 labels transfers zero-shot to the INT4 domain with Kendall's tau of 0.7219, versus 0.8352 in-domain and 0.8219 after fine-tuning, and it achieves higher normalized hypervolume than an INT4-specific surrogate by 12.26% under random search and 6.77% under a multi-objective evolutionary algorithm. The authors attribute this win to the FP32 surrogate's less noisy training signal and to sufficient correlation between FP32 and INT4 ranking landscapes. In addition, the paper presents the full three-stage hardware-aware NAS workflow that this finding supports.","pith_inferences":["If the 0% front-survival result generalizes to other search spaces, then the FP32 Pareto front should be treated only as a candidate pool for quantized deployment, never as the final ranking.","The reported hypervolume advantage may shrink or invert if the INT4 surrogate is trained with matched capacity, epochs, and data splits; a controlled head-to-head is a direct next experiment.","The same stability metrics could serve as a quantitative yardstick for quantization-aware NAS methods: raising front survival or lowering the dominance flip rate would demonstrate a genuine benefit over post-search quantization.","Extending the comparison to INT8 or mixed-precision policies would test whether the zero-shot advantage shrinks as quantized labels become less noisy relative to FP32."],"forward_implications":["A deployed workflow can keep a full-precision NAS surrogate and still cover the quantized Pareto space effectively; retraining the surrogate on quantized labels is not necessary for INT4.","Because the INT4 efficient frontier shares no architectures with the FP32 front, any post-search quantization pipeline must re-rank candidates after quantization rather than trusting FP32-optimality.","The INT4-trained surrogate's tendency toward higher-FLOPs architectures suggests that quantized training signals can bias search away from the low-compute region that hypervolume rewards.","Architecturally similar survivors converge to the same optimal accelerator configuration, so hardware mapping effort can be amortized across a family of candidate architectures.","The quantization bridge's feedback loop keeps the pipeline well-defined when quantization eliminates the entire candidate set, by triggering a fresh search."],"supporting_citations":[{"why":"Provides the cell-based search space and full FP32 ground-truth accuracy for all 15,625 architectures, the substrate for both surrogates and all Pareto metrics.","marker":"[3]"},{"why":"Supplies the Pareto rank-preserving surrogate architecture and listwise ranking loss that the Stage I frontend redesigns and retrains.","marker":"[8]"},{"why":"Provides the reconfigurable-array compiler and the analytical performance oracle used by the Stage III design-space exploration.","marker":"[5]"},{"why":"Provides the conversion layer from the quantized-model format to the compiler-compatible format, enabling the backend to consume the quantized architectures.","marker":"[14]"},{"why":"Documents the existing hardware-aware benchmark gap (FP32-only metrics with no quantization), motivating the quantization bridge contribution.","marker":"[2]"},{"why":"Supplies the image-classification dataset on which the benchmark accuracies and therefore the surrogate training are based.","marker":"[4]"}],"fun_headline_variants":["FP32 surrogate out-covers INT4-trained on quantized NAS-Bench-201","Zero-shot FP32 predictor wins on INT4 Pareto space despite 0% survival","PTQ reshuffles NAS Pareto; FP32 surrogate still leads in coverage","Survival rate 0% but FP32 surrogate wins on INT4 Pareto coverage"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes the INT4-trained surrogate is trained as fairly and strongly as the FP32 surrogate, yet the paper gives no training details for it; if that surrogate is undertuned or disadvantaged by noisy quantized labels, the conclusion that FP32 zero-shot transfer is superior would not follow.","fun_headline_variants_meta":{"raw":{"variants":["FP32 surrogate out-covers INT4-trained on quantized NAS-Bench-201","Zero-shot FP32 predictor wins on INT4 Pareto space despite 0% survival","PTQ reshuffles NAS Pareto; FP32 surrogate still leads in coverage","Survival rate 0% but FP32 surrogate wins on INT4 Pareto coverage"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000699,"raw_usage":{"total_tokens":3164,"prompt_tokens":962,"completion_tokens":2202,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":578,"completion_tokens_details":{"reasoning_tokens":2115}},"tokens_in":578,"tokens_out":2202,"duration_ms":18885,"temperature":1.0,"reasoning_tokens":2115,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:26:11.222153+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train an INT4 surrogate with the same architecture, epochs, data split, and hyperparameter budget as the FP32 surrogate, then rerun the 50-trial random-search and evolutionary comparisons on the same 15,625-architecture benchmark; if the FP32 zero-shot surrogate's normalized hypervolume advantage disappears or inverts, the paper's central claim fails.","supporting_citations":[{"cited_title":"”Multi-objective hardware-aware neural ar- chitecture search with Pareto rank-preserving surrogate models.” ACM Transactions on Architecture and Code Optimization 20.2 (2023): 1-21","cited_arxiv_id":null,"evidence_quote":"Supplies the Pareto rank-preserving surrogate architecture and listwise ranking loss that the Stage I frontend redesigns and retrains."},{"cited_title":"”A Unified FPGA/CGRA Acceleration Pipeline for Time-Critical Edge AI: Case Study on Autoencoder-Based Anomaly Detection in Smart Grids.” Electronics 15.2 (2026): 414","cited_arxiv_id":null,"evidence_quote":"Provides the conversion layer from the quantized-model format to the compiler-compatible format, enabling the backend to consume the quantized architectures."},{"cited_title":"Learning multiple layers of features from tiny images","cited_arxiv_id":null,"evidence_quote":"Supplies the image-classification dataset on which the benchmark accuracies and therefore the surrogate training are based."}],"review_version":1}