{"id":"65df4a5c-3abb-432f-b942-b5a524d5af48","arxiv_id":"2607.25884","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"CONQuER shifts mixed-precision quantisation search into the MLIR/TOSA compiler, uses dual surrogates to prune candidates, and calibrates online with hardware-in-the-loop runs, reporting up to 12.19x faster inference with small accuracy loss on an A100.","lead":"CONQuER is a compiler-integrated tool that searches for mixed-precision quantisation settings inside the MLIR compiler pipeline, testing only the most promising candidates on real hardware. If the approach generalises, it could make deploying neural networks on mobile and server hardware faster and less fragmented.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The hardware-in-the-loop accounting in §II.C/§III.A is self-contradictory: 1,440 total hardware inferences cannot cover 512-image evaluations for enough candidates to produce the reported Pareto fronts; the search budget must be audited before the central claim is accepted.","rationale":"The reader's CONDITIONAL verdict is appropriate, and my concern sharpens it rather than changing it. The reader identified the accuracy surrogate as the weakest assumption, but I find a more fundamental and easily checkable problem: the HIL budget as written contradicts the per-candidate measurement protocol. If the budget is genuinely 1,440 image inferences, the evolutionary search described in Section II.C cannot have run—there are simply not enough evaluations to populate a Pareto front. If the budget is instead 1,440 candidate evaluations, then the term 'hardware inferences' is used inconsistently and the comparison with SeQTO is compromised. Either way, the central claim that CONQuER discovers these policies through online-calibrated HIL search is not yet reproducible from the text. I would not reject outright because the released replication package may resolve the ambiguity; the concrete test above would settle it. Credit is due where the paper has independent support: the artifact is public, the ablation study is sensible, and the final reported accuracy is hardware-measured. But the budget inconsistency is a real threat to internal validity, and the paper should either report a corrected accounting or explicitly define 'hardware inference' before the core result can be accepted.","tokens_in":16270,"tokens_out":5689,"duration_ms":57158,"concrete_test":"Instrument the released CONQuER code (https://github.com/dakaidan/CONQuER-Replication) for one representative run, e.g., ResNet-50 on the NVIDIA A100, and log every HIL evaluation: count (a) candidate policies compiled/executed, (b) images per candidate (expected 512), and (c) timing runs. Compute total hardware inferences = candidates × 512 (+ timing overhead) and compare with the stated 1,440 cap, and with the number needed for ~34 generations of the reported population. Also inspect the search configuration files for population size and per-generation HIL candidate count. If the total is ≤1,440, explain how Figure 3 and Table IV were generated; if it is >1,440, either the cap or the per-candidate 512-image description must be corrected. This audit settles whether the empirical core of the paper is reproducible.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing issue is the search's hardware-in-the-loop accounting, not the accuracy surrogate. Section II.C states that each HIL-evaluated candidate measures accuracy on a distinct 512-image split, and Section III.A says the physical evaluation budget for both SeQTO and CONQuER was 'strictly capped at a maximum of 1,440 hardware inferences.' If 'hardware inference' means one image processed through the model, then the entire CONQuER run could HIL-evaluate at most floor(1440/512)=2 candidates (plus timing). That cannot produce the converged Pareto fronts reported in Table IV, the generational HV trajectories in Figure 3, or the headline 12.19x/1.44% result. If 'hardware inference' instead means one candidate-policy evaluation, then the cap is not comparable to SeQTO's per-image profiling, and the term is misleading. The paper never reports population size, number of generations, or the number of HIL-evaluated candidates, so the reader cannot determine which reading is intended. This is load-bearing because the central claim—that online-calibrated HIL search discovers these policies—requires enough HIL evaluations to actually drive the search. The accuracy-surrogate concern is secondary: final configurations are measured on hardware, so an imperfect surrogate can weaken the search but not falsify the reported measurements; an unworkable HIL budget would falsify the methodology as described.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents CONQuER, a compiler-integrated mixed-precision quantisation (MPQ) infrastructure. Quantisation is performed natively at the MLIR TOSA level; an NSGA-II evolutionary search generates candidate policies, a dual-surrogate pre-screening engine (a roofline-style hardware model and an entropy/isotropy-based accuracy model) filters candidates, and the most promising candidates are compiled and executed on physical hardware via IREE. An online logarithmic calibrator is used to align the surrogates with measured hardware behaviour. The paper reports speedups over an unquantised MLIR FP32 baseline across a Snapdragon 8 Elite, an Intel i5-1145G7, and an NVIDIA A100, with a headline result of 12.19x faster ResNet-50 inference on the A100 at a 1.44% top-1 accuracy drop. It also reports cross-target transfer penalties, ablations of the surrogate components, and comparisons against SeQTO and InfoQ.","tokens_in":16635,"tokens_out":6589,"duration_ms":66520,"significance":"If the experimental claims hold, the paper makes a useful contribution: moving MPQ search into the compiler IR and coupling it to physical execution is a sensible way to avoid the front-end/framework fragmentation that often makes quantised models unsupported or slow after lowering. The final Pareto points are measured directly on hardware rather than predicted by the surrogate, which limits circularity; the cross-target transfer experiments and the surrogate ablation are valuable evidence. The paper also provides a source-code link and repeats runs with five seeds. However, the reported hardware-in-the-loop budget is internally inconsistent, and the comparison against InfoQ is confounded by different runtime stacks. These issues currently prevent verification of the central claim and require a major revision before the results can be accepted.","major_comments":[{"comment":"The hardware-in-the-loop accounting is self-contradictory. §II.C states that each HIL-evaluated candidate measures accuracy on a distinct 512-image split, in addition to 3 warm-up and 10 timed latency runs. §III.A then states that the physical evaluation budget for both SeQTO and CONQuER was 'strictly capped at a maximum of 1,440 hardware inferences.' If 'hardware inference' means one image processed by the model, the entire CONQuER run can HIL-evaluate at most floor(1440/512)=2 accuracy candidates, which cannot produce the converged Pareto fronts in Table IV, the generational trajectories in Figure 3, or the reported 12.19x result. If 'hardware inference' instead means one candidate-policy evaluation, the cap is not comparable to SeQTO's per-image profiling and the term is misleading. The paper never reports population size, number of generations, or the number of HIL-evaluated candidat","section":"§II.C/§III.A"},{"comment":"The InfoQ comparison is not apples-to-apples. §III.A says InfoQ's execution latency is evaluated natively using the PyTorch runtime rather than the compiled IREE stack, while CONQuER is compiled and executed via IREE. The large speedup differences in Table II could therefore be due to the runtime/compiler stack rather than to the quantisation allocation method. This confound is especially important for the claim that InfoQ 'universally fails to achieve an absolute speedup.' Please measure InfoQ policies on the same IREE pipeline where they lower successfully, or, if that is impossible, report a same-runtime baseline (e.g., CONQuER-generated policies executed in PyTorch) and separate the effect of the search method from the effect of the deployment stack.","section":"§III.A/Table II"},{"comment":"The accuracy surrogate is an unvalidated additive model. The paper assumes that total accuracy degradation is the sum of per-node entropy-weighted noise terms, where the entropy H of activation-covariance eigenvalues is multiplied by quantisation noise amplitude ≈ 2^{-b}. No evidence is provided that this proxy is predictive of actual top-1 accuracy for the five models evaluated, nor that different layers are independent in their accuracy effects. The ablation in Table IV shows that the dual proxy outperforms unguided search, but it does not test whether the surrogate's ranking agrees with hardware-measured accuracy. Since the search uses the surrogate to discard candidates before HIL evaluation, a mis-calibrated proxy could eliminate the best policies. Please report, at minimum, a correlation or rank-agreement analysis between surrogate scores and measured accuracy for a sample of evalu","section":"§II.B"}],"minor_comments":[{"comment":"Typo: 'different of model layers' should be 'different model layers' or 'different combinations of model layers.'","section":"Abstract"},{"comment":"The online calibrator is described only as y = a ln(x) + b with a sliding window. It is unclear which surrogate output x is the input, which measured quantity y is the target, how often the coefficients are updated, and whether the calibrator is applied to the accuracy surrogate, the latency surrogate, or both. Please specify the algorithm precisely; this is a named contribution but is not currently reproducible from the text.","section":"§II.C"},{"comment":"In the caption and table header, SeQTO is referenced as '[3]' in the column heading but the bibliography gives SeQTO as reference [6]. Please correct the citation.","section":"Table I"},{"comment":"The InfoQ QAT rows are ambiguous: the column label 'QAT Acc↓' reports positive values such as +0.96%, which the text then describes as accuracy being 'resolved' or higher than the original model. Clarify the sign convention and whether the QAT numbers are measured on the training set or the validation set; as written they appear inconsistent.","section":"Table II"},{"comment":"In the Intel i5 column, the ResNet-50 CONQuER latency is listed as '69.06' without the 'ms' unit. Please add the unit.","section":"Table II"},{"comment":"The caption says hypervolumes are 'normalised,' but the y-axis values differ substantially across panels (e.g., MobileNetV2 around 0.02-0.045 vs. ResNet-50 around 0.40-0.48). Clarify what normalisation is used or remove the word 'normalised.'","section":"Figure 3"},{"comment":"The hardware target names are inconsistent: 'Intel I5 1145g7' in most places but 'Intel Core i5' in Figure 2 and 'i5' elsewhere. Use a single consistent identifier.","section":"§III.A"}],"recommendation":"major_revision","confidential_remarks":"The HIL budget audit is the key blocker. If the 1,440-inference cap is meant per candidate evaluation, the comparison with SeQTO is unfair; if it is meant per image, the described search cannot have run as reported. I would ask the authors for raw search logs, population sizes, generation counts, and exact HIL evaluation counts before considering the paper further. The InfoQ runtime confound and the unvalidated accuracy surrogate are also significant but potentially addressable in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: CONQuER is a legitimate addition to the quantization tooling literature. Moving mixed-precision search into MLIR's TOSA level and coupling NSGA-II with a roofline latency model plus an entropy-based accuracy filter is a sensible integration that I don't think anyone has done in exactly this form. The ablation showing that the dual proxy beats either single proxy or no proxy, and the transfer study showing policies are hardware-specific, are useful findings. The comparison also exposes real deployment friction in the baselines: InfoQ policies often don't lower through IREE, and SeQTO's greedy search gets stuck in local optima. That part reads honestly.\n\nThe problem is the hardware-in-the-loop budget. Section II.C says each HIL-evaluated candidate measures accuracy on a distinct 512-image split. Section III.A says both SeQTO and CONQuER were strictly capped at 1,440 hardware inferences. If \"hardware inference\" means one image, then CONQuER can HIL-evaluate at most two or three candidates—there's no way that produces the Pareto fronts in Table IV, the generational trajectories in Figure 3, or a meaningful online calibrator. If it means one candidate evaluation, the term is misleading and the comparison isn't apples-to-apples with SeQTO. The paper never reports population size, number of generations, or the actual number of HIL evaluations, so the reader can't disentangle this. That is load-bearing: the headline 12.19x speedup is only credible as a product of HIL-driven search.\n\nOther issues are smaller. The accuracy surrogate is never validated against measured accuracy, but since final configurations are hardware-measured, an imperfect surrogate only weakens search efficiency, not the reported measurements. A uniform-int8 baseline under the same IREE stack is missing—right now the speedups are all relative to FP32, which makes it hard to tell how much of the gain is from quantizing at all. Some of the variance numbers in Table I are alarming (the ±11.12% on MobileNetV2 accuracy). And Section IV.B's statement that ResNet-50 reduces to 24.11 ms natively on A100 doesn't square with the 8.43 ms in Table II.\n\nBottom line: the software engineering contribution is real and the paper deserves a refereed venue, but the budget accounting has to be fixed and the evaluation protocol clarified before the empirical claims are believable. If these are reporting errors, the correction is straightforward—but as written, the central experiment is uninterpretable.\n\nI'd take it to a reading group to discuss the methodology, and I'd cite it for the TOSA-integration idea, but I wouldn't rely on its numbers until the revision.","headline":"The TOSA-level integration is a genuine step forward, but the claimed 1,440-inference HIL budget contradicts the 512-image-per-candidate protocol, and the headline numbers are uninterpretable until that is resolved.","tokens_in":17142,"tokens_out":6675,"would_cite":true,"duration_ms":56885,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CONQuER shifts mixed-precision quantisation into the compiler and finds hardware-specific policies up to 12.19x faster inference, with top-1 accuracy within 1.44% of the unquantised baseline.","keywords":["mixed-precision quantisation","hardware-aware optimisation","compiler-integrated deployment","MLIR/TOSA","NSGA-II","surrogate-guided search","hardware-in-the-loop calibration","Pareto-optimal neural network deployment"],"falsifier":"A direct experiment that uses the accuracy surrogate alone to rank candidate mixed-precision policies, then measures their true top-1 accuracy on a held-out validation set, would settle it: if the surrogate's predicted ordering disagrees with the measured ordering on any of the benchmark models, the pre-screening can silently discard the best configurations.","tokens_in":16154,"feed_emoji":"⚙️","tokens_out":6119,"duration_ms":53268,"temperature":0.7,"pith_summary":"The paper's central claim is that the quality of a mixed-precision quantisation policy cannot be separated from the compiler pipeline and physical hardware that will execute it. It argues that current workflows, which quantise models in a hardware-agnostic front-end step, produce policies that map poorly to heterogeneous execution blocks and can even fail to compile. To fix this, it proposes CONQuER, a compiler-integrated infrastructure that generates and evaluates quantised graphs directly in the MLIR TOSA intermediate representation. A dual-surrogate filter prunes the exponential search space before expensive hardware-in-the-loop evaluation, and an online calibrator keeps the surrogates honest. If the central claim is right, developers should expect optimal bit-width assignments to be hardware-specific, and deployment toolchains should treat quantisation as a compiler optimisation rather than a preprocessing step.","feed_headline":"Move quantisation into the compiler: 12.19x faster inference","feed_subtitle":"Hardware-in-the-loop search finds bit-width policies that stay within 1.44% of full-precision accuracy.","key_machinery":"Dual-surrogate pre-screening engine: a lightweight two-part filter combining a roofline model of node latency (compute vs memory bounds plus dispatch overhead) with an information-theoretic accuracy fragility metric (normalised Shannon entropy of activation-covariance eigenvalues, weighted by 2^{-b} quantisation noise). It prunes the combinatorial space before any physical compilation; the NSGA-II evolutionary search then evaluates only the strongest candidates on hardware, and an online logarithmic calibrator aligns the surrogates with observed latency and accuracy across generations.","core_discovery":"The paper claims that mixed-precision quantisation is best done inside the compiler's middle-end, at the MLIR TOSA level, rather than as an external preprocessing step. It introduces CONQuER, which generates candidate quantised graphs natively in the IR, uses a dual-surrogate pre-screening engine — a roofline-based hardware cost model plus an information-theoretic accuracy proxy based on the Shannon entropy of activation-covariance eigenvalues — to discard weak configurations cheaply, and then compiles and executes only the most promising candidates on the target hardware via IREE. An online logarithmic calibrator continuously realigns the surrogates with measured hardware behaviour during a","pith_inferences":["A natural extension is to validate the accuracy surrogate as a standalone, no-training sensitivity score: if its predicted per-layer ordering matches measured accuracy drops, it could be reused outside CONQuER (e.g., in other search loops or as a diagnostic).","The online calibrator's logarithmic form could be evolved into a learned backend cost model that amortises calibration across many models compiled for the same target.","For dispatch-bound targets like large GPUs running small models, the paper's own data suggests that an early 'compute-proportionality' check could save search budget by falling back to uniform precision or other optimisations.","The same compiler-integrated search could be extended to other precision-related compiler decisions, such as operator tiling and vectorisation, turning mixed-precision quantisation into one axis of a broader co-optimisation problem."],"forward_implications":["If CONQuER's claims hold, mixed-precision policies should be tuned per deployment target; the paper reports that transferring a policy across targets misses the native optimum and adds measurable latency penalties in every tested case.","Compiler-integrated search prevents the silent deployment failures observed when externally quantised models cannot be lowered by the backend.","A strictly capped hardware evaluation budget (1,440 inferences) is sufficient to find strong Pareto-optimal policies when guided by dual surrogates.","On compute-proportional hardware, substantial speedups (up to 12.19x) are achievable with accuracy loss below 1.5%; on laptops with FP16 unpacking penalties, the search limits regression but cannot beat FP32.","Single-objective surrogates skew the Pareto front; the dual-surrogate configuration produces more stable, higher hypervolume convergence."],"fun_headline_variants":["Quantize inside the compiler for 12.19x faster inference","Compiler-integrated MPQ: 12.19x faster, within 1.44% accuracy","Evolutionary search picks bit-widths: 12.19x faster","Optimal quantization is hardware-specific — CONQuER finds it","MPQ in the compiler: 12.19x speedup, 1.44% accuracy loss"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The accuracy surrogate assumes that the Shannon entropy of activation-covariance eigenvalues, weighted by 2^{-b} quantisation noise, is an additive per-layer predictor of final top-1 accuracy, and the paper does not validate this predictor against the reported models.","fun_headline_variants_meta":{"raw":{"variants":["Quantize inside the compiler for 12.19x faster inference","Compiler-integrated MPQ: 12.19x faster, within 1.44% accuracy","Evolutionary search picks bit-widths: 12.19x faster","Optimal quantization is hardware-specific — CONQuER finds it","MPQ in the compiler: 12.19x speedup, 1.44% accuracy loss"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001208,"raw_usage":{"total_tokens":4862,"prompt_tokens":845,"completion_tokens":4017,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":589,"completion_tokens_details":{"reasoning_tokens":3921}},"tokens_in":589,"tokens_out":4017,"duration_ms":25219,"temperature":1.0,"reasoning_tokens":3921,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T01:11:46.379594+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct experiment that uses the accuracy surrogate alone to rank candidate mixed-precision policies, then measures their true top-1 accuracy on a held-out validation set, would settle it: if the surrogate's predicted ordering disagrees with the measured ordering on any of the benchmark models, the pre-screening can silently discard the best configurations.","supporting_citations":[],"review_version":1}