{"id":"57bbb837-29ba-4d24-b93c-dab3194f10ce","arxiv_id":"2505.13060","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"The paper derives an additive loss-MSE sensitivity metric and a hardware-aware time-gain model, then uses integer programming to assign per-layer FP8 or BF16 formats for LLM inference.","lead":"This paper presents an automatic mixed-precision quantization method that chooses which layers of a large language model use a lower-precision format to speed up inference while keeping the loss error under a preset limit. It combines a Taylor-expansion sensitivity metric, measured execution times of sequential subgraphs, and integer programming, with experiments on Intel Gaudi 2.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Loss-MSE additivity relies on unverified cross-layer noise independence; validation on 1B-selected configs does not establish the 8B constraint as a valid proxy.","rationale":"Good-faith reading: the paper makes an engineering claim that its sensitivity metric, empirical time-gain measurements, and IP formulation together select mixed-precision configurations that maximize time gain under a loss-MSE constraint. The paper explicitly acknowledges the additivity assumptions and provides validation on the 1B model, which is a genuine strength. However, the most load-bearing condition for the central claim is the validity of the additive loss-MSE model, because the IP constraint and the notion of optimality are defined through it. The assumption that quantization noise is independent across layers is mathematically suspect for sequential computation: activation errors from early layers propagate and enter the deviations at later layers, creating correlations. The paper's validation does not settle this because it uses only IP-ET-selected configurations on 1B, which are precisely the ones the model considers low-loss, and there is no 8B validation. Consequently, the reported 8B speedup, while empirically measured, cannot be attributed to the proposed constrained-optimization mechanism without additional evidence. This does not falsify the method; it makes the 8B claim conditional. The reader's CONDITIONAL verdict is appropriate and no change is needed.","tokens_in":11328,"tokens_out":14013,"duration_ms":138939,"concrete_test":"Estimate the actual loss MSE E[(\\hat g - g)^2] on a held-out calibration set for Llama-3.1-8B under (i) IP-ET solutions for tau in {0.1%, 0.3%, 0.5%, 0.7%}, (ii) all-FP8, and (iii) a random sample of configurations spanning a wide range of predicted loss MSE from Eq. (23). Plot predicted vs. measured loss MSE. If the relative error exceeds 25% for any configuration with predicted loss near the threshold, or if rank ordering differs substantially, the additivity assumption fails on 8B. Repeat the same comparison on the 1B model including non-IP-ET configurations to rule out selection bias in Fig. 3a.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Equation (23) defines the per-group loss MSE as a sum of independent per-layer contributions, and Eq. (5) uses this sum as the IP constraint. The independence assumption is not innocuous: the quantities \\tilde z_l in Eq. (14) are the deviations of the possibly-quantized extended inputs from their full-precision values. For sequential layers, \\tilde z_l and \\tilde z_{l-1} share propagated activation quantization error, so they are correlated and the cross-terms in E[\\tilde g^2] need not vanish. If those cross-terms are non-negligible, the IP constraint can misorder configurations and the selected solution need not satisfy the true loss-MSE bound. The paper's validation (Fig. 3a) is limited to the 1B model and to configurations generated by IP-ET at thresholds up to 0.7%; these are exactly the configurations the additive model judges safe, so the comparison does not probe where the approximation could break down. No 8B loss-MSE validation is provided. Since the 8B accuracy-latency curves are a headline result, the claimed mechanism for IP-ET on 8B is unsupported without a direct validation of Eq. (23) on 8B or an argument for why cross-layer correlations are negligible.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an automatic mixed-precision (MP) selection method for post-training quantization of LLMs. The method computes a per-layer sensitivity metric from a first-order Taylor expansion of the loss with respect to quantization noise in weights and activations, then assumes the resulting loss-MSE components are additive across layers. It partitions the model graph into sequential sub-graphs, measures empirical time gains for each sub-graph and each MP configuration, and solves an integer program that maximizes a performance metric (empirical time, theoretical MAC-based time, or memory) subject to a loss-MSE threshold. Experiments on Intel Gaudi 2 with Llama-3.2-1B and Llama-3.1-8B compare the proposed strategies (IP-ET, IP-TT, IP-M) with Random and Prefix baselines, reporting accuracy-versus-latency improvements, including the headline claim that IP-ET achieves accuracy loss below 0.1% at 450 ms TTFT on 8B whereas baselines require about 600 ms.","tokens_in":11543,"tokens_out":4154,"duration_ms":49220,"significance":"If the claims hold, the paper makes a useful engineering contribution: it gives a concrete, hardware-aware way to select MP configurations using an additive loss-MSE proxy and measured per-subgraph time gains, and it directly validates the additivity assumptions on the 1B model. The use of 10 seeds, multiple tasks, and a public evaluation harness strengthens the empirical part. The main weaknesses are that the loss-MSE independence assumption and the time-gain additivity are validated only on the 1B model and only on configurations selected by the proposed method itself, while the headline 8B result depends on these assumptions. The method's value is also constrained by the absence of a public code release and by the task-dependent correlation between the loss-MSE proxy and final accuracy.","major_comments":[{"comment":"The loss-MSE additivity in Eq. (23) rests on the assumption that quantization noise is statistically independent across layers. This is not innocuous for sequential layers: the noisy extended input z_l in Eq. (14) contains propagated activation quantization error from earlier layers, so the cross terms in E[(sum_l (z_l)^T grad g)^2] need not vanish. The validation in Fig. 3a covers only the 1B model and only the configurations produced by IP-ET for thresholds up to 0.7%; these are exactly the configurations the additive model judges safe, so the comparison does not probe configurations where the approximation could break down. Since the IP constraint in Eq. (5) is the mechanism that controls quality, the 8B accuracy-latency result is not fully supported without either direct 8B validation of Eq. (23) or an argument that cross-layer correlations are negligible for these models.","section":"§2.2, Eq. (23)"},{"comment":"The empirical-time-gain additivity is validated only on the 1B model, while the headline 8B result in Fig. 5b relies on IP-ET, which selects configurations using per-group time-gain sums. If the additive model misorders configurations on the 8B model, the IP-selected configuration may be suboptimal even though the final TTFT is measured directly. Please provide a validation of predicted versus measured total time gain for 8B configurations, preferably including configurations not selected by IP-ET.","section":"§3.2, Fig. 3b"},{"comment":"The paper treats loss-MSE as a proxy for accuracy, but Table 1 and Appendix C.1 show that this proxy can be misleading: on 8B LAMBADA, IP-ET has lower perplexity difference (0.922 vs. 1.075) but worse accuracy difference (-0.229 vs. -0.029) than Prefix. This does not invalidate the method, but it means the accuracy-latency comparison is task-dependent. Please report the correlation between the loss-MSE/constraint level and actual accuracy degradation, or at least discuss why the proxy is expected to track accuracy on the averaged tasks used for the headline claim.","section":"§3.4, Table 1"}],"minor_comments":[{"comment":"There are several typographical errors, e.g., 'weighta optimizer' in the Introduction and 'ℓ∈Llin SLBGEMM' around Eq. (17).","section":"Abstract / Introduction"},{"comment":"The text says Figure 3a and 3b correspond to empirical time gain and loss MSE 'respectively,' but the caption lists loss MSE in (a) and TTFT reduction in (b); the references should be aligned.","section":"§3.2, Fig. 3 caption"},{"comment":"The notation 'next[vertex]' is not defined in the algorithm or the text; please define the successor function explicitly.","section":"Algorithm 2"},{"comment":"The second sentence refers to memory gain as 'defined in Sec. 2.3.2,' but memory gain is defined in Sec. 2.3.3.","section":"Appendix C.3"},{"comment":"The caption ends with 'Random, Prefix' and omits a closing parenthesis; also, the caption for Fig. 9a/9b refers to IP-TT, but the figure appears to show IP-M results.","section":"Figure 7 caption"},{"comment":"The per-MAC time-gain parameter δ_{T,f} is introduced without specifying how it is obtained or calibrated; if it is a fixed hardware parameter, please state this explicitly.","section":"§2.3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is authored by Intel researchers and cites a recent Intel/Habana FP8 paper by some of the same authors; I do not see this as a circularity issue since the citation is used for FP8-on-Gaudi context and not to establish the paper's central claims. The main concern is the missing direct validation of the loss-MSE and time-gain additivity assumptions on the 8B model, which is the model featured in the headline speedup claim. If the authors can supply such validation, or convincingly argue that the 1B validation transfers, the manuscript could become publishable; as it stands, the empirical support for the central mechanism is incomplete."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis paper proposes an automatic way to select a per-layer FP8/BF16 mixed-precision configuration for LLM prefill on Intel Gaudi 2. The new pieces are a first-order Taylor-based loss-MSE sensitivity metric computed per layer from one forward/backward pass, an empirical time-gain model that partitions the graph into sequential subgraphs and measures each subgraph's gain directly, and an integer program that maximizes time gain under a loss-MSE budget. The combination is new, and Figure 1 makes a real point: per-layer time gains summed do not predict a parallel subgraph's gain, while the subgraph-level measurement does. Validation is genuine—10-seed accuracy on Llama-3.2-1B and Llama-3.1-8B, a direct check of the additive loss-MSE and time-gain estimates on the 1B, and a full table. The paper also states its assumptions rather than hiding them: independence of quantization noise across layers in Eq. 23, and MSE as a proxy for accuracy.\n\nThe soft spots are real but not fatal. The independence assumption is load-bearing for the IP constraint, and the stress-test point is correct: sequentially propagated quantization errors are correlated, so the cross-terms in the loss-MSE sum need not vanish. The validation only covers configurations selected by IP-ET on the 1B model, and it uses thresholds up to 0.7%—exactly the safe region. The headline 8B accuracy-latency curves come without a direct validation of Eq. 23 on 8B, so the constraint's validity on the larger model is taken on faith. That weakens the empirical claim but does not falsify the method; it is a testable assumption and the authors should test it. The baselines are weak (random and prefix), and no code or data is provided, which hurts reproducibility.\n\nThis is an honest engineering paper with a useful contribution for anyone doing post-training mixed precision on accelerators. The time-gain partitioning alone is worth citing. It deserves a serious review; with a revision that validates the additive loss-MSE on 8B (or argues why cross-layer correlation is negligible) and releases code, it would be a solid applied-ML/systems paper. The self-citation to Lee et al. is fine—it is used for FP8-on-Gaudi background, not to bias the evaluation. I would bring it to reading group as a good example of hardware-aware quantization with careful measurement.","headline":"A genuinely useful hardware-aware mixed-precision selector with honest assumptions, but the 8B headline result needs a direct check of the additive loss-MSE constraint.","tokens_in":12108,"tokens_out":2378,"would_cite":true,"duration_ms":24857,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A first-order Taylor expansion of loss under quantization noise, combined with measured time gains per sequential subgraph, lets integer programming pick the mixed-precision configuration that maximizes speed for a fixed loss-error budget.","keywords":["mixed precision","post-training quantization","loss MSE sensitivity","integer programming","sequential subgraphs","LLM inference","Gaudi 2","time-to-first-token"],"falsifier":"Quantize one early layer alone and measure end-to-end loss MSE, then one late layer alone, then both together; compare the joint measurement with the sum of the two single-layer measurements. If the joint error is not close to the sum, or if it reverses the ordering of two candidate configurations, the additive constraint that the integer program relies on fails.","tokens_in":11097,"feed_emoji":"⚡","tokens_out":6899,"duration_ms":63649,"temperature":0.7,"pith_summary":"The paper's aim is to make post-training mixed-precision selection automatic and hardware-aware: given a model, a calibration set, and a loss-error budget, choose which layers run in which floating-point format. It claims that two quantities can be treated as additive: the mean-squared error of the loss contributed by quantizing a layer, estimated from a first-order Taylor expansion of the loss as a function of weight and activation quantization noise, and the inference-time gain of a sequential subgraph, measured directly by running the model with each candidate configuration. With these values, an integer program selects the configuration that maximizes time, memory, or MAC-based gain while keeping loss MSE below a threshold. On Llama-3.2-1B and Llama-3.1-8B on Gaudi 2, the paper reports that this selection dominates random and prefix baselines across accuracy-versus-latency curves; for the 8B model it achieves accuracy loss below 0.1% at 450 ms time-to-first-token where baselines need about 600 ms.","feed_headline":"Mixed-precision selector beats random and prefix baselines","feed_subtitle":"On Llama-3.2-1B and Llama-3.1-8B, IP-based selection hits accuracy targets faster on Gaudi 2.","key_machinery":"The carrying object is the per-layer sensitivity scalar $s_\\ell = \\frac{1}{R}\\sum_r \\|z^r_\\ell \\odot \\dot z^r_\\ell\\|^2$, computed with high-precision forward and backward passes on a small calibration set. It converts quantization into an additive loss-MSE model through $d_{\\ell,f}=s_\\ell \\alpha_f$ with $\\alpha_f = 2^{-2m_f}/12$, and the group MSE is the sum of per-layer terms under an independence assumption. The second mechanism is graph partitioning: the computation DAG is split into single-entry/single-exit subgraphs whose execution is strictly sequential, and each subgraph's time gain for every candidate configuration is measured directly rather than predicted from MAC counts; the paper shows that per-layer time-gain sums badly miss the measured time gain of a parallel attention subgraph. The integer program then maximizes $\\sum_{j,p} c_{j,p} i_{j,p}$ subject to $\\sum_{j,p} d_{j,p} i_{j,p} \\le \\tau^2 \\mathbb{E}[g^2]$ and exactly one selected configuration per group.","core_discovery":"The central discovery is that mixed-precision configuration can be framed as an integer program with additive linear objectives and a single additive constraint. Each layer's quantization sensitivity $s_\\ell$ is the average squared norm of the element-wise product of its extended input and its loss gradient; multiplying by $\\alpha_f = 2^{-2m_f}/12$, the variance of a uniformly distributed mantissa-rounding error, gives that layer's loss-MSE contribution for format $f$. Summing these contributions over layers gives the predicted loss MSE of any configuration, and summing measured time-to-first-token gains over single-entry/single-exit sequential subgraphs gives the predicted time gain. The paper's validation on two Llama models shows the additive predictions match measurements well enough that the empirical-time variant, IP-ET, produces accuracy-versus-latency curves that dominate the random and prefix baselines on Gaudi 2.","pith_inferences":["The independence assumption behind additive loss MSE is the first thing to stress-test: if quantization errors in early layers are amplified or canceled by later layers, the ordering of configurations under the constraint could diverge from real accuracy.","The empirical time-gain model is hardware-specific; transferring the method to another accelerator would require re-measuring subgraph timings, though the loss-MSE side should carry over.","The paper's own LAMBADA-8B observation that IP-ET wins on perplexity while a baseline wins on accuracy suggests that loss MSE is a proxy tied to perplexity; users targeting one benchmark may need a task-aware threshold or a different surrogate.","The calibration cost of measuring every subgraph and configuration could grow for models with many parallel branches, so adaptive or sampled subgraph timing is a natural extension."],"forward_implications":["The IP-ET strategy yields accuracy-versus-latency curves that dominate random and prefix baselines on the tested 1B and 8B Llama models, with the 8B model keeping accuracy loss below 0.1% at 450 ms time-to-first-token versus roughly 600 ms for the baselines.","The same additive machinery works with alternative objectives: IP-TT maximizes MAC-based theoretical time gain and IP-M maximizes memory gain, and both usually outperform the baselines as well.","The loss-MSE estimate and time-gain estimate are validated on selected configurations, so the method can serve as a fast post-training quantization pipeline that needs only a small calibration set and one backward pass per sample.","Because the sensitivity metric is additive and the integer program is inexpensive, the approach extends naturally to many layers, formats, and model sizes without retraining."],"supporting_citations":[{"why":"Establishes FP8 inference on the Intel Gaudi accelerator and provides the precision-format context for the experiments.","marker":"Lee et al. [2025]"},{"why":"Supplies a prior practical mixed-precision post-training quantization algorithm whose layer-sensitivity reduction is a baseline the paper improves on.","marker":"Pandey et al. [2023]"},{"why":"Frames mixed-precision loss minimization as a multiple-choice knapsack problem, the optimization template the integer-programming approach extends.","marker":"Chen et al. [2021]"},{"why":"Provides the survey of mixed-precision frameworks used to position the search-based and optimization-based categories.","marker":"Rakka et al. [2024]"},{"why":"Establishes post-training quantization as an efficient deployment route, the setting the paper operates in.","marker":"Banner et al. [2019]"},{"why":"Supplies the standard quantization and integer-inference concepts that the numerical formats and rounding-noise model build on.","marker":"Jacob et al. [2018]"}],"fun_headline_variants":["IP-based mixed precision dominates baselines on Gaudi 2","Automatic MP: IP maximizes time gain under MSE cap","Partitioned mixed precision: IP wins on speed with loss bound","Auto MP: Taylor sensitivity plus subgraph time gain beats baselines"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that quantization noise in different layers is statistically independent, so total loss MSE is the plain sum of per-layer contributions; if errors amplify or cancel across layers, the integer program's constraint no longer tracks real quality.","fun_headline_variants_meta":{"raw":{"variants":["IP-based mixed precision dominates baselines on Gaudi 2","Automatic MP: IP maximizes time gain under MSE cap","Partitioned mixed precision: IP wins on speed with loss bound","Auto MP: Taylor sensitivity plus subgraph time gain beats baselines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000981,"raw_usage":{"total_tokens":4193,"prompt_tokens":1002,"completion_tokens":3191,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":618,"completion_tokens_details":{"reasoning_tokens":3119}},"tokens_in":618,"tokens_out":3191,"duration_ms":24509,"temperature":1.0,"reasoning_tokens":3119,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:20:57.355707+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Quantize one early layer alone and measure end-to-end loss MSE, then one late layer alone, then both together; compare the joint measurement with the sum of the two single-layer measurements. If the joint error is not close to the sum, or if it reverses the ordering of two candidate configurations, the additive constraint that the integer program relies on fails.","supporting_citations":[],"review_version":1}