{"id":"bb45fb7e-032d-41e7-ba01-2bcea73f3d6c","arxiv_id":"2505.14638","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"DPQ is a post-training quantization algorithm that stores weights in INT4, computes in FP8, and uses Hessian-based group-aware reordering to keep accuracy near the full-precision model.","lead":"Large AI models can be made faster and cheaper to run by storing weights in 4-bit integers and doing the math in 8-bit floating point. A new correction method, DPQ, keeps accuracy close to the full-precision model, and tests on Llama and Qwen models show the trade-off is practical.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Throughput claims in Sec. 4.3 rest on opaque 'LLM model projections' (Fig. 4) with no stated model for memory bandwidth, FP8 GeMM utilization, batch-size scaling, or INT4-to-FP8 dequantization cost; no end-to-end measurement supports the headline speedups.","rationale":"The reader's weakest-assumption analysis correctly identifies the speedup projections in Sec. 4.3 as the most load-bearing unvalidated component of the central claim. The accuracy half of the claim is backed by experiments on multiple model families and ablations, but the efficiency half—which is essential to the paper's practical value—rests entirely on an unspecified projection model. I agree with the reader's CONDITIONAL verdict: the algorithm appears sound, but the headline speedups need measured confirmation. A secondary concern, the mismatch between the claimed power-of-two scaling optimization in Sec. 2.2 and the non-power-of-two scale definitions in Sec. 3.1, reinforces the need for an actual hardware benchmark, because the projections may assume an optimization that the equations do not describe. The proposed end-to-end throughput test would directly settle whether the speedups materialize, so no verdict adjustment is needed.","tokens_in":17439,"tokens_out":9868,"duration_ms":83886,"concrete_test":"Measure end-to-end throughput on Gaudi 2 and Gaudi 3 for Llama-2-70B and Qwen2-72B with W4A8-DPQ, W4A16-GPTQ, and W8A8 at input token counts {512, 2048, 4096} and representative output lengths, using the batch sizes enabled by each format; report tokens/s and achieved memory bandwidth, and compare against Fig. 4. Separately profile the INT4-to-FP8 dequantization cost (cycles and ALU utilization) to check whether it is hidden in the memory-bound regime. If measured speedups are more than 20% below the projected 3x/1.4x, the efficiency claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central efficiency claim—'significant speedups' and specifically 'up to 3x speed-up over W4A16, and up to 1.4x speed-up over W8A8'—is supported only by 'LLM model projections' in Fig. 4. The paper does not specify the projection model: no assumed memory bandwidth, FP8 GeMM utilization, batch-size scaling law, or overhead for the online INT4-to-FP8 dequantization that Fig. 3 requires (read INT4, subtract zero-point, multiply by per-group scale, then by the FP8 scale). Because the W4A8 inference flow adds a dequantization step relative to W8A8, the 1.4x speedup over W8A8 must come from memory savings; whether the projection includes the dequant cost and per-group scale/zero-point storage is unknown. Additionally, Sec. 2.2 states that power-of-two FP8 scales are incorporated to exploit Gaudi's exponent-bias optimization, but Eqs. (6), (8), and (13) define scales as max(|X|)/Zmax (and Sec. 3.3 says the group scale is MSE-searched), which generally are not powers of two. It is therefore unclear whether the projected speedups assume the power-of-two optimization, and if so, how the equations reflect it. If the projections use optimistic bandwidth/utilization or ignore dequantization, the headline speedups will not reproduce on real Gaudi 2/3 workloads, undermining the production usefulness claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a W4A8 post-training quantization scheme for LLM/VLM inference: weights are stored as asymmetric INT4 with per-group scales and zero-points, then dequantized to FP8 at runtime, while activations are quantized to FP8 for FP8 matrix multiplication. To control accuracy loss from the two-stage weight quantization, the authors introduce DPQ, a GPTQ/OBQ-style algorithm that compensates quantization errors using Hessian information computed on the dequantized (INT4-to-FP8-to-BF16) weights, and GAR, a constrained reordering scheme that permutes weights within groups and permutes whole groups so that group scales remain aligned with consecutive memory locations at inference. Accuracy is evaluated on Qwen2-VL (MMMU, MMBench, MathVista) and Llama-2/Llama-3 (WikiText-2, common-sense reasoning, MMLU), with comparisons to QServe, QQQ, GPTQ, and RTN. Throughput is presented as LLM model projections in Fig. 4, claiming up to 3x speedup over W4A16 and up to 1.4x speedup over W8A8 on Gaudi 2/3.","tokens_in":17778,"tokens_out":9565,"duration_ms":91212,"significance":"If the accuracy results are representative, the paper makes a useful contribution: the W4A8 format with FP8 compute and INT4 storage is relevant to both memory-bound and compute-bound inference regimes, and GAR appears applicable beyond this specific setting. The accuracy experiments use independent external baselines (QServe, QQQ) and show that DPQ is competitive with or better than these methods on most reported tasks, which is a genuine strength; the code is also released through Intel Neural Compressor. However, the paper's central efficiency claim currently rests on unmeasured projections with unspecified assumptions, and the power-of-two scaling claim is not reconciled with the quantization equations, so the main practical advantage is not yet established.","major_comments":[{"comment":"The headline speedups ('up to 3x speed-up over W4A16, and up to 1.4x speed-up over W8A8') are based entirely on 'LLM model projections' rather than measured end-to-end throughput. The text never specifies the projection model: no assumed memory bandwidth, FP8 GeMM utilization, batch-size scaling, or cost of the online INT4-to-FP8 dequantization shown in Fig. 3 (read INT4, subtract zero-point, multiply by per-group scale and FP8 scale). Since this dequantization is an extra step relative to W8A8, the 1.4x claim over W8A8 cannot be assessed without knowing whether and how it is modeled. Please report measured Gaudi 2/3 throughput or fully specify the projection methodology, including assumptions and a sensitivity analysis.","section":"Sec. 4.3, Fig. 4"},{"comment":"The paper says power-of-two FP8 scales are incorporated to exploit Gaudi's exponent-bias optimization, but the algorithm as defined does not produce power-of-two scales: Eq. (6) sets sF = max(|W|)/Zmax, Eq. (13) does the same for activations, Eq. (8) computes sg from min/max, and Sec. 3.3 states the group scale is found by MSE search. These are generally not powers of two. Please state explicitly how (or whether) scales are rounded or constrained to powers of two and whether the Fig. 4 projections assume the exponent-bias optimization; otherwise the claimed hardware advantage is not connected to the quantization procedure.","section":"Sec. 2.2, Eqs. (6), (8), (13), Sec. 3.3"},{"comment":"The full-reordering W4A8 results are inconsistent between tables. For Llama-3.1-8B, Table 3 reports W4A8* (DPQ), described as using full activation reordering, with WikiText-2 perplexity 6.935, while Table 4 reports the 'Ordered' W4A8 configuration with perplexity 6.679 under the same BF16 reference (6.238). These two entries should describe the same configuration; please reconcile them and clarify which result supports the claim that GAR is only slightly worse than full reordering.","section":"Table 3 vs. Table 4"},{"comment":"The paper repeatedly states that DPQ/GAR achieves accuracy 'without introducing any inference overhead,' but the W4A8 flow in Fig. 3 includes an online step that reads INT4 weights, subtracts the zero-point, multiplies by the per-group scale, and then applies FP8 scaling. GAR removes the indexing overhead of full reordering, but it does not remove this arithmetic. Please either report the measured cost of this dequantization in the throughput numbers or qualify the 'no overhead' claim to mean 'no additional indexing overhead relative to no-reorder group quantization.'","section":"Sec. 3.2, Fig. 3"}],"minor_comments":[{"comment":"The text uses 'activation ordering' and 'activation reordering' to describe operations applied to weight groups; this terminology is likely to confuse readers, since the tensor being permuted is the weight tensor. Please use 'weight reordering guided by activation/Hessian statistics' consistently.","section":"Sec. 4.4, Appendix A"},{"comment":"The column headers 'A' and 'VG' are not defined; if 'A' is the average over the common-sense suite, say so and list all component tasks.","section":"Table 2"},{"comment":"The notation w16 and \\widehat{w}16 is introduced in the text but the equation uses both; please use a consistent symbol and define \\widehat{w}16 as the two-step dequantized value before Eqs. (10)-(11).","section":"Sec. 3.1, Eq. (12)"},{"comment":"The sentence 'since DPQ adds an additional level of quantization over GPTQ, it serves as an upper-bound to DPQ' is unclear; presumably W4A16 GPTQ is an upper bound for W4A8 DPQ, but the phrasing should be corrected.","section":"Table 3, footnote"},{"comment":"Each subplot's legend should state explicitly that all curves are normalized relative to W4A16, and the input-token ranges and the assumed output length should be given in the caption.","section":"Fig. 4 caption"},{"comment":"The statement that Llama 3.2 90B Vision can fit in an Nvidia H100 (80 GB HBM) with 4-bit weights should include the KV-cache and activation memory budget used for the estimate; as written, a 90B-parameter model at 4-bit weights plus activations may not fit an 80 GB device in all serving configurations.","section":"Sec. 4.3"}],"recommendation":"major_revision","confidential_remarks":"The paper's efficiency claims depend partly on the authors' own FP8 hardware paper [19] and on unmeasured projections; an editor may want to require measured throughput numbers before the paper is publicized as demonstrating significant speedups. Also, the arXiv category cs.CV is a poor fit for an LLM-inference systems paper; a systems or hardware-aware ML track would be more appropriate. The overall direction is sound, and the accuracy comparisons to external baselines are a strength."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The accuracy work is real and creditable. DPQ extends OBQ/GPTQ with two-level error compensation: it accounts for both the INT4 storage error and the FP8 compute error in one update, and the experiments on Llama-2/3 and Qwen2-VL show it beats QServe and QQQ on the reported tasks while staying close to BF16. The cleanest new idea is GAR, the constrained Hessian-based reordering that keeps group scales aligned with the original memory layout. That is a real fix for a real deployment problem, and the ablation shows it matters: without it, W4A8 perplexity on Llama-3.1-8B jumps from about 6.7 to 32.8. GAR is arguably useful beyond this specific setting.\n\nThe soft spots are mostly in the performance section, and the stress-test note lands. The headline speedups in Fig. 4 are 'LLM model projections' with no stated model for memory bandwidth, FP8 GeMM utilization, batch-size scaling, or the cost of the online INT4-to-FP8 dequantization (read INT4, subtract zero-point, multiply by group scale, then by FP8 scale). Because W4A8 adds a dequantization step relative to W8A8, the 1.4x speedup over W8A8 depends on memory savings that may be partially eaten by that overhead. Until the projection model is disclosed or end-to-end measurements appear, those numbers are upper bounds, not results. Also, the paper claims power-of-two FP8 scales to exploit Gaudi's exponent-bias optimization, but Eqs. (6), (8), and (13) define scales as max/|Zmax| and the text says group scales are MSE-searched. If scales are rounded to powers of two, say so; right now it is unclear whether the projections assume something the equations do not describe.\n\nMinor issues: the code lives in the Intel Neural Compressor repository, but no commit-level artifact is pinned. The W4A8 degradation on Llama-3.1-8B WikiText-2 is a bit larger than the wording 'minimal degradation' suggests, though still reasonable for a 4-bit weight scheme.\n\nOverall, the central accuracy argument holds up, and the GAR contribution is worth taking seriously. This paper is for practitioners doing PTQ for LLM/VLM deployment, and for researchers working on quantization algorithms. It deserves a serious referee, provided the reviewers push for either measured throughput or a fully specified projection model. I would accept it to peer review with that expectation.","headline":"Solid accuracy story for a W4A8 scheme with a genuinely useful reordering trick; the speedup numbers are projections, not measurements, and should be read as the paper's main caveat.","tokens_in":18313,"tokens_out":1703,"would_cite":true,"duration_ms":22670,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Storing LLM weights in 4-bit integers while computing in FP8 keeps accuracy near BF16 and promises up to 3x speedup over W4A16, via dual-precision error compensation and group-aware reordering.","keywords":["post-training quantization","W4A8 quantization","FP8 inference","INT4 weight storage","Dual Precision Quantization","Group-Aware Reordering","large language models","vision-language models"],"falsifier":"Measure end-to-end tokens per second on Gaudi 2 or Gaudi 3 for Llama-2 70B and Qwen2-72B quantized with DPQ at W4A8, and compare against W4A16 and W8A8 at matched batch sizes and sequence lengths; if the measured speedup does not reach the projected curves in Fig. 4, the performance claim is refuted. A second check would compare DPQ's FP8 activations against an INT8-activation W4A8 scheme with GAR applied to both, isolating whether the accuracy gain comes from the FP8 grid or from the reordering.","tokens_in":17239,"feed_emoji":"⚡","tokens_out":8727,"duration_ms":69108,"temperature":0.7,"pith_summary":"This paper tries to show that a hybrid W4A8 quantization scheme — weights stored in 4-bit integers, all matrix multiplications computed in 8-bit floating point — can run large language and vision-language models at near-BF16 accuracy while cutting memory traffic and exploiting FP8 tensor cores. The authors argue that prior W4A8 schemes quantize activations to INT8, whose uniform grid fits neither the roughly normal distribution of weights nor the outlier-heavy distribution of activations, whereas FP8's non-uniform grid and dynamic range preserve more accuracy. To make the double quantization (INT4 storage plus FP8 arithmetic) accurate, they introduce Dual Precision Quantization (DPQ), which compensates quantization error through both levels using Hessian-based updates, and Group-Aware Reordering (GAR), which prioritizes important weights without breaking the group layout that keeps inference overhead low. If the scheme works as claimed, models can be stored at half the memory of FP8 weights and computed at FP8 speed with accuracy close to the full-precision baseline.","feed_headline":"4-bit weights plus FP8 compute promise up to 3x speedup","feed_subtitle":"Dual Precision Quantization and group-aware reordering keep LLM accuracy close to BF16 while cutting memory traffic.","key_machinery":"The load-bearing object is the two-step dequantization chain inside DPQ: a weight is quantized from BF16 to FP8, then from FP8 to INT4 for storage; during inference it is dequantized INT4 → FP8 → BF16. The error term fed into the Hessian update is computed from the final BF16 reconstruction, so both quantization levels are compensated simultaneously rather than separately. The second mechanism, Group-Aware Reordering (GAR), permutes weights by Hessian diagonal importance subject to the constraint that permutations happen only inside groups or by swapping whole groups, which guarantees that after the tensor is re-permuted back to its original order each group's scale and zero-point still apply to consecutive weights; this preserves the single-multiplication dequantization of the original scheme. Hessian dampening and Cholesky-based computation of the inverse Hessian make the updates tractable, and power-of-two FP8 scales let the hardware fold scaling into the exponent bias.","core_discovery":"The central claim is that a W4A8 scheme with FP8 activations and computations, rather than INT8, is both more accurate and equally hardware-friendly, provided the weight quantization error is compensated across the full INT4-to-FP8-to-BF16 dequantization chain. DPQ does this by quantizing one weight at a time, dequantizing the stored INT4 value back through FP8 to BF16, and distributing the resulting two-level error to the remaining weights scaled by the inverse Hessian, exactly the second-order error-compensation update used in layer-wise quantization. GAR constrains the reordering to permutations within groups and whole-group swaps, so the per-group scales and zero-points stay aligned with consecutive memory after the tensor is restored to its original order; this removes the indexing overhead of unrestricted reordering while still quantizing the most Hessian-important weights first. The paper reports that on Qwen2-VL, Llama-2, and Llama-3, DPQ keeps accuracy within a few percent of BF16 on vision-language benchmarks, perplexity, common-sense reasoning, and MMLU, and outperforms existing W4A8 schemes that use INT8 activations. Performance projections on Gaudi 2 and 3 show up to 3x speedup over W4A16 and up to 1.4x over W8A8.","pith_inferences":["Beyond the paper's own results, the ablation suggests GAR's constrained reordering, not the FP8 grid, is the main accuracy lever, so a clean comparison of FP8 versus INT8 activations should hold the reordering scheme fixed.","The same two-level error compensation could be applied to other mixed-precision settings, such as INT4 storage with FP8 weight-only dequantization or KV-cache quantization, without changing inference cost.","The paper's speedup curves are projections, so a direct measurement of end-to-end tokens per second on Gaudi 2 and 3 across batch sizes would test the assumed FP8 utilization and memory bandwidth.","Because FP8's advantage over INT8 is largest for outlier-heavy activation distributions, models with known massive activation outliers should show a larger accuracy gap between the two formats; the paper's own discussion points to this test."],"forward_implications":["A W4A8 model quantized with DPQ and GAR stores weights in INT4, halving the memory footprint relative to W8A8 and cutting the bandwidth needed in memory-bound autoregressive decoding.","Because all matrix multiplications run in FP8, the scheme can use the doubled FP8 throughput of modern accelerators, so the same model serves larger batches than a W4A16 deployment.","The accuracy gap to the BF16 baseline is small enough on 7B-70B language models and 2B-72B vision-language models that W4A8 can replace W4A16 in serving settings where latency and memory dominate.","Since GAR keeps scales aligned with consecutive weights, it can be dropped into other group-wise quantization schemes such as W4A16 without adding inference overhead.","The reported speedups of up to 3x over W4A16 and 1.4x over W8A8 follow from combining the smaller INT4 memory footprint with FP8 compute, and they grow with input length in the projections."],"supporting_citations":[{"why":"Supplies the Hessian-based layer-wise error-compensation procedure that DPQ adapts to two quantization levels.","marker":"[12]"},{"why":"Provides the optimal brain quantization update rule and Hessian derivation used in Eq. (12).","marker":"[36]"},{"why":"Defines the W4A8 INT8 baseline that DPQ must match or beat on accuracy.","marker":"[23]"},{"why":"Provides another W4A8 INT8 comparison baseline whose results are quoted in Table 2.","marker":"[38]"},{"why":"Documents FP8 on Gaudi 2 and 3, including power-of-two scaling via exponent bias, which the scheme exploits.","marker":"[19]"},{"why":"Establishes the FP8 E4M3 format and its hardware throughput assumptions.","marker":"[18]"},{"why":"Shows that a small number of highly important weights dominate layers, justifying GAR's partial reordering.","marker":"[24]"},{"why":"Documents massive activations in LLMs, supporting the claim that FP8 handles outlier-heavy activations better.","marker":"[41]"}],"fun_headline_variants":["W4A8 with FP8 compute: DPQ and GAR keep accuracy close to BF16","Dual Precision Quantization: 4-bit weights, FP8 compute for 3x speedup","FP8 beats INT8 for W4A8: DPQ keeps LLM accuracy within a few percent","4-bit weights plus FP8 compute: DPQ cuts memory traffic, holds accuracy","W4A8 inference: DPQ error compensation enables near-BF16 quality"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline speedups come from LLM model projections rather than measured end-to-end throughput, so the load-bearing premise is that real Gaudi 2 and 3 runs will deliver the assumed FP8 matrix-multiplication utilization and memory bandwidth; if those assumptions are optimistic, the claimed gains shrink or vanish.","fun_headline_variants_meta":{"raw":{"variants":["W4A8 with FP8 compute: DPQ and GAR keep accuracy close to BF16","Dual Precision Quantization: 4-bit weights, FP8 compute for 3x speedup","FP8 beats INT8 for W4A8: DPQ keeps LLM accuracy within a few percent","4-bit weights plus FP8 compute: DPQ cuts memory traffic, holds accuracy","W4A8 inference: DPQ error compensation enables near-BF16 quality"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000347,"raw_usage":{"total_tokens":1930,"prompt_tokens":1008,"completion_tokens":922,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":624,"completion_tokens_details":{"reasoning_tokens":803}},"tokens_in":624,"tokens_out":922,"duration_ms":8302,"temperature":1.0,"reasoning_tokens":803,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:30:33.073233+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure end-to-end tokens per second on Gaudi 2 or Gaudi 3 for Llama-2 70B and Qwen2-72B quantized with DPQ at W4A8, and compare against W4A16 and W8A8 at matched batch sizes and sequence lengths; if the measured speedup does not reach the projected curves in Fig. 4, the performance claim is refuted. A second check would compare DPQ's FP8 activations against an INT8-activation W4A8 scheme with GAR applied to both, isolating whether the accuracy gain comes from the FP8 grid or from the reordering.","supporting_citations":[{"cited_title":"Optimal brain compression: A framework for accurate post-training quantization and prun- ing,","cited_arxiv_id":null,"evidence_quote":"Provides the optimal brain quantization update rule and Hessian derivation used in Eq. (12)."},{"cited_title":"Faster Inference of LLMs using FP8 on the Intel Gaudi","cited_arxiv_id":"2503.09975","evidence_quote":"Documents FP8 on Gaudi 2 and 3, including power-of-two scaling via exponent bias, which the scheme exploits."}],"review_version":1}