{"id":"b02fbb95-d6b1-4d6e-b1ac-350b442755da","arxiv_id":"2505.14302","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":26,"one_line_summary":"A unified QAT scaling law predicts 4-bit quantization error from model size, training tokens, and group size, showing activation outliers in the FC2 layer are the main W4A4 bottleneck.","lead":"This paper fits a formula that predicts how much performance is lost when training language models with 4-bit weights and activations, based on model size, training data, and quantization group size. It finds that more training data and coarser quantization increase the error, and that one activation layer is the main bottleneck.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The unified law's G variable is underdetermined: per-token/channel runs lack a numeric G, and the activation quantizer changes at G=256, so the fitted gamma_G confounds granularity with quantizer choice.","rationale":"The paper's central claim is that W4A4 QAT quantization error follows delta_p(N,D,G) = k * D^gamma_D * (log2 G)^gamma_G / N^gamma_N, with a single unified curve across group sizes. For that claim to hold, G must be a well-defined numeric quantity for every experiment, and the fitted gamma_G must isolate the effect of granularity. The text leaves both conditions unmet: per-token/channel granularity is a heterogeneous combination of per-token activation groups and per-channel weight groups, and the quantizer changes from AbsMax to LAC at G=256. This makes the fitted curve partly an artifact of the experimental recipe rather than a clean scaling law. The reader's weakest_assumption correctly pinpointed the missing per-token/channel mapping; I add the quantizer switch as an equally concrete confound that the reader mentioned only in passing. The concern is addressable: refit on a controlled subset, define the numeric mapping, or run all granularities with one quantizer. If those checks pass, the unified law is credible; if not, the central claim needs substantial qualification. The current conditional verdict is appropriate because the issues are clarifiable and the qualitative N and D trends are supported by direct observations. I therefore keep the reader's verdict unchanged while sharpening the required evidence.","tokens_in":18220,"tokens_out":4012,"duration_ms":41374,"concrete_test":"Re-fit Eq. (5) on the 80 W4A4 runs after (a) excluding the G=256 and per-token/channel points, and (b) rerunning G=256 with the same AbsMax activation quantizer used for G<256; if gamma_G changes sign or the held-out 973M relative error increases substantially, the granularity trend is confounded by the quantizer switch. Also report the specific numeric G assigned to per-token/channel and re-fit with G=sequence length versus G=hidden dimension to test whether the mapping matters.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Eq. (5) treats quantization granularity as a single real-valued group size G inside log2(G), but the experimental protocol never specifies a numeric G for the per-token/channel setting, and the activation quantizer is switched at G=256. Section 4.1 says AbsMax is used for fine group sizes (G<256) and LAC for coarse group sizes (G>=256), while per-token/channel is listed as a fifth granularity without a numeric value. Thus the 80-run fit and Figure 4c mix data from two different quantizers, and the fitted gamma_G=0.7779 reflects not only granularity but also quantizer identity. If the per-token/channel points were assigned G equal to sequence length, hidden dimension, or some other convention, Eq. (5) would produce different predictions for the same experiments, so the claimed single-curve unification in Fig. 1 and Table 2 is not fully determined by the text. The two held-out 973M points provide only weak discrimination because they are both far inside the same G regime and do not test the G encoding. This is a load-bearing gap because the central claim is that one law predicts W4A4 error across all granularities.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a unified scaling law for quantization-aware training (QAT) of 4-bit LLMs, modeling the quantization error gap δp between QAT and BFloat16 models as δp(N,D,G) = k · D^{γD} · (log2 G)^{γG} / N^{γN}. The authors fit this law on 80 W4A4 runs across model sizes 74–594M, token counts 10–100B, and granularities {32,64,128,256,per-token/channel}, report good fit (R²=0.94), and validate on held-out 973M-parameter models trained on 100B and 200B tokens. They also decompose W4A4 error into weight (W4A16) and activation (W16A4) components, fit separate scaling laws for each, and show that FC2-proj input activations, due to high kurtosis from SwiGLU, are the main bottleneck. A mixed-precision intervention (8-bit FC2 input) reduces this error and balances weight and activation contributions.","tokens_in":18750,"tokens_out":2581,"duration_ms":24770,"significance":"If the central claims hold, the paper contributes a practically useful extension of scaling-law methodology to QAT: it shows that quantization error increases with training tokens and coarser granularity, captures this in a compact three-variable law, and identifies FC2-input activations as the key bottleneck. The empirical base is substantial (268 experiments, 276K GPU-hours), the law is machine-checkable, and the paper includes explicit held-out validation on 973M models, which is a notable strength. The decomposition into W4A16 and W16A4 components is independently measured and provides falsifiable predictions about how weight-versus-activation error balances shift with data-to-parameter ratio. The main risk is that the granularity variable G is not explicitly mapped to a numerical value for the per-token/channel setting and is confounded by a quantizer change, so the single-curve unification claim is not fully determined by the text.","major_comments":[{"comment":"The variable G in Eq. (5) is underdetermined for the per-token/channel granularity. Section 4.1 lists G ∈ {32,64,128,256,per-token/channel}, but per-token activation quantization and per-channel weight quantization have heterogeneous group sizes, and no numerical G is assigned to the per-token/channel condition. The fitted γG in Table 1 therefore depends on an implicit mapping that is not stated; different mappings (e.g., sequence length, hidden dimension, or an arbitrary constant) would produce different predicted δp for the same experiments. Because the central claim is that Eq. (5) unifies all granularities into one curve, the absence of this mapping leaves the fit in Figure 4c and the parameter γG non-reproducible. Please specify the exact numeric G used for each run, or restrict the law to the numeric group sizes if no principled mapping exists.","section":"§4.1, Eq. (5)"},{"comment":"The activation quantizer switches at G = 256: AbsMax is used for G < 256 and LAC for G ≥ 256, which includes the per-token/channel setting. Figure 4c and the fitted γG = 0.7779 therefore combine data from two different quantizers, so the observed 'granularity effect' is confounded with quantizer identity. This undermines the attribution of the error trend solely to G. Please provide an ablation where the same quantizer is used across all granularities (e.g., LAC for all G) or otherwise demonstrate that the fitted exponent γG does not change materially when the quantizer is fixed.","section":"§4.1, §4.2, Fig. 4c"},{"comment":"The held-out validation on 973M models tests extrapolation in N and D but not in G: both validation points appear to use the same granularity regime, and the text does not report whether they use per-token/channel or a numeric group size. Since the G-encoding issue raised by Major Comment 1 is precisely about which numeric G is used for coarse granularities, the current validation does not discriminate between alternative assignments. Please report the granularity settings of the two 973M runs, and include at least one held-out data point at a new granularity if possible.","section":"§4.2, validation"},{"comment":"The decomposition δW4A4 ≈ 0.906·(δW4A16 + δW16A4) is presented as an observed correlation, but the paper then fits separate scaling laws for δW4A16 and δW16A4 and uses them to explain trends in δW4A4. This is reasonable as an approximation, yet the paper does not propagate the uncertainty of the 0.906 coefficient or verify that the fitted exponents are consistent with the W4A4 fit when combined. Since the approximation is load-bearing for the bottleneck analysis, please add a brief consistency check (e.g., compare δW4A4 predicted from the two-component fits against the direct W4A4 fit).","section":"§4.3, Eq. (5) usage"}],"minor_comments":[{"comment":"The definition of δW4A4 appears to have a sign error: the text says δW4A4 = loss_bf16 − loss_W4A4, which would be negative when QAT loss is higher, but Figure 4 and the surrounding discussion treat δ as positive. Please correct the sign or the ordering.","section":"§4.2, Evaluation metric"},{"comment":"In the sentence 'This strong correlation suggests that we can effectively analyze δW4A4 by separately examining the δW4A16 and δW4A4', the second term should read δW16A4.","section":"§4.3"},{"comment":"The phrase 'per-token/channel' is used inconsistently across figures; for example, Figure 4c labels the last x-axis tick 'Token/Channel' while the text and Table 2 use 'per-token/channel.' Please unify the terminology and define it once in the main text.","section":"§4.1"},{"comment":"The claim that 8-bit FC2 inputs 'provide an upper bound on the improvements possible' is somewhat informal, since outlier-suppression methods could in principle exceed 8-bit quantization if 8-bit itself is not perfect; please rephrase to avoid implying a rigorous bound.","section":"§4.4"}],"recommendation":"major_revision","confidential_remarks":"The paper's empirical scope is impressive and the main qualitative findings (error increases with D, FC2 activation outliers are a bottleneck) are likely to be robust. My concern is not with the existence of the trend but with the exact functional form: the G variable needs a precise definition for per-token/channel settings, and the AbsMax-to-LAC switch must be controlled before γG can be trusted. Both are fixable with additional experiments or a more clearly scoped law. I would not go to reject, but the manuscript in its current form claims more determinacy than the reported protocol supports."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this one. It is the first QAT scaling law I have seen that puts model size, training tokens, and quantization granularity in one formula, and it backs that formula with a serious experiment matrix: 268 runs, 276K GPU-hours, held-out 973M models. The fit is tight (R2 around 0.94 on W4A4), and the comparison against prior N-only laws shows a real improvement in relative error. The decomposition into weight and activation error, and the finding that FC2 input activation outliers dominate W4A4 error, are practically useful and empirically grounded. The mixed-precision intervention (keeping FC2 input at 8-bit) is a cheap, concrete fix that reduces error 20-43% depending on granularity. Credit where due: this is a genuine empirical contribution, and the authors are honest about limitations (no MoE, small scale, only W4A4).\n\nThe soft spots, in order of seriousness:\n\n1. The G variable is underdetermined. Eq. (5) uses log2(G), but the paper never says what numeric G is assigned to the per-token/per-channel runs. Those runs appear in Figure 4c and are included in the fit, so without that mapping the single-curve claim is not fully specified. This is a clarity gap, not a refutation, but an explicit mapping table is needed.\n\n2. The activation quantizer switches from AbsMax to LAC at G=256. The fitted gamma_G therefore mixes granularity with quantizer identity. LAC is designed for outliers, so the coarse-granularity error may be lower than with AbsMax, and the scaling law's G-dependence is partly quantizer-dependent. The ablations in Figure 14 help, but a separate fit per quantizer, or a quantizer indicator, would clean this up.\n\n3. Minor: the text defines delta as loss_bf16 minus loss_W4A4, which would be negative for degraded models, while all plots show positive values. Likely a sign typo, but confusing.\n\n4. No uncertainty estimates on the fitted parameters. For a scaling law meant to guide compute decisions, confidence intervals on gamma_N, gamma_D, and gamma_G would matter.\n\nNone of this kills the paper. The central qualitative claims (error decreases with N, increases with D and coarser G, FC2 input is the activation bottleneck) are robust to these issues. The held-out validation on 973M models gives some assurance, though it does not stress the G encoding.\n\nWho should read it: anyone doing model compression or choosing model size and data volume for quantized training. It deserves a serious referee: the experiment cost is high, the empirical contribution is real, and the issues are addressable in revision. I would send it to review, and I would ask for the G mapping, a quantizer-robustness check, and error bars before accepting.","headline":"A well-costed empirical scaling law for W4A4 QAT that is worth refereeing; the G variable needs pinning down and the quantizer switch confounds part of the granularity trend.","tokens_in":19284,"tokens_out":3073,"would_cite":true,"duration_ms":29398,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Low-bit quantized training error follows one power law in model size, training tokens, and quantization group size.","keywords":["quantization-aware training","scaling law","W4A4 quantization","quantization granularity","activation outliers","mixed-precision quantization","LLM compression","weight versus activation error"],"falsifier":"Train W4A4 QAT models at fixed $N$ and $D$ while sweeping activation group size and weight group size independently in opposite directions, such as 32-token/256-channel versus 256-token/32-channel; if a single $G$ cannot reproduce both measured errors, or if the fitted exponents $\\gamma_N,\\gamma_D,\\gamma_G$ change when the two group types are decoupled, the unified single-curve law is falsified.","tokens_in":18021,"feed_emoji":"📉","tokens_out":7923,"duration_ms":73667,"temperature":0.7,"pith_summary":"This paper claims that the extra loss introduced by 4-bit weight-and-activation quantization-aware training (W4A4 QAT) is not fixed by model size alone: it follows a single power law in three variables, $\\delta_p(N,D,G)=k D^{\\gamma_D}(\\log_2 G)^{\\gamma_G}/N^{\\gamma_N}$, where $N$ is the parameter count, $D$ the training tokens, and $G$ the quantization group size. Fitted on 80 W4A4 runs spanning 74M to 595M parameters and 10B to 100B tokens, the law predicts held-out 973M-parameter models and reduces relative prediction error from 8.5% to 4.7% compared with prior model-size-only QAT scaling laws. The experiments show that quantization error grows with more training tokens and coarser quantization granularity, and that the dominant error source is activation quantization at the FC2 projection input, whose outliers persist under QAT. If the law holds, QAT budgets become a design variable alongside model size: how much data a model sees and how finely weights and activations are grouped directly determine how much precision is lost.","feed_headline":"One power law captures 4-bit quantization-aware training error","feed_subtitle":"More data and coarser quantization groups raise the error; a single curve predicts it across 74M-to-973M models.","key_machinery":"The load-bearing object is the unified QAT error law $\\delta_p(N,D,G) = k D^{\\gamma_D}(\\log_2 G)^{\\gamma_G}/N^{\\gamma_N}$, fitted to smoothed training-loss gaps between QAT and BFloat16 models. The logarithmic term in $G$ makes the error vanish when there is no grouping, while the exponents $\\gamma_N,\\gamma_D,\\gamma_G$ measure how strongly model size, data volume, and granularity each drive error. The argument is carried by the decomposition $\\delta_{W4A4} \\approx 0.906\\,(\\delta_{W4A16}+\\delta_{W16A4})$, which converts one aggregate curve into two independently fitted component curves and lets the fitted exponents identify whether weights or activations are the bottleneck under each training regime.","core_discovery":"The central discovery is that W4A4 QAT quantization error has a predictable, factorized structure: it decreases as a power of model size, increases as a power of training tokens, and increases as a power of the logarithm of quantization group size. The same functional form fits weight-only quantization (W4A16) and activation-only quantization (W16A4) with different exponents, and their errors add to reproduce the W4A4 error with a fitted coefficient of 0.906. Activation error generally dominates, especially at coarse granularities, and the paper traces this to high-kurtosis outliers in the FC2 Proj input generated by the SwiGLU module. Keeping FC2 inputs at 8-bit precision drops W4A4 quantization error by 20.5% at group size 32 and 42.9% at group size 256, and makes weight and activation errors nearly equal; as the data-to-parameter ratio grows, weight quantization error eventually overtakes activation error.","pith_inferences":["Editorial extension: the law was fitted on models up to 595M parameters, so its predictions much beyond 1B parameters are extrapolation; the same exponents may not hold at frontier scale.","Editorial extension: the single numeric $G$ is underdetermined when activation groups are per-token and weight groups are per-channel with different sizes; a two-term law with separate granularity exponents for each component could fit equally well and resolve the ambiguity.","Editorial extension: the persistent FC2 input outliers suggest testing whether other outlier-prone parts of a transformer, such as early embedding projections or attention outputs, also deserve mixed precision as training data scales.","Editorial extension: since error grows with $D$, a QAT training schedule could anneal quantization granularity or introduce mixed precision later in training to counteract the widening gap."],"forward_implications":["Quantization error is predictable from a single curve across model size, data, and granularity, cutting relative prediction error on W4A4 from 8.5% to 4.7% compared with model-size-only QAT laws.","More training data makes quantization harder, so a model trained on 100B tokens instead of 10B needs finer quantization groups or higher precision to hold the same error.","Activation quantization at the FC2 Proj input is the main W4A4 bottleneck; keeping that input at 8-bit lowers error by 20.5% at group size 32 and 42.9% at group size 256.","Weight quantization error grows faster with training tokens than activation error, so at high data-to-parameter ratios weight quantization deserves as much attention as activation outliers.","Under the paper's cost assumptions, the effective parameter multiplier of W4A4 stays above 0.5, making 4-bit QAT a better compute-error trade-off than 8-bit QAT across the studied range."],"supporting_citations":[{"why":"Supplies the classical loss-scaling form that the paper extends and whose smoothed training loss serves as the evaluation metric.","marker":"[16]"},{"why":"Prior QAT scaling law that models quantization error through an effective parameter multiplier; the main baseline the proposed law must beat.","marker":"[12]"},{"why":"Prior precision scaling law that ignores training tokens and granularity; provides the per-granularity baseline and the 8-bit Pareto conclusion the paper revisits.","marker":"[20]"},{"why":"Establishes fine-grained 4-bit quantization as necessary for low-bit performance, motivating the granularity term in the law.","marker":"[8]"},{"why":"Provides the pretraining data mixture and training hyper-parameters used in all 268 experiments.","marker":"[30]"},{"why":"Defines the transformer architecture used in all runs, including the SwiGLU feed-forward block whose FC2 input becomes the outlier bottleneck.","marker":"[15]"},{"why":"Documents activation outliers as the central obstacle to low-bit activation quantization, the mechanism behind the FC2 bottleneck analysis.","marker":"[42]"}],"fun_headline_variants":["Scaling law predicts 4-bit quantization-aware training error across sizes","QAT error scales with model size, data, and group granularity","Activation outliers bottleneck W4A4 quantization-aware training","More training tokens actually increase QAT quantization error","Keep FC2 at 8 bits to cut W4A4 error by 43%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The law encodes every granularity choice as one number $G$ inside a log term, but activation groups (per token) and weight groups (per channel) can have different sizes, so the single-curve fit is not fully determined unless a mapping from both group types to one $G$ is specified.","fun_headline_variants_meta":{"raw":{"variants":["Scaling law predicts 4-bit quantization-aware training error across sizes","QAT error scales with model size, data, and group granularity","Activation outliers bottleneck W4A4 quantization-aware training","More training tokens actually increase QAT quantization error","Keep FC2 at 8 bits to cut W4A4 error by 43%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000728,"raw_usage":{"total_tokens":3296,"prompt_tokens":1017,"completion_tokens":2279,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":633,"completion_tokens_details":{"reasoning_tokens":2188}},"tokens_in":633,"tokens_out":2279,"duration_ms":56046,"temperature":1.0,"reasoning_tokens":2188,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:36:41.724108+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train W4A4 QAT models at fixed $N$ and $D$ while sweeping activation group size and weight group size independently in opposite directions, such as 32-token/256-channel versus 256-token/32-channel; if a single $G$ cannot reproduce both measured errors, or if the fitted exponents $\\gamma_N,\\gamma_D,\\gamma_G$ change when the two group types are decoupled, the unified single-curve law is falsified.","supporting_citations":[{"cited_title":"The case for 4-bit precision: k-bit inference scaling laws","cited_arxiv_id":null,"evidence_quote":"Establishes fine-grained 4-bit quantization as necessary for low-bit performance, motivating the granularity term in the law."}],"review_version":1}