{"id":"774ec690-bb11-4609-ba1a-c1303ee169a6","arxiv_id":"2505.03803","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"RWKVQuant quantizes RWKV models to about 3.275 bits per weight with under 1% average accuracy loss by mixing scalar and vector quantization chosen by an entropy-and-moment proxy.","lead":"This paper presents RWKVQuant, a post-training quantization framework that lets RWKV language models run at about 3 bits per weight with under 1% accuracy loss and roughly 2x speedup. It works by mixing two existing quantization styles, scalar and vector, and choosing per layer which to use based on the shape of the weight distribution.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline results are selected and calibrated on the evaluation benchmarks: Sec 4.1 uses test-set samples for calibration and sets tau_c/tau_f per model to force the 9:1 split, so the <1% loss and family-wide gains are not yet established under a held-out protocol.","rationale":"The reader correctly identifies the hand-set tau_c/tau_f thresholds as a load-bearing component, and Appendix A.5 explicitly concedes that the chosen allocation may not be optimal. My concern is broader and more direct: the same Section 4.1 that fixes the 9:1 split also states that calibration samples are drawn from the test datasets, and Table 12 shows threshold choices being compared on the very metrics used in the headline tables. This means the reported <1% accuracy loss is not a clean out-of-sample measurement; it is a measurement taken after the method has been allowed to see the evaluation distribution through both calibration and hyperparameter selection. That does not make the authors' numbers dishonest, but it does mean the central claim is not yet verified by the evidence in the paper. The proposed held-out rerun is a single, feasible experiment that would settle the point: if the numbers survive disjoint calibration and frozen thresholds, the concern is resolved; if they do not, the central claim needs substantial qualification. I therefore recommend moving from CONDITIONAL to UNVERDICTED pending that rerun and the promised code release.","tokens_in":22744,"tokens_out":13807,"duration_ms":139274,"concrete_test":"Rerun the full Table 2 protocol with all calibration and threshold selection restricted to data disjoint from the reported benchmarks: choose tau_c/tau_f and the 128 calibration samples from a training/validation split, then evaluate on the original LAMBADA and nine zero-shot test sets. In the same run, freeze the thresholds selected on RWKV-7-0.1B and apply them unchanged to all other RWKV models to test transfer. If RWKV6-14B average accuracy stays within 1 point of FP and Table 2 rankings persist, the concern is resolved; if the gap widens by more than 1-2 points or rankings change, the headline is an artifact of test-set-dependent tuning.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"The strongest empirical claim — 3.275 bpw RWKV-6-14B with <1% average zero-shot loss and consistent wins over SQ/VQ baselines — depends on how the quantization parameters are chosen. Section 4.1 states that 128 samples from the corresponding test datasets are used for calibration and that tau_c/tau_f are dynamically set to enforce a fixed 9:1 split of 3.25-bpw SQ to 3.5-bpw VQ layers. Appendix A.5 admits these values were empirically set and 'might not reflect the most balanced or effective proportion,' and Table 12 sweeps tau_c/tau_f while reporting the same zero-shot/LAMBADA metrics used in Table 2. Thus the headline bitrate is fixed by construction, and the accuracy is obtained under a protocol in which calibration and hyperparameter selection have access to the evaluation benchmarks. If that protocol is what produces the reported gains, the method's generality and the <1% loss claim are unsupported; a held-out calibration and threshold-selection procedure could shift the numbers materially.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript introduces RWKVQuant, a post-training quantization framework for RWKV-family models. The method combines scalar quantization (GPTQ-style) and vector quantization (GPTVQ-style) at the layer level, guided by a two-stage proxy: a coarse entropy-based uniformity measure and a fine-grained central-moment measure for detecting local outliers. A second contribution is a codebook-optimization scheme for element-wise multiplication layers, using activation-weighted KMeans with percentile clipping during batch integration. Experiments on RWKV-6, RWKV-7, and VRWKV models claim 3.275 bpw weight-only quantization with less than 1% average zero-shot accuracy loss and a 2.14x speedup on RWKV-6-14B, with consistent gains over both scalar-only and vector-only baselines.","tokens_in":22931,"tokens_out":9570,"duration_ms":82595,"significance":"If the empirical claims held, this would be a useful contribution to PTQ outside the Transformer family. The paper identifies a genuine structural obstacle—nonlinear operators in RWKV block fusion-based SQ—and a distributional obstacle—uniform weights degrade cluster-based VQ—and the proposed hybrid proxy is simple, interpretable, and has O(M) cost. The Taylor-expansion derivation of the fine-grained proxy is mathematically sound as a heuristic, and the element-wise codebook optimization is a principled adaptation that the ablations support. However, the current empirical evidence is compromised by test-set calibration, threshold selection on evaluation metrics, and multiple inconsistencies in the main results tables. The significance of the paper is therefore conditional on a properly controlled evaluation that separates calibration and threshold selection from test metrics.","major_comments":[{"comment":"Section 4.1 states that 128 samples from the corresponding test datasets are used for calibration and that tau_c and tau_f are dynamically set per model to force a 9:1 split between 3.25-bpw SQ layers and 3.5-bpw VQ layers. Appendix A.5 describes these thresholds as empirically set, and Table 12 sweeps tau_c and tau_f while reporting the same zero-shot and LAMBADA metrics used in Table 2. This means the headline 3.275 bpw is fixed by construction rather than determined by the proxy, and the reported accuracy is obtained under a protocol in which the evaluation benchmarks influence both calibration and hyperparameter selection. The claimed <1% accuracy loss and family-wide gains are not established under a held-out protocol; the authors should use a disjoint calibration/validation split and specify the threshold-selection rule before any test-set evaluation.","section":"Section 4.1, Appendix A.5, Table 12"},{"comment":"The main results table contains entries that do not match the appendix tables. For example, for RWKV7-0.5B at 3.25 bpw, Table 2 lists GPTQ as (0-shot9=41.16, LAMBADA PPL=23.29), but Table 9 lists GPTQ 3.25 as (avg=43.69, Wiki2=15.97) and AWQ 3.25 as (avg=41.16, Wiki2=23.29); the Table 2 AWQ entry for the same model, (68.92, 5.92), appears in neither Table 9 nor Table 10. Because Table 2 is the primary evidence for the hybrid method's superiority, these inconsistencies must be resolved and the table regenerated from the same run logs used for the appendix tables.","section":"Table 2 vs. Tables 9 and 10"},{"comment":"The ablation table reports a GPTVQ zero-shot accuracy of 48.29 for RWKV6-7B, whereas Table 2 reports 58.57 at 3.25 bpw and 59.70 at 3.5 bpw for the same model and method; similar mismatches appear for RWKV7-0.5B GPTQ (45.36 in Table 5 vs. 45.73 in Table 2). Since the ablation is intended to show that the hybrid beats each single method at a matched bitrate, these numbers must come from the same evaluation pipeline. As printed, the conclusion that hybrid quantization is responsible for the observed gain is unsupported.","section":"Table 5 vs. Table 2"},{"comment":"The text states that on 'larger models such as RWKV6-7B and RWKV7-14B' the method results in almost no increase in perplexity and less than one point accuracy drop, but RWKV7-14B does not appear in Table 2 and no results for it are reported anywhere in the paper. Either provide the RWKV7-14B experiments or correct the sentence to refer to the models actually evaluated.","section":"Section 4.2"}],"minor_comments":[{"comment":"There are several typos, including 'Scaler Quanzization' and 'a an essential technique'; these should be corrected.","section":"Section 1"},{"comment":"The heading uses 'RWK7' instead of 'RWKV7'.","section":"Table 6 heading"},{"comment":"The caption refers to thresholds as mu_c and mu_f, while the text and equations use tau_c and tau_f; please unify the notation.","section":"Figure 3 caption"},{"comment":"The header appears corrupted: it contains duplicated 'RWKV6-T' columns and an incomplete model list, and the corresponding model names in the body do not line up with the columns.","section":"Table 8 heading"},{"comment":"The sentence 'ensuring that the 3.25 bpw SQ proportion is approximately one-tenth and nine-tenths for 3.5 bpw VQ' is ambiguous and appears inverted relative to the 9:1 SQ:VQ ratio stated in Section 4.1.","section":"Appendix A.5"},{"comment":"The sentence 'For fairness, the weights of all multiplication operations are quantized using the RTN method' is unclear in context, since the ablation compares GPTQ, GPTVQ, and the hybrid; please clarify what is held fixed across the compared configurations.","section":"Section 4.3"},{"comment":"The appendix row for RWKV7-0.5B at 3.5 bpw RTN appears corrupted, with values (18.13, 19.36, 43.47, ...) that look like a shifted copy from another row; please regenerate the affected rows.","section":"Tables 9 and 10"}],"recommendation":"major_revision","confidential_remarks":"The threshold-tuning-on-test-set issue and the inconsistencies in Table 2 are substantial enough that I would not rely on the current numbers as published. The core idea is, however, salvageable, and the required fixes—rerunning with a held-out calibration split, reporting threshold selection before test evaluation, and reconciling tables—are within the manuscript's scope. I therefore recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look if you do PTQ for non-Transformer LLMs. The core idea is sensible: hybridize scalar and vector quantization per layer, using an entropy-based coarse proxy and a central-moment fine proxy to catch outliers. The activation-weighted codebook objective for RWKV's element-wise multiplications is a legitimate new tweak, and the ablations back it up. Tables are extensive across RWKV-6, RWKV-7, and VRWKV, and the method consistently beats the listed baselines at similar bits. I believe the central accuracy claims are roughly right. The math of the proxy is a heuristic, not a theorem, but it checks out and the paper says so honestly.\n\nThe soft spots are real but not ruinous. The headline 3.275 bpw is enforced by construction: tau_c and tau_f are tuned per model to force a 9:1 split of 3.25-bpw SQ to 3.5-bpw VQ layers. Appendix A.5 admits this, and Table 12 sweeps the thresholds on eval sets. So the exact bitrate is an artifact of tuning, and the claimed <1% loss is not demonstrated under a fully held-out protocol. Also, calibration uses 128 samples from the test sets, no error bars, and code is promised but not released. I also want to flag that Table 5's GPTVQ row for RWKV6-7B (48.29 zero-shot) looks off relative to Table 2's 58.57 for the same setting; likely a typo, but it needs a check.\n\nNone of these issues sink the method. The qualitative finding—RWKV has more uniform layers than LLaMA, so a per-layer SQ/VQ mix helps—is plausible and the hybrid beats both pure SQ and pure VQ in the ablations. The threshold tuning is a limitation of the evaluation, not a fatal flaw in the concept. A held-out threshold-selection protocol and released code would settle it.\n\nMy take: this is a solid paper for the efficient-inference crowd, especially anyone deploying RWKV on edge. It deserves a serious referee, not a desk reject. Recommend major revision: disclose the calibration protocol clearly, add a sensitivity analysis with thresholds chosen on a held-out set, fix the table inconsistency, and release the code before any camera-ready. I'd bring it to our reading group and would cite it once the code is out.","headline":"A useful PTQ recipe for RWKV that mostly delivers on its claims, with a real but fixable calibration-protocol weakness around the headline bitrate.","tokens_in":23585,"tokens_out":606,"would_cite":false,"duration_ms":7544,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RWKVQuant proposes a hybrid scalar/vector post-training quantization scheme that puts RWKV-6-14B at 3.275 bits per weight with less than 1% zero-shot accuracy loss and 2.14x faster generation.","keywords":["post-training quantization","RWKV","scalar quantization","vector quantization","hybrid quantization","codebook optimization","LLM compression","information entropy"],"falsifier":"Use the thresholds the paper reports for RWKV-7 ($\\tau_c=1.54$, $\\tau_f=30$) on RWKV-6-3B and check both the layer split and LAMBADA perplexity; if the split drifts from roughly 9:1 or perplexity rises by more than the paper's reported margin, the proxy has not isolated a transferable property of RWKV weights.","tokens_in":22478,"feed_emoji":"⚡","tokens_out":10726,"duration_ms":98727,"temperature":0.7,"pith_summary":"RWKVQuant claims that RWKV models fail under standard Transformer post-training quantization for two structural reasons: their non-linear operators block the parameter fusion used by smooth- and rotation-based scalar quantizers, and their unusually uniform weight distributions defeat cluster-based vector quantizers. The paper proposes a per-layer hybrid that routes each weight tensor to scalar quantization (GPTQ-style compensation) or vector quantization (GPTVQ-style codebooks), using a coarse information-entropy proxy for overall uniformity and a fine proxy built from weighted higher-order central moments for local outliers. For RWKV's element-wise multiplication layers, it adds an activation-weighted codebook optimization that directly minimizes the output error. If the method is right, RWKV-6-14B can run at 3.275 bits per weight with under 1% zero-shot accuracy loss and a 2.14x generation speedup, which is the concrete promise for deploying RWKV on memory-limited devices.","feed_headline":"RWKV goes to 3.275 bits with under 1% accuracy loss","feed_subtitle":"A per-layer mixture of scalar and vector codebooks keeps RWKV-6-14B within one point of full precision.","key_machinery":"The load-bearing object is the coarse-to-fine proxy: weights are flattened, sorted, and converted to adjacent gaps $G'$; the coarse proxy is the information-entropy gap $P_c = H(\\hat G') - H(G')$ between uniform gaps and observed gaps, and the fine proxy is $P_f = \\sum_{k=2}^K v_k |M_k|$, a weighted sum of higher-order central moments of $G'$ that magnifies local outliers. A layer gets scalar quantization only if $P_c < \\tau_c$ and $P_f < \\tau_f$; otherwise it gets vector quantization. The second mechanism is the element-wise codebook optimizer, a weighted K-means where the squared calibration activations $X^2$ weight each element, with percentile clipping before averaging over batches, so codebooks minimize $\\|X \\odot \\mu - X \\odot \\mathrm{Deq}(Q(\\mu))\\|_F^2$ rather than unweighted weight error.","core_discovery":"The central discovery is that no single existing quantizer fits RWKV, but the two families compensate each other. Smooth- and rotation-based scalar methods (GPTQ, AWQ, QuaRot) fail because RWKV's token-shift, sigmoid, and exponential operators sit in the fusion path, so the extra scaling or rotation parameters cannot be folded away and instead add computation. Cluster-based vector methods (K-Means, GPTVQ, VPTQ) fail because RWKV weights have a larger fraction of uniformly distributed values, which K-means represents poorly. RWKVQuant's answer is to decide per layer: when both a coarse entropy-based uniformity test and a fine weighted-moment outlier test come out low, scalar quantization is safe; otherwise vector quantization. On top of that, the codebook is fit with squared calibration activations as weights and percentile-clipped batch averaging, so the element-wise multiplication layers' actual output error is minimized. The paper reports that this hybrid beats both pure scalar and pure vector baselines across seven RWKV-6/7 language models and six VRWKV vision models, including the headline 3.275-bpw result on RWKV-6-14B.","pith_inferences":["The fixed 9:1 split is a knob, not a law: an automatic search over the threshold values would likely find a different allocation that trades bitrate against accuracy, so the 3.275 bpw number is better read as one point on a curve than as the method's natural operating point.","Applying the same proxy at channel or block granularity instead of whole layers, which the paper lists as future work, should recover either higher accuracy or a lower bitrate on the same models.","If the uniformity difference between RWKV and LLaMA is causal rather than incidental, then other architectures that rely on element-wise weighting or linear recurrences may show the same pattern, and the hybrid recipe would be a natural first thing to try on them.","A sensitivity test on calibration set size for the percentile-clipped weighted codebook would be the quickest way to see whether the reported accuracy depends on the exact 128-sample calibration choice."],"forward_implications":["A near-lossless 3.275-bit weight-only PTQ becomes possible across the RWKV family, not just on one model size: the same recipe holds for RWKV-7 0.1B-1.47B, RWKV-6 1B-14B, and VRWKV vision models.","The reported uniformity gap (about 60% of RWKV layers prefer scalar quantization versus about 10% for LLaMA) gives a measurable architectural signature that explains why Transformer PTQ transfers poorly to RWKV and where to look in other linear-RNN hybrids.","Activation-weighted codebook fitting improves LAMBADA perplexity and zero-shot accuracy on every RWKV model tested, so RWKV-style element-wise multiplication layers should not be quantized with unweighted K-means codebooks.","For an edge-deployment engineer, the concrete operating point is RWKV-6-14B at 3.275 bpw: memory drops from 26.07 GB to 9.21 GB and generation speed rises from 16.02 to 34.32 tokens/s on an A6000.","Because the proxy itself only inspects the weights, the per-layer SQ/VQ decision can be recomputed in O(M) time for each layer without running the model, which keeps the method cheap enough for many model sizes."],"supporting_citations":[{"why":"Supplies GPTQ, the compensation-based scalar quantizer that the hybrid's SQ branch uses and one of the main baselines.","marker":"Frantar et al., 2022"},{"why":"Supplies GPTVQ, the vector quantizer whose codebook training the hybrid's VQ branch and element-wise optimizer build on.","marker":"van Baalen et al., 2024"},{"why":"Supplies VPTQ, the vector-quantization baseline that drops more than 16 percent accuracy on RWKV-6-7B and motivates the hybrid design.","marker":"Liu et al., 2024a"},{"why":"Supplies QuaRot, the rotation-based scalar baseline whose fused parameters cannot be folded in RWKV and add over 99 percent FLOP overhead.","marker":"Ashkboos et al., 2024"},{"why":"Supplies K-means, the clustering algorithm used to measure relative cluster loss and to generate the codebooks.","marker":"Lloyd, 1982"},{"why":"Supplies information entropy, the basis of the coarse proxy that measures weight uniformity.","marker":"Shannon, 1948"},{"why":"Defines the RWKV-6 architecture and the Eagle/Finch models on which the language experiments run.","marker":"Peng et al., 2024"},{"why":"Defines VRWKV, the vision architecture family used for classification, detection, and segmentation experiments.","marker":"Duan et al., 2024"},{"why":"Supplies LAMBADA, the dataset whose perplexity is the paper's main language quality metric.","marker":"Radford et al., 2019"}],"fun_headline_variants":["RWKVQuant: 3.275-bit RWKV with <1% loss","Hybrid quantizer reduces RWKV-6-14B to 3.275 bits","Per-layer hybrid quantization brings RWKV to 3.275 bits","RWKV hits 3.275 bits with under 1% accuracy drop","3.275-bit RWKV-6-14B: under 1% loss, 2.14x faster"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central assumption is that the two cutoff values that pick scalar versus vector quantization for each layer carry over from one RWKV model to another, even though the paper tunes them per model to force a fixed 9:1 split and its own appendix warns that this split may be arbitrary.","fun_headline_variants_meta":{"raw":{"variants":["RWKVQuant: 3.275-bit RWKV with <1% loss","Hybrid quantizer reduces RWKV-6-14B to 3.275 bits","Per-layer hybrid quantization brings RWKV to 3.275 bits","RWKV hits 3.275 bits with under 1% accuracy drop","3.275-bit RWKV-6-14B: under 1% loss, 2.14x faster"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001615,"raw_usage":{"total_tokens":6465,"prompt_tokens":1021,"completion_tokens":5444,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":637,"completion_tokens_details":{"reasoning_tokens":5332}},"tokens_in":637,"tokens_out":5444,"duration_ms":37005,"temperature":1.0,"reasoning_tokens":5332,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:26:53.947842+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Use the thresholds the paper reports for RWKV-7 ($\\tau_c=1.54$, $\\tau_f=30$) on RWKV-6-3B and check both the layer split and LAMBADA perplexity; if the split drifts from roughly 9:1 or perplexity rises by more than the paper's reported margin, the proxy has not isolated a transferable property of RWKV weights.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies K-means, the clustering algorithm used to measure relative cluster loss and to generate the codebooks."}],"review_version":1}