{"id":"ce16397b-6fe8-4fc4-8384-c6242cb3a26e","arxiv_id":"2607.14618","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"PolyQ makes fractional-bit CPU LLM inference practical via per-channel bit allocation from {2,3,4,8,16} and compile-time channel-order merging.","lead":"PolyQ is a quantization-plus-compiler framework that lets edge LLMs run on CPUs at fractional average bit budgets (e.g., 3.4 bits/weight) by assigning different bit-widths to different weight channels and pre-arranging their layout. It matters because it replaces coarse 2/3/4-bit steps with a smooth memory-quality-speed knob for on-device inference.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 2's diagonal independence assumption is unvalidated; if it fails, the waterfilling allocations and the claimed 3-bit quality improvements are not established.","rationale":"The reader identified Eq. (2)'s factorization as the weakest assumption, and I agree. The central claim is that fractional-bit CPU deployment is practical, predictable, and energy-efficient. The 'practical' and 'predictable' quality components rest directly on the waterfilling allocation, which is driven by Eq. (2). If the diagonal-independence approximation is poor for transformer activations, the bit allocations themselves are suspect, and the reported quality gains over uniform baselines could be a proxy artifact rather than a real effect. The full-covariance trace formula is the correct linear-layer objective; Eq. (2) is a simplification that the paper neither justifies nor tests. This is a concrete, falsifiable technical concern, not a stylistic or reproducibility complaint. The compiler-side heuristic is a second fragility, but Section 5.5 provides measured end-to-end latency/throughput evidence that the system remains practical even if that heuristic is suboptimal; by contrast, the quality results have no independent check of the central proxy. I therefore do not recommend changing the conditional verdict: the paper is plausible and interesting, but the load-bearing assumption should be validated before acceptance. If the proposed full-covariance test shows Eq. (2) is accurate, the concern is resolved and the central claim is substantially strengthened. If it fails, the paper's quality claims need re-evaluation. The absence of code/data and error bars further supports keeping the conditional verdict, but those are secondary to the correctness of Eq. (2).","tokens_in":17715,"tokens_out":6806,"duration_ms":83194,"concrete_test":"On a Llama2-13B linear layer that exhibits a nontrivial 2/3/4/8/16-bit mix (e.g., the first attention output projection), use the same 128 WikiText-2 calibration sequences to compute the true expected squared output error trace(Delta W^T C_x Delta W) for every feasible single-channel promotion (2->3, 3->4, 4->8, 8->16) and compare that ranking with the ranking induced by Eq. (5). If the Spearman rank correlation is below ~0.8, or if the waterfilled bit map's full-covariance error is higher than uniform 3-bit's full-covariance error, Eq. (2)'s diagonal independence assumption is load-bearing. Then re-run the Figure 5 perplexity comparison replacing Eq. (2) with the full-covariance objective; if the 3-b quality gains shrink or reverse, the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The quality–budget curve is built on Eq. (2), which approximates a layer's expected squared output error by sum_j ||w_hat_j^(b) - w_j||^2 E[x_j^2]. For a linear layer, the true expectation is trace(Delta W^T C_x Delta W), where C_x is the full input-channel covariance; Eq. (2) is exact only if C_x is diagonal. Transformer channels are not independent: residual connections, LayerNorm, and attention mixing introduce correlations. If off-diagonal terms are material, the per-promotion gains G_{b->b'} in Eq. (5) can mis-rank channels, so the greedy waterfilling in Section 3.2 and the error-aware reassignment in Section 3.3 may allocate bits to the wrong channels. The claimed 2.4–32.1% perplexity improvements over uniform baselines at 3 b, and the 'stable scaling' shown in Figure 5, then become artifacts of the diagonal proxy rather than consequences of fractional budget control. The paper never validates Eq. (2) against actual layer-wise output error or a full-covariance proxy; it reports only end-task quality. Because the same proxy is reused in the quanta-matching step and in the bit-specific scaling of Section 3.4, an error here propagates through the entire quantization pipeline. This is the least secure link in the central claim; the compiler-side overhead is measured end-to-end in Section 5.5 and is therefore less load-bearing.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PolyQ, a quantization/compiler co-design for CPU-only LLM inference that assigns per-channel bit-widths from {2,3,4,8,16} under a user-specified fractional average-bit budget. The accuracy side uses an activation-aware per-channel error proxy (Eq. 2) to drive a greedy waterfilling allocation (Section 3.2), followed by ISA-aware quanta matching and bit-specific scaling. The compiler side permutes and clusters channels into bit-homogeneous blocks, generates LUT/SIMD kernels, and propagates compatible permutations across the operator DAG to avoid runtime reorders (Section 4). Experiments on Falcon-H1-3B, Llama2-13B, and Qwen3-32B report smooth quality scaling with budget, 2.4–32.1% perplexity improvements over prior methods at a 3-bit target, reduced modeled reorder traffic, and end-to-end CPU latency/throughput/energy measurements on three platforms.","tokens_in":18153,"tokens_out":5869,"duration_ms":61707,"significance":"If the claims hold, PolyQ is a meaningful step toward practical fractional-bit operation for CPU-only LLM inference, replacing coarse 2/3/4-bit points with a continuous budget-quality-latency design curve. The paper's strengths include end-to-end measurements on three CPU classes, a clearly specified waterfilling formulation, and a compiler design that explicitly treats layout regularization as a graph-labeling problem. The central direction is sound, but several load-bearing components—the error proxy, the DAG-labeling heuristic, and the calibration/evaluation split—require further validation before the headline claims can be fully credited.","major_comments":[{"comment":"The calibration/evaluation split is ambiguous. Calibration uses 128 randomly sampled WikiText-2 sequences; perplexity is then evaluated on WikiText-2. If the calibration and evaluation sets overlap, the reported perplexity improvements are optimistic and the comparison with baselines is unfair. State explicitly whether the calibration sequences are drawn from the training split and are strictly disjoint from the test set used for perplexity. If they are not disjoint, rerun the evaluation with a separated calibration set.","section":"§5.1"},{"comment":"The per-channel independent error proxy is load-bearing for the waterfilling allocation (Section 3.2–3.3) but is never validated. For a linear layer, the true expected squared output error is trace(ΔWᵀ C_x ΔW); Eq. (2) omits off-diagonal input covariance terms. Channel correlations induced by residual connections and LayerNorm may make these terms material. Provide a concrete validation: on at least one model, compare the proxy ranking and the resulting allocation against the full-covariance objective, or measure actual layer-wise output error after applying the proxy-based allocation.","section":"§3.1, Eq. (2)"},{"comment":"The DAG-labeling heuristic (topological propagation with local conflict resolution) is not proven to find near-optimal bases, and the headline reorder-traffic reduction (Figure 10) is a modeled quantity derived from this heuristic, not an end-to-end measured overhead. Add a comparison to an optimal/exhaustive labeling on small subgraphs, or report measured shuffle cost in the end-to-end runtime, to justify the 46.6–70.8% reduction claim.","section":"§4.4"},{"comment":"No error bars or seed variance are reported. The claims of 'stable quality scaling' and 'small local fluctuations' depend on the noise level, and the downstream accuracies in Figure 6 are computed once. Report mean and variance over at least 3 random seeds for calibration sampling and, where applicable, evaluation, especially at the 3-bit operating point that anchors the headline improvement.","section":"§5.2, Figures 5–6"}],"minor_comments":[{"comment":"The phrase 'using small residual corrections to preserve the total number of channels' is underspecified. Describe the rounding/correction procedure concretely (e.g., greedy adjustment by largest residual).","section":"Eq. (6)"},{"comment":"The criterion 'keep the label with the largest downstream reuse' is qualitative. Define 'reuse' formally or give pseudocode so the heuristic is reproducible.","section":"§4.4"},{"comment":"The term 'W3 overhead' is computed as 3/B_min − 1; this is a memory ratio, not an overhead. Consider renaming to 'memory savings of B_min vs. uniform W3' or clarifying the definition.","section":"Table 2"},{"comment":"The y-axis tick labels '0 9.0 18' are ambiguous. Label the axis as 'Average reorder traffic (MiB/token)' and use standard tick positions (e.g., 0, 5, 10, 15).","section":"Figure 10"},{"comment":"The sentence 'prefill latency stays within 4.7% of AWQ(T-MAC) on Ryzen 9950X, within 5.8–7.3% on Ryzen 7840U' mixes the summary for the first platform with a range for the second; clarify whether the range is across bit budgets or across repeated runs.","section":"§5.5"},{"comment":"The claim that Slim-LLM, RPTQ, and Atom 'implement GPU-only execution backends and have no CPU inference path' is asserted without a supporting reference or implementation detail; please clarify how this was determined.","section":"§5.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is well aligned with an ML-systems venue and the experimental scope is substantial. The main risks are the ambiguous calibration/evaluation split and the lack of direct validation of the error proxy and the labeling heuristic. These are addressable with additional experiments. I would also encourage the authors to release code or at least a detailed artifact, since many of the claims are tied to a specific implementation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth a serious referee. PolyQ's contribution is real: a per-channel bit-allocation over a {2,3,4,8,16} palette with fractional budget control, plus a graph-level compiler pass that propagates channel permutations across operators so mixed-precision execution does not incur runtime shuffles. That combination is new relative to Slim-LLM, AMQ, RPTQ, and Atom, and it is evaluated on three CPUs with energy and latency measurements that are careful and consistent.\n\nWhat I like most is the end-to-end budget scaling: prefill latency and decode throughput track the bit budget nearly proportionally, energy overhead is under 2%, and the budget realization after quanta matching stays within 0.045 b of target. Those numbers are concrete and the method is genuinely useful for edge deployment. The compiler layout analysis is also a real step forward; the reorder-traffic model is explicit and the 46-71% reduction over prior policies is plausible, though it is modeled rather than measured in end-to-end runtime.\n\nThe soft spots are not disqualifying but they matter. No code or artifact is provided, and no error bars or seed variance are reported. That makes the quality claims, especially the 2.4-32.1% perplexity improvements at 3 b, harder to trust than they should be. The reorder-traffic reduction is from a traffic model, not from profiling the actual generated kernels. The calibration set is randomly sampled from WikiText-2 and perplexity is also evaluated on WikiText-2; common practice, but worth confirming the model has not seen the exact sequences.\n\nThe deeper concern is the error proxy in Eq. (2). The paper weights per-channel weight reconstruction error by activation energy, implicitly assuming the input-channel covariance is diagonal. In transformer layers with residual connections, LayerNorm, and attention mixing, that is not true. The paper never validates Eq. (2) against actual layer-wise output error or a full-covariance proxy, and the same proxy drives both waterfilling and the scaling of Section 3.4. If the proxy misranks channels, the allocation could be suboptimal. That said, the method still delivers consistent end-task improvements on three models, and the proxy is a standard activation-aware heuristic; this is a robustness question, not a demonstrated fatal flaw. A good referee should ask for a covariance-based or empirical validation of Eq. (2) on at least one model.\n\nOverall, this is a well-executed systems paper with a new contribution and careful measurements. I would cite it, and I would bring it to a reading group. It needs an artifact or code release, error bars, a validation of the error proxy, and ideally a measured (not modeled) reorder-traffic comparison. Those are fixable and should be requested, not used to desk reject.\n\nRecommendation: send it to peer review.","headline":"Real contribution, careful measurements, but unvalidated error proxy and missing artifact; deserves a serious referee.","tokens_in":18583,"tokens_out":2827,"would_cite":true,"duration_ms":29218,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PolyQ shows that per-channel mixed precision can run efficiently on ordinary CPUs by moving all layout irregularity into the compiler, so fractional bit budgets like 3.7 bits per weight become deployable design points.","keywords":["LLM inference","CPU quantization","mixed precision","post-training quantization","activation-aware quantization","edge deployment","compiler optimization","quantization co-design"],"falsifier":"Take one of the three models and compare the greedy allocation against a near-exhaustive search over a few hundred random bit maps at the same average budget. If simple random or uniform maps match or beat the proxy-selected map on end-to-end perplexity, the proxy is not the reason for the reported gains.","tokens_in":17694,"feed_emoji":"⚙️","tokens_out":5304,"duration_ms":52988,"temperature":0.7,"pith_summary":"This paper claims that fine-grained, channel-wise bit allocation for large language models can be made practical on CPUs, where earlier mixed-precision methods were too irregular to run efficiently. It proposes assigning each weight channel its own bit-width from 2, 3, 4, 8, or 16 bits while keeping a user-chosen average budget, then moving the required channel reshuffling into the compile-time layout of weights and kernels. If correct, a single average-bit parameter would continuously trade memory footprint, output quality, latency, and energy, replacing coarse choices like '3-bit' or '4-bit' models with any fractional target such as 3.7 bits per weight. The measured claims are smooth quality scaling from 3 to 6 bits, perplexity gains of 2.4–32.1% over earlier methods at a 3-bit target, up to 70.8% less activation-reorder traffic, and under 2% energy overhead relative to an optimized low-bit backend.","feed_headline":"Fractional-bit CPU inference scales smoothly from 3 to 6 bits","feed_subtitle":"A per-channel bit allocator plus compile-time layout makes fractional weights run on ordinary CPUs.","key_machinery":"The load-bearing object is an activation-aware per-channel error proxy: for each channel, the squared reconstruction error between original and quantized weights, multiplied by that channel's calibration activation energy. A greedy waterfilling step repeatedly promotes the channel whose error reduction per extra bit is largest, under a fixed average bit budget, over the palette {2,3,4,8,16}. A compile-time compiler then permutes and clusters channels of equal bit-width, assigns a low-bit table-lookup or SIMD kernel to each block, and propagates compatible channel bases across graph edges so most activation shuffles never appear at runtime.","core_discovery":"The central discovery is that per-channel mixed precision no longer has to be execution-hostile. The paper shows that if the bit-width of every input channel is chosen greedily by a waterfilling procedure that weights each channel's weight-quantization error by its activation energy, the resulting irregular maps can be clustered into bit-homogeneous blocks, matched to SIMD or table-lookup kernels, and stitched together by propagating compatible channel permutations across the model's operator graph at compile time. The authors report that the same average-bit budget then predicts both memory footprint and runtime with only small residual overhead, making fractional deployment a smooth design","pith_inferences":["If the proxy's factorization holds broadly, the same waterfilling skeleton could be retargeted to other CPU or accelerator ISAs by swapping in the backend's block quanta, an abstraction the paper already sketches.","The collapse-to-recovery thresholds near 2.1–2.6 bits suggest a per-device memory-aware scheduler could automatically pick the smallest fractional budget that keeps a model usable, an extension beyond the paper's fixed-curve evaluation.","The method's quality claims rest on three models and one perplexity benchmark plus a small multitask set; a natural next test is whether the reported gains persist on a wider model family, especially models with correlated channels.","Because the compiler propagates channel bases through any basis-preserving operator, the layout machinery may also apply to attention-specific or non-linear structures beyond the transformer blocks evaluated."],"forward_implications":["The average-bit budget becomes a first-class design knob: setting it to, say, 3.4 or 3.9 bits yields corresponding memory, quality, latency, and energy points instead of only W2/W3/W4 choices.","Sub-3-bit operating points can be practical if chosen just past the model's collapse threshold, saving roughly 12–22% peak memory compared with the nearest uniform 3-bit deployment.","Compile-time layout regularization can remove most activation-reorder traffic (up to 70.8%) without changing the bit allocation itself.","Measured prefill latency and decode throughput scale nearly proportionally with the configured bit budget, so runtime can be predicted from the same parameter that controls memory and quality.","Fine-grained channel-wise precision on CPUs can be nearly energy-neutral, with energy per token within about 2% of a uniform low-bit backend."],"fun_headline_variants":["CPU inference scales 3-6 bits with per-channel bits and compile-time layout","PolyQ: fractional-bit LLMs on CPUs without the reorder penalty","Waterfilling bit allocation plus compiler layout speeds up edge LLM","Per-channel precision that runs: CPU layout cuts reorder traffic 70%","From 3 to 6 bits, CPU LLM inference stays smooth and low-energy"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The method assumes each channel's quantization error is fairly scored by its own weight error times its activation energy, independent of other channels, so spending extra bits on the highest-scoring channels is always the best allocation.","fun_headline_variants_meta":{"raw":{"variants":["CPU inference scales 3-6 bits with per-channel bits and compile-time layout","PolyQ: fractional-bit LLMs on CPUs without the reorder penalty","Waterfilling bit allocation plus compiler layout speeds up edge LLM","Per-channel precision that runs: CPU layout cuts reorder traffic 70%","From 3 to 6 bits, CPU LLM inference stays smooth and low-energy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000761,"raw_usage":{"total_tokens":3243,"prompt_tokens":802,"completion_tokens":2441,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":2354}},"tokens_in":546,"tokens_out":2441,"duration_ms":21576,"temperature":1.0,"reasoning_tokens":2354,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T01:33:45.582046+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take one of the three models and compare the greedy allocation against a near-exhaustive search over a few hundred random bit maps at the same average budget. If simple random or uniform maps match or beat the proxy-selected map on end-to-end perplexity, the proxy is not the reason for the reported gains.","supporting_citations":[],"review_version":1}