{"id":"c0e6780a-2122-44b0-a8bb-b21aa2b617c0","arxiv_id":"2501.12956","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"GANQ minimizes layer-wise output error for lookup-table based non-uniform weight quantization, improving LLM perplexity at 3-4 bits and enabling up to 2.57x inference speedup.","lead":"GANQ is a post-training method that compresses large language models to 3 or 4 bits per weight using a per-channel codebook, optimized to keep each layer's output close to the original. On standard benchmarks it improves perplexity over several established quantizers and runs up to 2.57x faster than a full-precision baseline on one GPU.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The greedy S-update in Eq. (22) is the load-bearing heuristic: it is explicitly suboptimal, has no approximation bound or ablation, and the unspecified initial codebook T0 makes the algorithm's behavior and reproducibility hard to assess.","rationale":"The paper's math is mostly internally consistent and the experiments are broad, including WikiText-2, C4, PTB, zero-shot tasks, LongBench, and latency profiling. The main soft spot is the one explicitly admitted by the authors: the back-substitution solve for S_i is sub-optimal with no bound and no ablation. I agree with the reader that this is the weakest load-bearing assumption, because S drives the codebook assignment and the T-update depends on S through Eq. (7). The unspecified T0 initialization is a secondary but real reproducibility gap that should be closed by reporting the initialization used in the released code. No evidence suggests a contrived comparison; the question is whether the heuristic in Eq. (22), and the missing initialization, are good enough to support the claimed advantage. An ablation with local search and varied T0 would settle this. Since the reader already assigned CONDITIONAL based on similar reasoning, I recommend no change.","tokens_in":19219,"tokens_out":6708,"duration_ms":72612,"concrete_test":"Add an ablation to the released code for OPT-125M and LLaMA-7B at 3 and 4 bits: after the greedy back-substitution in Algorithm 1, run several passes of column-wise local search over S_i, each time re-assigning every column to the best of the 2^N codebook values given all other columns and the current T, and report the layer-wise objective in Eq. (13) and the final WikiText-2 perplexity for K = 1, 5, 10. If local search changes perplexity by more than about 0.1 or the objective by more than 5%, then the suboptimality of Eq. (22) is performance-relevant; if not, the heuristic is adequate. Also run the same pipeline with at least three different T0 initializations (uniform grid, k-means, random) to bound the dependence on the unspecified initial codebook.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that GANQ achieves superior quantization by a principled, GPU-adaptive optimization rests on the alternating scheme in Eqs. (3)-(4). The T-update is exact, but the S-update is not: Section 3.2 states that Eq. (22) yields a 'sub-optimal' solution to the discrete subproblem (3). Because Eq. (22) assigns each column to the codebook entry that minimizes only the current residual term and ignores the effect of that choice on earlier columns j' < j, the objective in (13) is not globally minimized unless a bound or an ablation shows the greedy error is negligible. The paper provides neither. A poor S can bias the subsequent T-update through (7), so the whole alternating scheme may inherit an uncontrolled approximation error. This is especially consequential because the method is marketed as a principled alternative to heuristic clustering (Section 2). Reproducibility is further weakened by Algorithm 1 taking an unspecified initial codebook T0; the text never says whether T0 is a uniform grid, k-means centroids, or a random draw, and the final result can depend on it in a nonconvex alternating scheme. These are addressable empirical questions rather than demonstrated contradictions, so the reader's CONDITIONAL verdict is appropriate.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GANQ, a post-training weight-only quantization method for LLMs in which each layer's weights are represented by a per-channel codebook and a low-bit index matrix, enabling lookup-table-based mixed-precision matrix multiplication. The authors formulate the layer-wise quantization problem as a mixed-integer quadratic program and solve it by alternating between an exact closed-form update of the codebooks and a Cholesky-based greedy back-substitution update of the index assignments. Experiments on OPT, LLaMA, LLaMA-2, LLaMA-3, and LLaMA-3.2 models at 3 and 4 bits report perplexity improvements over RTN, GPTQ, OmniQuant, and SqueezeLLM in most configurations, plus zero-shot and long-context evaluations, and a reported up-to-2.57x inference speedup over FP16 on an RTX 4090.","tokens_in":19539,"tokens_out":5348,"duration_ms":57349,"significance":"If the central claims hold, GANQ is a useful contribution: it offers a principled optimization-based alternative to heuristic clustering for non-uniform LUT quantization, is training-free, parallelizes naturally across GPU rows, and is compatible with existing outlier-splitting techniques such as those used by SqueezeLLM. The authors also make their code publicly available and report a broad set of models, bit-widths, and tasks. The significance is conditional, however, because the discrete index-update step is a heuristic with no formal guarantee and no ablation, and because the empirical claim of consistent superiority is slightly stronger than the reported numbers support.","major_comments":[{"comment":"The greedy back-substitution that solves the S_i subproblem is explicitly described as producing a sub-optimal solution, yet the paper provides neither an approximation bound nor an ablation. This is a load-bearing step because the T-update in Eq. (7) is computed from the current S, so an uncontrolled error in the greedy S-update can propagate through the alternating iterations. I request either a formal error bound or an empirical study: on small layers where exhaustive search is feasible, compare the greedy solution with the optimal one; report the objective value and downstream perplexity for the greedy update versus k-means initialization, random restarts, or a few iterations of local search; and report sensitivity to the number of alternating iterations K.","section":"Section 3.2, Eq. (22)"},{"comment":"The initial codebook T0 appears only as an input to the algorithm; the text never specifies how T0 is chosen. In a nonconvex alternating scheme, the final codebook and index matrix can depend on this initialization, so the current description is incomplete for reproducibility. Please specify the initialization (e.g., uniform grid over the weight range, k-means centroids, or percentile-based values) and report a sensitivity experiment over several plausible initializations.","section":"Algorithm 1"},{"comment":"The abstract and Section 4.2 state that GANQ consistently outperforms state-of-the-art methods, but Table 5 shows that GANQ* ties SqueezeLLM on LLaMA-2-7B at 4-bit (5.57 vs. 5.57) and leads by only 0.01 on LLaMA-7B at 4-bit (5.76 vs. 5.77). No standard deviations, repeated runs, or significance tests are reported. Please soften the claim to 'generally improves' or add repeated-run statistics and significance tests, especially for the configurations where the margins are within 0.01-0.02 in perplexity.","section":"Tables 2 and 5"},{"comment":"The method has two free hyperparameters that directly affect the reported results: the number of alternating iterations K (given as K=10 in Section 4.4) and the outlier extraction ratio r (default r=0.5% in Appendix B). The paper reports no ablation for either parameter. Since the performance of the greedy S-update and the outlier-splitting step both depend on these choices, a sensitivity analysis is needed to establish that the reported gains are not tied to a narrow parameter setting.","section":"Section 4.4 and Appendix B"}],"minor_comments":[{"comment":"The phrase 'wight-only quantization' appears in the Related Work; this should be 'weight-only quantization'.","section":"Section 2"},{"comment":"In the GPTQ 4-bit row, the WinoGrande and Arc-e entries appear concatenated as '57.7275.25'; the table should be reformatted to show separate values.","section":"Table 3"},{"comment":"The back-substitution diagram is difficult to read in the provided version; please redraw it with larger fonts and clearer annotations.","section":"Figure 2"},{"comment":"The C4 perplexity results are reported after calibrating on C4 data, and PTB results are reported only for OPT models. While the text justifies the OPT-only PTB choice, a non-overlapping calibration/evaluation split or PTB numbers for LLaMA models would make the evaluation more convincing.","section":"Appendix C"},{"comment":"The outlier extraction uses floor and ceiling operations on percentile cutoffs, so the actual number of outliers per row may not equal r times the row length; please clarify how the reported ratio is enforced or state that it is approximate.","section":"Appendix B, Algorithm 2"},{"comment":"The GitHub URL in the paper contains a space ('https://github.com/Evans -Z/GANQ'); please correct it so the link resolves.","section":"Availability"}],"recommendation":"major_revision","confidential_remarks":"The central formulation and the exact T-update are sound, and the GPU-parallel row-wise design is appealing. The main risk is that the greedy S-update, which is load-bearing, has no analysis or ablation, and the initial codebook is unspecified. If the authors supply the requested experiments and specification, the paper could plausibly be accepted; without those, the reproducibility and the strength of the empirical claims remain insufficiently supported. The comparison against SqueezeLLM is also closer in some 4-bit cells than the text suggests, which the authors should address explicitly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The interesting part of GANQ is the formulation: it writes per-channel LUT quantization as a mixed-integer quadratic program with one-hot assignment constraints, then solves it with an alternating scheme. The T-update is exact; the S-update is a greedy back-substitution that exploits the Cholesky factor of XX^T. That is a clean algorithmic idea, and it parallelizes naturally on a GPU. The experiments are broader than most PTQ papers: OPT, LLaMA-2/3/3.2 at 3 and 4 bits, perplexity on three datasets, zero-shot accuracy, LongBench, and latency/memory profiling with a real LUT kernel. The speedups (up to 2.57x) are credible. Code is public and the method is simple enough to reimplement easily.\n\nThe soft spot is exactly what the paper admits but does not quantify: the S-update is suboptimal, with no approximation bound and no ablation. The greedy choice at each column ignores its effect on earlier columns, and since T is re-estimated from S, error in S propagates. This is not a fatal flaw—most practical quantizers are heuristic—but the paper markets itself as principled, so the mismatch matters. I would want to see either an ablation (e.g., compare against a slower exact or randomized assignment on small layers) or sensitivity analysis for the initial codebook T0, which is an input to Algorithm 1 but never specified in the text. In a nonconvex alternating scheme the initialization can matter a lot.\n\nThe empirical comparison is solid but the claim of consistent superiority is a bit strong given that GANQ ties SqueezeLLM in several 4-bit cases. The PTB table covers only OPT, with an explanation but still a gap. The C4 overlap (calibrating and evaluating on C4) is minor and standard.\n\nOverall, this is a useful incremental contribution. The formulation and solver are worth knowing, and the experiments are honest enough to serve as a baseline for future work. With a small revision that pins down T0 and gives an ablation or bound for the S-update, this would be a solid paper. I would send it to review.","headline":"GANQ is a solid, incremental quantization paper with a genuinely clean MIQP formulation and a fast GPU-parallel alternating solver, but the greedy S-update is an unquantified heuristic and the initial codebook is unspecified.","tokens_in":20072,"tokens_out":2361,"would_cite":true,"duration_ms":25612,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GANQ makes 3- and 4-bit weights rival full precision by giving each channel its own codebook, selected to minimize layer output error rather than rounding error.","keywords":["post-training quantization","non-uniform quantization","lookup table","mixed-integer quadratic programming","GPU acceleration","large language models","mixed-precision GEMM","weight-only quantization"],"falsifier":"A reader could settle the accuracy claim by running GANQ on a single small layer ($n \\le 16$, 4-bit) and comparing the final objective $\\|WX - T S X\\|^2$ to the global optimum found by enumerating all $16^n$ assignments; a large gap on realistic weight rows would show the method rests on an unquantified heuristic. A simpler test is to permute the columns before quantization: the optimal objective is permutation-invariant, but the greedy right-to-left solver is not, so any measurable degradation would expose where the heuristic loses accuracy.","tokens_in":19006,"feed_emoji":"⚡","tokens_out":6294,"duration_ms":65097,"temperature":0.7,"pith_summary":"GANQ argues that the right way to do post-training weight quantization is to let each output channel choose a small codebook of representable values and assign every weight to the nearest codebook entry in a way that minimizes the layer's output error, not just the weight error. The paper frames this as a mixed-integer quadratic program per layer and solves it with a training-free, GPU-parallel alternating algorithm, so the optimization itself is cheap enough to run on a single GPU. In experiments on OPT and LLaMA models, GANQ lowers 3- and 4-bit perplexity relative to RTN, GPTQ, and OmniQuant, and it can be combined with outlier-splitting methods for further gains. Because the quantized form is a lookup-table codebook, inference avoids dequantization and runs up to 2.57 times faster than the FP16 baseline on a single RTX 4090.","feed_headline":"GANQ quantizes LLMs to 3-4 bits with lower perplexity","feed_subtitle":"A training-free solver assigns per-channel codebooks, shrinking memory to a few GB and beating the FP16 baseline by 2.57x.","key_machinery":"The machinery is per-channel codebook quantization algebraically coupled to the input statistics. Define $H = XX^\\top$, take its Cholesky factor $L$, and rewrite the row objective as $\\|W_i L - T_i S_i L\\|^2$; because $L$ is lower triangular, the squared error decomposes into $n$ independent terms ordered by column, and the assignment for column $j$ can be fixed by a nearest-codebook lookup after subtracting the accumulated residual from earlier columns. This back-substitution turns a combinatorial search into $O(n)$ nearest-neighbor steps per row, and the same $L$ is shared across all rows, so all rows update in parallel as batched matrix operations. The codebook $T$ then updates in closed form via a Moore-Penrose inverse of the small $2^N \\times 2^N$ matrix $S H S^\\top$.","core_discovery":"The paper's central claim is that a principled optimization model for LUT-based non-uniform quantization—minimize $\\|WX - T S X\\|_F^2$ over a per-channel codebook $T$ and a one-hot assignment $S$—can be solved efficiently at LLM scale by exploiting row-wise separability. Each row's problem is a mixed-integer quadratic program; GANQ alternates between a closed-form least-squares update of the codebook $T$ and a greedy back-substitution update of the assignment $S$ along the Cholesky factor of $XX^\\top$. The paper reports that this yields lower perplexity than uniform-quantization baselines at 3 and 4 bits, near-FP16 zero-shot accuracy on LLaMA-2-7B, and, when paired with existing lookup-table inference kernels, a 2.57$\\times$ speedup over FP16 with peak memory under 4.1 GB on OPT-6.7B.","pith_inferences":["Beyond the paper: the row-separable formulation depends only on $XX^\\top$, not on the weight values themselves, so the same solver could be applied to other matrix-multiplication-bound tensors such as KV caches or MoE expert weights without re-deriving the algorithm.","Beyond the paper: the greedy back-substitution is analogous to successive interference cancellation, so a cheap local-refinement pass (re-assigning columns whose residual remains large) could close part of the suboptimality gap the paper concedes; this is a testable extension, not a claim in the paper.","Beyond the paper: the reported speedup uses existing LUT kernels, but the algorithm emits exactly the codebook-plus-index format those kernels consume, so GANQ's inference advantage should grow as kernel-level LUT engineering improves without changing the quantization procedure."],"forward_implications":["At 3 and 4 bits, GANQ reports lower WikiText-2 perplexity than RTN, GPTQ, and OmniQuant on every OPT and LLaMA model tested, including cases where a quantized model matches or beats the FP16 baseline.","Because the per-channel codebook adds only $2^N m$ extra parameters, the storage overhead stays within a few tenths of a percent of uniform per-channel quantization, so the memory savings of low-bit weights are preserved.","GANQ is orthogonal to outlier splitting: retaining 0.5% outliers or 10 full rows lets GANQ$^\\star$ beat the same baselines on OPT and LLaMA models while keeping a sparse component for extreme weights.","Using existing LUT-based kernels, GANQ-quantized OPT-6.7B achieves up to a 2.57$\\times$ speedup over FP16 and peak memory of 4.10 GB at 3 bits, compared with 12.91 GB for FP16."],"supporting_citations":[{"why":"Supplies GPTQ, the principal PTQ baseline, and the calibration-data protocol (C4 sequences of 2,048 tokens) that GANQ adopts.","marker":"Frantar et al., 2022"},{"why":"Supplies OmniQuant, the learnable-scale/zero-point baseline GANQ claims to outperform on perplexity.","marker":"Shao et al., 2024"},{"why":"Supplies SqueezeLLM, the sensitivity-based non-uniform clustering baseline, and the LUT-based inference kernel used for latency and memory measurements.","marker":"Kim et al., 2024"},{"why":"Supplies AffineQuant, a baseline that preprocesses weights with learnable affine transforms before uniform quantization.","marker":"Ma et al., 2024"},{"why":"Supplies SpQR's outlier-splitting idea that GANQ says it is compatible with for further gains.","marker":"Dettmers et al., 2023"},{"why":"Supplies NUPeS, the power-exponent non-uniform quantization approach contrasted with GANQ's optimization-based method.","marker":"Yvinec et al., 2023"},{"why":"Supplies an additional LUT-based mpGEMM kernel line that GANQ says could further improve its inference speed.","marker":"Mo et al., 2024"},{"why":"Supplies another fast LUT-based matrix-multiplication kernel that GANQ identifies as a source of further speedup.","marker":"Guo et al., 2024"}],"fun_headline_variants":["GANQ: 2.57x faster LLMs with 3-bit weights","Training-free GANQ shrinks LLMs to 4GB, keeps accuracy","Non-uniform quantization with GANQ: lower perplexity, 2.5x speed","GPU-adaptive GANQ cuts LLM memory and boosts speed","GANQ: non-uniform bits, 2.57x faster LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The accuracy claim depends on the unproven assumption that the greedy right-to-left back-substitution in Eq. (22), which chooses each column's codebook entry while ignoring its effect on earlier columns, produces a discrete assignment close enough to the true optimum that the layer-wise objective stays low.","fun_headline_variants_meta":{"raw":{"variants":["GANQ: 2.57x faster LLMs with 3-bit weights","Training-free GANQ shrinks LLMs to 4GB, keeps accuracy","Non-uniform quantization with GANQ: lower perplexity, 2.5x speed","GPU-adaptive GANQ cuts LLM memory and boosts speed","GANQ: non-uniform bits, 2.57x faster LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000687,"raw_usage":{"total_tokens":3120,"prompt_tokens":957,"completion_tokens":2163,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":573,"completion_tokens_details":{"reasoning_tokens":2059}},"tokens_in":573,"tokens_out":2163,"duration_ms":17514,"temperature":1.0,"reasoning_tokens":2059,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T16:35:19.632616+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could settle the accuracy claim by running GANQ on a single small layer ($n \\le 16$, 4-bit) and comparing the final objective $\\|WX - T S X\\|^2$ to the global optimum found by enumerating all $16^n$ assignments; a large gap on realistic weight rows would show the method rests on an unquantified heuristic. A simpler test is to permute the columns before quantization: the optimal objective is permutation-invariant, but the greedy right-to-left solver is not, so any measurable degradation would expose where the heuristic loses accuracy.","supporting_citations":[{"cited_title":"Fast matrix multiplications for lookup table-quantized llms","cited_arxiv_id":null,"evidence_quote":"Supplies another fast LUT-based matrix-multiplication kernel that GANQ identifies as a source of further speedup."}],"review_version":1}