{"id":"4fe1ed71-3887-4dc7-b6cc-db3411f6e06f","arxiv_id":"2502.06844","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A discrete hill-climbing search over permutation, scaling, and rotation invariances improves 2-bit quantized OPT models when applied on top of GPTQ, AWQ, and OmniQuant.","lead":"The paper proposes InvarExplore, a hill-climbing search that permutes, scales, and rotates layer weights of a quantized LLM to reduce quantization error. It reports consistent perplexity and accuracy gains on 2-bit OPT models when added to existing quantization methods.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The activation-matching term in Algorithm 1 is constant (MSE(H,H0) never depends on the candidate θ′), so the search as pseudocoded is not activation-guided and Table 4 cannot follow from the stated algorithm.","rationale":"Good-faith reading: the authors clearly intend the MSE to compare original unquantized activations with the candidate quantized model's activations; the text surrounding Eq. (23) says so, and Table 4 is presented as evidence. But the formal Algorithm 1 does not say this. The load-bearing claim is not merely rotation approximate invariance—the authors concede that—but that the search is activation-guided and that the activation-matching component explains part of the gain. As printed, that component is a constant and therefore cannot guide anything. This is an internal inconsistency that is checkable from the manuscript alone, unlike the rotation issue which requires empirical replication. It also strengthens the reader's reproducibility concern: if the pseudocode is wrong in this central spot, the reported numbers cannot be independently regenerated without guessing the correct objective. I therefore keep the conditional verdict (the method may be salvageable with a corrected algorithm and released code) but shift the justification from the rotation approximation to this more decisive algorithmic gap. Agreeing partially with the reader: they identified pseudocode inconsistency in passing but made rotation the weakest assumption.","tokens_in":15982,"tokens_out":5887,"duration_ms":66978,"concrete_test":"Implement Algorithm 1 literally, with H and H0 computed once and the MSE(H,H0) term left constant, across the 13B OPT, 2-bit group-128 AWQ setting, and run Table 4 with 0/1/5/10 matched layers. If the printed algorithm is followed, all layer counts yield identical perplexity; if Table 4 is reproduced, then the reported runs must have recomputed activations for each candidate, contradicting the pseudocode. As a complementary check, set α = 0 in the corrected implementation and compare to α > 0: if results are identical, the activation-matching loss has no effect and the 'activation-guided' claim fails.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Algorithm 1 (lines 1–2, 16) sets H = M(X, θ0) and H0 = M(X, quant(θ0)) once before the loop and then evaluates every candidate with L′ = LKL(X, quant(θ′)) + α MSE(H, H0). Since H and H0 are fixed, the second term is identical for all θ′ and cannot influence acceptance. The paper's core claim of an 'activation-guided discrete search' and the analysis in Table 4 ('matching more layers... leads to higher performance') are therefore not reproducible from the printed algorithm. The reader's rotation concern is real but explicitly acknowledged as approximate; this issue is unacknowledged and breaks the method as specified. A related ambiguity: line 16 should almost certainly use the activations of the candidate quantized model, e.g., MSE(M(X, θ0), M(X, quant(θ′))) over matched layers, but that is not what is written. Without released code, the discrepancy cannot be resolved by inspection.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes INVAREXPLORE, a framework for improving ultra-low-bit post-training quantization of LLMs by searching over permutation, scaling, and rotation transformations of Transformer feed-forward blocks. These transformations preserve (or approximately preserve) the unquantized model's function, while changing the rounding behavior under integer quantization. The authors present an activation-guided discrete hill-climbing search, apply it on top of GPTQ, AWQ, and OmniQuant for OPT models from 1.3B to 13B at 2-bit quantization, and report perplexity on WikiText-2 and C4 plus average accuracy across six reasoning tasks. They also provide ablations for individual transformations, bit widths, activation-matching layers, and calibration-set size.","tokens_in":16245,"tokens_out":8308,"duration_ms":89996,"significance":"If the empirical claims hold after a corrected algorithm is supplied, the contribution is useful: a model-agnostic add-on to existing PTQ methods, and one of the few attempts to optimize permutation invariance through discrete search rather than gradient-based methods that cannot easily handle permutation symmetry. The paper is clearly written, positions itself well against prior work, and openly acknowledges that rotation invariance is approximate. However, the central 'activation-guided' mechanism is not implemented as written in Algorithm 1, no code or random-seed details are provided, and the reported gains over the strongest baseline, OmniQuant, are often small or accompanied by task-level regressions. These issues prevent the current version from being reproducible and make the strength of the central claim difficult to assess.","major_comments":[{"comment":"As written, Algorithm 1 does not implement an activation-guided search. Lines 1–2 compute H = M(X, θ0) and H0 = M(X, quant(θ0)) once, before the search loop; every candidate θ′ is then evaluated at line 16 with L′ = LKL(X, quant(θ′)) + α MSE(H, H0). Because H and H0 are fixed, the activation-matching term is constant and cannot influence acceptance. The search as pseudocoded therefore reduces to calibration cross-entropy alone, and the result in Table 4 that matching more layers improves performance cannot follow from the stated algorithm. In addition, the text says that only 10% of the neurons in a layer are changed per update, but lines 12–14 sample full new permutation, scaling, and rotation vectors for the chosen layer. Please rewrite the pseudocode so that the activation-matching term is computed against the candidate quantized model, e.g., MSE(M(X, θ0), M(X, quant(θ′))) over the matched layers, make the subset update explicit, and release code or precise pseudocode that matches the experiments.","section":"Section 3.2, Algorithm 1, and Table 4"},{"comment":"The paper claims 'consistent' improvement over OmniQuant, but the point estimates show several ties and regressions. For OPT-2.7B, the average reasoning accuracy is unchanged at 50.62, with BoolQ dropping from 56.91 to 56.09; for OPT-13B, BoolQ drops from 66.09 to 65.78, PIQA from 73.07 to 72.96, and WinoGrande from 62.12 to 61.80. The gains over OmniQuant in Table 1 are often small, e.g., WikiText-2 perplexity 12.94 vs. 12.73 at 13B. Because the search involves random shuffles and Gaussian random walks, single point estimates are insufficient to support a claim of consistent improvement. Please report means and standard deviations over multiple random seeds, and where possible paired tests across tasks; the aggregate win/loss count in Appendix A is not a substitute for per-setting variance.","section":"Section 4, Tables 1 and 5"},{"comment":"The paper acknowledges that the rotation transform is only approximately invariant, but the supporting evidence is a single FP16 pilot measurement on 13B OPT (WikiText-2 cross-entropy 2.31528 vs. 2.31525). The approximation must hold after quantization and for every layer where rotations are applied, yet Table 2 shows rotation-only is a substantial contributor (PPL 30.21 vs. AWQ 35.89 on WikiText-2). Please add a per-layer or per-block analysis of the invariance violation, or quantify how much the rotation search changes the unquantized model's outputs across model sizes. Without this, the reader cannot determine whether the rotation component is exploiting the claimed invariance or merely fitting the calibration set by altering the model's behavior.","section":"Section 3.2, Eqs. (16)–(17), and Table 2"}],"minor_comments":[{"comment":"The sentence 'Here, Eqn. (13) follows because permutation...' appears to be a cross-reference error; the justification concerns the permutation invariance derivation in Eqns. (8)–(9), not the later scaling equation (13).","section":"Section 3.2, after Eq. (9)"},{"comment":"The transformed parameters reuse the same symbols Wup, bup, and Wdown as the original parameters. Using tildes or primes would avoid ambiguity when reading Eqns. (21)–(22).","section":"Section 3.2, Eqns. (10)–(22)"},{"comment":"The main text defines the loss using CE, while Algorithm 1 writes LKL without defining it. Please align the notation and define LKL explicitly.","section":"Section 3.2, Eq. (23) and Algorithm 1"},{"comment":"The hyperparameters α, σs, and σr are said to be obtained by a grid search on the calibration set. Please specify the grid and report sensitivity to these values; otherwise the method appears to have several free parameters tuned on the same objective used for search.","section":"Section 4.1"},{"comment":"The optimization curves in Figure 1 do not indicate whether they are single runs or averaged over seeds, and the reported acceptance ratio appears to be a single-trajectory quantity. Adding seed variance would strengthen the analysis.","section":"Section 4.2, Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The main obstacle is the mismatch between the text's description of activation-guided search and Algorithm 1 as printed. I would ask the editor to require a corrected algorithm and, ideally, code release as part of the revision, because the discrepancy cannot be resolved by inspection and directly affects the validity of Table 4. The paper is otherwise within scope and the idea is worth pursuing; the statistical concerns are secondary but should also be addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper has a genuinely new combination: a joint discrete search over permutation, scaling, and rotation for ultra-low-bit quantization. Permutation invariance is non-differentiable and gradient methods (SpinQuant, DuQuant, RPTQ) don't explore it well, so a hill-climbing search over permutations is a reasonable move. The empirical work is honest in structure: they replicate GPTQ and AWQ, use held-out test sets, and report add-on gains over three strong baselines. The ablations show each transformation contributes, and the gains over GPTQ and AWQ are real. Over OmniQuant the gains are modest but mostly consistent.\n\nThe soft spots are serious. The stress-test is correct: in Algorithm 1, H and H0 are set once, and every candidate is evaluated with L' = LKL(X, quant(theta')) + alpha * MSE(H, H0). Since H and H0 never change, the MSE term is constant, so the search as printed is pure cross-entropy. That contradicts the text and Eq. (23), which clearly intend the activation term to compare the candidate quantized model against the original. Table 4, showing that matching more layers helps, cannot follow from the stated algorithm. No code is released, so I can't tell whether this is a typo in the pseudocode or a deeper issue, but as written the central claim of \"activation-guided discrete search\" is not reproducible.\n\nOther issues are secondary but worth noting. The search is stochastic and no error bars or repeated runs are reported, so we can't assess variance. Rotation is only approximately invariant for ReLU networks; the paper acknowledges this and gives one pilot check, but that's thin support for a core component. Some reasoning tasks regress slightly, which the paper mentions but doesn't fully explore. Minor typos: \"Permuation\" in Table 2 and \"GTPQ\" in Section 4.2.\n\nBottom line: this is a promising add-on for 2-bit PTQ, but the printed algorithm doesn't match the described method. A reader who wants to build on this needs the code or a corrected pseudocode and a clearer characterization of the rotation approximation. I'd bring it to a reading group to discuss the discrete search idea and the discrepancy, and I would not cite it until the algorithm is fixed and code is available.\n\nMy recommendation for peer review: send it out. The core idea deserves referee time, but the reviewers should require a corrected algorithm and code before acceptance.","headline":"The activation-matching term in Algorithm 1 is constant, so the paper's core mechanism doesn't work as printed; the idea is still worth refereeing.","tokens_in":16725,"tokens_out":2365,"would_cite":false,"duration_ms":25907,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes InvarExplore, a discrete hill-climbing search over permutation, scaling, and rotation of a Transformer's weights that improves 2-bit quantization of OPT models from 1.3B to 13B on top of existing quantizers.","keywords":["post-training quantization","ultra-low-bit quantization","model invariance","permutation invariance","discrete hill-climbing search","weight-only quantization","large language models","activation matching"],"falsifier":"Run InvarExplore's rotation search on a pretrained model with GELU or SiLU activations at 2-bit and measure the unquantized cross-entropy before and after accepted moves; if the unquantized loss drifts by more than about 0.01% or the perplexity gain over AWQ disappears, the approximate-rotation premise, and with it the claimed generality of the PSR search, is falsified.","tokens_in":15804,"feed_emoji":"🎲","tokens_out":14236,"duration_ms":113477,"temperature":0.7,"pith_summary":"The paper tries to establish that permutation, scaling, and rotation of a trained Transformer's weights—each leaving the unquantized model's behavior intact or nearly intact—can be explored together to make 2-bit quantization markedly less damaging. It introduces a discrete hill-climbing search for this, since permutation is non-differentiable and previous gradient-based methods cannot easily reach it. The central claim is that the search is an add-on: applied after GPTQ, AWQ, or OmniQuant, it lowers perplexity and raises reasoning accuracy consistently for OPT models from 1.3B to 13B parameters. A sympathetic reader would care because 2-bit quantization is where existing methods degrade sharply, and the method needs only forward passes on a short calibration set.","feed_headline":"Discrete search improves 2-bit LLM accuracy","feed_subtitle":"InvarExplore's discrete search adds perplexity and reasoning gains on top of GPTQ, AWQ, and OmniQuant from 1.3B to 13B.","key_machinery":"The central object is the PSR transform acting on a Transformer feed-forward block $z = W_{\\mathrm{down}} f(W_{\\mathrm{up}} x + b_{\\mathrm{up}}) + b_{\\mathrm{down}}$. Replacing $(W_{\\mathrm{up}}, b_{\\mathrm{up}}, W_{\\mathrm{down}})$ by $(PSRW_{\\mathrm{up}}, PSRb_{\\mathrm{up}}, W_{\\mathrm{down}}R^{\\top}S^{-1}P^{\\top})$ leaves the unquantized output unchanged when the symmetry is exact (permutation; positive scaling for ReLU-type activations) and nearly unchanged for small rotations, which are built as a block-diagonal matrix of 2D rotations parameterized by angles $\\phi_1,\\dots,\\phi_{d/2}$. The search is a hill-climbing loop that at each step shuffles 10% of a sampled layer's neurons and takes Gaussian random-walk steps in scale ($\\sigma_s=10^{-2}$) and rotation angle ($\\sigma_r=10^{-5}$), then accepts the move only if the quantized model's cross-entropy plus $\\alpha$ times the activation MSE on the calibration batch improves. This machinery is what makes permutation searchable at all, and it is what lets the three invariance types reinforce each other.","core_discovery":"On the paper's own terms, the central discovery is that a trained Transformer has many exactly or nearly equivalent parameterizations—permuting neurons, rescaling features, and applying small rotations between the up and down projections of a feed-forward block—and some of these quantize far better than the original weights. Because round-to-nearest quantization depends on the min and max of each weight group, an invariant transform that reduces outlier spread lowers quantization error without changing the unquantized function. InvarExplore's hill-climbing search over the combined PSR transform, guided by cross-entropy plus activation MSE on 512-token sequences, is the means to find those parameterizations; the paper reports, for example, that on the 13B OPT model it cuts AWQ's WikiText-2 perplexity from 35.89 to 26.26 at 2-bit/group-128 and raises average six-task reasoning accuracy from 51.85% to 55.13%. Permutation, although a special case of orthogonal transformation, creates symmetric local optima that gradient-based rotation learners cannot escape, and because permutation is non-differentiable the paper argues a discrete search is required.","pith_inferences":["Beyond the paper, the same PSR search should transfer to leaky ReLU models, since the scaling symmetry is exact for any piecewise-linear activation with a fixed slope; the 0.001% rotation check would need to be rerun there.","A testable extension the paper does not run is wrapping the same forward-only hill-climbing search around non-integer quantizers such as vector quantization; the black-box objective should carry over, but this is conjecture.","An unstated corollary of the paper's Table 4 trend is that activation matching on more layers should keep improving final quantization quality when GPU memory permits, effectively trading search-time memory for accuracy."],"forward_implications":["On WikiText-2 at 2-bit/group-128, applying InvarExplore to AWQ lowers the 13B OPT perplexity from 35.89 to 26.26 and the C4 perplexity from 37.36 to 27.00.","The six-task reasoning average for the same setting rises from 51.85% to 55.13% on the 13B model, and consistent gains appear across OPT 1.3B, 2.7B, 6.7B, and 13B.","The 13B ablation shows each transformation alone beats the AWQ baseline on most tasks, and the combined PSR search beats each single transformation, supporting the synergy claim.","InvarExplore can be run with only forward passes on a short calibration set (32 sequences of 512 tokens), and even with activation matching disabled it still beats AWQ with no extra memory cost."],"supporting_citations":[{"why":"Defines the asymmetric integer group quantization with scale and zero-point that the whole method operates on.","marker":"Jacob et al. (2018)"},{"why":"AWQ baseline; its activation-based scaling is the special case of the framework and the main comparison method.","marker":"Lin et al. (2024b)"},{"why":"OmniQuant baseline; gradient-based scaling method that InvarExplore is applied on top of and compared against.","marker":"Shao et al. (2024)"},{"why":"Learned rotation method showing orthogonal transformations are explored by gradient descent but cannot reach permutations due to symmetric local optima.","marker":"Liu et al. (2024)"},{"why":"GPTQ baseline; sequential quantization method used as one of the base quantizers in the add-on evaluation.","marker":"Frantar et al. (2023)"},{"why":"DuQuant; manually designed zigzag permutation, the closest previous attempt to exploit permutation invariance.","marker":"Lin et al. (2024a)"},{"why":"Supplies the OPT family of models on which all experiments and the rotation-invariance pilot check are run.","marker":"Zhang et al. (2022)"}],"fun_headline_variants":["Discrete search finds better 2-bit LLM weights","InvarExplore: discrete search boosts 2-bit LLM quantization","Permutation search cuts 2-bit LLM perplexity","Synergy of invariants unlocks better 2-bit LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that small rotations are close enough to exact invariance: the paper checks this once, seeing a 0.001% cross-entropy change on a 13B OPT model, and the rotation part of the search—and with it part of the reported synergy—collapses if that approximation fails under quantization or on other architectures.","fun_headline_variants_meta":{"raw":{"variants":["Discrete search finds better 2-bit LLM weights","InvarExplore: discrete search boosts 2-bit LLM quantization","Permutation search cuts 2-bit LLM perplexity","Synergy of invariants unlocks better 2-bit LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000216,"raw_usage":{"total_tokens":1435,"prompt_tokens":951,"completion_tokens":484,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":567,"completion_tokens_details":{"reasoning_tokens":414}},"tokens_in":567,"tokens_out":484,"duration_ms":4744,"temperature":1.0,"reasoning_tokens":414,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T22:30:28.448857+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run InvarExplore's rotation search on a pretrained model with GELU or SiLU activations at 2-bit and measure the unquantized cross-entropy before and after accepted moves; if the unquantized loss drifts by more than about 0.01% or the perplexity gain over AWQ disappears, the approximate-rotation premise, and with it the claimed generality of the PSR search, is falsified.","supporting_citations":[],"review_version":1}