{"id":"f1523f73-617f-45fc-877d-087ff870710a","arxiv_id":"2412.01129","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"RILQ improves 2-bit LLM accuracy by optimizing a model-wise discrepancy loss at the final layer, making LoRA error compensation rank-insensitive.","lead":"This paper introduces RILQ, a method that uses a model-level loss to make low-rank LoRA adapters effectively compensate for accuracy loss in 2-bit quantized large language models. It matters because 2-bit LLMs could run with much smaller memory footprints, and this work shows a practical way to recover much of the lost accuracy.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Rank-insensitivity is attributed to Model-Loss, but the rank-sweep evidence uses RILQ (Model-Loss + GT-Loss); since Table 7 shows GT-Loss alone helps, the causal attribution may be confounded.","rationale":"The paper has genuine strengths: public code, evaluation across two model families and three quantizers, and a clear rank-sweep table. The central claim, however, rests on isolating the mechanism that makes low-rank compensation work at 2-bit precision. Table 5 demonstrates that the full RILQ recipe is rank-insensitive, but the recipe contains both Model-Loss and GT-Loss. Table 7 shows both losses contribute, so the rank-insensitivity could be attributable to the auxiliary objective rather than to the model-wise discrepancy scope. This is more load-bearing than the reader's generalization concern because it attacks the proposed explanation, not just the breadth of evidence. A single additional ablation across ranks with Model-Loss-only versus GT-Loss-only would settle it. If Model-Loss-only is rank-insensitive, the mechanistic story is supported; if not, the method may still be useful but the central claim needs reframing. I would therefore keep the reader's CONDITIONAL verdict, with the additional acceptance condition being this ablation.","tokens_in":18889,"tokens_out":6855,"duration_ms":60282,"concrete_test":"Re-run the Table 5 rank sweep (OmniQuant W2A16, LLaMA-2-7B, ranks 16/32/64/128/256) under four conditions with matched optimization budget and early stopping: (a) SVD/LoftQ, (b) Model-Loss only (Eq. 5), (c) GT-Loss only (Eq. 6), (d) RILQ (Eq. 5 + Eq. 6). Report C4/WikiText-2 perplexity and average CSQA accuracy, plus per-condition sigma across ranks. If (b) has sigma close to (d) and much smaller than (a), Model-Loss alone is rank-insensitive and the central claim holds. If (b) shows rank sensitivity similar to (a) while only (c) and (d) are flat, the paper's mechanistic attribution to Model-Loss fails and must be revised to credit GT-Loss or joint training.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that Model-Loss (Eq. 5) is what makes 2-bit LQEC rank-insensitive. The direct rank-sweep evidence in Table 5 compares 'SVD' (LoftQ) with 'RILQ', but RILQ is defined in 'Rank-Insensitive LQEC' as Model-Loss plus the ground-truth causal LM loss (GT-Loss, Eq. 6). Table 7 shows that GT-Loss alone already improves average CSQA accuracy to 53.28, nearly matching Model-Loss alone (52.77), and that the two losses together give 54.71. However, Table 7 is run at a single default rank and does not report rank sensitivity. Consequently, the flat perplexity across ranks in Table 5 (sigma = 0.07 for RILQ vs 0.69 for SVD at W2A16) could be driven by the auxiliary GT-Loss, or by the much larger optimization budget (up to 10,000 gradient steps with early stopping), rather than by the model-wise discrepancy scope. If GT-Loss alone is rank-insensitive, the mechanistic claim that wider discrepancy scope mitigates high-rank requirements is not supported; the method would still be useful, but for a different reason. The singular-vector analysis in Fig. 4(c) and Fig. 5 is post-hoc and does not isolate this confound.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses why LoRA-based quantization error compensation (LQEC) fails for 2-bit weight quantization and proposes RILQ, a method that trains low-rank adapters by minimizing a model-wise activation discrepancy loss (Model-Loss, Eq. 5) together with a ground-truth causal language modeling loss (GT-Loss, Eq. 6). The authors first present a rank sensitivity analysis showing that the relative error of LM-head outputs grows with rank as discrepancy scope narrows, and that the model-level loss is less rank-sensitive than linear- or layer-level losses. They then evaluate RILQ on LLaMA-2-7B and LLaMA-3-8B with OmniQuant, QuIP#, QuaRot, and LoftQ, reporting consistent perplexity and zero-shot CSQA/GSM8K improvements at 2-bit, plus gains in task-specific fine-tuning and QA-LoRA settings. Ablations examine rank sensitivity, loss scope, GT-Loss contribution, calibration size, and model scale up to 70B.","tokens_in":2029,"tokens_out":2142,"duration_ms":63575,"significance":"If the central claim holds, the paper makes a practically useful contribution: it shows that a simple final-layer output discrepancy objective can make 2-bit LQEC effective with rank-16 adapters, enabling adapter-merged weight-quantized inference without large accuracy loss. The strengths of the paper are its broad experimental coverage (two model families, four quantizers, direct correction and fine-tuning, QA-LoRA integration, model scales up to 70B) and the promised public code. The main weakness is that the causal attribution of rank-insensitivity to Model-Loss is not fully isolated from the auxiliary GT-Loss and from the larger optimization budget. Because RILQ is defined as Model-Loss plus GT-Loss, and Table 7 shows GT-Loss alone already improves accuracy, the rank-sweep evidence in Table 5 does not by itself establish the proposed mechanism. This underdetermination limits, but does not destroy, the significance of the empirical gains; the issue is addressable with targeted ablations.","major_comments":[{"comment":"The central claim is that Model-Loss mitigates the high-rank requirements of 2-bit quantization error and that this is what makes RILQ rank-insensitive. However, the rank-sweep evidence in Table 5 (sigma = 0.07 for RILQ vs 0.69 for SVD at W2A16) is obtained with RILQ, which is the combination of Model-Loss and GT-Loss. Table 7 shows that GT-Loss alone reaches 53.28 average CSQA accuracy, nearly matching Model-Loss alone (52.77) at a single rank, but Table 7 does not report a rank sweep. Consequently, the observed flatness across ranks could in principle be driven by GT-Loss, by the interaction of the two losses, or by the substantially larger optimization budget (up to 10,000 Adam steps with early stopping) rather than by the wider discrepancy scope per se. Please provide rank sweeps (e.g., ranks 16, 32, 64, 128, 256) for Model-Loss alone, GT-Loss alone, Model-Loss+GT-Loss, and the SVD baseline, with the same optimization budget and early-stopping criterion, on at least LLaMA-2-7B with OmniQuant and one additional quantizer. In addition, state explicitly which objective was used to produce Fig. 4(a) and Fig. 4(b); if those figures use Model-Loss alone, say so and reconcile with Table 5, which uses RILQ.","section":"Sec. 'Rank-Insensitive LQEC'; Eqs. (5)-(6); Tables 5 and 7"},{"comment":"The rank sensitivity analysis that motivates the central claim is performed on a single model (LLaMA-2-7B) with a single quantizer (OmniQuant for Fig. 4). The paper's broader conclusion, that Model-Loss is rank-insensitive for 2-bit LQEC, is used to justify the method across LLaMA-3-8B, QuIP#, QuaRot, and LoftQ. Given that Table 1 shows very different error magnitudes across models and quantizers (e.g., LoftQ W2A16 perplexity is 56168 on LLaMA-3-8B vs 1078 on LLaMA-2-7B), the rank-sensitivity phenomenon could be model- or quantizer-specific. Please replicate the rank sensitivity comparison (relative error E vs rank) for LLaMA-3-8B and for at least one additional 2-bit quantizer (e.g., QuIP# or QuaRot), or explicitly qualify the generalizability of the proposed mechanism.","section":"Sec. 'Rank Sensitivity Analysis'; Fig. 4"},{"comment":"All reported accuracy and perplexity numbers appear to come from single runs without error bars or multiple seeds. RILQ involves stochastic optimization (Adam, random calibration sampling of 256 C4 sentences, early stopping), and several reported differences are small in absolute terms: for example, in Table 7 the difference between GT-Loss alone (53.28) and Model-Loss alone (52.77) is 0.51 points, and the difference between Model-Loss alone and RILQ (54.71) is 1.94 points. Without variance estimates, it is not possible to tell which of these differences are significant. Please report mean and standard deviation over at least three seeds for the central tables (in particular Table 1, Table 7, and the rank sweep in Table 5), or provide a justification for why run-to-run variance is negligible for this setup.","section":"Tables 1, 7, and 9"}],"minor_comments":[{"comment":"The relative error metric E = |(Y - Y^q)/Y| is not defined precisely; please specify the norm, the averaging (over tokens, batch, and hidden dimensions), and the exact computation procedure for the values plotted in Fig. 4(a).","section":"Sec. 'Rank Sensitivity Analysis'"},{"comment":"The caption of Fig. 3(a) does not state the model or quantizer used; the text mentions LLaMA-2-7B, but the figure should include this information for self-containedness.","section":"Fig. 3(a) caption"},{"comment":"The layout of Table 3 is confusing: it is unclear whether 'RILQ Error Compensation' and 'Fine-Tuning' are separate rows or columns, and the text does not explain how the W2A16 OmniQuant baseline (47.42 CSQA) differs from the Table 1 OmniQuant result (51.88). Please clarify the experimental protocol and the relationship between the two tables.","section":"Table 3 and surrounding text"},{"comment":"There is a citation inconsistency for LoftQ: it is attributed to (Guo et al. 2024) in one place in the Related Work section and to (Li et al. 2024) elsewhere; please ensure LoftQ is consistently cited as Li et al. 2024 and LQ-LoRA as Guo et al. 2024.","section":"Related Work and References"},{"comment":"The appendix step 3 states that LoRA is initialized 'using gradient descent on Model-Loss (Eq. 5) and GT-Loss (Eq. 6)', which means RILQ always uses both losses; the terminology in the main text sometimes refers to RILQ and sometimes to Model-Loss alone, causing ambiguity. Please introduce distinct terms (e.g., 'Model-Loss-only' vs 'RILQ') and use them consistently.","section":"Appendix, 'Procedure of RILQ'"},{"comment":"The comparison to Apple's accuracy-recovery adapter (Gunter et al. 2024) with '200x sample efficiency' is not substantiated with a precise token count or a direct comparison under matched conditions; please either add the necessary details or soften the claim.","section":"Appendix, 'Memory Cost Analysis'"}],"recommendation":"major_revision","confidential_remarks":"The paper reports promising and reproducible-looking gains, and the code release is a plus. The main risk is the confounded attribution of rank-insensitivity: the authors need to show that Model-Loss alone, or the combination specifically, is what flattens the rank curve, not GT-Loss alone. The requested rank-sweep ablation is the key experiment. If the authors can provide it and replicate the phenomenon on a second model/quantizer, the paper would be suitable for acceptance. Also note that the paper's claim to be the first to investigate sub-4-bit LQEC limitations is somewhat overstated given RA-LoRA's rank-demand analysis, but this is not a blocking issue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version. RILQ is a genuinely useful empirical contribution: a model-level output discrepancy loss combined with a causal LM loss for LoRA-based 2-bit QEC, evaluated broadly and consistently improving accuracy across OmniQuant, QuIP#, QuaRot, and LoftQ on LLaMA-2-7B and LLaMA-3-8B. The gains are large and practical—QuIP# on LLaMA-3 jumps from 52.48 to 60.61 average CSQA accuracy, for example. The method is simple, reproducible, and the code is public. That part deserves credit.\n\nThe soft spot is the headline explanation. The stress-test concern lands. The paper claims Model-Loss is what makes LQEC rank-insensitive, but the rank-sweep evidence in Table 5 compares SVD against RILQ, which is Model-Loss plus GT-Loss. Table 7 shows GT-Loss alone nearly matches Model-Loss alone at the default rank (53.28 vs. 52.77), and there is no rank sweep for either loss in isolation. So the flat perplexity across ranks in Table 5 could be driven by the language modeling objective, or by the large optimization budget, rather than by the model-wise discrepancy scope. The singular-vector analysis in Fig. 4(c) and Fig. 5 is suggestive but post-hoc and does not resolve the confound. This does not break the method's practical value—RILQ works—but it weakens the mechanistic claim that the paper is built around.\n\nOther limitations are real but secondary. There are no error bars or multiple seeds, the rank-sensitivity analysis is on one model, and the calibration set is 256 C4 sentences. The \"no prior investigation\" framing is also a bit strong given BRECQ's granularity analysis, though the paper does engage with BRECQ and the low-rank angle is new. None of these are fatal; they are standard things a referee should ask for.\n\nWho gets value: anyone working on low-bit LLM quantization, LoRA-based error compensation, or deployment of 2-bit models. It deserves a serious referee. The empirical recipe is solid enough to publish, but the authors should be asked to add rank sweeps for Model-Loss alone and GT-Loss alone, report variance across seeds, and soften the causal attribution unless the confound is resolved.","headline":"RILQ is a useful empirical recipe for 2-bit LoRA-based quantization error compensation, but the paper's central rank-insensitivity story is not actually isolated in the experiments.","tokens_in":19745,"tokens_out":3097,"would_cite":true,"duration_ms":30921,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that 2-bit quantization errors, although high-rank, can be offset by small LoRA adapters if the adapters are trained against the whole model's output rather than layer-by-layer.","keywords":["2-bit quantization","low-rank adaptation","LoRA","quantization error compensation","large language models","model-wise discrepancy loss","rank sensitivity","LLM inference"],"falsifier":"Quantize a held-out model not used in calibration, such as Mistral-7B, to 2-bit, train adapters with Model-Loss at rank 16 and rank 256, and compare the LM-head relative error; if the rank-16 error is substantially worse than rank-256 (comparable to the SVD spread in Fig. 4(a)), the central claim is wrong.","tokens_in":18719,"feed_emoji":"🧠","tokens_out":7327,"duration_ms":57865,"temperature":0.7,"pith_summary":"The paper is trying to establish why LoRA-based quantization error compensation (LQEC) fails at 2-bit weight precision and to fix that failure. Its rank analysis shows that 2-bit quantization errors are high-rank: matching them with a low-rank adapter at a single linear module or even a single Transformer layer requires impractically large ranks. The proposed method, RILQ (Rank-Insensitive LoRA-based Quantization Error Compensation), instead trains all adapters jointly against a model-wise discrepancy loss measured at the output of the final Transformer layer, which the authors find is rank-insensitive—small-rank adapters recover a large share of the accuracy lost by 2-bit quantization. They demonstrate consistent accuracy and perplexity improvements on LLaMA-2-7B and LLaMA-3-8B across several quantizers, with no extra inference cost after the adapters are merged. If correct, the result makes aggressive 2-bit compression a more practical option for serving LLMs.","feed_headline":"A model-wide loss makes 2-bit LLMs accurate again","feed_subtitle":"RILQ shows LoRA adapters can fix 2-bit quantization error when trained on the model's final output instead of layer by layer.","key_machinery":"The central object is the model-wise activation discrepancy loss, called Model-Loss, defined as $\\|Y_N - Y_N^q\\|_F$, the Frobenius norm between the output activations of the final Transformer layer of the full-precision model and the quantized model with LoRA adapters. This loss performs the argument's load-bearing work: because it is applied after all Transformer layers, errors in individual layers are free to drift as long as the final output aligns, which makes the loss rank-insensitive and lets adapters with rank as low as 16 compensate errors created by 2-bit weights. The paper measures rank sensitivity through the relative error $E = |(Y - Y^q)/Y|$ at the LM-head output and supports the cooperative-compensation explanation by showing that Model-Loss raises the singular-vector magnitudes of rank-critical FFN1 modules relative to Q-Proj.","core_discovery":"The paper's central claim is that the rank requirement of LQEC is a property of the discrepancy scope, not of the quantization error alone. When the discrepancy loss is defined at a single linear module (Linear-Loss) or a single Transformer layer (Layer-Loss), 2-bit quantization error demands high-rank adapters, contradicting LoRA's low-rank premise. When the loss is defined at the output activation of the final Transformer layer (Model-Loss, $\\|Y_N - Y_N^q\\|_F$), the relative error at the LM-head stays low even at rank 16, because internal activation drift is allowed to happen freely and the adapters cooperate across layers to align the final output. The paper interprets this as balancing rank-critical modules like FFN1 against rank-redundant modules like Q-Proj, and combines Model-Loss with a causal language modeling loss (GT-Loss) for further gains.","pith_inferences":["The output-level compensation principle could generalize to other high-rank structured perturbations beyond 2-bit weights, such as aggressive pruning or extreme activation quantization, where a model-wide loss might again remove the need for high-rank corrections.","The internal-drift observation suggests a testable design principle for low-rank error compensation: allow intermediate feature mismatch as long as the final output is aligned, which is the opposite of layer-wise distillation objectives.","If the effect is a general property of transformer error propagation, then the optimal loss scope may scale with model depth, implying that larger models need even broader compensation scopes—this could be checked by measuring rank sensitivity across 7B, 13B, and 70B models."],"forward_implications":["RILQ makes low-rank LQEC practical at 2-bit precision, so serving a 2-bit quantized LLM can regain most of the accuracy lost to quantization while keeping the memory savings of 2-bit weights.","Because RILQ works as initialization, it can replace SVD-based initialization in LoRA fine-tuning pipelines, improving downstream task accuracy for models that are already quantized to 2 bits.","The rank-insensitivity result implies that rank selection for LQEC becomes far less critical: rank 16 nearly matches rank 256, removing the need for per-module rank allocation.","The gains transfer across different quantizers (OmniQuant, QuIP#, QuaRot) and both LLaMA-2 and LLaMA-3, indicating the effect depends on the loss scope rather than on a specific quantization scheme.","Since the adapters can be merged into quantized weights, RILQ adds no inference-time overhead relative to adapter-less quantized inference, including in the QA-LoRA setting."],"supporting_citations":[{"why":"Introduces LoRA, the low-rank adapter mechanism that RILQ tunes for quantization error compensation.","marker":"Hu et al. 2022"},{"why":"LoftQ, the Weight-SVD LQEC baseline whose high-rank failure at 2-bit motivates the paper's rank sensitivity analysis.","marker":"Li et al. 2024"},{"why":"OmniQuant, the quantizer used in the rank sensitivity analysis and the main calibration-based experiments.","marker":"Shao et al. 2024"},{"why":"QuIP#, the advanced 2-bit quantizer that shows the largest accuracy recovery with RILQ.","marker":"Tseng et al. 2024"},{"why":"QuaRot, the rotation-based quantizer evaluated with RILQ and used with GPTQ in weight-only experiments.","marker":"Ashkboos et al. 2024b"},{"why":"ApiQ, the per-linear-module loss method whose scope RILQ extends to the whole model.","marker":"Liao and Monz 2024"},{"why":"QLLM, the layer-wise loss predecessor that defines the discrepancy scope ladder the paper climbs.","marker":"Liu et al. 2024"},{"why":"RA-LoRA, which identified that linear modules differ in rank demand and is used to interpret why Model-Loss balances rank-critical and rank-redundant modules.","marker":"Kim et al. 2024"},{"why":"TSLD, the full-parameter QAT method whose model-wise objective and flexible signal propagation observation RILQ adapts to low-rank compensation.","marker":"Kim et al. 2023a"},{"why":"QLoRA, which supplies the NormalFloat 2-bit quantization used in LoftQ and the memory-equivalence baseline for RILQ fine-tuning.","marker":"Dettmers et al. 2023"}],"fun_headline_variants":["Model-wide loss fixes 2-bit LLMs with low-rank adapters","2-bit LLMs get a boost from loss on final output","RILQ: Rank-insensitive LoRA compensation for 2-bit LLMs","Why LoRA fails at 2-bit and how RILQ solves it","Final-output loss makes LoRA work for 2-bit quantization"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole method depends on the assumption that what the paper observes on LLaMA-2-7B with one calibration setup—that the model-level loss keeps working at rank 16—also holds on other models and quantizers, and that low output error really means better answers.","fun_headline_variants_meta":{"raw":{"variants":["Model-wide loss fixes 2-bit LLMs with low-rank adapters","2-bit LLMs get a boost from loss on final output","RILQ: Rank-insensitive LoRA compensation for 2-bit LLMs","Why LoRA fails at 2-bit and how RILQ solves it","Final-output loss makes LoRA work for 2-bit quantization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000213,"raw_usage":{"total_tokens":1433,"prompt_tokens":967,"completion_tokens":466,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":583,"completion_tokens_details":{"reasoning_tokens":371}},"tokens_in":583,"tokens_out":466,"duration_ms":4527,"temperature":1.0,"reasoning_tokens":371,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:39:21.903306+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Quantize a held-out model not used in calibration, such as Mistral-7B, to 2-bit, train adapters with Model-Loss at rank 16 and rank 256, and compare the LM-head relative error; if the rank-16 error is substantially worse than rank-256 (comparable to the SVD spread in Fig. 4(a)), the central claim is wrong.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"OmniQuant, the quantizer used in the rank sensitivity analysis and the main calibration-based experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"QuIP#, the advanced 2-bit quantizer that shows the largest accuracy recovery with RILQ."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ApiQ, the per-linear-module loss method whose scope RILQ extends to the whole model."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"RA-LoRA, which identified that linear modules differ in rank demand and is used to interpret why Model-Loss balances rank-critical and rank-redundant modules."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"QLoRA, which supplies the NormalFloat 2-bit quantization used in LoftQ and the memory-equivalence baseline for RILQ fine-tuning."}],"review_version":1}