Pith. sign in

REVIEW 3 major objections 6 minor 34 references

CLoQ: Enhancing Fine-Tuning of Quantized LLMs via Calibrated LoRA Initialization

T0 review · 3 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CLoQ uses calibration activations to set LoRA adapters, beating QLoRA and LoftQ on quantized Llama and Mistral models.

desk verdict Solid two-SVD theory and real INT2 gains, but the missing MagR ablation leaves the headline empirical claim under-supported. read the letter →

arxiv 2501.18475 v2 pith:GZ7HXQJJ submitted 2025-01-30 cs.LG cs.AI

Yanxia Deng , Aozhong Zhang , Selcuk Gurses , Naigang Wang , Zi Yang , Penghang Yin This is my paper · ORCID
classification cs.LGcs.AI
keywords quantizedLLMfine-tuningLoRAinitializationlow-rankadaptationpost-trainingquantizationcalibrationdataclosed-formSVD2-bitOPTQ
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

When a large language model is quantized to low precision, its weights no longer match the original, and the standard LoRA trick of starting adapters at zero leaves that mismatch uncorrected. CLoQ fixes the gap at initialization: using a small calibration set, it quantizes each layer and then solves for adapter matrices A and B that minimize the weighted discrepancy between the original and quantized-plus-adapter layer on the calibration activations. The paper proves that this problem has a closed-form solution via two singular value decompositions, needing no back-propagation. Empirically, the method consistently improves fine-tuning of 2- and 3-bit Llama and Mistral models over QLoRA, LoftQ, and ApiQ on language modelling and reasoning benchmarks, with the largest gains at 2-bit.

What carries the argument

The load-bearing object is the activation-weighted low-rank approximation problem min_{A,B} ||X(AB^T - ΔW)||$_F^{2}$, and its reduction through the non-symmetric square root R = $Σ_H^{{1/2}}$ U_H^T of H = X^T X. Because ||X(AB^T - ΔW)||$_F^{2}$ = ||R(AB^T - ΔW)||$_F^{2}$, the problem becomes ordinary truncated SVD of RΔW; two SVDs (one for H, one for RΔW) give the closed-form optimal product AB^T. The paper's particular choice of split A=$R^{{-1}}$U_{:r}Σ_{:r}, B=V_{:r} is one of infinitely many optimal factorizations and is selected empirically.

What would settle it

Run CLoQ on an untested model family (for example a 1B-3B model or a different architecture) with the default split and the two alternative optimal splits from Table 7; if the default split is not consistently the best, the central empirical claim loses support. A more direct check: replace the default split with ($R^{{-1}}$U_{:r}, V_{:r}Σ_{:r}) under identical fine-tuning hyperparameters; the paper already reports this collapses 2-bit WikiText perplexity from 6.51 to 880.6, demonstrating that the product-level optimality alone does not determine fine-tuning success.

Watch

Extended reading notes

Core claim

CLoQ's central claim is Theorem 3.1: for a full-rank activation Gram matrix H = X^T X with R = $Σ_H^{{1/2}}$ U_H^T, every pair (A,B) satisfying AB^T = $R^{{-1}}$ L_r(R ΔW) — where L_r is the best rank-r approximation of R ΔW — is optimal for min_{A,B} ||X(AB^T - ΔW)||$_F^{2}$. This reduces a linearly-transformed low-rank problem to an ordinary SVD truncation of RΔW, computed with two SVDs and no gradient. The paper then fixes the factorization A = $R^{{-1}}$U_{:r}Σ_{:r}, B = V_{:r} and, wrapping the quantization step with MagR-preprocessed OPTQ, reports consistent gains over existing LoRA-style fine-tuning of quantized LLMs, most clearly at 2-bit.

Load-bearing premise

The paper assumes that the particular factorization A = $R^{{-1}}$U_{:r}Σ_{:r}, B = V_{:r} — one of infinitely many optimal splits — will continue to train well across settings, even though the theorem only certifies the product AB^T and other equally optimal splits fail badly in the paper's own ablation.

Editorial extensions

If this is right

  • Fine-tuning of 2-bit quantized Llama2-13B with CLoQ reaches higher arithmetic-reasoning accuracy than 4-bit QLoRA, suggesting ultra-low-bit models can be practical after a calibration-based initialization.
  • Because CLoQ needs only two SVDs and no back-propagation, it makes high-quality LoRA initialization cheap enough for memory-limited settings (about 0.7 hours and 9 GB for Llama2-7B).
  • The closed-form result gives a general recipe for any quantization scheme: once Q is fixed, the optimal adapter product under a data metric is obtained by transforming the residual with R and truncating.
  • The performance gap over LoftQ widens as bit-width drops, indicating the calibration-weighted objective matters most when quantization error is large.
  • If the split choice holds up, adapter factorization itself (not just the product) becomes a meaningful design choice for fine-tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The theorem's optimality covers only the product AB^T; the paper does not explain why the chosen split trains better. Table 7 shows the equally optimal split (R^{-1}U_{:r}, V_{:r}Σ_{:r}) collapses at 2-bit (880.6 perplexity), so the empirical gains likely hinge on the interaction between the split's scale distribution and the optimizer, a question the authors leave open.
  • If the split's role is about gradient scale, then CLoQ's gains might be partially recoverable by rescaling any optimal product with a diagonal preconditioner—an inexpensive testable modification.
  • The method's reliance on calibration activations suggests it could generalize to other PEFT variants (e.g., DoRA or weight-decomposed adapters), but that extension is not tested in the paper.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces CLoQ, a LoRA initialization scheme for fine-tuning quantized LLMs. The method first quantizes the base model with OPTQ after MagR preprocessing and then computes low-rank adapters A and B by solving a layer-wise, activation-weighted reconstruction problem, min_{A,B} ||X(AB^T - (W-Q))||_F^2. Theorem 3.1 gives a closed-form characterization of all optimal products AB^T via two SVDs, and the authors select a specific factorization (A = R^{-1}U_{:r}Sigma_{:r}, B = V_{:r}) that performs best empirically. Experiments on Llama2-7B/13B, Llama3-8B, and Mistral-7B across WikiText, GSM8K, multi-task arithmetic reasoning, and commonsense reasoning report consistent improvements over baselines such as LoftQ and ApiQ, particularly at 2-bit weights. The paper also includes an ablation of different (A,B) splits, a calibration-size study, a sequence-length study, and initialization cost/memory comparisons.

Significance. The core mathematical result is clean and correct: Theorem 3.1 reduces the activation-weighted low-rank approximation to an unweighted SVD after a non-symmetric root transformation, and the proof is straightforward and valid under the stated full-rank assumption. The method is backpropagation-free, is computationally cheap (two SVDs per layer), and the paper provides code and detailed derivations, which are clear strengths. If the empirical gains over strong baselines such as ApiQ-bw are reproducible and attributable to the proposed initialization, CLoQ would be a useful practical contribution to low-bit fine-tuning. However, the current manuscript does not fully establish that attribution: the CLoQ pipeline bundles MagR preprocessing with the adapter initialization, and the headline INT2 improvements over ApiQ-bw are small (e.g., 0.1 perplexity or 0.2 accuracy points). The split-sensitivity result in Table 7 also indicates that the theory certifies only the product AB^T, not the trainability of the chosen factorization, which the paper explicitly defers to future work.

major comments (3)
  1. [Section 3.1.1, Section 4.1, Algorithm 1, Tables 1-5] The empirical central claim is that CLoQ's calibrated LoRA initialization improves fine-tuning of quantized LLMs. However, CLoQ differs from every baseline in two bundled ingredients: the MagR preprocessing applied before OPTQ and the calibrated (A,B) initialization. No experiment varies one ingredient while holding the other fixed. Since Section 3.1.1 states that MagR 'significantly improves OPTQ's effectiveness in the low-bit regime,' and since the INT2 gains over ApiQ-bw are small (e.g., Llama2-7B WikiText 6.51 vs. 6.61, GSM8K 33.7 vs. 33.5; Table 3 average 49.2 vs. 47.3), the observed gains could plausibly be due to MagR alone. I recommend adding an ablation that applies a baseline LoRA initialization (e.g., random A and zero B, or LoftQ's initialization) on top of MagR+OPTQ, and CLoQ's initialization on OPTQ without MagR, so the contribution of each component is isolated.
  2. [Section 3.1.2, Table 7] Theorem 3.1 characterizes all optimal products AB^T, but fine-tuning performance depends on the particular factorization, not just the product. The chosen split (R^{-1}U_{:r}Sigma_{:r}, V_{:r}) is selected empirically, and Table 7 shows that another equally optimal split, (R^{-1}U_{:r}, V_{:r}Sigma_{:r}), collapses at INT2 (WikiText perplexity 880.6, GSM8K accuracy 1.6). This means the closed-form optimality result does not by itself explain why CLoQ performs well; the method's success is contingent on a split that is not theoretically justified. The paper acknowledges this as future work, but for the central claim that CLoQ provides an optimal initialization for fine-tuning, the split dependence needs either a theoretical analysis of trainability or substantially more empirical evidence that the chosen split generalizes across models, tasks, and bit widths.
  3. [Section 4.2, Tables 1-5] Most main results are reported without error bars or multiple seeds, while several differences from strong baselines are very small. For example, Table 1 shows Llama2-7B at INT2 with WikiText 6.51 vs. 6.61 and GSM8K 33.7 vs. 33.5 for CLoQ vs. ApiQ-bw; the arithmetic reasoning improvement in Table 3 is about 1.9 points on average. Table 4 reports standard deviations for CLoQ only, not for the baselines. To substantiate the claim that CLoQ 'consistently outperforms' existing methods, the key INT2 and INT3 comparisons should be run with multiple seeds and reported with means and standard deviations, at least for the strongest baseline in each setting.
minor comments (6)
  1. [Section 2] There is a typo in the quantizer paragraph: 'respecitively' should be 'respectively.'
  2. [Section 4.2, Arithmetic reasoning (single task)] The sentence 'At INT2, CLoQ reaches an accuracy of 33.7% on Llama2-7B, which CLoQ achieves a 7.7% improvement in performance compared with ApiQ-lw' is grammatically broken and should be rephrased.
  3. [Tables 3 and 4] The column headers 'SV AMP' and 'MA WPS' contain spurious spacing; they should be 'SVAMP' and 'MAWPS.'
  4. [Appendix A, Tables 11-12] The hyperparameter tables cover only Llama2-7B and Llama2-13B, but Tables 2 and 4 report results for Llama3-8B and Mistral-7B; please provide the corresponding hyperparameters for those models to make the experiments reproducible.
  5. [Section 3.1] The paper states that a single alternating-minimization iteration 'suffices' based on observation, but no experiment compares one iteration with multiple iterations. Adding such an ablation would address whether the two-stage procedure is indeed sufficient.
  6. [Section 5, Baselines] LQ-LoRA is discussed in Related Work but is not included in the experimental comparisons. If it is omitted for practical reasons, this should be stated explicitly.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: Theorem 3.1 is a self-contained weighted low-rank approximation result, and the empirical claims rest on external benchmarks; the only circularity-adjacent element is a minor, non-load-bearing self-citation of MagR.

full rationale

The load-bearing theoretical claim is Theorem 3.1, which derives the closed-form solution to min_{A,B} ||X(AB^T - Delta W)||_F^2 by rewriting the objective as ||R(AB^T - Delta W)||_F^2 with H = X^T X = R^T R and then applying the Eckart-Young theorem. This is a parameter-free algebraic equivalence, not an assumption of the conclusion, so it is not self-definitional. The specific factorization A = R^{-1}U_{:r}Sigma_{:r}, B = V_{:r} is not certified by the theorem; Section 4.3 (Table 7) shows other equally optimal splits fail, and the paper explicitly defers a theory of the split to future work, so the theorem is not being used to justify the trainable split. The empirical evaluation is against external baselines (LoftQ, ApiQ, QLoRA) on held-out tasks. Two circularity-adjacent concerns remain but do not rise to circularity: (i) MagR (Zhang et al., 2024a) is prior work by overlapping authors and is bundled into every CLoQ run without an ablation isolating it, which weakens causal attribution of the INT2 gains; and (ii) the (A,B) split is selected by downstream performance on the same benchmarks. Both are empirical attribution issues, not reductions by construction, so no circular step meets the evidentiary bar.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical or architectural entities. It relies on standard hyperparameters, a regularization constant, and three load-bearing assumptions: layer-wise calibrated reconstruction is the right objective, one alternating step suffices, and the empirically chosen factorization of the optimal product remains trainable during fine-tuning.

free parameters (4)
  • lambda_reg = 0.01 * Tr(H) / m
    Added to the Gram matrix in Algorithm 1 to ensure invertibility; follows prior PTQ practice but means the solved objective is regularized rather than exactly equation (4).
  • lora_rank = 64
    Fixed across all methods and experiments; the method's performance at this rank may not transfer to other ranks.
  • calibration_size = 128
    Default number of WikiText-2 samples; Table 8 shows robustness from 32 to 256, so this is not a sensitive fitted constant.
  • quant_group_size = 64
    Group size for INT quantization used in all CLoQ experiments; hyperparameter, not derived.
assumptions (4)
  • domain assumption Layer-wise output reconstruction on calibration data is a valid proxy for downstream fine-tuning quality.
    The whole method optimizes ||X(Q+AB^T-W)||_F^2; if minimizing this discrepancy does not help fine-tuning, the closed-form optimum is irrelevant. Section 3.
  • ad hoc to paper One alternating step (quantize, then fit AB) suffices.
    The paper states it suffices to just perform a single iteration without proof; the joint problem (2) may have a better solution. Section 3.1.
  • ad hoc to paper The factorization (R^{-1}U Sigma, V) is trainable.
    Theorem 3.1 only characterizes AB^T; Table 7 shows other optimal factorizations fail at INT2, so this split is load-bearing but empirically chosen. Section 3.1.2.
  • domain assumption Regularized H approximates the unregularized objective.
    Algorithm 1 uses H+lambda I; the theorem assumes H full-rank. The paper gives no analysis of how lambda changes the optimum. Section 3.1.2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CLoQ: Enhancing Fine-Tuning of Quantized LLMs via Calibrated LoRA Initialization." pith.science (2026). https://pith.science/paper/GZ7HXQJJ

@misc{pith2026250118475,
  author       = {Pith},
  title        = {Pith review of: CLoQ: Enhancing Fine-Tuning of Quantized LLMs via Calibrated LoRA Initialization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GZ7HXQJJ}},
  note         = {Machine review of arXiv:2501.18475}
}
read the original abstract

Fine-tuning large language models (LLMs) using low-rank adaptation (LoRA) has become a highly efficient approach for downstream tasks, particularly in scenarios with limited computational resources. However, applying LoRA techniques to quantized LLMs poses unique challenges due to the reduced representational precision of quantized weights. In this paper, we introduce CLoQ (Calibrated LoRA initialization for Quantized LLMs), a simplistic initialization strategy designed to overcome these challenges. Our approach focuses on minimizing the layer-wise discrepancy between the original LLM and its quantized counterpart with LoRA components during initialization. By leveraging a small calibration dataset, CLoQ quantizes a pre-trained LLM and determines the optimal LoRA components for each layer, ensuring a strong foundation for subsequent fine-tuning. A key contribution of this work is a novel theoretical result that enables the accurate and closed-form construction of these optimal LoRA components. We validate the efficacy of CLoQ across multiple tasks such as language generation, arithmetic reasoning, and commonsense reasoning, demonstrating that it consistently outperforms existing LoRA fine-tuning methods for quantized LLMs, especially at ultra low-bit widths.

Figures

Figures reproduced from arXiv: 2501.18475 by the authors.

Figure 1
Figure 1. Fine-tuning results of Llama2-7B and Llama2-13B across various tasks. Left: the perplexity [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The discrepancy ∥X(Q + ABT −W)∥ between the LoRA initialization and the original pre-trained weight matrix, computed using the spectral norm and the Frobenius norm, respectively. The layer shown in the figures above is randomly selected from the Llama2-7B model. The initialization is derived using both CLoQ and LoftQ under INT2 quantization. Notably, CLoQ significantly reduces this discrepancy, demonstrating its eff… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

34 extracted references · 4 canonical work pages

  1. [2]

    Quantease: Optimization-based quantization for language models–an efficient and intuitive algorithm.arXiv preprint arXiv:2309.01885,

    Kayhan Behdin, Ayan Acharya, Aman Gupta, Sathiya Keerthi, and Rahul Mazumder. Quantease: Optimization-based quantization for language models–an efficient and intuitive algorithm.arXiv preprint arXiv:2309.01885,

    arXiv

  2. [4]

    Pact: Parameterized clipping activation for quantized neural networks.arXiv preprint arXiv:1805.06085,

    Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks.arXiv preprint arXiv:1805.06085,

    arXiv

  3. [7]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168,

    arXiv

  4. [11]

    Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685,

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685,

    arXiv

  5. [12]

    Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models

    Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Ka-Wei Lee. Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models. arXiv preprint arXiv:2304.01933,

    arXiv

  6. [13]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825,

    arXiv

  7. [14]

    Mawps: A math word problem repository

    Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. Mawps: A math word problem repository. InProceedings of the 2016 conference of the north american chapter of the association for computational linguistics: human language technologies, pp. 1152–1157,

    2016

  8. [16]

    Loftq: Lora-fine-tuning-aware quantization for large language models.arXiv preprint arXiv:2310.08659, 2023a

    Yixiao Li, Yifan Yu, Chen Liang, Pengcheng He, Nikos Karampatziakis, Weizhu Chen, and Tuo Zhao. Loftq: Lora-fine-tuning-aware quantization for large language models.arXiv preprint arXiv:2310.08659, 2023a. Zhijian Li, Biao Yang, Penghang Yin, Yingyong Qi, and Jack Xin. Feature affinity assisted knowledge distillation and quantization of deep neural network...

    arXiv

Show all 34 references
  1. [17]

    Program induction by rationale generation: Learning to solve and explain algebraic word problems.arXiv preprint arXiv:1705.04146,

    Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale generation: Learning to solve and explain algebraic word problems.arXiv preprint arXiv:1705.04146,

    arXiv

  2. [19]

    Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101,

    I Loshchilov. Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101,

    arXiv

  3. [20]

    Pointer sentinel mixture models

    14 Published in Transactions on Machine Learning Research (08/2025) Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843,

    2025 arXiv

  4. [21]

    Can a suit of armor conduct electricity? a new dataset for open book question answering.arXiv preprint arXiv:1809.02789,

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering.arXiv preprint arXiv:1809.02789,

    arXiv

  5. [22]

    Are nlp models really able to solve simple math word problems? arXiv preprint arXiv:2103.07191,

    Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are nlp models really able to solve simple math word problems? arXiv preprint arXiv:2103.07191,

    arXiv

  6. [23]

    Socialiqa: Commonsense reasoning about social interactions.arXiv preprint arXiv:1904.09728,

    Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsense reasoning about social interactions.arXiv preprint arXiv:1904.09728,

    1904 arXiv

  7. [24]

    Omniquant: Omnidirectionally calibrated quantization for large language models

    Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models. arXiv preprint arXiv:2308.13137,

    arXiv

  8. [25]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288,

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288,

    arXiv

  9. [26]

    Roselora: Row and column-wise sparse low-rank adaptation of pre-trained language model for knowledge editing and fine-tuning.arXiv preprint arXiv:2406.10777, 2024a

    Haoyu Wang, Tianci Liu, Ruirui Li, Monica Cheng, Tuo Zhao, and Jing Gao. Roselora: Row and column-wise sparse low-rank adaptation of pre-trained language model for knowledge editing and fine-tuning.arXiv preprint arXiv:2406.10777, 2024a. Naigang Wang, Jungwook Choi, Daniel Bra...

    arXiv

  10. [27]

    Lora-ga: Low-rank adaptation with gradient approximation.arXiv preprint arXiv:2407.05000, 2024b

    Shaowen Wang, Linxi Yu, and Jian Li. Lora-ga: Low-rank adaptation with gradient approximation.arXiv preprint arXiv:2407.05000, 2024b. Zhengbo Wang, Jian Liang, Ran He, Zilei Wang, and Tieniu Tan. Lora-pro: Are low-rank adapters properly optimized? arXiv preprint arXiv:2407.182...

    arXiv

  11. [28]

    Zeroquant-v2: Exploring post-training quantization in llms from comprehensive study to low rank compensation.arXiv preprint arXiv:2303.08302,

    Zhewei Yao, Xiaoxia Wu, Cheng Li, Stephen Youn, and Yuxiong He. Zeroquant-v2: Exploring post-training quantization in llms from comprehensive study to low rank compensation.arXiv preprint arXiv:2303.08302,

    arXiv

  12. [29]

    Quantization and training of low bit-width convolutional neural networks for object detection.arXiv preprint arXiv:1612.06052,

    Penghang Yin, Shuai Zhang, Yingyong Qi, and Jack Xin. Quantization and training of low bit-width convolutional neural networks for object detection.arXiv preprint arXiv:1612.06052,

    arXiv

  13. [30]

    Understanding straight-through estimator in training activation quantized neural nets

    Penghang Yin, Jiancheng Lyu, Shuai Zhang, Stanley Osher, Yingyong Qi, and Jack Xin. Understanding straight-through estimator in training activation quantized neural nets. InInternational Conference on Learning Representations, 2019a. 15 Published in Transactions on Machine Lea...

    2025 arXiv

  14. [31]

    Magr: Weight magnitude reduction for enhancing post-training quantization

    Aozhong Zhang, Naigang Wang, Yanxia Deng, Xin Li, Zi Yang, and Penghang Yin. Magr: Weight magnitude reduction for enhancing post-training quantization. InAdvances in neural information processing systems, 2024a. Aozhong Zhang, Zi Yang, Naigang Wang, Yingyong Qi, Jack Xin, Xin ...

    arXiv

  15. [32]

    Multiple task Following the framework proposed by Hu et al

    Model performance is evaluated at each epoch on the test set, and we report the highest recorded accuracy. Multiple task Following the framework proposed by Hu et al. (2023), we adopt a more integrated approach by training a single model across multiple tasks. Specifically, we...

    2023

  16. [33]

    Moreover, instead of conducting evaluations at every epoch, we assess model performance only after the final epoch. A.3 Commonsense reasoning To evaluate the commonsense reasoning capabilities of CLoQ, we consider eight key benchmark tasks: BoolQ, PIQA, SIQA, HellaSwag, WinoGr...

    2023

  17. [34]

    16 Published in Transactions on Machine Learning Research (08/2025) Table 11: Hyper-parameter for the finetuning of Llama2

    For evaluation, we forgo per-epoch assessments and instead report the final model’s performance after the last epoch. 16 Published in Transactions on Machine Learning Research (08/2025) Table 11: Hyper-parameter for the finetuning of Llama2. Hyper-parameter WikiT ext-2 GSM8K A...

    2025

  18. [1936]

    Gptq: Accurate post-training quantization for generative pre-trained transformers.arXiv preprint arXiv:2210.17323, 2022a

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers.arXiv preprint arXiv:2210.17323, 2022a. Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Optq: Accurate quantiza...

    arXiv

  19. [2016]

    On the crucial role of initialization for matrix factorization

    Bingcong Li, Liang Zhang, Aryan Mokhtari, and Niao He. On the crucial role of initialization for matrix factorization. arXiv preprint arXiv:2410.18965,

    arXiv

  20. [2017]

    Dora: Weight-decomposed low-rank adaptation.arXiv preprint arXiv:2402.09353,

    Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation.arXiv preprint arXiv:2402.09353,

    arXiv

  21. [2018]

    Boolq: Exploring the surprising difficulty of natural yes/no questions.arXiv preprint arXiv:1905.10044,

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions.arXiv preprint arXiv:1905.10044,

    1905 arXiv

  22. [2019]

    Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457,

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457,

    arXiv

  23. [2020]

    Olora: Orthonormal low-rank adaptation of large language models.arXiv preprint arXiv:2406.01775,

    Kerim Büyükakyüz. Olora: Orthonormal low-rank adaptation of large language models.arXiv preprint arXiv:2406.01775,

    arXiv

  24. [2021]

    Qlora: Efficient finetuning of quantized llms

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314,

    arXiv

  25. [2023]

    com/qwopqwop200/gptqlora

    URLhttps://github. com/qwopqwop200/gptqlora. Available athttps://github.com/qwopqwop200/gptqlora. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technica...

    arXiv

  26. [2024]

    Deepseek-coder: When the large language model meets programming–the rise of code intelligence

    13 Published in Transactions on Machine Learning Research (08/2025) Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. Deepseek-coder: When the large language model meets programming–the rise of code intelligence....

    2025 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.