REVIEW 5 major objections 6 minor 19 references
The Scaling Law for LoRA Base on Mutual Information Upper Bound
T0 review · 5 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read The paper claims a Jensen-Shannon upper bound on mutual information between base-model and LoRA hidden states tracks LoRA fine-tuning scaling more accurately and stably than cross-entropy or perplexity.
desk verdict The central bound is false and the reported metric doesn't match it; this is not a publishable paper. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is MIUB, the mutual information upper bound. The paper derives $I(O;L) \le \log(2)\,D_{\mathrm{JS}}(P_{OL}\|P_O P_L)$, then computes, per LoRA-injected layer, the Jensen-Shannon divergence between softmax-normalized hidden-state distributions of the frozen LLM branch and the LoRA branch after residual addition, summing over modules and averaging over samples. Lower MIUB is interpreted as less reliance of the adapter on base-model features and more learned task-specific generalization.
What would settle it
Compute both sides of the paper's inequality (6) on a concrete finite distribution where the joint and product distributions are known exactly, such as two binary variables with a non-independent joint distribution, and check whether $D_{\mathrm{JS}}(P_{OL}\|P_O P_L) \ge \tfrac{1}{2}I(O;L)$; a violating distribution would falsify the theorem. Separately, recompute Eq. (9) as the joint-versus-product Jensen-Shannon divergence rather than the marginal divergence actually implemented, and see whether the reported scaling trend still holds.
Extended reading notes
Core claim
The central discovery is that the dependency between the frozen LLM's feature space and the LoRA module's feature space, quantified by the mutual information upper bound, follows a scaling law: $\mathrm{MIUB}(N,R,D)=A(N_0/N)^\alpha+B(R_0/R)^\beta+C(D_0/D)^\gamma$. The paper identifies the hidden-state distributions after residual connection and softmax normalization as the right objects to compare, and shows in experiments that MIUB decreases monotonically when model size, LoRA rank, or data length increases, while cross-entropy and perplexity can move in the wrong direction. On this basis the paper proposes MIUB as an internal metric for evaluating LoRA fine-tuning in model compression and personalization settings.
Load-bearing premise
The proof of Theorem 3 assumes that the Jensen-Shannon divergence between the joint distribution of LLM and LoRA outputs and the product of their separate distributions is at least half their mutual information; if that inequality fails, the claimed upper bound does not follow.
Editorial extensions
If this is right
- LoRA fine-tuning on larger base models should show a smaller MIUB, meaning the adapter absorbs more task-specific knowledge and depends less on the frozen model.
- Increasing the LoRA rank should lower MIUB, so rank can be chosen by watching MIUB rather than validation cross-entropy.
- Longer or more complex fine-tuning data should lower MIUB, giving a data-complexity axis for LoRA scaling.
- MIUB is a candidate early-stop or model-selection signal for LoRA-based compression and personalization, since it tracks accuracy more stably than cross-entropy or perplexity in the reported experiments.
Reading between the lines
- If the reported trend generalizes beyond the two tested base models, MIUB could predict final fine-tuning accuracy from a single forward pass on a small sample, without requiring labels.
- The same joint-versus-product mutual-information framing could be applied to other parameter-efficient methods, such as prefix tuning or adapters, where frozen and trainable hidden states are also combined.
- A cheap test of the mechanism would be to track MIUB during training: if it falls while train loss falls and accuracy rises, the dependency interpretation is supported; if it rises, the claim that less dependence means more generalization needs revision.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes MIUB (Mutual Information Upper Bound), an internal metric for evaluating LoRA fine-tuning of large language models. It defines mutual information I(O;L) between LLM and LoRA hidden states, claims an upper bound via Jensen-Shannon divergence, and states a scaling law MIUB(N,R,D) = A(N0/N)^alpha + B(R0/R)^beta + C(D0/D)^gamma. Experiments with Llama3-8B and Phi3-3B on five benchmark tasks report that MIUB decreases with model size, LoRA rank, and data length, and that this trend is more stable than cross-entropy and perplexity. The central theoretical result is Theorem 3 together with Eq. (2).
Significance. The proposed internal metric addresses a real need: external metrics such as cross-entropy and perplexity often behave non-monotonically in LoRA fine-tuning, and an information-theoretic dependence measure is a reasonable idea. The paper has practical strengths: experiments span two base models, five tasks, three LoRA ranks, four compression levels, three data-length bins, and several prompt templates, and the qualitative MIUB trends in Table 1 mostly match the claimed directions. However, the theoretical bound is false, the quantity computed in Eq. (9) is not the divergence used in the theorem, the power-law form is never fitted, and Figure 3 is unreadable as rendered. Consequently the central contribution is not established, and the significance is presently limited.
major comments (5)
- [§3.1, Theorem 3 and Eq. (6)] The proof's key inequality D_JS(P_OL || P_O P_L) >= (1/2) I(O;L) is false. For binary O=L with P(0,0)=P(1,1)=1/2, I(O;L)=1 nat, while D_JS(P_OL || P_O P_L) = (3/4) ln(4/3) ≈ 0.216 nat, so the inequality would require 0.216 >= 0.5. Consequently Eq. (2) gives I <= log(2)*0.216 ≈ 0.150 nat < 1, a direct contradiction. Since D_JS is bounded above by ln 2 for any two distributions while mutual information is unbounded, no multiplicative constant can make such a bound valid. Theorem 3 and the corollaries built upon it are therefore unsupported.
- [§3.2, Eq. (9)] The quantity computed in Eq. (9) is not the MIUB of Theorem 3. Theorem 3 bounds I(O;L) using D_JS between the joint distribution P_OL and the product of marginals P_O P_L. Eq. (9) instead sums Jensen-Shannon divergences between the marginal softmax distributions of LLM and LoRA hidden states. The paper never describes estimating the joint distribution, so even if the theorem's inequality were correct, the experiments would not instantiate the proposed upper bound.
- [§3.1, Corollary 1; §4.4–4.6] The claimed scaling law is asserted rather than established. Corollary 1 postulates a three-term power law in N, R, and D, but no derivation from Theorem 3 is given, the exponents alpha, beta, gamma and constants A, B, C, N0, R0, D0 are never estimated, and no fitted curve or goodness-of-fit statistic is reported. Table 1 and Figure 3 show only point values and qualitative monotonicity, without error bars or variance across seeds. The reported monotonic decreases are weaker evidence than the stated power law, so the central scaling-law claim is unverified.
- [§4.6, Figure 3] As rendered in the manuscript, Figure 3 contains only Unicode escape sequences such as '/uni00000014/uni00000013/...' rather than a plottable figure, so the data-complexity scaling results cannot be checked. Because data length is one of the three axes of the claimed scaling law in Corollary 1, this missing evidence is load-bearing.
- [§4.7, Table 3] The MI versus MIUB comparison is internally suspicious. The reported MIUB and MI values agree to three decimal places (e.g., 1547.121 vs 1547.120, 1898.560 vs 1898.452), although MIUB is supposed to be an upper bound that can be substantially larger than MI. This near equality suggests the two columns may be computed from essentially the same estimator, or that the values are not the quantities defined by Eqs. (2) and (9). The authors need to clarify the computation before the claim that MIUB is more suitable than MI can be assessed.
minor comments (6)
- [Title] The title should read 'Based on' rather than 'Base on'.
- [§4.2] The text says 'seven zero-shot generation tasks' but lists five datasets (PIQA, ARC-Challenge, ARC-Easy, Winogrande, HellaSwag); the role of Wiki2 and PTB in the perplexity experiments should be clarified.
- [§4.4] The text mentions 'share8, share4, share3, share1', but Table 1 uses share2; the compression-level notation should be made consistent.
- [§4.7–4.8] Table cross-references are inconsistent: Section 4.7 refers to Table 2 when discussing MI vs MIUB (which is Table 3), and Section 4.8 refers to Table 3 when discussing prompt templates (which is Table 2).
- [§3.2, Eq. (9)] Equation (9) uses D^m_JS(P||Q) without defining P and Q in that equation; please specify whether P and Q are the softmax-transformed marginal hidden-state distributions and state explicitly how the joint distribution required by Theorem 3 is handled.
- [§3.1, Corollary 2] Corollary 2 is not a mathematical corollary of the previous results; it is an interpretive claim about MIUB and should be labeled as a remark or hypothesis.
Circularity Check
No significant circularity: the paper's core defect is an unproved and false inequality plus an empirical-quantity mismatch, neither of which reduces the claimed results to their inputs.
full rationale
The derivation chain does not contain a circular reduction. Theorem 3's upper bound depends on Eq. (6), D_JS(P_OL||P_O P_L) >= (1/2) I(O;L), which the paper asserts without proof; this inequality is in fact false (for perfectly correlated binary variables with P(0,0)=P(1,1)=1/2, I=ln 2 nats but D_JS=(3/4)ln(4/3)≈0.216 nats), so the theorem is unsupported. That is a correctness failure, not circularity: the bound is not defined in terms of the quantity it claims to bound, and the false step is not equivalent to the theorem's statement. Likewise, Eq. (9) computes the empirical 'MIUB' as an average Jensen-Shannon divergence between marginal softmax distributions of the LLM and LoRA hidden states, whereas Theorem 3's bound is log(2) times the JS divergence between the joint distribution P_OL and the product P_O P_L; the experiments therefore measure a different quantity from the one in the theorem. This is a validity/mislabeling gap, not a definitional circle. Corollary 1 states a power-law form with constants A,B,C and exponents alpha,beta,gamma but never estimates or fits these parameters, so there is no fitted input renamed as a prediction; the reported trends are empirical measurements of the metric under changes in model size, LoRA rank, and data length. No load-bearing self-citation appears: related work such as Wei et al. (2024) is cited only as inspiration. The central claims therefore fail on mathematical grounds, but they do not reduce to their own inputs by construction.
Assumptions & free parameters
free parameters (1)
- Scaling exponents alpha, beta, gamma and constants A, B, C, N0, R0, D0 =
not fitted
assumptions (3)
- domain assumption The dependency between LLM and LoRA feature spaces can be quantified by mutual information.
- ad hoc to paper D_JS(P_OL || P_O P_L) >= (1/2) I(O;L)
- ad hoc to paper MIUB follows the power-law form MIUB(N,R,D) = A(N0/N)^alpha + B(R0/R)^beta + C(D0/D)^gamma.
Cite this review
Pith. "Pith review of The Scaling Law for LoRA Base on Mutual Information Upper Bound." pith.science (2026). https://pith.science/paper/HSC5N3AX
@misc{pith2026250103152,
author = {Pith},
title = {Pith review of: The Scaling Law for LoRA Base on Mutual Information Upper Bound},
year = {2026},
howpublished = {\url{https://pith.science/paper/HSC5N3AX}},
note = {Machine review of arXiv:2501.03152}
}
read the original abstract
LoRA (Low-Rank Adaptation) is a widely used model fine-tuning method. In fine-tuning, the law among model performance, model parameters, and data complexity has been a focal issue in the field. Existing methods often leverage external metrics (such as cross-entropy or perplexity) to evaluate model performance. In the fine-tuning process for large models, two types of knowledge are typically involved: the frozen, general knowledge acquired by the model during pre-training and the new knowledge learned through the LoRA module from the current data. Generally, the less LoRA's learned knowledge relies on the large model, the more it captures the specific knowledge of new data, thereby enhancing its adaptability to new tasks. However, external metrics do not readily capture the dependency relationship between these two types of knowledge. Therefore, we designed an internal metric based on the Mutual Information Upper Bound (MIUB) theory to investigate the scaling law of large-model LoRA fine-tuning. In our experiments, we validated this approach on benchmark datasets, using the Llama3-8B and Phi3-3B models. The results show that the proposed MIUB metric aligns more accurately and stably with the scaling law of LoRA fine-tuning compared to cross-entropy and perplexity.
Figures
Reference graph
Works this paper leans on
-
[4]
arXiv preprint arXiv:2311.11696
Sparse low-rank adaptation of pre-trained language models. arXiv preprint arXiv:2311.11696. Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Wei Shen, Limao Xiong, Yuhao Zhou, Xiao Wang, Zhi- heng Xi, Xiaoran Fan, et al
-
[5]
Loramoe: Allevi- ating world knowledge forgetting in large language models via moe-style plugin. In Proceedings of the 62nd Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 1932–1945. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten,...
work page 1932
-
[6]
arXiv preprint arXiv:2407.21783
The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Yan Fang, Jingtao Zhan, Qingyao Ai, Jiaxin Mao, Wei- hang Su, Jia Chen, and Yiqun Liu
-
[7]
arXiv preprint arXiv:2307.06945
In-context autoencoder for con- text compression in a large language model. arXiv preprint arXiv:2307.06945. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen
-
[9]
arXiv preprint arXiv:2307.13269
Lorahub: Effi- cient cross-task generalization via dynamic lora com- position. arXiv preprint arXiv:2307.13269. Mladjan Jovanovic and Peter V oss
-
[10]
arXiv preprint arXiv:2404.18311
Trends and challenges of real-time learning in large lan- guage models: A critical review. arXiv preprint arXiv:2404.18311. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei
-
[12]
arXiv preprint arXiv:2310.18339
Moelora: An moe-based parameter efficient fine- tuning method for multi-task medical applications. arXiv preprint arXiv:2310.18339. Yuren Mao, Yuhang Ge, Yijiang Fan, Wenyi Xu, Yu Mi, Zhonghao Hu, and Yunjun Gao
-
[13]
arXiv preprint arXiv:2407.11046
A survey on lora of large language models. arXiv preprint arXiv:2407.11046. Mitch Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Ferguson, Karen Katz, and Britta Schasberger
Show all 19 references
-
[15]
arXiv preprint arXiv:2406.18406
Ircan: Mitigating knowledge conflicts in llm generation via identify- ing and reweighting context-aware neurons. arXiv preprint arXiv:2406.18406. Jue Wang, Yucheng Lu, Binhang Yuan, Beidi Chen, Percy Liang, Christopher De Sa, Christopher Re, and Ce Zhang
-
[16]
arXiv preprint arXiv:2401.17139
Large language model evaluation via matrix entropy. arXiv preprint arXiv:2401.17139. Shu Yang, Muhammad Asif Ali, Cheng-Long Wang, Li- jie Hu, and Di Wang
-
[17]
arXiv preprint arXiv:2402.11260
Moral: Moe augmented lora for llms’ lifelong learning. arXiv preprint arXiv:2402.11260. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi
-
[19]
arXiv preprint arXiv:2402.17193
When scaling meets llm finetuning: The effect of data, model and finetuning method. arXiv preprint arXiv:2402.17193
-
[2016]
arXiv preprint arXiv:1609.07843
Pointer sentinel mixture mod- els. arXiv preprint arXiv:1609.07843. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavat- ula, and Yejin Choi
-
[2018]
arXiv preprint arXiv:1803.05457
Think you have solved question an- swering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457. Ning Ding, Xingtai Lv, Qiaosen Wang, Yulin Chen, Bowen Zhou, Zhiyuan Liu, and Maosong Sun
-
[2019]
Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat
Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830. Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat
1905 arXiv
-
[2020]
arXiv preprint arXiv:2001.08361
Scaling laws for neural language models. arXiv preprint arXiv:2001.08361. Jeonghoon Kim, Jung Hyun Lee, Sungdong Kim, Joon- suk Park, Kang Min Yoo, Se Jung Kwon, and Dong- soo Lee
2001 arXiv
-
[2021]
arXiv preprint arXiv:2106.09685
Lora: Low-rank adap- tation of large language models. arXiv preprint arXiv:2106.09685. Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin
-
[2023]
arXiv preprint arXiv:2303.08774
Gpt-4 technical report. arXiv preprint arXiv:2303.08774. Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al
-
[2024]
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al
Phi-3 technical report: A highly capable language model locally on your phone.arXiv preprint arXiv:2404.14219. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.