{"id":"a06f4aaa-48ae-4b95-a068-2b7ad09ebd4a","arxiv_id":"2501.03152","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"The claimed mutual information upper bound for LoRA scaling laws is invalid because the key inequality in the proof is false.","lead":"This paper proposes an internal metric, MIUB, to track how LoRA fine-tuning depends on base model size, LoRA rank, and data length, and reports that the metric decreases with each factor. The intended use is a cheap signal for LoRA scaling laws, but the core theorem is mathematically false and the experimental support is weak.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central bound is false: Eq. (6) asserts D_JS(P_OL||P_O P_L) >= I/2, but a binary perfectly correlated example gives I=1 nat and D_JS≈0.216 nat, so MIUB is not an upper bound.","rationale":"In good faith, the paper's central claim is that MIUB, defined through Theorem 3, is an upper bound on the mutual information between LLM and LoRA hidden states and that it follows a scaling law. For that claim to hold, Eq. (6) must be true. It is not: a simple two-outcome joint distribution gives I=1 nat but D_JS≈0.216 nat, so the asserted inequality fails, and since D_JS is bounded while MI is not, no rescaling by log(2) can repair the argument. The proof step is not an external disagreement with consensus; it is internally invalid. Additionally, the experimental computation in Eq. (9) uses a different divergence than the theorem's, so even the empirical trends cannot be interpreted as measuring the proposed bound. Corollary 1 is a power-law ansatz stated without derivation from the theorem. Because the theoretical foundation collapses and the measured quantity does not match the claimed quantity, the paper does not support its scaling-law conclusions. The reader's verdict of REJECT is therefore confirmed; my stress-test adds no ground for changing that verdict.","tokens_in":11440,"tokens_out":7892,"duration_ms":69008,"concrete_test":"Evaluate Eq. (6) for binary O,L with uniform marginals and P(O=L)=1: I(O;L)=1 nat, D_JS(P_OL||P_O P_L)=(3/4)ln(4/3)≈0.216 nat, and log(2)*D_JS≈0.150 nat<1. If these values are reproduced, Theorem 3 is falsified and the MIUB scaling law has no valid theoretical basis.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The proof of Theorem 3 rests entirely on Eq. (6): D_JS(P_OL || P_O P_L) >= (1/2) I(O; L). This inequality is false. For binary variables O,L with P(0,0)=P(1,1)=1/2, I(O;L)=1 nat, while D_JS(P_OL||P_O P_L) = (3/4) ln(4/3) ≈ 0.216 nat. Hence Eq. (6) fails, and log(2)*D_JS ≈ 0.150 nat < 1, directly contradicting the theorem. More generally, JS divergence is bounded above by log(2) (or by 1 in bits), while mutual information is unbounded, so no constant multiple of D_JS(P_OL||P_O P_L) can be a universal upper bound on I(O;L). The standard inequality runs the opposite direction: D_JS(P||Q) ≤ D_KL(P||Q). Since Theorem 3 is the theoretical foundation for MIUB and for Corollaries 1 and 2, the central scaling-law claim is unsupported. There is also an independent gap in the implementation: Eq. (9) averages D_JS between the marginal softmax distributions of the LLM hidden state and the LoRA hidden state, not the joint-versus-product divergence D_JS(P_OL||P_O P_L) appearing in the theorem. Even if the inequality were corrected, the reported empirical quantity would not instantiate the proposed MIUB.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MIUB (Mutual Information Upper Bound), an internal metric for evaluating LoRA fine-tuning of large language models. It defines mutual information I(O;L) between LLM and LoRA hidden states, claims an upper bound via Jensen-Shannon divergence, and states a scaling law MIUB(N,R,D) = A(N0/N)^alpha + B(R0/R)^beta + C(D0/D)^gamma. Experiments with Llama3-8B and Phi3-3B on five benchmark tasks report that MIUB decreases with model size, LoRA rank, and data length, and that this trend is more stable than cross-entropy and perplexity. The central theoretical result is Theorem 3 together with Eq. (2).","tokens_in":11900,"tokens_out":11211,"duration_ms":92937,"significance":"The proposed internal metric addresses a real need: external metrics such as cross-entropy and perplexity often behave non-monotonically in LoRA fine-tuning, and an information-theoretic dependence measure is a reasonable idea. The paper has practical strengths: experiments span two base models, five tasks, three LoRA ranks, four compression levels, three data-length bins, and several prompt templates, and the qualitative MIUB trends in Table 1 mostly match the claimed directions. However, the theoretical bound is false, the quantity computed in Eq. (9) is not the divergence used in the theorem, the power-law form is never fitted, and Figure 3 is unreadable as rendered. Consequently the central contribution is not established, and the significance is presently limited.","major_comments":[{"comment":"The proof's key inequality D_JS(P_OL || P_O P_L) >= (1/2) I(O;L) is false. For binary O=L with P(0,0)=P(1,1)=1/2, I(O;L)=1 nat, while D_JS(P_OL || P_O P_L) = (3/4) ln(4/3) ≈ 0.216 nat, so the inequality would require 0.216 >= 0.5. Consequently Eq. (2) gives I <= log(2)*0.216 ≈ 0.150 nat < 1, a direct contradiction. Since D_JS is bounded above by ln 2 for any two distributions while mutual information is unbounded, no multiplicative constant can make such a bound valid. Theorem 3 and the corollaries built upon it are therefore unsupported.","section":"§3.1, Theorem 3 and Eq. (6)"},{"comment":"The quantity computed in Eq. (9) is not the MIUB of Theorem 3. Theorem 3 bounds I(O;L) using D_JS between the joint distribution P_OL and the product of marginals P_O P_L. Eq. (9) instead sums Jensen-Shannon divergences between the marginal softmax distributions of LLM and LoRA hidden states. The paper never describes estimating the joint distribution, so even if the theorem's inequality were correct, the experiments would not instantiate the proposed upper bound.","section":"§3.2, Eq. (9)"},{"comment":"The claimed scaling law is asserted rather than established. Corollary 1 postulates a three-term power law in N, R, and D, but no derivation from Theorem 3 is given, the exponents alpha, beta, gamma and constants A, B, C, N0, R0, D0 are never estimated, and no fitted curve or goodness-of-fit statistic is reported. Table 1 and Figure 3 show only point values and qualitative monotonicity, without error bars or variance across seeds. The reported monotonic decreases are weaker evidence than the stated power law, so the central scaling-law claim is unverified.","section":"§3.1, Corollary 1; §4.4–4.6"},{"comment":"As rendered in the manuscript, Figure 3 contains only Unicode escape sequences such as '/uni00000014/uni00000013/...' rather than a plottable figure, so the data-complexity scaling results cannot be checked. Because data length is one of the three axes of the claimed scaling law in Corollary 1, this missing evidence is load-bearing.","section":"§4.6, Figure 3"},{"comment":"The MI versus MIUB comparison is internally suspicious. The reported MIUB and MI values agree to three decimal places (e.g., 1547.121 vs 1547.120, 1898.560 vs 1898.452), although MIUB is supposed to be an upper bound that can be substantially larger than MI. This near equality suggests the two columns may be computed from essentially the same estimator, or that the values are not the quantities defined by Eqs. (2) and (9). The authors need to clarify the computation before the claim that MIUB is more suitable than MI can be assessed.","section":"§4.7, Table 3"}],"minor_comments":[{"comment":"The title should read 'Based on' rather than 'Base on'.","section":"Title"},{"comment":"The text says 'seven zero-shot generation tasks' but lists five datasets (PIQA, ARC-Challenge, ARC-Easy, Winogrande, HellaSwag); the role of Wiki2 and PTB in the perplexity experiments should be clarified.","section":"§4.2"},{"comment":"The text mentions 'share8, share4, share3, share1', but Table 1 uses share2; the compression-level notation should be made consistent.","section":"§4.4"},{"comment":"Table cross-references are inconsistent: Section 4.7 refers to Table 2 when discussing MI vs MIUB (which is Table 3), and Section 4.8 refers to Table 3 when discussing prompt templates (which is Table 2).","section":"§4.7–4.8"},{"comment":"Equation (9) uses D^m_JS(P||Q) without defining P and Q in that equation; please specify whether P and Q are the softmax-transformed marginal hidden-state distributions and state explicitly how the joint distribution required by Theorem 3 is handled.","section":"§3.2, Eq. (9)"},{"comment":"Corollary 2 is not a mathematical corollary of the previous results; it is an interpretive claim about MIUB and should be labeled as a remark or hypothesis.","section":"§3.1, Corollary 2"}],"recommendation":"reject","confidential_remarks":"The central theorem is false and the empirical metric does not instantiate the theorem, so the paper's main contributions are unsupported. The near-identical MI and MIUB values in Table 3 and the unreadable Figure 3 further undermine confidence in the reported experiments. I would not recommend a revision unless the authors replace the theoretical foundation and recompute the empirical quantity according to the corrected definition."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper's main theorem is false, and the empirical quantity it computes is not the bound in the theorem. I would not send this to review as is.\n\nThe paper does identify a real problem: external metrics like cross-entropy and perplexity are unstable guides to LoRA fine-tuning quality, and the scaling behavior of LoRA is under-studied. The idea that the dependency between the base model's hidden distribution and the LoRA module's hidden distribution could serve as an internal signal is worth exploring. That is a legitimate empirical direction.\n\nBut the theory collapses. Theorem 3 claims I(O;L) ≤ log(2)·D_JS(P_OL || P_O P_L), and the proof depends entirely on Eq. (6), D_JS(P_OL || P_O P_L) ≥ I/2. That inequality is false. For perfectly correlated binary variables, I = ln 2 ≈ 0.693 nats while D_JS ≈ 0.216 nats, so Eq. (6) fails by a factor of three. In general D_JS is bounded above by log 2 while mutual information is unbounded, so no constant multiple of that D_JS is a universal upper bound. The standard inequality runs the opposite direction: D_JS ≤ D_KL = I.\n\nThe implementation is also disconnected from the theory. Eq. (9) averages D_JS between the marginal softmax outputs of the LLM and LoRA, not D_JS(P_OL || P_O P_L) in the theorem. So even if the bound were correct, the reported numbers would not instantiate it. The MIUB values themselves (1500–3800) are impossible for a Jensen-Shannon divergence, which is at most ln 2 (about 0.69); the authors must be summing over many layers or tokens without saying so. There are no seeds, no error bars, and Table 1 contains corrupted entries like \"3779.9.2\". The gap between the claimed upper bound and the reported values is disqualifying.\n\nWho should read it: anyone curious whether internal divergences between base and adapter distributions trend with rank and size. But with an invalid theorem and uninterpretable numbers, there is nothing solid to build on. My recommendation: desk reject. If the authors resubmit, they need a real bound or an explicit reframing as a heuristic, plus transparent methodology.","headline":"The central bound is false and the reported metric doesn't match it; this is not a publishable paper.","tokens_in":12312,"tokens_out":3489,"would_cite":false,"duration_ms":28101,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims a Jensen-Shannon upper bound on mutual information between base-model and LoRA hidden states tracks LoRA fine-tuning scaling more accurately and stably than cross-entropy or perplexity.","keywords":["LoRA","mutual information upper bound","scaling law","Jensen-Shannon divergence","fine-tuning evaluation","large language models","model compression","parameter-efficient fine-tuning"],"falsifier":"Compute both sides of the paper's inequality (6) on a concrete finite distribution where the joint and product distributions are known exactly, such as two binary variables with a non-independent joint distribution, and check whether $D_{\\mathrm{JS}}(P_{OL}\\|P_O P_L) \\ge \\tfrac{1}{2}I(O;L)$; a violating distribution would falsify the theorem. Separately, recompute Eq. (9) as the joint-versus-product Jensen-Shannon divergence rather than the marginal divergence actually implemented, and see whether the reported scaling trend still holds.","tokens_in":11294,"feed_emoji":"📉","tokens_out":6830,"duration_ms":60401,"temperature":0.7,"pith_summary":"This paper tries to establish a scaling law for LoRA fine-tuning that is driven by the internal relationship between the frozen base model and the low-rank adapter, rather than by external test metrics. It defines MIUB, a mutual-information upper bound computed from the hidden-state distributions of the LLM and the LoRA module, and argues that this bound decays as the base model grows, as the LoRA rank increases, and as the fine-tuning data gets longer. The payoff, if the claim holds, is an internal evaluation signal that predicts a fine-tuned model's actual accuracy more reliably than cross-entropy or perplexity, which often behave erratically across model scales. The paper supports the claim with experiments on Llama3-8B and Phi3-3B across seven benchmark datasets and several prompt templates.","feed_headline":"Mutual-information bound tracks LoRA scaling better than loss","feed_subtitle":"A divergence between base-model and adapter hidden states falls as model size, rank, and data grow, beating cross-entropy and perplexity.","key_machinery":"The load-bearing object is MIUB, the mutual information upper bound. The paper derives $I(O;L) \\le \\log(2)\\,D_{\\mathrm{JS}}(P_{OL}\\|P_O P_L)$, then computes, per LoRA-injected layer, the Jensen-Shannon divergence between softmax-normalized hidden-state distributions of the frozen LLM branch and the LoRA branch after residual addition, summing over modules and averaging over samples. Lower MIUB is interpreted as less reliance of the adapter on base-model features and more learned task-specific generalization.","core_discovery":"The central discovery is that the dependency between the frozen LLM's feature space and the LoRA module's feature space, quantified by the mutual information upper bound, follows a scaling law: $\\mathrm{MIUB}(N,R,D)=A(N_0/N)^\\alpha+B(R_0/R)^\\beta+C(D_0/D)^\\gamma$. The paper identifies the hidden-state distributions after residual connection and softmax normalization as the right objects to compare, and shows in experiments that MIUB decreases monotonically when model size, LoRA rank, or data length increases, while cross-entropy and perplexity can move in the wrong direction. On this basis the paper proposes MIUB as an internal metric for evaluating LoRA fine-tuning in model compression and personalization settings.","pith_inferences":["If the reported trend generalizes beyond the two tested base models, MIUB could predict final fine-tuning accuracy from a single forward pass on a small sample, without requiring labels.","The same joint-versus-product mutual-information framing could be applied to other parameter-efficient methods, such as prefix tuning or adapters, where frozen and trainable hidden states are also combined.","A cheap test of the mechanism would be to track MIUB during training: if it falls while train loss falls and accuracy rises, the dependency interpretation is supported; if it rises, the claim that less dependence means more generalization needs revision."],"forward_implications":["LoRA fine-tuning on larger base models should show a smaller MIUB, meaning the adapter absorbs more task-specific knowledge and depends less on the frozen model.","Increasing the LoRA rank should lower MIUB, so rank can be chosen by watching MIUB rather than validation cross-entropy.","Longer or more complex fine-tuning data should lower MIUB, giving a data-complexity axis for LoRA scaling.","MIUB is a candidate early-stop or model-selection signal for LoRA-based compression and personalization, since it tracks accuracy more stably than cross-entropy or perplexity in the reported experiments."],"supporting_citations":[{"why":"Supplies the LoRA architecture whose frozen base branch and low-rank adapter branch define the two hidden-state distributions being compared.","marker":"Hu et al., 2021"},{"why":"Sets the scaling-law form that the paper adapts to LoRA fine-tuning, where performance changes predictably with model and data size.","marker":"Kaplan et al., 2020"},{"why":"Provides the precedent of using an internal information-theoretic quantity, matrix entropy, as a more stable evaluation metric than cross-entropy and perplexity.","marker":"Wei et al., 2024"},{"why":"Gives the existing fine-tuning scaling-law framework that the paper compares against when studying how data, model size, and method affect fine-tuning.","marker":"Zhang et al., 2024"},{"why":"Supplies Llama3-8B, one of the two base models used to test whether the MIUB scaling trend holds across model sizes.","marker":"Dubey et al., 2024"},{"why":"Supplies Phi3-3B, the second base model used to test the scaling trend at a smaller scale.","marker":"Abdin et al., 2024"}],"fun_headline_variants":["MIUB beats loss metrics for LoRA scaling laws","Loss metrics mislead: MIUB nails LoRA scaling","Why MIUB beats cross-entropy for LoRA scaling","Scaling law for LoRA: MIUB metric outperforms loss"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proof of Theorem 3 assumes that the Jensen-Shannon divergence between the joint distribution of LLM and LoRA outputs and the product of their separate distributions is at least half their mutual information; if that inequality fails, the claimed upper bound does not follow.","fun_headline_variants_meta":{"raw":{"variants":["MIUB beats loss metrics for LoRA scaling laws","Loss metrics mislead: MIUB nails LoRA scaling","Why MIUB beats cross-entropy for LoRA scaling","Scaling law for LoRA: MIUB metric outperforms loss"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000562,"raw_usage":{"total_tokens":2656,"prompt_tokens":924,"completion_tokens":1732,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":540,"completion_tokens_details":{"reasoning_tokens":1664}},"tokens_in":540,"tokens_out":1732,"duration_ms":11933,"temperature":1.0,"reasoning_tokens":1664,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:52:49.591150+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute both sides of the paper's inequality (6) on a concrete finite distribution where the joint and product distributions are known exactly, such as two binary variables with a non-independent joint distribution, and check whether $D_{\\mathrm{JS}}(P_{OL}\\|P_O P_L) \\ge \\tfrac{1}{2}I(O;L)$; a violating distribution would falsify the theorem. Separately, recompute Eq. (9) as the joint-versus-product Jensen-Shannon divergence rather than the marginal divergence actually implemented, and see whether the reported scaling trend still holds.","supporting_citations":[],"review_version":1}