{"id":"b91b9233-564a-4403-8289-34e5392f18c1","arxiv_id":"2502.03044","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"RepLoRA reparameterizes LoRA's A and B matrices through lightweight MLPs with shared inputs, and claims this changes estimation sample complexity from exponential to polynomial.","lead":"The paper reparameterizes LoRA's low-rank matrices as outputs of small MLPs and argues, through a mixture-of-experts analysis, that this makes fine-tuning more sample-efficient. It reports consistent gains over LoRA across language, image, video, and multimodal benchmarks, with the largest gains in small-data regimes.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 5.1 gives a polynomial n^{-1/2} lower bound; the claim that this implies 'slower than polynomial' and exponential sample complexity is not a valid logical consequence, so the central exponential-to-polynomial result is unproven.","rationale":"The reader's weakest assumption concerns the gap between the simplified theory (AQ=AV, BQ=BV) and the deployed RepLoRA (separate heads, diagonal shared inputs). That is a legitimate theory-practice concern. However, the more load-bearing problem is internal to the theory: even granting the simplification, Theorem 5.1 does not imply the exponential sample complexity that the abstract and Section 5 advertise. A lower bound of n^{-1/2} is polynomial; it cannot establish that vanilla LoRA needs exponential data. The paper's own proof constructs sequences with polynomial parameter-error rates, and the subsequent theorems give polynomial rates for the reparameterized case. Thus the central 'exponential to polynomial' dichotomy is unsupported. This is a correctness risk in the strongest claim, not merely a mismatch between assumptions and implementation. I still do not recommend outright rejection because the empirical contribution is consistently reported and the method may be valuable independent of the theory, but the theoretical claim must be withdrawn or substantially repaired. Hence the conditional verdict is unchanged, though the condition should explicitly require fixing or removing the exponential-rate claim.","tokens_in":42204,"tokens_out":6084,"duration_ms":53465,"concrete_test":"Analytically test the implication in Section 5.1: fix r=1 and exhibit a sequence of parameters in the model of Eq. (11) with total parameter error δ_n = n^{-1/2} for which D_{1,1}(G, G*) = O(n^{-1/2}). If such a sequence exists, the lower bound n^{-1/2} is compatible with polynomial estimation and cannot imply exponential sample complexity. A stronger check is to compute the actual minimax rate for estimating the low-rank matrices under the MoE regression model using standard Fano or Le Cam arguments; if the reassessed rate is polynomial, the 'exponential' claim is false. Re-deriving the displayed implication after Theorem 5.1 and checking whether an upper bound of order 1/log^τ n is ever proved will settle whether the exponential data requirement is a theorem or an unsupported assertion.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline claim is that reparameterization reduces the data required from exponential to polynomial. The 'without reparameterization' side rests entirely on Theorem 5.1, which establishes a minimax lower bound sup E[D_{1,r}] ≳ n^{-1/2}. This is a polynomial rate, not a super-polynomial one. Since D_{1,r} measures r-th powers of parameter deviations, a loss of order n^{-1/2} is consistent with parameter error of order n^{-1/(2r)}, which is polynomial for every fixed r. The text then asserts that the estimators are 'slower than any polynomial rates' and 'could become as slow as O(1/log^τ n)', leading to O(exp(ε^{-1/τ})) data. This inference does not follow: a lower bound cannot rule out polynomial upper bounds, and n^{-1/2} is itself polynomial. Moreover, the construction in Appendix A.1 explicitly produces parameter deviations of size n^{-1/r} with D_{1,r} of order n^{-r}, showing that polynomial rates are compatible with the loss. The later parametric rate in Theorem 5.2, √(log n/n), is also polynomial, so the claimed exponential gap is not established even under the paper's own simplified AQ=AV, BQ=BV assumption. The deployed RepLoRA's separate query/value heads and diagonal shared inputs are a separate theory-practice mismatch, but the logical gap in Theorem 5.1 is sufficient to invalidate the central theoretical contribution as stated.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RepLoRA, a parameter-efficient fine-tuning method that reparameterizes LoRA's low-rank matrices as outputs of lightweight MLPs with shared inputs. The authors first cast LoRA in a mixture-of-experts framework, then analyze a simplified regression model to argue that reparameterization improves sample complexity from exponential to polynomial. Theorems 5.2 and 5.3 give polynomial upper bounds for the reparameterized estimators, while Theorem 5.1 gives a minimax lower bound for the un-reparameterized estimator. The paper reports extensive experiments on language, vision, video, and multimodal tasks, claiming consistent gains over LoRA and matching LoRA's full-data performance with 30% of the training data.","tokens_in":42574,"tokens_out":6193,"duration_ms":54377,"significance":"If the exponential-to-polynomial claim were established, the paper would provide a substantial theoretical justification for a simple and widely applicable PEFT modification, and the empirical results across four domains would make the contribution practically relevant. The paper has clear strengths: the MoE-based reformulation of LoRA is instructive, the upper-bound analyses in Theorems 5.2 and 5.3 appear internally coherent, and the experimental section is extensive, including a sample-efficiency prediction that is tested directly. However, the central theoretical claim as stated is not supported by the provided results, and the theory is developed under an assumption that the deployed method does not satisfy.","major_comments":[{"comment":"Theorem 5.1 establishes sup_G E[D_{1,r}(Ĝ_n, G)] ≳ n^{-1/2}, which is a polynomial lower bound. The subsequent paragraph's inference that the low-rank estimators are 'slower than any polynomial rates' and may require O(exp(ε^{-1/τ})) samples does not follow: a lower bound of n^{-1/2} is compatible with a parameter-error rate of n^{-1/(2r)} and with a D_{1,r} rate of n^{-1/2}, both polynomial. Moreover, the construction in Appendix A.1 itself yields D_{1,r}(G_n, G*) = O(n^{-r}) with ∥f_{G_n} - f_{G*}∥/D_{1,r} → 0, demonstrating that polynomial loss rates are consistent with the paper's own singularity analysis. The central exponential-to-polynomial sample-complexity claim in the abstract and Section 5 therefore is not proven.","section":"Section 5.1, Theorem 5.1"},{"comment":"The reparameterized theory assumes A_Q = A_V and B_Q = B_V (Section 5.2, first paragraph, and the 'without loss of generalization' remark), while the deployed RepLoRA in Section 6, Eq. (18), uses separate output heads for A_Q/A_V and B_Q/B_V with diagonal inputs A and B. The theoretical upper bounds in Theorems 5.2 and 5.3 therefore apply to a different estimator than the one evaluated in Section 7, so the claim that the theory explains RepLoRA's empirical sample-efficiency gains is not supported. The paper should either extend the analysis to the separate-head architecture or explicitly present the theory as a stylized model that is not directly predictive of the deployed method.","section":"Section 5.2, Section 6"},{"comment":"The comparison between 'without reparameterization' and 'with reparameterization' is made across incomparable loss functions: Theorem 5.1 is a lower bound on D_{1,r}, which accumulates r-th powers of deviations of four matrices, whereas Theorems 5.2 and 5.3 are upper bounds on D_2 and D_3, which measure deviations of the product W_2 B W_1 A or of the pair (W_1 A, W_2 B) in linear or squared terms. The text does not translate the D_{1,r} lower bound into a lower bound on the metrics used for the reparameterized estimators, so the claimed reduction 'from exponential to polynomial' is not a formally established statement about a common estimation error.","section":"Sections 5.1 and 5.2"}],"minor_comments":[{"comment":"The phrase 'slower than any polynomial rates O_P(n^{-1/2r})' is internally contradictory because n^{-1/2r} is polynomial for every fixed r; the sentence should be rephrased.","section":"Section 5.1"},{"comment":"'without loss of generalization' should be 'without loss of generality'.","section":"Section 5.2"},{"comment":"In the definition of the regression function, the expression 'M^0_K × R^{d×d}' appears to be missing a symbol or a phrase; please clarify the intended meaning.","section":"Equation (11)"},{"comment":"The 'A VG' and 'PPT' columns are not visually separated from the dataset columns, making the table hard to read; consider adding clearer column separators or a note explaining that 'A VG' is the average over the nineteen datasets.","section":"Table 3"},{"comment":"The reference 'Langley, P., Crafting papers on machine learning' is listed but not cited in the text.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The empirical study is extensive and the upper-bound analyses have value, but the abstract's headline theoretical claim is not supported by the current proof of Theorem 5.1 and the associated interpretation. A revision that either proves a genuine super-polynomial lower bound or substantially reframes the contribution around the polynomial upper bounds and the singularity analysis would be needed for the paper to be acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the practical method in this paper is worth knowing about, and the experiments are extensive and mostly convincing. But the headline theoretical claim — that reparameterization cuts the data requirement from exponential to polynomial — is not proven. The lower bound they prove for vanilla LoRA is polynomial, n^{-1/2}, and they then assert that this means \"slower than any polynomial rates.\" That inference is invalid. A lower bound of n^{-1/2} does not rule out polynomial upper bounds; it is itself polynomial. So the exponential gap is unsupported.\n\nWhat is genuinely new: viewing LoRA through the mixture-of-experts lens, and showing that shared reparameterization yields parametric upper rates in a simplified MoE regression model. Theorems 5.2 and 5.3 give clean OP(sqrt(log n/n)) rates under the shared-structure assumption. The empirical section is broad — commonsense reasoning, image, video, multimodal — and RepLoRA consistently beats vanilla LoRA, sometimes by large margins at low data fractions. The sample-efficiency experiment is a real prediction, not a fitted curve. That part is solid.\n\nThe soft spots are in the theory-practice bridge. The theory assumes AQ=AV and BQ=BV, while the deployed RepLoRA uses separate output heads and diagonal shared inputs. The authors call the equality assumption \"without loss of generalization,\" but it is not WLOG; it changes the estimator. So even if the upper rates are correct, they apply to a different model than the one evaluated. Also, the paper omits DoRA in some tables and doesn't report error bars. Those are minor compared to the logical gap.\n\nWho should read this: PEFT practitioners who want a simple, effective reparameterization that seems to help in low-data regimes. The theory section needs a major rewrite or a much weaker claim before it can be trusted.\n\nMy recommendation: this deserves a serious referee. The empirical contribution is strong enough that the paper should not be desk-rejected, but the theoretical overclaim needs to be fixed. I'd send it to review with a clear note: correct the exponential-to-polynomial claim, or reframe the contribution as an empirical method plus provisional theory for a simplified setting.","headline":"A useful empirical PEFT method, but the exponential-to-polynomial theoretical claim does not follow from the paper's own lower bound.","tokens_in":43074,"tokens_out":2125,"would_cite":true,"duration_ms":19677,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that reparameterizing LoRA's low-rank matrices changes the data needed for a target estimation error from exponential to polynomial, and proposes RepLoRA to realize that gain.","keywords":["Low-Rank Adaptation","Parameter-Efficient Fine-Tuning","Mixture of Experts","Reparameterization","Sample Complexity","LoRA","Fine-tuning"],"falsifier":"Fit the same MoE regression with the exact RepLoRA parameterization used in practice (separate MLP heads for queries and values, diagonal shared inputs) and compute or simulate the minimax rate of low-rank matrix estimation; if the rate is no better than exponential, the paper's central theoretical claim does not apply to its own method.","tokens_in":41992,"feed_emoji":"📉","tokens_out":8081,"duration_ms":67061,"temperature":0.7,"pith_summary":"The paper tries to establish that the reason LoRA struggles with small datasets is structural: when LoRA's low-rank updates are viewed through the mixture-of-experts (MoE) form of an attention head, estimating the unshared low-rank matrices has minimax convergence slower than any polynomial rate, so reaching error $\\epsilon$ can require exponentially many samples. It then argues that reparameterizing the low-rank matrices as outputs of shared linear or nonlinear maps removes this bottleneck, giving rates $O_P(\\sqrt{\\log(n)/n})$ and polynomial sample complexity. On that basis it proposes RepLoRA, which generates the query and value low-rank matrices from two lightweight MLPs with shared diagonal inputs, and reports that RepLoRA consistently beats vanilla LoRA across language, image, video, and multimodal benchmarks, matching LoRA's full-data accuracy with about 30% of the training data.","feed_headline":"Reparameterizing LoRA cuts sample needs from exponential to polynomial","feed_subtitle":"A theoretical MoE analysis plus a lightweight MLP reparameterization lets RepLoRA match LoRA at 30% of the data.","key_machinery":"The central object is the MoE reading of an attention head: each output position is a softmax mixture over token-position experts $f_j(X)=W_V^\\top x_j$ with score functions $s_{i,j}(X)=x_i^\\top W_Q W_K^\\top x_j/\\sqrt{d_v}$, and LoRA's low-rank updates perturb both. The convergence analysis is carried out with a Voronoi loss that partitions fitted mixture atoms by nearest ground-truth atom; the loss is what turns the least-squares rate into a statement about the low-rank matrices themselves. The mechanism proposed to improve the rate is shared-structure reparameterization: instead of learning $A_Q,A_V,B_Q,B_V$ separately, generate them as outputs of shared maps from common inputs, which the theory shows changes the estimation rate from exponential-scale to polynomial-scale data requirements.","core_discovery":"Viewing each attention head as a softmax-gated MoE, LoRA's learned updates modify both the experts (through value projections) and the gating score functions (through query projections). In the paper's regression model, vanilla LoRA's parameter estimates are shown to have worst-case convergence slower than any polynomial in $n$, which implies exponential sample complexity; reparameterizing the low-rank matrices as outputs of shared maps $A_Q=A_V=W_1A$, $B_Q=B_V=W_2B$, or their nonlinear versions, yields $D_2(\\bar G_n,\\bar G^*)=O_P(\\sqrt{\\log(n)/n})$, hence a polynomial number of samples suffices. The paper takes this as evidence that shared-structure reparameterization is the mechanism behind RepLoRA's improved sample efficiency, and the experimental section shows the resulting method improving on LoRA consistently, with the largest gains at small data fractions.","pith_inferences":["A testable extension is to apply the same shared-input MLP reparameterization to other low-rank PEFT variants; if the theory is mechanistic, they should inherit part of the sample-efficiency gain.","The paper's strongest empirical claim is about data fraction, but the theory addresses estimation error of low-rank matrices rather than downstream task accuracy; connecting the two would require a task-level transfer argument the paper does not make.","The diagonal shared inputs used in practice are a stronger structural assumption than the theory's full shared matrices, so ablating diagonal versus dense shared inputs would reveal whether parameter efficiency or the shared structure itself drives the gain.","The lower bound for vanilla LoRA is minimax, so real optimizers with constrained initialization might see a smaller exponential-to-polynomial gap than the worst-case regression model predicts."],"forward_implications":["If the theory is right, shared-structure reparameterization is a statistical fix rather than an optimization trick: it removes a minimax inefficiency inherent to estimating LoRA matrices independently.","RepLoRA should keep its advantage in low-data regimes; on FGVC it reports gains up to 40.4% relative to LoRA at $f=1\\%$ training data and matches LoRA's full-data accuracy at $f=30\\%$.","Nonlinear reparameterization should dominate linear reparameterization, consistent with the rate improvement of Theorem 5.3 over Theorem 5.2.","Because the MLP is discarded after training, the sample-efficiency gain costs no extra inference overhead.","The LoRA-as-MoE formulation applies beyond queries and values, so the same reparameterization argument extends naturally to keys and to other low-rank PEFT updates expressed in this form."],"supporting_citations":[{"why":"Defines LoRA, the baseline method whose low-rank updates are reparameterized and the main comparison in all experiments.","marker":"(Hu et al., 2022)"},{"why":"Establishes the view of attention heads as mixtures of experts that this paper extends to LoRA.","marker":"(Le et al., 2024)"},{"why":"Shows statistical benefits of reparameterizing prompts, the template for the reparameterization analysis.","marker":"(Le et al., 2025)"},{"why":"Supplies the Voronoi-loss machinery used to convert regression-function rates into parameter-estimation rates.","marker":"(Manole & Ho, 2022)"},{"why":"Provides the empirical-process bound behind the parametric $O_P(\\sqrt{\\log(n)/n})$ regression-function rate.","marker":"(van de Geer, 2000)"},{"why":"Le Cam's lemma lower bound used to prove the $\\gtrsim n^{-1/2}$ minimax rate for vanilla LoRA.","marker":"(Yu, 1997)"},{"why":"Supplies the per-class subsampling protocol used to measure sample efficiency on FGVC datasets.","marker":"(d'Ascoli et al., 2021)"},{"why":"Motivates least-squares estimation of MoE experts and the convergence-rate framing adopted in Section 5.","marker":"(Nguyen et al., 2024d;c; 2025)"}],"fun_headline_variants":["Reparameterizing LoRA: exponential to polynomial sample complexity","RepLoRA matches LoRA with 30% of the data","RepLoRA: polynomial samples instead of exponential","LoRA reparameterization: from exponential to polynomial sample needs","RepLoRA uses 30% data to match LoRA's performance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The theoretical polynomial-rate result assumes the query and value low-rank matrices are exactly shared ($A_Q=A_V$ and $B_Q=B_V$), which the method deployed in experiments does not do; if that simplification is not representative, the proven sample-complexity gain may not explain RepLoRA's empirical advantage.","fun_headline_variants_meta":{"raw":{"variants":["Reparameterizing LoRA: exponential to polynomial sample complexity","RepLoRA matches LoRA with 30% of the data","RepLoRA: polynomial samples instead of exponential","LoRA reparameterization: from exponential to polynomial sample needs","RepLoRA uses 30% data to match LoRA's performance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001085,"raw_usage":{"total_tokens":4531,"prompt_tokens":936,"completion_tokens":3595,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":552,"completion_tokens_details":{"reasoning_tokens":3509}},"tokens_in":552,"tokens_out":3595,"duration_ms":24991,"temperature":1.0,"reasoning_tokens":3509,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T10:07:45.431785+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fit the same MoE regression with the exact RepLoRA parameterization used in practice (separate MLP heads for queries and values, diagonal shared inputs) and compute or simulate the minimax rate of low-rank matrix estimation; if the rate is no better than exponential, the paper's central theoretical claim does not apply to its own method.","supporting_citations":[],"review_version":1}