{"id":"f6f209fd-c658-4d7f-859c-fa8cc27d1343","arxiv_id":"2502.06820","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"LoCA reparameterizes fine-tuning weight updates with a sparse set of DCT coefficients and learns their locations, claiming better expressivity than LoRA under a Gaussian assumption on weight updates.","lead":"This paper introduces LoCA, a new way to fine-tune large models by working in the cosine frequency domain and learning which frequency components to update. If it holds up, it could reduce the number of trainable parameters while maintaining or improving accuracy.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central ranking proof assumes ∆W is i.i.d. Gaussian, but Proposition 1's assumption A2 is essentially that claim and the empirical support is not sufficient; real updates are likely structured, so the expressivity result may not transfer.","rationale":"The reader identified the same weakest assumption: Proposition 1 assumes the asymptotic i.i.d. Gaussian structure of Delta W, and Appendix A admits that strict i.i.d. is not established. I agree that this is the single most load-bearing concern because Theorem 1, the paper's central theoretical novelty, is a statement about expected reconstruction errors under W ~ N(0, I). If real fine-tuning updates are not well modeled by an i.i.d. Gaussian ensemble, the claimed expressivity advantage of carefully selected frequency components over low-rank decomposition is not established, regardless of how rigorously the inequality is proven for Gaussian matrices. The paper has independent support: the empirical evaluations across GLUE, E2E, instruction tuning, and vision tasks show LoCA is competitive and often better than FourierFT and LoRA at smaller parameter counts, and the toy convergence experiment in Appendix M is a useful sanity check. These empirical results justify the method itself, but they do not rescue the theoretical comparison from reliance on the i.i.d. Gaussian model. The proposed concrete test directly checks the model assumption on real fine-tuned weights: if, at equal parameter count, the optimal low-rank reconstruction of an actual Delta W has smaller error than the optimal sparse DCT reconstruction, then the premise of Theorem 1 is violated and the expressive-advantage claim should be presented only as a Gaussian-model result, not as a general property of fine-tuned weight updates. Because the reader already conditioned acceptance on strengthening or reframing this theoretical claim, my read does not change the recommended verdict.","tokens_in":40254,"tokens_out":7865,"duration_ms":84028,"concrete_test":"Using a fully fine-tuned RoBERTa-base on MNLI (or another GLUE task), extract the actual incremental matrices Delta W for query and value projections across several layers. For matched parameter budgets, compute (a) the best rank-r low-rank reconstruction error ||Delta W - U_r Sigma_r V_r^T||_F^2 and (b) the best sparse DCT reconstruction error keeping the top m DCT coefficients, where m is chosen so that the number of stored coefficients equals r(p+q), consistently counting parameters in both methods. If (a) is smaller than (b) for typical r in {8, 16, 32}, the i.i.d. Gaussian premise is falsified and Theorem 1's conclusion cannot be applied to real updates. Additionally, report the largest eigenvalue of Delta W^T Delta W relative to the Marchenko-Pastur upper edge to quantify any low-rank structure.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing foundation of Theorem 1 is Proposition 1, and Proposition 1 assumes the very property needed for the theorem to apply. Assumption (A2) states that W' - W is asymptotically i.i.d. and independent of W; Appendix A explicitly concedes that strict i.i.d. of trained neural-network parameters remains an open question. The supporting evidence, a bootstrap-like total-variation test and a Marchenko-Pastur fit, cannot establish independence: many non-i.i.d. ensembles (low-rank spikes, block correlations, sparsity) also have MP-like bulk spectral densities, and the TV test only checks closeness to a Gaussian within epsilon = 0.001 under a synthetic null of perturbed Gaussians. Consequently, Theorem 1's ordering E[L(W, W_F^(1))] > E[L(W, W_R)] > E[L(W, W_F^(2))] > E[L(W, W_F^(3))] is derived for W ~ N(0, I), but real fine-tuning updates are the output of gradient descent on a structured task and are plausibly low-rank or correlated. If they are, low-rank reconstruction can beat sparse frequency reconstruction at the same parameter budget, and the central expressivity claim fails to transfer. Appendix P studies only a uniform-correlation model rho * 1*1^T + I, not the low-rank or block-structured dependencies that actually arise in fine-tuning; the reported critical rho_c values do not address whether Theorem 1's premise holds for real updates.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces LoCA, a parameter-efficient fine-tuning method that reparameterizes weight updates as a sparse inverse-DCT with learnable coefficient values and locations. The authors first claim, via Proposition 1, that weight updates during fine-tuning are asymptotically i.i.d. Gaussian, and then use this to prove (Theorem 1) that, under an i.i.d. Gaussian model, sparse frequency-domain reconstruction with carefully selected components has lower expected Frobenius error than low-rank reconstruction at comparable parameter budgets, and that random frequency selection is worse than low-rank. Theorem 2 shows the DCT-based selection is equivalent to the optimal Fourier coefficient selection. The method alternates between optimizing coefficients and locations, using central-difference gradient estimates for the discrete locations. Experiments on GLUE, E2E, Alpaca instruction tuning, image classification, and Stable Diffusion are reported.","tokens_in":40592,"tokens_out":14522,"duration_ms":128045,"significance":"If the theoretical results were fully established, the paper would provide the first formal expressivity comparison between frequency-domain and low-rank PEFT, and LoCA itself shows consistently competitive or superior performance to FourierFT and near-LoRA results with fewer parameters. The connection to random matrix theory (Wishart eigenvalues, chi-square order statistics) is a useful framework. However, the central theorem is not fully proven and relies on assumptions that are not adequately validated, and the parameter accounting in both the theory and experiments needs correction. These issues currently limit the strength of the claims.","major_comments":[{"comment":"The key inequality (15) that establishes the ordering between low-rank and frequency methods is only verified numerically for general r and K; the analytic argument covers only r=1 and large K. Since Theorem 1 is stated as a theorem for all r < K/3, numerical verification is insufficient. This is load-bearing because the full ordering in Theorem 1 depends on (15). The authors should either provide a complete proof of (15) or explicitly restate the result as a conjecture supported by numerical evidence, and adjust the abstract and discussion accordingly.","section":"Appendix G, Theorem 1, Eq. (15)"},{"comment":"Assumptions A1 and A2 in Proposition 1 are load-bearing but not adequately established. The hypothesis test in Appendix B only checks closeness of the empirical distribution to a Gaussian under a synthetic null of perturbed Gaussians; it does not test independence. The Marchenko-Pastur fit is consistent with i.i.d. entries but also with many structured ensembles such as low-rank spikes or block-correlated matrices. The paper itself concedes in Appendix A that strict i.i.d. of trained neural-network parameters is an open question. To support Theorem 1, the authors should directly measure structure in real fine-tuning updates (e.g., singular value decay, effective rank, correlation between entries) or empirically verify the Theorem 1 ordering on actual full fine-tuning updates.","section":"Section 2, Proposition 1, Appendix A/B"},{"comment":"The comparison for W_F^(2) is not at the same parameter budget. In the theorem, N0 = (p+q)r = 2Kr for a square matrix, and N2 = 2/3 N0 locations. Each selected location in the non-redundant half carries two real coefficients (real and imaginary parts), so W_F^(2) uses 2N2 = 4/3 N0 learnable parameters, whereas W_R uses N0 parameters. Thus the inequality E[L(W, W_R)] > E[L(W, W_F^(2))] is an unequal-budget comparison and does not establish the claimed expressivity superiority. To make the comparison fair, N2 should be set to 1/2 N0 so that the total coefficient count equals N0, and the proof and numerical verification should be redone under that budget.","section":"Section 3, Theorem 1, parameter budget for W_F^(2)"},{"comment":"The reported parameter counts for LoCA exclude the location variables l, which are optimized for Bs steps and must be known to the model to place the learned coefficients. This undercounts trainable parameters and makes the comparisons with FourierFT (which has fixed random locations) and with LoRA potentially misleading. The authors should either include l in the reported parameter counts, or provide a concrete argument that the information content of the learned locations is negligible in the stated budgets and is not providing an unfair advantage over the baselines.","section":"Section 5, Implementation Details, Tables 1-4"},{"comment":"The robustness analysis only considers the uniform correlation model Sigma = rho * 1*1^T + I. Real fine-tuning updates are more plausibly low-rank or block-structured, which is exactly the regime most relevant to a comparison against low-rank adaptation. The authors should either study structured dependence models (e.g., a rank-k spike plus noise, or block correlations) or empirically test whether the Theorem 1 ordering holds on real weight updates. As presented, the reported critical rho_c values do not address the concern that structured dependencies may invalidate the i.i.d. premise of the theory.","section":"Appendix P, non-i.i.d. analysis"}],"minor_comments":[{"comment":"In the ViT-large LoCA 144K row, the RESISC45 entry '94.41±91.76' is clearly a typo and should be corrected.","section":"Table 4"},{"comment":"The text contains 'mean squeue error', which should be 'mean squared error'.","section":"Appendix M"},{"comment":"The caption mentions 'FHVC', which should be 'FGVC'.","section":"Figure 3 caption"},{"comment":"The empirical observations supporting the Gaussian model are based on a single model (LLaMA-7b) fine-tuned on a single dataset (Alpaca-52K); the authors should acknowledge this limited basis or include additional models/tasks in the analysis.","section":"Section 2"},{"comment":"The description of the hypothesis test is somewhat informal; in particular, the construction of the null distribution by sampling from perturbed Gaussians should be more clearly distinguished from a proper bootstrap, and the choice of epsilon and the perturbation variance should be justified.","section":"Appendix B"}],"recommendation":"major_revision","confidential_remarks":"The paper is an ICLR 2025 conference paper. The main concerns are the incomplete proof of Theorem 1, the unvalidated i.i.d. assumption, and the parameter-budget accounting in both the theoretical and experimental comparisons. These are fixable in principle, so I do not recommend rejection, but the authors should be asked to address them before publication in a journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You asked about the LoCA paper. Short version: the method is real, the experiments are honest, but the theory is doing more work than it can actually support.\n\nWhat's genuinely new: a frequency-domain PEFT that learns where to put its sparse frequency components, using iDCT and a finite-difference gradient estimate for the discrete locations. That's a concrete, sensible idea, and the paper shows it works: LoCA beats FourierFT at the same budget across GLUE, E2E, instruction tuning, and vision, and is competitive with LoRA while using fewer parameters. The appendix experiments, including Stable Diffusion, are a nice extra. Theorem 2, showing that selecting top coefficients in DCT matches the optimal Fourier variant, is clean and useful.\n\nThe soft spot is Theorem 1. The claimed ordering is the heart of the paper, but the proof only covers r=1 and large K analytically; for general r and K it relies on numerical integration of inequality (15). That's not a proof, and the paper should say so. Worse, Proposition 1 assumes (A2), which is essentially the i.i.d. Gaussian claim. The appendix admits strict i.i.d. is open. The MP fit and the custom hypothesis test are suggestive but don't establish independence; many structured ensembles have MP-like bulk spectra. Appendix P only considers uniform correlation, not the low-rank or block-structured dependencies you'd actually expect from gradient-based fine-tuning. So the expressivity advantage is proven only for a model that may not match real updates. The empirical results stand on their own, but they don't validate the theorem.\n\nIs it worth engaging? Yes, for the method and the experiments. The authors are thoughtful and don't hide the limitations. A serious referee should ask them to either produce a full proof of (15) or clearly label it as a conjecture, and to discuss whether the uniform-correlation study actually addresses the concern. Also ask for code. If I were an editor, I'd send it to review and expect a revision, not a desk reject.","headline":"Worth engaging for the method and experiments, but the central theory rests on an i.i.d. Gaussian assumption that is assumed more than shown.","tokens_in":41087,"tokens_out":3904,"would_cite":true,"duration_ms":33140,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","60B20","62E20"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proves that a sparse, deliberately chosen set of frequency components can reconstruct fine-tuning weight updates more accurately than a low-rank decomposition of the same size.","keywords":["parameter-efficient fine-tuning","low-rank adaptation","discrete cosine transform","frequency-domain adaptation","location selection","large language models","Gaussian weight updates","reconstruction expressivity"],"falsifier":"Take a real update matrix $\\Delta W$ from a fine-tuned model, such as a query or value projection in LLaMA fine-tuned on Alpaca, and compare the paper's reconstruction ordering directly: for each layer, compute the expected squared Frobenius error of the best rank-$r$ approximation and of keeping the $r$-largest DCT coefficients under the same parameter accounting. If for $r<K/3$ the low-rank error is smaller on the majority of layers, or if the eigenvalue mass of the sample correlation matrix outside the Marchenko-Pastur bounds exceeds the paper's critical values, the Gaussian-model assumption fails in a way that invalidates the ordering.","tokens_in":40007,"feed_emoji":"🎯","tokens_out":8200,"duration_ms":75624,"temperature":0.7,"pith_summary":"This paper tries to establish that frequency-domain parameter-efficient fine-tuning can be strictly more expressive than low-rank adaptation when the learnable frequency components are chosen deliberately. The theoretical engine is a Gaussian model of fine-tuning weight updates: the paper argues that in large trained models the update matrix $\\Delta W$ behaves like a matrix of independent, identically distributed Gaussian entries, and under that model it proves a strict ordering of expected reconstruction errors. Randomly placed Fourier coefficients reconstruct worse than a rank-$r$ approximation, but amplitude-selected Fourier coefficients reconstruct better, and selecting real or imaginary coefficients individually is best of all. An inverse discrete cosine transform (iDCT) reproduces that best case at lower cost, and the proposed LoCA method uses finite-difference location gradients to search for good coefficient locations during training. If the Gaussian model of weight updates is right, the practical payoff is that the same task performance can be obtained with fewer trainable parameters than low-rank adapters.","feed_headline":"Frequency beats low-rank for fine-tuning at equal budget","feed_subtitle":"A cosine-domain PEFT method picks the most informative coefficients during training and beats LoRA and FourierFT with fewer parameters.","key_machinery":"The load-bearing object is the sparse inverse discrete cosine transform reparameterization $\\Delta W = \\alpha C^T S(a,l,1) D$, where $C$ and $D$ are DCT matrices, $S$ scatters $B$ learnable coefficients $a$ onto locations $l$ of the DCT spectrum, and $\\alpha$ scales the update. The theoretical machinery behind it is the Gaussian update model: Proposition 1 models the vectorized update as asymptotically i.i.d. normal, which turns reconstruction quality into a pure random-matrix calculation. Theorem 1 then compares expected residual energies: the best low-rank approximation keeps the $r$ largest singular values, while frequency methods keep the largest Fourier or DCT coefficients, and the paper shows that for i.i.d. Gaussian data the latter carries more expected energy per parameter. Theorem 2 identifies iDCT as an exact real-valued implementation of the individually optimal Fourier coefficient selection. To make locations learnable, the paper uses a central-difference surrogate gradient that measures how much the reconstruction changes when a coefficient is shifted by one bin, and an alternating coefficients-then-locations optimization finds the informative part of the spectrum early in training.","core_discovery":"The paper's central claim is that, with an equal number of parameters, a sparse representation of a weight update in a frequency basis can have lower expected reconstruction error than the best low-rank representation, provided the frequency components are selected by magnitude rather than at random. Concretely, Theorem 1 states that for a $K\\times K$ weight matrix with i.i.d. standard normal entries and rank $r<K/3$, the expected squared Frobenius error of random Fourier coefficient selection exceeds that of low-rank approximation, which in turn exceeds that of selecting the largest-amplitude Fourier locations, which in turn exceeds that of selecting the largest individual real and imaginary Fourier coefficients. Theorem 2 states that selecting the largest coefficients of the discrete cosine transform attains exactly the same expected error as that individually optimal Fourier selection, while working entirely with real numbers. The proposed LoCA method implements this insight by reparameterizing $\\Delta W$ through iDCT, learning both the coefficients and the discrete locations, with location gradients estimated by central finite differences and an alternating optimization schedule. The method, tested on GLUE, E2E, instruction tuning, and image classification, matches or beats low-rank baselines while using far fewer parameters, and consistently beats FourierFT at equal budget.","pith_inferences":["Editorial inference: the theorem's ordering depends on updates being diffuse Gaussian noise; on tasks where $\\Delta W$ is genuinely low-rank or strongly correlated, LoRA may retain its edge, and the method's advantage is most likely for high-dimensional, noisy updates.","Editorial inference: the finite-difference location gradient is a general mechanism that amounts to sparse support search in any orthogonal basis, so applying the same selection scheme to wavelets or learned dictionaries would test whether DCT is special or merely convenient.","Editorial inference: the paper's appendix correlation analysis suggests the ordering tolerates mild uniform correlations among update entries; a stronger test would measure the actual covariance structure of $\\Delta W$ across layers and check whether its eigenvalue mass outside the Marchenko-Pastur interval exceeds the identified critical levels.","Editorial inference: if frequency selection is superior for i.i.d. updates, parameter budgets could be allocated adaptively per layer by the location-gradient signal, concentrating components where the spectrum is richest."],"forward_implications":["For the same parameter count, a sparse frequency-domain update can represent a fully fine-tuned $\\Delta W$ more faithfully than a low-rank update, so PEFT budgets can be smaller without sacrificing expressivity.","Under the paper's Gaussian model, random frequency-component placement is provably weaker than low-rank at the same budget; selection must be deliberate.","Because DCT selection matches optimal Fourier coefficient selection, real-valued iDCT is the practical way to realize the expressivity advantage without complex-valued optimization.","Dynamic location search during training offers a way to allocate the parameter budget to the spectrum's most informative bins, adapting per weight matrix.","On language and vision benchmarks, retaining the same component budget yields performance at or above LoRA-family baselines with a fraction of the parameters."],"supporting_citations":[{"why":"Defines LoRA, the low-rank decomposition baseline whose expected reconstruction error Theorem 1 is compared against.","marker":"(Hu et al., 2021)"},{"why":"Introduces FourierFT, the random-location frequency-domain method that the paper reanalyzes and improves on.","marker":"(Gao et al., 2024)"},{"why":"Provides the discrete cosine transform definition underlying the iDCT reparameterization.","marker":"(Ahmed et al., 1974)"},{"why":"Gives the asymptotic normality of M-estimators used to justify Proposition 1's Gaussian weight-update model.","marker":"(Yohai & Maronna, 1979)"},{"why":"Supplies the Marchenko-Pastur law used to test whether $\\Delta W$ behaves like an i.i.d. random matrix.","marker":"(Yang et al., 2012)"},{"why":"Provide the Tracy-Widom and large-deviation results that bound the expected largest singular value in the low-rank error calculation.","marker":"(Vivo et al., 2007; Johnstone, 2001; Johansson, 2000)"},{"why":"Supplies the Wishart density constant used in the eigenvalue-integral proof of Theorem 1.","marker":"(Muirhead, 2009)"}],"fun_headline_variants":["LoCA: DCT-selected frequencies outsmart LoRA with fewer params","Sparse DCT beats low-rank: LoCA's parameter-efficient edge","LoCA: Pick the right cosine coefficients, beat low-rank","Frequency selection wins: LoCA outperforms low-rank in PEFT","LoCA: Cosine-domain adaptation trumps low-rank with less"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ordering rests on treating a fine-tuning weight update as a matrix of independent, identically distributed Gaussian entries; if real updates carry low-rank or correlated structure, the frequency advantage could shrink or reverse.","fun_headline_variants_meta":{"raw":{"variants":["LoCA: DCT-selected frequencies outsmart LoRA with fewer params","Sparse DCT beats low-rank: LoCA's parameter-efficient edge","LoCA: Pick the right cosine coefficients, beat low-rank","Frequency selection wins: LoCA outperforms low-rank in PEFT","LoCA: Cosine-domain adaptation trumps low-rank with less"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00065,"raw_usage":{"total_tokens":3012,"prompt_tokens":1005,"completion_tokens":2007,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":621,"completion_tokens_details":{"reasoning_tokens":1915}},"tokens_in":621,"tokens_out":2007,"duration_ms":16146,"temperature":1.0,"reasoning_tokens":1915,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T10:47:44.632868+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a real update matrix $\\Delta W$ from a fine-tuned model, such as a query or value projection in LLaMA fine-tuned on Alpaca, and compare the paper's reconstruction ordering directly: for each layer, compute the expected squared Frobenius error of the best rank-$r$ approximation and of keeping the $r$-largest DCT coefficients under the same parameter accounting. If for $r<K/3$ the low-rank error is smaller on the majority of layers, or if the eigenvalue mass of the sample correlation matrix outside the Marchenko-Pastur bounds exceeds the paper's critical values, the Gaussian-model assumption fails in a way that invalidates the ordering.","supporting_citations":[],"review_version":1}