{"id":"8d2db70c-79ea-4740-af46-72f4af83b95b","arxiv_id":"2505.00580","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new PEFT method factorizes weight updates into interleaved diagonal and circulant matrices, trained with 1D FFTs, achieving similar accuracy to FourierFT with drastically fewer FLOPs and parameters.","lead":"This paper introduces CDVFT, a way to fine-tune large models by representing the weight update as a product of diagonal and circulant vectors trained with fast Fourier transforms. It cuts computation and trainable parameters versus FourierFT while keeping accuracy on GLUE, image classification, and instruction-tuning benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The unsupported m=2 factorization is the load-bearing assumption; an m-ablation would settle whether the FLOP/parameter savings actually preserve accuracy.","rationale":"The central claim is an empirical Pareto claim: CDVFT matches or beats LoRA/FourierFT accuracy while using 55K parameters and 27M FLOPs. That operating point is entirely a consequence of choosing m=2 and p=d. The Huhtanen-Peramaki factorization theorem cited as motivation only guarantees that up to 2n-1 factors can represent any square matrix; it provides no support for m=2. The experimental section fixes m=2 throughout, and the only stated justification is an informal assertion of sufficiency, with no ablation. The reader's weakest_assumption correctly identifies this as the most load-bearing gap. An m-sweep on the actual benchmarks would directly test it. I also note a secondary numerical inconsistency: Table 2 lists CDVFT ViT FLOPs as 2.72M, while the text's 51.9x ratio against 1.41G and Fig. 2 imply 27.2M. This does not change the verdict but reinforces the need for the requested checks. Since the reader already returns CONDITIONAL for addressable gaps, my read does not change the verdict.","tokens_in":13451,"tokens_out":16161,"duration_ms":162241,"concrete_test":"Run the RoBERTa-base GLUE suite and the ViT-base CIFAR-100/RESISC45 experiments with m in {1, 2, 3, 4}, keeping p=d and all other hyperparameters fixed, and report mean +/- std over 5 seeds for every dataset. The concern is settled if the m=2 average is within one standard deviation of the best m on each dataset and no task shows a sharp drop; finding a task where m=2 is substantially worse than a larger m would invalidate the fixed-m=2 efficiency claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 constructs delta W as a product of interleaved diagonal and circulant matrices (Eq. 3) and fixes m=2, i.e. three factors, in every experiment, with p=d for RoBERTa and ViT. This choice is load-bearing because the entire efficiency claim depends on m being small: the parameter count is (2m-1)*d per matrix and the FLOP cost is O(m*d*log d), so the advertised 55K-parameter, 27M-FLOP operating point exists only if m=2 is already expressive enough. The paper offers no ablation over m, no approximation-error analysis, and no bound from the cited Huhtanen-Peramaki theorem, which only guarantees representability with up to 2n-1 factors. The only justification is the informal statement in Section 1 that 'a few matrices and a small number of blocks is sufficient', and Fig. 1's caption even says m=1 while Section 4.1 says m=2, compounding the ambiguity. If some tasks require more factors, the FLOP savings shrink and the Pareto claim weakens; if m=2 is sufficient, the concern is resolved.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CDVFT, a parameter-efficient fine-tuning method that represents the weight update ΔW as a product of interleaved diagonal and circulant matrices and computes the adapter output with 1D FFT operations, without materializing ΔW during training. Non-square weight matrices are handled by partitioning the circulant factor into blocks. The method is evaluated on GLUE with RoBERTa-base, on CIFAR-100 and RESISC45 with ViT-base, and on Alpaca and GSM8K with LLaMA2-7B, using m=2 in all experiments. The reported results show accuracy comparable to LoRA and FourierFT while using fewer trainable parameters than LoRA and substantially fewer FLOPs than FourierFT.","tokens_in":13688,"tokens_out":14404,"duration_ms":141326,"significance":"If the fixed m=2 factorization is indeed sufficient across tasks, CDVFT is a useful contribution to the PEFT efficiency frontier: it achieves parameter counts close to FourierFT with much lower FLOPs and no accuracy loss on the tested benchmarks. The paper's strengths include the clean 1D-FFT formulation, the backward-pass rules in Eq. (11) that reuse forward FFTs via conjugation, and the breadth of evaluation across NLU, vision, and instruction tuning. The main open question is whether the small number of factors is an expressivity bottleneck; the paper currently provides no systematic evidence on this point, which is load-bearing for the central efficiency claim.","major_comments":[{"comment":"The efficiency claims in the abstract and Fig. 2 rest on fixing m=2, but the paper provides no ablation over m or over the block size p. The parameter count is (2m−1)d per square adapter and the FLOPs scale as O(m d log d), so the advertised 55K-parameter/27M-FLOP operating point is only meaningful if m=2 is expressive enough on all tested tasks. The cited Huhtanen–Perämäki theorem guarantees factorization with up to 2n−1 factors, not that two diagonal and one circulant factor suffice, and no approximation-error bound for m=2 is given. Please add an ablation on at least one GLUE task and one image task sweeping m (e.g., 1, 2, 3, 4) together with the corresponding parameter and FLOP counts, and sweep p in the same ablation. Without this, the Pareto comparison to LoRA/FourierFT is incomplete.","section":"Section 3.1/Eq. (3) and Section 4.1"},{"comment":"The caption states 'In practice, we find m = 1 (no loops required) can effectively fine-tune the model, where there are two diagonal matrices and one circulant matrix.' This contradicts Section 4.1, which sets 'a total of 3 factor matrices, i.e., m = 2,' and also contradicts Eq. (3), under which m=1 gives a single diagonal factor and no circulant matrix. The definition of m must be made consistent across the figure, the equations, and the experimental sections.","section":"Fig. 1 caption and Section 4.1"},{"comment":"The FLOPs values that support the '51.81× FLOPs reduction' claim are presented without a counting convention. It is not specified whether the numbers are per sample, per optimizer step, forward-only or forward+backward, or whether FFT operations are counted as complex or real operations. Please provide the exact formulas used for each method so the headline complexity comparison can be reproduced.","section":"Section 3.4 and Figs. 2/Table 3"},{"comment":"The factorization theorem of Huhtanen and Perämäki is quoted for complex matrices, while CDVFT trains real diagonal and circulant vectors. The paper does not explain whether the same 2n−1 factor bound holds over the reals or whether the theorem's factors can be chosen real for a real target matrix. Please either supply a real-matrix statement of the theorem or weaken the claim that the construction 'theoretically can approximate any dense matrix.'","section":"Section 2 and Section 3.1"}],"minor_comments":[{"comment":"The sentence 'we apply block circulant fine-tuning on query and value weight matrices inside the attention layer of two RoBERTa models and the LLaMA2-7B model fine-tuned on the alpaca dataset' appears to be a copy-paste error; the experiments use one RoBERTa model for GLUE and LLaMA2-7B for instruction tuning, and the sentence should be rewritten.","section":"Section 4.3, Implementation details"},{"comment":"Eq. (2) defines the circulant matrix as circ(a_{2j}) while the surrounding text and Eq. (3) use c_{2j} for circulant vectors; please unify the notation.","section":"Eq. (2)"},{"comment":"The text says 'we fine-tune on several challenging image classification datasets, only two are listed here for observation,' but Table 2 reports only two datasets; if additional datasets were evaluated, they should be reported, and if not, the sentence should be corrected.","section":"Section 4.2"},{"comment":"The phrase 'LaMDA++ is lack of rank information' should be 'LaMDA++ lacks rank information.'","section":"Table 3 caption"},{"comment":"The sentence containing '(Priority is primarily given to cases where the weight matrix is square.)' is incomplete or misplaced; please clean up this sentence.","section":"Section 3.4"},{"comment":"The block-partition description says matrices are 'automatically padded through replication'; please specify exactly how rows and columns are padded when d1 or d2 is not divisible by p, and whether padding affects the eventual merge into the pretrained weights.","section":"Section 3.3"},{"comment":"There is a typo on the first page: 'fullfill' should be 'fulfill.'","section":"Introduction"}],"recommendation":"major_revision","confidential_remarks":"The main risk is not the correctness of the linear-algebra derivations but whether the fixed m=2 structure is broadly expressive. An m-ablation would settle this; if the authors cannot provide one, the empirical claims should be narrowed. Making the FLOPs counting explicit and, ideally, releasing code would also materially strengthen the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid, incremental efficiency improvement over FourierFT for parameter-efficient fine-tuning. The math is correct and the FLOP numbers are plausible; the main thing that would make me fully comfortable is a systematic study of the number of factors m, which is the load-bearing assumption behind the efficiency claim.\n\nWhat's actually new: representing the weight update as a product of interleaved diagonal and circulant matrices, training directly in vector form with 1D FFTs, and adding a block partition for non-square matrices. That is a real and sensible extension of FourierFT. The forward and backward derivations (Eqs. 8-11) check out, and the trick of reusing forward FFTs in the backward step via conjugation is elegant. For m=2 you get 55K parameters and 27M FLOPs on RoBERTa-base, about 51.8x fewer FLOPs than FourierFT, and the reported accuracies across GLUE, ViT, and LLaMA are on par with LoRA and FourierFT. So the efficiency appears not to cost accuracy on the tested benchmarks.\n\nThe soft spots are concentrated in one place: m=2 is asserted, not validated. The Huhtanen-Peramaki theorem is a representability result with up to 2n-1 factors; it says nothing about whether three factors suffice for fine-tuning updates. The paper says 'we find only a few matrices ... is sufficient' but there is no ablation over m anywhere. Fig. 1's caption says m=1 while Section 4.1 says m=2, so the paper is internally inconsistent on this exact point. If some tasks need a larger m, the FLOP advantage shrinks linearly and the Pareto story weakens. This is easily fixable: run GLUE with m=1,2,3 and show where accuracy plateaus.\n\nMinor issues: no code release, and variance is reported only for GLUE. For a methods paper in this area, code is close to expected. There are also a few typos ('fullfill', 'moder performance') that suggest a rushed draft. None of these change the core result.\n\nBottom line: the derivation is careful, the experiments are honest, and the missing m-ablation is an addressable gap, not a fatal flaw. I'd send this to a serious referee, with the request that the revision include an m-ablation, code, and error bars for the non-GLUE tables. I'd also cite this as a representative example of structured matrix PEFT.\n\nRecommendation: worth engaging with as a conditional accept.","headline":"A solid incremental PEFT method with correct math and real FLOP savings, but the fixed m=2 factorization is asserted rather than validated.","tokens_in":14235,"tokens_out":3929,"would_cite":true,"duration_ms":37235,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning with circulant and diagonal vectors matches existing adapters while sharply reducing FLOPs and trainable parameters.","keywords":["parameter-efficient fine-tuning","circulant matrices","diagonal matrices","fast Fourier transform","low-rank adaptation","Fourier domain methods","block circulant partition","foundation models"],"falsifier":"Fine-tune RoBERTa-base on the six GLUE tasks with CDVFT at $m=2$ and at $m=3$ (adding one more circulant-diagonal pair), holding all other settings fixed; if $m=3$ improves any task's score by more than the reported standard deviation, the sufficiency claim for $m=2$ is refuted on that task.","tokens_in":13249,"feed_emoji":"⚡","tokens_out":6995,"duration_ms":57476,"temperature":0.7,"pith_summary":"The paper proposes a parameter-efficient fine-tuning method, CDVFT, that represents the weight-change matrix as a product of interleaved circulant and diagonal matrices instead of a low-rank or dense Fourier transform. Because a circulant matrix is defined by a single vector and its product with a vector can be computed with one-dimensional FFTs, the method never needs to build the dense update matrix. On GLUE with RoBERTa-base it reports a 51.81× reduction in FLOPs versus FourierFT and 5.33× fewer trainable parameters than LoRA, with an average score of 85.1 versus LoRA's 85.2. The same recipe transfers to ViT image classification and LLaMA2-7B instruction tuning, using block partitioning to handle non-square weight matrices.","feed_headline":"Circulant vectors slash fine-tuning cost 50x, keep accuracy","feed_subtitle":"Weight updates as circulant-diagonal products match LoRA accuracy with 5.33x fewer parameters.","key_machinery":"The central identity is that a circulant matrix-vector product equals $\\mathrm{IFFT}(\\mathrm{FFT}(\\mathbf{c}) \\odot \\mathrm{FFT}(\\mathbf{x}))$, where $\\mathbf{c}$ is the vector defining the circulant matrix. CDVFT iterates this identity with diagonal (elementwise) multiplications, so the whole adapter is a sequence of vector operations; the backward pass reuses forward FFTs via conjugation, $\\mathrm{IFFT}(\\mathrm{conj}(\\mathbf{F}) \\odot \\mathbf{F}_y)$, avoiding extra transforms. The block-partition scheme extends the square-only circulant construction to rectangular weight matrices by splitting them into square circulant blocks.","core_discovery":"Building on the theorem that any complex square matrix can be expressed as a product of at most $2n-1$ interleaved circulant and diagonal matrices, CDVFT factorizes the weight update $\\Delta W$ as $\\mathrm{diag}(\\mathbf{a}_{2m-1}) \\times \\mathrm{circ}(\\mathbf{a}_{2m-2}) \\times \\cdots \\times \\mathrm{diag}(\\mathbf{a}_1)$. In practice it sets $m=2$, i.e., two diagonal matrices and one circulant matrix, and computes the adapter output by an alternating chain of elementwise multiplications and 1D FFTs, so the quadratic cost of constructing and applying $\\Delta W$ becomes loglinear in the dimension. For non-square weights, the circulant factor is partitioned into square blocks, each block itself a circulant matrix. The paper claims this yields accuracy comparable to LoRA and FourierFT while using far fewer trainable parameters than LoRA and far fewer FLOPs than FourierFT across natural language understanding, image classification, and instruction tuning.","pith_inferences":["The factorization family suggests a natural expressivity-cost ladder: increasing $m$ from 1 to larger values approaches arbitrary dense updates, so CDVFT could serve as a bridge between extremely cheap structured adapters and full fine-tuning.","One testable extension is to apply CDVFT to other structured-matrix problems, such as compressing attention projections or convolution kernels, where circulant structure is a known inductive bias.","The paper's observation that larger block sizes sometimes do not hurt accuracy hints that the circulant-diagonal structure itself may act as a regularizer; a systematic study of this effect would separate regularization from capacity in the accuracy numbers."],"forward_implications":["Fine-tuning memory and compute scale with vector length $d$ and factor count $m$ rather than with the full weight-matrix dimension, making adaptation of larger models more feasible on limited hardware.","The $m=2$ configuration (diagonal-circulant-diagonal) is claimed to be sufficient for task-level accuracy on the evaluated benchmarks, so practitioners can use CDVFT without tuning the number of factors.","Block partitioning with larger block sizes $p$ reduces parameter count and FLOPs, giving a direct knob to trade cost against accuracy on non-square layers such as those in LLaMA2-7B.","Because the trained factors are vectors, the final update $\\Delta W$ can be reconstructed and merged into the frozen weights after training, so inference latency is unchanged."],"supporting_citations":[{"why":"Supplies the factorization theorem that any complex matrix is a product of interleaved circulant and diagonal matrices, the theoretical basis for the CDVFT parameterization.","marker":"Huhtanen and Perämäki 2015"},{"why":"The FourierFT baseline whose 2D-FFT reconstruction CDVFT replaces with 1D FFTs, and whose experimental settings CDVFT follows.","marker":"Gao et al. 2024a"},{"why":"The LoRA baseline that CDVFT compares against in parameter count and accuracy, and the source of the scaling constant $\\alpha$ and mergeable-adapter design.","marker":"Hu et al. 2021"},{"why":"Provides the circulant-matrix FFT derivative formulas and the fixed shift pattern that CDVFT's backward pass improves via conjugation.","marker":"Cheng et al. 2015"},{"why":"Introduces the block-circulant partitioning strategy that CDVFT adapts to handle non-square weight matrices.","marker":"Ding et al. 2017"},{"why":"The VeRA baseline that shares random low-rank matrices across layers; CDVFT compares its parameter/FLOP trade-off against it.","marker":"Kopiczko et al. 2024"}],"fun_headline_variants":["Circulant-diagonal fine-tuning matches LoRA with far fewer params","Fine-tuning with circulant and diagonal vectors: less cost, same accuracy","Efficient fine-tuning via circulant-diagonal factorization","Reduce fine-tuning cost with circulant-diagonal weight updates","Circulant-diagonal adapter rivals LoRA with fewer parameters"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's accuracy rests on the untested assumption that a single circulant matrix sandwiched between two diagonal matrices ($m=2$) is expressive enough to capture the useful weight update on any given task, with no ablation over $m$ provided.","fun_headline_variants_meta":{"raw":{"variants":["Circulant-diagonal fine-tuning matches LoRA with far fewer params","Fine-tuning with circulant and diagonal vectors: less cost, same accuracy","Efficient fine-tuning via circulant-diagonal factorization","Reduce fine-tuning cost with circulant-diagonal weight updates","Circulant-diagonal adapter rivals LoRA with fewer parameters"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000924,"raw_usage":{"total_tokens":3941,"prompt_tokens":904,"completion_tokens":3037,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":520,"completion_tokens_details":{"reasoning_tokens":2950}},"tokens_in":520,"tokens_out":3037,"duration_ms":22858,"temperature":1.0,"reasoning_tokens":2950,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:38:39.337952+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fine-tune RoBERTa-base on the six GLUE tasks with CDVFT at $m=2$ and at $m=3$ (adding one more circulant-diagonal pair), holding all other settings fixed; if $m=3$ improves any task's score by more than the reported standard deviation, the sufficiency claim for $m=2$ is refuted on that task.","supporting_citations":[{"cited_title":"Factoring matrices into the product of circulant and diagonal matrices","cited_arxiv_id":null,"evidence_quote":"Supplies the factorization theorem that any complex matrix is a product of interleaved circulant and diagonal matrices, the theoretical basis for the CDVFT parameterization."}],"review_version":1}