{"id":"f3ad0bbc-1ce4-4a11-b033-08423803e4f5","arxiv_id":"2501.12067","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"EDoRA combines DoRA's magnitude-direction decomposition with LoRA-XS's frozen SVD-based sandwich parameterization, cutting trainable parameters by roughly 30x versus LoRA and DoRA on GLUE with RoBERTa-base.","lead":"The paper proposes EDoRA, a parameter-efficient fine-tuning method that freezes SVD-initialized low-rank matrices and trains a small matrix plus a magnitude vector between them. On GLUE with RoBERTa-base, it reports competitive or better accuracy than LoRA and DoRA while using up to 30x fewer trainable parameters.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported accuracy advantage over LoRA/DoRA may be an artifact of unmatched baseline tuning and rank-selection noise; no baseline hyperparameters or significance testing are reported.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the empirical superiority claim depends on baselines having been tuned comparably, and the paper does not report the baseline tuning details or any significance testing. I agree with that assessment. The concern is concrete rather than fatal: it could be resolved by a controlled reproduction with a matched grid search and proper uncertainty quantification. Other potential issues, such as the fact that EDoRA still has an O(n) magnitude vector or that its frozen SVD matrices occupy memory during training, do not directly undermine the paper's explicit claim about trainable-parameter counts and are therefore secondary. The paper does provide useful independent evidence: the SVD-initialization ablation (Table 3) and the comparison with LoRA-XS at matched parameter budgets support the internal design choices. But those do not address the fairness of the headline comparison against LoRA and DoRA. Because the concern is about experimental controls rather than a fundamental flaw, the appropriate verdict remains conditional, matching the reader's original verdict.","tokens_in":13053,"tokens_out":14381,"duration_ms":153078,"concrete_test":"Using the released code, run a matched grid search for LoRA, DoRA, LoRA-XS, and EDoRA over the same learning-rate set (e.g., {1e-4, 5e-4, 1e-3, 5e-3}) and the same epoch counts (e.g., {20, 50}), for ranks 8, 16, and 32, on all six GLUE tasks with the same five seeds. Select the best configuration per method on a validation split, then compute a paired bootstrap or t-test comparing EDoRA's chosen model against each baseline's chosen model on the test metrics. If the 84.48 versus 83.66/83.78 gap is not significant or reverses, the superiority claim fails; if it persists under matched tuning, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that EDoRA is competitive or superior at 30x fewer parameters rests on the cross-method comparison in Table 2, and that comparison is not currently controlled. The paper states that hyperparameters were optimized through grid search, but Table 4 reports only EDoRA's per-rank learning rates and classifier learning rates; the LoRA, DoRA, and LoRA-XS configurations and search budgets are omitted. Because the small-data GLUE tasks (RTE, COLA, MRPC) are highly sensitive to learning rate and epoch count, and because EDoRA uses rates up to 5e-3 with 50 epochs on several tasks, an undertuned baseline could easily account for the 0.82-point average gap (84.48 vs 83.66 for LoRA and 83.78 for DoRA). The paper also selects the best result for each method after looking across all ranks (LoRA best at rank 20, DoRA at rank 20, EDoRA at rank 32) without a held-out model-selection protocol or significance testing. The reported per-task standard deviations are large relative to several per-task gaps (e.g., RTE 1.37, MRPC 1.53), so the observed differences may be within seed noise. These are addressable experimental-reporting issues, but until baselines are given the same tuning protocol and uncertainty is quantified, the central empirical claim is not settled.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes EDoRA, a parameter-efficient fine-tuning (PEFT) method that builds on the weight decomposition of DoRA. EDoRA decomposes a pre-trained weight matrix into a trainable magnitude vector and a directional component, freezes two low-rank matrices A and B obtained from a truncated SVD of the directional matrix, and introduces a small trainable square matrix R between them. The number of trainable parameters is n + r^2 per n x n weight matrix (magnitude vector plus R), which is far smaller than LoRA's 2nr or DoRA's n + 2nr. Experiments on six GLUE tasks with RoBERTa-base compare EDoRA with LoRA, DoRA, and LoRA-XS, reporting that EDoRA achieves the highest average accuracy (84.48%) at rank 32 while using 86.0K trainable parameters, versus LoRA's best 83.66% at 2.0M parameters and DoRA's best 83.78% at 2.1M parameters. An ablation study shows that SVD initialization improves average accuracy by about 1.5 points over random initialization.","tokens_in":13346,"tokens_out":8774,"duration_ms":78105,"significance":"If the empirical results are confirmed, EDoRA is a meaningful improvement in parameter efficiency: it reduces trainable parameters by roughly one to two orders of magnitude relative to LoRA and DoRA while preserving accuracy on the GLUE benchmark. The parameter-efficiency analysis in Section 3.1 is straightforward and correct, and the SVD-initialization ablation in Table 3 is a clean, self-contained comparison that supports the design choice. The authors provide code, which facilitates reproducibility. The main limitation is that the cross-method comparison in Table 2 is not fully controlled: baseline hyperparameters are absent, no significance tests are reported, and the rank-selection protocol is unclear. These are addressable issues, and the central efficiency claim (parameter reduction) does not depend on the small accuracy differences; the claim of 'competitive or superior' accuracy would be strengthened by rigorous experimental reporting.","major_comments":[{"comment":"The empirical comparison is not fully controlled because hyperparameters for the baselines are not reported. The text states that 'Hyperparameters were optimized through grid search, and the selected values are presented in Table 4,' but Table 4 lists only EDoRA's per-rank learning rates and classifier learning rates. Without the LoRA, DoRA, and LoRA-XS configurations and search budgets, it is impossible to rule out that the baselines were undertuned. On the small-data tasks (RTE, CoLA, MRPC), performance is highly sensitive to learning rate and epoch count, and EDoRA uses rates up to 5e-3 with 50 epochs on several tasks. Please report all baseline hyperparameters (or provide them via a public configuration file) and a description of the shared tuning protocol.","section":"§4.1, Table 2"},{"comment":"The claimed accuracy advantage is not accompanied by uncertainty quantification across the five seeds. The reported per-task standard deviations are large relative to several differences (e.g., EDoRA's RTE is 78.70±1.37 versus DoRA's 76.17±1.24, and EDoRA's MRPC 88.97±1.53 versus LoRA's 89.46±1.08), and the average advantage over DoRA is 0.70 points (84.48 vs 83.78). No significance tests or confidence intervals are reported, so it is unclear whether the cross-method differences are meaningful given seed noise. Report the mean and standard deviation of the per-method average over the five seeds, and perform paired tests (e.g., a paired t-test or Wilcoxon test) between EDoRA and each baseline, or provide effect sizes.","section":"§4.2, Table 2"},{"comment":"The initialization of the magnitude vector m is not specified. In DoRA, m is initialized as the column-wise norm of W0, which is necessary for the initial model to equal the pretrained network. EDoRA states that 'Initializing A, B, and R as mentioned, ensures that the learning process starts with a model nearly identical to the pre-trained model,' but this is only true if m is also set appropriately. Without this detail, the method is not fully reproducible and the initialization claim is not substantiated. Please specify the initialization of m and confirm whether it is trainable.","section":"§3, Eq. (4)"},{"comment":"The paper does not state whether the reported GLUE numbers are development-set or test-set results, nor does it describe the model-selection protocol for choosing the rank. Table 2 presents results for ranks 4–32 for each method, and the text selects the best rank per method (e.g., LoRA rank 20, DoRA rank 20, EDoRA rank 32) without a held-out validation procedure. If ranks were selected based on the same numbers that are then compared, the comparison is subject to post-hoc selection effects. Please clarify the data split used and specify how the best rank was chosen (e.g., by dev accuracy).","section":"§4.1–4.2"}],"minor_comments":[{"comment":"The sentence 'Parameter-efficient fine-tuning methods, such as LoRA, reduces the number of trainable parameters' contains a subject-verb agreement error; 'reduces' should be 'reduce'.","section":"Abstract"},{"comment":"The claim of 'over 45x fewer trainable parameters compared to LoRA and DoRA when applied to the GPT-3 model' is not directly supported by Table 1, which shows a 31.35x/32.33x reduction at r=16 and 59.08x/60.00x at r=32; please clarify the rank or specify that this is an average over the reported ranks.","section":"§1, Contributions"},{"comment":"The sentence 'The author showed that SPOT substantially improves...' should be 'The authors showed...' because the reference is to Vu et al.","section":"§2.1"},{"comment":"The text 'Zhang et. al (Zhang et al., 2023) introduced AdaLoRA in 2023' has a formatting issue ('et. al' should be 'et al.') and the phrase 'in 2023' is redundant given the citation year.","section":"§2.3"},{"comment":"Please report the optimizer, batch size, sequence length, warmup steps, and exact model checkpoint (e.g., 'roberta-base' from HuggingFace), as these details affect reproducibility even when code is available.","section":"§4.1"},{"comment":"The parameter-efficiency ratios are derived for a square weight matrix W ∈ R^{n×n}; for non-square weight matrices the ratios depend on the aspect ratio. Please state this generalization or clarify that the analysis is for a simplified square case.","section":"§3.1, Eqs. (7)–(8)"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for the journal, but the empirical claims need strengthening. The missing baseline hyperparameters are the most significant issue; the reader's report's skepticism on this point is justified. The method's novelty is incremental relative to LoRA-XS and DoRA, but the parameter reduction is substantial and the SVD ablation is a positive feature. If the authors can provide the missing experimental details and significance tests, the paper could be acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"EDoRA is DoRA with LoRA-XS's frozen SVD sandwich swapped in for LoRA's trainable A/B. That is a small but sensible combination, and the paper's real contribution is the parameter-efficiency analysis and the SVD-initialization ablation, both of which hold up. The math in Section 3.1 is correct: with hidden dimension n and rank r, EDoRA trains n + r^2 parameters per weight matrix, versus 2nr for LoRA and n + 2nr for DoRA, so the 30-60x reductions in Table 1 are real. The ablation in Table 3 is genuine evidence: SVD initialization beats random initialization at every rank, by about 1.5 points on average. Code is released.\n\nThe weak spot is the headline empirical claim. The GLUE comparison in Table 2 is not controlled. The paper says hyperparameters were tuned by grid search, but only EDoRA's configurations are reported; LoRA, DoRA, and LoRA-XS get no learning rates, epochs, or search budgets. The five-seed medians come with per-task standard deviations of 1-2.5 points, which are large relative to the cross-method gaps the paper emphasizes (RTE gap is 2.5 points, MRPC 0.5). There is no significance test, and the best rank for each method is selected after looking at all ranks, with no held-out protocol. The LoRA-XS rows are also confusing: the table shows only rank 32 while the text and Figure 3 imply a range. None of this is fatal, but until baselines are retuned under the same protocol and uncertainty is quantified, the 'competitive or superior at 30x fewer parameters' claim is not settled.\n\nThe learning-dynamics narrative is borrowed from DoRA and asserted, not measured. I would not lean on it, but it is motivation, not the core result.\n\nFor whom: PEFT people who want a simple, cheap variant of DoRA will find this useful. It deserves a serious referee, because the method is clearly specified, the efficiency analysis is correct, and the experimental issues are addressable. My recommendation: send it to peer review, but require the authors to report baseline hyperparameters and run significance tests or confidence intervals on the GLUE comparisons.","headline":"Plausible, clearly specified PEFT combination; the parameter-count win is real, but the GLUE superiority claim needs baseline transparency and significance testing.","tokens_in":13859,"tokens_out":3316,"would_cite":true,"duration_ms":32432,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that EDoRA, by freezing SVD-initialized low-rank matrices and training only a small square matrix between them, can match or beat LoRA and DoRA on GLUE with up to 30x fewer trainable parameters.","keywords":["EDoRA","parameter-efficient fine-tuning","low-rank adaptation","singular value decomposition","weight decomposition","magnitude-direction decomposition","GLUE benchmark","RoBERTa"],"falsifier":"Re-run the six GLUE tasks with identical hyperparameter search budgets and all five seeds for every method, then compare EDoRA's average against the best baseline; if the gap falls within the run-to-run spread reported in Table 2, the claimed advantage is not established.","tokens_in":12867,"feed_emoji":"📉","tokens_out":9271,"duration_ms":82870,"temperature":0.7,"pith_summary":"EDoRA is a parameter-efficient fine-tuning method that aims to reproduce the learning behavior of full fine-tuning while using far fewer trainable parameters than LoRA or DoRA. The method splits each pretrained weight matrix into a trainable magnitude vector and a directional component, freezes two low-rank matrices built from the top singular vectors of that direction, and trains only a small square matrix placed between them. On the GLUE benchmark with RoBERTa-base, the paper reports that EDoRA at rank 32 uses 86.0K trainable parameters and reaches 84.48% average performance, compared with the best LoRA result of 83.66% using 2.0M parameters and the best DoRA result of 83.78% using 2.1M parameters. The claimed implication is that memory-constrained and small-data deployments can get full fine-tuning-like adaptation without the parameter overhead.","feed_headline":"EDoRA tops GLUE with 30x fewer trainable parameters","feed_subtitle":"A frozen SVD frame plus a trainable r-by-r core hits 84.48% average GLUE with 86K parameters.","key_machinery":"The load-bearing object is the update rule $W' = m\\frac{W_0 + B R A}{\\lVert W_0 + B R A\\rVert_c}$, where $W_0$ is the frozen pretrained weight, $m$ is a trainable magnitude vector, $\\lVert \\cdot \\rVert_c$ denotes the vector-wise column norm, and $A = V_r^T$, $B = U_r\\Sigma_r$ are frozen low-rank matrices built from the top $r$ singular vectors of the directional matrix $D = W_0/\\lVert W_0\\rVert_c$. The small trainable matrix $R \\in \\mathbb{R}^{r\\times r}$, initialized with a near-zero Gaussian, carries the directional adaptation inside a subspace aligned with the most important singular directions of the pretrained weights. This restricts the trainable parameter count per weight matrix to $n + r^2$, versus $2nr$ for LoRA and $n + 2nr$ for DoRA, which is the source of the reported efficiency gain.","core_discovery":"The paper's central claim is that magnitude-direction decomposition, SVD-based initialization, and a frozen low-rank frame with a trainable $r\\times r$ core together recover the learning pattern of full fine-tuning at a fraction of the parameter cost. Concretely, EDoRA writes the adapted weight as $W' = m\\frac{W_0 + B R A}{\\lVert W_0 + B R A\\rVert_c}$, where $A$ and $B$ are frozen matrices from the truncated SVD of the directional part and $R$ is the only trainable low-rank matrix, with $m$ a trainable per-column magnitude. On six GLUE tasks the paper reports that EDoRA improves over LoRA, DoRA, and LoRA-XS on RTE, STSB, CoLA, and SST2, stays competitive on MRPC and QNLI, and achieves the best average score with the smallest parameter count. An ablation attributes a consistent part of the gain to SVD initialization, which outperforms random initialization at every rank by an average of about 1.46 percentage points.","pith_inferences":["A testable extension is to check whether the frozen singular frame caps performance on tasks that need directions outside the top-$r$ subspace, since EDoRA never updates $A$ or $B$ during training.","The recipe suggests a broader family: any frozen orthogonal frame plus a small trainable core could be compared against the SVD frame as an inexpensive ablation, separating the value of the frame from the value of the decomposition.","The reported average gap of roughly 0.8 points sits within the standard deviations in Table 2, so an independent replication with matched search budgets and more seeds is the natural next check before treating the ranking as settled."],"forward_implications":["Because the adapted weight merges into $W_0$ before inference, EDoRA adds no inference latency, matching LoRA and DoRA on that front.","Savings grow with model width: for GPT-3 at rank 16 the paper computes 31x fewer trainable parameters than LoRA and 32x fewer than DoRA, and estimates that 1 million personalized checkpoints would drop from 144TB to 4.59TB of memory.","On the small-data GLUE tasks RTE and CoLA, EDoRA posts its largest gains, consistent with the claim that its compact parameterization reduces overfitting.","SVD initialization is load-bearing within the method: the reported ablation gives it an average 1.46-point advantage over random initialization across all tested ranks."],"supporting_citations":[{"why":"Supplies the low-rank adaptation formulation and the LoRA baseline that EDoRA is measured against.","marker":"(Hu et al., 2022)"},{"why":"Defines the magnitude-direction decomposition that EDoRA inherits and is the strongest baseline to beat.","marker":"(Liu et al., 2024)"},{"why":"Introduces the frozen-SVD-frame plus trainable-middle-matrix structure that EDoRA extends, and is a baseline.","marker":"(Bałazy et al., 2024)"},{"why":"Provides the SVD parameterization of low-rank updates that motivates EDoRA's initialization strategy.","marker":"(Zhang et al., 2023)"},{"why":"Originates weight normalization, the conceptual source of separating magnitude from direction.","marker":"(Salimans & Kingma, 2016)"},{"why":"Defines the GLUE benchmark used for all reported evaluations.","marker":"(Wang et al., 2018)"},{"why":"Provides the RoBERTa-base backbone used in every experiment.","marker":"(Liu et al., 2019)"}],"fun_headline_variants":["EDoRA: SVD-powered LoRA with 30x fewer parameters","EDoRA outdoes LoRA, DoRA with 30x fewer params","Freeze SVD, train 86K: EDoRA hits 84.48 on GLUE","EDoRA: 30x fewer parameters, SVD-boosted GLUE scores","SVD-init LoRA variant EDoRA cuts params 30x, stays on top"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the LoRA, DoRA, and LoRA-XS baselines were tuned as carefully as EDoRA, so the reported accuracy gap reflects the method rather than uneven search effort.","fun_headline_variants_meta":{"raw":{"variants":["EDoRA: SVD-powered LoRA with 30x fewer parameters","EDoRA outdoes LoRA, DoRA with 30x fewer params","Freeze SVD, train 86K: EDoRA hits 84.48 on GLUE","EDoRA: 30x fewer parameters, SVD-boosted GLUE scores","SVD-init LoRA variant EDoRA cuts params 30x, stays on top"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000635,"raw_usage":{"total_tokens":2930,"prompt_tokens":951,"completion_tokens":1979,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":567,"completion_tokens_details":{"reasoning_tokens":1865}},"tokens_in":567,"tokens_out":1979,"duration_ms":14964,"temperature":1.0,"reasoning_tokens":1865,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:33:11.715841+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the six GLUE tasks with identical hyperparameter search budgets and all five seeds for every method, then compare EDoRA's average against the best baseline; if the gap falls within the run-to-run spread reported in Table 2, the claimed advantage is not established.","supporting_citations":[{"cited_title":"F., Cheng, K.-T., and Chen, M.-H","cited_arxiv_id":null,"evidence_quote":"Defines the magnitude-direction decomposition that EDoRA inherits and is the strongest baseline to beat."}],"review_version":1}