{"id":"4d4646f7-9804-4cac-beee-3a0f592db5a3","arxiv_id":"2505.20154","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"UORA is a LoRA/VeRA-style PEFT method that selectively reinitializes low-magnitude rows and columns of frozen random matrices, reaching LoRA-comparable performance with far fewer trainable parameters.","lead":"The authors introduce UORA, a fine-tuning method that keeps most weights frozen and trains only two small scaling vectors per layer, reinitializing unused low-rank directions during training. It reports performance close to LoRA with roughly 15 times fewer trainable parameters on GLUE and 8 times fewer on E2E, which matters for memory-constrained adaptation of large models.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"UORA's storage-efficiency claim is internally inconsistent: A and B are reinitialized during training, so the final adapter is not just the two scaling vectors, and Table 1 undercounts the storage required.","rationale":"The reader's weakest assumption (the magnitude heuristic) is reasonable, but the ablation in Table 12 shows k=0 (reinitialization disabled) yields 89.95 vs k=1 90.44 on MRPC, a 0.49-point difference with no error bars; even without the heuristic, UORA at rank 16 is competitive. So the empirical central claim does not hinge on the heuristic. The more fundamental issue is that UORA's efficiency claim conflates gradient-trainable parameters with the total learned state. A and B are modified during training (Eqs. 3-4) and are not recoverable from the seed and d,b alone; Table 1 counts only d,b for storage. This is an internal accounting inconsistency: if A,B must be stored, UORA's adapter is not smaller than LoRA's at the ranks used, and if they are merged into W0, the PEFT storage benefit is lost. This directly affects the paper's headline '15x/8x fewer parameters' and its state-of-the-art parameter-efficiency claim. The proposed test settles the question by checking whether the final adapter is reconstructible and measuring the omitted metadata. Agreement with the reader is 'disagree' because the reader focused on the performance heuristic rather than the parameter accounting. The verdict remains CONDITIONAL because the work is empirically broad and the trainable-parameter count is literally true; however, the paper should be required to clarify the storage accounting and provide the metadata size before the efficiency claim is accepted.","tokens_in":20316,"tokens_out":12346,"duration_ms":133970,"concrete_test":"Train UORA on MRPC using the released code with a fixed seed. Save only d, b, and the initial random seed (as Table 1 implies) and discard A,B. Reload and evaluate on the test set; if predictions differ from a run that kept the final A,B, then A,B are part of the learned state. Also instrument the code to log each reinitialization (step, layer, index, random draw) and compute the serialized size of this log; if log size plus d,b exceeds 2*d*r (LoRA's adapter size) at the ranks used (16 for UORA, 8 for LoRA), the storage-efficiency claim in Table 1 fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing flaw is in the parameter/storage accounting, not the magnitude heuristic. UORA's forward pass is dW = Lambda_b B Lambda_d A (Eq. 2), with A and B described as frozen. But the core novelty (Section 3.1, Eqs. 3-4) overwrites entries of A and B whenever a scaling entry stays below tau for k consecutive steps. These overwrites depend on the training trajectory (which d_i crosses tau at which step), so after training A and B are not the initial random matrices and cannot be regenerated from the initial seed plus d,b. Table 1 nevertheless counts UORA's stored weights as only d+b (e.g., 74KB for RoBERTa-base r=16), while LoRA is charged for A and B. To actually use the UORA adapter without merging, one must store the modified A,B (2*d*r per layer, the same shape as LoRA's A,B) or log all reinitialization events, which Table 1 omits. If one merges into W0, the PEFT storage advantage disappears. Thus the advertised 15x/8x parameter reduction is not a like-for-like storage comparison; at the rank values used in the paper, UORA's non-reconstructible A,B can be larger than LoRA's trained A,B. This is an internal inconsistency in the central efficiency claim, independent of whether the magnitude heuristic helps performance.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes UORA (Uniform Orthogonal Reinitialization Adaptation), a VeRA-style PEFT method in which two low-rank matrices A and B are randomly initialized and frozen, while trainable scaling vectors d and b parameterize the update. The novel mechanism is that whenever an entry of d remains below a threshold tau for k consecutive steps, the corresponding column of A and row of B are reinitialized using linear interpolation with factor alpha. The authors claim state-of-the-art parameter efficiency, reporting roughly 15x and 8x fewer trainable parameters than LoRA on GLUE and E2E, and present experiments on GLUE, E2E, arithmetic reasoning, and image classification, together with ablations of initialization, tau, k, and alpha.","tokens_in":20634,"tokens_out":3791,"duration_ms":42837,"significance":"If the claims held, the paper would make a useful empirical contribution to PEFT by showing that selectively reinitializing low-magnitude frozen directions can reduce the number of trainable parameters relative to LoRA while retaining competitive performance. The code is released, and the ablations over tau, k, alpha, and initialization are a strength. However, the central efficiency claim is weakened by a storage-accounting inconsistency and by incomplete benchmark reporting, so the significance as stated is not yet established.","major_comments":[{"comment":"The advertised parameter efficiency is not a like-for-like storage comparison. UORA overwrites entries of the frozen matrices A and B during training (Eqs. 3-4), and Section 6 explicitly acknowledges that \"UORA updates the frozen matrices.\" These modifications depend on the training trajectory, so the final adapter cannot be reconstructed from the initial random seed plus the trained vectors d and b unless all reinitialization events are logged. Without such logging, deploying the adapter requires storing the trained A and B, which have the same dimensions as LoRA's matrices (2*d_model*r per layer). Table 1 nonetheless charges UORA only for d+b, understating its storage footprint by roughly a factor of 2*d_model*r/(d_model+r). Consequently, the 15x and 8x parameter reductions claimed in the abstract and Section 5, and the storage-efficiency comparisons in Table 1, are not valid as stated.","section":"Section 3.1, Eqs. (3)-(4); Table 1; Section 6"},{"comment":"The GLUE evaluation omits MNLI and QQP, the two largest tasks in the benchmark, and the reported \"Avg.\" is computed over the remaining six tasks. This average is not directly comparable to standard GLUE averages reported in prior LoRA/VeRA papers, and the claim of a 15x parameter reduction is based on this nonstandard average. The paper should either complete the GLUE benchmark or clearly label the average as a partial-GLUE average and avoid direct comparisons with full-GLUE results from prior work.","section":"Section 4.1, Table 2"},{"comment":"The claim of achieving \"state-of-the-art\" parameter efficiency is not supported by the baselines chosen. The comparison set lacks recent PEFT methods that also report very low trainable-parameter counts, such as PiSSA, DoRA, or other LoRA variants, and the instruction-tuning and vision experiments include only LoRA, VeRA, and (in vision) head/full fine-tuning. Without a broader baseline set, the \"state-of-the-art\" assertion is unsupported. Additionally, the E2E results in Table 3 and instruction-tuning results in Table 4 are reported without standard deviations or significance tests, despite small performance differences among the methods.","section":"Sections 1 and 4; Tables 2-5"},{"comment":"The core mechanism rests on the heuristic that a small magnitude in the scaling vector d reliably marks the corresponding frozen row/column as unimportant. This is asserted with a citation to a pruning paper (Sun et al., 2024), but no direct evidence is provided for VeRA-style adapters, and the behavior is controlled by tuned thresholds tau and counts k that vary across tasks (Tables 6-9). The ablations in Appendix B show sensitivity to tau and k on single tasks, but they do not test the heuristic itself, e.g., by comparing the reinitialized dimensions against an oracle or a random-dimension baseline. Without such a test, the risk remains that the performance gains could come from the extra stochasticity of reinitialization rather than from the magnitude heuristic.","section":"Section 3.2 and Appendix B"}],"minor_comments":[{"comment":"The Limitations section contains a typo: \"Altough\" should be \"Although.\"","section":"Section 6"},{"comment":"The phrase \"UoRA\" is used in the Figure 1 caption and elsewhere with inconsistent capitalization; please use \"UORA\" uniformly.","section":"Section 3.2"},{"comment":"The dataset name \"SV AMP\" is written with an extra space and should be \"SVAMP\" throughout.","section":"Section 4.3"},{"comment":"The tuning strategy reports that rank 32 is needed for large models and that tau=1e-4 is best on AddSub, yet the GLUE experiments use tau values of 1e-5 and 8e-6. A brief explanation of how the final hyperparameters were selected across tasks would improve reproducibility.","section":"Appendix A.1"},{"comment":"Several references (e.g., Cai et al., 2025; Bi et al., 2025a; Sun et al., 2025) are not directly used by the method or the experiments; tightening the related-work section to the PEFT and reinitialization literature would improve readability.","section":"Related Work"}],"recommendation":"major_revision","confidential_remarks":"The storage-accounting issue is the main concern; I would ask the authors to either provide a correct deployment cost analysis (including the cost of storing the modified A and B or logging reinitialization events) or to reframe the claims as being about trainable parameters only. The paper appears to be an empirical techniques paper, and the current state-of-the-art claim is not supported by the baseline selection. I do not see grounds for rejection if the claims are revised properly, but the required changes are substantial."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, UORA is a real variant, not a repackaging: it takes VeRA's frozen-random-matrix setup and adds a training-time rule that reinitializes any row/column whose scaling entry stays small too long. That mechanism is new in the cited literature, and the GLUE/E2E numbers suggest it buys back some of the rank loss that VeRA suffers at low rank. Second, the paper's headline efficiency claim is overstated. The stress-test note is right: Table 1 counts only the two scaling vectors for UORA, but the reinitialization rule overwrites entries of A and B during training, and the final A and B are not the initial random matrices. You can't regenerate them from the seed plus d and b. So if you store the adapter without merging, you need the modified A and B (same shape as LoRA's), or you have to log the reinitialization events. That is not 15x or 8x storage savings; the trainable parameter count is lower, but the storage comparison is not like-for-like. This is fixable—the paper should either specify a deterministic replay mechanism or charge for the changed matrices—but as written it undermines a central claim.\n\nWhat the paper does well: the evaluation is broad (NLU, NLG, instruction tuning, vision), the ablations on tau, k, alpha are useful, and the practical tuning strategy in Appendix A is a nice touch. The hyperparameters are disclosed, code is linked. The performance is competitive with LoRA and often better than VeRA at much lower rank, which is worth knowing.\n\nSoft spots in proportion: the GLUE benchmark drops MNLI and QQP, so the average is not comparable to prior 8-task averages. The SOTA claim is unfounded given the missing recent baselines. Instruction tuning and vision results have no error bars. The magnitude heuristic is plausible but not proven; the paper relies on tuned thresholds and counts, and the ablations show sensitivity.\n\nMy take: this is a serious paper with a real new mechanism, but it needs a corrected storage analysis and a full GLUE run before acceptance. I'd send it to referees, with the note that the central efficiency claim has a hole.","headline":"A genuinely new PEFT mechanism that looks competitive, but the paper's storage accounting undercounts what UORA actually needs to ship.","tokens_in":21151,"tokens_out":2964,"would_cite":true,"duration_ms":36276,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a VeRA-style frozen-matrix adapter, whose only trainable parameters are two scaling vectors, can match or beat LoRA's fine-tuning performance across NLU, NLG, instruction-tuning, and image classification, provided…","keywords":["parameter-efficient fine-tuning","low-rank adaptation","frozen random projection","scaling vectors","interpolation-based reinitialization","LoRA","VeRA","large language models"],"falsifier":"Run UORA on a GLUE task such as MRPC with four configurations: reinitialization disabled (k=0), magnitude-guided reinitialization as proposed, random indices reinitialized at the same rate, and reinitialization of the largest-magnitude indices. If randomly reinitialized frozen directions match or beat the magnitude-guided version, or if reinitializing the largest-magnitude directions performs equally well, then the claim that small scaling-vector entries mark unimportant frozen dimensions is not what drives the reported gains.","tokens_in":20135,"feed_emoji":"⚡","tokens_out":4719,"duration_ms":46539,"temperature":0.7,"pith_summary":"The paper introduces Uniform Orthogonal Reinitialization Adaptation (UORA), a parameter-efficient fine-tuning method that keeps the frozen random projection matrices of VeRA and trains only two scaling vectors, but periodically reinitializes rows and columns of those frozen matrices when the learned scaling entries are small. This lets UORA use a rank close to LoRA's instead of VeRA's much larger ranks, reporting competitive or better performance with roughly 15x fewer trainable parameters than LoRA on GLUE and 8x fewer on E2E. The authors also show competitive results on arithmetic instruction tuning with LLaMA models and on ViT image classification. If the method holds up, it means the expensive low-rank matrices in LoRA-style adapters can be replaced by frozen matrices plus a cheap, selective refresh rule.","feed_headline":"Fine-tune LLMs with 15x fewer parameters than LoRA","feed_subtitle":"Frozen projection matrices plus a periodic reinitialization trick match LoRA's scores across NLP and vision.","key_machinery":"The central mechanism is interpolation-based reinitialization of frozen low-rank matrices guided by scaling-vector magnitude. UORA adopts VeRA's parameterization where A and B are frozen random matrices and only diagonal scaling vectors Λ_d and Λ_b are trained, but adds a rule: when a scaling entry falls below threshold τ for k consecutive steps, the matching frozen column and row are refreshed by linear interpolation v_new = α v_old + (1−α) v_rand. This refresh is what lets UORA operate at ranks comparable to LoRA (e.g., 16–32) instead of VeRA's typical 256–1024, keeping the trainable parameter count at roughly d_model + r per tuned layer.","core_discovery":"The paper claims that selective reinitialization of frozen projection matrices restores the expressiveness that VeRA loses when its rank is reduced, so that scaling-vector adaptation can match LoRA with far fewer trainable parameters. Concretely, UORA freezes orthogonally initialized random matrices A and B and trains scaling vectors d and b such that the weight update is ΔW = Λ_b B Λ_d A, and whenever an entry of d stays below a threshold τ for k consecutive steps, the corresponding column of A and row of B are replaced via v_new = α v_old + (1−α) v_rand. On GLUE, UORA reports an average of 86.5 on RoBERTa-base with 0.019M trainable parameters versus LoRA's 85.2 with 0.3M, and 88.5 on RoBERTa-large with 0.049M; on E2E it reports BLEU 66.67 on GPT-2 Medium with 0.051M parameters versus LoRA's 67.14 with 0.4M.","pith_inferences":["The magnitude heuristic is a proxy: a small learned scaling value does not necessarily prove the corresponding frozen direction is useless, so reinitialization may be acting as a structured exploration mechanism or as an implicit way to increase effective rank; a test that distinguishes these is comparing magnitude-guided reinitialization against random reinitialization at the same rate.","UORA's success suggests the frozen random matrices in VeRA are the bottleneck at low rank, and periodic refresh is a cheaper substitute for making them trainable, pointing to a broader design space where frozen components are refreshed rather than updated by gradients.","The threshold and count hyperparameters (τ, k) are tuned per task, and an adaptive criterion based on gradient signal or validation loss could be a natural testable extension that removes the manual tuning burden."],"forward_implications":["UORA cuts the trainable parameter count of LoRA-style adaptation by roughly an order of magnitude on GLUE (15x) and E2E (8x) with no reported loss in average performance.","Because the projection matrices stay frozen and only vectors are trained, the learned update can still be merged into the pretrained weights for zero inference latency.","UORA can use ranks close to LoRA's where VeRA needed much larger ranks, reducing both the compute during training and the bytes of stored adapter weights.","The reinitialization rule transfers across domains, including GLUE, E2E generation, arithmetic instruction tuning on LLaMA models, and ViT image classification."],"supporting_citations":[{"why":"VeRA is the direct predecessor that UORA extends: frozen random projection matrices with trainable scaling vectors, and the main efficiency baseline.","marker":"(Kopiczko et al., 2023)"},{"why":"LoRA defines the low-rank adaptation framework and the parameter-efficiency target that UORA compares against.","marker":"(Hu et al., 2021)"},{"why":"Supplies the reinitialization mechanism that UORA adapts to frozen adapter matrices.","marker":"(Zaidi et al., 2023)"},{"why":"Provides the heuristic that the magnitude of scaling values indicates the importance of corresponding dimensions in the frozen matrices.","marker":"(Sun et al., 2024)"},{"why":"AFLoRA is cited as evidence that reducing VeRA's rank degrades performance, motivating UORA's reinitialization fix.","marker":"(Liu et al., 2024c)"}],"fun_headline_variants":["UORA: LoRA-level scores with 15x fewer parameters","Reinitialize frozen matrices to fine-tune with 15x fewer params","UORA: same fine-tuning quality, 15x less trainable params","Cut PEFT params 15x, keep LoRA accuracy","UORA: fewer params, same GLUE scores as LoRA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's gains rest on the assumption that a small entry in the trained scaling vector reliably marks the corresponding frozen row and column as unimportant, so replacing those directions helps training rather than discarding useful structure.","fun_headline_variants_meta":{"raw":{"variants":["UORA: LoRA-level scores with 15x fewer parameters","Reinitialize frozen matrices to fine-tune with 15x fewer params","UORA: same fine-tuning quality, 15x less trainable params","Cut PEFT params 15x, keep LoRA accuracy","UORA: fewer params, same GLUE scores as LoRA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000764,"raw_usage":{"total_tokens":3387,"prompt_tokens":938,"completion_tokens":2449,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":554,"completion_tokens_details":{"reasoning_tokens":2356}},"tokens_in":554,"tokens_out":2449,"duration_ms":17727,"temperature":1.0,"reasoning_tokens":2356,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:58:24.852057+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run UORA on a GLUE task such as MRPC with four configurations: reinitialization disabled (k=0), magnitude-guided reinitialization as proposed, random indices reinitialized at the same rate, and reinitialization of the largest-magnitude indices. If randomly reinitialized frozen directions match or beat the magnitude-guided version, or if reinitializing the largest-magnitude directions performs equally well, then the claim that small scaling-vector entries mark unimportant frozen dimensions is not what drives the reported gains.","supporting_citations":[],"review_version":1}