{"id":"9bba8ade-2d22-4384-8e58-7bf7614e32ed","arxiv_id":"2412.20891","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Initializing tensor adapters from the MPO decomposition of pretrained weights boosts fine-tuning accuracy and parameter efficiency over random and SVD-based initialization in LLaMA2-7B and LLaMA3-8B.","lead":"DoTA is a method for fine-tuning large language models that starts small tensor adapters from a decomposition of the pretrained weights instead of random values. The paper reports higher accuracy with fewer trainable parameters than LoRA, PiSSA, FLoRA, and LoRETTA on reasoning benchmarks, plus a 4-bit quantized version.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported gains over baselines are not replicable as claimed: DoTA's hyperparameters are selected on the same evaluation tasks and several Table 1 numbers are imported, so the 0.6–1.5 pp margins over PiSSA may reflect tuning or protocol differences rather than MPO initialization.","rationale":"The reader's weakest_assumption already identified imported baselines and hyperparameter tuning on the evaluation tasks, and I agree that this is the load-bearing point. My concern is framed around the same evidence but with a concrete falsification test: rerun all methods under one controlled protocol with equal tuning budgets and report per-seed variability. I do not see an internal mathematical inconsistency in the MPO-based construction; the risk is empirical replicability of the reported margins. Since the reader's verdict is already CONDITIONAL and this is the same concern, no change to the verdict is needed.","tokens_in":10733,"tokens_out":9872,"duration_ms":106564,"concrete_test":"Obtain the released code (or reimplement from the paper) and run all methods from Table 1 plus QuanTA in one harness on LLaMA2-7B and LLaMA3-8B, using the same 100k-sample Commonsense-170K training split, the same 1,200-sample validation split, the same optimizer, bf16, adapter layers, and epoch count. Give every baseline the same hyperparameter search budget as DoTA (e.g., grid over its main rank parameter using the same validation protocol), and report per-seed results. If DoTA's average accuracy remains more than 1 pp above the best-tuned baseline on both models, the central claim survives; if the margin drops below the per-seed spread, the reported superiority is not established. Also report final task accuracy for the DoTA-Random ablation in Fig. 2.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim is that MPO-based initialization lets DoTA outperform random-initialization tensor PEFTs with fewer parameters (Abstract, Sec. 3.2, Table 1). The load-bearing support is Table 1. However, Section 4 states that N and R were selected by grid search on the same benchmark tasks (N in {3,5,7,9}, R in {16,32,64,128}), and Appendix B shows baseline hyperparameters are fixed values from their original papers. Table 1 also imports Full-FT, LoRA, DoRA, and DoRA† numbers from prior papers rather than recomputing them under the DoTA protocol (100k/1200 data split, bf16, one epoch, cosine schedule, same adapter layers). The margin over the strongest baseline, PiSSA, is only 0.6 pp on LLaMA3-8B and 1.5 pp on LLaMA2-7B; no error bars or significance tests are reported. The controlled ablation (Fig. 2) shows a training-loss gap between DoTA and a Gaussian-random tensor initialization of the same shape, but it does not report final task accuracy or standard deviations, and it does not include the imported baselines. QuanTA, the closest MPO-based tensor adaptation method, is mentioned only in the references and never benchmarked. Thus the 'outperforms with fewer parameters' claim is not yet controlled for selection bias and protocol mismatch.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DoTA, a parameter-efficient fine-tuning method that decomposes a pre-trained weight matrix into Matrix Product Operator (MPO) core tensors, initializes the trainable tensor adaptation from this decomposition, and keeps the truncation error in a frozen residual matrix. A quantized variant, QDoTA, is also introduced. The main empirical claim is that this initialization strategy lets DoTA outperform random-initialized tensor adaptation and existing PEFT baselines with fewer trainable parameters on commonsense and arithmetic reasoning tasks, supported by Table 1, a controlled initialization ablation in Figure 2, and rank-sensitivity experiments in Section 4.5.","tokens_in":10958,"tokens_out":4908,"duration_ms":49089,"significance":"If the central empirical claim holds, the contribution is a useful and timely one: it provides a concrete, structurally motivated initialization for tensor-based PEFT and shows a plausible mechanism by which preserving pretrained weight structure helps fine-tuning. The paper deserves credit for the controlled ablation in Section 4.4, which compares DoTA against a same-shape Gaussian-random tensor initialization and thereby directly tests the role of initialization rather than the tensor parameterization alone. The rank analysis in Section 4.5 also gives useful practical guidance. However, the headline comparison is currently weakened by hyperparameter selection on the evaluation benchmarks, by several baseline numbers imported from other papers, and by the absence of variance or significance measures for the small reported margins. The central claim is defensible but needs stronger experimental control before it can be accepted as stated.","major_comments":[{"comment":"The headline comparison in Table 1 is not yet controlled for hyperparameter selection. Section 4 states that DoTA's N and R were selected by grid search over N in {3,5,7,9} and R in {16,32,64,128} on the same benchmark tasks used for evaluation, while baseline hyperparameters are fixed values from their original papers. Because the reported margin over the strongest baseline PiSSA is small (0.6 percentage points on LLaMA3-8B and 1.5 on LLaMA2-7B), the possibility that grid-search selection, rather than MPO initialization, produces the margin is not excluded. Please select hyperparameters on a disjoint validation split, evaluate DoTA at the default baseline ranks, or report the performance variation across the searched grid.","section":"Section 4, first paragraph; Table 1"},{"comment":"Several baseline numbers are imported without being re-run under the protocol described in the paper. The Table 1 footnote says Full-FT, LoRA, DoRA, and DoRA† results are taken from prior papers, and Appendix B gives method-specific settings for the re-run methods but does not establish that the imported numbers were produced under the same training schedule, adapter layers, data split, one-epoch setting, and bf16 precision used for DoTA. Relative rankings can change with training length and learning-rate schedule, so the central claim of outperforming existing methods requires either re-running these baselines under the common protocol or documenting that the imported results were produced under equivalent conditions.","section":"Table 1 footnote; Appendix B"},{"comment":"The paper does not report variance or significance for the task-accuracy comparison, and the controlled initialization ablation stops at evaluation loss. Table 1 reports only point averages over three random seeds, so a 0.6-1.5 percentage-point margin over PiSSA is not distinguished from seed noise. Figure 2 shows the mean and standard deviation of evaluation loss for DoTA, DoTA-Random, and Full-FT, but it does not report final downstream task accuracy for the random-initialization control. Please add per-seed accuracy, standard deviations, and ideally a significance test, and extend the Figure 2 comparison to final task accuracy so the initialization gap is directly tied to the claimed accuracy benefit.","section":"Table 1; Section 4.4, Figure 2"},{"comment":"QuanTA, the closest MPO-based tensor adaptation method to DoTA, is cited in Related Work but never appears in the experiments. Without a QuanTA baseline evaluated under the same protocol, the claim that DoTA improves over tensor adaptation methods is incomplete for the most directly comparable prior method. Please add QuanTA to the commonsense and mathematical reasoning comparisons or explicitly explain why it cannot be fairly included.","section":"Section 5 (Related Work); Section 4"}],"minor_comments":[{"comment":"The abstract and Section 4.4 refer to validation loss, while the Figure 2 caption and axis label say evaluation loss; please standardize the terminology.","section":"Abstract; Figure 2 caption; Section 4.4"},{"comment":"The notation MPO(W0) is overloaded: it denotes the exact decomposition of W0 in the definition of Wres, but after rank truncation the contracted core tensors are trainable and no longer represent the exact MPO of W0. Using a distinct symbol for the truncated reconstruction, such as \\tilde{W}, would avoid ambiguity.","section":"Section 3.2, Equation (3)"},{"comment":"Figure 4 shows only PiSSA, LoRA, and DoTA, while the text says FLoRA and LoRETTA were also tested on mathematical reasoning but their results are not shown; providing the numerical values for all tested baselines would make the comparison complete and reproducible.","section":"Section 4.2, Figure 4"},{"comment":"The experimental settings list one epoch, batch size 16, cosine schedule, warmup ratio 0.03, and no dropout; please state explicitly whether these settings also apply to the imported Full-FT, LoRA, and DoRA numbers or only to the results re-run by the authors.","section":"Appendix B"}],"recommendation":"major_revision","confidential_remarks":"The core idea is interesting and the controlled random-initialization ablation is a good methodological choice, but the empirical evidence as written does not yet support the headline claim without a controlled re-evaluation. The margins over PiSSA are small, the baselines are not all re-run under a common protocol, and DoTA's hyperparameters are selected on the evaluation tasks. I recommend major revision rather than rejection because these issues are fixable within the paper's scope: re-running baselines under the same protocol, adding error bars and significance checks, evaluating QuanTA, and either moving hyperparameter selection to a held-out split or reporting sensitivity to the grid choices."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Let me cut to the chase. DoTA is a sensible, clearly written extension of PiSSA's idea to tensor adaptation: initialize the trainable tensor cores from an MPO decomposition of the pre-trained weight, keep the residual frozen, and fine-tune the cores. The genuinely new element is that initialization choice, and the paper's own controlled ablation (Figure 2) is the right way to test it — same tensor shapes, random vs MPO init, and the loss trajectories clearly diverge. That is real evidence for the central claim.\n\nWhat is also done well: the authors ran PiSSA, FLoRA, and LoRETTA themselves under a common protocol rather than importing all baseline numbers, and the rank sensitivity analysis (Figure 6) is a useful robustness check. The writing is direct and the method is simple enough to reimplement.\n\nThe soft spots are mostly about whether the headline margins are real. Hyperparameters N and R were selected by grid search on the very same benchmark tasks used for evaluation, and the margin over PiSSA is small — 1.5 points on LLaMA2-7B, 0.6 on LLaMA3-8B — with no significance tests reported. That margin could come from tuning budget rather than from the initialization itself. Omitting QuanTA is more than a minor gap: it is the closest MPO-based competitor, cited but never benchmarked. The Figure 2 ablation is also narrower than it looks: it plots validation loss, not final task accuracy, and gives no standard deviation. Some Table 1 numbers (Full-FT, LoRA, DoRA) are imported from other papers, so those comparisons are not fully apples-to-apples.\n\nNone of this sinks the paper. The controlled comparison in Figure 2 supports the direction of the claim, and the method is a reasonable addition to the PEFT toolbox. But the 'outperforms with fewer parameters' statement, as written, is conditional: it depends on tuned hyperparameters and a protocol that is not fully matched across all baselines.\n\nWho should read this: anyone working on PEFT initialization or tensor adaptation. It is a legitimate data point and a useful baseline for future comparisons. It deserves a serious referee, not a desk rejection. I would send it to review with three specific requests: report results across the full hyperparameter grid or use a separate validation split for model selection, add QuanTA to the comparison, and release code before publication.","headline":"A clean, plausible extension of PiSSA to tensor adaptation, with a controlled ablation supporting the initialization story, but the headline margins over the strongest baseline are conditional on tuning and protocol.","tokens_in":11572,"tokens_out":2975,"would_cite":true,"duration_ms":27249,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that initializing tensor adapters from the MPO decomposition of pretrained weights, rather than random noise, yields validation loss close to full fine-tuning and beats random-initialized tensor and low-rank methods with…","keywords":["Large Language Models","Parameter-Efficient Fine-Tuning","Tensor Decomposition","Matrix Product Operator","Initialization","Quantization","Low-Rank Adaptation"],"falsifier":"Re-run every baseline under identical training schedules, seed counts, and a shared hyperparameter search budget, and compare with random-initialized tensors of the same shape and parameter count; if DoTA's margin disappears or a random-initialized tensor matches its accuracy, the claimed initialization advantage is not sustained.","tokens_in":10431,"feed_emoji":"🧠","tokens_out":9353,"duration_ms":76467,"temperature":0.7,"pith_summary":"This paper tries to establish that how you initialize a tensor-based adapter matters as much as its shape. It claims that starting from the Matrix Product Operator (MPO) decomposition of a pretrained weight matrix, rather than from random noise, keeps the fine-tuned model close to the pre-training manifold and yields validation loss curves that track full fine-tuning. On eight commonsense reasoning tasks and two arithmetic datasets, the proposed DoTA is reported to outperform random-initialized tensor methods (LoRETTA, FLoRA) and low-rank baselines (LoRA, DoRA, PiSSA) while using a fraction of their parameters. A quantized variant, QDoTA, applies 4-bit NF4 quantization to the frozen residual and is claimed to match DoTA's commonsense performance while using less memory. If true, the result would make tensor decomposition a practical and parameter-efficient alternative to low-rank adaptation for LLM fine-tuning.","feed_headline":"DoTA: decomposing pretrained weights beats random tensor init","feed_subtitle":"MPO-initialized tensor adapters match full fine-tuning's loss curve using as little as 0.06% of parameters.","key_machinery":"The load-bearing tool is the Matrix Product Operator (MPO), a decomposition from quantum physics that represents a large matrix as a contracted chain of small fourth-order core tensors $\\{T^{(k)}\\}$. In DoTA, the pretrained weight $W_0$ is decomposed by iterative SVD (Algorithm 1), the core tensors become the trainable parameters, and the rank threshold $R$ controls their number. The frozen residual $W_{\\mathrm{res}} = W_0 - \\mathrm{MPO}(W_0)$ preserves the truncated detail and keeps the reconstruction exact before training. The number of trainable parameters is $\\rho = \\sum_{k=1}^{N} \\bar{R}_{k-1} I_k J_k \\bar{R}_k$, with $\\bar{R}_0 = \\bar{R}_N = 1$; for a $1024\\times1024$ matrix with $N=5$, $R=8$, and $I_k = J_k = 4$, this is roughly 3.3K parameters. The same construction, with the residual quantized to NF4 and computations in BFloat16, gives QDoTA.","core_discovery":"The central claim is that the information lost by truncating an MPO decomposition can be separated from the information worth adapting. DoTA writes the adapted weight as a frozen residual $W_{\\mathrm{res}} = W_0 - \\mathrm{MPO}(W_0)$ plus a trainable sequence of core tensors that initialize as $\\mathrm{MPO}(W_0)$. Because the tensors begin by reproducing the principal structure of the pretrained matrix, the model starts from the pretrained manifold instead of a random low-dimensional subspace, and the residual compensates for the truncation. The paper reports that this initialization, not the tensor format alone, is what drives performance: randomly initialized tensors of the same shape converge to a suboptimal validation loss, while DoTA's curve closely follows full fine-tuning. DoTA achieves 81.6% average accuracy on eight commonsense tasks with LLaMA2-7B at 0.15% trainable parameters, and 87.1% with LLaMA3-8B at 0.06%, surpassing all listed PEFT baselines.","pith_inferences":["Inference: any tensor or matrix adapter that freezes a residual and initializes from a spectral decomposition (SVD, MPO, Tucker) should inherit the same benefit; the active ingredient may be the initialization, not the tensor format.","Inference: the 4-bit quantization result suggests that the residual carries mostly low-precision detail, so even lower-bit formats or lossy compression of the residual could be tried while keeping the core tensors in higher precision.","Inference: a natural test the paper does not run: replace MPO with a plain truncated SVD of the same rank and shape; if it matches DoTA, the high-dimensional tensor structure is not the driver, only the principal-component initialization.","Inference: the validation-loss alignment with full fine-tuning implies DoTA may also serve as a better starting point for subsequent RLHF or continued pretraining than random-initialized adapters, since it stays closer to the pretrained distribution."],"forward_implications":["DoTA with $N=5, R=16$ uses 0.15% (LLaMA2-7B) or 0.06% (LLaMA3-8B) of parameters while beating LoRA, DoRA, PiSSA, FLoRA, and LoRETTA on commonsense reasoning averages.","QDoTA, using 4-bit NF4 residual quantization, stays close to full fine-tuning on commonsense tasks while using 0.2% parameters, outperforming QLoRA and QPiSSA.","Randomly initialized tensors of the same shape converge to a worse validation loss than MPO-initialized ones, isolating initialization as the cause of DoTA's advantage.","Rank 16 is the best default; rank 8 degrades performance, especially on math, while rank 32 gives little or no gain over rank 16.","DoTA also beats PiSSA on GSM8K by a wide margin, showing the benefit extends beyond commonsense."],"supporting_citations":[{"why":"Provides the low-rank baseline method and the random initialization convention DoTA replaces.","marker":"[13]"},{"why":"Supplies the matrix product operator decomposition algorithm that DoTA uses to initialize its tensors.","marker":"[9]"},{"why":"Introduces the residual-matrix adaptation principle and is a principal-component initialization baseline.","marker":"[19]"},{"why":"A random-initialized tensor-train baseline that DoTA claims to outperform.","marker":"[27]"},{"why":"A random-initialized Tucker-decomposition baseline that DoTA claims to outperform.","marker":"[23]"},{"why":"A weight-decomposed low-rank baseline and the source of the LoRA/DoRA numbers in Table 1.","marker":"[18]"},{"why":"The NF4 quantized-LoRA baseline against which QDoTA is compared.","marker":"[6]"},{"why":"The Commonsense-170K dataset used for all commonsense reasoning evaluations.","marker":"[14]"},{"why":"The MetaMathQA dataset used for arithmetic reasoning evaluations.","marker":"[29]"}],"fun_headline_variants":["DoTA: Pretrained MPO init beats random tensor adaptation","DoTA: 0.06% params, matches full fine-tuning","Tensor init matters: DoTA reuses pretrained MPO structure","DoTA: Start from MPO of pretrained weights, not random tensors","DoTA: Random tensor init loses; MPO init wins"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported margins assume that the baseline numbers imported from other papers (Full-FT, LoRA, DoRA in Table 1) were produced under settings equivalent to DoTA's runs, and that the hyperparameters $N=5$, $R=16$ selected by grid search on the same benchmark tasks do not inflate DoTA's advantage.","fun_headline_variants_meta":{"raw":{"variants":["DoTA: Pretrained MPO init beats random tensor adaptation","DoTA: 0.06% params, matches full fine-tuning","Tensor init matters: DoTA reuses pretrained MPO structure","DoTA: Start from MPO of pretrained weights, not random tensors","DoTA: Random tensor init loses; MPO init wins"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001114,"raw_usage":{"total_tokens":4655,"prompt_tokens":976,"completion_tokens":3679,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":592,"completion_tokens_details":{"reasoning_tokens":3586}},"tokens_in":592,"tokens_out":3679,"duration_ms":27981,"temperature":1.0,"reasoning_tokens":3586,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:07:27.689994+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run every baseline under identical training schedules, seed counts, and a shared hyperparameter search budget, and compare with random-initialized tensors of the same shape and parameter count; if DoTA's margin disappears or a random-initialized tensor matches its accuracy, the claimed initialization advantage is not sustained.","supporting_citations":[{"cited_title":"Physical Review Research 2(2), 023300 (2020)","cited_arxiv_id":null,"evidence_quote":"Supplies the matrix product operator decomposition algorithm that DoTA uses to initialize its tensors."},{"cited_title":"Advances in Neural Information Processing Systems 36 (2024)","cited_arxiv_id":null,"evidence_quote":"The NF4 quantized-LoRA baseline against which QDoTA is compared."}],"review_version":1}