{"id":"a5a11a72-2d76-46f8-bd90-4659d7f6595f","arxiv_id":"2506.15251","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"SoKA combines Kronecker-product SVD initialization with spectrum-based rank selection, using 25% fewer parameters than PiSSA on LLaMA2-7B while trailing PiSSA by less than one point on three benchmarks.","lead":"SoKA is a parameter-efficient fine-tuning method that combines Kronecker product factorization with SVD initialization and dynamic rank selection for large language models. On LLaMA2-7B it uses 0.99M trainable parameters, about 25% fewer than LoRA and PiSSA, while achieving accuracy about one point below PiSSA on GSM8K, MATH, and MBPP.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"KPSVD as written is dimensionally inconsistent unless p=n and q=m; in the valid square-factor case per-rank parameters equal LoRA's, so the 25% saving must come from rank selection, not Kronecker structure.","rationale":"The reader's REJECT verdict is well supported, and my stress-test does not change it. I focused on the KPSVD dimension consistency because the headline parameter-efficiency claim depends on it: if the factorization as written is undefined for general m,n,p,q, then the 0.99M parameter count cannot be trusted. The paper's own text alternates between N=mn=pq and N=mp=nq, and the Kronecker product shape (mp)×(nq) forces the unstated constraint p=n, q=m. In the only obviously valid square-factor setting, each Kronecker rank costs about the same as a LoRA rank, so the claimed 25% reduction is better explained by dynamic rank selection choosing an average rank near 96 than by the Kronecker structure itself. This is a concrete, checkable correctness concern rather than a stylistic overclaim. I also note the abstract's 'matching or exceeding baseline performance' is contradicted by Table 1, where SoKA trails PiSSA on all three benchmarks; however, the dimension inconsistency is the more load-bearing technical issue. The reader's weakest_assumption about pretrained singular-vector alignment is plausible but not the same concern, hence 'partial' agreement. A corrected paper could still be salvageable if the reshape convention is specified and the claims are scaled back, but as submitted the central claim is not adequately supported.","tokens_in":6923,"tokens_out":14335,"duration_ms":145118,"concrete_test":"Implement Algorithm 1 literally on a real LLaMA2-7B linear layer, e.g. a 4096×4096 q_proj, under two conventions: (a) W∈R^{mn×pq} with N=mn=pq, and (b) N=mp=nq. For each, verify that the reshaped U_i and V_i yield U_i⊗V_i of the same shape as W, and identify the condition p=n, q=m. Then fix m=n=p=q=64 and compute per-rank parameters 2N+1 versus LoRA's 2N; if they match, re-express the 0.99M versus 1.33M difference as an average-rank comparison and run PiSSA at r≈96 to see whether it reproduces SoKA's Table 1 numbers.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of 0.99M trainable parameters with matching or exceeding performance rests on KPSVD being a valid Kronecker factorization of each N×N weight matrix. Algorithm 1 requires W∈R^{mn×pq}, reshapes to (mn)×(pq), performs SVD, and reshapes singular vectors x_k∈R^{mn}, y_k∈R^{pq} into U_k∈R^{m×n}, V_k∈R^{p×q}. But U_k⊗V_k has shape (mp)×(nq). For this to equal the N×N update, one needs mp=mn and nq=pq, i.e. p=n and q=m. The paper never states this constraint; the complexity section instead writes N=mp=nq, an incompatible definition. A valid KPSVD can be recovered only with the block-permutation convention of Batselier and Wong (2017) and/or square factors, neither of which is specified. If the intended and valid case is square factors m=n=p=q=√N, then each Kronecker rank costs 2N+1 parameters, essentially identical to LoRA's 2N per rank. Consequently, the claimed 25% parameter reduction cannot come from the Kronecker form itself; it must be an artifact of dynamic rank selection producing an average rank near 96, and a fair comparison would use PiSSA/LoRA at the same average rank. Without specifying the reshape convention and factor dimensions, the 0.99M count and the reported results are not reproducible from the paper.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SoKA (SVD on Kronecker Adaptation), a parameter-efficient fine-tuning method for large language models. SoKA decomposes each weight update into a weighted sum of Kronecker products of factor matrices via a proposed Kronecker-Product SVD (KPSVD) procedure, initializes the adapter from principal singular vectors following PiSSA's heuristic, and selects the rank per layer using energy-threshold and elbow-point criteria. The experiments on LLaMA2-7B cover GSM8K, MATH, and MBPP, comparing against full fine-tuning, LoRA, and PiSSA. The paper claims 0.99M trainable parameters (25% fewer than LoRA/PiSSA), matching or exceeding baseline performance, faster convergence, and more stable gradients.","tokens_in":7228,"tokens_out":9227,"duration_ms":86576,"significance":"If the method worked as claimed, the combination of SVD-based initialization, Kronecker-structured adapters, and spectrum-aware rank selection would be a useful contribution to parameter-efficient fine-tuning, directly addressing the fixed-rank limitation of LoRA/PiSSA and the lack of spectrum adaptation in earlier Kronecker methods. The use of established PCA heuristics for rank selection is reasonable, and the algorithmic statement is explicit. However, the core Kronecker construction is dimensionally inconsistent as written, the reported numbers contradict the abstract's matching-or-exceeding claim, and the experimental section lacks the reproducibility details needed to verify the results. The paper also does not compare against the Kronecker-based PEFT baselines it cites. These issues are load-bearing for the central claims.","major_comments":[{"comment":"SoKA's results in Table 1 are below PiSSA on all three benchmarks: GSM8K 52.19 vs 53.37, MATH 7.93 vs 8.17, and MBPP 39.5 vs 40.4. The abstract's claim that SoKA achieves 'matching or exceeding baseline performance' is therefore contradicted by the paper's own data, and the text's statement that SoKA 'trails by less than 1 point' is inaccurate for GSM8K (a 1.18-point gap). Because the central contribution is a favorable parameter-efficiency/accuracy trade-off, this discrepancy is load-bearing.","section":"Experiments, Table 1"},{"comment":"The KPSVD construction is dimensionally inconsistent as written. The paper starts with W∈R^{mn×pq} and reshapes it to (mn)×(pq); the SVD gives x_k∈R^{mn} and y_k∈R^{pq}, which are reshaped to U_k∈R^{m×n} and V_k∈R^{p×q}. Then U_k⊗V_k has shape (mp)×(nq). For this to equal an N×N weight update, one requires mp=N and nq=N, which forces p=n and q=m (given N=mn=pq). The paper never states this constraint, and the later statement 'N=mp=nq' is a different condition unless p=n and q=m. The cited Batselier-Wong (2017) construction requires a specific permutation convention that is not described. In the valid general case, each Kronecker rank costs |U|+|V|+1 = 2N+1 parameters, essentially the same as LoRA's 2N parameters per rank; hence the reported 25% parameter saving (0.99M vs 1.33M) must come from the rank-selection step producing an average rank near 95 rather than 128, not from the Kronecker structure. Without stating this constraint and without a LoRA/PiSSA baseline at the same average rank, the parameter-efficiency claim is not supported.","section":"Kronecker-Product SVD for Adapter Initialization, Algorithm 1"},{"comment":"The singular-value sequence is defined as S=[σ1≥...≥σ_min(m,n)], but the KPSVD of a reshaped N×N matrix produces N singular values, not min(m,n). The energy-threshold criterion E(k) also sums over min(m,n) terms. This mismatch makes the rank-selection procedure ambiguous: it is unclear whether the spectrum is taken from the full KPSVD or from the factor dimensions, and how per-layer ranks are aggregated into the reported 0.99M trainable parameters. This is not a cosmetic issue, because the entire claimed parameter reduction rests on this selection rule.","section":"Dynamic Rank Selection"},{"comment":"The experimental section omits essential reproducibility information: learning rate, schedule, batch size, epochs, LoRA alpha and dropout, targeted modules, dataset versions and prompt formats, KPSVD block sizes (m,n,p,q), the energy threshold τ, rank bounds rmin and rmax, and hardware/software details. Figures 2 report loss and gradient-norm curves without training-configuration details or error bars. Given that the paper's central empirical claims cannot be checked or reproduced from the manuscript, this is a major omission.","section":"Experiments"},{"comment":"The empirical comparison includes only LoRA, PiSSA, and full fine-tuning; none of the Kronecker-based PEFT methods discussed in Related Works (Compacter, KronA, AdaKron, LyCORIS) is evaluated. The contribution bullet claiming 'state-of-the-art parameter efficiency' is therefore unsupported by the experiments, and the reader cannot assess whether SoKA improves over existing Kronecker-structured adapters.","section":"Experiments, Table 1"}],"minor_comments":[{"comment":"The per-block parameter count is written as 'r(mn+pq+1)'; this should be r_kp(mn+pq+1) to match the notation r_kp and to avoid confusion with the rank bound r.","section":"Kronecker-Product SVD for Adapter Initialization"},{"comment":"There are typos and stylistic inconsistencies: 'LLama2' in the contribution bullet should be 'LLaMA2'; 'an A,B' and 'an A,B ∈' in the complexity paragraph need grammatical correction; and the references list has inconsistent capitalization (e.g., 'mahabadi' vs 'Mahabadi').","section":"Throughout"},{"comment":"The paper should state whether the singular values σ_k are trainable or fixed, and how the adapter output is scaled at initialization, since this affects the effective learning rate compared with LoRA and PiSSA.","section":"Kronecker-Product SVD for Adapter Initialization"}],"recommendation":"reject","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the punchline: this is an incremental paper that deserves a serious referee only if it is heavily revised. The core idea is a plausible combination of existing pieces—PiSSA-style SVD initialization, a Kronecker-product update (KPSVD from Batselier and Wong), and dynamic rank selection—but the abstract overclaims, the KPSVD description is dimensionally inconsistent, and the parameter count cannot be reproduced without extra assumptions.\n\nWhat is actually new: the specific arrangement. No single component is novel; KPSVD, SVD initialization, and energy/elbow heuristics are all cited properly. The paper does report standard deviations, and the body text honestly notes that SoKA trails PiSSA by less than a point on each benchmark. That honesty doesn't carry into the abstract, which says SoKA matches or exceeds baseline performance when Table 1 shows it behind PiSSA on all three tasks (GSM8K 52.19 vs 53.37, MATH 7.93 vs 8.17, MBPP 39.5 vs 40.4).\n\nThe soft spots, in order of severity. First, the dimensional slip: Algorithm 1 takes W ∈ R^{mn×pq}, reshapes to (mn)×(pq), and reconstructs U_k⊗V_k of shape (mp)×(nq). For that to equal the original N×N matrix you need p=n and q=m, which is never stated. If you use square factors, each Kronecker rank costs 2N+1 parameters, essentially identical to LoRA's 2N per rank. So the 25% parameter reduction has to be coming from dynamic rank selection producing an average rank near 96, not from Kronecker structure per se. A fair comparison would pit SoKA against LoRA/PiSSA at the same average rank and same budget. Second, the experimental disclosure is thin: no hyperparameters, no module selection, no residual handling, no code. The convergence and gradient-stability claims rest on two qualitative plots with no quantitative backing. Third, the abstract's 'matching or exceeding' claim is simply wrong against PiSSA.\n\nI should say what the paper does well: it positions itself in the literature accurately, cites the prior work (Batselier, PiSSA, KronA) and uses established heuristics rather than inventing ad hoc rules. The combination is not crazy; a Kronecker adapter initialized from the principal components of W is a natural idea that could work if the details are fixed.\n\nWho gets value: PEFT researchers looking for yet another structured adapter variant. It's not a landscape-changer, but it's a legitimate incremental proposal.\n\nMy recommendation: send it to peer review with major revision. As submitted I would reject it, but the idea is worth engaging with because the flaws are fixable. Ask for a corrected KPSVD specification, an equal-rank comparison, and full experimental details.","headline":"Incremental combination of known ideas whose abstract overclaims and whose KPSVD dimension slip means the parameter saving actually comes from rank selection, not Kronecker structure; still a plausible idea worth a revised look.","tokens_in":7797,"tokens_out":4423,"would_cite":false,"duration_ms":43407,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SoKA represents weight updates as sums of SVD-initialized Kronecker products with dynamic rank pruning, cutting trainable parameters on LLaMA2-7B to 0.99M while matching PiSSA.","keywords":["parameter-efficient fine-tuning","Kronecker product","singular value decomposition","low-rank adaptation","dynamic rank selection","spectrum-aware pruning","LLaMA-2"],"falsifier":"Train SoKA and a randomly initialized Kronecker adapter on a synthetic task whose target update is deliberately orthogonal to the top singular subspace of the reshaped pretrained matrix; if SoKA's convergence and accuracy advantage disappears, the initialization claim is falsified. A simpler ablation would keep the same rank-selection rule but replace KPSVD initialization with random factors, isolating how much of the gain comes from the SVD rather than from the Kronecker structure.","tokens_in":6673,"feed_emoji":"🧩","tokens_out":9092,"duration_ms":78676,"temperature":0.7,"pith_summary":"This paper proposes SoKA, a parameter-efficient fine-tuning method that writes each weight update as a sum of Kronecker products whose factors come from a singular value decomposition of the reshaped pretrained weight matrix. The claim is that this SVD-based initialization captures the principal directions of the update, while a spectrum-aware rank-selection rule prunes negligible components, so adaptation needs fewer trainable parameters than LoRA or PiSSA without sacrificing accuracy. On LLaMA2-7B over GSM8K, MATH, and MBPP, SoKA uses 0.99 million trainable parameters, about 25% fewer than the 1.33 million used by LoRA and PiSSA, and the reported scores trail PiSSA by less than one point on each benchmark while beating LoRA. The authors also report faster convergence and smaller gradient norms than PiSSA. If these results hold, SoKA is a more parameter-efficient alternative to SVD-initialized low-rank adaptation at comparable accuracy.","feed_headline":"Kronecker SVD adapter matches PiSSA with 25% fewer parameters","feed_subtitle":"SoKA builds updates from SVD-based Kronecker factors and prunes rank per layer, matching PiSSA with fewer parameters.","key_machinery":"The load-bearing object is Kronecker-Product SVD (KPSVD): reshape the pretrained weight matrix, take its truncated SVD, then reshape each singular vector into a small matrix so that each singular triple becomes one Kronecker-product term $\\sigma_k U_k\\otimes V_k$. This converts a dense SVD into a structured adapter with $r(mn+pq+1)$ parameters instead of LoRA's $2Nr_{\\mathrm{lora}}$, and it makes the forward pass cheap through the identity $(U_k\\otimes V_k)\\operatorname{vec}(X)=\\operatorname{vec}(V_k X U_k^\\top)$. The supporting mechanism is dynamic rank selection, which sets the per-layer rank to $\\min(r_{\\mathrm{energy}}, r_{\\mathrm{elbow}})$ using cumulative singular-value energy and the largest successive gap in the spectrum.","core_discovery":"SoKA's central discovery is that a Kronecker-structured adapter initialized by a truncated SVD of the reshaped weight matrix can carry most of the adaptation signal with fewer parameters than a plain low-rank adapter. For an $N\\times N$ weight matrix with $N=mn=pq$, the method reshapes $W$ to $N\\times N$, computes its truncated SVD, reshapes each singular vector pair into factor matrices $U_k\\in\\mathbb{R}^{m\\times n}$ and $V_k\\in\\mathbb{R}^{p\\times q}$, and approximates the update as $\\Delta W\\approx\\sum_{k=1}^{r}\\sigma_k U_k\\otimes V_k$. The rank $r$ is chosen by taking the smaller of the energy-threshold rank and the elbow-point rank of the singular spectrum. The empirical claim is that this setup reaches the accuracy of PiSSA while using 0.99M trainable parameters, 25% fewer than LoRA/PiSSA, and converges faster with more stable gradients.","pith_inferences":["Inference: the KPSVD initialization should transfer to any square block of a pretrained network, not just the attention projections tested here, because the reshape-and-SVD step only requires $N=mn=pq$.","Inference: the energy-threshold and elbow-point rules could be tested against an oracle rank chosen by validation performance; the paper reports aggregate parameter counts but does not isolate which criterion drives the savings.","Inference: the reported gradient stability suggests SoKA may be a good fit for low-precision or communication-limited fine-tuning, where large gradient spikes are costly, but the paper does not test those settings."],"forward_implications":["Because each Kronecker term satisfies $(U_k\\otimes V_k)\\operatorname{vec}(X)=\\operatorname{vec}(V_k X U_k^\\top)$, SoKA avoids forming the full $N\\times N$ update and keeps the structured computation proportional to the small factor sizes.","Per-layer rank is set to $\\min(r_{\\mathrm{energy}}, r_{\\mathrm{elbow}})$, so layers with sharply decaying singular spectra automatically receive fewer parameters than layers with flat spectra.","SoKA's learned factors can be merged back into the frozen weights after training, so inference runs at the original model latency just as with LoRA.","On GSM8K, MATH, and MBPP with LLaMA2-7B, SoKA uses 0.99M trainable parameters, 25% fewer than LoRA/PiSSA, trails PiSSA by less than one point per benchmark, and outperforms LoRA on all three tasks."],"supporting_citations":[{"why":"Supplies the constructive Kronecker-product decomposition formalism that Algorithm 1 (KPSVD) applies to the weight matrix.","marker":"(Batselier and Wong 2017)"},{"why":"Defines the LoRA low-rank reparameterization that SoKA compares against and extends with Kronecker structure.","marker":"(Hu et al. 2022)"},{"why":"Introduces PiSSA, the SVD-initialized baseline whose parameter count and accuracy SoKA is designed to beat or match.","marker":"(Meng, Wang, and Zhang 2024)"},{"why":"Provides the GSM8K arithmetic-reasoning benchmark used to evaluate SoKA.","marker":"(Cobbe et al. 2021)"},{"why":"Provides the MATH formal-mathematics benchmark used to evaluate SoKA.","marker":"(Hendrycks et al. 2021)"},{"why":"Provides the MBPP code-generation benchmark used to evaluate SoKA.","marker":"(Austin et al. 2021)"},{"why":"Supplies the cumulative-energy criterion used in the dynamic rank selection.","marker":"(Jolliffe 1986)"},{"why":"Supplies the elbow-point/gap criterion used in the dynamic rank selection.","marker":"(Jackson 1993)"}],"fun_headline_variants":["SVD-Kronecker PEFT matches PiSSA with 25% fewer params","Adaptive-rank SVD-Kronecker trims 25% of PEFT params","SoKA: SVD-Kronecker PEFT at 25% fewer trainable weights","PiSSA-level accuracy with 25% fewer parameters","SVD-based Kronecker adapter: 25% params reduction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"SoKA assumes that the strongest singular directions of the reshaped pretrained weight matrix are also the directions a downstream task needs to change; if that alignment fails, the SVD initialization and spectrum-based rank pruning give no advantage over random or fixed-rank adapters.","fun_headline_variants_meta":{"raw":{"variants":["SVD-Kronecker PEFT matches PiSSA with 25% fewer params","Adaptive-rank SVD-Kronecker trims 25% of PEFT params","SoKA: SVD-Kronecker PEFT at 25% fewer trainable weights","PiSSA-level accuracy with 25% fewer parameters","SVD-based Kronecker adapter: 25% params reduction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001262,"raw_usage":{"total_tokens":5201,"prompt_tokens":1010,"completion_tokens":4191,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":626,"completion_tokens_details":{"reasoning_tokens":4089}},"tokens_in":626,"tokens_out":4191,"duration_ms":32104,"temperature":1.0,"reasoning_tokens":4089,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:38:43.073573+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train SoKA and a randomly initialized Kronecker adapter on a synthetic task whose target update is deliberately orthogonal to the top singular subspace of the reshaped pretrained matrix; if SoKA's convergence and accuracy advantage disappears, the initialization claim is falsified. A simpler ablation would keep the same rank-selection rule but replace KPSVD initialization with random factors, isolating how much of the gain comes from the SVD rather than from the Kronecker structure.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the constructive Kronecker-product decomposition formalism that Algorithm 1 (KPSVD) applies to the weight matrix."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces PiSSA, the SVD-initialized baseline whose parameter count and accuracy SoKA is designed to beat or match."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the cumulative-energy criterion used in the dynamic rank selection."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the elbow-point/gap criterion used in the dynamic rank selection."}],"review_version":2}