{"id":"55fa8730-0e96-4e2a-b788-b3042bbb82d2","arxiv_id":"2505.15090","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"DeFT-X applies SVD denoising to weight updates before magnitude pruning in composable sparse fine-tuning, showing small average gains over LT-SFT on NusaX and AmericasNLI.","lead":"DeFT-X improves sparse fine-tuning for zero-shot cross-lingual transfer by applying SVD-based denoising to the weight updates before pruning. It reports small average gains over the LT-SFT baseline on sentiment analysis and natural language inference for low-resource languages.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own ablation contradicts its denoising story: the retained 5% of higher-order components carries useful signal (Table 5: 81.4→80.6 when removed), so the gain over LT-SFT may rest on an unexamined retention hyperparameter rather than SVD denoising.","rationale":"The reader's weakest_assumption identifies exactly the internal tension in the paper: the claimed denoising rationale is contradicted by the ablation that shows higher-order components carry useful signal. The paper itself concedes this in §5, making it an in-scope piece of evidence. This is the most load-bearing concern because the central claim is not just an empirical gain but a mechanistic one: that SVD-based denoising improves the quality of sparse fine-tuned vectors. If the gain instead comes from the arbitrary 5% retention of higher-order components, the method reduces to an unexamined hyperparameter choice, and the paper provides no evidence that this choice generalizes. I also considered the selection across multiple rank configurations as a possible concern, but the reader already notes it, and the paper does report all configurations, making the empirical result more transparent. The proposed concrete test would directly distinguish the denoising mechanism from a hyperparameter artifact by varying n and comparing against a random-selection control at matched sparsity. The verdict of conditional acceptance remains appropriate pending this evidence, so I recommend no change to the reader's verdict.","tokens_in":14966,"tokens_out":4836,"duration_ms":42398,"concrete_test":"Run a controlled sweep on NusaX (XLM-RBASE, fixed r_l=r_t=100) varying the higher-order retention fraction n ∈ {0%, 1%, 5%, 10%, 25%, 50%}, plus a control where the retained n% entries are selected randomly rather than by magnitude, matched for sparsity. If F1 peaks at n=5% and the magnitude-selected variant clearly beats the random-selection control, the gain comes from the specific sparse residual, not from SVD denoising. If performance is flat across n or monotonic in n, the 'noise' hypothesis is contradicted and the method's improvement is an artifact of the unexamined 5% threshold.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In §3.1 the motivation claims that higher-order (low singular-value) components are 'more likely to capture uninformative or noisy artefacts' and should be pruned. Yet §5's ablation (Table 5) shows that retaining 5% of these components improves NusaX average F1 from 80.6 to 81.4, and the text explicitly states that 'higher-order components contain useful information, making it essential to retain them after denoising.' This is an internal inconsistency in the paper's own reported experiments. The full method's gain over LT-SFT (81.4 vs 80.2) is of the same magnitude as the gain from the retained sparse residual (0.8 F1), so the improvement could be driven entirely by the arbitrary choice to keep 5% of the higher-order components, rather than by a principled signal/noise separation. The paper does not vary the retention fraction, does not motivate the 5% figure, and reports no analysis showing this fraction generalizes across languages, tasks, or model sizes. The central claim that denoising by SVD before magnitude pruning is the source of the gains is therefore not supported by the evidence presented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DeFT-X, a modification of LT-SFT for zero-shot cross-lingual transfer. Before magnitude pruning and sparse fine-tuning, DeFT-X applies an SVD-based denoising step to the weight update matrix ΔW: each matrix is replaced by L + S, where L keeps the top-r singular components and S retains the top 5% of the residual by magnitude. Language-specific and task-specific sparse vectors are then learned as in LT-SFT and composed by addition. The method is evaluated on NusaX sentiment analysis (five Indonesian languages) and AmericasNLI (ten indigenous languages) with XLM-R base/large and mE5 base, reporting average F1 and accuracy. The best reported configurations outperform LT-SFT by 1.2 F1 on NusaX and 0.3 accuracy on AmericasNLI with XLM-R base.","tokens_in":15255,"tokens_out":5556,"duration_ms":48752,"significance":"If the results are robust, DeFT-X is a simple, modular improvement over LT-SFT that can be dropped into existing sparse fine-tuning pipelines, and the paper's focus on truly low-resource languages is valuable. The paper is clearly written, reports per-language numbers, and includes ablations isolating the contributions of higher-order components, magnitude pruning, and sparse fine-tuning. The contribution is empirical rather than theoretical, and the claimed gains are small, so the evidence must be stronger than a single run per configuration. The explicit admission in the Limitations section that the SVD rank requires manual tuning is honest but also highlights that the headline numbers are partially configuration-selected.","major_comments":[{"comment":"The motivation for SVD denoising is that higher-order (lower singular-value) components are 'more likely to capture uninformative or noisy artefacts' and should be pruned. Table 5 contradicts this: removing the retained higher-order components reduces NusaX average F1 from 81.4 to 80.6, and the text in §5 states that 'higher-order components contain useful information, making it essential to retain them after denoising.' Since the total gain over LT-SFT is 81.4 vs 80.2 (1.2 F1), the improvement could come entirely from the choice to retain 5% of the residual rather than from a principled signal-noise separation. The retention fraction is never varied and no evidence is given that 5% generalizes across languages, tasks, or model sizes. Please add an experiment sweeping n (e.g., 0%, 1%, 5%, 10%, 25%) and a control that retains random or smallest-magnitude residual entries, and report whether the benefit is monotone and where it saturates.","section":"§3.1, §4.2, Table 5"},{"comment":"The paper reports a single run per configuration and selects the best among several rank combinations (r_l, r_t) separately for each task and model size, yet the claimed improvements are small, especially on AmericasNLI (51.3 vs 51.0 average accuracy). With only 750 test examples per language and no error bars, confidence intervals, or significance tests, these differences are compatible with noise. In addition, the uniform rank values are described as 'selected by observing the overall rank that covers 90% variance,' with no held-out validation described, so the reported best numbers are optimistic. Please report multiple seeds or bootstrap confidence intervals, paired significance tests for per-language differences, and a rank-selection procedure on a validation split; without this, the abstract's 'at par or outperforms' claim is not supported.","section":"§4.2, §5, Tables 2 and 3"},{"comment":"There is a discrepancy about which source-language vector is used during task-vector training. Section 4.2 states that for XLM-R base, task adaptation 'always apply[ies] the source language SFT from LT-SFT,' while §3.2 and Algorithm 1 (Appendix A, line 24) describe DeFT-X as computing its own source-language vector via DEFT-X. If the reported DeFT-X task vectors were trained with an LT-SFT source vector, the final composed model is not purely DeFT-X, and the robustness claim in Table 4 is hard to interpret. Please clarify which source vector was used for each table, or rerun the base-model experiments with DeFT-X's own source vector.","section":"§3.2, §4.2, Appendix A"}],"minor_comments":[{"comment":"The symbol n is used both for the column dimension of W ∈ R^{m×n} and for the number of retained entries in the top-n mask; please rename one of them to avoid ambiguity.","section":"Eq. (4)"},{"comment":"The claim that DEFT-X 'results in lower overlap' is made without quantitative support; please report mean overlap, standard deviations, and the number of tasks/languages averaged over.","section":"Figure 2 and Figure 3"},{"comment":"The terms 'lower-order components' and 'higher-order components' are nonstandard and are used in the opposite direction of the usual spectral terminology; please define them explicitly at first use.","section":"§3.1"},{"comment":"There is a typo in 'initilizalized'; it should be 'initialized.'","section":"Appendix E"},{"comment":"The paper does not mention whether code will be released; given the number of hyperparameters and configuration choices, a public implementation would substantially aid reproducibility.","section":"§4.2, general"}],"recommendation":"major_revision","confidential_remarks":"The paper is a modest incremental contribution over LT-SFT, and the editor should weigh whether the journal's bar for empirical novelty is met. The main empirical claim rests on very small average gains and the best of several rank configurations; the internal inconsistency between the denoising rationale and Table 5 is the key correctness-risk point. I do not see evidence of circularity, and the per-language results are transparently reported, which is in the paper's favor."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"DeFT-X is a modest, clearly-described extension of LT-SFT: before magnitude pruning, it applies SVD low-rank approximation to the delta weight matrices, retaining the top r singular components plus the 5% largest-magnitude entries of the residual, then sparse fine-tunes and composes language/task vectors as in LT-SFT. The empirical contribution is honest and useful: average +1.2 F1 on NusaX and +0.3 accuracy on AmericasNLI over LT-SFT on XLM-R base, with the best configurations beating MAD-X as well. The ablations in Table 5 are genuinely informative: each pipeline step (denoised residual, magnitude pruning, sparse fine-tuning) contributes, and the parameter-overlap analysis is a nice extra.\n\nThe soft spots are real. First, no significance testing or error bars anywhere; the gains are small and the best rank/retention configurations are selected from many variants, so the reported averages are optimistic. Second, the paper's own Table 5 undercuts the motivating story in §3.1. The text says higher-order (low singular value) components are 'more likely to capture uninformative or noisy artefacts,' but the ablation shows that removing all higher-order components drops NusaX F1 from 81.4 to 80.6, and the paper itself says they contain useful information. That is not fatal—the method can be read as 'keep a sparse slice of the residual rather than all of it'—but the framing as denoising is sharper than the evidence supports. More importantly, the 0.8-point value of the retained residual is close to the whole 1.2-point gain over LT-SFT, so the improvement may owe more to the arbitrary choice of retaining 5% of the higher-order components than to the SVD denoising per se. The paper does not vary the retention fraction or show it generalizes. Third, no code is released, which matters for a method whose value is incremental.\n\nWho is this for? People working on parameter-efficient or sparse cross-lingual transfer will want to know about it, and the ablations are worth reading even if the headline result is thin. It deserves a serious referee—the method is plausible and the experiments are reproducible in principle—but the authors should be pushed to add significance tests, vary n, and reframe the noise story.","headline":"A modest, clearly-described extension of LT-SFT with a useful ablation table, but the paper's own ablation undercuts its denoising story and the gains are small and statistically untested.","tokens_in":15761,"tokens_out":2816,"would_cite":true,"duration_ms":24028,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SVD-denoised sparse fine-tuning improves zero-shot cross-lingual transfer.","keywords":["zero-shot cross-lingual transfer","sparse fine-tuning","singular value decomposition","low-rank denoising","lottery ticket hypothesis","NusaX","AmericasNLI","task arithmetic"],"falsifier":"Run DeFT-X on NusaX with the retention fraction for higher-order components set to 0%, 1%, 5%, 10%, and 20% while holding everything else fixed. If the average F1 does not peak near 5% and exceed LT-SFT's 80.2, then the paper's signal-noise narrative is not what carries the result.","tokens_in":14797,"feed_emoji":"🌐","tokens_out":5547,"duration_ms":42463,"temperature":0.7,"pith_summary":"The paper introduces DeFT-X, a composable sparse fine-tuning method for zero-shot cross-lingual transfer. It claims that applying singular value decomposition to denoise the weight-update matrix before magnitude-based pruning yields sparse fine-tuned vectors that transfer better to low-resource languages when composed. On NusaX sentiment analysis, DeFT-X reaches an average F1 of 81.4 versus 80.2 for the LT-SFT baseline, and on AmericasNLI it reaches 51.3 accuracy versus 51.0. The practical payoff would be a small but consistent way to improve modular, parameter-efficient transfer to languages with no labeled data.","feed_headline":"SVD-denoised sparse tuning beats LT-SFT on low-resource transfer","feed_subtitle":"SVD denoising before magnitude pruning lifts NusaX F1 to 81.4 and AmericasNLI accuracy to 51.3.","key_machinery":"The load-bearing object is the SVD-denoised weight update: for each matrix $W$ in $\\Delta W$, form $L = U_r \\Sigma_r V_r^T$ from the top-$r$ singular components, keep a mask $m$ over the top $n$ entries of $|W-L|$, set $S = m \\odot (W-L)$, and replace $W$ by $L+S$. This is followed by magnitude pruning of the entire denoised $\\Delta W$ and a sparse fine-tuning pass that resets to $\\theta^{(0)}$ and trains only the masked parameters. The mechanism is intended to separate high-singular-value structure from low-singular-value noise while retaining a sliver of the noise that the ablation shows is still useful, and it reduces overlap between language and task vectors.","core_discovery":"DeFT-X's central claim is that the difference between a fully fine-tuned model and the pretrained model, $\\Delta W = \\theta^{(1)} - \\theta^{(0)}$, contains noisy higher-order components that degrade the sparse vectors used for transfer. Each weight matrix in $\\Delta W$ is decomposed as $W = U\\Sigma V^T$, keeping a low-rank part $L$ built from the largest singular values, then magnitude-pruning the residual $W-L$ down to a small fraction (5%) rather than discarding it outright. The reconstructed, denoised $\\Delta W$ is then magnitude-pruned and sparsely fine-tuned to produce task-specific and language-specific vectors that are composed by addition, $\\theta_{TL} = \\theta^{(0)} + \\phi_T + \\phi_L$. In the paper's best configurations this improves zero-shot transfer over LT-SFT on both benchmarks, including a gain of 1.2 F1 points on NusaX and 0.3 accuracy points on AmericasNLI.","pith_inferences":["The improvement may owe more to the fixed 5% retention fraction and to the extra magnitude-pruning step than to a genuine separation of signal from noise: the paper's own ablation shows that removing the higher-order components entirely costs 0.8 F1 points.","If the 5% fraction is a hidden tuning knob, then a testable consequence is that the gain over LT-SFT should vary with that fraction; a sweep over retention rates would clarify whether denoising or the chosen fraction drives the result.","Because rank selection is manual and model-specific, a data-driven criterion (for example, choosing rank by validation performance on the source language) would turn DeFT-X from a recipe into a principle.","The same denoising step should apply to task arithmetic beyond cross-lingual transfer, e.g., merging models for multi-task learning, where noisy updates are also composed additively; the authors hint at this but do not test it."],"forward_implications":["On NusaX, the best DeFT-X configuration raises average F1 from 80.2 (LT-SFT) to 81.4 with XLM-R base, and improves several individual languages such as Minangkabau.","On AmericasNLI, DeFT-X with $r_l = 200$ and $r_t = 90\\%$ variance reaches 51.3 average accuracy versus 51.0 for LT-SFT.","DeFT-X preserves performance when task vectors are trained without source-language initialization, whereas LT-SFT degrades, suggesting the denoised vectors are more robust to composition.","Parameter overlap between language and task vectors drops with DeFT-X, implying less destructive interference when vectors are added.","DeFT-X also improves over LT-SFT on the mE5 base embedding model (from 72.1 to 74.0 average F1), indicating the benefit is not tied to XLM-R alone."],"supporting_citations":[{"why":"Provides the LT-SFT baseline and the composable sparse fine-tuning template that DeFT-X extends.","marker":"Ansell et al., 2022"},{"why":"Provides the MAD-X adapter baseline that DeFT-X compares against for zero-shot cross-lingual transfer.","marker":"Pfeiffer et al., 2020"},{"why":"Supplies the lottery ticket hypothesis that motivates identifying sparse, trainable subnetworks via pruning.","marker":"Frankle and Carbin, 2019"},{"why":"Supplies the 90% variance criterion used for selecting the SVD rank during denoising.","marker":"Chang et al., 2022"},{"why":"Shows that pruning small singular components of transformer weights can denoise internal representations, a direct precedent for the denoising step.","marker":"Sharma et al., 2023"},{"why":"Provides the low-rank and sparse merging approach that combines coarse singular-value pruning with fine magnitude pruning.","marker":"Zhao et al., 2025"},{"why":"Supplies the training data and protocol, including the de-contaminated NusaX test set used in the evaluation.","marker":"Ansell et al., 2023a"}],"fun_headline_variants":["SVD denoising improves sparse fine-tuning for cross-lingual transfer","Denoised sparse vectors beat LT-SFT on zero-shot transfer","DeFTX: SVD-denoised masks top LT-SFT on low-resource tasks","Cleaner sparse vectors from SVD pruning boost cross-lingual transfer","SVD-denoised sparse tuning beats LT-SFT on low-resource transfer"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that low singular value components of the weight update are mostly noise, yet the paper's own ablation shows that removing them outright hurts performance, so the benefit actually depends on the unexamined decision to keep 5% of them.","fun_headline_variants_meta":{"raw":{"variants":["SVD denoising improves sparse fine-tuning for cross-lingual transfer","Denoised sparse vectors beat LT-SFT on zero-shot transfer","DeFTX: SVD-denoised masks top LT-SFT on low-resource tasks","Cleaner sparse vectors from SVD pruning boost cross-lingual transfer","SVD-denoised sparse tuning beats LT-SFT on low-resource transfer"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000859,"raw_usage":{"total_tokens":3768,"prompt_tokens":1026,"completion_tokens":2742,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":642,"completion_tokens_details":{"reasoning_tokens":2639}},"tokens_in":642,"tokens_out":2742,"duration_ms":16866,"temperature":1.0,"reasoning_tokens":2639,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:24:05.002473+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run DeFT-X on NusaX with the retention fraction for higher-order components set to 0%, 1%, 5%, 10%, and 20% while holding everything else fixed. If the average F1 does not peak near 5% and exceed LT-SFT's 80.2, then the paper's signal-noise narrative is not what carries the result.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the lottery ticket hypothesis that motivates identifying sparse, trainable subnetworks via pruning."},{"cited_title":"Low-Rank and Sparse Model Merging for Multi-Lingual Speech Recognition and Translation","cited_arxiv_id":"2502.17380","evidence_quote":"Provides the low-rank and sparse merging approach that combines coarse singular-value pruning with fine magnitude pruning."}],"review_version":1}