{"id":"178e3f02-7c4b-45e9-bd4d-e23db9489ecd","arxiv_id":"2411.18615","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"Sparse training with fixed per-neuron top-magnitude masks reduces the measured incidence of gradient conflict and improves multi-task learning performance across several backbones and gradient manipulation methods.","lead":"This paper proposes training only a subset of model parameters to reduce conflicts between tasks in multi-task learning. It reports that this sparse training approach lowers measured gradient conflicts and improves performance across several vision benchmarks and existing multi-task methods.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The GC-incidence claim rests on an unspecified metric: if p% for ST is measured on masked gradients, the reported conflict reduction is partly a projection artifact.","rationale":"The reader's weakest_assumption identifies precisely this metric ambiguity, and I agree it is the most load-bearing concern. The paper's strongest claim is about reducing the incidence of gradient conflict, and the only quantitative support is the p% tables. If the metric is computed on masked gradients for ST and raw gradients for baselines, the comparison is inconsistent and the reduction is partly tautological. The theoretical section (Eq. 12 and Appendix A) also has an inequality-direction error, but the empirical claim does not depend on that theory, so I do not rank it as the primary concern. The performance improvements across benchmarks are broad and generally consistent, which suggests the method has practical value, but the stated mechanism is not established. A conditional verdict remains appropriate until the authors specify and apply a single consistent definition of p% for all methods. My concrete test would settle whether the GC-reduction claim is an artifact; if it passes, the paper's central novelty is validated; if it fails, the paper reduces to an empirical observation that sparse fine-tuning can improve MTL performance, which is weaker than the claimed proactive conflict mitigation.","tokens_in":42075,"tokens_out":3697,"duration_ms":130832,"concrete_test":"Recompute Table 1's p% for every method using the raw per-task shared-encoder gradients before any masking or manipulation at the same checkpoints, exactly as done for Joint Train and PCGrad. If ST's reductions (e.g., 31.89→26.33) largely disappear or reverse, the central claim is a measurement artifact; if they persist, the claim is confirmed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 defines p% as the average fraction of task-pairs with conflicting gradients per iteration, but it does not specify whether the conflict is computed on raw per-task gradients, on gradients after manipulation, or on the masked gradients that ST actually updates. This ambiguity is load-bearing because the paper's central contribution is that sparse training proactively reduces the incidence of gradient conflict. For baseline methods (Joint Train, PCGrad, CAGrad, etc.), the reported values are consistent with being computed on the raw, unmanipulated shared-encoder gradients, since manipulation methods like PCGrad alter the gradients but do not change the raw conflict rate. If ST's numbers are instead computed on M·g_i and M·g_j (the effective update directions after masking), then the measured reduction from 31.89% to 26.33% partly reflects the dimension reduction rather than a genuine change in the optimization landscape: zeroing out large parts of the gradient vector can remove conflicting components regardless of the underlying task interference. The paper's own Fig. 2 describes ST as an 'orthographic/coordinate projection' of gradients, so the metric choice is not a minor implementation detail. Without a consistent definition, the headline claim 'ST reduces the incidence of gradient conflict' is unfalsifiable from Tables 1, 6, 10, and 13. The performance gains in Tables 2 and 3 could still be real, but they would not be evidence for the proposed mechanism as stated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes sparse training (ST) as a proactive mechanism to reduce gradient conflict (GC) in multi-task learning. The method selects a fixed subset of encoder parameters by per-neuron top-magnitude (PSN) and updates only those parameters, leaving the rest frozen; it can be combined with gradient-manipulation methods such as PCGrad, CAGrad, GradDrop, MGDA, IMTL-G, and NashMTL. The authors report that ST lowers the average incidence of GC across multiple datasets (NYUv2, CityScapes, CelebA, Clevr, SmallNORB) and architectures (MTAN, SAM, ViT, Swin), and that it improves overall task performance as measured by relative task drop (∆m%). They also provide ablations on sparsity ratio, parameter-selection strategy, and model size.","tokens_in":42373,"tokens_out":5844,"duration_ms":54692,"significance":"If the central claim holds, the paper identifies a simple, architecture-preserving lever for multi-task optimization: freezing most shared-encoder parameters via a static mask reduces gradient conflicts and improves downstream performance. The empirical sweep is broad and internally consistent, covering several dense-prediction and classification benchmarks, six (plus FAMO) gradient-manipulation baselines, and both randomly initialized and pre-trained vision backbones. The supplementary material includes three-seed results for MTAN on NYUv2 and a clear statement of hyperparameters. The main weakness is that the GC-incidence metric is under-specified, which makes the headline mechanism claim currently ambiguous and potentially circular; the theoretical section also contains mislabeled inequalities. These issues are fixable without changing the overall experimental design, but they must be resolved before the mechanism claim can be accepted.","major_comments":[{"comment":"The definition of p% does not state whether gradient conflicts are measured on raw per-task gradients, on gradients after gradient manipulation, or on the masked gradients that sparse training actually updates. This is load-bearing because the core claim is that ST reduces the incidence of GC: if p% for ST is computed on M·g_i and M·g_j while p% for baselines is computed on raw g_i and g_j, the reported reductions (e.g., Joint Train 31.89%→26.33% in Table 1) are partly an artifact of coordinate projection, not an independent property of the optimization landscape. Figure 2 explicitly describes ST as an orthographic/coordinate projection, so the metric choice is not a minor implementation detail. Please specify the gradient representation for every method in the p% tables, and re-report the incidence values on a common representation (e.g., the actual update vectors used by each method).","section":"§4.1, Eq. for p%; Tables 1, 6, 10, 13"},{"comment":"The theoretical analysis is mathematically mislabeled and does not support the GC-reduction claim. The chain in Appendix A shows L = min_Θ max_λ ... ≥ max_λ min_Θ ... ≥ min_Θ (L + ||(I−M)(θ_sha−θ_in_sha)||_2), so the regularized objective in Eq. (12) is a lower bound on the primal value L, not an upper bound. The main text says \"This can be transformed to optimize the upper bound L of regularized problem,\" which inverts the direction of the inequality. Moreover, Eq. (12) is a penalized relaxation, not an equivalent reformulation of the constrained problem in Eq. (10), and no argument connects this regularized objective to the actual projected-gradient updates used by ST. Please correct the inequality/labeling and either provide a rigorous link between sparse training and GC incidence or present this material explicitly as intuition.","section":"§3.3 and Appendix A"},{"comment":"The main performance tables lack error bars for most configurations. Table 7 (MTAN on NYUv2) reports mean±std over three seeds, but the flagship results on SAM (Table 2) and the summaries on CelebA, Clevr, SmallNORB, and CityScapes (Table 3) are single point estimates. The paper repeatedly claims that ST \"consistently improves\" all methods, but without repeated-seed statistics it is impossible to assess whether differences such as Joint Train Δm% 6.763→5.314 are meaningful. Please provide repeated-seed results for at least the main comparisons, or state the number of seeds and quantify uncertainty.","section":"Tables 2, 3, 7, 10, 11, 12"}],"minor_comments":[{"comment":"The phrase \"any gradient magnitude methods\" appears twice and should be \"any gradient manipulation methods.\"","section":"§4.2, first paragraph"},{"comment":"The mask M is defined as an |Θ|×|Θ| diagonal matrix; this works formally, but it is more natural to define a boolean vector m ∈ {0,1}^|Θ| and write the update as θ − α m ⊙ ∇L(θ). Consider using the vector notation to avoid confusion.","section":"§3.4, Definition 2"},{"comment":"\"(C) Different sparse methods\" should be \"(c) Different sparse methods\" for consistency with the other panels.","section":"Figure 5c caption"},{"comment":"In the MTAN results (Table 6), the reduction for MGDA w/ ST is only 0.39% over all epochs and 0.85% in the last 50%; this is a much weaker effect than for other methods. The text should acknowledge this exception rather than claiming uniform effectiveness.","section":"§4.2, Table 6"},{"comment":"The sentence \"we employed a batch size of 3 and searched for the optimal learning rate from the set{2e-4, 5e-5}, and then the best results are reported\" is grammatically awkward; split into two sentences.","section":"Appendix D.2"},{"comment":"The Recon paper [29] is cited only for the distribution of gradient angles, but Recon also proposes proactive GC reduction by converting conflicting layers into task-specific layers. Given that the present paper addresses the same goal, a brief comparison/discussion of the relationship between ST and Recon would help position the contribution.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The manuscript has a solid empirical contribution and the supplementary material is unusually detailed (per-epoch GC curves, three-seed MTAN results, explicit hyperparameters). The main obstacle is the under-specified GC-incidence metric, which is central to the paper's narrative. If the authors can clarify that the p% comparisons are made on the same gradient representation, or alternatively reframe the contribution as \"ST uses a projected gradient with fewer conflicts within the selected subspace,\" the paper could become publishable. The theoretical section should be corrected or explicitly demoted to intuition. I do not see a circularity problem in the performance results, since those are grounded in external test metrics."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read of arXiv:2411.18615. The empirical sweep is genuinely useful: fixed-mask, per-neuron magnitude-based sparse training applied to the shared encoder consistently improves over joint training and six gradient manipulation baselines across NYUv2, CityScapes, CelebA, Clevr, and SmallNORB, on MTAN, SAM, ViT, and Swin. That is a solid practical contribution, and the observation that gradient manipulation methods do not lower raw conflict incidence (MGDA even raises it) is a good confirmation of earlier work. The paper is also honest about its computational limits and includes mean±std for at least one full table.\n\nThe soft spots are real, and the biggest one is exactly what the stress-test note flags. Section 4.1 defines p% as the fraction of task pairs with conflicting gradients but never states whether the conflict is computed on raw gradients, post-manipulation gradients, or the masked gradients that sparse training actually updates. The baseline numbers (PCGrad roughly equal to Joint Train) strongly imply raw gradients are used for the non-sparse methods; if the ST numbers are computed on M·g_i and M·g_j, the reported reduction is partly a projection artifact. Figure 2's own \"coordinate projection\" language makes this ambiguity concrete. This is not a minor implementation detail: the paper's central claim is that ST proactively reduces conflict, and without a consistent metric that claim is unfalsifiable from the tables. The fix is straightforward: report both raw and effective (post-mask) conflict incidence for all methods, and justify which one is the right comparison.\n\nThe theory section is also thinner than its label suggests — the Lagrangian reformulation plus a citation to Fu et al. is not a novel bound. I would call that a minor overclaim. The main tables in the body lack error bars, though the appendix provides three-seed std for MTAN, so this is fixable. Tuning the sparsity ratio on Joint Train and then reusing it across methods is a mild optimistic bias, not a fatal one.\n\nWho benefits: anyone training multi-task vision models, especially with large pre-trained encoders. The practical recipe is likely to be adopted even if the mechanism story is revised. This paper deserves peer review — the empirical study is substantial and the metric issue is clear enough that a good referee can push for the right experiments. I would not desk-reject it, but I would not accept the current version either.","headline":"The performance gains look real and broad, but the headline claim about reducing gradient conflict rests on an unstated metric choice that makes the mechanism unfalsifiable as written.","tokens_in":42871,"tokens_out":3458,"would_cite":true,"duration_ms":40958,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Sparse training — updating only a fixed subset of encoder parameters chosen per neuron — cuts the incidence of gradient conflict in multi-task learning and improves all tested gradient-manipulation methods.","keywords":["multi-task learning","gradient conflict","sparse training","parameter-efficient fine-tuning","gradient manipulation","per-neuron selection","vision transformers"],"falsifier":"Re-run the NYU-v2/SAM setup and compute p% on the raw per-task gradients for both Joint Train and Joint Train with ST, with the mask applied only to the parameter update; if the gap between 31.89% and 26.33% vanishes or inverts, the conflict-reduction claim is an artifact of measuring on masked gradients.","tokens_in":41893,"feed_emoji":"🎯","tokens_out":5971,"duration_ms":49077,"temperature":0.7,"pith_summary":"Multi-task models often train several tasks through one shared encoder, and the tasks' gradients can point against each other — a phenomenon the paper calls gradient conflict. The paper argues that updating only a fixed subset of the encoder's parameters, chosen per neuron by magnitude, reduces how often such conflicts occur, and that this simple change improves multi-task performance across datasets and architectures. If true, this means a one-time mask applied before training is a cheap, architecture-preserving lever for multi-task learning, and it can be bolted onto existing gradient-manipulation methods such as PCGrad, CAGrad, GradDrop, MGDA, IMTL-G, and NashMTL to make them work better. The paper supports this with experiments on CNN and transformer backbones (MTAN, SAM, ViT, Swin) over dense-prediction and classification benchmarks.","feed_headline":"Sparse training slashes gradient conflict in multi-task learning","feed_subtitle":"A one-time per-neuron mask improves joint training and every gradient-manipulation method tested.","key_machinery":"The central object is a fixed binary mask $M$ on the shared encoder, computed once before training by the parameter-selection rule the paper calls PSN (per-neuron selection). For each neuron, PSN keeps the top-$K$ input connections by magnitude, so every neuron stays trainable while most weights are frozen; the update is $\\hat{\\theta}_{sha} = \\theta_{sha} - M \\frac{1}{T}\\sum_{t=1}^T \\nabla_{\\theta_{sha}} L_t$. The paper interprets this mask as projecting the averaged task gradient onto a lower-dimensional subspace, which reduces interference between tasks. The theoretical justification is a Lagrangian reformulation of the constrained objective into a regularized problem, citing the stability and generalization bound of parameter-efficient fine-tuning.","core_discovery":"The paper's central claim is that sparse training (ST) — updating only the highest-magnitude incoming weights of each neuron in the shared encoder, with all other encoder parameters frozen — reduces the incidence of gradient conflict, defined as negative cosine similarity between per-task gradients. On the NYUv2 benchmark with the SAM backbone, the paper reports that ST lowers the average incidence of gradient conflict from 31.89% to 26.33% over all epochs, and from 35.85% to 29.14% over the last 50% of training. The same pattern holds when ST is combined with all six gradient-manipulation baselines, and the performance gains are consistent on CelebA, Clevr, SmallNORB, NYU-v2, and CityScapes. The paper also reports that larger pre-trained models exhibit more gradient conflict, making them the clearest beneficiaries of the strategy.","pith_inferences":["The paper's conflict metric may be measured on masked gradients for sparse training; if so, part of the reported reduction would be baked into the measurement. A cleaner comparison would compute the incidence on raw per-task gradients for all methods.","The per-neuron top-$K$ rule is effectively a structured-pruning prior; testing adaptive masks refreshed every few epochs could reveal whether the conflict reduction is a property of sparsity itself or of the fixed mask.","The paper's regularization argument implies that sparse training should improve fine-tuning stability even in single-task settings, which the paper does not test."],"forward_implications":["Sparse training can be added to any gradient-manipulation method without changing that method's update rule, and the paper shows consistent performance gains across all six tested methods.","The conflict reduction grows in later training epochs, suggesting the mask stabilizes optimization trajectories rather than merely shrinking the effective learning rate.","Larger pre-trained models show more gradient conflict and benefit more from sparsity, making this a low-cost route for multi-task fine-tuning of foundation models.","Because the mask is fixed before training, the computational overhead is negligible compared to methods that update masks per iteration."],"supporting_citations":[{"why":"Defines gradient conflict as negative cosine similarity and supplies the PCGrad baseline that the paper's conflict metric and experiments build on.","marker":"[35]"},{"why":"The CAGrad baseline; the paper compares its gradient-conflict incidence against this method.","marker":"[17]"},{"why":"Supplies the theoretical result that the regularized objective (freezing parameters) has better stability and a smaller generalization bound, which the paper uses to justify sparse training.","marker":"[11]"},{"why":"Prior work showing that gradient manipulation does not reduce the incidence of conflicting gradients; the paper cites it to frame its proactive approach.","marker":"[29]"},{"why":"The NashMTL baseline and the experimental protocol (hyperparameters, evaluation) the paper follows for several benchmarks.","marker":"[26]"},{"why":"MTAN is the randomly initialized CNN architecture used to show sparse training also helps models without pre-training.","marker":"[20]"},{"why":"The SAM backbone used for the main NYU-v2 experiments on pre-trained large models.","marker":"[3]"}],"fun_headline_variants":["Sparse training lowers multi-task gradient conflict","Update only top weights to ease multi-task training","Sparse updates cut task gradient clashes in joint models","Frozen parameters reduce task interference in multi-task training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's central comparison assumes the gradient-conflict metric is measured on the same gradient quantity for both full and sparse training; Section 4.2 never states whether the incidence p% is computed on raw per-task gradients, post-manipulation gradients, or masked gradients.","fun_headline_variants_meta":{"raw":{"variants":["Sparse training lowers multi-task gradient conflict","Update only top weights to ease multi-task training","Sparse updates cut task gradient clashes in joint models","Frozen parameters reduce task interference in multi-task training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000912,"raw_usage":{"total_tokens":3887,"prompt_tokens":884,"completion_tokens":3003,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":500,"completion_tokens_details":{"reasoning_tokens":2943}},"tokens_in":500,"tokens_out":3003,"duration_ms":21681,"temperature":1.0,"reasoning_tokens":2943,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:00:15.970172+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the NYU-v2/SAM setup and compute p% on the raw per-task gradients for both Joint Train and Joint Train with ST, with the mask applied only to the parameter update; if the gap between 31.89% and 26.33% vanishes or inverts, the conflict-reduction claim is an artifact of measuring on masked gradients.","supporting_citations":[{"cited_title":"Gradient surgery for multi-task learning","cited_arxiv_id":null,"evidence_quote":"Defines gradient conflict as negative cosine similarity and supplies the PCGrad baseline that the paper's conflict metric and experiments build on."},{"cited_title":"Conflict-averse gradient descent for multi-task learn- ing","cited_arxiv_id":null,"evidence_quote":"The CAGrad baseline; the paper compares its gradient-conflict incidence against this method."},{"cited_title":"Multi- task learning as a bargaining game, 2022","cited_arxiv_id":null,"evidence_quote":"The NashMTL baseline and the experimental protocol (hyperparameters, evaluation) the paper follows for several benchmarks."},{"cited_title":"End- to-end multi-task learning with attention","cited_arxiv_id":null,"evidence_quote":"MTAN is the randomly initialized CNN architecture used to show sparse training also helps models without pre-training."},{"cited_title":"Sam fails to segment anything? – sam- adapter: Adapting sam in underperformed scenes: Camou- flage, shadow, and more, 2023","cited_arxiv_id":null,"evidence_quote":"The SAM backbone used for the main NYU-v2 experiments on pre-trained large models."}],"review_version":1}