{"id":"2646e62e-c417-4680-8000-4785c53e07d8","arxiv_id":"2412.19547","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A two-stage multi-task training method that trains auxiliary tasks equally in task-specific decoders and weights their shared-encoder gradients by uncertainty and gradient norm improves primary-task performance relative to prior auxiliary-task weighting methods.","lead":"This paper proposes a two-stage weighting scheme for multi-task learning where auxiliary tasks are trained with equal emphasis in the decoder and then weighted by uncertainty and gradient norms in the shared encoder. The method reports the best primary-task results among compared methods on NYUv2, Cityscapes, PASCAL-Context, and CIFAR-100, including settings with noisy pseudo-labeled auxiliary tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The encoder-stage weighting rests on an unvalidated assumption that learned uncertainty tracks auxiliary-task training level; sigma_t is optimized against training loss and the mapping g in Eq. 9 is unspecified.","rationale":"The reader's weakest assumption identifies exactly the point on which the central claim depends: the encoder-stage weighting uses uncertainty as a proxy for auxiliary-task quality, and this proxy is neither derived nor validated. My reading of the method section confirms the gap: Eq. 3 defines the decoder-stage loss and Eq. 4 updates sigma_t based solely on L_t, which makes sigma_t a loss-tracking quantity rather than a measure of generalization or transfer. The paper's own statement in Sec. III-B defines 'training levels' by generalization performance, but no experiment connects the estimated sigma_t to any generalization measure. This is load-bearing because the encoder-stage weighting in Eq. 9 is the component that is supposed to 'promote well-optimized auxiliary tasks'; without it, the observed gains could be attributed entirely to the known Uncertainty Weight decoder stage plus the simple gradient re-normalization. The missing specification of g(.) compounds the problem: the exact algorithm cannot be checked or reproduced from the text, and the claim that weights 'typically do not exceed 0.6' is unverifiable. I agree with the reader that this warrants a conditional decision: the experimental results are consistently reported with five seeds and the ablations are informative, but the core mechanism needs an explicit definition and a direct validation. The internal contradictions in the pseudo-task section (e.g., the token-label discussion in Sec. IV-D2 and the Failure Case percentages in Sec. IV-E) further reduce confidence in the reporting but are not the primary vulnerability; I therefore keep the verdict unchanged rather than moving to reject or unverdict, while noting that the requested clarification and test should be mandatory before final acceptance.","tokens_in":26198,"tokens_out":7555,"duration_ms":70240,"concrete_test":"On NYUv2 with semantic segmentation as primary, log at every epoch: sigma_t for depth and normals, their validation metrics, and the actual f(sigma_t) used. Compute Spearman rank correlation between sigma_t (or 1/(2 sigma_t^2)) and each auxiliary validation metric across training. Then rerun the encoder-stage weighting replacing f(sigma_t) with a direct function of the auxiliary validation loss (same gradient normalization) and compare final primary mIoU and delta-MTL to Table VIII Ours. If the correlation is weak or negative and the validation-based weighting does not reproduce the reported improvement, the central mechanism is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that the uncertainty sigma_t estimated in the decoder stage (Eq. 3) is a faithful measure of an auxiliary task's 'training level' (generalization quality), because Eq. 9 converts it into the encoder weight f(sigma_t). This is not established. From Eq. 4, sigma_t is optimized on the task's training loss L_t, so at its stationary point sigma_t is roughly sqrt(L_t); it therefore tracks the instantaneous training loss, not validation performance or positive transfer to the primary task. The paper provides no correlation or intervention evidence linking low sigma_t to auxiliary tasks whose features improve the primary task. The mapping g(.) in Eq. 9 is never defined, so the actual encoder weights are unreproducible and the reported values ('typically not exceed 0.6') cannot be checked. If this proxy is invalid, the improvement attributed to 'well-trained auxiliary tasks' may simply be the effect of gradient-norm balancing (Eq. 8) plus the known Uncertainty Weight decoder stage (Eq. 3), and the paper's causal mechanism is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an Impartial Auxiliary Learning (IAL) framework for multi-task learning with a designated primary task. The method operates in two stages: in the decoder stage, task-specific decoders are trained with uncertainty-based weighting (Eq. 3) so that each task's decoder is trained independently of the primary task; in the encoder stage, auxiliary-task gradients are reweighted by a function of the same uncertainty (Eq. 9) and normalized in norm to the primary-task gradient (Eq. 8) before being summed with the primary-task gradient (Eq. 7). The authors evaluate on NYUv2, Cityscapes, PASCAL-Context, and Multi-CIFAR100, including settings where two large-scale pre-trained models provide noisy pseudo auxiliary tasks, and report that the method improves primary-task performance relative to existing task-weighting baselines. The central claim is that inadequately trained auxiliary tasks hurt the primary task, and that balancing auxiliary-task training through uncertainty weights and gradient-norm normalization improves the primary task.","tokens_in":26403,"tokens_out":8918,"duration_ms":79543,"significance":"If the central claim holds, the paper offers a simple and computationally light mechanism for improving a primary task in multi-task learning, with a distinctive focus on the training quality of auxiliary tasks rather than only on minimizing interference. The empirical evaluation is a genuine strength: four benchmarks, multiple tasks per benchmark, five random seeds, reimplemented baselines, and a set of ablations (Tables VIII-XI) that separately assess gradient normalization, uncertainty-based weighting, and impartial decoder training. The pseudo-task experiments address an interesting and practically relevant robustness question. However, the load-bearing mechanism is incompletely specified: the mapping g in Eq. (9) is never defined, and the claim that uncertainty sigma_t measures a task's 'training level' is not validated beyond a correlation with training loss. If the missing specification and the internal inconsistencies in the pseudo-task and ablation tables are fixed, the empirical contribution could be valuable for the multi-task learning community.","major_comments":[{"comment":"The function g(.) in Eq. (9) is never defined. This is load-bearing because f(sigma_t) determines the encoder-stage weight of every auxiliary task, and the statement in Section IV-G5 that 'these weights typically do not exceed 0.6' cannot be checked or reproduced without knowing g. Please provide the explicit form of g, its hyperparameters, and the admissible range of sigma_t; if sigma_t can exceed 1, the argument (1 - x) to g must also be defined and its behavior outside the nominal range clarified.","section":"III-C, Eq. (9)"},{"comment":"The claim that task uncertainty sigma_t measures a task's 'training level' (defined in Section III-B as generalization performance) is not established. From Eqs. (3)-(4), the stationary point satisfies sigma_t^2 approx L_t, so sigma_t tracks the instantaneous training loss, not validation performance or positive transfer to the primary task. The fixed-head experiment in Section IV-G3 is intended to support this, but the text says the uncertainty of the fixed (broken) depth task 'obviously drops', which is the opposite of what the proposed weighting direction would predict if the fixed task is poorly trained. Moreover, Table X appears to report the pseudo-task results rather than the standard NYUv2 numbers (see Major Comment 4). Please provide a direct correlation or intervention analysis, for example plotting sigma_t against per-task validation metrics over training, or deliberately degrading an auxiliary task and showing that f(sigma_t) decreases and the primary-task weight protects the encoder.","section":"III-B and IV-G3"},{"comment":"The encoder-stage update is not the gradient of the loss in Eq. (6). Eq. (7) replaces each gradient nabla_z L_t with the normalized gradient Norm(nabla_z L_t) from Eq. (8), which is not a gradient of L_t, so the method does not optimize a single well-defined objective. Additionally, the algorithm in Section III-C first updates the decoders and uncertainty (step 2) and then computes shared-feature gradients from Eq. (6) using the same forward activations (step 3); after the decoder update, those activations are stale. Please specify the exact computation order, for example whether nabla_z L_t is computed before the decoder update and whether a second forward pass is used, and state explicitly that Eq. (7) is a modified update rule rather than a gradient of Eq. (6).","section":"III-C, Eqs. (6)-(10)"},{"comment":"The noisy-task results are internally inconsistent. Table IV reports Ours with Delta-MTL +1.99% on NYUv2 and Table V reports +8.90% on Cityscapes, yet Section IV-D2 states that 'with token labels -0.45% v.s. -0.44% without auxiliary tasks' and Section IV-D3 states that the pseudo tasks 'do not promote but drop in some tasks'. In addition, Table X (ablation 'Influence of Worse Auxiliary Tasks') lists Ours as 0.5732/44.82/19.36 with Delta-MTL +2.43%, which is exactly the Table IV pseudo-task row, not the standard NYUv2 row from Table I (0.5751/44.60/20.62, -0.44%). The ablation conclusions in Section IV-G3 therefore appear to be based on a configuration that is not the standard NYUv2 benchmark. Please correct the tables and text, and label each table with the exact task configuration (with or without pseudo tasks, with or without token labels).","section":"IV-D2/D3 and Tables IV, V, X"}],"minor_comments":[{"comment":"Eq. (4) contains '1/x' where the derivative of log sigma_t should appear; this should be 1/sigma_t.","section":"III-C, Eq. (4)"},{"comment":"The definition of Delta-MTL is garbled: 'For the task li = 1, the lower values of M_i (classification accuracies), the better performance' and 'For the task li = 0, the higher values of M_i (L1 errors for depth prediction)' mix the two cases. It should read: li = 1 for lower-is-better metrics such as errors, li = 0 for higher-is-better metrics such as accuracies.","section":"IV-C1, Eq. (11)"},{"comment":"The heading 'Inplacement of the uncertainty in the encoder stage' should be 'Replacement of the uncertainty in the encoder stage'.","section":"IV-G4"},{"comment":"The phrase 'regardless the noise auxiliary tasks' pseudo labels' is grammatically incomplete; suggest 'regardless of noise in the auxiliary tasks' pseudo labels'.","section":"Abstract and Introduction"},{"comment":"Step 3 of the algorithm refers to Eq. (6) as the objective, but the actual encoder update uses the normalized gradient in Eq. (7) and Eq. (8); please reference the equations consistently so the reader can reproduce the update.","section":"III-C, step 3"}],"recommendation":"major_revision","confidential_remarks":"The paper has a solid empirical backbone and a plausible, simple idea, but the current version cannot be recommended for publication as is. The missing definition of g in Eq. (9) and the apparent mix-up between standard and pseudo-task results in Table X are fixable but require careful revision. I would like the editor to ensure the authors address the inconsistency between the textual claims in Section IV-D2/D3 and the positive Delta-MTL numbers in Tables IV and V, since that discrepancy directly affects the paper's headline robustness claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid MTL paper with a real empirical finding and a genuinely new two-stage recipe, but the central weighting function is never concretely defined, and a few text/table inconsistencies need cleaning up before the method can be reproduced.\n\nThe new thing is the split: decoders are trained with Kendall-style uncertainty weighting, then the encoder is updated with gradient norms rescaled to the primary task and multiplied by an uncertainty-derived weight. That combination is new, and the pseudo-task setting (using COCO/ImageNet pre-trained models to generate noisy auxiliary labels) is a useful stress test. The empirical work is better than the field average: four benchmarks, five seeds, all baselines reimplemented, training time reported, and the ablations (Tables VIII, IX, XI) show both modules contribute and that uncertainty beats alternatives like relative training speed and expected gradient norm. That is real evidence.\n\nThe soft spots are just as real. Eq. (9) defines f(x) = min(1, g(1-x)) but never says what g is. That means the core encoder weights cannot be computed or checked. The stress-test note is right that sigma_t is optimized on each task's own training loss (Eq. 4), so at stationarity it roughly tracks that loss, not generalization. Calling it a 'training level' is an unvalidated leap. The fixed-head experiment and Table XI give some empirical grounding, but the text is confused (it says uncertainty drops for a broken head, which is odd, and the caption doesn't clarify). The paper also contains several internal contradictions: Sec. IV-D2 says with token labels the gain is -0.45% vs -0.44%, while Table IV reports +1.99%; the failure-case section quotes decreases that don't match Table IV. These make the pseudo-task results hard to trust as written.\n\nNet: the central claim that better-trained auxiliary tasks help the primary is defensible and mostly supported. The mechanism is plausible but under-specified. This is a paper a serious referee should see, but it needs a revision: define g, validate the uncertainty-to-quality mapping (e.g., correlation with per-task generalization or an intervention), and fix the contradictory passages. I'd bring it to a reading group to discuss the uncertainty proxy issue, but I wouldn't cite it in this form.","headline":"Solid empirical MTL paper with a new two-stage weighting idea, but an undefined mapping and some inconsistent tables keep it from being reproducible as submitted.","tokens_in":26931,"tokens_out":3211,"would_cite":false,"duration_ms":28024,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Auxiliary tasks trained without bias improve the primary task in multi-task learning.","keywords":["multi-task learning","auxiliary tasks","task weighting","uncertainty estimation","gradient balancing","pseudo-label robustness","dense prediction","encoder-decoder networks"],"falsifier":"Train IAL on a dense-prediction benchmark while replacing the uncertainty weights with a deliberately scrambled ordering of the same per-task uncertainty values: if delta-MTL stays roughly the same, uncertainty ranking is not what drives the gain; if it drops, the ordering is load-bearing. A complementary test is to add an auxiliary task whose labels are randomized after it is well learned; the method should downweight it and keep primary performance near the no-auxiliary baseline, whereas a method that merely trains all tasks equally would suffer negative transfer.","tokens_in":25994,"feed_emoji":"🎯","tokens_out":9502,"duration_ms":90369,"temperature":0.7,"pith_summary":"Multi-task learning with auxiliary tasks usually gives the auxiliary losses much smaller weights than the primary loss, so the auxiliary decoders are undertrained. This paper argues that undertrained auxiliary tasks actively hurt the primary task, and proposes Impartial Auxiliary Learning (IAL), which trains every task's decoder with weights derived only from that task's uncertainty, then feeds the encoder with auxiliary gradients that are re-normalized to the primary gradient norm and scaled by an uncertainty-dependent weight below 1. The central claim is that this makes auxiliary tasks better trained, and that this in turn improves the generalization and accuracy of the primary task. The paper reports that IAL achieves the best relative multi-task improvement (delta-MTL) among compared methods on NYUv2, Cityscapes, PASCAL-Context, and Multi-CIFAR100, and remains beneficial when noisy pseudo-labels from large pre-trained models are added as auxiliary tasks. If the paper is right, a simple weighting scheme can replace careful auxiliary-task selection and still lift primary-task performance.","feed_headline":"Fair auxiliary-task training lifts primary-task performance","feed_subtitle":"Uncertainty-based weighting trains auxiliary tasks fully, then lets well-trained tasks improve the primary task.","key_machinery":"The central object is the two-stage loss decomposition around the shared encoder. In the decoder stage, task-dependent homoscedastic uncertainty $\\sigma_t$ is estimated from each task's own loss via the objective $L_{\\mathrm{MTL}}=\\sum_t \\frac{1}{2\\sigma_t^2}L_t + \\log\\sigma_t$, so each decoder's weight depends only on its own output. In the encoder stage, the auxiliary task's contribution is $f(\\sigma_t)\\operatorname{Norm}(\\nabla_z L_t)$, where $\\operatorname{Norm}(\\nabla_z L_t)=\\frac{|\\nabla_z L_{\\mathrm{pri}}|}{|\\nabla_z L_t|}\\nabla_z L_t$ re-scales the auxiliary gradient to the primary gradient's magnitude, and $f(\\sigma_t)=\\min(1,g(1-\\sigma_t))$ maps uncertainty to a weight in $[0,1)$; the mapping $g(\\cdot)$ is not specified. This mechanism lets the framework train auxiliary decoders without prejudice while still guaranteeing the primary task's dominance in the shared encoder, so that better-trained auxiliary tasks can transfer knowledge to the primary task.","core_discovery":"The paper's load-bearing claim is that inadequate training of auxiliary tasks is a main cause of poor primary-task performance in multi-task learning, and that fixing this rather than merely filtering out task interference is what unlocks positive transfer. Its proposed IAL framework splits optimization into two stages: in the decoder stage, each task loss is weighted by $1/(2\\sigma_t^2)$ with a learned task-dependent uncertainty $\\sigma_t$, so auxiliary decoders are trained as thoroughly as the primary decoder; in the encoder stage, auxiliary gradients are first re-normalized to the magnitude of the primary gradient and then weighted by $f(\\sigma_t)=\\min(1,g(1-\\sigma_t))$, keeping the primary task dominant while letting well-trained auxiliary tasks contribute more. The reported experiments support the claim: on NYUv2 IAL records the best delta-MTL among all compared methods ($-0.44\\%$), on Cityscapes it records $+8.22\\%$, and in a noisy setting where two pseudo-tasks built from pre-trained model predictions are added, IAL still improves the primary tasks ($+1.99\\%$ on NYUv2 and $+8.90\\%$ on Cityscapes) while baselines degrade. The paper concludes that better-optimized auxiliary tasks lead to improved primary-task performance, and that uncertainty estimated from each task's own loss is a usable measure of the auxiliary task's training level.","pith_inferences":["The paper leaves $g(\\cdot)$ unspecified; a natural test is whether any monotone mapping that ranks tasks by uncertainty reproduces the gains, which would show that the ordering, not the exact function, is what matters.","The pseudo-task setup opens a semi-supervised variant the paper does not explore: use a frozen large model to label an unlabeled dataset, and rely on the weighting to suppress domain-shifted labels while keeping primary-task training stable.","The failure-case analysis suggests equal training alone can cause negative transfer; a direct comparison of decoder-impartial-only versus encoder-gated variants on a strongly conflicting task pair would isolate where the encoder gate matters.","One could connect this to gradient-projection methods by replacing the norm re-scaling with orthogonal projection while keeping the uncertainty ranking; the ablations suggest uncertainty ordering would still be the main driver."],"forward_implications":["If IAL is right, a single weighting scheme—uncertainty for decoders, uncertainty plus gradient norm for the encoder—is enough to beat task-specific weighting heuristics and meta-learned weights on standard dense-prediction benchmarks.","Because the method improves the primary task by improving auxiliary-task training, even auxiliary tasks chosen for their similarity to the primary task are only useful if they are trained to a comparable degree.","The pseudo-task experiments imply that noisy auxiliary labels from large pre-trained models can be exploited rather than discarded; the weighting can suppress harmful tasks while letting informative ones contribute, expanding the available auxiliary data without manual annotation.","The results across NYUv2, Cityscapes, PASCAL-Context, and Multi-CIFAR100 suggest the benefit is not limited to dense prediction; classification and multi-domain settings also see primary-task gains."],"supporting_citations":[{"why":"Supplies the decoder-stage uncertainty weighting and the estimate of task-dependent homoscedastic uncertainty $\\sigma_t$.","marker":"[16]"},{"why":"Provides the NYUv2 benchmark used for the main dense-prediction evaluations.","marker":"[17]"},{"why":"Provides the Cityscapes benchmark used for the main dense-prediction evaluations.","marker":"[18]"},{"why":"Baseline auxiliary-task weighting method that the paper argues under-trains auxiliary tasks; used as a main comparison and in Fig. 1.","marker":"[13]"},{"why":"Meta-learning task-weighting baseline (Auto-lambda) that the paper compares against in auxiliary and pseudo-task settings.","marker":"[15]"},{"why":"Gradient-norm normalization baseline; the paper adapts its gradient magnitude idea for the encoder stage.","marker":"[31]"},{"why":"IMTL impartial multi-task learning baseline; the paper contrasts its decoder-stage impartiality with IMTL's global impartiality.","marker":"[22]"},{"why":"Gradient cosine similarity auxiliary task weighting baseline evaluated in the auxiliary setting.","marker":"[14]"},{"why":"Source of the method for constructing pseudo-tasks from pre-trained model predictions, used in the noisy auxiliary-task experiments.","marker":"[10]"}],"fun_headline_variants":["Impartial auxiliary training boosts primary task","Uncertainty weighting makes auxiliary tasks help more","Fully train auxiliary tasks to lift primary performance","Balanced multi-task training via learned uncertainty","Auxiliary tasks trained fully, primary task improves"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the uncertainty $\\sigma_t$ estimated from a task's own loss is a faithful measure of that task's training quality, so lower uncertainty means the auxiliary task is well trained and will transfer positively to the primary task; the mapping $g(\\cdot)$ that turns uncertainty into an encoder weight is never specified, so if uncertainty does not track useful transfer, the encoder weighting has no sound basis.","fun_headline_variants_meta":{"raw":{"variants":["Impartial auxiliary training boosts primary task","Uncertainty weighting makes auxiliary tasks help more","Fully train auxiliary tasks to lift primary performance","Balanced multi-task training via learned uncertainty","Auxiliary tasks trained fully, primary task improves"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000174,"raw_usage":{"total_tokens":1296,"prompt_tokens":976,"completion_tokens":320,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":592,"completion_tokens_details":{"reasoning_tokens":265}},"tokens_in":592,"tokens_out":320,"duration_ms":3085,"temperature":1.0,"reasoning_tokens":265,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T00:12:51.020909+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train IAL on a dense-prediction benchmark while replacing the uncertainty weights with a deliberately scrambled ordering of the same per-task uncertainty values: if delta-MTL stays roughly the same, uncertainty ranking is not what drives the gain; if it drops, the ordering is load-bearing. A complementary test is to add an auxiliary task whose labels are randomized after it is well learned; the method should downweight it and keep primary performance near the no-auxiliary baseline, whereas a method that merely trains all tasks equally would suffer negative transfer.","supporting_citations":[{"cited_title":"Multi-task learning using uncertainty to weigh losses for scene geometry and semantics,","cited_arxiv_id":null,"evidence_quote":"Supplies the decoder-stage uncertainty weighting and the estimate of task-dependent homoscedastic uncertainty $\\sigma_t$."},{"cited_title":"Indoor segmentation and support inference from rgbd images,","cited_arxiv_id":null,"evidence_quote":"Provides the NYUv2 benchmark used for the main dense-prediction evaluations."},{"cited_title":"The cityscapes dataset for semantic urban scene understanding,","cited_arxiv_id":null,"evidence_quote":"Provides the Cityscapes benchmark used for the main dense-prediction evaluations."},{"cited_title":"Adaptive auxiliary task weighting for reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"Baseline auxiliary-task weighting method that the paper argues under-trains auxiliary tasks; used as a main comparison and in Fig. 1."},{"cited_title":"Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks,","cited_arxiv_id":null,"evidence_quote":"Gradient-norm normalization baseline; the paper adapts its gradient magnitude idea for the encoder stage."},{"cited_title":"Towards impartial multi-task learning,","cited_arxiv_id":null,"evidence_quote":"IMTL impartial multi-task learning baseline; the paper contrasts its decoder-stage impartiality with IMTL's global impartiality."},{"cited_title":"Multi-task self-training for learning general representations,","cited_arxiv_id":null,"evidence_quote":"Source of the method for constructing pseudo-tasks from pre-trained model predictions, used in the noisy auxiliary-task experiments."}],"review_version":1}