{"id":"4e0d16ca-6182-4476-a7c2-71e84b66eb57","arxiv_id":"1909.02214","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Adding task-supervised auxiliary modules to shared layers during training improves hard-parameter-sharing multi-task learning on segmentation, depth, and surface normal prediction, with no inference-time overhead.","lead":"The paper proposes a training-time auxiliary module for multi-task neural networks that is removed at inference, and reports improved accuracy on indoor scene understanding tasks. The work matters because it offers a way to get multi-task benefits without extra inference cost.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The auxiliary-module benefit is not isolated from a loss-weight confound: deep supervision uses 0.1-scale losses while Eq. 3 uses unscaled auxiliary losses, so the architecture-specific claim is unsupported.","rationale":"I read the paper as making an empirical, architecture-level claim: equipping hard-sharing MTL with a trainable multi-level auxiliary module supervised by the same task labels improves all tasks at zero inference cost, and the module design (adaptors, aggregators, NAS-searched structure) is what delivers the gain. The strongest evidence is Table 3 (47.9 vs 42.9 mIoU) and Fig. 3. I looked for the weakest point in that chain. The paper's own deep-supervision experiment is the natural control, but it is not a fair control: the deep-supervision losses are scaled by 0.1 while the Eq. (3) auxiliary losses are not scaled. This confound is concrete and testable. I do not think the paper is fraudulent or the result impossible; the Table 3 gap is large enough that it is likely real, and the method is simple. But the explanatory claim—that the auxiliary module's inductive bias, rather than simply a stronger same-task gradient, produces the improvement—is not established. The reader's weakest_assumption about mechanism is in the same direction, but the reader did not identify the loss-weight mismatch, so my agreement is partial. A matched ablation would settle it. If the matched ablation shows no gap, the correct verdict remains conditional, but the condition should explicitly include showing a matched-weight deep-supervision baseline; if it shows a large gap in favor of Auxi, the central claim is supported. Either way, the paper would also benefit from error bars and code release, but those are secondary to the confound I identified.","tokens_in":12835,"tokens_out":5790,"duration_ms":56596,"concrete_test":"Run the Sec. 4.2 deep-supervision baseline on the same Baseline/Context networks with the per-branch auxiliary loss weight set to 1.0 (or with the total auxiliary weight matched to Eq. (3)) and with the same training schedule; compare against Auxi-both in Fig. 3 and Tab. 2. If the matched-weight deep supervision matches Auxi-both within a small margin, the auxiliary module's architecture is not the cause of the reported gains. Additionally, run Auxi-both with the auxiliary loss weight reduced to 0.1; if performance drops to the deep-supervision level, the effect is driven by loss weight rather than module design.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that the multi-level auxiliary module, not merely extra same-task supervision, is what improves hard-sharing MTL. The only control for this is the deep-supervision baseline, but it is not matched. In Sec. 4.2, deep supervision adds the same task losses at each downsampling layer 'with scale of 0.1'; in Eq. (3) the auxiliary losses enter with no coefficient, i.e. effective weight 1.0. The total auxiliary gradient magnitude is therefore different between the two conditions. The paper uses the limited gain of the 0.1-scaled deep supervision to argue that its architecture is responsible (Sec. 4.2 and Sec. 4.4), but no experiment varies the auxiliary loss weight, nor compares a simple per-layer classifier at weight 1.0 against Auxi-both. If the gain is simply due to a larger auxiliary gradient, then the proposed adaptor/aggregator design and the NAS search (Sec. 3.2, Sec. 4.4) are not the operative cause, and the paper's stated mechanism—an inductive bias from mimicking soft sharing—is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an auxiliary learning strategy for hard-parameter-sharing multi-task networks. During training, auxiliary modules are attached to intermediate outputs of the shared layers and are supervised by the same task losses as the main tasks, with the joint objective given in Eq. (3). At test time the auxiliary modules are removed, so the deployed network is the original hard-sharing model with no added inference cost. The method is evaluated on NYUD-v2 and SUNRGBD for semantic segmentation, depth estimation, and surface normal prediction, using MobileNetV2 and ResNet-50 backbones. The paper reports consistent improvements over joint-training baselines, comparisons with loss-weighting and prior MTL methods, and an automatically searched auxiliary module via reinforcement learning.","tokens_in":13064,"tokens_out":4418,"duration_ms":45287,"significance":"If the reported results hold, the method is a simple and practical training-only regularizer that improves hard-parameter-sharing MTL without changing the inference architecture. The paper's strengths include a clean objective with no tuned task weights, broad empirical coverage across backbones, datasets, and numbers of tasks, an efficient NAS procedure for the auxiliary module, and direct comparisons with Kendall et al.'s uncertainty weighting and other MTL baselines. The main weakness is that the central claim about the auxiliary module's architecture is not cleanly isolated from the scale of the auxiliary losses, so the mechanism attributed to the design is not yet fully supported.","major_comments":[{"comment":"The comparison between Deep supervision and Auxi is confounded by the scale of the auxiliary losses. Deep supervision is defined as adding losses \"at the end of each downsampling layer, and sum them all with scale of 0.1\" (Sec. 4.2), whereas Eq. (3) adds the auxiliary losses with no coefficient, i.e., effective weight 1.0. Therefore the observation that Deep supervision contributes little while Auxi improves both tasks cannot be attributed to the multi-level auxiliary module architecture; it may simply reflect a larger auxiliary gradient magnitude. To support the paper's mechanism claim, please report a matched ablation, for example deep supervision with per-loss weight 1.0 and/or Auxi with auxiliary losses scaled by 0.1, using the same set of tapped layers.","section":"Sec. 4.2, Eq. (3), Fig. 3"},{"comment":"The empirical conclusions rest on single-run point estimates with no error bars or significance tests. Several margins are small, e.g., Table 2 reports 35.02 vs 35.71 mIoU for Auxi-both with conv1x1 vs conv3x3, and Table 4 reports 53.7 vs 54.0 mIoU for Joint vs Auxi-T2 on segmentation. Without repeated runs or a significance test, these differences could be within run-to-run noise. Please report means and standard deviations over at least three runs, or an equivalent significance test, for the main comparisons in Tables 1-4 and Fig. 3.","section":"Tables 1-4"},{"comment":"The NAS component is evaluated without specifying the meta-train/meta-val split, the number of architectures sampled per search, or the variance of the search. Since the reported 1.9% mIoU gain of Auxi-NAS over Auxi-both (Fig. 3) is a selection result based on the meta-val reward, it is unclear whether this gain is a genuine property of the searched architecture or an artifact of selection bias. Please provide the split details and, ideally, evaluate the top-k searched structures or repeat the search to demonstrate stability.","section":"Sec. 3.2, Sec. 4.2"}],"minor_comments":[{"comment":"The phrase \"we get a significant 1.9% improvement\" uses \"significant\" in the non-statistical sense; please replace it with \"numerical improvement\" or provide a significance test to justify the statistical meaning.","section":"Sec. 4.2"},{"comment":"The notation for auxiliary-module parameters is inconsistent: the text uses θAt, while Eq. (3) writes θA1, ..., θAT. Please unify the notation to make clear which parameters belong to the t-th auxiliary module.","section":"Eq. (3)"},{"comment":"References He et al. 2016a and He et al. 2016b are the same paper (Deep Residual Learning for Image Recognition) and should be merged into a single citation.","section":"References"},{"comment":"Figure 3 is difficult to read because the markers and labels are small; a table with exact numerical values alongside the figure would make the comparisons clearer.","section":"Fig. 3"},{"comment":"The sentence \"we observe the auxiliary learning apparently boosts the performance\" is vague; please state the quantitative improvements and note whether they are consistent across the metrics reported in Table 4.","section":"Sec. 4.6"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the paper is in scope for the journal, the core idea is simple and potentially useful, and the experiments are broad. The main gap is a missing matched deep-supervision control that would isolate the architectural contribution from the loss-scale contribution; this is additive and feasible within the manuscript's scope. I have no concerns about citation patterns or novelty beyond the standard need to differentiate from deep supervision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper proposes a training-only auxiliary module for hard-parameter-sharing MTL, with the module removed at inference, and shows consistent gains across backbones and datasets. The specific combination — multi-level auxiliary inputs, trainable adaptors/aggregators, and NAS over the module — is new relative to the deep supervision literature.\n\nThe empirical work is substantial. The paper covers MobileNetV2 and ResNet-50, two and three tasks, NYUD-v2 and SUNRGBD, and reports improvements over Joint baselines that often exceed single-task performance. The objective in Eq. (3) is simple and the derivation is sound. That said, the central claim — that the multi-level architecture, not merely extra same-task supervision, explains the gains — is undercut by a confounded control. The deep supervision baseline in Sec. 4.2 uses a loss scale of 0.1, while the auxiliary losses in Eq. (3) are unscaled (coefficient 1). So the comparison conflates architecture with gradient magnitude. The paper even argues from the limited gain of the 0.1-scaled deep supervision that its architecture matters, but no experiment varies the auxiliary loss weight or tests a simple per-layer classifier at weight 1.0. That is a real gap.\n\nThere are also the usual empirical soft spots: no error bars or statistical tests, no code release, and the NAS search is only described at a high level. Those are not fatal but they do limit how much confidence one can place in the individual numbers.\n\nIf I were asked to referee this, I'd send it out. The idea is useful and testable, and the confound is fixable with a matched deep supervision baseline. A good referee could ask for that addition plus error bars, and the paper would come back much stronger.","headline":"Useful training-only auxiliary modules for MTL, but the architecture-specific claim is undercut by a confounded deep-supervision control; still worth peer review and a matched comparison.","tokens_in":13569,"tokens_out":2787,"would_cite":true,"duration_ms":27998,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Auxiliary modules supervised by task-specific losses improve hard-parameter-sharing multi-task networks with no inference cost.","keywords":["multi-task learning","hard parameter sharing","auxiliary module","inductive bias","semantic segmentation","depth estimation","surface normal prediction","neural architecture search"],"falsifier":"Train the same multi-task network with the auxiliary module replaced by direct deep supervision at the same intermediate feature maps, using the same auxiliary losses, loss scales, and training schedule. If this deep-supervision baseline matches the auxiliary-module results, the claimed benefit of the multi-level auxiliary architecture is not supported.","tokens_in":12624,"feed_emoji":"🧩","tokens_out":8117,"duration_ms":72377,"temperature":0.7,"pith_summary":"Multi-task networks that share one encoder across tasks are cheap to deploy but hard to train, because the different task losses pull the shared weights in competing directions. This paper claims that attaching temporary auxiliary modules to the shared hidden layers during training—each module reading multi-level features and supervised by one task's loss—regularizes the shared encoder and improves every task, at no extra inference cost because the modules are removed after training. On NYUD-v2, the method lifts joint semantic segmentation and depth estimation from 42.9 to 47.9 mIoU and from 0.143 to 0.127 relative error, and it also improves a three-task version on SUNRGBD. A careful reader would care because this is a training-only regularizer that makes hard parameter sharing competitive with single-task models while keeping deployment simple.","feed_headline":"Training-time auxiliaries boost multi-task vision at no inference cost","feed_subtitle":"Adding task-supervised side modules to shared encoder layers improves segmentation and depth with zero runtime overhead.","key_machinery":"The key machinery is the auxiliary module, a small side network formed from adaptors and aggregators. In the basic form, each adaptor is a $1\\times1$ convolution followed by batch normalization and ReLU, and the aggregator is either per-pixel summation or channel-wise concatenation; the module consumes $P$ intermediate feature maps from the shared encoder and outputs a prediction under the auxiliary task loss. Its role is to give the shared parameters extra gradient flow during backpropagation, acting as a regularizer that introduces inductive bias from the auxiliary task. For the searched variant, an LSTM controller with reinforcement learning chooses the adaptor operations, aggregation operations, and input locations for each task, using PPO to maximize a validation reward.","core_discovery":"The central claim is that the optimization difficulty of hard parameter sharing in multi-task learning can be relieved by explicit auxiliary modules that mimic soft parameter sharing during training. Each auxiliary module takes outputs from several shared hidden layers, passes them through trainable adaptors, aggregates the adapted features, and is supervised by the corresponding auxiliary task loss. Because the auxiliary module's parameters attach to the shared layers, its loss contributes extra gradient terms to the shared parameters, introducing a hierarchical inductive bias that helps balance shared and task-specific representations. After training, the auxiliary modules are discarded, so inference uses only the original multi-task network. The paper reports consistent gains across backbones, decoder designs, and two- and three-task settings, including results better than single-task baselines.","pith_inferences":["Because the auxiliary loss uses exactly the same ground truth as the main task, the reported gains cannot come from extra information; the likely active ingredient is the architectural pathway and multi-level aggregation. An ablation comparing the module with plain deep supervision at the same layers would isolate that ingredient.","The mechanism is not specific to dense prediction: any shared-encoder multi-task model with conflicting gradients should benefit from a training-only auxiliary pathway, so the method is a plausible plug-in for audio or NLP multi-task systems, though the paper does not test those settings.","If the architecture of the auxiliary module is what matters, one could search a single universal auxiliary cell on a small proxy pair of tasks and transplant it to new task combinations, avoiding a per-dataset search; this is a testable extension the paper leaves open."],"forward_implications":["Hard-parameter-sharing multi-task models can match or beat single-task baselines without any change to the inference-time architecture, so deployment speed and parameter count remain unchanged.","An auxiliary module for one task can improve that task even when the network is initialized from another task's pretrained weights, and auxiliary modules for related tasks such as surface normal and depth reinforce each other.","The gains appear across decoder variants and shared backbones, including MobileNetV2 and ResNet-50, so the training-time regularizer transfers across main architectures.","Automatically searching the auxiliary module architecture yields larger gains than the hand-designed basic module, and the search is cheap because it is restricted to the auxiliary module rather than the whole network."],"supporting_citations":[{"why":"Supplies the uncertainty-weighted joint-training baseline that the proposed auxiliary method is compared against and surpasses.","marker":"Kendall et al. (2018)"},{"why":"Provides the auxiliary-cell search space and fast architecture-search strategy used for the NAS variant.","marker":"Nekrasov et al. (2019)"},{"why":"Strong TRL-ResNet-50 multi-task baseline that the auxiliary method outperforms on NYUD-v2.","marker":"Zhang et al. (2018)"},{"why":"Provides the NYUD-v2 dataset used for the main two-task experiments.","marker":"Silberman et al. (2012)"},{"why":"Provides the SUNRGBD dataset used for the three-task semantic segmentation, depth, and surface normal experiments.","marker":"Song et al. (2015)"},{"why":"Supplies the MobileNetV2 shared encoder used as the default backbone.","marker":"Sandler et al. (2018)"},{"why":"Supplies the ResNet-50 backbone used for state-of-the-art comparisons.","marker":"He et al. (2016b)"},{"why":"Provides the real-time joint segmentation and depth baseline on NYUD-v2.","marker":"Nekrasov et al. (2018a)"},{"why":"Forms an earlier multi-task architecture baseline on NYUD-v2.","marker":"Eigen & Fergus (2015)"}],"fun_headline_variants":["Auxiliary side modules ease multi-task training, vanish at test","Training-only auxiliaries fine-tune shared layers in MTL","Drop-in training aids for multi-task nets, zero test overhead","Side losses for shared layers improve multi-task vision","Aux modules regularize training, then are dropped at inference"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the extra gradients from the auxiliary modules, which are computed from the same labels as the main tasks, push the shared encoder toward better representations rather than merely reweighting or duplicating the original supervision.","fun_headline_variants_meta":{"raw":{"variants":["Auxiliary side modules ease multi-task training, vanish at test","Training-only auxiliaries fine-tune shared layers in MTL","Drop-in training aids for multi-task nets, zero test overhead","Side losses for shared layers improve multi-task vision","Aux modules regularize training, then are dropped at inference"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001327,"raw_usage":{"total_tokens":5396,"prompt_tokens":935,"completion_tokens":4461,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":551,"completion_tokens_details":{"reasoning_tokens":4376}},"tokens_in":551,"tokens_out":4461,"duration_ms":31836,"temperature":1.0,"reasoning_tokens":4376,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:55:59.564371+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same multi-task network with the auxiliary module replaced by direct deep supervision at the same intermediate feature maps, using the same auxiliary losses, loss scales, and training schedule. If this deep-supervision baseline matches the auxiliary-module results, the claimed benefit of the multi-level auxiliary architecture is not supported.","supporting_citations":[{"cited_title":"Mobilenetv2: Inverted residuals and linear bottlenecks","cited_arxiv_id":null,"evidence_quote":"Supplies the MobileNetV2 shared encoder used as the default backbone."},{"cited_title":"Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture","cited_arxiv_id":null,"evidence_quote":"Forms an earlier multi-task architecture baseline on NYUD-v2."},{"cited_title":"Multi-task learning using uncertainty to weigh losses for scene geometry and semantics","cited_arxiv_id":null,"evidence_quote":"Supplies the uncertainty-weighted joint-training baseline that the proposed auxiliary method is compared against and surpasses."},{"cited_title":"Fast neural architecture search of compact semantic segmentation models via auxiliary cells","cited_arxiv_id":null,"evidence_quote":"Provides the auxiliary-cell search space and fast architecture-search strategy used for the NAS variant."},{"cited_title":"Indoor segmentation and support inference from rgbd images","cited_arxiv_id":null,"evidence_quote":"Provides the NYUD-v2 dataset used for the main two-task experiments."},{"cited_title":"Sun rgb-d: A rgb-d scene understanding benchmark suite","cited_arxiv_id":null,"evidence_quote":"Provides the SUNRGBD dataset used for the three-task semantic segmentation, depth, and surface normal experiments."}],"review_version":1}