Capacity and Redundancy Trade-offs in Multi-Task Learning

Asif Khan

REVIEW 2 major objections 4 minor 13 references

This paper claims that negative transfer in multi-task learning follows a quantitative identity: the total predictive information a shared encoder can supply across tasks is bounded by its capacity plus task redundancy, and clustered sharin

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 20:35 UTC pith:IVEIHZVA

load-bearing objection Useful capacity-redundancy frame and a clean Gaussian bridge, but the headline sharing rule (Thm 5.3) is unproved as stated due to a missing saturation condition. the 2 major comments →

arxiv 2607.16554 v1 pith:IVEIHZVA submitted 2026-07-17 cs.LG cs.AIcs.ITmath.IT

Capacity and Redundancy Trade-offs in Multi-Task Learning

Asif Khan This is my paper

classification cs.LG cs.AIcs.ITmath.IT

keywords multi-task learningcapacity-redundancy identitytotal correlationnegative transfertask clusteringlow-rank adaptationinformation bottleneckgradient cosine similarity

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that negative transfer in multi-task learning is not merely a training artifact but an information-theoretic consequence of limited shared capacity and weak task redundancy. The central claim is a capacity–redundancy identity: the sum over tasks of predictive information carried by a shared representation equals the joint predictive information plus the label redundancy (measured by total correlation) minus a residual coupling term that remains after conditioning on the representation; hence the sum can never exceed capacity plus redundancy. On top of that, the paper proves an if-and-only-if result for clustered sharing: replacing one global encoder with per-cluster encoders improves the sum of per-task informations precisely when the reduction in residual coupling exceeds the cross-cluster redundancy that is lost. If correct, this turns an intuitive trade-off into a measurable decision rule that can guide when to share or specialize parameters in parameter-efficient fine-tuning, including low-rank adapters (LoRA). The paper also proves a Gaussian bridge that justifies using gradient cosine similarity as a proxy for redundancy-based task clustering, and it reports experiments where similarity-clustered adapters reduce residual coupling and outperform size-matched random partitions.

Core claim

The core discovery is the capacity–redundancy identity sum_t I(Z_s;Y^(t)) = I(Z_s;Y^(1:T)) + TC(Y^(1:T)) − Delta, with Delta = TC(Y^(1:T)|Z_s), giving sum_t I(Z_s;Y^(t)) ≤ C_s + TC(Y^(1:T)). Theorem 5.3 turns this into a gain decomposition: clustered sharing beats global sharing by (Delta_shared − sum_k Delta_k) − TC_between(partition), so it is strictly better when interference reduction exceeds redundancy loss. Theorem 4.4 proves in a Gaussian model that whitened-gradient cosine similarity has the same sign as label correlation and, under matched noise and equal signal power, the same ordering; clustering by it yields the same partition as clustering by total correlation. Delta is estimate

What carries the argument

The load-bearing object is the exact capacity–redundancy identity, sum_t I(Z_s;Y^(t)) = I(Z_s;Y^(1:T)) + TC(Y^(1:T)) − Delta, where total correlation TC(Y^(1:T)) = sum_t H(Y^(t)) − H(Y^(1:T)) measures label redundancy and Delta = TC(Y^(1:T) | Z_s) is the residual coupling the shared representation leaves unexplained. The identity carries the entire argument: it converts the intuitive capacity-versus-redundancy trade-off into an equation, and subtracting the global and per-cluster versions of it yields Theorem 5.3's clustering-gap decomposition. A supporting mechanism is Theorem 4.4, the gradient–TC bridge, which licenses replacing total-correlation clustering with gradient cosine similarity

Load-bearing premise

The load-bearing premise is capacity saturation: the proof of the if-and-only-if clustering rule assumes the capacity-optimal global and per-cluster encoders achieve their budgets exactly (I(Z*_s;Y^(1:T)) = C and I(Z*_{S_k};Y^(S_k)) = C_k); without that, the gain identity carries an extra joint-predictive term, as the paper concedes after the proof.

What would settle it

In the paper's own Gaussian specialization, compute the left- and right-hand sides of the clustering-gap identity (Equation S4.36) for a rank-constrained encoder that is capacity-optimal but does not saturate its budget: if the joint-predictive term differs from zero, the inequality in Theorem 5.3 no longer predicts the sign of the gain. A concrete falsifying observation would be a case where clustered sharing improves validation performance yet Delta_shared − sum_k Delta_k is smaller than TC_between(P), showing the saturation-free term is decisive in practice.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

If tasks are nearly independent (small total correlation), adding tasks to a fixed-capacity shared encoder forces average per-task predictive information to decay like 1/T; keeping a per-task floor requires capacity to grow roughly linearly in T.
Any residual coupling Delta > 0 adds an irreducible term to the aggregate log-loss of decoupled per-task predictors, so private or clustered capacity is not a luxury but a requirement for heterogeneous task sets.
Theorem 5.3 supplies a concrete test: choose a partition with large Delta_shared − sum_k Delta_k and small TC_between(P); a partition that satisfies the inequality is provably better than global sharing at the same total budget.
For low-rank adapters, effective shared capacity grows at most linearly with rank and saturates when the feature covariance is low-rank, so clustered or private routes are needed once the number of distinct task directions outgrows the rank budget.
Validation-residual correlation estimates of Delta give a practical diagnostic: smaller estimated Delta under clustered routing tracks the interference-reduction side of the identity, matching the observed accuracy gains.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same CR identity could be used prospectively: estimate the two competing terms on a validation set before training and decide the cluster partition by the inequality of Theorem 5.3, rather than after measuring gains.
Because total correlation measures only redundancy, the identity cannot see synergistic task structure; extending the decomposition to O-information or partial information decomposition would reveal cases where tasks help each other beyond shared redundancy, which the present bound treats as unmodeled slack.
Under non-saturating encoders, the clustering rule needs an extra joint-predictive correction; an immediate testable extension is to estimate I(Z_s;Y^(1:T)) − sum_k I(Z_sk;Y^(S_k)) and add it to the gain formula to make the sharing decision robust in realistic under-trained regimes.
The linear rank-growth prediction for low-rank adapters is directly testable: measure saturation of validation performance as T increases at fixed rank, and check whether the rank needed to hold per-task performance roughly doubles when the number of weakly dependent tasks doubles.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Useful capacity-redundancy frame and a clean Gaussian bridge, but the headline sharing rule (Thm 5.3) is unproved as stated due to a missing saturation condition.

read the letter

First thing to know: the paper has a clean conceptual frame for the capacity-redundancy trade-off in MTL, but the advertised if-and-only-if sharing rule (Theorem 5.3) is not proved as stated. The proof in Appendix S4 gives a general gain identity with a joint-predictive difference term; that term only cancels under a saturation condition (Eq. S4.38) that is not included in the theorem statement in the main text. The paper admits this right after the proof. So the headline claim — "clustered sharing is strictly beneficial iff interference reduction exceeds redundancy loss" — is not true for arbitrary capacity-optimal encoders, only for those that saturate their budgets. That is a load-bearing gap, not a nitpick. The label "interference reduction ≥ 0" is also asserted rather than proved; conditional total correlation does not have to shrink under conditioning, so that term can be negative in general.

That said, the paper is not empty. The CR identity itself is elementary (DPI plus the chain rule), but the general clustering-gap decomposition (Eq. S4.36) is a useful way to split shared-representation gains into residual coupling, redundancy, and joint-predictive parts. Even without the saturation condition, it is a good diagnostic framing. The Gaussian gradient-TC bridge (Theorem 4.4) is small but genuine: whitened gradient cosine has the same sign as label correlation and, under matched noise, the same ordering, which licenses the common practice of clustering by gradient similarity. The empirical Δ estimator based on validation residuals is also a nice practical tool, and the synthetic bias analysis is a responsible touch.

The empirical section is honest but weaker than the abstract suggests. Gains on GLUE8 are modest, the random-partition comparison on GoEmotions is available only at one rank with a non-significant p-value around 0.09, and the PCGrad/GradNorm baselines are not compute-matched. The paper does call the random-partition comparison a sanity check, which is fair, but the significance language in the abstract overstates it.

Bottom line: worth reading for the conceptual vocabulary and the decomposition, and worth sending to peer review — the issues are addressable. But I would not cite Theorem 5.3 as a proved result until the saturation condition is explicit in the main text and the nonnegativity of the interference term is handled. Major revision needed.

Referee Report

2 major / 4 minor

Summary. The paper proposes a Capacity–Redundancy (CR) identity for multi-task learning: under a shared bottleneck with budget C_s, the total per-task mutual information is bounded by C_s plus the total correlation of the labels. It derives consequences (average per-task decay, capacity scaling), a shared–private extension, subset bounds, and a Gaussian gradient–TC bridge. It then states Theorem 5.3, a clustering-gap identity meant to give an iff condition for clustered sharing to outperform global sharing. The empirical section applies clustered LoRA on GoEmotions and GLUE8, estimating residual coupling from validation residuals and reporting multi-seed confidence intervals.

Significance. If Theorem 5.3 were correct as stated, it would provide a quantitative, distribution-level decision rule for choosing between global and clustered sharing, which is a useful conceptual contribution. The CR bound itself and the Gaussian gradient–TC bridge are correct and add a principled link between gradient similarity and redundancy. The synthetic verification and multi-seed empirical comparisons are strengths. However, the central theorem is overstated: the proof in Appendix S4 requires a saturation assumption that is absent from the main statement, and the 'interference reduction' term is not proven nonnegative. These issues do not invalidate the CR bound but require a substantive revision of the paper's main claim.

major comments (2)

[Theorem 5.3 / Appendix S4, Eq. (S4.36)–(S4.38)] The theorem states a decomposition with no joint-predictive term for 'capacity-optimal under budget' encoders. The general identity (S4.36) contains Σ_k I(Z*_S_k; Y(S_k)) − I(Z*_s; Y(1:T)), which cancels only under the saturation condition (S4.38): I(Z*_s; Y(1:T)) = C and I(Z*_S_k; Y(S_k)) = C_k. Capacity optimality under a budget does not imply saturation; the optimum may leave budget unused, e.g., when the label-relevant subspace has lower dimension than the budget. The proof itself concedes after (S4.37) that without the saturation condition the gain identity carries an additional joint-predictive difference term. Therefore the theorem's iff rule is false as stated. The statement must be revised to include saturation, or to state the general identity and give the iff only when the budgets are active.
[Theorem 5.3 body] The theorem labels Δ_sh − Σ_k Δ_k as 'interference reduction ≥ 0' and uses this label to interpret the decomposition. No proof is supplied, and the inequality is not guaranteed: Δ_sh = TC(Y | Z*_s) and Δ_k = TC(Y(S_k) | Z*_S_k) are conditional total correlations with different conditioners and different label subsets. Conditional TC is monotone with respect to the conditioning set only for the same set of variables, not across this comparison. It is easy to construct settings where per-cluster encoders leave within-cluster dependence unresolved while a global encoder with the same total budget removes cross-cluster dependence, giving a negative value. The algebraic identity under saturation remains valid, but the asserted sign is an unsupported claim and should be removed or proved.

minor comments (4)

[Section 7] The text refers to 'the subset CR bound (Theorem 6.1)', but the correct reference is Corollary 6.1.
[Appendix S4, after Eq. (S4.37)] There is a stray punctuation artifact: 'TCbetween(P),\n.' appears after the display equation.
[Theorem 3.2] The phrase 'Under the Markov chain' would be clearer if the chain Y(1:T) − X − Z_s were explicitly named in the theorem statement, rather than only in the preceding paragraph.
[Table 2 / Figure 2] The sentence 'Table 2 quantifies this effect ε∈{10^-6,10^-5} achieves small error' is a fragment and should be rewritten.

Circularity Check

0 steps flagged

No significant circularity: the theoretical claims are derived in-paper from classical information identities, with no self-citations and no fitted inputs fed back into the theorems.

full rationale

No significant circularity. The derivation chain is self-contained: Theorem 3.2 follows from the chain-rule sandwich (Lemma 3.1) plus the Markov property and DPI, all standard identities restated in-paper with external citations (Cover 1999; Watanabe 1960); no fitted constant feeds the theory. Theorem 5.3's gain identity is obtained by algebraically subtracting two instances of the exact CR identity (Appendix S4, Eqs. S4.33-S4.37); the 'iff' rule is the direct consequence of that exact decomposition, and both bracketed terms are defined independently of the gain. Theorem 4.4 is a closed-form Gaussian derivation whose assumptions (matched noise, equal signal power, whitened coordinates) are explicitly stated; the TC-log-det-correlation equivalence is standard Gaussian algebra. There are no self-citations (no prior work by the author appears in the reference list), no imported uniqueness theorem, and no ansatz adopted via citation. The empirical Delta-hat is estimated from validation residuals after training and used descriptively as a diagnostic; the synthetic sensitivity analysis calibrates epsilon against closed-form ground truth, and Delta-hat is not fed back into any theorem's assumptions. Two flagged limitations bear on correctness rather than circularity: (i) the main-text Theorem 5.3 states only 'capacity-optimal under budget' while the proof requires saturation (Eq. S4.38), and the paper concedes 'Without the saturation condition the exact gain identity consists of an additional joint-predictive difference term'; (ii) the label 'interference reduction >= 0' on Delta_sh - Sum Delta_k is asserted without proof. Both are unproven-assumption issues; neither reduces the derived claims to their inputs.

Axiom & Free-Parameter Ledger

2 free parameters · 6 axioms · 0 invented entities

The paper relies on standard information-theoretic definitions plus several domain assumptions about Markov structure and capacity budgets. The most fragile additions are the saturation condition used inside Theorem 5.3 and the Gaussian residual approximation used for Δ̂.

free parameters (2)

ε (ridge/log-det regularization) = 1e-5
Used in the Δ̂ estimator Δ̂ = -1/2 log det(R̂_e + εI); chosen from a synthetic bias-variance sweep and directly affects all reported Δ̂ values.
K (number of clusters) = 6
Number of task clusters chosen via a gap criterion constrained to K∈[2,6]; affects the feasibility of clustered/private runs at low rank budgets.

axioms (6)

domain assumption Test-time Markov chain Y^{1:T} − X − Z^s (and with private features Y^{1:T} − X − (Z^s, Z^{1:T}))
Assumed throughout Section 3 and Section 5 to apply data processing and the CR identities at evaluation time.
domain assumption Capacity budgets I(Z^s;X)≤C^s and I(Z^t;X|Z^s)≤C^t
The shared/private information constraints (Eq. 8) define the setting; real neural encoders have no literal information budget, so this is a modeling assumption.
ad hoc to paper Saturation of capacity-optimal encoders: I(Z*_s;Y^{1:T})=C and I(Z*_{S_k};Y^{S_k})=C_k for all k
Eq. S4.38 is used in the proof of Theorem 5.3 to cancel the joint-predictive difference term, but is not stated as a condition in the theorem itself.
ad hoc to paper Gaussian noise surrogate Z^s = Aϕ(X)+η with η∼N(0,σ²I) to make MI finite for deterministic LoRA encoders
Eq. S3.31 introduces a noise term so that I(Z^s;ϕ(X)) can be computed; the actual noise level σ² is unspecified.
domain assumption Matched noise levels and equal signal powers across tasks in Theorem 4.4
The ordering equivalence between gradient cosine and label correlation holds only under these equalities; without them, the ordering can differ.
domain assumption Validation residuals are approximately jointly Gaussian for Δ̂ to estimate TC(Y^{1:T}|Z^s)
Section 7 approximates conditional total correlation by -1/2 log det(R̂_e+εI), which is exact only for Gaussian residuals; GoEmotions tasks are binary and residuals are not Gaussian.

pith-pipeline@v1.3.0-alltime-deepseek · 21998 in / 14177 out tokens · 131368 ms · 2026-08-01T20:35:51.836310+00:00 · methodology

0 comments

read the original abstract

In multi-task learning (MTL) negative transfer is often considered as an optimization artifact, but it can also be viewed as a consequence of limited shared capacity and weak task redundancy. We investigate this effect through a Capacity--Redundancy (CR) identity that decomposes the sum of per-task predictive informations into joint predictive information that includes label redundancy defined via total correlation (TC), and a residual coupling term that quantifies interference left unresolved by the shared representation. Additionally, we show two key results: (i) a clustering-gap decomposition that gives a necessary and sufficient condition for clustered sharing to outperform global sharing, and (ii) a gradient--TC bridge in a Gaussian multi-task model that formally justifies gradient cosine similarity as a proxy for redundancy ordering. Empirically, we estimate the residual coupling $\Delta$ from validation residual correlations, showing that clustered LoRA substantially reduces $\widehat{\Delta}$, outperforms size-matched random partitions, and results in statistically significant gains with multi-seed confidence intervals.

Figures

Figures reproduced from arXiv: 2607.16554 by Asif Khan.

**Figure 1.** Figure 1: Capacity-Redundancy Tradeoff in Multi-Task Low-Rank Adaptation. (a) shows task entropies H(Y1), H(Y2) with overlap T C(Y ) = I(Y1; Y2) representing redundancy. The shared bottleneck Zs (gray) has capacity Cs. Blue arrows indicate task-specific information I(Zs; Yt). (b) rank-r matrices (A, B) add to frozen weights W, where rank r ≪ d limits capacity Cs. Sharing (A, B) across tasks creates the shared bottle… view at source ↗

**Figure 2.** Figure 2: Mean absolute error |∆b − TCexact| vs sample size N on the left. Signed bias ∆b − TCexact vs N on the right. Small ε converges toward the ground truth as N increases, while larger ε results in a persistent negative bias [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

**Figure 4.** Figure 4: ∆b computed from validation residuals vs total rank budget. Clustered routing results in a systematically smaller ∆b than sharing at matched budgets (rank ≥ 8), showing reduced residual dependence under similarity-guided finetuning. to heterogeneous task formats and limited sharing opportunities once syntax and sentiment tasks are isolated by similarity. This asymmetry is consistent with the CR predicti… view at source ↗

**Figure 5.** Figure 5: Shared is evaluated at all budgets, clustered/private [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗

discussion (0)

Reference graph

Works this paper leans on

13 extracted references · 1 canonical work pages

[4]

Demix layers: Disentan- gling domains for modular language modeling.arXiv preprint arXiv:2108.05036,

Suchin Gururangan, Mike Lewis, Ari Holtzman, Noah A Smith, and Luke Zettlemoyer. Demix layers: Disentan- gling domains for modular language modeling.arXiv preprint arXiv:2108.05036,

Pith/arXiv arXiv
[6]

doi: 10.18653/v1/2021.acl-long.353

Association for Computational Linguis- tics. doi: 10.18653/v1/2021.acl-long.353. URL https: //aclanthology.org/2021.acl-long.353/. Xi Lin, Hui-Ling Zhen, Zhenhua Li, Qing-Fu Zhu, and James Kwok. Controllable pareto multi-task learning. InAdvances in Neural Information Processing Systems (NeurIPS), volume 33, pages 20337–20348,

doi:10.18653/v1/2021.acl-long.353 2021
[8]

Pramod Kaushik Mudrakarta, Mark Sandler, Andrey Zh- moginov, and Andrew Howard

URL https://proceedings.mlr.press/ v162/momma22a.html. Pramod Kaushik Mudrakarta, Mark Sandler, Andrey Zh- moginov, and Andrew Howard. K for the price of 1: Parameter-efficient multi-task and transfer learning.arXiv preprint arXiv:1810.10703,

Pith/arXiv arXiv
[9]

Weizhu Qian, Bowei Chen, Yichao Zhang, Guanghui Wen, and Franck Gechter

URL https://openreview.net/ forum?id=B1eoyAVFwH. Weizhu Qian, Bowei Chen, Yichao Zhang, Guanghui Wen, and Franck Gechter. Multi-task variational information bottleneck.arXiv preprint arXiv:2007.00339,

Pith/arXiv arXiv 2007
[20]

Let A∈R T×d stack a⊤ t as rows

and defineTtask labels Y (t) =a ⊤ t X+ε t, ε t ∼ N(0, σ2 ε ),(S7.39) with independent {εt}T t=1 and σε = 0.1. Let A∈R T×d stack a⊤ t as rows. By varying the alignment of {at} we sweep label redundancy. Ifa i ≈a j then labels share signal andTC(Y 1:T )increases, while near-orthogonality yieldsTC(Y 1:T )≈0. Representations are produced by a rank-rlinear enc...

2088
[2000]

Small towers make big differences.arXiv preprint arXiv:2008.05808,

Yuyan Wang, Zhe Zhao, Bo Dai, Christopher Fifty, Dong Lin, Lichan Hong, and Ed H Chi. Small towers make big differences.arXiv preprint arXiv:2008.05808,

Pith/arXiv arXiv 2008
[2010]

Parameter-efficient multi- task fine-tuning for transformers via shared hypernet- works.arXiv preprint arXiv:2106.04489,

Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa De- hghani, and James Henderson. Parameter-efficient multi- task fine-tuning for transformers via shared hypernet- works.arXiv preprint arXiv:2106.04489,

Pith/arXiv arXiv
[2019]

An overview of multi-task learning in deep neural networks.arXiv preprint arXiv:1706.05098,

Sebastian Ruder. An overview of multi-task learning in deep neural networks.arXiv preprint arXiv:1706.05098,

Pith/arXiv arXiv
[2020]

doi: 10.18653/v1/2020.acl-main.372

Association for Computational Linguis- tics. doi: 10.18653/v1/2020.acl-main.372. URL https: //aclanthology.org/2020.acl-main.372/. Giulia Denevi, Massimiliano Pontil, and Carlo Ciliberto. Conditional meta-learning of linear representations. In Advances in Neural Information Processing Systems, vol- ume 35,

doi:10.18653/v1/2020.acl-main.372 2020
[2021]

doi: 10.18653/v1/2021.acl-long.568

Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.568. URL https:// aclanthology.org/2021.acl-long.568/. Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck.arXiv preprint arXiv:1612.00410,

Pith/arXiv arXiv 2021
[2022]

Towards a unified view of parameter-efficient transfer learning.arXiv preprint arXiv:2110.04366,

Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- Kirkpatrick, and Graham Neubig. Towards a unified view of parameter-efficient transfer learning.arXiv preprint arXiv:2110.04366,

Pith/arXiv arXiv
[2023]

doi: 10.18653/v1/2023.acl-long.144

Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.144. URL https:// aclanthology.org/2023.acl-long.144/. Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and An- drew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. InInternational conference on machine learning, pages 794–803. PMLR,

work page doi:10.18653/v1/2023.acl-long.144 2023
[2025]

The information bottleneck method.arXiv preprint physics/0004057,

Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method.arXiv preprint physics/0004057,

Pith/arXiv arXiv

[1] [4]

Demix layers: Disentan- gling domains for modular language modeling.arXiv preprint arXiv:2108.05036,

Suchin Gururangan, Mike Lewis, Ari Holtzman, Noah A Smith, and Luke Zettlemoyer. Demix layers: Disentan- gling domains for modular language modeling.arXiv preprint arXiv:2108.05036,

Pith/arXiv arXiv

[2] [6]

doi: 10.18653/v1/2021.acl-long.353

Association for Computational Linguis- tics. doi: 10.18653/v1/2021.acl-long.353. URL https: //aclanthology.org/2021.acl-long.353/. Xi Lin, Hui-Ling Zhen, Zhenhua Li, Qing-Fu Zhu, and James Kwok. Controllable pareto multi-task learning. InAdvances in Neural Information Processing Systems (NeurIPS), volume 33, pages 20337–20348,

doi:10.18653/v1/2021.acl-long.353 2021

[3] [8]

Pramod Kaushik Mudrakarta, Mark Sandler, Andrey Zh- moginov, and Andrew Howard

URL https://proceedings.mlr.press/ v162/momma22a.html. Pramod Kaushik Mudrakarta, Mark Sandler, Andrey Zh- moginov, and Andrew Howard. K for the price of 1: Parameter-efficient multi-task and transfer learning.arXiv preprint arXiv:1810.10703,

Pith/arXiv arXiv

[4] [9]

Weizhu Qian, Bowei Chen, Yichao Zhang, Guanghui Wen, and Franck Gechter

URL https://openreview.net/ forum?id=B1eoyAVFwH. Weizhu Qian, Bowei Chen, Yichao Zhang, Guanghui Wen, and Franck Gechter. Multi-task variational information bottleneck.arXiv preprint arXiv:2007.00339,

Pith/arXiv arXiv 2007

[5] [20]

Let A∈R T×d stack a⊤ t as rows

and defineTtask labels Y (t) =a ⊤ t X+ε t, ε t ∼ N(0, σ2 ε ),(S7.39) with independent {εt}T t=1 and σε = 0.1. Let A∈R T×d stack a⊤ t as rows. By varying the alignment of {at} we sweep label redundancy. Ifa i ≈a j then labels share signal andTC(Y 1:T )increases, while near-orthogonality yieldsTC(Y 1:T )≈0. Representations are produced by a rank-rlinear enc...

2088

[6] [2000]

Small towers make big differences.arXiv preprint arXiv:2008.05808,

Yuyan Wang, Zhe Zhao, Bo Dai, Christopher Fifty, Dong Lin, Lichan Hong, and Ed H Chi. Small towers make big differences.arXiv preprint arXiv:2008.05808,

Pith/arXiv arXiv 2008

[7] [2010]

Parameter-efficient multi- task fine-tuning for transformers via shared hypernet- works.arXiv preprint arXiv:2106.04489,

Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa De- hghani, and James Henderson. Parameter-efficient multi- task fine-tuning for transformers via shared hypernet- works.arXiv preprint arXiv:2106.04489,

Pith/arXiv arXiv

[8] [2019]

An overview of multi-task learning in deep neural networks.arXiv preprint arXiv:1706.05098,

Sebastian Ruder. An overview of multi-task learning in deep neural networks.arXiv preprint arXiv:1706.05098,

Pith/arXiv arXiv

[9] [2020]

doi: 10.18653/v1/2020.acl-main.372

Association for Computational Linguis- tics. doi: 10.18653/v1/2020.acl-main.372. URL https: //aclanthology.org/2020.acl-main.372/. Giulia Denevi, Massimiliano Pontil, and Carlo Ciliberto. Conditional meta-learning of linear representations. In Advances in Neural Information Processing Systems, vol- ume 35,

doi:10.18653/v1/2020.acl-main.372 2020

[10] [2021]

doi: 10.18653/v1/2021.acl-long.568

Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.568. URL https:// aclanthology.org/2021.acl-long.568/. Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck.arXiv preprint arXiv:1612.00410,

Pith/arXiv arXiv 2021

[11] [2022]

Towards a unified view of parameter-efficient transfer learning.arXiv preprint arXiv:2110.04366,

Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- Kirkpatrick, and Graham Neubig. Towards a unified view of parameter-efficient transfer learning.arXiv preprint arXiv:2110.04366,

Pith/arXiv arXiv

[12] [2023]

doi: 10.18653/v1/2023.acl-long.144

Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.144. URL https:// aclanthology.org/2023.acl-long.144/. Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and An- drew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. InInternational conference on machine learning, pages 794–803. PMLR,

work page doi:10.18653/v1/2023.acl-long.144 2023

[13] [2025]

The information bottleneck method.arXiv preprint physics/0004057,

Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method.arXiv preprint physics/0004057,

Pith/arXiv arXiv