Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Commute Your Domains: Trajectory Optimality Criterion for Multi-Domain Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The authors prove that swapping the order of two training domains changes the target loss through a commutator of gradient vector fields, computable from Hessian-vector products.

desk verdict A promising idea—using Lie brackets of gradient flows to diagnose training-order effects—but the central formula has a factor-of-two error, the toy validation disagrees with the printed theory, and the sign tables are inconsistent; the qualitative criterion may survive a fix, but as written the quantitative claims are not reliable. read the letter →

arxiv 2501.15556 v1 pith:WHFQHZY5 submitted 2025-01-26 cs.LG cs.CL

Alexey Rukhovich , Alexander Podolskiy , Irina Piontkovskaya This is my paper · ORCID
classification cs.LGcs.CL
keywords multi-domainlearningtrainingorderLiebracketgradientvectorfieldsweightscheduleHessian-vectorproductmultilingualpretrainingdatamixing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the effect of changing the order in which a model trains on two data domains is not an unpredictable nuisance but a second-order geometric quantity: the Lie bracket, or commutator, of the two gradient vector fields. The authors prove that swapping the order of two domains over a short window changes the target loss by a term proportional to $\langle \mathrm{Hess}\,X\,\nabla Y - \mathrm{Hess}\,Y\,\nabla X,\ \nabla Z \rangle$ evaluated along the trajectory. This yields a local optimality condition: wherever this quantity is nonzero, a locally optimal schedule must not mix the two domains at that instant. The paper validates the predicted sign of the loss change on a quadratic toy model and on bilingual English–Russian GPT-2 pretraining, where the direction of the change is correct for three of four checkpoints.

What carries the argument

The central object is the commutator of gradient vector fields, $R(L_i,L_j)=\mathrm{Hess}\,L_j\,\nabla L_i-\mathrm{Hess}\,L_i\,\nabla L_j$, and its pairing with the target gradient, $P(X,Y;Z)=\langle R(X,Y),\nabla Z\rangle$. This quantity is the second-order term in the difference between the two orders of training on domains $i$ and $j$, and it is computable from Hessian-vector products, making the criterion actionable at LLM scale.

What would settle it

A reproducible run where the computed commutator $P(L_2,L_1;L_i)$ and the measured excess loss have opposite signs, repeated enough times to rule out noise, would falsify the second-order prediction.

Watch

Extended reading notes

Core claim

The central claim is that, for smooth domain losses $L_1,\dots,L_K$ under gradient-flow dynamics $\dot\theta=-\sum_k w_k(t)\nabla L_k(\theta)$, the infinitesimal difference between two flow compositions is governed by the commutator $R(L_1,L_2)=\mathrm{Hess}\,L_2\,\nabla L_1-\mathrm{Hess}\,L_1\,\nabla L_2$. Corollary 3.2 turns this into a loss-change formula: perturbing a schedule so that domain $i$ is boosted and domain $j$ is cut for an interval of length $\varepsilon$, with the reverse for the following interval, changes the target loss $L$ by $\frac{1}{2}\,\delta\,\varepsilon^2\,P(L_i-L_j,\sum_k w_k L_k;L)+o(\delta\varepsilon^2)$, where $P(X,Y;Z)=\langle R(X,Y),\nabla Z\rangle$. Consequently, a locally optimal schedule at time $t$ must set either $w_i(t)=0$ or $w_j(t)=0$ whenever this $P$ is nonzero. For two domains this means the optimal schedule is always training on exactly one domain, switching only when $P(L_1,L_2;L)$ crosses zero.

Load-bearing premise

The load-bearing assumption is that training is a deterministic continuous-time gradient flow on the full-batch losses; if the discrete, stochastic, optimizer-modified trajectory is not well approximated by this flow at the scales used, the predicted direction of the loss change need not match the actual direction.

Editorial extensions

If this is right

  • A locally optimal multi-domain schedule for two losses never mixes both domains at the same instant; it must train on exactly one domain at a time, switching only when the commutator sign permits.
  • The optimality of any existing weight schedule can be checked locally, from checkpoints, using Hessian-vector products rather than full Hessians.
  • For the balanced bilingual LLM experiment, the sign of the predicted excess loss coincided with the measured sign at three of four checkpoints, and the appendix's imbalanced setting shows that ending with more low-resource data improves the low-resource loss with negligible high-resource cost.
  • The framework gives a principled way to align data-mixing schedules with the sign of the commutator rather than tuning them by hand, and the paper notes the quantity could be used for online data mixing.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the gradient-flow idealization holds, the same commutator identity should extend to three or more domains by repeated pairwise swaps; the paper only treats pairs, so a testable extension is to predict the net effect of a cyclic permutation of $K$ domains as a sum of pairwise commutators.
  • Because the formula uses the Hessian rather than the inverse Hessian, the relation to influence functions is complementary: influence functions answer which training points matter, while this commutator answers in which order to present them; one could combine both by weighting domains by influence and ordering by commutator sign.
  • The paper leaves the SDE case open. A natural test is to rerun the bilingual experiment with varying batch sizes: if the noise term $\Omega^{1/2}$ is large, the deterministic prediction should degrade as the noise intensity grows, providing a direct check of the limiting assumption.
  • For LLM pretraining with more than two domains, the optimality condition is more complex than 'never mix': it requires that any pair with nonzero commutator be separated, which may conflict; resolving that conflict is a concrete scheduling question the paper does not address.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies how the order of training on different data domains affects a target loss in multi-domain learning. It models training as a deterministic continuous-time gradient flow (Eq. 2) on the sum of domain losses with time-dependent weights, and derives, in Theorem 3.1 and Corollary 3.2, a local second-order formula for the change in target loss caused by swapping the order of two domains over a short window. The formula involves the Lie bracket of the gradient vector fields, expressed through Hessian-vector products. The authors also state an optimality condition asserting that a locally optimal weight schedule must spend almost all time on a single domain whenever the commutator P is nonzero. They validate the predictions on a quadratic toy example and on bilingual GPT-2 pre-training with C4 and mC4 data.

Significance. The central idea, that training-order effects in multi-domain learning can be characterized locally by the commutator of gradient vector fields, is elegant and potentially useful: it gives a computationally tractable criterion (Hessian-vector products) for when data mixing is suboptimal and suggests a bang-bang structure for optimal schedules. If correct, this would be a substantive contribution to curriculum and multi-domain learning. The paper also attempts an ambitious LLM-scale validation, and it is commendable that the reported P values are computed from actual model Hessians rather than fitted to observed losses. However, the manuscript currently contains load-bearing factor and sign inconsistencies in the main theorem and its proof, which undermine the quantitative claims until corrected.

major comments (4)
  1. [Corollary 3.2(a), Eq. (7)] The coefficient in the second-order expansion is incorrect. Expanding the ODE (2) with the swapping schedule of Corollary 3.2(a) directly, for a two-domain uniform base schedule, gives Δθ = δ ε² (Hess L_j ∇L_i − Hess L_i ∇L_j) and hence ΔL = δ ε² P(L_i−L_j, Σ w L; L) + o(δ ε²), with no factor 1/2. The 1/2 in Eq. (7) is therefore spurious, and the same error propagates to the 1/8 prefactor in Section 4.1 and to the proof of Corollary 3.2(b).
  2. [Theorem 3.1 and Appendix A] The sign of the commutator is not fixed consistently. The expansion in Appendix A computes, for v_i = −∇L_i, the difference Φ2(t2)∘Φ1(t1) − Φ1(t1)∘Φ2(t2) = t1t2 R + o(t1t2), but the final displayed line incorrectly claims the same expression for Φ1(t1)∘Φ2(t2) − Φ2(t2)∘Φ1(t1). These differ by a sign, so Eq. (5) and consequently Eq. (7) are ambiguous. Section 3.1, Eq. (4), likewise gives [v1,v2] as the negative of R for the quadratic example. The authors should state the commutator convention explicitly and verify the sign in the toy and LLM comparisons.
  3. [Tables 2 and 4] The P values for the same checkpoints are numerically identical in Table 2, labeled P(L2,L1;L_i), and Table 4, labeled P(L1,L2;L_i). Since P is antisymmetric in its first two arguments, the entries in one of these tables must have the opposite sign. At least one table is mislabeled, and this makes the sign-based comparison in Section 4.2 unreliable.
  4. [Proof of Corollary 3.2(b)] The displayed equation in the proof, L(θ^ε(t+2ε)) − L(θ(t+2ε)) = −½ ε² δ P(…), contradicts Eq. (7), which has the opposite sign. The argument that the swap improves the loss has the wrong direction for the schedule w^ε defined in part (a). The stated optimality criterion may survive a consistent sign fix, but as written the proof is internally inconsistent.
minor comments (5)
  1. [Section 4.1] The printed prediction 1/8 (Δt)² P(L1,L2;L) does not agree with the correct expansion (which would be ½ (Δt)² P for the full swap with δ=0.5), and Table 1 cannot be reproduced from the printed formula. The code apparently used a different coefficient, so the text should be corrected to match the experiments.
  2. [Section 4.2] The statement that the direction is predicted correctly for three of four checkpoints is weak as presented. It would be helpful to state the sign convention for θ12, quantify the disagreement at the failing checkpoint, and provide a simple binomial test to indicate whether the agreement is statistically meaningful.
  3. [Computational complexity paragraph] The estimate that the overhead is less than half a percent should include the cost of averaging over 600 samples for both gradient and Hessian-vector product, and should account for the fact that a Hessian-vector product generally costs about twice a gradient, not the same.
  4. [Section 5, first paragraph] There is a typo: 'Hesian-vector product' should be 'Hessian-vector product'.
  5. [Figures 2 and 3] The figures would be easier to read if the legends stated the Δw values corresponding to the annotated points and if the direction of increasing intervention were marked explicitly.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the commutator prediction is computed from Hessians and gradients and tested against measured excess losses, with no fitted parameters or load-bearing self-citations.

full rationale

The paper's central claim is a direct Taylor expansion of the flow commutator of gradient vector fields: Theorem 3.1 defines R(L1,L2) = Hess L2 ∇L1 − Hess L1 ∇L2, and Corollary 3.2 expresses the loss difference as (1/2)δ ε² P(...) with P built from that commutator. No parameter in this formula is fitted to the excess losses that the theory is later compared with. In the quadratic toy experiment, P is computed analytically from the known matrices A1,A2 and compared to observed differences; in the LLM experiment, P is estimated from Hessian-vector products and gradients, then compared to measured EL values (with only a stated γ² rescaling for the discrete-time ODE discretization), so the comparison is an independent test rather than a fitted prediction. There are no load-bearing self-citations: the cited related works [11] and [26] are not used as premises for the derivation, and no uniqueness theorem from the authors' own prior work is invoked. The limitations in Section 6.1 explicitly disclaim handling of stochasticity and optimizer choice, which is an honest scope restriction, not a circular reduction. The reviewer-flagged factor-of-two and sign-convention issues are arithmetic or labeling concerns about the published formulas and tables, not evidence that the derivation reduces to its own inputs. The derivation is self-contained and the empirical validation uses independently measured losses, so the circularity score is 0.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central derivation relies on a set of idealizing assumptions about the training process (smoothness, Euclidean metric, deterministic continuous-time gradient flow, whole-dataset losses, and validity of the second-order expansion at intervention scales). These are the main load-bearing structural assumptions. No free parameters or invented entities are introduced; the predictions are computed directly from Hessians and gradients of the model.

assumptions (5)
  • domain assumption Loss functions Lk are C^2-smooth on an open subset Theta of R^n, with gradients defined in the standard Euclidean metric (identity Riemannian metric).
    Section 3, first paragraph. Neural network losses with non-smooth activations or adaptive metrics are excluded; the paper acknowledges this in the Remark and Limitations.
  • domain assumption Training dynamics is the deterministic continuous-time gradient flow theta_dot = - sum_k w_k(t) grad L_k(theta(t)), with no stochasticity, no discretization, and no optimizer effects.
    Section 3, Eq. (2). The paper uses this ODE for the whole theory, then applies to Adam in Section 4.2 without derivation.
  • domain assumption Losses are aggregated over the entire dataset, not sample-dependent.
    Section 3: 'we consider the loss functions as something aggregated over the whole dataset'.
  • domain assumption The weight schedule w(t) is non-negative, right-continuous, and sums to one.
    Definition in Section 3. Needed for the ODE and for the perturbation in Corollary 3.2 to be a valid schedule.
  • ad hoc to paper The o(t1 t2) and o(delta epsilon^2) terms in Theorem 3.1 and Corollary 3.2 are negligible over the intervention windows used.
    The LLM experiment uses delta t = 4000 steps and total intervention 8000 steps, much larger than the infinitesimal limit; the paper lists non-locality as a source of discrepancy.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Commute Your Domains: Trajectory Optimality Criterion for Multi-Domain Learning." pith.science (2026). https://pith.science/paper/WHFQHZY5

@misc{pith2026250115556,
  author       = {Pith},
  title        = {Pith review of: Commute Your Domains: Trajectory Optimality Criterion for Multi-Domain Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WHFQHZY5}},
  note         = {Machine review of arXiv:2501.15556}
}
read the original abstract

In multi-domain learning, a single model is trained on diverse data domains to leverage shared knowledge and improve generalization. The order in which the data from these domains is used for training can significantly affect the model's performance on each domain. However, this dependence is under-studied. In this paper, we investigate the influence of training order (or data mixing) in multi-domain learning using the concept of Lie bracket of gradient vector fields. By analyzing the infinitesimal effects of changing the training order, we identify regions in the parameter space where altering the order between two training domains can benefit the target loss. We validate the predictions of our theoretical framework on the influence of training order (or data mixing) both on a toy example and bilingual LLM pre-training.

Figures

Figures reproduced from arXiv: 2501.15556 by the authors.

Figure 1
Figure 1. Example of vector fields Lie bracket. (a), (b): the level curves and gradients for two [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The results of intervening into domain weight schedule for bilingual LLM training. The [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. The results of intervening into domain weight schedule for bi-lingual LLM training with [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Domain losses dynamics. Here X and Y axes correspond to values of [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. First-Order Predictable but Pairwise Fragile: Local Task Adaptation in Trained Transformers

    cs.LG 2026-07 conditional novelty 5.0 of 10

    Single task-vector perturbations around a multitask LoRA point are first-order linear to 1e-2, but pairwise update-order sensitivity is set by a per-pair Lie bracket and can appear inside that window.

Reference graph

Works this paper leans on

43 extracted references · 27 canonical work pages · cited by 1 Pith paper

  1. [1]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023

    arXiv 2023

  2. [2]

    An introduction to the geometry of stochastic flows

    Fabrice Baudoin. An introduction to the geometry of stochastic flows. World Scientific, 2004

    work page 2004

  3. [3]

    Gradient-based bi-level optimization for deep learning: A survey

    Can Chen, Xi Chen, Chen Ma, Zixuan Liu, and Xue Liu. Gradient-based bi-level optimization for deep learning: A survey. arXiv preprint arXiv:2207.11719, 2022

    arXiv 2022

  4. [4]

    Gradnorm: Gra- dient normalization for adaptive loss balancing in deep multitask networks

    Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. Gradnorm: Gra- dient normalization for adaptive loss balancing in deep multitask networks. In International conference on machine learning, pages 794–803. PMLR, 2018

    work page 2018

  5. [5]

    Just pick a sign: Optimizing deep multitask models with gradient sign dropout

    Zhao Chen, Jiquan Ngiam, Yanping Huang, Thang Luong, Henrik Kretzschmar, Yuning Chai, and Dragomir Anguelov. Just pick a sign: Optimizing deep multitask models with gradient sign dropout. Advances in Neural Information Processing Systems, 33:2039–2050, 2020

    work page 2020

  6. [6]

    Order matters in the presence of dataset imbalance for multilingual learning

    Dami Choi, Derrick Xin, Hamid Dadkhahi, Justin Gilmer, Ankush Garg, Orhan Firat, Chih- Kuan Yeh, Andrew M Dai, and Behrooz Ghorbani. Order matters in the presence of dataset imbalance for multilingual learning. Advances in Neural Information Processing Systems, 36, 2024. 8

    work page 2024

  7. [7]

    The loss landscape of overparameterized neural networks

    Yaim Cooper. The loss landscape of overparameterized neural networks. arXiv preprint arXiv:1804.10200, 2018

    arXiv 2018

  8. [8]

    How to compute hessian- vector products? In The Third Blogpost Track at ICLR 2024, 2024

    Mathieu Dagréou, Pierre Ablin, Samuel Vaiter, and Thomas Moreau. How to compute hessian- vector products? In The Third Blogpost Track at ICLR 2024, 2024

    work page 2024

Show all 43 references
  1. [9]

    Laplace redux-effortless bayesian deep learning

    Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. Laplace redux-effortless bayesian deep learning. Advances in Neural Information Processing Systems, 34:20089–20103, 2021

    2021

  2. [10]

    Multiple-gradient descent algorithm (mgda) for multiobjective opti- mization

    Jean-Antoine Désidéri. Multiple-gradient descent algorithm (mgda) for multiobjective opti- mization. Comptes Rendus Mathematique, 350(5-6):313–318, 2012

    2012

  3. [11]

    Implicit biases in multitask and continual learningfrom a backward error analysis perspective

    Benoit Dherin. Implicit biases in multitask and continual learningfrom a backward error analysis perspective. In NeurIPS 2023 Workshop on Mathematics of Modern Machine Learning, 2023

    2023

  4. [12]

    Bilevel programming for hyperparameter optimization and meta-learning

    Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil. Bilevel programming for hyperparameter optimization and meta-learning. In International conference on machine learning, pages 1568–1577. PMLR, 2018

    2018

  5. [13]

    Gradual domain adaptation: Theory and algorithms, 2023

    Yifei He, Haoxiang Wang, Bo Li, and Han Zhao. Gradual domain adaptation: Theory and algorithms, 2023

    2023

  6. [14]

    Rotograd: Gradient homogenization in multitask learning

    Adrián Javaloy and Isabel Valera. Rotograd: Gradient homogenization in multitask learning. arXiv preprint arXiv:2103.02631, 2021

    2021 arXiv

  7. [15]

    How does adaptive optimization impact local neural network geometry? Advances in Neural Information Processing Systems, 36, 2024

    Kaiqi Jiang, Dhruv Malik, and Yuanzhi Li. How does adaptive optimization impact local neural network geometry? Advances in Neural Information Processing Systems, 36, 2024

    2024

  8. [16]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  9. [17]

    Understanding black-box predictions via influence functions

    Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pages 1885–1894. PMLR, 2017

    2017

  10. [18]

    Ubernet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory

    Iasonas Kokkinos. Ubernet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6129–6138, 2017

    2017

  11. [19]

    Understanding self-training for gradual domain adaptation

    Ananya Kumar, Tengyu Ma, and Percy Liang. Understanding self-training for gradual domain adaptation. ArXiv, abs/2002.11361, 2020

    2002 arXiv

  12. [20]

    Smooth manifolds

    John M Lee. Smooth manifolds. Springer, 2012

    2012

  13. [21]

    Sequential reptile: Inter-task gradient alignment for multilingual learning

    Seanie Lee, Hae Beom Lee, Juho Lee, and Sung Ju Hwang. Sequential reptile: Inter-task gradient alignment for multilingual learning. In International Conference on Learning Representations, 2022

    2022

  14. [22]

    Pareto multi-task learning

    Xi Lin, Hui-Ling Zhen, Zhenhua Li, Qing-Fu Zhang, and Sam Kwong. Pareto multi-task learning. Advances in neural information processing systems, 32, 2019

    2019

  15. [23]

    On the limited memory bfgs method for large scale optimization

    Dong C Liu and Jorge Nocedal. On the limited memory bfgs method for large scale optimization. Mathematical programming, 45(1):503–528, 1989

    1989

  16. [24]

    Towards impartial multi-task learning

    Liyang Liu, Yi Li, Zhanghui Kuang, J Xue, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. Towards impartial multi-task learning. In ICLR. iclr, 2021

    2021

  17. [25]

    Decoupled weight decay regularization

    I Loshchilov and F Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  18. [26]

    Abide by the law and follow the flow: Conservation laws for gradient flows

    Sibylle Marcotte, Rémi Gribonval, and Gabriel Peyré. Abide by the law and follow the flow: Conservation laws for gradient flows. Advances in neural information processing systems, 36, 2024

    2024

  19. [27]

    The benefit of multitask representation learning

    Andreas Maurer, Massimiliano Pontil, and Bernardino Romera-Paredes. The benefit of multitask representation learning. Journal of Machine Learning Research, 17(81):1–32, 2016

    2016

  20. [28]

    Multi-task learning as a bargaining game

    Aviv Navon, Aviv Shamsian, Idan Achituve, Haggai Maron, Kenji Kawaguchi, Gal Chechik, and Ethan Fetaya. Multi-task learning as a bargaining game. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th In...

    2022

  21. [29]

    Numerical optimization

    Jorge Nocedal and Stephen J Wright. Numerical optimization. Springer, 1999

    1999

  22. [30]

    Fast exact multiplication by the hessian

    Barak A Pearlmutter. Fast exact multiplication by the hessian. Neural computation, 6(1):147– 160, 1994

    1994

  23. [31]

    Estimating training data influence by tracing gradient descent

    Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 33:19920–19930, 2020

    2020

  24. [32]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020

    2020

  25. [33]

    Scalarization for multi-task and multi-domain learning at scale

    Amelie Royer, Tijmen Blankevoort, and Babak Ehteshami Bejnordi. Scalarization for multi-task and multi-domain learning at scale. Advances in Neural Information Processing Systems, 36, 2024

    2024

  26. [34]

    Scaling up influence functions

    Andrea Schioppa, Polina Zablotskaia, David Vilar, and Artem Sokolov. Scaling up influence functions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8179–8186, 2022

    2022

  27. [35]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  28. [36]

    Gradient vaccine: Investigating and improving multi-task optimization in massively multilingual models

    Zirui Wang, Yulia Tsvetkov, Orhan Firat, and Yuan Cao. Gradient vaccine: Investigating and improving multi-task optimization in massively multilingual models. In International Conference on Learning Representations, 2021

    2021

  29. [37]

    On the power-law hessian spectrums in deep learning

    Zeke Xie, Qian-Yuan Tang, Yunfeng Cai, Mingming Sun, and Ping Li. On the power-law hessian spectrums in deep learning. arXiv preprint arXiv:2201.13011, 2022

    2022 arXiv

  30. [38]

    mt5: A massively multilingual pre-trained text-to-text transformer

    L Xue, N Constant, A Roberts, M Kale, R Al-Rfou, A Siddhant, A Barua, and C Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934, 2020

    2010 arXiv

  31. [39]

    Adatask: A task-aware adaptive learning rate approach to multi-task learning

    Enneng Yang, Junwei Pan, Ximei Wang, Haibin Yu, Li Shen, Xihua Chen, Lei Xiao, Jie Jiang, and Guibing Guo. Adatask: A task-aware adaptive learning rate approach to multi-task learning. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 10745–10753, 2023

    2023

  32. [40]

    Gradient surgery for multi-task learning

    Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 5...

    2020

  33. [41]

    Gradual domain adaptation via gradient flow

    Zhan Zhuang, Yu Zhang, and Ying Wei. Gradual domain adaptation via gradient flow. InThe Twelfth International Conference on Learning Representations, 2024. A Proofs Proof of Theorem 3.1. This is actually a classic computation in differential geometry (see e.g. [20]). First, we...

    2024

  34. [42]

    = x + t1v1(x) + t2 1 2 (∇v1 · v1 (x)) + o(t2 1) 10 where ∇v1 · v1 (x) = P i ∂v1 ∂xi v1 i(x). Then the composition of two flows is Φ2(t2) ◦ Φ1(t1)x = Φ2(t2)(x + t1v1(x) + t2 1 2 (∇v1 · v1 (x)) + o(t2 1)) = = x + t1v1(x) + t2 1 2 (∇v1 · v1 (x)) + o(t2 1)+ + t2v2(x + t1v1(x) + t2...

  35. [43]

    = = x + t1v1(x) + t2 1 2 (∇v1 · v1 (x)) + o(t2 1)+ + t2v2(x) + t1t2 (∇v2 · v1 (x)) + o(t1t2)+ + t2 2 2 (∇v2 · v2 (x)) + o(t2 2) So, the commutator of two flows is Φ1(t1) ◦ Φ2(t2)x − Φ2(t2) ◦ Φ1(t1)x = t1t2((∇v2 · v1 (x)) − (∇v1 · v2 (x))) = = t1t2(Hess L2∇L1 − Hess L1∇L2) + o(...

    2000

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.