Pith. sign in

REVIEW 3 major objections 5 minor 48 references

Scalable Meta-Learning via Mixed-Mode Differentiation

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A reparameterization of the inner-loop update, requiring only a few lines of code, lets exact meta-gradients be computed in mixed-mode differentiation, cutting active memory by over 10x and wall-clock time by up to 25% in modern…

desk verdict Real, reproducible efficiency for exact-gradient meta-learning, but the 10x memory headline bundles a separately-invented checkpointing trick; on GPU the mixed-mode piece alone is near-neutral. read the letter →

arxiv 2505.00793 v2 pith:YPSGC3UN submitted 2025-05-01 cs.LG cs.AIstat.ML

Iurii Kemaev , Dan A Calian , Luisa M Zintgraf , Gregory Farquhar , Hado van Hasselt This is my paper · ORCID
classification cs.LGcs.AIstat.ML MSC 68T07
keywords bileveloptimizationmeta-learningautomaticdifferentiationmixed-modeHessian-vectorproductstruncatedbackpropagationthroughtimegradientcheckpointinglargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Meta-learning and hyperparameter optimisation require differentiating through the inner training loop itself, producing gradient-of-a-gradient calculations that standard autodiff libraries execute with two nested reverse passes, a graph shape that stores activations from both levels of differentiation. This paper argues that the graph is needlessly expensive and that a simple reparameterization, making the inner-loop gradient an explicit argument of the update and then transposing the second-derivative products via the symmetry of the Hessian, lets the same exact gradient be computed in mixed mode (forward-over-reverse), which never stores the inner backward pass's activations. The resulting algorithm, MixFlow-MG, requires only a few lines of code change and yields over 10x active-memory reductions and up to 25% wall-clock improvements in modern setups: on a 489M-parameter Chinchilla transformer active memory drops from 371 GiB to 55 GiB, and on the largest tested transformers the dynamic-memory reduction converges to 10x on GPUs and 23-25x on TPUs. If this holds, the practical reach of bilevel optimisation, including MAML, learned hyperparameters and loss weights, and algorithm discovery, expands from small inner models and short horizons to billion-parameter models and longer unrolls.

What carries the argument

The load-bearing object is the reparameterized update function $\Upsilon(\nabla L, \theta, \upsilon, \eta, x)$, which takes the inner-loop gradient as an explicit argument, together with the symmetry identities $(\partial^2 L/\partial\theta^2)^T = \partial^2 L/\partial\theta^2$ and $(\partial^2 L/\partial\eta\partial\theta)^T = \partial^2 L/\partial\theta\partial\eta$, valid when the loss has continuous second derivatives (Schwarz's theorem). These identities turn the vector-by-Hessian products and vector-by-mixed-derivative products that reverse-over-reverse evaluates inefficiently into Hessian-by-vector and mixed-derivative-matrix-by-vector products, which are computed by a custom VJP rule implementing forward-over-reverse differentiation (a custom VJP in JAX, a custom autograd function in PyTorch). Because forward mode at the outer level does not need the inner backward pass's saved activations, the activation buffer becomes independent of the model's depth. Two complementary optimisations, per-block rematerialisation of residual blocks and saving inner-loop gradients as part of gradient checkpointing, are layered on top to produce the headline numbers.

What would settle it

Sweep the number of layers $L$ of a fixed-width, fixed-context transformer while measuring peak dynamic memory for the default reverse-over-reverse implementation versus MixFlow-MG: the paper's mechanism predicts the default's dynamic memory grows roughly linearly in $L$ while the mixed-mode version stays roughly flat, so the ratio should rise linearly in $L$ toward $kL/\hat{k}$ at long contexts, and a flat ratio would falsify the depth-independence claim. Similarly, re-running the paper's Table 2 case on a compiler release that performs nested-gradient canonicalization internally should make the memory ratio collapse toward 1, which would show the advantage was compiler dependence rather than an algorithmic improvement.

Watch

Extended reading notes

Core claim

The paper's central claim is that the default reverse-over-reverse implementation of truncated-BPTT bilevel gradients multiplies by second-derivative matrices in an inefficient orientation, and that both product types can be transposed for free because the inner loss's second derivatives are symmetric. Rewriting the inner update as $(\theta_{i+1}, \upsilon_{i+1}) = \Upsilon(\nabla L_i, \theta_i, \upsilon_i, \eta, x_i)$ makes the swap a local code change: the needed Hessian-by-vector and mixed-derivative-matrix-by-vector products are then computed by a custom forward-over-reverse rule that never stores the inner backward pass's activations. For block-residual models such as transformers this removes the factor of layer count from the dynamic-memory cost, changing the scaling from $O(BL(S + kS^2))$ to $O(B(S + \hat{k}S^2))$, and the empirical gains grow with model size, converging to 23-25x dynamic-memory reduction on TPUs and about 10x on GPUs, with wall-clock wins up to 25% on GPUs and 20% on TPUs. The paper reports these numbers against dynamic (activation) memory; counting static checkpoints and optimiser states lowers the total gain to 4-6x, as the appendix discusses.

Load-bearing premise

The load-bearing premise is that the compiler does not already perform the equivalent graph optimisation: MixFlow-MG's gains are measured against today's compilers, which the paper says cannot make these rewrites on their own, and if a future compiler version canonicalises the default reverse-over-reverse graph into the mixed-mode form automatically, the reported over-10x memory and up-to-25% time advantages would vanish.

Editorial extensions

If this is right

  • Bilevel meta-optimisation with billion-parameter inner models becomes memory-feasible: the 489M-parameter case drops from 371 GiB to 55 GiB of active memory, and dynamic-memory reductions reach 10x on GPUs and 23-25x on TPUs at the 16B end of the scaling ladder.
  • For transformers with block-residual connections, dynamic-memory cost stops growing with the number of layers, scaling as $O(B(S + \hat{k}S^2))$ rather than $O(BL(S + kS^2))$, so the savings compound as models get deeper.
  • The gains are roughly constant in the number of inner updates and batch size, so users can spend the savings on longer inner horizons, larger batches, or longer contexts without losing the benefit.
  • The reparameterization is generic across bilevel tasks (MAML, per-parameter learning-rate learning, data-loss weighting), and the paper states the ideas can also be folded into approximate bilevel methods such as implicit-gradient and forward-mode schemes.
  • The appendix's static-memory reducers, including sharded data parallelism, reversible update inversion, and dynamic-programming checkpointing, are fully compatible with the mixed-mode core, so total-memory gains can be pushed toward the 10-25x dynamic gains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Re-running the paper's sweeps on future compiler releases is the sharpest way to separate algorithm from implementation: if a JAX or XLA version starts canonicalising nested reverse passes into the mixed-mode form on its own, the reported gains should shrink, which would attribute much of today's advantage to compiler immaturity.
  • The same custom-VJP trick should transfer to any program that differentiates twice through a symmetric second derivative, including long-horizon value-gradient reinforcement learning, trained optimisers, and second-order optimisation methods, none of which the paper benchmarks.
  • Because forward-over-reverse also avoids reading and writing inner backward activations, the wall-clock benefit on memory-bandwidth-bound accelerators should exceed the reported 25%; a bandwidth-limited backend is a direct place to test that prediction.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper addresses the computational cost of exact gradient-based bilevel optimization (Truncated-BPTT). It observes that default autodiff computes the meta-gradient in reverse-over-reverse mode, which materializes vector-Hessian and mixed-derivative products inefficiently. The authors reparameterize the inner update function as Υ(∇L_i, θ_i, υ_i, η, x_i) and use symmetry of second derivatives to replace vector-Hessian products with Hessian-vector products and mixed vector-matrix products with matrix-vector products, enabling forward-over-reverse or reverse-over-forward differentiation. They provide JAX and PyTorch implementations, a JAX custom_vjp rule for forward-over-reverse HVPs, and benchmarks on language-model meta-learning tasks (per-parameter learning rate learning, MAML, and loss weighting) on H100 and TPUv5p, reporting dynamic HBM ratios and step-time ratios. The headline claims are over 10x memory and up to 25% wall-clock time improvements over standard implementations.

Significance. The core reparameterization is simple, generic, and mathematically sound under the stated smoothness assumption, and it computes the same exact gradient as standard autodiff rather than an approximation. The paper includes concrete JAX/PyTorch code and extensive ablations, which are significant strengths. If the performance claims are correctly scoped, the method is likely to be useful to practitioners of meta-learning and bilevel optimization. However, the headline memory numbers mix the core mixed-mode differentiation with auxiliary optimizations (block rematerialization and saving inner gradients), and the paper inconsistently reports dynamic versus total memory; these issues must be clarified before the central performance claims can be assessed.

major comments (3)
  1. [§4 / Table 2 / §5.2] The headline "over 10x memory" is not cleanly attributable to mixed-mode differentiation. In Table 2, for the 489M model on GPU, default total HBM is 371.2 GiB; block rematerialization alone (`- + -`) gives 180.1 GiB; adding mixed-mode (`+ + -`) gives 174.8 GiB, only a 3% reduction. The advertised 54.8 GiB (`+ + +`) is reached only when "saving inner gradients" is also enabled, an optimization that Section 4 states is enabled only for MixFlow-MG and not found in existing libraries. Thus the >10x claim is a claim about the full MixFlow recipe, not about the core reparameterization, and the marginal contribution of the core is backend-dependent: on TPU the same comparison is 123.7 vs 43.8 GiB (about 2.8x). Please report the marginal contribution of each optimization separately and state explicitly whether the abstract's headline refers to the full recipe or to mixed-mode differentiation alone.
  2. [§5.1 / Abstract / Conclusion / Appendix A.2] The memory metric is used inconsistently. Section 5.1 defines the main metric as peak dynamic HBM ratio (Eq. 10), and the figures in Section 5 report dynamic-memory gains. The conclusion, however, states "up to 10x total memory reductions," while Table 2 reports total HBM: for the flagship 489M/GPU row the total ratio is 371.2/54.8 ≈ 6.8x, and Appendix A.2 explicitly states that total-memory gains are 4-6x, not 10x. Please harmonize these claims by distinguishing dynamic from total memory in every headline statement and in the conclusion.
  3. [§5.3 / Eq. (12)] Equation (12) presents a theoretical scaling ratio for dynamic memory that is linear in the number of layers L for default implementations and independent of L for the proposed method. This derivation is correct for the block-rematerialization comparison, but it does not include the "saving inner gradients" optimization that is essential for the empirical GPU gains in Table 2. Since the empirical sweeps in Figure 5 appear to use the full method, the reported agreement with Eq. (12) does not isolate the mechanism claimed in the equation. Please state which optimizations are active in each empirical curve and, if possible, show the same scaling plot with only the mixed-mode reparameterization active.
minor comments (5)
  1. [Algorithms 1 and 2] The placeholder lines "empty line" in Algorithm 1 and Algorithm 2 should be removed or replaced with explicit operation descriptions.
  2. [Throughout] There are several typos, including "Reparamererisation" in Proposition 3.1, "backpropogation" in Section 4, and "motivational example" in Appendix A.6.
  3. [§2.2 and Appendix A.1] Some displayed equations, particularly the three HVP-mode equations in Section 2.2, contain overlapping or garbled symbols in the manuscript source; please ensure the typeset equations clearly distinguish reverse-over-forward, forward-over-reverse, and reverse-over-reverse modes.
  4. [Table 2] Several rows in Table 2 report N/A for step time; please indicate whether those configurations exceeded device memory or were simply not benchmarked, and provide the corresponding measurements where available.
  5. [§4] Section 4 says block rematerialization is "kept enabled for all benchmarks," but Table 2 and Figure 10 include configurations without it; please clarify whether block rematerialization is considered part of the baseline or an optional component in different experiments.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the MixFlow-MG gradient derivation is self-contained and the reported memory gains are external benchmark comparisons with a disclosed, separately-enabled optimization.

full rationale

The paper's derivation chain is self-contained. Equations (4)-(8) and Appendix A.1 reparameterize the inner update, apply the chain rule, and use Schwarz's theorem to transpose vector-by-Hessian and vector-by-mixed-derivative products; the resulting mixed-mode gradient is algebraically identical to the default reverse-over-reverse gradient by construction, which is a correctness requirement rather than a circular prediction. No fitted parameter is renamed as a prediction, and no load-bearing uniqueness theorem is imported from the authors' prior work. Same-author citations (DeepMind JAX Ecosystem; Datarater) are used for software infrastructure and application context, not to justify the core claim. The reported memory gains combine mixed-mode differentiation with block rematerialization and 'saving inner gradients'; the paper explicitly discloses that the latter 'we enable it only for MixFlow-MG' (Section 4, Table 2). This means the >10x HBM reduction is a property of the full MixFlow recipe rather than of mixed-mode differentiation alone, and the marginal contribution of the core mechanism on GPU total HBM is modest (Table 2: 174.8 vs 180.1 GiB with block remat). However, this is a comparison-attribution caveat, not circularity: the benchmark is an external measurement against a stated baseline, and the mathematics is not equivalent to its inputs. The theoretical scaling model in Equation (12) uses stated compiler-dependent constants rather than fitted parameters, so it is not circular either.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

No new physical or mathematical entities are introduced. The main assumptions are standard smoothness, the reparameterization identity, and a compiler-behavior assumption. The scaling argument also relies on the attention mechanism memory model O(BL(S+kS^2)).

free parameters (1)
  • Inner step size 1e-3 in motivating example = 1e-3
    Ad hoc fixed learning rate for the toy example (Appendix A.5), used for demonstration only, and central claims are asymptotic and not dependent on it.
assumptions (4)
  • domain assumption The function approximator and loss function have continuous second-order derivatives, so Schwarz's theorem applies (Section 3).
    Required for swapping ∂²L/∂θ² and ∂²L/∂η∂θ with their transposes (Equations 7, 8). Standard for neural networks with smooth activations.
  • domain assumption Inner gradient ∂L/∂θ can be computed and passed as an explicit argument to the update function Υ without changing numerics.
    This is the core reparameterization in Equation 4; it holds by construction for gradient-based inner loops.
  • ad hoc to paper XLA/JAX does not automatically discover the mixed-mode transformation from the default graph.
    The paper's practical gains depend on the compiler's limitations; the authors state 'modern compilers are not able to make such improvements on their own' (Section 2.3/3). If the compiler later performs the optimization, the gain disappears.
  • domain assumption Activation memory dominates parameter memory in transformers (needed for the O(BL(S+kS^2)) vs O(B(S+ k_hat S^2)) scaling argument).
    This is stated in Section 4 and generally true for large transformers; the scaling analysis in Equation 12 relies on it.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Scalable Meta-Learning via Mixed-Mode Differentiation." pith.science (2026). https://pith.science/paper/YPSGC3UN

@misc{pith2026250500793,
  author       = {Pith},
  title        = {Pith review of: Scalable Meta-Learning via Mixed-Mode Differentiation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YPSGC3UN}},
  note         = {Machine review of arXiv:2505.00793}
}
read the original abstract

Gradient-based bilevel optimisation is a powerful technique with applications in hyperparameter optimisation, task adaptation, algorithm discovery, meta-learning more broadly, and beyond. It often requires differentiating through the gradient-based optimisation itself, leading to "gradient-of-a-gradient" calculations with computationally expensive second-order and mixed derivatives. While modern automatic differentiation libraries provide a convenient way to write programs for calculating these derivatives, they oftentimes cannot fully exploit the specific structure of these problems out-of-the-box, leading to suboptimal performance. In this paper, we analyse such cases and propose Mixed-Flow Meta-Gradients, or MixFlow-MG -- a practical algorithm that uses mixed-mode differentiation to construct more efficient and scalable computational graphs yielding over 10x memory and up to 25% wall-clock time improvements over standard implementations in modern meta-learning setups.

Figures

Figures reproduced from arXiv: 2505.00793 by the authors.

Figure 1
Figure 1. Peak HBM and step time across the number of (per inner step) transformations M in Equation (9) (GPU). 3.2. Motivating Example To illustrate the effects of MixFlow-MG, we consider the following simple BLO problem (Equation (3)), similar to Finn et al. (2017): η defines the initialisation point θ0 = η for the inner optimisation; the inner loss is a standard L2 loss which is independent of η; the update dynamics Φ(θi ,… view at source ↗
Figure 3
Figure 3. HBM footprints for each stage of optimisation for 489M chinchilla transformer on GPU: 1 – block rematerialization, 2 – saving inner gradients, 3 – mixed-mode differentiation. Each optimisation is responsible for reducing a specific chunk of HBM. In addition to enabling gradient checkpointing for inner loop unrolling, we found the following two optimisations impor￾tant for amplifying the gains of the proposed algorit… view at source ↗
Figure 4
Figure 4. Joint sweep over tasks, models, and hyperparameters from [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figures from the paper (7 more)
Figure 5
Figure 5. Figure 5: Sweep over data regimes for Chinchilla models (GPU): peak dynamic HBM ratio between default and mixed-mode diff-n. 20 40 60 Number of blocks 0 5 10 15 20 25 Peak dynamic HBM ratio 10 20 30 Number of MHA heads 1000 2000 Attention size 2500 5000 7500 Feed-forward layer s…
Figure 6
Figure 6. Figure 6: Sweep over transformer components: peak dynamic HBM ratio between default and mixed-mode differentiation. 4x less memory) for 80% of configurations, with peak reduc￾tions exceeding 90% (over 10x less memory) on both GPUs and TPUs. Previously, memory constraints severel…
Figure 7
Figure 7. Figure 7: Chinchilla scaling ladder: peak dynamic HBM gains across transformers of various sizes. our theoretical analysis. While the gains could be expected to be near-constant for the other structural parameters, the real numbers differ in practice, especially for small models…
Figure 8
Figure 8. Figure 8: Static and dynamic peak HBM w.r.t. model size. Combining them with Equation (16) we can rewrite the vector-by-hessian (VHP) and vector-by-mixed-derivatives-matrix (VMP) products into their transposed versions, i.e. hessian-by-vector (HVP) and mixed-derivatives-matrix-b…
Figure 9
Figure 9. Figure 9: HLO graph for the motivational example. Data nodes are depicted in orange, compute operations (multiplications, trigonometric functions etc.) in gray. It can be seen that the mixed-mode version contains far fewer data blocks. Also, this example demonstrates the complex…
Figure 10
Figure 10. Figure 10: All combinations of the used optimisation from Section 4 for 489M model. Note that GPU required saving inner gradients for peak memory gains, while TPU needs it only for reducing step time. Mixed-mode differentiation and model blocks’ rematerialisations are critical f…
Figure 11
Figure 11. Figure 11 [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 21 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Arnold, S. M. R., Mahajan, P., Datta, D., Bunner, I., and Zarkias, K. S. learn2learn: A library for meta-learning research, 2020. URL https://arxiv.org/abs/2008.12284

    arXiv 2020

  3. [3]

    and Strassen, V

    Baur, W. and Strassen, V. The complexity of partial derivatives. Theoretical Computer Science, 22 0 (3): 0 317--330, 1983. ISSN 0304-3975. doi:https://doi.org/10.1016/0304-3975(83)90110-X

    doi:10.1016/0304-3975(83)90110-x 1983

  4. [4]

    T., Duvenaud, D., and Jacobsen, J.-H

    Behrmann, J., Grathwohl, W., Chen, R. T., Duvenaud, D., and Jacobsen, J.-H. Invertible residual networks. In International conference on machine learning, pp.\ 573--582. PMLR, 2019

    2019

  5. [5]

    Gradient-based optimization of hyperparameters

    Bengio, Y. Gradient-based optimization of hyperparameters. Neural computation, 12 0 (8): 0 1889--1900, 2000

    work page 1900

  6. [6]

    and Roulet, V

    Blondel, M. and Roulet, V. The elements of differentiable programming. arXiv preprint arXiv:2403.14606, 2024

    arXiv 2024

  7. [7]

    Efficient and modular implicit differentiation

    Blondel, M., Berthet, Q., Cuturi, M., Frostig, R., Hoyer, S., Llinares-L \'o pez, F., Pedregosa, F., and Vert, J.-P. Efficient and modular implicit differentiation. Advances in neural information processing systems, 35: 0 5230--5242, 2022

    work page 2022

  8. [8]

    J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., Vander P las, J., Wanderman- M ilne, S., and Zhang, Q

    Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., Vander P las, J., Wanderman- M ilne, S., and Zhang, Q. JAX : composable transformations of P ython+ N um P y programs, 2018. URL http://github.com/google/jax

    2018

Show all 48 references
  1. [9]

    A., Farquhar, G., Kemaev, I., Zintgraf, L

    Calian, D. A., Farquhar, G., Kemaev, I., Zintgraf, L. M., Hessel, M., Shar, J., Oh, J., Gy \"o rgy, A., Schaul, T., Dean, J., et al. Datarater: Meta-learned dataset curation. arXiv preprint arXiv:2505.17895, 2025

    2025

  2. [10]

    V., Ahn, H., Neiswanger, W., Xie, P., Strubell, E., and Xing, E

    Choe, S., Mehta, S. V., Ahn, H., Neiswanger, W., Xie, P., Strubell, E., and Xing, E. Making scalable meta learning practical. Advances in neural information processing systems, 36: 0 26271--26290, 2023

    2023

  3. [11]

    How to compute hessian-vector products? In ICLR Blogposts 2024, 2024

    Dagréou, M., Ablin, P., Vaiter, S., and Moreau, T. How to compute hessian-vector products? In ICLR Blogposts 2024, 2024

    2024

  4. [12]

    The D eep M ind JAX E cosystem, 2020

    DeepMind, Babuschkin, I., Baumli, K., Bell, A., Bhupatiraju, S., Bruce, J., Buchlovsky, P., Budden, D., Cai, T., Clark, A., Danihelka, I., Dedieu, A., Fantacci, C., Godwin, J., Jones, C., Hemsley, R., Hennigan, T., Hessel, M., Hou, S., Kapturowski, S., Keck, T., Kemaev, I., Ki...

    2020

  5. [13]

    Model-agnostic meta-learning for fast adaptation of deep networks

    Finn, C., Abbeel, P., and Levine, S. Model-agnostic meta-learning for fast adaptation of deep networks. In Precup, D. and Teh, Y. W. (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp.\ 1126--...

    2017

  6. [14]

    Forward and reverse gradient-based hyperparameter optimization

    Franceschi, L., Donini, M., Frasconi, P., and Pontil, M. Forward and reverse gradient-based hyperparameter optimization. In Precup, D. and Teh, Y. W. (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Rese...

    2017

  7. [15]

    Bilevel programming for hyperparameter optimization and meta-learning

    Franceschi, L., Frasconi, P., Salzo, S., Grazzi, R., and Pontil, M. Bilevel programming for hyperparameter optimization and meta-learning. In International conference on machine learning, pp.\ 1568--1577. PMLR, 2018

    2018

  8. [16]

    M., Hauth, A., et al

    Gemini, T., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  9. [17]

    Achieving logarithmic growth of temporal and spatial complexity in reverse automatic differentiation

    Griewank, A. Achieving logarithmic growth of temporal and spatial complexity in reverse automatic differentiation. Optimization Methods and software, 1 0 (1): 0 35--54, 1992

    1992

  10. [18]

    and Walther, A

    Griewank, A. and Walther, A. Algorithm 799: revolve: an implementation of checkpointing for the reverse or adjoint mode of computational differentiation. ACM Transactions on Mathematical Software (TOMS), 26 0 (1): 0 19--45, 2000

    2000

  11. [19]

    Deep residual learning for image recognition

    He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 770--778, 2016

    2016

  12. [21]

    Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022 b

    2022 arXiv

  13. [22]

    Meta-learning online adaptation of language models

    Hu, N., Mitchell, E., Manning, C., and Finn, C. Meta-learning online adaptation of language models. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics,...

    2023

  14. [23]

    D., and Finn, C

    Hu, N., Mitchell, E., Manning, C. D., and Finn, C. Meta-learning online adaptation of language models. arXiv preprint arXiv:2305.15076, 2023 b

    2023 arXiv

  15. [24]

    B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  16. [25]

    Kingma, D. P. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  17. [26]

    Reformer: The efficient transformer

    Kitaev, N., Kaiser, ., and Levskaya, A. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451, 2020

    2001 arXiv

  18. [27]

    Darts: Differentiable architecture search

    Liu, H., Simonyan, K., and Yang, Y. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055, 2018

    2018 arXiv

  19. [28]

    Investigating bi-level optimization for learning and vision from a unified perspective: A survey and beyond

    Liu, R., Gao, J., Zhang, J., Meng, D., and Lin, Z. Investigating bi-level optimization for learning and vision from a unified perspective: A survey and beyond. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44 0 (12): 0 10045--10067, 2021

    2021

  20. [29]

    Optimizing millions of hyperparameters by implicit differentiation

    Lorraine, J., Vicol, P., and Duvenaud, D. Optimizing millions of hyperparameters by implicit differentiation. In International conference on artificial intelligence and statistics, pp.\ 1540--1552. PMLR, 2020

    2020

  21. [30]

    Gradient-based hyperparameter optimization through reversible learning

    Maclaurin, D., Duvenaud, D., and Adams, R. Gradient-based hyperparameter optimization through reversible learning. In International conference on machine learning, pp.\ 2113--2122. PMLR, 2015

    2015

  22. [31]

    Reversible vision transformers

    Mangalam, K., Fan, H., Li, Y., Wu, C.-Y., Xiong, B., Feichtenhofer, C., and Malik, J. Reversible vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 10830--10840, 2022

    2022

  23. [32]

    D., Harrison, J., Maheswaranathan, N., and Sohl-Dickstein, J

    Metz, L., Freeman, C. D., Harrison, J., Maheswaranathan, N., and Sohl-Dickstein, J. Practical tradeoffs between memory, compute, and performance in learned optimizers. In Conference on Lifelong Learning Agents (CoLLAs), 2022. URL http://github.com/google/learned_optimization

    2022

  24. [33]

    M., Xu, Z., van Hasselt, H

    Oh, J., Hessel, M., Czarnecki, W. M., Xu, Z., van Hasselt, H. P., Singh, S., and Silver, D. Discovering reinforcement learning algorithms. Advances in Neural Information Processing Systems, 33: 0 1060--1070, 2020

    2020

  25. [34]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    OpenAI, T., Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  26. [35]

    Automatic differentiation in pytorch

    Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A. Automatic differentiation in pytorch. NIPS 2017 Autodiff Workshop, 2017

    2017

  27. [36]

    Pearlmutter, B. A. Fast exact multiplication by the hessian. Neural Computation, 6 0 (1): 0 147--160, 1994. doi:10.1162/neco.1994.6.1.147

    1994 doi

  28. [37]

    Zero: Memory optimizations toward training trillion parameter models

    Rajbhandari, S., Rasley, J., Ruwase, O., and He, Y. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pp.\ 1--16. IEEE, 2020

    2020

  29. [38]

    M., and Levine, S

    Rajeswaran, A., Finn, C., Kakade, S. M., and Levine, S. Meta-learning with implicit gradients. Advances in neural information processing systems, 32, 2019

    2019

  30. [39]

    Truncated back-propagation for bilevel optimization

    Shaban, A., Cheng, C.-A., Hatch, N., and Boots, B. Truncated back-propagation for bilevel optimization. In The 22nd International Conference on Artificial Intelligence and Statistics, pp.\ 1723--1732. PMLR, 2019

    2019

  31. [40]

    Memory-efficient gradient unrolling for large-scale bi-level optimization

    Shen, Q., Wang, Y., Yang, Z., Li, X., Wang, H., Zhang, Y., Scarlett, J., Zhu, Z., and Kawaguchi, K. Memory-efficient gradient unrolling for large-scale bi-level optimization. arXiv preprint arXiv:2406.14095, 2024

    2024 arXiv

  32. [41]

    Learning by directional gradient descent

    Silver, D., Goyal, A., Danihelka, I., Hessel, M., and van Hasselt, H. Learning by directional gradient descent. In International Conference on Learning Representations, 2021

    2021

  33. [42]

    Roformer: Enhanced transformer with rotary position embedding

    Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 0 127063, 2024

    2024

  34. [43]

    Sutton, R. S. Adapting bias by gradient descent: An incremental version of delta-bar-delta. In AAAI, volume 92, pp.\ 171--176. Citeseer, 1992

    1992

  35. [44]

    Attention is all you need

    Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems, 2017

    2017

  36. [45]

    Werbos, P. J. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE, 78 0 (10): 0 1550--1560, 1990

    1990

  37. [46]

    W., Colmenarejo, S

    Wichrowska, O., Maheswaranathan, N., Hoffman, M. W., Colmenarejo, S. G., Denil, M., Freitas, N., and Sohl-Dickstein, J. Learned optimizers that scale and generalize. In International conference on machine learning, pp.\ 3751--3760. PMLR, 2017

    2017

  38. [47]

    P., and Silver, D

    Xu, Z., van Hasselt, H. P., and Silver, D. Meta-gradient reinforcement learning. Advances in neural information processing systems, 31, 2018

    2018

  39. [48]

    P., Silver, D., and Singh, S

    Zahavy, T., Xu, Z., Veeriah, V., Hessel, M., Oh, J., van Hasselt, H. P., Silver, D., and Singh, S. A self-tuning actor-critic algorithm. Advances in neural information processing systems, 33: 0 20913--20924, 2020

    2020

  40. [49]

    An introduction to bilevel optimization: Foundations and applications in signal processing and machine learning

    Zhang, Y., Khanduri, P., Tsaknakis, I., Yao, Y., Hong, M., and Liu, S. An introduction to bilevel optimization: Foundations and applications in signal processing and machine learning. IEEE Signal Processing Magazine, 41 0 (1): 0 38--59, 2024

    2024

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.