Memory-Reduced Meta-Learning with Guaranteed Convergence

Honglin Yang; Ji Ma; Xiao Yu

REVIEW 3 major objections 5 minor 64 references

Memory-Reduced Meta-Learning with Guaranteed Convergence

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims a memory-reduced meta-learning algorithm that keeps only the final inner-loop parameters and still converges sublinearly, with stochastic error $O(1/T)+O(1/|B|)$ and exact convergence in the deterministic case.

desk verdict A sensible memory-reduction trick with encouraging experiments, but the convergence proof rests on a task-alignment assumption the algorithm does not satisfy. read the letter →

arxiv 2412.12030 v1 pith:TXQZPCMP submitted 2024-12-16 cs.LG math.OC

Honglin Yang , Ji Ma , Xiao Yu This is my paper · ORCID

classification cs.LGmath.OC MSC 68T0590C26

keywords meta-learningbileveloptimizationmemoryreductionhypergradientestimationconjugategradientstochasticconvergencefew-shotlearning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Gradient-based meta-learning methods like MAML and ANIL adapt a model to new tasks with few examples, but their hypergradient estimates replay the whole inner adaptation history, so memory grows with the number of inner steps. The paper claims an algorithm that uses only the final inner-loop parameters, eliminating that history, and still converges sublinearly to an approximate stationary point. The stochastic guarantee is $O(1/T)$ in outer iterations plus an error floor $O(1/|B|)$ controlled by the task-batch size; in the deterministic setting the algorithm converges to an exact solution. The paper also claims computational complexity $O(\epsilon^{-1})$ and reports memory reductions greater than 50% relative to ANIL and an iterative-differentiation baseline on four few-shot benchmarks. If the claims hold, memory is no longer the bottleneck that limits how deep inner adaptation can go.

What carries the argument

The load-bearing object is the Hessian-inverse-vector product $v_i^* = [\nabla^2_\phi g_i(\theta,\phi_i^*)]^{-1}\nabla_\phi f_i(\theta,\phi_i^*)$, which is the only quantity needed to turn the implicit-differentiation hypergradient formula into a computable direction. Subroutine 1 computes it with conjugate gradient, and the warm-start rule $v^0_{i,t}=v^N_{i,t-1}$ (with $\phi^0_{i,t}=\phi^K_{i,t-1}$ for the inner loop) carries the approximation from one outer iteration to the next, making the residual $\zeta^N_{i,t}$ contract geometrically. The proof bundles this CG error with the inner-loop distance $\|\phi^0_{i,t}-\phi^*_{i,t}\|$ into a Lyapunov function $\Xi_t$; choosing $K$ and $N$ large enough makes the coupled system contract, and that contraction produces the sublinear bound.

What would settle it

Run Algorithm 1 on a small task distribution under two protocols: (a) a fixed cyclic order in which the same task returns at the next outer step, and (b) fully random resampling. If the bound from Theorem 1 is satisfied only in case (a), the warm-start recurrence is doing the work and the proof does not cover the random-resampling version; if both satisfy the bound, the gap is benign.

Watch

Extended reading notes

Core claim

The central claim is Theorem 1: under the paper's Assumptions 1 and 2, if the inner GD loop and the conjugate-gradient loop run for at least $K_0$ and $N_0$ iterations, the iterates satisfy $$\frac{1}{T+1}\sum_{t=0}^{T}\mathbb{E}[\|\nabla F(\theta_t)\|^2] \le O(1/T)+O(1/|B|).$$ The hypergradient is estimated as $\nabla_\theta f_B(\theta_t,\phi^K_t) - \frac{1}{|B|}\sum_i \nabla^2_{\theta\phi} g_i(\theta_t,\phi^K_{i,t}) v^N_{i,t}$, where $v^N_{i,t}$ is the output of $N$ conjugate-gradient steps for the Hessian-inverse-vector product and $\phi^K_{i,t}$ is the last inner iterate. Since no earlier inner iterates are stored or differentiated through, memory per outer iteration is nearly flat in $K$, and the same construction gives $O(1/T)$ convergence to an exact solution in the deterministic case (Theorem 2).

Load-bearing premise

The load-bearing premise is that the same task appears in consecutive outer iterations, so the warm-start identities $\phi^0_{i,t}=\phi^K_{i,t-1}$ and $v^0_{i,t}=v^N_{i,t-1}$ carry genuine information; the algorithm as written samples a fresh task batch each outer iteration, so that premise is not stated or guaranteed.

Editorial extensions

If this is right

At fixed memory, the inner adaptation loop can be made much deeper than in MAML or ANIL, because only the final adapted parameters and the final CG vector are kept.
The average squared gradient norm of the meta-objective can be pushed down by increasing the task-batch size $|B|$, which sets the $O(1/|B|)$ error floor.
In deterministic meta-learning the method reaches an exact stationary solution at rate $O(1/T)$, requiring $O(\kappa\epsilon^{-1})$ gradient evaluations on $\phi$ and $\theta$.
Hypergradient estimation costs $O(\max\{p,q\})$ per iteration rather than $O(q^2)$ or $O(pq)$, because only Hessian- and Jacobian-vector products are formed.
On CIFAR-FS, FC100, miniImageNet, and tieredImageNet, the method reaches a target validation accuracy faster than MAML, ANIL, and ITD-BiO while using more than 50% less memory.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The warm-start step $\phi^0_{i,t}=\phi^K_{i,t-1}$ presupposes that the same task $i$ appears in consecutive outer batches; with fresh task sampling the recurrence in Lemma 4 has no object to compare. Comparing fixed task order against random resampling would show whether the theorem describes the algorithm as written.
A variant that resets the CG vector or re-initializes inner parameters after every outer step would remove the task-alignment requirement; whether the $O(1/T)$ rate survives is a testable open question.
The $O(1/|B|)$ floor is inherited from finite-batch stochastic optimization, and the paper argues variance reduction is hard for hypergradients; a variance-reduced implicit-differentiation estimator would be a natural next step.
Because the method truncates implicit differentiation with a CG solver and stores no trajectory, it sits between first-order meta-learning and full implicit-differentiation methods; large $N$ should recover the implicit-gradient behavior.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The core idea here is worth taking seriously: run a fixed-K inner loop, then estimate the Hessian-inverse-vector product with warm-started conjugate gradient, and never store the lower-level trajectory. That genuinely cuts memory and the experiments back it up on four benchmarks. The writing is clear and the authors know the related bilevel/meta-learning literature. If the theory held, this would be a small but solid contribution.

It does not hold as written. The load-bearing problem is the warm-start in Algorithm 1, Step 6. The proof of Lemma 4 starts with phi^0_{i,t+1} = phi^K_{i,t} and then bounds ||phi^*_{i,t+1} - phi^*_{i,t}|| in terms of ||theta_{t+1} - theta_t||. That is only valid if task i is the same task in consecutive outer iterations. But Step 3 resamples the batch B ~ P(T) every iteration, so slot i at time t is generally a different task from slot i at time t-1. The warm-started parameters belong to one task while the optimal parameters in Lemma 4 belong to another, so the recurrence has no content. Theorem 1 does not follow from the given proof.

The second issue is that Lemma 3 bounds the hypergradient estimation error as a pure variance term plus the CG residual, but it never accounts for the deterministic bias from evaluating the hypergradient at phi^K instead of phi^*. That bias does not vanish as |B| grows, so even a fixed N and K would still leave a theta-dependent error that the descent argument ignores. Lemma 5 also applies the CG error bound in the A-norm to the residual zeta, which is not the same norm; the proof hand-waves the conditioning factor. That might be repairable, but it is not a minor typo.

The experiments are encouraging but they do not rescue the theory. The memory numbers are clean and the wall-clock times are respectable, but the claimed convergence guarantees are the paper's main selling point.

Bottom line: this is a paper for the optimization-based meta-learning crowd, and the algorithmic idea is worth remembering. But the central theorem is unproven, and the gaps are not cosmetic. I would not send this to peer review in its current form. A revised version that fixes the task-alignment issue—for example by reusing the same task batch across iterations or reworking the warm-start analysis—and properly handles the hypergradient bias could become a solid paper.

Referee Report

3 major / 5 minor

Summary. The paper proposes a memory-reduced meta-learning algorithm (Algorithm 1) that estimates hypergradients without storing historical lower-level parameters or gradients. The key idea is to approximate the Hessian-inverse-vector product in the implicit-differentiation formula by a conjugate-gradient subroutine, warm-started from the previous outer iteration, and to update only the last inner-loop point. The authors claim a sublinear convergence rate O(1/T)+O(1/|B|) for stochastic meta-learning, an exact-solution O(1/T) rate in the deterministic case, and O(epsilon^{-1}) complexity, with experiments on four few-shot benchmarks showing competitive accuracy and reduced memory.

Significance. If the convergence guarantees were valid, the paper would make a meaningful contribution: it combines the memory benefits of first-order MAML-type updates with a provable convergence rate matching implicit-differentiation methods, and it reduces per-iteration Hessian/Jacobian cost from O(q^2) (or O(pq)) to O(max{p,q}). The experimental section also directly documents memory reduction as a function of inner-loop iterations, which is a useful comparison point. However, the central theoretical claim is not established by the proof as written: the warm-start recurrence mixes task identities across independent task batches, the hypergradient estimation bound omits a finite-K bias, and the conjugate-gradient residual bound omits a conditioning factor. These are load-bearing issues, not presentation details.

major comments (3)

[Algorithm 1, Step 6; Appendix B.1, Lemma 4, Eq. (21)] The proof of Lemma 4 starts from the identity phi^0_{i,t+1} = phi^K_{i,t}, and Eq. (37) bounds E[||phi^*_{i,t+1} - phi^*_{i,t}||^2] by terms involving ||theta_{t+1} - theta_t||^2 through Eq. (29). This is valid only if phi^*_{i,t} and phi^*_{i,t+1} are optimal parameters of the same task i. But Algorithm 1, Step 3 samples a fresh task batch B ~ P(T) at every outer iteration; the index i in Step 6 is a slot in the current batch, not a persistent task identity. No fixed-task-alignment assumption is stated. Consequently the contraction in Lemma 4, and hence the Lyapunov argument (75)-(80) proving Theorem 1, do not apply to Algorithm 1 as written. The same issue affects Theorem 2 because its proof inherits Lemma 4.
[Appendix B.1, Lemma 3, Eq. (15)] Lemma 3 bounds E[||nabla-hat-F_B(theta_t) - nabla F(theta_t)||^2] by a pure variance term 2 sigma-bar^2/|B| plus a CG residual term. Here nabla-hat-F_B uses the K-step inner-loop point phi^K_{i,t}, while nabla F uses the exact lower-level optimum phi^*_{i,t}. For finite K, the difference is not mean-zero: it contains a deterministic bias of order E[||phi^K_{i,t} - phi^*_{i,t}||^2] that Assumption 2 does not control. The proof's step E[||nabla F_B(theta_t) - nabla F(theta_t)||^2] <= sigma-bar^2/|B| treats this bias as if it were only stochastic variance. This bias propagates into Eq. (62) and ultimately into the claimed O(1/T) + O(1/|B|) bound, so the proof of Theorem 1 is incomplete without an explicit bound on the truncation bias.
[Appendix B.1, Lemma 5, Eq. (38)] Lemma 2 provides a bound on the CG error in the A-norm: ||e^k||^2_A <= Gamma_k ||e^0||^2_A. To convert this into a bound on the residual norm ||zeta^N||^2 = ||A e^N||^2, one needs a conditioning factor: ||A e||^2 <= lambda_max(A) ||e||^2_A and ||e^0||^2_A <= lambda_min(A)^{-1} ||zeta^0||^2, giving a factor of order kappa = L_g/mu. The first term of (38) contains only Gamma_N (l_g^2 + sigma^2_g1)/mu^2 and omits this factor. The deterministic version in Eq. (84) does include a kappa^2 factor, which highlights the omission in the stochastic case. Without this factor, the contraction condition (70) on Gamma_N is too optimistic and the recurrence for zeta^N is not justified.

minor comments (5)

[Algorithm 1, Step 11] The phrase 'Jacobin-vector products' should be 'Jacobian-vector products'.
[Appendix A, Eq. (9)] The notation nabla-hat-F_B(.) appears in the appendix without being defined in the main text; the hat notation is used inconsistently between Eq. (8) and the proof of Lemma 3.
[References] The reference list is duplicated in full after the technical appendices; the duplicate copy should be removed.
[Appendix B.3, Corollary 1] The gradient-evaluation count for Step 11 is not derived explicitly and appears to conflate Jacobian-vector products with phi-gradient evaluations; please spell out which computations are counted.
[Eq. (37) and Eq. (36)] In Eq. (36) the constant multiplying ||theta_{t+1}-theta_t|| ||phi^*_{i,t+1}-phi^*_{i,t}|| is written as l_f, but the quantity being controlled is nabla_phi g, so the corresponding Lipschitz constant should be l_g (or the notation should be explained).

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the proof relies on external standard lemmas and algebraic recurrences, and the main flagged issue is a task-identity assumption gap, not a circular step.

full rationale

All load-bearing ingredients of Theorem 1 are either imported from external references (Lemma 1 from Ghadimi and Wang 2018; Lemma 2 from Shewchuk et al. 1994) or derived from Assumptions 1-2 through algebraic recurrences (Lemmas 3-5). The claimed O(1/T)+O(1/|B|) rate follows from summing these recurrences and choosing K0 and N0 so that certain coefficients are smaller than 1; no fitted numerical value is renamed as a prediction, and no conclusion of the theorem is used to define the algorithm or its constants. The only serious issue visible in the manuscript is a validity gap: Algorithm 1 Step 3 resamples B ~ P(T) every outer iteration, while Step 6 and Subroutine 1 carry forward phi^K_{i,t-1} and v^N_{i,t-1} under the same index i. Lemma 4 then compares phi*_{i,t} and phi*_{i,t+1} as if task i persists, which the algorithm as written does not guarantee. This is a hidden assumption or proof gap, not circularity: it is not an instance of the conclusion being used as an input, of a fitted parameter being called a prediction, or of a load-bearing self-citation. Therefore the circularity score is 0.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The proof imports standard smoothness assumptions and one prior lemma; the algorithm's warm-start introduces an unstated task-alignment premise. No new physical or mathematical entities are introduced.

free parameters (5)

Inner-loop iterations K = 20 in experiments; grid {5,10,15,20}
Hand-tuned; theory requires K≥K0, whose value depends on unknown constants and is not computed.
CG iterations N = 20 in experiments; grid {5,10,15,20}
Hand-tuned; theory requires N≥N0, which may be much larger than 20.
Upper-level step size λ_θ = 0.0001 (CIFAR-FS, FC100) or 0.001 (miniImageNet, tieredImageNet)
Chosen by hand; small enough for stability.
Lower-level step size λ_ϕ = 0.1 (CIFAR-FS, FC100) or 0.05 (miniImageNet, tieredImageNet)
Chosen by hand; used in the inner-loop gradient descent.
Task batch size |B| = 32
Fixed in experiments; the convergence error floor is O(1/|B|).

assumptions (5)

domain assumption Assumption 1: F is nonconvex and each g_i(θ,·) is μ-strongly convex in ϕ.
Stated in Section 2.1; needed for the implicit function formula and for linear convergence of the inner loop.
domain assumption Assumption 2: ∇f, ∇g, ∇²_{θϕ}g, ∇²_ϕg are Lipschitz and the stochastic gradients have bounded variance.
Stated in Section 2.1; used throughout the proofs of Lemmas 3-5 and Theorem 1.
ad hoc to paper Fixed task identity across outer iterations for the warm start ϕ^0_{i,t}=ϕ^K_{i,t-1}.
Algorithm 1 Step 6 and Lemma 4 require the same task i in consecutive batches; this is not stated in the paper.
standard math Lemma 1 from Ghadimi and Wang (2018): ∇F is Lipschitz continuous with constant L_F.
Imported lemma used in Eq. (59) of the proof of Theorem 1.
standard math CG convergence and Cayley-Hamilton theorem (Shewchuk 1994).
Used in Lemma 2 and Theorem 2 to bound the residual of the conjugate-gradient subroutine.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Memory-Reduced Meta-Learning with Guaranteed Convergence." pith.science (2026). https://pith.science/paper/TXQZPCMP

@misc{pith2026241212030,
  author       = {Pith},
  title        = {Pith review of: Memory-Reduced Meta-Learning with Guaranteed Convergence},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TXQZPCMP}},
  note         = {Machine review of arXiv:2412.12030}
}

abstract

The optimization-based meta-learning approach is gaining increased traction because of its unique ability to quickly adapt to a new task using only small amounts of data. However, existing optimization-based meta-learning approaches, such as MAML, ANIL and their variants, generally employ backpropagation for upper-level gradient estimation, which requires using historical lower-level parameters/gradients and thus increases computational and memory overhead in each iteration. In this paper, we propose a meta-learning algorithm that can avoid using historical parameters/gradients and significantly reduce memory costs in each iteration compared to existing optimization-based meta-learning approaches. In addition to memory reduction, we prove that our proposed algorithm converges sublinearly with the iteration number of upper-level optimization, and the convergence error decays sublinearly with the batch size of sampled tasks. In the specific case in terms of deterministic meta-learning, we also prove that our proposed algorithm converges to an exact solution. Moreover, we quantify that the computational complexity of the algorithm is on the order of $\mathcal{O}(\epsilon^{-1})$, which matches existing convergence results on meta-learning even without using any historical parameters/gradients. Experimental results on meta-learning benchmarks confirm the efficacy of our proposed algorithm.

Figures

Figures reproduced from arXiv: 2412.12030 by the authors.

**Figure 2.** Memory costs of Algorithm 1, MAML, ANIL, and ITD-BiO under different numbers of inner-loop iterations [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

64 extracted references · 41 canonical work pages

[1]

Antoniou, A.; Edwards, H.; and Storkey, A. 2019. How to train your MAML. arXiv:1810.09502

arXiv 2019
[2]

Arnold, S. M. R.; Mahajan, P.; Datta, D.; Bunner, I.; and Zarkias, K. S. 2020. learn2learn: A Library for Meta-Learning Research. arXiv:2008.12284

arXiv 2020
[3]

Baik, S.; Choi, J.; Kim, H.; Cho, D.; Min, J.; and Lee, K. M. 2021. Meta-learning with task-adaptive loss function for few-shot learning. In Proceedings of the IEEE/CVF international conference on computer vision, 9465--9474

work page 2021
[4]

F.; Torr, P

Bertinetto, L.; Henriques, J. F.; Torr, P. H. S.; and Vedaldi, A. 2018. Meta-learning with differentiable closed-form solvers. International Conference on Learning Representations

work page 2018
[5]

Chen, T.; Sun, Y.; Xiao, Q.; and Yin, W. 2022. A single-timescale method for stochastic bilevel optimization. In International Conference on Artificial Intelligence and Statistics, 2466--2488. PMLR

work page 2022
[6]

Chen, T.; Sun, Y.; and Yin, W. 2021. Closing the gap: Tighter analysis of alternating stochastic gradient methods for bilevel problems. Advances in Neural Information Processing Systems, 34: 25294--25307

2021
[7]

Chen, Z.; Liang, S.; Li, L.; and Cheng, S. 2024. Quantized Primal-Dual Algorithms for Network Optimization With Linear Convergence. IEEE Transactions on Automatic Control, 69(1): 471--478

work page 2024
[8]

Chen, Z.; and Wang, Y. 2024 a . Locally Differentially Private Decentralized Stochastic Bilevel Optimization with Guaranteed Convergence Accuracy. In Salakhutdinov, R.; Kolter, Z.; Heller, K.; Weller, A.; Oliver, N.; Scarlett, J.; and Berkenkamp, F., eds., Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of M...

work page 2024

Show all 64 references

[9]

Chen, Z.; and Wang, Y. 2024 b . Locally Differentially Private Gradient Tracking for Distributed Online Learning over Directed Graphs. IEEE Transactions on Automatic Control, 1--16

2024
[10]

Chi, Z.; Gu, L.; Liu, H.; Wang, Y.; Yu, Y.; and Tang, J. 2022. Metafscil: A meta-learning approach for few-shot class incremental learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 14166--14175

2022
[11]

Dagr \'e ou, M.; Ablin, P.; Vaiter, S.; and Moreau, T. 2022. A framework for bilevel optimization that enables stochastic and global variance reduction algorithms. Advances in Neural Information Processing Systems, 35: 26698--26710

2022
[12]

Denevi, G.; Stamos, D.; Ciliberto, C.; and Pontil, M. 2019. Online-within-online meta-learning. Advances in Neural Information Processing Systems, 32

2019
[13]

Domke, J. 2012. Generic methods for optimization-based modeling. In Artificial Intelligence and Statistics, 318--326. PMLR

2012
[14]

Fallah, A.; Mokhtari, A.; and Ozdaglar, A. 2020. On the convergence theory of gradient-based model-agnostic meta-learning algorithms. In International Conference on Artificial Intelligence and Statistics, 1082--1092. PMLR

2020
[15]

J.; Lin, Z.; and Zhang, T

Fang, C.; Li, C. J.; Lin, Z.; and Zhang, T. 2018. Spider: Near-optimal non-convex optimization via stochastic path-integrated differential estimator. Advances in neural information processing systems, 31

2018
[16]

Finn, C.; Abbeel, P.; and Levine, S. 2017. Model-Agnostic Meta -Learning for Fast Adaptation of Deep Networks. International Conference on Machine Learning

2017
[17]

Finn, C.; Yu, T.; Zhang, T.; Abbeel, P.; and Levine, S. 2017. One-shot visual imitation learning via meta-learning. In Conference on robot learning, 357--368. PMLR

2017
[18]

Franceschi, L.; Donini, M.; Frasconi, P.; and Pontil, M. 2017. Forward and Reverse Gradient-Based Hyperparameter Optimization. In International Conference on Machine Learning

2017
[19]

Franceschi, L.; Frasconi, P.; Salzo, S.; Grazzi, R.; and Pontil, M. 2018. Bilevel Programming for Hyperparameter Optimization and Meta -Learning. International Conference on Machine Learning

2018
[20]

Ghadimi, S.; and Wang, M. 2018. Approximation Methods for Bilevel Programming. arXiv:1802.02246

2018 arXiv
[21]

M.; Loizou, N.; Qian, X.; Sailanbayev, A.; Shulgin, E.; and Richt \'a rik, P

Gower, R. M.; Loizou, N.; Qian, X.; Sailanbayev, A.; Shulgin, E.; and Richt \'a rik, P. 2019. SGD: General analysis and improved rates. In International conference on machine learning, 5200--5209. PMLR

2019
[22]

Grazzi, R.; Franceschi, L.; Pontil, M.; and Salzo, S. 2020. On the iteration complexity of hypergradient computation. In Proc. International Conference on Machine Learning (ICML)

2020
[23]

He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 770--778

2016
[24]

Hiller, M.; Harandi, M.; and Drummond, T. 2022. On enforcing better conditioned meta-learning for rapid few-shot adaptation. Advances in Neural Information Processing Systems, 35: 4059--4071

2022
[25]

A.; Wang, L.; and Gong, B

Jamal, M. A.; Wang, L.; and Gong, B. 2021. A lazy approach to long-horizon gradient-based meta-learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 6577--6586

2021
[26]

D.; Liang, Y.; and Poor, H

Ji, K.; Lee, J. D.; Liang, Y.; and Poor, H. V. 2020. Convergence of meta-learning with task-specific adaptation over partial parameters. Advances in Neural Information Processing Systems, 33: 11490--11500

2020
[27]

Ji, K.; Yang, J.; and Liang, Y. 2020. Bilevel Optimization: Convergence Analysis and Enhanced Design. International Conference on Machine Learning

2020
[28]

P.; and Ba, J

Kingma, D. P.; and Ba, J. 2017. Adam: A Method for Stochastic Optimization. arXiv:1412.6980

2017 arXiv
[29]

Kirsch, L.; Flennerhag, S.; Van Hasselt, H.; Friesen, A.; Oh, J.; and Chen, Y. 2022. Introducing symmetries to black box meta reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, 7202--7210

2022
[30]

Krizhevsky, A.; Hinton, G.; et al. 2009. Learning multiple layers of features from tiny images. Toronto, ON, Canada

2009
[31]

Lee, J.; Yoo, J.; and Kwak, N. 2023. SHOT: suppressing the hessian along the optimization trajectory for gradient-based meta-learning. Advances in Neural Information Processing Systems, 36: 61450--61465

2023
[32]

Li, H.; Dong, W.; Mei, X.; Ma, C.; Huang, F.; and Hu, B.-G. 2019. LGM-Net: Learning to generate matching networks for few-shot learning. In International conference on machine learning, 3825--3834. PMLR

2019
[33]

Li, J.; Lu, T.; Cao, X.; Cai, Y.; and Wang, S. 2021. Meta-imitation learning by watching video demonstrations. In International Conference on Learning Representations

2021
[34]

Lian, X.; Zhang, C.; Zhang, H.; Hsieh, C.-J.; Zhang, W.; and Liu, J. 2017. Can decentralized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient descent. Advances in neural information processing systems, 30

2017
[35]

Lorraine, J.; Vicol, P.; and Duvenaud, D. 2020. Optimizing millions of hyperparameters by implicit differentiation. In International conference on artificial intelligence and statistics, 1540--1552. PMLR

2020
[36]

Maclaurin, D.; Duvenaud, D.; and Adams, R. 2015. Gradient-based hyperparameter optimization through reversible learning. In International conference on machine learning, 2113--2122. PMLR

2015
[37]

Nichol, A.; Achiam, J.; and Schulman, J. 2018. On First-Order Meta-Learning Algorithms. arXiv:1803.02999

2018 arXiv
[38]

Oh, J.; Yoo, H.; Kim, C.; and Yun, S. 2021. BOIL: Towards Representation Change for Few-shot Learning. In International Conference on Learning Representations

2021
[39]

Oreshkin, B.; Rodr \' guez L \'o pez, P.; and Lacoste, A. 2018. Tadam: Task dependent adaptive metric for improved few-shot learning. Advances in neural information processing systems, 31

2018
[40]

Park, E.; and Oliva, J. B. 2019. Meta-curvature. Advances in neural information processing systems, 32

2019
[41]

Peng, H. 2020. A Comprehensive Overview and Survey of Recent Advances in Meta-Learning. arXiv:2004.11149

2020 arXiv
[42]

Raghu, A.; Raghu, M.; Bengio, S.; and Vinyals, O. 2019. Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML. International Conference on Learning Representations. Freezing backbone---body and head ANIL and NIL

2019
[43]

Rajeswaran, A.; Finn, C.; Kakade, S.; and Levine, S. 2019. Meta-Learning with Implicit Gradients. Neural Information Processing Systems

2019
[44]

Ravi, S.; and Larochelle, H. 2016. Optimization as a model for few-shot learning. In International conference on learning representations

2016
[45]

J.; Hefny, A.; Sra, S.; Poczos, B.; and Smola, A

Reddi, S. J.; Hefny, A.; Sra, S.; Poczos, B.; and Smola, A. 2016. Stochastic variance reduction for nonconvex optimization. In International conference on machine learning, 314--323. PMLR

2016
[46]

B.; Larochelle, H.; and Zemel, R

Ren, M.; Triantafillou, E.; Ravi, S.; Snell, J.; Swersky, K.; Tenenbaum, J. B.; Larochelle, H.; and Zemel, R. S. 2018. Meta-Learning for Semi-Supervised Few-Shot Classification. arXiv:1803.00676

2018 arXiv
[47]

Santoro, A.; Bartunov, S.; Botvinick, M.; Wierstra, D.; and Lillicrap, T. 2016. Meta-learning with memory-augmented neural networks. In International conference on machine learning, 1842--1850. PMLR

2016
[48]

Shaban, A.; Cheng, C.-A.; Hatch, N.; and Boots, B. 2019. Truncated back-propagation for bilevel optimization. In The 22nd International Conference on Artificial Intelligence and Statistics, 1723--1732. PMLR

2019
[49]

R.; et al

Shewchuk, J. R.; et al. 1994. An introduction to the conjugate gradient method without the agonizing pain. Carnegie-Mellon University. Department of Computer Science Pittsburgh

1994
[50]

Shu, J.; Meng, D.; and Xu, Z. 2021. Learning an Explicit Hyper-parameter Prediction Function Conditioned on Tasks. Journal of machine learning research

2021
[51]

Snell, J.; Swersky, K.; and Zemel, R. 2017. Prototypical networks for few-shot learning. Advances in neural information processing systems, 30

2017
[52]

Soydaner, D. 2020. A comparison of optimization algorithms for deep learning. International Journal of Pattern Recognition and Artificial Intelligence, 34(13): 2052013

2020
[53]

N.; Sun, H.; Vanschoren, J.; and Vu, P

Ullah, I.; Carri \'o n-Ojeda, D.; Escalera, S.; Guyon, I.; Huisman, M.; Mohr, F.; van Rijn, J. N.; Sun, H.; Vanschoren, J.; and Vu, P. A. 2022. Meta-album: Multi-domain meta-dataset for few-shot image classification. Advances in Neural Information Processing Systems, 35: 3232--3247

2022
[54]

N.; Kaiser, .; and Polosukhin, I

Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, .; and Polosukhin, I. 2017. Attention is all you need. Advances in neural information processing systems, 30

2017
[55]

Vinyals, O.; Blundell, C.; Lillicrap, T.; Wierstra, D.; et al. 2016. Matching networks for one shot learning. Advances in neural information processing systems, 29

2016
[56]

Vuorio, R.; Sun, S.-H.; Hu, H.; and Lim, J. J. 2019. Multimodal model-agnostic meta-learning via task-aware modulation. Advances in neural information processing systems, 32

2019
[57]

T.; and Ni, L

Wang, Y.; Yao, Q.; Kwok, J. T.; and Ni, L. M. 2020. Generalizing from a few examples: A survey on few-shot learning. ACM computing surveys (csur), 53(3): 1--34

2020
[58]

Wu, Y.; Liang, T.; Feng, S.; Jin, Y.; Lyu, G.; Fei, H.; and Wang, Y. 2023. Metazscil: A meta-learning approach for generalized zero-shot class incremental learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, 10408--10416

2023
[59]

Yao, H.; Zhang, L.; and Finn, C. 2022. Meta-Learning with Fewer Tasks through Task Interpolation. arXiv:2106.02695

2022 arXiv
[60]

Zhang, B.; Luo, C.; Yu, D.; Li, X.; Lin, H.; Ye, Y.; and Zhang, B. 2024. Metadiff: Meta-learning with conditional diffusion for few-shot learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, 16687--16695

2024
[61]

Zhou, F.; Wu, B.; and Li, Z. 2018. Deep Meta-Learning: Learning to Learn in the Concept Space. arXiv:1802.03596

2018 arXiv
[62]

Zhou, P.; Yuan, X.; Xu, H.; Yan, S.; and Feng, J. 2019. Efficient meta learning via minibatch proximal update. Advances in Neural Information Processing Systems, 32

2019
[63]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...
[64]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

Antoniou, A.; Edwards, H.; and Storkey, A. 2019. How to train your MAML. arXiv:1810.09502

arXiv 2019

[2] [2]

Arnold, S. M. R.; Mahajan, P.; Datta, D.; Bunner, I.; and Zarkias, K. S. 2020. learn2learn: A Library for Meta-Learning Research. arXiv:2008.12284

arXiv 2020

[3] [3]

Baik, S.; Choi, J.; Kim, H.; Cho, D.; Min, J.; and Lee, K. M. 2021. Meta-learning with task-adaptive loss function for few-shot learning. In Proceedings of the IEEE/CVF international conference on computer vision, 9465--9474

work page 2021

[4] [4]

F.; Torr, P

Bertinetto, L.; Henriques, J. F.; Torr, P. H. S.; and Vedaldi, A. 2018. Meta-learning with differentiable closed-form solvers. International Conference on Learning Representations

work page 2018

[5] [5]

Chen, T.; Sun, Y.; Xiao, Q.; and Yin, W. 2022. A single-timescale method for stochastic bilevel optimization. In International Conference on Artificial Intelligence and Statistics, 2466--2488. PMLR

work page 2022

[6] [6]

Chen, T.; Sun, Y.; and Yin, W. 2021. Closing the gap: Tighter analysis of alternating stochastic gradient methods for bilevel problems. Advances in Neural Information Processing Systems, 34: 25294--25307

2021

[7] [7]

Chen, Z.; Liang, S.; Li, L.; and Cheng, S. 2024. Quantized Primal-Dual Algorithms for Network Optimization With Linear Convergence. IEEE Transactions on Automatic Control, 69(1): 471--478

work page 2024

[8] [8]

Chen, Z.; and Wang, Y. 2024 a . Locally Differentially Private Decentralized Stochastic Bilevel Optimization with Guaranteed Convergence Accuracy. In Salakhutdinov, R.; Kolter, Z.; Heller, K.; Weller, A.; Oliver, N.; Scarlett, J.; and Berkenkamp, F., eds., Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of M...

work page 2024