Pith. sign in

REVIEW 4 major objections 6 minor 20 references

Applications of fractional calculus in learned optimization

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Training a neural network to predict the fractional order of the gradient is possible, and on Rosenbrock 2D the resulting learned optimizer converges in 99.2% of runs.

desk verdict The learned prediction of alpha is a real idea, but Eq. 13 is not a Taylor expansion of any fractional derivative defined in the paper, so the reported gains do not support the claim about fractional order. read the letter →

arxiv 2411.14855 v1 pith:LOSI3HGU submitted 2024-11-22 cs.LG

Teodor Alexandru Szente , James Harrison , Mihai Zanfir , Cristian Sminchisescu This is my paper · ORCID
classification cs.LG
keywords fractionalcalculusgradientdescentlearnedoptimizationmeta-learningflowchaoticsystemsRosenbrockfunctionbackpropagationthroughtime
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that fractional-order gradient descent can be made practical by learning the order rather than tuning it by hand. It reports that a small neural network, fed the current state and gradient information, can output the fractional order $\alpha$ and step size $\eta$ so that the update $X_{t+1}=X_t-\eta D^{\alpha} f(X_t)$ converges on Rosenbrock 2D in 99.2% of 1000 random starts with supervision, and 71.8% when trained on other functions and tested on Rosenbrock, compared with at most 3.5% for standard optimizers. The same idea is applied to chaotic systems through a fractional gradient flow discretization, which the paper reports stabilizes backpropagation-through-time for Lorenz parameter optimization. If true, this would replace hand-designed update rules with a learned mechanism that behaves like a second-order method while only relying on first-order information.

What carries the argument

The carrying object is a learned map to fractional order: $F_\theta$ takes the current state, normalized gradient, gradient magnitude, and Fourier features and outputs $(\alpha_t, \eta_t)$, producing $X_{t+1}=X_t-\eta_t D^{\alpha_t} f(X_t)$. The fractional derivative is defined via the Riemann-Liouville, Caputo, and Grünwald-Letnikov formulations, but the paper approximates $D^{\alpha}$ by a first-order truncated Taylor expansion with weights $\frac{1}{\Gamma(1-\alpha)} f(X) + \frac{1}{\Gamma(\alpha)} \Delta X \frac{\partial f}{\partial X}$ (Eq. 13). For chaotic systems the mechanism is the fractional gradient flow $d^{\alpha} X/dt = -\nabla f(X)$, discretized by the multi-step scheme of Eq. (17), which is what stabilizes backpropagation-through-time on the Lorenz parameter-optimization task. Fourier features are an auxiliary but important ingredient: the paper reports they were necessary for the network to learn high-frequency structure and generalize across functions.

What would settle it

Compute the true Riemann-Liouville or Caputo fractional derivative (Eq. 4 or Eq. 6, via numerical quadrature) or the Grünwald-Letnikov finite-difference form (Eq. 7) along the optimizer's trajectory, and compare its update direction with the Eq. (13) surrogate; if the directions disagree substantially, the 99.2% convergence is an artifact of the surrogate update. Alternatively, rerun the supervised experiment with $\alpha$ clamped to 1 everywhere; if convergence remains near 99%, the learned fractional order itself is not doing the work.

Watch

Extended reading notes

Core claim

The central claim, stated on the paper's own terms, is that 'it is possible to train a neural network to predict the order of the gradient effectively.' Concretely, a learned optimizer $F_\theta$ maps the current state, normalized gradient, gradient magnitude, and Fourier features to $(\alpha, \eta)$; the update is a fractional gradient step with $D^{\alpha}$ approximated by the first-order truncated Taylor expression of Eq. (13). On Rosenbrock 2D this learned optimizer converges in 99.2% of runs with supervision and 71.8% without, whereas gradient descent converges in 0.6% and Adam in 1.6%. For chaotic systems, replacing the gradient update in backpropagation-through-time with the discretized fractional gradient flow of Eq. (17) makes the trajectory loss converge faster and more stably than the NRES gradient-estimator baseline. The paper frames these results as evidence that fractional calculus can be a viable component of learned optimization, while noting the current limitation to small-dimensional problems.

Load-bearing premise

The load-bearing premise is that the first-order truncated Taylor expansion in Eq. (13), with its Gamma-function weights, really does approximate the fractional derivative in the optimization setting; the paper gives no derivation or error bound, and if the approximation is inaccurate the reported convergence gains do not establish that fractional order prediction works.

Editorial extensions

If this is right

  • Fractional-order optimizers need not be hand-tuned: a learned policy can predict both the order and the step size on the fly, and on Rosenbrock 2D the supervised policy reaches 99.2% convergence, a departure from the sub-4% rates of classical optimizers.
  • The unsupervised results (71.8%) indicate that a policy trained on other classical functions transfers to a held-out target, suggesting fractional-order prediction is a learnable skill rather than a memorized fit.
  • The fractional gradient flow discretization renders backpropagation-through-time competitive on the Lorenz parameter task, converging faster and more stably than the noise-reuse evolution-strategies gradient estimator NRES.
  • Fourier features are a necessary ingredient for the learned policy to handle high-frequency behavior in low dimensions, which constrains the architecture choices for a practical learned fractional optimizer.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If Eq. (13) is a faithful local model, the same learning rule could be extended to per-dimension fractional orders, which the paper identifies as a major open problem; a natural next test is whether a Transformer or structured-state-space map predicts a vector of $\alpha$ values for high-dimensional losses.
  • The FGF discretization's stabilizing effect on BPTT suggests a cheaper alternative to evolutionary-search gradient estimators in chaotic control problems; one could test whether the same scheme stabilizes BPTT on partially observable or stochastic chaotic systems.
  • It would be informative to check whether the learned order $\alpha$ tracks curvature or conditioning (e.g., near the Rosenbrock valley), since a positive correlation would indicate the network is learning geometry rather than simply fitting a schedule.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a learned optimizer that predicts a fractional derivative order α and step size η, with the update X_{t+1}=X_t−ηD^α f(X_t) where D^α is approximated by the truncated expansion in Eq. (13). The method is tested on classical optimization functions, reporting a 99.2% convergence rate on Rosenbrock 2D when trained with supervision on the target function and 71.8% when trained without supervision, compared with 0.6% for gradient descent and 1.6% for Adam. A second set of experiments applies a fractional gradient flow discretization, Eq. (17), to Lorenz system parameter optimization and reports improved stability/convergence for backpropagation-through-time. The abstract concludes that a neural network can be trained to predict the order of the gradient effectively.

Significance. If the central claim were supported, the paper would make a useful contribution: learned selection of a fractional order in optimization is a novel idea, and the without-supervision Rosenbrock result (71.8% convergence vs. 0.6% for gradient descent) would be a striking demonstration that a small network can outperform standard optimizers on a held-out function. The paper also makes a reasonable effort to compare against strong baselines, including VeLO. However, the significance is entirely conditional on the validity of Eq. (13), the approximation that defines the optimizer's update. Because that approximation does not correspond to any standard fractional derivative and is not validated, the reported gains currently do not constitute evidence for the paper's abstract claim. The without-supervision experiment is a genuine attempt at independent evaluation, and the Lorenz experiment is an interesting probe, but both rely on the same unvalidated fractional-calculus foundation.

major comments (4)
  1. [§2.1, Eq. (13)] Equation (13) is not a Taylor expansion of the Riemann–Liouville or Caputo derivative defined in §1.1, and no derivation, error bound, or numerical validation is provided. For f(x)=x, both the RL and Caputo derivatives give x^{1−α}/Γ(2−α), whereas Eq. (13) evaluated at ΔX=0 gives x/Γ(1−α). For f(x)=x^2, the formula gives x^2/Γ(1−α)+2xΔX/Γ(α), whereas the actual fractional derivatives are x^{2−α}/Γ(3−α) plus terms involving the lower limit. The first term in Eq. (13) is a state-dependent scalar multiple of f(X) and does not match the X^{-α}-type prefactor produced by any standard fractional derivative. Since Eq. (12) uses this approximation for every update, the meta-learning results in Table 1 demonstrate only that a learned two-parameter update proportional to f(X) and ∇f(X) can optimize these functions; they do not show that the network predicts a meaningful fractional order.
  2. [Table 1 and §2.1] The headline 99.2% result is circular: row (8) is the model trained 'with supervision' on the target function and then tested on the same function. This is an in-sample fit, not evidence of generalization or of effective fractional-order prediction. The without-supervision row (9), which holds out the target function, is the relevant independent evaluation, and it should be the basis for any claim about predicting fractional order on unseen functions.
  3. [§2.2 and Fig. 2(b)] The chaotic-system experiment is reported without enough detail to assess the claim that the FGF discretization makes TBTT 'perform the best.' The fractional order α in Eq. (16) appears to be a manually chosen free parameter, not predicted by the learned network, and no values of α or η, no error bars, and no number of independent runs are given. The comparison between NRES and TBTT also lacks a precise definition of the convergence metric. These omissions make it impossible to determine whether the improved stability comes from the fractional flow or from a differently tuned experiment.
  4. [Eq. (17) and reference [3]] The discretization in Eq. (17) is attributed to reference [3], but that reference concerns approximations of fractional-order transfer functions and does not appear to contain this optimization-oriented discretization. The authors should provide a derivation or a correct citation, and they should state the conditions under which the claimed O(1/t^α) convergence rate applies to the optimization setting of §2.2.
minor comments (6)
  1. [Eq. (13)] The notation ΔX ∂f/∂X is ambiguous for vector-valued X; in a multivariate setting the second term should be written as ΔX·∇f or the equivalent scalar product.
  2. [§1.2, Eq. (10)] The discussion of the fractional Jacobian would benefit from an explicit statement of how fractional derivatives are extended to multivariate functions and why the transformation T_{J^α_f} is called a 'linear approximation only when α=1.'
  3. [Eq. (14)] The training loss Lθ = log(f(X_{t+1})) − log(f(X_t)) requires f to be positive on the sampled trajectories; the paper does not state that all classical functions are positive-valued, and the loss is undefined for negative function values.
  4. [Table 1] The column header 'Truncated trajectory length' is not defined, and the reported values near 1000 are puzzling given the stated maximum of 100 optimization steps; the authors should clarify what this quantity measures.
  5. [§3.1] The paper does not provide training details for Fθ, such as network architecture, hidden sizes, batch size, optimizer hyperparameters, or the number of training steps, which makes the experiments difficult to reproduce.
  6. [Fig. 2] The subplots in Fig. 2 lack axis labels and colorbar units, and the fractional order/learning-rate sweep for the FGF method is not described in the text.

Circularity Check

1 steps flagged · score 4.0 of 10

The 'with supervision' Rosenbrock result is an in-sample fit (trained and evaluated on the same function); the held-out 'without supervision' result supplies independent support, so the paper is only partially circular and has no load-bearing self-citation chain.

  1. fitted input called prediction [Section 3.1, Table 1 and caption]
    "(8) Ours w supervision 99.20% 12.156 ... (8) Our optimizer trained only on the target function with supervision (9) Our optimizer trained on other classical functions except the target function."

    Row (8) is trained on Rosenbrock 2D with a supervised loss that directly minimizes log f(X_{t+1}) on the same Rosenbrock 2D objective used for the convergence test; the 99.2% convergence rate is therefore an in-sample training statistic, not an independent prediction of the fractional order. The held-out row (9) samples only non-target functions during training and gives the independent 71.8% result.

full rationale

The only reduction-by-construction I can exhibit is the 'with supervision' row in Table 1, which is explicitly trained on the target Rosenbrock function and then evaluated on that same function; its 99.2% convergence rate is an in-sample fit, not a prediction. The 'w/o supervision' row (71.8%) is the genuine held-out result and gives the paper's central claim independent content, so the circularity is partial. I find no load-bearing self-citation chain: VeLO [14] and NRES [8] are baselines from overlapping authors but are used as external comparators, not as justification for the method's validity. The deeper problem with Eq. 13 — that it is not a valid first-order Taylor expansion of the RL/Caputo derivative and is never validated — is a correctness/soundness concern rather than circularity: a wrong approximation can invalidate the claim that the network predicts 'the order of the gradient' without making the claim equal to its inputs. I therefore do not count it as a circular step, though it should be weighed heavily in any correctness review.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central method relies on four unverified premises: the truncated Taylor expansion in Eq. 13, the applicability of the control-theory discretization in Eq. 17 to non-convex chaotic optimization, the log-ratio training objective, and the adequacy of a single scalar fractional order per problem. The network weights, the grid-searched α and η in the chaotic experiments, and the unspecified Fourier feature scale are fitted values that affect the reported results.

free parameters (4)
  • FGF fractional order α (chaotic experiments) = not reported
    Grid-searched over the test loss landscape (Fig. 2a: 'searching over fractional orders and learning rates'); selection on the test problem biases the comparison.
  • Step size η (FGF experiments) = not reported
    Grid-searched together with α on the test loss landscape; no cross-validation or error bars.
  • Neural network weights θ = trained on classical functions and, in the with-supervision regime, on the target function
    The network parameters are fitted to data; the with-supervision variant is fitted to the test function itself, making the reported 99.2% convergence an in-sample result.
  • Fourier feature scale
    The bandwidth of the Fourier features γ(X_t) is not specified; it affects the network's ability to fit high-frequency details and is a hand-tuned free parameter.
assumptions (4)
  • ad hoc to paper The first-order truncated Taylor expansion in Eq. 13 faithfully approximates the fractional derivative D^α f(X+ΔX).
    Eq. 13 is introduced without derivation or error bounds; it does not cleanly reduce to the standard derivative at α=1 and produces a nonlocal-looking first term.
  • domain assumption The discretization in Eq. 17, taken from control theory [3], is a valid and convergent optimizer update in the non-convex, chaotic setting of Lorenz parameter optimization.
    The paper cites a convergence rate for locally Lipschitz functions ([9]) but does not verify the assumptions hold for the chaotic loss landscape.
  • domain assumption The meta-objective L_θ = log(f(X_{t+1})) - log(f(X_t)) is a good training signal for learning an optimizer.
    Used without justification; log-scale differences can be dominated by early steps and may not reflect final convergence.
  • domain assumption A single scalar fractional order can meaningfully describe the update direction for all coordinates of a problem.
    The authors themselves note in the Limitations section that 'a single fractional order might not accurately describe all the dimensions'; this assumption underlies the whole method.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Applications of fractional calculus in learned optimization." pith.science (2026). https://pith.science/paper/LOSI3HGU

@misc{pith2026241114855,
  author       = {Pith},
  title        = {Pith review of: Applications of fractional calculus in learned optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LOSI3HGU}},
  note         = {Machine review of arXiv:2411.14855}
}
read the original abstract

Fractional gradient descent has been studied extensively, with a focus on its ability to extend traditional gradient descent methods by incorporating fractional-order derivatives. This approach allows for more flexibility in navigating complex optimization landscapes and offers advantages in certain types of problems, particularly those involving non-linearities and chaotic dynamics. Yet, the challenge of fine-tuning the fractional order parameters remains unsolved. In this work, we demonstrate that it is possible to train a neural network to predict the order of the gradient effectively.

Figures

Figures reproduced from arXiv: 2411.14855 by the authors.

Figure 1
Figure 1. Grid transform near origin for (a) TJf (b) TJ 1.2 f (c) TJ 1.25 f 2. Methodology 2.1. Meta-learning on classical functions We start by learning to optimize on a collection of classical functions1 . We consider each function to be parameterized by a state vector Xt . We train a neural network Fθ, that takes as input the current state, normalized gradients, magnitudes of the gradient and Fourier features γ(Xt) [19], a… view at source ↗
Figure 2
Figure 2. a) Loss surface for the function from eq. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 17 canonical work pages

  1. [3]

    Revisiting four approximation methods for fractional order transfer f unction implementations: Stabil- ity preservation, time and frequency response matching ana lyses

    Furkan Nur Deniz, Baris Baykant Alagoz, Nusret Tan, and M urat Koseoglu. Revisiting four approximation methods for fractional order transfer f unction implementations: Stabil- ity preservation, time and frequency response matching ana lyses. Annual Reviews in Control, 49:239–257, 2020. ISSN 1367-5788. doi: https://doi.org/1 0.1016/j.arcontrol.2020.03.003

    work page 2020

  2. [1]

    R´esum´e des lec ¸ons donn´ees `a l’ ´ecole royale polytechnique sur le calcul infinit ´esimal, volume 1

    Augustin-Louis Cauchy. R´esum´e des lec ¸ons donn´ees `a l’ ´ecole royale polytechnique sur le calcul infinit ´esimal, volume 1. Imprimerie royale, 1823. Reprint: Completes II( 4), Gauthier- Villars, Paris

    work page

  3. [2]

    FlashAttention-2: Faster attention with bette r parallelism and work partitioning

    Tri Dao. FlashAttention-2: Faster attention with bette r parallelism and work partitioning. In International Conference on Learning Representations (IC LR), 2024. 6 APPLICATIONS OF FRACTIONAL CALCULUS IN LEARNED OPTIMIZATIO N

    work page 2024

  4. [4]

    Transformer-based learned optimization

    Erik G¨ artner, Luke Metz, Mykhaylo Andriluka, C Daniel Freeman, and Cristian Sminchisescu. Transformer-based learned optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11970–11979, 2023

    work page 2023

  5. [5]

    Efficiently m odeling long sequences with struc- tured state spaces

    Albert Gu, Karan Goel, and Christopher R´ e. Efficiently m odeling long sequences with struc- tured state spaces. In The International Conference on Learning Representations (ICLR), 2022

    work page 2022

  6. [6]

    Rosenfeld

    Pham Viet Hai and Joel A. Rosenfeld. The gradient descent method from the perspective of fractional calculus. Mathematical Methods in the Applied Sciences , 44(7):5520–5547, 2021. doi: https://doi.org/10.1002/mma.7127

    work page doi:10.1002/mma.7127 2021

  7. [7]

    A closer look at learned optimiza- tion: Stability, robustness, and inductive biases

    James Harrison, Luke Metz, and Jascha Sohl-Dickstein. A closer look at learned optimiza- tion: Stability, robustness, and inductive biases. Advances in Neural Information Processing Systems, 35:3758–3773, 2022

    work page 2022

  8. [8]

    V ariance- reduced gradient estimation via noise-reuse in online evol ution strategies

    Oscar Li, James Harrison, Jascha Sohl-Dickstein, Virgi nia Smith, and Luke Metz. V ariance- reduced gradient estimation via noise-reuse in online evol ution strategies. Advances in Neural Information Processing Systems, 36:45489–45501, 2023

    work page 2023

Show all 20 references
  1. [9]

    Fractional differe ntial equation approach for convex optimization with convergence rate analysis

    Shu Liang, Le Yi Wang, and George Yin. Fractional differe ntial equation approach for convex optimization with convergence rate analysis. Optim. Lett., 14(1):145–155, 2020. doi: 10.1007/ S11590-019-01437-6

    2020

  2. [10]

    Note sur une formule pour les diff´ erentie lles ` a indices quelconques, ` a l’occasion d’un m´ emoire de m

    J Liouville. Note sur une formule pour les diff´ erentie lles ` a indices quelconques, ` a l’occasion d’un m´ emoire de m. tortolini. Journal de math ´ematiques pures et appliqu ´ees, 20, 1855

  3. [11]

    The novel adaptive fractional order gradient decent algorithms design via robust control

    Jiaxu Liu, Song Chen, Shengze Cai, and Chao Xu. The novel adaptive fractional order gradient decent algorithms design via robust control. ArXiv, abs/2303.04328, 2023

    2023 arXiv

  4. [12]

    Decoupled weight dec ay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight dec ay regularization. In International Conference on Learning Representations, 2019

    2019

  5. [13]

    V ariab le order fractional gradient descent method and its application in neural networks optim ization

    Weipu Lou, Wei Gao, Xianwei Han, and Yimin Zhang. V ariab le order fractional gradient descent method and its application in neural networks optim ization. In 2022 34th Chinese Control and Decision Conference (CCDC), pages 109–114, 2022. doi: 10.1109/CCDC55256. 2022.10033456

    2022

  6. [14]

    Daniel Freeman, Amil Merc hant, Lucas Beyer, James Bradbury, Naman Agrawal, Ben Poole, Igor Mordatch, Adam Rob erts, and Jascha Sohl- Dickstein

    Luke Metz, James Harrison, C. Daniel Freeman, Amil Merc hant, Lucas Beyer, James Bradbury, Naman Agrawal, Ben Poole, Igor Mordatch, Adam Rob erts, and Jascha Sohl- Dickstein. V eLO: Training V ersatile Learned Optimizers by Scaling Up. arXiv e-prints, art. arXiv:2211.09760, No...

    2022 doi arXiv

  7. [15]

    Oldham and J

    K. Oldham and J. Spanier. The Fractional Calculus Theory and Applications of Differentiation and Integration to Arbitrary Order. ISSN. Elsevier Science, 1974. ISBN 9780080956206. 7 APPLICATIONS OF FRACTIONAL CALCULUS IN LEARNED OPTIMIZATIO N

    1974

  8. [16]

    V ersuch einer allgemeinen Auffassung der Integration und Differentiation

    Bernhard Riemann. V ersuch einer allgemeinen Auffassung der Integration und Differentiation. Gesammelte Werke, Lipzig, 1876. ed. publ. posthumously

  9. [17]

    A caputo fractional derivative- based algorithm for optimization

    Y eonjong Shin, J´ erˆ ome Darbon, and George Em Karniadakis. A caputo fractional derivative- based algorithm for optimization. arXiv preprint arXiv:2104.02259, 2021

    2021 arXiv

  10. [18]

    State spac e approximation for general fractional order dynamic systems

    Zeng Liao Shu Liang, Cheng Peng and Y ong Wang. State spac e approximation for general fractional order dynamic systems. International Journal of Systems Science , 45(10):2203– 2212, 2014. doi: 10.1080/00207721.2013.766773

    2014

  11. [19]

    Fourier features let net- works learn high frequency functions in low dimensional dom ains

    Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sa ra Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Re n Ng. Fourier features let net- works learn high frequency functions in low dimensional dom ains. Advances in neural infor- mati...

    2020

  12. [20]

    Study on two-stage f ractional order gradient descend method

    Zhiguang Zhu, Ang Li, and Y ong Wang. Study on two-stage f ractional order gradient descend method. In 2021 40th Chinese Control Conference (CCC) , pages 7960–7964, 2021. doi: 10.23919/CCC52363.2021.9549324. 8

    2021

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.