Pith. sign in

REVIEW 4 cited by

Second-Order Forward-Mode Automatic Differentiation for Optimization

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2408.10419 v1 pith:JUQV4Z4Q submitted 2024-08-19 cs.LG stat.ML

Adam D. Cobb , At{i}l{i}m Güneş Baydin , Barak A. Pearlmutter , Susmit Jha This is my paper
classification cs.LGstat.ML
keywords second-ordersearchfomohforward-modehyperplanelineoptimizationbackpropagation
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

This paper introduces a second-order hyperplane search, a novel optimization step that generalizes a second-order line search from a line to a $k$-dimensional hyperplane. This, combined with the forward-mode stochastic gradient method, yields a second-order optimization algorithm that consists of forward passes only, completely avoiding the storage overhead of backpropagation. Unlike recent work that relies on directional derivatives (or Jacobian--Vector Products, JVPs), we use hyper-dual numbers to jointly evaluate both directional derivatives and their second-order quadratic terms. As a result, we introduce forward-mode weight perturbation with Hessian information (FoMoH). We then use FoMoH to develop a novel generalization of line search by extending it to a hyperplane search. We illustrate the utility of this extension and how it might be used to overcome some of the recent challenges of optimizing machine learning models without backpropagation. Our code is open-sourced at https://github.com/SRI-CSL/fomoh.

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Backpropagation-Free Metropolis-Adjusted Langevin Algorithm

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Forward-mode automatic differentiation can replace backpropagation inside the Metropolis-Adjusted Langevin Algorithm, yielding four MCMC samplers that are sometimes faster and lower-memory than standard MALA.

  2. IP-Basis PINNs: Efficient Multi-Query Inverse Parameter Estimation

    cs.LG 2025-09 conditional novelty 5.0 of 10

    A pre-trained basis network enables fast multi-query inverse parameter estimation by fitting only a linear readout online, demonstrated on harmonic oscillators, Lotka-Volterra, and quantum harmonic oscillator.

  3. Memory Savings at What Cost? A Study of Alternatives to Backpropagation

    cs.LG 2025-06 conditional novelty 5.0 of 10

    Checkpointed backpropagation beats forward-mode AD and zero-order optimization in accuracy, convergence speed, and compute for LLM fine-tuning, undermining claims that the alternatives are practical memory savers.

  4. Efficient Data Selection at Scale via Influence Distillation

    cs.CL 2025-05 conditional novelty 5.0 of 10

    Influence Distillation selects LLM fine-tuning data by approximating each sample's gradient influence on a target task via landmarks and JVP embeddings, matching or beating RDS+ accuracy at roughly one third the selec...

Pith tools