REVIEW 2 major objections 2 minor
Practical Newton-Type Distributed Learning using Gradient Based Approximations
T0 review · 2 major / 2 minor · reviewed 2026-05-24 · grok-4.3
Pith's one-line read Modifying DANE to solve local subproblems approximately with iterative methods reduces local computation costs while preserving accuracy in distributed convex optimization.
desk verdict The paper suggests using iterative approximations for local subproblems in DANE to cut computation, but the abstract supplies no analysis, bounds, or experiments to show the tradeoff actually works. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Approximate iterative solutions to local subproblems in the DANE framework, which replace exact Newton steps with cheaper gradient iterations.
What would settle it
Observing that the approximate version requires significantly more communication rounds than exact DANE on the same dataset while achieving lower accuracy would disprove the practicality claim.
Extended reading notes
Core claim
The paper claims that by solving the local subproblems in DANE approximately using iterative gradient-based methods rather than providing exact solutions, the algorithm achieves a better balance between local computation and communication, leading to improved practicality compared to standard DANE while retaining fast convergence properties of second-order methods over first-order alternatives like SGD.
Load-bearing premise
Approximate solutions to the local subproblems will not substantially increase the number of communication rounds required or degrade final accuracy compared to exact local solves.
Editorial extensions
If this is right
- The total local computation decreases per communication round.
- The number of communication rounds stays comparable to exact DANE.
- Final accuracy remains similar to exact methods.
- Tradeoffs can be adjusted by choosing different iterative methods and iteration counts.
Reading between the lines
- This could make Newton-type methods viable in environments with high communication latency.
- The method might generalize to other distributed optimization frameworks beyond DANE.
- Tuning the approximation level based on data size could optimize performance further.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript modifies the DANE framework for distributed convex expected-risk minimization by replacing exact local Newton-type subproblem solves with approximate iterative (gradient-based) methods. The goal is to lower per-round local computation cost while preserving the number of communication rounds and final accuracy, with empirical comparisons to SGD and studies of different iterative solvers.
Significance. If the empirical trade-off holds and the approach is shown to be robust, the work would supply a practical, lower-cost variant of communication-efficient second-order distributed methods for large-scale convex problems.
major comments (2)
- [Theoretical Analysis] § on convergence (likely §3 or §4): the manuscript does not extend DANE’s linear-rate analysis to bound the effect of inexact local solves; without such a bound the claim that communication rounds remain comparable rests on the unverified assumption that local approximation error stays negligible globally.
- [Experiments] Experiments section (tables/figures comparing communication rounds): no direct head-to-head count of outer iterations is reported between the approximate variant and exact DANE on identical problem instances, so the central cost-accuracy tradeoff cannot be verified.
minor comments (2)
- [Abstract] Abstract: the list of iterative methods examined and the concrete accuracy/communication metrics obtained are not stated.
- [Method] Notation: the precise tolerance used to terminate the inner iterative solvers is not defined, making reproducibility of the reported trade-offs difficult.
Simulated Author's Rebuttal
We thank the referee for the constructive feedback. We address the two major comments below.
read point-by-point responses
-
Referee: [Theoretical Analysis] § on convergence (likely §3 or §4): the manuscript does not extend DANE’s linear-rate analysis to bound the effect of inexact local solves; without such a bound the claim that communication rounds remain comparable rests on the unverified assumption that local approximation error stays negligible globally.
Authors: We agree that the manuscript does not derive a new convergence bound that explicitly accounts for inexact local Newton solves. The work is positioned as a practical modification of DANE, and the claim of comparable communication rounds is supported by the empirical results rather than a tightened theoretical guarantee. In revision we will add a short discussion paragraph noting that the original DANE linear-rate result continues to apply when the local iterative solvers are run to sufficient accuracy (as controlled by the inner stopping criteria we study), and we will cite standard results on inexact Newton methods to justify why the approximation error remains controlled in our setting. revision: partial
-
Referee: [Experiments] Experiments section (tables/figures comparing communication rounds): no direct head-to-head count of outer iterations is reported between the approximate variant and exact DANE on identical problem instances, so the central cost-accuracy tradeoff cannot be verified.
Authors: We will revise the experimental section to include an explicit table (or additional columns in existing tables) that reports the exact number of outer communication rounds required by both exact DANE and each approximate variant on every dataset and problem instance shown. This will make the communication-round comparison direct and verifiable. revision: yes
Circularity Check
No significant circularity; modification to DANE is an independent practical extension
full rationale
The paper describes a modification to the prior DANE algorithm by substituting iterative approximate solves for exact local Newton subproblems. No equations, fitted parameters, or self-citation chains appear in the provided text that reduce any claimed result to its inputs by construction. The central claim rests on empirical comparison to SGD and the assumption that approximation error remains tolerable, which is an external modeling choice rather than a definitional or self-referential reduction. The derivation is therefore self-contained.
Assumptions & free parameters
assumptions (2)
- domain assumption The objective consists of the average of convex empirical risk functions.
- domain assumption Local data instances can be accessed only on their respective machines.
Cite this review
Pith. "Pith review of Practical Newton-Type Distributed Learning using Gradient Based Approximations." pith.science (2026). https://pith.science/paper/6YP7ANPU
@misc{pith2026190709562,
author = {Pith},
title = {Pith review of: Practical Newton-Type Distributed Learning using Gradient Based Approximations},
year = {2026},
howpublished = {\url{https://pith.science/paper/6YP7ANPU}},
note = {Machine review of arXiv:1907.09562}
}
read the original abstract
We study distributed algorithms for expected loss minimization where the datasets are large and have to be stored on different machines. Often we deal with minimizing the average of a set of convex functions where each function is the empirical risk of the corresponding part of the data. In the distributed setting where the individual data instances can be accessed only on the local machines, there would be a series of rounds of local computations followed by some communication among the machines. Since the cost of the communication is usually higher than the local machine computations, it is important to reduce it as much as possible. However, we should not allow this to make the computation too expensive to become a burden in practice. Using second-order methods could make the algorithms converge faster and decrease the amount of communication needed. There are some successful attempts in developing distributed second-order methods. Although these methods have shown fast convergence, their local computation is expensive and could enjoy more improvement for practical uses. In this study we modify an existing approach, DANE (Distributed Approximate NEwton), in order to improve the computational cost while maintaining the accuracy. We tackle this problem by using iterative methods for solving the local subproblems approximately instead of providing exact solutions for each round of communication. We study how using different iterative methods affect the behavior of the algorithm and try to provide an appropriate tradeoff between the amount of local computation and the required amount of communication. We demonstrate the practicality of our algorithm and compare it to the existing distributed gradient based methods such as SGD.
Reviewed May 24, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.