{"id":"c466ef13-57cd-4fe7-89d5-42a5b85cc5db","arxiv_id":"2505.00982","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"DHO2 distributes FOSI's Hessian eigenvector computation with model parallelism and an ADMM-like update, claiming lower per-GPU memory and faster time-to-solution in distributed DNN training.","lead":"The paper introduces DHO2, a distributed version of the FOSI hybrid optimizer that splits Lanczos curvature computation across GPUs and adds an ADMM-style update rule. It reports 1.4x to 2.0x faster training time and 4% to 5% higher accuracy versus distributed K-FAC, Shampoo, and S-SGD on image classification.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Distributed Lanczos pseudocode omits global HVP aggregation and full-vector memory, so the memory bound and curvature correctness are unsupported as written; a code-level check is needed.","rationale":"The reader's weakest_assumption identifies the same load-bearing issue: Algorithm 2 does not specify the global Hessian-vector-product aggregation, and the memory derivation omits full-length vectors. My read confirms this and adds the concrete consequence for the claimed memory bound. This is not a disagreement with the empirical speedups; code is provided, so the missing pieces are checkable. Because the reader already marked the paper CONDITIONAL and this concern is precisely the condition that needs verification, the verdict should remain unchanged rather than being strengthened to rejection.","tokens_in":12422,"tokens_out":7724,"duration_ms":81852,"concrete_test":"Run the released DHO2 code on a small model (e.g., a two-layer MLP on CIFAR-10) with C=1 and C=4, using the same random seed and data order. Compare the tridiagonal matrix B produced by Algorithm 2 against the serial Lanczos B from Algorithm 1, and instrument lines 5-7 to log the sizes of vi and h. If the distributed B differs from serial B beyond machine precision, or peak per-GPU memory does not track mn/C, the correctness and scalability claims fail.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central scalability claim depends on Algorithm 2 producing the same Lanczos tridiagonalization as serial FOSI with per-GPU memory O(mn/C + m^2 + n/C). This is not established by the pseudocode. In Algorithm 2, line 5 all-gathers the complete n-dimensional vector vi onto every GPU, and line 6 computes h = hvpt(vi) without specifying how per-GPU mini-batch Hessian-vector products are aggregated. If each GPU uses only its local mini-batch, B[i,i] = h·vi is not the Rayleigh quotient of the global objective Hessian, so the curvature update (and hence the FOSI step) is invalid. If a global aggregation is intended, it must appear explicitly. The memory analysis in Section III-C counts D[sc:ec,:] and h[sc:ec] as O(n/C), but omits the full vi materialized by the line-5 all-gather (and the full h if line 6 is global). Therefore the per-GPU peak memory is at least O(n + mn/C + m^2 + n/C), not O(mn/C + m^2 + n/C), and the claimed approximate linear memory reduction does not follow from the presented algorithm.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DHO2, a distributed implementation of the hybrid-order optimizer FOSI. The main ideas are (i) a model-parallel distributed Lanczos algorithm that computes the low-rank eigendecomposition of the Hessian while splitting the n x m basis matrix D across GPUs, and (ii) an ADMM-like model update rule that alternates between a Gauss-Seidel style parameter update and inner FOSI updates on an augmented Lagrangian. The authors claim an approximately linear reduction of per-GPU memory with the number of devices, sublinear time-to-solution, and 1.4x to 2.1x training-time speedups over distributed first- and second-order baselines, supported by experiments on CIFAR-10/100 and tiny-imagenet with up to 64 GPUs. The manuscript also contains a source-code link.","tokens_in":12733,"tokens_out":10487,"duration_ms":112144,"significance":"If the technical claims were fully supported, the paper would be a valuable contribution: it is, to the best of my knowledge, the first distributed implementation of FOSI, and the combination of a model-parallel Lanczos iteration with an ADMM-style outer loop is a plausible way to reduce the memory bottleneck of curvature-based optimizers. The experimental setup is nontrivial, with results on up to 64 GPUs and comparisons against K-FAC, Shampoo, and S-SGD, and the public code link is a strength. However, the current manuscript has several load-bearing gaps: the linear-algebra derivation in Eq. (7) is incorrect, the distributed Lanczos pseudocode omits the global Hessian-vector-product aggregation and materializes full-length vectors, the memory analysis ignores the broadcast of the full V-hat matrix, and the ADMM-like update rule is not derived correctly and has no convergence analysis. The practical significance is therefore conditional on a corrected algorithm and analysis.","major_comments":[{"comment":"The equation H_t = D B D^T is asserted to follow from D^T H_t D = B by inserting D D^T = I on both sides. This is incorrect: for the Lanczos matrix D in R^{n x m} with m < n, only D^T D = I_m holds, while D D^T is the orthogonal projector onto the m-dimensional Krylov subspace, not the identity. Consequently H_t = D B D^T is not an equality but a rank-m Rayleigh-Ritz approximation. The derivation of the extreme-eigenvalue extraction should be restated as a projection/approximation, since the current text claims an exact factorization that is false and is load-bearing for the curvature computation.","section":"Section II-B, Eq. (7)"},{"comment":"The distributed Lanczos pseudocode does not specify how the global Hessian-vector product hvpt(v_i) is formed. In the data-parallel setting each GPU has a different mini-batch, and the full-batch HVP is the average of the per-GPU HVPs; without an all-reduce over h (or an equivalent global aggregation), B[i,i] is not the Rayleigh quotient of the true Hessian and the tridiagonal matrix B is not the Lanczos reduction of H_t. The notation is also inconsistent: line 7 computes B[i,i] from h and v_i as if h were a full n-vector, while line 9 uses h[sc:ec] as a partial vector. If h is full, the claimed per-GPU memory is violated; if h is partial, line 7 needs an additional reduction. The pseudocode must be rewritten to make the distributed HVP and the vector layout explicit.","section":"Algorithm 2, Section III-A"},{"comment":"The claimed per-GPU peak memory O(mn/C + m^2 + n/C) omits at least three objects that the algorithm as written materializes: the full n-dimensional v_i created by the all-gather in line 5 of Algorithm 2, the full n-dimensional h if hvpt is applied globally, and the complete n x (k+l) matrix V-hat that Section III-A says is obtained by broadcasting every GPU's partial V-hat to all other GPUs. Including these terms gives a per-GPU memory of at least O(n + mn/C + m^2 + n(k+l)), so the central claim of approximately linear memory reduction with the device count does not follow from the presented algorithm. The authors need to either revise the algorithm to avoid these full-vector materializations or revise the memory bound to reflect them.","section":"Section III-C, memory analysis"},{"comment":"The ADMM-like update rule is not a correct ADMM derivation as written. The augmented Lagrangian term sigma/2 * ||w_a - w||^2 is placed in L1, but the resulting sigma(w_a - w) gradient term is absent from the Newton step in Eq. (13). In addition, the stopping condition in Eq. (14) reduces to ||(1+sigma)(g_l + pi)||^2 < epsilon, which is not the ADMM stationarity condition ||nabla f(w_a) + pi|| = 0. Finally, Algorithm 3 runs a fixed number P of inner updates and never tests the stated condition. The paper's claim that ADMM enhances convergence is therefore not supported by the derivation; either the update must be derived correctly or it must be explicitly presented as an ADMM-inspired heuristic with empirical justification.","section":"Section III-B, Eqs. (12)-(14)"},{"comment":"The speedup comparison in Table I is partly against baselines that are reported as INF for the target accuracy, so the headline speedup figures do not apply uniformly. D-Shampoo has a shorter per-epoch time than DHO2 on CIFAR-10/100 but fails to reach 90%/67%, while D-KFAC reaches the target on CIFAR-10 but not on CIFAR-100. The text should state precisely for which target accuracies the 1.4x-2.1x speedup claim holds, and should discuss the accuracy-time tradeoff rather than treating INF entries as a comparison point.","section":"Section IV-B, Table I"}],"minor_comments":[{"comment":"The abstract states a 1.4x to 2.1x speedup, while the contributions and conclusions state 1.4x to 2.0x; these numbers should be reconciled.","section":"Abstract vs. Conclusions"},{"comment":"The variable m used for the Gram-Schmidt all-reduce result shadows the Lanczos iteration count m; a different name, such as t or q, would avoid the ambiguity.","section":"Algorithm 2, line 9"},{"comment":"The text says the extra computations in Algorithm 3 lines 5, 13, and 14 are trivial and 'will be proved in Section IV-B,' but Section IV-B contains experiments, not a proof; this claim should be removed or replaced with a concrete complexity estimate.","section":"Section III-C"},{"comment":"The symbol g_l is used in the stopping condition without a definition; it should be defined as the gradient of the augmented Lagrangian at the l-th inner iteration.","section":"Section III-B, Eq. (14)"},{"comment":"Reference [15] (Automon) appears unrelated to the second-order optimizer discussion in which it is cited; please verify whether it is the intended citation.","section":"References"},{"comment":"It would be clearer to state whether the reported peak memory usage is measured from the implementation or computed from the analytical bound, and to include error bars or run-to-run variability if available.","section":"Figure 6"}],"recommendation":"major_revision","confidential_remarks":"The authors provide a source-code link, but I did not inspect the code. If the released implementation already performs a global all-reduce for the Hessian-vector product and avoids materializing the full V-hat on every GPU, then the major comments about Algorithm 2 and the memory bound may be resolvable by a careful rewrite rather than a change in the system. I would encourage the editor to request the code-level verification of these points, because the memory-scalability claim is the paper's core contribution and the current text does not support it. The ADMM-like update section will require more than a small patch, since the derivation is not a valid ADMM recursion as written."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper is an honest engineering attempt to distribute FOSI, a hybrid first/second-order optimizer, using model parallelism and an ADMM-like update. The authors report 1.4–2.0x speedups and meaningful memory reductions on up to 64 GPUs, and they released code. That is the real substance.\n\nWhat is genuinely new: the first distributed FOSI implementation, a distributed Lanczos algorithm for curvature estimation, and a heuristic ADMM-style inner loop. The building blocks are known, but the integration is not trivial and the experiments are reasonably extensive (CIFAR, tiny-ImageNet, multiple models, 8–64 GPUs). The paper is clearly written and the comparison against K-FAC and Shampoo is fair in spirit, though some baselines are used at suboptimal accuracy.\n\nNow the soft spots, in proportion. The distributed Lanczos pseudocode (Algorithm 2) is underspecified at a load-bearing point: line 6 says `h <- hvpt(vi)` but never states how per-GPU mini-batch Hessian-vector products are aggregated. If each GPU computes on its local batch only, the B matrix is not the Lanczos reduction of the global Hessian, and the curvature update is wrong. The all-gather on line 5 also materializes the full n-dimensional vi on every GPU, and a global hvpt would materialize full h, yet the memory analysis in Section III-C claims per-GPU memory of O(mn/C + m^2 + n/C), omitting that O(n) full-vector cost. So the headline memory-reduction claim is not supported by the algorithm as written. Equation (7) also asserts DD^T = I for an n x m matrix with m < n, which is false; only D^T D = I holds. That is a mathematical error, though in context it is clearly a slip in an already approximate eigendecomposition. The ADMM-like update rule is asserted to improve convergence with no proof, and the stopping condition in equation (14) looks garbled.\n\nNone of these flaws necessarily invalidate the empirical speedups — the code might well do the right thing — but they do mean the paper's analytical claims are not credible as they stand. A serious referee should ask for a corrected and complete Algorithm 2, a revised memory bound with all full-length vectors counted, and a fix for equation (7). The experimental section can stay mostly intact.\n\nWho gets value from this: practitioners working on distributed training with second-order methods, and systems folks interested in memory-efficient optimizer implementations. It deserves peer review because the engineering is real and reproducible, but it needs heavy revision in the theory sections. I would not desk-reject it.","headline":"A useful distributed training engineering contribution with real experiments, but the theory sections have load-bearing gaps that need fixing before the memory and correctness claims are accepted.","tokens_in":13210,"tokens_out":2050,"would_cite":true,"duration_ms":22955,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DHO2 splits curvature computation across GPUs to scale a hybrid-order optimizer, reporting near-linear per-device memory reduction and 1.4–2.1× faster training than distributed first- and second-order baselines.","keywords":["distributed training","hybrid order optimizer","Lanczos algorithm","ADMM","model parallelism","curvature information","memory reduction","scalability"],"falsifier":"On a small model (e.g., logistic regression), compute the full Hessian explicitly, run Algorithm 2 with C=2 GPUs, and compare the resulting B to the single-device Lanczos B; any discrepancy in the off-diagonal β entries indicates the distributed aggregation is incorrect. Also instrument peak per-GPU memory while increasing C; if it does not fall roughly as n/C, the central memory claim fails.","tokens_in":12233,"feed_emoji":"⚡","tokens_out":4094,"duration_ms":37751,"temperature":0.7,"pith_summary":"The paper proposes DHO2, a distributed implementation of the hybrid-order optimizer FOSI that combines gradient and curvature information in each update. Its goal is to make FOSI practical on multiple GPUs: a distributed Lanczos algorithm computes the needed Hessian eigenvectors across devices via model parallelism, so each GPU stores only a slice of the large basis matrix. An ADMM-like update rule splits the model update into first- and second-order subproblems to keep convergence fast. If the design works as claimed, training large networks can be accelerated with limited per-device memory, a key concern for small compute clusters.","feed_headline":"Per-GPU memory falls roughly linearly as GPU count climbs","feed_subtitle":"DHO2 splits curvature computation and uses ADMM-style updates to train 1.4–2.1× faster than existing distributed optimizers.","key_machinery":"The central mechanism is the distributed Lanczos algorithm (Algorithm 2), which shares the Lanczos basis matrix D row-wise across GPUs and uses all-reduce to compute the global Gram-Schmidt projections and norms. The second mechanism is the ADMM-like update rule (Algorithm 3), which introduces a consensus variable wa and a Lagrange multiplier π, then alternates a closed-form w-update with an inner loop that optimizes wa using the hybrid first/second-order step. These two parts together let curvature be computed in parallel while the model update is accelerated by ADMM's convergence properties.","core_discovery":"On its own terms, the paper claims that FOSI's memory bottleneck, the n×m Lanczos basis matrix, can be partitioned across C devices so that each GPU holds an n/C×m slice, and the resulting curvature information matches the single-device result. Coupled with an ADMM-derived update that alternates between a consensus variable and a locally optimized model, DHO2 reports per-GPU memory that decreases roughly linearly with C and total training time 1.4–2.1× shorter than distributed K-FAC and distributed Shampoo baselines, with a 4–5% accuracy gain.","pith_inferences":["The memory bound assumes full-length vectors are not materialized on every GPU, but the all-gather and Hessian-vector product calls in Algorithm 2 materialize full vectors; a stricter analysis would include that cost.","The ADMM-like split suggests a generic recipe: any hybrid optimizer that splits into two subspaces could be wrapped with an ADMM consensus variable; one could test DHO2's update rule with a pure first-order base to isolate ADMM's contribution.","With many GPUs, communication of the all-gather may dominate; a natural extension is hierarchical all-reduce or layer-wise partitioning rather than row-wise splitting.","The 1.4–2.1× speedup is measured against specific baselines and hyperparameters; the method's advantage might diminish on models where Hessian structure is less low-rank."],"forward_implications":["Per-GPU memory for the curvature basis drops from O(mn) to O(mn/C + m^2 + n/C), allowing larger models on fixed hardware.","Training time to reach a target accuracy is reduced by 1.4–2.1× relative to distributed K-FAC and Shampoo, and 20–40% relative to S-SGD in the reported settings.","The ADMM-like update contributes convergence speed: removing it (DHO2-WA) slows training and lowers final accuracy.","The framework scales sublinearly in time and memory on up to 32–64 GPUs, making second-order-style optimization accessible to resource-constrained users."],"supporting_citations":[{"why":"Defines FOSI, the hybrid-order optimizer that DHO2 distributes.","marker":"[16]"},{"why":"Provides Pearlmutter's algorithm for exact Hessian-vector products, the operator used inside the Lanczos iterations.","marker":"[28]"},{"why":"Supplies the original Lanczos algorithm that the distributed version extends.","marker":"[21]"},{"why":"Contributes the model-parallel technique used to split the basis matrix across GPUs.","marker":"[20]"},{"why":"Serves as a baseline distributed second-order optimizer (distributed K-FAC) for comparison.","marker":"[11]"},{"why":"Serves as a baseline distributed Shampoo implementation for comparison.","marker":"[12]"},{"why":"Provides the ADMM background and update rules that inspire the ADMM-like model update.","marker":"[32]"}],"fun_headline_variants":["DHO2 splits curvature, cuts per-GPU memory linearly with device count","Parallel curvature and ADMM updates give 1.4-2.1x speedup","Distributed hybrid optimizer: linear memory drop, faster training","Slicing Lanczos basis across GPUs yields linear memory savings"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The distributed Lanczos algorithm assumes each GPU computes the exact global Hessian-vector product by properly combining all GPUs' local results; if that aggregation is missing, the tridiagonal matrix is not the Lanczos reduction of the true Hessian.","fun_headline_variants_meta":{"raw":{"variants":["DHO2 splits curvature, cuts per-GPU memory linearly with device count","Parallel curvature and ADMM updates give 1.4-2.1x speedup","Distributed hybrid optimizer: linear memory drop, faster training","Slicing Lanczos basis across GPUs yields linear memory savings"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000246,"raw_usage":{"total_tokens":1502,"prompt_tokens":867,"completion_tokens":635,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":483,"completion_tokens_details":{"reasoning_tokens":555}},"tokens_in":483,"tokens_out":635,"duration_ms":6092,"temperature":1.0,"reasoning_tokens":555,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:30:43.233936+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a small model (e.g., logistic regression), compute the full Hessian explicitly, run Algorithm 2 with C=2 GPUs, and compare the resulting B to the single-device Lanczos B; any discrepancy in the off-diagonal β entries indicates the distributed aggregation is incorrect. Also instrument peak per-GPU memory while increasing C; if it does not fall roughly as n/C, the central memory claim fails.","supporting_citations":[{"cited_title":"Fosi: Hybrid first and second order optimization,","cited_arxiv_id":null,"evidence_quote":"Defines FOSI, the hybrid-order optimizer that DHO2 distributes."},{"cited_title":"Fast exact multiplication by the hessian,","cited_arxiv_id":null,"evidence_quote":"Provides Pearlmutter's algorithm for exact Hessian-vector products, the operator used inside the Lanczos iterations."},{"cited_title":"An iteration method for the solution of the eigenvalue problem of linear differential and integral operators,","cited_arxiv_id":null,"evidence_quote":"Supplies the original Lanczos algorithm that the distributed version extends."},{"cited_title":"Deep neural network training with distributed k-fac,","cited_arxiv_id":null,"evidence_quote":"Serves as a baseline distributed second-order optimizer (distributed K-FAC) for comparison."}],"review_version":1}