Improving HISQ propagator solves using deflation

Carleton DeTar; Evan Weinberg; Leon Hostetler; M. A. Clark; Steven Gottlieb

Improving HISQ propagator solves using deflation

T0 review · 0 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Deflating the low eigenmodes of the HISQ operator cuts light-quark propagator solve time 10-fold at the physical mass on a 144^3 x 288 lattice.

desk verdict Solid LATTICE proceedings benchmark showing a real 10x solve-time speedup for deflated HISQ propagators at physical light quark mass, with the caveat that setup costs are explicitly excluded. read the letter →

arxiv 2502.00152 v1 pith:GG5RBPAM submitted 2025-01-31 hep-lat physics.comp-ph

Leon Hostetler , M. A. Clark , Carleton DeTar , Steven Gottlieb , Evan Weinberg This is my paper · ORCID

classification hep-latphysics.comp-ph

keywords latticeQCDHISQfermionsdeflationconjugategradientcriticalslowingdowneigenvaluecomputationGPUsolverpropagatorsolve

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that deflation—using the low eigenmodes of the Dirac operator as an initial guess for the iterative solver—removes most of the critical slowing down that makes light-quark propagator solves expensive. At the physical light quark mass on a $144^3 \times 288$ lattice, deflated conjugate gradient with 2048 eigenvectors took 4.79 s per solve versus 47.8 s for conventional even-odd preconditioned CG on the same 384 nodes, a 10x speedup. This matters because propagator solves dominate the cost of lattice QCD calculations, and multigrid methods for staggered fermions have been slow to mature. The paper also claims that single-precision eigenvectors with periodic re-deflation match double-precision convergence while halving memory and disk footprint. The authors offer this as a baseline for comparing deflation with multigrid on current large lattices.

What carries the argument

The machinery is deflated conjugate gradient on the normal equations. Eigenvectors $v_i$ of the massless squared Dirac operator $\not{D}^{\dagger} \not{D}$ are projected onto the source to form the initial guess $x = \sum_i v_i \frac{1}{\lambda_i} v_i^{\dagger} \eta$, so the CG solver works only on the high-mode part of the solution. Multi-deflation repeats this projection during the solve when the residual drops by a factor of $1/\delta_{tr}$, which keeps single-precision eigenvectors from stalling the residual; the re-deflation threshold was chosen empirically.

What would settle it

Run a production workflow at the physical light quark mass on the $144^3 \times 288$ lattice that computes a realistic number of propagators per configuration, and record total wall-clock time including eigenvector generation, disk read, and memory allocation; if the deflated total is not below undeflated CG, the amortization premise fails. A specific crossover test is to find the number of solves per configuration at which the two totals are equal.

Watch

Extended reading notes

Core claim

The central claim is a measured factor-of-10 reduction in solve time on the largest lattice in the study: with 2048 eigenvectors of the massless squared Dirac operator, the average time to solve the HISQ propagator equation $M^{\dagger} M \psi = M^{\dagger} \eta$ at the physical light quark mass on a $144^3 \times 288$ lattice fell from 47.8 s to 4.79 s, and the iteration count fell from 33,003 to 2,909. A second claim is that these eigenvectors need not be stored in double precision: if the solve is periodically re-deflated with single-precision vectors whenever the residual drops by a chosen factor, convergence matches the double-precision case while using half the memory and disk space. The paper presents this as evidence that deflation is a practical alternative to multigrid for HISQ propagator solves at current lattice sizes, even though deflation shifts, rather than eliminates, critical slowing down.

Load-bearing premise

The 10x speedup is measured for the solve only, because the paper excludes eigensolve, disk I/O, and memory allocation from its timings, relying on those costs being amortized over many propagator solves per configuration; if a workflow computes only a few propagators per configuration, the gain may vanish or reverse.

Editorial extensions

If this is right

On the $144^3 \times 288$ lattice at the physical light quark mass, deflated CG with 2048 eigenvectors gives a 10x solve-time speedup over conventional even-odd preconditioned CG when both runs use the same resources.
The method is also more cost-efficient per solve: 1839 node-seconds per solve with deflation versus 7914 node-seconds for undeflated CG run on fewer nodes at the light quark mass.
Single-precision eigenvectors with periodic re-deflation converge like double-precision ones, halving the memory and disk footprint, which lets the deflated problem run on half as many nodes.
At the strange quark mass the deflated solve is typically slower than undeflated CG; the speedup appears only near the physical light quark mass.
The measured factor of 10 matches a recent multigrid result for the same lattice size and quark mass, giving a direct performance comparison between the two approaches for HISQ.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A practical consequence not tested here is that the crossover number of solves per configuration, not the solve-time ratio, determines whether deflation pays off; reporting that crossover would let production teams decide when to use the method.
The empirical choice of the re-deflation threshold suggests a testable extension: a rule relating $\delta_{tr}$ to the precision of the stored eigenvectors could make the method self-tuning.
The paper notes a performance regression on one of the two large supercomputers used, so the 10x figure may not transfer automatically to every architecture; reproducing the timing on another GPU cluster would test how much of the gain is machine-dependent.
If batched multi-right-hand-side solves and eigenvector compression mature as outlined in the outlook, the amortized setup cost could drop enough to make deflation attractive for moderate numbers of propagator solves as well.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know two things about this one. First, it delivers a clean, internally consistent solve-time benchmark: on the 144^3 x 288 lattice at the physical light quark mass, deflated CG with 2048 eigenvectors cuts the average solve time from 47.8 s to 4.79 s, a factor of 10, with iteration counts dropping from 33,003 to 2,909. Second, the headline 10x is a solve-time figure only; the eigensolve, disk I/O, and memory allocation are excluded, and the paper says so directly. That is the main soft spot, but it is a disclosed scope decision rather than an oversight.

What is genuinely new and useful: this is the first deflated HISQ benchmark on a lattice this large and at the physical light quark mass, and the multi-deflation scheme with single-precision eigenvectors is a practical memory hack that works. The paper shows the stalling problem with sloppy eigenvectors and demonstrates that periodic re-deflation fixes it, halving the memory footprint while keeping performance close to double precision. That is a real contribution for practitioners who want to use deflation on memory-bound machines.

The data are handled honestly. They fixed the stopping criterion, reported iteration counts and wall times, separated solve time from setup time, and disclosed the Frontier regression that forced them to switch to Perlmutter. The node-seconds analysis is a good-faith attempt to address the fixed-resource criticism, and it still shows deflation winning by a factor of four in cost efficiency even when undeflated CG runs on the minimum nodes it needs.

Where I'd push back on the stress-test note: yes, the crossover point is unquantified, but the paper's stated target is the common workflow of many propagator solves per configuration, and within that scope the result is valid. The bigger real weaknesses are that each ensemble uses a single configuration, so there are no error bars on the speedup, and the empirically tuned re-deflation trigger delta_tr is not fully reported across lattice sizes. Those are worth fixing if this goes beyond proceedings.

Overall, this is a well-scoped performance study, not a deep algorithmic derivation. The math is standard, the measurements are reproducible in principle, and the claims match what was measured. It deserves a serious referee, and I would accept it with minor revisions. For anyone working on HISQ solvers or choosing between deflation and multigrid, it is worth a careful read.

Referee Report

0 major / 6 minor

Summary. This LATTICE2024 proceedings paper reports an empirical comparison of deflated versus undeflated even-odd preconditioned CG for HISQ propagator solves. Using the MILC and QUDA libraries, the authors study four ensembles at fixed physical volume (5.76 fm) with lattice spacings from 0.12 to 0.04 fm and quark masses ranging from the strange mass to the physical light mass. Section 3 introduces a multi-deflation scheme in which single-precision eigenvectors are used and the system is periodically re-deflated, triggered when the residual drops by a factor of 1/delta_tr, in order to avoid the stalling observed with sloppy eigenvectors. The central result, in Table 2 and Fig. 3, is that on the 144^3 x 288 lattice at the physical light quark mass, 2048 deflating eigenvectors reduce the average solve time from 47.8 s to 4.79 s, a factor of 10, and reduce the average CG iteration count from 33,003 to 2,909, with a true-residual stopping criterion of |r|/|b| < 1e-8. The paper also reports node-second cost comparisons and openly discloses a Frontier performance regression that led them to use Perlmutter for the final results. All reported solve times exclude eigensolve, eigenvector I/O, and memory allocation, with the stated assumption that the eigensolve cost is amortized over many propagator solves per configuration.

Significance. If the reported solve-time speedups hold, the paper fills a clear gap in the literature by providing HISQ deflation performance data at the largest lattice sizes currently in use, and it offers a useful baseline for future comparisons with HISQ multigrid. The paper is careful in several respects: stopping criteria are fixed, times are averaged over multiple solves with the first (setup-heavy) solve excluded, the Frontier regression is disclosed, and the node-second analysis addresses part of the fixed-resource criticism. The main limitation is that the end-to-end practical speedup depends on an unmeasured eigensolve and I/O cost; the paper is transparent about this scope, but the phrase 'time to solution' in Section 4 overstates what is actually measured. In my view this is a local wording and framing issue rather than an internal inconsistency, and it does not invalidate the scoped claim that deflation speeds up the iterative solve phase.

minor comments (6)

[Section 4] The sentence 'We show a factor of 10 speedup of the time to solution on the 144^3 x 288 lattice at the physical light quark mass' should be reworded to 'solve time' or should explicitly repeat the caveat that eigensolve, I/O, and memory allocation are excluded; as written, 'time to solution' conflicts with the definition used in Section 3 and Fig. 3.
[Section 3, Table 2] The text indicates that Table 2 tests multi-deflation with single-precision eigenvectors, but the table caption does not state the eigenvector precision or the delta_tr values used for the periodic re-deflation; please state in the caption that all deflated runs use single-precision eigenvectors and report the delta_tr values or note that they were tuned separately for each lattice size.
[Section 3, Fig. 2] In the text describing the right panel, the authors refer to 'the blue residual curve,' but neither the figure nor its caption identifies which curve is blue; please add a legend or explicitly state the correspondence so that the claim about delta_tr = 10^-3 is unambiguous.
[Section 1] The statement 'Because of the V^2 cost of deflation' would benefit from a one-sentence explanation that the number of eigenvectors scales with volume and each deflation application is O(V), so that readers do not conflate eigensolve scaling with the per-solve deflation cost.
[Section 3] Given that the 2048 single-precision single-parity eigenvectors for the 144^3 x 288 lattice are stated to be about 20 TB, a sentence on how the PARTFILE format is used in the timing runs, and whether disk-read time is included in any reported number, would help readers assess the I/O component of the setup that is excluded from the solve times.
[Section 4] The comparison with the multigrid result of Ref. [14] should state explicitly whether the cited 10x speedup is also a solve-time-only result with the same stopping criterion; otherwise the sentence 'This speedup is comparable to...' invites an end-to-end comparison that the present data do not support.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the 10x speedup is a direct measured benchmark with the eigensolve cost explicitly excluded and the delta_tr tuning parameter disclosed as an empirical choice.

full rationale

The paper's central claim is an empirical benchmark: on the 144^3 x 288 ensemble at the physical light quark mass, deflated CG with 2048 eigenvectors required 2,909 iterations and 4.79 s average solve time versus 33,003 iterations and 47.8 s for undeflated CG (Table 2 and Fig. 3). The comparison is a direct measurement using the MILC and QUDA codes, and the paper explicitly reports the resource-use caveat and node-seconds efficiency for the undeflated case. The only tunable input, delta_tr, is selected empirically to match double-precision iteration counts; this is disclosed as an empirical choice, and it is not used to generate a prediction from a fitted model. The amortization assumption that eigensolve cost is negligible when many solves are done per configuration is stated explicitly in Section 2, and it does not hide the setup cost from the wall-clock comparison; it simply defines the reported metric as solve time. Citations to QUDA and to earlier deflation papers are either software references, implementation details such as single-parity storage cited to Ref. [3], or non-load-bearing comparisons like the MG 10x speedup in Ref. [14]; none substitutes for a derivation that reduces to the claimed result. No circular step can be exhibited, so the circularity score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

No new physical entities are introduced. The central empirical claims rest on standard deflation theory plus two practical assumptions: amortized setup and single-configuration representativeness. The main hand-set performance parameters are the re-deflation trigger and the number of eigenvectors.

free parameters (2)

re-deflation trigger delta_tr (tol_restart) = 1e-3 for the 64^3x96 example; values for other lattices not reported
Controls when re-deflation occurs in the single-precision multi-deflation scheme; selected empirically and stated to vary with lattice size.
number of deflating eigenvectors N = 1024 or 2048
User-selected resource knob; the measured speedup depends strongly on N, with 5.5x at 1024 EVs versus 10x at 2048 EVs on the 144^3x288 lattice.

assumptions (4)

domain assumption The low eigenvectors of the massless squared Dirac operator D^dagger D form an effective deflation subspace for the massive squared operator M^dagger M.
The deflation step, Eq. (4), projects these eigenvectors onto the source; the paper does not demonstrate that this subspace suppresses the low modes of the massive operator for all masses studied.
domain assumption The eigensolve and eigenvector loading costs are negligible when amortized over many solves per configuration.
Explicitly assumed in Section 2; the reported 10x speedup covers solve time only.
domain assumption One gauge configuration per ensemble and five timed solves are sufficient to characterize solver performance.
Table 1 reports a single configuration per lattice; no configuration-to-configuration variability or error bars are provided.
domain assumption The QUDA and MILC implementations on Perlmutter A100 GPUs are representative of production HISQ workflows.
All timing results come from these specific software and hardware stacks; representativeness is not independently verified.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Improving HISQ propagator solves using deflation." pith.science (2026). https://pith.science/paper/GG5RBPAM

@misc{pith2026250200152,
  author       = {Pith},
  title        = {Pith review of: Improving HISQ propagator solves using deflation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GG5RBPAM}},
  note         = {Machine review of arXiv:2502.00152}
}

abstract

Typically, the conjugate gradient (CG) algorithm employs mixed precision and even-odd preconditioning to compute propagators for highly improved staggered quarks (HISQ). This approach suffers from critical slowing down as the light quark mass is decreased to its physical value. Multigrid is one alternative to combat critical slowing down; however, it involves setup costs that are not always easy to amortize. We consider deflation, which can also remove critical slowing down, but incurs its own setup cost to compute eigenvectors. Results using the MILC and QUDA software libraries to generate eigenvectors and to perform deflated solves on lattices up to $144^3 \times 288$ (with lattice spacing 0.04 fm) and with a range of quark masses from the physical strange down to the physical light quark values will be presented. We compare with CG and comment on deflation versus multigrid.

Figures

Figures reproduced from arXiv: 2502.00152 by the authors.

**Figure 1.** (Left) The number of CG iterations required to reach the solution versus the bare quark mass for several lattice sizes. (Right) The smallest eigenvalues of the squared massless Dirac operator 𝐷/ †𝐷/ for several lattice sizes. The ensembles approach the continuum limit in lattice spacing at fixed physical volume. with the deflation and CG offloaded to QUDA’s solver. In MILC, even-odd preconditioning (Schur decomposit… view at source ↗

**Figure 2.** (Left) The iterated residual versus CG iteration number for (a single) deflation on a 643 × 96 lattice with 2048 eigenvectors and at a lighter-than-physical quark mass of 0.000569. The black curve shows the result when using double precision eigenvectors, and the red curve shows the result when using single precision eigenvectors. In the double precision case, the residual reaches 10−8 in 708 CG iterations, whereas … view at source ↗

**Figure 3.** (Left) CG iterations versus bare quark mass for the 1443 × 288 system running on 1536 GPUs on Perlmutter. At the physical light quark mass 𝑎𝑚ℓ, 𝑝ℎ𝑦𝑠, the undeflated case required on average 33,003 iterations per solve versus 2,909 for deflation with 2K eigenvectors. This implies a potential speedup of ∼ 11× for the deflated case. (Right) Solve time versus bare quark mass for the same cases. At 𝑎𝑚ℓ, 𝑝ℎ𝑦𝑠, the average… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 17 canonical work pages

[1]

de Forcrand,Progress on lattice qcd algorithms,Nuclear Physics B - Proceedings Supplements 47(1996) 228

P. de Forcrand,Progress on lattice qcd algorithms,Nuclear Physics B - Proceedings Supplements 47(1996) 228

work page 1996
[2]

Computing and deflating eigenvalues while solving multiple right hand side linear systems in Quantum Chromodynamics

A. Stathopoulos and K. Orginos,Computing and deflating eigenvalues while solving multiple right-hand side linear systems with an application to quantum chromodynamics,SIAM Journal on Scientific Computing32(2010) 439 [0707.0131]. 8 Improving HISQ propagator solves using deflation Leon Hostetler

work page Pith review arXiv 2010
[3]

Numerical experiments using deflation with the HISQ action

C. Davies, C. DeTar, C. McNeile and A. Vaquero,Numerical experiments using deflation with the hisq action, EPJ Web of Conferences175 (2018) 14016 [1710.07219]

work page Pith review arXiv 2018
[4]

Romero, A

E. Romero, A. Stathopoulos and K. Orginos,Multigrid deflation for lattice qcd, Journal of Computational Physics409 (2020) 109356

work page 2020
[5]

Brannick, R.C

J. Brannick, R.C. Brower, M.A. Clark, J.C. Osborn and C. Rebbi,Adaptive multigrid algorithm for lattice qcd,Phys. Rev. Lett.100 (2008) 041601

work page 2008
[6]

Babich, J

R. Babich, J. Brannick, R.C. Brower, M.A. Clark, T.A. Manteuffel, S.F. McCormick et al., Adaptive multigrid algorithm for the lattice wilson-dirac operator,Phys. Rev. Lett.105 (2010) 201602

work page 2010
[7]

Osborn, R

J.C. Osborn, R. Babich, J. Brannick, R.C. Brower, M.A. Clark, S.D. Cohen et al.,Multigrid solver for clover fermions, LATTICE2010, 2010, https://arxiv.org/abs/1011.2775 [1011.2775]

arXiv 2010
[8]

Frommer, K

A. Frommer, K. Kahl, S. Krieg, B. Leder and M. Rottmann,Adaptive aggregation-based domain decomposition multigrid for the lattice wilson–dirac operator, SIAM Journal on Scientific Computing36 (2014) A1581 [1303.1377]

arXiv 2014

Show all 20 references

[9]

Richtmann, N

D. Richtmann, N. Meyer and T. Wettig,Mrhs multigrid solver for wilson-clover fermions, LATTICE2022, 2022, https://arxiv.org/abs/2211.13719 [2211.13719]

2022 arXiv
[10]

Cohen, R.C

S.D. Cohen, R.C. Brower, M.A. Clark and J.C. Osborn,Multigrid algorithms for domain-wall fermions, LATTICE2012, 2012, https://arxiv.org/abs/1205.2933 [1205.2933]

2012 arXiv
[11]

Boyle,Hierarchically deflated conjugate gradient, 2014

P.A. Boyle,Hierarchically deflated conjugate gradient, 2014

2014
[12]

Brower, M.A

R.C. Brower, M.A. Clark, E. Weinberg and D. Howarth,Multigrid for chiral lattice fermions: Domain wall, Phys. Rev. D102 (2020) 094517

2020
[13]

Brower, E

R.C. Brower, E. Weinberg, M.A. Clark and A. Strelchenko,Multigrid algorithm for staggered lattice fermions,Phys. Rev. D97 (2018) 114513

2018
[14]

Ayyar, E

V. Ayyar, E. Weinberg, R.C. Brower, M. Clark and M. Wagner,Optimizing staggered multigrid for exascale performance, inProceedings of The 39th International Symposium on Lattice Field Theory — PoS(LATTICE2022), LATTICE2022, Sissa Medialab, Jan., 2023, DOI [2212.12559]

2023 arXiv
[15]

Clark, R

M. Clark, R. Babich, K. Barros, R. Brower and C. Rebbi,Solving lattice qcd systems of equations using mixed precision solvers on gpus, Computer Physics Communications181 (2010) 1517–1528

2010
[16]

Babich, M.A

R. Babich, M.A. Clark, B. Joó, G. Shi, R.C. Brower and S. Gottlieb,Scaling lattice qcd beyond 100 gpus, inProceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis, SC ’11, p. 1–11, ACM, Nov., 2011, DOI [1109.2935]. 9 Improvi...

2011 arXiv
[17]

Stathopoulos, A.M

A. Stathopoulos, A.M. Abdel-Rehim and K. Orginos,Deflation for inversion with multiple right-hand sides in qcd, Journal of Physics: Conference Series180 (2009) 012073

2009
[18]

Clark,Rearchitecting quda for multi-rhs computations, LATTICE2024, 2024

K. Clark,Rearchitecting quda for multi-rhs computations, LATTICE2024, 2024

2024
[19]

Weinberg,Quda-accelerated batched solvers for lqcd workflows, LATTICE2024, 2024

E. Weinberg,Quda-accelerated batched solvers for lqcd workflows, LATTICE2024, 2024

2024
[20]

Clark, C

M.A. Clark, C. Jung and C. Lehner,Multi-grid lanczos, EPJ Web of Conferences175(2018) 14023. 10

2018

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.

[1] [1]

de Forcrand,Progress on lattice qcd algorithms,Nuclear Physics B - Proceedings Supplements 47(1996) 228

P. de Forcrand,Progress on lattice qcd algorithms,Nuclear Physics B - Proceedings Supplements 47(1996) 228

work page 1996

[2] [2]

Computing and deflating eigenvalues while solving multiple right hand side linear systems in Quantum Chromodynamics

A. Stathopoulos and K. Orginos,Computing and deflating eigenvalues while solving multiple right-hand side linear systems with an application to quantum chromodynamics,SIAM Journal on Scientific Computing32(2010) 439 [0707.0131]. 8 Improving HISQ propagator solves using deflation Leon Hostetler

work page Pith review arXiv 2010

[3] [3]

Numerical experiments using deflation with the HISQ action

C. Davies, C. DeTar, C. McNeile and A. Vaquero,Numerical experiments using deflation with the hisq action, EPJ Web of Conferences175 (2018) 14016 [1710.07219]

work page Pith review arXiv 2018

[4] [4]

Romero, A

E. Romero, A. Stathopoulos and K. Orginos,Multigrid deflation for lattice qcd, Journal of Computational Physics409 (2020) 109356

work page 2020

[5] [5]

Brannick, R.C

J. Brannick, R.C. Brower, M.A. Clark, J.C. Osborn and C. Rebbi,Adaptive multigrid algorithm for lattice qcd,Phys. Rev. Lett.100 (2008) 041601

work page 2008

[6] [6]

Babich, J

R. Babich, J. Brannick, R.C. Brower, M.A. Clark, T.A. Manteuffel, S.F. McCormick et al., Adaptive multigrid algorithm for the lattice wilson-dirac operator,Phys. Rev. Lett.105 (2010) 201602

work page 2010

[7] [7]

Osborn, R

J.C. Osborn, R. Babich, J. Brannick, R.C. Brower, M.A. Clark, S.D. Cohen et al.,Multigrid solver for clover fermions, LATTICE2010, 2010, https://arxiv.org/abs/1011.2775 [1011.2775]

arXiv 2010

[8] [8]

Frommer, K

A. Frommer, K. Kahl, S. Krieg, B. Leder and M. Rottmann,Adaptive aggregation-based domain decomposition multigrid for the lattice wilson–dirac operator, SIAM Journal on Scientific Computing36 (2014) A1581 [1303.1377]

arXiv 2014