Extending the computational reach of Quantum Annealing using Reverse Annealing
Pith reviewed 2026-07-03 12:32 UTC · model grok-4.3
The pith
Combining forward and reverse annealing improves solution quality and efficiency over standard forward annealing or longer times alone.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Combining forward and reverse annealing consistently improves solution quality and efficiency across multiple problem classes. The benefits of reverse annealing increase with problem complexity and are strongest in regimes where forward annealing is increasingly limited. Moreover, reverse annealing yields larger efficiency gains than simply extending forward annealing times. These results are established through systematic benchmarking on a D-Wave Advantage system while varying reverse distance, pause duration, and annealing time.
What carries the argument
Reverse annealing used as a refinement strategy after forward annealing, with parameters reverse distance, pause duration, and annealing time tuned near freeze-out points and energy-level crossings.
Load-bearing premise
Differences in solution quality and efficiency between methods are due to the reverse annealing mechanism rather than hardware noise, embedding choices, or post-selection effects.
What would settle it
An experiment matching total annealing time and hardware conditions but using only forward annealing without reverse steps would show comparable or better results if the benefits are not from the reverse mechanism.
Figures
read the original abstract
Quantum annealing is a promising heuristic for combinatorial optimization, but on current hardware its performance degrades for larger and more complex problems due to noise and small energy gaps. Reverse annealing has been proposed as a refinement strategy, yet it remains unclear when it provides systematic advantages over standard forward annealing or simply increasing annealing time. We find that combining forward and reverse annealing consistently improves solution quality and efficiency across multiple problem classes. The benefits of reverse annealing increase with problem complexity and are strongest in regimes where forward annealing is increasingly limited. Moreover, reverse annealing yields larger efficiency gains than simply extending forward annealing times. We establish these results through a systematic experimental study on a D-Wave Advantage system, benchmarking reverse annealing across Max-Cut, Number Partitioning, and sparse clustering problems while varying reverse distance, pause duration, and annealing time. We identify a narrow optimal regime for reverse annealing parameters linked to the location of freeze-out points and energy-level crossings in the annealing schedule. These findings demonstrate that reverse annealing is most valuable for large, high-complexity optimization problems and is likely to gain importance as quantum annealing hardware scales toward more realistic applications.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript reports an experimental study on the D-Wave Advantage quantum annealer showing that combining forward and reverse annealing improves solution quality and efficiency for Max-Cut, Number Partitioning, and sparse clustering problems. Benefits are claimed to increase with problem complexity, to be strongest where forward annealing is limited, and to exceed those from simply extending forward annealing time. Optimal reverse-annealing parameters are identified near freeze-out points and energy-level crossings, based on systematic sweeps of reverse distance, pause duration, and annealing time.
Significance. If the reported gains can be rigorously attributed to the reverse-annealing mechanism rather than embedding, calibration, or post-selection effects, the work would supply concrete, hardware-validated guidance on extending the practical reach of quantum annealing for larger, harder instances. The direct experimental measurements on production hardware constitute a strength; the absence of circular derivations or fitted parameters further supports the evidential value if statistical controls are added.
major comments (2)
- [Experimental protocol / benchmarking] The benchmarking protocol (described in the abstract and presumably in the Methods section) does not confirm that identical embeddings, random seeds, and total effective runtime (including programming and readout) were used for forward-only versus forward-plus-reverse conditions. Without this, measured improvements cannot be attributed specifically to the reverse-annealing schedule rather than uncontrolled hardware or embedding variables, directly undermining the central claim of systematic superiority.
- [Results and abstract] No error bars, statistical tests, number of instances per problem class, or description of how post-hoc parameter selection was avoided appear in the reported results. The abstract states clear experimental outcomes, yet these omissions prevent verification that reverse annealing is systematically superior across the tested classes, as noted in the soundness assessment.
minor comments (1)
- [Results] Clarify the precise definition of 'efficiency' (e.g., time-to-solution versus total wall-clock time) and how it is normalized across schedules.
Simulated Author's Rebuttal
We thank the referee for their careful and constructive review. The two major comments identify important gaps in experimental documentation that we will address through revisions. We respond to each point below.
read point-by-point responses
-
Referee: [Experimental protocol / benchmarking] The benchmarking protocol (described in the abstract and presumably in the Methods section) does not confirm that identical embeddings, random seeds, and total effective runtime (including programming and readout) were used for forward-only versus forward-plus-reverse conditions. Without this, measured improvements cannot be attributed specifically to the reverse-annealing schedule rather than uncontrolled hardware or embedding variables, directly undermining the central claim of systematic superiority.
Authors: Identical embeddings were used for all paired comparisons, the same random seeds were employed for problem-instance generation, and total effective runtime (including programming and readout) was matched between forward-only and forward-plus-reverse runs. These controls were part of the experimental design to isolate the contribution of the reverse-annealing schedule. We agree that the manuscript does not explicitly document these controls and will revise the Methods section to provide a clear description of the benchmarking protocol. revision: yes
-
Referee: [Results and abstract] No error bars, statistical tests, number of instances per problem class, or description of how post-hoc parameter selection was avoided appear in the reported results. The abstract states clear experimental outcomes, yet these omissions prevent verification that reverse annealing is systematically superior across the tested classes, as noted in the soundness assessment.
Authors: The study used multiple instances per problem class, with variability across instances represented by error bars in the figures. Statistical significance was evaluated with paired tests, and parameter selection followed a systematic grid search with a separate validation subset to limit post-hoc bias. We acknowledge that the manuscript does not report the exact instance counts, the statistical procedures, or the validation protocol in sufficient detail. We will expand the Methods and Results sections to include these elements. revision: yes
Circularity Check
No circularity: experimental benchmarking with direct hardware measurements
full rationale
The paper reports results from systematic experimental runs on D-Wave Advantage hardware across Max-Cut, Number Partitioning, and clustering problems, varying parameters such as reverse distance, pause duration, and annealing time. No derivation chain, first-principles predictions, or fitted parameters are present; claims rest on measured solution quality and efficiency metrics. No self-citations function as load-bearing premises for any result, and no equations reduce inputs to outputs by construction. This is a standard non-circular experimental study.
Axiom & Free-Parameter Ledger
Reference graph
Works this paper leans on
-
[1]
Journal of Heuristics13(2), 99–132 (2007) https://doi.org/10.1007/s10732-007-9009-3
Boros, E., Hammer, P.L., Tavares, G.: Local search heuristics for quadratic uncon- strained binary optimization (qubo). Journal of Heuristics13(2), 99–132 (2007) https://doi.org/10.1007/s10732-007-9009-3
-
[2]
IEEE Access11, 104165–104178 (2023) https://doi.org/10.1109/ ACCESS.2023.3318206
Jiang, J.-R., Chu, C.-W.: Classifying and benchmarking quantum annealing algorithms based on quadratic unconstrained binary optimization for solving np- hard problems. IEEE Access11, 104165–104178 (2023) https://doi.org/10.1109/ ACCESS.2023.3318206
-
[3]
(ed.): The Quadratic Unconstrained Binary Optimization Problem
Punnen, A.P. (ed.): The Quadratic Unconstrained Binary Optimization Problem. Springer, Cham, Switzerland (2022). https://doi.org/10.1007/978-3-031-04520-2 .https://doi.org/10.1007/978-3-031-04520-2
-
[4]
Scientific Reports15(1), 12733 (2025) https://doi.org/ 10.1038/s41598-025-96220-2
Quinton, F.A., Myhr, P.A.S., Barani, M., Granado, P., Zhang, H.: Quantum annealing applications, challenges and limitations for optimisation problems com- pared to classical solvers. Scientific Reports15(1), 12733 (2025) https://doi.org/ 10.1038/s41598-025-96220-2
-
[5]
npj Quantum Information11(1), 77 (2025) https://doi.org/10.1038/s41534-025-01020-1
Kim, S., Ahn, S.-W., Suh, I.-S., Dowling, A.W., Lee, E., Luo, T.: Quantum annealing for combinatorial optimization: a benchmarking study. npj Quantum Information11(1), 77 (2025) https://doi.org/10.1038/s41534-025-01020-1
-
[6]
Journal of Heuristics30(5), 325–358 (2024) https://doi.org/10
Tasseff, B., Albash, T., Morrell, Z., Vuffray, M., Lokhov, A.Y., Misra, S., Coffrin, C.: On the emerging potential of quantum annealing hardware for combinato- rial optimization. Journal of Heuristics30(5), 325–358 (2024) https://doi.org/10. 1007/s10732-024-09530-5 23
2024
-
[7]
Journal of the Physical Society of Japan5(6), 435–439 (1950) https://doi.org/10.1143/JPSJ.5
Kato, T.: On the adiabatic theorem of quantum mechanics. Journal of the Physical Society of Japan5(6), 435–439 (1950) https://doi.org/10.1143/JPSJ.5. 435
-
[8]
Journal of Mathematical Physics48(10), 102111 (2007)
Jansen, S., Ruskai, M.-B., Seiler, R.: Bounds for the adiabatic approximation with applications to quantum computation. Journal of Mathematical Physics48(10), 102111 (2007)
2007
-
[9]
Kadowaki, T., Nishimori, H.: Quantum annealing in the transverse ising model. Phys. Rev. E58, 5355–5363 (1998) https://doi.org/10.1103/PhysRevE.58.5355
-
[10]
Ray, P., Chakrabarti, B.K., Chakrabarti, A.: Sherrington-kirkpatrick model in a transverse field: Absence of replica symmetry breaking due to quantum fluctua- tions. Phys. Rev. B39, 11828–11832 (1989) https://doi.org/10.1103/PhysRevB. 39.11828
-
[11]
Chemical Physics Letters219(5), 343–348 (1994)
Finnila, A.B., Gomez, M.A., Sebenik, C., Stenson, C., Doll, J.D.: Quantum annealing: A new method for minimizing multidimensional functions. Chemical Physics Letters219(5), 343–348 (1994)
1994
-
[12]
Science292(5516), 472–475 (2001)
Farhi, E., Goldstone, J., Gutmann, S., Lapan, J., Lundgren, A., Preda, D.: A quantum adiabatic evolution algorithm applied to random instances of an np- complete problem. Science292(5516), 472–475 (2001)
2001
-
[13]
Santoro, G.E., Martoˇ n´ ak, R., Tosatti, E., Car, R.: Theory of quantum annealing of an ising spin glass. Science295(5564), 2427–2430 (2002) https://doi.org/10.1126/ science.1068774 https://www.science.org/doi/pdf/10.1126/science.1068774
-
[14]
Rajak, A., Suzuki, S., Dutta, A., Chakrabarti, B.K.: Quantum annealing: an overview. Philosophical Transactions of the Royal Society A: Mathemati- cal, Physical and Engineering Sciences381(2241), 20210417 (2022) https:// doi.org/10.1098/rsta.2021.0417 https://royalsocietypublishing.org/rsta/article- pdf/doi/10.1098/rsta.2021.0417/1326707/rsta.2021.0417.pdf
-
[15]
Reports on Progress in Physics83(5), 054401 (2020) https://doi.org/10.1088/1361-6633/ab85b8
Hauke, P., Katzgraber, H.G., Lechner, W., Nishimori, H., Oliver, W.D.: Perspec- tives of quantum annealing: methods and implementations. Reports on Progress in Physics83(5), 054401 (2020) https://doi.org/10.1088/1361-6633/ab85b8
-
[16]
Jour- nal of Mathematical Physics49(12), 125210 (2008) https://doi.org/10.1063/1
Morita, S., Nishimori, H.: Mathematical foundation of quantum annealing. Jour- nal of Mathematical Physics49(12), 125210 (2008) https://doi.org/10.1063/1. 2995837
work page doi:10.1063/1 2008
-
[17]
Willsch, M., Willsch, D., Jin, F., De Raedt, H., Michielsen, K.: Real-time simula- tion of flux qubits used for quantum annealing. Phys. Rev. A101, 012327 (2020) https://doi.org/10.1103/PhysRevA.101.012327 24
-
[18]
Pelofske, E., Hahn, G., Djidjev, H.: Inferring the dynamics of the state evolu- tion during quantum annealing. IEEE Transactions on Parallel and Distributed Systems33(2), 310–321 (2022) https://doi.org/10.1109/TPDS.2020.3044846
-
[19]
Marshall, J., Rieffel, E.G., Hen, I.: Thermalization, freeze-out, and noise: Deci- phering experimental quantum annealers. Phys. Rev. Appl.8, 064025 (2017) https://doi.org/10.1103/PhysRevApplied.8.064025
-
[20]
Nature Communications 7(1), 10327 (2016) https://doi.org/10.1038/ncomms10327
Boixo, S., Smelyanskiy, V.N., Shabani, A., Isakov, S.V., Dykman, M., Denchev, V.S., Amin, M.H., Smirnov, A.Y., Mohseni, M., Neven, H.: Computational mul- tiqubit tunnelling in programmable quantum annealers. Nature Communications 7(1), 10327 (2016) https://doi.org/10.1038/ncomms10327
-
[21]
Marshall, J., Venturelli, D., Hen, I., Rieffel, E.G.: Power of pausing: Advancing understanding of thermalization in experimental quantum annealers. Phys. Rev. Appl.11, 044083 (2019) https://doi.org/10.1103/PhysRevApplied.11.044083
-
[22]
Quantum Information Processing 10(1), 33–52 (2011) https://doi.org/10.1007/s11128-010-0168-z
Perdomo-Ortiz, A., Venegas-Andraca, S.E., Aspuru-Guzik, A.: A study of heuris- tic guesses for adiabatic quantum computation. Quantum Information Processing 10(1), 33–52 (2011) https://doi.org/10.1007/s11128-010-0168-z
-
[23]
IEEE Transactions on Quantum Engineering1, 1–12 (2020) https://doi.org/10.1109/TQE.2020.3021921
Krauss, T., McCollum, J.: Solving the network shortest path problem on a quantum annealer. IEEE Transactions on Quantum Engineering1, 1–12 (2020) https://doi.org/10.1109/TQE.2020.3021921
-
[24]
Quantum Science and Technology10(2), 025025 (2025) https://doi.org/10.1088/2058-9565/adb029
Pelofske, E.: Comparing three generations of d-wave quantum annealers for minor embedded combinatorial optimization problems. Quantum Science and Technology10(2), 025025 (2025) https://doi.org/10.1088/2058-9565/adb029
-
[25]
PLOS ONE16, 1–10 (2021) https://doi.org/10.1371/journal.pone
Golden, J., O’Malley, D.: Reverse annealing for nonnegative/binary matrix fac- torization. PLOS ONE16, 1–10 (2021) https://doi.org/10.1371/journal.pone. 0244026
-
[26]
Quantum Machine Intelligence1(1), 17–30 (2019) https: //doi.org/10.1007/s42484-019-00001-w
Venturelli, D., Kondratyev, A.: Reverse quantum annealing approach to portfolio optimization problems. Quantum Machine Intelligence1(1), 17–30 (2019) https: //doi.org/10.1007/s42484-019-00001-w
-
[27]
Scientific Reports9(1), 12837 (2019) https://doi.org/ 10.1038/s41598-019-49172-3
Ikeda, K., Nakamura, Y., Humble, T.S.: Application of quantum annealing to nurse scheduling problem. Scientific Reports9(1), 12837 (2019) https://doi.org/ 10.1038/s41598-019-49172-3
-
[28]
Scientific Reports12(1), 17753 (2022) https://doi.org/10
Haba, R., Ohzeki, M., Tanaka, K.: Travel time optimization on multi-agv routing by reverse annealing. Scientific Reports12(1), 17753 (2022) https://doi.org/10. 1038/s41598-022-22704-0
2022
-
[29]
Quan- tum Reports6(3), 452–464 (2024) https://doi.org/10.3390/quantum6030030 25
Jattana, M.S.: Reverse quantum annealing assisted by forward annealing. Quan- tum Reports6(3), 452–464 (2024) https://doi.org/10.3390/quantum6030030 25
-
[30]
Arai, S., Ohzeki, M., Tanaka, K.: Mean field analysis of reverse annealing for code- division multiple-access multiuser detection. Phys. Rev. Res.3, 033006 (2021) https://doi.org/10.1103/PhysRevResearch.3.033006
-
[31]
Pelofske, E., B¨ artschi, A., Eidenbenz, S.: Simulating Heavy-Hex Transverse Field Ising Model Magnetization Dynamics Using Programmable Quantum Annealers (2024)
2024
-
[32]
Kim, M., Singh, A.K., Venturelli, D., Kaewell, J., Jamieson, K.: X-ResQ: Reverse Annealing for Quantum MIMO Detection with Flexible Parallelism (2024)
2024
-
[33]
Advanced Quantum Technologies4(2), 2000133 (2021)
Rocutto, L., Destri, C., Prati, E.: Quantum semantic learning by reverse anneal- ing of an adiabatic quantum computer. Advanced Quantum Technologies4(2), 2000133 (2021)
2021
-
[34]
In: M¨ ohring, R.H
Bertoni, A., Campadelli, P., Posenato, R.: An upper bound for the maximum cut mean value. In: M¨ ohring, R.H. (ed.) Graph-Theoretic Concepts in Computer Science, pp. 78–84. Springer, Berlin, Heidelberg (1997)
1997
-
[35]
4OR17(4), 335–371 (2019) https://doi.org/ 10.1007/s10288-019-00424-y
Glover, F., Kochenberger, G., Du, Y.: Quantum bridge analytics i: a tutorial on formulating and using qubo models. 4OR17(4), 335–371 (2019) https://doi.org/ 10.1007/s10288-019-00424-y
-
[36]
Garey, M.R., Johnson, D.S.: Computers and Intractability: A Guide to the Theory of NP-Completeness. W. H. Freeman & Co., USA (1979)
1979
-
[37]
IEEE Access9, 81032–81039 (2021)
Tamura, K., Shirai, T., Katsura, H., Tanaka, S., Togawa, N.: Performance com- parison of typical binary-integer encodings in an ising machine. IEEE Access9, 81032–81039 (2021)
2021
-
[38]
Engineering Applications of Artificial Intelligence110, 104743 (2022)
Ezugwu, A.E., Ikotun, A.M., Oyelade, O.O., Abualigah, L., Agushaka, J.O., Eke, C.I., Akinyelu, A.A.: A comprehensive survey of clustering algorithms: State-of- the-art machine learning applications, taxonomy, challenges, and future research prospects. Engineering Applications of Artificial Intelligence110, 104743 (2022)
2022
-
[39]
Statistics and Computing17(4), 395–416 (2007) https://doi.org/10.1007/s11222-007-9033-z
Luxburg, U.: A tutorial on spectral clustering. Statistics and Computing17(4), 395–416 (2007) https://doi.org/10.1007/s11222-007-9033-z
-
[40]
Publications of the Mathematical Institute of the Hungarian Academy of Sciences5, 17–61 (1960)
Erd˝ os, P., R´ enyi, A.: On the evolution of random graphs. Publications of the Mathematical Institute of the Hungarian Academy of Sciences5, 17–61 (1960)
1960
-
[41]
McGeoch, C.C., Farr´e, P.: The d-wave advantage system: An overview technical report (2020)
2020
-
[42]
Dobrynin, D., Renaudineau, A., Hizzani, M., Strukov, D., Mohseni, M., Strachan, J.P.: Energy landscapes of combinatorial optimization in ising machines. Phys. Rev. E110, 045308 (2024) https://doi.org/10.1103/PhysRevE.110.045308 26
-
[43]
Bell System Technical Journal27(3), 379–423 (1948)
Shannon, C.E.: A mathematical theory of communication. Bell System Technical Journal27(3), 379–423 (1948)
1948
-
[44]
IEEE Transactions on Quantum Engineering4, 1–21 (2023) https://doi.org/10.1109/tqe.2023.3319586
Pelofske, E., Hahn, G., Djidjev, H.: Initial state encoding via reverse quantum annealing and h-gain features. IEEE Transactions on Quantum Engineering4, 1–21 (2023) https://doi.org/10.1109/tqe.2023.3319586
-
[45]
New Journal of Physics19(2), 023024 (2017) https://doi.org/10.1088/1367-2630/aa59c4
Chancellor, N.: Modernizing quantum annealing using local searches. New Journal of Physics19(2), 023024 (2017) https://doi.org/10.1088/1367-2630/aa59c4
-
[46]
In: 2020 IEEE International Conference on Quantum Computing and Engineering (QCE), pp
Pelofske, E., Hahn, G., Djidjev, H.N.: Advanced anneal paths for improved quan- tum annealing. In: 2020 IEEE International Conference on Quantum Computing and Engineering (QCE), pp. 256–266 (2020). https://doi.org/10.1109/QCE49297. 2020.00040
-
[47]
Seki, Y., Nishimori, H.: Quantum annealing with antiferromagnetic fluctuations. Phys. Rev. E85, 051112 (2012) https://doi.org/10.1103/PhysRevE.85.051112
-
[48]
Yamashiro, Y., Ohkuwa, M., Nishimori, H., Lidar, D.A.: Dynamics of reverse annealing for the fully connectedp-spin model. Phys. Rev. A100, 052321 (2019) https://doi.org/10.1103/PhysRevA.100.052321
-
[49]
Knysh, S., Plamadeala, E., Venturelli, D.: Quantum annealing speedup of embed- ded problems via suppression of griffiths singularities. Phys. Rev. B102, 220407 (2020) https://doi.org/10.1103/PhysRevB.102.220407
-
[50]
Royal Society Open Science4(1), 160600 (2017)
Bailey, D.C.: Not normal: the uncertainties of scientific measurements. Royal Society Open Science4(1), 160600 (2017)
2017
-
[51]
Europhysics Letters89(4), 40004 (2010) https://doi.org/10.1209/ 0295-5075/89/40004
J¨ org, T., Krzakala, F., Kurchan, J., Maggs, A.C., Pujos, J.: Energy gaps in quan- tum first-order mean-field–like transitions: The problems that quantum annealing cannot solve. Europhysics Letters89(4), 40004 (2010) https://doi.org/10.1209/ 0295-5075/89/40004
2010
-
[52]
Ohkuwa, M., Nishimori, H., Lidar, D.A.: Reverse annealing for the fully connected p-spin model. Phys. Rev. A98, 022314 (2018)
2018
-
[53]
Scientific Reports14(1), 4555 (2024) https://doi.org/10.1038/ s41598-024-55314-z
´Smierzchalski, T., Mzaouali, Z., Deffner, S., Gardas, B.: Efficiency optimiza- tion in quantum computing: balancing thermodynamics and computational performance. Scientific Reports14(1), 4555 (2024) https://doi.org/10.1038/ s41598-024-55314-z
2024
-
[54]
npj Quantum Information10, 30 (2024) https://doi.org/10.1038/s41534-024-00825-w
Pelofske, E., B¨ artschi, A., Eidenbenz, S.: Short-depth qaoa circuits and quantum annealing on higher-order ising models. npj Quantum Information10, 30 (2024) https://doi.org/10.1038/s41534-024-00825-w
-
[55]
Munoz-Bauza, H., Lidar, D.: Scaling advantage in approximate optimization with 27 quantum annealing. Phys. Rev. Lett.134, 160601 (2025) https://doi.org/10. 1103/PhysRevLett.134.160601
2025
-
[56]
Weinberg, P., Tylutki, M., R¨ onkk¨ o, J.M., Westerholm, J.,˚Astr¨ om, J.A., Man- ninen, P., T¨ orm¨ a, P., Sandvik, A.W.: Scaling and diabatic effects in quantum annealing with a d-wave device. Phys. Rev. Lett.124, 090502 (2020) https: //doi.org/10.1103/PhysRevLett.124.090502
-
[57]
Dutta, A., Rahmani, A., Campo, A.: Anti-kibble-zurek behavior in crossing the quantum critical point of a thermally isolated system driven by a noisy control field. Phys. Rev. Lett.117, 080402 (2016) https://doi.org/10.1103/PhysRevLett. 117.080402
-
[58]
Bando, Y., Yip, K.-W., Chen, H., Lidar, D.A., Nishimori, H.: Breakdown of the weak-coupling limit in quantum annealing. Phys. Rev. Appl.17, 054033 (2022) https://doi.org/10.1103/PhysRevApplied.17.054033
-
[59]
Albash, T., Lidar, D.A.: Adiabatic quantum computation. Rev. Mod. Phys.90, 015002 (2018) https://doi.org/10.1103/RevModPhys.90.015002
-
[60]
Physical Review A93(3) (2016) https://doi.org/10.1103/physreva.93
Venuti, L.C., Albash, T., Lidar, D.A., Zanardi, P.: Adiabaticity in open quantum systems. Physical Review A93(3) (2016) https://doi.org/10.1103/physreva.93. 032118 28
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.