{"id":"2c739778-bf31-4d4a-ac81-9464a03dbf45","arxiv_id":"2411.13742","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A 372-instance numerical benchmark of VQE for Fermi-Hubbard finds Momentum and Adam with finite differences achieve the best accuracy, while SPSA and CMAES minimize function calls.","lead":"This paper benchmarked 30 optimizers on hundreds of simulated Fermi-Hubbard model instances run through the variational quantum eigensolver. It finds that Momentum and Adam with finite-difference gradients were the most accurate, while SPSA and CMAES were more sample-efficient.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ranking may be an artifact of best-of-multiple hyperparameter runs for tuned optimizers versus a single default run for scipy optimizers.","rationale":"The reader's weakest assumption concerns the transfer of a single finite-difference step size (0.4) from four sweeping instances to all 372 instances. That is a reasonable and addressable concern, but I see a more directly load-bearing methodological issue: the final accuracy ranking in Fig. 6 uses the best energy over up to five hyperparameter configurations for tuned optimizers, while scipy optimizers with no tuned hyperparameters contribute only one run. Since the cost function is shot-noise limited, taking the minimum over several runs gives a statistical advantage to the tuned optimizers. This is not an internal inconsistency; the paper reports its protocol transparently in the figure caption and in Section 2.5. However, it weakens the strongest claim because the headline ranking may be an artifact of evaluation effort rather than optimizer merit. A default-set-only reanalysis is a concrete, low-cost check using the deposited data. If the ranking survives that check, the central empirical conclusion is much better supported. If not, the paper should present both comparisons and temper the claim accordingly. Because the concern is empirical and addressable, it does not warrant rejection; the paper remains a useful benchmark, but the conditional verdict should require this robustness check or a clear caveat about the unequal number of runs. I therefore keep the reader's CONDITIONAL verdict unchanged, noting partial agreement because my focus differs from the reader's step-size concern, though both concern hyperparameter-related fairness.","tokens_in":31270,"tokens_out":7118,"duration_ms":72175,"concrete_test":"Recompute the main comparisons (Fig. 6 final-energy boxplots and Fig. 9 call-count plots) using only the run with the default hyperparameter set for every optimizer on every instance, as the Zenodo dataset records each hyperparameter set separately. This gives exactly one noisy run per optimizer per instance, eliminating the best-of-k advantage. If Momentum/Adam-FD and SPSA/CMAES/BayesMGD still occupy the top ranks, the central ranking is robust; if scipy optimizers such as BFGS or SLSQP move substantially upward, the headline claim depends on the unequal number of attempts.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that Momentum/Adam with finite differences are the best for final accuracy rests on comparing optimizers under unequal evaluation effort. Section 2.5 says each tuned optimizer is run with 'at most 5 sets of hyperparameters' per instance (default plus the best found on each of the four sweeping instances). Figure 6's caption states that for each optimizer the reported value is 'the best final energy (over different runs, e.g. for multiple hyperparameter settings)'. Thus Momentum, Adam, SPSA, CMAES, BayesMGD and other tuned optimizers get up to five independent noisy runs and the minimum is used, while scipy optimizers with zero hyperparameters (COBYLA, Powell, BFGS, L-BFGS-B, TNC, SLSQP, CG, Newton-CG) contribute only a single run per instance. Because the cost function is stochastic (1,000 or 10,000 shots), the minimum of five noisy runs is a biased, optimistic estimator of an optimizer's typical performance relative to one run. The measured gap between the top tuned optimizers and the untuned scipy optimizers in Figs. 6 and 9 could therefore be inflated by the evaluation protocol rather than reflecting genuine optimizer quality. The paper acknowledges the subjectivity in choosing hyperparameters but does not address this unequal number of attempts in the ranking.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a large numerical benchmark of 30 optimisers applied to 372 instances of VQE for the Fermi-Hubbard model using the Hamiltonian variational ansatz. The authors record both exact and shot-noise cost functions, rank optimisers by final energy and by number of cost-function calls to reach fixed tolerances, analyse finite-difference versus simultaneous-perturbation gradients, and study the quantum natural gradient and imaginary-time evolution for 1D systems. The main claims are that Momentum and Adam with finite-difference gradients give the best final accuracy, that SPSA, CMAES, and BayesMGD are best in the low-call regime, and that the finite-difference step size has a strong effect on performance. Hyperparameters are selected by sweeping on four instances, and the full dataset is made available on Zenodo.","tokens_in":31574,"tokens_out":5821,"duration_ms":59011,"significance":"If the reported ranking is robust, the paper would be a useful reference for VQE practitioners, bringing together a wider set of optimisers, instances, and per-call data than most prior comparisons. The careful recording of every cost-function call, the use of exact ground states as baselines, the open data, and the detailed per-optimiser hyperparameter notes are genuine strengths that make the study reproducible in principle. However, the headline ranking is currently vulnerable to a protocol issue: tuned optimisers are evaluated as the best of up to five independent noisy runs, while zero-hyperparameter scipy optimisers are evaluated once. Because the cost function is stochastic, this unequal number of attempts can inflate the apparent gap between the top optimisers and the scipy baselines. The central conclusions are defensible but need additional analysis to control for this effect.","major_comments":[{"comment":"The headline ranking compares best-of-multiple-runs for tuned optimisers against a single run for the zero-hyperparameter scipy optimisers. Section 2.5 states that each tuned optimiser is run with at most five sets of hyperparameters (the default plus the best found on each of the four sweeping instances), and the caption of Fig. 6 says that for each optimiser the reported value is 'the best final energy (over different runs, e.g. for multiple hyperparameter settings)'. Since the cost function is stochastic with 1,000 or 10,000 shots, the minimum of five noisy runs is a biased optimistic estimator, whereas COBYLA, Powell, BFGS, L-BFGS-B, TNC, SLSQP, CG, and Newton-CG contribute only a single run per instance. The gaps in Figs. 6, 7, and 9 between the top tuned optimisers and the scipy optimisers could therefore be inflated by the evaluation protocol rather than reflecting genuine optimizer quality. I recommend adding a control that uses only the default hyperparameter set for every optimiser, and/or giving every optimiser the same number of independent restarts, and reporting both the best-of and the median performance.","section":"Section 2.5 and Fig. 6 caption"},{"comment":"The global finite-difference step size of 0.4 is the average of the best step sizes found on only four instances (0.417, 0.251, 0.324, and 0.550), and the per-optimiser hyperparameters are selected on those same four instances. These sweeping instances are all small (at most 3 × 2 grids and five layers) and do not include the 14-18 qubit or 8-10 layer cases in Benchmarks 3 and 4. Since Fig. 18 shows that optimiser performance varies systematically with U, filling, and number of layers, the representativeness of the four sweeping instances is an unverified assumption that is load-bearing for the claim that '0.4 seemed to be a good choice'. I ask the authors to quantify the sensitivity of the rankings to this choice, for example by re-running the gradient-based optimisers with step sizes near the ends of the observed range (e.g. 0.25 and 0.55) and reporting whether the leading optimisers remain the same.","section":"Section 2.4 and Section 2.5"},{"comment":"For the number-of-calls metric it is not stated whether the reported 'calls to reach tolerance' is taken from the best run over the multiple hyperparameter sets or from each run individually. If Fig. 9 uses the best run for each tuned optimiser and the single run for scipy optimisers, the same best-of-multiple-runs bias applies to the low-call ranking as to the final-energy ranking. The authors should state explicitly how the call counts are aggregated and should report both the best-run and the median-run call counts, so that readers can separate genuine optimizer quality from the number of attempts allowed.","section":"Section 3.2.2 and Fig. 9"}],"minor_comments":[{"comment":"In the row for '3 × 2', the expression '2m⌈ n−1 2 ⌋' mixes a ceiling bracket with a floor symbol; it should read '2m⌈(n−1)/2⌉'.","section":"Table 1"},{"comment":"The phrase 'some plots of individual runs are shown n Fig. 5' should read 'shown in Fig. 5'.","section":"Section 3.2, first paragraph"},{"comment":"The caption says 'Coordinate descent hyperparameters', but the table lists Nelder-Mead hyperparameters; the caption should be corrected.","section":"Table 15 caption"},{"comment":"The word 'Bernouilli' in the SPSA pseudocode description should be spelled 'Bernoulli'.","section":"Appendix C.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is a well-executed empirical study with open data, and the authors are transparent about the arbitrariness in hyperparameter choices. The central issue is the unequal evaluation effort between tuned optimisers and scipy optimisers; this is a protocol flaw that can be addressed with additional analysis, and it is not so deep as to require rejection. If the authors provide a default-only control and clarify the call-count aggregation, the paper could become a solid reference for the community."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague—\n\nThis is a genuinely useful benchmark, but I'd temper the headline claims. The paper runs 30 optimizers over 372 VQE instances for the Fermi-Hubbard model with the Hamiltonian variational ansatz, records per-call exact and noisy energies, and publishes the data on Zenodo. That alone makes it a reference point for near-term VQE practice. The hyperparameter sweeping on four instances is careful, and the comparison between finite-difference and simultaneous-perturbation gradient subroutines is a nice contribution. The quantum natural-gradient analysis is honest: faster per iteration, but the overhead usually eats the gain.\n\nThe soft spot is real and it's in the evaluation protocol. For tuned optimizers (Momentum, Adam, SPSA, CMAES, BayesMGD, etc.), the paper reports the best final energy over up to five hyperparameter sets per instance. The zero-hyperparameter scipy optimizers (COBYLA, Powell, BFGS, L-BFGS-B, SLSQP, CG, Newton-CG) get one run. Since the cost function is stochastic (1,000 or 10,000 shots), the minimum of five noisy runs is an optimistic bias compared to a single run. The stress-test note is correct: this likely inflates the gap between the top tuned optimizers and the scipy baseline. I don't think it flips the ranking entirely—Momentum/Adam with FD do look genuinely strong, and SPSA/CMAES/BayesMGD do well at low call counts, consistent with prior work—but the figures make the margin look bigger than the evidence supports. The fix is straightforward: report median (or mean) over repeated runs, or equalize the number of attempts per optimizer.\n\nTwo smaller concerns. The finite-difference step size 0.4 is averaged over four sweeping instances; the authors show the optimal step size varies from about 0.25 to 0.55, so 0.4 is reasonable but could be instance-dependent. And there's no code release, only data, which makes it harder to reproduce the exact pipeline. Minor.\n\nOverall: solid empirical work, useful to anyone choosing an optimizer for VQE on Fermi-Hubbard or similar systems. It deserves a serious referee—the central issue is addressable and the dataset is valuable. I would cite it, but with a footnote about the evaluation protocol.","headline":"Broad, useful VQE optimizer benchmark, but unequal evaluation effort across optimizers weakens the headline ranking.","tokens_in":32014,"tokens_out":3145,"would_cite":true,"duration_ms":30652,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Across 372 Fermi-Hubbard VQE instances, Momentum and Adam with finite-difference gradients reach the lowest ground-state energies, while SPSA, CMAES, and BayesMGD win when cost-function calls are scarce.","keywords":["variational quantum eigensolver","Fermi-Hubbard model","Hamiltonian variational ansatz","optimiser benchmarking","finite difference gradient","simultaneous perturbation","quantum natural gradient","hyperparameter sweeping"],"falsifier":"Run the same 372-instance benchmark with finite-difference step sizes of, say, 0.1 and 0.8 while keeping all hyperparameter sweeps fixed; if the top optimisers in the final-energy ranking shift away from Momentum and Adam, the central ranking is an artefact of the 0.4 step size chosen from four instances.","tokens_in":31120,"feed_emoji":"⚛️","tokens_out":4978,"duration_ms":46400,"temperature":0.7,"pith_summary":"This paper tries to establish which classical optimisers a practitioner should choose when running the variational quantum eigensolver on the Fermi-Hubbard model with the Hamiltonian variational ansatz. It benchmarks 30 optimisers on 372 instances that vary grid size, Coulomb interaction, filling, ansatz depth, and number of measurement shots. Its central finding is that no single optimiser wins on both metrics: Momentum and Adam with finite-difference gradients give the best final accuracy, whereas SPSA, CMAES, and BayesMGD reach a target accuracy in the fewest cost-function calls. The paper also shows that the finite-difference step size strongly controls gradient quality and that simultaneous-perturbation gradients trade final precision for speed. The practical payoff would be concrete guidance for near-term quantum hardware, where the number of circuit evaluations is the scarce resource.","feed_headline":"Momentum and Adam beat 28 optimisers on Fermi-Hubbard VQE","feed_subtitle":"Across 372 instances, they hit the lowest ground-state energies; SPSA, CMAES, and BayesMGD win when call counts matter.","key_machinery":"The central object is the benchmark itself: a fixed set of 372 VQE instances built from the Hamiltonian variational ansatz for the Fermi-Hubbard model, each evaluated with two shot counts (1,000 and 10,000), with exact ground energies computed classically as the reference. The two gradient subroutines are the load-bearing comparison: finite differences, which costs two cost-function evaluations per parameter and is run at step size 0.4 (the average of the best step sizes found by sweeping four instances), and simultaneous perturbation, which costs two evaluations total and is run at step size 0.15. The paper also uses the diagonal approximation of the quantum Fisher information matrix as the machinery for the quantum natural gradient comparison, restricted to one-dimensional systems because the HV ansatz's fixed FSWAP gates prevent a global parameter shift of shared-angle gates.","core_discovery":"The paper claims that for VQE on the Fermi-Hubbard model with the Hamiltonian variational ansatz under shot-noise-limited cost functions, optimisation performance separates cleanly into two regimes. On final energy accuracy, Momentum and Adam using a finite-difference gradient with step size 0.4 outperform the other optimisers across the 372 instances, with the advantage clearest at 10,000 shots and half filling. On efficiency, SPSA, CMAES, and BayesMGD need the fewest cost-function calls to reach a given tolerance, and CMAES offers the best balance between call count and final energy. The paper further claims that finite-difference gradients give more precise final energies but many more calls than simultaneous-perturbation gradients, and that the quantum natural gradient's per-iteration advantage over vanilla gradient descent is generally erased when counting the extra calls needed to estimate the Fisher information.","pith_inferences":["Beyond the paper's stated scope, the same two-regime pattern likely appears for other lattice Hamiltonians such as the transverse-field Ising model, because the Hamiltonian variational ansatz keeps parameter counts low and the noise structure is similar; this is a testable transfer, not a paper claim.","The step-size averaging over four instances suggests a cheaper protocol: run a small hyperparameter sweep on one or two representative instances before a hardware campaign, then fix the step size for the rest; the paper does not propose this as a protocol.","A testable extension is an adaptive schedule that begins with simultaneous-perturbation gradients to descend quickly and switches to finite differences near the end; the paper mentions this as a future idea, and the data here suggest it should beat either subroutine used alone."],"forward_implications":["A practitioner prioritising final ground-state energy should pick Momentum or Adam with a finite-difference gradient at a carefully chosen step size rather than a black-box optimiser.","A practitioner with a strict budget on circuit evaluations should pick SPSA, CMAES, or BayesMGD, with CMAES as the best compromise between call count and final energy.","Using simultaneous perturbation as a gradient subroutine cuts the number of calls but leaves the run further from the ground state than finite differences; switching from simultaneous perturbation to finite differences partway through a run is a plausible way to combine both advantages.","The step size for finite-difference gradients must be treated as a serious hyperparameter: an arbitrary small value such as 0.01 can make all gradient-based optimisers perform poorly.","For 1D Fermi-Hubbard systems, the quantum natural gradient and imaginary time evolution can improve per-iteration convergence, but their overhead in cost-function calls means they do not beat vanilla gradient descent when total calls are counted."],"supporting_citations":[{"why":"Supplies the Hamiltonian variational ansatz for Fermi-Hubbard, the FSWAP construction, the initial parameter choice, and the coordinate descent baseline used in the benchmark.","marker":"[6]"},{"why":"Introduces BayesMGD and provides the scalable Fermi-Hubbard VQE context that the paper extends to a wide optimiser comparison.","marker":"[3]"},{"why":"Defines quantum natural gradient and the quantum Fisher information approach that Section 3.3 tests on 1D systems.","marker":"[17]"},{"why":"Introduces model gradient descent, the predecessor of BayesMGD, and gives an earlier optimiser comparison the paper builds on.","marker":"[18]"},{"why":"Provides an earlier VQE optimiser comparison whose methods and baseline optimisers the paper adapts to the Fermi-Hubbard setting.","marker":"[23]"},{"why":"Gives the prior benchmark that found SPSA strong and CMAES competitive after hyperparameter tuning, forming the direct comparison point for the new results.","marker":"[25]"},{"why":"Supplies the SPSA algorithm and its default step-size conventions that the paper uses for the simultaneous perturbation gradient subroutine.","marker":"[38, 39]"},{"why":"Provides the scipy implementations of BFGS, L-BFGS-B, Nelder-Mead, Powell, and SLSQP used for exact-cost baseline runs and for comparison optimisers.","marker":"[34]"}],"fun_headline_variants":["VQE benchmark: Momentum and Adam top final energy, but SPSA and CMAES cut calls","Fermi-Hubbard VQE: Momentum and Adam for accuracy, SPSA and BayesMGD for speed","30 optimisers on Fermi-Hubbard VQE: two regimes, two winners","In Fermi-Hubbard VQE, Adam wins on energy but loses on calls to SPSA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The finite-difference step size 0.4, averaged from the best step sizes found on only four sweeping instances, is assumed to work well across all 372 instances; if the optimal step size varies by instance, the ranking of gradient-based optimisers could change.","fun_headline_variants_meta":{"raw":{"variants":["VQE benchmark: Momentum and Adam top final energy, but SPSA and CMAES cut calls","Fermi-Hubbard VQE: Momentum and Adam for accuracy, SPSA and BayesMGD for speed","30 optimisers on Fermi-Hubbard VQE: two regimes, two winners","In Fermi-Hubbard VQE, Adam wins on energy but loses on calls to SPSA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000762,"raw_usage":{"total_tokens":3392,"prompt_tokens":968,"completion_tokens":2424,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":584,"completion_tokens_details":{"reasoning_tokens":2321}},"tokens_in":584,"tokens_out":2424,"duration_ms":17289,"temperature":1.0,"reasoning_tokens":2321,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:55:46.662502+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 372-instance benchmark with finite-difference step sizes of, say, 0.1 and 0.8 while keeping all hyperparameter sweeps fixed; if the top optimisers in the final-energy ranking shift away from Momentum and Adam, the central ranking is an artefact of the 0.4 step size chosen from four instances.","supporting_citations":[{"cited_title":"Using models to improve optimizers for variational quantum algorithms","cited_arxiv_id":null,"evidence_quote":"Introduces model gradient descent, the predecessor of BayesMGD, and gives an earlier optimiser comparison the paper builds on."}],"review_version":1}