{"id":"f5018c1e-1a2a-434d-8079-9a6377bbc921","arxiv_id":"2508.10873","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A new ground-state energy estimation benchmark rates SHCI, DMRG, and double-factorized QPE, reporting near-universal SHCI solvability from an ML extrapolation that its own empirical table only partially supports.","lead":"This paper introduces an open benchmark that runs classical and quantum solvers on 228 ground-state energy problems and scores them with a machine-learning based 'solvability' metric. It finds optimized SHCI near-universal, DMRG solid on low-entanglement cases, and double-factorized QPE resource estimates viable on only 4 of 131 instances, while conceding the dataset is biased toward SHCI.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 1.0000 solvability for SHCI Opt is an ML-extrapolated ratio over synthetic latent points without physical-validity filtering (App. B.2) and contradicts the table's empirical 148/226 (65%); the ratio is non-monotonic in empirical pass rates, so the central claim is unsupported without held-out va","rationale":"The paper's central quantitative assertion is the solvability ratio, and the abstract's 'near-universal solvability' is the headline. The only evidence for it is the ML extrapolation in Table 2.1, which is internally inconsistent with the empirical counts and uses synthetic points that the authors explicitly admit lack a physical-validity filter. This is precisely the reader's weakest_assumption, so my read agrees and does not change the CONDITIONAL verdict. The concern is addressable—reporting empirical pass rates with confidence intervals and validating the ML estimator on held-out instances—so I do not escalate to REJECT; the framework and honest self-audit remain valuable. I also note a secondary reproducibility issue: Algorithm 1 specifies NNMF while the main text and figures use PCA, which must be resolved for any test to be meaningful.","tokens_in":23527,"tokens_out":5085,"duration_ms":52998,"concrete_test":"Leave-one-out or 70/30 hold-out validation: retrain the exact pipeline (resolving the PCA-vs-NNMF discrepancy) on a training subset; generate the 10,000 latent samples; then (i) measure the SVM's held-out predicted-vs-actual pass rate on the excluded real tasks, (ii) compute calibration (Brier score, reliability diagram) of predicted probabilities, and (iii) inverse-transform the 10,000 samples and check whether the resulting feature vectors obey physical constraints (non-negative DF eigenvalues, integer electron numbers, valid spin-orbital counts); for even stronger evidence, construct actual Hamiltonians from the 10-20 highest-probability synthetic points and run SHCI on them. If the held-out pass rate is ~65% rather than ~100%, or if the synthetic points are unphysical, the 1.0000 ratio is an artifact of the sampling distribution.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract claims 'fully optimized SHCI achieves near-universal solvability,' supported only by the ML-derived solvability ratio 1.0000 in Table 2.1. This ratio is computed (Alg. 1, §4.4) by training an SVM on actual solver labels, embedding features via PCA (though Alg. 1 specifies NNMF), generating 10,000 points in the latent-space bounding box, inverse-transforming them, and counting the fraction with predicted probability > 0.5. The authors state in Appendix B.2 that 'we do not yet have a filter for true physical validity for the generated points.' Hence the synthetic points may not correspond to realizable Hamiltonians, and the ratio measures the latent bounding box, not plausible GSEE problems. The empirical table contradicts the ratio: SHCI 2e-4: 0.8125 ratio vs 83/228 solved (36%); SHCI 1e-4: 0.6562 vs 91/228 (40%); SHCI 2e-5: 0.6486 vs 128/228 (56%); SHCI Opt: 1.0000 vs 148/226 (65%). The ratio is not even monotone in empirical success counts. Without a physical-validity filter and without calibration or held-out validation of the SVM probabilities, the headline is an ungrounded extrapolation. The framework's value is to track solver progress, so an unvalidated metric is central rather than cosmetic.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces QB-GSEE, an open benchmarking framework for Ground State Energy Estimation (GSEE), with a standardized problem-instance database, polynomial-time Hamiltonian features, and machine-learning-based solvability analysis. It presents case studies for three solver families: SHCI variants, DMRG, and DF QPE resource estimates. The paper's central claim, stated in the abstract and Section 2.2, is that fully optimized SHCI achieves near-universal solvability on the benchmark set, DMRG is competitive for low-entanglement systems, and DF QPE is currently limited by runtime and hardware constraints. The authors also explicitly acknowledge in Section 3 that the current dataset is biased toward SHCI-related instances, and they propose future expansion to more strongly correlated systems.","tokens_in":23875,"tokens_out":3056,"duration_ms":36086,"significance":"If the quantitative claims were supported, this would be a useful open infrastructure for comparing GSEE solvers: the repository is public, the schema is standardized, and the inclusion of DMRG and DF QPE resource estimates provides a broader view than most existing benchmarks. The paper's strength is its modular design and its explicit admission of dataset bias. However, the headline 'near-universal solvability' rests on an unvalidated machine-learning extrapolation over synthetic latent-space points, and the paper's own empirical counts contradict that headline. With validation and reframing, the benchmark could still make a solid contribution; as written, the central claim is not supported.","major_comments":[{"comment":"The '1.0000 solvability' for SHCI Opt is not an empirical result. It is the fraction of 10,000 synthetic latent-space points for which an SVM predicts a probability greater than 0.5 (Algorithm 1, §4.4). Appendix B.2 explicitly states that 'we do not yet have a filter for true physical validity for the generated points,' and no calibration or held-out validation of the SVM probabilities is reported. The same table reports 148/226 tasks solved (65%), while the prose in §2.2 says SHCI Opt 'achieves universal solvability' and 'very few problem instances remain unsolved.' This contradiction is load-bearing because the abstract's central claim rests on the ML-derived 1.0000, not on the empirical pass count.","section":"Table 2.1, §2.2, Algorithm 1, Appendix B.2"},{"comment":"The reported solvability ratios are not monotone in the empirical task counts. SHCI 2e-4 has ratio 0.8125 with 83/228 solved; SHCI 1e-4 has ratio 0.6562 with 91/228 solved; SHCI 2e-5 has ratio 0.6486 with 128/228 solved; SHCI Opt has ratio 1.0000 with 148/226 solved. A metric intended to measure solver capability should track empirical success rates more closely. Without calibration, held-out validation, or at least reporting both empirical fractions and ML ratios side by side, the ML-derived ratios cannot be interpreted as 'solvability.' The paper should either validate the SVM probabilities on held-out instances or restrict the headline claims to empirical pass rates.","section":"Table 2.1"},{"comment":"The dataset-composition circularity is a correctness risk for the central claim. Section 3 admits that many Hamiltonians are drawn from prior SHCI-centric studies, and the pass/fail labels are produced by running SHCI-type solvers on those instances. The ML model is then trained on those labels, and the solvability ratio is computed over synthetic points in the bounding box of the observed data. This makes the 'near-universal solvability' claim largely an echo of the benchmark's construction. I am not claiming the authors are being disingenuous; rather, the paper needs a concrete test of generalization, e.g., held-out instances from independent sources or an explicit demonstration that the ML decision boundary is stable when the training set is resampled. Without such a test, the headline is not about GSEE solvability in general but about this particular curated set.","section":"§3, §4.4"}],"minor_comments":[{"comment":"Typo: 'ighlighting' should be 'highlighting'.","section":"Abstract"},{"comment":"There is an inconsistency: §4.4 says PCA was ultimately chosen, while Algorithm 1 step 4 specifies Non-Negative Matrix Factorization (NNMF), and Appendix B.2 also discusses NNMF. Please clarify which latent-space method was used for the results in Table 2.1 and Figures 2.1–2.2.","section":"§4.4 and Algorithm 1"},{"comment":"The phrase 'principle component analysis' appears; it should be 'principal component analysis.'","section":"§4.4"},{"comment":"SHCI Opt reports 226 tasks attempted while all other rows report 228. The discrepancy should be explained in the text, especially since it affects the empirical 65% figure.","section":"Table 2.1"},{"comment":"The sentence 'Lowering ε_var increases the number of tasks solved, but it only results in a 17% improvement over SHCI 2e-5' is unclear; the relative increase from 128 to 148 is approximately 15.6%, not 17%, and it would help to state the comparison explicitly.","section":"§2.2"},{"comment":"The paper notes that for DF QPE, 'solved' means resource estimates are below a runtime threshold, not that the DF-QPE answer was checked against the reference energy. This is an important distinction; consider making it more prominent in the main text, not only in Section 2.3.","section":"Appendix C.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely salvageable if the authors re-center the narrative on the empirical results and treat the ML-derived solvability ratios as exploratory rather than authoritative. The central 'near-universal solvability' claim should be removed or substantially qualified until the SVM probabilities are validated, e.g., on held-out instances or with a physical-validity filter. I would encourage the editor to ask for a revised version that reports empirical pass rates with uncertainties and clearly labels ML extrapolations as such."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things worth knowing. First, the QB-GSEE framework itself is solid, genuinely useful infrastructure: a standardized instance schema, encoding-aware feature extraction (DF decomposition and Pauli hypergraph statistics), and a standard report format. Nothing in HamLib, VarBench, SupermarQ, QASMBench, or AppQSim pulls these together with ML-based solvability regions and an open repo. Second, the paper's marquee result--SHCI Opt at 1.0000 solvability--is not supported by the paper's own table. That number is the fraction of 10,000 synthetic points in a PCA/NNMF latent space where an SVM predicts >50% success probability. Appendix B.2 explicitly says there is no physical-validity filter for these generated points, and no calibration or held-out validation is reported. The same table shows SHCI Opt solved 148 of 226 tasks (65%). The ratios are not even monotone in the empirical pass counts: SHCI 2e-4 reports 0.8125 solvability while solving 83/228 (36%), while SHCI 2e-5 reports 0.6486 while solving 128/228 (56%). So the abstract's 'near-universal solvability' is an extrapolation artifact, not a measured fact.\n\nCredit where it's due: the paper is unusually transparent about its own weaknesses. It admits the dataset is SHCI-centric (Section 3), that some reference energies are not within chemical accuracy (Section 4.3), that the DMRG 'first run' protocol can land on excited states (Appendix C.1), and that DF QPE 'solved' means resource estimates fit a runtime budget rather than a verified answer (Section 2.3). That honesty makes the framework credible even though the headline number is shaky.\n\nThe fix is straightforward: report empirical pass rates with error bars, validate the ML solvability estimator on held-out or non-SHCI instances, and reconcile the prose with Table 2.1. Also minor: Algorithm 1 says NNMF while the text and figures use PCA, and SHCI Opt's attempted count is 226 vs 228 for everyone else. The central flaw is the unvalidated extrapolation, and it is load-bearing because the paper presents it as the main result. But the infrastructure does not depend on that claim. I'd send this to peer review. A good referee would push for a reworked abstract and a validated metric, and the result would be a benchmark that the community actually uses.","headline":"The benchmark framework is a real contribution and the paper is honest about its limits, but the 'near-universal solvability' headline rests on an unvalidated ML extrapolation that contradicts the paper's own empirical counts.","tokens_in":24504,"tokens_out":2313,"would_cite":true,"duration_ms":23382,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An open benchmarking framework claims that a fully optimized classical configuration-interaction solver, SHCI, solves essentially all current test instances, while DMRG handles low-entanglement systems and DF QPE remains resource-limited.","keywords":["ground state energy estimation","quantum chemistry","benchmarking","SHCI","DMRG","quantum phase estimation","solvability prediction","machine learning"],"falsifier":"Take a random sample of the 10,000 synthetic latent points, inverse-transform them into Hamiltonian features, construct actual Hamiltonians with those features, and run optimized SHCI on them; if the solved fraction falls well below 1.0000 — for example near the 65% empirical rate in the paper's table — the near-universal solvability claim would be refuted. A cheaper check: hold out a random subset of the 228 benchmark instances, retrain the classifier without them, and compare predicted solvability with the actually solved fraction.","tokens_in":23304,"feed_emoji":"⚛️","tokens_out":11553,"duration_ms":108276,"temperature":0.7,"pith_summary":"This paper introduces an open, modular benchmarking framework for ground-state energy estimation (GSEE) and uses it to compare three solver families: semistochastic heat-bath configuration interaction (SHCI), density matrix renormalization group (DMRG), and double-factorized quantum phase estimation (DF QPE). Its central claim is that on the current benchmark set, fully optimized SHCI — SHCI with perturbation-theory corrections and orbital optimization — achieves near-universal solvability, while DMRG solves the low-entanglement instances and DF QPE's resource estimates exceed runtime budgets on almost every attempt. The framework measures solvability with a machine-learning pipeline: Hamiltonian features are projected into a latent space, a machine-learning classifier predicts the probability of success at 10,000 synthetic points, and the solvability ratio is the fraction with probability above 0.5. The paper also acknowledges that many of its Hamiltonians come from SHCI-oriented datasets, so the comparison favors classical SHCI, and it proposes adding strongly correlated systems in future versions. If the framework is sound, it gives the field a common yardstick for tracking whether quantum or classical solvers are making real progress on chemically relevant instances.","feed_headline":"One classical solver covers nearly all ground-state test space","feed_subtitle":"Open benchmark maps where SHCI, DMRG, and quantum phase estimation succeed, and flags a dataset bias toward SHCI.","key_machinery":"The central machinery is the solvability-ratio pipeline: polynomial-time Hamiltonian features (electron count, log FCI dimension, double-factorization rank and eigenvalue gap, Pauli one-norm, Pauli-string count, and interaction-hypergraph statistics) projected by principal component analysis into a low-dimensional latent space; a support-vector machine trained on observed solver successes and failures; and generation of 10,000 new latent points whose inverse-transformed features are scored by the trained model. The solvability ratio is the fraction of those points with predicted success probability above 0.5, and this ratio is the paper's headline measure of how much of the GSEE problem spac","core_discovery":"The central claim is the QB-GSEE benchmark itself plus the first solvability landscape it produces. On the paper's own terms, solvability is the fraction of a 10,000-point synthetic latent space where a trained classifier predicts more than a 50% chance that the solver returns the ground-state energy within chemical accuracy and within the instance's runtime requirement. Using that measure, SHCI with optimized orbitals and perturbative corrections reaches 1.0000, the only evaluated solver to cover the full problem space; DMRG scores 0.4126, succeeding on low-entanglement systems with a sharp boundary; and DF QPE scores 0.0716, solving only 4 of 131 attempted tasks because estimated runtimes","pith_inferences":["The paper's own table reports only 148 of 226 empirical successes for optimized SHCI, so the 1.0000 solvability ratio is an extrapolation; a reader should not interpret it as a measured success rate, and the gap suggests the classifier's decision boundary may be optimistic.","If the synthetic latent points do not correspond to physically plausible Hamiltonians — the paper states no such validity filter exists yet — the solvability ratio could overstate coverage by sampling regions no real chemistry occupies; adding a validity filter or a held-out empirical check would settle this.","The same feature-plus-classifier pipeline could be used to actively select new benchmark instances near the current decision boundaries, sharpening the solvability maps and reducing the SHCI bias faster than adding random molecules.","For quantum solvers, the framework could pre-register algorithmic improvements (for example, tensor hypercontraction or symmetry-shift preprocessing) as new solver entries, giving a quantitative target for when QPE becomes competitive on existing classical instances."],"forward_implications":["Fully optimized SHCI (SHCI+PT with orbital optimization) is the only evaluated solver with solvability ratio 1.0000; if the machine-learning extrapolation is trusted, no current benchmark instance class defeats it.","Merely lowering the SHCI variational threshold is not enough: the paper's numbers show a modest gain in tasks solved, while the jump to full latent-space coverage comes from orbital optimization and perturbative corrections.","DMRG's sharper solvability boundary identifies low-entanglement systems as its reliable territory, so DMRG-based reference energies are safest for those instances.","DF QPE resource estimates exceed the runtime budget on 127 of 131 attempted tasks, meaning fault-tolerant quantum phase estimation at current double-factorization costs and hardware assumptions is not competitive on this benchmark set.","Because the current dataset is SHCI-biased, the benchmark's rankings cannot yet be read as a general statement about quantum versus classical advantage; the paper proposes adding strongly correlated, multi-reference systems to correct this."],"supporting_citations":[{"why":"Supplies the open benchmark infrastructure, standard report, and problem-instance schema whose results the paper analyzes.","marker":"[1]"},{"why":"Support-vector networks: the classifier whose predicted probabilities define the solvability boundaries and ratios.","marker":"[19]"},{"why":"Principal component analysis: the dimensionality-reduction method used to construct the latent space and solvability maps.","marker":"[41]"},{"why":"Planted-solution Hamiltonians with exactly known ground states, used as benchmark instances.","marker":"[31]"},{"why":"Provides the DMRG run protocol and DF QPE resource-estimate methodology that the classical and quantum solver procedures follow.","marker":"[37]"},{"why":"Low-rank double factorization of electronic structure Hamiltonians; supplies the DF rank, eigenvalue, and eigenvalue-gap features.","marker":"[46]"},{"why":"The quantum resource-estimation code used to compute DF QPE runtime and qubit estimates.","marker":"[21]"},{"why":"DF QPE algorithm and resource-estimation approach for quantum chemistry that the quantum resource estimates are based on.","marker":"[89]"},{"why":"An existing Hamiltonian library that motivates the need for reference energies and detailed performance breakdowns; the new benchmark positions itself against it.","marker":"[5]"}],"fun_headline_variants":["SHCI solves 100% of GSEE benchmark instances","DMRG solves 41%, DF QPE 5%, SHCI 100% on GSEE set","Benchmark reveals SHCI-dominated GSEE landscape with classical bias","GSEE benchmark exposes SHCI-trained dataset bias"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The whole ranking rests on the assumption that the machine-learning estimate of what fraction of the problem space a solver can handle matches how often the solver would actually succeed on real chemistry problems, even though the generated test points are not checked for physical plausibility.","fun_headline_variants_meta":{"raw":{"variants":["SHCI solves 100% of GSEE benchmark instances","DMRG solves 41%, DF QPE 5%, SHCI 100% on GSEE set","Benchmark reveals SHCI-dominated GSEE landscape with classical bias","GSEE benchmark exposes SHCI-trained dataset bias"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000306,"raw_usage":{"total_tokens":1617,"prompt_tokens":797,"completion_tokens":820,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":541,"completion_tokens_details":{"reasoning_tokens":743}},"tokens_in":541,"tokens_out":820,"duration_ms":9208,"temperature":1.0,"reasoning_tokens":743,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T20:15:21.537944+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of the 10,000 synthetic latent points, inverse-transform them into Hamiltonian features, construct actual Hamiltonians with those features, and run optimized SHCI on them; if the solved fraction falls well below 1.0000 — for example near the 65% empirical rate in the paper's table — the near-universal solvability claim would be refuted. A cheaper check: hold out a random subset of the 228 benchmark instances, retrain the classifier without them, and compare predicted solvability with the actually solved fraction.","supporting_citations":[{"cited_title":"isi-usc-edu/pyLIQTR: Release 1.1.1","cited_arxiv_id":null,"evidence_quote":"The quantum resource-estimation code used to compute DF QPE runtime and qubit estimates."}],"review_version":1}