{"id":"38b9746b-d923-495a-a160-16d2bbde4c95","arxiv_id":"2411.16412","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"ACTest automatically generates synthetic spectral functions and Green's functions to create reproducible benchmarks for analytic continuation methods, with a built-in 100-case standard dataset.","lead":"ACTest is an open-source Julia toolkit that generates synthetic spectral functions and their corresponding Green's functions to benchmark analytic continuation methods. It ships a standard 100-case dataset (ACT100) and is integrated with the ACFlow toolkit, so multiple methods can be tested automatically.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark noise model uses unrealistically low noise and nonphysical complex-valued Gaussian noise for real G(τ), so ACT100 results may not transfer to realistic QMC data.","rationale":"I agree with the reader that the complex-noise model in Eq. (28) is the weakest assumption, and I also share the reader's reservations about the loose pass threshold and the demonstrational nature of §4. However, I would weight the analysis differently: the most damaging gap is not Eq. (28) alone, but the combination of a nonphysical complex noise model and the benchmark's extremely low noise level (1e-6). That combination means the reported MaxEnt pass rates and error distributions say almost nothing about performance on realistic QMC data, which is the paper's stated motivation ('to mimic input data from realistic quantum many-body calculations'). The reader's verdict (CONDITIONAL) remains appropriate: the toolkit is real, open-source, and sound as a synthetic-data generator; the central benchmarking claim is not yet established. One should not REJECT because the paper's architecture—modular scripts, reproducible ACT100 parameters in actest/src/dataset.jl, and the integration with ACFlow—is verifiable and likely fixable. The concrete test I propose directly settles whether the low-noise/complex-noise choices change the benchmark conclusions, which is the load-bearing question.","tokens_in":17265,"tokens_out":1867,"duration_ms":15982,"concrete_test":"Regenerate the §4.2 benchmark at noise = 1e-3, 1e-4, and 1e-5 for both G(τ) (real noise only, drawn as δ·N(0,1) on the real part) and G(iωn) (complex noise), with all other settings fixed, and tabulate the pass rate and median Err of Eq. (29) per scenario. If the pass rate drops materially (e.g., from ~80% to below ~50%) as noise increases from 1e-6 to 1e-4, then the reported baseline benchmark is not representative of realistic data; additionally, recompute the pass-rate statistics using a stricter threshold such as Err ≤ 0.2 to quantify how loose Eq. (30) is.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that ACTest can generate benchmarks whose results assess real analytic continuation methods. Two load-bearing assumptions sit in §2.6–§2.7 and §4.2. First, Eq. (28) adds complex-valued Gaussian noise N_C(0,1) to G(τ), yet physical imaginary-time Green's functions are real; the imaginary component is nonphysical and changes the analytic structure the continuation method sees. Second, the only benchmark shown (§4.2) uses noise = 1e-6, which is far below realistic QMC error floors (typically 1e-4 to 1e-2 for many applications), so the reported ~80% pass rate and error statistics are not evidence about realistic noisy data. The pass criterion in Eq. (30) also counts Err < 1 as pass, which is an extremely loose threshold: an Acalc that is identically the zero function gives Err = 1 and passes nothing, but Acalc with 50% L1 error passes; the benchmark therefore does not discriminate between good and poor reconstructions. These issues do not invalidate ACTest as a synthetic-data generator, but they undermine the paper's implied claim that ACT100 results rank or validate methods for real QMC inputs.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces ACTest, a Julia toolkit for generating benchmark datasets for analytic continuation problems. Spectral functions are built as superpositions of randomly parameterized peaks (Gaussian, Lorentzian, delta-like, rectangular, and rise-and-decay), and exact Green's functions are computed on imaginary-time or Matsubara-frequency grids using fermionic, bosonic, or symmetric-bosonic kernels. Users can add noise, generate arbitrarily large datasets, or use the built-in 100-case ACT100 set. The toolkit is interfaced with ACFlow and includes scripts for running MaxEnt and other solvers, comparing reconstructed spectra to true spectra, and reporting error statistics. The paper's central claim is that ACTest provides a fair, reproducible, quantitative basis for comparing analytic continuation methods and codes.","tokens_in":17510,"tokens_out":5334,"duration_ms":53495,"significance":"If the toolkit is used as described, it addresses a genuine gap: analytic continuation methods are often tested on hand-picked examples without a shared standard. The kernel formulas in Eqs. (10)-(27) are standard and appear correctly stated, the ACT100 dataset is reproducible, and the ACFlow integration enables immediate use. The open-source availability, modular design, and built-in dataset are concrete strengths. However, the manuscript's own benchmark validation is limited by the noise model in Eq. (28), the very low noise level and small grid used in the Section 4.2 example, and the loose pass criterion in Eq. (30). The paper therefore establishes ACTest more convincingly as a generator of exact synthetic data than as a simulator of realistic QMC inputs.","major_comments":[{"comment":"The noise model G_noisy = G_exact[1 + δ N_C(0,1)] uses complex-valued Gaussian noise for all grid types. Imaginary-time Green's functions G(τ) for a Hermitian Hamiltonian are real; adding a complex random component is nonphysical and changes the analytic continuation problem in a way that is not representative of QMC data. Even for Matsubara data, the covariance of real and imaginary parts is not that of typical QMC estimators. In addition, the multiplicative form makes the noise singular wherever G_exact crosses zero, which can occur for off-diagonal correlators. Please replace or supplement Eq. (28) with grid-appropriate noise models (real noise for G(τ); realistic covariance for G(iωn)) and document the choice.","section":"§2.6, Eq. (28)"},{"comment":"The only reported benchmark uses ngrid = 10 Matsubara points and noise = 1e-6. Ten points is far fewer than typical QMC imaginary-time or Matsubara grids, and 1e-6 is well below realistic noise floors, which are often in the 10^-4 to 10^-2 range. The statement that changing the computational configurations would lead to similar conclusions is not supported by any experiment in the paper. As a result, the reported pass rates and timings do not demonstrate how ACTest-based benchmarks reflect realistic data. Please add at least one benchmark at a realistic noise level and grid size, or explicitly restrict the claims to near-exact synthetic data.","section":"§4.2, Fig. 3"},{"comment":"The pass criterion θ(1 - Err) classifies any reconstruction with normalized L1 error below 1 as a pass. Since Err is normalized by ||A_true||_1 via Eq. (29), a reconstruction with 99% error passes, and a trivial zero reconstruction sits exactly at Err = 1. This makes 'pass rate' a very loose, low-discrimination metric, and the reported ~80% pass rates are hard to interpret. The continuous error distributions shown in the left panel of Fig. 3 are more informative; please report those as the primary accuracy measure and define a stricter success threshold, or justify why Err < 1 is a meaningful passing criterion.","section":"§2.7, Eq. (30)"}],"minor_comments":[{"comment":"The typeset equation appears to contain a duplicated bracket, so it is unclear whether the factor (1 - Err) is squared or whether the second factor is a typographical artifact.","section":"§2.7, Eq. (30)"},{"comment":"The sentence 'The spirit of the maximum entropy entropy' contains a duplicated word 'entropy'.","section":"§4.2"},{"comment":"The comment lines contain the typo 'Typer' and several oddly spaced words; please proofread the input-file examples.","section":"§4.2, [Solver] block"},{"comment":"The parameter name 'offdiag' is used to switch between positive-definite and non-positive-definite spectra, but non-positive-definite spectra are not necessarily off-diagonal Green's functions; a comment or a clearer name would avoid confusion.","section":"§3.5, offdiag parameter"},{"comment":"The Program Summary still contains placeholders such as 'to be added by Technical Editor,' and the Data Availability statement says data will be provided on request even though the repository is public; please reconcile these statements.","section":"Program Summary and Data Availability"}],"recommendation":"major_revision","confidential_remarks":"The toolkit appears genuinely useful for the analytic-continuation community, and the core mathematical formulation is sound. My major concerns are about whether the benchmark protocol actually represents realistic QMC inputs: the noise model, the example's noise level and grid size, and the pass metric all affect the interpretation of the reported results. These are fixable with additional experiments and a revised noise/criterion design, so I do not see grounds for rejection, but the paper should not be accepted in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"ACTest fills a real gap: it's the first open toolkit specifically for benchmarking analytic continuation codes, with a fixed ACT100 dataset and a GitHub repo that actually runs. The core math is standard Lehmann-representation kernels, correctly stated in Eqs. (10)-(27), and the architecture (random peak superpositions, multiple grid/mesh types, fermionic and bosonic sectors) is sensible. The paper earns credit for shipping code and a reproducible dataset, and the integration with ACFlow makes it immediately useful. The central claim, that ACTest can generate synthetic spectra and matching Green's functions, is independently checkable and holds up. The soft spots are real but not fatal. Eq. (28) adds complex Gaussian noise to G(τ), which is unphysical because imaginary-time Green's functions are real; this changes the analytic structure the continuation method sees. The benchmark example uses noise = 1e-6, far below typical QMC error floors, so the ~80% pass rates are at best indicative of behavior in the nearly noiseless regime. The pass criterion in Eq. (30) is very loose: Err < 1 accepts a reconstruction with 50% L1 error, so the reported pass rates don't discriminate between good and poor results. Also, the example uses only 10 Matsubara points, which is light for a benchmark. The concluding remarks drift into speculation about what ACTest could do if extended, which should be trimmed. None of this undermines the toolkit as a data generator. But the paper's implied claim that ACT100 results can rank or validate methods on realistic QMC inputs is not yet supported. The author should add a realistic-noise example (δ ~ 1e-3 to 1e-2), fix the noise model for imaginary-time data (real-valued noise for G(τ)), tighten or justify the pass threshold, and explicitly note that the benchmark runs are a demonstration, not a full validation. The self-evaluation aspect (ACTest benchmarking the author's own ACFlow) should be acknowledged rather than buried. Who is this for? People developing or using analytic continuation methods, especially in quantum many-body; also machine-learning groups looking for training data. It's a serious contribution that deserves referee time. I'd take it for review, with revisions along the lines above. I'd also cite it if I were benchmarking a method.","headline":"A genuinely useful benchmarking toolkit for analytic continuation with a solid core, but the noise model, pass criterion, and demo-style example need tightening before the benchmarks can be trusted for realistic QMC data.","tokens_in":712,"tokens_out":918,"would_cite":true,"duration_ms":21090,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ACTest builds benchmark datasets with known exact spectra for analytic continuation methods.","keywords":["analytic continuation","spectral function","Green's function","maximum entropy method","quantum Monte Carlo","benchmark dataset","Julia toolkit","ACT100"],"falsifier":"Compare method rankings obtained on ACTest-generated data with rankings obtained on a set of small exactly solvable systems, such as a few-site Hubbard model solved by exact diagonalization where the exact spectral function is known; if the relative errors of the methods differ substantially at matched noise levels, the synthetic noise model does not fully capture what real data demand.","tokens_in":17024,"feed_emoji":"🧪","tokens_out":6272,"duration_ms":55974,"temperature":0.7,"pith_summary":"ACTest sets out to solve a practical problem in quantum many-body physics: analytic continuation methods that turn imaginary-time or Matsubara-frequency Green's functions into real-frequency spectral functions are numerous, but there has been no standard, quantitative way to compare them. The toolkit generates many synthetic spectral functions A(ω) from random superpositions of parameterized peaks, computes the corresponding exact Green's functions through Laplace transformation, and can add artificial noise to mimic quantum Monte Carlo data. Because the exact spectrum is known for every generated case, any analytic continuation method can be scored by the error of its reconstruction. ACTest ships a built-in standard dataset, ACT100, with 100 reproducible test cases covering fermionic and bosonic systems, positive and non-positive definite spectra, and various peak types. The paper also demonstrates the workflow by benchmarking the maximum entropy method as implemented in the ACFlow toolkit.","feed_headline":"ACTest gives analytic continuation a known-answer benchmark","feed_subtitle":"One hundred reproducible test cases let any method be scored against exact spectra.","key_machinery":"The load-bearing mechanism is the pair of exact spectrum and exact Green's function manufactured from parameterized peaks. Each spectral function is a sum A(ω)=Σ p_i(ω) of Gaussian, Lorentzian, narrow-Gaussian (δ-like), rectangular, or rise-and-decay peaks with random parameters; the Green's function follows exactly via the Laplace transform G(x)=∫dω K(x,ω)A(ω), with fermionic, bosonic, and symmetric-bosonic kernels on either the imaginary-time or Matsubara axis. The noise model G_noisy = G_exact[1+δN_C(0,1)] adds controlled complex Gaussian noise to mimic quantum Monte Carlo data, and the error metric Err = ∫|A_true − A_calc| / ∫|A_true| together with the pass-rate statistic f scores a method per test case. The included ACT100 dataset fixes 100 such pairs with predefined parameters so results are reproducible.","core_discovery":"The central claim is that a standard, reproducible benchmark for analytic continuation is feasible by construction: instead of relying on unknown real spectra, one builds the spectrum first as a sum of few randomly parameterized peaks (Gaussian, Lorentzian, δ-like, rectangular, or rise-and-decay), then obtains the exact Green's function by evaluating the Laplace-transform kernel K(x, ω) for the chosen imaginary-time or Matsubara grid. The exact A(ω) serves as ground truth, so an analytic continuation code can be scored objectively against it. The paper argues this enables a fair comparison of methods such as maximum entropy, stochastic analytic continuation, Nevanlinna continuation, and others, and it provides the ACT100 dataset to do so out of the box, together with integration with the ACFlow toolkit. Benchmark results on ACT100 for the maximum entropy method illustrate the workflow.","pith_inferences":["The synthetic noise in Eq. (28) is uncorrelated, complex, and multiplicative; if real quantum Monte Carlo noise is correlated or has different statistics, ranks obtained on ACTest cases may not fully transfer to real data, and injecting correlated or real-QMC noise would be a natural stress test.","The pass-rate statistic f only separates methods with Err above and below 1, so a stricter error threshold or a rank-based metric would better discriminate among high-accuracy methods on ACT100.","Because ACTest generates data from the same kernel and peak families that some solvers assume, methods that implicitly favor Gaussian-like features could look better on ACT100 than on out-of-distribution real spectra; cross-validation with a different spectral family would test for such bias."],"forward_implications":["Any analytic continuation method or code can be scored on the same 100 ACT100 cases, making comparisons between methods quantitative and reproducible across groups.","Users can generate arbitrarily large datasets with user-selected peak types, grids, noise levels, and spectral-sign constraints, which can directly serve as training and validation data for machine-learning analytic continuation methods.","The benchmark workflow exposes where a method struggles: varying lpeak, mesh types, and offdiag settings can reveal whether errors are driven by peak sharpness, non-linear grids, or sign changes in the spectrum.","The demonstration with maximum entropy on 100-case batches shows that, at noise 10^-6, diagonal fermionic and bosonic cases pass about 80% of the time while off-diagonal cases pass near 100%, despite off-diagonal errors being slightly larger, and the toolkit reports error, pass rate, and runtime for every test."],"supporting_citations":[{"why":"Supplies the artificial noise model G_noisy = G_exact[1+δN_C(0,1)] used to simulate quantum Monte Carlo noise.","marker":"[33]"},{"why":"Supplies the error metric Err(A_true, A_calc) that scores the accuracy of a reconstructed spectrum.","marker":"[36]"},{"why":"The ACFlow toolkit, which ACTest invokes to run analytic continuation solvers such as MaxEnt for the benchmark workflow.","marker":"[44]"},{"why":"Provides the chi2kink algorithm used to select the maximum entropy hyperparameter α in the example benchmarks.","marker":"[45]"},{"why":"The maximum entropy method references that define the solver being benchmarked on the ACT100 dataset.","marker":"[11–14]"}],"fun_headline_variants":["Known-answer benchmark puts analytic continuation to the test","ACTest scores any continuation method against exact spectra","100 cases with known spectra: ACTest benchmarks analytic continuation","Objective scoring for analytic continuation: ACTest delivers known answers","ACTest: benchmark analytic continuation with ground-truth spectra"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmarks inherit the assumption that adding uncorrelated, complex-valued Gaussian noise to an exact Green's function reproduces the noise found in real quantum Monte Carlo data, which is often correlated and, for imaginary-time data, real-valued.","fun_headline_variants_meta":{"raw":{"variants":["Known-answer benchmark puts analytic continuation to the test","ACTest scores any continuation method against exact spectra","100 cases with known spectra: ACTest benchmarks analytic continuation","Objective scoring for analytic continuation: ACTest delivers known answers","ACTest: benchmark analytic continuation with ground-truth spectra"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000158,"raw_usage":{"total_tokens":1235,"prompt_tokens":962,"completion_tokens":273,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":578,"completion_tokens_details":{"reasoning_tokens":195}},"tokens_in":578,"tokens_out":273,"duration_ms":2977,"temperature":1.0,"reasoning_tokens":195,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:08:08.620128+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare method rankings obtained on ACTest-generated data with rankings obtained on a set of small exactly solvable systems, such as a few-site Hubbard model solved by exact diagonalization where the exact spectral function is known; if the relative errors of the methods differ substantially at matched noise levels, the synthetic noise model does not fully capture what real data demand.","supporting_citations":[],"review_version":1}