{"id":"447ac7d5-b724-4cc8-8cef-7e16796c8422","arxiv_id":"2509.06489","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"In simulations, a new cross-correlation-only statistic NPMV detects a gravitational-wave background more often than the standard PTA statistic, improving detection probability by about 47% at the 5-sigma threshold.","lead":"This paper develops a new statistical test for detecting the faint gravitational-wave background in pulsar timing data. The test finds real signals more often than the current standard method, providing about a 47% improvement in detection probability at the usual detection threshold.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 47% improvement claim rests on unspecified realistic-model ROC calculations: Sec. 7.1 rules out the analytic CDF, but Sec. 7.3 never states the numerical method, sample counts, or uncertainties.","rationale":"The reader's weakest assumption identifies the same load-bearing gap: Sec. 7.3 does not specify the numerical method behind the realistic ROC curves. I agree this is the central issue. The NPMV construction itself appears mathematically sound: for block-diagonal N, the constrained minimization in Sec. 5 yields QNPMV=QNP-DiagQNP, and the toy-model comparison supports the qualitative ordering NP > NPMV ~ NPCC > DFCC. The 47% number, however, is read off a realistic simulation whose numerical procedure is absent from the text, and Sec. 7.1 explicitly disallows the analytic CDF. The code repository link is genuine supporting evidence and prevents this from being a rejection, but it does not replace a stated estimator, trial counts, or error bars. The paper should specify the ROC algorithm, sample sizes, and confidence intervals before the quantitative claim is accepted. The reader's CONDITIONAL verdict is therefore the right one, and my read does not change it.","tokens_in":15307,"tokens_out":11882,"duration_ms":150799,"concrete_test":"Run the public repository (https://github.com/vhaasteren/robust-ds-figures), identify the routine that generates Fig. 3, and check the numerical method. Then independently recompute the operating points at FAP=2.9e-7: if Monte Carlo, use at least 10^7 signal draws and a null threshold estimated from >=10^9 draws or an importance-sampling tail estimate, and quote binomial confidence intervals. If the code uses a deterministic generalized-chi-square routine, cross-check with an independent Davies/Imhof implementation. The headline is supported only if the NPMV-to-DFCC detection-probability ratio at this FAP is consistent with ~1.47 and inconsistent with 1.0 at, say, 95% confidence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline claim is that NPMV increases detection probability by ~47% over DFCC at FAP=2.9e-7 (Abstract; Sec. 7.3; Fig. 3). The entire quantitative claim rests on the realistic-model ROC curves, but Sec. 7.3 never states how those curves were computed. Sec. 7.1 explicitly says the analytic generalized-chi-square CDF (49)-(50) cannot be used for realistic data because the data are not described by a complex-valued random process; Sec. 7.3 gives no substitute. There is no mention of Monte Carlo, Davies/Imhof, importance sampling, trial counts, or convergence checks. The two relevant detection probabilities (2.7% vs 4.0%) differ by only 1.3 percentage points, so the 47% relative improvement could be within the numerical error of an unvalidated tail approximation. The linked public code is useful evidence of reproducibility, but it is not a substitute for a method statement in the paper, and no uncertainty is attached to the quoted figure.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a new quadratic detection statistic, NPMV, for pulsar timing array searches for a stochastic gravitational-wave background. The statistic is defined as the quadratic form that uses only cross-pulsar correlations and minimizes, under the null hypothesis, the variance of its difference from the Neyman-Pearson (NP) optimal quadratic statistic. For block-diagonal noise, the authors derive a simple closed form: the NPMV filter is the NP filter with its diagonal blocks removed. They also derive a generalized chi-square CDF for complex normal data with an indefinite quadratic form, and use it to compare DFCC, NP, NPCC, and NPMV in a toy model. They then present ROC curves for a more realistic NANOGrav-15-year-like model and report a 47% increase in detection probability for NPMV over DFCC at the 5-sigma FAP of 2.9e-7.","tokens_in":15573,"tokens_out":6866,"duration_ms":77939,"significance":"If correct, the NPMV statistic is a useful methodological contribution: it offers a cross-correlation-only statistic that lies closer to Neyman-Pearson optimality than the currently standard DFCC statistic, while retaining the practical robustness property of excluding autocorrelations from the quadratic form. The derivation of the NPMV filter is clean and is obtained from first principles with no free parameters fitted to the target result. The paper also ships code and uses external NANOGrav parameters as inputs, which are strengths. However, the headline quantitative claim rests on ROC curves whose numerical method is not described, and the central robustness motivation is not tested under noise mismodeling. These gaps need to be addressed before the performance claim can be considered established.","major_comments":[{"comment":"The method used to compute the realistic-model ROC curves is not stated. Section 7.1 explicitly says that the analytic CDF (49)-(50) cannot be used for realistic data because the data are not described by a complex-valued random process, but Section 7.3 only says \"We construct ROC curves\" and gives no substitute: no Monte Carlo sample count, no Davies/Imhof algorithm, no convergence checks. The headline 47% improvement is read off these curves; the two quoted detection probabilities (2.7% vs 4.0%) differ by only 1.3 percentage points, so the result could be within the numerical error of an unvalidated tail approximation. Please specify the numerical procedure and attach uncertainties to the quoted improvement.","section":"§7.3, Fig. 3"},{"comment":"The paper motivates NPMV as \"robust\" to pulsar noise modeling errors, but no mismodeling test is performed. The filter is constructed from N and C via (20)-(21); although the quadratic form uses only off-diagonal data blocks, the filter itself depends on the assumed noise covariance N. The claim that NPMV is \"less affected\" by pulsar noise modeling errors than NP or DFCC is therefore not demonstrated. Please add a quantitative robustness study, e.g. ROC curves under perturbed N for NPMV vs DFCC vs NP, or state explicitly that robustness is only a structural property with no empirical validation.","section":"Abstract; §§1, 4.2"}],"minor_comments":[{"comment":"The sentence about \"the autocorrelations can be reconstructed from the cross-correlations\" for n>=3 is unsupported and seems to conflict with realistic pulsar-specific noise. It is not used later, but as written it is misleading; please clarify or remove.","section":"§3"},{"comment":"Typo: \"modeling erorrs\" should be \"modeling errors\". Also, after Eq. (4), \"expectated value\" should be \"expected value\".","section":"§4.1"},{"comment":"The caption reports AUC=0.975 for both NPCC and NPMV, and equal detection probabilities at the 5-sigma FAP, while the text says NPCC curves lie above those of NPMV. Please state the precision to which the curves differ, or qualify the claim.","section":"§7.2, Fig. 2"},{"comment":"The discussion of catastrophic cancellation is useful, but the paper does not say whether the toy-model CDF evaluations used arbitrary precision or standard IEEE doubles. Please specify, since the NPCC optimization relies on these evaluations.","section":"§6.2"},{"comment":"The entry for Taylor et al. 2021 (enterprise_extensions) lacks a journal, arXiv identifier, or DOI; please complete the reference.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The central derivation appears sound and the toy-model comparison is internally consistent. The blocker is the unspecified numerical method behind the realistic-model ROC curves in Sec. 7.3, which is the sole basis for the 47% headline claim. If the authors supply the computational details and a robustness test under noise mismodeling, the paper could be suitable for publication. The missing-method issue is fixable within the scope of the manuscript, so I do not recommend rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe genuinely new thing here is the NPMV filter: a cross-correlation-only quadratic statistic that is the minimum-variance approximation to the Neyman-Pearson optimum. For block-diagonal noise it is simply the NP filter with diagonal blocks zeroed. That's elegant, and the Lagrange-multiplier derivation in Sec. 5 is clean. The closed-form CDF for indefinite complex quadratic forms in Sec. 6 is also a real contribution to the mathematical toolbox. On the toy model the ROC ordering is exactly what you'd expect: NP top, NPCC and NPMV tied, DFCC last. So the qualitative claim -- NPMV beats DFCC at fixed false-alarm probability -- has solid support.\n\nThe problem is Sec. 7.3, the realistic-model ROC curves. The paper explicitly says the analytic CDF from Sec. 7.1 cannot be used for realistic data, but then never states what replaces it. Monte Carlo? Davies/Imhof? Importance sampling? No trial counts, no convergence checks, no uncertainties. The headline 47% improvement is read off that figure, and the absolute numbers are 2.7% vs 4.0% detection probability -- a gap of 1.3 percentage points. Without knowing the numerical error in the tail, that 47% could be real or could be noise. The code is on GitHub, which is good, but the paper itself should describe the method.\n\nAlso, the word \"robust\" does a lot of work. They argue NPMV is less affected by pulsar noise modeling errors because it ignores autocorrelations, but they never test this under actual mismodeling. That's a plausible argument, not a demonstrated result.\n\nNet: this is a constructive methods paper that deserves serious refereeing. The core NPMV idea and the CDF derivation are worth citing regardless of how the simulation shakes out. But the manuscript needs a revision -- a method statement for Sec. 7.3, uncertainty quantification on the ROC curves, and ideally an explicit noise-mismodeling test. Send it to review.","headline":"The NPMV construction is clean and likely beats DFCC, but the headline 47% improvement rests on ROC curves whose computation is never described.","tokens_in":16032,"tokens_out":2367,"would_cite":true,"duration_ms":26854,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A cross-correlation-only detection statistic, NPMV, that is as close as possible to the Neyman-Pearson optimal test raises pulsar-array GWB detection probability by about 47% at the 5-sigma threshold.","keywords":["gravitational waves","pulsar timing arrays","Neyman-Pearson detection","quadratic detection statistics","Hellings-Downs correlation","stochastic gravitational-wave background","false-alarm probability","receiver operating characteristic"],"falsifier":"Recompute the Sec. 7.3 ROC curves using the published NANOGrav 15-year parameters (A_gw=2.1e-15, gamma=13/3, the quoted pulsar noise parameters) by Monte Carlo: draw many realizations under the null and signal hypotheses, evaluate the DFCC and NPMV statistics, and estimate the detection probabilities at FAP=2.9e-7. If the ratio is not close to 1.47, or the individual detection probabilities (≈2.7% vs ≈4.0%) are not reproduced, the central claim fails.","tokens_in":15216,"feed_emoji":"📡","tokens_out":8327,"duration_ms":80595,"temperature":0.7,"pith_summary":"Pulsar timing arrays currently judge gravitational-wave background claims with a statistic—the 'optimal cross-correlation' (DFCC) statistic—that maximizes signal-to-noise deflection, not detection probability at a fixed false-alarm probability. The truly Neyman-Pearson-optimal test is more powerful but uses autocorrelated pulsar noise, making it vulnerable to noise-modeling errors. This paper constructs a compromise quadratic statistic, NPMV, that uses only cross-correlations yet is as close as possible to the NP-optimal statistic in a minimum-variance sense. In a simulation matching the NANOGrav 15-year noise model and a background with amplitude 2.1e-15 and spectral index 13/3, NPMV raises the detection probability at the 5-sigma false-alarm threshold from about 2.7% to 4.0%—a roughly 47% relative gain over DFCC.","feed_headline":"New statistic boosts pulsar-array GWB detection odds by 47%","feed_subtitle":"At the 5-sigma false-alarm rate, NPMV detects 47% more real signals than the standard DFCC statistic.","key_machinery":"The central object is the quadratic detection statistic D(z|Q)=z†Qz and its filter Q. The NPMV filter is the solution to the constrained problem: minimize Var[D(z|Q)-D(z|Q_NP)] under the null hypothesis subject to Diag Q=0. For the block-diagonal noise covariance used in PTA analyses this solution collapses to Q_NPMV = Q_NP - Diag Q_NP, i.e., the NP-optimal filter with all autocorrelation blocks zeroed. The paper also derives the generalized chi-square CDF for indefinite filters via contour integration (Fourier representation of the step function and Cauchy residues), which is used to build ROC curves in the toy model and to compute p-values.","core_discovery":"For zero-mean complex Gaussian data, the NP-optimal quadratic filter is Q_NP = N^{-1} - C^{-1}. The standard DFCC filter instead removes the block-diagonal (autocorrelation) part from the deflection-optimal filter Q_DF, which maximizes signal-to-noise ratio, not detection probability. The paper proves that when the noise covariance is block-diagonal (the CURN null hypothesis), the quadratic filter that minimizes the null-hypothesis variance of its statistic's difference from the NP statistic, subject to using only cross-correlations, has the simple closed form Q_NPMV = Q_NP - Diag Q_NP. This statistic can be built directly from the whitened DF filter already used in PTA pipelines. ROC compar","pith_inferences":["The minimum-variance projection idea is generic: any detection problem where the likelihood-ratio statistic is unusable because of nuisance-parameter sensitivity could define a 'practically optimal' statistic by projecting the NP filter onto the subspace of robust filters, and the closed-form solution would hold whenever the noise covariance is block-diagonal.","The 47% gain is tied to the specific NANOGrav-like noise realization and GWB parameters; the improvement factor will likely vary with array sensitivity, number of pulsars, and signal amplitude, so the headline number should not be read as a universal sensitivity gain.","The paper's Sec. 7.3 realistic-model ROC curves rely on an unspecified numerical method; until that method is documented, the 47% figure should be treated as a computational result to be verified by independent Monte Carlo."],"forward_implications":["PTA collaborations can adopt NPMV as the detection statistic for FAPs and p-values, replacing DFCC, and obtain higher sensitivity at the same false-alarm rate without opening the analysis to pulsar-noise autocorrelation mismodeling.","Because the NPMV filter is computed from the same whitened DF filter already in existing pipelines, retrofitting current analyses requires only minor code changes; the paper provides an implementation in the enterprise_extensions package.","Reported detection significances for current datasets would change under NPMV: for a real GWB at the modeled amplitude, the same false-alarm threshold is crossed more often, so a claimed '5-sigma' threshold corresponds to a higher true detection rate.","The NPCC statistic—the true constrained NP-optimal cross-correlation statistic—performs slightly better than NPMV, but it requires expensive numerical optimization; NPMV captures most of the available gain in closed form."],"supporting_citations":[{"why":"introduces the cross-correlation detection statistic for PTA stochastic backgrounds, the lineage DFCC descends from.","marker":"Anholm et al. 2009"},{"why":"formulates the standard optimal cross-correlation statistic used by PTA searches, the baseline DFCC that NPMV is compared against.","marker":"Chamberlin et al. 2015"},{"why":"introduces the CURN null hypothesis (N = Diag C) that the paper adopts as H0 throughout.","marker":"Ellis et al. 2013"},{"why":"derives the DF and NP filters and the whitened reduced-basis framework that the paper uses to construct NPMV from the existing DF filter.","marker":"van Haasteren 2025"},{"why":"provides the NANOGrav 15-year pulsar positions and the fixed-gamma noise/GWB parameter values used in the realistic simulation and ROC curves.","marker":"Agazie et al. 2023a"},{"why":"supplies the generalized chi-square CDF methods for quadratic statistics in PTAs that the paper extends to the indefinite-filter case.","marker":"Hazboun et al. 2023"},{"why":"the Isserlis theorem used to reduce the variance of the difference of quadratic forms to Tr(Δ N Δ N), the core of the NPMV derivation.","marker":"Isserlis 1918"},{"why":"derives the optimal cross-correlation filter for Hellings-Downs correlations, providing the detection-statistic framework DFCC sits within.","marker":"Allen & Romano 2023"}],"fun_headline_variants":["Cross-correlation statistic lifts PTA detection odds 47%","Pulsar-array statistic finds 47% more signals at 5-sigma","New PTA detection statistic: 47% better at fixed FAP","Robust cross-stat beats standard for PTA GWB detection","NPMV statistic: 47% boost in pulsar GWB detection"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The 47% improvement is read off ROC curves for the realistic NANOGrav-like model in Sec. 7.3, but the paper does not state how those curves were computed—the analytic generalized chi-square CDF cannot be used for realistic data, and no substitute method is specified—so the headline performance claim is only as reliable as that unspecified numerical procedure.","fun_headline_variants_meta":{"raw":{"variants":["Cross-correlation statistic lifts PTA detection odds 47%","Pulsar-array statistic finds 47% more signals at 5-sigma","New PTA detection statistic: 47% better at fixed FAP","Robust cross-stat beats standard for PTA GWB detection","NPMV statistic: 47% boost in pulsar GWB detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000184,"raw_usage":{"total_tokens":1221,"prompt_tokens":879,"completion_tokens":342,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":623,"completion_tokens_details":{"reasoning_tokens":246}},"tokens_in":623,"tokens_out":342,"duration_ms":4256,"temperature":1.0,"reasoning_tokens":246,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T23:29:25.540794+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the Sec. 7.3 ROC curves using the published NANOGrav 15-year parameters (A_gw=2.1e-15, gamma=13/3, the quoted pulsar noise parameters) by Monte Carlo: draw many realizations under the null and signal hypotheses, evaluate the DFCC and NPMV statistics, and estimate the detection probabilities at FAP=2.9e-7. If the ratio is not close to 1.47, or the individual detection probabilities (≈2.7% vs ≈4.0%) are not reproduced, the central claim fails.","supporting_citations":[{"cited_title":"A., Siemens, X., & van Haasteren, R","cited_arxiv_id":null,"evidence_quote":"introduces the CURN null hypothesis (N = Diag C) that the paper adopts as H0 throughout."}],"review_version":1}