{"id":"3c4c41a5-926f-4b7d-8296-7db1861e5130","arxiv_id":"2505.09090","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A sequential test based on ratios of scoring rules is shown to be a generalized e-value, yielding finite-sample error control for forecast method selection.","lead":"Forecasters can now choose between competing prediction methods using a sequential test that compares cumulative scoring-rule ratios and stops as soon as one method is favored. The paper connects this scheme to e-values, which can give finite-sample error control, and tests it against the standard Diebold-Mariano approach.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Uniform-in-n validity of the e-process claim is unproven: Lemma 2 bounds omega pointwise, while the implemented grid must satisfy omega <= |h_P(n)| for every n, with only an informal heuristic offered.","rationale":"The reader's weakest_assumption correctly identifies the omega-selection problem: the implemented procedure requires omega <= |h_P| without a principled way to guarantee it. I sharpen this into a more load-bearing theoretical gap: the e-process claim itself needs the bound to hold uniformly in n, whereas Lemma 2 proves only a pointwise bound. This is a substantive, concrete weakness in the central claim, not merely an implementation inconvenience. Other issues the reader notes (filtration error in the Lemma 3 proof, boundary-constant arithmetic) are real but repairable or minor; the uniform-omega issue is not shown repairable without new assumptions such as inf_n |h_P(n)| > 0, which the paper never states. Because the paper's own framing is conditional on omega and the simulations do use a fixed grid, the conditional verdict is appropriate; the weakness reinforces the need for revision rather than overturning the contribution entirely. The proposed Monte Carlo check would directly test whether the recommended grid can break the advertised error control in the paper's own Example 1.","tokens_in":24686,"tokens_out":11814,"duration_ms":118688,"concrete_test":"Use Example 1 with rho = 1 (HQ) and log-score. For n in {50, 200, 500, 1000, 5000}, estimate h_P(n), the nonzero root of E[exp(h Delta_n)] = 1, by Monte Carlo (e.g., 100,000 replications). If inf_n |h_P(n)| < 1/4, run the SSRE scheme from Section 5.1 with omega = 1 and k_u = (1-beta)/beta for beta = 1/10 under the same DGP, and record the empirical frequency of selecting P. If the frequency exceeds beta/(1-beta) = 1/9, the advertised error control fails for a grid-valid omega.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central advertised guarantee is that C_n^{(omega)}(Q,P) is an e-process under HQ for any omega in [0, |h_P|], so Ville's inequality bounds finite-sample error probabilities. This requires E[exp(omega Delta_n)] <= 1 for all n simultaneously. Lemma 2, however, establishes this only pointwise in n: h_P is defined in Lemma 1 as the nonzero root of E[exp(h Delta_n)] = 1 for a fixed n, and the proof of Lemma 2 uses convexity at that single n. No argument shows that a single omega <= |h_P(n)| works uniformly over n, and the subscript n in h_P is suppressed throughout Section 4.1. If |h_P(n)| varies with n and inf_n |h_P(n)| < 1/4, then the recommended grid {1/4, 1/2, 1} includes values that are not valid e-process weights at some sample sizes, and the Ville-based error control in Lemma 3 and Theorem 1 is unsupported. The Appendix proof of Lemma 3 has an additional adaptation error (R_n is B_{n+1}-measurable, so C_n is not adapted to B_n) and a mistaken tower-property equality; these can be repaired by shifting the filtration, but the uniform-omega issue cannot be repaired without an extra condition bounding |h_P(n)| away from zero. Section 5.1's training-sample heuristic only checks whether the chosen omega values appear to behave similarly over one finite window; it does not verify the uniform bound, and a finite sample cannot rule out later violation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a sequential scoring-rule evaluation (SSRE) framework for choosing between two forecasting methods. The key statistic is C_n(Q,P)=∏ exp(S(φ[Q],Y)-S(φ[P],Y)), and a boundary-crossing rule on C_n defines the stopping time N. The authors prove (i) a large-deviations/tilting lemma giving a root h_P of the MGF of Δ_n; (ii) termination with probability one and existence of all moments under weak nondegeneracy conditions; (iii) that powered statistics C_n^{(ω)} are generalized universal e-values under a null HQ defined by E_P[R_n|B_n]≤1, with Ville-based finite-sample error control; and (iv) asymptotic power for detecting a false null. A practical protocol chooses ω∈{1/4,1/2,1} by an informal training-sample check, sets β=1/10, and simulates comparisons against Diebold-Mariano tests in three examples.","tokens_in":25081,"tokens_out":10443,"duration_ms":105618,"significance":"If established, this would provide a genuinely sequential, finite-sample alternative to asymptotic forecast-comparison tests, and the termination/moment results under composite hypotheses are of independent interest. The paper is also careful to compare against DM tests in regimes where the DM assumptions fail. However, the headline error-control guarantee is not fully established as stated: the tilting parameter h_P is n-dependent, the proof of Lemma 3 has filtration errors, and the proof of Proposition 1 uses an invalid product inequality. The e-process interpretation is valuable but, under the paper's own definition of HQ, the ω=1 case follows almost by construction; the independent content is the powered/ω-parameterized version and the asymptotic power results.","major_comments":[{"comment":"The proof establishes E_P[exp(ω Δ_n(Q,P))]≤1 only pointwise in n, because h_P in Lemma 1 is the root of E_P[exp(h log C_n)]=1 for that fixed n and is n-dependent. The text suppresses this subscript when asserting that 'for any ω∈[0,|h_P|], C_n^(ω)(Q,P) is an e-process.' An e-process requires a single ω to satisfy the moment inequality for every n simultaneously, and the recommended grid {1/4,1/2,1} in Section 5.1 may contain values exceeding |h_P(n)| at some sample sizes if inf_n |h_P(n)| < 1/4. Since Lemma 3 and Theorem 1 rely on the uniform-in-n e-process property, this gap is load-bearing; a condition such as ω≤inf_{n,P∈HQ}|h_P(n)|, or a genuinely adaptive choice of ω, is needed.","section":"Section 4.1, Lemma 2"},{"comment":"The proof states that each R_n(Q,P) is adapted to B_n and uses E_P[R_n|B_{n-1}]≤1, while Section 3.1 defines HQ through E_P[R_n|B_n]≤1 and R_n involving Y_{n+τ} is B_{n+1}-measurable when τ=1. The displayed tower-property equality also treats conditional expectations of the product as a product of conditional expectations, which is not valid. This is repairable by consistently shifting the filtration or by re-indexing R_n, but as written the supermartingale/e-process argument for Lemma 3 is incorrect.","section":"Appendix, proof of Lemma 3"},{"comment":"The proof asserts Pr(N>n) ≤ ∏_{m=1}^n Pr(k_l < C_m < k_u). For arbitrary events, the probability of an intersection is not bounded above by the product of the marginal probabilities, so this inequality is not available without additional independence or positive-correlation structure. The subsequent Cauchy-product argument depends on this bound, and therefore the claimed exponential tail bound, termination with probability one, and Corollary 1's moment result are not proven as written. A conditional contraction argument or another mechanism is required.","section":"Appendix, proof of Proposition 1"},{"comment":"The text sets β=1/10 and states k_u≈11.11, but the formulas in Section 4.1 and Section 5.1 give k_u=(1−β)/β=9 (and β_Q=β/(1−β)=1/9≈0.111). If the simulations actually used 11.11, the bound controlled by Theorem 1 is not the one advertised, and the reported rejection frequencies in Table 1 should be compared with the correct threshold.","section":"Section 5.2, simulation protocol"}],"minor_comments":[{"comment":"The proof introduces G^(ω)_{n_j}(Q,P) without definition; this appears to be a typo for C^(ω)_{n_j}(Q,P).","section":"Section 4.2, Theorem 2 proof"},{"comment":"The notation for the continuation event alternates between E^{Q∩P}_n and E^{QXP}_n; a single consistent symbol would improve readability.","section":"Section 3.2 and Figure 1"},{"comment":"There are numerous typographical errors in the references and text, including 'Wolforwitz' for Wolfowitz, 'Mathemetical' in the Brown reference, 'Chpman' for Chapman in Wetherill, and 'Role's theorem' for Rolle's theorem.","section":"Throughout"},{"comment":"The informal training-sample check only compares the trajectories of exp(−ω sign(Δ_R) Δ_R) over one finite window; as the authors acknowledge, this cannot verify the uniform moment condition required by Lemma 2, and the paper should state this limitation explicitly in the main text rather than only in the surrounding discussion.","section":"Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"I see no evidence of misconduct or problematic citation practices. The paper's topic fits the journal, but the proof of the central e-process guarantee and the termination proof need substantive repair before publication; the simulations cannot substitute for a correct theorem."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead Frazier and Poskitt's SSRE paper. The new thing is real: they frame cumulative scoring-rule ratios exp(omega Delta_n) as generalized universal e-values and use Ville's inequality to claim finite-sample control of sequential forecast-method selection, plus termination and moment results for the stopping time. That is a useful direction, and the simulations sensibly target settings where DM tests are known to be unreliable (unit root vs AR, forecast combinations). The DM comparisons are fair enough, though no code or data is shipped.\n\nThe soft spots are not cosmetic. The main advertised guarantee—C_n^(omega) is an e-process for any omega in [0, |h_P|]—is only proven pointwise in n. Lemma 2 fixes n and finds h_P(n); the proof gives no reason a single omega works for all n, and the grid {1/4, 1/2, 1} may cross |h_P(n)|. Section 5.1's in-sample \"do the statistics look similar\" heuristic does not check this, and cannot rule out a later violation. Since the error control in Lemma 3 and Theorem 1 rests on the e-process property, this is load-bearing, not a technicality. The authors need an additional condition such as inf_n |h_P(n)| > omega, stated at the hypothesis level, or a different argument.\n\nThere are also proof-level slips. Lemma 3's appendix proof treats C_n as adapted to B_n when R_n uses Y_{n+1}; that can be fixed by shifting indices, but as written the supermartingale step is wrong. Lemma 2 and Theorem 1 repeat the same n-suppression. Proposition 1's proof bounds P(N>n) by the product of marginal probabilities P(kl < C_m < ku) for dependent events; the product bound is not valid in general, so the claimed exponential tail and finite moments need a conditional-probability argument. Finally, with beta = 1/10, k_u = (1 - beta)/beta = 9, but the text says approximately 11.11; trivial to correct.\n\nThe central idea is still worth engaging. The paper would be a useful addition to the forecast-evaluation literature if the uniform-omega condition and the proof of Proposition 1 are fixed. I'd send it to a serious referee, not desk reject it, but I would not rely on the current finite-sample error guarantees as stated until the e-process claim is repaired.","headline":"The SSRE-to-e-value connection is a genuine new idea, but the advertised finite-sample error control is not yet proven because the omega-grid is only validated pointwise while the e-process claim needs a uniform bound.","tokens_in":25556,"tokens_out":4574,"would_cite":false,"duration_ms":50049,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62M10","62M15","62G09"],"pacs":[],"model":"deepseek-v4-flash","headline":"Sequential forecast selection via scoring-rule ratios becomes an anytime-valid e-value test, with finite-sample error control at every stopping time.","keywords":["sequential scoring rule evaluation","e-values","generalized universal e-values","forecast method selection","proper scoring rules","exponential tilting","large deviations","anytime-valid inference"],"falsifier":"In a simple experiment with a known data-generating process where $|h_P|$ can be computed exactly by solving $E_P[\\exp(h\\Delta_n)]=1$, generate data under the null $H_Q$, set $\\omega$ strictly larger than that computed value, and run the SSRE boundary rule. If the empirical rejection rate for falsely selecting method $P$ exceeds the claimed bound $\\beta/(1-\\beta)$ in finite samples while the same experiment with $\\omega<|h_P|$ stays below the bound, the central e-process claim is falsified.","tokens_in":24490,"feed_emoji":"📈","tokens_out":3708,"duration_ms":40783,"temperature":0.7,"pith_summary":"The paper claims that choosing between two forecasting methods can be done sequentially, as data arrive, by watching a product of scoring-rule ratios cross upper and lower boundaries. This sequential scoring rule evaluation (SSRE) scheme is shown to terminate with probability one, to have stopping times with all moments finite, and to control error probabilities in finite samples rather than only asymptotically. The key move is to recast the accumulated score-ratio product as a generalized e-value, so that anytime-valid bounds from Ville's inequality apply. A sympathetic reader would care because it removes the stationarity and weak-dependence assumptions that underpin standard forecast-comparison tests, and it is aimed at method selection rather than merely rejecting a no-inferiority null.","feed_headline":"Score-ratio product makes forecast selection anytime-valid","feed_subtitle":"Sequential test stops with finite-sample error bounds, without the stationarity assumptions that break standard forecast comparisons.","key_machinery":"The load-bearing object is the accumulated score-ratio product $C_n(Q,P)=\\exp\\{\\sum_{m=1}^n [S(\\varphi[Q],Y_m)-S(\\varphi[P],Y_m)]\\}$, combined with an exponential tilting change of measure whose tilting parameter $h_P$ is the nonzero root of $E_P[\\exp(h\\log C_n(Q,P))]=1$. Lemma 2 shows that $C_n^{\\omega}(Q,P)=\\exp(\\omega\\Delta_n(Q,P))$ is an e-process under $H_Q$ for every $\\omega\\in[0,|h_P|]$, and the analogous statistic is an e-process under $H_P$. An e-value is a nonnegative random variable with expectation at most one under the null; an e-process extends this property to every stopping time, so Ville's inequality converts it into a sequential error bound. The same e-process structure is then scaled and averaged, following the e-value false-discovery-rate construction, to control errors across repeated comparisons.","core_discovery":"The central claim is that the SSRE process $C_n(Q,P)=\\prod_{m=1}^n \\exp\\{S(\\varphi[Q],Y_m)-S(\\varphi[P],Y_m)\\}$, with boundaries $k_l$ and $k_u$, behaves like a sequential probability ratio test even though it is not a likelihood ratio. Under the null that method $Q$ is at least as good as $P$, each multiplicative factor is an e-variable, and the product is an e-process, so $\\Pr(\\sup_{n\\le T} C_n^{\\omega}(Q,P) \\ge 1/\\pi)\\le \\pi$ for every $\\omega\\in[0,|h_P|]$, where $h_P$ is the tilting parameter that solves the moment generating function equation. This gives finite-sample control over the probability of wrongly selecting method $P$ when $Q$ is better, and, with probability converging to one, the procedure eventually selects the better method when the alternative is correct. The paper also proves the SSRE stopping time is finite almost surely and has an exponentially decaying tail.","pith_inferences":["A direct extension, not pursued in the paper, is to use the same e-process to construct a sequential test of equal predictive ability rather than a two-way selection rule, with the indifference region $k_l < C_n < k_u$ interpreted as a zone of practical equivalence.","The averaging over several $\\omega$ values suggests a calibration recipe one could test empirically: estimate $|h_P|$ from the training sample by solving the moment generating equation, then verify that all chosen $\\omega$ lie below the estimate; the paper's informal visual comparison over the training set is a lighter-weight version of this.","The e-process property may allow a betting-style interpretation: the score-ratio product is a wealth process in a game where the forecaster wagers on method $Q$ versus method $P$, which could connect SSRE to portfolio and prediction-market formulations of forecast evaluation.","One could stress-test the method by deliberately choosing $\\omega>|h_P|$ in a simulation to confirm the advertised error control fails exactly when the key inequality is violated."],"forward_implications":["A practitioner can compare two forecasting methods in real time and stop as soon as the accumulated scoring-rule ratio crosses a prechosen boundary, with error probabilities bounded in finite samples rather than relying on asymptotic normality.","The procedure terminates almost surely and has finite stopping-time moments, so expected stopping time can be budgeted in advance using Markov's inequality.","Because the construction needs no stationarity or weak-dependence assumption on the score differences, it applies exactly in settings where Diebold-Mariano style tests are known to fail, such as estimated models and nonstationary series.","The e-value formulation allows false-discovery-rate control across multiple hypothesis tests of forecast accuracy, via ordered and scaled e-values, without requiring independence among the tests.","The framework naturally extends beyond a single benchmark versus a single alternative; the authors note it should support construction of sequential model confidence sets.","The authors conclude that the exponential tilting structure can be represented as a universal generalized e-value, tying forecast selection to the wider anytime-valid inference literature."],"supporting_citations":[{"why":"Supplies the sequential probability ratio test template, the boundary-crossing decision rule, and the moment generating function lemma that the paper generalizes to scoring-rule ratios.","marker":"Wald (1947)"},{"why":"Defines proper scoring rules and the framework for measuring forecast accuracy that the SSRE product is built from.","marker":"Gneiting and Raftery (2007)"},{"why":"Provides the notion of a forecasting method as a model plus estimation and tuning choices, and the conditional predictive ability testing perspective the paper adapts.","marker":"Giacomini and White (2006)"},{"why":"Supplies the standard forecast-comparison test used as the baseline in the simulation studies.","marker":"Diebold and Mariano (1995)"},{"why":"Formalizes e-values and their combination and calibration properties, which are used in the SSRE testing procedure.","marker":"Vovk and Wang (2021)"},{"why":"Supplies the ordered and scaled e-value construction that the paper uses to control false discovery rates across repeated SSRE tests.","marker":"Wang and Ramdas (2022)"},{"why":"Introduces generalized universal e-values for risk minimizers, which the paper connects to the SSRE statistic.","marker":"Dey, Martin and Williams (2024a)"},{"why":"Extends generalized universal e-values to multiple testing, supporting the paper's error-control claims.","marker":"Dey, Martin and Williams (2024b)"},{"why":"Provides the textbook treatment of SPRT boundary values and operating characteristics that Proposition 2 parallels.","marker":"Wetherill (1975)"},{"why":"Supplies the ARMA(2,1) parameter configurations used in the simulation example where two forecasting models are nearly equivalent.","marker":"Poskitt (1987)"}],"fun_headline_variants":["Sequential scoring rules pick better forecasters","E-values turn forecast selection into a sequential test","Stopping-time guarantee for forecast method choice","Scoring-rule e-process makes forecast selection anytime-valid","Fast and error-controlled forecast method selection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The procedure requires the user to pick values of $\\omega$ that do not exceed the unknown tilting parameter $|h_P|$; if $\\omega>|h_P|$, the e-process property fails and the advertised finite-sample error control does not hold.","fun_headline_variants_meta":{"raw":{"variants":["Sequential scoring rules pick better forecasters","E-values turn forecast selection into a sequential test","Stopping-time guarantee for forecast method choice","Scoring-rule e-process makes forecast selection anytime-valid","Fast and error-controlled forecast method selection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000816,"raw_usage":{"total_tokens":3580,"prompt_tokens":957,"completion_tokens":2623,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":573,"completion_tokens_details":{"reasoning_tokens":2567}},"tokens_in":573,"tokens_out":2623,"duration_ms":20201,"temperature":1.0,"reasoning_tokens":2567,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:40:35.989290+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"In a simple experiment with a known data-generating process where $|h_P|$ can be computed exactly by solving $E_P[\\exp(h\\Delta_n)]=1$, generate data under the null $H_Q$, set $\\omega$ strictly larger than that computed value, and run the SSRE boundary rule. If the empirical rejection rate for falsely selecting method $P$ exceeds the claimed bound $\\beta/(1-\\beta)$ in finite samples while the same experiment with $\\omega<|h_P|$ stays below the bound, the central e-process claim is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the sequential probability ratio test template, the boundary-crossing decision rule, and the moment generating function lemma that the paper generalizes to scoring-rule ratios."},{"cited_title":"White , Halbert H","cited_arxiv_id":null,"evidence_quote":"Provides the notion of a forecasting method as a model plus estimation and tuning choices, and the conditional predictive ability testing perspective the paper adapts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the standard forecast-comparison test used as the baseline in the simulation studies."},{"cited_title":"Wang , Ruodu R","cited_arxiv_id":null,"evidence_quote":"Formalizes e-values and their combination and calibration properties, which are used in the SSRE testing procedure."},{"cited_title":"Ramdas , Aaditya A","cited_arxiv_id":null,"evidence_quote":"Supplies the ordered and scaled e-value construction that the paper uses to control false discovery rates across repeated SSRE tests."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the textbook treatment of SPRT boundary values and operating characteristics that Proposition 2 parallels."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the ARMA(2,1) parameter configurations used in the simulation example where two forecasting models are nearly equivalent."}],"review_version":1}