Pith. sign in

REVIEW 4 major objections 5 minor 13 references

Assessing Extrapolation of Peaks Over Thresholds with Martingale Testing

T0 review · 4 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a gambler betting on the sign of discrepancies between model-simulated and observed top order statistics yields an agnostic score of extrapolation power, and that choosing the Peaks-Over-Threshold quantile that minimi

desk verdict A genuinely new idea—betting-game wealth on top order statistics to choose a POT threshold— wrapped in a winning competition write-up, but the martingale test as stated is not valid and the extrapolation-transfer claim is asserted, not shown. read the letter →

arxiv 2512.03116 v3 pith:5S5CIF4V submitted 2025-12-02 stat.ME stat.ML

Joseph de Vilmarest , Olivier Wintenberger (LPSM (UMR_8001)) This is my paper · ORCID
classification stat.MEstat.ML MSC 62G3260G4262L10
keywords PeaksOverThresholdextremevalueextrapolationmartingaletestingbybettingselectionEVA2025DataChallengetoporderstatisticsprecipitationextremes
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that threshold selection in Peaks Over Threshold extreme-value modeling—normally a matter of expert judgment—can be automated by treating extrapolation as a sequential betting game. A gambler bets on the sign of the error between the model's largest simulated order statistics and the observed ones; under a correct model, the gambler's capital is a martingale, and a wealth that grows means the model predicts the tail wrongly. The authors select the quantile level p that minimizes the final wealth, claiming this agnostic score chose the high thresholds that won the EVA2025 data challenge. If they are right, the key obstacle to routine extreme-value extrapolation—where to put the threshold—has a principled, assumption-light answer.

What carries the argument

The capital process $$W_k = \prod_{i=0}^k \left(1 + (\gamma_{1,k} - 0.5)(\tilde{Y}_{(K-i)} - Y_{(K-i)})\right)$$, where γ is the exponential-weighting betting fraction; it is a martingale under the null that the model's and the data's top order statistics have the same conditional expectations, so wealth grows only when the model systematically over- or under-predicts the largest observations. Testing by betting converts extrapolation checking into a game on the K largest order statistics; the method is called 'agnostic' because it does not specify an alternative hypothesis.

What would settle it

Take a large climate-model dataset, hold out 46 of 50 runs, fit the POT model on 4 runs, and compute the terminal wealth at K=3 for the candidate p-grid; if the p with minimal wealth does not give frequency estimates closer to the 50-run truth than, say, the median grid p, the score isn't selecting extrapolation power. A cheaper test: on Target 1, where the true frequency is now known to be 0.24, verify that the K=3 wealth-minimizing level (p≈0.995 or 0.9) beats the Q-Q-chosen p=0.999.

Watch

Extended reading notes

Core claim

The central claim is that the terminal wealth of an exponentially weighted betting strategy on the signs of discrepancies between model-sampled and observed top order statistics is a valid agnostic measure of an extrapolation model's quality, and that minimizing this wealth over the grid of candidate quantile levels selects a threshold whose extrapolations fare well. In the EVA2025 challenge, this procedure selected p=0.9995 for targets 2 and 3, which the authors state contributed to winning; for target 1, the score pointed to lower levels, but the authors overrode it based on Q-Q plots, a choice they later report was wrong.

Load-bearing premise

The method assumes that a three-round betting game on a single simulated sample of the largest order statistics reveals whether the model will extrapolate correctly beyond the observed range, and the power guarantees assume bounded prediction errors that the authors never verify for extreme precipitation.

Editorial extensions

If this is right

  • If true, threshold selection in Peaks Over Threshold can be based on an automated score instead of Q-Q plots or expert heuristics.
  • The score picks very high quantiles (e.g., 0.9995) where only a few hundred exceedances remain, enabling extrapolation from sparse data.
  • The approach extends to any generative model of extremes, not just exponential POT, provided top order statistics can be simulated.
  • The authors' Target 1 error shows the score can be more reliable than graphical diagnostics, because the Gumbel bias distorts Q-Q comparisons.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The betting score could be repurposed as a general validation layer for AI or machine-learning extreme-value generators, letting a nonparametric test decide whether a fitted generator extrapolates.
  • The choice of K=3 is admitted arbitrary and the scores are sensitive to K; a stability-based heuristic (e.g., choose p that is optimal across several K) is a natural testable extension.
  • The terminal wealth has no calibrated interpretation; converting it into an e-value or a valid p-value would give error control and could let the method replace subjective threshold choice in operational settings.
  • Since the paper's power guarantees require bounded errors and conditional variance bounds never verified for precipitation data, a diagnostic check of these bounds on exceedances would tell when the guarantees actually apply.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper describes the winning strategy for the EVA2025 Data Challenge, where the task was to estimate frequencies of extreme precipitation events from four climate-model runs. The authors reduce each target to a univariate variable, fit a Peaks Over Threshold model with exponential exceedances after seasonal adjustment, and use a martingale testing-by-betting procedure to select the POT quantile level p. The claimed novelty is that the terminal wealth of an adversarial gambler betting on the sign of the prediction error between model-simulated and observed top order statistics is an agnostic score of extrapolation power, and that minimizing this wealth selects the threshold. The paper reports the resulting estimates, discusses the final choices, and includes a post-competition assessment showing that the martingale-selected thresholds would have improved Target 1.

Significance. If the central claim is valid, the paper offers a principled, data-driven way to choose the POT threshold in extreme value analysis, replacing expert judgment with an agnostic sequential-testing score. The EVA2025 win is concrete external evidence that the overall pipeline performs well in a realistic extrapolation task, and the post-hoc Target 1 analysis is a useful case study. However, the theoretical core is thin: the martingale property is not established for the actual wealth process used, the imported power guarantees rely on unverified boundedness assumptions, and the transfer from observed top order statistics to unobservable extrapolation steps is asserted rather than proven. The paper itself acknowledges several of these limitations, which is honest but does not remove their load-bearing status for the novelty claim.

major comments (4)
  1. [§3.2] The wealth process W_k^C is defined as a product of factors 1 + (γ_{1,k} − 0.5)(\tilde{Y}_{(K−i)} − Y_{(K−i)}). This is a valid nonnegative martingale only if the factors remain nonnegative. For γ = 0 or γ = 1, a prediction error outside [−2, 2] makes the factor negative; the EWA weights can then leave [0,1], and log(W_k^C) can become undefined. No bound or normalization is reported for precipitation order statistics on the Leadbetter scale, and the data include values above 5.7. Since the terminal wealth W_{K−1}^C is the score used to select p, this invalidates the martingale-testing interpretation of the selection criterion.
  2. [§3.3] The power guarantees are imported from reference [10] under assumptions |\tilde{Y} − Y| ≤ 1 and E[(\tilde{Y} − Y)^2 | F_k] ≤ m/4. The paper states 'These results are valid in our non-standard game' without proving that these conditions hold for the order-statistics game with exponential-tail models and observed precipitation extremes. Since [10] is a preprint co-authored by one of the present authors, this is not independent support. The O(log(1/α)/m) rate and Equation (3) therefore do not currently apply to the procedure as implemented.
  3. [§3.1] The load-bearing transfer assumption — that coherence of the top K−1 order statistics assesses extrapolation at unfeasible steps k ≥ K — is asserted without derivation. The implementation makes the assumption fragile: K = 3 is chosen 'arbitrarily' (§4.2), the scores are 'very sensitive to the choice of K' (§4.1), and the game is run on a single simulated sample of top order statistics. No experiment or argument shows that minimizing wealth at round K−1 selects thresholds that extrapolate better beyond the observed range. This is the central novelty, and it remains unsupported.
  4. [§4.1 and §5] The paper explicitly validates the model on the training set: 'we validate the model on the training set due to the lack of top order statistics at the Targets level' (§4.1). An in-sample self-coherence check with one realization and an arbitrary K does not make the threshold selection agnostic. The conclusion concedes that the method is 'lacking rigorous guarantees' and that 'the literature lacks clear interpretations of the terminal wealth value, particularly for tests that should be rejected at unfeasible steps k larger than the deterministic stopping time K − 1.' These admissions are consistent with the referee's reading that the statistical claim is not yet established.
minor comments (5)
  1. [§4.2] The text refers to 'using (7)' but no Equation (7) appears in the manuscript; the numbering appears to be missing or broken.
  2. [References] Several Springer references list '???' as the place of publication; this needs completion.
  3. [§4.1] Figure 7 shows scores for K=3 and K=5, but the text states these are 'very sensitive' without quantifying the sensitivity or providing any uncertainty assessment. Since only one simulation is used, the figure should at least be described as a single realization, not a stable score surface.
  4. [§2.4] The display '1\tilde Y^{(3)} = ...' appears to contain a typo or formatting artifact; the meaning of the leading '1' is unclear.
  5. [§1 and §5] The paper alternates between 'EV A' and 'EVA' for the conference/challenge; please standardize the spelling.

Circularity Check

2 steps flagged · score 4.0 of 10

Threshold selection reduces to in-sample self-coherence; power guarantees rest on an unverified self-citation.

  1. self definitional [§3.1; implemented in §4.1]
    "The extrapolation power of the model ˜Y for unfeasible steps (k ≥ K) is assessed by the coherence of the top order statistics over the first K − 1 rounds."

    Extrapolation for unfeasible steps k ≥ K is never tested directly; it is defined as the coherence of the first K−1 rounds. The terminal wealth W^C_{K−1} is built from exactly those K−1 rounds using the same 4 runs on which the POT model was fitted. Selecting p by minimizing W is therefore, by construction, selecting the threshold whose model is most self-coherent on the training sample. The paper's own §4.1 admission, 'we validate the model on the training set', confirms that no out-of-sample extrapolation is involved, so the claimed 'extrapolation power' assessment is a definitional relabeling of an in-sample goodness-of-fit check.

  2. self citation load bearing [§3.3]
    "They established power guarantees of the form: E[τ C α ] = O(log(1/α)/m), under H1 ... These results are valid in our non-standard game, so martingale testing can be used to agnostically test the coherence of top order statistics from a generative model."

    The power guarantee that justifies using the betting score as an extrapolation test is imported from reference [10], Durand & Wintenberger, a preprint co-authored by the present paper's second author. It is not proved here, not machine-checked, and its conditions (|error| ≤ 1 and conditional variance ≤ m/4) are not verified for precipitation order statistics on the Leadbetter scale. The sentence 'These results are valid in our non-standard game' is asserted rather than derived, so the theoretical support for the central novelty reduces to an unverified self-citation. The conclusion's own admission that the approach is 'lacking rigorous guarantees' further confirms the imported guarantee is not independently established.

full rationale

The paper's central selection procedure is: fit a POT model at each candidate p, simulate top order statistics from that model, compare them with the observed top order statistics from the same 4 runs via the betting wealth W, and pick p that minimizes W. This is an in-sample self-coherence check, as the authors explicitly concede in §4.1 ('we validate the model on the training set'). The only bridge from 'coherence in rounds 1..K−1' to 'extrapolation beyond the observable range' is the asserted transfer in §3.1, which is definitional rather than demonstrated. In addition, the power guarantees for the betting test are imported from a preprint by the second author, with no independent formalization or code and with unverified conditions. These features make the derivation partially circular: the score being optimized and the quantity being claimed ('extrapolation power') are connected only by definition, while the theoretical strength of the test rests on a self-citation. However, the paper is transparent about the limitations, the final frequencies were externally evaluated in the EVA2025 challenge, and there is no single fitted parameter that is directly renamed as a prediction. Thus the circularity is partial rather than total, giving a score of 4.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The method rests on standard EVT (Pickands–Balkema–de Haan), the exponential-tail choice γ=0, an i.i.d. assumption after detrending, the transfer of testing-by-betting power guarantees from a self-cited preprint, and a uniform-angle assumption for Target 3. Free parameters: p_i (chosen by the martingale score), K (arbitrarily 3), the candidate grid, the ~92% confidence level, and the GAM basis size (unreported). No invented physical entities are introduced; the 'extrapolation game' is a methodological construct, not an entity.

free parameters (5)
  • POT quantile level p_i = p*_2 = p*_3 = 0.9995; p*_1 = 0.999 chosen graphically against the method's {0.9, 0.995}
    The central tuning parameter. Chosen to minimize the martingale wealth score (§4.1), except for Target 1 where the authors overrode the score and chose by Q-Q plots (§4.2). The frequency estimates are heavily dependent on p (§5).
  • Number of game rounds K = 3 (5 also displayed in Figure 7)
    Chosen arbitrarily: 'We arbitrarily chose K = 3' (§4.2). The scores are 'very sensitive to the choice of K' (§4.1); no data-driven selection or sensitivity analysis is provided.
  • Candidate grid for p = {0.9, 0.99, 0.995, 0.999, 0.9992, 0.9995, 0.9997, 0.9999}
    The grid resolution is chosen by hand; the largest value is excluded from selection because only 24 exceedances remain (§4.1).
  • Confidence level for intervals (~92%) = ~0.92 instead of 0.95
    Chosen to improve sharpness of the scoring-rule-optimizing intervals (§2.4); the actual coverage/score trade-off is not analyzed.
  • GAM smoothness (cyclic cubic spline basis size) = Unreported ('few splines')
    The seasonal scale function f_i is fitted with mgcv using cyclic cubic splines and quadratic loss, but the number of basis functions is not stated; the authors note the fits are 'sensitive to large exceedances' (§5).
assumptions (6)
  • domain assumption The target sequences (y_t^(i)) are independent realizations of Y^(i) in the max-domain of attraction of a GPD (Pickands–Balkema–de Haan theorem).
    §2.2: 'We thus assume that (y_t)_t is a sequence of independent realizations... Y ∈ MDA(G_γ)'. Stationarity is asserted with 'no clear trend observed', but independence across time is assumed, not tested.
  • domain assumption Shape parameter γ = 0 (exponential tail) for all targets and all days of the year.
    §2.3: 'we simplify and stabilize the extrapolation procedure by fixing γ_i,d = 0'. Supported only graphically (Q-Q plots, Figure 5); the extrapolated far-tail probabilities depend entirely on this choice.
  • standard math Under H0: E[Y_{(K−k)}|F_k] = E[~Y_{(K−k)}|F_k], the EWA wealth process W^C_k is a martingale.
    §3.2: This is the standard testing-by-betting capital process (Ville's martingale, [5]); the validity is classical if the null is true. The open question is whether H0 is the right coherence null for extrapolation.
  • ad hoc to paper The power guarantees and O(log(1/α)/m) rate from [10] carry over to this 'non-standard' order-statistics game.
    §3.3: 'These results are valid in our non-standard game' — asserted, not proven here. The conditions |~Y_{(K−k)} − Y_{(K−k)}| ≤ 1 and E[(~Y−Y)²|F_k] ≤ m/4 are invoked but never checked for the data; [10] is self-cited (second author Wintenberger).
  • domain assumption For Target 3, the angular distribution of (y^(3.1), y^(3.2)) given y^(3) > u equals (sin θ, cos θ) with θ ~ U([0, π/2]).
    §2.1, Figure 2: claimed from a histogram of arcsin(y^(3.1)/y^(3)); treated as exact in equation (2), with no goodness-of-fit test.
  • standard math The number of occurrences of the rare events follows a Poisson distribution (point-process approximation).
    §2.4: Standard Poisson approximation for rare-event point processes, cited to [9] (Mikosch & Wintenberger, self-cited but a standard textbook result).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Assessing Extrapolation of Peaks Over Thresholds with Martingale Testing." pith.science (2026). https://pith.science/paper/5S5CIF4V

@misc{pith2026251203116,
  author       = {Pith},
  title        = {Pith review of: Assessing Extrapolation of Peaks Over Thresholds with Martingale Testing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5S5CIF4V}},
  note         = {Machine review of arXiv:2512.03116}
}
read the original abstract

We present the winning strategy for the EVA2025 Data Challenge, which aimed to estimate the probability of extreme precipitation events. These events occurred at most once in the dataset making the challenge fundamentally one of extrapolating extreme values. Given the scarcity of extreme events, we argue that a simple, robust modeling approach is essential. We adopt univariate models instead of multivariate ones and model Peaks Over Thresholds using Extreme Value Theory. Specifically, we fit an exponential distribution to model exceedances of the target variable above a high quantile (after seasonal adjustment). The novelty of our approach lies in using martingale testing to evaluate the extrapolation power of the procedure and to agnostically select the level of the high quantile. While this method has several limitations, we believe that framing extrapolation as a game opens the door to other agnostic approaches in Extreme Value Analysis.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 2 linked inside Pith

  1. [10]

    arXiv preprint arXiv:2504.00593 (2025)

    Durand, A., Wintenberger, O.: Power comparison of sequential testing by betting procedures. arXiv preprint arXiv:2504.00593 (2025)

    arXiv 2025

  2. [1]

    arXiv preprint arXiv:2508.15724 (2025)

    Zhang, Z., Fischer, E., Zscheischler, J., Engelke, S.: Numerical models outperform ai weather forecasts of record-breaking extremes. arXiv preprint arXiv:2508.15724 (2025)

    arXiv 2025

  3. [2]

    Technometrics 61(1), 123–135 (2019)

    Kiriliouk, A., Rootz´ en, H., Segers, J., Wadsworth, J.L.: Peaks over thresh- olds modeling with multivariate generalized pareto distributions. Technometrics 61(1), 123–135 (2019)

    2019

  4. [3]

    Springer, ??? (2006)

    De Haan, L., Ferreira, A.: Extreme Value Theory: an Introduction. Springer, ??? (2006)

    2006

  5. [4]

    Coles, S., Bawa, J., Trenner, L., Dorazio, P.: An Introduction to Statistical Modeling of Extreme Values vol. 208. Springer, ??? (2001)

    2001

  6. [5]

    Journal of the Royal Statistical Society Series A: Statistics in Society 184(2), 407–431 (2021)

    Shafer, G.: Testing by betting: A strategy for statistical and scientific commu- nication. Journal of the Royal Statistical Society Series A: Statistics in Society 184(2), 407–431 (2021)

    2021

  7. [6]

    The Annals of probability, 792–804 (1974)

    Balkema, A.A., De Haan, L.: Residual life time at great age. The Annals of probability, 792–804 (1974)

    1974

  8. [7]

    the Annals of Statistics, 119–131 (1975)

    Pickands III, J.: Statistical inference using extreme order statistics. the Annals of Statistics, 119–131 (1975)

    1975

Show all 13 references
  1. [8]

    R news 1(2), 20–25 (2001)

    Wood, S.N.: mgcv: Gams and generalized ridge regression for r. R news 1(2), 20–25 (2001)

    2001

  2. [9]

    Springer, ??? (2024)

    Mikosch, T., Wintenberger, O.: Extreme Value Theory for Time Series. Springer, ??? (2024)

    2024

  3. [11]

    Cambridge university press, ??? (2006) 13

    Cesa-Bianchi, N., Lugosi, G.: Prediction, Learning, and Games. Cambridge university press, ??? (2006) 13

    2006

  4. [12]

    http://eudml.org/doc/192893

    Ville, J.: ´Etude Critique de la Notion de Collectif, (1939). http://eudml.org/doc/192893

    1939

  5. [13]

    arXiv preprint arXiv:2410.23614 (2024) 14

    Ramdas, A., Wang, R.: Hypothesis testing with e-values. arXiv preprint arXiv:2410.23614 (2024) 14

    2024 arXiv

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.