pith. sign in

arxiv: 2605.00779 · v2 · submitted 2026-05-01 · 📊 stat.ME

Evaluating the performance of GCM trajectories using Weather Type frequencies for persistence and transitions: the Iberian Peninsula and Lamb classification

Elsa Barrio-Torres , Swen Brands , Jes\'us As\'in , Jes\'us Abaurrea , Zeus Gracia-Tabuenca , Jorge Castillo-Mateo This is my paper

Pith reviewed 2026-05-09 19:00 UTC · model grok-4.3

classification 📊 stat.ME
keywords GCM evaluationWeather typesLamb classificationCMIP6Iberian PeninsulaTransition probabilitiesOverlap coefficientPersistence
0
0 comments X

The pith

Assessing weather type persistence and transitions offers a more discriminative way to select GCMs for the Iberian Peninsula than matching daily distributions alone.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper evaluates 36 CMIP6 global climate models for their ability to reproduce summer atmospheric circulation over the Iberian Peninsula using the Lamb Weather Type classification. It compares model outputs to ERA5 reanalysis by checking not only the frequency of each weather type but also how those types persist and transition from one day to the next. Models that match daily frequencies well sometimes fail on the dynamics of day-to-day changes, leading to a stricter selection of only 12 trajectories. The EC-Earth3 family performs best overall, with models generally better in the northwest than in central and southern areas. This shows that including transition information gives a stronger test for model usefulness in regional climate applications.

Core claim

By applying the Lamb classification to both observations and model simulations, the study demonstrates that while many models can replicate the overall occurrence rates of weather types, fewer accurately capture the probabilities of staying in the same type or switching between types over 24 hours. A filtering process using an overlap coefficient threshold retains only those models that perform adequately on both aspects across multiple grid points, resulting in the selection of 12 trajectories, with the EC-Earth3 aerchem variant showing the most consistent performance.

What carries the argument

The Lamb Weather Type classification scheme applied to daily sea-level pressure fields, combined with calculation of transition probability matrices and persistence statistics, quantified via the overlap coefficient between model and reference distributions.

Load-bearing premise

The Lamb Weather Type classification and the chosen overlap coefficient threshold together capture the aspects of atmospheric circulation that matter most for model usefulness, with ERA5 serving as an unbiased reference.

What would settle it

A direct comparison of simulated and observed sequences of weather types in an independent validation period or region, checking if models passing the transition filter actually produce more realistic event chains than those that do not.

read the original abstract

This study evaluates the performance of 36 historical CMIP6 GCM trajectories (1979-2005) in reproducing atmospheric circulation over the Iberian Peninsula in the summer months (June-September) using the Lamb Weather Type (WT) classification scheme. Using ERA5 reanalysis as the observational reference, we introduce a methodological framework-applicable to any region worldwide-to evaluate GCM performance. This approach extends traditional daily frequency analysis by evaluating both the daily frequency distribution of WTs and their 24-hour dynamic evolution (i.e., transition probabilities and persistence). Model performance is quantified using the Overlap coefficient. A filtering process is applied where only trajectories that successfully reproduce both daily and conditional distributions with a minimum Overlap threshold $t_{sim}$ across a set number of grid points are retained. The findings show that while several models can adequately reproduce daily WT frequencies (16 out of 36), some struggle to capture day-to-day atmospheric transitions. This leads to a final selection of 12 trajectories over the Iberian Peninsula. Model performance across the region is then evaluated using integrated metrics assessing daily reproduction, conditional reproduction, and transition dynamics. Overall, models from the ec earth3 family-specifically the ec earth3 aerchem trajectory-exhibit the best and most consistent performance across the region. Additionally, the results highlight a geographical performance gap: while models generally represent circulation well in the northwest, they face significant challenges in the central and southern Mediterranean regions of the Peninsula. Ultimately, this study establishes that assessing WT persistence and transitions provides a far more discriminative, objective tool for GCM selection than evaluating daily distributions alone.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The paper evaluates 36 CMIP6 GCM historical trajectories (1979-2005) over the Iberian Peninsula during summer (June-September) against ERA5 reanalysis using the Lamb Weather Type (WT) classification. It extends standard daily frequency analysis by also assessing 24-hour persistence and transition probabilities, quantifying performance via the overlap coefficient. A filtering procedure retains models that meet a minimum overlap threshold t_sim across grid points for both daily and conditional distributions, yielding 16 models that pass frequencies and 12 that pass both; the EC-Earth3 family (especially ec-earth3-aerchem) performs best overall, with a noted northwest-to-southeast performance gradient. The central claim is that incorporating persistence and transitions yields a substantially more discriminative, objective GCM selection tool than daily frequencies alone.

Significance. If the result holds, the work supplies a reproducible, region-agnostic framework that improves GCM filtering for circulation-sensitive applications such as regional downscaling and impact modeling. The concrete reduction from 16 to 12 trajectories and the identification of consistent EC-Earth3 performance provide actionable guidance for the Iberian Peninsula while highlighting systematic geographical biases in model skill.

major comments (3)
  1. [Methods (filtering procedure)] Methods section describing the filtering process: the overlap threshold t_sim is applied to reduce the acceptable set from 16 (daily frequencies) to 12 (frequencies plus transitions), yet neither the value of t_sim nor any justification or sensitivity tests are provided. Because the claimed increase in discrimination rests entirely on this cutoff, the stability of the final set size and of the EC-Earth3 ranking under modest perturbations to t_sim must be demonstrated.
  2. [Results (model selection and metrics)] Results section reporting model counts and integrated metrics: the manuscript states that 16 trajectories pass the frequency test and 12 pass both tests, but supplies no error bars, bootstrap intervals, or full per-model/per-grid-point overlap tables. Without these, it is impossible to judge whether the performance gap between retained and rejected models is statistically robust or sensitive to sampling variability in the 1979-2005 period.
  3. [Discussion] Discussion of the superiority claim: the assertion that persistence and transition statistics are 'far more discriminative' is supported only by the single observed reduction in model count. An explicit comparison (e.g., rank correlation or number of models retained under frequency-only versus frequency-plus-transition criteria across a range of thresholds) is required to substantiate that the added information is not an artifact of the particular t_sim choice.
minor comments (2)
  1. [Notation and Methods] Ensure the symbol t_sim is defined with its numerical value at first use and that the exact number of grid points and the overlap-coefficient formula are stated unambiguously.
  2. [Figures] Figures displaying spatial performance maps or bar charts of overlap coefficients should include uncertainty indicators and a clear legend distinguishing the daily-frequency versus conditional-distribution results.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for their thorough review and valuable suggestions. We address each of the major comments in detail below and will incorporate the necessary revisions to improve the clarity and robustness of the manuscript.

read point-by-point responses

  1. Referee: [Methods (filtering procedure)] Methods section describing the filtering process: the overlap threshold t_sim is applied to reduce the acceptable set from 16 (daily frequencies) to 12 (frequencies plus transitions), yet neither the value of t_sim nor any justification or sensitivity tests are provided. Because the claimed increase in discrimination rests entirely on this cutoff, the stability of the final set size and of the EC-Earth3 ranking under modest perturbations to t_sim must be demonstrated.

    Authors: We appreciate the referee's observation regarding the filtering procedure. We recognize that the exact value of t_sim and associated sensitivity analyses were not explicitly presented in the submitted manuscript. In the revised version, we will clearly specify the value of t_sim used, provide justification for its selection based on achieving a balance between retaining sufficient models and ensuring high fidelity to observed distributions, and include sensitivity tests showing the stability of the 12-model selection and the EC-Earth3 ranking under perturbations to t_sim. revision: yes

  2. Referee: [Results (model selection and metrics)] Results section reporting model counts and integrated metrics: the manuscript states that 16 trajectories pass the frequency test and 12 pass both tests, but supplies no error bars, bootstrap intervals, or full per-model/per-grid-point overlap tables. Without these, it is impossible to judge whether the performance gap between retained and rejected models is statistically robust or sensitive to sampling variability in the 1979-2005 period.

    Authors: We agree that providing measures of uncertainty would strengthen the results. We will revise the Results section to include bootstrap confidence intervals for the overlap coefficients and add supplementary tables containing the full per-model and per-grid-point overlap values. This will enable a better assessment of the statistical robustness of the performance differences and model selection. revision: yes

  3. Referee: [Discussion] Discussion of the superiority claim: the assertion that persistence and transition statistics are 'far more discriminative' is supported only by the single observed reduction in model count. An explicit comparison (e.g., rank correlation or number of models retained under frequency-only versus frequency-plus-transition criteria across a range of thresholds) is required to substantiate that the added information is not an artifact of the particular t_sim choice.

    Authors: We acknowledge that the superiority claim would benefit from more explicit supporting analyses. We will expand the Discussion to include direct comparisons, such as the number of models retained under frequency-only and combined criteria across a range of thresholds, as well as rank correlations between the two sets of metrics. These additions will provide stronger evidence that the transition information adds discriminative power beyond the specific choice of t_sim. revision: yes

Circularity Check

0 steps flagged

No significant circularity; direct comparison to independent ERA5 using standard overlap metric

full rationale

The paper computes overlap coefficients between GCM-simulated and ERA5-observed daily WT frequency distributions as well as 24-hour transition and persistence probabilities. These quantities are obtained from separate data sources and use a conventional overlap measure that is not defined in terms of the final performance ranking or model selection. The threshold t_sim functions as an external selection cutoff rather than a parameter fitted from the same data in a way that would force the outcome by construction. No self-citations, ansatzes, or uniqueness theorems are invoked to justify core steps, and the observed reduction from 16 to 12 acceptable trajectories is an empirical result of applying the independent metrics, not a definitional tautology. The framework therefore remains self-contained against the external ERA5 benchmark.

Axiom & Free-Parameter Ledger

1 free parameters · 2 axioms · 0 invented entities

The central claim depends on the Lamb classification being a faithful representation of circulation, ERA5 being an accurate benchmark, and the overlap coefficient plus chosen threshold being sufficient discriminators; the threshold t_sim is an explicit free parameter.

free parameters (1)
  • t_sim
    Minimum overlap threshold used to retain trajectories; its specific value is not stated in the abstract but directly controls which models are kept.
axioms (2)
  • domain assumption ERA5 reanalysis provides an unbiased and sufficiently accurate representation of observed atmospheric circulation over the Iberian Peninsula.
    Used as the sole observational reference without additional validation or uncertainty quantification in the abstract.
  • domain assumption The Lamb Weather Type classification captures the dynamically relevant features of summer circulation for model evaluation purposes.
    Invoked as the basis for both frequency and transition analysis.

pith-pipeline@v0.9.0 · 5622 in / 1474 out tokens · 41622 ms · 2026-05-09T19:00:27.816037+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

51 extracted references · 16 canonical work pages

  1. [1]

    Theoretical and Applied Climatology , volume=

    Acero, Francisco Javier and Fern. Theoretical and Applied Climatology , volume=. 2018 , publisher=

    2018

  2. [2]

    2018 , month =

    Achebak, Hicham AND Devolder, Daniel AND Ballester, Joan , journal =. 2018 , month =. doi:10.1371/journal.pmed.1002617 , pages =

    work page doi:10.1371/journal.pmed.1002617 2018

  3. [3]

    A survey of distance/similarity measures for categorical data , year=

    Alamuri, Madhavi and Surampudi, Bapi Raju and Negi, Atul , booktitle=. A survey of distance/similarity measures for categorical data , year=

  4. [4]

    Elsa Barrio-Torres and Jesús Abaurrea and Jesús Asín and Jorge Castillo-Mateo and Ana Carmen Cebrián and Zeus Gracia , year=

  5. [5]

    and Shaw, Tiffany and Simpson, I

    Bordoni, Silvia and Kang, S. and Shaw, Tiffany and Simpson, I. and Zanna, Laure , year =. The futures of climate modeling , volume =. npj Climate and Atmospheric Science , doi =

  6. [6]

    Climate dynamics , volume=

    Brands, Swen and Herrera, Sixto and Fern. Climate dynamics , volume=. 2013 , publisher=

    2013

  7. [7]

    2022 , publisher =

    Brands, Swen , title =. 2022 , publisher =

    2022

  8. [8]

    Reductions in daily continental-scale atmospheric circulation biases between generations of Global Climate Models: CMIP5 to CMIP6 , volume =

    Cannon, Alex , year =. Reductions in daily continental-scale atmospheric circulation biases between generations of Global Climate Models: CMIP5 to CMIP6 , volume =. Environmental Research Letters , doi =

  9. [9]

    2025 , issn =

    Changes in large-scale circulation behind the increase in extreme heat events in the Apennines (Italy) , journal =. 2025 , issn =. doi:10.1016/j.atmosres.2025.108013 , author =

    work page doi:10.1016/j.atmosres.2025.108013 2025

  10. [10]

    Spatio-temporal modeling for record-breaking temperature events in

    Jorge Castillo-Mateo and Alan E Gelfand and Zeus Gracia-Tabuenca and Jes. Spatio-temporal modeling for record-breaking temperature events in. Journal of the American Statistical Association , volume=. 2025 , publisher=

    2025

  11. [11]

    Joint space-time modelling for upper daily maximum and minimum temperature record-breaking , journal =

    Castillo-Mateo, Jorge and Gracia-Tabuenca, Zeus and As. Joint space-time modelling for upper daily maximum and minimum temperature record-breaking , journal =. 2025 , doi =

    2025

  12. [12]

    International Journal of Climatology , volume =

    Corell, David and Estrela, María José and Miró, Juan Javier and Orgambides-García, David and Niclós, Raquel , title =. International Journal of Climatology , volume =. doi:10.1002/joc.70182 , year =

    work page doi:10.1002/joc.70182

  13. [13]

    Artificial Intelligence for the Earth Systems , volume=

    A data-driven probabilistic network approach to assess model similarity in CMIP ensembles , author=. Artificial Intelligence for the Earth Systems , volume=

  14. [14]

    Hoffman, F. M. and Hassler, B. and Swaminathan, R. and Lewis, J. and Andela, B. and Collier, N. and Hegedűs, D. and Lee, J. and Pascoe, C. and Pfl\"uger, M. and Stockhause, M. and Ullrich, P. and Xu, M. and Bock, L. and Chun, F. and Gier, B. K. and Kelley, D. I. and Lauer, A. and Lenhardt, J. and Schlund, M. and Sreeush, M. G. and Weigel, K. and Blockley,...

    2025

  15. [15]

    and Briffa, K.R

    Hulme, M. and Briffa, K.R. and Jones, P.D. and others , title =. Climate Dynamics , volume =. 1993 , doi =

    1993

  16. [16]

    Geophysical Research Letters , volume=

    Predictability of semiannual atmospheric circulation type frequencies in the extratropics using the EC-Earth3 decadal forecasting system , author=. Geophysical Research Letters , volume=. 2025 , publisher=

    2025

  17. [17]

    International Journal of Climatology , volume=

    The signature of the main modes of climatic variability as revealed by the Jenkinson-Collison classification over Europe , author=. International Journal of Climatology , volume=. 2024 , publisher=

    2024

  18. [18]

    Climate Dynamics , volume=

    Exploring the limits of the Jenkinson--Collison weather types classification scheme: A global assessment based on various reanalyses , author=. Climate Dynamics , volume=. 2023 , publisher=

    2023

  19. [19]

    Geoscientific Model Development , volume=

    A circulation-based performance atlas of the CMIP5 and 6 models for regional climate studies in the Northern Hemisphere mid-to-high latitudes , author=. Geoscientific Model Development , volume=. 2022 , publisher=

    2022

  20. [20]

    Geophysical Research Letters , volume=

    A global climate model performance atlas for the Southern Hemisphere extratropics based on regional atmospheric circulation patterns , author=. Geophysical Research Letters , volume=. 2023 , publisher=

    2023

  21. [21]

    Environmental Research , volume =

    Spatio-temporal evolution of heat waves severity and expansion across the. Environmental Research , volume =. 2023 , issn =. doi:10.1016/j.envres.2022.114864 , author =

    work page doi:10.1016/j.envres.2022.114864 2023

  22. [22]

    Dunne, J. P. and Hewitt, H. T. and Arblaster, J. M. and Bonou, F. and Boucher, O. and Cavazos, T. and Dingley, B. and Durack, P. J. and Hassler, B. and Juckes, M. and Miyakawa, T. and Mizielinski, M. and Naik, V. and Nicholls, Z. and O'Rourke, E. and Pincus, R. and Sanderson, B. M. and Simpson, I. R. and Taylor, K. E. , TITLE =. Geoscientific Model Develo...

    2025

  23. [23]

    and Bony, S

    Eyring, V. and Bony, S. and Meehl, G. A. and Senior, C. A. and Stevens, B. and Stouffer, R. J. and Taylor, K. E. , TITLE =. Geoscientific Model Development , VOLUME =. 2016 , NUMBER =

    2016

  24. [24]

    and Francisco, Raquel , year =

    Giorgi, F. and Francisco, Raquel , year =. Uncertainties in regional climate change prediction: A regional analysis of ensemble simulations with the HADCM2 coupled AOGCM , volume =. Climate Dynamics , doi =

  25. [25]

    Quarterly Journal of the Royal Meteorological Society , volume=

    The ERA5 global reanalysis , author=. Quarterly Journal of the Royal Meteorological Society , volume=. 2020 , publisher=

    2020

  26. [26]

    2023 , doi=

    ERA5 hourly data on single levels from 1940 to present, Copernicus Climate Change Service (C3S) Climate Data Store (CDS)[data set] , author=. 2023 , doi=

    1940

  27. [27]

    Jenkinson, A. F. and Collison, F. P. , title =. 1977 , number =

    1977

  28. [28]

    Jones, P. D. and Harpham, C. and Briffa, K. R. , title =. International Journal of Climatology , volume =. doi:https://doi.org/10.1002/joc.3498 , year =

    work page doi:10.1002/joc.3498

  29. [29]

    and L\'opez-Moreno, J

    El Kenawy, A. and L\'opez-Moreno, J. I. and Vicente-Serrano, S. M. , TITLE =. Natural Hazards and Earth System Sciences , VOLUME =. 2011 , NUMBER =

    2011

  30. [30]

    and Hall, Alex , title =

    Klein, Stephen A. and Hall, Alex , title =. Current Climate Change Reports , year =

  31. [31]

    Legasa, M. N. and Casanueva, A. and Manzanas, R. , title =. International Journal of Climatology , volume =. 2025 , keywords =. doi:10.1002/joc.70183 , note =

    work page doi:10.1002/joc.70183 2025

  32. [32]

    2021 , publisher =

    Climate Change 2021: The Physical Science Basis , author =. 2021 , publisher =

    2021

  33. [33]

    and Lee, Robert and Rowell, David , year =

    Mcsweeney, Carol and Jones, R. and Lee, Robert and Rowell, David , year =. Selecting CMIP5 GCMs for downscaling over multiple regions , volume =. Climate Dynamics , doi =

  34. [34]

    and Ronald, Stouffer , year =

    Meehl, Gerald and Boer, George and Covey, Curt and Latif, M. and Ronald, Stouffer , year =. CMIP Coupled Model Intercomparison Project , volume =. Bulletin of the American Meteorological Society , doi =

  35. [35]

    Miró, J. J. and Estrela, M. J. and Millán, M. , title =. International Journal of Climatology , volume =. 2006 , keywords =

    2006

  36. [36]

    Palmer, T. E. and McSweeney, C. F. and Booth, B. B. B. and Priestley, M. D. K. and Davini, P. and Brunner, L. and Borchert, L. and Menary, M. B. , TITLE =. Earth System Dynamics , VOLUME =. 2023 , NUMBER =

    2023

  37. [37]

    Weather and Climate Extremes , volume =

    Understanding the. Weather and Climate Extremes , volume =. 2023 , issn =. doi:10.1016/j.wace.2023.100631 , author =

    work page doi:10.1016/j.wace.2023.100631 2023

  38. [38]

    International Journal of Climatology , volume =

    Peña-Angulo, Dhais and Gonzalez-Hidalgo, José Carlos and Sandonís, Leire and Beguería, Santiago and Tomas-Burguera, Miquel and López-Bustins, Joan Albert and Lemus-Canovas, Marc and Martin-Vide, Javier , title =. International Journal of Climatology , volume =. doi:10.1002/joc.7006 , year =

    work page doi:10.1002/joc.7006

  39. [39]

    Atmospheric Research , volume=

    P. Atmospheric Research , volume=. 2023 , publisher=

    2023

  40. [40]

    Journal of statistical software , volume=

    Multivariate and propensity score matching software with automated balance optimization: the matching package for R , author=. Journal of statistical software , volume=

  41. [41]

    Summer heatwaves trends and hotspots in the

    Serra, Carina and Lana, Xavier and Mart. Summer heatwaves trends and hotspots in the. Theoretical and Applied Climatology , volume=. 2024 , publisher=

    2024

  42. [42]

    An updated review on recent trends in observational surface atmospheric variables and their extremes over

    Serrano, Sergio Mart. An updated review on recent trends in observational surface atmospheric variables and their extremes over. Cuadernos de investigaci. 2017 , publisher=

    2017

  43. [43]

    2023 , issn =

    Atmospheric Research , volume =. 2023 , issn =. doi:10.1016/j.atmosres.2023.106931 , author =

    work page doi:10.1016/j.atmosres.2023.106931 2023

  44. [44]

    , year =

    Shepherd, Theodore G. , year =. Atmospheric circulation as a source of uncertainty in climate change projections , volume =. doi:10.1038/ngeo2253 , pages =

    work page doi:10.1038/ngeo2253

  45. [45]

    and DelSole, T

    Shukla, J. and DelSole, T. and Fennessy, M. and Kinter, J. and Paolino, D. , title =. Geophysical Research Letters , volume =. doi:https://doi.org/10.1029/2005GL025579 , year =

    work page doi:10.1029/2005gl025579

  46. [46]

    Benestad and Claas Teichmann and Ole B

    Stefan Sobolowski and Samuel Somot and Jesus Fernandez and Guillaume Evin and Swen Brands and Douglas Maraun and Sven Kotlarski and Martin Jury and Rasmus E. Benestad and Claas Teichmann and Ole B. Christensen and Katharina Bülow and Erasmo Buonomo and Eleni Katragkou and Christian Steger and Silje Sørland and Grigory Nikulin and Carol McSweeney and Andre...

    work page doi:10.1175/bams-d-23-0189.1

  47. [47]

    2019 , issn =

    Weather and Climate Extremes , volume =. 2019 , issn =. doi:10.1016/j.wace.2019.100224 , author =

    work page doi:10.1016/j.wace.2019.100224 2019

  48. [48]

    2025 , issn =

    Rapid attribution prototype for extreme high temperature events in China , journal =. 2025 , issn =. doi:10.1016/j.accre.2025.06.004 , author =

    work page doi:10.1016/j.accre.2025.06.004 2025

  49. [49]

    An Overview of CMIP5 and the Experiment Design

    Karl E. Taylor and Ronald J. Stouffer and Gerald A. Meehl , title =. Bulletin of the American Meteorological Society , year =. doi:10.1175/BAMS-D-11-00094.1 , pages =

    work page doi:10.1175/bams-d-11-00094.1

  50. [50]

    and DaCamara, Carlos C

    Trigo, Ricardo M. and DaCamara, Carlos C. , title =. International Journal of Climatology , volume =. doi:https://doi.org/10.1002/1097-0088(20001115)20:13<1559::AID-JOC555>3.0.CO;2-5 , year =

    work page doi:10.1002/1097-0088(20001115)20:13

  51. [51]

    Climate Dynamics , volume=

    Analysis of synoptic weather patterns of heatwave events , author=. Climate Dynamics , volume=. 2023 , publisher=

    2023