{"id":"253b3ca0-f752-4bf0-92d0-d41cca178ce3","arxiv_id":"2501.14809","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A seismic AI evaluation framework jointly estimates training and data uncertainty and learning efficiency, and shows no single PhaseNet training approach wins at all data budgets.","lead":"This paper presents a framework for evaluating earthquake-picking AI models that separates two sources of performance uncertainty: randomness in training and randomness in which data is chosen. It applies the framework to compare three ways of training the PhaseNet model and shows that the best choice depends on how much training data is available.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Source-location clustering does not prevent station-ID leakage: shared stations between train and test could bias every learning-efficiency and uncertainty result.","rationale":"The strongest claim is that the framework prevents misleading model rankings and reveals budget-dependent winners. For that claim, the split design must ensure that adding clusters actually adds source diversity and that test data are not contaminated by training data. The paper itself identifies station proximity as a leakage pathway (Section 2), yet the implemented stratification (Section 3.1) only clusters sources and assigns noise by station. Since the same station appears in both earthquake and noise waveforms across many source clusters, station identity is a concrete confound not controlled by the design. The paper's appendix validates diversity of source-region features but provides no station-overlap diagnostic. A station-disjoint split is a natural and cheap validation; if it changes rankings, the central claim is an artifact of leakage. There is also a secondary design concern: for the 12-cluster budget, all 12 'random' cluster sets are identical (only one combination exists), so the data-sampling variance at the largest budget is zero by construction rather than by learning; this makes the reported decline in data variance with data quantity partly an artifact. That issue is acknowledged in the Discussion, but it reinforces that the framework's uncertainty estimates need a design-level check. The station-overlap test is the single most decisive check because it affects every training budget and every model comparison.","tokens_in":15910,"tokens_out":7748,"duration_ms":79116,"concrete_test":"Use SeisBench INSTANCE metadata to extract station IDs for every waveform in the training, validation, and test splits. Report (i) the number of stations shared between training and test, (ii) the fraction of test earthquake waveforms recorded at shared stations, and (iii) the fraction of test noise waveforms at shared stations. If any of these fractions is non-negligible, construct a station-disjoint split by assigning all waveforms recorded at a station to a single split (or clustering by station location), rerun the 1/3/6/9/12-cluster comparisons, and check whether the rank order of Standard, TL Free, and TL Frozen, or the data-budget crossing point, changes. A change would indicate that the original k-means source split was not adequately controlling the leakage it was designed to prevent.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 constructs INSTANCE splits by k-means clustering on source latitude/longitude (20 clusters), with noise waveforms assigned to clusters by station location. Section 2 explicitly lists 'nearby stations' as a source of data leakage, but the split never blocks on station identity. A fixed seismic station records events from many source clusters, so a test waveform from a northern or southern source can be recorded at a station whose other waveforms (central-source earthquake waveforms, or noise waveforms assigned by station location) are in the training pool. If this occurs, station-specific noise, instrument response, and site conditions are shared across splits. The paper reports cluster-level metadata distributions (Figures 3, A1-A5) but never reports train/test station overlap. Without station-disjoint splits, the learning-efficiency curves and the data-uncertainty estimates can be optimistically biased, and the framework's claim of 'partially control[ling] for data leakage' is not supported. This is the load-bearing premise for the central conclusion that model rankings depend on training data budget.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an evaluation framework for seismic deep learning models that jointly quantifies performance uncertainty (from stochastic training and random data sampling) and learning efficiency as a function of training data budget. The authors construct train/validation/test splits from the INSTANCE dataset by k-means clustering on source locations, train 720 PhaseNet instances under three training approaches (standard, transfer learning with free weights, and transfer learning with frozen encoder) across five data-budget levels (1, 3, 6, 9, and 12 clusters), and summarize classification and regression metrics using a mixed-effects model (Eq. 1). They report that no training approach is globally best across budgets, that uncertainty decreases with more data, and that ignoring uncertainty creates a substantial risk of mis-ranking models. The paper also discusses negative variance estimates as a known limitation and suggests future improvements such as more cluster sets or a Bayesian framework.","tokens_in":16086,"tokens_out":6905,"duration_ms":72437,"significance":"If the framework's leakage controls are adequate, this paper makes a useful contribution to seismic AI benchmarking by drawing attention to learning efficiency and uncertainty, and by providing a concrete experimental design (720 model instances on public data) that others can emulate. The statistical model is straightforward and its Gaussian assumptions are checked with QQ plots. The authors are transparent about limitations, including negative variance estimates and the single-feature stratification. The main practical claim—that model rankings depend on the amount of available training data—is nontrivial and relevant to practitioners choosing between training from scratch and transfer learning, and to the seismic foundation-model community. However, the validity of the learning-efficiency and uncertainty conclusions depends on the split adequately preventing data leakage, which is the central issue I raise in the major comments.","major_comments":[{"comment":"The split construction in Section 3.1 assigns earthquake waveforms to clusters by source location and noise waveforms to clusters by station location, but it never blocks on station identity. Because a fixed station records events from many source clusters, test waveforms from northern and southern sources can be recorded at stations whose other waveforms—from central sources, or noise waveforms assigned by station location—appear in the training pool. The manuscript itself lists 'nearby stations' as a source of data leakage in Section 2, and Figures A1-A5 show cluster-dependent signal properties, so station-level site effects are likely shared across the split. Table 1 reports only source and waveform counts, not station overlap, and Figure 2 (bottom) does not identify which stations are in training versus test. This is load-bearing because the learning-efficiency curves in Figures 6-7 and the uncertainty decomposition in Figure 8 could all be optimistically biased if test waveforms are more similar to training waveforms than a station-disjoint split would permit. I request (i) reporting the number and proportion of stations shared between the training and test waveform sets, and (ii) a sensitivity analysis using a station-disjoint split, or an alternative demonstration that station sharing does not affect the ranking conclusions.","section":"Section 3.1, Figure 2, Table 1"},{"comment":"The paper acknowledges that data variance is often estimated as negative for larger training budgets and is not displayed. Because the framework's stated contribution includes segregating performance uncertainty arising from stochastic training and random data sampling (Section 5.2), the current design does not support reliable statements about the relative contribution of data variance for budgets of 9 and 12 clusters, where the number of distinct cluster draws is small (and exactly one for 12 clusters). The suppression of negative estimates in Figure 8 makes the relative-contribution bars potentially misleading. Please report the frequency of negative variance estimates across all metrics and budgets, and state explicitly which budget ranges the variance decomposition is trustworthy for.","section":"Section 7, Figure 8"},{"comment":"The test set uses only the 4 northernmost and 4 southernmost clusters, while all training and validation data come from the 12 central clusters (Section 3.1). The paper positions this as a semi-out-of-distribution evaluation, which is a reasonable choice, but the central conclusion 'no model is a global winner in terms of learning efficiency' (Section 8) is therefore about performance on a particular semi-OOD test distribution. The ranking could differ for an in-distribution test set sampled from the central clusters, and the current paper provides no evidence about that case. Adding an in-distribution test set, or explicitly re-scoping the conclusion to semi-OOD generalization, would clarify the claim.","section":"Section 6, Figures 6-7"}],"minor_comments":[{"comment":"The affiliations use 'V A' with a space; this should be 'VA', both in the header and anywhere else it appears.","section":"Author affiliations"},{"comment":"The validation split description says 'Maintaining a ∼11.4% earthquake to noise ratio', but the text immediately before refers to the 'noise to earthquake waveform ratio' in INSTANCE; the ratio should be stated consistently as noise-to-earthquake.","section":"Section 3.1"},{"comment":"The caption says 'the opaque and transparent bars are the contributions of data and training variances' and then says 'the training contribution is transparent and the data contribution is opaque'; this is internally inconsistent and should be clarified.","section":"Figure 8 caption"},{"comment":"The word 'wholistic' should be 'holistic'.","section":"Section 7"},{"comment":"The manuscript does not include a data or code availability statement; since the framework is meant to be reusable, stating whether code and trained model weights will be released would aid reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The station-overlap issue is the main risk to the paper's central claim; it is addressable with an additional split and sensitivity analysis, so I do not recommend rejection. The paper is otherwise a solid empirical contribution, and the authors are transparent about limitations. The scope seems appropriate for a seismological methods journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper is a solid, honest empirical study of how to compare seismic pickers with quantified uncertainty. Its main caveat is not hidden: the train/test split is geographic by source location, not by station, so station-specific effects can leak across splits.\n\nWhat is actually new: the authors combine deep ensembles, stratified cluster sampling, and a mixed-effects model to separate training variance from data variance, then apply it to PhaseNet under three training schemes with 720 model instances. That is a real contribution to the seismic ML benchmarking literature, which mostly compares point estimates. The statistical model in Eq. 1 is appropriate, the QQ plot checks give reasonable support, and the ranking simulation in Figure 9 is a nice practical illustration of why single-instance comparisons are risky.\n\nThe weak spot is load-bearing. Section 2 lists 'nearby stations' as a source of data leakage, but the split never blocks on station identity. A fixed station records events from many source clusters, so a test waveform from a northern or southern source can be recorded at a station whose other waveforms—central-source earthquakes or noise assigned by station location—are in the training pool. The authors report cluster-level metadata distributions but never report train/test station overlap. Without a station-disjoint split, or at least a random-split baseline, the learning-efficiency curves and the data-uncertainty estimates may be optimistically biased. This does not necessarily destroy the paper's conclusions—all models see the same splits, so relative rankings may be robust—but it does mean the framework's claim of 'partially controlling for data leakage' is not empirically supported as written.\n\nMinor soft spots: the choice of 20 clusters, the validation-based threshold, and the Gaussian error assumption for bounded metrics are all stated and discussed; none are hidden. The negative variance estimates are handled transparently. The absence of released code and split definitions is a real reproducibility gap, especially for a methodology paper.\n\nWho is this for? Seismic ML practitioners who want a template for evaluating models with uncertainty, and researchers comparing transfer-learning approaches. It is not a new algorithm but a methodology contribution. It deserves a serious referee: the framework is useful, the empirical work is extensive, and the leakage concern is addressable in revision.\n\nI would send it to peer review with a request for code or detailed split artifacts and a station-overlap analysis. If the leakage turns out to be severe, the empirical conclusions may shift, but the framework itself remains worth publishing.","headline":"A useful evaluation framework for seismic deep learning models, but the split design lets station identity leak across train and test, and without code or a station-disjoint baseline the central learning-efficiency claims are not fully supported.","tokens_in":16611,"tokens_out":2242,"would_cite":true,"duration_ms":24788,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Seismic AI's best training approach depends on the amount of available data, not a single global winner.","keywords":["uncertainty quantification","model benchmarking","deep learning","learning efficiency","seismic phase picking","transfer learning","data leakage","statistical experimental design"],"falsifier":"Compute pairwise waveform similarities between clusters: if waveforms drawn from different source-location clusters are on average as similar as waveforms within a cluster, then location clustering has not controlled leakage and the training budgets are not adding independent information, which would invalidate the learning-efficiency comparison.","tokens_in":15672,"feed_emoji":"📡","tokens_out":5559,"duration_ms":55869,"temperature":0.7,"pith_summary":"This paper argues that standard evaluations of seismic deep learning models can mislead because they ignore two sources of variability: random data sampling and stochastic training, which jointly make performance uncertain. It proposes an evaluation framework that measures both sources and links them to learning efficiency, defined as performance gain per amount of diverse training data. The paper demonstrates the framework on PhaseNet, a widely used phase-picking model, comparing three training approaches across five data budgets. The central finding is that no training approach wins across all budgets: transfer learning helps most at the smallest data size, while training from scratch overtakes it once roughly 30,000 waveforms are available, so model choice should depend on a practitioner's data budget.","feed_headline":"Seismic AI's best training approach depends on data budget","feed_subtitle":"A framework that separates data and training uncertainty prevents misleading model superiority claims.","key_machinery":"The load-bearing mechanism is a three-part evaluation design. First, INSTANCE earthquake sources are clustered into 20 groups by latitude and longitude with k-means; 12 central clusters supply training and validation data, while 8 northern and southern clusters form a balanced test set, so that data budgets add whole geographic clusters rather than random waveforms and leakage from similar signals is reduced. Second, each model is trained on 12 random selections of 1, 3, 6, 9, or 12 clusters, with 4 deep-ensemble initializations per selection, yielding 720 model instances. Third, a mixed-effects statistical model, $y_{madi} = \\gamma + \\mu_m + \\alpha_a + \\theta_{ma} + \\epsilon^{\\mathrm{data}}_{mad} + \\epsilon^{\\mathrm{train}}_{madi}$, estimates the effect of training approach, data budget, and their interaction while splitting total variance into data-sampling and training components, and produces confidence intervals for every metric used in comparisons.","core_discovery":"The paper's central claim is that a fair comparison of seismic AI models requires jointly estimating performance uncertainty and learning efficiency, and that once these are accounted for, no model is a global winner. Using a mixed-effects statistical model applied to 720 PhaseNet instances, the authors separate data uncertainty (which cluster subset of training sources is drawn) from training uncertainty (which of 4 initializations is used). They find that the transfer-learning model with free weights has the best mean recall at one training cluster, while the from-scratch model is best at three or more clusters, and that rankings from a single model instance carry substantial mis-ranking risk, such as a 2% chance that the worst average model appears to be the best on recall. The paper concludes that expending effort on very large labeled datasets may be wasteful, since performance gains diminish beyond roughly 30,000 to 60,000 waveforms.","pith_inferences":["Because the framework treats the data split as part of the experiment, the same design could benchmark seismic foundation models against simpler pickers, testing whether pretraining on global data beats from-scratch training at each fine-tuning budget.","The choice of source-location clustering is one of many possible stratifications; a natural extension is to test whether clustering on time, network, magnitude, or signal-to-noise ratio changes the learning-efficiency conclusions, since the authors note that a single feature captures diversity only partially.","The frequent negative estimates of data variance at large budgets suggest the design under-samples cluster combinations; increasing the number of random cluster sets beyond 12, or sampling with replacement, could give sharper uncertainty estimates where the paper currently reports near-zero data variance.","Outside seismology, the same template could apply to any benchmark where geographic or domain structure creates leakage, such as medical imaging from different hospitals, by replacing source-location clusters with site or device clusters."],"forward_implications":["Practitioners can choose a training approach by matching their labeled-data budget to the estimated performance curves instead of trusting a single benchmark ranking.","Developers know where to spend effort: when data uncertainty dominates, label or acquire more diverse data; when training uncertainty dominates, run more initializations or ensembles.","Reported state-of-the-art claims for seismic pickers are unreliable unless they come with variance estimates; a single run can put the worst average model first.","Acquiring ever-larger training sets has diminishing returns; roughly 30,000 to 60,000 waveforms may suffice for reasonable PhaseNet performance even from scratch.","Transfer learning is not automatically better than training from scratch; its advantage is limited to small data budgets in this setting, which matters for foundation-model fine-tuning."],"supporting_citations":[{"why":"PhaseNet: the deep-neural-network phase-picking architecture whose three training approaches are compared.","marker":"[1]"},{"why":"STEAD: the global dataset used to pretrain the upstream models for transfer learning.","marker":"[29]"},{"why":"INSTANCE: the Italian dataset whose sources are clustered for splits and on which all models are validated and tested.","marker":"[30]"},{"why":"SeisBench: the toolbox through which both datasets are accessed and whose default STEAD splits are reused.","marker":"[6]"},{"why":"Deep ensembles: the method used to capture training uncertainty through multiple initializations.","marker":"[38]"},{"why":"Snapshot ensembles: the cosine-annealing restart idea adapted for the ensemble training design.","marker":"[39]"},{"why":"Establishes that variance from training and data sampling affects ML benchmarks, motivating the two uncertainty sources.","marker":"[8]"},{"why":"Scikit-learn's k-means: the clustering algorithm that defines the geographic training and test splits.","marker":"[31]"}],"fun_headline_variants":["Seismic AI rankings shift with data size","No universal best seismic AI model","Data budget decides seismic AI winner","Uncertainty reveals seismic AI trade-offs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework's conclusions rest on the assumption that k-means clustering by source location sufficiently reduces data leakage and captures the diversity that matters for learning efficiency; if waveforms from different clusters still share substantial information through common stations, networks, or propagation paths, the learning-efficiency curves and uncertainty estimates would be biased.","fun_headline_variants_meta":{"raw":{"variants":["Seismic AI rankings shift with data size","No universal best seismic AI model","Data budget decides seismic AI winner","Uncertainty reveals seismic AI trade-offs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000226,"raw_usage":{"total_tokens":1422,"prompt_tokens":853,"completion_tokens":569,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":469,"completion_tokens_details":{"reasoning_tokens":518}},"tokens_in":469,"tokens_out":569,"duration_ms":5924,"temperature":1.0,"reasoning_tokens":518,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:13:19.623386+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute pairwise waveform similarities between clusters: if waveforms drawn from different source-location clusters are on average as similar as waveforms within a cluster, then location clustering has not controlled leakage and the training budgets are not adding independent information, which would invalidate the learning-efficiency comparison.","supporting_citations":[{"cited_title":"PhaseNet: A Deep-Neural-Network-Based Seismic Arrival Time Picking Method.Geophys","cited_arxiv_id":null,"evidence_quote":"PhaseNet: the deep-neural-network phase-picking architecture whose three training approaches are compared."},{"cited_title":"Mostafa Mousavi, Yixiao Sheng, Weiqiang Zhu, and Gregory C","cited_arxiv_id":null,"evidence_quote":"STEAD: the global dataset used to pretrain the upstream models for transfer learning."},{"cited_title":"INSTANCE – the Italian seismic dataset for machine learning.Earth Syst","cited_arxiv_id":null,"evidence_quote":"INSTANCE: the Italian dataset whose sources are clustered for splits and on which all models are validated and tested."},{"cited_title":"SeisBench—A Toolbox for Machine Learning in Seismology.Seismol","cited_arxiv_id":null,"evidence_quote":"SeisBench: the toolbox through which both datasets are accessed and whose default STEAD splits are reused."}],"review_version":1}