{"id":"c6616016-6091-45fc-abed-75f999673831","arxiv_id":"2504.15487","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A CNN subgrid model trained on one quasi-geostrophic regime underestimates out-of-distribution activation and output spectra, and retraining only the first hidden layer with target data corrects the spectral bias.","lead":"This paper studies why a neural-network model of ocean turbulence fails on new flow types and how transfer learning, retraining just one network layer with a small amount of new data, fixes the problem. Fourier analysis of the network's filters and internal signals shows that the mismatch is a spectral one: the model under-produces high-frequency energy and one-layer retraining corrects that.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fig. 5's comparison of BNN0 on in-distribution vs OOD inputs conflates input-spectrum differences with weight-induced underestimation, so the claimed spectral mechanism is not isolated.","rationale":"The paper's headline contribution is a mechanistic explanation of TL failure, not merely the empirical result that one-layer retraining helps. The causal story requires that the learned weights are the source of the spectral underestimation, but Fig. 5 Row a cannot distinguish weight-driven suppression from input-driven differences. Without a same-input control, the 'underestimation begins in early layers' observation is predicted by the input spectra alone. This is the load-bearing point because the abstract and §3.2 state the mechanism as a finding, and the Discussion's caveats (single-maximum assumption, clustering subjectivity, case specificity) do not address this confound. The reader's weakest assumption concerns seed noise; that is valid but would not resolve the confound. Verdict stays CONDITIONAL: the empirical TL improvements and spectral diagnostics are useful, but the mechanistic claim requires the same-input comparison (and ideally seeded ensembles) before it can be treated as established.","tokens_in":18849,"tokens_out":8151,"duration_ms":72383,"concrete_test":"Evaluate BNN0, BNNi,i, and TLNN0,i on the same held-out target-case samples (e.g., 100 Case-1 snapshots) and compute channel-averaged, meridionally averaged activation spectra at each layer for 5+ training seeds, with error bands. If BNN0's early-layer spectra are not systematically below BNNi,i's after controlling for input spectra (e.g., the layer-1 gap equals the input-spectrum gap, or the gap does not grow between layer 1 and the output), then the 'weights underestimate OOD spectra' mechanism is unsupported. Report the layer-1 gap normalized by the input-spectrum gap to quantify the weight contribution.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2's mechanism claim—that weights/biases learned on Case 0 'underestimate the out-of-distribution sample spectra as they pass through the network' (Abstract, §3.2)—rests on Fig. 5 Row a, which compares BNN0 applied to in-distribution Case 0 inputs with the same BNN0 applied to OOD Cases 1–3 inputs. The first-layer activations are the fixed first-layer kernels convolved with the input velocities (Eq. 8, with ℓ=1); therefore any difference in input spectra between cases (visible in Fig. 2b–d) immediately appears as a layer-1 activation-spectrum gap, independent of whether the frozen weights are 'biased.' Calling this gap an underestimation by the learned weights is thus not established: the same gap would occur if the weights were optimal for the target but the inputs had lower spectral energy. The proper control is a same-input comparison of BNN0 vs BNNi,i on identical target samples. Fig. 5 Rows b–d provide such a comparison only after retraining (TLNN0,i vs BNNi,i), not for the pre-TL failure. The single-seed/error-bar issue is real but secondary; even a many-seed version of Fig. 5 Row a would retain this confound.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies transfer learning (TL) for a 9-layer convolutional neural network that predicts subgrid-scale potential vorticity forcing in a two-layer quasi-geostrophic ocean model. Four dynamical regimes are considered: two isotropic eddy configurations and two anisotropic jet configurations. The authors train a base network on Case 0 (BNN0), evaluate its out-of-distribution performance on Cases 1–3, and show that retraining only the first hidden layer on small amounts of target data (TLNN0,i) improves both offline metrics and online kinetic-energy spectra. Using Fourier analysis of kernels and hidden-layer activations, the paper argues that BNN0 fails to generalize because its learned weights and biases systematically underestimate out-of-distribution activation spectra as signals propagate through the network, and that retraining one layer corrects this spectral mismatch. The paper also claims that the learned convolutional kernels act as low-pass, Gabor, and high-pass filters regardless of whether the training data are isotropic or anisotropic.","tokens_in":19124,"tokens_out":5167,"duration_ms":45647,"significance":"If the proposed mechanism is correct, the paper would provide a physically interpretable explanation for when and why TL helps in data-driven subgrid-scale parameterization, potentially guiding layer-selection strategies in geophysical machine learning. The study is strengthened by its combination of offline and online evaluation, by covering both isotropic and anisotropic regimes, and by releasing code. The spectral diagnostics (kernel Fourier magnitudes and layer-wise activation spectra) are a useful and transferable interpretability tool. However, the central causal claim rests on a comparison that conflates input-spectrum differences with weight-induced underestimation, and the kernel-taxonomy claim is supported mainly by a qualitatively chosen clustering analysis. These issues are fixable but currently leave the headline conclusions less secure than the abstract suggests.","major_comments":[{"comment":"The claim that BNN0's learned weights 'underestimate the out-of-distribution sample spectra' is not established by the comparison shown in Fig. 5 Row a. That row compares BNN0,0 applied to in-distribution Case 0 inputs with BNN0,i applied to out-of-distribution Case i inputs. Since the first-layer activations are convolutions of the fixed first-layer kernels with the input velocities (Eq. 8 with ell = 1), any difference in input spectra between cases (visible in Fig. 2b-d) appears directly in the layer-1 activation spectra, independent of whether the frozen weights are 'biased' for the target regime. The same gap would occur even if the weights were optimal for the target but the inputs had different spectral content. The appropriate control is a same-input comparison: feed identical target-case samples to BNN0 and to BNNi,i and compare their activation spectra. Fig. 5 Rows b-d provide such a comparison only after retraining (TLNN0,i vs BNNi,i), not for the pre-TL failure that the mechanism claim is about. Without this control, the central explanatory story in the abstract and Section 3.2 is not supported by the displayed evidence.","section":"Section 3.2, Fig. 5 Row a"},{"comment":"The claim that the learned kernels are 'low-pass, Gabor, and high-pass filters regardless of whether the training data are isotropic or anisotropic' rests on k-means clustering with the number of clusters chosen 'until qualitatively similar patterns are observed' (Fig. 6 caption) and on visual classification of the resulting cluster centers. No quantitative criterion is given for the cluster count, no error bars or seed ensembles are provided for the histograms in Fig. 7, and Section 4 acknowledges that the analysis assumes a single global maximum per kernel. These limitations are real and affect the cross-case generality claim. To make the claim load-bearing, the authors should report a reproducible cluster-selection rule, quantify the classification of filters (e.g., by peak location and bandwidth), and show stability across multiple training runs.","section":"Section 3.3, Fig. 6 and Fig. 7"},{"comment":"The manuscript does not report error bars or seed ensembles for the activation spectra, which are channel-averaged and meridionally averaged. Fig. 5 and Fig. 7 appear to be based on a single training run per configuration. Because the paper's mechanism claim is that the spectral underestimation is a systematic property of the learned filters rather than an artifact of one initialization, the authors should either provide ensemble statistics over several independent training runs or explicitly justify why a single run is representative. This is especially important given that the central comparison in Fig. 5 Row a is confounded by input-spectrum differences, as noted above.","section":"Section 3.2, Fig. 5; Section 2.3"}],"minor_comments":[{"comment":"The symbol ⊛ is overloaded: it denotes spatial convolution in Eqs. (8) and (10), but in Eq. (12) it is used for convolution in the frequency domain. Please introduce a separate notation for frequency-domain convolution. Also, the statement that the Fourier transform 'can still be derived analytically' is misleading, because the resulting expression depends on the input-dependent set {h_j^ell > 0}, which is not known in closed form.","section":"Eq. (12), Section 2.4"},{"comment":"The RMSE formula in Eq. (13) is typeset incorrectly: the square root appears to apply only to the numerator, leaving the denominator outside the root in a way that does not match the intended normalized RMSE. Please fix the equation and clarify the normalization.","section":"Eq. (13), Section 2.5"},{"comment":"The caption says 'Rows a to d show spectra of hidden layers for different models and cases,' but it does not specify which networks are compared in each row (e.g., BNN0,0 vs BNN0,i in Row a, TLNN0,i vs BNNi,i in Rows b-d). Please make the row-by-row comparison explicit.","section":"Fig. 5 caption"},{"comment":"The phrase 'the learned weights and biases from one dataset underestimate the out-of-distribution sample spectra' presumes the causal conclusion that the paper is trying to establish. Until the same-input control is added, I suggest rewording to something like 'are associated with lower activation spectra on out-of-distribution inputs' in the abstract and in the Section 3.2 mechanism statement.","section":"Abstract and Section 3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper has a clear and useful empirical contribution, and I do not see circularity in the main analysis: the spectral quantities are computed from trained networks, not fitted to the conclusion. The key risk is that Fig. 5 Row a, which is the primary evidence for the paper's central mechanism, conflates input-spectrum differences with weight-induced underestimation. This is fixable by adding a same-input comparison of BNN0 and BNNi,i on identical target samples, plus ensemble statistics. The paper's strong abstract claims should be tempered until that control is added. I recommend major revision rather than rejection because the proposed mechanism is plausible and the experimental setup is otherwise well designed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth taking seriously. It extends Subel et al.'s spectral analysis of transfer learning to a two-layer quasi-geostrophic ocean model with anisotropic jets, and it adds layer-wise activation spectra and a kernel-maxima histogram method. The one-layer retraining result is clean and the recommendation that spectral RMSE is a better generalization metric than MSE or CC is backed by the offline and online comparisons. The code is on GitHub, which is a plus. On those grounds, the paper deserves a serious referee.\n\nThe soft spots are mostly statistical, but one is more serious. The central claim — that frozen weights/biases from the source 'underestimate' out-of-distribution sample spectra as they propagate — is not actually isolated by the evidence. Figure 5 Row a compares BNN0 on Case 0 inputs against the same BNN0 on Case 1–3 inputs. Since the first-layer activations are just the fixed kernels convolved with the input velocities, the activation-spectrum gap can be entirely due to the inputs having different spectra, independent of any 'underestimation' by the weights. The proper control is a same-input comparison: run BNN0 and BNNi,i on identical target samples and show BNN0's activations sit below BNNi,i's. That comparison is only shown after retraining (TLNN0,i vs BNNi,i), not for the pre-TL failure. Until that control is added, the mechanism is plausible but not demonstrated.\n\nThe other issues are minor-to-moderate and fixable: activation spectra are shown without seed ensembles or error bars, the k-means cluster count is chosen by visual inspection, and the abstract's 'broadly applicable' overpromises in a way the discussion quietly retracts. None of these are fatal, and the single-seed issue would lose much of its sting if the confound were addressed.\n\nBottom line: this is a paper for people working on data-driven subgrid parameterization and transfer learning for geophysical turbulence. It gives a useful diagnostic framework and a specific, checkable claim about why one-layer retraining fixes spectral bias. I would not accept it as is, but I would send it to peer review and push for the same-input pre-TL comparison plus a small seed ensemble. With those changes, the mechanistic story would stand on firmer ground.","headline":"A useful and largely well-executed extension of spectral analysis to transfer learning for ocean SGS models, but the central mechanistic claim in Fig. 5 is confounded by input-spectrum differences and needs a proper same-input control before it carries weight.","tokens_in":19612,"tokens_out":1814,"would_cite":false,"duration_ms":17594,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Spectral mismatch explains why a CNN for ocean subgrid forcing fails to generalize, and retraining one hidden layer corrects it.","keywords":["transfer learning","subgrid-scale parameterization","quasi-geostrophic turbulence","convolutional neural networks","activation spectra","Fourier analysis","out-of-distribution generalization","ocean eddy parameterization"],"falsifier":"Train the same BNN architecture several times from different random seeds and compare out-of-distribution activation spectra: if the underestimation does not recur across seeds, the spectral mechanism is not systematic. Alternatively, retrain a late layer instead of the first hidden layer; if the output spectrum then matches the FDNS reference, the claim that the early-layer spectral bias is the bottleneck is falsified.","tokens_in":18675,"feed_emoji":"🌊","tokens_out":9095,"duration_ms":71388,"temperature":0.7,"pith_summary":"A 9-layer convolutional neural network trained to predict subgrid forcing in a two-layer quasi-geostrophic ocean turbulence model fails when the dynamics change—from isotropic eddies to anisotropic jets—because the frozen weights and biases underestimate the spectra of out-of-distribution activations as signals pass through the network. Retraining only the first hidden layer with 2–10 percent of target data corrects this underestimation, so the output spectrum approaches the filtered high-resolution reference. This spectral mechanism matters because transfer learning is a standard way to adapt climate and ocean models to new regimes, and the paper offers a concrete offline diagnostic—spectrum RMSE—that exposes generalization failure before expensive online simulation. It also finds that the learned kernels act as low-pass, Gabor, and high-pass filters regardless of flow isotropy, so transfer learning adjusts filter amplitudes and scale preferences rather than learning a new filter vocabulary.","feed_headline":"Spectral mismatch explains why ocean AI fails on new regimes","feed_subtitle":"Retraining one hidden layer on target data corrects the underestimated spectra, the paper shows.","key_machinery":"The central mechanism is the spectral evolution of activations through the CNN. For each layer, the Fourier transform of activation $j$ in layer $\\ell$, $\\hat{g}^j_\\ell$, is a sum over the ReLU-positive mask of the linear pre-activation spectrum, which itself is built from the padded-kernel spectra $\\widehat{W}^{\\beta,j}_\\ell$, the bias term $\\hat{b}^j_\\ell$ (nonzero only at zero wavenumber), and the previous activation spectrum $\\hat{g}^{\\beta}_{\\ell-1}$. By comparing channel-averaged, meridionally averaged activation spectra between BNN$_{0,0}$ and BNN$_{0,i}$ for each layer, the paper localizes the failure to early layers and shows that re-training layer 2 changes both the kernel spectra and the ReLU-positive regions, propagating an upshift that aligns the output spectrum with FDNS. The learned kernels themselves act as low-pass, high-pass, and Gabor (oriented band-pass) filters, and the analysis tracks how their dominant-wavenumber footprint changes under transfer learning.","core_discovery":"The paper claims that a CNN trained entirely on Case 0 and applied to Cases 1–3 underestimates the channel-averaged, meridionally averaged activation spectra from the earliest hidden layers onward, and this underestimation compounds toward the output, producing a subgrid-forcing spectrum that falls short of the filtered direct-numerical-simulation (FDNS) reference. Transfer learning with only the first hidden layer ($\\ell=2$) retrained on a small fraction of target data (2% and 10%) reverses this: the layer-2 spectra shift upward and the correction propagates through the frozen layers, so transfer-learned network spectra approach the base-network-on-target spectra and the FDNS reference. The Fourier-transformed kernels of layer 2, clustered by k-means, consistently fall into low-pass, Gabor, and high-pass filter families in all four cases; retraining mostly increases the amplitude at unchanged dominant wavenumbers and moves many dominant wavenumbers to lower $\\kappa=\\sqrt{k_x^2+k_y^2}$, meaning toward larger scales.","pith_inferences":["Inference: The layer-localized spectral gap could be used as a selection criterion for which layer to retrain in other architectures: retrain the first layer where the in- versus out-of-distribution activation-spectrum gap appears, rather than a fixed layer.","Inference: Because transfer learning shifts many kernel maxima toward lower wavenumbers, regimes dominated by large-scale energy may need only early-layer fine-tuning, while small-scale-dominated regimes may require retraining deeper layers; this is testable but not tested here.","Inference: The single-global-maximum analysis ignores kernels with multiple significant spectral peaks; counting all local maxima might reveal whether transfer learning works mainly by amplitude scaling or by spectral redistribution.","Inference: The same activation-spectrum diagnostic could be applied to other data-driven parameterizations (such as atmospheric convection or boundary-layer closures) to predict beforehand whether a pretrained model will fail on a warmer climate and how much adaptation data is needed."],"forward_implications":["Spectrum RMSE, not RMSE or correlation coefficient, is the offline metric that reveals whether a parameterization will generalize; models that look strong on standard errors can still be spectrally wrong.","Re-training only the first hidden layer with 2–10% of target data is enough to correct the spectral gap, so targeted transfer learning can replace full retraining in similar subgrid-scale applications.","The filter vocabulary (low-pass, Gabor, high-pass) is universal across isotropic and anisotropic training data, so adaptation between regimes is a matter of adjusting the amplitude and scale preference of existing filters, not of inventing new ones.","Online (a posteriori) tests show transfer-learned networks improve kinetic-energy spectra and potential-vorticity PDF tails wherever the base CNN leaves room, although scale-selective dissipation can mask these gains."],"supporting_citations":[{"why":"Supplies the kernel-spectral analysis approach for explaining transfer learning in data-driven turbulence modeling, which this paper extends to anisotropic ocean quasi-geostrophic flows.","marker":"Subel et al. [2023]"},{"why":"Establishes the CNN-based subgrid-scale closure with transfer learning for higher Reynolds numbers, providing the architectural and training-template basis.","marker":"Guan et al. [2022a]"},{"why":"Provides the data-generation, filtering, and coarse-graining procedure for the two-layer quasi-geostrophic benchmark that this study follows.","marker":"[Ross et al., 2023]"},{"why":"Demonstrates that transfer learning improves generalization of data-driven parameterization across Lorenz 96 systems, motivating the study's focus on TL for subgrid models.","marker":"Chattopadhyay et al. [2020]"},{"why":"Uses Fourier analysis of CNN kernels to connect learned filters to physical dynamics, supporting the interpretation of kernels as spectral filters.","marker":"Pahlavan et al. [2024b]"},{"why":"Provides the scaling framework distinguishing eddy and jet regimes, which justifies the choice of the four base and target configurations.","marker":"[Gallet and Ferrari, 2021]"}],"fun_headline_variants":["Retrain one layer to fix ocean AI spectral underestimation","Why ocean AI fails: underestimated spectra, fixed by one layer","Fourier analysis exposes spectral fix for ocean transfer learning","One-layer retrain corrects spectral bias in ocean turbulence AI"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The mechanism is inferred from a single trained network: if the underestimation in the shown layer-wise activation spectra is an artifact of one initialization or of the specific channel and meridional averaging, the claim that transfer learning works by correcting frozen-filter spectral bias collapses.","fun_headline_variants_meta":{"raw":{"variants":["Retrain one layer to fix ocean AI spectral underestimation","Why ocean AI fails: underestimated spectra, fixed by one layer","Fourier analysis exposes spectral fix for ocean transfer learning","One-layer retrain corrects spectral bias in ocean turbulence AI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000244,"raw_usage":{"total_tokens":1539,"prompt_tokens":961,"completion_tokens":578,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":577,"completion_tokens_details":{"reasoning_tokens":509}},"tokens_in":577,"tokens_out":578,"duration_ms":5469,"temperature":1.0,"reasoning_tokens":509,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:25:22.288440+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same BNN architecture several times from different random seeds and compare out-of-distribution activation spectra: if the underestimation does not recur across seeds, the spectral mechanism is not systematic. Alternatively, retrain a late layer instead of the first hidden layer; if the output spectrum then matches the FDNS reference, the claim that the early-layer spectral bias is the bottleneck is falsified.","supporting_citations":[],"review_version":1}