Initial Model Incorporation for Deep Learning FWI: Pretraining or Denormalization?

Bangyu Wu; Kai Yang; Meng Li; Ruihua Chen

REVIEW 4 major objections 6 minor 9 references

Initial Model Incorporation for Deep Learning FWI: Pretraining or Denormalization?

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read For neural FWI, denormalization outperforms pretraining as a way to incorporate the starting model.

desk verdict A useful empirical comparison of pretraining vs denormalization for DRFWI, but the general claim rests on a single Marmousi benchmark with test-set-tuned hyperparameters. read the letter →

arxiv 2506.05484 v1 pith:RFSJ3COX submitted 2025-06-05 cs.LG physics.geo-ph

Ruihua Chen , Bangyu Wu , Meng Li , Kai Yang This is my paper

classification cs.LGphysics.geo-ph

keywords fullwaveforminversionneuralreparameterizationpretrainingdenormalizationlow-frequencybiasnegativetransferlossofplasticityimplicitFWI

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Full waveform inversion (FWI) reconstructs subsurface velocity models by fitting observed seismic data, but it needs a good starting model and often stalls in local minima. Neural-network reparameterization (DRFWI) generates the velocity model from a network, and the starting model can be injected either by pretraining the network on that model or by adding the network output directly to it (denormalization). This paper argues that the two-stage pretraining workflow is the weaker option: it forces the network to fit a constant-mean model, makes its parameters inactive, and causes negative transfer, while denormalization lets the network represent only the high-frequency perturbation. Using Marmousi experiments with smooth and linear starting models, the paper reports that denormalization converges faster, recovers more deep-structure detail, and achieves lower error on all reported metrics. If correct, this simplifies the DRFWI workflow to a single stage with no pretuning and better accuracy.

What carries the argument

The load-bearing object is the denormalization operator $D_2(f_\Theta(I), m_{\mathrm{init}}) = f_\Theta(I) \odot S + m_{\mathrm{init}}$, which replaces the global mean matrix $M$ with the actual initial velocity model. It turns the initial model into a fixed (or adaptively learnable) baseline, so the network only needs to learn the residual perturbation; this is what decouples the prior from the network parameters, avoids the two-stage objective mismatch, and lets high-frequency components contribute more to gradients. The pretraining route is analyzed through the same operator with $M$ in place of $m_{\mathrm{init}}$, showing that the network must then fit a constant-velocity background plus perturbation, which is what triggers low-frequency bias and loss of plasticity.

What would settle it

Run both incorporations on a second synthetic model with strong lateral velocity contrasts and a deliberately poor starting model (e.g., a linear gradient far from the true background), keeping the same network, optimizer, and epoch budget; if Pretrain-DRFWI equals or beats Denorm-DRFWI in final MSE/SSIM, the claimed universal superiority of denormalization is refuted.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes a comparative claim: for implicit neural reparameterization of FWI, the denormalization operator $D_2(f_\Theta(I), m_{\mathrm{init}}) := f_\Theta(I) \odot S + m_{\mathrm{init}}$ (and its adaptive variant where $m_{\mathrm{init}}$ is learnable) outperforms the pretraining pipeline that first fits the network to $m_{\mathrm{init}}$ through $D_1(f_\Theta(I)) := f_\Theta(I) \odot S + M$ with a constant mean $M$. The mechanism is that pretraining requires the network to represent the full velocity model, including low frequencies that the network learns first and slowly refines, and the two-stage objective mismatch traps the parameters in a local optimum near the pretrained configuration; denormalization instead lets the network represent only the perturbation, so high-frequency gradients dominate and plasticity is preserved. The paper supports the claim with Marmousi-based experiments across smooth and linear starting models and diagnostic cosine-similarity and parameter-distance measurements.

Load-bearing premise

The paper's general conclusion rests on the premise that the ranking observed on one Marmousi model with two hand-built starting models carries over to other geometries, acquisition designs, and data conditions; nothing in the experiments establishes that external validity.

Editorial extensions

If this is right

Adopting denormalization removes the pretraining stage entirely, so users need not tune pretraining epochs or learning rates.
The network capacity is spent on the perturbation rather than the background model, which should improve recovery of deep high-frequency structure in FWI.
The adaptive variant A-Denorm-DRFWI, which lets the initial model itself be learnable, is the best-performing configuration on every reported metric.
The diagnosis in terms of low-frequency bias and negative transfer implies that the advantage of denormalization should grow as the initial model gets less accurate, while pretraining degrades.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same decoupling argument likely extends to other reparameterized inversion schemes (e.g., wavefield- or noise-input networks), but the paper only tests a coordinate-based implicit network, so that extension is an inference, not a demonstrated claim.
The paper's parameter-distance diagnostic suggests a cheap early-stopping rule: monitor cosine similarity between successive parameter states; if it stays near 1 early in FWI, the network has lost plasticity and denormalization should be preferred.
A natural stress test not run in the paper is to compare the two incorporations under noisy data, fewer sources, and deeper targets; the mechanism predicts denormalization's margin widens.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

This short letter does a clean head-to-head comparison of two ways to embed an initial velocity model in neural-reparameterized FWI: the two-stage pretraining approach from IFWI and the denormalization trick from DIP-FWI. The authors argue, with supporting experiments, that denormalization is simpler, converges faster, and gives lower error on a Marmousi test. That is a useful practical message for people building DRFWI workflows.

The genuinely new part is the mechanism analysis: relative low-frequency gradient content, negative transfer via parameter cosine similarity and Euclidean distance, and the plasticity-loss argument. These are plausible and give the empirical ranking more than just a table of numbers. I also appreciate that they test two initial models (smooth and linear) rather than one, and the metrics are consistent across four model/metric combinations.

The soft spots are real but not fatal. The entire evidence base is one 2D Marmousi model with one acquisition geometry and one network architecture. There are no repeated runs or error bars, so we have no idea whether the ranking is stable across seeds or optimizer states. More importantly, the pretraining hyperparameters were selected by evaluating the final inversion MSE against ground truth—that is test-set tuning, and it biases the comparison in favor of denormalization. The best denorm variant, A-Denorm, also adds a learnable initial-model matrix with 94×288 extra parameters that the pretraining baseline does not have. Neither factor is controlled, so the abstract's general claim that denormalization 'enhances inversion accuracy' should be read as 'on this benchmark, under these hyperparameter choices.'

That said, the direction of the result is plausible and consistent with the low-frequency bias argument. The paper is honest about the pretraining sensitivity, and the authors do not hide the tuning procedure. I would not call the central claim unsupported; I would call it not yet generalized.

Who is this for? Practitioners of DRFWI who want a quick answer to which initialization method to use. It deserves a serious referee, but the referee should push for a second benchmark (even a modified Marmousi or a simpler model) and repeated runs before publication. As it stands, a cautious 'revise' is the right call.

Referee Report

4 major / 6 minor

Summary. The paper compares two ways of embedding an initial velocity model into neural-network-reparameterized full waveform inversion (DRFWI): the two-stage pretraining approach (Pretrain-DRFWI) and the single-stage denormalization approach (Denorm-DRFWI), with the latter further split into static (S-Denorm) and adaptive (A-Denorm) variants. Using IFWI as the representative DRFWI method, experiments on a synthetic Marmousi model with smooth and linear initial models report that both denormalization variants converge faster and achieve lower MSE/MAE and higher R2/SSIM than pretraining, with A-Denorm-DRFWI performing best. The authors attribute this to denormalization's avoidance of low-frequency bias and negative transfer caused by the two-stage pretraining workflow.

Significance. The question is practically relevant: if denormalization is consistently better than pretraining, it would simplify DRFWI workflows and improve accuracy without additional tuning. The paper offers a clear comparison on a standard benchmark and a plausible mechanistic explanation in terms of low-frequency bias and negative transfer. The experimental setup is described in sufficient detail to be reproducible in principle. However, the external validity of the central claim is not yet established: the evidence comes from a single Marmousi model, a single acquisition geometry, and a single run per configuration, with no uncertainty quantification.

major comments (4)

[Section III.B and Fig. 3] The pretraining hyperparameters (epochs and learning rate) are selected by evaluating the MSE between the inverted model and the ground-truth velocity model. This is a test-set leakage: the ground truth of the inference problem is used to tune hyperparameters, and no validation split is used. The reported metrics are therefore not unbiased estimates of generalization, and the comparison is not controlled because Denorm-DRFWI hyperparameters are not tuned in the same way. Please use a validation set or explicitly report that all methods share the same tuning protocol, and reassess whether the ranking is unchanged.
[Section III.C and Table I] The central claim of the abstract and conclusion — that denormalization accelerates convergence and enhances inversion accuracy compared with pretraining — rests on a single Marmousi model with a single acquisition geometry and a single run per method. No seed variation or repeated runs are reported, so the stability of the ranking across random initializations is unknown, and no error bars are given. Without at least one additional benchmark (e.g., a different subsurface model or acquisition geometry) or multiple seeds, the general conclusion is not empirically established. Please add such experiments or explicitly restrict the conclusion to the tested configuration.
[Section II.B and Table I] A-Denorm-DRFWI treats the initial model minit as a learnable matrix, adding 94×288 additional trainable parameters compared with Pretrain-DRFWI and S-Denorm-DRFWI. The superior accuracy of A-Denorm could therefore be due to increased model capacity rather than the denormalization operator itself. To attribute the gain to the denormalization mechanism, the parameter count should be controlled, for example by adding an equivalent number of parameters to the competing methods or by ablating the learnable minit.
[Section III.D2 and Table II] The negative-transfer/loss-of-plasticity analysis presents cosine similarity and Euclidean distance between network parameters before and after training. These are descriptive quantities with no control condition, no statistical test, and no direct link to the inversion metrics. The inference of 'loss of plasticity' from small parameter changes between stages would be more compelling with additional evidence such as gradient norms, effective Jacobian rank, or a comparison against a random-initialization baseline.

minor comments (6)

[Section III.A] There is a typo: 'Denorm-DeRFWI' should be 'Denorm-DRFWI'.
[Equations (2) and (3)] The symbol M is used for the mean matrix in Eq. (2) and for the acquisition mask in Eq. (3). This reuse of notation is confusing; rename one of them (e.g., use μ for the mean and A for the mask).
[Section II.B and III.A] The phrase 'standard variance matrix' should be 'standard deviation matrix'.
[Fig. 6] The term 'Target fitting model' is not defined. Clarify whether this is the network output after pretraining or after some stage of the inversion, and how it is computed.
[Section III.A] The paper mentions 'the source code of IFWI [1]' but provides no link or version information. Please include a reference or repository URL for reproducibility.
[Section III.B] The sentence 'Fig. 4 (b) and (f) denote the final inverted model' is ambiguous because (b) and (f) correspond to different initial models; specify 'with the smooth and linear initial velocity model, respectively' at the point of reference.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the pretraining-vs-denormalization comparison is empirical, and the paper's architectural distinction (Eq. 2 vs Eq. 4) is the method being tested, not a hidden reuse of the conclusion.

full rationale

The paper's central claim is that Denorm-DRFWI outperforms Pretrain-DRFWI on a Marmousi benchmark, supported by inversion-error curves, wavenumber spectra, and cosine-similarity statistics. The key operators D1(f_Theta(I)) := f_Theta(I) * S + M and D2(f_Theta(I), minit) := f_Theta(I) * S + minit differ by design: D2 replaces the global mean with the initial model, so the network represents a perturbation around minit. This is an architectural fact about the method being proposed, not a conclusion derived from itself. The paper does not fit a parameter and then report that parameter as a prediction. The pretraining epochs and learning rate are selected using test MSE against the ground truth in Section III.B, which is a hyperparameter-selection weakness and could even favor Pretrain-DRFWI, but it does not force the denormalization advantage. The mechanistic explanations (low-frequency bias, negative transfer, loss of plasticity) are supported by measured error curves, wavenumber spectra, and parameter-distance/cosine-similarity tables in Section III.D, not by citing a theorem from the present authors. References [1] and [6] are prior baseline works by other research groups and are used as baselines or building blocks, not as load-bearing self-citations. There is no self-citation chain, no uniqueness theorem imported from the authors' own prior work, and no ansatz smuggled in via citation. The main weakness is external validity: the general conclusion rests on one 2D Marmousi model, two hand-built initial models, one acquisition geometry, and single runs per method, with no seed variation or second benchmark. That is a correctness/evidence concern, not a circularity concern. Consequently, no specific circular step can be exhibited, and the appropriate score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

No new physical entities are introduced. The main free parameters are pretraining hyperparameters tuned on the test target; the denormalization normalization constants are inherited from IFWI. The result depends on standard assumptions about acoustic modeling, network capacity, and representativeness of the Marmousi benchmark.

free parameters (3)

Pretraining epochs per initial model = 1000 (smooth), 2500 (linear)
Chosen by empirical search to minimize final inversion MSE against ground truth on the test Marmousi model (Sec. III-B), so the comparison is partially fit to the target.
Pretraining learning rate = 5e-5
Selected from the sweep in Fig. 3 based on the same test-MSE criterion.
Denormalization scale S and global mean M = S = 1 km/s, M = 3 km/s
Taken from IFWI settings rather than fitted here; they define the normalization but do not drive the comparison (Sec. III-A).

assumptions (3)

domain assumption The acoustic wave equation and the chosen finite-difference/RNN forward operator accurately model the observed seismograms.
Used throughout the forward modeling and inversion; not validated against field data.
domain assumption The 4-layer 128-unit MLP with sine activation can represent the velocity models needed for both methods, including high-frequency perturbations.
Network capacity is fixed and assumed sufficient; no capacity check is reported.
domain assumption The Marmousi model and the two initial models are representative of realistic FWI starting models.
The conclusion is drawn from these two cases; no other subsurface models are tested.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Initial Model Incorporation for Deep Learning FWI: Pretraining or Denormalization?." pith.science (2026). https://pith.science/paper/RFSJ3COX

@misc{pith2026250605484,
  author       = {Pith},
  title        = {Pith review of: Initial Model Incorporation for Deep Learning FWI: Pretraining or Denormalization?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RFSJ3COX}},
  note         = {Machine review of arXiv:2506.05484}
}

read the original abstract

Subsurface property neural network reparameterized full waveform inversion (FWI) has emerged as an effective unsupervised learning framework, which can invert stably with an inaccurate starting model. It updates the trainable neural network parameters instead of fine-tuning on the subsurface model directly. There are primarily two ways to embed the prior knowledge of the initial model into neural networks, that is, pretraining and denormalization. Pretraining first regulates the neural networks' parameters by fitting the initial velocity model; Denormalization directly adds the outputs of the network into the initial models without pretraining. In this letter, we systematically investigate the influence of the two ways of initial model incorporation for the neural network reparameterized FWI. We demonstrate that pretraining requires inverting the model perturbation based on a constant velocity value (mean) with a two-stage implementation. It leads to a complex workflow and inconsistency of objective functions in the two-stage process, causing the network parameters to become inactive and lose plasticity. Experimental results demonstrate that denormalization can simplify workflows, accelerate convergence, and enhance inversion accuracy compared with pretraining.

Figures

Figures reproduced from arXiv: 2506.05484 by the authors.

**Figure 1.** Initial Model Incorporation for DRFWI (e.g., Implicit FWI [1]) with pretraining and denormalization. (a) Pretrain-DRFWI includes two stages (i.e., [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

**Figure 2.** Marmousi model. The red stars denote the source locations. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 4.** Comparisons of different methods using different initial models. (a) Smooth initial model; (e) Linear initial model; (b) - (d) Inversion results from [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗

Figures from the paper (3 more)

**Figure 5.** Figure 5: The error of model w.r.t. training epochs using different methods. (a) [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]

**Figure 6.** Figure 6: Target fitting model. (a) Pretrain-DRFWI; (b) Denorm-DRFWI [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]

**Figure 7.** Figure 7: Wavenumber spectrum along four distinct velocity profiles at different [PITH_FULL_IMAGE:figures/full_fig_p005_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

9 extracted references · 7 canonical work pages

[1]

Implicit seismic full wave- form inversion with deep neural representation,

J. Sun, K. Innanen, T. Zhang, and D. Trad, “Implicit seismic full wave- form inversion with deep neural representation,” Journal of Geophysical Research: Solid Earth, vol. 128, no. 3, p. e2022JB025964, 2023

work page 2023
[2]

An overview of full-waveform inversion in exploration geophysics,

J. Virieux and S. Operto, “An overview of full-waveform inversion in exploration geophysics,” Geophysics, vol. 74, no. 6, pp. WCC1–WCC26, 2009

2009
[3]

Review of crosshole ground-penetrating radar full-waveform inversion of experimental data: Recent developments, challenges, and pitfalls,

A. Klotzsche, H. Vereecken, and J. van der Kruk, “Review of crosshole ground-penetrating radar full-waveform inversion of experimental data: Recent developments, challenges, and pitfalls,” Geophysics, vol. 84, no. 6, pp. H13–H28, 2019

work page 2019
[4]

Physics-guided data-driven seismic inversion: Recent progress and future opportunities in full-waveform inversion,

Y . Lin, J. Theiler, and B. Wohlberg, “Physics-guided data-driven seismic inversion: Recent progress and future opportunities in full-waveform inversion,” IEEE Signal Processing Magazine, vol. 40, no. 1, pp. 115– 133, 2023

work page 2023
[5]

Parametric convolutional neural network- domain full-waveform inversion,

Y . Wu and G. A. McMechan, “Parametric convolutional neural network- domain full-waveform inversion,” Geophysics, vol. 84, no. 6, pp. R881– R896, 2019

work page 2019
[6]

Integrating deep neural networks with full-waveform inversion: Reparameterization, regularization, and uncertainty quantification,

W. Zhu, K. Xu, E. Darve, B. Biondi, and G. C. Beroza, “Integrating deep neural networks with full-waveform inversion: Reparameterization, regularization, and uncertainty quantification,” Geophysics, vol. 87, no. 1, pp. R93–R109, 2022

work page 2022
[7]

Overview frequency princi- ple/spectral bias in deep learning,

Z.-Q. J. Xu, Y . Zhang, and T. Luo, “Overview frequency princi- ple/spectral bias in deep learning,” Communications on Applied Math- ematics and Computation, pp. 1–38, 2024

work page 2024
[8]

A survey on negative transfer,

W. Zhang, L. Deng, L. Zhang, and D. Wu, “A survey on negative transfer,” IEEE/CAA Journal of Automatica Sinica, vol. 10, no. 2, pp. 305–329, 2023

work page 2023

Show all 9 references

[9]

Loss of plasticity in deep continual learning,

S. Dohare, J. F. Hernandez-Garcia, Q. Lan, P. Rahman, A. R. Mahmood, and R. S. Sutton, “Loss of plasticity in deep continual learning,” Nature, vol. 632, no. 8026, pp. 768–774, 2024

2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Implicit seismic full wave- form inversion with deep neural representation,

J. Sun, K. Innanen, T. Zhang, and D. Trad, “Implicit seismic full wave- form inversion with deep neural representation,” Journal of Geophysical Research: Solid Earth, vol. 128, no. 3, p. e2022JB025964, 2023

work page 2023

[2] [2]

An overview of full-waveform inversion in exploration geophysics,

J. Virieux and S. Operto, “An overview of full-waveform inversion in exploration geophysics,” Geophysics, vol. 74, no. 6, pp. WCC1–WCC26, 2009

2009

[3] [3]

Review of crosshole ground-penetrating radar full-waveform inversion of experimental data: Recent developments, challenges, and pitfalls,

A. Klotzsche, H. Vereecken, and J. van der Kruk, “Review of crosshole ground-penetrating radar full-waveform inversion of experimental data: Recent developments, challenges, and pitfalls,” Geophysics, vol. 84, no. 6, pp. H13–H28, 2019

work page 2019

[4] [4]

Physics-guided data-driven seismic inversion: Recent progress and future opportunities in full-waveform inversion,

Y . Lin, J. Theiler, and B. Wohlberg, “Physics-guided data-driven seismic inversion: Recent progress and future opportunities in full-waveform inversion,” IEEE Signal Processing Magazine, vol. 40, no. 1, pp. 115– 133, 2023

work page 2023

[5] [5]

Parametric convolutional neural network- domain full-waveform inversion,

Y . Wu and G. A. McMechan, “Parametric convolutional neural network- domain full-waveform inversion,” Geophysics, vol. 84, no. 6, pp. R881– R896, 2019

work page 2019

[6] [6]

Integrating deep neural networks with full-waveform inversion: Reparameterization, regularization, and uncertainty quantification,

W. Zhu, K. Xu, E. Darve, B. Biondi, and G. C. Beroza, “Integrating deep neural networks with full-waveform inversion: Reparameterization, regularization, and uncertainty quantification,” Geophysics, vol. 87, no. 1, pp. R93–R109, 2022

work page 2022

[7] [7]

Overview frequency princi- ple/spectral bias in deep learning,

Z.-Q. J. Xu, Y . Zhang, and T. Luo, “Overview frequency princi- ple/spectral bias in deep learning,” Communications on Applied Math- ematics and Computation, pp. 1–38, 2024

work page 2024

[8] [8]

A survey on negative transfer,

W. Zhang, L. Deng, L. Zhang, and D. Wu, “A survey on negative transfer,” IEEE/CAA Journal of Automatica Sinica, vol. 10, no. 2, pp. 305–329, 2023

work page 2023