{"id":"8435d4f7-0904-48eb-8a2c-c58753cdc042","arxiv_id":"2505.06331","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A learnable mask F(z)=1-exp(-α²z²), applied pointwise before activation, improves PINN accuracy and stability on several PDE benchmarks and activation functions.","lead":"This paper proposes Mask-PINNs, which add a learnable pointwise mask function to physics-informed neural networks to keep hidden feature distributions stable during training. The authors report lower solution errors on convection, wave, and Helmholtz equations, along with better scaling to wider networks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Local Lipschitz contraction near zero does not establish global training-time stability; the mask saturates to 1 away from zero, leaving the central ICS mechanism unproven.","rationale":"The reader's weakest_assumption identifies exactly the same gap: Proposition 1 is a local statement near z=0, while the paper's conclusions require global, training-time control of feature distributions. My review reconfirms this as the most load-bearing concern. The mask F(z)=1-exp(-α²z²) has the property F(z)≈α²z² near zero but F(z)≈1 for large |z|, so the contraction is inherently local; the paper never proves that pre-activations remain in that local regime throughout optimization. This is not a matter of disagreement with community consensus; it is an internal gap between what is proven (a local asymptotic bound) and what is claimed (mitigation of internal covariate shift over full training). The empirical results are consistently in the proposed method's favor and are reported over five trials, but without error bars or code, the quantitative strength of the improvements is difficult to assess. Those are secondary concerns; the theory-to-practice bridge is primary. The reader's CONDITIONAL verdict remains appropriate: the empirical work is promising, but the theoretical framing should be tempered or supplemented with evidence that the local regime is actually the operating regime. No change to the verdict is needed from this stress-test pass.","tokens_in":12556,"tokens_out":4663,"duration_ms":49551,"concrete_test":"Reproduce the convection equation experiment (Section 4.1) with Tanh and SoftPlus, initializing α=1.0 as in the paper. Log per-layer pre-activations at initialization and every 1,000 iterations for the full 50,000 iterations. For each layer, compute the fraction of pre-activation entries with |z| < sqrt(ln 2)/α (the regime where F(z)<0.5 and the quadratic contraction is active) and the ratio L_H(r)/L_σ(r) at r = per-layer standard deviation and 95th percentile. If, for layers that drive accuracy, this fraction drops below roughly 0.5 or the ratio approaches 1 during early training while the accuracy gain persists, then Proposition 1's local regime is not the operating regime and the theoretical bridge fails; if the fraction remains high throughout training, the proposed mechanism is confirmed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's theoretical support for the central stability claim rests on Proposition 1 (Section 3.2), which shows only that the local Lipschitz constant of H=F·σ on [-r,r] is asymptotically smaller than that of σ as r→0 (Eqs. 19-31). Remark 2 then asserts, without proof, that under Xavier initialization pre-activations 'concentrate near zero' and that this local contraction keeps the output 'zero centered and stable around initialization' and, by extension, throughout training. However, F(z)=1-exp(-α²z²) saturates to 1 for |z| ≳ 1/α, so the contraction is confined to a neighborhood of zero; once pre-activations drift beyond that neighborhood, the mask acts essentially as the identity and the claimed mechanism disappears. No result in the paper bounds ||z_l(t)|| or the feature distribution over training, and no analysis connects the local Lipschitz bound to the PDE loss landscape or to gradient flow. Thus the theoretical mechanism asserted to mitigate ICS is unproven at the global, training-time scale; the empirical gains could be driven by a different effect (e.g., effective early-layer scaling or regularization) without the ICS story being correct. This is the load-bearing gap in the central claim, and the observed improvements alone do not close it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Mask-PINNs, a modification of standard PINN architectures in which a learnable pointwise mask F(z)=1-exp(-alpha^2 z^2) multiplies the activation output at each hidden layer. The mask is intended to reduce internal covariate shift (ICS) while preserving the deterministic, pointwise input-output mapping required by PDE-based losses. The authors provide a local Lipschitz analysis near zero (Proposition 1), then evaluate the method on convection, wave, and Helmholtz equations with Tanh, GELU, SiLU, and SoftPlus activations, reporting consistent relative-L2-error improvements over several baselines. They also report that Mask-PINNs allow wider networks to be trained more effectively and present an NTK eigenvalue-spectrum comparison on Burgers' equation. The central empirical claim is that Mask-PINNs improve accuracy, convergence stability, and robustness across PDE benchmarks and activations.","tokens_in":12800,"tokens_out":5980,"duration_ms":60553,"significance":"If the empirical results hold, Mask-PINNs would be a simple, inexpensive architectural change with broad applicability to PINN training. The paper's strengths are its systematic benchmarking across three PDEs and four activation functions, the straightforward and physically pointwise mechanism, and the demonstration that wider networks remain trainable. The authors also make a genuine attempt at a theoretical justification, although, as detailed below, that justification is only local and does not currently support the global ICS-stability claim. The consistent improvements over strong baselines, even without error bars, make this a useful contribution to the literature.","major_comments":[{"comment":"The theoretical support for the central claim is incomplete. Proposition 1 shows only that, as r->0, the local Lipschitz constant of H=F*sigma on [-r,r] is asymptotically smaller than that of sigma. Since F(z)=1-exp(-alpha^2 z^2) saturates to 1 for |z| large relative to 1/alpha, the contraction is confined to a neighborhood of zero. Remark 2 asserts without proof that under Xavier initialization pre-activations concentrate near zero and that this local contraction keeps outputs stable throughout training. No bound on ||z_l(t)|| or on feature distributions over the training trajectory is provided, and no analysis connects the local Lipschitz bound to gradient flow or to the PDE loss landscape. The paper should either supply a global, training-time argument (for example, a mean-field or dynamical-isometry analysis that bounds pre-activation variances under the masked activation) or substantially soften the theoretical claims and present the mechanism as an empirical regularizer.","section":"Section 3.2, Proposition 1 and Remark 2"},{"comment":"All quantitative results are reported as averages over 5 independent trials, but no standard deviations, confidence intervals, or individual trial values are given. For claims of 'consistent and significant improvements', the absence of variance information is a problem, especially for close entries such as Table 2, GELU row, where Mask-PINN gives 2.90e-3 versus LAAF's 4.37e-3. Please report mean +/- std or provide the per-seed results, and state the random seed handling.","section":"Section 4, Tables 1-3"},{"comment":"The mask scaling parameter alpha is initialized differently across problems and activations: 1.0 for most settings, 5.0 for Tanh and 2.0 for SoftPlus in the Helmholtz experiments. The conclusion explicitly acknowledges that the initial value of alpha has a considerable impact on training dynamics. As alpha controls the sharpness of the mask, the claimed robustness is conditional on this hyperparameter. The authors should provide a sensitivity study over alpha initializations or a principled, task-independent initialization strategy.","section":"Section 4.3 and Section 6"},{"comment":"The NTK analysis is not derived for the masked network. The kernel formulas in Eqs. (44)-(50) are written for a generic network and are not specialized to Mask-PINNs; the text does not show how F enters the kernel, and no analytical claim about eigenvalues is proved. The eigenvalue-spectrum plots are empirical observations. Either derive the NTK of the masked architecture and connect its spectral properties to the mask, or explicitly present this section as an empirical diagnostic rather than a theoretical analysis.","section":"Section 5, Eqs. (44)-(50)"}],"minor_comments":[{"comment":"For the case sigma(0)=0, the expansion of H'(u) appears to be H'(u)=3 alpha^2 sigma'(0) u^2 + O(u^3), not alpha^2 sigma'(0) u^2 + O(u^3); the asymptotic order is unchanged, but the displayed equation should be corrected.","section":"Section 3.2, proof of Proposition 1"},{"comment":"Remark 2 refers to 'Proposition 2' when the relevant result is Proposition 1; please correct the cross-reference.","section":"Remark 2"},{"comment":"The claim that placing the mask before the activation function 'can therefore reintroduce ICS' is not supported by an experiment or by a quantitative argument. A small ablation comparing mask-before-activation, mask-after-activation, and the proposed side-by-side placement would make the design choice convincing.","section":"Section 3.3"},{"comment":"The pre-activation distribution figures and variance curves would be more informative with axis labels, units, and error bars or multiple trial overlays; as presented, they are qualitative.","section":"Section 4, Figures 4-9"},{"comment":"The wider-network experiment uses only 3 hidden layers and does not compare against other wide-network methods (e.g., PirateNets or residual architectures). The claim that Mask-PINNs 'enable effective use of wider networks' would be stronger with a comparison to the best-performing baseline at the same widths.","section":"Section 4.4"},{"comment":"No code or data availability statement is provided. Releasing the implementation and random seeds would substantially improve reproducibility of the reported five-trial averages.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The empirical contribution is promising, but the manuscript currently overclaims the theoretical mechanism: the local Lipschitz result does not imply global training-time ICS mitigation, and the NTK section is not actually derived for the masked network. I recommend asking the authors to add error bars, report alpha sensitivity, and either strengthen the theory or reframe the paper as an empirical method with a local stability heuristic. The core idea is simple and likely useful, so I do not see the need for rejection if the framing is corrected."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Colleague],\n\nThe empirical core of this paper is in good shape, and I'd bet the effect is real: a pointwise mask F(z)=1-exp(-α²z²) applied alongside the activation stabilizes PINN training and gives consistent accuracy gains on convection, wave, and Helmholtz equations across Tanh, GELU, SiLU, and SoftPlus. That is a genuinely useful result for practitioners, and the mask is simple enough that others will try it. Credit where due: the authors compare against several baselines, run five trials, check wider networks, and give an intuitive explanation of why BN/LN are unsuitable for PINNs.\n\nThe soft spot is the theory, and it is load-bearing. Proposition 1 shows only that the local Lipschitz constant of F·σ near zero is asymptotically smaller than σ's. But F saturates to 1 for |z| ≳ 1/α, so the contraction is confined to a small neighborhood of zero. Remark 2 then asserts, without proof, that this keeps features 'zero centered and stable' throughout training and across depths. No bound on feature distributions over training is given. So the ICS-mitigation mechanism is not established; the observed gains could come from a different effect, e.g., an implicit scaling or regularization. The NTK section is descriptive, not a proof. This is fixable: either prove a global contraction (unlikely with this mask) or reframe the paper as an empirical stabilization technique with a local-sensitivity rationale.\n\nOther issues: only means are reported, no error bars, and 5 trials is enough to hope but not to know — several table entries are within a factor of two of the runner-up. The α initialization is tuned per problem/activation (Helmholtz uses 5.0 for Tanh, 2.0 for SoftPlus), which the authors themselves flag in the conclusion; it doesn't sink the paper, but it tempers the robustness claim. No code is released, which makes the deeper-PINN and NTK results hard to verify.\n\nWho's this for? Anyone working on PINN training stability or coordinate-based MLPs. It deserves a serious referee, but the revision really should add uncertainty quantification, release code, and soften the theory from 'mitigates ICS' to 'stabilizes feature distributions near initialization, with consistent empirical gains.' I'd want to see those changes before building on it myself.","headline":"A simple mask layer that empirically stabilizes PINN training, but the paper's central ICS explanation is not proven by the local Lipschitz analysis.","tokens_in":13283,"tokens_out":2830,"would_cite":false,"duration_ms":28692,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A learnable mask stabilizes PINN training and cuts solution error.","keywords":["physics-informed neural networks","internal covariate shift","learnable mask function","activation function","partial differential equations","feature distribution stabilization","neural tangent kernel","wide neural networks"],"falsifier":"Train a 30-layer Mask-PINN on the convection equation, logging per-layer pre-activation mean, variance, and gradient norms every 1,000 iterations: if deeper layers' distributions drift away from zero or gradient norms collapse while $\\alpha$ stays positive, then the local contraction proved near initialization does not guarantee the global stability the paper claims.","tokens_in":12363,"feed_emoji":"🧠","tokens_out":4961,"duration_ms":47432,"temperature":0.7,"pith_summary":"Mask-PINNs claims that a single learnable, input-conditioned mask placed at every hidden layer can tame the internal covariate shift that destabilizes physics-informed neural networks, without using the batch or feature statistics that break PINNs' pointwise physical constraints. The mask multiplies each pre-activation by $F(z)=1-\\exp(-\\alpha^2 z^2)$, a smooth inverted-Gaussian curve that contracts large activations and, close to zero, behaves like $\\alpha^2 z^2$, lowering the local sensitivity of the activation. The paper argues this keeps feature distributions centered and compact, and reports consistently lower relative L2 errors than vanilla, ResNet, weight-normalized, LAAF, PirateNet, and ABU baselines on convection, wave, and Helmholtz equations across Tanh, GELU, SiLU, and SoftPlus. It also reports that masked networks keep improving as width grows, where vanilla PINNs degrade. If right, the method is a simple architectural change that widens the usable design space of PINNs.","feed_headline":"One mask tames drift inside physics-informed neural nets","feed_subtitle":"Pointwise modulation preserves physical constraints while cutting error and enabling wide networks.","key_machinery":"The load-bearing object is the mask function $F(u)=1-\\exp(-\\alpha^2 u^2)$, applied pointwise to pre-activations and multiplied before the activation, so each hidden layer computes $H=F(z)\\sigma(z)$. Since $0\\le F\\le 1$, it globally contracts activation magnitudes; near $u=0$ the Taylor expansion $F(u)=\\alpha^2 u^2+O(u^4)$ makes the derivative of the masked activation vanish faster than the bare activation's, which Proposition 1 turns into a reduction of the local minimal Lipschitz constant. That local contraction, combined with the globally bounded magnitude, is the mechanism the paper uses to explain why feature distributions stay centered and compact, and why wider networks do not saturate.","core_discovery":"The paper's central claim is that inserting the mask $H = F(z)\\sigma(z)$, with $F(z)=1-\\exp(-\\alpha^2 z^2)$ and $\\alpha$ learnable, into each hidden layer counteracts the drift in pre-activation distributions that makes PINNs hard to train. The mask is pointwise and deterministic, so the PDE constraints, which require derivatives of the network output with respect to inputs, remain intact—unlike batch or layer normalization, which couple outputs across samples or force every feature to the same statistics. The paper proves that near zero the mask reduces the activation's local Lipschitz constant by an order: for activations with $\\sigma(0)=0$ the leading term scales as $r^3$ versus $r$ for vanilla, and for $\\sigma(0)\\neq 0$ as $r^2$ versus $r$. Empirically, Mask-PINN reports the best relative L2 error under every activation on the convection and wave benchmarks and on Helmholtz, and is the only method that avoids failure on that problem for SiLU and SoftPlus. The authors interpret the results as evidence that stabilizing feature distributions is itself a sufficient mechanism for improved PINN training, and they support this with pre-activation histograms, loss-landscape smoothness, and NTK eigenvalue spectra that stay higher and converge to a steady shape earlier than vanilla PINN.","pith_inferences":["Editorial inference: the mask's benefit likely compounds with existing cures for PINN training pathologies—adaptive activations, loss balancing, or residual connections—because it acts on a different failure axis (distribution drift) than those methods.","Editorial inference: the mask's dependence on the initialization of $\\alpha$, which the paper flags as impactful, suggests a principled initialization rule (e.g. tied to layer width or activation curvature) could be derived; the reported experiments tune $\\alpha$ per activation, and a rule would remove that tuning.","Editorial inference: the same mechanism may transfer to other coordinate-based MLP tasks such as neural radiance fields or signed distance functions, where internal covariate shift and saturation also limit training, but this is not tested in the paper."],"forward_implications":["If the central claim holds, PINNs can be made wider without the usual degradation, because the mask keeps pre-activations in a Gaussian-like band even at large width.","Training becomes robust to activation choice: the mask restores competitive accuracy for Tanh, GELU, SiLU, and SoftPlus, including cases where vanilla PINNs fail entirely.","The reported NTK analysis implies faster convergence and a stable optimization trajectory, so Mask-PINNs should reach lower loss with fewer iterations in other PDE settings.","Because the mask is pointwise and deterministic, it can be dropped into existing PINN code without changing the loss formulation or sampling strategy.","The authors suggest the same mask may transfer to other coordinate-based networks such as implicit neural representations."],"supporting_citations":[{"why":"Defines the PINN framework and the PDE-constrained loss that Mask-PINN is designed to stabilize.","marker":"[1]"},{"why":"Introduces internal covariate shift and batch normalization, the problem and the standard remedy that Mask-PINN argues are not directly usable in PINNs.","marker":"[27]"},{"why":"Introduces layer normalization, the other standard normalization baseline that the paper argues distorts physical consistency.","marker":"[31]"},{"why":"Provides the weight normalization baseline and is also the source of the NTK derivation style used in the paper's spectral analysis.","marker":"[32]"},{"why":"Supplies the locally adaptive activation function (LAAF) baseline that Mask-PINN compares against and outperforms.","marker":"[33]"},{"why":"PirateNets is a key deep-architecture baseline and the source of the claim that PINNs degrade with width and depth.","marker":"[15]"},{"why":"Specialized activation functions baseline (ABU-PINN) that Mask-PINN compares against for adaptive activation behavior.","marker":"[13]"},{"why":"Documents gradient flow pathologies in PINNs, providing the training-stability context that motivates the mask.","marker":"[23]"}],"fun_headline_variants":["Mask-PINN quells internal covariate shift in PDE solvers","Pointwise mask keeps physics intact while stabilizing PINNs","Learnable mask lifts PINN accuracy and enables wide nets","Stabilize PINNs with a mask that preserves PDE constraints","Mask-PINN: drift-free training for physics-informed networks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central assumption is that the mask's provable contraction of activation sensitivity near zero, established locally at initialization, continues to hold over the entire training trajectory and at every depth, so that feature distributions never drift far enough to reintroduce saturation or vanishing gradients.","fun_headline_variants_meta":{"raw":{"variants":["Mask-PINN quells internal covariate shift in PDE solvers","Pointwise mask keeps physics intact while stabilizing PINNs","Learnable mask lifts PINN accuracy and enables wide nets","Stabilize PINNs with a mask that preserves PDE constraints","Mask-PINN: drift-free training for physics-informed networks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000606,"raw_usage":{"total_tokens":2869,"prompt_tokens":1034,"completion_tokens":1835,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":650,"completion_tokens_details":{"reasoning_tokens":1750}},"tokens_in":650,"tokens_out":1835,"duration_ms":13644,"temperature":1.0,"reasoning_tokens":1750,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:47:30.372980+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a 30-layer Mask-PINN on the convection equation, logging per-layer pre-activation mean, variance, and gradient norms every 1,000 iterations: if deeper layers' distributions drift away from zero or gradient norms collapse while $\\alpha$ stays positive, then the local contraction proved near initialization does not guarantee the global stability the paper claims.","supporting_citations":[{"cited_title":"Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations,","cited_arxiv_id":null,"evidence_quote":"Defines the PINN framework and the PDE-constrained loss that Mask-PINN is designed to stabilize."},{"cited_title":"Batch normalization: accelerating deep network training by reducing internal covariate shift,","cited_arxiv_id":null,"evidence_quote":"Introduces internal covariate shift and batch normalization, the problem and the standard remedy that Mask-PINN argues are not directly usable in PINNs."},{"cited_title":"Weight normalization: A simple reparameterization to accelerate training of deep neural networks,","cited_arxiv_id":null,"evidence_quote":"Provides the weight normalization baseline and is also the source of the NTK derivation style used in the paper's spectral analysis."},{"cited_title":"Locally adaptive activation functions with slope recovery for deep and physics -informed neural networks,","cited_arxiv_id":null,"evidence_quote":"Supplies the locally adaptive activation function (LAAF) baseline that Mask-PINN compares against and outperforms."},{"cited_title":"Learning Specialized Activation Functions for Physics-Informed Neural Networks,","cited_arxiv_id":null,"evidence_quote":"Specialized activation functions baseline (ABU-PINN) that Mask-PINN compares against for adaptive activation behavior."},{"cited_title":"Understanding and mitigating gradient flow pathologies in physics -informed neural networks,","cited_arxiv_id":null,"evidence_quote":"Documents gradient flow pathologies in PINNs, providing the training-stability context that motivates the mask."}],"review_version":1}