{"id":"e23ffd98-a6a9-4cd6-b1dc-fffe532bd2e5","arxiv_id":"2504.19270","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A variance-matching initialization for implicit neural representations keeps preactivation variance stable for any activation function and improves image, audio, and 3D reconstruction for Gaussian and sinc activations.","lead":"This paper presents a weight initialization scheme that keeps signal variance stable across layers in implicit neural networks, for almost any activation function. The scheme improves image, audio, and 3D shape reconstruction, especially when used with Gaussian activations.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Backward variance formula Eq. (20) assumes independence of gradient, weight, and activation derivative; this fails for nonlinear activations, so the 'any activation' gradient-stability claim is not established.","rationale":"The paper's central claim has two parts: forward variance matching and backward gradient-variance matching. The forward part is solid: Eq. (4) follows from exact second-moment computation, and the Monte Carlo estimates are accurate and cheap. The backward part is the load-bearing weak point. The derivation of Eq. (10) in supplement §6.1 factorizes a variance of products under an unstated independence assumption that is violated for any nonlinear activation because the backpropagated gradient at layer i depends on the same weight W_i whose variance is being matched. The paper does not present this as a mean-field approximation or give a limit in which correlations vanish, so the statement that the initialization 'applies to any activation' overreaches. This does not invalidate the empirical gains, which are demonstrated for specific activations and tasks, but it means the contribution should be read as a heuristic with strong empirical support rather than a proven general initialization. The reader's conditional verdict is appropriate; no change is needed.","tokens_in":15138,"tokens_out":14529,"duration_ms":150242,"concrete_test":"Run the 100-layer/1000-unit controlled experiment of Table 3 with Gaussian activation (σ_a=0.05) using VI3NR weights with σ_p from Eq. (13); over 10^4 fresh initializations, backpropagate a fixed unit-variance output gradient and measure the empirical variance of ∂L/∂z_i at each layer. Compare with the layer-by-layer prediction of Eq. (10). If the ratio departs from 1 by more than ~20% at any depth, the independence factorization in Eq. (20) is empirically false and Eq. (13) does not ensure stable backward variance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that Eq. (13) yields stable gradient variance for arbitrary activations rests on the backward-pass variance identity Eq. (10), derived in supplement Eq. (20). That derivation writes Var(Σ_k δ_k W_kj f'(z_{i-1,j})) as Σ_k Var(δ_k) Var(W_kj)(E[f'(z_{i-1,j})]^2 + Var(f'(z_{i-1,j}))). This factorization requires δ_k = ∂L/∂z_i,k to be independent of W_kj and of f'(z_{i-1,j}). But δ_k = [W_{i+1}^T δ_{i+1}]_k f'(z_i,k) with z_i,k = Σ_l W_kl x_i,l, so δ_k depends on W_kj through f'(z_i,k); and x_i,j = f(z_{i-1,j}) couples W_kj to f'(z_{i-1,j}). Hence the factorization is not exact and can be badly wrong (e.g., for f(z)=z^2, fourth moments of W enter). The paper presents Eqs. (9)-(13) as derivations rather than mean-field approximations, yet the 'applies to any activation' claim is unsupported for the backward pass. A related index slip: Eq. (8) involves f'(z_{i-1}) but Eq. (10)/(21) switches to f'(z_i); the two are equal only if the preactivation distribution is the same at every layer, which is exactly what the backward condition is supposed to enforce.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes VI3NR, a weight initialization scheme for implicit neural representations (INRs). The method sets layer weight variances according to Eq. (4) so that preactivation variance is preserved across layers at a user-chosen value σ_p^2, with activation statistics computed either analytically or by Monte Carlo estimation. The authors further derive a backward-pass condition, Eq. (13), intended to keep gradient variance stable across layers, and argue that this generalizes Xavier and Kaiming initialization and applies to any activation function. Experiments on image, audio, and 3D shape reconstruction show that the initialization improves performance over random-normal baselines for Gaussian and sinc activations, with an additional claim that the theory of the initialization matches task performance, particularly for Gaussian INRs.","tokens_in":15520,"tokens_out":8039,"duration_ms":82382,"significance":"If fully correct, this would be a useful and practical contribution: a general initialization recipe for coordinate-MLPs that removes much of the activation-specific trial-and-error currently needed. The forward-pass variance analysis is clean, the Monte Carlo procedure for activation statistics is simple and efficient, and the controlled experiments (Tabs. 2–3) provide a useful comparison of initialization methods. The empirical gains on image, audio, and SDF reconstruction over random initialization are encouraging. However, the backward-pass derivation rests on an unstated independence/mean-field assumption that is not generally valid for nonlinear activations, and the claimed quantitative match between theory and task performance is only approximate, with the reported results relying on an additional task-specific line search. These issues undermine the strength of the 'any activation' and 'theory matches task' claims as currently stated.","major_comments":[{"comment":"The derivation of the backward variance condition treats ∂L/∂z_i, [W_i]_kj, and f'([z_{i-1}]_j) as mutually independent when passing from Eq. (17) to Eq. (20). This independence does not hold for nonlinear activations: ∂L/∂z_i contains a factor f'(z_i) with z_i = W_i x_i, so ∂L/∂z_i depends on the same weight matrix W_i that appears explicitly in Eq. (8), and x_i = f(z_{i-1}) couples f'([z_{i-1}]_j) to the activations appearing in ∂L/∂z_i. The factorization in Eq. (20) therefore neglects higher-order moments of W_i and is at best a mean-field approximation. Because Eq. (13) is derived from this factorization, the claim that the initialization stabilizes gradient variance for 'any activation' is not established. The authors should state this assumption explicitly, justify its use in the mean-field regime, or provide a direct empirical validation of Eq. (10) for the activations used. Table 3, where Eb remains 20–36% even after selecting σ_p via Eq. (13), suggests the approximation is not highly accurate for Gaussian and sinc activations.","section":"Supplementary §6.1, Eqs. (17)–(21)"},{"comment":"The paper claims that the theory of the initialization matches task performance (abstract and Sec. 4.2). However, in Fig. 2 the theoretical slope of the backward condition is σ_r = σ_a/σ_p ≈ 0.66, while the image-reconstruction optimum follows σ_r ≈ 0.33 and the SDF optimum is much wider and offset. Since the main experiments (Tabs. 5–7) use 8-layer networks and perform an additional task-specific line search over σ_p (Sec. 3.5), the reported gains do not validate Eq. (13) as a predictor of the optimal σ_p for shallow INRs; they validate the combination of the initialization family plus task-level tuning. The qualitative trend in Fig. 3 for deeper networks is supportive, but the wording should be adjusted to state that Eq. (13) provides a good prior for deep networks and a useful starting point for shallow ones.","section":"Sec. 4.2, Fig. 2 and Sec. 3.5"},{"comment":"The text says that when σ_p is selected from Eq. (13) the method achieves low forward and backward errors for INR activations. The reported Eb values (sine 20.4, Gaussian 35.6, sinc 21.4, wavelet 22.5) are much lower than the σ_p = 1 case (100) but are not low in absolute terms; a 20–36% SMAPE means gradient variance still changes substantially over 100 layers. This weakens the 'stable gradient variance' claim even as an approximation and needs to be discussed, especially in light of the independence issue raised above.","section":"Table 3 and Sec. 3.4"}],"minor_comments":[{"comment":"The statement that the elements [z_i]_k are independent is not correct unconditionally, since they share the same random vector x_i; they are conditionally independent given x_i. This does not affect the marginal variance computation, but the proof and wording should be corrected.","section":"Sec. 3.3, Proposition 3.1"},{"comment":"The notation switches from f'([z_{i-1}]_j) in Eq. (8) to f'(z_i) in Eq. (10). The two are equal only when the preactivation distribution is the same at every layer, which is the desired invariant; this should be stated explicitly before the switch.","section":"Sec. 3.4, Eq. (10)"},{"comment":"The 'small local line search' is not specified in detail. It would strengthen reproducibility to state the range and number of σ_p values tested.","section":"Sec. 3.5"},{"comment":"For ReLU, the note says all σ_p satisfy Eq. (13), but the grid search then reports σ_p = 1. It would be clearer to explain how σ_p = 1 was selected in that case.","section":"Table 1, bottom row"},{"comment":"The abstract's phrase 'applies to any activation function' is stronger than the derivations support; the authors should qualify this to activations for which the mean-field assumptions hold and whose statistics can be estimated reliably.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope for the journal. The core issue is the gap between the strong theoretical framing and the mean-field nature of the backward-pass derivation; the paper's own Table 3 numbers already reveal the limits of the approximation. If the authors reframe the contribution as an empirically effective initialization with a theoretically motivated mean-field basis, rather than an exact 'any activation' guarantee, the paper could be suitable after revision. I would also encourage the authors to engage with the mean-field signal-propagation literature, which makes similar independence assumptions, as it would help position the contribution and clarify what is new."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about this paper. First, it delivers a genuinely general forward-pass initialization for coordinate MLPs, extending SIREN and Kumar to arbitrary activations while adding a controllable preactivation variance sigma_p. Second, the backward-pass variance condition, which is the theoretical heart of the paper's 'any activation' claim, rests on an unstated independence assumption that is not exact. The paper deserves a serious referee, but it needs a major revision.\n\nWhat is new and good: The forward-pass formula (Eq. 4) is clean: set weight variance to keep preactivation variance at a chosen sigma_p, and compute the activation moments analytically or with cheap Monte Carlo. The paper correctly recovers Xavier and Kaiming as special cases, and the practical recipe works. The experiments show large PSNR/IoU gains over random initialization for Gaussian and sinc activations across images, SDFs, and audio. The MC estimator for activation statistics is accurate with 10K samples. The demonstration that the theoretically optimal sigma_p becomes more predictive as network depth increases (Fig. 3) is an honest and informative empirical finding.\n\nSoft spots: The backward variance derivation (Eq. 10, supplement Eq. 20) factorizes Var(∂L/∂z_{i-1}) as a product of variances of the gradient, weight, and activation derivative. That factorization requires these three to be independent. They are not: ∂L/∂z_{i,k} depends on W_{kj} through the forward pass, and the activation derivative f'(z_{i-1,j}) is coupled through the same weight. The paper never flags this as a mean-field approximation. In the large-width limit the dependence may be asymptotically negligible, but the paper presents it as an exact derivation. A referee should ask for either a rigorous asymptotic statement or an explicit approximation with supporting numerics. There is also a smaller issue in Proposition 3.1: the products p_{kl} are not independent because they share x_i; the CLT result survives under a conditional argument, but the proof as written has a gap. Finally, the 'theory matches task performance' claim is overstated. The theoretical slope from Eq. (13) and the task-optimal slope differ (Fig. 2), and the final numbers rely on a task-specific line search over sigma_p. The authors acknowledge this, and the depth-dependent trend is a nice nuance, but the headline should be 'theory guides' rather than 'theory matches.'\n\nWho is this for? Researchers working on INR activations, especially Gaussian and sinc, or on initialization theory. It is a solid conditional contribution, not a breakthrough.\n\nRecommendation: send to peer review. Ask for major revision to address the independence assumption, fix the CLT proof, and soften the 'theory matches' claim.","headline":"Useful generalization of SIREN/Kumar for arbitrary INR activations, but the backward-pass variance condition is an unstated mean-field approximation and the 'theory matches task performance' claim is softer than advertised.","tokens_in":15991,"tokens_out":4607,"would_cite":true,"duration_ms":44062,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"One variance rule keeps implicit neural nets stable for any activation","keywords":["implicit neural representations","weight initialization","variance propagation","preactivation variance","activation functions","Gaussian activation","sinc activation","Monte Carlo estimation"],"falsifier":"On a 100-layer untrained MLP with weights set by Eq. (4) and $\\sigma_p$ chosen from Eq. (13), measure the ratio $\\mathrm{Var}(\\partial L/\\partial z_{i-1})/\\mathrm{Var}(\\partial L/\\partial z_i)$ layer by layer; if the ratio drifts away from 1 with depth, or if the empirical correlation between $\\partial L/\\partial [z_i]_k$ and $f'([z_{i-1}]_j)$ is non-negligible, the backward condition does not deliver what the derivation claims.","tokens_in":14972,"feed_emoji":"⚖️","tokens_out":9412,"duration_ms":81675,"temperature":0.7,"pith_summary":"Implicit neural representations (INRs) encode images, audio, and shapes as functions learned by small fully connected networks, and their performance depends heavily on how the weights are initialized. The paper claims that for any activation function there is a single free parameter—the preactivation variance—that can be tuned so that both the forward signal variance and the backward gradient variance stay constant across layers. It derives explicit formulas for the required weight variance and identifies the preactivation variance that satisfies both the forward and backward conditions. Because the activation statistics in those formulas are hard to compute in closed form, the paper shows they can be replaced by fast Monte Carlo estimates with little loss. If correct, the method turns a per-activation initialization design problem into a one-parameter choice, and experimentally it improves reconstruction for Gaussian and sinc activations in images, audio, and 3D surfaces.","feed_headline":"One variance rule keeps implicit neural nets stable for any activation","feed_subtitle":"Matching forward and backward variance across layers lifts image, audio, and 3D reconstruction for Gaussian and sinc nets.","key_machinery":"The load-bearing object is the variance-matching identity at Eq. (4): choosing $\\sigma^2(W_i)=\\sigma_p^2/(M_i(\\mu^2(x_i)+\\sigma^2(x_i)))$ makes the preactivation variance $\\sigma_p^2$ survive every layer. The companion condition Eq. (13) imposes the same invariance on the backward pass, and because $\\sigma_p$ appears in both, it acts as a free parameter that lets the two constraints be satisfied simultaneously rather than averaged. Activation statistics are computed either analytically or by Monte Carlo sampling of $N(0,\\sigma_p^2)$; for Gaussian activations the needed statistic reduces to the ratio $\\sigma_a/\\sigma_p$.","core_discovery":"On its own terms, the paper establishes a variance-preserving initialization scheme that works for any activation. For layers with zero-mean i.i.d. weights, the preactivations converge to a Gaussian as width grows (Proposition 3.1), so preserving distributions reduces to preserving variances. The paper sets the weight variance at every layer to $\\sigma^2(W_i)=\\sigma_p^2/(M_i(\\mu^2(x_i)+\\sigma^2(x_i)))$, with expectations taken over $z\\sim N(0,\\sigma_p^2)$, so that each layer's preactivations keep variance $\\sigma_p^2$. It then derives a backward-pass condition, $\\sigma_p^2 (M_{i+1}/M_i)(\\mu^2(f'(z_i))+\\sigma^2(f'(z_i)))/(\\mu^2(x_i)+\\sigma^2(x_i))=1$, that keeps gradient variance constant as well, and solves for $\\sigma_p$ by grid search since the condition depends on $\\sigma_p$ itself. Previous initialization recipes emerge as special cases. For Gaussian and sinc activations the paper reports that this initialization reduces forward and backward variance drift in controlled deep-network tests and improves image, audio, and 3D surface reconstruction over random initialization.","pith_inferences":["Extension: the same ratio-based analysis should transfer to other radial basis activations, such as inverse multiquadrics, where the Gaussian derivation's dependence on $\\sigma_a/\\sigma_p$ is likely to recur.","Testable extension: if the backward independence assumption is violated, Eq. (13)'s predicted $\\sigma_p$ should still suppress gradient drift for deep tanh and sigmoid networks; measuring layer-by-layer gradient variance on a 100-layer net would isolate where the mean-field approximation breaks.","Implicit consequence: the task-dependence of the optimal $\\sigma_p$ suggests that an input-statistics-aware or loss-aware predictor of $\\sigma_p$ could replace the line search in future work.","Unstated corollary: because the forward condition depends only on second-order statistics, the method may interact with normalization layers that alter those statistics, requiring $\\sigma_p$ to be recomputed for the modified network."],"forward_implications":["Any new INR activation can be used with a principled initialization by estimating two scalar statistics of $f(z)$ and $f'(z)$ under a Gaussian; no per-activation mathematical derivation is required.","Deep coordinate networks with Gaussian, sinc, or other nonstandard activations should no longer suffer vanishing or exploding gradients at initialization when $\\sigma_p$ is chosen from Eq. (13).","The initialization covers earlier schemes: the sine-activation recipe and the ReLU/tanh-style recipes correspond to particular choices of $\\sigma_p$, so existing results are recovered rather than contradicted.","Because the best $\\sigma_p$ also depends on the task, the method converts the initialization question into a small, cheap hyperparameter search that can be done once per task.","For Gaussian activations the theory predicts a linear relationship between the activation scale $\\sigma_a$ and $\\sigma_p$; the paper observes this trend in reconstruction performance, suggesting the same relationship can guide future activation-parameter choices."],"supporting_citations":[{"why":"supplies the sine-activation INR formulation and the specific variance choice that this scheme generalizes","marker":"[23]"},{"why":"provides the baseline forward/backward variance initialization that this scheme derives as a special case","marker":"[7]"},{"why":"provides the ReLU fan-in/fan-out initialization that this scheme recovers as a special case","marker":"[9]"},{"why":"previous arbitrary-activation forward variance result, extended here with a formal CLT proof and without Taylor approximation","marker":"[10]"},{"why":"states the central limit theorem used to justify the Gaussian preactivation distribution","marker":"[1]"},{"why":"gives the gain-based initialization heuristics and the controlled forward/backward error experiment used as a comparison","marker":"[18]"},{"why":"supplies the Gaussian activation family and the activation-scale value used in experiments","marker":"[19]"},{"why":"provides the 3D shape benchmark used for signed-distance-function reconstruction","marker":"[12]"},{"why":"provides the image benchmark used for PSNR reconstruction comparisons","marker":"[6]"},{"why":"supplies the sinc activation test case from the sampling-theory literature","marker":"[22]"}],"fun_headline_variants":["Stable variance for any activation in implicit neural nets","Variance-informed initialization improves INR reconstruction","General init rule keeps INR layers stable","One variance fix for all INR activations","Stable variance init boosts image, audio, 3D"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The backward-pass derivation assumes that at initialization the gradient arriving at a layer, the weight matrix, and the activation derivative are uncorrelated when variances multiply; in reality the gradient is built from those same weights and activation derivatives, so this independence is not guaranteed.","fun_headline_variants_meta":{"raw":{"variants":["Stable variance for any activation in implicit neural nets","Variance-informed initialization improves INR reconstruction","General init rule keeps INR layers stable","One variance fix for all INR activations","Stable variance init boosts image, audio, 3D"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000316,"raw_usage":{"total_tokens":1805,"prompt_tokens":973,"completion_tokens":832,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":589,"completion_tokens_details":{"reasoning_tokens":762}},"tokens_in":589,"tokens_out":832,"duration_ms":8447,"temperature":1.0,"reasoning_tokens":762,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:56:38.709264+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a 100-layer untrained MLP with weights set by Eq. (4) and $\\sigma_p$ chosen from Eq. (13), measure the ratio $\\mathrm{Var}(\\partial L/\\partial z_{i-1})/\\mathrm{Var}(\\partial L/\\partial z_i)$ layer by layer; if the ratio drifts away from 1 with depth, or if the empirical correlation between $\\partial L/\\partial [z_i]_k$ and $f'([z_{i-1}]_j)$ is non-negligible, the backward condition does not deliver what the derivation claims.","supporting_citations":[{"cited_title":"Implicit neural representa- tions with periodic activation functions","cited_arxiv_id":null,"evidence_quote":"supplies the sine-activation INR formulation and the specific variance choice that this scheme generalizes"},{"cited_title":"Understanding the diffi- culty of training deep feedforward neural networks","cited_arxiv_id":null,"evidence_quote":"provides the baseline forward/backward variance initialization that this scheme derives as a special case"},{"cited_title":"Zhang, Shaoqing Ren, and Jian Sun","cited_arxiv_id":null,"evidence_quote":"provides the ReLU fan-in/fan-out initialization that this scheme recovers as a special case"},{"cited_title":"On weight initialization in deep neural networks, 2017","cited_arxiv_id":null,"evidence_quote":"previous arbitrary-activation forward variance result, extended here with a formal CLT proof and without Taylor approximation"},{"cited_title":"Ash and C.A","cited_arxiv_id":null,"evidence_quote":"states the central limit theorem used to justify the Gaussian preactivation distribution"},{"cited_title":"Pytorch: An imperative style, high-performance deep learning library","cited_arxiv_id":null,"evidence_quote":"gives the gain-based initialization heuristics and the controlled forward/backward error experiment used as a comparison"},{"cited_title":"Beyond periodicity: Towards a unifying framework for activations in coordinate- mlps","cited_arxiv_id":null,"evidence_quote":"supplies the Gaussian activation family and the activation-scale value used in experiments"},{"cited_title":"Bacon: Band-limited coordinate net- works for multiscale scene representation","cited_arxiv_id":null,"evidence_quote":"provides the 3D shape benchmark used for signed-distance-function reconstruction"},{"cited_title":"Kodak Lossless True Color Image Suite","cited_arxiv_id":null,"evidence_quote":"provides the image benchmark used for PSNR reconstruction comparisons"}],"review_version":1}