{"id":"f375fe03-36c7-49fb-b85c-7aa380ebf7e6","arxiv_id":"2501.04182","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Heavy-tailed random weights in square feedforward DNNs produce multiple stable fixed point attractors, while Gaussian weights yield a single fixed point, with a non-monotone dependence on depth.","lead":"This paper uses simulations to compare fixed points of autoencoder-style neural networks with light-tailed (Gaussian) versus heavy-tailed (Cauchy) random weights. It reports that heavy-tailed weights create multiple stable fixed points, and that the number of fixed points rises then falls with network depth.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The non-monotone Q(L) claim rests on a schematic figure and a grid-based counting rule that can miss fixed points; it is currently unsupported by reported numerical data.","rationale":"The reader's weakest_assumption correctly identifies the grid-resolution and convergence-criteria issue in Section III as load-bearing. My stress-test agrees but goes further: the paper itself labels Fig. 2 as 'schematic' and provides no realization counts, error bars, or raw data for Q(L), so the non-monotone depth dependence is even less supported than a mere discretization artifact. This is a missing-support finding of the kind the review rules require flagging. I found no internal mathematical contradiction in the Gaussian contraction analysis of Section V: the g < 1 threshold at beta ≈ 1/2 and the exponential decay with L in Eq. (9) are plausible and consistent with the composition structure. However, that analysis concerns the light-tailed case, which is not the contested part of the central claim. The contested parts are (i) the emergence of multiple fixed points for heavy-tailed Cauchy initialization and (ii) the non-monotone Q(L) dependence. Both currently rest on an unpublished numerical enumeration and a schematic figure. The trained-DNN 'verification' in Section IV is weak because the loss (3) directly encodes the target fixed points, so finding K fixed points near the centers is expected and does not independently validate the distribution-driven mechanism. Given these gaps, the reader's REJECT verdict is appropriate. I set verdict_should_be to UNCHANGED because the reader already rejected; my concern reinforces that rejection rather than moving it to a different category.","tokens_in":8652,"tokens_out":3768,"duration_ms":40120,"concrete_test":"Recompute Q(L) for the Cauchy-initialized untrained family (L = 2, 3, 4, 5, 10, 20; widths {2, 100, ..., 2}) with at least 100 independent realizations per L, using refined grids delta = 0.025 and 0.0125 and stricter convergence criteria epsilon = 1e-8, N0 = 500. Report the full histogram of Q for each L and each refinement, not just the mode, and overlay the curves to check whether the non-monotone peak persists. Additionally, cross-check grid counts with an independent basin-sampling/root-finding method (e.g., many random starts with Newton iteration) to determine whether fixed points with basins smaller than a grid cell are being missed. If the non-monotone peak survives refinement and realization averaging, the discretization concern is settled; otherwise the central depth-dependence claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is the numerical basis for the claimed depth dependence Q(L). In Section III, Q is measured by iterating Eq. (5) from grid points defined in Eq. (4) with step delta = 0.05, declaring convergence when |x_{m+1} - x_m| < epsilon = 1e-5 within N0 = 50 steps. This counting rule silently assumes that every true fixed point has a basin of attraction containing at least one grid point and that iteration (5) reaches that basin within 50 steps. For Cauchy-distributed weights the map is not a contraction on Omega, so Banach's theorem does not apply; small basins, thin fractal boundaries, slow convergence, or limit cycles would cause fixed points to be missed or misclassified. Moreover, the paper reports Q only as a 'mode' with no number of realizations, no error bars, and no data table, and Fig. 2 is explicitly labelled 'schematic'. Thus the advertised non-monotone Q(L) curve is not backed by published numerical evidence; it could be an artifact of the discretization or of a single favorable realization. The trained-DNN section does not repair this gap, because the loss (3) directly pushes outputs toward the chosen K centers, so finding K fixed points near those centers is partly baked into the training objective and cannot independently confirm the distribution-driven mechanism.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies fixed points of fully connected feedforward networks with equal input and output dimensions. It reports numerical experiments claiming that Gaussian i.i.d. weight initialization gives a single stable fixed point for a range of architectures, while Cauchy initialization gives multiple stable fixed points whose basins partition the input domain, and that the number of fixed points Q(L) depends non-monotonically on depth L. A toy trained autoencoder is used to show that fixed points coincide with training centers, and a contraction-mapping analysis for Gaussian weights identifies beta_cr approximately 1/2 for Tanh and approximately 1/4 for Sigmoid.","tokens_in":8936,"tokens_out":5552,"duration_ms":55887,"significance":"If correct, the tail behavior of random weights would be an important determinant of the fixed-point landscape, connecting heavy-tailed self-regularization to a concrete dynamical-systems phenomenon. The paper makes clear falsifiable predictions and provides an explicit numerical protocol, and the CLT scaling argument for beta_cr is a useful heuristic. However, the central non-monotone Q(L) claim and the trained-network verification need substantially stronger evidence before the conclusions can be accepted.","major_comments":[{"comment":"The non-monotone Q(L) claim is not supported by the reported data. Fig. 2 is explicitly labeled 'schematic illustration', no table of Q values is given, no number of random realizations or variance is reported, and Q is only described as a mode. The grid counting rule in Eqs. (4)-(5) silently assumes every true fixed point has a basin containing at least one grid point and that iteration (5) converges within N0=50 steps; for non-contractive Cauchy maps this can miss small-basin fixed points, misclassify slow orbits, or confuse limit cycles with fixed points. A resolution study and ensemble statistics are needed before the depth dependence can be claimed.","section":"Section III and Fig. 2"},{"comment":"The trained-DNN experiment does not independently verify the heavy-tail mechanism. The loss in Eq. (3) explicitly drives Phi(x) toward the chosen centers x*_k for all training inputs, including x*_k itself, so the appearance of fixed points near x*_k is partly built into the training objective. To support the claimed analogy with untrained heavy-tailed DNNs, the authors should either measure the empirical weight distribution after training and compare its tail behavior with untrained Cauchy DNNs, or design a training objective that does not directly target the fixed points.","section":"Section IV, Eq. (3)"},{"comment":"The statement that for Cauchy distributions there is 'full correspondence between the areas of contraction of Phi (i.e., the areas where g<1) and basins' is inconsistent with the definition of g in Eq. (8) as the global maximum of the ratio over all pairs of grid points. A single global constant cannot localize regions where contraction holds; the authors need to define and compute a local or restricted contraction constant if they wish to make claims about areas of contraction.","section":"Section V, Eq. (8)"},{"comment":"Calling Section V a 'computer-assisted proof' is an overstatement. The contraction check is performed on a finite grid, and the beta_cr argument relies on a CLT variance scaling rather than a rigorous bound valid for all weight matrices and architectures. The text should be rephrased as numerical evidence and should explicitly state the finite-sample, finite-grid nature of the verification.","section":"Section V, heading"}],"minor_comments":[{"comment":"The figure contains garbled placeholder text such as '/s48 /s53' and is described as 'schematic'; it should be replaced by a data plot with error bars or a table of the underlying Q values.","section":"Fig. 2"},{"comment":"The 'weak similarity effect' is invoked to justify the claim that the activation function does not change the number of fixed points, but the term is not defined or explained in this manuscript; at minimum a precise statement or a more specific reference is needed.","section":"Section III"},{"comment":"There are several typos and inconsistencies, including 'employer' for 'employee' in the Fig. 3 caption, 'N711 4' in the acknowledgments, and 'IEEE Acess' in reference 22; the manuscripts should be carefully proofread.","section":"Captions and references"},{"comment":"The manuscript does not state whether the code or the full numerical data will be made available; for a paper whose claims are entirely numerical, a data-availability statement or a link to code is strongly recommended.","section":"Reproducibility"}],"recommendation":"major_revision","confidential_remarks":"The core observation is potentially interesting and fits the journal's scope, but the current evidence is too thin, especially for the non-monotone Q(L) claim. I would encourage requesting a revised version with ensemble statistics, resolution studies, and a reworked trained-DNN comparison. The authors should also tone down the 'computer-assisted proof' language and address the global-versus-local contraction issue in Section V."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper has a plausible and interesting observation—Cauchy-initialized autoencoders show multiple stable fixed points while Gaussian ones show one—but the evidence as reported does not support the depth-dependence claim, and the trained network confirmation is partly circular. Worth a referee's time, but not acceptance as is.\n\nWhat is actually new: the distribution-tail dependence of fixed point multiplicity in feedforward square DNNs, and the basin partition observation, are not in the cited literature. The Gaussian case reduces to known contraction properties, but the heavy-tail case is a genuinely different phenomenon. The CLT-based argument for beta_cr = 1/2 with odd activations, and the numerical g composition result, are sensible and give the paper some formal grounding, though they are not deep.\n\nSoft spots: The main issue is that the central non-monotone Q(L) curve is presented only as a schematic, Figure 2, with no error bars, no number of random realizations, and no data table. The fixed point counting itself rests on a fixed grid (delta=0.05) and convergence criterion (epsilon=1e-5, N0=50), which silently assumes every true fixed point has a basin containing a grid point and that iteration converges in 50 steps. For Cauchy weights, the map is not a contraction on the whole domain, so small basins or slow convergence could be missed; the reported Q could be an artifact of discretization. The trained-DNN section does not repair this gap, since the loss (3) explicitly pushes outputs toward the chosen centers, so finding fixed points near them is baked into the objective and does not independently confirm the mechanism. These are load-bearing weaknesses for the paper's main claim, not minor caveats.\n\nOn balance: the core idea is worth exploring, and the paper makes an honest attempt to connect fixed point structure to Heavy-Tailed Self-Regularization. But the current manuscript does not provide reproducible evidence for the non-monotone depth dependence, and the numerical methodology is under-specified. A serious referee could ask for actual data, statistics over realizations, and a more robust fixed point solver, and the paper would be much stronger if those were provided.\n\nRecommendation: send to peer review, but with the expectation that the current version would need major revision. The phenomenon, if real, is useful for understanding autoencoder memorization and the role of initialization. Not something I'd cite yet.","headline":"Interesting but unproven: heavy-tail fixed point multiplicity is plausible, yet the depth-dependence claim rests on a schematic and grid-counting assumptions.","tokens_in":9441,"tokens_out":2276,"would_cite":false,"duration_ms":21908,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","37C25"],"pacs":[],"model":"deepseek-v4-flash","headline":"In square DNNs, the tail of random weights controls the fixed-point landscape.","keywords":["fixed points","random weights","heavy-tailed distributions","light-tailed distributions","autoencoder networks","contraction mapping","basins of attraction","deep neural networks"],"falsifier":"Repeat the Cauchy-initialized depth-$L=5$ and $L=20$ experiments with a finer grid (e.g., $\\delta=0.005$) and with random starting points off the grid; if the count $Q$ changes materially, or if $L=20$ yields more than one stable fixed point from off-grid initializations, then the reported non-monotone $Q(L)$ is a grid artifact. A cleaner test is to run iteration (5) from several thousand uniformly random points, count distinct limits, and compare the result with the grid-based $Q$.","tokens_in":8462,"feed_emoji":"🎯","tokens_out":7841,"duration_ms":71117,"temperature":0.7,"pith_summary":"This paper studies fully-connected feedforward networks whose input and output dimensions are equal (autoencoder-type maps), asking how many stable fixed points such a map has when its weight matrices are drawn i.i.d. at random. The central claim is that the tail of the weight distribution, not just the architecture or the training procedure, decides the fixed-point set: Gaussian (light-tailed) initialization produces a single stable fixed point whose basin of attraction is the whole domain, while Cauchy (heavy-tailed) initialization produces several stable fixed points whose basins partition the domain. The number of fixed points $Q(L)$ depends on depth $L$ non-monotonically, first growing and then falling back to one. The paper further reports that in trained DNNs with standard initialization, multiple fixed points form at positions coinciding with the 'true' images of the training set, with each basin containing its training set. This matters because it connects the widely observed heavy-tailed statistics of trained weights to a concrete, observable property of the network map: how many distinct inputs it can stably recognise.","feed_headline":"Heavy-tailed random weights create many fixed points in a DNN","feed_subtitle":"Gaussian-initialized networks have one stable attractor; heavy-tailed weights produce several, with basins covering the input space.","key_machinery":"The central object is the layer map $\\Phi_l(x)=\\phi(W_l x+b_l)$ and its composition $\\Phi=\\Phi_{L-1}\\circ\\cdots\\circ\\Phi_0$; fixed points and basins are computed numerically by iterating $x_{m+1}=\\Phi(x_m)$ from grid points with step $\\delta=0.05$, convergence tolerance $\\epsilon=10^{-5}$ and at most $N_0=50$ steps. The argument's load-bearing mechanism is the Banach contraction mapping: for light-tailed weights the layer normalization $\\sigma=N^{-\\beta}$ with $\\beta\\geq\\beta_{cr}\\approx 1/2$ keeps the single-layer Lipschitz constant $g_0<1$, so $g=g_0^L<1$ and the composed map has a unique attracting fixed point. For heavy-tailed weights the contraction condition fails in parts of the domain, allowing several stable fixed points whose basins of attraction are precisely the regions where the iteration converges. The critical exponent $\\beta_{cr}$ and the exponential composition law $g=g_0^L$ are what carry the depth dependence.","core_discovery":"On the paper's own terms: for $\\Phi(x)=x_L$ built from layers $x_{l+1}=\\phi(W_l x_l+b_l)$ with $n_0=n_L$, and with i.i.d. entries in $W_l,b_l$, the tail of the entry distribution determines the number and stability of fixed points. For Gaussian entries normalized with variance $\\sigma_l^2=(n_l)^{-1}$, numerical iteration from a grid on $\\Omega=[-1,1]^2$ finds exactly one stable fixed point, $x^*=0$, with the whole grid converging to it; the map satisfies the Banach contraction criterion $g<1$ when the variance exponent $\\beta$ in $\\sigma=N^{-\\beta}$ is at least $\\beta_{cr}\\approx 1/2$ (for odd activations), and $g=g_0^L$ grows or decays exponentially with depth. For Cauchy entries with the same normalization, depths $L=2,3,5$ yield $Q=2,3,5$ stable fixed points, each with a basin of attraction partitioning $\\Omega$, whereas $L=20$ returns $Q=1$; hence $Q(L)$ is non-monotone. In trained DNNs with Gaussian initialization and a two-dimensional toy photo-encoding task, fixed points after training coincide numerically with the centers of the training-set circles, and each basin contains the corresponding training set. The paper treats these as numerical evidence and a computer-assisted contraction proof rather than a rigorous theorem.","pith_inferences":["Beyond the paper: the reported $Q(L)$ counts only attractors whose basins contain at least one grid point with $\\delta=0.05$; refining the grid should reveal smaller basins if they exist, so the true fixed-point count is likely at least the reported $Q$, and the non-monotone peak could be sharper or shifted.","Beyond the paper: the mechanism behind non-monotone $Q(L)$ may be a bifurcation as the effective Jacobian norm of the random composition crosses 1; one could test this by computing the largest Lyapunov exponent of the weight matrices as a function of $L$ and checking whether the peak in $Q$ coincides with the transition.","Beyond the paper: because the trained-DNN experiment uses two-dimensional toy data and HardTanh activation, the same 'fixed points coincide with class centers' finding could be tested on real autoencoders with higher-dimensional images, checking whether the number of stable fixed points matches the number of classes.","Beyond the paper: the critical normalization $\\beta_{cr}\\approx 1/2$ for odd activations matches the standard scaling used in practical initialization; measuring $g$ at finite width near $\\beta=1/2$ could predict where $Q$ jumps from 1 to multiple fixed points without extensive grid searches."],"forward_implications":["With standard Gaussian initialization, an untrained autoencoder-type network generically has one stable fixed point, so it cannot by itself distinguish multiple 'true' inputs; recognition requires training.","When weights acquire heavy-tailed statistics — as training is claimed to do — the network generically develops multiple stable fixed points whose basins partition the input domain, giving a dynamical account of how distinct learned patterns are stored.","The non-monotone dependence of $Q(L)$ on depth means that for a fixed width there is an intermediate depth range that maximizes the number of attractors; architecture choice can therefore tune how many patterns a random network can hold.","In trained networks, fixed points coincide with training-set centers and each basin contains its training set, so the set of attracting fixed points can serve as a stopping or cross-validation diagnostic.","The contraction parameter $g$ follows $g=g_0^L$, so depth composes the single-layer contraction factor; changing the variance exponent $\\beta$ near $\\beta_{cr}$ should sharply switch the network between one and many fixed points."],"supporting_citations":[{"why":"Source of Heavy-Tailed Self-Regularization; used to justify modelling trained DNNs by untrained heavy-tailed (Cauchy) initialization.","marker":"27"},{"why":"Random-matrix analysis of general i.i.d. weight distributions; supplies the 'weak similarity' of activation functions and the $\\sigma\\sim N^{-1/2}$ normalization used in Section V.","marker":"30"},{"why":"Spectral universality result connecting heavy-tailed empirical spectral distributions of trained DNNs to the fixed-point analysis.","marker":"28"},{"why":"Textbook definition of the layer map $x_{l+1}=\\varphi(W_l x_l+b_l)$ that the model is built on.","marker":"14"},{"why":"Banach fixed-point theorem reference underlying the contraction criterion $g<1$ and uniqueness of the attracting fixed point.","marker":"36"},{"why":"Prior fixed-point study for single-layer nonnegative DNNs that motivates the autoencoder fixed-point formulation.","marker":"17"}],"fun_headline_variants":["Heavy-tailed random weights lead to multiple DNN fixed points","Gaussian weights give one attractor; Cauchy many","Fixed-point count in DNNs depends on weight tail not depth","Non-monotonic fixed points appear with heavy-tailed DNN weights"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported fixed-point counts assume that every true fixed point has a basin of attraction large enough to contain at least one grid point (step $\\delta=0.05$) and that iteration (5) converges within $N_0=50$ steps to tolerance $\\epsilon=10^{-5}$; if a basin is smaller than a grid cell, that fixed point is missed and the non-monotone $Q(L)$ curve could be an artifact of discretization.","fun_headline_variants_meta":{"raw":{"variants":["Heavy-tailed random weights lead to multiple DNN fixed points","Gaussian weights give one attractor; Cauchy many","Fixed-point count in DNNs depends on weight tail not depth","Non-monotonic fixed points appear with heavy-tailed DNN weights"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000755,"raw_usage":{"total_tokens":3452,"prompt_tokens":1133,"completion_tokens":2319,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":749,"completion_tokens_details":{"reasoning_tokens":2248}},"tokens_in":749,"tokens_out":2319,"duration_ms":17814,"temperature":1.0,"reasoning_tokens":2248,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:39:38.742958+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Repeat the Cauchy-initialized depth-$L=5$ and $L=20$ experiments with a finer grid (e.g., $\\delta=0.005$) and with random starting points off the grid; if the count $Q$ changes materially, or if $L=20$ yields more than one stable fixed point from off-grid initializations, then the reported non-monotone $Q(L)$ is a grid artifact. A cleaner test is to run iteration (5) from several thousand uniformly random points, count distinct limits, and compare the result with the grid-based $Q$.","supporting_citations":[{"cited_title":"Martin and M.W","cited_arxiv_id":null,"evidence_quote":"Source of Heavy-Tailed Self-Regularization; used to justify modelling trained DNNs by untrained heavy-tailed (Cauchy) initialization."},{"cited_title":"Pastur, V","cited_arxiv_id":null,"evidence_quote":"Random-matrix analysis of general i.i.d. weight distributions; supplies the 'weak similarity' of activation functions and the $\\sigma\\sim N^{-1/2}$ normalization used in Section V."},{"cited_title":"Berlyand, P.-E","cited_arxiv_id":null,"evidence_quote":"Textbook definition of the layer map $x_{l+1}=\\varphi(W_l x_l+b_l)$ that the model is built on."},{"cited_title":"Jachymski, I","cited_arxiv_id":null,"evidence_quote":"Banach fixed-point theorem reference underlying the contraction criterion $g<1$ and uniqueness of the attracting fixed point."},{"cited_title":"Fixed points of nonnegative neural networks","cited_arxiv_id":"2106.16239","evidence_quote":"Prior fixed-point study for single-layer nonnegative DNNs that motivates the autoencoder fixed-point formulation."}],"review_version":1}