Pith. sign in

REVIEW 1 major objections 6 minor 17 references

Training Deep Morphological Neural Networks as Universal Approximators

T0 review · 1 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Deep networks built purely from max-plus and min-plus perceptrons cannot approximate all continuous functions, but adding learnable linear scalings between morphological layers restores universal approximation with only $O(N)$ extra…

desk verdict Solid negative theorems on pure morphological networks; the positive universality results are right but Theorem 4 needs an explicit density lemma stated and proved. read the letter →

arxiv 2505.09710 v4 pith:2AKXNHMW submitted 2025-05-14 cs.LG

Konstantinos Fotopoulos , Petros Maragos This is my paper · ORCID
classification cs.LG
keywords morphologicalneuralnetworksmax-plusalgebramin-plusuniversalapproximationDilation-ErosionPerceptronMaxoutgradientsparsitynetworkpruning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that depth alone does not make a morphological network powerful. It proves that a network built only from max-plus and min-plus morphological perceptrons has gradients that are almost everywhere either zero or a single coordinate unit vector, so it cannot even represent a sloping linear function, let alone approximate every continuous function. The paper then proposes two architectures---the Max-Plus-Min network and the Hybrid-MLP---that place learnable linear operations between morphological layers, and proves both are universal approximators on compact domains with only $O(N)$ learnable activation parameters per layer. Experiments on MNIST, Fashion-MNIST, and CIFAR-10 show the resulting networks train, generalize, and, under pruning, degrade less than linear networks. The upshot is a design rule with practical weight: morphological networks need a small amount of linearity, not more morphology, to become expressive.

What carries the argument

The load-bearing object is the Max-Plus-Min (MPM) morphological layer. For an input $x$ it computes $(w_{i0}\vee \max_j (x_j+w_{ij}))+(m_{i0}\wedge \min_j (x_j+w_{ij}))$, a dilation and an erosion that share weights but have different biases, and then multiplies each output by a learnable scalar $\alpha_i$; the MPM-SVD variant instead applies a learnable diagonal matrix between two fixed random orthogonal matrices. The universality proof works by choosing a large constant $C$: weights near $+C$, $-C$, or $0$ make each input feed the maximum branch, the minimum branch, or neither, so the layer can first build affine functions $a^\top x+b$ and then combine them into $\max_k(a_k^\top x+b_k)+\min_m(c_m^\top x+d_m)$. Because this last class is dense in the continuous functions---the paper takes this from the Maxout universality argument---the architecture inherits universal approximation.

What would settle it

Compute the input gradient of any pure max-plus/min-plus network at a point where every maximizer and minimizer is unique: any gradient other than $0$ or a standard basis vector refutes Theorem 1. For the MPM claim, try to approximate $2x_1$ uniformly on $[-1,1]^d$ with finite-width MPM networks; if no width reaches error below $1/2$, Theorem 4 is false, whereas success is expected because the max-plus-min affine class is a standard difference-of-convex dense family.

Watch

Extended reading notes

Core claim

The central claim is a pair of obstruction and recovery theorems. Any single-output network that only composes max-plus and min-plus morphological perceptrons is Lipschitz and, almost everywhere, its input gradient is either $0$ or a standard basis vector $e_i$; hence its functions lie in a class that cannot be dense in the continuous functions, and the same sparsity limits training through the parameters (Theorems 1 and 2). DEP-style blends of dilation and erosion are also non-universal: their gradients are componentwise nonnegative with $\ell^1$ norm at most $1$, which again excludes simple linear maps such as $2x_1$ (Theorem 3). The proposed Max-Plus-Min layer---a max-plus dilation plus a min-plus erosion sharing weights, with separate biases, followed by a learnable scaling---overcomes the obstruction, and the paper proves the resulting MPM network and the Hybrid-MLP are universal approximators on compact domains (Theorems 4 and 5). In the Hybrid-MLP case the proof is explicit: any fully connected ReLU or Maxout network is a special case of the Hybrid-MLP.

Load-bearing premise

The MPM universality proof rests on the unstated assumption that functions of the form $\max_k(a_k^\top x+b_k)+\min_m(c_m^\top x+d_m)$ can approximate any continuous function on the compact domain; the paper cites this as a byproduct of Maxout universality but never states or proves it as a lemma.

Editorial extensions

If this is right

  • Pure max-plus networks of depth $L\ge 2$ collapse, in the semiring, to a single morphological layer, so any expressive fully morphological network needs some linear operation inserted between morphological layers.
  • The MPM and MPM-SVD networks are universal approximators on compact domains while keeping activation parameters at $O(N)$ per layer, so the sparsity and prunability of morphological networks is not lost.
  • The Hybrid-MLP is a strict generalization of ReLU and Maxout MLPs, and with large batches it converges faster than a standard MLP or Maxout network on MNIST and Fashion-MNIST.
  • Morphological networks are substantially more prunable than linear networks: at 90% unstructured $\ell^1$ pruning, MPM keeps about 92% test accuracy on MNIST while the MLP collapses to about 38%, and SNIP can prune MPM to 1173 parameters on MNIST with no accuracy drop.
  • Residual connections and weight dropout improve generalization, with RMPM-Drop reaching 97.49% test accuracy on MNIST, within 0.52 points of a standard MLP.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The unstated density fact behind Theorem 4 is the statement that functions $\max_k(a_k^\top x+b_k)+\min_m(c_m^\top x+d_m)$ are dense, which is the difference-of-convex (DC) property; a direct DC-density proof would make the MPM argument self-contained.
  • The paper's initialization analysis predicts a testable pattern: morphological weights should be initialized near a common mean while learnable scalings absorb variance, and ablating initialization over mean and standard deviation should show this explicitly.
  • The large-batch requirement of the Hybrid-MLP suggests morphological layers inject high-variance gradient noise; variance-reduced optimizers or a smaller effective mini-batch may restore trainability at batch 64.
  • If diagonal scaling suffices for universality in MPM, similar summed-dilation-erosion layers with cheap linear couplings may make convolutional morphological networks universal approximators while keeping the pruning advantage.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 6 minor

Summary. The paper studies deep morphological neural networks (DMNNs) built from max-plus and min-plus morphological perceptrons. It proves that existing 'pure' DMNNs that use only max-plus/min-plus MPs are not universal approximators (Theorem 1) and that DEP-based networks are not universal approximators (Theorem 3), arguing that their output gradients are almost everywhere restricted to zero or a single coordinate unit vector (respectively, nonnegative with L1 norm at most one). It then proposes new architectures—MPM, RMPM, MPM-SVD, and Hybrid-MLP—that insert linear activations between morphological layers, and proves universal approximation theorems for MPM (Theorem 4) and Hybrid-MLP (Theorem 5). The experiments demonstrate trainability, pruning advantages, and convergence behavior with large batches.

Significance. The paper makes a clear theoretical contribution by identifying algebraic and analytic obstructions to universality in morphological networks, and by providing constructive architectures that overcome these obstructions while keeping the number of learnable activation parameters modest. The non-universality proofs are rigorous and the universality proofs are constructive, which is valuable for the morphological deep learning community. The empirical results support the claims of trainability and strong pruning properties, which are relevant for edge computing; if the theoretical results hold, they settle a question in the literature about the expressivity of 'pure' morphological networks.

major comments (1)
  1. [Appendix B, proof of Theorem 4] The proof asserts, without stating a lemma or giving a proof or precise citation, that the class of functions of the form max_{k in [K]}(a_k^T x + b_k) + min_{m in [M]}(c_m^T x + d_m) is dense in C(K) for compact K, calling it 'a byproduct of the proof of universality of Maxout networks.' This density claim is load-bearing: it is exactly what converts the constructive result that every MPM can compute such a function into the universal approximation statement. The claim is true (for example, the class contains max-affine functions by taking M=1, c=0, d=0, and max-affine functions are dense by the Maxout universality theorem), but as written Theorem 4 is not fully self-contained. Please state and prove this density fact explicitly as a lemma, or cite a specific theorem from which it follows directly.
minor comments (6)
  1. [Section 4.2, Tables 2 and 4] The text claims 'zero accuracy drop' for the 1173-parameter pruned MPM, but on Fashion-MNIST the unpruned MPM (Table 2) reports 82.86±0.17 while the SNIP-pruned network (Table 4) reports 82.73±0.33, a drop of 0.13 percentage points. Even if this is within one standard deviation, the phrase 'zero accuracy drop' is inaccurate; please rephrase to 'negligible accuracy drop' or specify the exact comparison being made.
  2. [Appendix C] The appendix explicitly states that 'Most of the claims in this appendix are qualitative and lack formal proofs.' Since the main text (Section 3) refers to this appendix to justify design choices such as setting lambda = 1/2, please add a sentence in the main text indicating that those arguments are heuristic, so that readers do not mistake them for proved results.
  3. [Appendix B, proof of Theorem 4] In the construction of the second layer, the bias w^(2)_{(d-1)0} for the output that sums two terms is not specified; the reader must infer that it is set to 0 or another value that makes the displayed equalities hold. Please specify all weights explicitly for that unit.
  4. [Throughout] The abstract says 'Only O(N) parameters (or learnable parameters) per layer of size N belong to the activations,' but in Setting 2 the activation includes fixed O(N^2) matrices U and V; only the learnable parameters are O(N). Please clarify this distinction in the abstract and in the table of parameter counts.
  5. [References] The reference for the bound on the expectation of the maximum of Gaussian random variables is a Mathematics Stack Exchange answer (Sivaraman, 2011), which is not a standard archival source; consider replacing it with a textbook reference or a brief self-contained derivation.
  6. [Section 3, Theorems 4 and 5] The universal approximation theorems are stated for the class of all MPM (resp. Hybrid-MLP) networks, but the proofs construct networks whose depth grows with the input dimension and the number of affine pieces. Please add a sentence clarifying that the density claim is over architectures of arbitrary depth and width, not over a fixed architecture, to avoid a potential misinterpretation.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: impossibility results are first-principles gradient arguments; MPM/Hybrid-MLP universality rests on external Maxout/ReLU universality, with only a proof-completeness gap in Theorem 4's unstated density lemma.

full rationale

The paper's central negative results (Theorems 1–3) are established by direct gradient computations in Appendix A: Lipschitz continuity, the a.e. gradient characterization for max-plus/min-plus networks, the sparse-gradient-in-parameters result, and the l1-gradient bound for DEP networks are all proved from elementary lemmas (Lemmas A.1–A.4). Non-universality then follows from the non-density of functions with a.e. ||grad||1 ≤ 1, proved in Lemmas A.1–A.2. No fitted value or earlier result by the authors is needed. Theorem 4 is constructive: the proof shows explicitly that an MPM can compute functions of the form max_k(a_k^T x+b_k)+min_m(c_m^T x+d_m), then invokes the fact that this class is dense in C(K). The closest thing to an external load-bearing input is the sentence in Appendix B: 'A byproduct of the proof of universality of Maxout networks, is that the class of functions of the form max_{k in [K]}(a_k^T x+b_k) - max_{m in [M]}(c_m^T x+d_m) = ... is a universal approximator on R^d.' This is attributed to Goodfellow et al. (2013), an external benchmark, not to the present authors, and it is not an input of the MPM construction; it is a standard external approximation result. The absence of a stated and proved density lemma is a proof-completeness gap, not circularity. Theorem 5 is also non-circular: it constructs explicit Hybrid-MLP weights that simulate ReLU and Maxout networks layer by layer (using Lemmas B.1 and B.2), so universality transfers from those external benchmarks. Self-citations (notably Maragos and Schafer 1987 and Maragos et al. 2021) appear in the motivation and in Appendix E, where the authors argue explicitly that Theorem 1 is complementary to, rather than derived from, the Representation Theorem; no load-bearing proof invokes these self-citations. Appendix C's initialization discussion is expressly qualitative and not used in the universality proofs. No fitted parameter is relabeled as a prediction, and no architecture is defined in terms of the target function class. Therefore the derivation chain is not circular.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The theoretical core introduces no fitted constants: the large constants C in the universality proofs are existential choices, not learnable parameters. The free parameters listed are experimental hyperparameters and baseline initializations. The main unstated load-bearing assumption is the density of DC functions in Theorem 4, a standard but unproved result.

free parameters (4)
  • MP baseline initialization mean = -5/3
    Hand-tuned in Appendix C ('after extensive trial and error') for the max-plus MP baseline; does not affect the universality theorems but affects baseline comparisons.
  • MP baseline initialization std = 3
    Hand-tuned for the max-plus MP baseline in Appendix C.
  • weight dropout rate = 0.3
    Used for RMPM-Drop in Tables 1 and 2; chosen by the authors, not theoretically derived.
  • activation initialization std (Setting 1) = 1/3.46
    Hand-tuned for the linear activations of fully connected Setting-1 networks (Appendix C).
assumptions (5)
  • standard math The class of functions max_k(a_k^T x + b_k) + min_m(c_m^T x + d_m) is dense in C(K) for compact K (density of DC functions).
    Used without proof in Theorem 4 to conclude MPM universality; a standard consequence of the fact that piecewise-linear functions are dense and are differences of convex functions.
  • standard math Rademacher's theorem: Lipschitz functions on R^d are differentiable almost everywhere.
    Used in Lemma A.1/A.2 and the proofs of Theorems 1 and 3 to justify a.e. differentiability.
  • standard math Max-plus matrix multiplication is associative (Rmax is a semiring).
    Used in Section 3 to show that a multi-layer max-plus MP network collapses to a single morphological layer.
  • domain assumption Universal approximation by Maxout networks (Goodfellow et al. 2013).
    Invoked as the source of the DC-density 'byproduct' in Theorem 4.
  • domain assumption Representation Theorem of Maragos and Schafer (1987).
    Used in Section 2 and Appendix E to contextualize; the paper argues its theorems complement, not rely on, it.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Training Deep Morphological Neural Networks as Universal Approximators." pith.science (2026). https://pith.science/paper/2AKXNHMW

@misc{pith2026250509710,
  author       = {Pith},
  title        = {Pith review of: Training Deep Morphological Neural Networks as Universal Approximators},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2AKXNHMW}},
  note         = {Machine review of arXiv:2505.09710}
}
read the original abstract

We investigate deep morphological neural networks (DMNNs), studying how changes in algebraic structure affect the expressivity and trainability of deep architectures. We show that despite the inherent non-linearity of morphological operations, existing deep morphological architectures fail to be universal approximators and exhibit optimization limitations related to sparse and uninformative gradients. To address these issues, we introduce architectures incorporating constrained "linear" activations between morphological layers and averaging max-plus and min-plus neurons. Only O(N) parameters (or learnable parameters) per layer of size N belong to the activations, with the remaining parameters constrained to morphological operations. We prove universal approximation results for the proposed architectures without requiring substantially larger parameter counts than comparable linear networks. Residual connections and weight dropout further improve generalization. Our experiments show that our networks are trainable and compact, despite the imposed architectural restrictions.

Figures

Figures reproduced from arXiv: 2505.09710 by the authors.

Figure 1
Figure 1. Cases of differentiable and non-differentiable networks with respect to input and weights. [PITH_FULL_IMAGE:figures/full_fig_p016_1.png] view at source ↗
Figure 2
Figure 2. Different landscapes of morphological networks. [PITH_FULL_IMAGE:figures/full_fig_p025_2.png] view at source ↗
Figure 3
Figure 3. Distributions of MP and MPM networks. 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 Value 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 20.0 Density Distribution - Input [PITH_FULL_IMAGE:figures/full_fig_p027_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Distribution of MNIST values. Initialization in experiments. In our experiments, we did not ablate the effect of initialization. Rather, we tried to initialize each network properly according to the above remarks and give it as much of an advantage as possible. Specifi…
Figure 5
Figure 5. Figure 5: Convergence rate of different models on MNIST and Fashion-MNIST for [PITH_FULL_IMAGE:figures/full_fig_p029_5.png]
Figure 6
Figure 6. Figure 6: Regression using MLP, MPM, and non-activated MPM on noisy samples for different underlying functions. [PITH_FULL_IMAGE:figures/full_fig_p030_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 17 canonical work pages

  1. [1]

    slack" and continue the path from this argument. We continue this process until either 1) we reach a

    ˜f is Lipschitz:| ˜f(t2)− ˜f(t1)|=|f(x 0 +t 2ˆv)−f(x 0 +t 1ˆv)|≤L∥x 0 +t 2ˆv−x 0 +t 1ˆv∥=L|t 2−t 1|∥ˆv∥= L|t2−t 1| 2.| ˜f′(t)|≤1 a.e.: Sincef is Lipschitz, by Rademacher’s theorem, it is differentiable a.e. on int(B). SinceB is convex, its boundary has zero measure, and hence its interior int(B) has same measure asB. This means that f is differentiable a....

    work page 2017

  2. [2]

    Consider 3 samples (−1.7,1; 2.3),(5,−2.2; 3.7),(1,1; 4.7)

    Consider an unbiased max-plus MP with 2 inputs and a single output, and an unbiased linear perceptron with 2 inputs and a single output. Consider 3 samples (−1.7,1; 2.3),(5,−2.2; 3.7),(1,1; 4.7) . The loss functions of the two percetrons are illustrated in Figure 2a. As we can see, for this particular dataset, the loss function of the MP (blue) has a smal...

    work page

  3. [3]

    maximum of Gaussians

    Now consider a hybrid network, with a first layer of 2 unbiased max-plus MPs of 2 inputs, and a single output unbiased linear perceptron. We fix the weights of the output perceptron to 1, and the weights w12 =w 21 = 0 , study the loss function for variable weights w11,w 22. Consider again 3 samples (1.2,−2.4; 1.4),(−3.36,2.34; 2.16),(−2.1,−1.5; 2.4) . The...

    work page

  4. [4]

    After extensive trial and error, we found that initializing the weights with a mean of−5/3 and a standard deviation of 3 yielded the best results

    Max-Plus MP-based Networks:As discussed earlier, properly initializing this network required setting the weight variance to a value greater than 1 and the mean to a negative value. After extensive trial and error, we found that initializing the weights with a mean of−5/3 and a standard deviation of 3 yielded the best results. Therefore, this initializatio...

    work page

  5. [5]

    As a result, we used the same initialization as for DEP networks withλ= 1/2

    DEP (λ= 3/4 ):Properly initializing this network proved to be impractical. As a result, we used the same initialization as for DEP networks withλ= 1/2

    work page

  6. [6]

    Given this, the ideal weight initialization follows the same approach as DEP networks withλ= 1/2

    DEP (learnableλ):The parameter λ was initialized from a uniform distributionU([0,1]) , which has a mean of1/2. Given this, the ideal weight initialization follows the same approach as DEP networks withλ= 1/2

    work page

  7. [7]

    finalMNNs*.ipynb

    DEP (λ= 1/2 ) and MPM:The initialization of networks with our proposed fixed λ= 1/2 and MPM networks is significantly simpler than that of MP-based networks. All morphological layers are initialized to follow a standard distribution. 27 Table 12: Parameter count of all models. NetworkMLP MP DEP DEP (λ= 1/2) Act-MP Act-DEP Params.466698 466698 932106 93081...

    work page

  8. [9]

    Increasing:h(x)≥0,∀x,

    work page

Show all 17 references
  1. [10]

    Increasing

    Translation-invariant:P xh(x) = 1, then we can represent the linear operator as a supremum of weighted erosions: Γ(f)(x) = (h∗f)(x) = _ g∈Bas(Γ) ^ y∈Zd f(y)−g(y−x), where the basisBas(Γ)is given by Bas(Γ) ={g∈S: X y∈spt(h) h(y)g(−y) = 0∧g(−x) =−∞⇔h(x) = 0} Example 1.If we take...

  2. [11]

    This is a condition for the proof to work

    In Theorem 1 suprema and infima are only over finite domains, exactly how they are taken in neural networks. This is a condition for the proof to work

  3. [12]

    the weights of the supremum is not inside the previous infimum like they are in the idenities obtained from the Representation Theorem

    In Theorem 1 we assume the use of max-plus and min-plus MPs, i.e. the weights of the supremum is not inside the previous infimum like they are in the idenities obtained from the Representation Theorem

  4. [13]

    linear" activations. We perform regression of simple single-variate single-output functions sampled with zero-mean i.i.d. gaussian noise. To ablate the effect of our

    Our findings indicate that the network requires a large batch size to be trainable, suggesting that the inclusion of morphological layers introduces significant noise in the gradient estimation of stochastic optimization methods like Adam. However, for sufficiently large batch...

    1987

  5. [14]

    First, we see how having suprema and infima over finite domains is implicitly used in the proof

    If we take the limit as the domain tends to be infinite, then Theorem 1 proves the impossibility of representing linear perceptrons with negative weights, or weights that sum up to more that 1. First, we see how having suprema and infima over finite domains is implicitly used ...

  6. [15]

    When we have a finite family of functions, with each being differentiable a.e., then they aresimultaneously differentiable a.e. (i.e. the Lebesgue-measure of the set of points for which at least one of the functions is non-differentiable is zero)

  7. [16]

    Let us see the above in more detail

    When we have a supremum or infimum over a finite domain, then it is definitely attained. Let us see the above in more detail. In the proof of Theorem 1, we write x(n+1) i = maxjf(n+1) ij . We note that each f(n+1) ij is differentiable a.e.. Then, becausej runs over a finite (a...

  8. [17]

    Sinceyn is Lipschitz, so isgn

    Definegn(t) =y n(xj =t,x −j). Sinceyn is Lipschitz, so isgn. Moreover, we are given that∂xjyn(x)≥0 a.e., which implies that dgn dt (t)≥0 a.e.. Now defineg(t) =y(x j =t,x −j). We have thatgn→g pointwise, sinceyn→y pointwise. Therefore, we can apply Lemma E.2 to obtain dg dt (t)...

  9. [18]

    35 Definegn(t) =y n(x+t1)andg(t) =y(x+t1)

    We are given that X j (∇yn(x))j =⟨1,∇y n(x)⟩∈{0,1} ⇒ ⟨1,∇y n(x)⟩≤1,a.e. 35 Definegn(t) =y n(x+t1)andg(t) =y(x+t1). Then dgn dt (t) =⟨1,∇y n(x+t1)⟩≤1,a.e. Sinceyn is Lipschitz, so isgn. Sinceyn→y pointwise, it also holds thatgn→g pointwise. Hence, Lemma E.2 applies, and we have...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.