{"id":"40f61ee6-3e75-466f-8287-09482b3f9999","arxiv_id":"2505.02537","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Alternating one-sided saturating activations, or non-positive weights with ReLU, make four-layer monotonic MLPs universal approximators, and a new sign-switch layer avoids constrained weights in practice.","lead":"The paper proves that monotonic neural networks can keep the ReLU activation and still approximate any monotonic function with a constant number of layers, and it introduces a parameter scheme that drops the usual non-negative weight constraint. This matters because monotonic models are used where outputs must respect known cause-effect directions, and ReLU trains faster than the sigmoid and tanh activations previously required.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The proof overclaims: Theorem 3.5 is finite-point interpolation, and the advertised universal approximation lacks the compactness argument; the tie-breaking in Theorem 3.5 is also false as printed.","rationale":"I read the central theoretical claim as likely correct but not fully proven as advertised. The reader's weakest assumptions identify the same two soft spots: unhandled ties in the geometric separation and the unstated transition from finite-point interpolation to universal approximation. My stress-test confirms both. The tie problem is concrete and easy to exhibit, but it is repairable by an explicit tie-breaking rule that respects componentwise order among equal f-values. The universal-approximation gap is also real: the theorem proves interpolation on finitely many points, and uniform approximation on compacta requires an additional epsilon-delta argument that is standard but absent from the paper. Because these are fixable and do not appear to threaten the main construction, a CONDITIONAL verdict remains appropriate; I would not escalate to REJECT. The empirical section is supportive but depends on external hyperparameters and an external codebase, which further supports keeping the verdict conditional rather than ACCEPT. I found no evidence of a deeper inconsistency that would invalidate the four-layer construction.","tokens_in":22600,"tokens_out":22267,"duration_ms":294850,"concrete_test":"Run the Layer-1 construction of Theorem 3.5 on the two-point set f≡0 with x1=(1,1), x2=(0,0) in the order stated; verify that no non-negative α exists with α^T x2 > α^T x1, then re-run with the order swapped to confirm the tie-breaking repair. Separately, write out the missing compactness argument for a continuous monotone f on a compact cube: choose a product grid, apply the finite construction to grid values, and bound sup_x |g(x)-f(x)| via monotonicity and uniform continuity. If the finite-to-uniform step cannot be completed, the universal-approximation claim should be weakened to finite-point interpolation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Theorem 3.5, as stated and proved, establishes approximate interpolation on a finite set of n points, not uniform approximation on compact domains. The abstract and Propositions 3.9/3.11 claim universal approximation, but the paper never supplies the missing compactness argument: choose a finite grid, apply the finite construction to grid values, and then use monotonicity of both the target f and the constructed network plus uniform continuity of f to bound the sup-norm error inside each grid cell. Without this step, the central advertised claim goes beyond what is demonstrated. A second, more localized gap appears in the same theorem: the proof says ties are 'resolved arbitrarily', but if two points have equal f-value and one is componentwise at most the other, the later point in an arbitrary tie order can be componentwise smaller than the earlier point. In that case no non-negative normal vector can place the later point on the positive side of the Layer-1 hyperplane, so the required hyperplane does not exist. This is repairable by breaking ties so that componentwise-smaller points come first, which is always possible, but the printed statement and proof do not say this. Both issues are fixable and do not appear to invalidate the underlying result, but the paper as written overstates what is rigorously established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies MLPs with constrained weights for monotonic function approximation. It claims that (i) a non-negative-weight MLP with three hidden layers and monotone activations saturating on alternating sides can interpolate any monotone function on any finite point set; (ii) as a consequence, four-layer non-negative MLPs with alternating sigma and its point reflection, and four-layer non-positive MLPs with any one-sided saturating activation (including ReLU), are universal approximators for monotone functions; and (iii) an 'activation-switch' parametrization that uses the positive and negative parts of the weight matrix achieves the same expressivity without explicit weight constraints and trains stably. The proof constructs half-space indicators with large weights, forms level sets through intersections and complements, and sums a telescoping series in the output layer. Experiments compare the proposed parametrization with several monotonic architectures on five datasets.","tokens_in":22827,"tokens_out":17265,"duration_ms":216999,"significance":"If the theoretical claims were fully established, the paper would make a useful contribution: it extends the four-layer universal approximation result of Mikulincer and Reichman (2022) from threshold activations to all one-sided saturating monotone activations, and it gives the practically relevant result that non-positive weight constraints with ReLU avoid the convexity limitation of non-negative constraints. The proposed activation-switch parametrization is simple, removes the need to preselect alternating activations, and the reported empirical results are competitive. However, the advertised universal approximation is not proved as written: the paper establishes only finite-sample approximate interpolation, and one step in that proof is flawed. The gaps are repairable, but they are load-bearing for the central claims.","major_comments":[{"comment":"Theorem 3.5 proves only approximate interpolation on an arbitrary finite set of points; it does not by itself imply universal approximation on a compact domain. Propositions 3.9 and 3.11 assert universal approximation without supplying the usual finite-grid argument, so the paper's central advertised claim is not established as written. The missing step is load-bearing: for a compact K and tolerance epsilon, one must choose a finite net whose cells have small f-variation (using uniform continuity of f), apply the finite construction to the values of f on the net, and then use monotonicity of both f and the constructed network to bound the sup-norm error in each cell. Please add this argument and explicitly state the target class and topology, e.g., continuous monotone functions on compact subsets of R^d with the sup norm.","section":"Sections 3.2-3.3, Propositions 3.9 and 3.11"},{"comment":"The proof assumes that for every pair with i<j there is a non-negative normal alpha such that x_j is in the positive half-space and x_i is in the negative half-space. This is false when f(x_i)=f(x_j) and x_j is componentwise at most x_i, because every alpha>=0 then satisfies alpha^T x_j <= alpha^T x_i. The phrase 'ties resolved arbitrarily' does not prevent this situation. The gap is repairable by ordering equal-valued points so that componentwise-smaller points come first, but as printed the proof is invalid at this step.","section":"Theorem 3.5, proof of Layer 1"},{"comment":"The theorem states that the network can 'interpolate' the target function, but the proof uses approximations throughout: Lemma 3.6 and Lemma 3.7 only show that indicators are approximated for large lambda, and no limit is taken. With one-sided saturating activations such as ReLU, exact step functions are not representable, so the exact-interpolation statement is stronger than what is proved. Please either weaken the theorem to epsilon-approximate interpolation with an explicit error bound, or give an exactness argument for the activations for which exact interpolation holds.","section":"Theorem 3.5 and Lemmas 3.6-3.7"}],"minor_comments":[{"comment":"In the proof of Lemma 3.6 (and its appendix counterpart), the parameters w=lambda alpha^T and b=lambda alpha^T beta give sigma(lambda alpha^T(x+beta)), not sigma(lambda alpha^T(x-beta)); the correct bias is b=-lambda alpha^T beta. Since beta is arbitrary, this is a local algebraic slip that does not affect the existence of the desired half-space indicators, but it should be corrected.","section":"Lemma 3.6 and Lemma A.3"},{"comment":"Theorem 3.5 says '3 hidden layers' while the proof uses four affine layers (three hidden layers plus an output layer). The layer count should be stated consistently with Propositions 3.9 and 3.11, which refer to 'at least 4 layers'.","section":"Theorem 3.5"},{"comment":"The intersection in Lemma 3.7 is written as A=intersection_{i=0}^n A_i, but the following text refers to A_1,...,A_n; the index should run from 1 to n.","section":"Lemma 3.7"},{"comment":"Proposition 3.10 is stated without proof; since it is the basis for Proposition 3.11, a short derivation showing how the bias terms transform under the double sign flip would make the equivalence easier to verify.","section":"Proposition 3.10"},{"comment":"The experimental section does not report the number of random seeds or how the confidence intervals in Table 1 were computed; please add these details so the empirical comparison is reproducible.","section":"Section 5, Table 1"}],"recommendation":"major_revision","confidential_remarks":"The core constructive idea appears sound and the gaps identified above are repairable: the missing compactness argument and the tie-breaking fix are routine, and the sign error is local. I do not see circularity or hidden assumptions beyond those noted. I recommend major revision rather than rejection because the paper's central claims as stated exceed what is rigorously demonstrated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a real result worth a serious referee. The paper extends the known four-layer universal approximation bound for threshold activations (Mikulincer–Reichman 2022) to all monotone activations that saturate on at least one side, and the corollary — non-positive weights with ReLU give universal approximation for monotone functions — is new and genuinely surprising. The activation-switch parametrization in Section 4 is a useful practical contribution on top of the theory.\n\nThe construction itself is sound. I checked the layer-by-layer logic of Theorem 3.5: Lemma 3.7's saturation-side conditions line up with the alternation pattern, the union/intersection bookkeeping is consistent, and the fourth-layer telescoping sum interpolates correctly. Proposition 3.10 (equivalence between weight-sign flips and point-reflected activations) is straightforward algebra, and Proposition 4.1 correctly shows the switch parametrization covers both weight-sign branches. The empirical results look favorable but are secondary.\n\nThree soft spots, in order of seriousness. First, the abstract and the propositions claim universal approximation, but Theorem 3.5 as stated and proved is finite-point interpolation. The missing compactness argument is routine — interpolate on a uniform grid, then use monotonicity of both the target and the network plus uniform continuity to bound errors within grid cells — but it is not in the paper. That is the largest gap, and it is fillable. Second, the separation claim in Theorem 3.5 is false as printed for tied values: if f(x_i) = f(x_j) and x_j is componentwise at most x_i, no non-negative normal vector puts x_j on the positive side, so 'ties resolved arbitrarily' is not enough. This is repairable by ordering ties so componentwise-smaller points come first, but the printed proof overclaims. Third, Lemma 3.6 and its appendix twin have a sign slip: b = λα^Tβ yields λα^T(x + β), not λα^T(x − β). Since β is a free parameter the lemma still holds, but the algebra is off as written.\n\nNo code is released and the experimental pipeline is only described by reference to an external codebase, so the empirical part is not fully reproducible from the paper alone.\n\nThis paper is for people working on provably monotone architectures and anyone building on Mikulincer–Reichman. It deserves peer review rather than a desk reject — a conditional accept, with the tie handling fixed and the finite-to-compact argument added. I'd bring it to a reading group.","headline":"Genuine extension of the four-layer monotone universal approximation result to all one-sided saturating activations, with a clean non-positive-ReLU corollary; the printed proof has a sign slip, a fixable tie gap, and stops at finite-point interpolation.","tokens_in":23360,"tokens_out":12643,"would_cite":true,"duration_ms":129858,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","41A30"],"pacs":[],"model":"deepseek-v4-flash","headline":"A monotone MLP with non-negative weights and activations that saturate on alternating sides can interpolate any monotone function on any finite point set with three hidden layers, and flipping the weights to non-positive makes four-layer…","keywords":["monotonic neural networks","universal approximation","weight constraints","saturating activations","ReLU networks","monotone interpolation","non-positive weights","activation switch"],"falsifier":"Take the constant function $f \\equiv 0$ on the two points $x_1 = (1,1)$ and $x_2 = (0,0)$ in $\\mathbb{R}^2$, ordered $x_1$ before $x_2$ as the theorem's 'ties resolved arbitrarily' permits. For every non-negative vector $\\alpha$, we have $\\alpha \\cdot x_2 = 0 \\leq \\alpha \\cdot x_1$, so no hyperplane with non-negative normal can put $x_2$ strictly on its positive side and $x_1$ strictly on its negative side; the Layer-1 separation the proof invokes for every pair therefore cannot be instantiated on this input, showing the proof's claim that monotonicity alone guarantees such a separating hyperplane is false as stated.","tokens_in":22389,"feed_emoji":"🧠","tokens_out":19850,"duration_ms":205268,"temperature":0.7,"pith_summary":"Monotonic neural networks are models whose predictions are forced by construction to rise (or fall) with their inputs, and most architectures build this in with bounded activations like sigmoid, which are slow to train and cannot extrapolate beyond the data range. This paper argues that boundedness is not what gives these networks their universal approximation power; what matters is that each activation saturates on one side and consecutive layers saturate on alternating sides. The main theorem proves by explicit construction that a non-negative-weight MLP with three hidden layers interpolates any monotone function on any finite set of points under this alternation, matching the best depth bound previously known only for step-function activations. A second result shows that flipping the weight constraint to non-positive makes four-layer ReLU networks universal approximators for monotone functions, something the non-negative setting provably cannot do with convex activations. An 'activation switch' layer derived from this insight removes the need for weight reparameterization, and the paper's experiments show it matches or outperforms existing monotonic architectures.","feed_headline":"Alternating saturation lets 4-layer MLPs fit any monotone function","feed_subtitle":"Universal approximation now works with unbounded activations, and flipped weight signs let plain ReLU qualify.","key_machinery":"The load-bearing device is the saturation side of a monotone activation: a function is right-saturating (in $S^+$) when it has a finite limit at $+\\infty$ and left-saturating (in $S^-$) when it has a finite limit at $-\\infty$; ReLU, ELU, and softplus are left-saturating, while sigmoid and tanh saturate both sides. The proof mechanism is a three-stage geometric construction in which each hidden layer refines a set-valued approximation: first-layer units, using weight vectors $\\alpha \\geq 0$ scaled toward infinity, approximate indicator functions of half-spaces; second-layer units, whose activation saturates in the opposite direction, combine those indicators into indicator functions of intersections (the sets $A_i^{(2)}$ built from the half-spaces that contain $x_i$ and exclude later points); third-layer units, saturating back in the original direction, form the indicator of the superlevel set via complements of intersections; and the fourth layer's non-negative weights $w_j = (f(x_j) - f(x_{j-1}))/\\gamma^{(3)}$ telescope into an exact interpolation. The other load-bearing identity is the sign-flip equivalence, Proposition 3.10: two adjacent layers with non-positive weights and activation $\\sigma$ compute exactly what two non-negative layers would compute with the point-reflected activation $\\sigma'(x) = -\\sigma(-x)$, and repeated application of this flip converts the alternating-saturation theorem into the non-positive-weight universal approximation result.","core_discovery":"The paper's central claim is that the saturation side of a monotone activation, not boundedness, confers universal approximation power on weight-constrained MLPs. Theorem 3.5 states that an MLP with non-negative weights and three hidden layers can interpolate any monotone non-decreasing function on any finite set of points, provided the activations are monotone and alternate saturation sides: $\\sigma^{(1)} \\in S^-$, $\\sigma^{(2)} \\in S^+$, $\\sigma^{(3)} \\in S^-$, or the reverse, where $S^-$ and $S^+$ denote activations with finite limits at $-\\infty$ and $+\\infty$. The construction builds indicator functions of half-spaces in the first layer, of intersections in the second, of the superlevel sets $\\{x_j : f(x_j) \\geq f(x_i)\\}$ in the third, and finishes with a fourth layer whose non-negative weights telescope the differences between consecutive function values, so the output equals $f(x_i)$ at every training point. Because this bound does not depend on input dimension, it matches the best-known constant-layer result, previously available only for threshold activations. The companion result, Proposition 3.11, converts the alternation theorem into a statement about the sign of the weights: with four layers and non-positive weights, any saturating monotone activation, plain ReLU included, yields a universal approximator for monotone functions, in contrast to the convex-function-only limitation of non-negative ReLU networks. The paper also packages this into a practical layer form, the post-activation switch $\\hat{f}(x) = W^+ \\sigma(x) + W^- \\sigma(-x) + b$, which lets the network learn its activation's saturation side rather than fixing it in advance.","pith_inferences":["The paper proves interpolation on finite point sets and asserts universal approximation on compact domains; the unstated bridge is a grid-density argument for continuous monotone functions, and spelling it out would make the abstract's claim fully rigorous.","The proof's phrase 'ties resolved arbitrarily' hides an assumption: equal-valued point pairs must still be separable by a hyperplane with non-negative normal. Ordering ties so that no later point is componentwise smaller than an earlier equal-valued point would repair the construction, if such an ordering always exists.","Because the activation-switch layer learns its saturation direction per neuron, the alternating pattern could be dropped or learned end-to-end; a testable variant is per-neuron gating of the $W^+$/$W^-$ split, which would preserve the monotonicity guarantee only if the weights feeding the gate stay non-negative.","The paper leaves open whether non-saturating activations such as Leaky-ReLU can be used; a concrete next test is whether a minimally saturating modification, any finite limit on one side, inherits the theorem, which would settle the boundary of the result."],"forward_implications":["Any one-sided saturating monotone activation, including ReLU, ELU, CELU, SELU, softplus, exponential, tanh, or sigmoid, can replace bounded activations in monotonic MLPs while preserving universal approximation at constant four-layer depth.","Non-positive weight constraints make four-layer ReLU networks universal approximators for monotone functions, whereas the non-negative counterpart with ReLU can only approximate convex functions.","The activation-switch layer, $\\hat{f}(x) = W^+ \\sigma(x) + W^- \\sigma(-x) + b$, needs no weight reparameterization and no a priori choice of alternating activations, which the paper's experiments show eases initialization and improves training stability.","Existing constrained architectures that use three activation classes to obtain universal monotonic approximation need only an activation and its point reflection; the third, bounded class is redundant.","On the five benchmark datasets reported, the proposed layer matches or outperforms prior monotonic architectures without special weight initialization."],"supporting_citations":[{"why":"Supplies the four-layer interpolation bound for threshold-activated monotone networks that Theorem 3.5 matches and generalizes to one-sided saturating activations.","marker":"(Mikulincer & Reichman, 2022)"},{"why":"Defines the non-negative-weight and bounded-activation monotonic MLP framework, whose dimension-dependent layer count and convexity limitation frame the paper's contributions.","marker":"(Daniels & Velikova, 2010)"},{"why":"Provides the three-activation constrained architecture whose empirical performance the theory explains and whose third activation class the paper shows is redundant.","marker":"(Runje & Shankaranarayana, 2023)"},{"why":"Supplies the vanishing-gradient analysis of bounded activations that motivates the switch to unbounded saturating activations.","marker":"(Glorot & Bengio, 2010)"}],"fun_headline_variants":["Monotone nets need saturation side, not boundedness","Alternating saturation unlocks universal monotone approximation","Unbounded activations can still be universal for monotone functions","Flipped weight sign turns plain ReLU into universal monotone net","Saturation side, not boundedness, decides monotone net power"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The construction in the proof of Theorem 3.5 assumes that every pair of training points with $i < j$, including pairs with equal function values, can be separated by a hyperplane with non-negative normal so that $x_j$ falls on the positive side; this can fail when the points are comparable in the componentwise order but their function values tie, and the proof's 'ties resolved arbitrarily' does not ensure a separating orientation.","fun_headline_variants_meta":{"raw":{"variants":["Monotone nets need saturation side, not boundedness","Alternating saturation unlocks universal monotone approximation","Unbounded activations can still be universal for monotone functions","Flipped weight sign turns plain ReLU into universal monotone net","Saturation side, not boundedness, decides monotone net power"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000801,"raw_usage":{"total_tokens":3604,"prompt_tokens":1110,"completion_tokens":2494,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":726,"completion_tokens_details":{"reasoning_tokens":2409}},"tokens_in":726,"tokens_out":2494,"duration_ms":15460,"temperature":1.0,"reasoning_tokens":2409,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:50:48.291013+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the constant function $f \\equiv 0$ on the two points $x_1 = (1,1)$ and $x_2 = (0,0)$ in $\\mathbb{R}^2$, ordered $x_1$ before $x_2$ as the theorem's 'ties resolved arbitrarily' permits. For every non-negative vector $\\alpha$, we have $\\alpha \\cdot x_2 = 0 \\leq \\alpha \\cdot x_1$, so no hyperplane with non-negative normal can put $x_2$ strictly on its positive side and $x_1$ strictly on its negative side; the Layer-1 separation the proof invokes for every pair therefore cannot be instantiated on this input, showing the proof's claim that monotonicity alone guarantees such a separating hyperplane is false as stated.","supporting_citations":[{"cited_title":"and Reichman, D","cited_arxiv_id":null,"evidence_quote":"Supplies the four-layer interpolation bound for threshold-activated monotone networks that Theorem 3.5 matches and generalizes to one-sided saturating activations."},{"cited_title":"and Velikova, M","cited_arxiv_id":null,"evidence_quote":"Defines the non-negative-weight and bounded-activation monotonic MLP framework, whose dimension-dependent layer count and convexity limitation frame the paper's contributions."},{"cited_title":"and Shankaranarayana, S","cited_arxiv_id":null,"evidence_quote":"Provides the three-activation constrained architecture whose empirical performance the theory explains and whose third activation class the paper shows is redundant."}],"review_version":1}