{"id":"0c2cd98c-147a-4a8e-8d47-4ccde7be56d8","arxiv_id":"2504.14356","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"This paper presents a unified MILP training framework for dense and convolutional networks, but the equations force pre-activations to be nonnegative, so the exact ReLU encoding and the global optimality claims are not supported as written.","lead":"This paper proposes a mixed-integer programming framework for training neural networks that jointly learns weights, prunes layers and filters, and encodes ReLU, max-pooling, and sparsity constraints in one optimization problem. It aims to make small neural networks exactly optimizable and verifiable, but the formulations as written contain a load-bearing error and the experiments only approach optimality.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Constraint (1i) forces hidden pre-activations to be nonnegative, so the ReLU zero branch is unreachable and the claimed exact encoding fails.","rationale":"I read the paper as a claim of new exact mixed-integer formulations for training and architecture selection. The correctness of the formulation can be checked directly from the equations. The reader's weakest assumption identifies exactly the defect I find: constraints (1i) and (2o) impose z ≥ 0 for hidden pre-activations, which is internally inconsistent with the stated z_min < 0 and with constraints (1g)/(2f). This removes the negative branch of ReLU, so the claimed exact encoding fails as printed. This is a mathematical error in the formulation, not a dispute with the broader literature. The paper's own experiments also report MIP gaps up to 20.3%, so the 'globally optimal' claim is stronger than the evidence, and the absence of code, data, and MNIST subset size further limits reproducibility. However, those issues are secondary to the core correctness defect. The proposed concrete test settles the matter: a one-neuron network with a fixed negative pre-activation is feasible under any exact ReLU encoding but infeasible under the printed (1i). I agree with the reader's REJECT verdict and recommend no change.","tokens_in":14485,"tokens_out":4358,"duration_ms":40092,"concrete_test":"Instantiate the dense MILP with one hidden neuron and fixed parameters W1 = -1, b1 = 0, W2 = 1, b2 = 0, with input x = 1. The true ReLU network computes z = -1, a = 0, output 0. Solving the printed model with these fixed parameters should report infeasible because constraint (1i) requires z = -1 ≥ 0. Remove the erroneous 0 ≤ z lower bound from (1i), keep z_min ≤ z ≤ Mγ, and the same instance becomes feasible with a = 0. If the formulation cannot represent this elementary ReLU unit with a negative pre-activation, the claimed exact encoding is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is an exact MILP encoding of ReLU networks, and the load-bearing defect is in the printed bounds. Constraint (1i) states 0 ≤ a,z ≤ Mγ_l for all hidden layers, and constraint (2o) states 0 ≤ z,a ≤ Mγ_c for convolutional layers. These constraints make hidden pre-activations nonnegative. For z ≥ 0, the ReLU zero branch is unreachable: with δ = 0, (1e)-(1h) require a ≥ z > 0 and a ≤ 0, which is infeasible; with δ = 1, the activation equals z. Thus every feasible hidden unit is in the identity regime, and the MILP represents only affine maps rather than general ReLU networks. The paper explicitly sets z_min < 0 and describes these as precomputed bounds essential to the big-M encoding, but (1i) overwrites the lower bound with 0. The same contradiction appears in (2o) and is reinforced by (2r), which states z ∈ [0,M]. This is not a matter of consensus or experimental setup; the equations as printed do not encode ReLU. The defect is repairable by replacing the lower bound in (1i) with z_min ≤ z ≤ Mγ and using the standard big-M pair, but as written the formulation is not exact.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes mixed-integer linear programming (MILP) models for training fully connected and convolutional neural networks with ReLU activations, integrating parameter learning, architecture selection, and structural sparsity regularization into a single optimization problem. The dense model is presented in Section 2 and the convolutional model in Section 3, with experiments on IRIS, Wine, WBC, and MNIST reported in Section 4. The central claims are that the formulations provide an exact encoding of ReLU network behavior and that solving them yields globally optimal solutions with respect to a composite objective that balances prediction accuracy, weight sparsity, and architectural compactness.","tokens_in":14710,"tokens_out":4491,"duration_ms":38864,"significance":"If correct, the proposed framework would be a useful contribution to the growing area of exact, verifiable neural network training, offering a unified way to handle small-scale training, pruning, and architecture search with formal guarantees. The paper is clearly written and the intended scope is appropriate for a methods paper. However, the correctness of the core formulation is not established: the printed constraints do not encode general ReLU networks, and the experimental results do not support the global optimality claim. As written, the central contribution is undermined, so the significance is currently not realized.","major_comments":[{"comment":"Constraint (1i) imposes 0 ≤ z(l)_{i,j} ≤ M γ_l for all hidden layers, which forces every hidden pre-activation to be nonnegative. In the ReLU encoding (1e)-(1h), this makes the zero branch (δ=0) feasible only when z=0 and infeasible for any z<0. The text explicitly states that z_min < 0 is a precomputed lower bound essential for the big-M encoding, so (1i) directly contradicts the stated design. A correct big-M ReLU encoding requires z(l)_{i,j} to range over an interval with a negative lower bound, e.g., z_min ≤ z ≤ Mγ and a(l)_{i,j} ≤ Mγ_l. As printed, the dense model only represents affine maps rather than general ReLU networks, invalidating the exactness claims in the abstract and Section 5.","section":"Section 2, constraint (1i)"},{"comment":"The convolutional formulation repeats the same defect. Constraint (2o) forces 0 ≤ z(l)_{i,c,h,w} ≤ M γ(l)_c, and constraint (2r) states z(l)_{i,c,h,w} ∈ [0, M]. Consequently all convolutional pre-activations are nonnegative, and the ReLU encoding (2d)-(2g) cannot represent the zero branch for negative pre-activations. The statement in Section 3 that constraints (2d)-(2g) yield an exact piecewise-linear representation of ReLU is therefore false without a valid negative lower bound on z, so the convolutional model also fails to encode general CNN behavior.","section":"Section 3, constraints (2o) and (2r)"},{"comment":"The reported MIP gaps of 8.6%, 4.0%, 10.2%, and 20.3% indicate that the solver did not prove optimality within the time budget. The abstract claims that the framework yields globally optimal solutions with respect to the composite objective, but these experiments provide only incumbent solutions without optimality certificates. This is a separate issue from the encoding defect, but it is another load-bearing discrepancy between the claims and the evidence.","section":"Section 4, Tables 1 and 2"}],"minor_comments":[{"comment":"The term \"MLIP\" appears in the first paragraph and should be \"MILP\".","section":"Section 3, first paragraph"},{"comment":"The notation l ∈ L_pool is used but the set L_pool is never defined in the paper.","section":"Section 3, constraint (2h)"},{"comment":"The sentence \"The ReLU activity dropped by over 30%\" is not defined or quantified against a clear baseline; please state how this metric is measured.","section":"Section 4, experimental description"},{"comment":"The symmetry-breaking constraint is claimed to preserve global optimality, but the paper does not prove that every functional equivalence class contains a feasible representative satisfying the row-sum ordering.","section":"Section 2, constraint (1o)"},{"comment":"The choice of the big-M constant for the max-pooling encoding is described only as needing to be \"conservative\"; the paper should specify how M is computed or bounded to ensure correctness.","section":"Section 3, constraint (2j)"},{"comment":"For reproducibility, the exact train-test splits (including whether a subset of MNIST was used) and the number of samples per split should be reported.","section":"Section 4, dataset splits"}],"recommendation":"reject","confidential_remarks":"The central ReLU encoding error is substantial and affects the validity of the experiments, because the solved models are effectively linear if all pre-activations are constrained to be nonnegative. The formulation could in principle be repaired by replacing (1i), (2o), and (2r) with appropriate lower bounds, but the current version does not support the claims. Additionally, the lack of code or detailed reproducibility information made it difficult to verify the experimental results beyond what is reported in the tables."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the central claim does not hold up. Constraints (1i) and (2o) set a lower bound of 0 on hidden pre-activations, so the ReLU zero branch is unreachable and the MILP encodes only affine maps. This is not a minor typo; it contradicts the paper's own definition of ReLU and the described big-M mechanism with z_min < 0.\n\nWhat the paper does well: it is clearly organized, and the intent is sensible—a unified MILP for training, pruning, and architecture search. The big-M ReLU encoding is standard (the authors cite the verification literature), but they add layer-level and filter-level gating and max-pooling selection, which are reasonable extensions. The symmetry-breaking constraints are sensible, and they are upfront about NP-hardness.\n\nSoft spots beyond the fatal bug: the global optimality claim is overblown. Table 2 reports a final MIP gap of 20.3%, and Table 1 shows gaps up to 10.2%, so the solutions are not proven globally optimal within the stated tolerance. There is no code, no data subset size for MNIST, no error bars, and no comparison with the cited MILP training works (e.g., Thorbjarnarson & Yorke-Smith, Fischetti & Jo, ElAraby et al.). That makes novelty and reproducibility hard to assess.\n\nThis paper is for researchers in exact optimization-based ML. They will catch the constraint error quickly but may still find the pruning and pooling formulations useful as a template once fixed. As written, it is a cautionary example rather than a usable method.\n\nRecommendation: do not publish in current form. If the constraint is corrected to z_min ≤ z ≤ Mγ, the core idea might be salvageable, but the experimental section needs major work—code, baselines, and tighter gaps. I would still send this to peer review rather than desk-reject, because the topic is relevant and the flaw is concrete; a referee can give detailed, actionable feedback. Expected outcome: rejection or major revision.","headline":"The paper's central exact-encoding claim fails because constraints (1i) and (2o) force pre-activations to be nonnegative, making ReLU's zero branch unreachable and reducing the model to affine maps.","tokens_in":15307,"tokens_out":3193,"would_cite":false,"duration_ms":27517,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["90C11","90C10","68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"Training, pruning, and architecture selection for small ReLU networks can be cast as one globally optimal mixed-integer linear program.","keywords":["mixed-integer programming","neural networks","ReLU modeling","convolutional networks","network pruning","sparsity","interpretability","formal verification"],"falsifier":"Take the stated MILP with one hidden ReLU neuron, one sample $(x,t)=(-1,-1)$, and a linear output. An exact ReLU network fits this with a negative pre-activation and zero activation, but the constraint $0 \\le z \\le M\\gamma_1$ forbids negative $z$, so the solver cannot reach the true optimum; the returned optimal value and the sign of $z$ settle whether the encoding is exact.","tokens_in":1637,"feed_emoji":"🧠","tokens_out":5746,"duration_ms":93808,"temperature":0.7,"pith_summary":"This paper claims that training small fully connected and convolutional ReLU networks can be cast as one mixed-integer linear program in which weights, activations, and architecture are decided together. ReLU switches, max-pooling choices, and layer or filter pruning are encoded by binary variables, so a solver can return a globally optimal network for a composite objective that trades prediction error against weight sparsity and architectural compactness. The authors report experiments on IRIS, Wine, WBC, and MNIST showing that sparse networks can be found in a single run. A sympathetic reader would take the paper as an argument that formal optimization, not gradient heuristics, can build small interpretable networks with certificates.","feed_headline":"One optimization problem trains, prunes, and verifies small networks","feed_subtitle":"ReLU switches and pruning choices become binary variables, so the solver finds a globally optimal sparse architecture.","key_machinery":"The load-bearing object is the big-M piecewise-linear encoding of a ReLU unit as $a = \\max(0,z)$, implemented with a binary switch $\\delta$ through the constraints $a \\ge 0$, $a \\ge z$, $a \\le z - z_{\\min}(1-\\delta)$, and $a \\le z_{\\max}\\delta$, where $z_{\\min}<0<z_{\\max}$ are precomputed pre-activation bounds. Around this core, the model adds binary switches $\\gamma_l$ for dense layers and $\\gamma_c^{(l)}$ for convolutional output channels that zero all weights, biases, and activations of a pruned unit through $M\\gamma$ big-M bounds; auxiliary variables $u$ that make $u=|W|$ at optimum, giving an exact $\\ell^1$ penalty; and binary selection variables $\\zeta$ that implement max pooling. These objects convert nonlinear training and architecture selection into one MILP.","core_discovery":"On its own terms, the paper's central claim is that the forward pass of a dense ReLU network and of a CNN—convolution with weight sharing, ReLU, max pooling, flattening, and the final affine output—admits a linear encoding with binary helpers, and that the same encoding can be extended to structural decisions. Each ReLU unit is modeled by a binary $\\delta$ that picks the identity or zero branch via big-M bounds; each layer (dense) or output channel (CNN) gets a binary $\\gamma$ that zeros all associated weights, biases, and activations when pruned; the $\\ell^1$ penalty is linearized by auxiliary variables $u=|W|$; max pooling is modeled by binary selection variables $\\zeta$. The paper's conclusion is that any feasible solution of the MILP is a completely specified network and any optimal solution minimizes the composite objective globally, so training, pruning, and architecture selection no longer need separate pipelines.","pith_inferences":["The ReLU encoding itself is not tied to the squared loss; swapping in an $\\ell^1$ or hinge-style objective should preserve the same forward constraints, which would extend the framework to robust or quantile-style training.","The stated pruning bounds $0 \\le z \\le M\\gamma$ contradict the negative pre-activation bounds $z_{\\min}<0$; a corrected model would need $z \\ge z_{\\min}\\gamma$ (or equivalent) to keep the zero branch of ReLU reachable, so the exactness claim should be read as intended rather than as literally displayed.","Because the binary count grows as samples times units times spatial locations, the practical ceiling is small tabular or low-resolution image problems; the paper's own MNIST configuration already exceeds 1.5 million ReLU binaries.","A quick single-neuron test on a point requiring a negative pre-activation would show whether an implemented solver, not just the displayed equations, honors the claimed exactness."],"forward_implications":["Small dense networks can be trained with hidden layers pruned automatically while keeping accuracy; the paper reports 96.7–99.1% test accuracy on IRIS, WBC, and Wine, with the WBC model collapsing to one hidden layer.","Convolutional filters can be pruned during training without a separate pruning pass; the paper reports 4 of 6 filters retained with 91% test accuracy on MNIST.","Because the forward pass is a set of linear constraints, constraints encoding fairness, monotonicity, or logic rules can be added at training time and are satisfied by construction.","Every feasible MILP solution is a completely specified network, so the same object used for training can be handed to a verifier without re-encoding.","The composite objective lets the user interpolate between accuracy, weight sparsity, and architectural compactness by tuning $\\alpha$, $\\lambda$, and $\\beta$."],"supporting_citations":[{"why":"Establishes the standard approach of encoding ReLU networks for robustness verification in MILP, which the paper adapts into a training formulation.","marker":"[10]"},{"why":"Provides strong mixed-integer programming formulations for trained neural networks, the technical basis for tight big-M ReLU encodings.","marker":"[12]"},{"why":"Shows how to model deep neural networks as mixed-integer linear programs, supplying the core encoding idea extended here.","marker":"[17]"},{"why":"Demonstrates optimal architecture search with mixed-integer programming, the precedent for treating structural choices as binary decisions inside optimization.","marker":"[19]"},{"why":"Target heuristic for pruning and compression that the joint MILP training is meant to replace or unify.","marker":"[8]"},{"why":"Classical optimal-pruning baseline whose exact combinatorial counterpart the paper claims to realize.","marker":"[9]"},{"why":"Prior work casting neural network configuration as an MILP, which the paper generalizes to modern ReLU and convolutional architectures.","marker":"[14]"},{"why":"Recent MILP-based optimal training of integer-valued networks, showing that joint training and structural optimization is a feasible research direction.","marker":"[15]"}],"fun_headline_variants":["MILP unifies training, pruning, and architecture selection","One solver, globally optimal sparse neural networks","Exact MILP encodes ReLU and pruning for optimal nets","Binary variables make training and pruning one optimization","Interpretable networks via a single mixed-integer program"],"cache_read_input_tokens":17408,"weakest_assumption_plain":"The argument stands or falls on the big-M encoding being an exact representation of ReLU for every unit; the displayed constraints $0 \\le z \\le M\\gamma$ force all pre-activations nonnegative, which removes the negative branch of ReLU and so the model as written cannot represent a general ReLU network.","fun_headline_variants_meta":{"raw":{"variants":["MILP unifies training, pruning, and architecture selection","One solver, globally optimal sparse neural networks","Exact MILP encodes ReLU and pruning for optimal nets","Binary variables make training and pruning one optimization","Interpretable networks via a single mixed-integer program"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000485,"raw_usage":{"total_tokens":2363,"prompt_tokens":884,"completion_tokens":1479,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":500,"completion_tokens_details":{"reasoning_tokens":1403}},"tokens_in":500,"tokens_out":1479,"duration_ms":9477,"temperature":1.0,"reasoning_tokens":1403,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:50:28.680321+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the stated MILP with one hidden ReLU neuron, one sample $(x,t)=(-1,-1)$, and a linear output. An exact ReLU network fits this with a negative pre-activation and zero activation, but the constraint $0 \\le z \\le M\\gamma_1$ forbids negative $z$, so the solver cannot reach the true optimum; the returned optimal value and the sign of $z$ settle whether the encoding is exact.","supporting_citations":[{"cited_title":"Mathemat- ical Programming 183(1), 3–39 (2020) 19","cited_arxiv_id":null,"evidence_quote":"Provides strong mixed-integer programming formulations for trained neural networks, the technical basis for tight big-M ReLU encodings."},{"cited_title":"Constraints 23, 296–309 (2018)","cited_arxiv_id":null,"evidence_quote":"Shows how to model deep neural networks as mixed-integer linear programs, supplying the core encoding idea extended here."},{"cited_title":"In: Pr oceedings of the AAAI Conference on Artificial Intelligence, vol","cited_arxiv_id":null,"evidence_quote":"Demonstrates optimal architecture search with mixed-integer programming, the precedent for treating structural choices as binary decisions inside optimization."},{"cited_title":"In: Adva nces in Neural Information Processing Systems, pp","cited_arxiv_id":null,"evidence_quote":"Classical optimal-pruning baseline whose exact combinatorial counterpart the paper claims to realize."},{"cited_title":"Chemical Engineering Research and De sign 88(1), 55– 60 (2010)","cited_arxiv_id":null,"evidence_quote":"Prior work casting neural network configuration as an MILP, which the paper generalizes to modern ReLU and convolutional architectures."},{"cited_title":"Plos one 18(2), 0261029 (2023)","cited_arxiv_id":null,"evidence_quote":"Recent MILP-based optimal training of integer-valued networks, showing that joint training and structural optimization is a feasible research direction."}],"review_version":1}