{"id":"33b09dee-594c-4027-8fcb-55638702037a","arxiv_id":"1908.06209","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Parametric majorizers replace a bi-level training problem with single-level surrogate losses that upper-bound the original objective, enabling efficient learning of energy-based models.","lead":"The paper introduces 'parametric majorizers', surrogate single-level optimization problems that upper-bound the loss of a bi-level energy-minimization training problem, so parameters can be learned with standard tools. It proves a hierarchy of surrogates and shows applications in CT reconstruction, segmentation, and image denoising.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The implemented iterative surrogate (Eq. 22) is not a parametric majorizer under Def. 1: it can be strictly positive when the true loss is zero, so the paper's 'without collapse' guarantee does not cover the algorithm used in the experiments.","rationale":"The reader's weakest assumption was Eq. (8), with the CT rank-deficient case and the iterative over-approximation noted as secondary. I agree that Eq. (8) is a genuine limitation for the CT application, and the paper's parenthetical 'practically even works beyond this setting' is exactly the kind of practical claim that lies outside the theory. But the more load-bearing gap for the central claim is the iterative surrogate (22): it fails the zero-equivalence half of Definition 1, so it is not even a parametric majorizer under the paper's own definition. This matters because the denoising experiments, which provide the paper's strongest quantitative evidence, are run with (22)-(23), and the abstract promises 'efficient algorithms without collapse' from parametric majorization. The exact iterative majorizer of Prop. 4 is internally sound, and the non-iterative surrogates in Prop. 2 appear correct, so this does not invalidate the core theoretical framework. It does mean that the practical headline claim is not guaranteed by the paper's theory for the algorithm actually implemented. The proposed check is decisive and inexpensive: evaluating Eq. (22) at a perfect θ with a non-optimal xbar directly demonstrates the failure of Definition 1. This supports the reader's CONDITIONAL verdict rather than a rejection, because the gap is localized to the iterative approximation and is partially acknowledged in the appendix.","tokens_in":22700,"tokens_out":26241,"duration_ms":252183,"concrete_test":"Compute Eq. (22) on the one-dimensional example of Section 3.3 (x* = 0.3, y = 1.5) at a parameter θ for which x(θ) = x*, with a previous iterate xbar = 0.8. Since the true loss is zero, Definition 1 requires the surrogate to be zero, but Eq. (22) evaluates to l(x*, xbar) + W_E(∇l(x*, xbar), xbar) > 0. This single calculation settles whether the iterative surrogate satisfies the zero-equivalence clause of Definition 1. A companion run of iterations (23) from xbar = 0.8, recording the true bilevel loss at each step, would further confirm whether Prop. 4's monotonicity transfers to Eq. (22).","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central object is the parametric majorizer of Def. 1, which requires both l(x,x(θ)) ≤ S(x,y,θ) and the zero-equivalence condition l(x,x(θ)) = 0 ⇒ S(x,y,θ) = 0. The iterative surrogate (22) is an over-approximation obtained by applying Fenchel's inequality to the linearized majorizer (21), and it does satisfy the upper-bound property. However, it does not inherit zero-equivalence. Fix a perfect parameter θ with x(θ) = x* and take a previous iterate xbar ≠ x*. Then l(x*, x(θ)) = 0, while Eq. (22) evaluates to l(x*, xbar) − ⟨p, xbar⟩ + E(xbar) + E*(p), with p = ∇l(x*, xbar), which equals l(x*, xbar) + W_E(p, xbar) > 0 for the squared-Euclidean and cross-entropy losses used in the paper. Hence the objective actually minimized in (23), and used for the denoising results in Table 1, is not a parametric majorizer in the paper's own sense. The appendix (A.1.5) concedes that the descent lemma of Prop. 4 holds only 'approximately' for this over-approximation; the zero-equivalence failure is a sharper and independent reason that the monotonicity and no-collapse guarantees do not transfer to the implemented iterative scheme.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses the bilevel training problem for parametric energy minimization models, in which parameters θ are learned so that the minimizer x(θ)=argmin_x E(x,y,θ) matches a supervised target x* according to a loss l. Because direct differentiation through the argmin is costly and problematic for nonsmooth energies, the authors propose to replace the bilevel objective by a single-level surrogate S(x,y,θ) that majorizes the loss l(x,x(θ)) and satisfies a zero-equivalence condition, then minimize Σ_i S(x_i^*, y_i, θ). They construct a chain of surrogates: the Bregman distance of the lower-level energy, its dual Bregman-conjugate form, partial surrogates via the W-function, and a gradient penalty under strong convexity, and they prove an ordering result in Proposition 2. They also introduce an iterative majorizer based on a linearized bound, prove a descent lemma for the exact iteration (Proposition 4), and propose an over-approximated iterative surrogate (Eqs. 22-23) that is used in the experiments. Applications are presented for computed tomography, variational segmentation, and analysis-operator denoising.","tokens_in":22982,"tokens_out":16488,"duration_ms":144285,"significance":"If the claims held, the framework would be a useful bridge between energy-based modeling and deep learning: it gives a single-level training objective, avoids differentiating through an argmin, is applicable to convex nonsmooth energies, and empirically trains denoising models substantially faster than the implicit-differentiation baseline of [26]. The ordering of surrogates in Proposition 2 is elegant, the toy example in Section 3.3 is consistent, and the authors provide a public implementation. The main caveat is that the implemented iterative scheme used for the headline denoising results is not a parametric majorizer in the paper's own sense, and the CT experiment uses the gradient penalty outside the regime in which it is proven to be a majorizer. Thus the central 'without collapse' guarantee is established only for the non-iterative surrogates, not for all algorithms used in the experiments.","major_comments":[{"comment":"The iterative surrogate in Eq. (22) is not a parametric majorizer under Definition 1. For a parameter θ with x(θ)=x* and a previous iterate xbar≠x*, the true loss is l(x*,x(θ))=0, but Eq. (22) evaluates to l(x*,xbar)-⟨∇l(x*,xbar),xbar⟩+E(xbar)+E*(∇l(x*,xbar)) = l(x*,xbar)+W_E(∇l(x*,xbar),xbar), which is strictly positive for the squared-Euclidean and cross-entropy losses used in Sections 4.2 and 4.3 because ∇l(x*,xbar) is not a subgradient of E at xbar. Hence the objective actually minimized in Eq. (23) fails the zero-equivalence clause of Definition 1. Appendix A.1.5 already concedes that Proposition 4 holds only approximately for Eq. (22); the zero-equivalence failure is a separate, sharper reason that the descent and no-collapse guarantees do not transfer to the implemented iterative scheme. Since Table 1 reports denoising results from exactly this scheme, the paper's central claim of 'without collapse' is not established for the algorithm used in its main experiment. The revision should either prove a descent/no-collapse property for the over-approximation under explicit conditions or clearly label Eq. (23) as a heuristic with no monotonicity guarantee.","section":"§3.4, Eq. (22) and Appendix A.1.5"},{"comment":"The CT experiment uses the gradient penalty (17) for E(x)=1/2||Ax-y||²+βR(x)+⟨x,N(θ,y)⟩. The text says this is a parametric majorizer for the Euclidean loss 'if A has full rank (and practically even works beyond this setting...)'. When A is rank-deficient, the energy is not strongly convex and the modulus m(θ,y) in Eq. (17) is zero (or undefined), so the last inequality of Proposition 2 does not apply. The parenthetical 'practically works' is an empirical claim, not a proof; in particular, the data term controls only the range of Aᵀ, not the nullspace component of x*-x(θ). The revision should state the full-rank condition as an explicit assumption for the CT experiment or provide a separate bound that handles rank-deficient A.","section":"§4.1, Eq. (17)"},{"comment":"Proposition 4 is proved for the exact iterative procedure that minimizes the right-hand side of Eq. (20b), but the implemented iteration (23) minimizes a different objective obtained by an additional Fenchel upper bound and by dropping the constant C. The paper should state explicitly that Eq. (23) is not the algorithm covered by Proposition 4, and should report which loss values are monitored during the step-size reduction heuristic described in Appendix A.2.3. As written, the connection between the theoretical descent lemma and the experimental iterative scheme is too loose to support the claims made in Section 3.4.","section":"§3.4, Eq. (23)"}],"minor_comments":[{"comment":"The argument order of the Bregman distance is inconsistent: Proposition 3 writes l(x,y)=D_w(y,x), while Eq. (26) writes the log loss as D_h(x_i^*, x_i(θ)). For nonsymmetric distances these two expressions are not equal. Please use one convention throughout and adjust the proof of Proposition 3 accordingly.","section":"§3.4 / §4.2"},{"comment":"The caption says the orange curve is the partial surrogate obtained from Eq. (15) by inserting z=∇E1(x*), while Eq. (16) defines the partial surrogate by choosing z∈∂E2(x*). The correspondence between the plotted curves and the displayed equations should be clarified.","section":"Figure 1"},{"comment":"The statement of Proposition 2 upper-bounds by 1/(m(θ,y))||q_i||², but the proof arrives at 1/(2m(θ,y))||q_i||². Since the former is the larger bound, the proof is not wrong, but the constants should be reconciled for clarity.","section":"Appendix A.1.3"},{"comment":"Eq. (9) writes D_Eθ without the superscript 0 that is introduced in Eq. (10); please use D^0_Eθ consistently, since the choice of subgradient matters for the identity.","section":"§3.2, Eq. (9)"}],"recommendation":"major_revision","confidential_remarks":"The conceptual core is sound for the non-iterative surrogates, and the experimental speedups are promising. The main issue is that the implemented iterative algorithm, which produces the headline denoising results, is outside the paper's own theoretical framework. The authors should either restrict the claims, prove additional properties for the over-approximation, or make the heuristic status of the iterative scheme explicit. The CT rank-deficiency gap should also be addressed before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Daniel,\n\nQuick read of 1908.06209. The core idea is genuine: define a parametric majorizer S(x,y,θ) that upper-bounds the bi-level loss l(x, x(θ)) and vanishes when the loss vanishes, then train by minimizing the majorizer. That's a clean way to avoid differentiating through argmin, and it connects Bregman distances, SSVM duals, and the generalized perceptron into one ordering (Prop 2). The convex analysis looks correct to me, and the toy example is actually illuminating. They also ship code, which helps.\n\nWhat's good: the surrogate hierarchy gives practitioners a clear menu—Bregman surrogate, partial surrogate, gradient penalty—with computational costs. The denoising experiment (96 9x9 filters trained in minutes rather than days) makes a real point about scalability. The segmentation example is a nice demonstration that non-smooth energies fit the same frame. The paper is honestly written: the appendix explicitly says Eq. (22) is an over-approximation and Prop. 4 holds only approximately.\n\nThat last point is also the main soft spot. The abstract says the framework yields 'efficient algorithms without collapse of the energy function.' But the zero-equivalence condition in Def. 1 is what prevents collapse, and Eq. (22)—the iterative surrogate actually used in Table 1—does not satisfy it. For a perfect parameter with x(θ)=x*, the surrogate at a previous iterate xbar is l(x*, xbar) + W_E(p, xbar) > 0, not zero. So the descent and no-collapse guarantees proven for Prop. 4 do not transfer to the implemented scheme. The paper admits this in A.1.5, but it undercuts the abstract's claim. This is addressable—either weaken the claim or find a different iterative majorizer that preserves zero-equivalence.\n\nOther soft spots are minor. The CT application uses the gradient penalty when A is rank-deficient, which goes beyond the theory; the text says it 'practically works,' but that's an empirical assertion. The experiments have no error bars and the segmentation run uses four images, so treat the quantitative claims as illustrative.\n\nBottom line: the framework is real and useful for imaging researchers who want model-based training without bi-level pain. The theory is sound for the static surrogates; the iterative part is a heuristic that works in practice. That deserves a serious referee, but the abstract and conclusion need to be brought in line with what is proven.","headline":"A solid, transparent framework for majorizer-based training of energy models, but the headline 'without collapse' claim outruns the theory for the iterative scheme used in the experiments.","tokens_in":23503,"tokens_out":3143,"would_cite":true,"duration_ms":30107,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["90C25","90C30","68U10"],"pacs":[],"model":"deepseek-v4-flash","headline":"Bi-level training of parametric energy models can be replaced by a single-level majorizing surrogate, and the paper proves a hierarchy from exact Bregman distances down to a cheap gradient penalty.","keywords":["parametric majorization","bilevel optimization","energy minimization","Bregman distance","gradient penalty","variational image reconstruction","surrogate loss","convex analysis"],"falsifier":"On the CT setup with a rank-deficient angular sampling operator $A$, evaluate $l(x^*_i, x_i(\\theta))$ and the gradient-penalty surrogate for a dense random grid of $\\theta$ values; if any sample has surrogate value below the true loss, the majorization property that the practical claim relies on is violated, and the CT result would rest on an unproven inequality rather than the paper's theorem.","tokens_in":22488,"feed_emoji":"⚡","tokens_out":13703,"duration_ms":116081,"temperature":0.7,"pith_summary":"The paper's project is to make parametric energy minimization trainable at scale. In an energy-based model the prediction $x(\\theta)$ is the minimizer of $E(x,y,\\theta)$, and the natural training objective $l(x^*, x(\\theta))$ is bi-level: a gradient step would require differentiating through an argmin. The authors propose to replace that objective with a parametric majorizer, a single-level surrogate $S(x,y,\\theta)$ that lies above the true loss for every $\\theta$ and vanishes exactly when the loss vanishes. They prove that Bregman distances of the energy and their relaxations form a chain of such majorizers, ending in a simple gradient penalty, and show experimentally that minimizing these surrogates trains usable CT, segmentation, and denoising energies without the energy collapsing to a constant.","feed_headline":"Bilevel energy training becomes one surrogate objective","feed_subtitle":"Parametric majorizers bound the true loss from above, so gradients need no argmin differentiation.","key_machinery":"The central object is the parametric majorizer $S(x,y,\\theta)$, a function satisfying $l(x,x(\\theta)) \\le S(x,y,\\theta)$ for every $\\theta$ and vanishing exactly when the loss vanishes. The paper's canonical example is the Bregman distance of the lower-level energy, $D^0_{E_\\theta}(x^*, x(\\theta)) = E(x^*) - E(x(\\theta))$, a convex-analysis measure of deviation from a linear lower bound. Bregman duality rewrites it as $D^{x^*}_{E^*_\\theta}(0,q)$, a Bregman distance in the convex conjugate, and decomposing $E=E_1+E_2$ yields a partial surrogate $\\min_{z\\in\\partial E_2(x^*)} W_{E_1}(-z,x^*)$ plus, under strong convexity, the final gradient penalty $\\frac{1}{m(\\theta,y)}\\|q\\|^2$. The iterative variant uses the Bregman three-point identity to linearize the loss around the current solution $x(\\theta_k)$, producing a descent lemma for the exact majorizer and an implementable over-approximation.","core_discovery":"On the paper's own terms, the central claim is that the bi-level problem $\\min_\\theta \\sum_i l(x^*_i, x_i(\\theta))$ with $x_i(\\theta) = \\arg\\min_x E(x,y_i,\\theta)$ can be solved by minimizing a single-level surrogate provided $l(x,z) \\le D_{E_\\theta}(x,z)$ for all $x,z$ and the surrogate vanishes when the loss vanishes. The Bregman surrogate $D^0_{E_\\theta}(x^*_i,x_i(\\theta)) = E(x^*_i) - E(x_i(\\theta))$ is the tightest member of the family; by Bregman duality it equals $D^{x^*_i}_{E^*_\\theta}(0,q_i)$ with $q_i \\in \\partial E(x^*_i)$, and when $E$ is $m(\\theta,y)$-strongly convex this is bounded by the gradient penalty $\\frac{1}{m(\\theta,y)}\\|q_i\\|^2$. Proposition 2 records the whole chain $l(x^*_i,x_i(\\theta)) \\le D^0_{E_\\theta}(x^*_i,x_i(\\theta)) \\le \\min_{z\\in\\partial E_2(x^*_i)} W_{E_1}(-z,x^*_i) \\le \\frac{1}{m(\\theta,y)}\\|q_i\\|^2$, so the practitioner may choose the cheapest bound that still majorizes the loss. The iterative variant re-linearizes around the current solution and, for the exact majorizer, yields a descent property; the implemented version is an acknowledged over-approximation of that ideal. Experiments on learned CT corrections, TV-entropy segmentation, and analysis-operator denoising show that the surrogates are trainable with standard first-order tools and, in the denoising case, cut training time by an order of magnitude relative to implicit-differentiation baselines while matching their quality.","pith_inferences":["One can read the Bregman surrogate as a continuous analogue of margin-based structured prediction: instead of enforcing a fixed margin, the energy is required to grow at least as fast as the loss away from the optimum, which suggests a direct bridge between energy-based learning and large-margin classifiers.","If the gradient penalty is a valid majorizer, the training objective becomes a regression on the first-order optimality residuals of the energy; this could let practitioners attach a learned correction to a known physical model and train purely on violations of the optimality conditions, a testable recipe for inverse problems beyond CT.","Because the exact descent guarantee applies only to the unimplemented iterative majorizer, a natural empirical check is to monitor the true bilevel loss and the surrogate per iteration; the paper's step-size-reduction heuristic already hints that the over-approximation can break monotonicity.","Since Proposition 2 gives a nested sequence of bounds, the gap between the Bregman surrogate and the gradient penalty could serve as a diagnostic for how much accuracy is being traded for speed, guiding adaptive selection of the surrogate during training."],"forward_implications":["Training an energy-based model can be done with ordinary first-order optimization on the surrogate, avoiding second-order or implicit differentiation of the argmin.","A zero surrogate value certifies a perfect match, because $S(x,y,\\theta)=0$ implies $l(x,x(\\theta))=0$, so the training signal is aligned with the model's actual minimizer.","The ordering in Proposition 2 gives a principled trade-off between surrogate tightness and computational cost, with the gradient penalty usable exactly when the energy is strongly convex.","The iterative surrogate offers a way to correct for non-separable problems, and the experiments indicate that iterating is needed to reach competitive denoising performance.","The approach applies to non-smooth convex energies, as demonstrated by the TV-plus-entropy segmentation model, so it is not restricted to differentiable variational models."],"supporting_citations":[{"why":"This reference supplies the Bregman-distance identity used to pass between primal and dual surrogate formulations.","marker":"[11]"},{"why":"It provides the Bregman duality and convex-conjugate machinery that the surrogate derivations rely on.","marker":"[10]"},{"why":"It establishes the relative strong convexity condition that makes the loss dominated by the energy's Bregman distance.","marker":"[94]"},{"why":"It defines relative smoothness and relative strong convexity, which underpin Proposition 1.","marker":"[65]"},{"why":"It formulates the structured-prediction margin setting that the parametric-majorizer approach generalizes to continuous non-smooth energies.","marker":"[92]"},{"why":"It describes SSVM margin-rescaling strategies that motivate treating the separable zero-loss case as a single-level problem.","marker":"[95]"},{"why":"It provides the nonconvex majorization-minimization theory that the iterative surrogate builds on.","marker":"[39]"},{"why":"It is the analysis-operator learning baseline whose setup the denoising experiment repeats and whose cost the surrogate approach cuts.","marker":"[26]"}],"fun_headline_variants":["One-step majorizer replaces bilevel energy optimization","Train energy minimizers without argmin gradients","Surrogate majorizers make bilevel training single-level","Tightest majorizer kills bi-level pain","Skip argmin differentiation with a majorizer bound"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is Equation (8), that the chosen loss is everywhere at most the Bregman distance of the energy; when that fails, as the authors acknowledge it formally does for rank-deficient measurement operators in their CT experiment, the surrogates are no longer guaranteed to bound the true loss.","fun_headline_variants_meta":{"raw":{"variants":["One-step majorizer replaces bilevel energy optimization","Train energy minimizers without argmin gradients","Surrogate majorizers make bilevel training single-level","Tightest majorizer kills bi-level pain","Skip argmin differentiation with a majorizer bound"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000825,"raw_usage":{"total_tokens":3698,"prompt_tokens":1125,"completion_tokens":2573,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":741,"completion_tokens_details":{"reasoning_tokens":2502}},"tokens_in":741,"tokens_out":2573,"duration_ms":16235,"temperature":1.0,"reasoning_tokens":2502,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:52:30.642521+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On the CT setup with a rank-deficient angular sampling operator $A$, evaluate $l(x^*_i, x_i(\\theta))$ and the gradient-penalty surrogate for a dense random grid of $\\theta$ values; if any sample has surrogate value below the true loss, the majorization property that the practical claim relies on is violated, and the CT result would rest on an unproven inequality rather than the paper's theorem.","supporting_citations":[{"cited_title":"A simplified view of first order methods for optimization","cited_arxiv_id":null,"evidence_quote":"It establishes the relative strong convexity condition that makes the loss dominated by the energy's Bregman distance."},{"cited_title":"Efficient Piecewise Training of Deep Structured Models for Semantic Segmentation","cited_arxiv_id":null,"evidence_quote":"It defines relative smoothness and relative strong convexity, which underpin Proposition 1."},{"cited_title":"Max- Margin Markov Networks","cited_arxiv_id":null,"evidence_quote":"It describes SSVM margin-rescaling strategies that motivate treating the separable zero-loss case as a single-level problem."},{"cited_title":"Composite Optimiza- tion by Nonconvex Majorization-Minimization","cited_arxiv_id":null,"evidence_quote":"It provides the nonconvex majorization-minimization theory that the iterative surrogate builds on."},{"cited_title":"Insights Into Analysis Operator Learning: From Patch-Based Sparse Models to Higher Order MRFs","cited_arxiv_id":null,"evidence_quote":"It is the analysis-operator learning baseline whose setup the denoising experiment repeats and whose cost the surrogate approach cuts."}],"review_version":1}