{"id":"ae3c3465-a2ee-49d2-8ea6-2dbcc842211b","arxiv_id":"2411.15795","paper_version":3,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":9,"one_line_summary":"F-CMA is a random-reshuffling optimizer with a line search and early stopping that the authors claim converges deterministically and trains image classifiers faster than Adam and CMAL.","lead":"The paper introduces F-CMA, an optimizer for training neural networks that reshuffles data each pass and adjusts its step size with a line search. Its authors claim it trains models up to 68% faster and to higher accuracy than popular optimizers, with a deterministic convergence guarantee.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Proposition 5's monotonicity of ζ is contradicted by Algorithm 3 Step 16, so ζ_k→0 and the global convergence theorem are unproven.","rationale":"The paper's headline claim is a deterministic global convergence guarantee for F-CMA. Section 3.3 is built around applying Proposition 1, which requires both boundedness of {w_k} and ζ_k→0. The only result establishing ζ_k→0 is Proposition 5, and its proof is invalid: it assumes an exhaustive dichotomy of the learning-rate update that omits Step 16, where ζ_{k+1}=max(α~_k,α_min) and DFL can return α~_k larger than ζ_k. This is not a matter of missing assumptions or consensus disagreement; it is an internal inconsistency between the algorithm specification and the proof. The same monotonicity assumption is used in Proposition 4 for boundedness, so both premises of Proposition 1 are compromised. The reader's weakest_assumption identifies exactly this flaw, and my reading of the full text confirms it. The experimental comparison is also under-powered, but the theoretical gap alone is sufficient to reject the paper's central claim as written. A simple logging run or a formal trace of the update rules would settle whether ζ ever increases; even if ζ never increases in practice, the proof as written does not establish the claimed theorem.","tokens_in":19873,"tokens_out":4288,"duration_ms":39955,"concrete_test":"Instrument the public F-CMA code to log ζ_k, the branch taken, and the DFL return α~_k at every epoch for one Table 2 configuration (e.g., ResNet-18/CIFAR10 with default hyperparameters). If any epoch executes Step 16 with α~_k > ζ_k and ζ_{k+1} > ζ_k, monotonicity and Proposition 5 are empirically false. Analytically, trace all assignments in Algorithm 3: Step 16 is an update rule permitting ζ_{k+1}=max(α~_k,α_min), and DFL can output α~_k = ζ_kη/δ^j for arbitrarily large j since δ<1; exhibit such a path with Step 13 false to refute the dichotomy used in the proof.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The proof that ζ_k→0 (Proposition 5) starts with the dichotomy \"either ζ_{k+1}=ζ_k or ζ_{k+1}=θζ_k<ζ_k,\" making {ζ_k} monotonically non-increasing. This dichotomy is false: Algorithm 3 Step 16 sets ζ_{k+1}=max(α~_k, α_min) whenever Step 13 fails, and the DFL (Algorithm 2) can return α~_k > ζ_k because it initializes α=ζ_kη and repeatedly multiplies by 1/δ>1 under the while condition. Step 13 failing only requires α~_k‖d_k‖² > τζ_k, which is compatible with α~_k > ζ_k. Therefore ζ can increase, the claimed monotonicity fails, and the contradiction argument in Proposition 5 collapses. Moreover, Proposition 4's proof of boundedness of {w_k} explicitly relies on \"{ζ_k} is by definition a non-increasing sequence,\" so the sequence {w_k} is also not established to be bounded. Since Proposition 1 requires ζ_k→0, it cannot be applied, and Proposition 6's global stationarity result is unsupported. The central theoretical claim of the paper therefore does not follow from the presented proof.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces F-CMA, a random-reshuffling minibatch algorithm for the nonconvex finite-sum problem (1)/(3). It combines a per-epoch sufficient-decrease check with a derivative-free line search (DFL) that uses a coercive surrogate model and at most two evaluations of the true objective. The authors claim a deterministic global convergence result, inf_k ||∇f(w_k)||→0, without convexity of f or a gradient-related search direction, and they report experiments on CIFAR-10 and CIFAR-100 with six architectures showing up to 68% training-time reduction, up to 20% per-epoch efficiency gain, and up to 5% accuracy improvement over eight baselines. The paper also proposes a learning-rate-based early stopping rule and argues that the method has lower memory requirements than adaptive optimizers.","tokens_in":20058,"tokens_out":16087,"duration_ms":130013,"significance":"If the proof were correct, F-CMA would be a notable contribution: a line-search optimizer for deep networks with a deterministic nonconvex convergence guarantee, lower memory than adaptive methods, and a derivative-free search that can reuse any surrogate model. The empirical campaign is broad (54 model-optimizer combinations, six architectures), the code is promised in a public repository, and the theoretical analysis is self-contained, with no fitted constants entering the theorem. These strengths are real. However, the convergence proof contains load-bearing errors: the claimed monotonicity of the step-size sequence is contradicted by the algorithm itself, and the final stationarity argument has sign and evaluation-point mistakes. These flaws invalidate the paper's central theoretical claim as it stands.","major_comments":[{"comment":"Proposition 5 asserts that {ζ_k} is monotonically non-increasing because at every iteration either ζ_{k+1}=ζ_k or ζ_{k+1}=θζ_k<ζ_k. This dichotomy is incomplete: when the condition at Step 13 of Algorithm 3 fails, Step 16 sets ζ_{k+1}=max{α~_k, α_min}. Algorithm 2 (DFL) can return α~_k>ζ_k because it initializes α=ζ_kη and repeatedly multiplies by 1/δ>1, while the failure of Step 13 only requires α~_k||d_k||²>τζ_k and is compatible with α~_k>ζ_k. Hence ζ_k can increase, the claimed monotonicity fails, and the contradiction argument showing ζ_k→0 collapses. Since Proposition 1 requires ζ_k→0 and Proposition 4 explicitly uses the monotonicity to prove boundedness of {w_k}, the global convergence theorem (Proposition 6) is not established.","section":"Section 3.3, Proposition 5; Algorithm 3 Step 16"},{"comment":"The proof of Proposition 6 contains multiple sign and evaluation-point errors. In the case where ¯K′ is infinite, the proof compares the vector d_k with the scalar τζ_k in an inequality, which is not a valid vector inequality; the stationarity conclusion requires passing to norms. In the case where ¯K′′ is infinite, the proof states that ˜α_k≤0, but the line-search return ˜α_k is a nonnegative step size. The Mean-Value theorem is also misapplied: the correct identity is f(w_k+˜α_k d_k)-f(w_k)=˜α_k ∇f(w_k+ξ_k d_k)^T d_k with ξ_k∈(0,˜α_k), and the Armijo inequality gives ∇f(w_k+ξ_k d_k)^T d_k ≤ -γ||d_k||², not ∇f(w_k)^T d_k ≤ γ||d_k||² as written. Once the signs and evaluation points are corrected, the displayed limit argument does not yield ||∇f(w̄)||=0 without additional work; as printed, the proof is invalid.","section":"Section 3.3, Proposition 6"},{"comment":"The boundedness proof depends crucially on the assertion that '{ζ_k} is by definition a non-increasing sequence' when taking limits along a divergent subsequence. Because that monotonicity is false (see the comment on Proposition 5), the boundedness of {w_k} is not proved. Independently, even under a monotonicity assumption, the proof replaces ζ_k by its limit ζ̄ in an upper bound of the form f(w_k)≤f(w0)+P²L_fζ_k(CM+D); since a non-increasing sequence satisfies ζ_k≥ζ̄, the correct upper bound would use the initial value ζ_0, so the displayed inequality has the wrong direction.","section":"Section 3.3, Proposition 4"}],"minor_comments":[{"comment":"Proposition 1 states 'lim_{k→0}ζ_k=0'; the limit should be as k→∞. It also says the sequence {w_k} is produced by Algorithm 1, but the relevant generating method is Algorithm 3.","section":"Section 3.3, Proposition 1"},{"comment":"In the proof of Proposition 5 the text refers to 'Step 17' as the case where the algorithm sets ζ_{k+1}, whereas the instruction that sets ζ_{k+1}=max{α~_k,α_min} is Step 16; Step 17 is only the closing 'end if'.","section":"Section 3.3, Proposition 5"},{"comment":"Assumption 3 is labeled 'f_p bounded below' but defines compact sublevel sets, which is a coercivity condition; Proposition 5 then uses Assumption 3 to conclude φ_k≥0, which requires a specific lower bound (such as f≥0) that is not stated.","section":"Section 3.1, Assumption 3"},{"comment":"In the proof of Lemma 3, the chain of inequalities introduces an extra factor C: from Assumption 4 and ||∇f||≤M the correct bound is ||∇f_p||≤CM+D, so the middle expression should be L_fζ_kp(C||∇f||+D), not L_fζ_kpC(||∇f||+D).","section":"Section 3.1, Lemma 3"},{"comment":"Equation (16) contains the unexplained placeholder '=CO'; this appears to be a typesetting artifact of the bound that follows in the proof of Proposition 1.","section":"Equation (16)"},{"comment":"The dataset names are typeset as 'CIF AR10' and 'CIF AR100' in several sentences; they should be 'CIFAR-10' and 'CIFAR-100'.","section":"Section 4"}],"recommendation":"reject","confidential_remarks":"The manuscript has an extensive and reproducible empirical study, and the idea of a derivative-free line search based on a surrogate is interesting. My rejection is based on the theory: the main convergence theorem is contradicted by the algorithm itself (Step 16 of Algorithm 3 can increase ζ), and the final stationarity proof has sign errors. These are not presentation issues; they invalidate the headline claim. The authors might repair the theory by capping ζ_{k+1} at ζ_k and re-checking the proof of Proposition 6, but that would be a substantive revision of both algorithm and analysis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the paper is not a crank submission: F-CMA is a concrete optimizer that extends the controlled-minibatch line (CMA/CMAL) with a derivative-free extrapolation line search, an internal η scheduler, and gradient clipping in place of per-function coercivity. The line search uses at most two full objective evaluations, and the early-stopping rule conditioned on the learning rate is a sensible energy-saving idea. The authors also ship code and run a 54-combination benchmark across six architectures and two datasets. That is real work.\n\nSecond, the central proof that the method converges to a stationary point does not hold as written. Proposition 5 claims ζ_k is monotonically non-increasing because each step either leaves it or multiplies by θ<1. But Algorithm 3 Step 16 sets ζ_{k+1}=max{α~_k, α_min} whenever the DFL step size fails the small-direction test, and the DFL can return α~_k > ζ_k. So ζ can increase, the dichotomy is false, and the argument that ζ_k→0 collapses. Proposition 4's boundedness proof explicitly leans on ζ being non-increasing, so that conclusion is also unsupported. Proposition 6 then applies a mean-value theorem with a sign error and evaluates the gradient at w_k instead of an intermediate point, so the claimed contradiction ∇f(w̄)=0 doesn't follow. These are not typos; they are load-bearing.\n\nThe empirical section is also weaker than the abstract implies. The headline time savings come mostly from early stopping (F-CMA stops at 78-122 epochs while baselines run 250), not from per-epoch efficiency—Table 2 shows T/Ep for F-CMA is comparable to Adam and worse than SGD. There are no error bars, only one run per configuration, and the comparison omits the line-search optimizers cited in Section 2. The accuracy gains are modest and within what single-seed noise could produce.\n\nOn the positive side, the work is not circular: no fitted constants enter the theorem, and the paper does not define the target result into existence. The code link is a plus. But a proof that fails at two consecutive propositions cannot support a deterministic convergence claim. I would want the authors to fix the learning-rate update (or the proof) and redo the experiments with multiple seeds and proper baselines before believing the main conclusions.\n\nFor a journal, I would send this to review rather than desk-reject outright — the algorithmic idea is worth an expert look and the flaws are specific enough for a referee to pinpoint. But my own recommendation is reject in current form; it needs major revision. Reading group: maybe, as a case study in checking algorithm/proof consistency.","headline":"F-CMA has a genuinely new line-search minibatch design and an honest empirical sweep, but the main convergence theorem is not proven as written: the learning-rate sequence can increase, and the final stationarity argument has a sign/evaluation error.","tokens_in":20659,"tokens_out":4002,"would_cite":false,"duration_ms":34896,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"F-CMA, a random-reshuffling optimizer with a derivative-free line search, claims deterministic convergence to a stationary point for nonconvex finite-sum problems without the memory overhead of adaptive gradient methods.","keywords":["nonconvex optimization","mini-batch","random reshuffling","line search","convergence theory","deep learning","computer vision"],"falsifier":"Run F-CMA on a smooth nonconvex problem with logging of ζ_k, and check whether any iteration at Step 16 sets ζ_{k+1} = max{α̃_k, α_min} with α̃_k > ζ_k; a single such increase contradicts the monotone-decrease premise of Proposition 5, and a run where ζ_k does not converge to zero would directly refute the claimed theorem.","tokens_in":19534,"feed_emoji":"⚡","tokens_out":7777,"duration_ms":63163,"temperature":0.7,"pith_summary":"F-CMA is a mini-batch optimizer for the unconstrained minimization of a sum of smooth, possibly non-convex loss functions. Its central claim is a deterministic global convergence guarantee: even though samples are randomly reshuffled every epoch, the learning rate is adjusted by a sufficient-decrease test and a derivative-free line search, and the sequence of iterates admits a stationary limit point with inf_k ‖∇f(w_k)‖ → 0. The paper argues this needs neither convexity of the objective nor a gradient-related search direction, and replaces the individual coercivity assumption of prior controlled mini-batch methods with gradient clipping. If correct, the method delivers a convergence guarantee comparable to adaptive methods while using less memory, and the reported experiments claim training time cuts of up to 68%, per-epoch efficiency gains up to 20%, and accuracy gains up to 5% on CIFAR-10 and CIFAR-100.","feed_headline":"F-CMA claims 68% faster training with a convergence proof","feed_subtitle":"Mini-batch random reshuffling plus derivative-free line search reaches stationary points deterministically, beating Adam on CIFAR-10/100.","key_machinery":"The load-bearing object is the learning-rate sequence {ζ_k} paired with the per-epoch sufficient-decrease test at Step 6 of Algorithm 3. Each epoch runs random reshuffling (Algorithm 1) producing a tentative point w̃^k, a direction d_k accumulating the per-batch gradients, and an estimate f̃^k of f; if f̃^{k+1} ≤ min{φ_k − γ ζ_k, f(w0)} the point is accepted unchanged, otherwise the algorithm checks whether ‖d_k‖ ≤ τ ζ_k and, failing that, launches the derivative-free line search (DFL, Algorithm 2). DFL scales ζ_k by η, extrapolates a step by repeatedly dividing by δ while a cheap surrogate model ψ satisfies an Armijo-type reduction, and then evaluates the true objective once; the returned step α̃_k and threshold τ decide whether ζ_{k+1} is left unchanged, decreased by θ, or set to max{α̃_k, α_min}. The proof mechanics are: Lemma 3 bounds |f̃^{k+1} − f(w_k)| by $P^{2}$ L_f ζ_k(CM + D), so the estimate tracks the true function as ζ_k shrinks; Assumption 4 (a growth condition, enforced by gradient clipping) keeps gradients bounded; coercivity keeps iterates in compact level sets; and Proposition 5's claimed monotone decrease ζ_k → 0 lets RR's cumulative direction approach the full gradient, which via Proposition 1 yields stationarity.","core_discovery":"On its own terms, the paper's discovery is that the controlled mini-batch idea can be made faster and memory-lighter while keeping a deterministic convergence proof. For the nonconvex finite-sum problem min_w f(w) = Σ_i f_i(w), the authors prove (Proposition 6) that the points produced by F-CMA admit limit points and that at least one is stationary, i.e., inf_k ‖∇f(w_k)‖ → 0. The mechanism is a learning-rate control loop: after each random-reshuffling epoch, the accumulated batch-loss estimate f̃^k must satisfy a sufficient decrease bound; if it does not, the algorithm either shrinks the learning rate or invokes a derivative-free line search (DFL) that uses an arbitrary approximation model ψ of f to extrapolate a trial step, verifying it on the true objective. The convergence argument relies on three lemmas: the gap between the estimated and true objective is O(ζ_k) (Lemma 3), the iterates stay bounded (Proposition 4), and the learning rate tends to zero (Proposition 5).","pith_inferences":["A testable consequence the authors do not draw: if F-CMA's monotone learning-rate premise fails (Step 16 can raise ζ_{k+1} above ζ_k), the proof of ζ_k → 0 is not supported, and one should instrument a run to see whether ζ_k ever increases in practice.","The 68% time reduction is reported as total training time with the early-stopping rule; a stricter comparison that fixes the number of epochs, or reports time-to-accuracy curves, would separate the effect of the optimizer's step quality from the effect of stopping earlier.","The surrogate model ψ could be a subsampled batch; using the same permutation for ψ and the true check might couple the two evaluations, so an ablation varying the size of ψ would clarify how much of the line-search benefit comes from the proxy.","Because the proof only needs a coercive approximation ψ, other cheap models could be plugged into DFL without changing the convergence argument."],"forward_implications":["If the convergence theorem holds, deep networks trained with mini-batch random reshuffling can be certified to reach a stationary point deterministically, closing part of the gap between adaptive-method practice and convergence theory.","The gradient-clipping substitution for individual coercivity lets the theory apply to standard training pipelines without per-component lower-bound assumptions.","The learning-rate-driven early stopping rule (ε threshold) yields an automatic stopping signal, which the authors tie to lower energy consumption and carbon footprint during training.","Because ψ can be any approximation model, the line search can be decoupled from the exact loss, opening the scheduler to cheaper surrogates while preserving the true-objective verification."],"supporting_citations":[{"why":"Provides the ease-controlled random reshuffling framework and the safeguard rule from which F-CMA is adapted.","marker":"[50]"},{"why":"The approximated controlled mini-batch algorithm whose per-epoch aggregated-loss decrease condition F-CMA borrows while using fewer full loss evaluations.","marker":"[51]"},{"why":"Supplies the random reshuffling iteration (Algorithm 1) and its convergence properties that F-CMA builds on.","marker":"[20]"},{"why":"Supplies the model for gradient error bounds used in Lemma 3 and Assumption 4.","marker":"[17]"},{"why":"The primary adaptive optimizer baseline that F-CMA compares against and whose memory and convergence limitations motivate the method.","marker":"[1]"},{"why":"The CIFAR-10 and CIFAR-100 datasets used in the empirical comparison.","marker":"[85]"},{"why":"The ResNet architecture family used in the experiments.","marker":"[86]"},{"why":"Supports the gradient-clipping technique that enforces bounded gradients in place of individual coercivity.","marker":"[52]"},{"why":"Also supports the gradient-clipping technique and its effect on training speed for non-adaptive methods.","marker":"[53]"}],"fun_headline_variants":["F-CMA: 68% faster training with a deterministic convergence proof","Controlled minibatch optimizer slashes training time 68%, proves convergence","New optimizer: 68% faster training, guaranteed stationarity","F-CMA beats Adam on CIFAR with 68% speedup and proof"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The convergence theorem stands on the claim that the learning rate never increases, yet the algorithm's line-search branch can raise it, and if that happens the proof that the learning rate tends to zero no longer goes through.","fun_headline_variants_meta":{"raw":{"variants":["F-CMA: 68% faster training with a deterministic convergence proof","Controlled minibatch optimizer slashes training time 68%, proves convergence","New optimizer: 68% faster training, guaranteed stationarity","F-CMA beats Adam on CIFAR with 68% speedup and proof"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000637,"raw_usage":{"total_tokens":2930,"prompt_tokens":935,"completion_tokens":1995,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":551,"completion_tokens_details":{"reasoning_tokens":1914}},"tokens_in":551,"tokens_out":1995,"duration_ms":13311,"temperature":1.0,"reasoning_tokens":1914,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:57:31.452417+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run F-CMA on a smooth nonconvex problem with logging of ζ_k, and check whether any iteration at Step 16 sets ζ_{k+1} = max{α̃_k, α_min} with α̃_k > ζ_k; a single such increase contradicts the monotone-decrease premise of Proposition 5, and a run where ζ_k does not converge to zero would directly refute the claimed theorem.","supporting_citations":[{"cited_title":"Convergence of ease-controlled Random Reshuffling gradient Algorithms under Lipschitz smoothness","cited_arxiv_id":"2212.01848","evidence_quote":"Provides the ease-controlled random reshuffling framework and the safeguard rule from which F-CMA is adapted."},{"cited_title":"CMA Light: a novel Minibatch Algorithm for large-scale non convex finite sum optimization","cited_arxiv_id":"2307.15775","evidence_quote":"The approximated controlled mini-batch algorithm whose per-epoch aggregated-loss decrease condition F-CMA borrows while using fewer full loss evaluations."},{"cited_title":"Mishchenko, A","cited_arxiv_id":null,"evidence_quote":"Supplies the random reshuffling iteration (Algorithm 1) and its convergence properties that F-CMA builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the model for gradient error bounds used in Lemma 3 and Assumption 4."},{"cited_title":"Krizhevsky, G","cited_arxiv_id":null,"evidence_quote":"The CIFAR-10 and CIFAR-100 datasets used in the empirical comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The ResNet architecture family used in the experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supports the gradient-clipping technique that enforces bounded gradients in place of individual coercivity."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Also supports the gradient-clipping technique and its effect on training speed for non-adaptive methods."}],"review_version":1}