{"id":"b476bbaa-6470-4c16-87ca-a4f6a368965a","arxiv_id":"2508.12270","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"L-SR1 is a learned second-order optimizer whose rank-one preconditioner is trained with a secant penalty, and it outperforms LGD on 3DPW human mesh recovery while using a smaller model.","lead":"L-SR1 trains a small neural network to act as a learned preconditioner for the classical SR1 quasi-Newton algorithm, adding a soft penalty that encourages the secant condition. The authors report faster convergence on several analytic benchmarks and a lower 3D human mesh recovery error than the learned-gradient-descent baseline LGD.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed learned projection is never computed at inference: Algorithm 1 uses raw vector-generator outputs, and Eq. (9) only adds a soft secant penalty to the meta-loss, so the central quasi-Newton guarantee is unverified and no test-time secant residual is reported.","rationale":"The reader's weakest assumption is the same one I would stress: the meta-loss penalty in Eq. (9) is treated as though it were a projection enforcing Eq. (4) at inference. I agree with that identification. I do not see a more fundamental internal contradiction: the outer-product preconditioner is PSD by construction, the elementwise MLP design is consistent with dimension invariance, and the empirical comparisons, while thin (single PA-MPJPE number, no error bars, no code release), would be interpretable if the method were implemented as claimed. The reason this is load-bearing rather than a wording quibble is that the paper's advertised contribution is precisely the projection mechanism: the title, the PGSM abstract sentence, the contribution list, and Sec. 4.2 all say the projection 'enforces' the secant condition and PSD. The actual algorithm never invokes the projection defined in Eq. (8). A soft penalty is a legitimate training technique, and it might work, but the paper must either implement a true projection or reframe the contribution as 'secant-regularized learned preconditioning' and provide test-time residual measurements. Because the claims as written are not supported by the described method, and because the code is withheld until acceptance, the reader's REJECT verdict remains appropriate. The concrete test I propose would settle the empirical half of the concern: if the λ_sec-trained model has substantially lower test-time residuals than the no-penalty model, the regularization is effective and the practical impact of the missing projection is limited.","tokens_in":14580,"tokens_out":8002,"duration_ms":88546,"concrete_test":"Run the trained L-SR1 checkpoint on the 32 quadratic test functions (N=10) and, at each inner step k, compute the secant residual r_k = ||p_k - (I + Σ_{v∈B_L} vv^T)q_k||_2 / (||p_k||_2 + ε), averaged over steps and problems, for models trained with and without λ_sec R_sec, and for L-BFGS as a reference. If the λ_sec-trained model's residual is not substantially smaller than the no-penalty model's, the PGSM claim that meta-training enforces the secant relation empirically fails; if it is substantially smaller, the soft-penalty design is effective and the practical impact of the missing explicit projection is largely mitigated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central novelty (PGSM, Sec. 4.2) is a 'learned projection' that is supposed to keep B_k PSD and satisfy the secant condition (Eq. 4). PSD holds by construction because B_k = B0 + Σ vv^T with B0 = I, but the secant condition is never enforced at inference. Eq. (6) defines B* as an argmin projection; Eq. (8) restricts this to the structured family; Eq. (9) then replaces this projection by adding λ_sec ||p_k - B_k q_k||^2 to the meta-loss. Algorithm 1 (lines 4-11) simply uses the unprojected v_k = P(f_k) in the buffer and computes d_k = Σ vv^T g_{k-1}; no projected vectors are computed or applied. The text even says this 'does not add computational overhead during inference,' confirming that no projection is performed at test time. Therefore the claimed guarantee that L-SR1 'enforces' the secant relation is not implemented; at best, the network is regularized toward it. Whether the soft penalty suffices at test time is an empirical question, but the paper never reports the test-time secant residual ||p_k - B_k q_k|| for the analytic benchmarks or HMR. This matters because the method's name, the second contribution bullet, and the quasi-Newton grounding (Sec. 3.2) all rest on this mechanism; without it, L-SR1 is just a learned PSD preconditioner regularized toward secant, and the claimed 'projection-guided' improvement over the no-projection variant (Fig. 3) is attributed to a component that is never executed. The concern is internal to the paper: Eq. (8) describes a projection, Alg. 1 does not compute it, and Eq. (9) uses a penalty instead.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes L-SR1, a learned second-order optimizer that uses MLP-generated vectors to build a limited-memory preconditioner of the form B_k = I + sum_i v_i v_i^T, together with elementwise learned learning rates. The headline component, the \"Projection-Guided Secant Mechanism\" (PGSM), is described as a learned projection that enforces the secant condition and positive semidefiniteness. The method is evaluated on random quadratics, on performance profiles over benchmark functions, and on monocular Human Mesh Recovery (HMR), where Table 1 reports PA-MPJPE 51.58 for L-SR1 versus 55.90 for LGD on 3DPW. The paper claims state-of-the-art results among optimization-based HMR methods and strong generalization across problem dimensions without fine-tuning.","tokens_in":14979,"tokens_out":6627,"duration_ms":71966,"significance":"If fully supported, the paper would contribute a compact, dimension-invariant learned second-order preconditioner that can be dropped into optimization pipelines such as HMR. The empirical setup is broad, including analytic benchmarks, performance profiles, runtime and memory measurements, ablations, and a real-world integration. The PSD construction via outer products is simple and cheap, and the runtime comparison in Appendix A is useful. However, the two headline properties are not actually implemented as stated: the secant condition is only softly penalized in the meta-loss and is not enforced at inference, and the claimed self-supervision is contradicted by the ground-truth term in the HMR meta-loss. The HMR comparison also rests on a single unpaired number. As it stands, the evidence supports a weaker claim: a learned PSD preconditioner regularized toward the secant relation, which may still be useful but is not the \"learned projection\" advertised.","major_comments":[{"comment":"The central \"learned projection\" is never computed at inference. Eq. (8) defines a projection as an argmin over the structured family, but Eq. (9) replaces it by adding the soft penalty lambda_sec * ||p_k - B_k q_k||^2 to the meta-loss. Algorithm 1 (lines 4-11) uses the raw vector-generator output v_k = P(f_k) directly in the buffer and computes d_k = sum_{v in B_L} v v^T g_{k-1}; no projected vector is applied. Thus the claim that L-SR1 \"enforces\" the secant condition is not implemented. This is load-bearing because the method's name, the second contribution bullet, and the quasi-Newton grounding in Sec. 3.2 all rest on this mechanism. The authors should either apply an actual projection at inference or revise the claims and report the test-time secant residual ||p_k - B_k q_k|| to show that the soft penalty suffices.","section":"Sec. 4.2, Eq. (8)-(9); Algorithm 1"},{"comment":"The paper repeatedly describes the method as self-supervised and claims it works \"without the need for annotated data or supervised meta-training,\" but Eq. (13) contains the term lambda_self * ||Theta_k - Theta_gt||_1, where Theta_gt are ground-truth SMPL parameters from AMASS. This is supervised meta-training with annotated ground truth. The claim should be corrected, or the loss and training protocol must be changed so that no ground-truth parameters are used.","section":"Sec. 5.2, Eq. (13); Abstract; Contributions"},{"comment":"The HMR comparison is reported as a single number per method with no standard deviations, no number of seeds, and no significance testing. L-SR1's headline value of 51.58 is reached after 13 steps, while LGD's 55.90 is reached after 4; the bracketed value 51.74 after 4 steps is the relevant point comparison, but even that is one run. The statement that L-SR1 \"consistently outperforms\" LGD in 3D accuracy is therefore not statistically supported. Multiple seeds and an error bar or significance test are needed.","section":"Table 1; Sec. 5.2"},{"comment":"The performance-profile claim that L-SR1 achieves the highest profile is presented without repeated runs, confidence intervals, or sensitivity analysis. The profile depends on the tuned learning rates of the non-trainable baselines and on the buffer sizes and secant weights chosen in Table 6, so the claimed consistent effectiveness needs robustness evidence across these choices.","section":"Sec. 5.1.2; Fig. 3c; Table 6"}],"minor_comments":[{"comment":"The header contains a typo: \"PA-MPJEPE\" should be \"PA-MPJPE\".","section":"Table 1"},{"comment":"The symbol gamma is used for both the learning-rate generator exponent scale and the meta-learning-rate decay; rename one of them to avoid ambiguity.","section":"Appendix C.2.1; Appendix C.2.3"},{"comment":"Reference [29] is missing venue and year information and should be completed.","section":"References"},{"comment":"The symbol R_sec is used in Eq. (13) without being defined in that equation; please restate or point explicitly to Eq. (9) at first use.","section":"Eq. (13)"},{"comment":"The caption says \"with and without learned projection,\" but neither variant applies a projection at inference; the difference is the presence of the secant penalty in the meta-loss. Please rephrase to avoid implying that the projection is executed.","section":"Fig. 3 caption"},{"comment":"The paper states that code will be released upon acceptance, but currently provides no seeds or exact train/validation checkpoint selection details; these should be included to make the single-number HMR result reproducible.","section":"Reproducibility"}],"recommendation":"major_revision","confidential_remarks":"The self-supervision claim is misleading and should be handled prominently in the revision; there is no indication of citation impropriety. The paper is potentially salvageable if the strong projection and self-supervision claims are replaced by accurate descriptions and if the secant residual and multi-seed HMR experiments are added."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is genuinely useful and mostly well executed. L-SR1 takes the SR1 structure, replaces the fixed secant update with a learned vector generator that is elementwise and dimension-invariant, and adds a secant penalty to the meta-loss. That combination is new as far as I know, and the analytic experiments support its value: the version with the penalty converges faster and tracks the Newton direction more closely than the version without it, and the method generalizes across problem sizes without retraining. The HMR result (PA-MPJPE 51.58 vs LGD's 55.90) is also encouraging, with a smaller model and lower runtime. The paper is clearly written and gives enough implementation detail to reproduce. The soft spots are not fatal to the method, but they are load-bearing for the claims. Most importantly, the 'learned projection' described in Sec. 4.2 is never computed at inference. Algorithm 1 uses the raw v_k from the generator; the projection objective in Eq. (8) only appears as a soft penalty in the meta-loss (Eq. 9). So the claim that L-SR1 'enforces the secant condition' is unverified. The authors should either implement an actual projection at test time or, more honestly, reframe the mechanism as 'secant regularization' and report the test-time secant residual. The current framing overstates the quasi-Newton grounding. Second, the abstract's 'no reliance on annotated data' is directly contradicted by Eq. 13, which includes λ_self||Θ_k - Θ_gt||_1 using AMASS ground truth. That is annotated data. The 'self-supervised' language is misleading. Third, the HMR comparison is weak on statistics: a single number with no error bars or significance testing, and the closely related learned optimizer of Gärtner et al. [17] is omitted from the table. That omission is odd given the paper explicitly cites [17] as a quasi-Newton-inspired learned optimizer. These are the reasons the paper is not acceptable in its current form. But the underlying method is sound enough to deserve a serious referee. The empirical claims, if tightened, would be a modest but real contribution to learned optimization. I would send it to peer review, with a strong request to align the claims with the actual mechanism and to provide proper comparisons and uncertainty estimates. The authors have the right idea; they just need to stop overpromising.","headline":"A solid empirical learned-optimizer paper that overstates its 'learned projection'—the actual method is a secant-regularized SR1-style preconditioner, and the no-annotated-data claim is contradicted by Eq. 13.","tokens_in":731,"tokens_out":884,"would_cite":false,"duration_ms":28661,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["90C53","65K05","68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"L-SR1, a compact learned second-order optimizer, reports PA-MPJPE 51.58 on 3DPW by generating SR1-style preconditioners that are trained to satisfy the quasi-Newton secant relation.","keywords":["learned optimization","second-order optimization","quasi-Newton methods","symmetric rank-one","preconditioning","human mesh recovery","meta-learning"],"falsifier":"Run a trained L-SR1 checkpoint on held-out quadratics and record, for each inner iteration, the normalized secant residual $\\lVert p_k - \\tilde{B}_k q_k \\rVert_2 / \\lVert p_k \\rVert_2$. If that residual is no smaller than for a model trained with $\\lambda_{\\text{sec}} = 0$, the projection mechanism is not doing the claimed work and the performance gains come from the learned preconditioning alone.","tokens_in":14326,"feed_emoji":"🧍","tokens_out":11383,"duration_ms":99064,"temperature":0.7,"pith_summary":"L-SR1 is a learned second-order optimizer that extends the classical Symmetric Rank-One method: a small neural network produces vectors whose outer products precondition gradient updates, and a training-time penalty pushes those preconditioners toward the quasi-Newton secant relation. The paper's central claim is that this compact, self-supervised design can converge faster and to better solutions than both classical optimizers and prior learned optimizers on analytic benchmarks and on monocular human mesh recovery. On the 3DPW benchmark the paper reports PA-MPJPE 51.58 for L-SR1, versus 55.90 for the LGD baseline it replaces, with a smaller model and no task-specific fine-tuning. If true, this would make learned second-order preconditioning a practical drop-in component for optimization-based fitting pipelines, including in settings where dimension changes across problems.","feed_headline":"Rank-one learned optimizer beats prior optimization-based HMR","feed_subtitle":"On 3DPW it reaches PA-MPJPE 51.58, beating LGD's 55.90 with fewer parameters and no fine-tuning.","key_machinery":"The load-bearing object is the preconditioner $\\tilde{B}_k = B_0 + \\sum_{i=1}^L v_i v_i^{\\top}$, where $B_0 = I$ and each $v_i$ is generated by a shared multilayer perceptron called the Vector Generator from an encoded optimization state and kept in a fixed-size buffer. The outer-product form automatically makes $\\tilde{B}_k$ symmetric positive semi-definite, which turns every update into a descent direction. The secant condition $B_k q_k = p_k$, the quasi-Newton requirement that the approximate inverse Hessian map the change in gradient to the change in parameters, is converted into a projection objective and then into the training penalty $\\lambda_{\\text{sec}}\\lVert p_k - \\tilde{B}_k q_k \\rVert_2^2$ in the meta-loss. That penalty is the mechanism meant to make the learned preconditioner behave like an inverse-Hessian approximation at inference, even though no explicit projection step is run during optimization.","core_discovery":"The paper's central claim is that a learned preconditioner built from rank-one outer products can transfer the strengths of quasi-Newton methods into a trainable optimizer without sacrificing stability. At each iteration L-SR1 stores the last $L$ vectors produced by a learned vector generator and computes the descent direction as $d_k = \\sum_{v \\in \\mathcal{B}_L} v v^{\\top} g_{k-1}$; because the buffer stores only vectors, the effective preconditioner $\\tilde{B}_k = I + \\sum_i v_i v_i^{\\top}$ is always symmetric positive semi-definite, guaranteeing a descent direction. The quasi-Newton structure is imposed by a projection objective, minimizing $\\lVert p_k - \\tilde{B}_k q_k \\rVert_2^2$, which is implemented as a secant penalty in the meta-loss rather than as an exact projection at inference. The reported consequences are that L-SR1 obtains the highest performance profile among six solvers on a 30-problem analytic suite and, on 3DPW, lowers PA-MPJPE from 55.90 (LGD) to 51.58 with a 10.4M-parameter model that needs no annotated 3D data or fine-tuning.","pith_inferences":["Editorial inference: if the secant penalty is what carries the method, then sweeping $\\lambda_{\\text{sec}}$ on the HMR task should reproduce the accuracy gap that the paper shows with and without the projection on quadratics, but the paper does not report that sweep.","Editorial inference: the same preconditioner could be tested as a drop-in update rule for non-convex problems such as neural network training, where the paper's analytic evidence does not directly apply.","Editorial inference: the method's dimension invariance suggests one trained optimizer could be shared across a family of tasks of very different sizes, but the paper evaluates transfer from a single training dimension rather than multi-task training on mixed dimensions."],"forward_implications":["A single L-SR1 checkpoint, trained on quadratics and two benchmark functions in dimension 100, attains the top performance profile on a test suite with dimensions 50 to 1000, indicating the elementwise design transfers across problem sizes.","Replacing the LGD update module with L-SR1 in a human mesh recovery pipeline improves PA-MPJPE on 3DPW from 55.90 to 51.58, with a smaller model of 10.4M versus 17.4M parameters, so learned second-order preconditioning is competitive in a high-dimensional vision task.","Because all learnable modules operate elementwise and the vector buffer has a fixed size, memory use grows only linearly with dimension and per-iteration runtime is reported as lower than LGD's (91 ms versus 166 ms in the computational appendix).","The self-supervised training on a 2D reprojection loss plus the secant penalty means the optimizer does not need annotated 3D meshes or per-task fine-tuning, lowering the barrier to inserting it into other iterative fitting pipelines."],"supporting_citations":[{"why":"Provides the LGD HMR framework whose update module L-SR1 replaces, and supplies the training and evaluation protocol used on 3DPW.","marker":"[54]"},{"why":"Prior transformer-based learned optimizer that also uses outer-product positive semi-definite preconditioning; L-SR1 builds on and contrasts with its approach.","marker":"[17]"},{"why":"L-BFGS, the classical limited-memory quasi-Newton baseline used in the analytic and HMR comparisons.","marker":"[43]"},{"why":"Adam, the first-order optimizer baseline in the performance-profile and runtime comparisons.","marker":"[25]"},{"why":"AdaHessian, a second-order adaptive baseline included in the performance-profile experiments.","marker":"[66]"},{"why":"The classical SR1 method whose rank-one update structure L-SR1 extends with learned vectors.","marker":"[15]"},{"why":"The SMPL body model used to reconstruct meshes in the HMR evaluation.","marker":"[32]"},{"why":"The AMASS dataset used for self-supervised meta-training of the HMR pipeline.","marker":"[36]"}],"fun_headline_variants":["L-SR1: learned rank-one optimizer beats LGD on 3DPW","Learned SR1 preconditioner: no fine-tuning, faster HMR","Rank-one learned preconditioner accelerates optimization without annotations","L-SR1: quasi-Newton memory for stable, faster learned optimization"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the training-time penalty that pushes the learned preconditioner toward the quasi-Newton secant relation, the requirement that the approximate inverse Hessian map the change in gradient to the change in parameters, is strong enough to make that relation hold on test problems, where no exact projection is applied.","fun_headline_variants_meta":{"raw":{"variants":["L-SR1: learned rank-one optimizer beats LGD on 3DPW","Learned SR1 preconditioner: no fine-tuning, faster HMR","Rank-one learned preconditioner accelerates optimization without annotations","L-SR1: quasi-Newton memory for stable, faster learned optimization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001185,"raw_usage":{"total_tokens":4934,"prompt_tokens":1029,"completion_tokens":3905,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":645,"completion_tokens_details":{"reasoning_tokens":3827}},"tokens_in":645,"tokens_out":3905,"duration_ms":30303,"temperature":1.0,"reasoning_tokens":3827,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:24:47.707041+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a trained L-SR1 checkpoint on held-out quadratics and record, for each inner iteration, the normalized secant residual $\\lVert p_k - \\tilde{B}_k q_k \\rVert_2 / \\lVert p_k \\rVert_2$. If that residual is no smaller than for a model trained with $\\lambda_{\\text{sec}} = 0$, the projection mechanism is not doing the claimed work and the performance gains come from the learned preconditioning alone.","supporting_citations":[{"cited_title":"Human body model fitting by learned gradient descent","cited_arxiv_id":null,"evidence_quote":"Provides the LGD HMR framework whose update module L-SR1 replaces, and supplies the training and evaluation protocol used on 3DPW."},{"cited_title":"Transformer- based learned optimization","cited_arxiv_id":null,"evidence_quote":"Prior transformer-based learned optimizer that also uses outer-product positive semi-definite preconditioning; L-SR1 builds on and contrasts with its approach."},{"cited_title":"Updating quasi-newton matrices with limited storage","cited_arxiv_id":null,"evidence_quote":"L-BFGS, the classical limited-memory quasi-Newton baseline used in the analytic and HMR comparisons."},{"cited_title":"secant constraint","cited_arxiv_id":null,"evidence_quote":"AdaHessian, a second-order adaptive baseline included in the performance-profile experiments."},{"cited_title":"Convergence of quasi-newton matrices generated by the symmetric rank one update","cited_arxiv_id":null,"evidence_quote":"The classical SR1 method whose rank-one update structure L-SR1 extends with learned vectors."},{"cited_title":"Troje, Gerard Pons-Moll, and Michael J","cited_arxiv_id":null,"evidence_quote":"The AMASS dataset used for self-supervised meta-training of the HMR pipeline."}],"review_version":1}