Pith. sign in

REVIEW 3 major objections 5 minor 19 references

This paper claims that a multi-agent search over a typed optimizer DSL discovered RS-MR, a reduced-state matrix optimizer that improves transformer pretraining over Muon and other strong baselines under the authors' native evaluation protoc

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 12:25 UTC pith:MGIHMTZU

load-bearing objection Worth a look for the typed-DSL search framework, but the single-run evaluation doesn't support the claim that RS-MR beats Muon. the 3 major comments →

arxiv 2607.20486 v1 pith:MGIHMTZU submitted 2026-06-02 cs.AI

OPTScientist: Multi-Agent Discovery of Typed Optimizer Programs for Transformer Pretraining

Zhongzheng Li , Tiancan Feng , Wenhao Li , Qingsong Ran , Shikun Feng , Xiaoyuan Zhang , Yue Wang , Xiaoguang Zhao This is my paper
classification cs.AI
keywords optimizer discoverytyped DSLmulti-agent LLM searchevolutionary searchtransformer pretrainingmatrix optimizerRS-MRcompiler validation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

OPTScientist is a framework that turns optimizer design into a constrained scientific search: instead of generating free-form Python code, four LLM agents propose hypotheses, write candidates in a typed domain-specific language, compile and evaluate them, and critique results. The paper's central claim is that this process discovered RS-MR, a matrix optimizer that improves transformer pretraining over strong baselines under the authors' native evaluation protocol. On a 21-layer, 1.1B-parameter transformer trained for 3000 steps, RS-MR reaches 0.798375 validation bits-per-byte, beating Muon by about 0.57% while adding only 7.45 MB of optimizer-state memory. The authors state that broader validation on larger models, longer horizons, more datasets, and other architectures is still needed, and that RS-MR is a matrix-parameter optimizer embedded in a hybrid stack rather than a full-model replacement for AdamW.

Core claim

The paper's core discovery is RS-MR (Reduced-State MAGMA-RowNorm), a matrix-parameter optimizer found by the OPTScientist search loop. RS-MR keeps only two optimizer states—a momentum matrix and a compact blockwise gate—and forms its update by row-normalizing the momentum matrix, taking a polar direction, damping the magnitude with a lightweight RMS-based factor, and gating blocks by their agreement with the current gradient. In the paper's native 21-layer transformer pretraining benchmark, RS-MR achieves the best final validation BPB among sixteen optimizers (0.798375), improving on Muon by 0.004574 BPB (about 0.57%) with only 0.39% additional optimizer-state memory. The authors emphasize t

What carries the argument

The central object is a typed optimizer DSL that expresses updates via direction, scaling, preconditioning/geometry, regularization, state, and grouping modules, plus a compiler that type-checks and lowers programs before training. The search loop has four role agents (Theorist, Designer, Engineer, Reviewer) and a two-stage evolution that extends the DSL with safe macro operators when repeated failures signal a representational bottleneck. The mechanism that carries the argument is RS-MR's update rule: normalize-then-orthogonalize (row-norm of momentum before polar), lightweight RMS damping, and soft blockwise gating that combines a slow history gate with a fast alignment gate.

Load-bearing premise

The central empirical claim rests on a single training run per optimizer, with no error bars; if the 0.0046 BPB gap between RS-MR and Muon is within run-to-run noise, the superiority claim collapses.

What would settle it

Run the native 21-layer benchmark with RS-MR and Muon for at least five independent seeds and compare final validation BPB distributions; the claim is falsified if the RS-MR distribution overlaps Muon's within one standard deviation or if RS-MR's best result does not consistently beat Muon's mean.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • RS-MR outperforms Muon at every validation checkpoint from 250 to 3000 steps, so the reported improvement is not a final-step artifact.
  • The improvement over Muon costs only 7.45 MB of additional optimizer state, keeping RS-MR in the favorable Pareto region on both validation BPB and memory.
  • All ten top-ranked DSL-discovered optimizers beat Muon, suggesting the typed search space contains multiple practical alternatives to classical baselines.
  • Because RS-MR is a matrix-parameter optimizer in a hybrid stack, the practical claim is about structured matrix updates, not a full-model replacement for AdamW.
  • Compiler-backed validation makes discovered optimizers inspectable and reproducible as compact typed programs, reducing wasted training budget.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: If the 0.57% gain survives repeated-seed evaluation, the mechanism worth isolating is the row-normalization-before-polarization step; an ablation that removes row_norm from RS-MR would test whether that is the active ingredient.
  • Editorial inference: Because RS-MR was selected during search on proxy versions of the same architecture and data, the reported native validation is not fully independent; a stronger transfer test would freeze the search and then evaluate the discovered optimizer on a held-out architecture or data split.
  • Editorial inference: The same typed-DSL loop could be redirected to other auditable program-synthesis settings, such as normalization layers, learning-rate schedules, or data-augmentation pipelines, where compiler-backed validity and long-horizon evaluation matter.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces OPTScientist, a multi-agent framework that searches for optimizer programs in a typed domain-specific language (DSL) with compiler-backed validation. The discovered optimizer RS-MR is a reduced-state matrix optimizer combining row-normalized polar directions, RMS damping, and blockwise gating. On a 3000-step, 1.1B-parameter nanochat benchmark on FineWeb-Edu, RS-MR achieves final validation BPB 0.798375, beating Muon (0.802949) by 0.004574 BPB (about 0.57%) with a 7.45 MB optimizer-state memory overhead. The paper claims this demonstrates that typed, compiler-validated optimizer discovery can find practical optimizers that improve transformer pretraining.

Significance. The framework is well-motivated, and the DSL/compiler design is a genuine strength: it addresses the expressivity/reliability tradeoff in optimizer search, produces auditable typed programs, and the discovered RS-MR is compact and interpretable. A memory-efficient matrix optimizer that consistently outperforms Muon would be a useful result. However, the empirical evidence as presented is insufficient to establish the central claim because the reported improvement rests on single runs and the final validation set is used for selection among discovered candidates. The significance of the paper therefore depends on whether the authors can provide statistically grounded and selection-bias-free evidence in a revision.

major comments (3)
  1. [Section 6.3, Table 1] The central empirical claim — that RS-MR improves over Muon — rests on a single run per optimizer with no seeds, confidence intervals, or repeated runs. The reported gap is 0.004574 BPB (~0.57%). The appendix (Table 2) does not fill this gap; the near-identical BPB values among OE-1..OE-5 on the same fixed benchmark are not an estimate of run-to-run variance. The statement that 'all optimizer variants are trained on the same data prefix' makes the comparison controlled but does not by itself rule out variance from model initialization or data shuffling. If the pipeline is fully deterministic, the paper should say so and argue why a single deterministic comparison is sufficient; otherwise, multiple seeds with error bars are required. The paper's own Limitations (Sec. 7) acknowledge the need for broader validation, but this does not resolve the statistical support for the specific claim.
  2. [Section 4.5, Section 6.2, Table 2] RS-MR is not an independently validated winner; it is OE-1, the best of the 10 OpenEvolve candidates evaluated on the exact native d21 benchmark used for 'final validation.' The promoted performance is thus a maximum selected from the evaluation set, not an unbiased estimate of a typical or generalizable candidate. The multi-scale search stages (Appendix C.6) use the same architecture family and FineWeb-Edu corpus, so selection pressure is strongly aligned with the final benchmark. To support the improvement claim, the paper needs a validation protocol in which the final comparison is not used for selection, or a correction for selection bias (e.g., nested validation, multiple independent searches, or reporting all candidates together with repeated-seed error bars).
  3. [Section 5, Algorithm 2] The RS-MR update is not fully specified. Algorithm 2 uses clip_gamma without giving a value, block_cossim without specifying the block size (the appendix mentions a 16x16 MAGMA mask, but this is not in the algorithm), and the relationship between the hyperparameter tau and the fast-gate temperature 0.15 is unclear. Since RS-MR is claimed as a discovered artifact, the exact program and all numerical constants should be supplied; otherwise the result cannot be reproduced or audited. The DSL snippet in Appendix B.1 contains only beta1 and tau, omitting the other constants used in Algorithm 2.
minor comments (5)
  1. [Abstract / Section 6.1] Bits-per-byte (BPB) is used without definition at first occurrence. Define it in the abstract or in Section 6.1 where the metric is introduced.
  2. [Section 6.2] The text says 'All 16 optimizer variants in the native d21 comparison', but Table 1 lists only 7 optimizers. The reader must go to the appendix Table 2 for the full list; refer to it explicitly in the main text.
  3. [Figure 2] The caption mentions 'First-500-step training loss' but the figure combines training and validation dynamics. Clarify the smoothing, the evaluation schedule, and what is shown in each panel.
  4. [Section 5, Algorithm 2] The use of tau in the slow gate (line 2) and the constant 0.15 in the fast gate (line 6) is confusing. If tau is 0.15, say so explicitly; if the fast gate uses a different fixed temperature, name it as a separate hyperparameter.
  5. [Section 4.4] The evaluator is described as 'deterministic', but no seed or determinism mechanism is reported. If deterministic training is intended, specify the seed and the fixed data order; otherwise 'deterministic' is misleading.

Circularity Check

1 steps flagged

RS-MR's improvement over Muon is reported on the very native d21 benchmark used to select it, so the central empirical claim is a selection result rather than an independent prediction.

specific steps
  1. fitted input called prediction [Algorithm 1 (line 17); Section 4.5 'Final Validation'; Section 6.2-6.3; Appendix Table 2 caption]
    "return best final-validation candidate in H ... The optimizer promoted as the main artifact is selected by native long-horizon validation rather than by proxy score alone. ... All 16 optimizer variants on the native d21 NANOCHAT3000-step benchmark, ranked by final validation BPB. RS-MR is the same DSL program as OE-1."

    The d21 validation BPB is both the selection objective and the reported evidence. Algorithm 1 returns the best final-validation candidate; Sec. 4.5 says the artifact is selected by native long-horizon validation; Table 2 ranks all OE candidates on that same d21 benchmark and identifies RS-MR as OE-1, the top entry. Thus 'RS-MR improves transformer pretraining over strong baselines under our native evaluation protocol' restates the selection outcome rather than an independent prediction. No held-out benchmark, separate data split, or repeated-seed estimate is provided; the paper's Limitations section itself concedes broader validation is needed. The claim is therefore partially forced by the evaluation-design choice: any candidate promoted as best on this protocol would trivially be expecte

full rationale

The paper's own algorithm exposes the circularity: the final artifact is chosen by the same native long-horizon validation whose results are then presented as evidence of improvement. Algorithm 1 returns 'best final-validation candidate in H'; Section 4.5 states the promoted optimizer is 'selected by native long-horizon validation rather than by proxy score alone'; and Appendix Table 2 ranks the OE candidates on exactly the native d21 benchmark, with RS-MR equal to OE-1 at the top. The abstract's 'improves transformer pretraining over strong baselines under our native evaluation protocol' is therefore not an out-of-sample test of RS-MR, but a description of the selection outcome. The paper does not report a separate external benchmark, a different data split, or repeated runs with error bars, so the reported 0.004574 BPB advantage over Muon cannot be separated from selection bias. This is partial circularity: the empirical prediction reduces in large part to the criterion used to choose the optimizer. The rest of the contribution—the typed DSL, the compiler-backed validation, and the multi-agent search loop—does not itself depend on the questionable empirical claim, so the score is moderate rather than maximal.

Axiom & Free-Parameter Ledger

6 free parameters · 4 axioms · 0 invented entities

The central claim draws on several fitted hyperparameters and unproven effectiveness assumptions. Most importantly, the search process itself tunes the optimizer on proxy tasks of the same benchmark family; this is the main circularity burden.

free parameters (6)
  • beta1 (momentum decay) = 0.95
    RS-MR Algorithm 2 line 1; chosen by evolutionary search, not derived.
  • slow gate smoothing (0.9, 0.1) = 0.9 / 0.1
    EMA in st update; part of the DSL hyperparameter set tuned during search.
  • RMS damping scale (0.3) and normalizer (0.1) = 0.3, 0.1
    αt formula in Algorithm 2; fitted constants.
  • fast gate temperature 0.15 = 0.15
    a_t gate in Algorithm 2; selected via search.
  • slow gate temperature tau = 0.15
    st gate in Algorithm 2; DSL hyperparameter.
  • block size for blockwise state = 16x16 (per appendix)
    MAGMA-style block masking granularity; not justified independently.
axioms (4)
  • domain assumption Muon-style polar momentum directions are a strong update geometry for transformer weight matrices.
    RS-MR inherits Muon's polar update; paper provides no proof of optimality, only prior art.
  • domain assumption Blockwise cosine similarity between momentum and gradient is a reliable indicator of block relevance; gating on it improves updates.
    Core gating mechanism in Algorithm 2; no empirical or theoretical justification beyond final performance.
  • domain assumption Short-horizon multi-scale proxy evaluation (49.8M–514.3M params) transfers to the 1.1B-parameter d21 target.
    Appendix C.6; search-time target-stage promotion assumes transfer.
  • domain assumption A 20M-token validation slice after 3.15B training tokens provides a stable, meaningful ranking of optimizers.
    Section 6.1; no error bar or stability analysis.

pith-pipeline@v1.3.0-alltime-deepseek · 13767 in / 14013 out tokens · 117593 ms · 2026-08-02T12:25:32.159498+00:00 · methodology

0 comments
read the original abstract

Designing optimizers for modern deep learning remains a challenging scientific problem, requiring the joint consideration of optimization geometry, state dynamics, numerical stability, implementation constraints, and empirical generalization. Existing automated optimizer discovery methods typically search either over unconstrained code spaces or within narrowly parameterized optimizer families. The former is flexible but often produces invalid or uninterpretable programs, while the latter is stable but limits novelty. We introduce OPTScientist, a theory-guided multi-agent framework for optimizer discovery in a typed domain-specific language (DSL). OPTScientist formulates optimizer design as a constrained scientific search process, where candidate updates are expressed through direction, scaling, preconditioning, regularization, state, and grouping modules. Four role agents, Theorist, Designer, Engineer, and Reviewer, collaborate within a single orchestration loop to propose hypotheses, synthesize DSL candidates, compile and evaluate optimizers, and critique results. To overcome the limitations of a fixed search space, OPTScientist combines evolutionary search over optimizer programs with a second-stage mechanism that proposes small DSL extensions when repeated failures reveal representational bottlenecks. Using this framework, we discover RS-MR, a reduced-state matrix optimizer that improves transformer pretraining over strong baselines under our native evaluation protocol. Our results suggest a path toward automated optimizer science grounded in theory, typed programs, compiler validation, and closed-loop experimentation.

Figures

Figures reproduced from arXiv: 2607.20486 by Qingsong Ran, Shikun Feng, Tiancan Feng, Wenhao Li, Xiaoguang Zhao, Xiaoyuan Zhang, Yue Wang, Zhongzheng Li.

Figure 1
Figure 1. Figure 1: Overview of OPTSCIENTIST. The framework discovers optimizer programs through a theory-guided multi-agent loop. A THEORIST proposes optimizer hypotheses, a DESIGNER instantiates them as typed DSL programs, an ENGINEER compiles and evaluates candidates through transformer pretraining, and a REVIEWER scores candidates and updates shared memory. The system alternates between fixed-DSL optimizer evolution and c… view at source ↗
Figure 2
Figure 2. Figure 2: Native d21 NANOCHAT training dynamics. RS-MR maintains a consistent advantage over Muon throughout the 3000-step validation trajectory and improves early training loss. performance differences primarily reflect optimizer behavior rather than changes in data or training configuration. RS-MR is first discovered through the OPTSCIENTIST search process and then re-evaluated in the native d21 benchmark. This fi… view at source ↗
Figure 3
Figure 3. Figure 3: Top-16 optimizer comparison on the native d21 benchmark. RS-MR achieves the best [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

19 extracted references · 15 linked inside Pith

  1. [5]

    Muon is scalable for llm training.arXiv preprint arXiv:2502.16982, 2025a

    Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, et al. Muon is scalable for llm training.arXiv preprint arXiv:2502.16982, 2025a. Liming Liu, Zhenghao Xu, Zixuan Zhang, Hao Kang, Zichong Li, Chen Liang, Weizhu Chen, and Tuo Zhao. Cosmos: A hybrid adaptive optimizer for memory-efficien...

    Pith/arXiv arXiv

  2. [7]

    Evolving deep learning optimizers.arXiv preprint arXiv:2512.11853,

    Mitchell Marfinetz. Evolving deep learning optimizers.arXiv preprint arXiv:2512.11853,

    arXiv

  3. [8]

    Celo: Training versatile learned optimizers on a compute diet.arXiv preprint arXiv:2501.12670,

    10 Abhinav Moudgil, Boris Knyazev, Guillaume Lajoie, and Eugene Belilovsky. Celo: Training versatile learned optimizers on a compute diet.arXiv preprint arXiv:2501.12670,

    Pith/arXiv arXiv

  4. [9]

    Alphae- volve: A coding agent for scientific and algorithmic discovery.arXiv preprint arXiv:2506.13131,

    Alexander Novikov, Ngân V˜u, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wag- ner, Sergey Shirobokov, Borislav Kozlovskii, Francisco JR Ruiz, Abbas Mehrabian, et al. Alphae- volve: A coding agent for scientific and algorithmic discovery.arXiv preprint arXiv:2506.13131,

    Pith/arXiv arXiv

  5. [10]

    The ademamix optimizer: Better, faster, older

    Matteo Pagliardini, Pierre Ablin, and David Grangier. The ademamix optimizer: Better, faster, older. arXiv preprint arXiv:2409.03137,

    Pith/arXiv arXiv

  6. [11]

    Agent laboratory: Using llm agents as research assistants.Findings of the Association for Computational Linguistics: EMNLP 2025, pages 5977–6043,

    Samuel Schmidgall, Yusheng Su, Ze Wang, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Michael Moor, Zicheng Liu, and Emad Barsoum. Agent laboratory: Using llm agents as research assistants.Findings of the Association for Computational Linguistics: EMNLP 2025, pages 5977–6043,

    2025

  7. [12]

    Soap: Improving and stabilizing shampoo using adam.arXiv preprint arXiv:2409.11321,

    Nikhil Vyas, Depen Morwani, Rosie Zhao, Mujin Kwun, Itai Shapira, David Brandfonbrener, Lucas Janson, and Sham Kakade. Soap: Improving and stabilizing shampoo using adam.arXiv preprint arXiv:2409.11321,

    Pith/arXiv arXiv

  8. [13]

    V oyager: An open-ended embodied agent with large language models.arXiv preprint arXiv:2305.16291,

    Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. V oyager: An open-ended embodied agent with large language models.arXiv preprint arXiv:2305.16291,

    Pith/arXiv arXiv

  9. [14]

    Fantastic pretraining optimizers and where to find them.arXiv preprint arXiv:2509.02046,

    Kaiyue Wen, David Hall, Tengyu Ma, and Percy Liang. Fantastic pretraining optimizers and where to find them.arXiv preprint arXiv:2509.02046,

    Pith/arXiv arXiv

  10. [15]

    React: Synergizing reasoning and acting in language models.arXiv preprint arXiv:2210.03629,

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models.arXiv preprint arXiv:2210.03629,

    Pith/arXiv arXiv

  11. [16]

    Mars: Unleashing the power of variance reduction for training large models.arXiv preprint arXiv:2411.10438,

    11 Huizhuo Yuan, Yifeng Liu, Shuang Wu, Xun Zhou, and Quanquan Gu. Mars: Unleashing the power of variance reduction for training large models.arXiv preprint arXiv:2411.10438,

    Pith/arXiv arXiv

  12. [17]

    Adam-mini: Use fewer learning rates to gain more.arXiv preprint arXiv:2406.16793,

    Yushun Zhang, Congliang Chen, Ziniu Li, Tian Ding, Chenwei Wu, Diederik P Kingma, Yinyu Ye, Zhi-Quan Luo, and Ruoyu Sun. Adam-mini: Use fewer learning rates to gain more.arXiv preprint arXiv:2406.16793,

    Pith/arXiv arXiv

  13. [18]

    Galore: Memory-efficient llm training by gradient low-rank projection.arXiv preprint arXiv:2403.03507,

    Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient llm training by gradient low-rank projection.arXiv preprint arXiv:2403.03507,

    Pith/arXiv arXiv

  14. [19]

    The main paper only introduces the high-level idea: optimizers are represented as modular update programs

    12 OPTScientist: Multi-Agent Discovery of Typed Optimizer Programs for Transformer Pretraining (Appendix) A Unified Optimizer Design Space This appendix gives a more detailed description of the optimizer design space used to motivate our DSL. The main paper only introduces the high-level idea: optimizers are represented as modular update programs. Here we...

    1913

  15. [2016]

    Researchagent: Iterative research idea generation over scientific literature with large language models

    Jinheon Baek, Sujay Kumar Jauhar, Silviu Cucerzan, and Sung Ju Hwang. Researchagent: Iterative research idea generation over scientific literature with large language models. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (V olume 1: Long Papers), pa...

    2025

  16. [2019]

    The ai scientist: Towards fully automated open-ended scientific discovery.arXiv preprint arXiv:2408.06292,

    Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist: Towards fully automated open-ended scientific discovery.arXiv preprint arXiv:2408.06292,

    Pith/arXiv arXiv

  17. [2023]

    Evolution of heuristics: Towards efficient automatic algorithm design using large language model.arXiv preprint arXiv:2401.02051,

    Fei Liu, Xialiang Tong, Mingxuan Yuan, Xi Lin, Fu Luo, Zhenkun Wang, Zhichao Lu, and Qingfu Zhang. Evolution of heuristics: Towards efficient automatic algorithm design using large language model.arXiv preprint arXiv:2401.02051,

    Pith/arXiv arXiv

  18. [2024]

    Sophia: A scalable stochastic second-order optimizer for language model pre-training.arXiv preprint arXiv:2305.14342,

    Hong Liu, Zhiyuan Li, David Hall, Percy Liang, and Tengyu Ma. Sophia: A scalable stochastic second-order optimizer for language model pre-training.arXiv preprint arXiv:2305.14342,

    Pith/arXiv arXiv

  19. [2025]

    com/karpathy/nanochat

    URL https://github. com/karpathy/nanochat. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980,

    Pith/arXiv arXiv