{"id":"a23412d7-76bd-45d2-90f3-93fa6bce9dbd","arxiv_id":"2607.20486","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"A multi-agent typed-DSL search discovers RS-MR, a Muon-like matrix optimizer that beats Muon by 0.57% validation BPB on one 3.15B-token transformer pretraining run, without error bars.","lead":"This paper introduces OPTScientist, a multi-agent LLM framework that searches over optimizer programs written in a typed domain-specific language, and reports that the discovered RS-MR optimizer improves validation bits-per-byte over Muon on a 1.1B-parameter transformer pretraining benchmark. The finding matters as a demonstration of automated optimizer discovery, but the reported gain is small, based on a single run per variant, and no code is released.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed 0.004574 BPB advantage of RS-MR over Muon rests on a single run per optimizer with no error bars; RS-MR was also selected as the best of 10 candidates evaluated on the same native benchmark, so the improvement is not statistically established.","rationale":"I read the paper in good faith and the central claim is that RS-MR improves transformer pretraining over strong baselines. The evidence for this is Table 1, which shows a single BPB value per optimizer. The reader's weakest assumption is exactly that this single-run gap exceeds run-to-run variance. I agree that this is the load-bearing point. The appendix Table 2 shows the top five discovered optimizers differ by only 1.5e-5 BPB, so the metric is highly reproducible across similar optimizers; but that does not quantify seed variance. Without error bars, the 0.004574 gap to Muon could be within noise, especially given the search process evaluated many candidates and selected the best one on the same native benchmark. The paper's own limitations concede the evidence is narrow. I do not see an internal inconsistency in the DSL or framework, but the empirical claim is not yet supported. Therefore, I concur with the REJECT verdict, but I would note that the framework could become acceptable with multi-seed evaluation and a truly held-out selection procedure. Since the reader already rejected, I keep the verdict unchanged.","tokens_in":14032,"tokens_out":6102,"duration_ms":62003,"concrete_test":"Run the native d21 3000-step benchmark with at least 5 independent random seeds (varying data order and initialization) for RS-MR, Muon, and AdamW under the identical hyperparameters and data prefix. Report the mean and standard deviation of final validation BPB, and compute the difference between RS-MR and Muon with a 95% confidence interval. If the confidence interval includes zero or the mean gap is less than two standard errors, the claimed improvement is not established. To address selection bias, pre-register RS-MR as the only candidate for this confirmatory evaluation, or evaluate on a data split/model configuration never used during the search.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that RS-MR improves transformer pretraining over strong baselines—is supported by Table 1, which reports a single final validation BPB per optimizer (RS-MR: 0.798375, Muon: 0.802949). No seeds, confidence intervals, or repeated runs are reported. Appendix Table 2 compounds the issue: the top five discovered optimizers (OE-1..OE-5) span only 1.5e-5 BPB, indicating the metric is extremely stable across similar optimizers, yet the claimed advantage over Muon is 0.004574 BPB. Without an estimate of run-to-run variance (different data shuffles, model initializations), we cannot determine whether this gap is signal or noise. Furthermore, RS-MR is the best of 10 OpenEvolve candidates all evaluated on the same native d21 benchmark (Table 2). This means the reported performance is a selected maximum, not an unbiased estimate of a typical candidate. The paper's own Limitations section acknowledges the evidence is narrow and calls for broader validation. Consequently, the abstract's claim of 'improves transformer pretraining' is not yet supported by the presented data.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces OPTScientist, a multi-agent framework that searches for optimizer programs in a typed domain-specific language (DSL) with compiler-backed validation. The discovered optimizer RS-MR is a reduced-state matrix optimizer combining row-normalized polar directions, RMS damping, and blockwise gating. On a 3000-step, 1.1B-parameter nanochat benchmark on FineWeb-Edu, RS-MR achieves final validation BPB 0.798375, beating Muon (0.802949) by 0.004574 BPB (about 0.57%) with a 7.45 MB optimizer-state memory overhead. The paper claims this demonstrates that typed, compiler-validated optimizer discovery can find practical optimizers that improve transformer pretraining.","tokens_in":14405,"tokens_out":6118,"duration_ms":62640,"significance":"The framework is well-motivated, and the DSL/compiler design is a genuine strength: it addresses the expressivity/reliability tradeoff in optimizer search, produces auditable typed programs, and the discovered RS-MR is compact and interpretable. A memory-efficient matrix optimizer that consistently outperforms Muon would be a useful result. However, the empirical evidence as presented is insufficient to establish the central claim because the reported improvement rests on single runs and the final validation set is used for selection among discovered candidates. The significance of the paper therefore depends on whether the authors can provide statistically grounded and selection-bias-free evidence in a revision.","major_comments":[{"comment":"The central empirical claim — that RS-MR improves over Muon — rests on a single run per optimizer with no seeds, confidence intervals, or repeated runs. The reported gap is 0.004574 BPB (~0.57%). The appendix (Table 2) does not fill this gap; the near-identical BPB values among OE-1..OE-5 on the same fixed benchmark are not an estimate of run-to-run variance. The statement that 'all optimizer variants are trained on the same data prefix' makes the comparison controlled but does not by itself rule out variance from model initialization or data shuffling. If the pipeline is fully deterministic, the paper should say so and argue why a single deterministic comparison is sufficient; otherwise, multiple seeds with error bars are required. The paper's own Limitations (Sec. 7) acknowledge the need for broader validation, but this does not resolve the statistical support for the specific claim.","section":"Section 6.3, Table 1"},{"comment":"RS-MR is not an independently validated winner; it is OE-1, the best of the 10 OpenEvolve candidates evaluated on the exact native d21 benchmark used for 'final validation.' The promoted performance is thus a maximum selected from the evaluation set, not an unbiased estimate of a typical or generalizable candidate. The multi-scale search stages (Appendix C.6) use the same architecture family and FineWeb-Edu corpus, so selection pressure is strongly aligned with the final benchmark. To support the improvement claim, the paper needs a validation protocol in which the final comparison is not used for selection, or a correction for selection bias (e.g., nested validation, multiple independent searches, or reporting all candidates together with repeated-seed error bars).","section":"Section 4.5, Section 6.2, Table 2"},{"comment":"The RS-MR update is not fully specified. Algorithm 2 uses clip_gamma without giving a value, block_cossim without specifying the block size (the appendix mentions a 16x16 MAGMA mask, but this is not in the algorithm), and the relationship between the hyperparameter tau and the fast-gate temperature 0.15 is unclear. Since RS-MR is claimed as a discovered artifact, the exact program and all numerical constants should be supplied; otherwise the result cannot be reproduced or audited. The DSL snippet in Appendix B.1 contains only beta1 and tau, omitting the other constants used in Algorithm 2.","section":"Section 5, Algorithm 2"}],"minor_comments":[{"comment":"Bits-per-byte (BPB) is used without definition at first occurrence. Define it in the abstract or in Section 6.1 where the metric is introduced.","section":"Abstract / Section 6.1"},{"comment":"The text says 'All 16 optimizer variants in the native d21 comparison', but Table 1 lists only 7 optimizers. The reader must go to the appendix Table 2 for the full list; refer to it explicitly in the main text.","section":"Section 6.2"},{"comment":"The caption mentions 'First-500-step training loss' but the figure combines training and validation dynamics. Clarify the smoothing, the evaluation schedule, and what is shown in each panel.","section":"Figure 2"},{"comment":"The use of tau in the slow gate (line 2) and the constant 0.15 in the fast gate (line 6) is confusing. If tau is 0.15, say so explicitly; if the fast gate uses a different fixed temperature, name it as a separate hyperparameter.","section":"Section 5, Algorithm 2"},{"comment":"The evaluator is described as 'deterministic', but no seed or determinism mechanism is reported. If deterministic training is intended, specify the seed and the fixed data order; otherwise 'deterministic' is misleading.","section":"Section 4.4"}],"recommendation":"major_revision","confidential_remarks":"The reader's reject recommendation is understandable, but I would not reject outright because the core issues — missing error bars and selection on the final benchmark — are fixable with additional runs and a cleaner validation protocol. However, the cost of such experiments is substantial, and if the authors cannot provide them, the empirical contribution of the paper will remain unsupported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe punchline: the DSL-based multi-agent search framework is a thoughtful piece of engineering, but the paper's central empirical claim—that RS-MR improves transformer pretraining over Muon—rests on one 3000-step run per optimizer, with no error bars or repeated seeds. That gap of 0.004574 BPB (~0.57%) is too small to take seriously without variance estimates.\n\nWhat's genuinely new: the typed DSL with compiler-backed validation, the four-role agent loop, and two-stage evolution. These are sensible combinations of known ideas (FunSearch-style program search, AutoML-Zero, Muon), but putting them together in a constrained, auditable optimizer DSL is a reasonable contribution. The paper is also honest: the limitations section admits the evidence is narrow and the theory isn't understood. That's refreshing.\n\nThe soft spots are concentrated in the evaluation. Table 1 gives a single run for each optimizer. The appendix shows the top five discovered optimizers separated by only 1.5e-5 BPB, which suggests the metric is either very stable across similar optimizers or near its noise floor—either way, without multiple runs you can't tell if the 0.004574 gap is signal or seed noise. On top of that, RS-MR is the best of ten candidates selected on the same native benchmark (Table 2), so the reported number is a selected maximum, not an unbiased estimate. The paper tries to distinguish proxy search from final validation, but the final validation is on the same architecture and data family used during search, so it's not an independent test.\n\nThat said, none of this is hidden. The authors explicitly flag the need for broader validation. The framework itself is plausible and the DSL is a nice way to keep search interpretable. What's missing is the evidence to back the headline claim.\n\nFor peer review: I'd send it out, but with a clear expectation that the empirical section needs major work—multi-seed runs, confidence intervals, and either an independent holdout or a more careful selection-bias discussion. The framework is worth engaging with; the result isn't yet.","headline":"Worth a look for the typed-DSL search framework, but the single-run evaluation doesn't support the claim that RS-MR beats Muon.","tokens_in":14869,"tokens_out":2246,"would_cite":false,"duration_ms":23272,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a multi-agent search over a typed optimizer DSL discovered RS-MR, a reduced-state matrix optimizer that improves transformer pretraining over Muon and other strong baselines under the authors' native evaluation protoc","keywords":["optimizer discovery","typed DSL","multi-agent LLM search","evolutionary search","transformer pretraining","matrix optimizer","RS-MR","compiler validation"],"falsifier":"Run the native 21-layer benchmark with RS-MR and Muon for at least five independent seeds and compare final validation BPB distributions; the claim is falsified if the RS-MR distribution overlaps Muon's within one standard deviation or if RS-MR's best result does not consistently beat Muon's mean.","tokens_in":13960,"feed_emoji":"🔬","tokens_out":5606,"duration_ms":54822,"temperature":0.7,"pith_summary":"OPTScientist is a framework that turns optimizer design into a constrained scientific search: instead of generating free-form Python code, four LLM agents propose hypotheses, write candidates in a typed domain-specific language, compile and evaluate them, and critique results. The paper's central claim is that this process discovered RS-MR, a matrix optimizer that improves transformer pretraining over strong baselines under the authors' native evaluation protocol. On a 21-layer, 1.1B-parameter transformer trained for 3000 steps, RS-MR reaches 0.798375 validation bits-per-byte, beating Muon by about 0.57% while adding only 7.45 MB of optimizer-state memory. The authors state that broader validation on larger models, longer horizons, more datasets, and other architectures is still needed, and that RS-MR is a matrix-parameter optimizer embedded in a hybrid stack rather than a full-model replacement for AdamW.","feed_headline":"New optimizer beats Muon on transformer pretraining","feed_subtitle":"Typed-program multi-agent search discovered RS-MR, which improves validation BPB by 0.57% with under 1% extra memory.","key_machinery":"The central object is a typed optimizer DSL that expresses updates via direction, scaling, preconditioning/geometry, regularization, state, and grouping modules, plus a compiler that type-checks and lowers programs before training. The search loop has four role agents (Theorist, Designer, Engineer, Reviewer) and a two-stage evolution that extends the DSL with safe macro operators when repeated failures signal a representational bottleneck. The mechanism that carries the argument is RS-MR's update rule: normalize-then-orthogonalize (row-norm of momentum before polar), lightweight RMS damping, and soft blockwise gating that combines a slow history gate with a fast alignment gate.","core_discovery":"The paper's core discovery is RS-MR (Reduced-State MAGMA-RowNorm), a matrix-parameter optimizer found by the OPTScientist search loop. RS-MR keeps only two optimizer states—a momentum matrix and a compact blockwise gate—and forms its update by row-normalizing the momentum matrix, taking a polar direction, damping the magnitude with a lightweight RMS-based factor, and gating blocks by their agreement with the current gradient. In the paper's native 21-layer transformer pretraining benchmark, RS-MR achieves the best final validation BPB among sixteen optimizers (0.798375), improving on Muon by 0.004574 BPB (about 0.57%) with only 0.39% additional optimizer-state memory. The authors emphasize t","pith_inferences":["Editorial inference: If the 0.57% gain survives repeated-seed evaluation, the mechanism worth isolating is the row-normalization-before-polarization step; an ablation that removes row_norm from RS-MR would test whether that is the active ingredient.","Editorial inference: Because RS-MR was selected during search on proxy versions of the same architecture and data, the reported native validation is not fully independent; a stronger transfer test would freeze the search and then evaluate the discovered optimizer on a held-out architecture or data split.","Editorial inference: The same typed-DSL loop could be redirected to other auditable program-synthesis settings, such as normalization layers, learning-rate schedules, or data-augmentation pipelines, where compiler-backed validity and long-horizon evaluation matter."],"forward_implications":["RS-MR outperforms Muon at every validation checkpoint from 250 to 3000 steps, so the reported improvement is not a final-step artifact.","The improvement over Muon costs only 7.45 MB of additional optimizer state, keeping RS-MR in the favorable Pareto region on both validation BPB and memory.","All ten top-ranked DSL-discovered optimizers beat Muon, suggesting the typed search space contains multiple practical alternatives to classical baselines.","Because RS-MR is a matrix-parameter optimizer in a hybrid stack, the practical claim is about structured matrix updates, not a full-model replacement for AdamW.","Compiler-backed validation makes discovered optimizers inspectable and reproducible as compact typed programs, reducing wasted training budget."],"fun_headline_variants":["Multi-agent search discovers RS-MR, beating Muon on pretraining","Typed DSL optimizer RS-MR edges out Muon by 0.57%","OPTScientist finds RS-MR, surpassing Muon by 0.57%","Automated optimizer search: RS-MR beats Muon on transformers","RS-MR: Muon-beating optimizer from multi-agent search"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The central empirical claim rests on a single training run per optimizer, with no error bars; if the 0.0046 BPB gap between RS-MR and Muon is within run-to-run noise, the superiority claim collapses.","fun_headline_variants_meta":{"raw":{"variants":["Multi-agent search discovers RS-MR, beating Muon on pretraining","Typed DSL optimizer RS-MR edges out Muon by 0.57%","OPTScientist finds RS-MR, surpassing Muon by 0.57%","Automated optimizer search: RS-MR beats Muon on transformers","RS-MR: Muon-beating optimizer from multi-agent search"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000554,"raw_usage":{"total_tokens":2504,"prompt_tokens":801,"completion_tokens":1703,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":545,"completion_tokens_details":{"reasoning_tokens":1602}},"tokens_in":545,"tokens_out":1703,"duration_ms":13672,"temperature":1.0,"reasoning_tokens":1602,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T12:25:32.159498+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the native 21-layer benchmark with RS-MR and Muon for at least five independent seeds and compare final validation BPB distributions; the claim is falsified if the RS-MR distribution overlaps Muon's within one standard deviation or if RS-MR's best result does not consistently beat Muon's mean.","supporting_citations":[],"review_version":1}