{"id":"dbb0a085-e319-4b9d-86d4-ac7fe91b6e79","arxiv_id":"2505.07608","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"MiMo-7B-RL, trained from scratch with a reasoning-focused pretraining mix and GRPO on 130K verifiable math and code problems, reports higher benchmark scores than o1-mini on AIME 2025 and LiveCodeBench v5 and v6.","lead":"Xiaomi's MiMo team trained a 7-billion-parameter language model from scratch with data mixes intended to strengthen reasoning, then added reinforcement learning on 130,000 verifiable math and coding problems. The final model, MiMo-7B-RL, reports scores above OpenAI's o1-mini on AIME 2025 and LiveCodeBench v5 and v6, and the checkpoints are public.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The o1-mini comparisons depend on unverified decontamination: the 25T-token pretraining corpus and 130K RL set are filtered only via internal 16-gram/n-gram checks, so disguised or near-duplicate AIME 2025 and LiveCodeBench v5/v6 items could explain the reported margins.","rationale":"The reader's weakest assumption identifies benchmark contamination as the key risk to the headline result, and my reading agrees. The paper's decontamination descriptions are internal and partial: SFT filters at 16-gram overlap, RL data are 'carefully decontaminated' without details, and pretraining — the component most likely to cause memorization of web-scraped contest problems — has no stated decontamination procedure at all. The model's public checkpoints make a concrete falsification test possible, which is why the correct verdict is CONDITIONAL rather than outright rejection or acceptance. The reader's verdict already captures this, so no adjustment is needed. I also noted minor internal inconsistencies (Section 3.1 says ~500K SFT samples while the Discussion reports scaling to 6M, and the abstract's 'general reasoning' claim contradicts Table 4's MMLU-Pro and IFEval scores), but these do not change the central math/code claim as much as the contamination risk does.","tokens_in":22996,"tokens_out":4060,"duration_ms":39521,"concrete_test":"Run a membership-inference probe on the public MiMo-7B-Base checkpoint: compute per-token perplexity or Min-K% Prob for each AIME 2025 and LiveCodeBench v5/v6 problem statement, and compare against a matched control set of same-difficulty problems published after the model's training cutoff (e.g., AIME 2025 if the base checkpoint predates its release; otherwise use contests released after the checkpoint date). If benchmark statements are systematically more memorized than the control set, decontamination failed and the headline margins are suspect.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim is that MiMo-7B-RL surpasses o1-mini on AIME 2025 and LiveCodeBench v5/v6. That claim is load-bearing on the assumption that those evaluation problems were not seen during pretraining, SFT, or RL. The paper's own defense is thin: Section 3.1 says SFT queries with 16-gram overlap with evaluation benchmarks are removed, and Section 3.2 says RL data is 'carefully decontaminated' without specifying the method or reporting overlap statistics. Critically, no decontamination of the 25T-token pretraining corpus is described at all. The pretraining pipeline in Section 2.1 includes synthetic reasoning data generated from 'mathematics and code problems' and web extraction, with no stated exclusion of benchmark items. AIME 2025 was released in February 2025 and LiveCodeBench v5/v6 cover contests through May 2025; if the pretraining corpus or the 'newly collected problem set' for RL scraped the same web sources after those releases, exact or near-duplicate problems could leak. The 16-gram filter catches verbatim copies but not paraphrases, template rewrites, or translated statements. Because the data are private and the decontamination is not independently auditable, the reported margins (4.7 points on AIME 2025, 4.0 and 2.5 on LiveCodeBench v5/v6) could reflect contamination artifacts rather than genuine reasoning gains. This is a correctness risk, not a demonstration that contamination occurred, but it is the weakest link in the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript reports a full-stack recipe for building MiMo-7B, a 7B-parameter reasoning-focused LLM trained from scratch on 25 trillion tokens. The pretraining contribution is a three-stage data mixture with synthetic reasoning data, an MTP auxiliary objective, and a data-processing pipeline aimed at increasing reasoning-pattern density. The posttraining contribution is an SFT stage followed by GRPO-based RL on 130K verifiable math and code problems, with a test-difficulty-driven code reward, dynamic sampling, and an engineered rollout engine. The paper claims that MiMo-7B-Base outperforms comparable open 7B/8B/9B base models on several reasoning benchmarks, that MiMo-7B-RL-Zero surpasses a 32B RL baseline, and that MiMo-7B-RL beats OpenAI o1-mini on AIME 2025 and LiveCodeBench v5/v6 while maintaining competitive general performance. The open-sourced checkpoints and detailed hyperparameters are notable assets, but the headline comparison depends on decontamination assurances and an internally evaluated benchmark protocol that are not fully documented.","tokens_in":23419,"tokens_out":6381,"duration_ms":58353,"significance":"If the claims survive scrutiny, the paper is significant: a 7B model trained with an openly described recipe would outperform o1-mini on selected math and code benchmarks, at a scale where such results are still uncommon, and would provide a reproducible open-source reference for both pretraining and RL posttraining. The MTP-based speculative decoding with reported acceptance rates is a practical inference contribution, and the Seamless Rollout Engine addresses a real systems bottleneck in RL training. Credit is also due for releasing model checkpoints and for reporting repeated-sampling averages on several benchmarks rather than single draws. However, the general-reasoning claim in the abstract is contradicted by Table 4, the head-to-head comparison against o1-mini depends on unverified decontamination of private training data, and the evaluation protocol for the headline numbers needs clarification. These issues do not necessarily invalidate the core empirical findings, but they must be corrected before the central claims can be accepted.","major_comments":[{"comment":"This is a correctness risk rather than a demonstration of contamination, but it is the weakest load-bearing link in the headline comparison.","section":"Section 2.1, Section 3.1, Section 3.2, Tables 4-5"},{"comment":"This is not merely a wording issue: the current text makes a general-reasoning claim that the paper's own table refutes.","section":"Abstract, Section 1, Table 4"},{"comment":"This directly affects the validity of the o1-mini margin reported in the abstract.","section":"Section 3.5.1, Table 4, Figure 1"},{"comment":"Without this information, the reader cannot judge whether the reported gains on LiveCodeBench come from the reward scheme or from other components of the RL pipeline.","section":"Section 3.3.1, Figure 5"}],"minor_comments":[{"comment":"","section":"Section 3.5.2"},{"comment":"","section":"Section 3.2"},{"comment":"","section":"Section 3.1"},{"comment":"","section":"Section 2.2"},{"comment":"","section":"References"},{"comment":"","section":"Table 6"},{"comment":"","section":"Table 2 and Table 3"},{"comment":"","section":"Figure 5"}],"recommendation":"major_revision","confidential_remarks":"The paper is a systems-and-recipe report from an industrial lab, and the headline comparison to a closed model like o1-mini is inherently hard to audit. My main concerns, in order, are the decontamination gap for the pretraining corpus and RL data, the unsupported 'general reasoning' claim in the abstract, and the ambiguity about whether the reported numbers are pass@1 or pass@k. I do not see evidence of deliberate misreporting, but the authors should be asked to release their evaluation harness and exact contamination-filtering scripts, and to temper the abstract before acceptance. The open-sourced checkpoints are a strong positive; the paper would be more valuable if the evaluation protocol were independently reproducible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is worth reading as a 7B reasoning recipe. Two components are genuinely new – the test-difficulty-driven code reward and the easy-data resampling for dynamic sampling – and the authors release checkpoints, so the headline math/code numbers can be checked by anyone. The base model numbers are also striking: a 7B base at 32.9 pass@1 on AIME 2024 and LCB v5 before any RL is not something I'd have predicted from the current open 7B crop. If the RL numbers reproduce, this changes cost assumptions for reasoning models.\n\nThe paper does a lot well. The three-stage pretraining mix with MTP is applied at real scale, the base-model pass@k analysis is the right way to look for reasoning potential, and the post-training section documents concrete problems (sparse code rewards, dynamic-sampling efficiency loss, math/code interference) instead of just reporting wins. The infrastructure work is a sensible engineering contribution, and the RL-Zero comparison against a 32B baseline is informative.\n\nSoft spots, in rough order of importance. First, the abstract overclaims: on the paper's own Table 4, MiMo-7B-RL trails o1-mini by 21.7 points on MMLU-Pro and 23.8 on IFEval. 'General reasoning' superiority is not supported. Second, the o1-mini comparison depends on decontamination that is described thinly and not independently auditable: SFT gets a 16-gram overlap filter, RL data is 'carefully decontaminated' with no method given, and no pretraining decontamination is described at all. Given the 25T-token corpus includes web and synthetic reasoning data, and AIME 2025/LCB v6 postdate portions of it, this is a genuine correctness risk. It is not evidence contamination happened, and public checkpoints make independent re-runs easy, but the risk is real. Third, there is an internal inconsistency about SFT scale: Section 3.1 says the final SFT dataset is about 500K samples, while the discussion says scaling from 500K to 6M produced the gains in Table 6 and implies the main RL model used the 6M version. That needs resolving before the recipe can be taken at face value. Finally, several recipe details (difficulty-group thresholds, pass-rate cuts, resampling schedule) are missing.\n\nThe math and code results are plausible and checkable. The paper deserves a serious referee, not a desk reject. My recommendation: send it out, and tell the referees to focus on decontamination evidence and the SFT-scale inconsistency.","headline":"Real 7B reasoning recipe with public checkpoints and two genuinely new RL ideas, but the o1-mini headline depends on unverifiable decontamination and the paper overclaims general reasoning.","tokens_in":24189,"tokens_out":3670,"would_cite":true,"duration_ms":34712,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 7-billion-parameter model trained with reasoning-dense data and graded code rewards beats o1-mini on math and code benchmarks.","keywords":["MiMo-7B","reasoning language models","multi-token prediction","reinforcement learning","GRPO","three-stage data mixture","difficulty-graded code reward","pass@k evaluation"],"falsifier":"Run MiMo-7B-RL on a competition set released after its training cutoff — the next AIME or a LiveCodeBench window that postdates the model — and compare pass@32 with o1-mini; if the reported lead shrinks or inverts on the fresh set while the paper's benchmarks stay high, the advantage is contamination rather than reasoning ability. A cheaper inspection is to search the released RL and SFT data for near-duplicates of evaluation problems at an 8-gram threshold and read the retrieved passages for paraphrased restatements.","tokens_in":22808,"feed_emoji":"🧠","tokens_out":13070,"duration_ms":97808,"temperature":0.7,"pith_summary":"The paper claims that a 7-billion-parameter language model can reach the reasoning level of much larger proprietary systems if pretraining and post-training are both engineered for reasoning. On the pretraining side, it trains a base model on 25 trillion tokens using a three-stage mixture that raises math and code to roughly 70 percent of the data and then adds synthetic reasoning responses, alongside a multi-token prediction objective. On the post-training side, it applies reinforcement learning to 130K verifiable math and code problems, with a test-difficulty-graded reward that gives partial credit on hard coding problems. The resulting model, MiMo-7B-RL, scores 55.4 on AIME 2025 and 57.8 and 49.3 on LiveCodeBench v5 and v6, above the 50.7, 53.8, and 46.8 reported for o1-mini. If correct, the work shows that data and reward design, not scale alone, determines how much reasoning a small model can unlock.","feed_headline":"7B open model beats o1-mini on math and code","feed_subtitle":"Reasoning-dense pretraining and difficulty-graded code rewards carry a 7B model past a proprietary reasoner.","key_machinery":"The argument is carried by four mechanisms. The three-stage data mixture builds a general curated distribution, then raises math and code to about 70 percent of tokens, then adds about 10 percent synthetic reasoning responses while extending the context window to 32,768 tokens. Multi-token prediction (MTP) is an auxiliary objective that predicts several future tokens at once and, at inference, is reused for speculative decoding, with the first MTP layer reported to accept about 90 percent of drafted tokens on AIME 2024. The test-difficulty-driven code reward, inspired by International Olympiad in Informatics scoring, clusters each problem's test cases into difficulty levels by their pass rates over many model rollouts and returns the sum of per-group scores, so a solution earns partial credit for passing harder subtasks instead of an all-or-nothing verdict. Easy-data re-sampling keeps perfectly solved problems in a pool that is sampled with probability 10 percent, which stabilizes policy updates in late RL training. These sit on top of GRPO modifications the paper adopts from recent work: removal of the KL loss, dynamic sampling that filters prompts with pass rates of 0 or 1, and an asymmetric clip bound.","core_discovery":"MiMo-7B-RL, a 7B model trained from scratch and then tuned on verifiable problems, reaches 55.4 on AIME 2025, 57.8 on LiveCodeBench v5, and 49.3 on LiveCodeBench v6, surpassing o1-mini (50.7, 53.8, 46.8) on all three while remaining below it on MMLU-Pro and IFEval. The authors attribute the outcome to two linked investments. Pretraining targets what they call reasoning pattern density: extraction tooling that preserves equations and code, small fine-tuned taggers that replace heuristic filters, synthetic reasoning data, and a three-stage mixture whose middle stage devotes about 70 percent of tokens to math and code, with a final stage that adds roughly 10 percent synthetic reasoning responses and extends the context window. Post-training applies a GRPO-style reinforcement learning recipe with rule-based rewards only, no KL term, dynamic sampling, and a test-difficulty-driven code reward that clusters test cases by pass rate and awards partial credit so hard problems yield learnable signal. They also report that RL applied directly to the base model outperforms RL trained on a 32B base model on both math and code, which they take as evidence that reasoning potential is largely set during pretraining.","pith_inferences":["Because the paper reports no ablation of the three-stage mixture, a controlled study that varies only the stage-2 math/code ratio or the stage-3 synthetic share would say which stage actually drives the base model's pass@k advantage.","The pass-rate clustering behind the code reward should transfer to any domain with verifiable sub-results — theorem proving with per-tactic checks, agent tasks with sub-goal verification, scientific computation with partial output checks — wherever all-or-nothing rewards are too sparse.","The reported margins over o1-mini may not transfer to future contest sets: the 16-gram decontamination filter removes verbatim overlaps only, so disguised or paraphrased evaluation problems inside the 25T-token corpus would inflate the paper's numbers while leaving freshly written competitions as the true test."],"forward_implications":["A 7B open model can exceed a proprietary reasoning model on math and code benchmarks using only rule-based rewards, with no format or length penalties.","The ceiling of RL training is set largely by the base model: RL run directly on MiMo-7B-Base outperforms RL run on a 32B base on both math and code, suggesting curated pretraining can substitute for scale.","MTP pays for itself twice: it contributes a training signal and, via speculative decoding, cuts the latency of the long generations that reasoning models produce.","Scaling SFT from 500K to 6M instances improves reasoning and dialogue and does not blunt later RL gains; a follow-on checkpoint trained with on-policy RL and a longer generation budget reaches 80.1 on AIME 2024.","The o1-mini advantage is specific to verifiable math and code reasoning: on MMLU-Pro and IFEval, MiMo-7B-RL scores below o1-mini, so the headline comparison is not a claim of general superiority."],"supporting_citations":[{"why":"Defines the rule-based-reward RL recipe for reasoning models that MiMo-7B-RL builds on, and reports the language-mixing issue that the paper tries to penalize.","marker":"Guo et al., 2025"},{"why":"Supplies the GRPO update that Equation 1 modifies with dynamic sampling, clip-higher, and no KL.","marker":"Shao et al., 2024"},{"why":"Source of the no-KL-loss objective change that the paper adopts to raise the policy's ceiling.","marker":"Hu et al., 2025"},{"why":"Source of the dynamic sampling and asymmetric clip techniques used to calibrate difficulty and preserve exploration.","marker":"Yu et al., 2025"},{"why":"Provides the Multi-Token Prediction module that MiMo-7B adapts for pretraining and replicates for speculative decoding.","marker":"Liu et al., 2024a"},{"why":"Supplies the pass@k evaluation methodology used to measure the base model's reasoning potential against 32B baselines.","marker":"Yue et al., 2025"},{"why":"Defines LiveCodeBench, the code benchmark where MiMo-7B-RL reports its largest leads over o1-mini.","marker":"Jain et al., 2024"},{"why":"Defines AIME 2025, the math contest on which MiMo-7B-RL reports a 4.7-point lead over o1-mini.","marker":"MAA, 2025"},{"why":"Provides the vLLM engine that the paper extends with MTP support for RL rollout.","marker":"Kwon et al., 2023"},{"why":"Supplies the verl RL training framework on which the Seamless Rollout Engine is built.","marker":"Sheng et al., 2024"}],"fun_headline_variants":["7B model tops o1-mini on math and code","Small 7B reasoner outguns o1-mini","Reasoning-dense pretraining lifts 7B past o1-mini","MiMo-7B: 7B that beats o1-mini at math","Pretraining secrets: 7B model beats o1-mini"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline comparisons assume that AIME 2025 and LiveCodeBench v5 and v6 problems did not appear, in original or paraphrased form, in the 25-trillion-token pretraining corpus, the multi-million-sample SFT set, or the 130K RL problem set, since the only stated defense is a 16-gram overlap filter that catches verbatim but not disguised copies.","fun_headline_variants_meta":{"raw":{"variants":["7B model tops o1-mini on math and code","Small 7B reasoner outguns o1-mini","Reasoning-dense pretraining lifts 7B past o1-mini","MiMo-7B: 7B that beats o1-mini at math","Pretraining secrets: 7B model beats o1-mini"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000903,"raw_usage":{"total_tokens":3920,"prompt_tokens":1012,"completion_tokens":2908,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":628,"completion_tokens_details":{"reasoning_tokens":2813}},"tokens_in":628,"tokens_out":2908,"duration_ms":16824,"temperature":1.0,"reasoning_tokens":2813,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:12:28.297467+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run MiMo-7B-RL on a competition set released after its training cutoff — the next AIME or a LiveCodeBench window that postdates the model — and compare pass@32 with o1-mini; if the reported lead shrinks or inverts on the fresh set while the paper's benchmarks stay high, the advantage is contamination rather than reasoning ability. A cheaper inspection is to search the released RL and SFT data for near-duplicates of evaluation problems at an 8-gram threshold and read the retrieved passages for paraphrased restatements.","supporting_citations":[],"review_version":1}