{"id":"6f42e931-ce9c-48f5-8d4a-cad43c2a691a","arxiv_id":"2412.06845","paper_version":6,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"The authors trained and openly released a 7B LLM, an instruction-tuned variant, a GRPO-based reasoning variant, and a VLM, claiming competitive or superior performance on zero-shot, few-shot, CoT, and VLM benchmarks.","lead":"This paper releases Moxin 7B, a fully open-source 7B language model family with pretraining code, data, checkpoints, as well as a reasoning model trained with GRPO and a vision-language model. A generalist might read it to see a concrete, relatively low-cost attempt at open full-stack model development and the pitfalls of benchmarking such models.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Base-model benchmark scores are uninterpretable: sections 3.3.3 and 7.2 show both Moxin-7B-Original and the released Moxin-7B-Base were trained on evaluation-benchmark training data, so Tables 2 and 3 cannot support the superior-performance claim against clean baselines.","rationale":"Good-faith reading: the paper is primarily an open-science release, and the released code/data/checkpoints are real evidence in favor of reproducibility. The reasoning and VLM sections are plausible and use standard open frameworks. The central performance claim, however, depends on benchmark comparisons where the paper's own text discloses that the released base model was fine-tuned on the training splits of the evaluation benchmarks (Section 7.2), and Section 3.3.3 shows that even the 'Original' model consumed evaluation-benchmark training data during pretraining. This is not an external-consensus disagreement; it is an internal evaluation-design flaw. The concrete test can settle whether the disclosed contamination materially moved Tables 2 and 3. If it does, the abstract and Section 5 superiority claims need to be withdrawn or re-run with decontaminated/held-out benchmarks. Since the reader already conditioned acceptance on this issue, my read leaves the CONDITIONAL verdict unchanged.","tokens_in":26430,"tokens_out":6785,"duration_ms":65570,"concrete_test":"Run a full decontamination audit using the released training data and checkpoints. Assemble the exact training mixture used for Moxin-7B-Original (including the Section 3.3.3 capability-enhancement data) and for Moxin-7B-Enhanced (including the Section 7.2 evaluation-benchmark fine-tuning data). For each of ARC-C, ARC-E, HellaSwag, PIQA, Winogrande, and MMLU, detect test items or near-duplicates present in those corpora via exact matching and fuzzy n-gram overlap. Recompute Tables 2 and 3 on the subset of test items with zero overlap. If the revised Moxin averages fall by more than 1-2 points or no longer beat the clean baselines, the superior-performance claim is an evaluation artifact rather than evidence of model quality.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline 'superior performance' claim for the base model rests on Tables 2 and 3, but both models in those tables are contaminated. Section 3.3.3 states that the final pretraining phase ('Capability Enhancement') uses open-source data from HuggingFace, 'primarily utilizing the training datasets of various evaluation benchmarks such as MMLU and HellaSwag.' This means Moxin-7B-Original, the supposedly clean foundation model, already saw MMLU/HellaSwag training examples during pretraining. Section 7.2 then says Moxin-7B-Enhanced is produced by 'subsequent partial fine-tuning of Moxin-7B-Original on the training data of the evaluation datasets' and is released as Moxin-7B-Base; Table 9 confirms Moxin-7B-Enhanced is the released base model. The comparison baselines (Mistral 7B, LLaMA 2/3, Gemma, Qwen, etc.) were not trained on those benchmark training splits. Consequently the Table 2 average gain (75.44 vs. 73.83 best baseline) and Table 3 result (70.55 vs. 71.57 best baseline) may reflect benchmark familiarity or memorization rather than genuine capability. No error bars or decontamination analysis are provided, so the inflation cannot be bounded. The GRPO/reasoning claim in Table 7 is less directly affected because those math benchmarks are not the stated capability-enhancement data, but the base and instruct models inherit the tainted foundation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Moxin 7B, a fully open-source 7B language model that releases pretraining code, training data, intermediate checkpoints, and final weights, together with Instruct, Reasoning, and VLM variants. The authors report zero-shot and few-shot base-model comparisons, post-trained instruct-model evaluations, GRPO-based chain-of-thought reinforcement learning results on math benchmarks, and VLM evaluations. The central claims are that the base model achieves superior or competitive performance against stronger 7B/8B baselines, the reasoning model demonstrates the effectiveness of RL for small LLMs, and the VLM outperforms comparable open VLM baselines.","tokens_in":26822,"tokens_out":5490,"duration_ms":49933,"significance":"If the empirical results were clean, the paper would be a valuable demonstration that a fully open 7B model can be competitive with widely used open-weight models and that GRPO-based RL can transfer from smaller and larger scales to 7B. The release of code, data, checkpoints, and configurations, including the reported $160,000 pretraining cost, is a genuine contribution to reproducibility and aligns with the Model Openness Framework. However, the base-model evaluation is compromised because both reported base models were trained on the training splits of the evaluation benchmarks themselves, so the headline 'superior performance' claim for the base model is not supported by the evidence as presented.","major_comments":[{"comment":"The base-model comparisons in Tables 2 and 3 cannot support the claimed 'superior performance' because both Moxin-7B-Original and Moxin-7B-Enhanced were trained on the training splits of the evaluation benchmarks. Section 3.3.3 states that the Capability Enhancement phase uses 'the training datasets of various evaluation benchmarks such as MMLU and HellaSwag,' so even Moxin-7B-Original has already seen these benchmark training examples before the additional fine-tuning described in Section 7.2. Section 7.2 then defines Moxin-7B-Enhanced as the result of 'subsequent partial fine-tuning of Moxin-7B-Original on the training data of the evaluation datasets' and Table 9 confirms that Moxin-7B-Enhanced is released as Moxin-7B-Base. The baseline models in Tables 2 and 3 were not trained on those benchmark training splits. Consequently, the reported gains (e.g., 75.44 vs. the 73.83 best baseline in Table 2, and 70.55 vs. 71.57 in Table 3) may reflect benchmark familiarity or memorization rather than general capability. The manuscript provides no decontamination analysis, no per-task overlap statistics, and no error bars, so the magnitude of any inflation cannot be bounded.","section":"§7.2, Tables 2 and 3; §3.3.3; Table 9"},{"comment":"There is an internal inconsistency in the definition of the models being compared. Section 7.2 calls Moxin-7B-Original 'the foundation model before fine-tuning on the training data of the evaluation datasets,' but Section 3.3.3 describes training on those same benchmark training datasets as part of the pretraining Capability Enhancement phase. The paper should clarify the exact chronology and composition of the training data, and it should either report evaluations for a truly clean foundation model or explicitly state that all reported base-model variants have seen evaluation-benchmark training data, in which case the comparisons against clean baselines must be reframed.","section":"§7.2 (definition of Moxin-7B-Original) vs. §3.3.3"},{"comment":"All performance claims are based on single-point accuracy numbers with no error bars, confidence intervals, or significance tests. Several margins are small, for example Table 8 reports Ours at 64.68 vs. Mistral v0.1 7B at 62.83, and Table 5 reports Moxin-7B-DPO at 71.66 vs. Mistral 8B Instruct at 71.62. Because the 'outperforms' claims in the abstract and introduction depend on these differences, the paper should report variance estimates from multiple evaluation runs or otherwise justify that the evaluation protocol is deterministic and the differences are reliable.","section":"Tables 4, 5, 7, and 8"}],"minor_comments":[{"comment":"The text says 'Table 3 presents our zero-shot evaluation results,' but Table 3 is in the Few-Shot Evaluation section and uses few-shot settings; the sentence should say 'few-shot.'","section":"§7.2.2"},{"comment":"The phrase 'few show evaluation' should be corrected to 'few-shot evaluation.'","section":"§7.3"},{"comment":"The RL framework name is spelled inconsistently as 'AReal' in the text and 'AReaL' in the caption of Table 7; the authors should pick one spelling and use it consistently.","section":"§5 and Table 7"},{"comment":"The average for Moxin-7B-II is reported as 71.095 with three decimal places while all other entries have two; this formatting inconsistency should be fixed.","section":"Table 5"},{"comment":"The MMLU benchmark is cited as reference [141], which is an AI Sandbagging paper; this appears to be an incorrect reference, and the intended MMLU citation is likely [107].","section":"§7.1, MMLU bullet"},{"comment":"The claim that 'our model outperforms all other VLM baselines' is based on the average score, but Ours is lower than several baselines on individual tasks (e.g., VizWiz: 54.08 vs. 56.39 for Llama-2 Chat 7B); the claim should be qualified to 'on average' or a per-task breakdown should be discussed.","section":"Table 8"}],"recommendation":"major_revision","confidential_remarks":"The contamination issue is the central problem: the paper's own text in Sections 3.3.3 and 7.2 shows that both base-model variants were trained on evaluation-benchmark training data, making the headline comparisons against clean baselines unsupported. This is not a subtle methodological choice but a direct contradiction between the abstract's 'superior performance' claim and the paper's own training-data description. A major revision should either provide a clean evaluation of a model that did not see benchmark training splits, or substantially reframe the paper's claims. The open-source release and the GRPO/reasoning experiments are of separate value, and the authors should be given the opportunity to present them in a corrected form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the artifact is real and worth having, but the headline benchmark comparison is not. I agree with the reader's conditional verdict, and the stress-test note lands.\n\nWhat's new: a genuinely open 7B model family—pretraining code, data, intermediate checkpoints, SFT/DPO recipes, GRPO runs, and a VLM—all released. That is a real contribution, especially the GRPO-at-7B demonstration in Table 7. Those math results are not based on the contaminated benchmarks, and the comparison to Qwen2.5-Math-7B and Llama-3.1-70B-Instruct is credible, modulo the usual missing error bars.\n\nThe problem is the base-model evaluation. Section 3.3.3 says the final pretraining phase used the training data of MMLU and HellaSwag, and Section 7.2 says the released 'Base' is actually Moxin-7B-Enhanced, fine-tuned on the training splits of ARC, HellaSwag, PIQA, Winogrande, and MMLU. Tables 2 and 3 then compare this model against baselines that never saw those training splits. The 'superior performance' claim in the abstract and Section 7.2 cannot stand. The paper is transparent about the procedure, which suggests good faith, but transparency doesn't make the comparison valid.\n\nThe instruct and VLM results inherit the tainted foundation, so I'd treat Tables 4-6 and 8 as suggestive, not conclusive. The VLM table at least has an internally controlled comparison (same framework, different backbones), which is more defensible.\n\nMinor point: no error bars or variance anywhere; the Section 7.2.2 text says 'zero-shot evaluation results' where it means few-shot. The generation examples are cherry-picked.\n\nVerdict: this is an artifact paper, not a science-of-LLMs paper. The artifacts deserve referee attention, but the evaluation needs to be redone with a clean base model and uncertainty estimates. This paper is for people building open LLM stacks, reproducibility researchers, and groups interested in RL at small scale. I'd recommend sending it to review with a request for major revision: re-run base evaluations on a model not trained on eval splits, report variance, and soften the 'superior performance' language. If the authors do that, it's a useful contribution to the open-model ecosystem.","headline":"A genuinely open 7B model family with a working GRPO recipe, but the headline base-model numbers are contaminated by training on the eval benchmarks themselves.","tokens_in":27418,"tokens_out":3591,"would_cite":false,"duration_ms":32442,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Moxin 7B claims a fully open pipeline—pretraining, post-training, GRPO-based reasoning, and a VLM—matches several leading 7B/8B open models on standard benchmarks.","keywords":["large language model","open-source LLM","reinforcement learning","GRPO","chain-of-thought reasoning","vision language model","model openness framework","7B model"],"falsifier":"Re-run Moxin-7B-Original (the pretrained-only checkpoint) on the same lm-evaluation-harness/opencompass pipeline and compare its zero-shot and few-shot averages—the paper reports 67.21% and 64.74%—against the released base's 75.44% and 70.55%. If the gap is almost fully accounted for by the partial fine-tune on evaluation-benchmark training data, or if retraining any baseline with the same capability-enhancement step closes the gap, the 'superior performance' claim for the base model is falsified. For the reasoning claim, a falsifier is a 7B run that keeps the CoT SFT but omits GRPO; if its competition-math scores stay near 43.2%, the claim that reinforcement learning provides the gain is falsified.","tokens_in":26231,"feed_emoji":"🤖","tokens_out":10360,"duration_ms":89257,"temperature":0.7,"pith_summary":"The paper is trying to establish that a 7B model built end-to-end from fully open components—pretraining data, training code, post-training recipes, and reinforcement learning—can compete with widely used 7B/8B open models while releasing everything needed for reproduction and audit. Its evidence is a family of models: a base model averaging 75.44% on five zero-shot tasks, an instruct model averaging 75.92% on the same suite, a reasoning model trained with GRPO that averages 43.2% Pass@1 on four competition-math benchmarks, and a vision-language model on the same backbone that outperforms comparable VLM baselines. If the results hold, they matter because most 'open' releases still withhold training data or code; Moxin claims the highest completeness level of the Model Openness Framework, so every stage of the pipeline can be inspected and rebuilt. The released base model, however, is the capability-enhanced version, produced by partial fine-tuning on the training data of the evaluation benchmarks.","feed_headline":"Fully open 7B model matches leading open 7B-8B rivals","feed_subtitle":"Moxin releases every stage from pretraining to reinforcement learning, with a reasoning model and VLM on the same backbone.","key_machinery":"Two mechanisms carry the argument. The first is the three-phase pretraining schedule whose final 'capability enhancement' phase mixes high-quality data, including training splits of evaluation benchmarks such as MMLU and HellaSwag, into the corpus; this step converts Moxin-7B-Original into Moxin-7B-Enhanced, and the authors release the Enhanced checkpoint as the base model. The second is GRPO (Group Relative Policy Optimization), a reinforcement-learning algorithm that uses an outcome reward model—returning 1 if an answer passes LaTeX/Sympy checks and 0 otherwise—and optimizes a group-relative advantage rather than a learned value model. The reasoning pipeline first SFTs the instruct model on DeepSeek-R1-distilled chain-of-thought data (OpenThoughts and OpenR1-Math-220k), then applies GRPO through the DeepScaleR framework. The VLM uses the Prismatic architecture with DINOv2 and SigLIP visual features fused and Moxin-7B as the language backbone.","core_discovery":"The paper's own claim is that a completely open 7B pipeline can be strong, not just transparent. On the zero-shot suite, the released base model reaches 75.44% average accuracy, above all listed 7B/8B baselines including Gemma-7B (73.83%), LLaMA 3.1-8B (73.79%), and Mistral-7B (73.29%). The reasoning model reaches 43.2% average Pass@1 on MATH-500, AMC 2023, Minerva Math, and OlympiadBench, above Qwen2.5-Math-7B-Base (33.55%) and Llama-3.1-70B-Instruct (40.48%), which the authors present as evidence that GRPO-based RL works at 7B scale. The VLM averages 64.68 across seven visual benchmarks, above LLaVA-v1.5-7B and above the same Prismatic framework with Llama-2 or Mistral backbones. An important framing note: the released base model is Moxin-7B-Enhanced, created by partially fine-tuning the pretrained model on the training splits of the evaluation benchmarks; the paper also reports the pretrained-only Moxin-7B-Original, which averages 67.21% on the same zero-shot suite.","pith_inferences":["Editorial: The paper's own tables show the pretrained-only Moxin-7B-Original averaging 67.21% zero-shot and 64.74% few-shot, below most listed baselines; readers should attribute the 'superior' base-model claim to the released enhanced checkpoint, not to the raw pretrained weights.","Editorial: The reasoning pipeline does not ablate the GRPO stage from the preceding CoT SFT, so the marginal contribution of reinforcement learning itself is not isolated; a no-RL control run would settle whether the 43.2% average comes from the reward optimization or from the distilled reasoning traces.","Editorial: The VLM comparison is confined to one architecture family and one evaluation suite; testing the released VLM on broader multimodal benchmarks would show whether the advantage generalizes.","Editorial: Because the paper releases intermediate checkpoints, an independent audit could rerun the capability-enhancement phase and the GRPO run to check whether the reported scores reproduce within noise."],"forward_implications":["If the results hold, a fully open 7B pipeline can match or beat several popular 7B/8B open models on standard zero-shot and few-shot benchmarks, so openness need not cost competitive performance.","The GRPO result would extend evidence for outcome-reward reinforcement learning from 1.5B and 67B scales to a 7B model, strengthening the case that RL is a viable reasoning booster for small LLMs.","The released checkpoint names mean that users of 'Moxin-7B-Base' are actually getting the capability-enhanced model, not the raw pretrained weights.","The VLM result would imply that the Moxin-7B backbone transfers well to multimodal tasks in the Prismatic-style architecture, outperforming Llama-2 and Mistral backbones trained identically."],"supporting_citations":[{"why":"Supplies the GRPO recipe and the R1-distilled chain-of-thought data approach the reasoning model follows.","marker":"[75]"},{"why":"Provides the Tulu 3 SFT/DPO datasets, training configurations, and OLMES evaluation harness used for the instruct model.","marker":"[73]"},{"why":"Provides the DeepScaleR RL framework that scales GRPO training to the 7B reasoning model.","marker":"[78]"},{"why":"Provides the DCLM-BASELINE pretraining corpus used alongside SlimPajama.","marker":"[64]"},{"why":"Provides the SlimPajama 627B-token cleaned and deduplicated text corpus used in pretraining.","marker":"[93]"},{"why":"Provides OpenThoughts CoT reasoning data distilled from DeepSeek R1 for the reasoning SFT stage.","marker":"[76]"},{"why":"Provides OpenR1-Math-220k, the math reasoning traces used in the CoT SFT stage.","marker":"[77]"},{"why":"Provides the Infinity Instruct dataset used to train the alternate instruct model version.","marker":"[74]"},{"why":"Provides the Prismatic VLMs framework and evaluation suite for the vision-language model.","marker":"[126]"}],"fun_headline_variants":["Fully open 7B: pretraining to GRPO, plus VLM and checkpoints","Open 7B model beats 70B on math, and rivals on zero-shot","Moxin 7B: open-sources pipeline, tops list of 7B/8B baselines","7B fully open model: all code, data, and RL-tuned reasoning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark comparisons rest on the assumption that fine-tuning the base model on the training splits of the evaluation benchmarks does not materially inflate its test scores, so comparisons against baselines that did not receive that fine-tuning remain fair.","fun_headline_variants_meta":{"raw":{"variants":["Fully open 7B: pretraining to GRPO, plus VLM and checkpoints","Open 7B model beats 70B on math, and rivals on zero-shot","Moxin 7B: open-sources pipeline, tops list of 7B/8B baselines","7B fully open model: all code, data, and RL-tuned reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000946,"raw_usage":{"total_tokens":4154,"prompt_tokens":1175,"completion_tokens":2979,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":791,"completion_tokens_details":{"reasoning_tokens":2882}},"tokens_in":791,"tokens_out":2979,"duration_ms":22850,"temperature":1.0,"reasoning_tokens":2882,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:22:22.732161+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run Moxin-7B-Original (the pretrained-only checkpoint) on the same lm-evaluation-harness/opencompass pipeline and compare its zero-shot and few-shot averages—the paper reports 67.21% and 64.74%—against the released base's 75.44% and 70.55%. If the gap is almost fully accounted for by the partial fine-tune on evaluation-benchmark training data, or if retraining any baseline with the same capability-enhancement step closes the gap, the 'superior performance' claim for the base model is falsified. For the reasoning claim, a falsifier is a 7B run that keeps the CoT SFT but omits GRPO; if its competition-math scores stay near 43.2%, the claim that reinforcement learning provides the gain is falsified.","supporting_citations":[],"review_version":1}