{"id":"756f1242-6e8c-4db3-9109-ef7b4fb6ccb7","arxiv_id":"2505.22375","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Pangu Embedded, a 7B reasoner trained with iterative distillation, RL, and an adaptive fast/slow thinking scheme, reports superior benchmark scores to similarly sized Qwen3-8B and GLM-4-9B.","lead":"Huawei reports that Pangu Embedded, a 7-billion-parameter language model, outperforms similarly sized Qwen3-8B and GLM-4-9B on math, science, and coding benchmarks while adding user-controllable fast and slow thinking modes. The practical significance is a potential route to cheaper, lower-latency reasoning models that can be deployed on Ascend NPUs.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 2's headline margins over Qwen3-8B rest on internally measured baseline cells with no protocol details or error bars; under the paper's own 500-sample rule, the 2.5-point AIME24 gap is within sampling noise.","rationale":"I agree with the reader's weakest_assumption: the non-daggered baseline cells are the soft spot in the comparative claim. The manuscript has genuine substance, including a detailed training recipe, internal ablations (Tables 5-7), and a concrete dual-system mechanism, which makes the engineering contribution plausible. However, the abstract makes a quantitative comparative claim about beating Qwen3-8B and GLM-4-9B, and that claim is not secured by the reported evaluation. The reader's concern is fairness of internal baseline measurements; I would add that even fair internal runs would not settle the question, because Eq. (11)'s 500-sample floor produces wide confidence intervals on a 30-problem benchmark like AIME24. A 2.5-point lead is under 1.5 standard errors, so the headline margin is not distinguishable from noise. The paper also reuses AIME24, LiveCodeBench, and GPQA Diamond in development and ablation decisions, which increases the risk of selection bias; that is a further reason the reported numbers cannot be treated as unbiased estimates. None of this proves the claim false, and the official GLM-4-9B cells are consistent with a real advantage, so I would not reject the paper. The correct verdict is the same as the reader's: CONDITIONAL acceptance pending an independent, protocol-matched evaluation. The textual artifacts in the manuscript, such as the duplicated 'Data Synthesis' passage in Section 2.3.2 and the 'Changed from to sentence start' notes in Section 2.5.2, reinforce the need for a clean public release but are not the central issue.","tokens_in":34108,"tokens_out":10982,"duration_ms":127461,"concrete_test":"Replace every non-daggered Qwen3-8B cell in Table 2 with the corresponding published number from the Qwen3 technical report, using the same benchmark and answer-extraction convention; then compute bootstrap 95% confidence intervals for the Pangu-vs-Qwen3 differences under Eq. (11) sampling (N=17 for AIME24, roughly 600 samples for GPQA Diamond). If the AIME24 or LiveCodeBench margin reverses, or its confidence interval includes zero, the claimed advantage over Qwen3-8B is not established. A stronger version is to release the weights and harness so a third party can run both models with identical prompts, decoding, and sample counts.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim ('outperforms similar-size models like Qwen3-8B and GLM4-9B', Abstract) rests on Table 2. The tightest comparison, Qwen3-8B Thinking, is supported only partly by daggered official numbers; the Qwen3-8B Thinking AIME24 (79.4), LiveCodeBench (61.8), ArenaHard (89.9), and MMLU-Pro (72.5) cells are from 'our internal evaluation.' Section 4.2 does not state the prompt templates, few-shot counts, decoding temperatures, top-n-sigma/top-p settings, max tokens, or answer-extraction rules for those internal baseline cells, nor whether they used the same harness as Pangu Embedded. Eq. (11) fixes at least 500 effective samples per benchmark, so AIME24 (M=30) gets N=17 stochastic runs per problem. The paper gives no per-cell N, no standard errors, and no confidence intervals. Under the paper's own sampling rule, the AIME24 Thinking gap (81.9 vs 79.4, 2.5 points) has standard error about 1.7 points with p near 0.8 and 510 samples, so it is less than 1.5 standard errors; the GPQA gap (68.0 vs 62.0) is borderline. Thus even if the internal baselines are measured in good faith, the headline 'outperforms' is not statistically established. The official daggered GLM-4-9B cells are more favorable to Pangu, but the abstract and intro emphasize the Qwen3-8B comparison, and no weights or evaluation harness are released for independent reproduction.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript describes Pangu Embedded, a 7B-parameter LLM reasoner trained on Ascend NPUs. Stage 1 combines model-aware iterative distillation with inter-iteration model merging, followed by GRPO-style reinforcement learning guided by the Multi-source Adaptive Reward System (MARS). Stage 2 adds fast/slow thinking modes via fusion SFT, supporting both manual mode switching and automatic complexity-aware mode selection. The central claim is that Pangu Embedded outperforms similarly sized models such as Qwen3-8B and GLM-4-9B on AIME 2024, GPQA, LiveCodeBench, and other benchmarks (Abstract, Table 2).","tokens_in":34541,"tokens_out":4866,"duration_ms":50651,"significance":"If the central comparison were fully established, the result would be practically significant: a single 7B model delivering reasoning quality competitive with 8-9B rivals while also offering an efficient fast mode. The paper also contains genuinely useful engineering contributions: model-aware data complexity selection, inter-iteration merging, a repetition self-repair mechanism, detailed RL infrastructure for Ascend NPUs, and informative internal ablations (Tables 3-7, Figures 13-14). These parts are described with enough specificity to be partially reimplemented, which is a strength. However, the headline 'outperforms' claim is not yet statistically grounded: several baseline cells in Table 2 are internal and protocol-free, no confidence intervals are reported, and AIME 2024 is used both as a tuning target and as a headline result. No code, weights, or evaluation harness are released, which further limits independent verification.","major_comments":[{"comment":"The central comparison against Qwen3-8B and GLM-4-9B mixes daggered official numbers with 'our internal evaluation' cells, but Section 4.2 does not report the prompt templates, few-shot counts, decoding temperatures, top-n-sigma/top-p settings, max tokens, or answer-extraction rules for the internal baseline cells, nor whether the same harness was used as for Pangu Embedded. Under the paper's own Eq. (11) sampling rule, AIME24 (M=30) receives 17 stochastic runs per problem, so the reported 2.5-point Thinking-mode gap (81.9 vs 79.4) is about one standard error (roughly 2.5 points at 510 effective samples per model) and is not statistically significant; the GPQA gap is borderline. No per-cell N, standard errors, or confidence intervals are provided anywhere. Because the abstract's 'outperforms' claim rests on these margins, the evaluation needs either per-cell protocols with uncertainty quantification or a weakened claim.","section":"Table 2, Section 4.2"},{"comment":"AIME 2024 is used to select the data-complexity mean mu in Eq. (2) via the experiments in Figure 12, and Table 5 uses AIME 2024 to decide that three SFT iterations suffice. Consequently the reported AIME24 Thinking score (81.9) is a selected maximum over configurations, not an unbiased estimate; the same benchmark is both a tuning target and a headline result. The paper should either perform selection on a separate development set or explicitly adjust for this selection when presenting AIME24 as evidence of superiority.","section":"Section 4.5, Figure 12, Table 5"},{"comment":"The text states that 'the majority of our assessments employ few-shot inputs, with a minority using zero-shot prompts,' but does not specify which benchmark uses which setting or how many shots. This matters because baseline numbers from official reports may use different few-shot protocols than the internally measured cells, and the paper claims exact matching for gold-answer benchmarks without stating the extraction rules. This lack of protocol detail makes it impossible for a reader to reproduce any single cell of Table 2, including the Pangu Embedded cells.","section":"Section 4.2, 'Evaluation Baselines and Metrics'"}],"minor_comments":[{"comment":"The paragraph on 'Other Rewards' contains editorial artifacts '(Changed from to sentence start)' that should be removed before publication.","section":"Section 2.5.2"},{"comment":"The axis labels in Figure 10 are rendered as raw unicode escape sequences (e.g., '/uni0000002a/uni00000033/...'), making the figure unreadable; the figure needs to be regenerated with proper text.","section":"Figure 10"},{"comment":"The 'Data Synthesis' paragraph appears twice with near-identical wording, once after the description of the pipeline and once after Eq. (2); the duplication should be removed.","section":"Section 2.3.2"},{"comment":"There is a typo in 'In GRPO algirithm' that should read 'algorithm'.","section":"Section 2.5.1"},{"comment":"Reference [29] lists 'Codeforces' as the title of the AIME 2024 citation, which appears to be an error; the correct source title should be used.","section":"Reference [29]"},{"comment":"The figure caption contains the typo 'Curribulum Data Mixing' instead of 'Curriculum Data Mixing'.","section":"Figure 5"}],"recommendation":"major_revision","confidential_remarks":"The central engineering contributions appear real and the ablations are internally consistent, but the headline comparison is not statistically established because of the protocol gaps in Table 2 and the use of AIME 2024 for model selection. These issues are fixable within the scope of the paper: report per-cell protocols and confidence intervals, add a development-set holdout for selection, and soften claims where margins are within noise. I therefore recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a serious engineering report with a plausible 7B model, but the abstract overstates the evidence. The AIME 2024 headline is not an independent test result — Figure 12 and Table 5 use that benchmark for data-selection and early-stopping decisions — and the gaps over Qwen3-8B in Table 2 are within sampling noise under the paper's own 500-sample rule (Eq. 11). I would not take the 'outperforms' claim as established without the numbers being re-verified on a fresh split.\n\nWhat's genuinely good: the paper is transparent about the recipe — model-aware iterative distillation with inter-iteration merging, GRPO with a zero-advantage mask, a multi-source reward system, curriculum data mixing, and a dual-system SFT for fast/slow switching. The ablations are real: fusion training clearly beats naive co-training, the merging helps, and the repetition self-repair shows a sensible 8-seed evaluation (Table 7). They also cite the parallel fast/slow works (FAST, AutoThink, AdaptThink) rather than hiding them. The Ascend RL infrastructure — SSP scheduling, co-located train/inference, Ascend-aware chunking — is a useful contribution in itself.\n\nThe soft spots, in order of severity. First, the circularity: AIME 2024 was used to pick the complexity mean mu (Figure 12) and to decide when to stop iterative SFT (Table 5). So the AIME 2024 number in Table 2 is not a held-out result. Second, the baseline comparison: Table 2 mixes official daggered numbers with 'our internal evaluation,' but Section 4.2 doesn't state the prompt templates, few-shot counts, decoding temperatures, or answer extraction for the internal cells. Third, the statistical point: with their own minimum of 500 effective samples, the 2.5-point AIME24 gap has a standard error of about two points, so it is not significant; the GPQA gap is borderline. The absence of error bars makes every close comparison unreadable. Fourth, LiveCodeBench and MATH500 appear as test curves during RL validation (Figures 13-14), so those benchmarks may have influenced training decisions too. Finally, no weights or evaluation harness are released, so independent reproduction is impossible.\n\nVerdict on the work: the engineering is credible, the components are each reasonable, and the model very likely does something useful on Ascend. But the paper's central empirical claim is not yet demonstrated. It deserves a serious referee, and a good referee will send it back for: (1) re-evaluation of AIME 2024 as a strictly held-out benchmark, (2) a public evaluation harness or at least full protocol disclosure, (3) error bars and per-cell details for every baseline, and (4) ideally a release of the weights. If those are addressed, this could be a useful reference for the fast/slow-thinking line. I'd bring it to a reading group as a case study in how engineering reports and benchmark claims interact.","headline":"Serious engineering report, but the AIME 2024 result is tuned, and the headline gaps are within sampling noise under the paper's own rule; needs a serious referee and major revision before the outperformance claim is credible.","tokens_in":35160,"tokens_out":5515,"would_cite":true,"duration_ms":57044,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a single 7-billion-parameter model can switch between fast and slow reasoning and beat similar-size rivals at both, using a two-stage distillation-plus-RL training framework.","keywords":["Pangu Embedded","dual-system reasoning","fast and slow thinking","adaptive mode selection","iterative distillation","reinforcement learning","reward system","Ascend NPU"],"falsifier":"Rerun the Table 2 comparison under one fixed protocol: identical prompts, identical decoding settings (including sampling temperature and top-p), identical few-shot counts, and at least 500 effective samples per benchmark, for Pangu Embedded and for Qwen3-8B and GLM-4-9B in both modes. If the rival models reproduce or exceed 81.9, 68.0, and 67.1 under those matched conditions, the claimed lead would be a measurement artifact rather than a model property.","tokens_in":33905,"feed_emoji":"🧠","tokens_out":12439,"duration_ms":114660,"temperature":0.7,"pith_summary":"The paper sets out to show that a single 7-billion-parameter model can behave as both a fast, low-latency answerer and a slow, deliberative reasoner, and that it can outperform similarly sized rivals in both roles. It reports that Pangu Embedded's slow 'thinking' mode scores 81.9 on AIME 2024, 68.0 on GPQA Diamond, and 67.1 on LiveCodeBench, ahead of Qwen3-8B and GLM-4-9B, while its fast mode stays competitive and an automatic, complexity-aware mode selector acts as a lightweight metacognitive controller that cuts output tokens sharply on easy problems. The construction is a two-stage pipeline: iterative distillation with inter-iteration model merging and reinforcement learning build a strong base reasoner, then 'fusion training' teaches the same weights to emit either a direct answer or a chain-of-thought, on user command or by self-assessed complexity. If the results hold, strong reasoning would not require a giant model or a long thinking chain for every query.","feed_headline":"One 7B model out-scores 8-9B rivals in both thinking modes","feed_subtitle":"Fast, direct answers and deep chain-of-thought in a single checkpoint, with up to 88% fewer tokens on easy queries.","key_machinery":"The load-bearing object is the dual-system training dataset and the fusion-training curriculum built on it: roughly 300K fast-mode and 300K slow-mode samples, where easy queries are paired with concise answers and hard queries with <think>...</think> chains, and where training starts from the already capable slow reasoner and replays slow data while adding fast data. The model learns to emit or omit the thinking block, which makes the mode switch observable and lets a meta-prompt or natural-language instruction select the behaviour at inference time. Supporting machinery includes the model-aware complexity score $C(x,y;G_{t-1})=1-\\frac{1}{k}\\sum_i I(\\mathrm{Eq}(y_i,y))$ that selects distillation data, the inter-iteration merging rule $\\Theta^t_{\\mathrm{merged}}=\\Theta^{t-1}_{\\mathrm{merged}}+\\lambda_t\\bar{\\delta}^t$, and the Multi-source Adaptive Reward System (MARS) with rule-based and LLM-based verifiers.","core_discovery":"Pangu Embedded is a 7-billion-parameter reasoner whose central claim is that one set of weights can deliver both System 1 and System 2 behaviour without sacrificing either. In slow mode the authors report 81.9 on AIME 2024, 68.0 on GPQA Diamond, and 67.1 on LiveCodeBench, edging out Qwen3-8B (79.4, 62.0, 61.8) and GLM-4-9B (76.4, 58.5, 51.8); in fast mode the model remains competitive, and an adaptive variant holds accuracy nearly constant on GSM8K while reducing average output tokens from 2,721 to 325. The mechanism is a two-stage training framework: model-aware iterative distillation with inter-iteration checkpoint merging followed by GRPO reinforcement learning under a Multi-source Adaptive Reward System (MARS), and then a fusion-training stage that replays mastered slow-thinking data while introducing fast-thinking exemplars, so the presence or absence of the <think>...</think> block becomes the model's learned switch between cognitive modes.","pith_inferences":["Because adaptive mode was trained only on mathematics, an obvious extension is to build difficulty-labelled datasets for code, legal, and general QA and test whether the token-reduction effect transfers.","The mode selector is learned by imitating labels rather than optimized for a latency-accuracy trade-off; adding an efficiency-aware reward to the RL stage could push the frontier further.","The manual-switch design implies a practical serving architecture: one checkpoint can offer a cheap 'quick answer' tier and a premium 'show your work' tier with the same weights."],"forward_implications":["In slow mode, the 7B model reports higher scores than Qwen3-8B and GLM-4-9B on AIME 2024, GPQA Diamond, and LiveCodeBench, which would put a 7B-class model at the front of the 8-9B reasoning pack.","In fast mode the same checkpoint stays within a few points of those rivals while generating far shorter outputs, so one deployed model can serve both low-latency and high-accuracy traffic.","The adaptive mode reduces average output tokens by about 88 percent on GSM8K at nearly unchanged accuracy, implying most efficiency gains come from routing simple queries to fast responses.","Fusion training with replay beats naive co-training across all reported benchmarks, indicating that the sequencing of fast and slow data matters as much as the data themselves.","Domain adaptation on legal data raises LawBench average accuracy from 49.97 to 54.59, suggesting the post-training recipe transfers beyond math and code."],"supporting_citations":[{"why":"Supplies the teacher model and the long-chain-of-thought paradigm that the iterative distillation pipeline distills.","marker":"[12]"},{"why":"Provides the GRPO policy-optimization algorithm used in the reinforcement-learning stage.","marker":"[38]"},{"why":"Defines the principal same-scale baseline (Qwen3-8B) that the headline comparison must beat.","marker":"[50]"},{"why":"Supplies the pretrained base model and tokenizer that Pangu Embedded starts from.","marker":"[55]"},{"why":"Supplies the clip-higher configuration ($\\epsilon=0.28$) adopted in the RL hyperparameters.","marker":"[57]"},{"why":"Provides the ZIP entropy-based compression criterion used to keep the training pool diverse.","marker":"[54]"},{"why":"Defines LiveCodeBench, one of the three headline benchmarks for the comparison.","marker":"[17]"},{"why":"Defines LawBench, the benchmark used for the domain-adaptation extension.","marker":"[8]"}],"fun_headline_variants":["7B model outshines 8-9B rivals with dual fast-slow thinking","One 7B model, two cognitive modes, beats larger rivals on benchmarks","Pangu Embedded cuts tokens 88% while keeping accuracy in fast mode","Metacognition in 7B LLM: fast and slow thinking, higher benchmark scores"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the comparison numbers for Qwen3-8B and GLM-4-9B were measured under equally favorable, comparable settings; some baseline cells in Table 2 come from the authors' own internal evaluation rather than the rivals' official reports.","fun_headline_variants_meta":{"raw":{"variants":["7B model outshines 8-9B rivals with dual fast-slow thinking","One 7B model, two cognitive modes, beats larger rivals on benchmarks","Pangu Embedded cuts tokens 88% while keeping accuracy in fast mode","Metacognition in 7B LLM: fast and slow thinking, higher benchmark scores"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00089,"raw_usage":{"total_tokens":3905,"prompt_tokens":1079,"completion_tokens":2826,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":695,"completion_tokens_details":{"reasoning_tokens":2738}},"tokens_in":695,"tokens_out":2826,"duration_ms":21388,"temperature":1.0,"reasoning_tokens":2738,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:08:48.749837+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the Table 2 comparison under one fixed protocol: identical prompts, identical decoding settings (including sampling temperature and top-p), identical few-shot counts, and at least 500 effective samples per benchmark, for Pangu Embedded and for Qwen3-8B and GLM-4-9B in both modes. If the rival models reproduce or exceed 81.9, 68.0, and 67.1 under those matched conditions, the claimed lead would be a measurement artifact rather than a model property.","supporting_citations":[],"review_version":1}