{"id":"fb45f39d-b900-42b2-bfd8-5118b119beb8","arxiv_id":"2607.22083","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A 3B-parameter model using layer-reused Transformers and multi-stage RL reports higher scores than 9B and 12B rivals on agentic code, office, and tool-use benchmarks.","lead":"Nanbeige4.2-3B is a 3-billion-parameter language model for agentic use, pretrained from scratch with a Looped Transformer that reuses its layers, then trained with a multi-stage RL recipe. The authors report it beats larger open models (Qwen3.5-9B, Gemma4-12B) on code-, office-, and tool-use agent benchmarks while staying competitive on math and science reasoning.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No contamination audit for code-agent benchmarks: the repository-to-task SFT pipeline (§3.1.1) plausibly overlaps with SWE-bench Suite, so the reported outperformance may reflect memorization rather than general agentic skill.","rationale":"Stress-test reasoning: The paper's strongest claim is comparative. For it to hold, the evaluation must be a faithful measure of agentic skill. The reader's weakest_assumption (benchmark integrity) is correct, and my pass confirms it is load-bearing. I further localize the risk: the data pipeline in §3.1.1 generates SFT tasks from GitHub repositories and patches, which are exactly the same kind of data used to construct SWE-bench Verified/Pro. The paper provides no contamination audit or statement of exclusion at any stage (pretraining, SFT, RL). This is particularly concerning because the reported advantage over Qwen3.5-9B is large (10.5 points on SWE-bench Verified, 14.9 on Terminal-Bench) and the model is only 3B. Memorization is a plausible explanation. The proposed probe (direct patch generation) is a decisive test: if the model can produce gold-like patches without any interaction, then the agentic score does not measure tool-use and planning. I also considered the Looped Transformer's doubled inference compute and the in-house benchmark affiliations; these are legitimate but less central. The absence of a contamination audit is the single most load-bearing gap. The reader's CONDITIONAL verdict is therefore appropriate; no verdict change.","tokens_in":14288,"tokens_out":9634,"duration_ms":99549,"concrete_test":"Run a zero-shot memorization probe on SWE-bench Verified using the released checkpoint: prompt the model with the issue text and the base-commit repository files (no tool access, temperature 0) and ask for a patch. Measure direct pass@1 and the fraction of outputs with >95% similarity to the gold patch. If direct pass@1 exceeds 20%, contamination is plausible and the reported 63.6% agentic score is suspect. If it is below 5%, the memorization concern is largely resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Central claim: Nanbeige4.2-3B outperforms larger models on agentic benchmarks (Table 3). This assumes scores measure transferable agentic skill, not training-data memorization. §3.1.1 (Fig. 2) describes a repository-to-task pipeline that mines GitHub repositories, synthesizes tasks from real patches, and uses them for SFT. SWE-bench Verified/Pro and Terminal-Bench are also derived from real GitHub issues. The paper never states that benchmark repositories or patches were excluded from this pipeline or from the 28T-token pretraining corpus. Because the same class of data is used to build training tasks, there is a concrete risk that the benchmark instances (or near-duplicates) appear in training. The reported 10+ point gaps over larger baselines could then be an artifact of memorization, not evidence of general agentic capability. No decontamination analysis, overlap statistics, or reproduction code is provided. This is the least secure condition for the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Nanbeige4.2-3B, a 3B non-embedding parameter model pretrained from scratch on 28T tokens with a Looped Transformer architecture that reuses a shared layer stack. Post-training combines a three-stage SFT curriculum over reasoning, general, and agentic data with a multi-stage RL pipeline: two-stage RLHF for Think/Non-Think response quality, length-controlled reasoning RL, and agentic RL with action-centric process rewards. The central empirical claim is that the model outperforms substantially larger open models, including Qwen3.5-9B and Gemma4-12B, across general-agent, code-agent, reasoning, and alignment benchmarks (Table 3), and that it transfers to a local personal-assistant setting under the OpenClaw framework (Table 4). The paper also reports base-model improvements from the looped architecture and updated data mix (Table 1).","tokens_in":14618,"tokens_out":3848,"duration_ms":45402,"significance":"If the reported results hold up, this would be a notable demonstration that a compact model can serve as a general-purpose agent across code, office, and tool-use settings while retaining competitive reasoning, with practical implications for local deployment. The paper's strengths include a detailed account of the training recipe, a transparently described data-synthesis pipeline, release of the checkpoint, and same-scaffold comparisons in the OpenClaw evaluation (Table 4). The claimed gains at 3B scale are significant for the agentic-small-model line of work. However, the central evidence is entirely benchmark-based, and the evaluation infrastructure is partly author-controlled: several benchmarks or harnesses are in-house or co-authored by team members, no contamination audit is provided, and no variance information is reported despite 8-run averaging. These issues place the headline outperformance claim on an insecure footing that needs to be addressed before the result can be credited.","major_comments":[{"comment":"The repository-to-task SFT pipeline mines real GitHub repositories, selects reference patches, and reconstructs executable tasks from historical development activities. SWE-bench Verified, SWE-bench Pro, and Terminal-Bench 2.0 are likewise derived from real code repositories/issues. The paper never states that the training pipeline excluded the benchmark repositories, patches, or near-duplicates, nor does it provide any overlap statistics or decontamination analysis. Since the headline outperformance on code-agent benchmarks rests on these numbers, the absence of a contamination audit is load-bearing. A concrete overlap analysis, a release of task provenance, or an evaluation on a held-out, non-GitHub-derived benchmark is required.","section":"§3.1.1, Fig. 2, and Table 3"},{"comment":"Several evaluation components are author-affiliated or in-house: Recruit-Bench is described as 'our in-house benchmark'; GDPval and AgentIF-Oneday are run with 'our in-house harness' and an agent judge; ClawGym (ref. [3]) has current Nanbeige team members among its authors. Combined with LLM judges used for scoring, this creates a risk that favorable scoring conventions, prompt formats, or judge behavior advantage the proposed model. The manuscript should provide exact evaluation prompts, judge outputs or human-audit subsets, and ideally third-party or publicly hosted harness runs for at least the non-author-affiliated benchmarks. Without this, the cross-model comparisons in Table 3 and Table 4 are hard to verify independently.","section":"§3.3.1, §3.3.2, and Appendix B.3"},{"comment":"The paper reports that SWE-bench Verified, SWE-bench Pro, and Terminal-Bench 2.0 results are averaged over 8 independent runs, but Table 3 reports only single point values with no standard deviation, confidence interval, or significance test. On several rows the gaps to the next-best baseline are modest (e.g., OfficeQA-Pro 21.1 vs 15.8, HLE 17.8 vs 14.8, IF-Bench 54.6 vs 55.3). Without variance information it is impossible to know whether the claimed superiority over Qwen3.5-9B and Gemma4-12B is statistically meaningful on those metrics. Per-run scores and significance testing should be reported, especially for the headline agentic benchmarks.","section":"Appendix B.2 and Table 3"},{"comment":"The architecture section claims that training the looped transformer from scratch performs 'significantly better' than upcycling, that a two-pass loop retains approximately 75% of token efficiency, and that deeper loops give only marginal gains. No experimental table, learning curves, or ablations are shown to support these claims. Since the Looped Transformer is a key component of the paper's contribution, these claims should be documented with concrete numbers and controlled comparisons; otherwise the base-model improvement in Table 1 cannot be attributed to the architecture as opposed to the data mix.","section":"§2.1 and Table 1"}],"minor_comments":[{"comment":"Please specify the exact number of layers, hidden width, loop pass count, and inference-time FLOPs/ latency figures; the current prose leaves the configuration underspecified.","section":"§2.1"},{"comment":"Use consistent notation for pass rates: 'Claw-Eval Pass^3' appears with a superscript but is not defined in the table caption or Appendix B; state whether this is pass@3 or another aggregation.","section":"Table 3"},{"comment":"Missing entries for AgentIF-Oneday under Gemma4-E4B and Gemma4-12B are shown as dashes; add a footnote explaining why these baselines were not evaluated.","section":"Table 3"},{"comment":"General inference settings list temperature 0.6, but code-agent evaluations use temperature 1.0. Clarify why the temperature differs and whether the general agent evaluations ever use temperature 1.0.","section":"Appendix B.1 and B.2"},{"comment":"The penalty term uses p_q, the fraction of fully correct responses in the 'current rollout group.' The group size and how p_q is computed are not defined; also, the claim that α<1 guarantees a correct response is always preferred assumes binary base rewards. State this assumption explicitly.","section":"§3.2.3, Eq. (1)"},{"comment":"Recruit-Bench is described as in-house, but no description of its size, task distribution, or construction protocol is provided. At minimum, include a benchmark card or public release to allow scrutiny.","section":"§3.3.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is technically interesting and the training recipe is unusually detailed, but the empirical core is not yet secured. The authors should be asked to provide a contamination audit, variance information, and external or fully released evaluation harnesses before the claim of outperforming larger models can be accepted. I do not see evidence of deliberate manipulation, but the combination of in-house benchmarks, author-affiliated co-authors on ClawGym, and LLM judges makes the evaluation fragile. A major revision with those additions could be sufficient."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know is that this is a solid system report, not a slam dunk. The from-scratch comparison of looped pretraining against upcycling is real evidence, and the length-controlled reasoning RL objective in Eq. (1) is a concrete, usable increment. The checkpoint is released, the data pipelines are described in enough detail to be actionably reproduced, and the internal ablations (loop depth, KV-cache sharing, stage-wise SFT mixtures) are honest. Those are genuine contributions.\n\nThe soft spot is exactly what the stress-test note flags: the headline claim that a 3B model beats Qwen3.5-9B and Gemma4-12B on agentic benchmarks depends on Table 3, and Table 3 is missing variance, relies on in-house harnesses and judge models for GDPRval and AgentIF, uses the team's own ClawGym, and has no contamination audit. The repository-to-task SFT pipeline mines GitHub patches, and nothing in the text says SWE-bench repositories or instances were excluded. Given that the largest gaps over the 9B baseline appear precisely on SWE-bench Verified and Pro, the overlap risk is live, not hypothetical. That doesn't make the paper worthless; it makes the central comparison a hypothesis that needs external confirmation.\n\nProportionately, the architecture and post-training sections are the strong parts; benchmark integrity is the load-bearing weak point. That is fixable in revision with overlap statistics and independent or shared-harness reproduction. I would send this to serious peer review, not desk reject it, but I would want reviewers to push hard on decontamination and variance before any comparative claims are trusted.\n\nThe paper is for people building compact agentic models and anyone working on looped architectures. It deserves a careful read, but cite the released model and the length-control detail, not the benchmark beats.","headline":"A competent system report for a 3B agentic model whose headline benchmark claims rest too heavily on unchecked, partially self-referential evaluation.","tokens_in":15145,"tokens_out":1988,"would_cite":true,"duration_ms":24440,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Nanbeige4.2-3B claims a 3B-parameter model can outdo 9B and 12B models on agentic benchmarks while staying competitive on reasoning.","keywords":["Looped Transformer","compact agentic model","agentic reinforcement learning","trajectory synthesis","tool use","office agent","code agent","local personal assistant"],"falsifier":"An independent replication that re-runs the seven general-agent and three code-agent benchmarks under the stated settings and finds the 3B model no longer ranks first, or a contamination audit showing that synthesized training trajectories overlap with held-out test tasks.","tokens_in":14221,"feed_emoji":"🤖","tokens_out":3832,"duration_ms":37375,"temperature":0.7,"pith_summary":"This paper tries to establish that a compact language model with 3B non-embedding parameters can be a general-purpose agent across software engineering, office work, and complex tool use, without giving up reasoning strength. The recipe combines a Looped Transformer that reuses its layer stack, pre-training from scratch on 28T tokens, execution-grounded data synthesis across three domains, and a four-stage reinforcement-learning pipeline. If the claim holds, it means parameter count is not the main barrier to agentic ability; the data and reward design can unlock it in small models. The evidence is a set of benchmark tables where the 3B model beats larger open models on every reported agent benchmark and on five of six reasoning benchmarks.","feed_headline":"Looped transformer lifts a 3B model past 9B and 12B agents","feed_subtitle":"Trained on 28T tokens with a staged RL recipe, it tops agentic benchmarks while staying competitive on reasoning.","key_machinery":"The Looped Transformer — a transformer that passes hidden states through the same layer stack for a second pass, increasing effective depth at no parameter cost — is the architectural carrier of the argument. Around it, the post-training recipe does the heavy lifting: a three-stage SFT curriculum shifting from reasoning to agentic tokens, turn-level loss masking that keeps recovery context without training on bad turns, two-stage RLHF for think and non-think responses, length-controlled reasoning RL with a difficulty-aware penalty, and agentic RL with action-centric process rewards.","core_discovery":"The central discovery, stated on the paper's own terms, is that a 3B non-embedding model can be a general agent that outperforms substantially larger models (9B and 12B) across code-agent, office-agent, and complex tool-use benchmarks while remaining competitive on mathematical, coding, and scientific reasoning. The authors attribute this to three interacting ingredients: a Looped Transformer that increases effective depth without adding parameters, pre-training from scratch on a 28T-token corpus that includes a small share of agentic trajectories, and a post-training pipeline whose SFT data is produced by closed-loop environment synthesis and whose RL stages progressively stabilize generati","pith_inferences":["The paper's benchmarking rests partly on in-house harnesses and an in-house benchmark; a neutral third-party replication would be the quickest test of the headline claim.","A natural extension would be ablating the loop depth to isolate how much of the gain comes from the architecture versus the data and RL recipe.","The difficulty-aware length penalty suggests a generic method for balancing reasoning effort and cost that could transfer to other small models without modification."],"forward_implications":["A locally deployable 3B model can carry daily-assistant, office-automation, and research workflows that today are assumed to need much larger models.","Agentic capability in small models may come primarily from execution-grounded data and reward design rather than parameter count.","The reported cross-task and cross-mode generalization of RLHF implies that behavior-level regularization can improve reasoning and agentic performance simultaneously.","The closed-loop data-synthesis pipelines suggest a path to keep raising task difficulty as a model improves, potentially extending to other domains.","If the results reproduce, the gap between small and large models in agentic settings is smaller than commonly assumed."],"fun_headline_variants":["3B model with looped transformer tops 9B and 12B on agent tasks","Looped transformer enables 3B model to outdo 12B in agentic tasks","Compact 3B agent beats 9B and 12B rivals on tool-use benchmarks","3B model, trained on 28T tokens, outperforms larger agents","Small model, big agent: 3B beats 12B in agentic tasks"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The reported benchmark scores faithfully compare the models — that they are unbiased, not inflated by training-data overlap with the test tasks, and obtained with equivalent scaffolds and protocols.","fun_headline_variants_meta":{"raw":{"variants":["3B model with looped transformer tops 9B and 12B on agent tasks","Looped transformer enables 3B model to outdo 12B in agentic tasks","Compact 3B agent beats 9B and 12B rivals on tool-use benchmarks","3B model, trained on 28T tokens, outperforms larger agents","Small model, big agent: 3B beats 12B in agentic tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001279,"raw_usage":{"total_tokens":5070,"prompt_tokens":751,"completion_tokens":4319,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":495,"completion_tokens_details":{"reasoning_tokens":4207}},"tokens_in":495,"tokens_out":4319,"duration_ms":28431,"temperature":1.0,"reasoning_tokens":4207,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T05:49:14.700854+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An independent replication that re-runs the seven general-agent and three code-agent benchmarks under the stated settings and finds the 3B model no longer ranks first, or a contamination audit showing that synthesized training trajectories overlap with held-out test tasks.","supporting_citations":[],"review_version":1}