{"id":"781d415a-f572-4888-aa7f-28d01b7a03c0","arxiv_id":"2607.09885","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"low","formal_verification":"none","parameter_count":6,"one_line_summary":"Index-1.9B-Base reaches 64.92 average benchmark score via WSD training with late curated data plus Norm-Head, with open Pure/Boost controls isolating instruction-data inflation.","lead":"Bilibili open-sources Index-1.9B, a 1.9B-parameter language-model family trained on 2.8T mostly Chinese-English tokens that scores 64.92 average on standard exams, reasoning, math and code. The report supplies controlled Pure/Boost ablations that quantify how instruction data and late curated data inflate those scores.","discovery_kind":"extension","skeptic_critique":{"model":"grok-4.5","headline":"No significant objection identified","rationale":"The central claim is an empirical leaderboard result, not a causal mechanism claim. The manuscript already supplies the most important controls (Pure vs Boost, WSD vs cosine × curated data) and openly flags the mid-training surge, so the attribution gap the reader flags is a limitation of interpretation rather than a threat to the reported numbers. Reproducibility of the scores via the released weights and evaluation code is the decisive check; if it holds, ACCEPT remains the right verdict. I therefore leave the reader's verdict and confidence unchanged.","tokens_in":16311,"tokens_out":437,"duration_ms":4951,"concrete_test":"Independently re-evaluate the released Index-1.9B and Index-1.9B-Pure checkpoints on the exact OpenCompass configuration used for Table 4 (5-shot MMLU/C-Eval/CMMLU; 0-shot ARC/HellaSwag with both PPL protocols; generation+extraction for GSM8K/HumanEval). If the average remains within ~1–2 points of 64.92 / 50.61, the headline claim stands.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's strongest claim is the empirical score of Index-1.9B-Base (avg 64.92 on the OpenCompass suite in Table 4), competitive with larger open models. That number is directly measured under a released protocol and is not itself in dispute. The reader's weakest_assumption correctly notes that causal attribution of late-stage gains to WSD+curated/instruction data is under-determined by the unexplained 1.0–1.2 T constant-LR surge (Fig. 7, §6.7) and by the incomplete data release. However, the claim as stated does not require that causal story; the Pure/Boost pair and schedule×data ablations already quantify the inflation transparently, and the authors document the surge rather than hide it. No internal inconsistency or measurement error undermines the reported scores or the utility of the released controls.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"This technical report introduces Index-1.9B, a family of open 1.9B-parameter (non-embedding) small language models pretrained on 2.8T predominantly Chinese–English tokens. The release comprises a base model, a Pure control trained without instruction-like data, a Chat model aligned with SFT and DPO, and a Character model that adds RAG for few-shot role-playing. Pretraining uses a Warmup–Stable–Decay schedule with raised curated-data concentration in the decay phase and a Norm-Head output layer. On an OpenCompass suite (MMLU, C-Eval, CMMLU, HellaSwag, ARC-C/E), Index-1.9B-Base reports an average of 64.92, competitive with larger open models (Table 4). The paper also reports controlled ablations on depth vs. width, learning-rate magnitude and schedules, schedule×data coupling, and the effect of instruction data in decay, and documents an unexplained benchmark surge during the constant-LR phase (Fig. 7, §6.7). Models and evaluation code are released.","tokens_in":16564,"tokens_out":1366,"duration_ms":30290,"significance":"If the reported scores and controls hold under independent re-evaluation, the work is a useful contribution to the open SLM literature: a competitive ~2B bilingual base model, an explicit Pure/Boost pair that quantifies how much instruction data in late pretraining inflates standard benchmarks (Table 8, §6.6), and several design ablations (Norm-Head, depth, LR magnitude, WSD×curated data) run under a shared evaluation suite. Releasing both Pure and Boost, an intermediate constant-LR checkpoint, and evaluation code is a concrete strength for transparent leaderboard comparison and for studying training dynamics. The CharacterEval result (Table 7) further shows that a 1.9B RAG-augmented role-playing model can approach much larger systems on that Chinese benchmark. Mathematics and code remain weaker (Table 5), which the authors acknowledge.","major_comments":[{"comment":"§6.5–6.7 and Figures 6–7: The design narrative attributes late-stage gains to coupling WSD decay with raised curated (and optionally instruction) data. The same manuscript documents a large, unexplained surge in C-Eval/MMLU between 1.0–1.2T tokens under constant LR and an unchanged natural-text mixture, already before decay. Because the schedule×data ablations are single-run and the surge is not mechanistically accounted for, causal claims that the combination “is what matters” (Fig. 6) are under-determined. The central score claim in Table 4 does not require that causal story, but the discussion should more sharply separate measured outcomes from causal attribution and state what the Pure/Boost and schedule×data runs can and cannot identify.","section":"§6.5–6.7, Figures 6–7"},{"comment":"§5.1 and Table 4: For ARC and HellaSwag the authors compute two perplexity protocols and report the higher score. This choice is disclosed but is easy to miss when comparing against numbers taken from other technical reports (many of which use a single fixed protocol). Given that Index-1.9B’s HellaSwag (80.69) is a large driver of the English average and of the gap vs. Pure (62.63), the paper should report both protocol scores side-by-side for Index and, where possible, for key baselines, or re-evaluate a subset of baselines under the same dual-protocol rule so that the 64.92 average is not over-interpreted as protocol-matched.","section":"§5.1, Table 4"}],"minor_comments":[{"comment":"Ablations in §6 (depth, LR, schedules, Norm-Head) report single trajectories with no multi-seed variance or error bars. A short note on expected run-to-run variability, or at least that conclusions are from single runs, would help readers weight the design recommendations.","section":"§6"},{"comment":"Table 4 mixes self-measured OpenCompass numbers with † entries from other reports and includes MiniCPM-SFT among base-model comparisons. A clearer column or footnote separating base vs. instruction-tuned checkpoints would reduce misreading.","section":"Table 4"},{"comment":"Figure 1 pie shares and the 4:5 Chinese–English ratio are useful; stating approximate token counts per major bucket (not only percentages) would aid reproducibility discussions even without full data release.","section":"§2.1, Figure 1"},{"comment":"§3.2 and Appendix B: The preference for self-generated refusals over human-written ones is interesting; a brief quantitative comparison (refusal rate on benign vs. unsafe prompts before/after DPO under both schemes) would strengthen the safety-data claim beyond the qualitative example.","section":"§3.2, Appendix B"},{"comment":"Minor presentation: “INDEX-1.9B-CONSTANT-LR” / “INDEX-1.9B-32K” casing is inconsistent with the Index-1.9B-* naming used elsewhere; unify. Also fix the occasional missing space before citations and the “Instr. Knowledge” column wrapping in Tables 3 and 6.","section":"Throughout / Tables 3, 6"}],"recommendation":"minor_revision","confidential_remarks":"Fit is appropriate for a systems/technical-report track in CL/ML venues that publish open model reports. Novelty is primarily empirical and engineering (recipe + transparent Pure/Boost control + documented surge), not a new algorithm. I would not block on that if the venue accepts technical reports of this type. The dual-PPL “take the max” choice and the large Boost vs. Pure gap are the main points where a careful reader could over-read the headline 64.92; the minor-revision requests above are aimed at that, not at rejecting the work. No integrity red flags; the authors’ decision to release Pure and to plot the unexplained surge is a positive signal."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful part of this paper is not a new architecture. It is a competitive open 1.9B Chinese–English base (avg 64.92 on the OpenCompass suite, beating several larger open models) plus two things the field usually omits: a Pure vs Boost pair that isolates ~7 points of MMLU-style inflation from 7% instruction data in the decay phase, and an explicit WSD + late curated-data upweighting ablation. They also ship weights, an intermediate constant-LR checkpoint, and evaluation code.\n\nWhat they do well is the controlled branching. Pure and Boost share the same stable-phase checkpoint; the schedule×data 2×2 is clean; depth-vs-width, Norm-Head, and peak-LR ablations are reported on the same suite. Math and code are weak and they say so. The unexplained jump at 1.0–1.2T tokens under constant LR and fixed mixture is plotted and left unexplained rather than papered over. That honesty is rare.\n\nSoft spots, in proportion: causal credit for the late gains is under-determined by that earlier surge, so the WSD+curated story is suggestive rather than airtight. No multi-seed error bars, incomplete data release, and the usual free parameters (peak LR, layer count, instruction fraction, etc.). None of that invalidates the measured scores or the utility of the Pure/Boost control. Novelty is incremental (WSD from MiniCPM, Norm-Head from Baichuan 2); the contribution is the transparent measurement package for this size class.\n\nThis is for people training or evaluating bilingual SLMs who want a reproducible baseline and a clear warning about instruction-data inflation. It is not foundational theory. I would send it to peer review; the evidence is empirical, the controls are real, and the community gets usable artifacts. Worth a reading-group slot if anyone is currently training under 3B.","headline":"Solid open SLM report: competitive 1.9B bilingual numbers plus unusually transparent Pure/Boost and schedule×data controls; the mid-training surge is documented rather than hidden.","tokens_in":17167,"tokens_out":501,"would_cite":true,"duration_ms":5392,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"A 1.9-billion-parameter open model trained on 2.8 trillion tokens reaches an average benchmark score of 64.92, matching or beating open models several times its size.","keywords":["small language models","pre-training","Norm-Head","Warmup-Stable-Decay","instruction data","Chinese-English bilingual","role-playing","direct preference optimization"],"falsifier":"Run a matched 1.9B training that keeps the learning rate constant through the final 400 billion tokens while still raising curated-data concentration, then compare final average benchmark score to the decaying-LR run; a large drop supports the coupling claim, while a near-equal score would falsify it.","tokens_in":17219,"feed_emoji":"🤖","tokens_out":1132,"duration_ms":27972,"temperature":0.7,"pith_summary":"This report introduces Index-1.9B, a family of open small language models built around a 1.9-billion-parameter base model pre-trained on 2.8 trillion mostly Chinese and English tokens. The authors show that a deeper architecture, a Norm-Head output layer that stabilizes large learning rates, and a Warmup-Stable-Decay schedule that raises the share of curated data during the long decay phase produce a compact model competitive with much larger open models on examination, reasoning, mathematics, and code suites. They release a pure control model with all instruction-like data removed, an aligned chat model, and a retrieval-augmented character model for few-shot role-playing, along with controlled studies of depth, learning-rate size and schedule, data quality in the decay phase, and the inflation of scores from instruction data in pre-training. They also document an unexplained jump in benchmark scores midway through the constant-learning-rate phase. The work matters for anyone who wants strong, efficient bilingual models and transparent evidence about which training choices actually move the numbers.","feed_headline":"1.9B open model hits 64.92 avg, rivals far larger models","feed_subtitle":"WSD schedule, curated late data, and Norm-Head let a compact bilingual model match models several times its size","key_machinery":"Warmup-Stable-Decay (WSD) learning-rate schedule coupled with a large late-stage raise in curated-data concentration, plus the Norm-Head output layer that normalizes LM-head weights so training remains stable at high peak learning rates. Together they create a long, high-signal decay regime in which the model continues to improve on benchmarks.","core_discovery":"Index-1.9B-Base attains an average score of 64.92 (English average 69.93) across standard examination, reasoning, mathematics, and code benchmarks, competitive with or exceeding open models of several times its size, including outperforming Llama-2-13B on the overall suite. The result is obtained by pre-training on 2.8T tokens with a 36-layer deep design, a Norm-Head that stabilizes a peak learning rate of 5×10⁻⁴, and a Warmup-Stable-Decay schedule whose decay phase (400B tokens) substantially concentrates curated data, optionally with a small fraction of instruction data.","pith_inferences":["The unexplained surge under constant learning rate and fixed mixture between 1.0–1.2T tokens suggests that much of the capability may already be available before any decay phase, so high-quality data under a large constant learning rate alone may be sufficient for a large share of the final score.","If the Pure-versus-Boost gap generalizes, leaderboards that do not disclose instruction-like data in pre-training systematically overstate the foundation capability of small models.","The same WSD-plus-curated-late-data recipe is a natural candidate for other bilingual or domain-specialized small models where embedding memory is a large fraction of the budget and high peak learning rates are desirable."],"forward_implications":["Open 1.9B-scale models can match or beat much larger open models on exam and reasoning suites when trained with WSD, late curated data, and Norm-Head.","Adding only about 7% instruction data in the decay phase can lift MMLU and related scores by roughly seven points, enough to reorder public leaderboards, so pure and boosted releases are needed for fair comparison.","At fixed parameter count, a moderately deeper, narrower Transformer outperforms a shallower, wider one on downstream benchmarks.","Self-generated safety refusals in preference optimization avoid the high-perplexity and forgetting problems of forcing alignment to human-written refusals.","Retrieval over character dialogues lets a 1.9B model support few-shot role-playing customization that ranks with much larger systems on CharacterEval."],"fun_headline_variants":["Index-1.9B hits 64.92 avg, matches models several times larger","1.9B open model reaches 64.92, tops some 13B peers","Index-1.9B-Base: 64.92 suite avg via WSD and Norm-Head","Compact bilingual 1.9B scores 64.92 across exam and code","Open 1.9B with deep design and curated decay hits 64.92"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The large late gains are mainly caused by coupling learning-rate decay with higher-quality curated (and optional instruction) data, rather than by the unexplained capability surge already observed under constant learning rate and unchanged data between 1.0 and 1.2 trillion tokens.","fun_headline_variants_meta":{"raw":{"variants":["Index-1.9B hits 64.92 avg, matches models several times larger","1.9B open model reaches 64.92, tops some 13B peers","Index-1.9B-Base: 64.92 suite avg via WSD and Norm-Head","Compact bilingual 1.9B scores 64.92 across exam and code","Open 1.9B with deep design and curated decay hits 64.92"]},"model":"grok-4.5","effort":"low","cost_usd":0.005454,"raw_usage":{"total_tokens":1583,"prompt_tokens":915,"num_sources_used":0,"completion_tokens":119,"cost_in_usd_ticks":54540000,"prompt_tokens_details":{"text_tokens":915,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":549,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":915,"tokens_out":119,"duration_ms":4407,"temperature":1.0,"reasoning_tokens":549,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-14T14:49:53.773014+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Run a matched 1.9B training that keeps the learning rate constant through the final 400 billion tokens while still raising curated-data concentration, then compare final average benchmark score to the decaying-LR run; a large drop supports the coupling claim, while a near-equal score would falsify it.","supporting_citations":[],"review_version":1}