Pith. sign in

REVIEW 2 major objections 5 minor 26 references

A 1.9-billion-parameter open model trained on 2.8 trillion tokens reaches an average benchmark score of 64.92, matching or beating open models several times its size.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 14:49 UTC pith:XS24D2GA

load-bearing objection Solid open SLM report: competitive 1.9B bilingual numbers plus unusually transparent Pure/Boost and schedule×data controls; the mid-training surge is documented rather than hidden. the 2 major comments →

arxiv 2607.09885 v1 pith:XS24D2GA submitted 2026-07-10 cs.CL

Index SLM Technical Report

Lusheng Zhang , Shien He , Tianxing Yan , Mengran Yu , Ziang Cui , Kai Zhao , Xiaojing Liu , Tianjiao Li This is my paper
classification cs.CL
keywords small language modelspre-trainingNorm-HeadWarmup-Stable-Decayinstruction dataChinese-English bilingualrole-playingdirect preference optimization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This report introduces Index-1.9B, a family of open small language models built around a 1.9-billion-parameter base model pre-trained on 2.8 trillion mostly Chinese and English tokens. The authors show that a deeper architecture, a Norm-Head output layer that stabilizes large learning rates, and a Warmup-Stable-Decay schedule that raises the share of curated data during the long decay phase produce a compact model competitive with much larger open models on examination, reasoning, mathematics, and code suites. They release a pure control model with all instruction-like data removed, an aligned chat model, and a retrieval-augmented character model for few-shot role-playing, along with controlled studies of depth, learning-rate size and schedule, data quality in the decay phase, and the inflation of scores from instruction data in pre-training. They also document an unexplained jump in benchmark scores midway through the constant-learning-rate phase. The work matters for anyone who wants strong, efficient bilingual models and transparent evidence about which training choices actually move the numbers.

Core claim

Index-1.9B-Base attains an average score of 64.92 (English average 69.93) across standard examination, reasoning, mathematics, and code benchmarks, competitive with or exceeding open models of several times its size, including outperforming Llama-2-13B on the overall suite. The result is obtained by pre-training on 2.8T tokens with a 36-layer deep design, a Norm-Head that stabilizes a peak learning rate of 5×10⁻⁴, and a Warmup-Stable-Decay schedule whose decay phase (400B tokens) substantially concentrates curated data, optionally with a small fraction of instruction data.

What carries the argument

Warmup-Stable-Decay (WSD) learning-rate schedule coupled with a large late-stage raise in curated-data concentration, plus the Norm-Head output layer that normalizes LM-head weights so training remains stable at high peak learning rates. Together they create a long, high-signal decay regime in which the model continues to improve on benchmarks.

Load-bearing premise

The large late gains are mainly caused by coupling learning-rate decay with higher-quality curated (and optional instruction) data, rather than by the unexplained capability surge already observed under constant learning rate and unchanged data between 1.0 and 1.2 trillion tokens.

What would settle it

Run a matched 1.9B training that keeps the learning rate constant through the final 400 billion tokens while still raising curated-data concentration, then compare final average benchmark score to the decaying-LR run; a large drop supports the coupling claim, while a near-equal score would falsify it.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Open 1.9B-scale models can match or beat much larger open models on exam and reasoning suites when trained with WSD, late curated data, and Norm-Head.
  • Adding only about 7% instruction data in the decay phase can lift MMLU and related scores by roughly seven points, enough to reorder public leaderboards, so pure and boosted releases are needed for fair comparison.
  • At fixed parameter count, a moderately deeper, narrower Transformer outperforms a shallower, wider one on downstream benchmarks.
  • Self-generated safety refusals in preference optimization avoid the high-perplexity and forgetting problems of forcing alignment to human-written refusals.
  • Retrieval over character dialogues lets a 1.9B model support few-shot role-playing customization that ranks with much larger systems on CharacterEval.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The unexplained surge under constant learning rate and fixed mixture between 1.0–1.2T tokens suggests that much of the capability may already be available before any decay phase, so high-quality data under a large constant learning rate alone may be sufficient for a large share of the final score.
  • If the Pure-versus-Boost gap generalizes, leaderboards that do not disclose instruction-like data in pre-training systematically overstate the foundation capability of small models.
  • The same WSD-plus-curated-late-data recipe is a natural candidate for other bilingual or domain-specialized small models where embedding memory is a large fraction of the budget and high peak learning rates are desirable.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. This technical report introduces Index-1.9B, a family of open 1.9B-parameter (non-embedding) small language models pretrained on 2.8T predominantly Chinese–English tokens. The release comprises a base model, a Pure control trained without instruction-like data, a Chat model aligned with SFT and DPO, and a Character model that adds RAG for few-shot role-playing. Pretraining uses a Warmup–Stable–Decay schedule with raised curated-data concentration in the decay phase and a Norm-Head output layer. On an OpenCompass suite (MMLU, C-Eval, CMMLU, HellaSwag, ARC-C/E), Index-1.9B-Base reports an average of 64.92, competitive with larger open models (Table 4). The paper also reports controlled ablations on depth vs. width, learning-rate magnitude and schedules, schedule×data coupling, and the effect of instruction data in decay, and documents an unexplained benchmark surge during the constant-LR phase (Fig. 7, §6.7). Models and evaluation code are released.

Significance. If the reported scores and controls hold under independent re-evaluation, the work is a useful contribution to the open SLM literature: a competitive ~2B bilingual base model, an explicit Pure/Boost pair that quantifies how much instruction data in late pretraining inflates standard benchmarks (Table 8, §6.6), and several design ablations (Norm-Head, depth, LR magnitude, WSD×curated data) run under a shared evaluation suite. Releasing both Pure and Boost, an intermediate constant-LR checkpoint, and evaluation code is a concrete strength for transparent leaderboard comparison and for studying training dynamics. The CharacterEval result (Table 7) further shows that a 1.9B RAG-augmented role-playing model can approach much larger systems on that Chinese benchmark. Mathematics and code remain weaker (Table 5), which the authors acknowledge.

major comments (2)
  1. [§6.5–6.7, Figures 6–7] §6.5–6.7 and Figures 6–7: The design narrative attributes late-stage gains to coupling WSD decay with raised curated (and optionally instruction) data. The same manuscript documents a large, unexplained surge in C-Eval/MMLU between 1.0–1.2T tokens under constant LR and an unchanged natural-text mixture, already before decay. Because the schedule×data ablations are single-run and the surge is not mechanistically accounted for, causal claims that the combination “is what matters” (Fig. 6) are under-determined. The central score claim in Table 4 does not require that causal story, but the discussion should more sharply separate measured outcomes from causal attribution and state what the Pure/Boost and schedule×data runs can and cannot identify.
  2. [§5.1, Table 4] §5.1 and Table 4: For ARC and HellaSwag the authors compute two perplexity protocols and report the higher score. This choice is disclosed but is easy to miss when comparing against numbers taken from other technical reports (many of which use a single fixed protocol). Given that Index-1.9B’s HellaSwag (80.69) is a large driver of the English average and of the gap vs. Pure (62.63), the paper should report both protocol scores side-by-side for Index and, where possible, for key baselines, or re-evaluate a subset of baselines under the same dual-protocol rule so that the 64.92 average is not over-interpreted as protocol-matched.
minor comments (5)
  1. [§6] Ablations in §6 (depth, LR, schedules, Norm-Head) report single trajectories with no multi-seed variance or error bars. A short note on expected run-to-run variability, or at least that conclusions are from single runs, would help readers weight the design recommendations.
  2. [Table 4] Table 4 mixes self-measured OpenCompass numbers with † entries from other reports and includes MiniCPM-SFT among base-model comparisons. A clearer column or footnote separating base vs. instruction-tuned checkpoints would reduce misreading.
  3. [§2.1, Figure 1] Figure 1 pie shares and the 4:5 Chinese–English ratio are useful; stating approximate token counts per major bucket (not only percentages) would aid reproducibility discussions even without full data release.
  4. [§3.2, Appendix B] §3.2 and Appendix B: The preference for self-generated refusals over human-written ones is interesting; a brief quantitative comparison (refusal rate on benign vs. unsafe prompts before/after DPO under both schemes) would strengthen the safety-data claim beyond the qualitative example.
  5. [Throughout / Tables 3, 6] Minor presentation: “INDEX-1.9B-CONSTANT-LR” / “INDEX-1.9B-32K” casing is inconsistent with the Index-1.9B-* naming used elsewhere; unify. Also fix the occasional missing space before citations and the “Instr. Knowledge” column wrapping in Tables 3 and 6.

Circularity Check

0 steps flagged

No circularity: empirical SLM training report with external benchmarks and explicit Pure/Boost controls; no derivation reduces a claimed prediction to a fitted input by construction.

full rationale

Index-1.9B is an engineering and ablation report, not a first-principles derivation. The strongest claim (avg 64.92 on the OpenCompass suite, Table 4) is a direct measurement under a released protocol against public benchmarks (MMLU, C-Eval, CMMLU, HellaSwag, ARC, GSM8K, HumanEval, CharacterEval). Architectural choices (Norm-Head, 36-layer depth, WSD schedule, late-stage curated/instruction data) are justified by controlled ablations (Sections 6.1–6.6, Figures 2–7, Tables 3 and 8) that compare variants on the same external metrics; none of those ablations redefine the target score as a fitted parameter. The Pure vs. Boost pair explicitly isolates instruction-data inflation rather than concealing it. The unexplained 1.0–1.2 T constant-LR surge is documented, not used as a hidden premise. Self-reference to the authors’ larger Index-Large model is limited to conservative data-label bootstrapping and is not load-bearing for any reported score. No uniqueness theorem, ansatz smuggled via self-citation, or renaming of a known result appears. Therefore the circularity score is 0.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 0 invented entities

The central performance claim rests on standard Transformer training assumptions, a handful of hand-chosen hyper-parameters (LR, depth, data ratios), and the authors’ proprietary cleaned corpus. No new physical or mathematical entities are postulated; Norm-Head is adopted from prior work. Free parameters are the usual training knobs whose values were selected by ablation rather than derived.

free parameters (6)
  • peak learning rate (stable phase) = 5e-4
    Set to 5e-4 (decayed to 5e-6); chosen because Norm-Head tolerates large rates; ablated against 2e-4 on 1 B models.
  • number of layers at fixed 1.9 B non-embedding parameters = 36
    Set to 36 after depth-vs-width ablation at 1.01 B; not derived from theory.
  • instruction-data fraction in Boost decay = 7%
    7 % added only in the decay branch; the sole variable between Pure and Boost ablations.
  • curated-data up-weighting window = final 10% / 400B tokens
    Raised substantially over the final ~400 B tokens / final 10 % of training; schedule chosen to coincide with WSD decay.
  • vocabulary size and Chinese max piece length = 65029 / max-piece 5
    65 029 tokens; Chinese BPE max piece length reduced to 5; chosen for memory and compression trade-offs.
  • SFT instruction-token loss share and optimizer-state reuse = ~60% + load opt state
    Best configuration keeps ~60 % instruction tokens in the loss and loads pre-training optimizer state; selected by internal 3-point rubric.
axioms (5)
  • domain assumption Decoder-only Transformer with RoPE, SwiGLU and RMSNorm is a competitive architecture for bilingual SLMs
    Adopted wholesale from Llama conventions (Section 2.3); not re-derived.
  • domain assumption Warmup-Stable-Decay learning-rate schedule permits a long high-LR stable phase followed by rapid late learning
    Taken from MiniCPM (Hu et al., 2024) and validated by the authors’ own schedule ablations (Section 6.4).
  • domain assumption Norm-Head (weight normalization of the LM head) stabilizes training under large learning rates
    Adopted from Baichuan 2; authors confirm higher gradient-norm tolerance and better scores (Section 6.1).
  • domain assumption Standard public benchmarks (MMLU, C-Eval, CMMLU, HellaSwag, ARC, GSM8K, HumanEval) under OpenCompass protocols are valid proxies for general capability
    Used throughout Section 5; no independent validation of contamination or construct validity is supplied.
  • ad hoc to paper Heuristic + conservatively applied classifier cleaning removes harmful or low-quality documents without introducing large selection bias
    Described in Section 2.1; classifiers bootstrapped from the authors’ larger Index-Large model; full rules and residual bias unquantified.

pith-pipeline@v1.1.0-grok45 · 20466 in / 3448 out tokens · 43953 ms · 2026-07-14T14:49:53.773014+00:00 · methodology

0 comments
read the original abstract

We present Index-1.9B, a series of open small language models developed at Bilibili. The series comprises four models: Index-1.9B-Base, a foundation model with 1.9 billion non-embedding parameters pre-trained on 2.8 trillion predominantly Chinese and English tokens; Index-1.9B-Pure, a control variant trained with an identical recipe but with all instruction-like data strictly filtered from the corpus; Index-1.9B-Chat, aligned from the base model with supervised fine-tuning and direct preference optimization; and Index-1.9B-Character, which augments the chat model with retrieval-augmented generation for few-shot role-playing customization. Pre-training employs a Warmup-Stable-Decay learning-rate schedule in which the concentration of curated data is raised substantially during the decay phase, together with a Norm-Head output layer that stabilizes training under large learning rates. On a suite of standard benchmarks covering examination, reasoning, mathematics, and code, Index-1.9B-Base attains an average score of 64.92, competitive with or exceeding open models of several times its size. We further report controlled studies on model depth, learning-rate magnitude and scheduling, the interaction between learning-rate decay and data quality, and the effect of including instruction data during pre-training, and we document an unexplained surge in benchmark performance midway through the constant-learning-rate phase. All models, together with evaluation code, are released at https://github.com/bilibili/Index-1.9B.

Figures

Figures reproduced from arXiv: 2607.09885 by Kai Zhao, Lusheng Zhang, Mengran Yu, Shien He, Tianjiao Li, Tianxing Yan, Xiaojing Liu, Ziang Cui.

Figure 1
Figure 1. Figure 1: Composition of the Index-1.9B pre-training corpus. Substantial effort was devoted to data cleaning, organized around three components. Bias-aware filtering. To avoid introducing avoidable model-induced bias, the great majority of the corpus is cleaned with heuristic rules. Classifiers are trained only for samples that heuristics cannot reliably handle, with labels bootstrapped by annotation from our larger… view at source ↗
Figure 2
Figure 2. Figure 2: Average benchmark score of a 1B model trained on 1T tokens, with and without Norm-Head. Two caveats apply. First, at fixed parameter count, increasing depth increases activation memory, which scales with L × h while parameters and FLOPs scale with L × h 2 (for L layers and hidden size h). Second, we have not yet established at what depth the benefit saturates; we leave this to future work. 200 400 600 800 … view at source ↗
Figure 3
Figure 3. Figure 3: Deep (36-layer) versus wide-and-shallow (9-layer) models at equal parameter count (1.01B non-embedding). 6.3 Learning-rate Magnitude Seemingly mundane hyperparameter choices can have deep effects, and the learning rate is the canonical ex￾ample. Varying only the peak learning rate of a 1B model trained on 1T tokens under a cosine schedule (2×10−4 vs. 5 × 10−4 ), we observe a stable and significant advantag… view at source ↗
Figure 4
Figure 4. Figure 4: Effect of peak learning rate on a 1B model trained on 1T tokens (cosine schedule). 0 50 100 150 200 250 Training step (thousands) 0.2 0.4 0.6 0.8 1.0 Learning rate (×10 −3 ) Cosine Linear WSD [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: The three learning-rate schedules compared in Section 6.4, shown with a peak rate of 10−3 decaying to 10−4 . (c) 2024 Bilibili Index LLM Team. All Rights Reserved. 10 [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: shows that the combination is what matters: WSD with curated data achieves the best score (38.10), exceeding either intervention alone. Interestingly, cosine with curated data scores slightly below plain cosine; we conjecture that the model requires an adaptation period after the distribution shift while the cosine tail leaves too little learning rate for it, and we plan further experiments to test this. C… view at source ↗
Figure 7
Figure 7. Figure 7: MMLU across pre-training. The stable phase (constant learning rate, natural text) is followed by two 50K-step decay branches from the same checkpoint: natural text only, and natural text plus 7% instruction data. Scores are measured on intermediate checkpoints. The unexplained surge at 1.0–1.2T tokens is discussed in Section 6.7. 7 Limitations Throughout training we applied compliance checks, among other m… view at source ↗
Figure 8
Figure 8. Figure 8: shows a knowledge-oriented exchange with Index-1.9B-Chat under the default system prompt. Fig￾ure 9 shows the released role-playing interface, through which users instantiate a custom persona from a small uploaded dialogue corpus [PITH_FULL_IMAGE:figures/full_fig_p015_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: The few-shot role-playing interface of Index-1.9B-Character. Uploading a dialogue corpus and a role description instantiates a new persona. B Safety Preference-pair Construction [PITH_FULL_IMAGE:figures/full_fig_p016_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

26 extracted references · 22 linked inside Pith

  1. [1]

    The Falcon series of open language models.arXiv preprint arXiv:2311.16867,

    Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, et al. The Falcon series of open language models.arXiv preprint arXiv:2311.16867,

    Pith/arXiv arXiv

  2. [2]

    Qwen technical report.arXiv preprint arXiv:2309.16609,

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report.arXiv preprint arXiv:2309.16609,

    Pith/arXiv arXiv

  3. [3]

    DeepSeek LLM: Scaling open-source language models with longtermism.arXiv preprint arXiv:2401.02954,

    Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. DeepSeek LLM: Scaling open-source language models with longtermism.arXiv preprint arXiv:2401.02954,

    Pith/arXiv arXiv

  4. [4]

    (c) 2024 Bilibili Index LLM Team. All Rights Reserved.12 Index Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374,

    Pith/arXiv arXiv 2024

  5. [5]

    Think you have solved question answering? Try ARC, the AI2 reasoning challenge.arXiv preprint arXiv:1803.05457,

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? Try ARC, the AI2 reasoning challenge.arXiv preprint arXiv:1803.05457,

    Pith/arXiv arXiv

  6. [6]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    Pith/arXiv arXiv

  7. [7]

    BTLM-3B-8K: 7B parameter performance in a 3B parameter model.arXiv preprint arXiv:2309.11568,

    Nolan Dey, Daria Soboleva, Faisal Al-Khateeb, Bowen Yang, Ribhu Pathria, Hemant Khachane, Shaheer Muham- mad, Zhiming Myers, Jacob Robert Steeves, Natalia Vassilieva, et al. BTLM-3B-8K: 7B parameter performance in a 3B parameter model.arXiv preprint arXiv:2309.11568,

    Pith/arXiv arXiv

  8. [8]

    Gemma: Open models based on Gemini research and technology.arXiv preprint arXiv:2403.08295,

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on Gemini research and technology.arXiv preprint arXiv:2403.08295,

    Pith/arXiv arXiv

  9. [9]

    Training compute-optimal large lan- guage models.arXiv preprint arXiv:2203.15556,

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large lan- guage models.arXiv preprint arXiv:2203.15556,

    Pith/arXiv arXiv

  10. [10]

    MiniCPM: Unveiling the potential of small language models with scalable training strate- gies.arXiv preprint arXiv:2404.06395,

    Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. MiniCPM: Unveiling the potential of small language models with scalable training strate- gies.arXiv preprint arXiv:2404.06395,

    Pith/arXiv arXiv

  11. [11]

    Mistral 7B.arXiv preprint arXiv:2310.06825,

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7B.arXiv preprint arXiv:2310.06825,

    Pith/arXiv arXiv

  12. [12]

    Scaling laws for neural language models.arXiv preprint arXiv:2001.08361,

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361,

    Pith/arXiv arXiv 2001

  13. [13]

    SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing

    Taku Kudo and John Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71,

    2018

  14. [14]

    All Rights Reserved.13 Index Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini

    (c) 2024 Bilibili Index LLM Team. All Rights Reserved.13 Index Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. InProceedings of the 60th An- nual Meeting of the Association for Computational Linguistics, pages 8424–8445,

    2024

  15. [15]

    CMMLU: Measuring massive multitask language understanding in Chinese.arXiv preprint arXiv:2306.09212,

    Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. CMMLU: Measuring massive multitask language understanding in Chinese.arXiv preprint arXiv:2306.09212,

    Pith/arXiv arXiv

  16. [16]

    LLM360: Towards fully transparent open-source LLMs.arXiv preprint arXiv:2312.06550,

    Zhengzhong Liu, Aurick Qiao, Willie Neiswanger, Hongyi Wang, Bowen Tan, Tianhua Tao, Junbo Li, Yuqi Wang, Suqi Sun, Omkar Pangarkar, et al. LLM360: Towards fully transparent open-source LLMs.arXiv preprint arXiv:2312.06550,

    Pith/arXiv arXiv

  17. [17]

    GPT-4 technical report.arXiv preprint arXiv:2303.08774,

    OpenAI. GPT-4 technical report.arXiv preprint arXiv:2303.08774,

    Pith/arXiv arXiv

  18. [18]

    GLU variants improve transformer.arXiv preprint arXiv:2002.05202,

    Noam Shazeer. GLU variants improve transformer.arXiv preprint arXiv:2002.05202,

    Pith/arXiv arXiv 2002

  19. [19]

    LLaMA: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023a

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Bap- tiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023a. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay...

    Pith/arXiv arXiv

  20. [20]

    All Rights Reserved.14 Index Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin

    (c) 2024 Bilibili Index LLM Team. All Rights Reserved.14 Index Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. InAdvances in Neural Information Processing Systems, volume 30,

    2024

  21. [21]

    Skywork: A more open bilingual foundation model.arXiv preprint arXiv:2310.19341,

    Tianwen Wei, Liang Zhao, Lichang Zhang, Bo Zhu, Lijie Wang, Haihua Yang, Biye Li, Cheng Cheng, Weiwei Lü, Rui Hu, et al. Skywork: A more open bilingual foundation model.arXiv preprint arXiv:2310.19341,

    Pith/arXiv arXiv

  22. [22]

    Baichuan 2: Open large-scale language models.arXiv preprint arXiv:2309.10305,

    Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. Baichuan 2: Open large-scale language models.arXiv preprint arXiv:2309.10305,

    Pith/arXiv arXiv

  23. [23]

    Qwen2 technical report.arXiv preprint arXiv:2407.10671,

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report.arXiv preprint arXiv:2407.10671,

    Pith/arXiv arXiv

  24. [24]

    Tensor programs V: Tuning large neural networks via zero-shot hyperparam- eter transfer.arXiv preprint arXiv:2203.03466,

    Greg Yang, Edward J Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs V: Tuning large neural networks via zero-shot hyperparam- eter transfer.arXiv preprint arXiv:2203.03466,

    Pith/arXiv arXiv

  25. [25]

    CharacterGLM: Customizing Chinese conversational AI characters with large language models.arXiv preprint arXiv:2311.16832,

    Jinfeng Zhou, Zhuang Chen, Dazhen Wan, Bosi Wen, Yi Song, Jifan Yu, Yongkang Huang, Libiao Peng, Jiaming Yang, Xiyao Xiao, et al. CharacterGLM: Customizing Chinese conversational AI characters with large language models.arXiv preprint arXiv:2311.16832,

    Pith/arXiv arXiv

  26. [26]

    Fig- ure 9 shows the released role-playing interface, through which users instantiate a custom persona from a small uploaded dialogue corpus

    A Qualitative Examples Figure 8 shows a knowledge-oriented exchange with Index-1.9B-Chat under the default system prompt. Fig- ure 9 shows the released role-playing interface, through which users instantiate a custom persona from a small uploaded dialogue corpus. Figure 8:A conversation with Index-1.9B-Chat (web demo). The model answers an open-ended ques...

    2024