Pith. sign in

REVIEW 5 major objections 5 minor 5 cited by

BitNet b1.58 2B4T Technical Report

T0 review · 5 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A natively trained 1-bit LLM can match full-precision models of similar size at 2-billion-parameter scale.

desk verdict Valuable open 1-bit 2B model with real efficiency gains, but the parity claim rests on an internally inconsistent Table 1 and single-run benchmarks; deserves peer review after fixing the averages. read the letter →

arxiv 2504.12285 v2 pith:2U7CKZVZ submitted 2025-04-16 cs.CL cs.LG

Shuming Ma , Hongyu Wang , Shaohan Huang , Xingxing Zhang , Ying Hu , Ting Song , Yan Xia , Furu Wei This is my paper · ORCID
classification cs.CLcs.LG
keywords 1-bitLLMternaryweightsBitNetb1.58nativequantizationsmalllanguagemodelsefficientCPUinferencelow-bittrainingParetofrontier
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces BitNet b1.58 2B4T, a 2-billion-parameter language model whose weights are stored as ternary values {-1, 0, +1} (1.58 bits) and whose activations are 8-bit, trained from scratch on 4 trillion tokens. The central claim is that a natively trained 1-bit model, not a post-hoc quantized one, can reach performance comparable to leading open-weight full-precision instruction-tuned models in the 1B-2B range. On a 16-benchmark average it scores 54.19 against 55.23 for the strongest full-precision comparator, while using 0.4 GB of non-embedding memory, 29 ms per token on CPU, and an estimated 0.028 J per token. If the claim holds, full-precision weights are not required for competitive small-scale language models, and capable models become practical on CPUs and memory-limited devices.

What carries the argument

The load-bearing object is the BitLinear layer with absmean weight quantization: weights are mapped to ternary values {-1, 0, +1} by thresholding against the tensor's absolute mean, giving the 1.58-bit representation, while activations are quantized to int8 per token with absmax scaling. Around that core, the model uses subln normalization, squared ReLU in the feed-forward network, rotary position embeddings, and no bias terms, and it is trained with a two-stage learning-rate/weight-decay schedule (a high-learning-rate cosine phase followed by a low-learning-rate cooldown on curated data), then SFT and DPO. This machinery is what converts the efficiency of ternary arithmetic into measured reductions in memory, latency, and energy, and the paper argues the native training is what keeps quality from collapsing.

What would settle it

Re-running the 16 benchmarks several times with different seeds and computing standard errors would settle the central claim: if BitNet's average with error bars overlaps or exceeds Qwen2.5-1.5B's, parity is supported; if repeated runs place it clearly below, or if removing any single benchmark (for example GSM8K, where BitNet leads) changes the ranking, the claimed parity is an artifact of benchmark selection.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that extreme quantization can be baked into training rather than applied afterward: BitNet b1.58 2B4T is the first open-source native 1-bit LLM at 2B scale, and its BitLinear layers quantize weights to {-1, 0, +1} during every forward pass and activations to 8-bit integers per token. The paper reports that this model achieves results on par with leading full-precision open-weight models of similar size across language understanding, math, code, and conversation benchmarks, and that it outperforms both smaller native 1-bit models and much larger post-training-quantized 1.58-bit models (7B and 8B). The efficiency figures are the point: 0.4 GB non-embedding memory versus 2.6 GB for Qwen2.5-1.5B, 29 ms per token versus 65 ms on CPU, and 0.028 J versus 0.347 J estimated energy per token, with only a 1.04-point gap in the benchmark average.

Load-bearing premise

The parity claim rests on a single-run average over 16 benchmarks in which BitNet's 54.19 is only 1.04 points below the top full-precision model's 55.23; if that gap is within run-to-run noise, or if a different benchmark set flips the ordering, parity is not established.

Editorial extensions

If this is right

  • Competitive small language models do not need full-precision weights; training quantization into the model from scratch preserves quality that post-training quantization loses.
  • A 2B-parameter model can run on CPU-only hardware at 29 ms per token with a 0.4 GB non-embedding footprint, opening local deployment on laptops and edge devices.
  • Estimated decoding energy drops by over an order of magnitude (0.028 J vs 0.347 J per token), which changes the economics of high-volume or always-on inference.
  • Native 1-bit training beats post-training 1.58-bit quantization of models several times larger, suggesting that for extreme low-bit regimes, training-time quantization is the more effective use of compute.
  • The open release of weights and inference kernels gives other groups a direct basis for reproducing the efficiency numbers and building on them.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that the benchmark average is measured from single runs; whether the 1.04-point gap to Qwen2.5-1.5B is real parity or run-to-run noise is untested, and a multi-seed evaluation would settle it.
  • If the parity holds at 2B, the natural next test is whether it survives at 7B-13B scale, where full-precision models have more capacity to absorb data; the paper's own scaling-law discussion flags this as open.
  • The efficiency numbers assume 7nm arithmetic energy and custom kernels; on commodity GPUs without native ternary support, the practical speedups may be smaller unless the hardware co-design the paper calls for materializes.
  • A testable extension suggested by the comparison table: PTQ methods on Qwen2.5 lose 3-4 points on average, so a direct comparison of BitNet against a 2-bit or 1.5-bit PTQ of a similarly sized model would probe whether native training is specifically better than better PTQ, not just PTQ as practiced here.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. This technical report presents BitNet b1.58 2B4T, a 2-billion-parameter native 1-bit (ternary-weight) LLM trained from scratch on 4 trillion tokens, followed by SFT and DPO. It claims that the model reaches performance 'on par' with leading open-weight full-precision instruction-tuned models in the 1B-2B range across 16 benchmarks, while reducing non-embedding memory to 0.4 GB, CPU latency to 29 ms per token, and estimated arithmetic energy to 0.028 J. The paper also compares against PTQ INT4 baselines and other 1-bit models, and releases weights plus GPU and CPU inference code.

Significance. BitNet b1.58 2B4T is a potentially significant contribution to efficient LLM deployment: it is, to the authors' credit, an open release of a natively trained 1-bit model at a larger scale than previous open 1-bit efforts, with open-source CPU (bitnet.cpp) and GPU inference implementations. If the efficiency claims are confirmed, the work would provide a strong proof-of-concept that extreme quantization can be trained rather than post-hoc applied, with real deployment advantages. However, the quantitative support for the headline parity claim is currently fragile: the 'Average' row in Table 1 is not reproducible from the displayed rows, the evaluation is single-run with no uncertainty quantification, and the energy and latency metrics rely on self-implemented estimation and runtime comparisons rather than independent measurements. The strengths are the public release and the broad, externally run benchmark evaluation; the weaknesses are load-bearing and require revision.

major comments (5)
  1. [Section 4.1, Table 1] The reported 'Average' row is not reproducible from the displayed 16 benchmark rows. Arithmetic means of the listed per-task scores are LLaMA 3.2 1B 46.30, Gemma-3 1B 45.64, Qwen2.5 1.5B 55.44, SmolLM2 1.7B 49.86, MiniCPM 2B 47.04, and BitNet b1.58 2B 54.19, whereas Table 1 reports 44.90, 43.74, 55.23, 48.70, 42.05, and 54.19. The BitNet average happens to match, but the comparator averages do not, and the parity claim rests on the 1.04-point gap between BitNet (54.19) and Qwen (55.23). The paper must state the exact aggregation rule, for example a particular 11-benchmark subset as suggested by Figure 1, with any normalization, provide the underlying per-task table or code, and reconcile the discrepancy. If an unweighted mean over all displayed rows is intended, the table's average row is internally inconsistent.
  2. [Section 4.1, Table 1] All benchmark scores are single-run point estimates with no seeds, standard errors, confidence intervals, or number of runs. For generative tasks such as HumanEval+, MATH-500, and GSM8K, pass@1 and exact-match scores can vary by several points across decoding seeds, and the aggregate Qwen-vs-BitNet gap is only 1.04 points. Without uncertainty quantification, the claim that the two models are 'on par' is not robustly supported; the authors should report multiple seeds or bootstrap intervals, or at least bound the evaluation noise.
  3. [Section 4.1 and Appendix B (Energy)] The energy values in Table 1 are arithmetic-operation-energy (AOE) estimates for matrix multiplication using the Horowitz (2014) and Zhang et al. (2022) energy model, as described in Appendix B, not measured system energy. In transformer inference, memory-access energy for activations and weights typically dominates arithmetic energy, so the reported 0.028 J for BitNet versus 0.347 J for Qwen2.5-1.5B likely overstates the real energy advantage. The paper should either report direct power measurements, for example RAPL or external metering, or explicitly scope the claim to arithmetic energy and discuss why the omitted memory and activation terms do not change the comparison.
  4. [Section 5.2 and Appendix B (Latency)] The CPU latency comparison is confounded by implementation: BitNet uses the authors' own bitnet.cpp, while all full-precision models are evaluated with llama.cpp, as stated in Appendix B. A 29 ms versus 65 ms difference could reflect kernel maturity or optimizations rather than the model format itself. For a fair efficiency claim, the comparison should be repeated with a common runtime or a third-party implementation of both, and the hardware and software configurations, thread counts, and token lengths should be held fixed.
  5. [Section 4.1 (Benchmark selection)] The 16-benchmark set is introduced without inclusion criteria, weighting, or a sensitivity analysis, and it is not a principled aggregate. Per-task deltas against Qwen2.5-1.5B in Table 1 range from -12.2 (HumanEval+), -9.6 (MATH-500), and -7.08 (MMLU) to +9.1 (WinoGrande) and +1.6 (GSM8K), so the average is highly sensitive to benchmark composition. The paper should justify the benchmark set, report per-category means, and show how the parity conclusion changes under alternative benchmark subsets or weighting schemes.
minor comments (5)
  1. [Tables 1 and 2] Table 1 misspells 'ARC-Challenge' as 'ARC-Challange' and Table 2 has a typo in the AWQ column header; these should be corrected.
  2. [Figure 1] Figure 1's axes are not described in the caption, the x-axis appears to be non-embedding memory in GB, and no error bars or confidence intervals are shown; the caption should define all quantities and note that scores are point estimates.
  3. [Appendix B (Energy)] Appendix B states that energy is computed for a fixed sequence length of 512, but it does not specify the batch size, the treatment of attention, or whether embedding lookups are included; without these details the energy estimate is not fully reproducible.
  4. [Section 7 and overall paper] The manuscript has no explicit limitations section; Section 7 lists future directions such as scaling, multilingual support, and longer context, but it does not discuss the uncertainty in the evaluation or the English-centric data limitation as limitations of the current release.
  5. [References] Some references are formatted inconsistently, for example several arXiv entries lack volume and page information and the Allal et al. entry omits the venue; a final reference cleanup is needed.

Circularity Check

0 steps flagged · score 0.0 of 10

No material circularity: the performance claim rests on external benchmarks and public harnesses, not on self-defined quantities.

full rationale

The paper makes no formal derivation whose inputs and outputs could coincide. Its central claim is an empirical comparison: BitNet b1.58 2B4T is evaluated on 16 external benchmarks against full-precision models using a public evaluation pipeline, as stated in Section 4 and Appendix B. The benchmark tasks, datasets, and scoring procedures are external to the paper, so the reported scores are not constructed from any fitted parameter or prior result of the authors. The architecture section cites prior BitNet work by the same authors, and the CPU latency and energy estimates use implementations and energy models associated with the authors' line of work, but these are the artifacts under evaluation rather than evidence that forces the comparison outcome. No equation is shown to be equivalent to another by construction, and no fitted quantity is renamed as a prediction. The internal inconsistency in Table 1's 'Average' row is a correctness and reproducibility concern about the strength of the parity claim, but it is not a circularity: rechecking that arithmetic does not make the benchmark scores depend on the paper's assumptions. Because the load-bearing claim is independently checkable against external benchmarks, the paper is self-contained with respect to circularity.

Assumptions & free parameters 7 free parameters · 4 assumptions · 0 invented entities

The paper makes no analytical derivation, so there are no fitted constants in the mathematical sense. The entries above are the training hyperparameters and modeling assumptions that the empirical claim depends on. No new entities, fields, forces, or conserved quantities are introduced; the only novelty is a trained model artifact and its inference kernels.

free parameters (7)
  • Stage-1 peak pretraining learning rate
    Section 3.1.1 says the peak was relatively high but never reports the value; the two-stage LR schedule is a central training choice.
  • Stage-2 cooldown learning rate and token midpoint = not reported
    Section 3.1.1: LR is abruptly decayed approximately midway; exact values and token count are omitted, blocking replication.
  • Weight decay peak (Stage 1) = 0.1
    Section 3.1.2: chosen by hand to regularize the high-LR phase.
  • SFT learning rate and epochs = not reported
    Section 3.2.3 says 1-bit models needed larger LR and more epochs, but values are not given.
  • DPO beta = 0.1
    Section 3.3.2: chosen by hand; controls distance from reference policy.
  • DPO learning rate = 2e-7
    Section 3.3.2: chosen by hand.
  • Number of DPO epochs = 2
    Section 3.3.2: chosen by hand.
assumptions (4)
  • domain assumption The 16 publicly available benchmarks used in Table 1 are a valid measure of language understanding, math, code, and conversation for comparing small LLMs.
    Section 4 selects ARC-E, ARC-C, HellaSwag, WinoGrande, PIQA, OpenbookQA, CommonsenseQA, TruthfulQA, MMLU, TriviaQA, BoolQ, GSM8K, MATH-500, HumanEval+, IFEval, and MT-bench; the paper provides no justification that this set or its unweighted average represents capability.
  • domain assumption The Horowitz and Zhang arithmetic-operation energy model applied to matrix multiply at 7nm estimates real decoding energy.
    Appendix B converts ADD and MUL pJ numbers into per-token energy, omitting memory access, activation movement, kernel overhead, and measured device power.
  • domain assumption Absmean ternary weights and per-token int8 activations preserve enough information for the target tasks.
    Section 2 assumes this; the paper's own benchmarks are the only evidence offered, making it a premise rather than a proven theorem.
  • domain assumption Public web corpora (DCLM, FineWeb-EDU) and synthetic math data provide sufficient, uncontaminated pretraining coverage.
    Section 3.1.3 names sources but gives no mixture proportions, deduplication details, or contamination analysis.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BitNet b1.58 2B4T Technical Report." pith.science (2026). https://pith.science/paper/2U7CKZVZ

@misc{pith2026250412285,
  author       = {Pith},
  title        = {Pith review of: BitNet b1.58 2B4T Technical Report},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2U7CKZVZ}},
  note         = {Machine review of arXiv:2504.12285}
}
read the original abstract

We introduce BitNet b1.58 2B4T, the first open-source, native 1-bit Large Language Model (LLM) at the 2-billion parameter scale. Trained on a corpus of 4 trillion tokens, the model has been rigorously evaluated across benchmarks covering language understanding, mathematical reasoning, coding proficiency, and conversational ability. Our results demonstrate that BitNet b1.58 2B4T achieves performance on par with leading open-weight, full-precision LLMs of similar size, while offering significant advantages in computational efficiency, including substantially reduced memory footprint, energy consumption, and decoding latency. To facilitate further research and adoption, the model weights are released via Hugging Face along with open-source inference implementations for both GPU and CPU architectures.

Figures

Figures reproduced from arXiv: 2504.12285 by the authors.

Figure 1
Figure 1. BitNet b1.58 2B4T advances the Pareto frontier defined by leading open-weight LLMs under 3B parameters in terms of performance versus memory, demonstrating superior efficiency. ∗ Equal contribution. ⋄ Corresponding author. S. Ma, S. Huang, X. Zhang, T. Song, Y. Xia and F. Wei are with Microsoft Research. H. Wang is with University of Chinese Academy of Sciences. Y. Hu is with Tsinghua University. arXiv:2504.12285v2 … view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. When Can Depth Replace Precision? A Resource Theory of Quantized Neural Computation

    cs.LG 2026-07 conditional novelty 7.0 of 10 partial

    For a fixed low-bit residual library, the distance to the closed relaxed reachable set is an exact structural floor that pure depth approaches at O(1/D), while write-back arithmetic can reverse the gain and accuracy m...

  2. Attend to Your Own Thoughts: Breaking the Barrier for Post-Training Quantization of Reasoning LLMs through the Lens of 1.58-Bit Quantization

    cs.CL 2026-08 conditional novelty 6.0 of 10

    Using the target model's own self-generated reasoning traces as post-training quantization calibration data lets 1.58-bit ternary LLMs retain math and coding performance, beating a QAT-trained baseline with far fewer tokens.

  3. BitNet Text Embeddings

    cs.CL 2026-06 unverdicted novelty 6.0 of 10

    BITEMBED trains 1.58-bit ternary-weight LLM embedders with contrastive pre-training, supervised distillation, and multi-precision output training, matching FP16 teachers within ~0.6 MMTEB points at ~2x CPU speed.

  4. Ultra-Quantisation: Efficient Embedding Search via 1.58-bit Encodings

    cs.LG 2025-05 conditional novelty 6.0 of 10

    A top-magnitude sign quantisation, grounded in a family of convex polytopes, encodes embeddings as ternary vectors and delivers fast, accurate approximate similarity search.

  5. Transforming Decoder-Only Transformers for Accurate WiFi-Telemetry Based Indoor Localization

    cs.NI 2025-05 reject novelty 5.0 of 10

    Fine-tuned decoder-only LLaMA models predict WiFi transmitter-receiver distance from raw RSSI, FTM, and CSI tokens, with reported sub-meter median errors, though the evaluation lacks baselines and has split-design concerns.

Reference graph

Works this paper leans on

49 extracted references · 26 canonical work pages · cited by 5 Pith papers

  1. [1]

    B., Lozhkov, A., Bakouch, E., Blázquez, G

    Allal, L. B., Lozhkov, A., Bakouch, E., Blázquez, G. M., Penedo, G., Tunstall, L., Marafioti, A., Kydlíček, H., Lajarín, A. P., Srivastav, V., Lochner, J., Fahlgren, C., Nguyen, X.-S., Fourrier, C., Burtenshaw, B., Larcher, H., Zhao, H., Zakka, C., Morlon, M., Raffel, C., von Werra, L., and Wolf, T. (2025). Smollm2: When smol goes big - data-centric train...

    arXiv 2025

  2. [2]

    Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., Zhong, H., Zhu, Y., Yang, M.-H., Li, Z., Wan, J., Wang, P., Ding, W., Fu, Z., Xu, Y., Ye, J., Zhang, X., Xie, T., Cheng, Z., Zhang, H., Yang, Z., Xu, H., and Lin, J. (2025). Qwen2.5-vl technical report. CoRR , abs/2502.13923

    arXiv 2025

  3. [3]

    L., Gao, J., and Choi, Y

    Bisk, Y., Zellers, R., Bras, R. L., Gao, J., and Choi, Y. (2019). PIQA: reasoning about physical commonsense in natural language. CoRR , abs/1911.11641

    arXiv 2019

  4. [4]

    Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. (2019). Boolq: Exploring the surprising difficulty of natural yes/no questions. CoRR , abs/1905.10044

    arXiv 2019

  5. [5]

    Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. (2021). Training verifiers to solve math word problems. CoRR , abs/2110.14168

    arXiv 2021

  6. [6]

    Cui, G., Yuan, L., Ding, N., Yao, G., He, B., Zhu, W., Ni, Y., Xie, G., Xie, R., Lin, Y., Liu, Z., and Sun, M. (2024). ULTRAFEEDBACK: boosting language models with scaled AI feedback. In ICML . OpenReview.net

    work page 2024

  7. [7]

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravankumar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gregerson, A., Spataru, A., Rozière, B., Biron, B., Tang, B., Chern, B., Caucheteux, C., Nayak, C., Bi, C., Marra...

    arXiv 2024

  8. [8]

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. (2021a). Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021

    work page 2021

Show all 49 references
  1. [9]

    Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. (2021b). Measuring mathematical problem solving with the MATH dataset. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS ...

    2021

  2. [10]

    Horowitz, M. (2014). 1.1 computing's energy problem (and what we can do about it). In 2014 IEEE International Conference on Solid-State Circuits Conference, ISSCC 2014, Digest of Technical Papers, San Francisco, CA, USA, February 9-13, 2014 , pages 10--14

    2014

  3. [11]

    Hsu, P.-L., Dai, Y., Kothapalli, V., Song, Q., Tang, S., Zhu, S., Shimizu, S., Sahni, S., Ning, H., and Chen, Y. (2024). Liger kernel: Efficient triton kernels for LLM training. CoRR , abs/2410.10989

    2024 arXiv

  4. [12]

    L., Zhang, K., Wang, C., Yao, Y., Zhao, C., Zhou, J., Cai, J., Zhai, Z., Ding, N., Jia, C., Zeng, G., Li, D., Liu, Z., and Sun, M

    Hu, S., Tu, Y., Han, X., He, C., Cui, G., Long, X., Zheng, Z., Fang, Y., Huang, Y., Zhao, W., Zhang, X., Thai, Z. L., Zhang, K., Wang, C., Yao, Y., Zhao, C., Zhou, J., Cai, J., Zhai, Z., Ding, N., Jia, C., Zeng, G., Li, D., Liu, Z., and Sun, M. (2024). Minicpm: Unveiling the p...

    2024 arXiv

  5. [13]

    S., and Zettlemoyer, L

    Joshi, M., Choi, E., Weld, D. S., and Zettlemoyer, L. (2017). Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July...

    2017

  6. [14]

    Li, H., Dong, Q., Tang, Z., Wang, C., Zhang, X., Huang, H., Huang, S., Huang, X., Huang, Z., Zhang, D., Gu, Y., Cheng, X., Wang, X., Chen, S.-Q., Dong, L., Lu, W., Sui, Z., Wang, B., Lam, W., and Wei, F. (2024a). Synthetic data (almost) from scratch: Generalized instruction tu...

    2024 arXiv

  7. [15]

    Y., Bansal, H., Guha, E., Keh, S

    Li, J., Fang, A., Smyrnis, G., Ivgi, M., Jordan, M., Gadre, S. Y., Bansal, H., Guha, E., Keh, S. S., Arora, K., Garg, S., Xin, R., Muennighoff, N., Heckel, R., Mercat, J., Chen, M. F., Gururangan, S., Wortsman, M., Albalak, A., Bitton, Y., Nezhurina, M., Abbas, A., Hsieh, C.-Y...

    2024

  8. [16]

    Slimorca: An open dataset of gpt-4 augmented flan reasoning traces, with verification

    Lian, W., Wang, G., Goodson, B., Pentland, E., Cook, A., Vong, C., and "Teknium" (2023). Slimorca: An open dataset of gpt-4 augmented flan reasoning traces, with verification

    2023

  9. [17]

    Lin, S., Hilton, J., and Evans, O. (2022). Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022 , pages 3214--3252

    2022

  10. [18]

    S., Wang, Y., and Zhang, L

    Liu, J., Xia, C. S., Wang, Y., and Zhang, L. (2023). Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems , 36:21558--21572

    2023

  11. [19]

    Ma, S., Wang, H., Ma, L., Wang, L., Wang, W., Huang, S., Dong, L., Wang, R., Xue, J., and Wei, F. (2024). The era of 1-bit llms: All large language models are in 1.58 bits. CoRR , abs/2402.17764

    2024 arXiv

  12. [20]

    Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. (2018). Can a suit of armor conduct electricity? A new dataset for open book question answering. CoRR , abs/1809.02789

    2018 arXiv

  13. [21]

    B., Lozhkov, A., Mitchell, M., Raffel, C

    Penedo, G., Kydl \' cek, H., Allal, L. B., Lozhkov, A., Mitchell, M., Raffel, C. A., von Werra, L., and Wolf, T. (2024). The fineweb datasets: Decanting the web for the finest text data at scale. In Globersons, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J. M.,...

    2024

  14. [22]

    D., Ermon, S., and Finn, C

    Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. (2023). Direct preference optimization: Your language model is secretly a reward model. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S., editors, Advances in Neural Inf...

    2023

  15. [23]

    L., Bhagavatula, C., and Choi, Y

    Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. (2020). WinoGrande: an adversarial winograd schema challenge at scale. In AAAI , pages 8732--8740

    2020

  16. [24]

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. CoRR , abs/1707.06347

    2017 arXiv

  17. [25]

    K., Wu, Y., and Guo, D

    Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Zhang, M., Li, Y. K., Wu, Y., and Guo, D. (2024). Deepseekmath: Pushing the limits of mathematical reasoning in open language models. CoRR , abs/2402.03300

    2024 arXiv

  18. [26]

    Shazeer, N. (2020). GLU variants improve transformer. CoRR , abs/2002.05202

    2020 arXiv

  19. [27]

    Su, J., Ahmed, M. H. M., Lu, Y., Pan, S., Bo, W., and Liu, Y. (2024). Roformer: Enhanced transformer with rotary position embedding. Neurocomputing , 568:127063

    2024

  20. [28]

    Talmor, A., Herzig, J., Lourie, N., and Berant, J. (2019). Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techno...

    2019

  21. [29]

    Tang, Z., Zhang, X., Wang, B., and Wei, F. (2024). Mathscale: Scaling instruction tuning for mathematical reasoning. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net

    2024

  22. [30]

    Team, F.-L. (2024). The falcon 3 family of open models

    2024

  23. [31]

    Team, G., Kamath, A., Ferret, J., Pathak, S., Vieillard, N., Merhej, R., Perrin, S., Matejovicova, T., Ram 'e , A., Rivi `e re, M., et al. (2025). Gemma 3 technical report. arXiv preprint arXiv:2503.19786

    2025 arXiv

  24. [32]

    N., Kaiser, L., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. In Guyon, I., von Luxburg, U., Bengio, S., Wallach, H. M., Fergus, R., Vishwanathan, S. V. N., and Garnett, R., editors, Advances in ...

    2017

  25. [33]

    Wang, H., Ma, S., Dong, L., Huang, S., Wang, H., Ma, L., Yang, F., Wang, R., Wu, Y., and Wei, F. (2023a). Bitnet: Scaling 1-bit transformers for large language models. CoRR , abs/2310.11453

    2023 arXiv

  26. [34]

    Wang, H., Ma, S., Huang, S., Dong, L., Wang, W., Peng, Z., Wu, Y., Bajaj, P., Singhal, S., Benhaim, A., Patra, B., Liu, Z., Chaudhary, V., Song, X., and Wei, F. (2022). Foundation transformers. CoRR

    2022

  27. [35]

    Wang, H., Ma, S., Wang, R., and Wei, F. (2024a). Q-sparse: All large language models can be fully sparsely-activated. CoRR , abs/2407.10969

    2024 arXiv

  28. [36]

    Wang, H., Ma, S., and Wei, F. (2024b). Bitnet a4.8: 4-bit activations for 1-bit llms. CoRR , abs/2411.04965

    2024 arXiv

  29. [37]

    Wang, J., Zhou, H., Song, T., Cao, S., Xia, Y., Cao, T., Wei, J., Ma, S., Wang, H., and Wei, F. (2025). Bitnet.cpp: Efficient edge inference for ternary llms. CoRR , abs/2502.11880

    2025 arXiv

  30. [38]

    Wang, L., Ma, L., Cao, S., Zheng, N., Zhang, Q., Xue, J., Miao, Z., Cao, T., , and Yang, Y. (2023b). Ladder: Efficient tensor compilation on customized data format. In OSDI

    2023

  31. [39]

    Xu, C., Sun, Q., Zheng, K., Geng, X., Zhao, P., Feng, J., Tao, C., Lin, Q., and Jiang, D. (2024a). Wizardlm: Empowering large pre-trained language models to follow complex instructions. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Aus...

    2024

  32. [40]

    Xu, Y., Han, X., Yang, Z., Wang, S., Zhu, Q., Liu, Z., Liu, W., and Che, W. (2024b). Onebit: Towards extremely low-bit large language models. In Globersons, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J. M., and Zhang, C., editors, Advances in Neural Informatio...

    2024

  33. [41]

    Xu, Z., Jiang, F., Niu, L., Deng, Y., Poovendran, R., Choi, Y., and Lin, B. Y. (2024c). Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing. CoRR , abs/2406.08464

    2024 arXiv

  34. [42]

    Yadav, V., Bethard, S., and Surdeanu, M. (2019). Quick and (not so) dirty: Unsupervised selection of justification sentences for multi-hop question answering. In EMNLP-IJCNLP

    2019

  35. [43]

    Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Lin, J., Dang, K., Lu, K., Bao, K., Yang, K., Yu, L., Li, M., Xue, M., Zhang, P., Zhu, Q., Men, R., Lin, R., Li, ...

    2024 arXiv

  36. [44]

    Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. (2019). HellaSwag: can a machine really finish your sentence? In Proceedings of the 57th Conference of the Association for Computational Linguistics , pages 4791--4800

    2019

  37. [45]

    Zhang, Y., Zhang, Z., and Lew, L. (2022). PokeBNN: A binary pursuit of lightweight accuracy. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12465--12475. IEEE

    2022

  38. [46]

    Zhao, W., Ren, X., Hessel, J., Cardie, C., Choi, Y., and Deng, Y. (2024). Wildchat: 1m chatgpt interaction logs in the wild. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net

    2024

  39. [47]

    P., Gonzalez, J

    Zheng, L., Chiang, W.-L., Sheng, Y., Li, T., Zhuang, S., Wu, Z., Zhuang, Y., Li, Z., Lin, Z., Xing, E. P., Gonzalez, J. E., Stoica, I., and Zhang, H. (2024). Lmsys-chat-1m: A large-scale real-world LLM conversation dataset. In The Twelfth International Conference on Learning R...

    2024

  40. [48]

    P., Zhang, H., Gonzalez, J

    Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., and Stoica, I. (2023). Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing Systems 36

    2023

  41. [49]

    Zhou, J., Lu, T., Mishra, S., Brahma, S., Basu, S., Luan, Y., Zhou, D., and Hou, L. (2023). Instruction-following evaluation for large language models. CoRR , abs/2311.07911

    2023 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.