Pith. sign in

REVIEW 12 cited by

JetMoE: Reaching Llama2 Performance with 0.1M Dollars

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2404.07413 v1 pith:K4VSHN72 submitted 2024-04-11 cs.CL cs.AI

Yikang Shen , Zhen Guo , Tianle Cai , Zengyi Qin This is my paper · ORCID
classification cs.CLcs.AI
keywords jetmoe-8bmodeltrainingaccessibledevelopmentefficientjetmoellama2-7b
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
abstract

Large Language Models (LLMs) have achieved remarkable results, but their increasing resource demand has become a major obstacle to the development of powerful and accessible super-human intelligence. This report introduces JetMoE-8B, a new LLM trained with less than $0.1 million, using 1.25T tokens from carefully mixed open-source corpora and 30,000 H100 GPU hours. Despite its low cost, the JetMoE-8B demonstrates impressive performance, with JetMoE-8B outperforming the Llama2-7B model and JetMoE-8B-Chat surpassing the Llama2-13B-Chat model. These results suggest that LLM training can be much more cost-effective than generally thought. JetMoE-8B is based on an efficient Sparsely-gated Mixture-of-Experts (SMoE) architecture, composed of attention and feedforward experts. Both layers are sparsely activated, allowing JetMoE-8B to have 8B parameters while only activating 2B for each input token, reducing inference computation by about 70% compared to Llama2-7B. Moreover, JetMoE-8B is highly open and academia-friendly, using only public datasets and training code. All training parameters and data mixtures have been detailed in this report to facilitate future efforts in the development of open foundation models. This transparency aims to encourage collaboration and further advancements in the field of accessible and efficient LLMs. The model weights are publicly available at https://github.com/myshell-ai/JetMoE.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Monet: Mixture of Monosemantic Experts for Transformers

    cs.AI 2024-12 conditional novelty 8.0 of 10

    Monet composes experts from shared sub-layers to scale to 262,144 experts per layer with parameter growth proportional to the square root of the expert count, yielding more monosemantic and manipulable LLMs.

  2. SHMoAReg: Spark Deformable Image Registration via Spatial Heterogeneous Mixture of Experts and Attention Heads

    cs.CV 2025-09 conditional novelty 6.0 of 10

    A deformable image registration network with mixture-of-attention heads in the encoder and per-voxel, per-direction mixture-of-experts convolutions in the decoder improves abdominal CT Dice from 60.58% to 65.58%.

  3. HierMoE: Accelerating MoE Training with Hierarchical Token Deduplication and Expert Swap

    cs.DC 2025-08 conditional novelty 6.0 of 10

    HierMoE reduces MoE training time by removing duplicate token copies at each GPU-hierarchy level and swapping experts for load balance, measured at 1.18-1.27x end-to-end speedup on 32 GPUs.

  4. Occult: Optimizing Collaborative Communication across Experts for Accelerated Parallel MoE Training and Inference

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Occult co-locates frequently co-activated experts on the same device and optionally prunes routing to fewer devices, cutting all-to-all communication and speeding up MoE training and inference.

  5. Mixture of Sparse Attention: Content-Based Learnable Sparse Attention via Expert-Choice Routing

    cs.LG 2025-05 conditional novelty 6.0 of 10

    MoSA, an expert-choice style sparse attention that selects per-head top-k tokens, outperforms dense transformers on C4 language modeling under matched FLOPs with up to 27% perplexity improvement and reduces wall-clock...

  6. Hymba: A Hybrid-head Architecture for Small Language Models

    cs.CL 2024-11 conditional novelty 6.0 of 10

    Hymba-1.5B, a hybrid-head model with parallel attention and Mamba heads plus learnable meta tokens, achieves state-of-the-art accuracy among sub-2B models and outperforms Llama-3.2-3B with an 11.67x smaller cache and ...

  7. ZUNA1.1: A more flexible EEG foundation model for Denoising and Super-resolution

    cs.LG 2026-07 conditional novelty 5.0 of 10

    ZUNA1.1, an open-source 380M EEG diffusion autoencoder, reconstructs variable-length, flexibly masked EEG at least as well as its predecessor and far better than spherical spline interpolation.

  8. LLM360 K2: Building a 65B 360-Open-Source Large Language Model from Scratch

    cs.LG 2025-01 conditional novelty 5.0 of 10

    K2 Diamond is a fully open 65B-parameter LLM that reaches Llama 2 70B-level performance on standard benchmarks.

  9. LLaMA-MoE v2: Exploring Sparsity of LLaMA from Perspective of Mixture-of-Experts with Post-Training

    cs.CL 2024-11 conditional novelty 5.0 of 10

    A dense instruction-tuned LLaMA-3-8B can be partitioned into Attention-MoE and MLP-MoE and re-trained with about 7B instruction tokens, recovering usable math/code skills with half the activated parameters.

  10. The Evolution of Mixture-of-Experts Architectures in Large Language Models: Routing, Topology, Load Balancing, and Expert Parallelism

    cs.CL 2026-08 conditional novelty 4.0 of 10

    A survey proposes eight architectural milestones and four control planes for MoE LLMs, arguing the field is moving toward decoupling routing, compute budgets, and physical execution.

  11. MoE$^2$: Optimizing Collaborative Inference for Edge Large Language Models

    cs.NI 2025-01 reject novelty 4.0 of 10

    MoE2 trains a gating network once on all edge LLMs, then selects a resource-feasible subset and a per-prompt top-k of experts to fuse token probabilities under latency and energy budgets.

  12. A Survey on Inference Optimization Techniques for Mixture of Experts Models

    cs.LG 2024-12 conditional novelty 3.0 of 10

    A structured survey of MoE inference optimization that categorizes existing techniques into model, system, and hardware levels and summarizes reported speedups and memory savings.

Pith tools