Pith. sign in

REVIEW 20 cited by

OpenELM: An Efficient Language Model Family with Open Training and Inference Framework

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2404.14619 v2 pith:OSBNGAQT submitted 2024-04-22 cs.CL cs.AIcs.LG

Sachin Mehta , Mohammad Hossein Sekhavat , Qingqing Cao , Maxwell Horton , Yanzi Jin , Chenfan Sun , Iman Mirzadeh , Mahyar Najibi
show 3 more authors
Dmitry Belenko Peter Zatloukal Mohammad Rastegari
This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords modelopenopenelmlanguagereleasetrainingapplecode
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
abstract

The reproducibility and transparency of large language models are crucial for advancing open research, ensuring the trustworthiness of results, and enabling investigations into data and model biases, as well as potential risks. To this end, we release OpenELM, a state-of-the-art open language model. OpenELM uses a layer-wise scaling strategy to efficiently allocate parameters within each layer of the transformer model, leading to enhanced accuracy. For example, with a parameter budget of approximately one billion parameters, OpenELM exhibits a 2.36% improvement in accuracy compared to OLMo while requiring $2\times$ fewer pre-training tokens. Diverging from prior practices that only provide model weights and inference code, and pre-train on private datasets, our release includes the complete framework for training and evaluation of the language model on publicly available datasets, including training logs, multiple checkpoints, and pre-training configurations. We also release code to convert models to MLX library for inference and fine-tuning on Apple devices. This comprehensive release aims to empower and strengthen the open research community, paving the way for future open research endeavors. Our source code along with pre-trained model weights and training recipes is available at \url{https://github.com/apple/corenet}. Additionally, \model models can be found on HuggingFace at: \url{https://huggingface.co/apple/OpenELM}.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 20 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Crisp: Cognitive Restructuring of Negative Thoughts through Multi-turn Supportive Dialogues

    cs.CL 2025-04 conditional novelty 7.0 of 10

    Fine-tuning on the CRISP dataset lets a 14B open-source model match or beat GPT-4o in human-rated cognitive restructuring dialogues.

  2. FantasyTalking2: Timestep-Layer Adaptive Preference Optimization for Audio-Driven Portrait Animation

    cs.CV 2025-08 unverdicted novelty 6.0 of 10

    A three-part system, Talking-Critic, Talking-NSQ, and TLPO, aligns diffusion portrait animation models to human preferences and improves lip-sync, motion naturalness, and visual quality.

  3. Minifinetuning: Low-Data Generation Domain Adaptation through Corrective Self-Distillation

    cs.LG 2025-05 conditional novelty 6.0 of 10

    MFT, a corrective self-distillation objective, reports 2 to 10 times better specialization-to-degeneralization ratios than standard finetuning across many models and three specialized domains.

  4. Next Token Perception Score: Analytical Assessment of your LLM Perception Skills

    cs.CL 2025-05 conditional novelty 6.0 of 10

    A subspace-overlap score computed from frozen LLM representations and labels is proposed, with a bounding theorem in a linear setting and empirical correlations with linear probe accuracy and LoRA gains.

  5. Factual Knowledge in Language Models: Robustness and Anomalies under Simple Temporal Context Variations

    cs.CL 2025-02 conditional novelty 6.0 of 10

    Across 2,003 popular temporal facts, even the best of 18 language models perfectly distinguishes correct from incorrect years, months, or dates for only 11% of facts.

  6. Towards Responsible Governing AI Proliferation

    cs.CY 2024-12 conditional novelty 6.0 of 10

    The paper proposes a 'Proliferation' paradigm of AI, where small, hidden, augmented, decentralized, and open-weight models challenge compute-centric governance.

  7. MAS-Attention: Memory-Aware Stream Processing for Attention Acceleration on Resource-Constrained Edge Devices

    cs.DC 2024-11 conditional novelty 6.0 of 10

    MAS-Attention pipelines tiled MatMul and softmax workloads onto separate MAC and vector units of edge NPUs, achieving up to 2.75x simulated and 1.76x measured speedups over FLAT while keeping outputs exact.

  8. Crown, Frame, Reverse: Layer-Wise Scaling Variants for LLM Pre-Training

    cs.CL 2025-09 conditional novelty 5.0 of 10

    At 180M parameters and 5B tokens, all layer-wise scaling variants beat the paper's 18-layer uniform baseline, yet the 12-layer uniform baseline remains best.

  9. RefineX: Learning to Refine Pre-training Data at Scale from Expert-Guided Programs

    cs.CL 2025-07 conditional novelty 5.0 of 10

    A deletion-only program refiner, trained on expert end-to-end edits converted via minimum edit distance, improves LLM pretraining data and downstream accuracy.

  10. EdgeLoRA: An Efficient Multi-Tenant LLM Serving System on Edge Devices

    cs.DC 2025-07 conditional novelty 5.0 of 10

    EdgeLoRA combines automatic adapter routing, LRU caching with a memory pool, and grouped LoRA batching to serve thousands of LoRA adapters on edge devices with up to 4x higher throughput than llama.cpp.

  11. AlignHuman: Improving Motion and Fidelity via Timestep-Segment Preference Optimization for Audio-Driven Human Animation

    cs.CV 2025-06 conditional novelty 5.0 of 10

    Timestep-segment preference optimization with separate motion and fidelity LoRAs improves audio-driven human animation quality and allows a 3.3x inference speedup.

  12. Beyond Text Compression: Evaluating Tokenizers Across Scales

    cs.CL 2025-06 conditional novelty 5.0 of 10

    Tokenizer choice matters mostly for multilingual tasks, and 350M-parameter models can predict 2.7B model ranking on translation but not on English benchmarks.

  13. Rethinking Data Mixture for Large Language Models: A Comprehensive Survey and New Perspectives

    cs.CL 2025-05 accept novelty 5.0 of 10

    A survey organizing LLM data mixture methods into offline and online families, with a fine-grained taxonomy based on optimization frameworks.

  14. Lossless Compression of Large Language Model-Generated Text via Next-Token Prediction

    cs.LG 2025-05 conditional novelty 5.0 of 10

    Using an LLM as a next-token predictor with arithmetic coding compresses LLM-generated text about 20x, roughly 4 to 7 times better than Gzip, LZMA, or neural compressors in the paper's benchmarks.

  15. Sigma: Differential Rescaling of Query, Key and Value for Efficient Language Models

    cs.CL 2025-01 conditional novelty 5.0 of 10

    An attention variant that compresses Key heads harder than Value heads and widens Query heads yields up to 33.36% faster long-context attention, powering a system-domain LLM that reportedly outperforms GPT-4 on the ne...

  16. RedPajama: an Open Dataset for Training Large Language Models

    cs.CL 2024-11 conditional novelty 5.0 of 10

    The paper releases open pretraining corpora, RedPajama-V1 and V2, and shows that web-data quality signals can be used to filter V2 into competitive training sets.

  17. Collaborative Inference and Learning between Edge SLMs and Cloud LLMs: A Survey of Algorithms, Execution, and Open Challenges

    cs.DC 2025-07 conditional novelty 4.0 of 10

    A survey that builds a taxonomy of edge-cloud LLM-SLM collaboration for inference and training, claiming to be the first to unify both phases.

  18. TensorSLM: Energy-efficient Embedding Compression of Sub-billion Parameter Language Models on Low-end Devices

    cs.CL 2025-06 conditional novelty 4.0 of 10

    TensorSLM applies per-vector tensor-train SVD to compress SLM token embeddings training-free, showing competitive task performance at roughly 2x embedding compression on Raspberry Pi with an estimated, pre-decoder ene...

  19. Effective and Efficient Schema-aware Information Extraction Using On-Device Large Language Models

    cs.CL 2025-05 conditional novelty 4.0 of 10

    DLISC, a dual-LoRA two-stage schema-aware extraction method with incremental schema caching, reports better F1 and lower latency than three RAG baselines on two IE datasets, though the comparison lacks error bars and code.

  20. Bridging the Safety Gap: A Guardrail Pipeline for Trustworthy LLM Inferences

    cs.AI 2025-02 conditional novelty 3.0 of 10

    A guardrail pipeline combining detection, retrieval grounding, rule-based wrappers, and a repair model is reported to match OpenAI moderation and fix 80.7 percent of hallucinated HaluEval answers.

Pith tools