Pith. sign in

REVIEW 5 cited by

SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2312.15166 v3 pith:4HSQ7NMH submitted 2023-12-23 cs.CL cs.AIcs.LG

Dahyun Kim , Chanjun Park , Sanghoon Kim , Wonsung Lee , Wonho Song , Yunsu Kim , Hyeonwoo Kim , Yungi Kim
show 10 more authors
Hyeonju Lee Jihoo Kim Changbae Ahn Seonghoon Yang Sukyung Lee Hyunbyung Park Gyoungjin Gim Mikyoung Cha Hwalsuk Lee Sunghun Kim
This is my paper
classification cs.CLcs.AIcs.LG
keywords scalingsolarlanguagellmsup-scalingdeptheffectiveefficiently
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We introduce SOLAR 10.7B, a large language model (LLM) with 10.7 billion parameters, demonstrating superior performance in various natural language processing (NLP) tasks. Inspired by recent efforts to efficiently up-scale LLMs, we present a method for scaling LLMs called depth up-scaling (DUS), which encompasses depthwise scaling and continued pretraining. In contrast to other LLM up-scaling methods that use mixture-of-experts, DUS does not require complex changes to train and inference efficiently. We show experimentally that DUS is simple yet effective in scaling up high-performance LLMs from small ones. Building on the DUS model, we additionally present SOLAR 10.7B-Instruct, a variant fine-tuned for instruction-following capabilities, surpassing Mixtral-8x7B-Instruct. SOLAR 10.7B is publicly available under the Apache 2.0 license, promoting broad access and application in the LLM field.

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Chain-of-Experts: Unlocking the Communication Power of Mixture-of-Experts Models

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Chain-of-Experts replaces one parallel MoE routing step with several sequential expert steps inside a layer, reporting lower loss and memory use in small-scale experiments.

  2. CHERRY: Compressed Hierarchical Experts with Recurrent Representational Yield

    cs.CL 2026-06 unverdicted novelty 5.0 of 10

    Selective-pivot-token training plus layer-averaging-with-recurrence reportedly gives 2.5x parameter compression on a small Korean LLM, but the efficiency claim lacks its decisive controls and the abstract advertises r...

  3. Expanding Foundational Language Capabilities in Open-Source LLMs through a Korean Case Study

    cs.CL 2025-09 conditional novelty 5.0 of 10

    A 102B Korean-English model, expanded from Llama 3 70B with LlamaPro and Masked Structure Growth and trained on 194B tokens, scores 64.74 on KMMLU and 83.34 on KorMedMCQA, roughly matching GPT-4 on Korean benchmarks.

  4. SCOUT: Teaching Pre-trained Language Models to Enhance Reasoning via Flow Chain-of-Thought

    cs.AI 2025-05 conditional novelty 4.0 of 10

    SCOUT combines progressive distillation with a cross-attention module to make recursive latent reasoning work through fine-tuning, yielding up to 1.8% accuracy gains over standard fine-tuning.

  5. Efficient Speech Translation through Model Compression and Knowledge Distillation

    cs.CL 2025-05 conditional novelty 4.0 of 10

    Iterative decoder pruning plus QLoRA and knowledge distillation compress Qwen2-Audio-7B by up to 50% with 97-100% of teacher translation quality on English-German and English-Chinese.

Pith tools