REVIEW 16 cited by
SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
We introduce SOLAR 10.7B, a large language model (LLM) with 10.7 billion parameters, demonstrating superior performance in various natural language processing (NLP) tasks. Inspired by recent efforts to efficiently up-scale LLMs, we present a method for scaling LLMs called depth up-scaling (DUS), which encompasses depthwise scaling and continued pretraining. In contrast to other LLM up-scaling methods that use mixture-of-experts, DUS does not require complex changes to train and inference efficiently. We show experimentally that DUS is simple yet effective in scaling up high-performance LLMs from small ones. Building on the DUS model, we additionally present SOLAR 10.7B-Instruct, a variant fine-tuned for instruction-following capabilities, surpassing Mixtral-8x7B-Instruct. SOLAR 10.7B is publicly available under the Apache 2.0 license, promoting broad access and application in the LLM field.
Forward citations
Cited by 16 Pith papers
-
Chain-of-Experts: Unlocking the Communication Power of Mixture-of-Experts Models
Chain-of-Experts replaces one parallel MoE routing step with several sequential expert steps inside a layer, reporting lower loss and memory use in small-scale experiments.
-
Resilient LLM-Empowered Semantic MAC Protocols via Zero-Shot Adaptation and Knowledge Distillation
A hybrid MAC protocol that starts with LLM-generated control messages and switches to a distilled neural model after retraining improves resilience to user-count changes at lower compute than pure LLM inference.
-
Latent Flow Transformer
LFT replaces up to 13 of 24 transformer layers of Pythia-410M with a single flow-based layer trained with Flow Walking, achieving KL 0.736 vs 0.932 for skipping three layers.
-
Fine, I'll Merge It Myself: A Multi-Fidelity Framework for Automated Model Merging
An automated multi-fidelity search framework discovers layer-wise and depth-wise model merging recipes that improve single- and multi-objective LLM reasoning performance without retraining.
-
Unraveling the Capabilities of Language Models in News Summarization
A 20-model benchmark on three news datasets finds GPT-3.5/GPT-4 lead, a few small models are competitive, and three-shot demonstrations with low-quality gold summaries fail to help.
-
Channel Merging: Preserving Specialization for Merged Experts
Channel Merging clusters similar channel parameters from fine-tuned LLMs and reconstructs the selected expert at inference, matching unmerged accuracy with about 53% of the ensemble's parameters.
-
Fine-Grained Behavior Simulation with Role-Playing Large Language Model on Social Media
A new fine-grained benchmark and an observation-and-memory chain-of-thought fine-tuning method for simulating social media user behavior.
-
CHERRY: Compressed Hierarchical Experts with Recurrent Representational Yield
Selective-pivot-token training plus layer-averaging-with-recurrence reportedly gives 2.5x parameter compression on a small Korean LLM, but the efficiency claim lacks its decisive controls and the abstract advertises r...
-
Expanding Foundational Language Capabilities in Open-Source LLMs through a Korean Case Study
A 102B Korean-English model, expanded from Llama 3 70B with LlamaPro and Masked Structure Growth and trained on 194B tokens, scores 64.74 on KMMLU and 83.34 on KorMedMCQA, roughly matching GPT-4 on Korean benchmarks.
-
An Empirical Study of Evaluating Long-form Question Answering
In long-form QA, LLM-based evaluators correlate with human judgments better than ROUGE or BERTScore, but they are biased by answer length, question type, self-reinforcement, and rare-word usage, and fine-grained promp...
-
Speculate, then Collaborate: Fusing Knowledge of Language Models during Decoding
CoSD fuses two LLMs at decoding time: it keeps the draft model's tokens unless a confidence rule or a decision tree over token probabilities says the assistant model's token is better.
-
Large Language Models for Scholarly Ontology Generation: An Extensive Analysis in the Engineering Field
Zero-shot LLMs, especially Claude 3 Sonnet and a fine-tuned 7B Mistral variant, classify semantic relations between engineering research topics with high F1 on the new IEEE-Rel-1K benchmark.
-
ResidualDroppath: Enhancing Feature Reuse over Residual Connections
A two-phase training algorithm alternating droppath steps with frozen-path steps gives modest accuracy improvements on small image datasets, with inconsistent ImageNet results.
-
SCOUT: Teaching Pre-trained Language Models to Enhance Reasoning via Flow Chain-of-Thought
SCOUT combines progressive distillation with a cross-attention module to make recursive latent reasoning work through fine-tuning, yielding up to 1.8% accuracy gains over standard fine-tuning.
-
Efficient Speech Translation through Model Compression and Knowledge Distillation
Iterative decoder pruning plus QLoRA and knowledge distillation compress Qwen2-Audio-7B by up to 50% with 97-100% of teacher translation quality on English-German and English-Chinese.
-
DNA 1.0 Technical Report
DNA 1.0 8B Instruct is an 8-billion-parameter Korean-English model that reports state-of-the-art results on Korean benchmarks through continual pre-training, SLERP merging, DPO, and distillation.
Discussion (0). Continue with ORCID to comment.