Sparse LLMs in data-scarce multi-epoch regimes follow a scaling law based on active parameters, unique tokens, repetition count, and sparsity level that predicts performance and delays data saturation.
Title resolution pending
3 Pith papers cite this work. Polarity classification is still indexing.
3
Pith papers citing it
representative citing papers
LLaDA2.0 scales discrete diffusion language models to 100B parameters via systematic conversion from autoregressive models using a 3-phase WSD training scheme and releases open-source 16B and 100B MoE variants.
Introduces a standardized evaluation setup and SQL-D1 agent for diffusion language models on NL2SQL, claiming structural robustness advantages over autoregressive models.
citing papers explorer
-
When Data Is Scarce: Scaling Sparse Language Models with Repeated Training
Sparse LLMs in data-scarce multi-epoch regimes follow a scaling law based on active parameters, unique tokens, repetition count, and sparsity level that predicts performance and delays data saturation.
-
LLaDA2.0: Scaling Up Diffusion Language Models to 100B
LLaDA2.0 scales discrete diffusion language models to 100B parameters via systematic conversion from autoregressive models using a 3-phase WSD training scheme and releases open-source 16B and 100B MoE variants.
-
Are Diffusion Language Models Good Database Analysts?
Introduces a standardized evaluation setup and SQL-D1 agent for diffusion language models on NL2SQL, claiming structural robustness advantages over autoregressive models.