For Hindi vocabulary extension of a 30B LLM, the best embedding initialization is uniform subword averaging with Hindi norm calibration on the input and character-length-weighted averaging on the output, cutting continued pre-training steps roughly 6-fold.
Proceedings of the 2018 conference on empirical methods in natural language processing: System demonstrations , pages=
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.CL 1years
2026 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Beyond Initialization Loss: A Systematic Study of Token Embedding Initialization Strategies for LLM Vocabulary Extension
For Hindi vocabulary extension of a 30B LLM, the best embedding initialization is uniform subword averaging with Hindi norm calibration on the input and character-length-weighted averaging on the output, cutting continued pre-training steps roughly 6-fold.