LangMAP adapts UnigramLM for multilingual use to deliver language-specific tokenization from a shared vocabulary, boosting boundary alignment metrics across natural and programming languages with mixed downstream fine-tuning gains.
Tokenization Impacts Multilingual Language Modeling: Assessing Vocabulary Allocation and Overlap Across Languages
6 Pith papers cite this work, alongside 6 external citations. Polarity classification is still indexing.
6
Pith papers citing it
6
external citations · OpenAlex
fields
cs.CL 6years
2026 6representative citing papers
A new multilingual corpus for Cham, Khmer, and Tay-Nung, plus a script-aware ELECTRA-style recipe, achieves the best topic-classification accuracy among the tested encoders — and shows why lexical-overlap retrieval can mislead low-resource benchmarks.
In compute-optimal regimes, language model parameter count scales proportionally with data bytes rather than tokens, and the optimal compression rate decreases with increasing compute.
Interpreting harmful Discord messages requires integrating external knowledge and extended context, not just local message-level classification; LLMs leverage local context better than humans but still fail on coded language and community-specific references.
Replacing tokens, freezing the corresponding embeddings, and tuning the rest of the model improves NLU performance on low-resource languages compared to full fine-tuning.
SLO-Guard, a crash-aware two-phase autotuner for vLLM serving, achieves no best-latency improvement over random search but demonstrates more consistent budget allocation across 150 trials on Qwen2-1.5B/A100.
citing papers explorer
-
LangMAP: A Language-Adaptive Approach to Tokenization
LangMAP adapts UnigramLM for multilingual use to deliver language-specific tokenization from a shared vocabulary, boosting boundary alignment metrics across natural and programming languages with mixed downstream fine-tuning gains.
-
Echoes Across Vietnam's Highlands, Delta, and Coast: A Multilingual Corpus for Cham, Khmer, and Tay-Nung
A new multilingual corpus for Cham, Khmer, and Tay-Nung, plus a script-aware ELECTRA-style recipe, achieves the best topic-classification accuracy among the tested encoders — and shows why lexical-overlap retrieval can mislead low-resource benchmarks.
-
Compute Optimal Tokenization
In compute-optimal regimes, language model parameter count scales proportionally with data bytes rather than tokens, and the optimal compression rate decreases with increasing compute.
-
Understanding Interpretation Difficulty in Harmful Online Communication: Insights from Cybercrime Communities
Interpreting harmful Discord messages requires integrating external knowledge and extended context, not just local message-level classification; LLMs leverage local context better than humans but still fail on coded language and community-specific references.
-
Modular Monolingual Adaptation using Pretrained Language Models
Replacing tokens, freezing the corresponding embeddings, and tuning the rest of the model improves NLU performance on low-resource languages compared to full fine-tuning.
-
Copy First, Translate Later: Interpreting Translation Dynamics in Multilingual Pretraining
SLO-Guard, a crash-aware two-phase autotuner for vLLM serving, achieves no best-latency improvement over random search but demonstrates more consistent budget allocation across 150 trials on Qwen2-1.5B/A100.