REVIEW 11 cited by
On the Cross-lingual Transferability of Monolingual Representations
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
State-of-the-art unsupervised multilingual models (e.g., multilingual BERT) have been shown to generalize in a zero-shot cross-lingual setting. This generalization ability has been attributed to the use of a shared subword vocabulary and joint training across multiple languages giving rise to deep multilingual abstractions. We evaluate this hypothesis by designing an alternative approach that transfers a monolingual model to new languages at the lexical level. More concretely, we first train a transformer-based masked language model on one language, and transfer it to a new language by learning a new embedding matrix with the same masked language modeling objective, freezing parameters of all other layers. This approach does not rely on a shared vocabulary or joint training. However, we show that it is competitive with multilingual BERT on standard cross-lingual classification benchmarks and on a new Cross-lingual Question Answering Dataset (XQuAD). Our results contradict common beliefs of the basis of the generalization ability of multilingual models and suggest that deep monolingual models learn some abstractions that generalize across languages. We also release XQuAD as a more comprehensive cross-lingual benchmark, which comprises 240 paragraphs and 1190 question-answer pairs from SQuAD v1.1 translated into ten languages by professional translators.
Forward citations
Cited by 11 Pith papers
-
TyDi QA-WANA: A Benchmark for Information-Seeking Question Answering in Languages of West Asia and North Africa
TyDi QA-WANA is a new 28,000-example QA benchmark covering 10 under-represented languages with long-context, information-seeking questions and baseline evaluations.
-
skLEP: A Slovak General Language Understanding Benchmark
A nine-task Slovak-language understanding benchmark with translated and newly curated datasets, plus the first broad fine-tuned model comparison for Slovak.
-
Judging Quality Across Languages: A Multilingual Approach to Pretraining Data Filtering with Language Models
JQL trains small multilingual quality scorers from LLM judgments and human annotations, and filtering pretraining data with them improves downstream multilingual model performance over heuristic baselines.
-
The UD-NewsCrawl Treebank: Reflections and Challenges from a Large-scale Tagalog Syntactic Annotation Project
The authors manually annotated a 15,619-sentence Tagalog news treebank following Universal Dependencies and evaluated transformer-based dependency parsers against it, plus quality and topic analyses.
-
ALoFTRAG: Automatic Local Fine Tuning for Retrieval Augmented Generation
ALoFTRAG self-generates Q&A from unlabeled RAG texts, filters them with the same local LLM, and LoRA fine-tunes to lift citation accuracy by 8.3% and answer accuracy by 3.0% on average across 26 languages.
-
Neural FOXP2 -- Language Specific Neuron Steering for Targeted Language Improvement in LLMs
A three-stage SAE-plus-SVD steering recipe claims to make Hindi or Spanish the default language of an LLM at inference time, but the visible manuscript reports only expected, not measured, outcomes.
-
Targeted Lexical Injection: Unlocking Latent Cross-Lingual Alignment in Lugha-Llama via Early-Layer LoRA Fine-Tuning
Using Layer 2 embeddings in Lugha-Llama, the paper reports a 28% relative increase in final-layer cosine similarity for Swahili-English pairs, but the supporting layer scan contains an internal contradiction and the c...
-
Transfer of Structural Knowledge from Synthetic Languages
A new synthetic language, flat_shuffle, transfers more structure to English fine-tuning than earlier synthetic bracket languages, though still far short of training on English from scratch.
-
Multilingual Prompt Engineering in Large Language Models: A Survey Across NLP Tasks
A survey that categorizes multilingual prompting techniques by NLP task and language family, and designates potential state-of-the-art prompting methods for each dataset.
-
Beyond English: The Impact of Prompt Translation Strategies across Languages and Tasks in Multilingual LLMs
Selective pre-translation, translating only some prompt components into English, generally outperforms both full prompt translation and direct inference across tasks and languages, with the largest gains for low-resou...
-
IndicSQuAD: A Comprehensive Multilingual Question Answering Dataset for Indic Languages
IndicSQuAD is a large extractive QA dataset for ten Indic languages, translated from SQuAD 2.0, with baseline evaluations using monolingual BERT models and MuRIL-BERT.
Discussion (0). Continue with ORCID to comment.