REVIEW 4 cited by
Predicting the Performance of Multilingual NLP Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Recent advancements in NLP have given us models like mBERT and XLMR that can serve over 100 languages. The languages that these models are evaluated on, however, are very few in number, and it is unlikely that evaluation datasets will cover all the languages that these models support. Potential solutions to the costly problem of dataset creation are to translate datasets to new languages or use template-filling based techniques for creation. This paper proposes an alternate solution for evaluating a model across languages which make use of the existing performance scores of the model on languages that a particular task has test sets for. We train a predictor on these performance scores and use this predictor to predict the model's performance in different evaluation settings. Our results show that our method is effective in filling the gaps in the evaluation for an existing set of languages, but might require additional improvements if we want it to generalize to unseen languages.
Forward citations
Cited by 4 Pith papers
-
Analyzing the Effect of Linguistic Similarity on Cross-Lingual Transfer: Tasks and Experimental Setups Matter
Cross-lingual transfer success is best predicted by syntactic similarity for POS tagging and parsing, by trigram overlap for n-gram topic models, and by mBERT pretraining coverage for mBERT-based topic models.
-
SenWiCh: Sense-Annotation of Low-Resource Languages for WiC using Hybrid Methods
The authors release sense-annotated WSD/WiC datasets for ten low-resource languages and report that English-based zero-shot transfer often beats small in-language fine-tuning, while mixed training usually helps.
-
Zero-shot Cross-lingual Transfer Learning with Multiple Source and Target Languages for Information Extraction: Language Selection and Adversarial Training
A weighted typological distance metric predicts zero-shot cross-lingual transfer for information extraction and guides multilingual source-language selection and adversarial training.
-
Analysis of Indic Language Capabilities in LLMs
A desk-research review finds that LLM performance is strongest for Hindi, Bengali, Marathi, Telugu, and Tamil, and recommends prioritizing these five languages for safety benchmarks.
Discussion (0). Continue with ORCID to comment.