REVIEW 9 cited by
Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Datasets are foundational to many breakthroughs in modern artificial intelligence. Many recent achievements in the space of natural language processing (NLP) can be attributed to the finetuning of pre-trained models on a diverse set of tasks that enables a large language model (LLM) to respond to instructions. Instruction fine-tuning (IFT) requires specifically constructed and annotated datasets. However, existing datasets are almost all in the English language. In this work, our primary goal is to bridge the language gap by building a human-curated instruction-following dataset spanning 65 languages. We worked with fluent speakers of languages from around the world to collect natural instances of instructions and completions. Furthermore, we create the most extensive multilingual collection to date, comprising 513 million instances through templating and translating existing datasets across 114 languages. In total, we contribute four key resources: we develop and open-source the Aya Annotation Platform, the Aya Dataset, the Aya Collection, and the Aya Evaluation Suite. The Aya initiative also serves as a valuable case study in participatory research, involving collaborators from 119 countries. We see this as a valuable framework for future research collaborations that aim to bridge gaps in resources.
Forward citations
Cited by 9 Pith papers
-
ChiKhaPo: A Large-Scale Multilingual Benchmark for Evaluating Lexical Comprehension and Generation in Large Language Models
ChiKhaPo is an 8-subtask benchmark that measures word-level comprehension and generation in 2,700+ languages and shows state-of-the-art models perform poorly on low-resource languages.
-
PoLoRA: A Preconditioned Orthogonalized LoRA Optimizer
A preconditioned, product-aware spectral optimizer for LoRA reaches Adam's final loss in 1.2–1.7x fewer steps on instruction-tuning, with ~3% per-step overhead.
-
Are we Merging the Right Models? Impact of Expert Training Duration on Model Merging for LLMs
The optimal training duration for expert models before merging depends on the merging method; sparsification-based merging benefits from overtrained experts.
-
The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text
A new 8TB openly-licensed text corpus trains 7B LLMs that are competitive with Llama 1/2, showing that performant models need not depend on unlicensed web data.
-
CC-Tuning: A Cross-Lingual Connection Mechanism for Improving Joint Multilingual Supervised Fine-Tuning
CC-Tuning fuses English feed-forward activations into non-English inputs during multilingual supervised fine-tuning, using a trainable Decision Maker and a least-squares Transform Matrix to simulate the connection at ...
-
Test-Time Scaling with Repeated Sampling Improves Multilingual Text Generation
Repeated sampling with a verifier improves open-ended multilingual generation, and reward-based verifiers are needed for math and code tasks.
-
Maintaining MTEB: Towards Long Term Usability and Reproducibility of Embedding Benchmarks
The MTEB maintainers document their infrastructure for versioning and validating benchmark components, plus a zero-shot score that flags models trained on benchmark tasks.
-
Optimising Language Models for Downstream Tasks: A Post-Training Perspective
A dissertation that repackages the author's previously published papers on continued pre-training, prompt tuning, and instruction modelling into a single narrative.
-
The Multilingual Divide and Its Impact on Global AI Safety
The language gap in AI models creates safety disparities across languages, and closing it requires funding multilingual datasets, transparency, and research.
Discussion (0). Continue with ORCID to comment.