Y-NQ: English-Yor\`ub\'a Evaluation dataset for Open-Book Reading Comprehension and Text Generation

Christophe Ropers; Eduardo S\'anchez; Ifeoluwanimi Adebara; Joe Chuang; Joy Chen; Marta R. Costa-juss\`a

REVIEW 4 major objections 5 minor 16 references

Y-NQ: English-Yor\`ub\'a Evaluation dataset for Open-Book Reading Comprehension and Text Generation

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An English–Yorùbá evaluation dataset shows that current large language models answer open-book reading-comprehension questions worse in Yorùbá than in English, despite Yorùbá documents being far shorter and the task therefore easier.

desk verdict A useful Yorùbá QA dataset whose headline 2.5x disparity claim rests on a tiny, inconsistently described subset and on unvalidated ROUGE scores. read the letter →

arxiv 2412.08279 v1 pith:R7RBKLJG submitted 2024-12-11 cs.CL

Marta R. Costa-jussà , Joy Chen , Ifeoluwanimi Adebara , Joe Chuang , Christophe Ropers , Eduardo Sánchez This is my paper · ORCID

classification cs.CL

keywords Yorubareadingcomprehensionopen-bookquestionansweringtextgenerationlow-resourcelanguagesmultilingualevaluationlargelanguagemodelsROUGE

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces Y-NQ, a parallel English–Yorùbá dataset for open-book reading comprehension and text generation, built from Wikipedia pages paired with Natural Questions. It asks whether the reading-comprehension abilities of current English-centric large language models transfer to Yorùbá, a low-resource language. Across GPT-4o, o1-mini, and Llama-3.1-8B, the answer is no: Yorùbá scores are consistently lower on ROUGE-1/2/L even though Yorùbá documents average about 430 words versus roughly 10,000 for English, which should make the task easier. On a small matched-length subset, Yorùbá performance is about 2.5 times lower on ROUGE-2, and Yorùbá scores collapse for documents over 1,500 words while English does not. The paper positions the dataset as a reusable benchmark for testing whether multilingual capabilities extend to Yorùbá.

What carries the argument

The central object is the Y-NQ dataset itself: a curated parallel set of English and Yorùbá Wikipedia documents, each paired with the same questions and with field-level annotations that include the Yorùbá question, short and long answers, a rewrite flag, paragraph location, and a literal-semantic alignment flag. The evaluation machinery is an open-book prompting protocol plus ROUGE-1/2/L scoring, with document-length bucketing and a six-document matched-length subset to control for the confound that Yorùbá documents are shorter. The comparison logic is the load-bearing mechanism: since shorter documents should make answering easier, consistently lower Yorùbá scores under the same protocol constitute evidence of a genuine capability gap rather than task difficulty.

What would settle it

Randomly sample about 50 Y-NQ questions, have bilingual annotators write two or three acceptable Yorùbá answers per question, and recompute ROUGE-L against the best-matching reference; if Yorùbá scores rise to English parity, the reported gap is a single-reference scoring artifact, while if they stay lower, the comprehension deficit is real.

Watch

Extended reading notes

Core claim

Y-NQ contains 358 question–answer pairs covering 338 English and 208 Yorùbá Wikipedia articles, with English documents averaging 10,363 words and Yorùbá documents 430 words. The authors compare three large language models under a uniform open-book prompt: read the passage and answer the question in a single paragraph using only the passage. ROUGE evaluation shows Yorùbá behind English for every model and metric, for example GPT-4o ROUGE-1 0.39 versus 0.34, ROUGE-2 0.23 versus 0.19, and ROUGE-L 0.30 versus 0.27; Llama-3.1-8b scores 0.31 versus 0.20 on ROUGE-1. Because Yorùbá documents are much shorter, the task is easier for Yorùbá, so the deficit is evidence that English-oriented reading comprehension does not extend to Yorùbá. Length-bucket analysis shows Yorùbá performance drops sharply for documents around 1,500 words while English stays nearly flat, and on six matched topics with near-equal average length, English ROUGE-2 is more than 2.5 times higher (0.23 versus 0.09). A by-product of annotation was the discovery of 26 incorrect English answers in the source Wikipedia material, underscoring that cross-lingual reference quality matters.

Load-bearing premise

The central claim assumes that ROUGE scores against a single reference answer measure reading-comprehension quality in Yorùbá as faithfully as in English, even though Yorùbá's morphology and word order allow many correct answers that share few n-grams with the reference.

Editorial extensions

If this is right

Open-book benchmarks in low-resource languages need controlled or matched document lengths; otherwise shorter documents can mask or amplify the true capability gap.
Developers evaluating multilingual LLMs should expect that high-resource-language reading comprehension does not automatically transfer to Yorùbá, and should test with generative open-book tasks rather than span or multiple-choice only.
The 1,500-word degradation point gives a concrete target: improving long-context handling for Yorùbá may be a separate problem from overall language capability.
Y-NQ provides a reusable test set for future models; a model that closes the Yorùbá–English gap despite Yorùbá's shorter documents would demonstrate genuine cross-lingual reading comprehension.
The annotation finding that 26 English Wikipedia answers were incorrect for Yorùbá-related content indicates that cross-lingual reference sets need verification, and that model errors are not the only source of mismatch.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The sharp Yorùbá drop beyond 1,500 words may stem from context-window degradation interacting with sparser token representations; this could be tested by measuring accuracy on Yorùbá passages of increasing length against English passages of the same length, and by probing where in the passage the needed answer sits.
ROUGE's n-gram overlap could systematically under-count correct Yorùbá answers, since Yorùbá is morphologically rich and permits flexible word order; a human rating pass or a semantic-similarity metric on the same outputs could change the size of the reported gap even if not its direction.
If the gap is confirmed by human evaluation, the likely mechanism is pretraining data imbalance, which would predict that the gap narrows for languages with more pretraining tokens and widens for morphologically richer low-resource languages; Y-NQ-style datasets for other African languages could test this.
The 26 wrong English Wikipedia answers imply that part of the English–Yorùbá gap might be due to reference errors; extending the same annotation pass to Yorùbá references could reveal whether some Yorùbá failures are actually correct answers to bad or incomplete references.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Y-NQ is a genuinely useful resource: 358 human-annotated open-book QA pairs for Yorùbá built on NQ, with parallel English documents. If you work on multilingual evaluation, this is worth having. The annotation pipeline is described in useful detail, including the failed SONAR pre-annotation attempt, cleaning of English-contaminated Wikipedia pages, and the discovery of 26 incorrect English Wikipedia answers. The paper is honest about its limits: small size, Wikipedia domain, contamination risk, and the non-comparability of document lengths across languages.

The main experimental claim is the problem. The abstract's '2.5x drop' comes from a subset described as six documents in one place and four in another. The ROUGE ratios in Table 5 are 1.41x for R-1, 1.58x for R-L, and 2.56x for R-2, so the 2.5x is the maximum metric, not a central estimate. With four to six documents, one anomalous Yorùbá document can dominate. That claim should be re-reported with per-question paired scores and confidence intervals, or dropped in favor of the full-dataset result, which is more robust: consistent Yorùbá underperformance of roughly 0.05–0.15 ROUGE points across all three models.

The other soft spot is ROUGE itself. No human evaluation is actually reported, despite the Table 4 caption mentioning 'Human Score.' For Yorùbá, ROUGE-1/2/L against a single reference can under-score valid answers that use different word forms or word order. The limitations section acknowledges human evaluation as a possible complement, but it is not included. This leaves the central disparity partly a measurement concern. It is fixable: a small human judgment sample or an analysis of ROUGE failure modes would help.

None of this undermines the dataset as a contribution. The direction of the finding likely holds—English LLMs do worse in Yorùbá even with shorter documents—but the paper oversells the strength of the evidence. A careful revision that anchors the conclusion in the full-dataset numbers, reports the comparable-length subset with explicit counts and error bars, and adds a small human validation of ROUGE would make this a solid resource paper.

Recommendation: send it to review. With revisions, this is citable and useful.

Referee Report

4 major / 5 minor

Summary. The paper introduces Y-NQ, an English–Yorùbá dataset for open-book reading comprehension and generative question answering, built by extending Natural Questions with 358 human-annotated question–answer pairs over 338 English and 208 Yorùbá Wikipedia documents. The authors evaluate GPT-4o, o1-mini, and Llama-3.1-8b using ROUGE scores, report consistently lower automatic scores for Yorùbá than English, and further claim that on a small set of comparable-length documents Yorùbá performance drops by a factor of 2.5. The paper also reports an annotation by-product: 26 incorrect English Wikipedia answers identified during annotation. The main contribution is the dataset itself, and the experiments are presented as evidence that current English-oriented LLMs do not extend their reading comprehension capabilities to Yorùbá.

Significance. If the central claim holds, Y-NQ is a useful resource for multilingual open-book reading comprehension: it is human-annotated, provides document-level context, includes parallel English–Yorùbá documents on shared topics, and is one of few datasets targeting Yorùbá generative QA rather than multiple-choice span retrieval. The documentation of annotation guidelines, the detailed pre-annotation pipeline, and the finding about English Wikipedia inaccuracies are also valuable. However, the experimental evidence for the headline claim is currently fragile: the strongest quantitative statement relies on at most six documents, and the automatic metric (ROUGE) is not validated for Yorùbá. The dataset and the qualitative direction of the results are solid enough to merit revision, but the paper should not be accepted in its current form without strengthening or carefully re-scoping the performance claims.

major comments (4)

[Abstract; §3, Table 5] The headline claim that 'performance of Yorùbá drops by x2.5 times' is not supported by the reported evidence. Table 5 shows ROUGE-1 and ROUGE-L ratios of 1.41x and 1.58x; only ROUGE-2 gives 2.56x (0.23 vs 0.09). Selecting the maximum ratio across metrics is not a robust central estimate, and the comparison is computed at the document level on six documents, with the text elsewhere referring to 'only 4 documents that are over 900 words long' and Table 5 labeled as 'six comparable English and Yorùbá documents.' The authors should either re-report the comparison per question with counts and confidence intervals, or de-emphasize the x2.5 statement and make the full-dataset results in Table 4 the primary evidence for the disparity.
[§3, Evaluation and Table 4] The paper uses ROUGE-1, ROUGE-2, and ROUGE-L as the sole automatic metrics for Yorùbá without any validation that these metrics track answer correctness in a morphologically rich language. Since the conclusion that 'reading comprehension capabilities of current English LLMs do not extend to Yorùbá' depends on this assumption, the authors should provide either a human evaluation of model outputs, a cross-lingual metric validation (e.g., correlation with manual judgments on a sample), or a clear statement that the results are about n-gram overlap rather than comprehension. This concern is reinforced by the Table 4 caption, which says 'Human Score is computed on 358 questions' but no human scores appear anywhere in the paper; the human evaluation should be reported, or the caption corrected.
[§3, Automatic metrics] The sentence 'e.g., losing 0.4 in Rouge-1' is numerically inconsistent with Table 4: the ROUGE-1 gaps are 0.05 (GPT-4o), 0.15 (o1-mini), and 0.11 (Llama-3.1-8b), not 0.4. This appears to be a typo, but it should be corrected because it changes the magnitude of the reported disparity.
[§2, Dataset statistics; Table 2] There are several internal inconsistencies in the dataset counts that need to be reconciled. The abstract and Table 2 state 358 questions, while §2 states '356 unique questions'; Table 2 lists 338 English and 208 Yorùbá documents, whereas the abstract repeats these numbers, and §2 says '664 Yorùbá documents and 1,566 questions were sent for human annotation.' The authors should clarify the exact final counts, the relationship between the 1,566 annotated questions and the 358 released questions, and why the English and Yorùbá question counts differ if the dataset is parallel.

minor comments (5)

[§2, Annotator findings] The sentence ending '(d) answers existing in multiple paragraphs in the document for which they annotated the row with all paragraphs where' is incomplete and should be finished to describe what was recorded for such cases.
[Throughout] There are several typos and naming inconsistencies: 'Rouge' should be 'ROUGE', 'LlaMA-3.1-8b' should be 'Llama-3.1-8B', 'Bebebele' should be 'Belebele', 'close-book' should be 'closed-book', and 'therevy' should be 'thereby'.
[References] Several citations use the placeholder form 'et al., 2024a' and 'et al., 2024b' without the full author lists; the reference entries should be completed to standard citation style.
[Figure 1] The caption for Figure 1 does not state how the equal-size length buckets were constructed, how many documents fall into each bucket, or whether the curves are for GPT-4o only; adding this information would make the length analysis reproducible.
[§4, Limitations] The dataset availability statement says Y-NQ is 'freely available on HuggingFace' but no URL or dataset identifier is given; this should be included for provenance and reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper reports direct automatic-metric comparisons against an independently annotated human reference set.

full rationale

No load-bearing derivation in this paper reduces to its own inputs. Y-NQ is an annotated dataset, and the paper's conclusions are observed ROUGE comparisons between model outputs and human references. There are no fitted parameters, no uniqueness theorems, no formal construction in which one equation equals another by definition, and no prediction that is statistically forced by a prior fit. The SONAR-based pre-annotation was explicitly abandoned and is not used in the reported results. The marginal reporting concern that the x2.5 disparity is computed on a small subset described inconsistently (Section 2 says six documents, Section 3 says only four documents over 900 words, and Table 5 reports six comparable documents) is a measurement-reporting and cherry-picking risk, not circularity: the observed ROUGE ratios are not forced by dataset construction. The limitations section discloses possible training contamination and cross-lingual non-comparability, which are external-validity caveats and do not make the central empirical claim equivalent to its assumptions. The score is therefore 0.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

The paper introduces no free parameters fitted to data and no new entities. The central claim rests on domain assumptions about metric validity and reference quality, listed above.

assumptions (2)

domain assumption ROUGE scores against a single reference are a valid measure of answer quality for Yorùbá.
The entire comparison in Section 3 rests on ROUGE without human validation, so this is an unverified assumption about the metric.
domain assumption The English NQ gold answers are correct and the Yorùbá annotations are correct.
The benchmark treats the reference answers as ground truth, including 8 known-incorrect English answers that the authors chose to keep.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Y-NQ: English-Yor\`ub\'a Evaluation dataset for Open-Book Reading Comprehension and Text Generation." pith.science (2026). https://pith.science/paper/R7RBKLJG

@misc{pith2026241208279,
  author       = {Pith},
  title        = {Pith review of: Y-NQ: English-Yor\`ub\'a Evaluation dataset for Open-Book Reading Comprehension and Text Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/R7RBKLJG}},
  note         = {Machine review of arXiv:2412.08279}
}

read the original abstract

The purpose of this work is to share an English-Yor\`ub\'a evaluation dataset for open-book reading comprehension and text generation to assess the performance of models both in a high- and a low- resource language. The dataset contains 358 questions and answers on 338 English documents and 208 Yor\`ub\'a documents. The average document length is ~ 10k words for English and 430 words for Yor\`ub\'a. Experiments show a consistent disparity in performance between the two languages, with Yor\`ub\'a falling behind English for automatic metrics even if documents are much shorter for this language. For a small set of documents with comparable length, performance of Yor\`ub\'a drops by x2.5 times. When analyzing performance by length, we observe that Yor\`ub\'a decreases performance dramatically for documents that reach 1500 words while English performance is barely affected at that length. Our dataset opens the door to showcasing if English LLM reading comprehension capabilities extend to Yor\`ub\'a, which for the evaluated LLMs is not the case.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 3 canonical work pages

[1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
[4]

Smith, and Yulia Tsvetkov

Orevaoghene Ahia, Anuoluwapo Aremu, Diana Abagyan, Hila Gonen, David Ifeoluwa Adelani, Daud Abolade, Noah A. Smith, and Yulia Tsvetkov. Voices unheard: NLP resources and models for Y or \`u b \'a regional dialects. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Pro...

work page 2024
[5]

NaijaRC: A Multi-choice Reading Comprehension Dataset for Nigerian Languages

Anuoluwapo Aremu, Jesujoba O. Alabi, Daud Abolade, Nkechinyere F. Aguobi, Shamsuddeen Hassan Muhammad, and David Ifeoluwa Adelani. Naijarc: A multi-choice reading comprehension dataset for nigerian languages, 2024. URL https://arxiv.org/abs/2308.09768

work page Pith review arXiv 2024
[6]

The belebele benchmark: a parallel reading comprehension dataset in 122 language variants

Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. The belebele benchmark: a parallel reading comprehension dataset in 122 language variants. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting ...

2024
[7]

Boolq: Exploring the surprising difficulty of natural yes/no questions

Christopher Clark, Matthew Gardner, Tom Fevry, and Robert Weischedel. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044, 2019

arXiv 1905
[8]

Sonar: Sentence-level multimodal and language-agnostic representations, 2023

Paul-Ambroise Duquenne, Holger Schwenk, and Benoît Sagot. Sonar: Sentence-level multimodal and language-agnostic representations, 2023. URL https://arxiv.org/abs/2308.11466

arXiv 2023

Show all 16 references

[9]

The llama 3 herd of models, 2024 a

Abhimanyu Dubey et al. The llama 3 herd of models, 2024 a . URL https://arxiv.org/abs/2407.21783

2024 arXiv
[10]

Gpt-4 technical report, 2024 b

OpenAI et al. Gpt-4 technical report, 2024 b . URL https://arxiv.org/abs/2303.08774

2024 arXiv
[11]

Triviaqa: A large scale question-answer dataset for reading comprehension

Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. Triviaqa: A large scale question-answer dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017

2017 arXiv
[12]

Pairing W ikipedia articles across languages

Marcus Klang and Pierre Nugues. Pairing W ikipedia articles across languages. In Key-Sun Choi, Christina Unger, Piek Vossen, Jin-Dong Kim, Noriko Kando, and Axel-Cyrille Ngonga Ngomo, editors, Proceedings of the Open Knowledge Base and Question Answering Workshop ( OKBQA 2016)...

2016
[13]

Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav...

2019
[14]

ROUGE : A package for automatic evaluation of summaries

Chin-Yew Lin. ROUGE : A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74--81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL https://aclanthology.org/W04-1013

2004
[15]

Cross-lingual open-retrieval question answering for A frican languages

Odunayo Ogundepo, Tajuddeen Gwadabe, Clara Rivera, Jonathan Clark, Sebastian Ruder, David Adelani, Bonaventure Dossou, Abdou Diop, Claytone Sikasote, Gilles Hacheme, Happy Buzaaba, Ignatius Ezeani, Rooweither Mabuya, Salomey Osei, Chris Emezue, Albert Kahira, Shamsuddeen Muham...

2023
[16]

Squad: 100,000+ questions for machine comprehension of text

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016

2016 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

[4] [4]

Smith, and Yulia Tsvetkov

Orevaoghene Ahia, Anuoluwapo Aremu, Diana Abagyan, Hila Gonen, David Ifeoluwa Adelani, Daud Abolade, Noah A. Smith, and Yulia Tsvetkov. Voices unheard: NLP resources and models for Y or \`u b \'a regional dialects. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Pro...

work page 2024

[5] [5]

NaijaRC: A Multi-choice Reading Comprehension Dataset for Nigerian Languages

Anuoluwapo Aremu, Jesujoba O. Alabi, Daud Abolade, Nkechinyere F. Aguobi, Shamsuddeen Hassan Muhammad, and David Ifeoluwa Adelani. Naijarc: A multi-choice reading comprehension dataset for nigerian languages, 2024. URL https://arxiv.org/abs/2308.09768

work page Pith review arXiv 2024

[6] [6]

The belebele benchmark: a parallel reading comprehension dataset in 122 language variants

Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. The belebele benchmark: a parallel reading comprehension dataset in 122 language variants. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting ...

2024

[7] [7]

Boolq: Exploring the surprising difficulty of natural yes/no questions

Christopher Clark, Matthew Gardner, Tom Fevry, and Robert Weischedel. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044, 2019

arXiv 1905

[8] [8]

Sonar: Sentence-level multimodal and language-agnostic representations, 2023

Paul-Ambroise Duquenne, Holger Schwenk, and Benoît Sagot. Sonar: Sentence-level multimodal and language-agnostic representations, 2023. URL https://arxiv.org/abs/2308.11466

arXiv 2023