TyDi QA-WANA: A Benchmark for Information-Seeking Question Answering in Languages of West Asia and North Africa

Jonathan H. Clark; Parker Riley; Siamak Shakeri; Waleed Ammar

REVIEW 2 major objections 4 minor 26 references

TyDi QA-WANA: A Benchmark for Information-Seeking Question Answering in Languages of West Asia and North Africa

T0 review · 2 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read TyDi QA-WANA offers 28,197 native information-seeking questions across ten West Asian and North African language varieties, each paired with a full Wikipedia article so that long-context reading is part of the task.

desk verdict Useful new benchmark with careful dataset construction, but the Farsi and Tajik dev/test splits are so small after NULL consensus that those two languages can't yet support reliable within-language comparisons. read the letter →

arxiv 2507.17709 v1 pith:IHVIPK5A submitted 2025-07-23 cs.CL

Parker Riley , Siamak Shakeri , Waleed Ammar , Jonathan H. Clark This is my paper

classification cs.CL

keywords TyDiQA-WANAmultilingualquestionansweringinformation-seekingquestionslow-resourcelanguageslong-contextQAMinimalAnswerSpanArabicdialectsWestAsiaandNorthAfrica

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper presents TyDi QA-WANA, a question-answering benchmark of 28,197 examples across ten language varieties of West Asia and North Africa, built without translating questions. The authors aim to give the field an evaluation resource that captures genuine information-seeking questions, requires reading an entire Wikipedia article as context, and covers under-represented varieties such as Algerian Arabic, Tajik, and Azerbaijani. If the benchmark works as intended, it lets researchers measure and compare long-context QA ability in languages where few native datasets exist, and it demonstrates that two large language models can already extract answers from full articles without fine-tuning.

What carries the argument

The load-bearing mechanism is the Minimal Answer Span (MinSpan) task and the TyDi-style data pipeline. MinSpan requires, for a question plus a full article, either the minimal byte span that answers the question, a YES or NO answer, or NULL when no answer exists. The pipeline has three stages: question elicitation from short article prompts, retrieval of the first Wikipedia result for each question, and answer labeling by one or three annotators. The NULL-consensus majority rule, applied to the three-annotation evaluation splits, is what makes the benchmark robust to annotation noise and to gaming by always predicting "no answer."

What would settle it

Have independent annotators judge whether the retrieved article is on-topic for the question before any answer labeling; if in high-NULL languages such as Tajik and Armenian most retrieved articles are judged off-topic, then the benchmark's non-NULL subset is too small and selective to support the claim that it measures information-seeking comprehension.

Watch

Extended reading notes

Core claim

The paper's central claim is that TyDi QA-WANA is a valid long-context information-seeking QA benchmark for ten language varieties. To support this, it describes a collection procedure in which annotators write questions they are actually curious about after seeing the first 150 characters of a Wikipedia article; a search engine retrieves one Wikipedia article for each question; and annotators label a minimal answer span, a yes/no answer, or NULL if no answer exists. Evaluation uses max-over-annotations F1 and exact match with a NULL-consensus rule that prevents models from being rewarded for predicting "no answer" whenever any single annotator said NULL. Baseline results show the two tested models score roughly 35 to 76 F1 depending on language, with high NULL rates in varieties whose Wikipedias are small; the paper argues this still reflects answerability rather than always-NULL gaming because performance on the non-NULL portion is reasonable except in Algerian Arabic.

Load-bearing premise

The dataset's usefulness assumes that each question's single retrieved Wikipedia article is a plausible place to find the answer, so that "no answer" labels reflect genuine reading difficulty rather than failed search.

Editorial extensions

If this is right

Researchers get a public benchmark for ten WANA language varieties with a defined metric, so per-language model improvements can be tracked over time.
Because the full article is the context, the dataset can reveal whether long-context models actually use distant evidence rather than only nearby sentences.
The NULL/non-NULL split exposes where retrieval and small-Wikipedia coverage, not comprehension, are the bottleneck.
In-language, untranslated collection means dialect questions, such as four Arabic varieties, can be studied against Modern Standard Arabic articles, enabling cross-variety transfer research.
Baselines show off-the-shelf large language models can run the task without fine-tuning, lowering the barrier for other groups to use the benchmark.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A likely consequence not developed in the paper: in high-NULL varieties like Tajik and Armenian, overall scores mostly measure retrieval relevance plus NULL detection, so cross-language comparisons of comprehension should use the non-NULL subset or a better retrieval stage.
The same data collection could be rerun with multiple retrieved articles per question, or with relevance-filtered retrieval, to separate reading comprehension from open-retrieval quality.
The paper's model-based no-answer critic is a hidden auxiliary classifier; reporting its agreement with human NULL labels would make baseline numbers more interpretable.
Because Arabic questions are in dialect while the articles are Modern Standard Arabic, the dataset could double as a cross-variety retrieval benchmark, a use the authors mention but do not exploit in their evaluation.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

TyDi QA-WANA is a genuine contribution: a natively elicited, information-seeking QA dataset for ten under-represented language varieties, with full Wikipedia articles as context. The construction pipeline is careful: the rater screening, the NULL-consensus rule, the decision to elicit questions instead of translating them, and the release of code and data all earn credit. For Arabic dialect work alone, having Algerian, Egyptian, Iraqi, and Jordanian questions paired with MSA articles is new and useful. The baseline results from Gemini 1.5 Pro and 2.0 Flash, including the NULL/non-NULL breakdown, give the community a reasonable starting point. The soft spots are real but localized. The stress-test note is right: after NULL consensus, Farsi and Tajik effectively have about 50 and 15 non-NULL examples per split, respectively. With roughly 15 binary examples, a 10-point F1 gap is within sampling noise. The paper telling readers to compare models within a language is not enough when the non-NULL subset is that small. The authors should either expand those splits, add confidence intervals, or explicitly caution that Farsi and Tajik are not yet reliable evaluation sets. The missing confidence intervals across all languages are a minor but pervasive issue. The Algerian Arabic 145-byte average answer length is an odd outlier, worth a sentence of explanation. The retrieval step (first Google hit, discard no-result questions) creates a selection bias that the paper mentions but does not analyze; it is a design choice, not a fatal flaw. The central claim, that this is a valid and useful benchmark for most of its ten languages, holds up. The paper is honest about high NULL rates and cross-language incomparability; it just does not go far enough on the small-split problem. I would bring this to a reading group as an example of careful multilingual benchmark construction, and I would cite it if I worked on low-resource QA or long-context evaluation. It deserves a serious peer review, with the recommendation that the authors address the Farsi and Tajik sample-size problem and add error bars before final acceptance.

Referee Report

2 major / 4 minor

Summary. The paper introduces TyDi QA-WANA, a question-answering dataset of 28,197 examples in 10 language varieties of West Asia and North Africa, following the TyDi QA MinSpan task. Questions are elicited in-language from short Wikipedia prompts, paired with a full Wikipedia article retrieved via a restricted Google search, and labeled by one annotator (train) or three annotators (dev/test). A NULL-consensus majority vote determines answerability. The paper presents dataset statistics, baseline results from Gemini 1.5 Pro and Gemini 2.0 Flash on dev and test splits, and splits results by NULL/non-NULL consensus. The authors release code and data.

Significance. If the benchmark is valid, it fills a real gap: a culturally grounded, in-language, long-context QA resource for several under-represented varieties, with a deliberate focus on information-seeking questions and large article contexts. The data collection and quality-control procedures are described in detail, including native-speaker verification, training quizzes, and the NULL-consensus protocol, which is a sensible way to handle annotator disagreement on unanswerability. Releasing the data and code is a concrete contribution to reproducibility. The main value is as an evaluation resource for future models, and the baseline experiments demonstrate the feasibility of using long-context LLMs on the task. However, the usefulness of the benchmark for Farsi and Tajik is substantially weakened by the very small effective size of the non-NULL evaluation sets in those varieties, and the high NULL rates raise questions about whether the retrieval step yields sufficiently relevant articles.

major comments (2)

[Table 1, §7.1, Table 3] The effective evaluation size for Farsi and Tajik is too small to support the benchmark-level claims. Farsi dev/test splits contain 105/104 examples with 47.9% NULL consensus; Tajik dev/test contain 86/88 examples with 82.7% NULL consensus. This leaves roughly 55 Farsi and 15 Tajik non-NULL examples per split. With about 15 examples, the reported Tajik test F1 gap between Gemini 1.5 Pro (68.2) and Gemini 2.0 Flash (57.6) is well within sampling noise, as is the 13.7-point gap on the non-NULL subset in Table 5. The paper reports no confidence intervals or significance tests. Since the central claim is that the dataset provides a useful evaluation resource across 10 language varieties, this undercuts that claim for Farsi and Tajik specifically. I recommend either expanding these splits, or transparently reporting bootstrap confidence intervals and explicitly cautioning that per-language comparisons for Farsi and Tajik are low-powered.
[§3, Table 1] The article retrieval step uses only the first Google search result and discards questions with no result, but does not verify that the first result is topically relevant to the question. For varieties with high NULL rates, particularly Tajik (82.7%) and Armenian (76.7%), the resulting non-NULL subset may be small and unrepresentative, and the task risks measuring retrieval failure and NULL prediction rather than comprehension. The paper's explanation that small Wikipedias cause NULL is plausible, but it is not demonstrated. A concrete test would be to sample NULL-consensus examples and have annotators judge whether the retrieved article is topically relevant to the question; if most retrieved articles are relevant, then NULL reflects absence of answer, whereas if many are irrelevant, the benchmark under-represents answerable information-seeking questions.

minor comments (4)

[Abstract] There is a typo in the abstract: 'Each question in paired with an entire article' should be 'Each question is paired with an entire article.'
[Tables 4 and 5] The table headers use the symbols '!' and '%' to denote NULL and non-NULL consensus, but these symbols are not defined in the captions or in the surrounding text; please define them explicitly.
[§9, Table 5] The statement that non-NULL performance is 'reasonable' with 'the possible exception of Algerian Arabic' understates the outlier: Algerian Arabic non-NULL test F1 is 16.1 for Gemini 1.5 Pro and 16.3 for Gemini 2.0 Flash, far below the next-lowest variety (Tajik, 58.3/44.6). This deserves a substantive discussion, especially in light of the unusually large average answer length for Algerian Arabic (145 bytes) in Table 1.
[§8] The no-answer critic is a second inference pass that decides whether the model's output is a misformatted 'no answer' response, but no evaluation of the critic's accuracy is provided. Since this affects the final F1/EM scores, a small validation of the critic (e.g., on a sample of ambiguous outputs) would strengthen the baseline reporting.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the dataset labels are human-produced, the baseline scores are independent measurements, and no claim reduces to a fitted input or self-citation chain.

full rationale

This paper constructs a dataset and reports baseline model scores; there is no derived quantity that is equivalent to an input by construction. The labels come from human annotators through question elicitation, article retrieval, and answer labeling (Section 3), and the NULL-consensus procedure (Section 7.1) is an explicit post-hoc aggregation rule rather than a fitted parameter. The baseline results in Tables 2-5 are external measurements on held-out splits and are not used to define the dataset. The paper borrows the MinSpan task definition and data collection procedure from Clark et al. (2020), but this is an adopted task framing, not a circular justification: the cited prior work is an independent, published benchmark, and the present contribution is a new human-annotated dataset in new language varieties. Self-citations to TyDi QA and XTREME-UP appear only as related work and task provenance, not as evidence that the new dataset works. The high NULL rates and small non-NULL evaluation subsets for Farsi and Tajik are legitimate statistical-power concerns for benchmark utility, but they are not circularity: they concern whether the evaluation is informative, not whether the reported numbers reduce to the inputs. No equation, fitted parameter, or uniqueness claim is invoked to force any reported result, so the appropriate finding is no significant circularity.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

No new entities are introduced. The free parameters are design choices that affect dataset shape but are not fitted to optimize performance. The axioms concern the data collection pipeline and label reliability, which are the load-bearing assumptions for a benchmark paper.

free parameters (2)

Prompt length for question elicitation = 150 characters
Chosen by hand in Section 3 to give annotators a topic seed without revealing the answer.
NULL consensus threshold = 2 out of 3 annotations
Chosen by hand in Section 7.1 to establish a majority vote for answerability.

assumptions (3)

domain assumption Google search restricted to Wikipedia retrieves a relevant article for each question
Section 3, Article Retrieval, uses the first result; if none, the question is discarded, so the dataset depends on this retrieval being meaningful.
domain assumption Annotators are reliable native speakers and follow the information-seeking guidelines
Section 3 describes QC steps including native-speaker verification and a 90% quiz, but the entire dataset rests on annotator behavior.
domain assumption The NULL consensus majority vote produces correct answerability labels
Section 7.1 defines consensus to handle annotation noise; incorrect consensus would bias evaluation.

how reviews work

0 comments

Cite this review

Pith. "Pith review of TyDi QA-WANA: A Benchmark for Information-Seeking Question Answering in Languages of West Asia and North Africa." pith.science (2026). https://pith.science/paper/IHVIPK5A

@misc{pith2026250717709,
  author       = {Pith},
  title        = {Pith review of: TyDi QA-WANA: A Benchmark for Information-Seeking Question Answering in Languages of West Asia and North Africa},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IHVIPK5A}},
  note         = {Machine review of arXiv:2507.17709}
}

read the original abstract

We present TyDi QA-WANA, a question-answering dataset consisting of 28K examples divided among 10 language varieties of western Asia and northern Africa. The data collection process was designed to elicit information-seeking questions, where the asker is genuinely curious to know the answer. Each question in paired with an entire article that may or may not contain the answer; the relatively large size of the articles results in a task suitable for evaluating models' abilities to utilize large text contexts in answering questions. Furthermore, the data was collected directly in each language variety, without the use of translation, in order to avoid issues of cultural relevance. We present performance of two baseline models, and release our code and data to facilitate further improvement by the research community.

Figures

Figures reproduced from arXiv: 2507.17709 by the authors.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

26 extracted references · 13 canonical work pages

[1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Joshua Ainslie, Santiago Ontanon, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang. 2020. Etc: Encoding long and structured inputs in transformers. arXiv preprint arXiv:2004.08483

arXiv 2020
[4]

Shane Arora, Marzena Karpinska, Hung-Ting Chen, Ipsita Bhattacharjee, Mohit Iyyer, and Eunsol Choi. 2024. Calmqa: Exploring culturally specific long-form question answering across 23 languages. arXiv preprint arXiv:2406.17761

work page Pith review arXiv 2024
[5]

Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2019. On the cross-lingual transferability of monolingual representations. arXiv preprint arXiv:1910.11856

arXiv 2019
[6]

Akari Asai, Jungo Kasai, Jonathan H Clark, Kenton Lee, Eunsol Choi, and Hannaneh Hajishirzi. 2020. Xor qa: Cross-lingual open-retrieval question answering. arXiv preprint arXiv:2010.11856

arXiv 2020
[7]

Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150

arXiv 2020
[8]

Houda Bouamor, Sabit Hassan, and Nizar Habash. 2019. The madar shared task on arabic fine-grained dialect identification. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 199--207

work page 2019

Show all 26 references

[9]

ElMehdi Boujou, Hamza Chataoui, Abdellah El Mekki, Saad Benjelloun, Ikram Chairi, and Ismail Berrada. 2021. An open access nlp dataset for arabic dialects: Data collection, labeling, and model construction. arXiv preprint arXiv:2102.11000

2021 arXiv
[10]

Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki

Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. https://doi.org/10.1162/tacl_a_00317 T y D i QA : A benchmark for information-seeking question answering in typologically diverse languages . Transac...

2020 doi
[11]

Gulshan Dovudov, V \' t Suchomel, and Pavel S merk. 2012. Pos annotated 50m corpus of tajik language. Language Technology for Normalisation of Less-Resourced Languages, page 93

2012
[12]

Leon Engl \"a nder, Hannah Sterz, Clifton Poth, Jonas Pfeiffer, Ilia Kuznetsov, and Iryna Gurevych. 2024. M2qa: Multi-domain multilingual question answering. arXiv preprint arXiv:2407.01091

2024 arXiv
[14]

Gemini Team . 2024 b . http://arxiv.org/abs/2312.11805 Gemini: A family of highly capable multimodal models

2024 arXiv
[15]

Matthias Hecking and Tatiana Sarmina-Baneviciene. 2010. A tajik extension of the multilingual information extraction system zenon. In Proceedings of the 15th International Command and Control Research and Technolgy Symposium (ICCRTS), Santa Monica, CA. Citeseer

2010
[16]

Jafar Isbarov, Kavsar Huseynova, Elvin Mammadov, and Mammad Hajili. 2024. Open foundation models for azerbaijani language. arXiv preprint arXiv:2407.02337

2024 arXiv
[17]

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for C...

2019
[18]

Patrick Lewis, Barlas O g uz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2019. Mlqa: Evaluating cross-lingual extractive question answering. arXiv preprint arXiv:1910.07475

2019 arXiv
[19]

Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157--173

2024
[20]

Shayne Longpre, Yi Lu, and Joachim Daiber. 2021. Mkqa: A linguistically diverse benchmark for multilingual open domain question answering. Transactions of the Association for Computational Linguistics, 9:1389--1406

2021
[21]

Rayyan Merchant and Kevin Tang. 2024. Parstext: A digraphic corpus for tajik-farsi transliteration. In Proceedings of the Second Workshop on Computation and Written Language (CAWL)@ LREC-COLING 2024, pages 1--7

2024
[22]

Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for squad. arXiv preprint arXiv:1806.03822

2018 arXiv
[23]

Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arX...

2024 arXiv
[24]

Uma Roy, Noah Constant, Rami Al-Rfou, Aditya Barua, Aaron Phillips, and Yinfei Yang. 2020. Lareqa: Language-agnostic answer retrieval from a multilingual pool. arXiv preprint arXiv:2004.05484

2020 arXiv
[25]

Sebastian Ruder, Jonathan H Clark, Alexander Gutkin, Mihir Kale, Min Ma, Massimo Nicosia, Shruti Rijhwani, Parker Riley, Jean-Michel A Sarr, Xinyi Wang, et al. 2023. Xtreme-up: A user-centric scarce-data benchmark for under-represented languages. arXiv preprint arXiv:2305.11938

2023 arXiv
[26]

Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33:17283--17297

2020
[27]

Jihad Zahir. 2022. Iadd: An integrated arabic dialect identification dataset. Data in Brief, 40:107777

2022

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Joshua Ainslie, Santiago Ontanon, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang. 2020. Etc: Encoding long and structured inputs in transformers. arXiv preprint arXiv:2004.08483

arXiv 2020

[4] [4]

Shane Arora, Marzena Karpinska, Hung-Ting Chen, Ipsita Bhattacharjee, Mohit Iyyer, and Eunsol Choi. 2024. Calmqa: Exploring culturally specific long-form question answering across 23 languages. arXiv preprint arXiv:2406.17761

work page Pith review arXiv 2024

[5] [5]

Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2019. On the cross-lingual transferability of monolingual representations. arXiv preprint arXiv:1910.11856

arXiv 2019

[6] [6]

Akari Asai, Jungo Kasai, Jonathan H Clark, Kenton Lee, Eunsol Choi, and Hannaneh Hajishirzi. 2020. Xor qa: Cross-lingual open-retrieval question answering. arXiv preprint arXiv:2010.11856

arXiv 2020

[7] [7]

Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150

arXiv 2020

[8] [8]

Houda Bouamor, Sabit Hassan, and Nizar Habash. 2019. The madar shared task on arabic fine-grained dialect identification. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 199--207

work page 2019