{"id":"a4e33991-4947-416f-b74a-ef666db5430e","arxiv_id":"2507.17709","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"TyDi QA-WANA is a new 28,000-example QA benchmark covering 10 under-represented languages with long-context, information-seeking questions and baseline evaluations.","lead":"This paper introduces TyDi QA-WANA, a question-answering benchmark with 28,000 examples across 10 languages of West Asia and North Africa, built from natively elicited information-seeking questions paired with full Wikipedia articles. It provides baseline results for two Gemini models and releases data and code for the research community.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Farsi and Tajik evaluation splits are too small to support the 10-language benchmark claim: after NULL consensus, Tajik has ~15 non-NULL test examples, so reported F1 differences are statistically meaningless.","rationale":"The reader's weakest assumption focused on the retrieval step producing irrelevant articles, leading to high NULL rates. I agree that high NULL rates are a concern, but the more precise, load-bearing issue is their consequence: the effective non-NULL evaluation set for Farsi and Tajik is tiny. Applying the paper's own NULL rates (Table 1, column '% With NULL Consensus') to the small dev/test sizes (Table 1) yields about 55 non-NULL Farsi and 15 non-NULL Tajik examples per split. With 15 examples, any F1 difference must exceed roughly 12 points to be significant, yet the paper reports Tajik test F1s of 68.2 vs 57.6 (Table 3) without error bars. This directly threatens the central claim that the dataset is a useful evaluation benchmark for all 10 varieties. The paper does provide a non-NULL/NULL breakdown (Tables 4-5), which is good, but it does not quantify the statistical power of these splits. The Algerian Arabic non-NULL F1 outlier (16-17) and the 'first dataset' claim are secondary; they do not affect the whole benchmark as much as the statistical emptiness of two languages. I recommend CONDITIONAL acceptance: the dataset is a real contribution, but the authors should provide confidence intervals or explicitly acknowledge the limited reliability of Farsi/Tajik evaluations, and ideally report aggregated results across languages to compensate for small per-language samples.","tokens_in":9765,"tokens_out":6423,"duration_ms":70168,"concrete_test":"Compute the actual number of non-NULL-consensus examples per language in the dev and test splits (using the released data and the NULL consensus rule of Section 7.1), then bootstrap each baseline model's F1 scores on the Farsi and Tajik test sets with 10,000 resamples to obtain 95% confidence intervals. If the intervals for the two Gemini models overlap by more than half the absolute difference in the reported F1 scores, the benchmark cannot statistically distinguish models for those varieties; report the minimum detectable effect size at the observed sample sizes.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that TyDi QA-WANA is a valid, useful benchmark covering 10 language varieties. For an evaluation benchmark to support that claim, each variety's dev/test splits must contain enough answerable examples to distinguish systems. Table 1 shows Farsi dev=105/test=104 and Tajik dev=86/test=88, with NULL-consensus rates of 47.9% and 82.7% respectively. Applying these rates to the dev/test splits (assuming the rates are representative) leaves roughly 55 Farsi and 15 Tajik non-NULL evaluation examples per split. With ~15 binary-score examples, the standard error of an F1 estimate is about 10-12 points, so the Tajik test F1 gap between Gemini 1.5 Pro (68.2) and Gemini 2.0 Flash (57.6) in Table 3 is well within sampling noise. The paper reports no confidence intervals or significance tests, and its recommendation to compare within-language scores does not solve the problem when the non-NULL subset is this small. The reader's concern about high NULL rates is the root cause; the missing sharpening is that the effective evaluation set for two of the ten claimed languages is nearly empty. This undercuts the benchmark's utility for Farsi and Tajik specifically, and weakens the '28K examples divided among 10 language varieties' claim as an evaluation resource.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces TyDi QA-WANA, a question-answering dataset of 28,197 examples in 10 language varieties of West Asia and North Africa, following the TyDi QA MinSpan task. Questions are elicited in-language from short Wikipedia prompts, paired with a full Wikipedia article retrieved via a restricted Google search, and labeled by one annotator (train) or three annotators (dev/test). A NULL-consensus majority vote determines answerability. The paper presents dataset statistics, baseline results from Gemini 1.5 Pro and Gemini 2.0 Flash on dev and test splits, and splits results by NULL/non-NULL consensus. The authors release code and data.","tokens_in":10134,"tokens_out":4039,"duration_ms":44819,"significance":"If the benchmark is valid, it fills a real gap: a culturally grounded, in-language, long-context QA resource for several under-represented varieties, with a deliberate focus on information-seeking questions and large article contexts. The data collection and quality-control procedures are described in detail, including native-speaker verification, training quizzes, and the NULL-consensus protocol, which is a sensible way to handle annotator disagreement on unanswerability. Releasing the data and code is a concrete contribution to reproducibility. The main value is as an evaluation resource for future models, and the baseline experiments demonstrate the feasibility of using long-context LLMs on the task. However, the usefulness of the benchmark for Farsi and Tajik is substantially weakened by the very small effective size of the non-NULL evaluation sets in those varieties, and the high NULL rates raise questions about whether the retrieval step yields sufficiently relevant articles.","major_comments":[{"comment":"The effective evaluation size for Farsi and Tajik is too small to support the benchmark-level claims. Farsi dev/test splits contain 105/104 examples with 47.9% NULL consensus; Tajik dev/test contain 86/88 examples with 82.7% NULL consensus. This leaves roughly 55 Farsi and 15 Tajik non-NULL examples per split. With about 15 examples, the reported Tajik test F1 gap between Gemini 1.5 Pro (68.2) and Gemini 2.0 Flash (57.6) is well within sampling noise, as is the 13.7-point gap on the non-NULL subset in Table 5. The paper reports no confidence intervals or significance tests. Since the central claim is that the dataset provides a useful evaluation resource across 10 language varieties, this undercuts that claim for Farsi and Tajik specifically. I recommend either expanding these splits, or transparently reporting bootstrap confidence intervals and explicitly cautioning that per-language comparisons for Farsi and Tajik are low-powered.","section":"Table 1, §7.1, Table 3"},{"comment":"The article retrieval step uses only the first Google search result and discards questions with no result, but does not verify that the first result is topically relevant to the question. For varieties with high NULL rates, particularly Tajik (82.7%) and Armenian (76.7%), the resulting non-NULL subset may be small and unrepresentative, and the task risks measuring retrieval failure and NULL prediction rather than comprehension. The paper's explanation that small Wikipedias cause NULL is plausible, but it is not demonstrated. A concrete test would be to sample NULL-consensus examples and have annotators judge whether the retrieved article is topically relevant to the question; if most retrieved articles are relevant, then NULL reflects absence of answer, whereas if many are irrelevant, the benchmark under-represents answerable information-seeking questions.","section":"§3, Table 1"}],"minor_comments":[{"comment":"There is a typo in the abstract: 'Each question in paired with an entire article' should be 'Each question is paired with an entire article.'","section":"Abstract"},{"comment":"The table headers use the symbols '!' and '%' to denote NULL and non-NULL consensus, but these symbols are not defined in the captions or in the surrounding text; please define them explicitly.","section":"Tables 4 and 5"},{"comment":"The statement that non-NULL performance is 'reasonable' with 'the possible exception of Algerian Arabic' understates the outlier: Algerian Arabic non-NULL test F1 is 16.1 for Gemini 1.5 Pro and 16.3 for Gemini 2.0 Flash, far below the next-lowest variety (Tajik, 58.3/44.6). This deserves a substantive discussion, especially in light of the unusually large average answer length for Algerian Arabic (145 bytes) in Table 1.","section":"§9, Table 5"},{"comment":"The no-answer critic is a second inference pass that decides whether the model's output is a misformatted 'no answer' response, but no evaluation of the critic's accuracy is provided. Since this affects the final F1/EM scores, a small validation of the critic (e.g., on a sample of ambiguous outputs) would strengthen the baseline reporting.","section":"§8"}],"recommendation":"major_revision","confidential_remarks":"The paper is a dataset contribution with a clear reproducibility benefit, but the effective evaluation-set size for Farsi and Tajik is a load-bearing issue for the '10 language varieties' claim. I would encourage the authors to address it head-on, either by expanding those splits or by adding explicit uncertainty quantification and hedging the per-language claims. The retrieval-relevance concern is also worth a targeted analysis even if it does not change the main conclusions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"TyDi QA-WANA is a genuine contribution: a natively elicited, information-seeking QA dataset for ten under-represented language varieties, with full Wikipedia articles as context. The construction pipeline is careful: the rater screening, the NULL-consensus rule, the decision to elicit questions instead of translating them, and the release of code and data all earn credit. For Arabic dialect work alone, having Algerian, Egyptian, Iraqi, and Jordanian questions paired with MSA articles is new and useful. The baseline results from Gemini 1.5 Pro and 2.0 Flash, including the NULL/non-NULL breakdown, give the community a reasonable starting point. The soft spots are real but localized. The stress-test note is right: after NULL consensus, Farsi and Tajik effectively have about 50 and 15 non-NULL examples per split, respectively. With roughly 15 binary examples, a 10-point F1 gap is within sampling noise. The paper telling readers to compare models within a language is not enough when the non-NULL subset is that small. The authors should either expand those splits, add confidence intervals, or explicitly caution that Farsi and Tajik are not yet reliable evaluation sets. The missing confidence intervals across all languages are a minor but pervasive issue. The Algerian Arabic 145-byte average answer length is an odd outlier, worth a sentence of explanation. The retrieval step (first Google hit, discard no-result questions) creates a selection bias that the paper mentions but does not analyze; it is a design choice, not a fatal flaw. The central claim, that this is a valid and useful benchmark for most of its ten languages, holds up. The paper is honest about high NULL rates and cross-language incomparability; it just does not go far enough on the small-split problem. I would bring this to a reading group as an example of careful multilingual benchmark construction, and I would cite it if I worked on low-resource QA or long-context evaluation. It deserves a serious peer review, with the recommendation that the authors address the Farsi and Tajik sample-size problem and add error bars before final acceptance.","headline":"Useful new benchmark with careful dataset construction, but the Farsi and Tajik dev/test splits are so small after NULL consensus that those two languages can't yet support reliable within-language comparisons.","tokens_in":694,"tokens_out":710,"would_cite":true,"duration_ms":31153,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TyDi QA-WANA offers 28,197 native information-seeking questions across ten West Asian and North African language varieties, each paired with a full Wikipedia article so that long-context reading is part of the task.","keywords":["TyDi QA-WANA","multilingual question answering","information-seeking questions","low-resource languages","long-context QA","Minimal Answer Span","Arabic dialects","West Asia and North Africa"],"falsifier":"Have independent annotators judge whether the retrieved article is on-topic for the question before any answer labeling; if in high-NULL languages such as Tajik and Armenian most retrieved articles are judged off-topic, then the benchmark's non-NULL subset is too small and selective to support the claim that it measures information-seeking comprehension.","tokens_in":1412,"feed_emoji":"🌍","tokens_out":1528,"duration_ms":93656,"temperature":0.7,"pith_summary":"This paper presents TyDi QA-WANA, a question-answering benchmark of 28,197 examples across ten language varieties of West Asia and North Africa, built without translating questions. The authors aim to give the field an evaluation resource that captures genuine information-seeking questions, requires reading an entire Wikipedia article as context, and covers under-represented varieties such as Algerian Arabic, Tajik, and Azerbaijani. If the benchmark works as intended, it lets researchers measure and compare long-context QA ability in languages where few native datasets exist, and it demonstrates that two large language models can already extract answers from full articles without fine-tuning.","feed_headline":"28,197 questions test long-context QA in 10 WANA languages","feed_subtitle":"Questions written without translation pair with whole Wikipedia articles, testing whether models can use long contexts.","key_machinery":"The load-bearing mechanism is the Minimal Answer Span (MinSpan) task and the TyDi-style data pipeline. MinSpan requires, for a question plus a full article, either the minimal byte span that answers the question, a YES or NO answer, or NULL when no answer exists. The pipeline has three stages: question elicitation from short article prompts, retrieval of the first Wikipedia result for each question, and answer labeling by one or three annotators. The NULL-consensus majority rule, applied to the three-annotation evaluation splits, is what makes the benchmark robust to annotation noise and to gaming by always predicting \"no answer.\"","core_discovery":"The paper's central claim is that TyDi QA-WANA is a valid long-context information-seeking QA benchmark for ten language varieties. To support this, it describes a collection procedure in which annotators write questions they are actually curious about after seeing the first 150 characters of a Wikipedia article; a search engine retrieves one Wikipedia article for each question; and annotators label a minimal answer span, a yes/no answer, or NULL if no answer exists. Evaluation uses max-over-annotations F1 and exact match with a NULL-consensus rule that prevents models from being rewarded for predicting \"no answer\" whenever any single annotator said NULL. Baseline results show the two tested models score roughly 35 to 76 F1 depending on language, with high NULL rates in varieties whose Wikipedias are small; the paper argues this still reflects answerability rather than always-NULL gaming because performance on the non-NULL portion is reasonable except in Algerian Arabic.","pith_inferences":["A likely consequence not developed in the paper: in high-NULL varieties like Tajik and Armenian, overall scores mostly measure retrieval relevance plus NULL detection, so cross-language comparisons of comprehension should use the non-NULL subset or a better retrieval stage.","The same data collection could be rerun with multiple retrieved articles per question, or with relevance-filtered retrieval, to separate reading comprehension from open-retrieval quality.","The paper's model-based no-answer critic is a hidden auxiliary classifier; reporting its agreement with human NULL labels would make baseline numbers more interpretable.","Because Arabic questions are in dialect while the articles are Modern Standard Arabic, the dataset could double as a cross-variety retrieval benchmark, a use the authors mention but do not exploit in their evaluation."],"forward_implications":["Researchers get a public benchmark for ten WANA language varieties with a defined metric, so per-language model improvements can be tracked over time.","Because the full article is the context, the dataset can reveal whether long-context models actually use distant evidence rather than only nearby sentences.","The NULL/non-NULL split exposes where retrieval and small-Wikipedia coverage, not comprehension, are the bottleneck.","In-language, untranslated collection means dialect questions, such as four Arabic varieties, can be studied against Modern Standard Arabic articles, enabling cross-variety transfer research.","Baselines show off-the-shelf large language models can run the task without fine-tuning, lowering the barrier for other groups to use the benchmark."],"supporting_citations":[{"why":"Defines the MinSpan task, the elicitation/retrieval/labeling procedure, and the NULL-consensus scoring rationale that this dataset adopts.","marker":"Clark et al. (2020)"},{"why":"Establishes the information-seeking QA paradigm, where questions are written independently of the passage, motivating the collection design.","marker":"Kwiatkowski et al. (2019)"},{"why":"Provides the long-context model capability that makes it feasible to feed an entire article to the baseline systems.","marker":"Reid et al. (2024)"},{"why":"Supplies one of the two baseline models evaluated on the new dataset, anchoring expected performance.","marker":"Gemini Team (2024a)"},{"why":"Supports the motivation for culturally relevant, in-language question collection rather than translation.","marker":"Arora et al. (2024)"}],"fun_headline_variants":["28K long-context QA questions in 10 WANA languages","WANA QA benchmark: 28K questions, 10 varieties, long-article contexts","No-translation QA benchmark for 10 WANA languages with long contexts","Information-seeking QA in 10 WANA languages: 28K long-context pairs"],"cache_read_input_tokens":12672,"weakest_assumption_plain":"The dataset's usefulness assumes that each question's single retrieved Wikipedia article is a plausible place to find the answer, so that \"no answer\" labels reflect genuine reading difficulty rather than failed search.","fun_headline_variants_meta":{"raw":{"variants":["28K long-context QA questions in 10 WANA languages","WANA QA benchmark: 28K questions, 10 varieties, long-article contexts","No-translation QA benchmark for 10 WANA languages with long contexts","Information-seeking QA in 10 WANA languages: 28K long-context pairs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000473,"raw_usage":{"total_tokens":2309,"prompt_tokens":863,"completion_tokens":1446,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":479,"completion_tokens_details":{"reasoning_tokens":1361}},"tokens_in":479,"tokens_out":1446,"duration_ms":14204,"temperature":1.0,"reasoning_tokens":1361,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T14:38:19.583279+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have independent annotators judge whether the retrieved article is on-topic for the question before any answer labeling; if in high-NULL languages such as Tajik and Armenian most retrieved articles are judged off-topic, then the benchmark's non-NULL subset is too small and selective to support the claim that it measures information-seeking comprehension.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the information-seeking QA paradigm, where questions are written independently of the passage, motivating the collection design."},{"cited_title":"CaLMQA: Exploring culturally specific long-form question answering across 23 languages","cited_arxiv_id":"2406.17761","evidence_quote":"Supports the motivation for culturally relevant, in-language question collection rather than translation."}],"review_version":1}