{"id":"97867ac9-127c-43b0-9b87-4d87fc7f3598","arxiv_id":"2505.03688","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"IndicSQuAD is a large extractive QA dataset for ten Indic languages, translated from SQuAD 2.0, with baseline evaluations using monolingual BERT models and MuRIL-BERT.","lead":"This paper introduces IndicSQuAD, a large multilingual question-answering dataset for ten Indian languages, created by translating the English SQuAD 2.0 dataset. It reports baseline fine-tuning results with monolingual BERT models and MuRIL-BERT, showing that monolingual models generally perform better.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Answer-span alignment quality is asserted, not verified; the 1% similarity threshold from Marathi is ported without evidence to nine languages, so the dataset's core usability claim is unsupported.","rationale":"The reader's weakest assumption is the same load-bearing concern I identified: the similarity-based span alignment, ported from Marathi, is unverified for the nine new languages. My review read the pipeline description (§3.3) and the dataset statistics (§3.4) and confirmed that there is no human evaluation, no error analysis, and no gold verification set for the translated spans, while the prior MahaSQuAD work at least had a 500-example manually verified gold test set. This directly affects the dataset's primary contribution: if spans are misaligned, extractive QA training labels are wrong and the reported F1/EM baselines are not trustworthy evidence of model capability. I do not see a separate fatal flaw: the resource is public, the baselines are plausible, and the 'largest' claim, though under-supported, is less damaging because even if comparable benchmarks exist, the dataset's own scale is documented. The fix is feasible without discarding the work: add a human or high-precision automated verification study of span alignment across languages, and report per-language alignment accuracy, ideally conditioned on answer type and sentence length. If verification succeeds, the paper can be accepted; if alignment is poor for some languages, the paper needs revision. Hence the verdict remains CONDITIONAL, matching the reader.","tokens_in":8687,"tokens_out":1337,"duration_ms":12333,"concrete_test":"Sample at least 200 randomly chosen examples per new language (1,800 total) plus 100 per language from SQuAD 2.0's unanswerable set, have a native speaker mark whether the stored answer span exactly matches the translated answer, and compare per-language exact-span agreement and unanswerable-case handling. If any language falls below roughly 95% exact-span agreement, the ported 1% similarity threshold is not language-robust and the data-quality claim needs revision.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that IndicSQuAD provides 'accurate answer-span alignment across diverse languages' (Abstract), which is what makes the translated SQuAD instances usable for extractive QA. Section 3.3's algorithm selects the answer span by similarity-based substring matching inside each translated sentence, with the 1% extension threshold described as derived for Marathi. The paper reports no manual verification for the nine new languages and no error analysis on the span alignment. If similarity-based matching degrades on morphologically rich Dravidian or agglutinative languages such as Tamil, Telugu, Kannada, or Malayalam, shifted or wrong spans would directly corrupt training labels and inflate the apparent model performance gap or hide label noise. The paper's own MahaSQuAD predecessor included a gold test set of 500 manually verified examples; IndicSQuAD does not provide an equivalent verification set. Without a labeled evaluation of alignment accuracy, the Abstract's 'accurate answer-span alignment' claim is unsupported for nine languages.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces IndicSQuAD, an extractive question-answering dataset constructed by translating SQuAD 2.0 into ten Indic languages (Marathi plus nine new languages) using a pipeline adapted from the MahaSQuAD work. The paper reports splits of 118,516 training, 11,873 validation, and 11,803 test examples per language, evaluates monolingual BERT models and MuRIL-BERT on each language, and claims that the translation procedure preserves linguistic fidelity and accurate answer-span alignment. It further claims that IndicSQuAD is the largest multi-Indic QA resource to date and that monolingual models generally outperform the multilingual baseline.","tokens_in":8988,"tokens_out":5534,"duration_ms":53401,"significance":"If the span-alignment and linguistic-fidelity claims were verified, IndicSQuAD would be a substantial resource: roughly 118k training examples per language for ten languages would support training and benchmarking of extractive QA in languages that currently lack large-scale resources. The public release of the dataset and model links is a practical strength, and building on the documented MahaSQuAD pipeline gives a concrete starting point. However, the strongest claims in the abstract are not backed by human verification or error analysis for the newly added languages, and the methodology section contains language-specific artifacts that call into question whether the described procedure was actually applied uniformly to all target languages.","major_comments":[{"comment":"The algorithm is written entirely for Marathi: Step 3 says the sentence and answer are 'translated into Marathi (target language)', and Step 6 says named entities and numerical values are 'transliterated into the Devanagari script using the AI4Bharat Transliteration Engine'. Table 2, however, lists Tamil, Telugu, Kannada, and Malayalam, which use non-Devanagari scripts. As written, applying this pipeline to those languages would produce Devanagari text instead of the target script, which contradicts the dataset's stated script coverage and leaves the 'accurate answer-span alignment' claim unsupported for those languages. The manuscript must specify how the translation and transliteration steps are specialized for each target language and script.","section":"Section 3.3, Steps 3 and 6"},{"comment":"The paper reports no manual verification or error analysis of the automatically selected answer spans for the nine languages added in this work. Section 2.1 notes that MahaSQuAD included a gold test set of 500 manually verified examples, but no equivalent is reported for IndicSQuAD. Because the span-selection method is an unsupervised similarity search (Step 4) with a fixed 1% extension tolerance (Step 5), and because morphologically rich Dravidian languages may challenge substring matching, the central claim of 'accurate answer-span alignment across diverse languages' is not currently supported. A language-wise alignment accuracy estimate on a manually checked sample, or a substantial weakening of the claim, is needed.","section":"Section 3.3, Steps 4-5; Section 2.1"},{"comment":"The baseline comparisons are presented without error bars, multiple random seeds, or significance tests. Several differences are small (e.g., Bengali EM 50.07 vs 49.36; Punjabi EM 51.04 vs 50.80), and the conclusion that monolingual models outperform MuRIL-BERT across most languages could be an artifact of run-to-run variance. The authors should report standard deviations and statistical tests, or soften the comparative claims accordingly.","section":"Section 4.3, Table 1"},{"comment":"The language count is inconsistent: the abstract and conclusion describe 'nine major Indic languages', while contribution 1 and Table 2 list ten languages including Marathi, and Section 3.2 says the dataset covers 9 languages but then discusses ten. Since 'nine new languages plus Marathi' versus 'ten total' is a basic factual claim about the resource, the manuscript should state the count consistently.","section":"Abstract; Section 3.2; Table 2; Conclusion"}],"minor_comments":[{"comment":"The same dataset is introduced twice as 'ChaII Dataset' and 'ChAII Dataset', and Section 2.3 attributes it to Singh et al. (2025) although the surrounding text and the original reference are Thirumala and Ferracane (2022). Please merge the duplicate descriptions and correct the citation.","section":"Section 2.2 and Section 2.3"},{"comment":"The hyperparameter n_best_size is set to 2, which is unusually low for extractive QA and may strongly restrict the candidate answer spans. Please clarify whether this is the intended value and whether it affects the reported baseline numbers.","section":"Section 4.2"},{"comment":"The rows of Table 3 are labeled 'entities' rather than 'instances' or 'examples', and Section 3.1 says the source is SQuAD 2.0 but does not specify whether the counts correspond to the full SQuAD 2.0 split. Please align the terminology and identify the exact source split.","section":"Section 3.4 and Table 3"},{"comment":"The MuCoT work appears twice as Kumar et al. 2022a and 2022b with slightly different venue formatting; the duplicate entry should be removed or merged.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The dataset could be a useful resource, but the verifiability of span alignment is the crux of the contribution. At minimum, the authors should mirror the MahaSQuAD gold-test-set idea with a manually verified sample and report alignment accuracy per language, and they must correct the Marathi-specific copy-paste artifacts in Section 3.3. If the authors cannot provide this verification, the abstract's 'accurate answer-span alignment' claim should be removed and the contribution reframed as a resource release with algorithmically aligned spans."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The real contribution here is simple and legitimate: a large translated SQuAD 2.0 dataset for ten Indic languages, publicly released, with monolingual and multilingual baselines. That is what a resource paper should do, and the authors have done it. The scale (118k training examples per language) and the uniform coverage across Indo-Aryan and Dravidian languages make it a potentially valuable training and benchmarking asset. I also credit them for building openly on MahaSQuAD and for releasing the data and models.\n\nWhat the paper does not do is verify the thing that makes the dataset usable: the claim that answer spans are accurately aligned in the translated passages. Section 3.3 describes an automated similarity-based substring selection with a 1% extension threshold, ported directly from the Marathi pipeline. There is no manual verification for the nine new languages, no error analysis, and no equivalent of MahaSQuAD's 500-example gold test set. For morphologically rich languages like Tamil or Telugu, the alignment could degrade silently, which would corrupt training labels and make the baselines meaningless as quality evidence. The abstract calls this \"accurate answer-span alignment\" — that is an assertion, not a demonstrated result.\n\nThe copy-paste artifacts do not help. Step 3 of the algorithm says the answer is translated \"into Marathi (target language)\", and step 6 says named entities are transliterated into Devanagari, which is wrong for Bengali, Tamil, Telugu, etc. That tells me the methodology section was written from the MahaSQuAD template and not fully adapted. It is a fixable editorial problem, but it makes me less confident that the pipeline was actually run correctly for all languages.\n\nThe \"largest multi-Indic QA resource\" claim is also weakly substantiated. The related work mentions the Indic-QA Benchmark covering 11 languages, and the paper does not provide a size comparison that would support \"largest.\" The baselines lack error bars or significance tests, though for a dataset paper I treat that as minor.\n\nNone of this is fatal. The dataset can still be useful even if the alignment quality is imperfect, and the authors have shared it so others can check. But the paper needs a serious revision: manual verification on a sample for each new language, or at least a rigorous error analysis; a corrected methodology section; and a honest comparison with existing benchmarks. I would send this to peer review with a request for major revision. The resource deserves scrutiny, and with the right evidence it could be a solid contribution.","headline":"A genuinely useful translated QA resource for ten Indic languages, but the paper's central claim about answer-span accuracy is unverified; needs human evaluation before it can be fully trusted.","tokens_in":9386,"tokens_out":1331,"would_cite":false,"duration_ms":15068,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"IndicSQuAD delivers translated SQuAD datasets for ten Indic languages, claiming the largest multi-Indic extractive QA resource to date.","keywords":["question answering","Indic languages","extractive QA","translated datasets","answer span alignment","low-resource NLP","multilingual BERT","SQuAD"],"falsifier":"Have native speakers annotate a random sample of, say, 300 answer spans per language from the released test sets, marking whether the stored span is the exact and only correct answer to the question within the passage; if the exact-match rate is substantially below the near-perfect alignment the pipeline claims, the central fidelity claim is weakened.","tokens_in":8499,"feed_emoji":"🌐","tokens_out":8328,"duration_ms":74857,"temperature":0.7,"pith_summary":"The paper tries to establish that a single translation pipeline can turn the English SQuAD 2.0 dataset into large, usable extractive question-answering datasets for nine new Indic languages alongside the previously released Marathi set. The core claim is that the pipeline preserves high linguistic fidelity and accurate answer-span alignment, yielding 118,516 training, 11,873 validation, and 11,803 test examples per language. If true, this matters because most of these languages have no QA resource of this scale, giving researchers a common controlled source for training and benchmarking. The paper also reports that monolingual BERT models generally outperform the multilingual MuRIL-BERT on these datasets, which points to open challenges in low-resource Indic-language modeling.","feed_headline":"Ten Indic languages get 118,000-question QA datasets","feed_subtitle":"The resource gives each of ten Indic languages enough extractive QA data to train and benchmark models","key_machinery":"The load-bearing mechanism is a similarity-based answer-span alignment algorithm. After sentence segmentation, the sentence containing the answer and the answer string are translated independently; then every substring of the translated sentence is compared against the translated answer using embedding-based similarity scores. The best substring is chosen as the base answer, and adjacent words are appended iteratively as long as the score stays within 1% of the maximum, with named entities and numbers transliterated in a final step. This mechanism is what is supposed to guarantee that translated answers line up with their passages despite word-order and morphological differences.","core_discovery":"The paper's central claim is that a robust, automated translation-and-alignment recipe, originally built for Marathi, scales to a ten-language Indic QA resource. For each language, the recipe segments English contexts into sentences, translates the answer-bearing sentence and the answer separately, then uses embedding similarity to select the substring of the translated sentence that best matches the translated answer, expanding it word by word while similarity stays within 1% of the maximum. Named entities and numbers are transliterated to maintain script consistency. The paper reports that this produces, for each of the ten languages, a dataset identical in size to the Marathi one and presents the collection as the largest multi-Indic QA resource to date, with baseline results favoring language-specific BERT models over MuRIL-BERT.","pith_inferences":["A per-language audit of answer-span boundaries would test whether the 1% similarity acceptance threshold is equally appropriate for Dravidian and Indo-Aryan languages, since the reported human verification is limited to Marathi.","Because all ten datasets derive from a common English source, cross-language performance gaps mix genuine linguistic difficulty with translation artifacts; treating them purely as language difficulty would overread the benchmark.","The transliteration of named entities into Devanagari may introduce script-mixed passages in languages that use other scripts, so script normalization is a cheap, testable modification that could shift the reported scores.","A small, manually verified gold set per language, along the lines of the one reported for Marathi, would let future work quantify span-alignment noise and could serve as a filtered training subset."],"forward_implications":["Each of the ten languages gains 118,516 training examples, a scale that previously existed for only a few of them.","Researchers can fine-tune language-specific BERT models on identical data sizes, making cross-language comparisons of model ability more direct.","The pipeline, if truly language-agnostic, can be rerun on additional low-resource languages to produce comparable QA sets.","Because all languages are derived from the same English source, the ten datasets form a controlled testbed for measuring how well multilingual models transfer across Indic languages.","The reported baselines indicate that monolingual models remain the stronger choice when per-language training data is available, while multilingual models lag on the lower-resource languages."],"supporting_citations":[{"why":"Supplies the English SQuAD contexts, questions, and answer spans that are translated into each Indic language.","marker":"(Rajpurkar et al., 2016)"},{"why":"Defines the Marathi dataset and its robust span-alignment algorithm, which the paper extends to nine more languages.","marker":"(Ruturaj et al., 2023)"},{"why":"Provides the SimilarityAnalyzer tool used to score substrings against the translated answer during span alignment.","marker":"(Joshi, 2022c; Magdum et al., 2023)"},{"why":"Supplies the embedding models that produce the similarity scores used in the alignment step.","marker":"(Deode et al., 2023)"},{"why":"Provides MuRIL-BERT, the multilingual baseline model compared against the monolingual BERT models.","marker":"(Khanuja et al., 2021)"},{"why":"Supplies a language-specific BERT baseline used for evaluation in the Hindi case.","marker":"(Joshi, 2022a)"},{"why":"Supplies the monolingual Marathi BERT baseline and the Marathi corpus context for the earlier dataset.","marker":"(Joshi, 2022b)"}],"fun_headline_variants":["IndicSQuAD: 118K QA questions for 10 Indic languages","Ten Indic languages get 118K-question QA datasets","Automated translation yields QA datasets for 10 Indic languages","New QA benchmark covers 10 Indic languages with 118K questions each","Largest multi-Indic QA resource: 10 languages, 118K questions each"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The dataset's usefulness rests on the assumption that the 1% similarity-based substring selection finds the correct answer span in every translated passage, even though this was only manually verified for Marathi and not for the nine new languages.","fun_headline_variants_meta":{"raw":{"variants":["IndicSQuAD: 118K QA questions for 10 Indic languages","Ten Indic languages get 118K-question QA datasets","Automated translation yields QA datasets for 10 Indic languages","New QA benchmark covers 10 Indic languages with 118K questions each","Largest multi-Indic QA resource: 10 languages, 118K questions each"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001309,"raw_usage":{"total_tokens":5303,"prompt_tokens":880,"completion_tokens":4423,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":496,"completion_tokens_details":{"reasoning_tokens":4328}},"tokens_in":496,"tokens_out":4423,"duration_ms":26593,"temperature":1.0,"reasoning_tokens":4328,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:44:08.547600+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have native speakers annotate a random sample of, say, 300 answer spans per language from the released test sets, marking whether the stored span is the exact and only correct answer to the question within the passage; if the exact-match rate is substantially below the near-perfect alignment the pipeline claims, the central fidelity claim is weakened.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Marathi dataset and its robust span-alignment algorithm, which the paper extends to nine more languages."}],"review_version":1}