{"id":"5d30a856-93fe-4e06-b453-875096a7884a","arxiv_id":"2502.02047","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A translated Amharic version of SQuAD 2.0 is created with a proximity-aware alignment method, and fine-tuning XLM-R on it improves Amharic extractive QA scores on synthetic and human-curated dev sets.","lead":"This paper translates the English SQuAD 2.0 question-answering dataset into Amharic using Google Translate and a custom alignment step, creating a new benchmark called AmaSQuAD. The authors fine-tune an XLM-R model on this synthetic data and report modest gains on Amharic QA tasks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No independent validation of AmaSQuAD's answer spans: the only quality evidence is the same similarity-scoring algorithm that generated the labels, so a systematic alignment failure would invalidate both the benchmark and the reported fine-tuning gains.","rationale":"I agree with the Reader's overall CONDITIONAL verdict, but I locate the load-bearing concern one level above the proximity heuristic. The proximity assumption is one mechanism by which alignment can fail; the controlling issue is that nothing external checks whether Algorithm 1's spans are correct. The paper's own Section V-A calls human feedback 'essential,' which is an explicit admission that the current validation is incomplete. The Reader's weakest_assumption is therefore partial, not identical: if a human-annotation study showed high agreement, the proximity concern would be empirically answered; if it showed low agreement, the benchmark would be unsound regardless of how the proximity heuristic behaves. The proposed human-annotation check directly settles this. I do not recommend moving to REJECT because a usable resource may still exist and the conditional verdict already requests the missing validation; no verdict change is needed.","tokens_in":9028,"tokens_out":5454,"duration_ms":55869,"concrete_test":"Sample 300 answerable AmaSQuAD training instances stratified by the Algorithm 1 similarity score (e.g., 100 in 0.4-0.6, 100 in 0.6-0.8, 100 in 0.8-1.0). Ask two fluent Amharic speakers to independently locate the answer span in the translated context using the translated question and the English answer as reference, then adjudicate disagreements and compare human spans to Algorithm 1 outputs by exact match and token F1. Report agreement per similarity stratum; if overall exact match is below about 90%, or if low-similarity strata have substantially lower agreement, the labels are too unreliable to support the benchmark and fine-tuning claims. Additionally record the normalized positional shift between the original English answer location and the human-approved Amharic span to test the proximity prior directly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"For the central claim to hold, each AmaSQuAD instance's answer span must be the correct location of the answer in the translated Amharic context. This is the load-bearing premise, and the paper never tests it against human judgment. Algorithm 1 (Section III-B) selects the span by maximizing a weighted cosine-similarity and LCS score while preferring spans near the original English answer position. The only reported quality evidence, Figures 1 and 2, plots the distribution of exactly that same score between the extracted span and the translated answer, so the peak at 0.9-1.0 is partly by construction: the extractor is rewarded for producing high scores. Section V-A explicitly states that incorporating human feedback on dataset quality is 'essential,' but no such evaluation is reported. The proximity heuristic is a concrete failure mode: Amharic is SOV with freer word order than English, so the translated answer can move relative to its English position; if it moves far enough, the near-original-position constraint in Algorithm 1 will select the wrong span. Since the same noisy labels are used to train and evaluate the model on AmaSQuAD dev, the reported +7.86 EM and +7.54 F1 gains could partly reflect learning the label-generation pattern rather than genuine Amharic QA ability. The AmQA improvements (+0.16 EM, +1.00 F1) are too small and unreplicated to independently confirm the method.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AmaSQuAD, an Amharic extractive question-answering benchmark created by translating SQuAD 2.0 with Google Translate. To cope with translation-induced misalignment, the authors propose Algorithm 1, which scores candidate answer spans in the translated context by a weighted combination of cosine similarity and LCS, while preferring spans near the original English answer position. The dataset is filtered to answer spans with similarity at least 0.6, and unanswerable questions are downsampled to 6,000 training and 700 development instances. The authors then fine-tune XLM-R large on this synthetic dataset and report EM/F1 improvements on AmaSQuAD dev (36.55/50.01 to 44.41/57.55) and on the human-curated AmQA dataset (52.50/67.80 to 52.66/68.80). The paper claims this demonstrates a framework for translating extractive QA datasets into low-resource languages.","tokens_in":9338,"tokens_out":2195,"duration_ms":23816,"significance":"If the dataset labels are of sufficient quality, AmaSQuAD would be a valuable and much-needed resource: it is, to the authors' knowledge, the first large-scale Amharic extractive QA dataset, and it is publicly released. The paper also provides a reproducible pipeline and establishes a detailed baseline on the existing AmQA dataset, which previous work had not reported. These strengths are real and should be credited. However, the significance of the central claims—that AmaSQuAD is a usable benchmark and that fine-tuning on it improves Amharic QA—depends on label quality and on the reliability of the reported gains, and the current evidence for both is incomplete.","major_comments":[{"comment":"The load-bearing premise of the alignment algorithm is that the English answer position is a reliable prior for locating the Amharic answer, stated in Section III-B as the intuition that 'the answer ordering in the translated Amharic context won't deviate significantly from the answer in the original English context.' This premise is never tested against human judgment. The only quality evidence, Figures 1 and 2, plots the distribution of the same weighted cosine/LCS score that Algorithm 1 uses to select spans, so high similarity is partly by construction. Because the same automatically produced labels are used for both training and evaluation on AmaSQuAD dev, a systematic alignment failure would inflate the reported gains and invalidate the benchmark. The paper needs an independent validation of a sample of answer spans, e.g., human annotation of a few hundred instances or comparison against an alternative alignment method, to establish that the labels are correct.","section":"Section III-B, Algorithm 1, Figures 1-2"},{"comment":"All reported numbers come from single training runs with no error bars or significance testing. The improvements on the human-curated AmQA dataset are +0.16 EM and +1.00 F1, which are within the run-to-run variation typically observed when fine-tuning transformer models. Since the claim that fine-tuning on AmaSQuAD improves real Amharic QA rests on these small differences, the authors should report means and standard deviations over multiple seeds, or at least show that the difference is stable across hyperparameter settings. Without this, the AmQA improvement cannot be distinguished from noise.","section":"Section IV, Table 1"},{"comment":"The similarity filter threshold of 0.6 and the downsampling of unanswerable questions to 6,000/700 are arbitrary choices that materially change the task distribution. In SQuAD 2.0, unanswerable questions are roughly half of the dataset, whereas in AmaSQuAD they become about 9.4% of the training set and 4.8% of the development set. This affects both the difficulty of the benchmark and the meaning of the EM/F1 numbers. The paper should justify these choices, report sensitivity to the threshold, and clarify whether the development-set performance reflects the intended evaluation protocol for the released dataset.","section":"Section IV (dataset filtering and downsampling)"}],"minor_comments":[{"comment":"The abstract reports the post-fine-tuning F1 on AmaSQuAD dev as 57.5%, while Table 1 reports 57.55%; these should be made consistent.","section":"Abstract and Table 1"},{"comment":"The pseudocode uses undefined functions such as extw, st, and prox, and the stride loop is not clearly explained. Please define these terms and clarify how the proximity prior interacts with the similarity score when multiple candidate spans tie.","section":"Algorithm 1"},{"comment":"The conclusion contains an incomplete sentence ending with 'from 36.55' and should be finished or removed.","section":"Section VI"},{"comment":"The dataset name is spelled inconsistently as AmaSQuAD, AmaSQUAD, and AmSQuAd; please standardize it.","section":"Throughout"},{"comment":"Reference [12] (Abedissa and Libsie) is cited in the related-work section as '[7] developed a non-factoid...' but the citation number does not match; the in-text citation appears to be incorrect.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper's contribution is primarily a new dataset resource, and the writing and experimental depth are currently closer to a workshop paper than a full archival journal article. The central dataset-creation idea is reasonable, but the missing independent label validation and the lack of multiple runs are substantive gaps that need to be addressed before the claims can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper delivers a genuinely new, large Amharic QA resource and a workable translation pipeline, but the quality evidence for the answer spans is internal to the algorithm that produces them, and the cross-dataset gain is too small to independently confirm the method.\n\nWhat is new: AmaSQuAD is a real contribution. It's the first large Amharic extractive QA dataset, and it's public. The closest prior resource, AmQA, has only 2,628 examples and no unanswerable questions. The translation and alignment recipe follows ParSQuAD and TAR, but the proximity-weighted combination of cosine similarity and LCS is a sensible adaptation for Amharic. The paper is clearly written, and the release of the dataset is the kind of concrete artifact that makes it worth engaging.\n\nThe soft spots are real. The big one is circularity. The similarity distributions in Figures 1 and 2 are generated by the same cosine/LCS scoring that selects the answer spans, so a peak at 0.9-1.0 is partly by construction. The paper says human feedback is essential (Section V-A) but no such evaluation is reported. If the proximity heuristic fails—Amharic is SOV with freer word order, and the algorithm explicitly prefers spans near the English answer position—the dataset will contain systematic label errors. Because the same labels are used for training and evaluation on AmaSQuAD dev, the reported +7.86 EM and +7.54 F1 gains could partly reflect learning the label-generation pattern.\n\nThe AmQA results are within plausible noise: +0.16 EM and +1.00 F1 from a single run, no error bars. The claim of being 'the first to establish a detailed baseline' on AmQA also overstates things; the AmQA paper already reports EM/F1 for XLM-R on that dataset, albeit without fine-tuning on AmQA. The hand-picked 0.6 similarity threshold and the heavy downsampling of unanswerable questions are additional choices that shift the task distribution; they need justification or sensitivity analysis.\n\nWho is this for? People working on low-resource QA or multilingual benchmarks. The dataset itself is useful, and the pipeline is cheap to reproduce. But the benchmark is not yet trustworthy as-is. It deserves a serious referee, because the resource is valuable and the main problems are fixable: human-validate a sample of spans, report multiple runs, and correct the characterization of AmQA prior work. I'd send it out, but with the expectation of a substantial revision.","headline":"Genuinely new Amharic QA dataset and a workable pipeline, but the benchmark's validity is undercut by circular validation and the cross-lingual gain is too small to confirm the method.","tokens_in":9886,"tokens_out":3073,"would_cite":true,"duration_ms":28188,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A machine-translated Amharic version of SQuAD 2.0, with answer spans repaired by similarity and proximity, yields a usable QA benchmark and improves Amharic extractive QA when XLM-R is fine-tuned on it.","keywords":["Amharic question answering","AmaSQuAD","SQuAD 2.0","extractive question answering","machine translation alignment","XLM-R fine-tuning","low-resource NLP"],"falsifier":"Construct a control set of Amharic translations in which the answer has been deliberately moved far from its English position; if Algorithm 1 still selects the span nearest the English position rather than the true translated answer, the proximity prior is producing systematic label errors.","tokens_in":8811,"feed_emoji":"📚","tokens_out":9857,"duration_ms":87609,"temperature":0.7,"pith_summary":"This paper sets out to show that a large extractive question-answering dataset can be created for a low-resource language by machine-translating an existing English benchmark and repairing the damage translation does to answer locations. The result is AmaSQuAD, an Amharic version of SQuAD 2.0 whose answer spans are recovered by scoring candidate spans with cosine similarity and longest-common-subsequence similarity while preferring positions near the original English answer. The authors then fine-tune the multilingual XLM-R large model on the synthetic Amharic data and report that exact match and F1 rise from 36.55 to 44.41 percent and from 50.01 to 57.55 percent on AmaSQuAD development data, with smaller gains on the human-curated AmQA dataset. If the method holds, it offers a recipe for bootstrapping QA resources in other languages that lack them.","feed_headline":"Amharic QA benchmark from translated SQuAD lifts F1 by 7.5 points","feed_subtitle":"Fine-tuning XLM-R large on the synthetic set also nudges human-curated AmQA scores up","key_machinery":"The load-bearing mechanism is Algorithm 1, the proximity-similarity-LCS answer extractor. For each translated question, it translates the English answer, splits the translated context into words, and slides a window of the answer's word length across the context with strides 0 through 3. Each candidate span receives a score $$s = w_1 \\cdot \\cos(\\text{emb}(\\text{span}), \\text{emb}(\\text{answer})) + w_2 \\cdot \\frac{\\text{LCS}(\\text{span}, \\text{answer})}{\\max(|\\text{span}|, |\\text{answer}|)}$$ with $w_1 = 2/3$ and $w_2 = 1/3$, and the best-scoring span is kept only if it is also the closest to the original English answer position. This weighted combination is what repairs the misalignment that raw machine translation introduces, and the proximity heuristic is what disambiguates multiple answer instances. A second mechanism is dataset filtering: only answers with similarity at least 0.6 are kept, and unanswerable questions are downsampled to keep the training set tractable.","core_discovery":"The central claim is that answer alignment after machine translation is the real obstacle to creating QA data in low-resource languages, and that a cheap geometric proxy can overcome it. Translating SQuAD 2.0 into Amharic with machine translation does not preserve exact answer spans, so the paper's Algorithm 1 scans the translated context with a window the size of the translated answer, scores each window by $2/3$ cosine similarity (from a BERT model fine-tuned on Amharic) plus $1/3$ LCS-based string similarity, and selects the highest-scoring span whose position is closest to the English answer's position. The paper claims this proximity prior is reliable because the answer ordering in the translated Amharic context does not deviate significantly from the original English context. On the resulting filtered AmaSQuAD dataset, fine-tuning XLM-R large raises EM/F1 from 36.55/50.01 to 44.41/57.55 on AmaSQuAD development and from 52.50/67.80 to 52.66/68.80 on AmQA, which the authors take as evidence that synthetic translated data can support Amharic extractive QA.","pith_inferences":["The proximity prior is the fragile link: for languages whose grammar moves answers further from their English positions, Algorithm 1 would likely select the wrong span even when the similarity signal is decent.","The reported gains compare XLM-R large with and without fine-tuning on the same base; a natural extension is to benchmark mBERT or XLM-R base on the same splits to see whether the improvement is about data quality or model capacity.","Because unanswerable questions were downsampled to 6,000/700, an evaluation set with a realistic mix of answerable and unanswerable Amharic questions would be needed to test whether AmaSQuAD actually teaches rejection of unanswerable questions."],"forward_implications":["AmaSQuAD gives Amharic its first large extractive QA training set, with about 57,650 filtered answerable training questions plus downsampled unanswerable questions in SQuAD 2.0 format.","Fine-tuning on AmaSQuAD is learnable and effective in-domain: EM jumps 7.86 points and F1 7.54 points on AmaSQuAD dev.","The synthetic data transfers, though modestly, to human-curated questions: AmQA EM rises 0.16 points and F1 1.00 point, so training on translated data does not appear to hurt real-language QA.","The same translate-align-filter recipe is proposed for other low-resource languages and other extractive QA source datasets."],"supporting_citations":[{"why":"Supplies the English SQuAD 2.0 contexts, questions, and answer spans that AmaSQuAD translates.","marker":"[6]"},{"why":"Provides the human-curated AmQA dataset used for out-of-domain evaluation and the previous Amharic QA resource.","marker":"[7]"},{"why":"XLM-R is the cross-lingual model family the paper fine-tunes for Amharic QA, chosen because Amharic is in its pretraining data.","marker":"[8]"},{"why":"Establishes the translate-align-retrieve approach for creating synthetic QA data in a low-resource language, which the paper extends.","marker":"[10]"},{"why":"ParSQuAD supplies the sentence-location method for recovering translated answer starts, which the paper adapts with similarity scoring.","marker":"[13]"},{"why":"The fine-tuned multilingual BERT model provides the embeddings used to compute cosine similarity between candidate spans and translated answers.","marker":"[16]"},{"why":"Source of the Longest Common Subsequence algorithm used in the answer-span similarity score.","marker":"[17]"}],"fun_headline_variants":["Amharic QA benchmark from translated SQuAD boosts F1 by 7.5 points","SQuAD translated to Amharic yields QA benchmark with 7.5 F1 gain","Improved Amharic QA via translated SQuAD and similarity-based alignment","From SQuAD to Amharic: new QA benchmark and better extraction","New Amharic QA dataset from SQuAD translation lifts extraction scores"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline assumes the answer's location in the Amharic translation stays close to its location in the original English passage, so location can guide span selection.","fun_headline_variants_meta":{"raw":{"variants":["Amharic QA benchmark from translated SQuAD boosts F1 by 7.5 points","SQuAD translated to Amharic yields QA benchmark with 7.5 F1 gain","Improved Amharic QA via translated SQuAD and similarity-based alignment","From SQuAD to Amharic: new QA benchmark and better extraction","New Amharic QA dataset from SQuAD translation lifts extraction scores"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00084,"raw_usage":{"total_tokens":3703,"prompt_tokens":1030,"completion_tokens":2673,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":646,"completion_tokens_details":{"reasoning_tokens":2566}},"tokens_in":646,"tokens_out":2673,"duration_ms":18333,"temperature":1.0,"reasoning_tokens":2566,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T13:32:24.866061+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a control set of Amharic translations in which the answer has been deliberately moved far from its English position; if Algorithm 1 still selects the span nearest the English position rather than the true translated answer, the proximity prior is producing systematic label errors.","supporting_citations":[{"cited_title":"Automatic Spanish Translation of the SQuAD Dataset for Multilingual Question Answering","cited_arxiv_id":"1912.05200","evidence_quote":"Establishes the translate-align-retrieve approach for creating synthetic QA data in a low-resource language, which the paper extends."},{"cited_title":"Parsquad: machine translated squad dataset for persian question answering,","cited_arxiv_id":null,"evidence_quote":"ParSQuAD supplies the sentence-location method for recovering translated answer starts, which the paper adapts with similarity scoring."},{"cited_title":"Davlan/bert-base-multilingual-cased-finetuned-amharic,","cited_arxiv_id":null,"evidence_quote":"The fine-tuned multilingual BERT model provides the embeddings used to compute cosine similarity between candidate spans and translated answers."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of the Longest Common Subsequence algorithm used in the answer-span similarity score."}],"review_version":1}