{"id":"41b76b59-7311-4d68-a873-013ad0553818","arxiv_id":"2501.17449","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Translation, dataset expansion, and fine-tuning together raise Quran QA retrieval scores, but the experiment does not separate translation's contribution.","lead":"This paper tests whether translating Arabic Quran questions into English and expanding the question set improves automatic Quran answer retrieval. The authors report better scores after fine-tuning seven language models, but the experiment mixes translation with other changes, so the cross-language effect is not isolated.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 2's 'Base vs Ours' comparison confounds cross-language translation with dataset expansion, English corpus choice, and SQuAD v2 pre-fine-tuning, and 'Base' is undefined; therefore the central claim that cross-language translation drives the gain is not supported.","rationale":"The paper's stated contribution is specifically the cross-language approach, yet the experimental design bundles that approach with dataset expansion, a change in the Quran text version/language, and an additional SQuAD v2 fine-tuning stage. Because 'Base' is not defined and no ablation is run, the headline result cannot be attributed to cross-language processing. This is not a matter of disagreeing with community consensus; it is an identification problem in the experimental design. The reader identified the same core weakness, and my read does not change that verdict. The concrete check proposed above would settle the concern: an Arabic-only control with all other interventions unchanged would show whether translation is actually responsible for the improvements. Until such a control exists, the central claim remains unsupported, and the appropriate disposition remains rejection/unverified.","tokens_in":7369,"tokens_out":4800,"duration_ms":45256,"concrete_test":"Run the full 'Ours' pipeline with Arabic source questions and the Arabic Quran instead of English-translated questions and the Pickthall English Quran, holding the expanded dataset, paraphrase procedure, SQuAD v2 pre-fine-tuning, and evaluation split fixed; compare MAP@10, MRR, Recall@10, and Precision@10 against the Table 2 'Ours' numbers. If the Arabic-only variant is within statistical noise of RoBERTa-Base's MAP@10 of 0.34, the cross-language translation and English corpus choice are not the load-bearing factors.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract and conclusion claim that 'this cross-language approach significantly improves model performance', but the evidence in Table 2 does not isolate the cross-language component. The 'Ours' condition simultaneously (1) expands the dataset from 251 to 1,895 questions, (2) translates Arabic questions to English via Google Translate, (3) switches the answer corpus from the Arabic Quran to Pickthall's English translation, and (4) pre-fine-tunes on SQuAD v2 before Quran QA fine-tuning. The 'Base' row is never defined in Section 3, Section 4, or the table caption, so the reader cannot tell whether it is a zero-shot model, a SQuAD-only model, an Arabic-input model, or an official shared-task baseline. No ablation or Arabic-only control separates translation from the other interventions. Additionally, the paper never states how the 1,895-question final dataset is split for fine-tuning versus evaluation; if the original 52 test questions were included in the expanded set, the reported gains could partly reflect training on the test instances. Any one of these confounds would make the causal attribution to 'cross-language' unidentifiable, so the central claim is currently unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a cross-language approach to Quranic question answering. The authors expand a small Arabic QA dataset (251 questions) into 1,895 English questions via external resources, paraphrasing, and Google Translate, retrieve answers from Pickthall's English translation of the Quran, and fine-tune several language models and large language models with a cross-encoder architecture. They report evaluation results on MAP@10, MRR, Recall@10, and Precision@10, claiming that the cross-language approach significantly improves performance, with RoBERTa-Base achieving the highest MAP@10 (0.34) and MRR (0.52), and DeBERTa-v3-Base excelling in Recall@10 (0.50) and Precision@10 (0.24). However, the paper's central comparison in Table 2 (Base vs. Ours) is not interpretable because 'Base' is never defined and the 'Ours' condition bundles several simultaneous changes, so the central claim is not supported by the reported experiments as written.","tokens_in":7606,"tokens_out":5277,"duration_ms":47253,"significance":"If the claimed improvements were properly isolated, the work could be a useful contribution to low-resource Quranic QA, an important domain for a large population of users. The paper addresses a genuine linguistic challenge (MSA questions versus Classical Arabic passages), evaluates multiple models, and uses standard retrieval metrics. The main strength is the breadth of the experimental effort. However, the current experimental design does not allow one to attribute the reported gains to the cross-language component: there is no defined baseline, no ablation, and no control condition. The also-important issue of whether the test set overlaps with the expanded training set is left unexplained. As a result, the paper's central claim is not yet verified, and the practical significance of the numbers is unclear without comparison to prior published results on the same test set.","major_comments":[{"comment":"The 'Base' condition in Table 2 is never defined. The caption says 'baseline (Base)' but the text provides no description of what Base is (e.g., zero-shot, Arabic-input, SQuAD-only, or a shared-task baseline). At the same time, the 'Ours' condition changes four variables jointly: dataset size increases from 251 to 1,895 questions, questions are translated from Arabic to English via Google Translate, the answer corpus changes from the Arabic Quran to Pickthall's English translation, and models are pre-fine-tuned on SQuAD v2 before Quran QA fine-tuning. Because the experiment does not vary these factors independently, the improvement from Base to Ours cannot be attributed specifically to the cross-language approach, which is the paper's central claim. The authors should define Base explicitly and run ablations (e.g., Arabic-only with the expanded dataset and the same fine-tuning schedule) to isolate the effect of translation.","section":"Section 3.1-3.3, Table 2"},{"comment":"The paper never states how the 1,895-question expanded dataset is split into training, development, and test sets. The original Quran QA 2023 dataset has 174 training, 52 test, and 25 development questions. If any of the 52 original test questions were included in the expansion and used during fine-tuning, the reported scores in Table 2 would be optimistic and potentially invalid. The authors must specify exactly which questions were used for the evaluation in Table 2 and confirm that no test question, original or expanded, was present in the training data.","section":"Section 3.1 and Section 4"},{"comment":"The paper reports only the relative improvements of 'Ours' over the undefined 'Base' and does not compare against previously published results on the same Quran QA 2023 shared task. For example, related work in Section 2 mentions a MAP of 0.25 by Elkomy and Sarhan; the paper's own DeBERTa-v3-Base achieves MAP@10 of 0.32, but it is unclear whether these numbers are comparable because the evaluation protocols (test sets, metrics, and passage pools) are not stated to be the same. To demonstrate practical significance, the authors should evaluate their models under the official shared-task setting or at least directly compare with prior systems on the same test questions and metrics.","section":"Section 4, Results"},{"comment":"The training procedure is underspecified, making the results difficult to reproduce or assess. The paper does not report hyperparameters (learning rate, batch size, number of epochs), the number of negative passages per question, the loss function, or how the cross-encoder is implemented (e.g., which model serves as the encoder, how positive and negative passages are sampled). The statement in Section 3.4 that 'contrastive learning could further help' is speculative and not actually implemented, adding to the ambiguity. These details are necessary for confidence in the reported numbers.","section":"Section 3.3 and 3.4"}],"minor_comments":[{"comment":"The phrase 'Fine-tuned LLMs, such as BERT-Medium' is inaccurate because BERT-Medium is a small language model, not a large language model; the authors should consistently use 'LMs and LLMs' throughout.","section":"Section 1, last paragraph"},{"comment":"Bloom is listed with 7 billion parameters, but the cited reference is for the 176B-parameter Bloom; if a smaller Bloom variant was used, the reference should be updated and the model checkpoint should be specified.","section":"Table 1"},{"comment":"The caption says the retrieved passage is 'translated back into Arabic,' but no back-translation step is described in the methodology or shown to be part of the evaluation; the authors should either clarify or remove this step from the figure.","section":"Figure 2 caption"},{"comment":"The notation for the metrics is inconsistent: 'MAP10' in the table and results text versus 'MAP@10' in the abstract; the same applies to Recall and Precision. Please unify the notation.","section":"Section 4 and Abstract"},{"comment":"Reference [15] is cited as a HuggingFace dataset URL rather than a proper citation to the SQuAD v2 paper; the citation should be completed with the original reference.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses an important problem, but the experimental design is currently not sufficient to support the central claim. The lack of a defined baseline and the absence of ablations are serious but potentially fixable with additional experiments. The missing description of the train/test split raises a potential contamination concern that must be resolved. I recommend major revision rather than outright rejection because the core idea is defensible and the required changes (defining baselines, adding ablations, and reporting the split) are within the scope of a revised manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know before you look at this one. First, the paper's headline claim—that cross-language translation improves Quranic QA—is not actually demonstrated by the experiments. The 'Ours' condition bundles four changes at once: dataset expansion from 251 to 1,895 questions, translation via Google Translate, a switch to Pickthall's English Quran, and SQuAD v2 pre-fine-tuning. The 'Base' row in Table 2 is never defined in the methodology or the caption, so you cannot tell what is being compared. Second, even if the claim held, the approach is not new; Alawwad et al. (2023), cited in the paper, already used translation and paraphrasing for the same Quranic passage retrieval task. What is new is the scale: a larger augmented dataset, a wider model zoo, and the SQuAD v2 pre-fine-tuning step.\n\nTo give credit: the paper is honest about its relation to prior work, the workflow is clearly diagrammed, and the dataset expansion effort is real—though no data or code is released. The held-out test set does mean the absolute numbers are not circularly fitted, and the comparisons across seven models are systematic.\n\nThe soft spots are real and load-bearing. The attribution problem is central: without an Arabic-only control or ablations, the gains in Table 2 could come from any of the four changes, including the English corpus switch. The undefined 'Base' makes the comparison unidentifiable. A second concern, which the paper leaves ambiguous, is whether the 52 original test questions were included in the expanded 1,895-question set before fine-tuning; Figure 1 suggests expansion is applied to the whole dataset, but the text doesn't clarify. If test instances were trained on, that is a leak. I'd call this a moderate concern rather than a confirmed flaw, but the authors need to state the split explicitly. Minor issues: no error bars or significance tests, so 'significantly improves' is rhetorical, and the absolute numbers (MAP@10 0.34) are modest.\n\nWho is this for? Researchers working on low-resource religious text QA or cross-language retrieval. The dataset, if released, could be a useful asset. As it stands, the paper is a well-organized but under-controlled extension of known ideas. I would send it to peer review rather than desk reject, because the confound is fixable and the dataset has potential value, but I would require an Arabic-only baseline, a defined and justified 'Base', a clear train/test split, and a release of data and code. Without those, the conclusion should be limited to 'translation enables scale' rather than 'translation improves performance.'","headline":"A well-organized but under-controlled extension of known translation-based Quranic QA; the headline claim is confounded by an undefined baseline and bundled changes, so the paper needs major revision before its conclusions can be trusted.","tokens_in":8128,"tokens_out":2752,"would_cite":false,"duration_ms":27182,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that translating Arabic Quran questions into English and fine-tuning pre-trained language models on the expanded English dataset significantly improves Quranic passage retrieval, with RoBERTa-Base reaching the best…","keywords":["Quran Question Answering","Passage Retrieval","Modern Standard Arabic","Classical Arabic","Dataset Expansion","Fine-Tuning","Cross-Language Approach"],"falsifier":"Run the same 1,895-question expanded dataset entirely in Arabic—Arabic questions, an Arabic Quran text, and an Arabic cross-encoder—and compare MAP@10 and MRR with the English pipeline; if the Arabic run matches or beats the English numbers, translation is not the active ingredient. A complementary test would translate the questions but keep the original 251-question dataset, isolating translation from dataset expansion.","tokens_in":7155,"feed_emoji":"📖","tokens_out":7966,"duration_ms":65729,"temperature":0.7,"pith_summary":"This paper proposes a cross-language recipe for Quranic question answering: machine-translate the Arabic questions into English, paraphrase each question to multiply the data, retrieve answers from an English translation of the Quran, and fine-tune English-optimized transformer models on top. The authors report that this recipe clearly raises passage-retrieval scores over the base versions of the same models, with RoBERTa-Base reaching MAP@10 of 0.34 and MRR of 0.52 and DeBERTa-v3-Base reaching Recall@10 of 0.50 and Precision@10 of 0.24. The motivation is to sidestep the gap between Modern Standard Arabic questions and Classical Arabic Quranic verses by moving the whole task into English, where stronger pre-trained models are available. A sympathetic reader would take this as evidence that cross-language transfer is a practical path for low-resource scriptural QA.","feed_headline":"Arabic-to-English translation boosts Quran QA retrieval scores","feed_subtitle":"RoBERTa-Base lifts MAP@10 to 0.34 and MRR to 0.52 on 1,895 translated questions.","key_machinery":"The mechanism carrying the argument is the cross-language data pipeline: Arabic questions are machine-translated, paraphrased twice for linguistic diversity, and paired with passages from an English translation of the Quran, so that English-optimized transformer models can be fine-tuned as cross-encoders—networks that take a question and a passage as a single input and emit a relevance score. That pipeline converts 251 original questions into 1,895 English training items. The paper's conclusion is carried by the 'Base' versus 'Ours' comparison in Table 2, where 'Ours' bundles translation, dataset expansion, and SQuAD v2 pre-fine-tuning.","core_discovery":"The central claim, in the authors' terms, is that the combination of dataset expansion and cross-language retrieval—growing the original 251-question shared-task set to 1,895 paraphrased English questions, using an English translation of the Quran as the passage corpus, pre-fine-tuning on SQuAD v2, and then fine-tuning seven transformer models as cross-encoders—improves performance over each model's baseline on every metric reported. In their results, ELECTRA-Large shows the largest relative jump (MAP@10 from 0.04 to 0.31), RoBERTa-Base gives the best ranking (MAP@10 0.34, MRR 0.52), and DeBERTa-v3-Base gives the best coverage (Recall@10 0.50, Precision@10 0.24). The authors interpret these gains as showing that English-language models can overcome the MSA-to-Classical Arabic linguistic divide.","pith_inferences":["A natural next step the authors leave implicit is an ablation that separates translation from dataset expansion; without it, the reported gains cannot be attributed specifically to the cross-language choice.","The same translate-and-fine-tune recipe could be tested on other classical-language corpora, such as hadith or medieval Arabic texts, where a high-resource target language might unlock otherwise unavailable retrieval quality.","A hybrid system that uses the English model for ranking but maps passages back to the original Arabic verses would preserve access to the Arabic text while keeping the ranking gains; the paper stops at the English translation."],"forward_implications":["Teams working on Quranic QA can adopt the expanded 1,895-question English dataset and cross-encoder setup as a ready baseline instead of building Arabic-specific systems from scratch.","RoBERTa-Base is the recommended model when ranking quality matters most, since it attains the highest MAP@10 (0.34) and MRR (0.52).","DeBERTa-v3-Base is the recommended model when coverage matters most, with Recall@10 of 0.50 and Precision@10 of 0.24.","The reported gains imply that even large 7-billion-parameter models (Bloom, Falcon) benefit from the translated dataset, suggesting that resource scale does not remove the need for cross-language data preparation."],"supporting_citations":[{"why":"Supplies the original 251-question shared-task dataset and the passage-retrieval task definition that the paper expands.","marker":"[5]"},{"why":"Defines a hybrid dual-encoder/cross-encoder baseline with MAP 0.25 that motivates the need for dataset expansion and cross-language methods.","marker":"[9]"},{"why":"Establishes the earlier translation-based passage retrieval approach that this paper extends to a larger dataset and more models.","marker":"[13]"},{"why":"Provides the SQuAD v2 corpus used to pre-fine-tune all models before task-specific fine-tuning.","marker":"[15]"},{"why":"Supplies the English translation of the Quran used as the retrieval corpus.","marker":"[18]"},{"why":"Cited to support the premise that pre-trained models perform more effectively on English data, which motivates translating the questions.","marker":"[19]"}],"fun_headline_variants":["Cross-lingual retrieval lifts Quran QA: RoBERTa tops MAP@10","English Quran translation boosts answer retrieval","RoBERTa-Base wins Quran QA retrieval with cross-lingual data","Cross-language fine-tuning improves Quranic QA retrieval scores"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion depends on the 'Base' versus 'Ours' comparison in Table 2 being a valid isolation of the cross-language approach, but 'Ours' changes the dataset size, the language, and the pre-training step all at once, and the paper never defines what 'Base' means.","fun_headline_variants_meta":{"raw":{"variants":["Cross-lingual retrieval lifts Quran QA: RoBERTa tops MAP@10","English Quran translation boosts answer retrieval","RoBERTa-Base wins Quran QA retrieval with cross-lingual data","Cross-language fine-tuning improves Quranic QA retrieval scores"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001164,"raw_usage":{"total_tokens":4848,"prompt_tokens":1008,"completion_tokens":3840,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":624,"completion_tokens_details":{"reasoning_tokens":3770}},"tokens_in":624,"tokens_out":3840,"duration_ms":26198,"temperature":1.0,"reasoning_tokens":3770,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T04:42:36.070160+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 1,895-question expanded dataset entirely in Arabic—Arabic questions, an Arabic Quran text, and an Arabic cross-encoder—and compare MAP@10 and MRR with the English pipeline; if the Arabic run matches or beats the English numbers, translation is not the active ingredient. A complementary test would translate the questions but keep the original 251-question dataset, isolating translation from dataset expansion.","supporting_citations":[{"cited_title":"In: Proceedings of ArabicNLP 2023, pp","cited_arxiv_id":null,"evidence_quote":"Supplies the original 251-question shared-task dataset and the passage-retrieval task definition that the paper expands."},{"cited_title":"In: Proceedings of ArabicNLP 2023, pp","cited_arxiv_id":null,"evidence_quote":"Defines a hybrid dual-encoder/cross-encoder baseline with MAP 0.25 that motivates the need for dataset expansion and cross-language methods."},{"cited_title":"In: Proceedings of ArabicNLP 2023, pp","cited_arxiv_id":null,"evidence_quote":"Establishes the earlier translation-based passage retrieval approach that this paper extends to a larger dataset and more models."},{"cited_title":"Huggingface (2018)","cited_arxiv_id":null,"evidence_quote":"Provides the SQuAD v2 corpus used to pre-fine-tune all models before task-specific fine-tuning."},{"cited_title":"Huggingface Repository","cited_arxiv_id":null,"evidence_quote":"Supplies the English translation of the Quran used as the retrieval corpus."},{"cited_title":"Procedia Computer Science225, 12–21 (2023)","cited_arxiv_id":null,"evidence_quote":"Cited to support the premise that pre-trained models perform more effectively on English data, which motivates translating the questions."}],"review_version":1}