{"id":"7dccef56-85f7-41be-a66a-781a807be40c","arxiv_id":"2507.23404","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"An Arabic dense retriever using a trainable attentive scoring module instead of dot-product similarity reports improved top-k passage retrieval on ArabicaQA.","lead":"This paper adds a learned relevance-scoring module to a dense passage retrieval system for Arabic, replacing the standard dot-product similarity between questions and passages. On an Arabic question-answering benchmark it reports small but consistent top-k retrieval gains over existing Arabic retrievers.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never isolates the ARS module: Section 5 explicitly defers ablations to future work, so the central claim that ARS drives the Table 1 gains is unsupported even if baseline numbers are perfectly controlled.","rationale":"The reader's weakest assumption concerns whether baseline numbers came from controlled comparisons. That is a legitimate threat to the existence of the reported improvement. However, the more load-bearing issue is attribution of the improvement to ARS: Section 5 admits ablations are future work, so even a perfectly controlled Table 1 would not establish the paper's central claim. The reader mentions missing ablations in the rationale but identifies baseline comparability as the weakest assumption. I partially agree: baseline comparability is necessary, but the absent ARS ablation is more decisive because it targets the stated novelty. Since the paper is otherwise coherent and the missing evidence is addressable, the CONDITIONAL verdict remains appropriate; I do not recommend moving to ACCEPT or REJECT based on this pass.","tokens_in":7980,"tokens_out":4223,"duration_ms":44499,"concrete_test":"Run a single ablation matrix on the ArabicaQA train/validation/test split using the same Wikipedia index, preprocessing, and negative sampling as the paper: (A) MiniBERT dual encoder with InfoNCE only, scored by dot product; (B) MiniBERT dual encoder with InfoNCE + Ldyn + Lreg, still scored by dot product; (C) full APR with ARS. If Top-10 accuracy of (B) and (C) differ by less than about one absolute point, or if (A) matches (C), then ARS is not the driver; if (C) clearly beats both (A) and (B), the central claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central novelty claim is that the Attentive Relevance Scoring (ARS) module—not the MiniBERT encoder, the dynamic relevance loss, or the regularization loss—produces the observed retrieval gains. That claim is never tested. Section 5 explicitly states: 'we recognize that we need a more detailed analysis. Future work will focus on doing thorough ablation studies to separate the contributions of our proposed components.' APR differs from the DPR and AraDPR baselines along at least four dimensions simultaneously: (i) encoder initialization (MiniBERT vs. AraBERT), (ii) the ARS scoring function in Eqs. 2–4, (iii) the dynamic relevance loss Ldyn in Eq. 7, and (iv) the logit regularization Lreg in Eq. 8. Because no configuration removes ARS while keeping the other components fixed, the sentence in Section 4.3 claiming that the results 'demonstrate that APR effectively leverages the ARS module' is an attribution without a control. The baseline-comparability question is real, but it is secondary: even if all Table 1 numbers came from the same harness, split, and index, the causal role of ARS would remain unsupported. This is a missing-support issue, not an internal inconsistency.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes APR, an Arabic dense passage retrieval model built on dual MiniBERT encoders and a trainable Attentive Relevance Scoring (ARS) module that replaces dot-product scoring with a learned interaction layer (Eqs. 2–4). Training combines an InfoNCE contrastive loss with a dynamic relevance loss and a logit regularization term (Eqs. 5–8). The authors evaluate on ArabicaQA and report top-k retrieval accuracy (Table 1) showing APR above TF-IDF, BM25, DPR, and AraDPR at all cutoffs. The central claim is that the ARS module is responsible for these improvements, but the paper provides no ablation that isolates ARS or the auxiliary losses, and the baseline comparison setup is not documented in enough detail to rule out confounds.","tokens_in":8246,"tokens_out":4262,"duration_ms":42825,"significance":"If the central claim were established, a lightweight learned relevance scoring function that improves Arabic retrieval over standard dot-product DPR would be a useful contribution, particularly for morphologically rich and low-resource languages. The paper has positive elements: the evaluation is on a held-out test split (so the scoring is not circular), the model adds only 0.13M parameters over the dual encoders, and the code is promised publicly. However, the causal role of the ARS module is not tested, baseline comparability is undocumented, no statistical reliability is reported, and there are numeric inconsistencies in the results section. These issues currently prevent the paper from supporting its main claim.","major_comments":[{"comment":"The central claim that the ARS module drives the observed gains is not tested. APR differs from the DPR and AraDPR baselines in at least four ways: the encoder initialization (MiniBERT vs. AraBERT), the ARS scoring function in Eqs. (2)–(4), the dynamic relevance loss L_dyn in Eq. (7), and the logit regularization L_reg in Eq. (8). No ablation configuration removes ARS while holding the other components fixed, and the paper explicitly defers 'thorough ablation studies' to future work in Section 5. Consequently, the statement in Section 4.3 that the results 'demonstrate that APR effectively leverages the ARS module' is an attribution without a control. This is a missing-support issue for the paper's main contribution, not merely a question of presentation.","section":"Section 5 and Section 4.3"},{"comment":"The comparability of the baseline numbers is not documented. The paper does not state whether the DPR and AraDPR rows were produced in the same harness as APR, with the same ArabicaQA train/validation/test splits, the same Arabic Wikipedia passage index, the same preprocessing, and the same negative sampling. If those rows are quoted from prior papers under different evaluation conditions, the reported gains are not apples-to-apples and the superiority claim in Section 4.3 is not established. Please either rerun all baselines under identical settings or provide explicit provenance and settings for each row, including the exact passage corpus and retrieval procedure (e.g., exhaustive search vs. ANN).","section":"Table 1 and Section 4.2"},{"comment":"The results section contains numeric inconsistencies with Table 1. The text says APR achieves 'about 38%' at k = 5 and cites AraDPR (36.10%) and DPR (36.40%) at that same cutoff, but those two baseline values are the Top-1 entries in Table 1, not k = 5 values. The text also states APR reaches 75.01% at k = 100, whereas Table 1 reports 73.43. These discrepancies must be resolved. Additionally, the caption of Figure 2 should clarify whether it shows validation or test accuracy and what the x-axis represents, since the discussion draws conclusions from these numbers.","section":"Section 4.3, text vs. Table 1"},{"comment":"No error bars, multiple-seed results, or statistical significance tests are reported for any method. The Top-1 gain over AraDPR is only +0.91 percentage points, which may be within run-to-run noise. Without at least a few random seeds or a paired significance test, the claim that APR 'outperforms all baseline systems across all top-k retrieval thresholds' is not statistically supported, even setting aside the ablation issue. Please report variance across seeds or a significance test.","section":"Table 1 and Section 4.2"}],"minor_comments":[{"comment":"Typo: 'the overall architecture of our pro APR' should read 'the overall architecture of our proposed APR'.","section":"Section 3, paragraph 1"},{"comment":"The phrase 'Answer Relevance Scoring' should be 'Attentive Relevance Scoring' to match the method name used elsewhere in the paper.","section":"Section 4.3, paragraph 3"},{"comment":"Reference [25] is cited as the source of 'MiniBERT', but the listed reference describes 'KUISAIL at SemEval-2020 Task 12: Bert-CNN for offensive speech identification in social media'. The connection to MiniBERT is not evident and the citation appears incorrect; please provide the correct reference for the Arabic MiniBERT model or specify the model version and source.","section":"Reference [25]"},{"comment":"The term 'attention' for a single learned weight vector w_a applied to an element-wise product is unconventional. If this is intended as an attention mechanism, the paper should clarify how attention weights are derived; if it is a learned interaction layer, the discussion should position it relative to existing interaction-based rankers (e.g., DSSM or attentive matching).","section":"Section 3.2, Eqs. (2)–(4)"},{"comment":"The training description does not state the number of epochs or total training steps, nor which negative passage is used in L_dyn (Eq. 7) when multiple negatives are available. Adding these details would improve reproducibility.","section":"Section 4.2"},{"comment":"The superscripts '1' and '2' after 'ArabicaQA dataset' and 'Arabic Wikipedia' appear to refer to dataset URLs or footnotes, but these are not expanded in the bibliography; please add formal citations or a footnote with the dataset links.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper's contribution is not yet supported because the key attribution claim (ARS drives the gains) lacks an ablation and the baseline setup is not controlled. This is a missing-support issue that can, in principle, be fixed within the manuscript's scope by adding ablations, rerunning or documenting baselines, and reporting statistical reliability. I would encourage the editor to request a major revision rather than reject, given the promising direction and the positive aspects of a held-out evaluation and public code."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read for you on arXiv:2507.23404. The paper is a small, honest Arabic dense retrieval system: it takes DPR, swaps in a lightweight Arabic encoder, and adds a tiny learned interaction head (ARS, ~0.13M parameters) on top of [CLS] embeddings. The empirical claim is a consistent 1–5 point gain over AraDPR on ArabicaQA, and the code is public. That is real progress of the incremental sort.\n\nWhat it does well: the architecture is easy to follow, the loss is clearly specified, and the authors are upfront that the scoring module builds on DSSM and attentive FAQ matching. They also explicitly disclaim having done ablations — Section 5 says future work will separate the contributions of the proposed components. That is rare honesty.\n\nThe soft spots are mostly about attribution, not reproducibility. The central sentence in Section 4.3 ('APR effectively leverages the ARS module') is not supported by any experiment. APR differs from AraDPR in at least four ways: encoder initialization, the ARS head, the dynamic relevance loss, and the logit regularization. With no configuration that removes ARS while keeping the rest fixed, the causal role of ARS is unproven. The paper admits this. Also, baseline comparability is undocumented: I do not see confirmation that the DPR and AraDPR rows were rerun on the same ArabicaQA split, same Wikipedia index, and same negative sampling. If those numbers are quoted from other papers, the gains are not apples-to-apples.\n\nTwo smaller things. The reference for MiniBERT points to a SemEval-2020 offensive language paper, which does not describe a MiniBERT Arabic model. And there are internal numeric inconsistencies: the text mentions about 75% at k=100 and about 38% at k=5, but Table 1 says 73.43 and has no k=5 row. These need cleanup.\n\nNet: the paper's headline attribution overreaches, but the system is well-defined, the code is public, and the benchmark numbers may be useful. The honest limitation statement actually strengthens my confidence that the authors are not hiding anything. I would send this to review, but ask for ablations and either rerun baselines or clearly cite where they came from. It is a workshop-scale contribution, not a breakthrough.","headline":"A small, honest Arabic dense retrieval system whose gains look real but whose central attribution to the new scoring module is untested — worth a revision, not a rejection.","tokens_in":8798,"tokens_out":2102,"would_cite":false,"duration_ms":21695,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that replacing dot-product relevance with a small trainable Attentive Relevance Scoring module lets a lightweight Arabic DPR model beat AraDPR and other baselines on every top-k cutoff in ArabicaQA.","keywords":["Arabic NLP","Dense Passage Retrieval","Attentive Relevance Scoring","question answering","ArabicaQA","Arabic Wikipedia","contrastive learning","low-resource retrieval"],"falsifier":"Train APR on the same ArabicaQA split with the ARS head replaced by a plain dot-product scorer while keeping the encoders and all three losses unchanged; if the top-k accuracy at every cutoff stays within noise of the reported Table 1 numbers, the claim that attentive relevance scoring drives the gains is refuted, and likewise rerunning DPR and AraDPR in the identical evaluation harness would test whether the baseline comparison is fair.","tokens_in":7800,"feed_emoji":"🔍","tokens_out":10759,"duration_ms":108154,"temperature":0.7,"pith_summary":"Arabic dense retrieval usually scores a question against a passage with a single dot product between vector embeddings, a step that is cheap but insensitive to Arabic's morphology and near-synonymy. The paper proposes APR, a Dense Passage Retrieval system that keeps a lightweight Arabic dual encoder and inserts a trainable Attentive Relevance Scoring (ARS) head in place of the fixed dot product. ARS projects both embeddings, combines them with an element-wise interaction and tanh activation, and learns a weight vector that turns the interaction into a relevance score. The authors report that APR beats DPR, AraDPR, BM25, and TF-IDF at every top-k cutoff on ArabicaQA, with the largest absolute gain over AraDPR at Top-10. They attribute the improvement to ARS's ability to model semantic relevance in Arabic, while leaving detailed component ablation to future work.","feed_headline":"Arabic passage retrieval gains 4.8% at Top-10 with attentive scoring","feed_subtitle":"A small adaptive scoring module on an Arabic dual encoder beats dense and sparse baselines at every ranking cutoff.","key_machinery":"The load-bearing object is the Attentive Relevance Scoring module, a scoring head with about 0.13 million parameters. Given $\\ell^2$-normalized [CLS] embeddings $q$ and $p$, it computes $h_q = W_q q$, $h_p = W_p p$, then $a = \\tanh(h_q \\odot h_p)$, and finally $r = \\sigma(w_a^\\top a)$; this scalar is the ranking score that replaces the dot product. Because passage embeddings are pre-computed, the module only needs the query embedding at inference and can score any number of stored passages. The module is trained jointly with an InfoNCE contrastive loss on the [CLS] embeddings, a dynamic relevance loss that pushes positive scores toward 1 and negative scores toward 0, and a logit-standard-deviation regularizer, weighted by $\\alpha=1$, $\\beta=1$, and $\\gamma=0.1$.","core_discovery":"The paper's central claim is that the final relevance score in Arabic dense retrieval should be learned rather than fixed. APR keeps a lightweight dual-encoder architecture initialized from an Arabic MiniBERT model but replaces dot-product or cosine scoring with the trained Attentive Relevance Scoring module. On the ArabicaQA test set the authors report top-1/10/20/50/100 accuracy of 37.01/63.17/66.36/70.77/73.43 percent, against AraDPR's 36.10/58.40/63.40/68.60/71.90 and DPR's 36.40/57.80/62.10/66.60/69.50. The consistency of the gap, especially +4.77 at top-10, is presented as evidence that ARS separates semantically similar but incorrect passages from truly relevant ones, improving the candidates handed to downstream readers.","pith_inferences":["If ARS is the true driver, the same two-vector interaction head should transfer to other morphologically rich, under-resourced languages by swapping in a suitable pretrained encoder, with the largest gains expected where dot-product similarity is a weak relevance proxy.","The reported system combines ARS with two auxiliary losses, so the contribution of the scoring head alone is not yet isolated; a natural test is to fix the encoders and losses and compare dot-product scoring with ARS scoring.","A practical two-stage retrieval design follows directly from the architecture: use ANN search with the CLS embeddings to obtain candidates quickly, then re-rank them with ARS, combining fast recall with finer semantic scoring.","If the dynamic relevance loss keeps scores well separated on hard negatives, ARS scores may also serve as confidence estimates, allowing Arabic QA systems to abstain when no passage scores highly."],"forward_implications":["Downstream Arabic question-answering readers should receive higher-quality candidate passages, with the largest benefit at Top-10 where the reported gain over AraDPR is +4.77 percent.","Because the ARS head adds only 0.13 million parameters, the ranking improvement does not require a larger or heavier encoder, which is useful for low-resource deployments of Arabic retrieval.","Since passage embeddings remain pre-computable, ARS can be added as a cheap reranking step over an approximate-nearest-neighbor shortlist without re-encoding the corpus.","The consistent lead at every k value suggests the method improves overall ranking quality rather than only the first retrieved item."],"supporting_citations":[{"why":"Defines the dense passage retrieval baseline and dot-product scoring that APR is designed to improve.","marker":"[1]"},{"why":"Supplies the ArabicaQA dataset, training negatives, and the AraDPR baseline that APR must beat.","marker":"[2]"},{"why":"Provides the BM25 sparse-retrieval baseline used in the comparison table.","marker":"[6]"},{"why":"The dual-encoder architecture on which APR builds, with ARS inserted at its scoring step.","marker":"[24]"},{"why":"Cited for the Arabic pretrained MiniBERT encoder that initializes both the question and passage encoders.","marker":"[25]"},{"why":"Supplies the TF-IDF sparse baseline in the comparison table.","marker":"[27]"}],"fun_headline_variants":["Arabic retrieval: learned relevance scoring beats fixed dot-product","Attentive scoring module lifts Arabic retrieval across every cutoff","Replace fixed dot-product with trained ARS: Arabic passage retrieval improves","Arabic retrieval: training the relevance scorer, not just the encoder","Attentive scoring gives Arabic retrieval a 4.8% Top-10 gain"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The fragile premise is that the Table 1 baseline numbers come from the same ArabicaQA split, Wikipedia passage index, preprocessing, and negative sampling as APR, so the reported gains can be credited to the attentive scoring module rather than to differences in experimental setup.","fun_headline_variants_meta":{"raw":{"variants":["Arabic retrieval: learned relevance scoring beats fixed dot-product","Attentive scoring module lifts Arabic retrieval across every cutoff","Replace fixed dot-product with trained ARS: Arabic passage retrieval improves","Arabic retrieval: training the relevance scorer, not just the encoder","Attentive scoring gives Arabic retrieval a 4.8% Top-10 gain"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000852,"raw_usage":{"total_tokens":3671,"prompt_tokens":880,"completion_tokens":2791,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":496,"completion_tokens_details":{"reasoning_tokens":2702}},"tokens_in":496,"tokens_out":2791,"duration_ms":22872,"temperature":1.0,"reasoning_tokens":2702,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T10:47:39.497844+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train APR on the same ArabicaQA split with the ARS head replaced by a plain dot-product scorer while keeping the encoders and all three losses unchanged; if the top-k accuracy at every cutoff stays within noise of the reported Table 1 numbers, the claim that attentive relevance scoring drives the gains is refuted, and likewise rerunning DPR and AraDPR in the identical evaluation harness would test whether the baseline comparison is fair.","supporting_citations":[{"cited_title":"A key development stemming from this is DPR, which has become a corner- stone for modern open-domain Question Answering (QA) and search systems","cited_arxiv_id":null,"evidence_quote":"Supplies the ArabicaQA dataset, training negatives, and the AraDPR baseline that APR must beat."},{"cited_title":"Dense passage retrieval for open-domain question answer- ing,","cited_arxiv_id":null,"evidence_quote":"Provides the BM25 sparse-retrieval baseline used in the comparison table."},{"cited_title":"Arabic information retrieval,","cited_arxiv_id":null,"evidence_quote":"The dual-encoder architecture on which APR builds, with ARS inserted at its scoring step."},{"cited_title":"Arquad: An expert-annotated arabic machine reading comprehension dataset,","cited_arxiv_id":null,"evidence_quote":"Cited for the Arabic pretrained MiniBERT encoder that initializes both the question and passage encoders."},{"cited_title":"Learning deep structured semantic models for web search using clickthrough data,","cited_arxiv_id":null,"evidence_quote":"Supplies the TF-IDF sparse baseline in the comparison table."}],"review_version":1}