{"id":"00605e66-aa10-49d5-9ae8-4a9c30e6b05a","arxiv_id":"2501.04153","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Zero-shot question-generation reranking boosts Korean and Japanese retrieval but hurts Finnish and Bengali, and translated training data yields no consistent QA improvement.","lead":"This paper tests two ideas for a multilingual question-answering system: re-ranking retrieved passages with a zero-shot question-generation model, and adding machine-translated training data. The re-ranking improved two languages, hurt two others, and the added data gave no consistent gains.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Korean/Japanese gains are not tested for statistical significance, and the 'high-resource' condition is asserted post hoc even though Table 1 lists Korean with the fewest training examples; the central claim may rest on selection on noise.","rationale":"The reader's weakest assumption correctly identifies that relying on p(q|z) from an untrained multilingual model is not universally reliable, and the Finnish/Bengali results confirm this. My stress-test concern is narrower and more empirical: even the surviving claim about Korean and Japanese is not established because the reported gains lack significance testing, the Japanese gains are near zero, the Korean dev set is small, and the post-hoc 'high-resource' label conflicts with the paper's own Table 1. I do not see an internal contradiction that would invalidate the method in principle; the idea of zero-shot question-likelihood reranking is plausible and the Korean improvement is suggestive. However, the paper has not supplied enough evidence to support the abstract's broad 'simple and effective' claim or even the conclusion's narrowed version. The reader's REJECT verdict remains appropriate, so no verdict change is needed.","tokens_in":8754,"tokens_out":4626,"duration_ms":48321,"concrete_test":"Run QGPR-mBART re-ranking on the Arabic and Russian XOR-TyDi dev sets from Table 1, using the same mDPR top-50 passages and scoring procedure as in Section 5.1, and compute paired bootstrap 95% confidence intervals for the R@5/R@15 gains across all languages in Tables 2 and 3. If Arabic/Russian do not improve over mDPR, or if the Korean/Japanese confidence intervals include zero, the narrowed 'high-resource language' claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Tables 2 and 3 evaluate QGPR-mBART on exactly four languages: Korean, Japanese, Finnish, and Bengali. Only Korean shows a non-trivial improvement over mDPR (R@5 +3.9, R@15 +4.2); Japanese gains are +0.1 and +0.6, well within plausible sampling noise on dev sets of 473 and 693 questions. No significance tests, confidence intervals, or error bars are reported. The conclusion then narrows the claim to 'reasonably high resource languages like Korean and Japanese,' but Table 1 lists Korean with the fewest training examples (1856) of any language, fewer than Bengali (2428), the language where the method degrades most. No truly high-resource language such as Arabic or Russian is ever evaluated for QGPR, so the proposed condition is untested. The Finnish and Bengali degradations already show the method is not generally effective; the only defense is a post-hoc language-resource explanation that is contradicted by the paper's own dataset statistics and not validated on any held-out language. Thus the central claim rests on a small, possibly noisy improvement in one language and a tiny effect in another, with no statistical support.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper, submitted to the MIA shared task, proposes two modifications to a CORA-style multilingual open QA pipeline (mDPR retrieval followed by mT5 generation). The first contribution, Question-Generation based Re-ranking (QGPR), re-scores the mDPR top-50 passages with a zero-shot question-generation likelihood log p(q|z) computed by mBART or mT5, with additional variants that translate the question or passage before scoring. The second contribution is machine-translation-based data augmentation of Korean and Bengali training triples for the reader. Retrieval results are reported for Korean, Japanese, Finnish, and Bengali, and QA F1 results for seven languages. The paper claims that QGPR outperforms the mDPR baseline for 'reasonably high resource languages like Korean and Japanese' and that the augmentation approach gives only marginal, inconsistent gains.","tokens_in":8992,"tokens_out":3462,"duration_ms":36703,"significance":"If the QGPR claim were robust, a training-free reranker applicable to any retrieval method would be a useful contribution to low-resource CLIR, because it avoids labeled retrieval data. The paper has genuine positive features: the method is simple, the reranking score in Eq. (4) is a model likelihood rather than a fitted value, the evaluation is against external XOR-TyDi ground truth, and the authors report negative results as well as positive ones. However, the evidence offered does not establish the central claim. The only non-trivial improvement is for Korean; Japanese gains are tiny, Finnish and Bengali degrade, and the paper's own resource-based explanation is contradicted by its dataset statistics. The QA augmentation comparison is additionally confounded by a change in reader input length.","major_comments":[{"comment":"The central claim rests on a single language. The conclusion narrows the claim to 'reasonably high resource languages like Korean and Japanese', but Table 1 lists Korean with the fewest training examples (1,856) of all seven languages, fewer than Bengali (2,428), where QGPR-mBART degrades most. Japanese gains are +0.1 R@5 and +0.6 R@15 on a dev set of 693 questions, which is within plausible sampling noise. No significance tests, confidence intervals, or error bars are reported anywhere. The paper therefore provides no statistical support for the claim that the method is effective, and the proposed high-resource explanation is a post hoc assertion contradicted by the paper's own table.","section":"§6.1, Tables 2–3"},{"comment":"Recall@K is defined with 'the ground truth positives in the top-50 results obtained by mDPR as our total positives set'. This means QGPR can only reorder passages that mDPR already retrieved; any passage missed by mDPR is invisible to the evaluation. The abstract and introduction describe the method as improving 'passage retrieval', but the experiments only measure re-ranking within the mDPR top-50. The claims should be scoped accordingly, and the metric definition should be stated in the abstract or introduction if retrieval recall is discussed.","section":"§6.1, metrics paragraph"},{"comment":"The QA comparison is confounded by the reader input length. Section 5.2 states that the maximum input length was reduced from the baseline default of 1000 to 600 tokens, so the augmented reader sees fewer passages than the baseline system. Table 5 therefore does not provide a controlled comparison of the data augmentation method. The authors themselves note that they 'do not see any consistent tendency' and attribute non-augmented-language changes to training hyperparameters. As written, no conclusion about the augmentation approach can be drawn from this experiment.","section":"§5.2, §6.2, Table 5"},{"comment":"The method's core assumption is that a pretrained mBART likelihood p(q|z) is a reliable cross-lingual relevance signal. The paper gives no analysis of when this assumption holds: the Finnish and Bengali degradations show it fails for some languages, but the paper does not examine characteristics such as passage-language mix, question answerability, or language relatedness. Without such analysis, the 'high-resource language' caveat is an ad hoc explanation rather than a tested condition.","section":"§4.1, Eq. (4)"}],"minor_comments":[{"comment":"There are numerous typographical errors, including 'Japaneese', 'vice-e-versa', 'mulltingual', 'space space', 'had to tracks', and 'doesn't require'. A careful proofreading pass is needed.","section":"Throughout"},{"comment":"The sentence describing the 768-dimensional passage representation is duplicated: 'mBERTp(p)' and the following sentence both introduce the mBERT encoder. One copy should be removed.","section":"§3.1"},{"comment":"The citation style is inconsistent: the paper alternates between 'XOR-TYDI QA', 'XOR QA', and 'XOR-TYDI', and Asai et al. 2020/2021a appear to refer to the same work. The 'F oundations' spacing in the Robertson and Zaragoza reference should also be fixed.","section":"References"},{"comment":"The text says the mBART variant gave an improvement of '4% in Recall@5 and Recall@15' for Korean. This is consistent with Table 3 (+3.9, +4.2), but the preceding sentence about 'Positives@K and Recall@K' should make clear which metric is being quoted, since Table 2 shows much smaller absolute changes in P@K.","section":"§6.1"},{"comment":"The table caption calls the dataset 'XOR-TyDi-QA' and includes English with 91,876 training examples, but English has no development split in the table. Clarify whether English is used only for training and how the 'target dataset' label applies.","section":"Table 1"}],"recommendation":"reject","confidential_remarks":"This is a workshop-style report with a promising but unproven idea. My main concern is that the headline claim is supported only by Korean results and contradicted by the paper's own resource statistics; adding significance tests or re-running the baseline at equal input length would help, but the scope of the paper would still need substantial new experiments to justify the 'effective' claim. The paper may be more suitable for a workshop where preliminary negative results are acceptable, but it does not meet the standard for a journal publication in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a shared-task report with one genuinely interesting empirical finding and a central claim that its own tables undermine. Read it for the negative results and the translation-asymmetry observation, not for the abstract.\n\nWhat's new and good: applying Sachan-style zero-shot question-generation reranking (QGPR) to a multilingual setting is a sensible adaptation, and the paper cites the prior work openly. The analysis that translating passages into the question language beats translating questions into the passage language is a real, fairly clean observation. The MRR breakdown by same-language vs cross-lingual passages is also useful. Credit where it's due: the authors report the Finnish and Bengali degradations plainly and admit the augmentation experiment gave no consistent gain. They don't hide the mess.\n\nThe soft spots are load-bearing. Table 3 shows QGPR improves Korean by +3.9 R@5, gives Japanese +0.1, and hurts Finnish and Bengali by 4-5 points. The abstract calls the method 'simple and effective,' but two of four languages get worse. No significance tests, confidence intervals, or error bars anywhere, and the dev sets are 473 and 693 questions, so the Korean gain could easily be sampling noise. The 'high-resource languages like Korean and Japanese' explanation is post hoc: Table 1 lists Korean with the fewest training examples of all eight languages (1856), below Bengali (2428), which degrades the most. No truly high-resource language like Arabic or Russian is tested for QGPR, so the proposed condition is untested. The QA augmentation comparison is also confounded by cutting the reader's max input length from 1000 to 600 tokens, which the authors themselves note.\n\nSo the stress-test concern holds up. The paper is a legitimate exploratory report, but the central claim does not survive its own data.\n\nWho should read it: people working on cross-lingual retrieval, especially those considering zero-shot reranking, will want to know that the method helps some languages and hurts others with no clear predictor. That is a useful cautionary result. It deserves a serious referee if the claims are scaled back and the resource explanation is removed or tested; as written, it overreaches. I'd send it to review with the expectation of heavy revision, not desk-reject it—the experiments are real, and the translation-direction asymmetry is worth examining further.","headline":"An honest but overclaimed shared-task report: the one clear Korean re-ranking gain is likely noise, and the 'high-resource' explanation contradicts the paper's own table.","tokens_in":9528,"tokens_out":2972,"would_cite":false,"duration_ms":28983,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Zero-shot question-generation likelihood can re-rank retrieved passages in multilingual open QA, improving Korean and Japanese recall with no training while hurting Finnish and Bengali.","keywords":["cross-lingual information retrieval","question generation","re-ranking","open question answering","zero-shot learning","multilingual language models","dense passage retrieval","XOR-TyDi QA"],"falsifier":"On the XOR-TyDi dev sets, compute the QGPR score for each language's ground-truth positive passages and for sampled negative passages, then check whether positives rank above negatives; the paper's own recall drops for Finnish and Bengali already suggest that in those languages the score is not functioning as a cross-lingual relevance signal.","tokens_in":8556,"feed_emoji":"🔍","tokens_out":8029,"duration_ms":67411,"temperature":0.7,"pith_summary":"This paper tries to establish that a simple, training-free re-ranking step improves passage retrieval in multilingual open question answering. The re-ranker takes the top-50 passages from an existing dense retriever, mDPR, and re-scores each one by the average log-likelihood that a pretrained multilingual model assigns to the question given that passage — a zero-shot question-generation score. On the XOR-TyDi QA benchmark, this mBART-based score raises Recall@5 and Recall@15 for Korean and Japanese, but it lowers recall for Finnish and Bengali. The authors therefore claim the method helps for reasonably high-resource languages when the pretrained model generates well, and that it can be layered on any base retriever without labelled retrieval data. The paper's second contribution, machine-translation data augmentation, did not yield consistent gains in answer generation.","feed_headline":"Zero-shot mBART question likelihood beats mDPR on Korean and Japanese, not Finnish or…","feed_subtitle":"A training-free re-ranker re-scores retrieved passages by question-generation likelihood, helping some languages and hurting others.","key_machinery":"The central object is the question-generation re-ranking score, QGPR, which is $\\log p(q|z)$: the average per-token log-likelihood of the question $q$ when a pretrained multilingual model generates it conditioned on passage $z$. This score is computed by a zero-shot mBART-50 model, with a language tag for the target language, over the top-50 passages produced by mDPR; passages are then re-ranked solely by this likelihood, ignoring the original retrieval score. The mechanism is token-level cross-attention between question and passage inside the language model, which the authors argue is more expressive than the dense inner-product matching of mDPR. It lets the same pretrained model serve as a training-free relevance scorer across languages.","core_discovery":"On the paper's terms, the central discovery is that the conditional probability of the question given a retrieved passage, estimated by a zero-shot pretrained multilingual model, carries enough relevance signal to re-rank mDPR's top-50 passages in a completely unsupervised way. The authors instantiate this as QGPR, computing $\\log p(q|z) = \\frac{1}{|q|} \\sum_t \\log p(q_t | q_{<t}, z; \\Theta)$ with mBART, and report Recall@5 improvements from 12.1 to 16.0 (Korean) and 5.5 to 5.6 (Japanese), with Recall@15 improvements from 21.4 to 25.6 and 9.1 to 9.7, against the mDPR baseline. The same procedure with mT5, and with either questions or passages machine-translated into the other language, failed to beat the baseline. The authors interpret the gains as evidence that question generation forces the model to explain every token in the question through rich cross-attention, and they qualify the claim to languages where the zero-shot generator is competent: Korean and Japanese improve, Finnish and Bengali degrade.","pith_inferences":["A testable extension the paper does not run is to use the QGPR score with a question-generation model that is itself strong in Finnish and Bengali; if the gain reappears, the bottleneck is per-language generator quality rather than the scoring principle.","The near-zero cross-lingual MRR values in the paper's Table 4 suggest the method is mostly reordering same-language passages, so the cross-lingual retrieval problem remains largely open even if this re-ranker is adopted.","The same average-likelihood criterion could in principle re-rank passages in other retrieval-heavy tasks, such as fact verification or entity linking, wherever a multilingual generator exists, but nothing in this paper tests that transfer."],"forward_implications":["Any open-QA retriever, sparse or dense, can be coupled with this re-ranker, so languages without labelled retrieval data can still get a ranking signal from a pretrained multilingual model.","The benefit is conditional on the quality of the zero-shot generator: Korean and Japanese gain, Finnish and Bengali lose, so the re-ranker should be deployed selectively rather than uniformly.","Because re-ranking ignores the base retriever's scores, it provides an orthogonal signal that can improve the ordering of passages already retrieved, but it cannot recover positives that the base retriever failed to include in its top-50.","The machine-translation data augmentation approach does not reliably improve QA F1, and the paper attributes this mainly to input-length limits in the reader model that keep most translated contexts out.","The paper's MRR analysis indicates the re-ranking helps mainly for same-language positives, while cross-lingual MRR improves in some languages but remains low in absolute terms."],"supporting_citations":[{"why":"Supplies the mDPR retriever whose top-50 outputs are re-ranked, and serves as the baseline system.","marker":"(Asai et al., 2021b)"},{"why":"Defines the XOR-TyDi QA dataset and task used for evaluation.","marker":"(Asai et al., 2020)"},{"why":"Provides the mBART-50 pretrained model used as the zero-shot question-generation re-ranker and as the translation model for data augmentation.","marker":"(Liu et al., 2020)"},{"why":"Provides mT5, used for the alternative QGPR variants and as the reader model in the data-augmentation experiments.","marker":"(Xue et al., 2020)"},{"why":"Establishes the question-generation likelihood re-ranking idea that the paper adapts to the multilingual setting.","marker":"(Sachan et al., 2022)"},{"why":"Defines the dense passage retriever formulation that mDPR extends to the multilingual setting.","marker":"(Karpukhin et al., 2020a)"}],"fun_headline_variants":["Zero-shot mBART re-ranker boosts Korean, Japanese QA recall","Training-free re-ranker uses question likelihood to aid low-resource QA","mBART question odds re-rank passages, helping some languages, not all","QGPR: Zero-shot re-ranking lifts recall on Korean and Japanese"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the average token log-likelihood $\\log p(q|z)$ from a pretrained multilingual model ranks relevant passages above irrelevant ones even when the question and passage are in different languages and the model was never trained for question generation — a premise the paper's own Finnish and Bengali results violate.","fun_headline_variants_meta":{"raw":{"variants":["Zero-shot mBART re-ranker boosts Korean, Japanese QA recall","Training-free re-ranker uses question likelihood to aid low-resource QA","mBART question odds re-rank passages, helping some languages, not all","QGPR: Zero-shot re-ranking lifts recall on Korean and Japanese"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00031,"raw_usage":{"total_tokens":1811,"prompt_tokens":1028,"completion_tokens":783,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":644,"completion_tokens_details":{"reasoning_tokens":702}},"tokens_in":644,"tokens_out":783,"duration_ms":6458,"temperature":1.0,"reasoning_tokens":702,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:39:48.980343+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On the XOR-TyDi dev sets, compute the QGPR score for each language's ground-truth positive passages and for sampled negative passages, then check whether positives rank above negatives; the paper's own recall drops for Finnish and Bengali already suggest that in those languages the score is not functioning as a cross-lingual relevance signal.","supporting_citations":[],"review_version":1}