{"id":"79284522-f6f9-4e4e-89b8-ff8155842db4","arxiv_id":"1908.01519","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Multilingual BERT fine-tuned on English RACE answers Bulgarian multiple-choice questions at 42.23% accuracy using Wikipedia retrieval, on a new 2,633-question benchmark, well above the 24.89% random baseline.","lead":"This paper creates a 2,633-question Bulgarian reading comprehension benchmark and shows that a multilingual BERT model trained only on English can answer 42.23% of it when Wikipedia passages are retrieved as context, versus 24.89% for random guessing. The result suggests zero-shot cross-lingual transfer works partially, but the setup has important evaluation caveats.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No control without English RACE fine-tuning, so the claimed transfer from English reading comprehension is not established.","rationale":"The reader's weakest assumption focuses on lexical retrieval leakage, which is well-grounded but partially conceded by the authors ('more of a word-matching nature') and affects interpretation of the task. The missing RACE ablation is, in my view, more load-bearing for the paper's explicit 'zero-shot transfer' claim: the Conclusion asserts that RACE pre-training was 'very helpful,' yet no row in Table 4 or Table 5 compares a model that has not been fine-tuned on RACE. Without such a control, the 42.23% figure cannot be attributed to transfer from English reading comprehension; it could be a property of the retriever plus the multilingual pretrained encoder. A shuffled-label control is a clean, inexpensive way to isolate the contribution of the RACE supervision while holding every other component fixed. If the control performs near 42%, the paper's central claim collapses to 'a retrieval and pretrained-multilingual pipeline works on Bulgarian' rather than 'English RC skill transfers zero-shot.' If the control performs at chance, the current claim is strengthened. Thus the verdict should remain conditional, but the required condition should explicitly include a no-RACE or shuffled-label control. I partially agree with the reader: both concerns question whether the accuracy demonstrates genuine cross-lingual reading-comprehension transfer, but they target different necessary conditions.","tokens_in":15227,"tokens_out":8860,"duration_ms":90137,"concrete_test":"Run the best configuration (paragraph split, fields title.bg^2, passage, passage.ngram, passage.bg^2) with Multilingual BERT fine-tuned on RACE with randomly shuffled answer labels (same epochs, batch size 8, learning rate 1e-5), keeping retrieval and voting identical. If this control achieves accuracy near 42.23%, the RACE supervision is not the driving factor and the transfer claim fails; if it drops toward the 24.89% random baseline, the transfer claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is zero-shot transfer of reading-comprehension skill from English RACE to Bulgarian, and the Conclusion states that 'additional pre-training on the English RACE corpus [was] very helpful.' But the experiments never vary the English fine-tuning. In Table 4, every non-random row (29.62% to 42.23%) uses Multilingual BERT fine-tuned on RACE for three epochs; the Slavic BERT row is also RACE-fine-tuned. There is no condition with Multilingual BERT not fine-tuned on RACE, no control English task, and no shuffled-label control. The only non-model baseline is 'Random 24.89%', which does not separate the contributions of the retriever, the pretrained multilingual representations, and the RACE supervision. Consequently, the observed 42.23% accuracy may arise from Bulgarian Wikipedia retrieval plus Multilingual BERT's existing zero-shot cross-lingual ability, with RACE fine-tuning contributing little. This matters because the paper's stated contribution is the effectiveness of RACE-fine-tuned transfer, and the Conclusion makes a causal claim the data cannot support. This concern is distinct from the lexical-retrieval leakage the reader identified: even if the benchmark is a valid reading-comprehension test, the experiments do not demonstrate that English reading-comprehension training is what enables the Bulgarian answers.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies zero-shot multilingual transfer of reading-comprehension models from English to Bulgarian. The authors collect a new Bulgarian multiple-choice dataset of 2,633 questions from matriculation exams and online quizzes, and build a pipeline that retrieves relevant passages from Bulgarian Wikipedia using Elasticsearch and then scores answer candidates with Multilingual BERT fine-tuned on the English RACE dataset. The paper reports an overall accuracy of 42.23% on the Bulgarian test set, compared with a random baseline of 24.89%, and presents ablations over retrieval fields, indexing strategies, query-result list sizes, and fine-tuning epochs. The central claim is that zero-shot transfer of English reading-comprehension skill, aided by Wikipedia retrieval, is effective for Bulgarian.","tokens_in":15483,"tokens_out":4389,"duration_ms":41646,"significance":"If the results were established, the paper would make a useful contribution by releasing a new Bulgarian multiple-choice reading-comprehension dataset, demonstrating an open-domain retrieval pipeline for a low-resource language, and providing evidence on the feasibility of zero-shot cross-lingual transfer. The dataset and code are publicly released, which is a clear strength. However, the two main design choices — the absence of a control without English RACE fine-tuning and the answer-option-conditional retrieval design — leave the central claim of cross-lingual reading-comprehension transfer unsubstantiated. The paper's contribution is better framed as an exploratory pipeline and a benchmark resource, pending additional control experiments.","major_comments":[{"comment":"The claim in the Conclusion that 'additional pre-training on the English RACE corpus [was] very helpful' is not supported by the experimental design. Every non-random configuration in Table 4 uses a model fine-tuned on RACE (Multilingual BERT or Slavic BERT), and the only non-RACE baseline is 'Random 24.89'. Without a condition in which Multilingual BERT is not fine-tuned on RACE, or is fine-tuned on a control English task, the observed accuracy cannot be attributed to English reading-comprehension supervision rather than to the retriever or to the multilingual pretrained representations alone. This is load-bearing because the stated contribution is the effectiveness of RACE-fine-tuned zero-shot transfer.","section":"Section 5.3, Table 4; Conclusion"},{"comment":"The retriever forms queries by concatenating the question with each answer option, and Appendix B confirms that 'queries are formed by concatenating the question with an answer option.' This creates a lexical-overlap shortcut: the retrieved passage for the correct option is more likely to contain that option's exact wording, and the comprehension model can then assign high probability to the option without performing cross-lingual inference. Section 5.3 concedes that the online quiz questions are 'more of a word-matching nature rather then a reasoning one.' The paper does not provide a control (e.g., question-only retrieval, retrieval on a per-question passage set independent of the options, or an analysis of accuracy on questions where the correct option has no lexical overlap with the retrieved passage), so the 42.23% accuracy does not yet establish zero-shot reading-comprehension transfer.","section":"Section 3.1; Appendix B; Section 5.3"},{"comment":"The best configuration (paragraph splitting, 42.23%) was selected after evaluating many indexing and query-field combinations on the same Bulgarian test set, with no validation split or significance testing. Consequently, the reported accuracy is an optimistically biased estimate of the pipeline's true performance, and small absolute differences (e.g., 39.69 vs. 40.26) are not shown to be reliable. The authors should report results on a held-out portion of the data or use cross-validation, and provide error bars or significance tests for the main comparisons.","section":"Section 5.3, Table 4"}],"minor_comments":[{"comment":"The dataset size is inconsistent: the Abstract says 2,221 matriculation questions plus 412 quiz questions (2,633), the Introduction says 2,636, and Table 1 reports 2,633. Please reconcile these numbers.","section":"Abstract, Section 1, Table 1"},{"comment":"Equation (2) writes 'P r(A|p; q)' where A is the set of answer candidates; the probability should refer to a specific answer a, as in Equation (1), to avoid notational ambiguity.","section":"Equation (2)"},{"comment":"The retrieval field names are used inconsistently: Section 5.2 mentions 'title.bg' and 'passage.bg', while Table 4 and Appendix A use 'title.bulgarian' and 'passage.bulgarian'. Please unify the notation.","section":"Section 5.2, Table 4"},{"comment":"The description of Slavic BERT is abbreviated; it is unclear whether the 'Slavic pre-training' is an additional pretraining stage or a domain-adaptation fine-tuning of Multilingual BERT, and the phrase 'pre-training' is used in both senses. A precise description is needed.","section":"Section 5.1, Table 3"},{"comment":"There are several typos and stylistic slips, including 'rather then' (Section 5.3), 'expect for' (Section 4), and 'V ocab Size' (Table 1). Please copyedit the manuscript.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The decision between major_revision and reject is a close call. I chose major_revision because the two central omissions (no no-RACE-fine-tuning control and no control for the answer-option retrieval leakage) are addressable with additional experiments, and the released dataset and pipeline are useful resources. The revised manuscript must show that the reported accuracy is not primarily a retrieval artifact and that the RACE fine-tuning contributes beyond the multilingual pretrained model; otherwise the paper's central claim should be restated as a resource-and-pipeline contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"My quick read on 1908.01519. The genuinely new thing is the Bulgarian multiple-choice dataset: 2,633 questions from matriculation exams and online quizzes, with a Wikipedia retrieval pipeline to supply contexts. That resource is useful, the paper is honest about its limitations, and the code/data appear to be public. The zero-shot setup is straightforward and the ablation over indexing strategies is systematic. So there is real value here.\n\nThe soft spots are real too, and two of them matter for the central claim. First, every non-random row is RACE-fine-tuned; there is no condition with multilingual BERT but zero RACE fine-tuning, no shuffled-label control, no control task. The conclusion that \"additional pre-training on the English RACE corpus was very helpful\" is therefore not actually demonstrated by these experiments. You can say the pipeline works at 42.23% with RACE fine-tuning, but you cannot say RACE fine-tuning is what made it work. That is a missing control, not a minor edit. Second, the retrieval query concatenates the question with each answer option, so passages are retrieved that lexically match the correct option. The paper itself concedes the quiz questions are more word-matching than reasoning. A retrieval-only baseline (e.g., choose the answer whose options retrieve the strongest evidence) would tell you how much of that 42.23% is just lexical overlap. The paper includes no such baseline.\n\nThere are smaller issues: the best configuration is selected by many ablations on the same test set, no error bars or significance tests, and the dataset is small. These are proportionate concerns for a 2019 arXiv paper; they are not fatal to the resource, but they cap the strength of the transfer claim.\n\nWho is this for? Anyone building or evaluating low-resource reading-comprehension benchmarks, or working on zero-shot cross-lingual transfer, will get something from the dataset and the retrieval recipe. It deserves a serious referee: the resource is citable and the experiment is reproducible, but a reviewer should insist on the RACE-free control and a retrieval-only baseline before the 42.23% figure is treated as evidence of cross-lingual reading comprehension.","headline":"New Bulgarian MRC dataset is a real resource, but the RACE-transfer claim lacks a no-RACE control and the answer-option retrieval leaks lexical signal; worth revising and reviewing, not rejecting.","tokens_in":15982,"tokens_out":1637,"would_cite":true,"duration_ms":18378,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A multilingual BERT fine-tuned only on English RACE can answer Bulgarian multiple-choice exam questions at 42.23% accuracy, well above the 24.89% random baseline, when paired with Bulgarian Wikipedia passage retrieval.","keywords":["multilingual BERT","zero-shot transfer","reading comprehension","Bulgarian","RACE dataset","Wikipedia retrieval","multiple-choice QA","low-resource NLP"],"falsifier":"Re-run the full pipeline with the answer options removed from the retrieval query, keeping the model and indexing unchanged. If accuracy falls toward the 24.89% random baseline, the reported transfer is driven by option-word matching in retrieval rather than by cross-lingual reading comprehension; if it stays near 42%, the model is genuinely using retrieved evidence to reason in Bulgarian.","tokens_in":15029,"feed_emoji":"🎓","tokens_out":8765,"duration_ms":79212,"temperature":0.7,"pith_summary":"The paper asks whether a reading-comprehension model trained only on English can answer multiple-choice questions in Bulgarian, a language it never saw during fine-tuning. To test this, the authors build a new benchmark of 2,633 Bulgarian questions from twelfth-grade matriculation exams and online history quizzes, with no supplied reading passages, and a pipeline that retrieves supporting paragraphs from Bulgarian Wikipedia using the question plus each candidate answer as the query. They report that multilingual BERT fine-tuned on the English RACE dataset, applied with no Bulgarian training, reaches 42.23% accuracy, compared with 24.89% for random guessing and 29.62% for the weakest retrieval-based setup. The authors conclude that cross-lingual transfer works in this setting, and that retrieval and indexing choices, rather than extra Slavic pre-training, drive most of the gain.","feed_headline":"English-trained BERT answers Bulgarian exam questions at 42%","feed_subtitle":"Zero-shot transfer plus Wikipedia retrieval beats random guessing by 17 points on a new 2,633-question benchmark.","key_machinery":"The load-bearing object is the triplet input used for scoring, [CLS] passage [SEP] question + option [SEP], fed through BERT fine-tuned on RACE to produce a probability for each answer option. Around this, the retriever forms queries by concatenating each candidate option to the question and searches a Bulgarian Wikipedia index using BM25 and cosine similarity over stemmed and n-gram fields. The final answer is the option with the largest summed probability across retrieved passages. The query design, question plus option, is what lets a model with no Bulgarian training find evidence passages, but it also ties the reported accuracy to lexical overlap between options and retrieved text.","core_discovery":"The central claim is that zero-shot multilingual transfer to Bulgarian multiple-choice reading comprehension works: a multilingual BERT model fine-tuned on English RACE, combined with Wikipedia passage retrieval, answers the authors' new Bulgarian dataset at 42.23% accuracy. The best configuration indexes Bulgarian Wikipedia by paragraph, analyzes titles and passages with Bulgarian stemming in addition to word n-grams, queries with each question-plus-option pair, and sums the model's option probabilities over the top retrieved passages. The paper also introduces the dataset itself: 2,633 multiple-choice questions across biology, philosophy, geography, and history, designed to mirror the RACE setup without provided contexts. Additional pre-training on Slavic corpora lowered accuracy to 33.27%, which the authors attribute to catastrophic forgetting of English, while paragraph-level splitting outperformed sliding-window indexing.","pith_inferences":["The question-plus-option query makes it hard to know how much of the 42.23% is cross-lingual reasoning and how much is lexical retrieval; a version of the benchmark with distractors that share no wording with Wikipedia could separate the two.","If the mechanism is shared multilingual representations rather than lexical overlap, the same pipeline should transfer to other low-resource languages with adequate Wikipedia coverage, and performance should track how well the target language is represented in the multilingual model.","A cheap testable extension is to re-rank retrieved passages by the model's own confidence before voting, or to combine question-only retrieval with option-aware retrieval; either could raise accuracy without new training data.","The negative result for Slavic pre-training suggests that further language-model adaptation can destroy English task skills, so zero-shot transfer may be better served by stronger retrieval than by more target-language pretraining."],"forward_implications":["If 42.23% holds up, a single English-fine-tuned multilingual model can be deployed for Bulgarian question answering without any Bulgarian training labels.","Retrieval quality is the dominant lever: paragraph splitting added about two points over small windows, while Slavic pre-training cost about nine points, so effort is better spent on indexing than on language-specific model adaptation.","The new 2,633-question Bulgarian benchmark gives the community a public testbed for zero-shot machine reading comprehension in a low-resource language.","Because longer retrieval lists hurt accuracy, the model should trust only the top one or two evidence passages per option.","The reported accuracy is only 17 points above random on four-option questions, so the transfer is partial and substantial headroom remains for reasoning-oriented questions."],"supporting_citations":[{"why":"Supplies the multilingual BERT model and the fine-tuning recipe used for multiple-choice reading comprehension.","marker":"Devlin et al., 2019"},{"why":"Supplies RACE, the large English multiple-choice dataset used for all fine-tuning in the zero-shot transfer.","marker":"Lai et al., 2017"},{"why":"Supplies the BM25 relevance model used by the context retriever over Wikipedia.","marker":"Robertson and Zaragoza, 2009"},{"why":"Supplies the Bulgarian stemming and stop-word analysis used to index and query Wikipedia in Bulgarian.","marker":"Savoy, 2007"},{"why":"Establishes the open-domain retrieval-from-Wikipedia paradigm that the paper adapts to a low-resource language.","marker":"Chen et al., 2017a"},{"why":"Provides evidence that multilingual BERT transfers zero-shot across languages on XNLI, the premise the paper extends to reading comprehension.","marker":"Conneau et al., 2018"}],"fun_headline_variants":["Zero-shot BERT answers Bulgarian exams at 42%","New Bulgarian benchmark: zero-shot BERT beats baseline","English-trained BERT tackles Bulgarian multiple choice","42% on Bulgarian exams with zero-shot transfer from English","Wikipedia retrieval boosts BERT to 42% on Bulgarian MCQs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central assumption is that this benchmark measures reading comprehension rather than lexical matching, since each search query is the question plus one answer option, which biases the retrieved Wikipedia passages toward whichever option's wording appears in Wikipedia; the authors themselves note that many quiz questions are word-matching rather than reasoning questions.","fun_headline_variants_meta":{"raw":{"variants":["Zero-shot BERT answers Bulgarian exams at 42%","New Bulgarian benchmark: zero-shot BERT beats baseline","English-trained BERT tackles Bulgarian multiple choice","42% on Bulgarian exams with zero-shot transfer from English","Wikipedia retrieval boosts BERT to 42% on Bulgarian MCQs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00101,"raw_usage":{"total_tokens":4251,"prompt_tokens":911,"completion_tokens":3340,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":527,"completion_tokens_details":{"reasoning_tokens":3271}},"tokens_in":527,"tokens_out":3340,"duration_ms":21349,"temperature":1.0,"reasoning_tokens":3271,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T15:10:24.674561+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the full pipeline with the answer options removed from the retrieval query, keeping the model and indexing unchanged. If accuracy falls toward the 24.89% random baseline, the reported transfer is driven by option-word matching in retrieval rather than by cross-lingual reading comprehension; if it stays near 42%, the model is genuinely using retrieved evidence to reason in Bulgarian.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the multilingual BERT model and the fine-tuning recipe used for multiple-choice reading comprehension."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies RACE, the large English multiple-choice dataset used for all fine-tuning in the zero-shot transfer."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the BM25 relevance model used by the context retriever over Wikipedia."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Bulgarian stemming and stop-word analysis used to index and query Wikipedia in Bulgarian."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides evidence that multilingual BERT transfers zero-shot across languages on XNLI, the premise the paper extends to reading comprehension."}],"review_version":1}