Beyond English-Only Reading Comprehension: Experiments in Zero-Shot Multilingual Transfer for Bulgarian

Ivan Koychev; Momchil Hardalov; Preslav Nakov

REVIEW 3 major objections 5 minor 45 references

Beyond English-Only Reading Comprehension: Experiments in Zero-Shot Multilingual Transfer for Bulgarian

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A multilingual BERT fine-tuned only on English RACE can answer Bulgarian multiple-choice exam questions at 42.23% accuracy, well above the 24.89% random baseline, when paired with Bulgarian Wikipedia passage retrieval.

desk verdict New Bulgarian MRC dataset is a real resource, but the RACE-transfer claim lacks a no-RACE control and the answer-option retrieval leaks lexical signal; worth revising and reviewing, not rejecting. read the letter →

arxiv 1908.01519 v2 pith:IK6DL5L4 submitted 2019-08-05 cs.CL cs.IR

Momchil Hardalov , Ivan Koychev , Preslav Nakov This is my paper · ORCID

classification cs.CLcs.IR

keywords multilingualBERTzero-shottransferreadingcomprehensionBulgarianRACEdatasetWikipediaretrievalmultiple-choiceQAlow-resourceNLP

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether a reading-comprehension model trained only on English can answer multiple-choice questions in Bulgarian, a language it never saw during fine-tuning. To test this, the authors build a new benchmark of 2,633 Bulgarian questions from twelfth-grade matriculation exams and online history quizzes, with no supplied reading passages, and a pipeline that retrieves supporting paragraphs from Bulgarian Wikipedia using the question plus each candidate answer as the query. They report that multilingual BERT fine-tuned on the English RACE dataset, applied with no Bulgarian training, reaches 42.23% accuracy, compared with 24.89% for random guessing and 29.62% for the weakest retrieval-based setup. The authors conclude that cross-lingual transfer works in this setting, and that retrieval and indexing choices, rather than extra Slavic pre-training, drive most of the gain.

What carries the argument

The load-bearing object is the triplet input used for scoring, [CLS] passage [SEP] question + option [SEP], fed through BERT fine-tuned on RACE to produce a probability for each answer option. Around this, the retriever forms queries by concatenating each candidate option to the question and searches a Bulgarian Wikipedia index using BM25 and cosine similarity over stemmed and n-gram fields. The final answer is the option with the largest summed probability across retrieved passages. The query design, question plus option, is what lets a model with no Bulgarian training find evidence passages, but it also ties the reported accuracy to lexical overlap between options and retrieved text.

What would settle it

Re-run the full pipeline with the answer options removed from the retrieval query, keeping the model and indexing unchanged. If accuracy falls toward the 24.89% random baseline, the reported transfer is driven by option-word matching in retrieval rather than by cross-lingual reading comprehension; if it stays near 42%, the model is genuinely using retrieved evidence to reason in Bulgarian.

Watch

Extended reading notes

Core claim

The central claim is that zero-shot multilingual transfer to Bulgarian multiple-choice reading comprehension works: a multilingual BERT model fine-tuned on English RACE, combined with Wikipedia passage retrieval, answers the authors' new Bulgarian dataset at 42.23% accuracy. The best configuration indexes Bulgarian Wikipedia by paragraph, analyzes titles and passages with Bulgarian stemming in addition to word n-grams, queries with each question-plus-option pair, and sums the model's option probabilities over the top retrieved passages. The paper also introduces the dataset itself: 2,633 multiple-choice questions across biology, philosophy, geography, and history, designed to mirror the RACE setup without provided contexts. Additional pre-training on Slavic corpora lowered accuracy to 33.27%, which the authors attribute to catastrophic forgetting of English, while paragraph-level splitting outperformed sliding-window indexing.

Load-bearing premise

The central assumption is that this benchmark measures reading comprehension rather than lexical matching, since each search query is the question plus one answer option, which biases the retrieved Wikipedia passages toward whichever option's wording appears in Wikipedia; the authors themselves note that many quiz questions are word-matching rather than reasoning questions.

Editorial extensions

If this is right

If 42.23% holds up, a single English-fine-tuned multilingual model can be deployed for Bulgarian question answering without any Bulgarian training labels.
Retrieval quality is the dominant lever: paragraph splitting added about two points over small windows, while Slavic pre-training cost about nine points, so effort is better spent on indexing than on language-specific model adaptation.
The new 2,633-question Bulgarian benchmark gives the community a public testbed for zero-shot machine reading comprehension in a low-resource language.
Because longer retrieval lists hurt accuracy, the model should trust only the top one or two evidence passages per option.
The reported accuracy is only 17 points above random on four-option questions, so the transfer is partial and substantial headroom remains for reasoning-oriented questions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The question-plus-option query makes it hard to know how much of the 42.23% is cross-lingual reasoning and how much is lexical retrieval; a version of the benchmark with distractors that share no wording with Wikipedia could separate the two.
If the mechanism is shared multilingual representations rather than lexical overlap, the same pipeline should transfer to other low-resource languages with adequate Wikipedia coverage, and performance should track how well the target language is represented in the multilingual model.
A cheap testable extension is to re-rank retrieved passages by the model's own confidence before voting, or to combine question-only retrieval with option-aware retrieval; either could raise accuracy without new training data.
The negative result for Slavic pre-training suggests that further language-model adaptation can destroy English task skills, so zero-shot transfer may be better served by stronger retrieval than by more target-language pretraining.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

My quick read on 1908.01519. The genuinely new thing is the Bulgarian multiple-choice dataset: 2,633 questions from matriculation exams and online quizzes, with a Wikipedia retrieval pipeline to supply contexts. That resource is useful, the paper is honest about its limitations, and the code/data appear to be public. The zero-shot setup is straightforward and the ablation over indexing strategies is systematic. So there is real value here.

The soft spots are real too, and two of them matter for the central claim. First, every non-random row is RACE-fine-tuned; there is no condition with multilingual BERT but zero RACE fine-tuning, no shuffled-label control, no control task. The conclusion that "additional pre-training on the English RACE corpus was very helpful" is therefore not actually demonstrated by these experiments. You can say the pipeline works at 42.23% with RACE fine-tuning, but you cannot say RACE fine-tuning is what made it work. That is a missing control, not a minor edit. Second, the retrieval query concatenates the question with each answer option, so passages are retrieved that lexically match the correct option. The paper itself concedes the quiz questions are more word-matching than reasoning. A retrieval-only baseline (e.g., choose the answer whose options retrieve the strongest evidence) would tell you how much of that 42.23% is just lexical overlap. The paper includes no such baseline.

There are smaller issues: the best configuration is selected by many ablations on the same test set, no error bars or significance tests, and the dataset is small. These are proportionate concerns for a 2019 arXiv paper; they are not fatal to the resource, but they cap the strength of the transfer claim.

Who is this for? Anyone building or evaluating low-resource reading-comprehension benchmarks, or working on zero-shot cross-lingual transfer, will get something from the dataset and the retrieval recipe. It deserves a serious referee: the resource is citable and the experiment is reproducible, but a reviewer should insist on the RACE-free control and a retrieval-only baseline before the 42.23% figure is treated as evidence of cross-lingual reading comprehension.

Referee Report

3 major / 5 minor

Summary. The paper studies zero-shot multilingual transfer of reading-comprehension models from English to Bulgarian. The authors collect a new Bulgarian multiple-choice dataset of 2,633 questions from matriculation exams and online quizzes, and build a pipeline that retrieves relevant passages from Bulgarian Wikipedia using Elasticsearch and then scores answer candidates with Multilingual BERT fine-tuned on the English RACE dataset. The paper reports an overall accuracy of 42.23% on the Bulgarian test set, compared with a random baseline of 24.89%, and presents ablations over retrieval fields, indexing strategies, query-result list sizes, and fine-tuning epochs. The central claim is that zero-shot transfer of English reading-comprehension skill, aided by Wikipedia retrieval, is effective for Bulgarian.

Significance. If the results were established, the paper would make a useful contribution by releasing a new Bulgarian multiple-choice reading-comprehension dataset, demonstrating an open-domain retrieval pipeline for a low-resource language, and providing evidence on the feasibility of zero-shot cross-lingual transfer. The dataset and code are publicly released, which is a clear strength. However, the two main design choices — the absence of a control without English RACE fine-tuning and the answer-option-conditional retrieval design — leave the central claim of cross-lingual reading-comprehension transfer unsubstantiated. The paper's contribution is better framed as an exploratory pipeline and a benchmark resource, pending additional control experiments.

major comments (3)

[Section 5.3, Table 4; Conclusion] The claim in the Conclusion that 'additional pre-training on the English RACE corpus [was] very helpful' is not supported by the experimental design. Every non-random configuration in Table 4 uses a model fine-tuned on RACE (Multilingual BERT or Slavic BERT), and the only non-RACE baseline is 'Random 24.89'. Without a condition in which Multilingual BERT is not fine-tuned on RACE, or is fine-tuned on a control English task, the observed accuracy cannot be attributed to English reading-comprehension supervision rather than to the retriever or to the multilingual pretrained representations alone. This is load-bearing because the stated contribution is the effectiveness of RACE-fine-tuned zero-shot transfer.
[Section 3.1; Appendix B; Section 5.3] The retriever forms queries by concatenating the question with each answer option, and Appendix B confirms that 'queries are formed by concatenating the question with an answer option.' This creates a lexical-overlap shortcut: the retrieved passage for the correct option is more likely to contain that option's exact wording, and the comprehension model can then assign high probability to the option without performing cross-lingual inference. Section 5.3 concedes that the online quiz questions are 'more of a word-matching nature rather then a reasoning one.' The paper does not provide a control (e.g., question-only retrieval, retrieval on a per-question passage set independent of the options, or an analysis of accuracy on questions where the correct option has no lexical overlap with the retrieved passage), so the 42.23% accuracy does not yet establish zero-shot reading-comprehension transfer.
[Section 5.3, Table 4] The best configuration (paragraph splitting, 42.23%) was selected after evaluating many indexing and query-field combinations on the same Bulgarian test set, with no validation split or significance testing. Consequently, the reported accuracy is an optimistically biased estimate of the pipeline's true performance, and small absolute differences (e.g., 39.69 vs. 40.26) are not shown to be reliable. The authors should report results on a held-out portion of the data or use cross-validation, and provide error bars or significance tests for the main comparisons.

minor comments (5)

[Abstract, Section 1, Table 1] The dataset size is inconsistent: the Abstract says 2,221 matriculation questions plus 412 quiz questions (2,633), the Introduction says 2,636, and Table 1 reports 2,633. Please reconcile these numbers.
[Equation (2)] Equation (2) writes 'P r(A|p; q)' where A is the set of answer candidates; the probability should refer to a specific answer a, as in Equation (1), to avoid notational ambiguity.
[Section 5.2, Table 4] The retrieval field names are used inconsistently: Section 5.2 mentions 'title.bg' and 'passage.bg', while Table 4 and Appendix A use 'title.bulgarian' and 'passage.bulgarian'. Please unify the notation.
[Section 5.1, Table 3] The description of Slavic BERT is abbreviated; it is unclear whether the 'Slavic pre-training' is an additional pretraining stage or a domain-adaptation fine-tuning of Multilingual BERT, and the phrase 'pre-training' is used in both senses. A precise description is needed.
[Throughout] There are several typos and stylistic slips, including 'rather then' (Section 5.3), 'expect for' (Section 4), and 'V ocab Size' (Table 1). Please copyedit the manuscript.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the reported accuracy is an empirical measurement on a newly collected Bulgarian test set, not a consequence of the model definition or a self-citation chain.

full rationale

The paper's central claim is that a multilingual BERT model fine-tuned on English RACE, combined with Wikipedia passage retrieval, achieves 42.23% accuracy on a newly collected Bulgarian multiple-choice benchmark in a zero-shot fashion. This is not definitionally circular: the model is trained on English RACE labels and evaluated on Bulgarian questions that were not used for fine-tuning, and the Bulgarian ground-truth answers are not inputs to the model. The retrieval and scoring equations (Eqs. 1-2) define a prediction rule, while the reported accuracy is an empirical outcome measured against held-out labels. The paper does not fit any parameter to Bulgarian labels and then report that fit as a prediction. The main validity concerns are experimental, not circular: the final configuration was selected by tuning on the Bulgarian test set, and there is no control condition without English RACE fine-tuning, so the causal claim that RACE pre-training was 'very helpful' is not fully established by the experiments. These concerns concern benchmark validity and experimental design, not a reduction of the result to its own inputs. The self-citations (e.g., Nakov 2003 for a Bulgarian stemmer, Joty et al. 2017 for cross-language learning, Simov et al. 2012 for prior Bulgarian QA work) are related-work or implementation references and are not load-bearing for the central derivation. No uniqueness theorem or ansatz is smuggled in via self-citation. Therefore no circular step can be identified from the paper's own equations or cited evidence, and the appropriate circularity score is 0.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The central claim depends on standard BERT fine-tuning plus an Elasticsearch retrieval pipeline; no new physical or theoretical entities are introduced. The main free parameters are hyperparameters and retrieval settings selected by ablation on the target test set.

free parameters (5)

Number of RACE fine-tuning epochs = 3
Selected on the Bulgarian test set; Table 4 shows 1 epoch gives 40.26%, 2 epochs 41.89%, 3 epochs 42.23%.
Retrieval index splitting strategy = Paragraph split
Chosen because it yielded the highest test accuracy (42.23% vs 36.54% for a large window and 40.26% for a small window) in Table 4.
Retrieval field weights = title.bulgarian^2, passage.ngram, passage, passage.bulgarian^2
Selected by ablation in Table 4; the weights are manually chosen boosts that affect which passages are retrieved.
Number of retrieved passages per answer option = 2 for the best overall result
Figure 2 and Table 5 show the best overall accuracy at Sq=2; other categories peak at different lengths, so this value was selected on the test set.
BERT fine-tuning hyperparameters = batch size 8, max sequence 320, learning rate 1e-5
Given in Section 5.1; chosen by hand following standard BERT practice and not tuned on Bulgarian target data.

assumptions (3)

domain assumption Multilingual BERT's token embeddings are aligned across languages well enough for zero-shot transfer from English to Bulgarian.
Unstated precondition for the central transfer experiment; Section 3.2 and Section 5.3 assume the model fine-tuned on RACE can process Bulgarian text with no Bulgarian training.
domain assumption Bulgarian Wikipedia contains passages that, when retrieved with question+option queries, can provide evidence for the correct answer.
The whole context retriever pipeline in Section 3.1 depends on the availability and relevance of Wikipedia content for these school-level questions.
domain assumption The 2,633 exam and quiz questions are valid multiple-choice reading comprehension items with exactly one correct answer among the listed options.
Dataset construction in Section 4 filters non-textual, ordering, and calculation questions, but does not independently verify answer keys or check for ambiguity.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Beyond English-Only Reading Comprehension: Experiments in Zero-Shot Multilingual Transfer for Bulgarian." pith.science (2026). https://pith.science/paper/IK6DL5L4

@misc{pith2026190801519,
  author       = {Pith},
  title        = {Pith review of: Beyond English-Only Reading Comprehension: Experiments in Zero-Shot Multilingual Transfer for Bulgarian},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IK6DL5L4}},
  note         = {Machine review of arXiv:1908.01519}
}

read the original abstract

Recently, reading comprehension models achieved near-human performance on large-scale datasets such as SQuAD, CoQA, MS Macro, RACE, etc. This is largely due to the release of pre-trained contextualized representations such as BERT and ELMo, which can be fine-tuned for the target task. Despite those advances and the creation of more challenging datasets, most of the work is still done for English. Here, we study the effectiveness of multilingual BERT fine-tuned on large-scale English datasets for reading comprehension (e.g., for RACE), and we apply it to Bulgarian multiple-choice reading comprehension. We propose a new dataset containing 2,221 questions from matriculation exams for twelfth grade in various subjects -history, biology, geography and philosophy-, and 412 additional questions from online quizzes in history. While the quiz authors gave no relevant context, we incorporate knowledge from Wikipedia, retrieving documents matching the combination of question + each answer option. Moreover, we experiment with different indexing and pre-training strategies. The evaluation results show accuracy of 42.23%, which is well above the baseline of 24.89%.

Figures

Figures reproduced from arXiv: 1908.01519 by the authors.

**Figure 1.** BERT for multiple-choice reasoning. As recommended by Devlin et al. (2019), we introduce a new task-specific parameter vector L, L ∈ R H, where H is the hidden size of the model. In order to obtain a score for each passagequestion-answer triplet, we take the dot product between L and the final hidden vector for the classification token ([CLS]), thus ending up with N unbounded numbers: one for each option. Finally,… view at source ↗

**Figure 2.** Accuracy per question category based on the number of query results per answer option. Next, we study the impact of the number of finetuning epochs on the model’s performance. We observe an increase in accuracy as the number of epochs grows, which is in line with previously reported results for English tasks. While this correlation is not as strong as for the original RACE task (see [PITH_FULL_IMAGE:figures/full… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 33 canonical work pages

[1]

Roee Aharoni, Melvin Johnson, and Orhan Firat. 2019. Massively multilingual neural machine translation. In Proceedings of the Conference of the North A merican Chapter of ACL\/ . Minneapolis, MN, USA, NAACL-HLT '19, pages 3874--3884

work page 2019
[2]

Mikel Artetxe and Holger Schwenk. 2018. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. arXiv preprint arXiv:1812.10464\/

arXiv 2018
[3]

Akari Asai, Akiko Eriguchi, Kazuma Hashimoto, and Yoshimasa Tsuruoka. 2018. Multilingual extractive reading comprehension by runtime machine translation. arXiv preprint arXiv:1809.03275\/

arXiv 2018
[4]

Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017 a . Reading W ikipedia to answer open-domain questions. In Proceedings of the Meeting of the Association for Computational Linguistics\/ . Vancouver, Canada, ACL '17, pages 1870--1879

work page 2017
[5]

Yun Chen, Yang Liu, Yong Cheng, and Victor O.K. Li. 2017 b . A teacher-student framework for zero-resource neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics\/ . Vancouver, Canada, ACL '17, pages 1925--1935

work page 2017
[6]

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? T ry ARC , the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457\/

arXiv 2018
[7]

Peter Clark, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Turney, and Daniel Khashabi. 2016. Combining retrieval, statistics, and inference to answer elementary science questions. In Proceedings of the 13th AAAI Conference on Artificial Intelligence\/ . Phoenix, AZ, USA, AAAI '16, pages 2580--2586

work page 2016
[8]

Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI : Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing\/ . Brussels, Belgium, EMNLP '18, pages 2475--2485

work page 2018

Show all 45 references

[9]

Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer- XL : Attentive language models beyond a fixed-length context. In Proceedings of the Meeting of the Association for Computational Linguistics\/ . Florence, Italy, ACL '1...

2019
[10]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT : Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistic...

2019
[11]

Yarman Vural, and Kyunghyun Cho

Orhan Firat, Baskaran Sankaran, Yaser Al-Onaizan, Fatos T. Yarman Vural, and Kyunghyun Cho. 2016. Zero-resource translation with multi-lingual neural machine translation. In Proc. of the Conference on Empirical Methods in Natural Language Processing\/ . Austin, TX, USA, EMNLP ...

2016
[12]

Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018. Learning word vectors for 157 languages. In Proceedings of the Conference on Language Resources and Evaluation\/ . Miyazaki, Japan, LREC '18

2018
[13]

Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics\/ . Melbourne, Australia, ACL '18, pages 328--339

2018
[14]

Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Vi \'e gas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean

Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Vi \'e gas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017. G oogle ' s multilingual neural machine translation system: Enabling zero-shot transl...

2017
[15]

Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. T rivia QA : A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the Annual Meeting of the Association for Computational Linguistics\/ . Vancouver, Canada, ACL '17...

2017
[16]

Shafiq Joty, Preslav Nakov, Llu \' s M \`a rquez, and Israa Jaradat. 2017. Cross-language learning with adversarial neural networks. In Proc. of the Conference on Computational Natural Language Learning\/ . Vancouver, Canada, CoNLL '17, pages 226--237

2017
[17]

Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. RACE : Large-scale R e A ding comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing\/ . Copenhagen, Denmark, EMNLP '17, pages 785--794

2017
[18]

Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291\/

2019 arXiv
[19]

Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? A new dataset for open book question answering. In Proceedings of the Conference on Empirical Methods in Natural Language Processing\/ . Brussels, Belgium, EMNLP '18,...

2018
[20]

Preslav Nakov. 2003. Building an inflectional stemmer for B ulgarian. In Proceedings of the 4th International Conference Conference on Computer Systems and Technologies: E-Learning\/ . Rousse, Bulgaria, CompSysTech '03, pages 419--424

2003
[21]

Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. In Proceedings of the Workshop on Cognitive Computation: Integrating Neural and Symbolic Approaches\/ . Bar...

2016
[22]

Jianmo Ni, Chenguang Zhu, Weizhu Chen, and Julian McAuley. 2019. Learning to attend on essential terms: An enhanced retriever-reader model for open-domain question answering. In Proceedings of the Conference of the North A merican Chapter of ACL\/ . Minneapolis, MN, USA, NAACL...

2019
[23]

Rodrigo Nogueira and Kyunghyun Cho. 2019. Passage re-ranking with BERT . arXiv preprint arXiv:1901.04085\/

2019 arXiv
[24]

Xiaoman Pan, Kai Sun, Dian Yu, Heng Ji, and Dong Yu. 2018. Improving question answering with external knowledge. arXiv preprint:1902.00993\/

2018 arXiv
[25]

Anselmo Pe \ n as, Eduard Hovy, Pamela Forner, Álvaro Rodrigo, Richard Sutcliffe, Corina Forascu, Yassine Benajiba, and Petya Osenova. 2012. Overview of QA4MRE at CLEF 2012: Q uestion answering for machine reading evaluation. In CLEF Working Note Papers\/ . Rome, Italy, pages 1--24

2012
[26]

Anselmo Pe \ n as, Christina Unger, and Axel-Cyrille Ngonga Ngomo. 2014. Overview of CLEF question answering track 2014. In Information Access Evaluation. Multilinguality, Multimodality, and Interaction\/ . pages 300--306

2014
[27]

Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics\/ ...

2018
[28]

Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training

2018
[29]

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners

2019
[30]

Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don ' t know: Unanswerable questions for SQ u AD . In Proceedings of the Meeting of the Association for Computational Linguistics\/ . Melbourne, Australia, ACL '18, pages 784--789

2018
[31]

Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. C o QA : A conversational question answering challenge. Transactions of the Association for Computational Linguistics\/ 7:249--266

2019
[32]

Burges, and Erin Renshaw

Matthew Richardson, Christopher J.C. Burges, and Erin Renshaw. 2013. MCT est: A challenge dataset for the open-domain machine comprehension of text. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing\/ . Seattle, WA, USA, EMNLP '13, pages...

2013
[33]

Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and beyond. Found. Trends Inf. Retr.\/ 3(4):333--389

2009
[34]

Jacques Savoy. 2007. Searching strategies for the B ulgarian language. Inform. Retrieval\/ 10(6):509--529

2007
[35]

Kiril Ivanov Simov, Petya Osenova, Georgi Georgiev, Valentin Zhikov, and Laura Tolosi. 2012. Bulgarian question answering for machine reading. In CLEF Working Note Papers\/ . Rome, Italy

2012
[36]

Kai Sun, Dian Yu, Jianshu Chen, Dong Yu, Yejin Choi, and Claire Cardie. 2019 a . DREAM : A challenge data set and models for dialogue-based reading comprehension. Transactions of the Association for Computational Linguistics\/ 7:217--231

2019
[37]

Kai Sun, Dian Yu, Dong Yu, and Claire Cardie. 2019 b . Improving machine reading comprehension with general reading strategies. In Proceedings of the North A merican Chapter of ACL\/ . Minneapolis, MN, USA, NAACL-HLT '19, pages 2633--2643

2019
[38]

Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Proceedings of the 27th Annual Conference on Neural Information Processing Systems\/ . Montreal, Canada, NIPS '14, pages 3104--3112

2014
[39]

Yi Tay, Luu Anh Tuan, and Siu Cheung Hui. 2018. Multi-range reasoning for machine comprehension. arXiv preprint arXiv:1803.09074\/

2018 arXiv
[40]

Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. N ews QA : A machine comprehension dataset. In Proceedings of the 2nd Workshop on Representation Learning for NLP \/ . Vancouver, Canada, RepL4NLP '19, pages 191--200

2017
[41]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the Annual Conference on Neural Information Processing Systems\/ . Long Beach, CA, USA, NIPS '17, page...

2017
[42]

Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. 2019 a . End-to-end open-domain question answering with BERT serini. In Proceedings of the Conference of the North A merican Chapter of ACL\/ . Minneapolis, MN, USA, NAACL-HLT '19, pages 72--77

2019
[43]

Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019 b . XLNet : Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237\/

2019 arXiv
[44]

, " * write output.state after.block = add.period write newline

ENTRY address author booktitle chapter doi edition editor howpublished institution journal key month note number organization pages publisher school series title type url volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.se...
[45]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Roee Aharoni, Melvin Johnson, and Orhan Firat. 2019. Massively multilingual neural machine translation. In Proceedings of the Conference of the North A merican Chapter of ACL\/ . Minneapolis, MN, USA, NAACL-HLT '19, pages 3874--3884

work page 2019

[2] [2]

Mikel Artetxe and Holger Schwenk. 2018. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. arXiv preprint arXiv:1812.10464\/

arXiv 2018

[3] [3]

Akari Asai, Akiko Eriguchi, Kazuma Hashimoto, and Yoshimasa Tsuruoka. 2018. Multilingual extractive reading comprehension by runtime machine translation. arXiv preprint arXiv:1809.03275\/

arXiv 2018

[4] [4]

Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017 a . Reading W ikipedia to answer open-domain questions. In Proceedings of the Meeting of the Association for Computational Linguistics\/ . Vancouver, Canada, ACL '17, pages 1870--1879

work page 2017

[5] [5]

Yun Chen, Yang Liu, Yong Cheng, and Victor O.K. Li. 2017 b . A teacher-student framework for zero-resource neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics\/ . Vancouver, Canada, ACL '17, pages 1925--1935

work page 2017

[6] [6]

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? T ry ARC , the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457\/

arXiv 2018

[7] [7]

Peter Clark, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Turney, and Daniel Khashabi. 2016. Combining retrieval, statistics, and inference to answer elementary science questions. In Proceedings of the 13th AAAI Conference on Artificial Intelligence\/ . Phoenix, AZ, USA, AAAI '16, pages 2580--2586

work page 2016

[8] [8]

Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI : Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing\/ . Brussels, Belgium, EMNLP '18, pages 2475--2485

work page 2018