{"id":"46824f8c-4a35-4343-bcb0-dcf3c2b24806","arxiv_id":"1908.05416","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"XCMRC is a cross-lingual cloze-style machine reading comprehension benchmark built from Chinese-English parallel news articles, with baselines showing significant room for improvement.","lead":"The authors introduce XCMRC, a new benchmark for cross-lingual machine reading comprehension where a question and its candidate answers are in one language and the supporting passage is in another. The paper provides baseline results showing that current methods still lag far behind an upper bound, especially under a pseudo low-resource setting.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Passage-independent baseline scoring 59.83% versus passage-using 58.35–61.64% undercuts the claim that XCMRC requires cross-lingual reading comprehension; construct validity needs a control test.","rationale":"The reader's weakest assumption correctly identifies the passage-independent baseline's near-parity with passage-using models as the central threat to construct validity. My independent read of Section 3.2 and Table 5 confirms this is the most load-bearing issue: the benchmark's raison d'être is to require cross-lingual reading, and the paper's own numbers suggest a question-only model captures most of the signal. The reader's verdict of CONDITIONAL is appropriate because the concern is addressable with a control experiment and does not by itself falsify the dataset's usefulness for other purposes. I do not see an internally inconsistent derivation or a clear mathematical error; the issue is empirical validity. The proposed document-shuffle control is the most direct way to determine whether D is actually load-bearing. Other weaknesses, such as the partially multilingual BERT baseline and the same-seed multiple runs, are secondary and would not change the verdict if the construct-validity concern were resolved.","tokens_in":9527,"tokens_out":4314,"duration_ms":45719,"concrete_test":"On the released dev/test sets, build a document-shuffle control: for each item, replace D with a randomly selected English document from a different article, preserving Q, C, and A, then run the naive BiDAF_Candidates and translation-based BiDAF_Cloze models. If accuracy changes by only a few points relative to the original D and remains near the passage-independent 59.83%, the passage contributes little and the benchmark is not measuring cross-lingual MRC. Additionally, restrict to items where the English translation of the gold answer does not appear anywhere in D; on that subset, a passage-using model that merely copies the translated noun should not outperform PI_Candidates. Such a control would settle whether the high passage-independent score reflects genuine question-only answerability or an artifact of the particular dev split.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that XCMRC tests cross-lingual reading comprehension, i.e., that the source-language passage is necessary to answer the target-language cloze question. The paper's own Section 4.3 introduces a passage-independent model, PI_Candidates, which ignores D entirely and scores 59.83% on the passage/query configuration where the passage is English and the question is Chinese. The passage-using pseudo low-resource baseline, BiDAF_Candidates, scores only 58.35% and 61.64% on the two directions in Table 5. This near-parity means that, for a large fraction of items, the question plus the ten candidate nouns carry enough signal to choose the answer without reading the passage. The generation procedure in Section 3.2 makes this plausible: the question is formed from a Chinese paragraph, the aligned English paragraph is removed, but the remaining English paragraphs come from the same bilingual article and may or may not contain the answer's translation. Thus the benchmark risks measuring lexical and collocation statistics in Q and C rather than requiring the reader to understand D. Because the dataset's stated purpose is to 'test machines on their cross-lingual reading comprehension ability,' this construct-validity gap is the most load-bearing threat. It does not require assuming the authors were careless; passage-independent baselines are common diagnostic tools, but here the diagnostic result is close to the passage-based result, so the benchmark's central requirement is not satisfied for the pseudo low-resource setting and only weakly satisfied for the common setting.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes XCMRC, a cross-lingual cloze-style machine reading comprehension benchmark constructed from 25,467 aligned Chinese-English Economist articles. It consists of two cross-lingual sub-datasets, EPCQ (English passages, Chinese questions) and CPEQ (Chinese passages, English questions), each with roughly 56k samples, and two monolingual sub-datasets (EPEQ, CPCQ) intended as an upper-bound setting. The task is to fill a blank in a target-language question by choosing one of ten noun candidates after reading a passage in the source language. The authors define a common XCMRC task and a pseudo low-resource task and evaluate four baseline families: translation-based BiDAF Cloze, multilingual-BERT-based BERT-Candidates, naive BiDAF-Candidates, and a passage-independent PI-Candidates baseline. Their main findings are that translation-based methods perform best (up to 67.28%) but remain far below the monolingual upper bound (72.97%/68.81%), and that pseudo low-resource methods perform close to the passage-independent baseline, indicating that they learn little cross-lingual document information.","tokens_in":9728,"tokens_out":7569,"duration_ms":69603,"significance":"The paper addresses a real gap: at the time of submission there was no public cross-lingual MRC benchmark, and an automatically constructed bilingual cloze dataset is a plausible and scalable route. The main strengths are the scale of the resource, the clear formalization of two task variants (common and pseudo low-resource), the release of data and code, and the inclusion of a passage-independent control baseline, which is scientifically honest. If the construct-validity concerns are resolved, the benchmark could be useful for evaluating cross-lingual representation learning. However, the near-parity of the passage-independent and passage-based baselines currently undermines the claim that XCMRC measures cross-lingual reading comprehension rather than shallow lexical statistics.","major_comments":[{"comment":"The passage-independent baseline PI_Candidates achieves 59.83% accuracy, while the passage-using BiDAF_Candidates achieves only 58.35% and 61.64% in the two directions (Table 5). This near-parity means that for a large fraction of items the question and candidates alone suffice, so the benchmark may not require reading the passage. This is not merely a curiosity: the EPCQ example in Table 1 has the answer 'discrimination' appearing verbatim in the visible text of the question itself. The paper's own discussion in Section 5.2 admits that the naive approach 'has learned very limited information about the document.' To establish construct validity, the authors should report an item-level analysis: (i) the proportion of dev items for which PI_Candidates is correct; (ii) the proportion for which a passage-based model is correct while PI_Candidates is wrong; (iii) the same statistics on a filtered subset where the answer does not appear in the question; and (iv) a human ceiling on a sample. If the fraction of items requiring the passage is small, the benchmark should be revised (e.g., by filtering such items) before it can support the paper's central claim.","section":"Section 4.3, Table 5, Table 1 sample"},{"comment":"Only dev set results are reported (Section 5.2: 'we only present the performance on dev set'), so there is no evidence on the held-out test set that the benchmark defines. Since the dataset's purpose is to support evaluation, at least test-set numbers (or a clear statement that the test set is reserved for a leaderboard with hidden labels) are needed. In addition, Section 5.1 states that models were 'run 5 times independently with the same random seed 1234'; identical seeds do not yield independent trials, so the reported averages cannot be used to assess variance. The authors should run with different seeds and report mean ± standard deviation.","section":"Section 5.1 and Section 5.2"},{"comment":"Table 5 does not make clear which cells correspond to which dataset or language direction. The row labels 'English' and 'Chinese' under 'Passage' are not matched with explicit question-language columns, and the prose in Section 5.2 refers to 'EPEQ' when discussing the naive approach's performance on the cross-lingual CPEQ/EPCQ pair. Please reformat the table with explicit columns for EPCQ and CPEQ (and EPEQ/CPCQ where relevant), and correct the acronym usage so the central numerical claims can be verified.","section":"Table 5 and Section 5.2"}],"minor_comments":[{"comment":"Typographical errors: 'Tabel 2' in Section 3.2 should be 'Table 2', and 'addictive' in the Table 1 sample should be 'additive'.","section":"Section 3.2 and Table 1"},{"comment":"In the EPCQ sample, the answer 'discrimination' appears in the question text; if this sample is representative, such leakage should be explicitly measured and discussed, or the sample replaced.","section":"Table 1"},{"comment":"The definition of accuracy is top-1 accuracy on the ten candidates; please state this explicitly in Section 5.2, as the candidate-selection setting is not obvious from the model equations.","section":"Section 5.2"},{"comment":"The paper should report whether the answer word (or its translation) appears in the document D after the aligned paragraph is removed; this statistic is crucial for interpreting whether D can contain the answer at all and for choosing a meaningful upper bound.","section":"Section 3.2"}],"recommendation":"major_revision","confidential_remarks":"If the item-level analysis confirms that PI_Candidates solves a large share of items, the benchmark's contribution would be substantially weakened; the editor may wish to weight this in assessing novelty. The paper's own example leaking the answer suggests the issue is not hypothetical."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here is my read. What is actually new: XCMRC is the first cross-lingual cloze-style MRC benchmark I know of, with two dual sub-datasets (EPCQ, CPEQ) built automatically from aligned English-Chinese news, and a pseudo low-resource task that restricts external language resources. The construction is reasonable, the scale is sufficient for training, and the authors include an explicit monolingual upper bound. They also deserve credit for reporting a passage-independent baseline at all; that is the right diagnostic instinct.\n\nThe problem is what that diagnostic shows. On EPCQ (English passage, Chinese question), the passage-independent model scores 59.83%, slightly higher than the passage-using naive BiDAF_Candidates at 58.35%. On CPEQ, the naive model does better (61.64% vs 58.20%), but the gap is small. So for a large share of items, the question plus ten candidate nouns carry enough signal to choose the answer without reading the source-language passage. The paper acknowledges the closeness but reads it as the naive model failing to learn cross-lingual information. The alternative reading is more worrying: the task itself may not require cross-lingual reading comprehension, at least in the pseudo low-resource setting. Since the stated purpose is to 'test machines on their cross-lingual reading comprehension ability,' this is a load-bearing validity issue, not a footnote. It is fixable—candidate filtering, artifact analysis, or a control showing passage-using models can clearly beat question-only—but as is, the central claim is not established.\n\nOther soft spots are smaller. Only dev set results are reported, despite having a test split. Five runs with the same random seed are not independent; calling the result an average across runs is misleading. The BERT baseline is only partly multilingual: English passage is encoded with mBERT, Chinese question with separate pre-trained Chinese word embeddings, so it is not a clean multilingual-sentence-encoder baseline. The table layout is confusing, and the upper bound is simply a monolingual model, which is honest but loose.\n\nWho should read this: groups working on cross-lingual QA and low-resource MRC will want to know about the dataset and the pseudo low-resource setting. I would send this to review; it deserves a serious referee, but I would expect a revision that addresses the artifact problem and reports test results. I would not cite it in its current form, but I would track a revised version.","headline":"New cross-lingual cloze MRC dataset with a real construct-validity problem: passage-independent baseline nearly ties passage-based models.","tokens_in":10341,"tokens_out":4729,"would_cite":false,"duration_ms":43943,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"XCMRC is presented as the first public cross-lingual cloze-style machine reading comprehension benchmark, with English/Chinese passage–question pairs and baselines that fall well short of a monolingual upper bound.","keywords":["cross-lingual machine reading comprehension","cloze-style reading comprehension","benchmark dataset","pseudo low-resource task","multilingual sentence encoding","translation-based baseline","BiDAF","English-Chinese"],"falsifier":"Run a strong passage-independent model that attends only to the question and candidates on the XCMRC test sets and compare its accuracy with the translation-based and upper-bound baselines; if it matches or exceeds them, the passage is not the source of the answer signal.","tokens_in":9265,"feed_emoji":"🌐","tokens_out":3922,"duration_ms":36396,"temperature":0.7,"pith_summary":"The paper introduces XCMRC, a benchmark for cross-lingual machine reading comprehension built from parallel English–Chinese articles. Each sample asks a model to fill a masked noun in a question written in one language by reading a passage written in the other, with ten candidate nouns provided. The authors claim this is the first public cross-lingual language understanding benchmark for reading comprehension, and they show that translation-based and multilingual-encoder baselines achieve reasonable but incomplete accuracy, while pseudo low-resource approaches lag far behind the upper bound. The benchmark is meant to push research on machines that can read across languages, which matters because most real-world NLP systems encounter many languages without enough annotated data in each.","feed_headline":"First cross-lingual reading-comprehension benchmark debuts","feed_subtitle":"Chinese and English passages stump QA models, with translation baselines far from the monolingual upper bound.","key_machinery":"The organizing mechanism is the automatic dataset generation pipeline: starting from strictly aligned Chinese–English paragraph pairs, the pipeline counts nouns, selects an answer word with corpus frequency 3–10, builds a cloze question by masking that word in a paragraph, removes the corresponding paragraph from the document, and samples nine distractor nouns from the same noun pool. This pipeline is what creates the cross-lingual test: the question and candidates are in the target language, the document is in the source language, and the answer must be recovered by connecting the two. The identical pipeline applied monolingually produces the EPEQ and CPCQ sub-datasets that act as the upper bound oracle.","core_discovery":"The central claim is that XCMRC is the first public benchmark for cross-lingual cloze-style machine reading comprehension, and that it is usable to measure both a general cross-lingual task and a pseudo low-resource variant. The corpus contains two dual subsets: EPCQ (English passages, Chinese questions) with 57,599 samples and CPEQ (Chinese passages, English questions) with 55,990 samples, generated automatically from an aligned bilingual corpus by choosing a noun, blanking it in a sentence, removing the corresponding paragraph from the passage, and adding nine distractor nouns. The same generator produces monolingual EPEQ and CPCQ sub-datasets that serve as an upper bound. Baselines show that the translation-based BiDAF Cloze reaches 67.28% on CPEQ and 65.99% on EPCQ, the multilingual BERT-based model reaches 63.28%, while naive and passage-independent models on the pseudo low-resource task stay near 58–62%, compared with monolingual upper bounds of 72.97% and 68.81%.","pith_inferences":["I infer that the near-parity between the passage-independent baseline (59.83%) and the passage-using naive and translation baselines (58.35–67.28%) indicates that many XCMRC questions can be answered from question and candidate statistics alone, so a revised version should filter such questions to keep the passage load-bearing.","I infer that the same generation pipeline could produce pseudo low-resource XCMRC benchmarks for other language pairs, provided an aligned bilingual corpus exists, making the resource restriction a genuinely controllable experimental variable.","I infer that newer multilingual encoders not tested in this paper could close or even reverse the gap between translation-based and encoder-based approaches, and the reported results give a concrete baseline for such comparisons."],"forward_implications":["If XCMRC is a valid measure, cross-lingual reading comprehension between English and Chinese has substantial headroom: the best common-task baseline is about 3–6 points below the monolingual upper bound.","Pseudo low-resource cross-lingual MRC remains largely unsolved: the naive cross-lingual model learns little beyond what the question alone provides, matching the passage-independent baseline.","The upper bound of the benchmark is not a static ceiling; as monolingual cloze models improve on EPEQ and CPCQ, the estimated headroom for cross-lingual models will need to be revised upward.","Translation-based methods currently outperform multilingual sentence encoders on this benchmark, suggesting that for long-document cloze tasks, explicit translation transfers more information than shared embedding spaces."],"supporting_citations":[{"why":"Defines the extractive MRC task and evaluation setup whose architecture and training paradigm XCMRC's baselines adapt to cloze format.","marker":"[23]"},{"why":"Supplies the BiDAF model used as the base for BiDAF Cloze, BiDAF Candidates, and the passage-independent baseline.","marker":"[26]"},{"why":"Provides the automatic cloze-generation methodology that XCMRC's construction pipeline follows, extended here to parallel bilingual text.","marker":"[11]"},{"why":"Establishes the cloze-style reading comprehension benchmark with answer candidates that XCMRC extends to the cross-lingual setting.","marker":"[12]"},{"why":"Provides the multilingual BERT model used as the sentence encoder in the MSE-based baseline.","marker":"[7]"},{"why":"The XNLI cross-lingual benchmark whose result pattern, translation-based methods performing best, is echoed by the XCMRC findings.","marker":"[5]"}],"fun_headline_variants":["XCMRC: first cross-lingual cloze-style reading test","Cross-lingual reading gap exposed by first benchmark","Machines read across languages, far behind monolingual","New benchmark quantifies cross-lingual reading gap","Translation-based baselines lag in cross-lingual reading"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the passage is necessary to answer the cloze question -- yet the paper's own passage-independent baseline achieves 59.83%, close to the passage-using baselines, which would undermine that premise.","fun_headline_variants_meta":{"raw":{"variants":["XCMRC: first cross-lingual cloze-style reading test","Cross-lingual reading gap exposed by first benchmark","Machines read across languages, far behind monolingual","New benchmark quantifies cross-lingual reading gap","Translation-based baselines lag in cross-lingual reading"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00035,"raw_usage":{"total_tokens":1944,"prompt_tokens":1011,"completion_tokens":933,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":627,"completion_tokens_details":{"reasoning_tokens":851}},"tokens_in":627,"tokens_out":933,"duration_ms":9539,"temperature":1.0,"reasoning_tokens":851,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:14:10.173436+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a strong passage-independent model that attends only to the question and candidates on the XCMRC test sets and compare its accuracy with the translation-based and upper-bound baselines; if it matches or exceeds them, the passage is not the source of the answer signal.","supporting_citations":[{"cited_title":"In: EMNLP","cited_arxiv_id":null,"evidence_quote":"Defines the extractive MRC task and evaluation setup whose architecture and training paradigm XCMRC's baselines adapt to cloze format."},{"cited_title":"I n: NIPS","cited_arxiv_id":null,"evidence_quote":"Provides the automatic cloze-generation methodology that XCMRC's construction pipeline follows, extended here to parallel bilingual text."},{"cited_title":"Co mputer Science (2015)","cited_arxiv_id":null,"evidence_quote":"Establishes the cloze-style reading comprehension benchmark with answer candidates that XCMRC extends to the cross-lingual setting."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the multilingual BERT model used as the sentence encoder in the MSE-based baseline."},{"cited_title":"In: EMNLP","cited_arxiv_id":null,"evidence_quote":"The XNLI cross-lingual benchmark whose result pattern, translation-based methods performing best, is echoed by the XCMRC findings."}],"review_version":1}