{"id":"4b6c87e2-d384-4adb-8e8f-9145fbe9704f","arxiv_id":"1909.00361","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Dual BERT, a multilingual BERT model with a bilingual decoder, improves machine reading comprehension in low-resource languages by jointly modeling machine-translated source data and target data.","lead":"The paper introduces cross-lingual machine reading comprehension and a model called Dual BERT that uses English data to improve reading comprehension in Chinese and other low-resource languages. It reports large gains on Chinese, Japanese, and French datasets, suggesting a practical path for building MRC systems without large local training sets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cross-lingual attention in Eq. 4 is asserted but never validated; observed gains may reflect a second-view regularizer rather than token-level bilingual transfer.","rationale":"The reader's conditional verdict is appropriate, and the weakest assumption identified by the reader matches my main concern. The empirical results appear internally consistent: adding an English-translated view of the same training data and an auxiliary span loss improves Chinese MRC over the target-only baseline, and the ablations in Table 4 show the source branch contributes. The problem is that the paper's stated mechanism—token-level cross-lingual attention transferring span information—is underdetermined by the experiments. The source and target sequences are translations of each other but are not word-aligned, and multilingual BERT's cross-lingual token alignment for Chinese-English is imperfect. Thus the improvement could come from a second-view regularizer rather than from genuine bilingual semantic modeling. This concern is checkable without redoing the main experiments: extract the attention matrix and test its alignment quality against external word alignments. Because the concern is about missing validation rather than a demonstrated error, I keep the reader's CONDITIONAL verdict unchanged.","tokens_in":10849,"tokens_out":6520,"duration_ms":66708,"concrete_test":"On a random sample of 200 CMRC dev examples, run the trained Dual BERT model and extract the self-attentive attention matrix A~_TS from Eq. 4. For each token in the predicted target span, record the source token with maximum attention. Independently, obtain Chinese–English word alignments for the same passages using GNMT attention or a standard aligner such as fast_align on the parallel GNMT outputs. Compute the fraction of target span tokens whose attended source token falls within the aligned source span of the gold answer, and compare this with random attention and with self-attention. If the cross-lingual attention is no more aligned than random, the claimed token-level transfer mechanism is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The key mechanism of Dual BERT is the bilingual decoder in Section 4.2: Equation 4 applies softmax-filtered dot-product attention between Chinese and English BERT token representations, and the paper claims this transfers span information across languages. No evidence is provided that this attention is actually cross-lingually aligned. Multilingual BERT provides imperfect token-level alignment for a distant pair like Chinese-English, and the source input is an independent GNMT translation with different segmentation and no word-level alignment supervision. If the attention is not aligned, the observed gains over the target-only baseline are equally consistent with an auxiliary-training or regularization effect: the source branch is a second view of the same training instances, and the auxiliary loss in Section 4.3 adds a second span-supervision signal. Under that explanation, the central claim that Dual BERT learns semantic relations between passage and question in a bilingual context is not established, and portability to other distant language pairs remains unsupported. This is a missing validation of the core mechanism, not an allegation of incorrect reporting.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Cross-Lingual Machine Reading Comprehension (CLMRC) as a task and proposes two families of approaches: zero-shot back-translation pipelines (GNMT, SimpleMatch, Answer Aligner, Answer Verifier) and a supervised model called Dual BERT that encodes target-language and machine-translated source-language inputs with multilingual BERT and combines them through a bilingual decoder with self-adaptive attention and dynamic auxiliary-loss weighting. Experiments are conducted on CMRC 2018, DRCD, and 304-sample Japanese and French SQuAD sets. The authors report state-of-the-art results on the Chinese datasets and claim that the proposed systems surpass human performance on some metrics.","tokens_in":10968,"tokens_out":3526,"duration_ms":34732,"significance":"If the reported gains are reliable, the paper offers a practical recipe for improving extractive MRC in low-resource languages by exploiting English SQuAD through machine translation and a shared multilingual encoder, and it would be one of the first systematic evaluations of such cross-lingual transfer on large realistic Chinese benchmarks. The paper's strengths include experiments on public Chinese datasets, a clear ablation study, comparisons within a controlled experimental setup, and a public code release. However, the central mechanism of Dual BERT is not directly validated, the significance testing is under-specified, and at least one headline claim is contradicted by the paper's own table; these issues need to be addressed before the claims are fully supported.","major_comments":[{"comment":"The Abstract and Section 1 claim that the proposed systems achieve state-of-the-art performance and 'even surpass human performance on some metrics.' Table 2 does not support this: on CMRC 2018 the best system reaches 73.6 EM/90.2 F1 on the test set while human performance is 92.4 EM/97.9 F1, and on DRCD the best system reaches 85.4 EM/91.6 F1 while human performance is 80.4 EM/93.3 F1 (the EM is lower but F1 is higher for the system, and no overall surpassing is visible). Please correct or explicitly qualify this overclaim.","section":"Abstract/§1 and Table 2"},{"comment":"The text states that all improvements over the baselines are 'verified by p-test with p < 0.05,' but it does not specify which statistical test was used, how many random seeds or runs were averaged, or whether significance was computed on the development, test, or challenge sets. Since the paper's central claim is of consistent and significant improvements, this reporting gap is load-bearing. Please provide the test name, the number of runs, and the exact procedure, or soften the significance claim accordingly.","section":"§5.2"},{"comment":"The bilingual decoder in Equation (4) computes softmax-filtered dot-product attention between target and source BERT token representations, and the ablation in Section 5.4 attributes the gain to cross-lingual transfer. However, no experiment verifies that this attention is actually aligned at the token level between Chinese and English, especially given that the source is produced by an independent GNMT translation with different segmentation and no alignment supervision. The observed gain is equally consistent with an auxiliary-training or regularization effect: the source branch is a second view of the same instance and the auxiliary loss adds an extra span-supervision signal. Please add a direct validation of the alignment mechanism, for example by measuring attention quality against word-level translation links, by shuffling the source token order, or by comparing against an auxiliary-loss-only baseline without cross-attention.","section":"§4.2, Eq. (4) and §5.4"},{"comment":"The Japanese and French SQuAD experiments use only 304 samples per language, and the improvement over Asai et al. (2018) is measured against a different base system (BERT-Large as the English reader versus their model). No significance test or confidence interval is reported for these differences. Please frame these zero-shot results accordingly, and either provide significance measures or explicitly describe the comparison as indicative rather than statistically verified.","section":"§5.3 and Table 3"}],"minor_comments":[{"comment":"The notation 'BERT Bcn' and 'BERT Bmul' in Table 2 is not defined in the text; please clarify which pre-trained checkpoints are used for the Chinese and multilingual variants.","section":"§5.1"},{"comment":"In the sentence after Equation (4), the text refers to calculating the dot product between 'AST' and 'BS', while the equation uses Ã_TS; please make the notation consistent.","section":"§4.2"},{"comment":"The phrase 'resulting in the start and target prediction PsS, PeS' appears to contain a typo; it should likely read 'start and end prediction.'","section":"§4.3"},{"comment":"The relax parameter δ is introduced with a typical range but its exact value in the reported experiments is not specified; please state how δ was chosen.","section":"§3.2"},{"comment":"The curves in Figure 3 would be easier to interpret with error bars or an explicit statement of the number of random seeds used, since differences between small pre-training data sizes appear modest.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"This is a useful empirical paper that would likely be publishable after revision. The main reasons for major revision are the unsupported 'surpass human performance' claim, the underspecified significance testing, and the lack of direct evidence for the cross-lingual attention mechanism, which is the core novelty of Dual BERT. The paper is not fatally flawed; the issues can be addressed with additional analyses and corrected reporting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead this if you work on multilingual QA or transfer. The paper’s concrete contribution is a set of back-translation pipelines plus a Dual BERT model that jointly encodes machine-translated English and original Chinese instances, with a dynamic lambda weighting for the auxiliary source loss. The evaluation on CMRC 2018 and DRCD is real and realistic, unlike earlier work on tiny SQuAD translations, and the ablations show each component adds something. Code and datasets are available. That is the honest core, and it is a legitimate contribution.\n\nThe main soft spots are the usual for this kind of paper, plus one deeper one. The reported p-test is not specified: no test type, no number of runs, and no variance. The Japanese/French zero-shot comparison uses 304 samples per language, so those numbers are suggestive, not decisive. The 'state-of-the-art' claim is slightly overstated because several previous systems used different base models or training regimes; the paper acknowledges some are not directly comparable, but the abstract still says 'by a large margin.'\n\nThe deeper issue is the central mechanism. Section 4.2 says Dual BERT transfers span information through cross-lingual attention between multilingual BERT representations (Eq. 4), but no analysis shows those attention weights actually align Chinese and English token spans. Multilingual BERT is not reliably token-aligned for distant pairs, and the English side is a separate GNMT translation with different segmentation. Without such validation, the gains over the target-only baseline are equally consistent with the source branch acting as a second-view regularizer plus the auxiliary loss. That does not invalidate the empirical result, but it means the paper’s theoretical explanation is under-supported. The authors could check this with attention probes or by testing on a truly distant pair. I would not call it a fatal flaw; the empirical contribution stands on its own.\n\nWho is this for? Researchers building MRC systems for low-resource languages and people working on cross-lingual transfer. It deserves serious peer review; the missing mechanism evidence should be requested in revision, not grounds for rejection.\n\nRecommendation: engage; referee it, and ask for a proper significance test and at least one diagnostic of the bilingual attention.","headline":"A solid, honest empirical study of cross-lingual MRC with real gains on Chinese benchmarks; the central bilingual-attention mechanism is asserted rather than proven, but the paper is worth refereeing.","tokens_in":11532,"tokens_out":2801,"would_cite":true,"duration_ms":23653,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Cross-lingual transfer through a shared multilingual BERT lets English reading-comprehension data improve Chinese machine reading comprehension, and the proposed Dual BERT achieves state-of-the-art results on CMRC 2018 and DRCD…","keywords":["cross-lingual machine reading comprehension","Dual BERT","multilingual BERT","back-translation","span extraction","low-resource languages","self-adaptive attention","machine translation"],"falsifier":"Run Dual BERT on a language pair with poor multilingual-BERT alignment (for example, English and a typologically distant language with little shared vocabulary) and check whether the bilingual decoder's gain over cascade training disappears or reverses; if the shared embedding space cannot align the two languages, the transfer mechanism fails.","tokens_in":10609,"feed_emoji":"🌐","tokens_out":9870,"duration_ms":78762,"temperature":0.7,"pith_summary":"The paper tries to show that machines can learn to read in a low-resource language by importing skill from a high-resource language, even when the languages are very different. It formalizes a Cross-Lingual Machine Reading Comprehension task, gives back-translation pipelines that translate passages and questions into English, answer them there, and map the answers back, and then proposes Dual BERT, which reads both the Chinese sample and its English translation in one shared multilingual BERT and decodes the Chinese answer from the bilingual context. The results on two public Chinese datasets show consistent gains over systems trained only on Chinese, with state-of-the-art scores and a human-beating exact-match score on one benchmark. If this holds, languages with little labeled reading-comprehension data can leverage English SQuAD-scale data through translation plus a shared representation.","feed_headline":"Dual BERT lifts Chinese reading comprehension past human scores","feed_subtitle":"A bilingual decoder lets English training data improve Chinese question answering where labeled data is scarce.","key_machinery":"The load-bearing mechanism is the Self-Adaptive Attention in the bilingual decoder: it multiplies the raw inter-lingual attention $A_{TS}=B_T B_S^\\top$ on both sides by the self-attention maps $A_T=\\mathrm{softmax}(B_T B_T^\\top)$ and $A_S=\\mathrm{softmax}(B_S B_S^\\top)$, producing $\\tilde A_{TS}=A_T A_{TS} A_S^\\top$, and then attends to source representations with it. This lets each language first filter its own irrelevant tokens and then lets target tokens query the filtered source context. A cosine-similarity weight $\\lambda=\\max\\{0,\\cos\\langle \\tilde H_S,\\tilde H_T\\rangle\\}$ on the auxiliary source-span loss down-weights translated samples whose ground-truth span sits far from the target span in representation space.","core_discovery":"The paper's central claim is that cross-lingual transfer for extractive reading comprehension works by decoding target spans through a shared multilingual BERT that simultaneously sees the machine-translated source version of the same passage-question pair. Dual BERT uses the target BERT representation as the query and the source representation as key and value in a modified multi-head attention layer, then fuses the attended source representation into the target representation with a residual connection. The auxiliary source-language span loss is weighted by a dynamic lambda, computed as the cosine similarity between the ground-truth span representations in the two languages, so noisy translations contribute less. On CMRC 2018 and DRCD the model improves over target-only multilingual BERT, cascade training, and mixed English-Chinese training, and it surpasses human exact-match on the DRCD test set.","pith_inferences":["The same recipe should extend to other language pairs covered by multilingual BERT and a decent translation system, but the Japanese and French results are zero-shot baselines only, so the full Dual BERT training loop on those languages is untested.","The dynamic lambda could be reused as a general noise-weighting signal for any machine-translated training data, since it measures whether source and target representations of the same answer are close.","The discussion's conclusion that data size beats language proximity, if right, suggests a practical selection rule: choose the largest available source corpus rather than the most typologically similar language."],"forward_implications":["Dual BERT with SQuAD pre-training sets state-of-the-art results on CMRC 2018 and DRCD, including a human-beating exact-match score on DRCD test.","The character-level SimpleMatch alignment improves zero-shot Japanese and French SQuAD results, raising Japanese F1 by 11.8 points, more than the prior NMT-attention alignment method.","Back-translation with an Answer Verifier surpasses multilingual BERT trained only on target-language data, so translated English samples can strengthen supervised Chinese MRC.","When source pre-training data is plentiful, using a distant but data-rich language (English) can work as well as or better than using a close but data-poor language (Traditional Chinese) for a Simplified Chinese task."],"supporting_citations":[{"why":"supplies multilingual BERT, the shared encoder whose embedding space carries the cross-lingual alignment the whole method relies on.","marker":"Devlin et al. 2019"},{"why":"supplies SQuAD, the source-language training and pre-training resource used for both back-translation and Dual BERT.","marker":"Rajpurkar et al. 2016"},{"why":"provides the runtime machine-translation baseline and the Japanese and French SQuAD evaluation sets used for zero-shot comparison.","marker":"Asai et al. 2018"},{"why":"provides CMRC 2018, one of the two target Chinese span-extraction datasets.","marker":"Cui et al. 2019"},{"why":"provides DRCD, the traditional-Chinese target dataset.","marker":"Shao et al. 2018"},{"why":"supplies the neural machine translation system and WordPiece tokenization used to create the translated training samples.","marker":"Wu et al. 2016"},{"why":"supplies multi-head self-attention, which Self-Adaptive Attention modifies for the cross-lingual decoder.","marker":"Vaswani et al. 2017"}],"fun_headline_variants":["Dual BERT surpasses human on DRCD Chinese MRC test","Cross-lingual MRC: English data boosts Chinese Q&A","Bilingual attention model beats human on Chinese reading","Dual BERT uses English data to lift Chinese comprehension","Back-translation and dual attention improve Chinese MRC"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that multilingual BERT's embedding space aligns English and Chinese tokens well enough for the bilingual decoder's cross-lingual attention to transfer span information; if that token-level alignment fails, the model cannot transfer reading-comprehension skill from source to target.","fun_headline_variants_meta":{"raw":{"variants":["Dual BERT surpasses human on DRCD Chinese MRC test","Cross-lingual MRC: English data boosts Chinese Q&A","Bilingual attention model beats human on Chinese reading","Dual BERT uses English data to lift Chinese comprehension","Back-translation and dual attention improve Chinese MRC"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000206,"raw_usage":{"total_tokens":1378,"prompt_tokens":910,"completion_tokens":468,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":526,"completion_tokens_details":{"reasoning_tokens":385}},"tokens_in":526,"tokens_out":468,"duration_ms":4669,"temperature":1.0,"reasoning_tokens":385,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T05:54:04.637651+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Dual BERT on a language pair with poor multilingual-BERT alignment (for example, English and a typologically distant language with little shared vocabulary) and check whether the bilingual decoder's gain over cascade training disappears or reverses; if the shared embedding space cannot align the two languages, the transfer mechanism fails.","supporting_citations":[{"cited_title":"Multilingual Extractive Reading Comprehension by Runtime Machine Translation","cited_arxiv_id":"1809.03275","evidence_quote":"provides the runtime machine-translation baseline and the Japanese and French SQuAD evaluation sets used for zero-shot comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides CMRC 2018, one of the two target Chinese span-extraction datasets."}],"review_version":1}