Pith. sign in

REVIEW 2 major objections 5 minor 33 references

X-WikiRE: A Large, Multilingual Resource for Relation Extraction as Machine Comprehension

T0 review · 2 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read One multilingual reading-comprehension model can extract facts in five languages by borrowing relations seen in other languages.

desk verdict X-WikiRE is a genuinely useful multilingual resource, but the abstract's 'zero-shot' claim overstates what the experiments actually show. read the letter →

arxiv 1908.05111 v2 pith:ZO3S32EY submitted 2019-08-14 cs.CL

Mostafa Abdou , Cezar Sas , Rahul Aralikatte , Isabelle Augenstein , Anders S{o}gaard This is my paper · ORCID
classification cs.CL
keywords X-WikiREmultilingualrelationextractionreadingcomprehensionzero-shotcross-lingualtransferknowledgebasepopulationdistantsupervisionmachine
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces X-WikiRE, a multilingual relation-extraction dataset covering English, German, French, Spanish, and Italian, built by turning each relation into natural-language question templates and aligning encyclopedia pages with facts from a public knowledge base. The authors claim that a model trained on this resource transfers across languages: with only 10,000 target-language examples for fine-tuning, an English-trained model nearly matches the performance of a model trained on the full target-language training set. They further claim that a single model trained jointly on all five languages exploits differences in how well each language's knowledge base is populated, roughly doubling zero-shot relation-extraction F1 in most languages compared with a monolingual model. If these results hold, relation-extraction systems could populate under-resourced knowledge bases from well-resourced ones with very little in-language annotation.

What carries the argument

The load-bearing mechanism is the relation-as-question reformulation combined with a nil-aware reading-comprehension model. Each relation type is mapped to one or more question templates with an entity slot, so extracting a fact becomes finding the answer span in a paragraph or returning NIL when no answer is present. The dataset is produced automatically by aligning knowledge-base statements with encyclopedia text, and the model uses multilingually aligned word vectors so the same parameters can process all five languages. The nil-aware component decomposes each context word's evidence into a part relevant to answering the question and an orthogonal part, and this decomposition is what lets the model decide between outputting a span and declining to answer.

What would settle it

Sample positive triples from X-WikiRE, have annotators mark whether the extracted sentence truly states the relation, and recompute precision on the verified subset; if precision drops sharply, the distant supervision is overly optimistic. A complementary test is to evaluate the trained models on a hand-annotated multilingual relation-extraction benchmark and compare with the dataset-reported scores.

Watch

Extended reading notes

Core claim

The central discovery is that relation extraction, reformulated as answering questions, can be shared across languages. The authors construct the dataset by distant supervision: for each knowledge-base triple, they take the first sentence of the encyclopedia article that mentions both entities, and negatives are built by swapping contexts between triples with the same entity type. They then show, empirically, that (1) an English-trained nil-aware reading-comprehension model recovers most of a target language's monolingual performance with 1,000 to 10,000 fine-tuning examples, and (2) a jointly trained multilingual model matches or exceeds monolingual baselines when the target is unseen entities and roughly doubles zero-shot F1 when the target is unseen relation types. The paper attributes the zero-shot gain to the fact that different languages populate the same relations to very different degrees, so a relation that is missing from one language's training data is likely present in another's.

Load-bearing premise

The entire dataset rests on the unverified assumption that the first sentence mentioning both entities in an encyclopedia article actually states the relation recorded in the knowledge base; if that fails, the gold labels are noisy and reported F1 scores may not reflect real extraction quality.

Editorial extensions

If this is right

  • A single trained system can populate several low-resource knowledge bases from one well-resourced language, needing only a few thousand labeled examples per new language.
  • Zero-shot relation-extraction evaluation becomes feasible at much larger scale, because relations seen in any of the five languages provide supervision for the others.
  • Parameter efficiency improves: one multilingual model replaces five monolingual models while matching or beating them in most settings.
  • Representation choice matters: in this setup, multilingually aligned word vectors with high vocabulary coverage beat a widely used multilingual masked-language model, so vocab coverage should be checked before choosing a representation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the key mechanism is cross-lingual relation sharing, then adding a sixth language with a very different topical profile should still help, but the gain will depend on how many of its relations overlap with the existing five; this can be measured by leaving out one language and re-running the zero-shot evaluation.
  • The same distant-supervision recipe could be applied to languages not in the paper, and the small fine-tuning requirement suggests that even a few thousand hand-verified examples per language could unlock large gains.
  • A natural stress test is to check how much of the zero-shot improvement survives when the source and target languages are from different families, since the current five include four Romance or Germanic languages; the paper does not report this comparison.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper introduces X-WikiRE, a large multilingual relation-extraction dataset for English, German, Spanish, French, and Italian, constructed from Wikidata and Wikipedia by distant supervision and by human translation of question templates. Relation extraction is framed as machine comprehension, and experiments with a nil-aware reading-comprehension model (NAMANDA) compare monolingual baselines, cross-lingual transfer with target-language finetuning, and joint multilingual training. The central claims are that models can be transferred cross-lingually with little in-language data, and that multilingual training improves zero-shot relation extraction, especially when relations are not observed in a target language but are observed in other languages.

Significance. If the claims hold, X-WikiRE is a useful large-scale resource for multilingual relation extraction, and the cross-lingual transfer results would support a practical route to populating low-resourced knowledge bases from well-resourced ones. The paper is commendable for releasing the dataset and code (though no URL is provided), for human-translating question templates, and for benchmarking against Levy et al. (2017). The experiments are extensive and, with one important caveat, support the weaker claim that multilingual supervision improves target-language relation extraction. The stronger claim about zero-shot relation extraction, however, requires an entity-disjoint control to rule out cross-lingual entity memorization.

major comments (2)
  1. [§5.3, Table 3 (UnREL)] The abstract's claim that multilingual support improves zero-shot relation extraction is not supported as stated, because the multilingual UnREL setup only removes a relation from the target language's training set while the same relation remains in the other four languages' training sets, and test entities may appear in those languages' training data. The large gains (e.g., ES F1 34.54→67.64, IT 35.62→61.13, FR 31.78→53.06) could therefore arise from the model seeing a specific entity-relation pair in another language and copying the answer, rather than from generalizing to a truly unseen relation. The paper should report an entity-disjoint control in which test entities are removed from all languages' training data, or should re-frame the claim as cross-lingual supervision for target-language RE rather than zero-shot relation extraction.
  2. [Section 3, 'Slot-filling data'] The dataset is constructed by distant supervision: for each Wikidata triple, the first sentence containing both entity1 and entity2 is taken as evidence that the relation holds, with no human verification or sample-quality analysis. Since X-WikiRE is itself a central contribution, the absence of any manual evaluation of label precision leaves the reliability of all reported F1 scores unquantified. A sample-based human evaluation or at least a detailed error analysis should be added to establish the quality of the benchmark.
minor comments (5)
  1. [Table 3] In the UnREL columns, the table reports a single 'Multi.' score, but the text describes both a 200k-per-language model and a full-data model for the UnENT setting; it should be clarified which setup the UnREL multilingual model uses.
  2. [Appendix B, Table 4] The table header contains a typo: 'Avarage' should be 'Average'.
  3. [Section 5.4 / Appendix D] The comparison between multilingual BERT and fastText is not fully fair because BERT is not fine-tuned and only its static final-layer features are used; this should be stated more explicitly as a limitation of the comparison.
  4. [Conclusion] The paper says 'we release our code and dataset' but no URL or repository identifier is given; please add a link for reproducibility.
  5. [Section 5.2, Figure 5] Cross-lingual transfer results are reported without confidence intervals or significance tests; since the differences between finetuning sizes are sometimes small, statistical testing would strengthen the claims.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: held-out empirical evaluation with external comparison; the only overlapping-author citation is incidental and not load-bearing.

full rationale

This paper does not derive a theoretical prediction from an input that is defined to be that prediction. The central contributions are empirical: a new distant-supervision dataset built from Wikidata/Wikipedia alignments, a machine comprehension model taken from prior work (Kundu and Ng, 2018), and held-out evaluations. Monolingual baselines, cross-lingual transfer curves, and multilingual training results are produced by training on disjoint splits and measuring F1 against gold spans. The claimed gains of multilingual UnREL training (Table 3) are empirical outcomes on test folds, not quantities that were fitted or defined into existence. The only overlapping-author citation is Saha et al. (2018) for DuoRC in a list of nil-aware datasets (Section 4); it is not load-bearing. The zero-shot protocol is explicitly described in Section 5.3: a relation is held out from the target language's training set but may appear in other languages. That is a transparent design choice about cross-lingual supervision, and while it could weaken the interpretation of "zero-shot" and the lack of an entity-disjoint control is a real evaluation concern, it does not make the result equivalent to its inputs by construction. No fitted parameter is relabeled as a prediction, no uniqueness theorem is imported, and no known result is repackaged as a derivation. Hence no circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

All three axioms are domain assumptions inherited from the automatic construction pipeline. They are not proven or validated against external benchmarks, and they bound the trust one can place in the absolute numbers.

assumptions (3)
  • domain assumption Distant supervision validity: a sentence containing both entity1 and entity2 expresses the Wikidata relation of the triple.
    Section 3, Slot-filling data: contexts are extracted as the first sentence containing both entities; no human verification of label correctness is performed.
  • domain assumption Question template translation fidelity: human translations preserve the relation semantics and entity-type answer constraints across the five languages.
    Section 3, Querification: translators paraphrased 1192 English templates and added gender or agreement placeholders; no inter-annotator agreement or quality control is reported.
  • domain assumption Cross-lingual embedding alignment: supervised fastText spaces align words well enough to support transfer.
    Section 4, Multilingual representations: all transfer experiments rely on this, and the paper itself shows multilingual BERT fails due to vocabulary coverage, so the representation choice is load-bearing.

how reviews work

0 comments
Cite this review

Pith. "Pith review of X-WikiRE: A Large, Multilingual Resource for Relation Extraction as Machine Comprehension." pith.science (2026). https://pith.science/paper/ZO3S32EY

@misc{pith2026190805111,
  author       = {Pith},
  title        = {Pith review of: X-WikiRE: A Large, Multilingual Resource for Relation Extraction as Machine Comprehension},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZO3S32EY}},
  note         = {Machine review of arXiv:1908.05111}
}
read the original abstract

Although the vast majority of knowledge bases KBs are heavily biased towards English, Wikipedias do cover very different topics in different languages. Exploiting this, we introduce a new multilingual dataset (X-WikiRE), framing relation extraction as a multilingual machine reading problem. We show that by leveraging this resource it is possible to robustly transfer models cross-lingually and that multilingual support significantly improves (zero-shot) relation extraction, enabling the population of low-resourced KBs from their well-populated counterparts.

Figures

Figures reproduced from arXiv: 1908.05111 by the authors.

Figure 1
Figure 1. The overlap of triples between languages. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The number of triples for the top 10 properties in each language. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. An overview of Namanda’s architecture. (Trischler et al., 2017; Rajpurkar et al., 2018; Saha et al., 2018). We employ the architecture de￾scribed in Kundu and Ng (2018) as our standard reading comprehension model for all the experi￾ments. This nil-aware answer extraction frame￾work (NAMANDA) is briefly described below. In a set of initial trials (see [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Our cross-lingual transfer and multilingual training setups. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: F1-scores for the cross-lingual transfer experiments in the [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Precision, Recall and F1-scores for the cross-lingual transfer experiments in UnREL setting. The results are the mean of 5-fold cross-validation. The MONO line shows the corresponding monolingual model’s F1-score. tions in this framework. 5.3 One Model, Multiple Langua…
Figure 10
Figure 10. Figure 10: Property start time. B Context size We computed the average length of the context in out dataset and Levy et al. (2017)’s dataset. Ob￾serve that in our dataset contexts are longer on av￾erage. Also observe that, on average, contexts in test set have more tokens than t…
Figure 7
Figure 7. Figure 7: Property place of death. DE EN FR ES EN FR ES IT Property: military rank (P410) 66 431 [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Property military rank. DE EN FR ES EN FR ES IT Property: noblefamily(P53) 64 483 [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 9
Figure 9. Figure 9: Property ‘noble f amily. DE EN FR ES EN FR ES IT Property: start time (P580) 187 629 [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 17 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Zeljko Agic and Natalie Schluter. 2018. http://dblp.uni-trier.de/db/conf/lrec/lrec2018.html#AgicS18 Baselines and Test Data for Cross-Lingual Inference . In LREC. European Language Resources Association (ELRA)

    work page 2018

  4. [4]

    Nguyen Bach and Sameer Badaskar. 2007. http://www.cs.cmu.edu/\ nbach/papers/A-survey-on-Relation-Extraction.pdf A Review of Relation Extraction

    work page 2007

  5. [5]

    Michele Banko, Michael J Cafarella, Stephen Soderland, Matthew Broadhead, and Oren Etzioni. 2007. http://www.aaai.org/Papers/IJCAI/2007/IJCAI07-429.pdf Open information extraction from the web. In IJCAI, volume 7, pages 2670--2676

    work page 2007

  6. [6]

    Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. http://aclweb.org/anthology/Q17-1010 Enriching Word Vectors with Subword Information . Transactions of the Association for Computational Linguistics, 5:135--146

    2017

  7. [7]

    Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. https://arxiv.org/pdf/1708.00055 SemEval-2017 Task 1: Semantic Textual Similarity-Multilingual and Cross-lingual Focused Evaluation . arXiv preprint arXiv:1708.00055

    arXiv 2017

  8. [8]

    Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. https://doi.org/10.18653/v1/P17-1171 Reading wikipedia to answer open-domain questions . In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1870--1879. Association for Computational Linguistics

    doi:10.18653/v1/p17-1171 2017

Show all 33 references
  1. [9]

    Alexis Conneau, Guillaume Lample, Marc'Aurelio Ranzato, Ludovic Denoyer, and Herv \'e J \'e gou. 2017. https://arxiv.org/pdf/1710.04087 Word translation without parallel data . arXiv preprint arXiv:1710.04087

    2017 arXiv

  2. [10]

    Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. http://aclweb.org/anthology/D18-1269 XNLI: Evaluating Cross-lingual Sentence Representations . In Proceedings of the 2018 Conference on Empirical Methods i...

    2018

  3. [11]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. https://arxiv.org/pdf/1810.04805 BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding . arXiv preprint arXiv:1810.04805

    2018 arXiv

  4. [12]

    Joe Ellis, Jeremy Getman, Dana Fore, Neil Kuster, Zhiyi Song, Ann Bies, and Stephanie M Strassel. 2015. https://www.ldc.upenn.edu/sites/www.ldc.upenn.edu/files/tackbp2016-linguistic-resources-tackbp-1.pdf Overview of Linguistic Resources for the TAC KBP 2015 Evaluations: Metho...

    2015

  5. [13]

    Anthony Fader, Stephen Soderland, and Oren Etzioni. 2011. http://aclweb.org/anthology/D11-1142 Identifying Relations for Open Information Extraction . In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 1535--1545. Association for C...

    2011

  6. [14]

    Manaal Faruqui and Shankar Kumar. 2015. https://doi.org/10.3115/v1/N15-1151 Multilingual Open Relation Extraction Using Cross-lingual Projection . In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua...

    2015 doi

  7. [15]

    Daniel Hewlett, Alexandre Lacoste, Llion Jones, Illia Polosukhin, Andrew Fandrianto, Jay Han, Matthew Kelcey, and David Berthelot. 2016. https://doi.org/10.18653/v1/P16-1145 WikiReading: A Novel Large-scale Language Understanding Task over Wikipedia . In Proceedings of the 54t...

    2016 doi

  8. [16]

    Lucie-Aimée Kaffee and Elena Simperl. 2018. http://dblp.uni-trier.de/db/conf/wikis/opensym2018.html#KaffeeS18 Analysis of Editors' Languages in Wikidata . In OpenSym, pages 21:1--21:5. ACM

    2018

  9. [17]

    Diederik P Kingma and Jimmy Ba. 2014. https://arxiv.org/pdf/1412.6980 Adam: A method for stochastic optimization . arXiv preprint arXiv:1412.6980

    2014 arXiv

  10. [18]

    Souvik Kundu and Hwee Tou Ng. 2018. http://aclweb.org/anthology/D18-1456 A Nil-Aware Answer Extraction Framework for Question Answering . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4243--4252. Association for Computational ...

    2018

  11. [19]

    Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017. https://doi.org/10.18653/v1/K17-1034 Zero-Shot Relation Extraction via Reading Comprehension . In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 333--342. Associ...

    2017 doi

  12. [20]

    Makoto Miwa and Mohit Bansal. 2016. https://doi.org/10.18653/v1/P16-1105 End-to-End Relation Extraction using LSTMs on Sequences and Tree Structures . In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1105...

    2016 doi

  13. [21]

    Abiola Obamuyide and Andreas Vlachos. 2018. Zero-shot relation classification as textual entailment. In Proceedings of the First Workshop on Fact Extraction and VERification (FEVER), pages 72--78

    2018

  14. [22]

    Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. https://doi.org/10.3115/v1/D14-1162 Glove: Global Vectors for Word Representation . In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532--1543. Associati...

    2014 doi

  15. [23]

    Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. http://aclweb.org/anthology/P18-2124 Know What You Don't Know: Unanswerable Questions for SQuAD . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784-...

    2018

  16. [24]

    Sebastian Riedel, Limin Yao, Andrew McCallum, and Benjamin M. Marlin. 2013. http://aclweb.org/anthology/N13-1008 Relation Extraction with Matrix Factorization and Universal Schemas . In Proceedings of the 2013 Conference of the North American Chapter of the Association for Com...

    2013

  17. [25]

    Tim Rockt \"a schel, Sameer Singh, and Sebastian Riedel. 2015. Injecting logical background knowledge into embeddings for relation extraction. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language T...

    2015

  18. [26]

    Khapra, and Karthik Sankaranarayanan

    Amrita Saha, Rahul Aralikatte, Mitesh M. Khapra, and Karthik Sankaranarayanan. 2018. http://aclweb.org/anthology/P18-1156 DuoRC: Towards Complex Language Understanding with Paraphrased Reading Comprehension . In Proceedings of the 56th Annual Meeting of the Association for Com...

    2018

  19. [27]

    Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016. https://arxiv.org/pdf/1611.01603 Bidirectional attention flow for machine comprehension . arXiv preprint arXiv:1611.01603

    2016 arXiv

  20. [28]

    Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. https://doi.org/10.18653/v1/W17-2623 NewsQA: A Machine Comprehension Dataset . In Proceedings of the 2nd Workshop on Representation Learning for NLP, pages 191-...

    2017 doi

  21. [29]

    Patrick Verga, David Belanger, Emma Strubell, Benjamin Roth, and Andrew McCallum. 2016. https://doi.org/10.18653/v1/N16-1103 Multilingual Relation Extraction using Compositional Universal Schema . In Proceedings of the 2016 Conference of the North American Chapter of the Assoc...

    2016 doi

  22. [30]

    Denny Vrande c i \'c and Markus Kr \"o tzsch. 2014. Wikidata: a free collaborative knowledge base

    2014

  23. [31]

    Alexander Yates, Michele Banko, Matthew Broadhead, Michael J Cafarella, Oren Etzioni, and Stephen Soderland. 2007. http://acl.ldc.upenn.edu/N/N07/N07-4013.pdf TextRunner: Open Information Extraction on the Web . In HLT-NAACL (Demonstrations), pages 25--26

    2007

  24. [32]

    Dmitry Zelenko, Chinatsu Aone, and Anthony Richardella. 2003. http://www.jmlr.org/papers/volume3/zelenko03a/zelenko03a.pdf Kernel methods for relation extraction . Journal of Machine Learning Research, 3:1083--1106

    2003

  25. [33]

    GuoDong Zhou, Jian Su, Jie Zhang, and Min Zhang. 2005. http://aclweb.org/anthology/P05-1053 Exploring Various Knowledge in Relation Extraction . In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL'05), pages 427--434. Association for...

    2005

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.