REVIEW 3 major objections 4 minor 2 references
Challenges in Expanding Portuguese Resources: A View from Open Information Extraction
T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read A theory-grounded annotation methodology produces a high-quality Portuguese Open IE corpus with near-perfect annotator agreement.
desk verdict A genuinely useful Portuguese Open IE corpus with an annotation scheme worth reading, but the validation table has impossible MCC values and the abstract overclaims what was validated. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is the structured-propositions thesis — the claim that the structure of the proposition expressed by a sentence is a function of that sentence's structure — together with a set of seven structural and two contextual annotation rules (S1–S5, S4.1–S4.2, R4) that convert dependency trees into valid extractions. The thesis licenses the use of syntactic dependency structure as a guide to propositional content, so a relationship is recognized when its structure is embedded in, or obtainable by sense-preserving transformation from, the sentence's structure. The rules determine what counts as a valid triple (arg1, rel, arg2): which phrases may serve as arguments, when prepositions attach to the relation descriptor, when adverbs merge with verbs, and when an extraction is informative rather than merely well-formed.
What would settle it
Annotate the silver set again using the finalized guidelines and measure inter-annotator agreement and classifier AUC; if the reannotated silver set does not approach the gold set's kappa of 0.94, the improvement in agreement may come from text difficulty rather than the rules. More directly, if two Portuguese sentences with identical dependency structure but different propositional content yield different correct extractions under the rules S1–S5, the structured-propositions thesis is contradicted.
Extended reading notes
Core claim
The central discovery is that an Open IE corpus can be built with both theoretical grounding and practical reliability: the OIEC-PT corpus provides 473 manually validated extractions from 300 Portuguese sentences, with a gold-standard subset of 136 extractions from 100 sentences reaching near-perfect inter-annotator agreement (kappa 0.94). The annotation process rests on a distinction between relations (abstract types) and relationships (concrete instances), and on the claim that relationships expressed in text can be recognized because the structure of a sentence's proposition is a function of its syntactic structure. Seven structural rules and two contextual rules translate this principle into concrete annotation decisions, handling prepositional phrases, pronouns, agreement, and lexical expressions. The paper argues that the high classifier performance on the gold set compared with the silver set confirms that the finalized rules improve annotation consistency.
Load-bearing premise
The load-bearing premise is that the proposition a sentence expresses has the same structure as the sentence's syntax, so that annotation rules built on dependency trees preserve meaning; if this fails, the corpus may systematically miss or distort real relationships.
Editorial extensions
If this is right
- Portuguese Open IE systems can now be evaluated against a common benchmark with a gold-standard subset, instead of ad hoc or borrowed English datasets.
- The annotation rules provide a reusable guideline for future Portuguese annotation projects, and the iterative methodology (silver set to calibrate, gold set to finalize) can be replicated for other languages.
- Because the corpus is drawn from a parallel treebank, the same sentences can be annotated in other languages, enabling cross-lingual evaluation and multilingual Open IE research.
- Classifier validation suggests that annotation consistency directly affects learnability, meaning that investing in finalized guidelines improves downstream supervised Open IE models.
- Removing morphosyntactically erroneous sentences from the source treebank yields a cleaner evaluation resource, but also means the corpus excludes some authentic Portuguese text.
Reading between the lines
- The structured-propositions grounding could be extended to build similar Open IE corpora for other low-resource languages that have Universal Dependencies treebanks, not just Portuguese.
- The paper's binary-only focus leaves n-ary relations unannotated; a natural extension would be to generalize the rules to n-ary extraction, which the authors identify as future work.
- Reannotating the silver set with the finalized guidelines — a step the authors could not complete — would separate the effect of annotation consistency from text difficulty and is the most direct test of the paper's quality claim.
- The use of classifier AUC as a quality probe could be applied during annotation as a real-time consistency check, flagging sentences where annotators disagree for re-review.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents OIEC-PT, a manually annotated corpus for Open Information Extraction (Open IE) in Portuguese, built from 300 sentences of the Parallel Universal Dependencies (PUD) Portuguese treebank. The corpus is divided into a silver set (200 sentences) and a gold set (100 sentences), with a total of 473 extractions (136 in the gold set). The authors describe an iterative annotation process involving five annotators, report high inter-annotator agreement (Randolph's kappa reaching 0.94 on the gold set), and propose a set of structural (S1–S5) and contextual (R4) annotation rules grounded in a structured-propositions view of semantics. To validate the corpus, they train two classifiers (CrossOIE-CNN and TABOIE-Catboost) to distinguish valid from invalid extractions and report precision, recall, F1, accuracy, MCC, and AUC on three evaluation scenarios (PUD100, PUD200, PUD300). The central claim is that the gold set is higher quality than the silver set and that the methodology yields a reliable resource for Portuguese Open IE.
Significance. If the resource and its validation were sound, this would be a useful contribution to Portuguese Open IE, which currently lacks annotated corpora. The paper's strengths include the explicit theoretical grounding of the annotation rules, the iterative annotation methodology with high kappa, the use of a parallel treebank that enables cross-lingual extension, and the candid acknowledgment of limitations (e.g., the absence of silver-set reannotation). These features make the corpus potentially valuable for developing and evaluating Portuguese Open IE systems. However, the validation evidence as presented is flawed in a way that undermines the central quality claim: the MCC values in Table 3 are arithmetically inconsistent with the reported precision and recall, and the abstract's claim of evaluating 'state-of-the-art Open IE systems' is not supported by the experiments, which use two self-authored classifiers rather than Open IE extractors.
major comments (3)
- [Section 5.2, Table 3] The reported MCC values for TabOIEC are arithmetically impossible given the precision and recall in the same rows. For a binary confusion matrix, the Matthews correlation coefficient is bounded by sqrt(Precision × Recall). For PUD100, sqrt(0.603 × 0.282) ≈ 0.41, but the table reports MCC = 0.937; for PUD200 the bound is ≈0.34 versus 0.947; for PUD300 the bound is ≈0.52 versus 0.985. Since these MCC values are used to conclude that 'The results presented by TabOIEC validate our progress and the methodology' (Section 5.2), the validation evidence is internally inconsistent and must be corrected before the quality claim can be accepted.
- [Abstract and Section 5.1] The abstract states that the corpus is validated by 'evaluating the performance of state-of-the-art Open IE systems,' but the experiments use only two classifiers (CrossOIE-CNN and TABOIE-Catboost) from the authors' own previous work (Cabral et al. 2020a,b). These are validity classifiers, not Open IE extraction systems, and no comparison with actual Open IE extractors is provided. The claim of evaluating state-of-the-art Open IE systems is therefore not supported by the reported experiments.
- [Section 5.1] The validation is partially circular: both classifiers were designed and trained within the same research group and encode annotation assumptions similar to those used to build the corpus. High classifier performance may simply indicate that the labels are learnable under those assumptions, rather than providing independent evidence of corpus coherence. A more convincing check would use classifiers trained on independently produced annotations (e.g., from Glauber et al. 2018) or evaluate against alternative annotation guidelines; at minimum, this circularity should be explicitly discussed as a limitation.
minor comments (4)
- [Section 5.1] The description of the third evaluation scenario says 'train the classifier in our silver set and test on the golden set (PUD200),' but PUD200 refers to the silver set; this is confusing and should be reworded (e.g., 'train on the silver set (PUD200) and test on the golden set (PUD100)').
- [Table 3] The table header 'Metricsscoresforlanguagesclassifiers' contains typographical errors and should be corrected to 'Metrics scores for language classifiers' or similar.
- [Keywords] The keywords field contains template placeholders 'Proceedings, Template, BCS, ContentRepository, Indexing' and should be replaced with actual keywords relevant to the paper's content.
- [Section 4.3] The phrase 'The resulting OIEC-PT corpus is (anonymously) available at https://hyperalgesic-combs.000webhostapp.com/corpora.zip' will need a stable, persistent repository link (e.g., Zenodo or a university repository) for a journal publication.
Circularity Check
The corpus construction is not circular, but the classifier-based validation leans on the authors' own morpho-syntactic classifier family, making that validation partly self-referential; the impossible MCC values are a separate correctness error.
-
self citation load bearing
[Section 5.2 and Section 5.1, Table 3]
"Recent work, such as Cabral et al. [2020a,b], has proposed the use of classifiers to assess the quality of Open IE extractions to assist the creation of an Open IE dataset. We adopted them ... The results presented by TabOIEC validate our progress and the methodology used in the corpus construction."
The validating classifiers are from Cabral et al. [2020a,b], whose authors include three of the present paper's authors (Cabral, Souza, Claro). TABOIE-Catboost is described in the references as an 'Explainable open IE classifier with morpho-syntactic rules,' while the annotation guidelines in Section 4.2 are precisely structural/morpho-syntactic rules (S1-S5). The high MCC/AUC therefore shows that the gold labels are learnable by a classifier built on the same rule family that generated the labels, rather than demonstrating agreement with an independent standard. The kappa agreement and the manual iterative process provide separate, non-circular support for corpus quality, so the circularity is partial rather than total.
full rationale
The paper's derivation chain is: structured-propositions theory (Section 2) motivates structural annotation rules (Section 4.2), which are applied by human annotators to produce OIEC-PT (Section 4.4), and the corpus is then validated by classifier performance (Section 5). The theoretical grounding and the annotation rules are not circular: the rules are operationalizations of a philosophical thesis, and the corpus is not defined as 'whatever the classifiers score highly.' The inter-annotator kappa (0.94 in the gold step) is an independent, internally meaningful quality signal. The main circularity concern is the validation step: the classifiers adopted as quality probes are prior works by the same research group and encode morpho-syntactic notions of valid extractions that overlap with the annotation rules. Thus the sentence 'The results presented by TabOIEC validate our progress and the methodology' is partly self-referential: it confirms that the labels are consistent with the authors' own rule family, not that the rules match an external ground truth. I do not treat the arithmetically impossible MCC values in Table 3 as circularity; that is a correctness/consistency error that independently undermines the validation paragraph, but it is not a case of a prediction reducing to its inputs by construction. Overall, the central corpus-construction claim has independent support, so a moderate score of 4 is appropriate.
Assumptions & free parameters
assumptions (4)
- domain assumption Structured propositions thesis: the structure of the proposition expressed by a sentence is a function of the structure of that sentence.
- ad hoc to paper The annotation rules S1-S5 and R4 are sense-preserving transformations of sentence structure into propositional structure.
- domain assumption The PUD corpus is a suitable and representative base for Portuguese Open IE annotations.
- standard math High inter-annotator agreement (Randolph's kappa) is a valid indicator of corpus quality.
Cite this review
Pith. "Pith review of Challenges in Expanding Portuguese Resources: A View from Open Information Extraction." pith.science (2026). https://pith.science/paper/2GUQKT5F
@misc{pith2026250111851,
author = {Pith},
title = {Pith review of: Challenges in Expanding Portuguese Resources: A View from Open Information Extraction},
year = {2026},
howpublished = {\url{https://pith.science/paper/2GUQKT5F}},
note = {Machine review of arXiv:2501.11851}
}
read the original abstract
Open Information Extraction (Open IE) is the task of extracting structured information from textual documents, independent of domain. While traditional Open IE methods were based on unsupervised approaches, recently, with the emergence of robust annotated datasets, new data-based approaches have been developed to achieve better results. These innovations, however, have focused mainly on the English language due to a lack of datasets and the difficulty of constructing such resources for other languages. In this work, we present a high-quality manually annotated corpus for Open Information Extraction in the Portuguese language, based on a rigorous methodology grounded in established semantic theories. We discuss the challenges encountered in the annotation process, propose a set of structural and contextual annotation rules, and validate our corpus by evaluating the performance of state-of-the-art Open IE systems. Our resource addresses the lack of datasets for Open IE in Portuguese and can support the development and evaluation of new methods and systems in this area.
Figures
Reference graph
Works this paper leans on
-
[1]
we need to discuss the relationship
Bender,E.(2019).Englishisn’tgenericforlanguage,despite whatnlppapersmightleadyoutobelieve. In Symposium and Data Science and Statistics . [Online; accessed 15- may-2020]. Bender, E. M. (2009). Linguistically naïve != language in- dependent: Why NLP needs linguistic typology. InPro- ceedings of the EACL 2009 Workshop on the Interaction between Linguistics ...
arXiv 2019
-
[2005]
Available at: https://www.eric.ed.gov/contentdelivery/ servlet/ERICServlet?accno=ED490661. Soames,S.(2019). Propositionsascognitiveacts. Synthese, 196(4):1369–1383. Souza, F., Nogueira, R., and Lotufo, R. (2019). Portuguese named entity recognition using bert-crf.arXiv preprint arXiv:1909.10649. Stanovsky, G. and Dagan, I. (2016). Creating a large benchma...
arXiv 2019
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.