{"id":"a45dea67-4cae-4da4-b0fa-9e0030f34a5d","arxiv_id":"2501.11851","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The authors introduce OIEC-PT, a 300-sentence Portuguese Open IE corpus with 473 extractions and a formal annotation rule set.","lead":"This paper presents a new manually annotated Portuguese corpus for Open Information Extraction, built from 300 sentences of a multilingual treebank. It also proposes a set of annotation rules and evaluates the resource with machine learning classifiers.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported MCC values in Table 3 are arithmetically impossible given the precision and recall in the same rows, so the classifier-based validation of corpus quality cannot be trusted as written.","rationale":"I read the paper as an attempt to provide a high-quality manually annotated Portuguese Open IE corpus with a theoretically grounded annotation methodology. The strongest quantitative evidence for quality beyond the annotation process is the classifier validation in Section 5. The reader's weakest_assumption concerned the structured-propositions thesis and the sense-preserving nature of rules S1-S5; that is a legitimate philosophical risk, but it is not the most immediately actionable problem. The reader's rationale also noted the MCC inconsistency, but did not identify it as the weakest assumption. My pass shows the inconsistency is not merely suspicious but mathematically impossible: for any binary confusion matrix, MCC cannot exceed sqrt(Precision * Recall), so the three TabOIEC MCC values (0.937, 0.947, 0.985) cannot co-exist with the precision and recall in the same rows. This directly undermines the sentence in Section 5.2 that the MCC scores validate the methodology. I give credit where due: the paper reports high inter-annotator agreement, provides explicit annotation rules, and candidly acknowledges that the silver set would need reannotation to fully separate annotation consistency from text difficulty. However, the Table 3 issue is a concrete internal inconsistency that must be fixed before the validation-based quality claim can be accepted. Because the corpus itself may still be valuable and the fix could be as simple as replacing erroneous MCC values with correctly computed ones, the conditional verdict is appropriate rather than outright rejection.","tokens_in":9912,"tokens_out":4953,"duration_ms":51184,"concrete_test":"Ask the authors to release the per-class confusion matrices for CrossOIE and TabOIEC on PUD100, PUD200, and PUD300 at the threshold used for the precision/recall columns. Independently recompute MCC from those confusion matrices. If the TabOIEC rows cannot be reproduced (for example, PUD100 with Precision=0.603 and Recall=0.282 must have MCC <= 0.412), the validation section should be revised or the corpus quality claim weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quality claim rests partly on the classifier validation in Section 5. In Table 3, each TabOIEC row reports (Precision, Recall, MCC) = (0.603, 0.282, 0.937), (0.553, 0.212, 0.947), and (0.426, 0.630, 0.985). For any confusion matrix with TP=a, FP=b, FN=c, TN=d, MCC = (ad-bc) / sqrt((a+b)(a+c)(b+d)(c+d)) <= a / sqrt((a+b)(a+c)) = sqrt(Precision * Recall). The upper bounds for these rows are about 0.412, 0.342, and 0.518, respectively, so the reported MCC values are impossible. Since the authors use these MCC scores to state that the results validate their progress and methodology (Section 5.2), the empirical support for the high-quality corpus claim is internally inconsistent. The high kappa (0.94) and the explicit limitation about silver-set reannotation are acknowledged, but the validation table must be corrected before the quality claim is fully supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents OIEC-PT, a manually annotated corpus for Open Information Extraction (Open IE) in Portuguese, built from 300 sentences of the Parallel Universal Dependencies (PUD) Portuguese treebank. The corpus is divided into a silver set (200 sentences) and a gold set (100 sentences), with a total of 473 extractions (136 in the gold set). The authors describe an iterative annotation process involving five annotators, report high inter-annotator agreement (Randolph's kappa reaching 0.94 on the gold set), and propose a set of structural (S1–S5) and contextual (R4) annotation rules grounded in a structured-propositions view of semantics. To validate the corpus, they train two classifiers (CrossOIE-CNN and TABOIE-Catboost) to distinguish valid from invalid extractions and report precision, recall, F1, accuracy, MCC, and AUC on three evaluation scenarios (PUD100, PUD200, PUD300). The central claim is that the gold set is higher quality than the silver set and that the methodology yields a reliable resource for Portuguese Open IE.","tokens_in":10148,"tokens_out":5446,"duration_ms":52751,"significance":"If the resource and its validation were sound, this would be a useful contribution to Portuguese Open IE, which currently lacks annotated corpora. The paper's strengths include the explicit theoretical grounding of the annotation rules, the iterative annotation methodology with high kappa, the use of a parallel treebank that enables cross-lingual extension, and the candid acknowledgment of limitations (e.g., the absence of silver-set reannotation). These features make the corpus potentially valuable for developing and evaluating Portuguese Open IE systems. However, the validation evidence as presented is flawed in a way that undermines the central quality claim: the MCC values in Table 3 are arithmetically inconsistent with the reported precision and recall, and the abstract's claim of evaluating 'state-of-the-art Open IE systems' is not supported by the experiments, which use two self-authored classifiers rather than Open IE extractors.","major_comments":[{"comment":"The reported MCC values for TabOIEC are arithmetically impossible given the precision and recall in the same rows. For a binary confusion matrix, the Matthews correlation coefficient is bounded by sqrt(Precision × Recall). For PUD100, sqrt(0.603 × 0.282) ≈ 0.41, but the table reports MCC = 0.937; for PUD200 the bound is ≈0.34 versus 0.947; for PUD300 the bound is ≈0.52 versus 0.985. Since these MCC values are used to conclude that 'The results presented by TabOIEC validate our progress and the methodology' (Section 5.2), the validation evidence is internally inconsistent and must be corrected before the quality claim can be accepted.","section":"Section 5.2, Table 3"},{"comment":"The abstract states that the corpus is validated by 'evaluating the performance of state-of-the-art Open IE systems,' but the experiments use only two classifiers (CrossOIE-CNN and TABOIE-Catboost) from the authors' own previous work (Cabral et al. 2020a,b). These are validity classifiers, not Open IE extraction systems, and no comparison with actual Open IE extractors is provided. The claim of evaluating state-of-the-art Open IE systems is therefore not supported by the reported experiments.","section":"Abstract and Section 5.1"},{"comment":"The validation is partially circular: both classifiers were designed and trained within the same research group and encode annotation assumptions similar to those used to build the corpus. High classifier performance may simply indicate that the labels are learnable under those assumptions, rather than providing independent evidence of corpus coherence. A more convincing check would use classifiers trained on independently produced annotations (e.g., from Glauber et al. 2018) or evaluate against alternative annotation guidelines; at minimum, this circularity should be explicitly discussed as a limitation.","section":"Section 5.1"}],"minor_comments":[{"comment":"The description of the third evaluation scenario says 'train the classifier in our silver set and test on the golden set (PUD200),' but PUD200 refers to the silver set; this is confusing and should be reworded (e.g., 'train on the silver set (PUD200) and test on the golden set (PUD100)').","section":"Section 5.1"},{"comment":"The table header 'Metricsscoresforlanguagesclassifiers' contains typographical errors and should be corrected to 'Metrics scores for language classifiers' or similar.","section":"Table 3"},{"comment":"The keywords field contains template placeholders 'Proceedings, Template, BCS, ContentRepository, Indexing' and should be replaced with actual keywords relevant to the paper's content.","section":"Keywords"},{"comment":"The phrase 'The resulting OIEC-PT corpus is (anonymously) available at https://hyperalgesic-combs.000webhostapp.com/corpora.zip' will need a stable, persistent repository link (e.g., Zenodo or a university repository) for a journal publication.","section":"Section 4.3"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of the Journal of the Brazilian Computer Society and addresses a real gap in Portuguese NLP resources. The corpus itself may be a useful contribution, and the annotation methodology is thoughtfully presented. However, the validation section as written contains an impossible result that must be corrected, and the abstract overstates what the experiments show. These issues are fixable within the manuscript's scope, so I recommend major revision rather than rejection. Please verify the MCC computations and re-run or re-report the experiments before considering acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this one for the corpus, not for the validation. OIEC-PT is real: 300 PUD sentences, 473 extractions, a 100-sentence gold set with kappa 0.94, and a fairly careful set of structural and contextual rules grounded in structured propositions. It fills a real gap — Portuguese Open IE had essentially one 25-sentence corpus with no formal relation definition. The annotation guide is the most valuable part; S1-S5 and R4 show the kind of explicit decisions that make a low-resource corpus reusable.\n\nThe stress-test note is correct and it matters. In Table 3, the TabOIEC rows report MCC of 0.937, 0.947, and 0.985 alongside precision/recall pairs whose theoretical MCC upper bound is around 0.41, 0.34, and 0.52. Those numbers cannot come from the same confusion matrix. The authors use those MCC scores to claim the classifier validates their methodology, so the empirical support for the quality claim is internally inconsistent as printed. This is not a rounding issue; it is an order-of-magnitude mismatch. The precision-recall curves and AUC values (0.66-0.84) are more believable and tell a more modest story. The authors should recompute or re-report the table, and the abstract should not say 'state-of-the-art Open IE systems' when they only ran their own two classifiers from Cabral et al., which share annotation assumptions with the corpus. That is a circularity worth noting, though not fatal if reframed as 'we probe learnability with classifiers we built.'\n\nThe removed-sentences point is also underreported: they say they filtered erroneous PUD sentences but never say how many. For an evaluation corpus, that count matters.\n\nSo: the central resource claim likely holds — the corpus is what it says it is, and the kappa plus explicit rules give it credibility. The validation section needs a corrective pass. This paper deserves a serious referee; the fixes are straightforward and the resource is useful to anyone working on Portuguese or multilingual Open IE. I would engage with it after the table is corrected.","headline":"A genuinely useful Portuguese Open IE corpus with an annotation scheme worth reading, but the validation table has impossible MCC values and the abstract overclaims what was validated.","tokens_in":10658,"tokens_out":1637,"would_cite":true,"duration_ms":16511,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A theory-grounded annotation methodology produces a high-quality Portuguese Open IE corpus with near-perfect annotator agreement.","keywords":["Open Information Extraction","Portuguese corpus","annotation methodology","structured propositions","semantic relations","Universal Dependencies","inter-annotator agreement","low-resource NLP"],"falsifier":"Annotate the silver set again using the finalized guidelines and measure inter-annotator agreement and classifier AUC; if the reannotated silver set does not approach the gold set's kappa of 0.94, the improvement in agreement may come from text difficulty rather than the rules. More directly, if two Portuguese sentences with identical dependency structure but different propositional content yield different correct extractions under the rules S1–S5, the structured-propositions thesis is contradicted.","tokens_in":9720,"feed_emoji":"🇧🇷","tokens_out":6168,"duration_ms":51027,"temperature":0.7,"pith_summary":"This paper claims that a theoretically grounded annotation methodology can produce a reliable Open Information Extraction corpus for Portuguese, a language that lacks such resources. The authors built OIEC-PT from 300 sentences of a parallel Universal Dependencies treebank, producing 473 extractions, with a gold set of 100 sentences and 136 extractions annotated at a Randolph's kappa of 0.94. The annotation rules are justified by the structured-propositions thesis, which holds that a sentence's propositional structure mirrors its syntactic structure. The authors validate the corpus by showing that classifiers trained on the gold set achieve higher AUC than on the silver set, evidence of annotation consistency. If the resource is sound, it gives Portuguese Open IE an evaluation benchmark and a path toward multilingual standards.","feed_headline":"Portuguese Open IE corpus hits 0.94 annotator agreement","feed_subtitle":"A 300-sentence resource with 473 extractions gives Portuguese NLP its own Open IE evaluation benchmark.","key_machinery":"The machinery is the structured-propositions thesis — the claim that the structure of the proposition expressed by a sentence is a function of that sentence's structure — together with a set of seven structural and two contextual annotation rules (S1–S5, S4.1–S4.2, R4) that convert dependency trees into valid extractions. The thesis licenses the use of syntactic dependency structure as a guide to propositional content, so a relationship is recognized when its structure is embedded in, or obtainable by sense-preserving transformation from, the sentence's structure. The rules determine what counts as a valid triple (arg1, rel, arg2): which phrases may serve as arguments, when prepositions attach to the relation descriptor, when adverbs merge with verbs, and when an extraction is informative rather than merely well-formed.","core_discovery":"The central discovery is that an Open IE corpus can be built with both theoretical grounding and practical reliability: the OIEC-PT corpus provides 473 manually validated extractions from 300 Portuguese sentences, with a gold-standard subset of 136 extractions from 100 sentences reaching near-perfect inter-annotator agreement (kappa 0.94). The annotation process rests on a distinction between relations (abstract types) and relationships (concrete instances), and on the claim that relationships expressed in text can be recognized because the structure of a sentence's proposition is a function of its syntactic structure. Seven structural rules and two contextual rules translate this principle into concrete annotation decisions, handling prepositional phrases, pronouns, agreement, and lexical expressions. The paper argues that the high classifier performance on the gold set compared with the silver set confirms that the finalized rules improve annotation consistency.","pith_inferences":["The structured-propositions grounding could be extended to build similar Open IE corpora for other low-resource languages that have Universal Dependencies treebanks, not just Portuguese.","The paper's binary-only focus leaves n-ary relations unannotated; a natural extension would be to generalize the rules to n-ary extraction, which the authors identify as future work.","Reannotating the silver set with the finalized guidelines — a step the authors could not complete — would separate the effect of annotation consistency from text difficulty and is the most direct test of the paper's quality claim.","The use of classifier AUC as a quality probe could be applied during annotation as a real-time consistency check, flagging sentences where annotators disagree for re-review."],"forward_implications":["Portuguese Open IE systems can now be evaluated against a common benchmark with a gold-standard subset, instead of ad hoc or borrowed English datasets.","The annotation rules provide a reusable guideline for future Portuguese annotation projects, and the iterative methodology (silver set to calibrate, gold set to finalize) can be replicated for other languages.","Because the corpus is drawn from a parallel treebank, the same sentences can be annotated in other languages, enabling cross-lingual evaluation and multilingual Open IE research.","Classifier validation suggests that annotation consistency directly affects learnability, meaning that investing in finalized guidelines improves downstream supervised Open IE models.","Removing morphosyntactically erroneous sentences from the source treebank yields a cleaner evaluation resource, but also means the corpus excludes some authentic Portuguese text."],"supporting_citations":[{"why":"Defines the Open Information Extraction task that the corpus serves.","marker":"Etzioni et al. [2008]"},{"why":"Foundational source for structured propositions, the theoretical basis of the annotation rules.","marker":"King [1995]"},{"why":"Supplies the distinction between relations and relationships as truth-makers, used to define what annotators extract.","marker":"Guarino and Guizzardi [2015]"},{"why":"Source corpus (PUD) from which the 300 sentences are drawn, enabling parallel extension.","marker":"Nivre et al. [2020]"},{"why":"Provides the free-marginal kappa statistic used to measure inter-annotator agreement.","marker":"Randolph [2005]"},{"why":"Prior English Open IE benchmark whose methodology the paper critiques and contrasts with its own.","marker":"Stanovsky and Dagan [2016]"},{"why":"WiRe57 English dataset; comparison for annotation methodology.","marker":"Lechelle et al. [2019]"},{"why":"Earlier Portuguese Open IE corpus that this work extends and improves upon.","marker":"Glauber et al. [2018]"},{"why":"CrossOIE-CNN classifier used in validation experiments.","marker":"Cabral et al. [2020a]"},{"why":"TABOIE-Catboost classifier used in validation experiments.","marker":"Cabral et al. [2020b]"}],"fun_headline_variants":["Portuguese Open IE corpus reaches 0.94 inter-annotator agreement","OIEC-PT: 473 extractions, 300 sentences, kappa 0.94","High-agreement Portuguese Open IE corpus enables evaluation","New Portuguese gold set for Open IE: 0.94 annotation agreement"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the proposition a sentence expresses has the same structure as the sentence's syntax, so that annotation rules built on dependency trees preserve meaning; if this fails, the corpus may systematically miss or distort real relationships.","fun_headline_variants_meta":{"raw":{"variants":["Portuguese Open IE corpus reaches 0.94 inter-annotator agreement","OIEC-PT: 473 extractions, 300 sentences, kappa 0.94","High-agreement Portuguese Open IE corpus enables evaluation","New Portuguese gold set for Open IE: 0.94 annotation agreement"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000803,"raw_usage":{"total_tokens":3487,"prompt_tokens":863,"completion_tokens":2624,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":479,"completion_tokens_details":{"reasoning_tokens":2544}},"tokens_in":479,"tokens_out":2624,"duration_ms":19233,"temperature":1.0,"reasoning_tokens":2544,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:47:19.748457+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Annotate the silver set again using the finalized guidelines and measure inter-annotator agreement and classifier AUC; if the reannotated silver set does not approach the gold set's kappa of 0.94, the improvement in agreement may come from text difficulty rather than the rules. More directly, if two Portuguese sentences with identical dependency structure but different propositional content yield different correct extractions under the rules S1–S5, the structured-propositions thesis is contradicted.","supporting_citations":[],"review_version":1}