{"id":"9d6e1139-3a09-42a3-876f-946ddb971966","arxiv_id":"2505.00479","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A dependency-parsing method and a LEGAL-BERT-based classifier both reach roughly 80-84% accuracy for identifying regulatory statements in EU law on a new 7,200-sentence dataset, with a Krippendorff's alpha of 0.58 between them.","lead":"This paper describes two automated ways to identify regulatory statements in EU legislation: a rule-based dependency parser and a machine-learning classifier, reporting accuracies of about 80% and 84%. Reliable large-scale identification could help scholars and practitioners measure the density and strictness of EU law across time and policy areas.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 0.80/0.84 accuracies and the 0.58 model agreement rest on a single-annotator gold standard with no measured agreement; if the two annotators label differently, the headline numbers and complementarity claim are unsupported.","rationale":"The main empirical result is not just the raw accuracy values; it is the combination of 'both methods are good' and 'they disagree' as evidence that a hybrid could improve. The single-annotator design is the point where this inference is least secure. A low inter-annotator agreement would not necessarily invalidate the models if the annotators are expert and consistent, but currently there is no evidence of consistency. The paper openly discusses OCR and XAI limitations in Section 5, but it does not acknowledge the absence of label reliability evidence for the main corpus, so this is a genuine gap rather than an already-flagged caveat. I also considered the evaluation-window limitation in Section 5, where the evaluation sample was drawn primarily from 2000-2023 despite the stated 1970-2022 corpus; that is important for external validity but is acknowledged by the authors, whereas the annotation issue affects the internal validity of the headline accuracy and the hybrid motivation. The public code and data are a real strength and make the proposed inter-annotator check feasible and inexpensive. The reader's conditional verdict remains appropriate: the concern is substantive but addressable, and the paper's core observation could survive if the check shows high annotator agreement.","tokens_in":13880,"tokens_out":5809,"duration_ms":65524,"concrete_test":"Take a random sample of 300 sentences from the existing 7,200 (or from the test set), have both annotators label them independently without conferring, and compute Krippendorff's alpha between the two label sets. Then have the two annotators adjudicate disagreements, or use a third expert, to produce a consensus gold label for each sampled sentence. Recompute both classifiers' accuracy and the model-pair alpha on this adjudicated subset. If the annotator alpha is below 0.667, or if either accuracy shifts by more than about 0.05 compared with the single-annotator labels, the reported 0.80/0.84 numbers and the complementarity conclusion are not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that both classifiers reach roughly 0.80-0.84 accuracy and disagree enough (Krippendorff alpha = 0.58) that a hybrid could improve. This conclusion depends on the quality of the 7,200-sentence gold standard described in Section 3.4. As stated there, the two annotators split the batch and each labeled separate halves, with only 'exceptionally few' cases requiring conferral. No overlap is reported, so no inter-annotator agreement can be computed. If the two scholars apply the IGT definition differently, the two halves carry different label semantics. Since the train/test split is random over the pooled halves, each model is trained and evaluated against a mixture of two potentially inconsistent standards. The reported accuracies then conflate model error with annotator disagreement, and the low between-model agreement of 0.58 may reflect annotation inconsistency rather than complementary modeling strengths. Section 5 openly discusses OCR and XAI limitations but does not flag the absence of label reliability evidence. Thus both the absolute accuracy values and the argument that low model agreement indicates a promising hybrid are not secure.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses the task of automatically identifying regulatory statements in EU legislation, framed as binary sentence classification. It defines a regulatory statement using the Institutional Grammar Tool (IGT), requiring at least Attribute-Deontic-Aim components with the attribute expressed as an agent noun. The authors construct a corpus of about 7,200 human-labeled sentences sampled from roughly 120,000 EU legislative documents, stratified by adoption year and policy area. They compare two approaches: a hand-crafted dependency-parsing rule set built on SpaCy and ConceptNet, and a transfer-learning classifier using LEGAL-BERT features fed to XGBoost. They report accuracies of 0.80 and 0.84, respectively, a between-model Krippendorff alpha of 0.58, and provide a LIME-based XAI analysis. The paper argues that the two methods have complementary strengths and proposes a fusion strategy, while also discussing limitations such as OCR artifacts and attribute identification difficulties.","tokens_in":14077,"tokens_out":5033,"duration_ms":52852,"significance":"If the reported results are robust, the paper would make a useful contribution to computational legal-text analysis: it provides a concrete, IGT-based definition of regulatory statements, a sizable publicly available labeled corpus, and a comparison of a grammar-based method with a transformer-based method on the same test set. The reproducibility effort is a clear strength: code repositories, raw data, and processed data are made available. The hand-crafted dependency rules are tested on a separate holdout set rather than tuned on it, which is methodologically sound. The XAI analysis, though preliminary, is a useful step toward interpreting the transformer model. However, the central empirical claims are currently not fully supported because the ground truth was produced without measured inter-annotator reliability and the test set covers only a limited time period relative to the claimed scope.","major_comments":[{"comment":"The main empirical claim—that both classifiers reach 0.80/0.84 accuracy and that their 0.58 agreement supports a hybrid—rests on a gold standard whose reliability is not established. Section 3.4 states that the two annotators split the 7,200-sentence batch into two equal halves and labeled them separately, with no overlapping annotation and no reported inter-annotator agreement. Because the train/test split is random over the pooled halves, the test labels are a mixture of two potentially different interpretations of the IGT definition, and the reported accuracy values and the between-model Krippendorff alpha of 0.58 conflate model error with annotator disagreement. The authors should either add an overlapping annotation sample and report agreement, or explicitly reframe all headline numbers as conditional on a single-annotator label convention. This issue is load-bearing for the abstract and for Section 4.1.","section":"Section 3.4 and Section 4.1"},{"comment":"The claim that the classifiers are applicable to the full 1970–2022 corpus is not supported by the evaluation data. Section 5 states that, because of OCR splicing errors, sentences for evaluation were sampled primarily from 2000–2023; Section 3.2, however, states that 2023 and 2024 were excluded and that the corpus covers 1970–2022. Taken together, the test set contains essentially no pre-2000 sentences, so the reported test-set accuracy cannot validate generalization to pre-1990 documents. The statement in Section 5 that manual inspection showed no significant linguistic differences is an anecdotal justification and should be either backed by a quantitative comparison or replaced by a narrower claim about post-2000 legislation.","section":"Section 5 (OCR issue) and Section 3.2"},{"comment":"Table 1 reports point estimates with no confidence intervals or significance tests. With roughly 1,450 test sentences, the difference between 0.80 and 0.84 may be within sampling variability, and the claim that both approaches 'performed similarly well' is not backed by any statistical comparison. Similarly, the 0.58 between-model alpha is interpreted as evidence of complementary strengths, but a low agreement metric alone is not a direct measure of error complementarity; a direct analysis of the sets of misclassified sentences, or an oracle-combination experiment, would be needed to substantiate the complementarity claim.","section":"Section 4.1, Table 1"}],"minor_comments":[{"comment":"The citation for BERT appears to be wrong: 'Bidirectional Encoder Representations from Transformers (BERT) [8]' should cite Devlin et al. [7], not Dragoni et al. [8].","section":"Section 2"},{"comment":"The text says 'approximately 7,200 sentences,' but the described equal allocation implies 1,034 strata × 7 sentences = 7,238 sentences; please state the exact number and clarify the discrepancy.","section":"Section 3.4"},{"comment":"The abstract cites approximately 180,000 published legal acts between 1952 and 2023, while Section 3.2 reports roughly 120,000 documents between 1970 and 2022; please reconcile these numbers or explain the different scopes.","section":"Abstract and Section 3.2"},{"comment":"The proposed fusion strategy is described as benefiting from both methods' strengths, and Section 7 states it 'could significantly improve performance,' but the strategy is not evaluated; please either add a small experiment or soften the wording.","section":"Section 4.3"},{"comment":"The table reports Krippendorff alpha values for agreement with ground truth, while the text separately reports alpha = 0.58 for inter-model agreement; the table caption should clarify this distinction to avoid confusion.","section":"Table 1 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the journal's scope and the reproducibility effort is commendable. The main risk is that the headline accuracy numbers and the complementarity argument may not be robust to annotation variation; if the authors can provide a small overlap sample with measured agreement, or significantly temper the generalization claims, the paper would be publishable. I would not reject at this stage, as the issue is addressable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things up front. First, the paper ships a new 7,200-sentence corpus for binary regulatory-statement classification, with code and data, which is real value for legal-NLP. Second, the headline 0.80/0.84 accuracies rest on a gold standard produced by two annotators splitting the batch in half with no overlap, so no inter-annotator agreement is reported. That is the soft spot that matters most: if the two annotators apply the IGT definition differently, the two halves carry different label semantics, and the train/test split mixes them. The reported accuracies then conflate model error with annotator disagreement, and the inter-model alpha of 0.58 may partly reflect annotation inconsistency rather than only complementary modeling strengths. The stress-test note lands.\n\nWhat the paper does well: the IGT-based definition is clear and domain-grounded, and the task definition is new for EU-wide legislation. The comparison of dependency parsing versus transformer feature extraction is more detailed than prior work, including XAI analysis, and the authors are honest about other limitations (OCR, attribute identification, generalizability). They explicitly flag the post-1990 evaluation window and its effect on generalization, which is good practice.\n\nOther soft spots: no confidence intervals on the accuracy estimates, and the hybrid claim is plausible but untested. The selective evaluation window is acknowledged, and the claim that results generalize back to 1970 is based on human inspection, not quantitative evidence. These are addressable in revision.\n\nWho this is for: anyone working on text classification of legal corpora, especially EU law or institutional grammar applications. The corpus and comparison are useful even if absolute accuracy values require caution. It deserves a serious referee because the contribution is concrete and reproducible, and the main flaw is fixable with overlap annotation and confidence intervals. Send to peer review, conditional on revision.","headline":"A workmanlike empirical comparison of two classifiers for regulatory sentences in EU law, with a genuinely new corpus and open code/data, but the missing inter-annotator agreement on the gold standard makes the headline accuracy numbers less secure than they look.","tokens_in":624,"tokens_out":1696,"would_cite":true,"duration_ms":30650,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Two automated methods — one grammar-based, one transformer-based — identify regulatory statements in EU legislation with 80–84% accuracy, and their low agreement points toward a better hybrid.","keywords":["regulatory statements","EU legislation","institutional grammar tool","text classification","dependency parsing","transfer learning","legal text classification","explainable AI"],"falsifier":"Have both annotators independently label the same random sample of a few hundred sentences from the corpus and compute Krippendorff's alpha between them, and also implement the proposed grammar-with-transformer-fallback hybrid on the released test set; low annotator agreement would undermine the accuracy numbers, and a hybrid that fails to beat 0.84 would undermine the complementary-strengths claim.","tokens_in":13680,"feed_emoji":"⚖️","tokens_out":9965,"duration_ms":92026,"temperature":0.7,"pith_summary":"The paper attacks a bottleneck for measuring how dense and strict EU law has become: deciding which sentences actually impose a legal rule on a specific actor, as opposed to conditions, amendments, or boilerplate about entry into force. It fixes a definition of a regulatory statement using the Institutional Grammar Tool's ADICO scheme, requiring an attribute expressed as an agent noun, a deontic auxiliary such as 'must' or 'shall', and an aim. Two classifiers are built to that definition, one from hand-written dependency-parsing rules and one from a transformer pretrained on legal text with a tree-boosted classifier on top. On a new 7,200-sentence corpus stratified across policy areas and years, both reach accuracies above 0.80 (0.80 and 0.84). Since the two models agree only moderately (Krippendorff's alpha 0.58), the paper argues that a hybrid using the grammar rules to certify attributes and the transformer to catch implicit or unfamiliar actors should do better.","feed_headline":"Two automated methods spot EU regulatory sentences at 80–84%","feed_subtitle":"A 7,200-sentence test shows the methods disagree often; fusing them may push accuracy higher.","key_machinery":"The load-bearing mechanism is the ADICO decomposition from the Institutional Grammar Tool, cut down to ADI: every regulatory sentence must name the actor (Attribute) as an agent noun, carry a deontic auxiliary ('must' or 'shall'), and state an action (Aim). The dependency approach operationalizes ADI as a small set of parse-tree rules — a lexical verb whose auxiliary is deontic, with a path through the tree to an agent noun checked against a knowledge graph. The transformer approach operationalizes the same definition statistically by training a shallow classifier on sentence features from a legal-text pretrained transformer. The two mechanisms are deliberately orthogonal, and their disagreement is treated as reusable signal rather than noise.","core_discovery":"The paper's central claim is that the notion of a regulatory statement can be pinned down by an IGT-based definition — a sentence must explicitly carry the Attribute, Deontic, and Aim components, with the attribute realized as an agent noun — and that this definition can be learned by two very different machines. The dependency approach encodes the definition directly as parse-tree conditions: a lexical verb governed by a 'must' or 'shall' auxiliary, plus a dependency path from that verb to an agent noun, where agenthood is decided with the help of a knowledge graph. The transfer-learning approach instead extracts sentence features from a legal-domain transformer and classifies them with gradient-boosted trees. On a stratified corpus of 7,200 sentences drawn from roughly 120,000 EU legal acts across 1971-2022, both methods surpass 0.80 accuracy, the transformer reaching 0.84. The grammar rules are more precise on the regulatory class while the transformer has higher regulatory recall, and the model-model agreement is only 0.58; those two facts are the paper's evidence that the methods capture different linguistic signals.","pith_inferences":["If the hybrid direction is right, a confidence-weighted ensemble over the two classifiers is a natural variant; since both models and the test set are released, this can be checked directly without new data collection.","The single-annotator ground truth is the main fragility; an immediate validation experiment would have both scholars independently label a shared random subsample, since the paper never reports annotator-annotator agreement.","The choice to count only core ADI sentences means that amendments, preconditions, and or-else sanctions are excluded from the density metric; in heavily amended policy areas, regulatory strictness may therefore be understated, and a complementary metric counting those clauses would test how much this matters.","For the pre-1990 documents, OCR-induced sentence splicing is acknowledged as a corruption source; applying the trained models to the earliest decades of the corpus will require a layout-aware sentence segmenter, and the authors' expectation that results generalize to those documents is a testable assumption."],"forward_implications":["Regulatory density and strictness can now be measured automatically over the full corpus of approximately 180,000 EU legal acts, broken down by year and policy area.","A sentence flagged as regulatory by the dependency rules is highly trustworthy (high precision), while the transformer recovers regulatory sentences whose actor is implicit or whose name the grammar rules do not know.","Because the error sets are largely disjoint, a concrete hybrid — let the grammar rules identify the attribute, and fall back to the transformer when they fail — is a direct next step that the paper argues should improve accuracy.","The explanation analysis indicates that the transformer keys on action verbs for regulatory sentences and on amendment and context lexicon (such as 'replaced by the following') for non-regulatory ones, giving an interpretable handle on its decisions.","The code, data, and models are released publicly, so the comparison can be reproduced, audited, and extended by other groups."],"supporting_citations":[{"why":"Supplies the ADICO/IGT grammar that the paper's definition of a regulatory statement is built on.","marker":"[6]"},{"why":"Book-length source of the Institutional Grammar Tool vocabulary used to justify the attribute-deontic-aim requirement.","marker":"[25]"},{"why":"Provides the legal-domain transformer whose extracted features drive the transfer-learning approach.","marker":"[4]"},{"why":"Provides the tree-boosted classifier trained on the transformer features.","marker":"[5]"},{"why":"Provides the knowledge graph the dependency approach uses to decide whether a noun is an agent noun.","marker":"[34]"},{"why":"Provides the neural dependency parser that generates the parse trees the grammar rules operate on.","marker":"[12]"},{"why":"Baseline pretrained transformer used to compare feature extractors.","marker":"[7]"},{"why":"Defines the Krippendorff alpha agreement measure used to compare models to ground truth and to each other.","marker":"[16]"},{"why":"Supplies the local explanation method used in the XAI analysis of the transformer's predictions.","marker":"[31]"},{"why":"Closest prior transfer-learning work on deontic rule classification, which this paper extends from a single regulation to general EU legislation.","marker":"[18]"}],"fun_headline_variants":["Two AI methods spot EU regulatory sentences at 80–84%","Complementary algorithms identify EU law regulations with 84% hit rate","Grammar rules and transformers both read EU laws, suggest fusion","EU regulatory text: two AI approaches agree enough to combine"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ground-truth labels for the 7,200 sentences were each produced by one of two annotators who split the batch, and no inter-annotator agreement is reported; if the two scholars interpret the institutional-grammar definition differently, the reported 0.80 and 0.84 accuracy figures are not reliable as absolute numbers.","fun_headline_variants_meta":{"raw":{"variants":["Two AI methods spot EU regulatory sentences at 80–84%","Complementary algorithms identify EU law regulations with 84% hit rate","Grammar rules and transformers both read EU laws, suggest fusion","EU regulatory text: two AI approaches agree enough to combine"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000608,"raw_usage":{"total_tokens":2819,"prompt_tokens":917,"completion_tokens":1902,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":533,"completion_tokens_details":{"reasoning_tokens":1831}},"tokens_in":533,"tokens_out":1902,"duration_ms":16609,"temperature":1.0,"reasoning_tokens":1831,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:40:44.623439+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have both annotators independently label the same random sample of a few hundred sentences from the corpus and compute Krippendorff's alpha between them, and also implement the proposed grammar-with-transformer-fallback hybrid on the released test set; low annotator agreement would undermine the accuracy numbers, and a hybrid that fails to beat 0.84 would undermine the complementary-strengths claim.","supporting_citations":[{"cited_title":"following","cited_arxiv_id":null,"evidence_quote":"Book-length source of the Institutional Grammar Tool vocabulary used to justify the attribute-deontic-aim requirement."},{"cited_title":"It is customary to require α ≥ .800 (for statistically significant reliability). Where tentative conclusions are still acceptable, α ≥ .667 is the lowest conceivable limit","cited_arxiv_id":null,"evidence_quote":"Provides the legal-domain transformer whose extracted features drive the transfer-learning approach."},{"cited_title":"Citizens must separate their recyclables","cited_arxiv_id":null,"evidence_quote":"Baseline pretrained transformer used to compare feature extractors."},{"cited_title":"In 1st Workshop on MIning and REasoning with Legal texts (MIREL 2016)","cited_arxiv_id":null,"evidence_quote":"Defines the Krippendorff alpha agreement measure used to compare models to ground truth and to each other."}],"review_version":1}