{"id":"e52a683a-e955-4e2b-a95e-a093ad24e2bc","arxiv_id":"2412.02427","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"On a German legal norm corpus, a fine-tuned XLM-RoBERTa outperforms a rule-based system and a prompted LLM (LeoLM) in macro F1 for ten annotation classes.","lead":"The paper compared three ways to automatically find text spans in German legal norms: hand-written rules, fine-tuned neural models, and a large language model. The fine-tuned transformer (XLM-R) performed best, matching patterns seen in other NER domains, but the claim that class heterogeneity is the cause is not directly tested.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LLM prompt mismatch and missing uncertainty bounds leave the headline ranking plausible but not fully settled; the conditional verdict is appropriate.","rationale":"The reader's weakest assumption correctly identifies the representativeness of the compared implementations. The paper itself concedes that the LLM prompt examples come from a smaller dataset with a different annotation guideline, which directly undermines the claim that the deep generative family is well represented. The rule-based system's lack of syntactic information is also acknowledged. These limitations do not invalidate the reported numbers for these specific systems, but they weaken the broader generalization in the abstract. The lack of error bars compounds the issue, though the large gap between XLM-R and LeoLM(opt) makes it unlikely that simple prompt noise would flip the ranking. A targeted re-run with dev-set prompts would settle whether the prompt mismatch is material. Since the reader's conditional verdict already reflects this uncertainty, no verdict change is needed.","tokens_in":13536,"tokens_out":3572,"duration_ms":37917,"concrete_test":"Re-run the deep generative pipeline with few-shot examples drawn from the GerPS-NER dev set under the same annotation guideline, keeping all other settings identical, and compute macro F1 on the same test split. If the re-run remains below XLM-R by at least 0.10 macro F1 (e.g., below 0.55), the ranking is robust; if it approaches or exceeds 0.6455, the original comparison is confounded. Report 5 runs with varied example orderings to bound variance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central ranking claim depends on each implementation being representative of its method family. The deep generative approach was prompted with examples from a smaller, differently annotated dataset (Section 4.4; Section 8), so its macro F1 of 0.4488 may understate what a well-tuned LLM could achieve on GerPS-NER. The rule-based system was intentionally limited to flat token/phrase patterns without dependency syntax (Section 4.2), also potentially understating its family. The authors acknowledge this in Limitations: prompt examples 'may not be representative' and the annotation guideline differs. No error bars or repeated runs are reported, so the 0.197 macro-F1 gap between XLM-R and LeoLM(opt) cannot be separated from implementation or prompt-choice variance. The causal claim about class heterogeneity (Section 6.4) is speculative, but the load-bearing issue for the headline claim is the representativeness of the compared implementations.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"GerPS-Compare reports a three-way comparison of NER approaches on GerPS-NER, a German legal-administration corpus with ten classes. The authors evaluate a SpaCy EntityRuler-based rule system, two deep discriminative models (BiLSTM-CRF and XLM-R with FLERT), and LeoLM, a generative LLM, using a prompting scheme from prior work. They report token-level F1 and a span-level Jaccard metric, with macro F1 as the main summary. The main empirical finding is that XLM-R obtains the highest macro F1 (0.6455) and that deep discriminative models generally outperform the rule-based and generative approaches, which perform at similar levels. The paper also interprets this ranking as caused by the semantic and syntactic heterogeneity of the GerPS-NER classes.","tokens_in":13708,"tokens_out":5907,"duration_ms":57912,"significance":"This is a useful, clearly written empirical benchmark for a realistic German legal-administrative NER task. The corpus is close to a real application, the paper reports both token F1 and a span-level Jaccard measure, and the code is promised on Zenodo, which supports reproducibility. The internal consistency between the F1 and Jaccard rankings is a strength: the ordering of approaches is largely monotonic across the two metric families, and the paper is honest about implementation limitations. If the ranking is robust to the implementation concerns raised below, the paper would be a valuable data point for practitioners choosing among rule-based, fine-tuned encoder, and LLM-prompting approaches. The main limitations are that each method family is represented by one particular implementation with acknowledged restrictions, and there is no uncertainty quantification.","major_comments":[{"comment":"The deep generative implementation is not a representative instance of the LLM method family: the prompts use example annotations from the smaller, differently annotated dataset of Bachinger et al. (2024), and the authors acknowledge in Section 8 that these examples 'may not be representative of the classes in the overall corpus.' Because the headline claim is that deep discriminative models outperform the deep generative approach, this prompt-example mismatch directly weakens the comparison; a tuned prompt set could shift the LeoLM scores. The paper should either re-run the generative approach with prompt examples drawn from the GerPS-NER dev set or explicitly restrict the conclusion to the specific prompt configuration tested.","section":"§4.4, §8"},{"comment":"The evaluation of the generative approach is dominated by the choice of consolidation scheme: Table 1 shows macro F1 of 0.4488 under optimistic consolidation versus 0.1067 under pessimistic consolidation. Optimistic consolidation selects, among multiple model predictions, the one that matches the gold standard; this is not a realistic deployment setting and can only overestimate performance. The paper should report the pessimistic result as the primary LLM score, or at least justify why the optimistic oracle is informative for the method comparison.","section":"§4.4, Table 1"},{"comment":"No uncertainty quantification is provided: each model is evaluated once, with no repeated runs, standard deviations, or significance tests. The gaps that drive the ranking—XLM-R 0.6455 vs BiLSTM-CRF 0.6058, rule-based 0.5082 vs LeoLM opt 0.4488—may be within run-to-run variation, particularly for deep models whose training is stochastic. The central ranking claim would be more defensible if the paper reported variance across seeds or at least a significance test (e.g., bootstrap over test sentences).","section":"§4.3, Table 1"},{"comment":"The rule-based system is explicitly limited to flat SpaCy EntityRuler patterns without syntactic information, because of early design decisions (Section 4.2 and Section 8). The paper nevertheless draws a general conclusion that deep discriminative models are 'better equipped' than 'human linguists designing rule-based NER systems' (Section 6.4). A restricted implementation does not license a claim about the rule-based family; a syntax-aware system could plausibly perform better on classes such as Bedingung and Signalwort, as the authors themselves note. I would either soften the family-level conclusion to 'the specific rule-based system implemented here' or add an additional rule-based variant using dependency patterns.","section":"§4.2, §6.4, §8"},{"comment":"The explanation that class heterogeneity is the 'common cause' of the similar performance of the rule-based and generative approaches is not tested by the experimental design. The compared implementations differ in prompt examples, rule expressivity, and training data, so the similar score levels do not isolate the effect of class definitions. The authors hedge ('arguably'), but the Conclusion repeats the causal claim as a main finding. This should be reframed as a hypothesis, with a proposed test such as comparing the same models on a corpus with more standard, linguistically motivated entity classes.","section":"§6.4"}],"minor_comments":[{"comment":"The text says 'The micro F1-scores for the model predictions are presented in Table 1,' but Table 1 reports macro F1-scores; this inconsistency should be corrected.","section":"§5.1"},{"comment":"There are several typos, including 'heterogenerity' in the abstract and 'notriously difficult' in Section 5.2; a proofreading pass is needed.","section":"Abstract and §5.2"},{"comment":"The legend reads 'F1 ScoreJaccard Score' without a separator; this should be 'F1 Score / Jaccard Score'.","section":"Figure 2"},{"comment":"The description of the Jaccard aggregation says 'the arithmetic mean of the Jaccard score values of all file inputs'; the term 'file inputs' is vague and should be defined precisely.","section":"§4.1"},{"comment":"The code availability is given as 'Anonymous, 2024'; this placeholder should be replaced with the actual author and repository information before publication.","section":"§3 and References"},{"comment":"Some class names run into their scores in the table (e.g., 'Recipient of service0.77'), which hurts readability; formatting should be fixed.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The paper's self-citations (Feddoul et al. 2024 for the corpus, Bachinger et al. 2024 for the prompting scheme) are appropriate inputs rather than a circularity problem. The main risk is that the conclusions are stated at the method-family level whereas the evidence supports only the specific implementations tested. I see no scope or novelty problem for an applied NLP venue; the requested revisions concern strengthening the comparison and softening or testing the causal interpretation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid, honestly-reported empirical comparison of three NER approaches on a real German legal corpus. The headline ranking (XLM-R > BiLSTM-CRF > rule-based > LeoLM optimistic) is plausible and consistent across token F1 and span Jaccard. The genuinely new pieces are the three-way comparison on the full GerPS-NER corpus and the per-class Jaccard analysis with zero-overlap ratios, which shows more nuance than token F1 alone.\n\nWhat the paper does well: the implementation details are transparent, code is on Zenodo, and the authors report both macro F1 and span-level metrics. They also explicitly list limitations, including the LLM prompt issue and the rule-based system's lack of syntax, which is more honesty than many benchmark papers.\n\nWhere I'd push back: the deep generative baseline is not representative of what a tuned LLM system could do. The prompts come from a smaller, differently annotated dataset (Sections 4.4 and 8), and the optimistic consolidation picks the prediction that matches gold when multiple predictions exist—that's an upper bound, not a deployment setting. So the gap between XLM-R and LeoLM is real for these exact configurations, but it's not a fair family-level comparison. That's the main soft spot. Second, there are no error bars or repeated runs; given the gap is 0.2 F1, it's probably robust, but for a benchmark I'd want variance estimates. Third, the causal claim in Section 6.4—that class heterogeneity explains both rule-based and LLM underperformance—is suggestive but untested; it should be framed as a hypothesis.\n\nNet: this is a useful paper for anyone building NER for German administrative/legal text. The ranking is plausible, the data work is careful, and the limitations are openly stated. It deserves serious peer review, but I'd ask for the LLM prompt setup to be either fixed or reframed as a comparison of specific systems, not families, and for the causal story to be softened.\n\nMy recommendation: send to review, with a request for the authors to either re-tune the LLM prompts on the dev set or explicitly declare the comparison as system-level, and to add confidence intervals if resources allow.","headline":"A transparent, useful three-way NER benchmark on German legal norms with a plausible ranking, but the LLM baseline is under-tuned and the causal story is speculative; still deserves review.","tokens_in":14236,"tokens_out":2378,"would_cite":false,"duration_ms":23633,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"XLM-R beats rules and LLMs on legal norm NER","keywords":["legal NER","German legal texts","public administration","rule-based system","deep discriminative models","large language models","heterogeneous classes","span-based evaluation"],"falsifier":"Re-run the deep generative approach with few-shot prompts drawn from the GerPS-NER training split, or add dependency-based patterns to the rule-based system; if either then matches or exceeds XLM-R's 0.6455 macro F1, the paper's heterogeneity explanation would be called into question.","tokens_in":13374,"feed_emoji":"⚖️","tokens_out":8042,"duration_ms":70795,"temperature":0.7,"pith_summary":"This paper compares three approaches to named entity recognition on German legal norms governing public administration, using the ten-class GerPS-NER corpus. It finds that deep discriminative models, in particular the XLM-R transformer, clearly outperform both a hand-built rule-based system and a prompted German open-source language model. The authors argue the gap is not an implementation accident but follows from the classes themselves, which mix linguistic categories with administrative ones. A sympathetic reader would take away that for this kind of heterogeneous annotation scheme, fine-tuned discriminative models are the safer baseline, while rules and LLM prompting still help for specific classes.","feed_headline":"XLM-R beats rules and LLMs on legal norm NER","feed_subtitle":"Macro F1 of 0.646 on the ten-class GerPS-NER corpus makes fine-tuned discriminative models the baseline to beat.","key_machinery":"The key machinery is the GerPS-NER annotation scheme, whose ten classes deliberately mix linguistically motivated categories such as 'signaling word' and 'condition' with legal-administrative ones such as 'legal grounds for action' and 'data field.' The comparison also rests on a dual evaluation: token-level F1 and span-level Jaccard (intersection-over-union) scores, the latter added because most entities span several tokens. The heterogeneity of the classes is the explanatory mechanism that carries the argument.","core_discovery":"The paper's central claim is that on GerPS-NER, a corpus of German legal norms with ten classes defined by public administration professionals, deep discriminative models are the best-performing family. XLM-R reaches the highest macro F1-score of 0.6455 and wins in 9 of 10 classes; the rule-based system reaches 0.5082 and the optimistic interpretation of the LLM outputs reaches 0.4488. The authors attribute this result to the semantic and syntactic heterogeneity of the classes, which makes the task unusually hard for both human-written rules and generic LLMs but learnable for discriminative models. Per-class analysis supports the claim: the rule-based system beats XLM-R only on the notoriously difficult data-field class, and span-based Jaccard scores show that token F1 alone can hide poor phrase-boundary detection.","pith_inferences":["The heterogeneity explanation likely generalizes to other domain-specific NER tasks, such as clinical or administrative text, where annotation schemes mix linguistic and domain-specific categories.","The LLM's underperformance may be partly an artifact of off-domain prompting; an in-domain few-shot prompt set is an immediate, low-cost experiment the paper leaves open.","The dual F1-plus-Jaccard evaluation could become a standard for legal NER, because it distinguishes 'found the entity' from 'found the right span.'","A hybrid pipeline that uses the rule-based system to select examples for LLM prompting is testable today and directly addresses the paper's suggested integration."],"forward_implications":["In 9 of 10 classes, deep discriminative models outperform both the rule-based system and the LLM, with XLM-R as the single best model.","The rule-based and LLM approaches perform at similar overall levels, but their strengths are class-specific: the rule-based system wins on the data-field class.","The performance gap is driven mainly by class heterogeneity rather than by implementation details of the individual systems.","Combining deep discriminative models with rules is a promising route for improving low-scoring classes, and combining rules with LLM prompting could cut development costs.","For practical deployment in public administration, a fine-tuned discriminative model is the recommended starting baseline."],"supporting_citations":[{"why":"Supplies the GerPS-NER corpus and its ten-class scheme, which all evaluated approaches are tested on.","marker":"Feddoul et al., 2024"},{"why":"Provides the two discriminative architectures (BiLSTM-CRF and XLM-R) and the data-augmentation backdrop the comparison builds on.","marker":"Erd et al., 2022"},{"why":"Gives the prompt design and the optimistic/pessimistic consolidation variants used for the deep generative approach.","marker":"Bachinger et al., 2024"},{"why":"Provides the XLM-RoBERTa model that achieves the best macro F1 in the comparison.","marker":"Conneau et al., 2020"},{"why":"Supplies the framework and BiLSTM-CRF implementation used as the second deep discriminative model.","marker":"Akbik et al., 2019"},{"why":"Provides the document-level extension used for fine-tuning XLM-R.","marker":"Schweter and Akbik, 2021"},{"why":"Provides the German open-source language model used for the deep generative prompting.","marker":"Plüster, 2023"},{"why":"Offers the contrasting result that a rule-based system beat deep learning on health records, which motivates the paper's heterogeneity explanation.","marker":"Gorinski et al., 2019"},{"why":"Supplies the standard German legal NER dataset and baseline that anchors the deep discriminative approach.","marker":"Leitner et al., 2020"}],"fun_headline_variants":["Discriminative NER wins on German legal norms","XLM-R tops rules and LLMs for legal NER","Legal norm NER: discriminative models rule","Class heterogeneity explains discriminative NER win","Fine-tuned XLM-R outdoes rules and LLMs on legal norms"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes each implementation faithfully represents its method family, but the LLM was prompted with examples from a smaller, differently annotated dataset and the rule-based system was limited to flat patterns without syntactic parsing, so a better-tuned version of either could shift the ranking.","fun_headline_variants_meta":{"raw":{"variants":["Discriminative NER wins on German legal norms","XLM-R tops rules and LLMs for legal NER","Legal norm NER: discriminative models rule","Class heterogeneity explains discriminative NER win","Fine-tuned XLM-R outdoes rules and LLMs on legal norms"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000619,"raw_usage":{"total_tokens":2845,"prompt_tokens":894,"completion_tokens":1951,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":510,"completion_tokens_details":{"reasoning_tokens":1871}},"tokens_in":510,"tokens_out":1951,"duration_ms":13213,"temperature":1.0,"reasoning_tokens":1871,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:27:17.874769+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the deep generative approach with few-shot prompts drawn from the GerPS-NER training split, or add dependency-based patterns to the rule-based system; if either then matches or exceeds XLM-R's 0.6455 macro F1, the paper's heterogeneity explanation would be called into question.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the GerPS-NER corpus and its ten-class scheme, which all evaluated approaches are tested on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the two discriminative architectures (BiLSTM-CRF and XLM-R) and the data-augmentation backdrop the comparison builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the prompt design and the optimistic/pessimistic consolidation variants used for the deep generative approach."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the standard German legal NER dataset and baseline that anchors the deep discriminative approach."}],"review_version":1}