{"id":"415133dc-a7f8-4dff-8b40-a8555de6c264","arxiv_id":"2504.16353","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A Legal-BERT pipeline classifies definition paragraphs in the U.S. Code, reporting 98.2% F1, but term and scope extraction are not separately evaluated.","lead":"This paper describes an AI system that finds legal definitions inside the United States Code, the large collection of federal laws. It uses a legal-language transformer model and reports high accuracy on identifying definition paragraphs, though the data and code are not released.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table I's 98.2% F1 is the paper's only quantitative support, but no train/test split, label provenance, title breakdown, or annotation reliability is reported; without proof of no leakage, the score cannot be taken as evidence of generalized definition extraction.","rationale":"I read the paper as making a two-part central claim: (1) the Legal-BERT detector achieves a near-98% F1 on statutory definition detection, and (2) the multi-stage pipeline extracts terms and scope. The reader's weakest assumption targets the gold-standard validity and lack of leakage behind part (1). I agree that this is the most load-bearing issue: Table I is the only quantitative evidence in the paper, and without a documented evaluation protocol the number cannot be checked. The absence of any title breakdown, split description, or annotation protocol is concrete, not stylistic; it means a random split and formulaic definitions could easily yield exactly the kind of high-F1 result reported, regardless of true generalization. I also note that the paper claims term and scope extraction but never evaluates either; this is a real secondary gap, but fixing it would not address the validity of Table I, whereas fixing the evaluation protocol would. The reader's verdict of CONDITIONAL remains appropriate: the work is plausible but the central numeric claim needs data release and a held-out-title evaluation before it can be accepted. No change to the reader's verdict is needed.","tokens_in":8298,"tokens_out":7404,"duration_ms":73964,"concrete_test":"Request or reconstruct the dataset from a versioned U.S. Code XML snapshot and rerun definition detection with a title-level held-out split: train on all titles except 7 and 26, tune on one held-out title, and test on titles 7 and 26. Compare this title-level F1 to the random-split Table I result. If the F1 drops by more than about 2 points, or near-duplicate paragraphs are found between train and test, the reported 98.2% overstates generalized definition detection; additionally, recompute Table I using the standard F1 formula to check how much of the reported gain is an artifact of the non-standard weighting.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section IV-B/Table I is that Legal-BERT achieves 96.82% precision, 98.93% recall, and 98.25% F1 on definition detection. That claim is only meaningful if the test set is correctly labeled and representative of the 54-title U.S. Code, with no overlap between training and test paragraphs. Section III-C reports only '3,500 definitions and 1,500 non-definitions from various titles'; it omits the split ratio, the random-vs-hierarchical splitting method, the list of titles, deduplication checks, and the gold-label generation protocol. There is no inter-annotator agreement. Because statutory definitions are highly formulaic ('the term X means Y'), a random paragraph-level split can make a model look deceptively strong: near-identical definitional sentences from the same section can appear in both train and test, and easy pattern-based positives can dominate. The paper's own Limitations section admits that implicit definitions without standard keywords remain hard; if the dataset was built by pattern search, those hard cases are under-represented and the 98.2% F1 does not describe real-world performance. In addition, Table I's F1 uses a non-standard weighted formula (Section IV-A), so the headline number is not directly comparable to standard F1.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a multi-stage pipeline for extracting statutory definitions from the U.S. Code. The system parses the XML hierarchy of the Code, classifies paragraphs as definitional using a fine-tuned Legal-BERT model, aggregates related paragraphs into definitional units, and then extracts defined terms and jurisdictional scope using a combination of attention mechanisms, sequence labeling, and rule-based patterns. The only quantitative evaluation reported is for the paragraph-level definition-detection component (Table I), where the authors claim Legal-BERT achieves 96.82% precision, 98.93% recall, and a weighted F1-score of 98.25%, outperforming logistic regression, generic BERT, and a rule-based baseline. The paper also describes a multi-task loss (Eq. 10) and the system's architecture, but it does not report evaluation results for term extraction, scope detection, definition aggregation, or the end-to-end pipeline.","tokens_in":8529,"tokens_out":4377,"duration_ms":40481,"significance":"If validated, a reliable definition detector for the U.S. Code would be practically valuable for legal informatics, especially for downstream tasks such as statutory reasoning, legislative drafting, and legal knowledge graph construction. The paper's use of a legal-domain transformer (Legal-BERT) and its attention to document-structure-aware aggregation are reasonable design choices that extend prior work. However, the current manuscript does not support the central end-to-end claim: only the paragraph-level definition-detection module has reported numbers, and those numbers rest on an underspecified evaluation protocol. The paper introduces useful components—such as the multi-task formulation and the hierarchical document graph—but the lack of evaluation for three of the four main components, combined with an undefined non-standard F1 variant, means the significance of the contribution is not yet established.","major_comments":[{"comment":"The definition-detection evaluation is underspecified to the point that the reported 98.25% F1-score cannot be interpreted. The paper does not state the train/test split ratio, whether the split was random or hierarchical (e.g., by title, chapter, or section), the list of titles in train versus test, the test-set size, deduplication checks, or the label-generation protocol (manual annotation vs. pattern-based extraction). Because statutory definitions are highly formulaic and often repeated, a random paragraph-level split can leak near-identical definitional sentences into both train and test, inflating scores. The paper's own Limitations section (Section V-B) admits that implicit definitions remain challenging, which suggests the dataset may be dominated by explicit pattern-based definitions. Please provide a full description of the dataset construction, a title breakdown, the split method, and inter-annotator agreement if applicable.","section":"Section IV-B, Table I"},{"comment":"The abstract claims the system 'automatically extract[s] defined terms, their definitions, and their scope,' but the evaluation covers only the paragraph-level definition-detection component. No quantitative results are reported for term extraction (Section III-E1), scope detection (Section III-E2), definition aggregation (Section III-D), or the multi-task heads whose losses appear in Eq. (10) (L_term and L_scope). Consequently, the paper's end-to-end claim is unsupported. To support the central contribution, the authors should report task-level metrics for each component and an end-to-end evaluation on a gold standard of complete definitions with their terms and scopes.","section":"Section IV and Abstract"},{"comment":"The 'weighted F1' measure is not defined and is not a standard F-beta score. The text says the authors use β = 0.7 for recall and β = 0.3 for precision, but the standard F-beta formula is (1+β²)PR/(β²P+R), which weights recall by β² relative to precision, not by an additive weight. The reported 98.25% F1 is therefore not comparable to the standard F1 values in the literature or to the baselines in Table I unless the same non-standard formula is applied everywhere. Please state the exact formula used and report the standard precision, recall, and F1 (or standard F-beta with β=1 and β=2) for transparency.","section":"Section IV-A"},{"comment":"The construction of the training corpus is not described. The paper states only that the dataset contains '3,500 definitions and 1,500 non-definitions from various titles,' without explaining how these paragraphs were selected, how 'definition' was operationally defined, whether the gold labels were produced by legal experts, or how the non-definition examples were sampled (e.g., random paragraphs vs. structurally similar paragraphs that are not definitions). If the corpus was built by searching for explicit definitional patterns, the benchmark will under-represent the implicit definitions that Section V-B acknowledges are problematic, and the reported accuracy will overstate real-world performance. Please document the annotation protocol and provide a distribution of definition types and titles.","section":"Section III-C"}],"minor_comments":[{"comment":"The abstract states the U.S. Code is 'over 200,000 pages,' while Section I states 'over 240,000 pages' (citing a 20% increase since 2014). Please reconcile these numbers.","section":"Abstract vs. Section I"},{"comment":"Reference [18] (Niklaus et al.) is described as proposing 'hierarchical evaluation metrics that consider partial matches and nested structures,' but the cited paper is 'A survey on open information extraction.' This citation appears inaccurate; please replace it with the correct source or correct the description.","section":"Section II-E"},{"comment":"Reference [12] (Katz et al.) is cited as demonstrating 'how legal concept networks can support predictive analytics in legal domains,' but the cited paper is about predicting Supreme Court decisions. The connection to legal concept networks is not evident; please clarify or correct the citation.","section":"Section II-D"},{"comment":"The term-extraction example shows an input sentence and a BIO tag sequence, but the tokenization is not shown, so the alignment between tokens and labels is unclear. Please include the tokenized sequence.","section":"Section III-E1"},{"comment":"The multi-task loss weights λ1, λ2, and λ3 are reported as tuned on a development set, but no development set is described. Please specify the size and composition of the development set and the tuning procedure.","section":"Section III-G, Eq. (10)"},{"comment":"The table reports point estimates without test-set size, confidence intervals, or standard deviations across folds. Please add these to assess the reliability of the differences between models.","section":"Table I"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is plausible and the domain-specific application is potentially useful, but the current manuscript is far below the evidentiary standard for a journal publication: only one module is evaluated, the evaluation protocol is underspecified, and the headline metric is not standard. I would be willing to review a revised version that reports task-level evaluations for all pipeline components, a rigorous description of the dataset and its split, and standard metrics. If the authors cannot provide such evaluations within the scope of a revision, the paper is better suited to a workshop venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. First, the paper's only measured result is paragraph-level definition detection; the end-to-end claim about extracting terms and scope has no evaluation at all. Second, the headline 98.2% F1 is computed with a weighted average they call F1, and the evaluation section omits test-set size, split method, title breakdown, and label provenance, so that number is not interpretable as reported.\n\nWhat is genuinely new: an annotated corpus of 3,500 statutory definitions and 1,500 non-definitions from multiple U.S. Code titles, and a multi-stage pipeline that uses Legal-BERT, hierarchical attention, XML structure, aggregation, term extraction, and scope detection. Applying domain-adapted transformers to statutory definition extraction is a reasonable extension of known techniques, not a new paradigm. The comparison to logistic regression and generic BERT is a sensible baseline choice, and the limitations section is honest about implicit definitions and multi-paragraph challenges.\n\nWhere it falls short. The central claim in the abstract—automatically extracting defined terms, definitions, and scope—is unsupported because only the definition-detection binary classification gets numbers. Term extraction, scope detection, and aggregation have no reported evaluation. Table I lacks the test-set size, whether the split was random or hierarchical, which titles were used, and any inter-annotator agreement. Statutory definitions are highly formulaic, so a random paragraph split can leak near-identical sentences into train and test, inflating F1. The paper's own limitations admit implicit definitions remain hard; if the dataset was built by pattern search, the 98.2% number likely overstates real-world performance. Also, Section IV-A describes a weighted F1 with beta=0.7 for recall and 0.3 for precision; that is not the standard harmonic F1, so the headline number is not comparable to the baselines' F1 unless they used the same non-standard formula. No code or data are released, which makes the corpus claim unverifiable. These are fixable problems, but they are load-bearing: the paper's only quantitative evidence is one number whose validity cannot be checked.\n\nWho this is for. People working on legal information extraction or U.S. Code analytics might find the pipeline description useful as a starting point, but they should not cite the accuracy claims. It deserves a serious referee because the dataset and task are relevant and the evaluation can be repaired with moderate effort, but as submitted the core claim is not supported.\n\nRecommendation: send to peer review with a demand for full evaluation of all pipeline stages, split and label details, standard F1 (or clear explanation), and ideally data/code release. If the authors cannot supply those, the paper should be rejected.","headline":"A plausible legal-NLP pipeline with a new corpus, but the only reported number is an uninterpretable weighted F1 for one stage; the end-to-end claim is unsupported.","tokens_in":9107,"tokens_out":2561,"would_cite":false,"duration_ms":22531,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A transformer pipeline extracts statutory definitions and their scope from the U.S. Code at 98.2% F1.","keywords":["statutory definition extraction","Legal-BERT","U.S. Code","legal NLP","definition detection","scope extraction","multi-task learning"],"falsifier":"Have two independent legal annotators label a random sample of paragraphs from all 54 titles as definitional or not, then run the published model on those titles and compare agreement and out-of-title F1. If annotator agreement is low or the F1 drops more than a few points on titles never used in fine-tuning, the 98.2% figure would not hold as a general estimate for the U.S. Code.","tokens_in":8049,"feed_emoji":"⚖️","tokens_out":9162,"duration_ms":83881,"temperature":0.7,"pith_summary":"This paper claims that a multi-stage natural language processing pipeline can automatically find statutory definitions in the United States Code, pull out the defined terms, and identify the jurisdictional scope of each definition. The central evidence is a Legal-BERT paragraph classifier that detects definitional paragraphs with 96.8% precision and 98.9% recall, for an F1-score of 98.2%, beating logistic regression, generic BERT, and rule-based pattern matching on the paper's test sets. The system works from the XML version of the U.S. Code, preserves the document hierarchy, and aggregates related paragraphs into complete definitional units rather than isolated sentences. If the result holds, it would remove a major bottleneck for legal readers and downstream systems: definitions in the U.S. Code are scattered across more than 200,000 pages and 54 titles, often far from where the defined term is used. The paper presents this as extending definition extraction from general text to complex, multi-paragraph statutory language.","feed_headline":"Legal-BERT finds U.S. Code definitions at 98.2% F1","feed_subtitle":"Five-stage pipeline connects defined terms to their definitions and jurisdictional scope across federal statutes.","key_machinery":"The load-bearing component is a fine-tuned Legal-BERT encoder: a BERT language model pre-trained on legal corpora, used as a paragraph-level definition detector. Around it, the system has a five-stage pipeline: XML structure parsing into a document graph; the transformer classifier with hierarchical attention over tokens, sentences, and paragraphs; a definition aggregator that merges adjacent definitional paragraphs through structural and semantic relatedness; hybrid term extraction combining pattern matching with a BERT-based begin-inside-outside token-labeling head; and scope detection that matches explicit scoping phrases or infers scope from document hierarchy. The machinery is designed so that definition detection uses both legal semantics and the U.S. Code's hierarchy, rather than relying only on explicit phrases such as \"the term X means Y.\"","core_discovery":"The paper's central claim is that domain-adapted transformer models combined with document-structure awareness can extract statutory definitions from the U.S. Code at near-production accuracy. Concretely, the authors assert that their best definition detector, Legal-BERT fine-tuned on 3,500 definitions and 1,500 non-definitions, reaches 96.8% precision, 98.9% recall, and 98.2% F1, with an AUPRC of 0.984; they further claim that the surrounding pipeline recovers coherent definitional units, defined terms, and scope across multiple titles. They present this as an improvement over feature-based classifiers and pattern-based methods, and as an advance over prior definition-extraction work that focused on general or non-statutory text.","pith_inferences":["The reported 98.2% F1 is a paragraph-level detection score; the paper does not report end-to-end accuracy on whole definitional units, and stage-by-stage errors would probably make the unit-level figure lower.","Because fine-tuning uses a 3,500/1,500 corpus from selected titles, applying the unchanged model to all 54 titles or to state statutes is an immediate test of whether the score generalizes to definition formats outside that corpus.","The paper's own limitations list (implicit definitions, cross-references, non-adjacent scope references, and multi-paragraph definitions) identifies exactly where the headline F1 is most fragile; a stratified evaluation by definition format would show how much the score varies."],"forward_implications":["Legal research platforms could automatically link any occurrence of a defined term to the definition that governs it, including the title or section scope.","A reader query such as \"what does this term mean in Title 7?\" could return the definition, its exceptions, and the sections where it applies.","The extracted term and scope pairs could feed statutory-reasoning systems that must decide which definition controls a given legal context.","Definition networks built from the outputs would let analysts see how a term defined in one title is referenced by definitions elsewhere in the code."],"supporting_citations":[{"why":"Supplies the XML version of the U.S. Code that the document-structure processor parses.","marker":"[1]"},{"why":"Supplies the Legal-BERT encoder that the definition detector is fine-tuned from.","marker":"[3]"},{"why":"Provides the joint-learning formulation for definition extraction and relationship identification that motivates the multi-task heads.","marker":"[6]"},{"why":"Frames the definition-extraction task and provides a multi-domain benchmark that situates the evaluation.","marker":"[7]"},{"why":"Demonstrates that BERT-based models can capture implicit definitional cues on domain-specific corpora, the basis for replacing feature-based classifiers.","marker":"[16]"},{"why":"Establishes the legal importance and varied formats of statutory definitions, which define the task.","marker":"[26]"},{"why":"Provides earlier linguistic and structural feature methods for definition extraction that the paper builds on and compares against.","marker":"[27]"},{"why":"Provides the genetic-algorithm pattern-based baseline whose precision and recall the paper contrasts with its transformer results.","marker":"[28]"}],"fun_headline_variants":["Legal-BERT extracts U.S. Code definitions with 98.2% F1","Transformer model parses U.S. Code definitions at 98.2% F1","Fine-tuned Legal-BERT hits 98.2% F1 on U.S. Code definitions","Who defines 'defines'? Legal-BERT does, at 98.2% F1","U.S. Code definition extraction: 98.2% F1 with Legal-BERT"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported accuracy depends on the assumption that the 3,500 definitions and 1,500 non-definitions used for training are labeled correctly by legal standards, cover the range of definition formats across the U.S. Code, and are split so that test paragraphs do not reuse the same structural patterns seen in training.","fun_headline_variants_meta":{"raw":{"variants":["Legal-BERT extracts U.S. Code definitions with 98.2% F1","Transformer model parses U.S. Code definitions at 98.2% F1","Fine-tuned Legal-BERT hits 98.2% F1 on U.S. Code definitions","Who defines 'defines'? Legal-BERT does, at 98.2% F1","U.S. Code definition extraction: 98.2% F1 with Legal-BERT"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001321,"raw_usage":{"total_tokens":5391,"prompt_tokens":968,"completion_tokens":4423,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":584,"completion_tokens_details":{"reasoning_tokens":4305}},"tokens_in":584,"tokens_out":4423,"duration_ms":27374,"temperature":1.0,"reasoning_tokens":4305,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:05:16.239615+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have two independent legal annotators label a random sample of paragraphs from all 54 titles as definitional or not, then run the published model on those titles and compare agreement and out-of-title F1. If annotator agreement is low or the F1 drops more than a few points on titles never used in fine-tuning, the 98.2% figure would not hold as a general estimate for the U.S. Code.","supporting_citations":[{"cited_title":"Available: http://uscode.house.gov/","cited_arxiv_id":null,"evidence_quote":"Supplies the XML version of the U.S. Code that the document-structure processor parses."},{"cited_title":"Chalkidis, M","cited_arxiv_id":null,"evidence_quote":"Supplies the Legal-BERT encoder that the definition detector is fine-tuned from."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the joint-learning formulation for definition extraction and relationship identification that motivates the multi-task heads."},{"cited_title":"Spala, N","cited_arxiv_id":null,"evidence_quote":"Frames the definition-extraction task and provides a multi-domain benchmark that situates the evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Demonstrates that BERT-based models can capture implicit definitional cues on domain-specific corpora, the basis for replacing feature-based classifiers."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the legal importance and varied formats of statutory definitions, which define the task."},{"cited_title":"Westerhout, ”Definition extraction using linguistic and structural features,” in Proceedings of the 1st Workshop on Definition Extraction , 2009, pp","cited_arxiv_id":null,"evidence_quote":"Provides earlier linguistic and structural feature methods for definition extraction that the paper builds on and compares against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the genetic-algorithm pattern-based baseline whose precision and recall the paper contrasts with its transformer results."}],"review_version":1}