{"id":"e904f4fb-b983-406d-8a65-a76969c6ff63","arxiv_id":"2501.15990","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A new Spanish-language contract clause extraction corpus of 373 tenders, 19 categories, and 4,782 tags, with baseline fine-tuning results.","lead":"3CEL is a manually annotated corpus of 373 Spanish public procurement tenders with 4,782 span-level tags across 19 contract-related categories. It aims to fill a gap in Spanish legal NLP resources and is benchmarked by fine-tuning four transformer language models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The quality claim rests on an undefined 0.61 IAA with no external benchmark or data release, so the reliability of 3CEL gold labels is unverified.","rationale":"The paper's contribution is a resource, so the load-bearing claim is that the annotations are good enough to serve as gold labels. The reported IAA is the only direct evidence, and it is uninterpretable without defining the metric or providing a reference point. The fine-tuning results (macro F1 0.73 on 18 labels, 0.84 on 15) are suggestive that the corpus contains learnable signal, but they are single-run and the conclusion that MEL is best is not robust: on 18 labels xlm-roberta-large scores 0.73 versus MEL's 0.72, and the stated reason for preferring MEL (trained on more data, faster learning) does not follow from the reported test metric. This is secondary to the annotation-reliability issue, but both point to the need for release and replication. I do not think the argument is unsound; the reader's CONDITIONAL verdict is appropriate pending corpus release and a defined, benchmarked IAA.","tokens_in":9514,"tokens_out":7486,"duration_ms":74097,"concrete_test":"Release the corpus (or a stratified 50-tender sample) and the annotation guidelines; have two annotators who did not participate in the project label the sample under the same blind protocol, and report the exact IAA statistic with per-label values. If external strict category+span agreement is materially below 0.61, the reliability claim fails; if it is comparable, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that 3CEL is a high-quality resource and that fine-tuning validates it depends on the annotations being reliable ground truth. The only quantitative evidence is \"mean IAA 0.61\" (Table 4, Section 3). The paper never states which IAA statistic is used (span-level F1, Cohen's kappa, token accuracy), gives no per-block or per-label breakdown, and provides no external benchmark or threshold for strict span-category agreement in long legal clauses. Because the guidelines were revised during annotation and blocks were re-annotated for homogeneity, the released labels reflect an undocumented harmonization process; raw IAA alone cannot establish their reliability. If per-label agreement is low for rare categories such as canon, surety board and intellectual property (the categories removed in the 15-label experiments), then the macro-F1 results and the \"valuable resource\" conclusion are not well supported. The corpus link is marked \"to be published\", so independent reannotation is currently impossible; this makes the concern untestable rather than resolved.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces 3CEL, a corpus of 373 manually annotated Spanish public-procurement tenders with 4,782 span-level tags in 19 categories, intended for legal information extraction. The authors describe the full creation pipeline (data collection, tag-set design, transcription, cleaning, anonymization, and blind peer annotation) and report fine-tuning experiments for span categorization with four transformer models (xlm-roberta-large, legal-xlm-roberta-large, RoBERTalex, and MEL), using 18 labels and a reduced 15-label set. The paper concludes that 3CEL is a high-quality and valuable resource and that the in-house MEL model performs best when considering macro-F1 and learning speed.","tokens_in":9735,"tokens_out":2529,"duration_ms":24232,"significance":"If the corpus is released and its annotation quality is properly established, 3CEL would be a valuable and, to my knowledge, the first comparable Spanish legal clause extraction corpus, filling a clear gap given the scarcity of Spanish legal NLP resources. The detailed documentation of the annotation workflow, the anonymization strategy, and the use of strict span-and-category agreement are commendable and provide a useful template for similar efforts. However, the current evidence for the central quality claim is incomplete: the corpus is not yet publicly accessible, the reported inter-annotator agreement is not fully specified, and the model-comparison experiments rest on a single seed and a post-hoc label subset. These issues are fixable but must be addressed before the resource can be adopted as reliable ground truth.","major_comments":[{"comment":"The manuscript reports a mean inter-annotator agreement of 0.61 but does not state which statistic this is (Cohen's kappa, span-level F1, token accuracy, or another metric), does not provide per-block or per-category breakdowns, and gives no external benchmark or quality threshold for span-category annotation of long legal clauses. Since IAA is the only quantitative evidence for the core claim that 3CEL is a \"high quality resource,\" the authors should specify the exact metric, report agreement separately for each of the 19 categories and each annotation block, and justify why the achieved level is sufficient for use as gold labels in fine-tuning.","section":"Section 3, Table 4"},{"comment":"The corpus link is marked \"to be published,\" which makes the central artifact unavailable for independent reannotation, reproducibility checks, or verification of the annotation guidelines and harmonization process. Given that the paper's main contribution is the resource itself, the authors should either provide a public release link (even a preview or sample) or state a concrete availability plan; without this, the quality claims cannot be independently assessed.","section":"Section 3, footnote 2"},{"comment":"All fine-tuning experiments use a single seed (seed 12345), and the reported macro-F1 differences between MEL and xlm-roberta-large are only 0.01 in the 18-label setting and 0.01 in the 15-label setting. These differences are well within the range of seed-to-seed variation for transformer fine-tuning, so the conclusion that MEL is the best model is not statistically supported. The authors should run multiple seeds (e.g., 3--5) and report mean and standard deviation, or use a significance test, before drawing conclusions about model ranking.","section":"Section 4.2, Table 6 and Table 5"},{"comment":"The 15-label experiment is defined after the fact by removing the three least frequent categories (canon, surety board, intellectual property), and the removal is justified only by the observation that they are underrepresented while simultaneously being used to explain a 0.1 macro-F1 improvement. This is a post-hoc label subset selection that makes the 18-vs-15 comparison difficult to interpret as a controlled evaluation. The authors should either pre-register the subset decision, report per-category F1 for all 18 labels, or treat the 15-label result as a descriptive diagnostic rather than a validated experimental condition.","section":"Section 4.1 and Section 4.2"},{"comment":"The model concluded to be best, MEL, is developed within the same INESData project and is cited as \"to be published,\" so its architecture, training data, and weights are not available to readers. This makes the comparison non-reproducible and creates a potential conflict of interest in the evaluation. The authors should disclose MEL's full specification, release the model if possible, or at minimum provide a detailed technical description so that the comparison can be independently reproduced.","section":"Section 4.2 and reference [33]"}],"minor_comments":[{"comment":"The abstract and introduction state that 3CEL contains 19 defined categories, while Table 4 reports 18 categories present in the corpus and the fine-tuning experiments use 18 labels. Please clarify the relationship between the 19 defined categories, the 18 annotated/present categories, and the 18 labels used in the experiments.","section":"Abstract and Table 4"},{"comment":"The task description says that \"the number of annotated tags and the number of labels in the tag set are doubled, including 36 classification tags instead of the original 18.\" Since Table 2 lists 19 labels, the count should be reconciled (e.g., if one of the 19 labels is unused or merged).","section":"Section 4.1"},{"comment":"The row \"metric_for_best_Bearing in mindmodel macro_f1\" appears to contain a copy-paste artifact; it should read \"metric_for_best_model: macro_f1\" or similar.","section":"Table 5"},{"comment":"There are minor language errors: \"perfomance\" should be \"performance\" in Section 4.1, and the Spanish label \"indemización\" should be \"indemnización\" in Table 2.","section":"Section 4.1 and Table 2"},{"comment":"Figure 9's caption says \"3CEL tag distribution\" but does not specify whether the counts include both B- and I-prefixed tags; please make this explicit.","section":"Section 3, Figure 9"},{"comment":"When citing CUAD, the number of labels (41) and annotated labels (over 13,000) is given, but for 3CEL the analogous numbers (19 categories, 4,782 tags) are not shown in the same passage; adding this comparison would help readers situate the corpus size.","section":"Section 2"}],"recommendation":"major_revision","confidential_remarks":"The paper describes a potentially useful resource, but its central validation currently depends on an undefined IAA metric and an unreleased corpus. The fine-tuning evaluation also has a problematic post-hoc label selection and an in-house unpublished model as the recommended system. These concerns are addressable with additional analysis and transparency, so I see no need for rejection, but the manuscript requires substantial revision before it meets the standards for a resource paper. I would also encourage the editor to ask the authors to make the corpus available at least in a preview form during review, since no independent verification is otherwise possible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a real contribution—a first Spanish corpus for contract clause span categorization, built with a sensible pipeline and a tag set tailored to public tenders rather than a CUAD translation. The fine-tuning results are a sanity check, not the main event. The problem is the paper's quality claim rests on a single IAA number (0.61) that is never defined, with no per-label breakdown and no external benchmark, and the corpus link says 'to be published', so the central artifact is currently untestable. For a resource paper, that is the whole ballgame.\n\nWhat's good: the corpus fills a documented gap. I see no comparable Spanish legal contract clause extraction corpus in the citations. The five-step creation methodology is described in detail: data collection from the Spanish procurement platform with a sample stratified by contract type, OCR tool selection by manual peer review, cleaning, anonymization with consistent fake entities, and blind peer annotation following MATTER. The 19-label tag set is a thoughtful adaptation of CUAD to Spanish tenders, and the decision to use span categorization rather than token-level NER is appropriate for long contract clauses. The fine-tuning setup is standard, and the authors are honest about small differences: 0.73 vs 0.72 macro F1 between xlm-roberta-large and MEL is called 'not relevant,' which is the correct read.\n\nSoft spots: (1) The IAA statistic is not identified. 'Annotators must match both category and span' could be token-level exact match, span-level F1, or something else, and the paper does not say. There is no per-block or per-label table, so we cannot see whether agreement is high for common categories and terrible for the rare ones—canon, surety board, intellectual property—which are exactly the labels removed in the 15-label experiment. The 0.1 F1 improvement from dropping them then gets used to support the 'valuable resource' conclusion, which is post hoc. (2) Single seed, no error bars; the MEL vs xlm-roberta-large differences are within noise. (3) MEL is an unpublished same-project model; concluding it is best while citing it as 'to be published' is a presentation problem, even if the conclusion is modest. (4) The corpus link is 'to be published,' so independent reannotation or even inspection is impossible right now.\n\nNone of this is fatal to the resource's potential. A strict-agreement IAA of 0.61 is not automatically bad for long, multi-label spans. But without the corpus and a defined metric, the reliability claim is unverified, not verified.\n\nWho should read this: anyone building Spanish legal IE resources or seeking a benchmark in this niche. It deserves a serious referee if the authors release the data and annotation guidelines. I would want the IAA statistic defined, per-label agreement reported, and at least a second seed before accepting the fine-tuning numbers. If the corpus becomes public, this is a citable resource; as it stands, I would only cite it as 'described, pending release.'\n\nRecommendation: send to peer review, conditional on releasing the corpus and tightening the evaluation reporting. Not a desk reject.","headline":"A genuinely new Spanish legal contract clause corpus with a solid pipeline, but the reliability claim rests on an undefined IAA and unreleased data; deserves peer review conditional on release and better evaluation reporting.","tokens_in":10260,"tokens_out":3298,"would_cite":false,"duration_ms":28985,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper presents 3CEL, a manually annotated corpus of 373 Spanish public tenders containing 4,782 span-level tags across 19 legal categories, and shows through fine-tuning experiments that the corpus supports automatic clause…","keywords":["Spanish legal NLP","contract clauses","information extraction","span categorization","manually annotated corpus","public procurement tenders","legal corpus annotation","fine-tuning"],"falsifier":"Re-annotate a random subsample of about 50 tenders with two new legal annotators using the same published guidelines and measure category-span agreement; agreement materially below the reported 0.61, or high variation across the 19 categories, would indicate that the learned models reflect annotator noise rather than stable contract content.","tokens_in":9391,"feed_emoji":"⚖️","tokens_out":6305,"duration_ms":57622,"temperature":0.7,"pith_summary":"3CEL is a new manually annotated corpus for extracting legal clauses from Spanish public-procurement tenders. It contains 373 anonymized contracts, 19 legally defined categories, and 4,782 span-level tags, and it is offered as the first corpus of its kind in Spanish. The paper argues that this resource is valuable by fine-tuning four transformer models on it; the best configuration reaches 0.84 macro F1 on the 15 most frequent categories. If correct, this gives Spanish legal NLP a reusable benchmark for clause-level information extraction that did not previously exist.","feed_headline":"First Spanish contract-clause corpus covers 373 tenders","feed_subtitle":"Fine-tuned models identify legal clauses in Spanish tenders at 0.84 macro F1, making clause review automatable.","key_machinery":"The central object is the corpus itself, named 3CEL, whose annotations are the load-bearing element. Each annotation is a span of text, typically a sentence or several sentences, assigned one or more of 19 legal categories covering contract provisions, contract crisis, and compliance; the categories are encoded with B/I token tags, doubling the classification labels to 36. The corpus was built through a five-step pipeline from Spanish public tenders, including data collection, tag-set definition, transcription, cleaning and anonymization, and annotated under a MATTER-style workflow with blind peer review and a final harmonization pass. This machinery matters because it converts unstructured contract text into a form that standard token classifiers can learn from, while the anonymization ensures the corpus can be shared.","core_discovery":"The paper's central claim is that 3CEL is a high-quality resource for Spanish legal information extraction and that fine-tuning experiments validate it for span categorization. On the paper's own terms, the discovery is that a manually curated corpus of sentence-level legal labels, constructed from real Madrid-region tenders and harmonized through blind peer annotation, supports automatic clause extraction at a practically useful level: macro F1 of roughly 0.73 with the full 18-label set and 0.84 when the three rarest categories are dropped. The paper also reports a mean inter-annotator agreement of 0.61 under the strict rule that annotators must agree on both the legal category and the exact span boundary. These numbers are presented as evidence that a Spanish legal clause extraction benchmark can be built and that current language models can learn it.","pith_inferences":["Editorial inference: a testable extension would be to run a simple keyword-plus-window span-matching baseline against the fine-tuned models; if that baseline approaches 0.84 macro F1, the corpus's added value over cheap pattern matching would need re-examination.","Editorial inference: the 0.1 macro-F1 gain when moving from 18 to 15 labels suggests that annotation effort for rare legal categories may be better spent on collecting more examples than on adding labels, a trade-off future corpus designers could plan around.","Editorial inference: because a harmonization step revises annotations after disagreement, the final labels may be more consistent than the raw 0.61 inter-annotator agreement suggests, meaning the corpus's practical training value could exceed what the headline agreement number implies.","Editorial inference: the anonymization strategy, replacing real entities with realistic fake ones, could be reused to publish other sensitive legal corpora, but its effect on model performance when the model is later applied to non-anonymized text remains untested."],"forward_implications":["Contract reviewers can use a fine-tuned model to locate and label clauses such as termination, penalties, guarantees, and confidentiality in Spanish tenders, reducing manual review time.","The reported 0.73 macro F1 on 18 categories and 0.84 on the 15 most represented categories establish a benchmark that future Spanish legal span-categorization systems would need to beat.","Removing the three least-represented categories raises macro F1 by about 0.1, indicating that low-frequency legal clauses are the main bottleneck for automatic extraction.","Because the corpus is anonymized with realistic fake entities, it can be shared and reused without exposing personal data, supporting further research in the Spanish legal domain."],"supporting_citations":[{"why":"Supplies the CUAD contract-review annotation scheme and 41-label tag set that 3CEL adapts to Spanish tenders.","marker":"[14]"},{"why":"Provides the five-step corpus creation methodology that the authors follow to build 3CEL.","marker":"[22]"},{"why":"Selected OCR tool used to transcribe the PDF tenders into raw text for annotation.","marker":"[24]"},{"why":"Pre-trained NER model used to detect and anonymize person and location entities in the tenders.","marker":"[28]"},{"why":"Defines the MATTER annotation methodology covering guides, segmentation, pre-annotation, and annotation.","marker":"[29]"},{"why":"Annotation tool used for the actual span-level labeling of the corpus.","marker":"[30]"},{"why":"Multilingual transformer fine-tuned on 3CEL that attains the best 18-label macro F1 of 0.73.","marker":"[31]"},{"why":"Legal Spanish language model that is the fastest learner and best performer on the 15-label set with 0.84 macro F1.","marker":"[33]"}],"fun_headline_variants":["Spanish contract-clause corpus debuts with 373 tenders","New Spanish legal corpus: 373 tenders, 19 categories, 4,782 tags","Spanish contract clause extraction reaches 0.84 F1 on common labels","First annotated Spanish contract corpus for NLP now available","373 Spanish tenders tagged for automated contract clause review"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a mean inter-annotator agreement of 0.61, scored only when annotators match both the legal category and the exact text span, is good enough for the 3CEL annotations to serve as reliable ground truth for fine-tuning.","fun_headline_variants_meta":{"raw":{"variants":["Spanish contract-clause corpus debuts with 373 tenders","New Spanish legal corpus: 373 tenders, 19 categories, 4,782 tags","Spanish contract clause extraction reaches 0.84 F1 on common labels","First annotated Spanish contract corpus for NLP now available","373 Spanish tenders tagged for automated contract clause review"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001239,"raw_usage":{"total_tokens":5039,"prompt_tokens":850,"completion_tokens":4189,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":466,"completion_tokens_details":{"reasoning_tokens":4099}},"tokens_in":466,"tokens_out":4189,"duration_ms":31670,"temperature":1.0,"reasoning_tokens":4099,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T13:48:55.780402+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a random subsample of about 50 tenders with two new legal annotators using the same published guidelines and measure category-span agreement; agreement materially below the reported 0.61, or high variation across the 19 categories, would indicate that the learned models reflect annotator noise rather than stable contract content.","supporting_citations":[{"cited_title":"Prodigy: A new annotation tool for radically efficient machine teaching","cited_arxiv_id":null,"evidence_quote":"Annotation tool used for the actual span-level labeling of the corpus."},{"cited_title":"Anotación de corpus lingüísticos: metodología utilizada en el instituto de ingeniería del conocimiento (iic)","cited_arxiv_id":null,"evidence_quote":"Provides the five-step corpus creation methodology that the authors follow to build 3CEL."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Selected OCR tool used to transcribe the PDF tenders into raw text for annotation."},{"cited_title":"Mmg/xlm-roberta-large-ner-spanish","cited_arxiv_id":null,"evidence_quote":"Pre-trained NER model used to detect and anonymize person and location entities in the tenders."},{"cited_title":"Natural Language Annotation for Machine Learning","cited_arxiv_id":null,"evidence_quote":"Defines the MATTER annotation methodology covering guides, segmentation, pre-annotation, and annotation."},{"cited_title":"Mel: Legal Spanish Language Model","cited_arxiv_id":null,"evidence_quote":"Legal Spanish language model that is the fastest learner and best performer on the 15-label set with 0.84 macro F1."}],"review_version":1}