REVIEW 5 major objections 6 minor 34 references
3CEL: A corpus of legal Spanish contract clauses
T0 review · 5 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read This paper presents 3CEL, a manually annotated corpus of 373 Spanish public tenders containing 4,782 span-level tags across 19 legal categories, and shows through fine-tuning experiments that the corpus supports automatic clause…
desk verdict A genuinely new Spanish legal contract clause corpus with a solid pipeline, but the reliability claim rests on an undefined IAA and unreleased data; deserves peer review conditional on release and better evaluation reporting. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the corpus itself, named 3CEL, whose annotations are the load-bearing element. Each annotation is a span of text, typically a sentence or several sentences, assigned one or more of 19 legal categories covering contract provisions, contract crisis, and compliance; the categories are encoded with B/I token tags, doubling the classification labels to 36. The corpus was built through a five-step pipeline from Spanish public tenders, including data collection, tag-set definition, transcription, cleaning and anonymization, and annotated under a MATTER-style workflow with blind peer review and a final harmonization pass. This machinery matters because it converts unstructured contract text into a form that standard token classifiers can learn from, while the anonymization ensures the corpus can be shared.
What would settle it
Re-annotate a random subsample of about 50 tenders with two new legal annotators using the same published guidelines and measure category-span agreement; agreement materially below the reported 0.61, or high variation across the 19 categories, would indicate that the learned models reflect annotator noise rather than stable contract content.
Extended reading notes
Core claim
The paper's central claim is that 3CEL is a high-quality resource for Spanish legal information extraction and that fine-tuning experiments validate it for span categorization. On the paper's own terms, the discovery is that a manually curated corpus of sentence-level legal labels, constructed from real Madrid-region tenders and harmonized through blind peer annotation, supports automatic clause extraction at a practically useful level: macro F1 of roughly 0.73 with the full 18-label set and 0.84 when the three rarest categories are dropped. The paper also reports a mean inter-annotator agreement of 0.61 under the strict rule that annotators must agree on both the legal category and the exact span boundary. These numbers are presented as evidence that a Spanish legal clause extraction benchmark can be built and that current language models can learn it.
Load-bearing premise
The load-bearing premise is that a mean inter-annotator agreement of 0.61, scored only when annotators match both the legal category and the exact text span, is good enough for the 3CEL annotations to serve as reliable ground truth for fine-tuning.
Editorial extensions
If this is right
- Contract reviewers can use a fine-tuned model to locate and label clauses such as termination, penalties, guarantees, and confidentiality in Spanish tenders, reducing manual review time.
- The reported 0.73 macro F1 on 18 categories and 0.84 on the 15 most represented categories establish a benchmark that future Spanish legal span-categorization systems would need to beat.
- Removing the three least-represented categories raises macro F1 by about 0.1, indicating that low-frequency legal clauses are the main bottleneck for automatic extraction.
- Because the corpus is anonymized with realistic fake entities, it can be shared and reused without exposing personal data, supporting further research in the Spanish legal domain.
Reading between the lines
- Editorial inference: a testable extension would be to run a simple keyword-plus-window span-matching baseline against the fine-tuned models; if that baseline approaches 0.84 macro F1, the corpus's added value over cheap pattern matching would need re-examination.
- Editorial inference: the 0.1 macro-F1 gain when moving from 18 to 15 labels suggests that annotation effort for rare legal categories may be better spent on collecting more examples than on adding labels, a trade-off future corpus designers could plan around.
- Editorial inference: because a harmonization step revises annotations after disagreement, the final labels may be more consistent than the raw 0.61 inter-annotator agreement suggests, meaning the corpus's practical training value could exceed what the headline agreement number implies.
- Editorial inference: the anonymization strategy, replacing real entities with realistic fake ones, could be reused to publish other sensitive legal corpora, but its effect on model performance when the model is later applied to non-anonymized text remains untested.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces 3CEL, a corpus of 373 manually annotated Spanish public-procurement tenders with 4,782 span-level tags in 19 categories, intended for legal information extraction. The authors describe the full creation pipeline (data collection, tag-set design, transcription, cleaning, anonymization, and blind peer annotation) and report fine-tuning experiments for span categorization with four transformer models (xlm-roberta-large, legal-xlm-roberta-large, RoBERTalex, and MEL), using 18 labels and a reduced 15-label set. The paper concludes that 3CEL is a high-quality and valuable resource and that the in-house MEL model performs best when considering macro-F1 and learning speed.
Significance. If the corpus is released and its annotation quality is properly established, 3CEL would be a valuable and, to my knowledge, the first comparable Spanish legal clause extraction corpus, filling a clear gap given the scarcity of Spanish legal NLP resources. The detailed documentation of the annotation workflow, the anonymization strategy, and the use of strict span-and-category agreement are commendable and provide a useful template for similar efforts. However, the current evidence for the central quality claim is incomplete: the corpus is not yet publicly accessible, the reported inter-annotator agreement is not fully specified, and the model-comparison experiments rest on a single seed and a post-hoc label subset. These issues are fixable but must be addressed before the resource can be adopted as reliable ground truth.
major comments (5)
- [Section 3, Table 4] The manuscript reports a mean inter-annotator agreement of 0.61 but does not state which statistic this is (Cohen's kappa, span-level F1, token accuracy, or another metric), does not provide per-block or per-category breakdowns, and gives no external benchmark or quality threshold for span-category annotation of long legal clauses. Since IAA is the only quantitative evidence for the core claim that 3CEL is a "high quality resource," the authors should specify the exact metric, report agreement separately for each of the 19 categories and each annotation block, and justify why the achieved level is sufficient for use as gold labels in fine-tuning.
- [Section 3, footnote 2] The corpus link is marked "to be published," which makes the central artifact unavailable for independent reannotation, reproducibility checks, or verification of the annotation guidelines and harmonization process. Given that the paper's main contribution is the resource itself, the authors should either provide a public release link (even a preview or sample) or state a concrete availability plan; without this, the quality claims cannot be independently assessed.
- [Section 4.2, Table 6 and Table 5] All fine-tuning experiments use a single seed (seed 12345), and the reported macro-F1 differences between MEL and xlm-roberta-large are only 0.01 in the 18-label setting and 0.01 in the 15-label setting. These differences are well within the range of seed-to-seed variation for transformer fine-tuning, so the conclusion that MEL is the best model is not statistically supported. The authors should run multiple seeds (e.g., 3--5) and report mean and standard deviation, or use a significance test, before drawing conclusions about model ranking.
- [Section 4.1 and Section 4.2] The 15-label experiment is defined after the fact by removing the three least frequent categories (canon, surety board, intellectual property), and the removal is justified only by the observation that they are underrepresented while simultaneously being used to explain a 0.1 macro-F1 improvement. This is a post-hoc label subset selection that makes the 18-vs-15 comparison difficult to interpret as a controlled evaluation. The authors should either pre-register the subset decision, report per-category F1 for all 18 labels, or treat the 15-label result as a descriptive diagnostic rather than a validated experimental condition.
- [Section 4.2 and reference [33]] The model concluded to be best, MEL, is developed within the same INESData project and is cited as "to be published," so its architecture, training data, and weights are not available to readers. This makes the comparison non-reproducible and creates a potential conflict of interest in the evaluation. The authors should disclose MEL's full specification, release the model if possible, or at minimum provide a detailed technical description so that the comparison can be independently reproduced.
minor comments (6)
- [Abstract and Table 4] The abstract and introduction state that 3CEL contains 19 defined categories, while Table 4 reports 18 categories present in the corpus and the fine-tuning experiments use 18 labels. Please clarify the relationship between the 19 defined categories, the 18 annotated/present categories, and the 18 labels used in the experiments.
- [Section 4.1] The task description says that "the number of annotated tags and the number of labels in the tag set are doubled, including 36 classification tags instead of the original 18." Since Table 2 lists 19 labels, the count should be reconciled (e.g., if one of the 19 labels is unused or merged).
- [Table 5] The row "metric_for_best_Bearing in mindmodel macro_f1" appears to contain a copy-paste artifact; it should read "metric_for_best_model: macro_f1" or similar.
- [Section 4.1 and Table 2] There are minor language errors: "perfomance" should be "performance" in Section 4.1, and the Spanish label "indemización" should be "indemnización" in Table 2.
- [Section 3, Figure 9] Figure 9's caption says "3CEL tag distribution" but does not specify whether the counts include both B- and I-prefixed tags; please make this explicit.
- [Section 2] When citing CUAD, the number of labels (41) and annotated labels (over 13,000) is given, but for 3CEL the analogous numbers (19 categories, 4,782 tags) are not shown in the same passage; adding this comparison would help readers situate the corpus size.
Circularity Check
No circular derivation: 3CEL's claims rest on manual annotation, reported IAA, and held-out fine-tuning; the only same-project self-citation (MEL) is not load-bearing.
full rationale
The paper does not contain a derivation chain in which a predicted quantity is defined in terms of the evidence, nor a fitted parameter that is later relabeled as a prediction. The central claim is that 3CEL is a valuable Spanish legal span-categorization resource; the support is the annotation process, the reported mean inter-annotator agreement of 0.61, and fine-tuning results on a train/test split (75%/25%) of the corpus. Those macro-F1 numbers are computed on held-out test chunks, so the evaluation is not circular by construction: the test spans are not used to fit the models. The tag set is derived from CUAD and expert legal review, not from the corpus itself, so there is no self-definitional loop. The only same-project citation is MEL [33], described in a footnote as 'also developed in the context of INESData' and 'to be published'. That is a genuine self-citation, and the paper's conclusion that MEL is the best model is partly based on MEL's unpublished training setup and on learning-speed curves. However, this model-comparison conclusion is not load-bearing for the paper's main claim: the statement that 'the experiments carried out validate 3CEL as a valuable resource for span categorization' is supported by the held-out macro-F1 values of several independently published models (xlm-roberta-large, legal-xlm-roberta-large, RoBERTalex) alone, and the MEL comparison is an ancillary result. The concern that a mean IAA of 0.61 is undefined as to statistic, and that no external benchmark or released corpus currently permits independent reannotation, is a legitimate quality/correctness risk but not a circularity: it does not make the corpus claim equivalent to its inputs. Therefore the circularity score is low, reflecting only the minor, non-load-bearing self-citation of MEL and of the IIC annotation methodology [22].
Assumptions & free parameters
free parameters (1)
- Reduced label set cutoff =
>40 appearances
assumptions (4)
- domain assumption The 19 expert-defined categories and annotation guidelines capture information relevant to contract review.
- domain assumption Mean IAA of 0.61 under strict matching indicates acceptable annotation consistency.
- domain assumption Fine-tuning on text chunks preserves the span-categorization task and the validity of performance estimates.
- domain assumption Consistent pseudo-anonymization replacing entities with fake but realistic strings preserves the contexts needed for annotation and fine-tuning.
Cite this review
Pith. "Pith review of 3CEL: A corpus of legal Spanish contract clauses." pith.science (2026). https://pith.science/paper/Z6GQ2ACI
@misc{pith2026250115990,
author = {Pith},
title = {Pith review of: 3CEL: A corpus of legal Spanish contract clauses},
year = {2026},
howpublished = {\url{https://pith.science/paper/Z6GQ2ACI}},
note = {Machine review of arXiv:2501.15990}
}
read the original abstract
Legal corpora for Natural Language Processing (NLP) are valuable and scarce resources in languages like Spanish due to two main reasons: data accessibility and legal expert knowledge availability. INESData 2024 is a European Union funded project lead by the Universidad Polit\'ecnica de Madrid (UPM) and developed by Instituto de Ingenier\'ia del Conocimiento (IIC) to create a series of state-of-the-art NLP resources applied to the legal/administrative domain in Spanish. The goal of this paper is to present the Corpus of Legal Spanish Contract Clauses (3CEL), which is a contract information extraction corpus developed within the framework of INESData 2024. 3CEL contains 373 manually annotated tenders using 19 defined categories (4 782 total tags) that identify key information for contract understanding and reviewing.
Figures
Figures from the paper (8 more)
Reference graph
Works this paper leans on
-
[1]
Information extraction: Techniques and challenges
Ralph Gishman. Information extraction: Techniques and challenges. In SCIE ’97: International Summer School on Information Extraction: A Multidisciplinary Approach to an Emerging Information Technology, pages 10–27. 10 3CEL: a Corpus of Legal Spanish Contract Clauses SCIE, 1997
work page 1997
-
[2]
Information Extraction: Past, Present and Future, pages 23–49
Jakub Piskorski and Roman Yangarber. Information Extraction: Past, Present and Future, pages 23–49. Springer, United States, 2012
work page 2012
-
[3]
Rosen, Gerbrand Ceder, Kristin A
John Dagdelen, Alexander Dunn, Sanghoon Lee, Nicholas Walker, Andrew S. Rosen, Gerbrand Ceder, Kristin A. Persson, and Anubhav Jain. Structured information extraction from scientific text with large language models. Nature Communications, 2024
work page 2024
-
[4]
Legalruleml core specification version 1.0
Monica Palmirani, Guido Governatori, Tara Athan, Harold Boley, Adrian Paschke, and Adam Wyner. Legalruleml core specification version 1.0. https://docs.oasis-open.org/legalruleml/legalruleml-core-spec/ v1.0/legalruleml-core-spec-v1.0.html , 2021
work page 2021
-
[5]
Financial industry business ontology - fibo
Enterprise Data Management Council EDM Council. Financial industry business ontology - fibo. https: //edmcouncil.org/frameworks/industry-models/fibo/, 1999–2024
work page 1999
-
[6]
CSM Lab School of Electrical Engineering and Computer Science (EECS) University of Ottawa. Contract ontology. https://sites.google.com/uottawa.ca/csmlab/research/contract-ontology, 2020
work page 2020
-
[7]
Thesaurus enhanced extraction of hohfeld’s relations from Spanish labour law
Patricia Martín-Chozas and Artem Revenko. Thesaurus enhanced extraction of hohfeld’s relations from Spanish labour law. In Sarra Ben Abbès, Rim Hantach, Philippe Calvez, Davide Buscaldi, Danilo Dessì, Mauro Dragoni, Diego Reforgiato Recupero, and Harald Sack, editors, Joint Proceedings of the 2nd International Workshop on Deep Learning meets Ontologies an...
work page 2021
-
[8]
Pproc, an ontology for transparency in public procurement
José Muñoz-Soro, Guillermo Esteban, Oscar Corcho, and Francisco Serón. Pproc, an ontology for transparency in public procurement. Semantic Web, 7:295–309, 03 2016
work page 2016
Show all 34 references
-
[9]
https://theybuyforyou.eu
They buy for you. https://theybuyforyou.eu
-
[10]
Nextprocurement
Barcelona supercomputing Center, Universidad Carlos III de Madrid, Universidad Politécnica de Madrid, Lo- caliData, Generalitat de Catalunya, Centre de Telecomunicacions i Tecnologies de la informació, Ayuntamiento de Madrid, and Ayuntamiento de Zaragoza. Nextprocurement. http...
-
[11]
Building the legal knowledge graph for smart compliance services in multilingual europe
Elena Montiel-Ponsoda, Víctor Rodríguez-Doncel, and Jorge Gracia. Building the legal knowledge graph for smart compliance services in multilingual europe. In In Proceedings of the 1st Workshop on Technologies for Regulatory Compliance., 2017
2017
-
[12]
Language Resources as Linked Data for the Legal Domain, pages 170–180
Patricia Martín-Chozas, Elena Montiel-Ponsoda, and Víctor Rodríguez-Doncel. Language Resources as Linked Data for the Legal Domain, pages 170–180. IOS Press BV , Amsterdam, 2019
2019
-
[13]
The atticus project
The Atticus Project. The atticus project. https://www.atticusprojectai.org, 2023
2023
-
[14]
CUAD: an expert-annotated NLP dataset for legal contract review
Dan Hendrycks, Collin Burns, Anya Chen, and Anya Ball. CUAD: an expert-annotated NLP dataset for legal contract review. CoRR, abs/2103.06268, 2021
2021 arXiv
-
[15]
Bert: Pre-training of deep bidirectional transformers for language understanding, 2019
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019
2019
-
[16]
Albert: A lite bert for self-supervised learning of language representations, 2020
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations, 2020
2020
-
[17]
Roberta: A robustly optimized bert pretraining approach, 2019
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019
2019
-
[18]
Deberta: Decoding-enhanced bert with disentan- gled attention, 2021
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentan- gled attention, 2021
2021
-
[20]
Joel Niklaus, Veton Matoshi, Matthias Sturmer, Ilias Chalkidis, and Daniel E. Ho. Multilegalpile: A 689gb multilingual legal corpus. ArXiv, abs/2306.02069, 2023
2023 arXiv
-
[21]
bge-m3-spa-law-qa
Little John AI. bge-m3-spa-law-qa. https://huggingface.co/littlejohn-ai/bge-m3-spa-law-qa, 2024
2024
-
[22]
Anotación de corpus lingüísticos: metodología utilizada en el instituto de ingeniería del conocimiento (iic)
Nuria Aldama, Marta Guerrero, Helena Montoro, and Doaa Samy. Anotación de corpus lingüísticos: metodología utilizada en el instituto de ingeniería del conocimiento (iic). https://www.iic.uam.es/whitepapers/ anotacion-corpus-linguisticos-metodologia-utilizada-iic/ , 2022
2022
-
[23]
Plataforma de contratación del sector público
Dirección General del Patrimonio del Estado del Ministerio de Hacienda y Administraciones Públicas. Plataforma de contratación del sector público. https://contrataciondelestado.es/wps/portal/plataforma, 2008. 11 3CEL: a Corpus of Legal Spanish Contract Clauses
2008
-
[24]
James R. Barlow. Ocrmypdf. https://ocrmypdf.readthedocs.io/en/latest/, 2023
2023
-
[25]
pdftotext 2.2.2
Jason Alan Palmer. pdftotext 2.2.2. https://pypi.org/project/pdftotext/, 2021
2021
-
[26]
pdfplumber 0.11.4
Jeremy Singer-Vine. pdfplumber 0.11.4. https://pypi.org/project/pdfplumber/, 2024
2024
-
[27]
Pypdf2 3.0.1
Mathieu Fenniak. Pypdf2 3.0.1. https://pypi.org/project/PyPDF2/, 2022
2022
-
[28]
Mmg/xlm-roberta-large-ner-spanish
dezzai. Mmg/xlm-roberta-large-ner-spanish. https://huggingface.co/MMG/ xlm-roberta-large-ner-spanish , 2023
2023
-
[29]
Natural Language Annotation for Machine Learning
James Pustejovsky and Amber Stubbs. Natural Language Annotation for Machine Learning. O’Reilly Media, Inc., 2012
2012
-
[30]
Prodigy: A new annotation tool for radically efficient machine teaching
Ines Montani and Matthew Honnibal. Prodigy: A new annotation tool for radically efficient machine teaching. Artificial Intelligence, 2018
2018
-
[31]
Unsupervised cross-lingual representation learning at scale
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. CoRR, abs/1911.02116, 2019
1911 arXiv
-
[32]
Spanish legalese language model and corpora, 2021
Asier Gutiérrez-Fandiño, Jordi Armengol-Estapé, Aitor Gonzalez-Agirre, and Marta Villegas. Spanish legalese language model and corpora, 2021
2021
-
[33]
Mel: Legal Spanish Language Model
David Betancur Sánchez, Nuria Aldama García, Álvaro Barbero Jiménez, Marta Guerrero Nieto, Patricia Marsà Morales, Nicolás Serrano Salas, Carlos García Hernán, Pablo Haya Coll, Elena Montiel Ponsoda, and Pablo Calleja Ibáñez. Mel: Legal Spanish Language Model. ArXiv, To be published
-
[34]
Finetune a span categorizer with bert and transformers
La Javaness R&D. Finetune a span categorizer with bert and transformers. https://lajavaness.medium. com/1-token-classification-vs-span-categorization-52a685e4674a , 2022
2022
-
[35]
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...
2020
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.