{"id":"7a35bfd2-a4ad-4dea-80ab-73b030025119","arxiv_id":"2502.00865","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The authors create the first Spanish-language multi-label dataset of potentially abusive clauses in Chilean terms of service and benchmark fine-tuned and few-shot language models on it.","lead":"This paper builds and tests a dataset of 50 Chilean online terms of service, annotated for potentially abusive clauses under Chilean law, and reports how well various AI language models detect and classify those clauses. It matters because it extends legal-language AI beyond English and EU law to Spanish-speaking jurisdictions, potentially giving consumers in Chile and Latin America a tool to flag unfair contract terms.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 3's classification splits are internally inconsistent: per-label counts sum to 295/127/181 for Illegal (total 603) while the table claims 335/127/181 (total 643), so the benchmark setup is not reproducible.","rationale":"The reader's weakest assumption concerns label correctness (no inter-annotator agreement). That is a legitimate and fundamental concern, but it is not demonstrated by the text; the described two-expert review process is a reasonable quality-control step, and the absence of kappa is a common limitation. The split inconsistency in Table 3 is concrete, internally verifiable, and directly undermines the benchmark evaluation, which is half of the paper's central contribution. A machine-checkable reconciliation of the dataset statistics is impossible from the paper alone because the text gives contradictory counts. This does not require external resources or assumptions about expert judgment; it is a clear reporting or methodological error. If the splits are wrong, the reported F1 numbers may be inflated or meaningless, yet the dataset could still be salvageable after correction and release. Therefore the appropriate disposition remains CONDITIONAL (major revision), same as the reader's verdict. I do not see grounds to reject outright because the dataset novelty and annotation effort are plausible, but the paper cannot be accepted as-is. My concern differs from the reader's weakest assumption, hence 'partial' agreement.","tokens_in":35291,"tokens_out":12247,"duration_ms":115699,"concrete_test":"Reconstruct the classification splits from the 5,209 annotated clauses by applying the Sechidis et al. (2011) iterative stratification exactly as described, using the multi-label annotations. Verify whether the per-label counts per split sum to the reported instance totals (335/127/181 for Illegal, etc.). If the sums do not reconcile, or if any test instance appears in the training set after the claimed oversampling, the reported F1 scores are not reproducible from the paper's description.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on a usable dataset and a trustworthy benchmark. Both are undermined by the split statistics in Table 3. For Illegal Classification, the per-label rows sum to 295 train, 127 val, 181 test (total 603 label assignments), exactly matching the detection split's abusive counts. Yet the table's 'Sum' row reports 335 train, 127 val, 181 test, total 643 instances. This 40-instance excess is attributed to 'selective oversampling' by iterative stratified sampling. But Sechidis et al. (2011) iterative stratification partitions the given instances; it does not create new ones. Oversampling would require duplicating instances, which risks train/test leakage and changes the evaluation. The same problem appears in Dark Classification (label sums 170/74/105 vs reported 170/75/105) and Gray Classification (label sums 185/79/113, total 377, vs reported 384). Additionally, the per-label totals in Table 3 (e.g., ltd 156, des risk 83) do not match the corpus-level counts in Table A1 (ltd 138, des risk 64). These inconsistencies mean the exact train/validation/test splits used to produce the reported macro-F1 and micro-F1 scores cannot be reconstructed from the paper, so the benchmark results are not verifiable. The dataset itself may still be a valuable contribution, but the evaluation claims are compromised.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a Spanish-language corpus of clauses from 50 Chilean terms of service, annotated by two legal experts with a 20-class scheme spanning illegal, dark, and gray potentially abusive clauses. It constructs six detection/classification benchmarks, fine-tunes a range of transformer models, and evaluates several multilingual LLMs in few-shot settings, reporting detection macro-F1 up to 79–89% and classification macro-F1 of 60–70%. The authors claim the first Spanish-language multi-label classification dataset for legal clauses and the first evaluation of RoBERTalex on a Spanish-language legal task.","tokens_in":35553,"tokens_out":7722,"duration_ms":74467,"significance":"If the corpus and benchmark are reliable, this is a genuinely useful contribution: it addresses a real gap in non-English, non-EU legal NLP, provides a substantial annotated resource grounded in Chilean consumer law, and offers a broad, seed-averaged comparison of fine-tuned and few-shot models, including a first task evaluation of RoBERTalex. The detailed error analysis is a further strength. However, the benchmark's trustworthiness is currently compromised by internal inconsistencies in the reported split statistics and label counts, and by ambiguities in how the detection negative sets were constructed. The dataset itself may still be valuable, but the evaluation numbers need to be corrected or supported by released splits before they can be relied upon.","major_comments":[{"comment":"The split statistics for the classification tasks are internally inconsistent, so the reported results in Tables 5–7 cannot be reconstructed from the paper. For Illegal Classification, the six label rows sum to 295 train / 127 val / 181 test (total 603), while the 'Sum' row reports 335 / 127 / 181 (total 643); for Dark Classification the label rows sum to 170 / 74 / 105 (total 349) but the 'Sum' row reports 170 / 75 / 105; and for Gray Classification the label rows sum to 185 / 79 / 113 (total 377) but the 'Sum' row reports 384, with the bfe row itself summing to 76 rather than 83. The stated explanation that iterative stratified sampling (Sechidis et al., 2011) 'selectively oversamples' to obtain 643 instances from 603 is not plausible, since that algorithm partitions the given instances and does not create new ones; if oversampling was deliberately performed, its details and safeguards against train/test leakage are not described. Moreover, per-label totals in Table 3 (e.g., ltd=156, des risk=83) disagree with Table A1 (ltd=138, des risk=64) and with the counts given in Section 3 (156 and 117). The exact data splits must be released, or the tables corrected, before the benchmark claims are verifiable.","section":"Section 3.7, Table 3"},{"comment":"The detection tasks appear to exclude clauses that are abusive in other categories from both the positive and negative sets. The 'ok' count is identical (4,165) in all three detection splits even though the corpus contains 1,044 clauses with at least one abusive label; if dark- or gray-labeled clauses are removed from the negative class for Illegal Detection, the binary task becomes 'illegal clause vs. completely clean clause,' which is easier than the realistic setting in which a detector must also distinguish illegal clauses from other abusive but not illegal clauses. The paper should state explicitly whether other-category abusive clauses are included as negatives; if they are excluded, the detection experiments should be re-run on the full corpus, or the reduced setting should be justified as the intended task definition.","section":"Section 3.7, Table 2"},{"comment":"The gold standard rests entirely on the joint work of two in-house Chilean legal experts, and the authors state that they do not report agreement measures 'due to the expert review.' Since every F1 score in Tables 5–7 is computed against these labels, the absence of any inter-annotator agreement statistic, adjudication audit, or independent verification leaves the reliability of the labels unsupported. At minimum, the authors should report agreement on a held-out subset, provide an audit trail, or release the detailed annotation guidelines, and they should resolve the apparent discrepancies in label counts (e.g., ltd and des risk) that suggest the version of the data used in the experiments differs from the version summarized in Table A1.","section":"Section 3.7"}],"minor_comments":[{"comment":"The text says 'Figure 7 shows classification reports,' but the material is presented as Table 7; the cross-reference should be corrected.","section":"Section 5.3"},{"comment":"The text refers to 'Gwen2.5 72b' where the model name should be 'Qwen2.5 72b.'","section":"Section 5.2"},{"comment":"The abstract and introduction emphasize 'four categories and 20 classes,' but only three categories are used for machine learning because the formal-requirements category is excluded in Section 3.2; this should be stated more prominently to avoid overstating the scope of the annotated and predicted scheme.","section":"Abstract and Section 3"},{"comment":"The caption describes the classification tasks as 'multi-class classification,' while the text and the multi-label formulation in Section 4.1 describe them as multi-label tasks; the terminology should be made consistent.","section":"Table 3 caption"}],"recommendation":"major_revision","confidential_remarks":"The dataset and the broad evaluation are potentially solid contributions, but the experimental section currently contains arithmetic inconsistencies that a careful reader cannot resolve, and the construction of the detection negatives is under-specified. I would condition acceptance on release of the exact data splits and code, corrected statistics, and a clear statement of the detection task definition. The lack of an explicit data-availability statement is also a concern for a paper whose central contribution is a dataset."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the dataset is the contribution, and it's a real one. First Spanish-language multi-label legal clause corpus from a non-EU jurisdiction, 50 Chilean ToS, 20-class scheme grounded in the LPC. The model sweep is thorough: fine-tuned encoders plus few-shot LLMs, 426 experiments, and a first legal evaluation of RoBERTalex. That part is worth having.\n\nBut the evaluation section has a load-bearing reproducibility problem. Table 3's per-label rows sum to 295/127/181 for Illegal (total 603), while the printed Sum row says 335/127/181 (total 643). The paper attributes the extra 40 instances to 'selective oversampling' by Sechidis iterative stratification. That method partitions existing instances; it doesn't synthesize new ones. The same mismatch shows up in Dark (label sums 170/74/105 vs reported 170/75/105) and Gray (label sums total 377 vs reported 384). Worse, per-label totals in Table 3 don't match Table A1: ltd is 156 in Table 3 but 138 in Table A1, des risk 83 vs 64, ter 10 vs 8. And the running text says 117 des risk clauses. Three numbers for the same quantity. So the exact train/val/test splits cannot be reconstructed, and every F1 score in Tables 5 and 6 is anchored to splits we can't verify.\n\nTwo other soft spots, both correctable. There's no data or code link, and no inter-annotator agreement metric; the authors say expert review 'does away with' kappa, which isn't standard practice. The detection negative sets exclude clauses that are abusive in the other categories, which inflates detection ease relative to a realistic deployment where you don't know which clauses are abusive.\n\nNone of this kills the corpus as a contribution. The legal analysis is careful, the annotation scheme is documented, and the performance ranges reported are plausible for this kind of task. But the paper as submitted cannot be used as a benchmark source until the numbers are fixed and the data is out. A serious referee should ask for exactly that: corrected tables, a link to the dataset and code, and an IAA figure or a clear justification for its absence.\n\nI'd send it to review: the dataset is novel and the evaluation breadth is useful. But I'd hold off citing the reported F1 scores until the splits are verifiable.","headline":"A genuinely useful new Chilean ToS corpus and a broad model benchmark, but the printed split tables don't add up and the data isn't released; the benchmark numbers as-is are not verifiable.","tokens_in":36051,"tokens_out":5364,"would_cite":false,"duration_ms":46403,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces a Spanish-language dataset of 5,209 clauses from 50 Chilean Terms of Service and shows that transformer-based models can detect potentially abusive clauses with macro-F1 scores up to 89%.","keywords":["consumer protection law","abusive clauses","terms of service","Spanish-language NLP","legal NLP","multi-label classification","Chilean law","transformer fine-tuning"],"falsifier":"An independent pair of Chilean legal experts, blind to the original labels, could re-annotate a random subset of the 5,209 clauses using the same scheme; if the new annotation disagrees with the original labels on a substantial fraction of clauses, or if models trained on one annotation fail badly when evaluated against the other, the reported F1 scores would be measuring a noisy gold standard rather than the true clause categories.","tokens_in":35095,"feed_emoji":"⚖️","tokens_out":5880,"duration_ms":48870,"temperature":0.7,"pith_summary":"This paper introduces the first Spanish-language dataset of legal clauses annotated for potentially abusive content, built from 50 online Terms of Service used in Chile. The annotation scheme distinguishes illegal, dark, and gray clauses across 20 classes, grounded in Chilean consumer protection law. The paper benchmarks fine-tuned transformer models and few-shot large language models on detection and classification tasks, reporting macro-F1 scores of 79–89% for detection and 60–70% for classification. If the approach holds up, consumers and regulators in Chile and Latin America could automatically screen contracts for abusive terms.","feed_headline":"Detecting abusive clauses in Chilean terms of service hits 89% F1","feed_subtitle":"New Spanish-language dataset benchmarks abusive-clause detection and classification in Chilean ToS.","key_machinery":"The central object is the annotation scheme: four categories of potentially abusive clauses under Chilean law — formal-requirement infringements, illegal clauses, dark clauses, and gray clauses — with the last three expanded into 20 labeled classes (six illegal, six dark, eight gray) tied to specific legal sources such as Article 16 of the Consumer Protection Law. The scheme carries the argument because it converts an open-ended legal judgment into a fixed set of multi-label classes that can be learned, and it is applied to 5,209 clauses segmented from 50 ToS, with six derived datasets for binary detection and multi-label classification.","core_discovery":"The paper's central claim is that a carefully designed annotation scheme, applied by Chilean legal experts to 50 Spanish-language ToS, yields a corpus on which transformer-based models can reliably detect potentially abusive clauses and classify them into meaningful legal categories. It reports that the best models detect abusive clauses with macro-F1 scores up to 89% and micro-F1 scores up to 96%, while the harder multi-label classification task reaches macro-F1 scores of 60–70% and micro-F1 scores of 64–80%. The work positions this dataset as the first Spanish-language legal-clause classification resource and the first to apply Chilean law, and it shows that language- and domain-specific pre-training materially affects performance, with few-shot prompting of large models remaining behind fine-tuning on most tasks but competitive on the most ambiguous gray-clause category.","pith_inferences":["If the dataset is adopted, a natural extension is to expand beyond the 50 companies to cover more sectors or other Latin American jurisdictions with similar consumer-protection laws, testing how well the models transfer.","The exclusion of formal-requirement violations (font size, language, printability) leaves a gap that could be closed cheaply with rule-based parsers; a combined system would cover more of the legal risk surface.","A testable hypothesis implicit in the paper is that inter-annotator disagreement concentrates in the gray-clause classes; measuring agreement per class could reveal which categories truly need more examples or clearer guidelines.","Because the labels were agreed by two in-house experts without a reported agreement metric, an independent audit of a random sample of clauses would measure the gold standard's stability."],"forward_implications":["If detection is as reliable as reported, a screening tool could flag potentially abusive clauses in a ToS before a consumer accepts it, lowering the information asymmetry.","The dataset gives Spanish-language legal NLP a benchmark, so future models can be compared on a non-EU, non-English task.","The distinction between illegal, dark, and gray clauses means the system can tell users which clauses are presumed abusive by law and which require a court's judgment.","The finding that few-shot prompting lags fine-tuning on most tasks but helps on ambiguous gray clauses suggests a hybrid workflow: fine-tune for detection and prompt large models for the hardest classification cases.","The reported evaluation of the Spanish legal-domain model RoBERTalex on a Spanish legal task provides a reference result for that model."],"supporting_citations":[{"why":"Supplies the original unfair-ToS dataset and detection methodology that this corpus extends.","marker":"(Lippi et al., 2019)"},{"why":"Extends the European dataset and provides memory-network baselines that inform the classification approach.","marker":"(Ruggeri et al., 2022)"},{"why":"Provides the multilingual unfair-ToS corpus that motivates cross-language transfer and annotation comparison.","marker":"(Drawzeski et al., 2021)"},{"why":"Contributes the fine-tuning code base and baseline setup the experiments build on.","marker":"(Chalkidis et al., 2022)"},{"why":"Introduces RoBERTalex, the Spanish legal-domain model evaluated here for the first time on a Spanish legal task.","marker":"(Gutiérrez-Fandiño et al., 2021)"},{"why":"Provides BETO, the Spanish pre-trained model that achieves the best detection results.","marker":"(Cañete et al., 2023)"},{"why":"Supplies the iterative stratified sampling method used to create the multi-label dataset splits.","marker":"(Sechidis et al., 2011)"},{"why":"Grounds the annotation scheme in Chilean doctrine on adhesion contracts and abusive clauses.","marker":"(de la Maza Gazmuri, 2003)"},{"why":"Provides the official Chilean definition of abusive clauses used to shape the categorization.","marker":"(SERNAC, 2021)"}],"fun_headline_variants":["AI flags abusive clauses in Chilean ToS with up to 89% F1","First Spanish-language dataset benchmarks abusive ToS clauses","Chilean ToS abuse detection hits 89% macro-F1","New NLP tool detects unfair terms in Chilean contracts","Spanish legal AI identifies abusive clauses in Chile's ToS"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The gold-standard labels are assumed correct: the two Chilean legal experts who annotated the corpus reached consensus without any reported inter-annotator agreement metric, so systematic errors in their legal interpretation or clause segmentation would propagate into every reported score.","fun_headline_variants_meta":{"raw":{"variants":["AI flags abusive clauses in Chilean ToS with up to 89% F1","First Spanish-language dataset benchmarks abusive ToS clauses","Chilean ToS abuse detection hits 89% macro-F1","New NLP tool detects unfair terms in Chilean contracts","Spanish legal AI identifies abusive clauses in Chile's ToS"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000221,"raw_usage":{"total_tokens":1461,"prompt_tokens":967,"completion_tokens":494,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":583,"completion_tokens_details":{"reasoning_tokens":409}},"tokens_in":583,"tokens_out":494,"duration_ms":5415,"temperature":1.0,"reasoning_tokens":409,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T17:25:27.816030+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An independent pair of Chilean legal experts, blind to the original labels, could re-annotate a random subset of the 5,209 clauses using the same scheme; if the new annotation disagrees with the original labels on a substantial fraction of clauses, or if models trained on one annotation fail badly when evaluated against the other, the reported F1 scores would be measuring a noisy gold standard rather than the true clause categories.","supporting_citations":[{"cited_title":"CLAUDETTE: an Automated Detector of Potentially Unfair Clauses in Online Terms of Service","cited_arxiv_id":"1805.01217","evidence_quote":"Supplies the original unfair-ToS dataset and detection methodology that this corpus extends."},{"cited_title":", Lagioia , F","cited_arxiv_id":null,"evidence_quote":"Extends the European dataset and provides memory-network baselines that inform the classification approach."},{"cited_title":": Contratos por adhesi \\'o n y cl \\'a usulas abusivas Por qu \\'e el Estado y no solamente el mercado? Revista chilena de derecho privado (1), 109--148 (2003) botherref","cited_arxiv_id":null,"evidence_quote":"Grounds the annotation scheme in Chilean doctrine on adhesion contracts and abusive clauses."},{"cited_title":"Aprueba Circular Interpretativa Sobre Criterios de Equidad En Las Estipulaciones Contenidas En Contratos de Adhesi \\'o n de Consumo","cited_arxiv_id":null,"evidence_quote":"Provides the official Chilean definition of abusive clauses used to shape the categorization."}],"review_version":1}