REVIEW 4 major objections 6 minor 26 references
Novel Benchmark for NER in the Wastewater and Stormwater Domain
T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read This paper introduces the starwars corpus, a sentence-aligned French–Italian NER benchmark for wastewater and stormwater texts, and shows that cross-lingual annotation projection with a fine-tuned word aligner reaches 0.66 F1, beating the…
desk verdict A genuinely new French-Italian NER resource for a niche technical domain, but the missing inter-annotator agreement and the unreleased corpus mean the benchmark claims are not fully supported yet. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the starwars corpus itself: a parallel French–Italian collection whose sentences are aligned one-to-one, with 14 entity types drawn from the RAEPA geostandard and an application ontology. The argument runs through this alignment because each French sentence has an Italian translation, so labels can be transported from source to target by word alignment. The paper's projection baselines come from SimAlign and AWESoME-align, both using mBERT contextual embeddings at the sub-word level; AWESoME-a adds fine-tuning on downstream alignment tasks and is the mechanism that pushes projection above the Italian monolingual baselines. Decoder-only LLMs are prompted to perform the same projection in IOB2 or HTML-like tag format, but most fail to respect the fixed 14-tag scheme.
What would settle it
Have two hydrology experts independently re-annotate a random sample of the Italian corpus, measure how often the experts choose the same tags, and re-run the Italian BERT and projection evaluations against a corrected gold standard; if agreement is low, or if the corrected Italian F1 moves toward the French level and the projection advantage shrinks, the benchmark's cross-lingual comparisons were partly artifacts of gold-standard noise.
Extended reading notes
Core claim
The central claim is that the starwars corpus is a usable, sentence-aligned French–Italian benchmark: 110 documents per language, 3,103 aligned sentences, and 5,357 French and 5,324 Italian annotated entities across 14 domain-specific tags. On this benchmark, the paper's baselines show that fine-tuned CamemBERT reaches 0.63 F1 on French, while Italian BERT reaches 0.53 and GilBERTo 0.50; the gap is attributed to smaller Italian pre-training corpora and translation-induced label noise. The main positive result is that fine-tuned AWESoME-align projection from French to Italian reaches 0.66 F1 and from Italian to French 0.65 F1, outperforming the monolingual Italian models without any Italian training data; the authors interpret this as evidence that annotation projection can substitute for costly manual annotation when extending the corpus to new languages.
Load-bearing premise
The gold-standard labels for Italian are assumed to be correct, but they were created by machine translation, manual correction, and hand-transposition of the French annotations, with no reported agreement between annotators; if residual translation mistakes remain, every F1 number and the comparison between languages rests on uncertain ground.
Editorial extensions
If this is right
- French–Italian NER in this domain now has a fixed reference: any new model can be compared against CamemBERT at 0.63 F1 and Italian BERT at 0.53 F1 on this corpus.
- Annotation projection with a fine-tuned aligner can create usable Italian annotations from French at 0.65–0.66 F1 without any Italian training labels, so extending the corpus to a third language appears feasible with only translation and alignment effort.
- The gains from more monolingual training data are not exhausted: all fine-tuned models improve monotonically up to 100% of the training set, so collecting more annotated documents should raise the baseline scores.
- Decoder-only LLMs in the tested prompting regime are not yet a reliable projection tool: most do not consistently respect the 14-tag scheme, and none beats the aligner-based baseline across both language pairs.
- Because the French and Italian documents and sentences are aligned, the corpus supports future work on relation extraction and entity linking, not only NER.
Reading between the lines
- Beyond the paper: the reported French–Italian performance gap may be inflated by noise in the Italian gold standard; if untranslated terms such as 'piquages' remain, both the Italian model scores and the projection scores are measured against a moving target, and true model quality could differ.
- Beyond the paper: the same projection pipeline is directly testable on the planned English extension; if English projection reaches French-level F1, that would confirm alignment-based transfer, rather than monolingual pre-training size, as the bottleneck for low-resource domain NER.
- Beyond the paper: the overall F1 score hides tag-specific behavior; a per-tag breakdown of projection errors would reveal whether low-frequency tags such as Shape and Fault drive the misses, and whether tag imbalance explains the LLM failures.
- Beyond the paper: the near-systematic failure of decoder-only LLMs to respect the fixed 14-tag set suggests the problem is instruction following rather than missing domain knowledge; constrained decoding or output-format validation could change that picture.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces the 'starwars' corpus, a French-Italian sentence-aligned benchmark for named entity recognition in the wastewater and stormwater management domain, with 110 documents per language and a 14-tag annotation scheme. It reports three sets of experiments: fine-tuning French and Italian BERT-family models for monolingual annotation (CamemBERT reaching 0.63 F1 on French; ItalianBERT and GilBERTo reaching 0.53 and 0.50 F1 on Italian), projecting annotations between the aligned corpora with SimAlign and AWESoME (fine-tuned AWESoME reaching 0.66 F1 French-to-Italian and 0.65 F1 Italian-to-French), and zero-shot annotation projection with decoder-only LLMs of various sizes, where only Gemma3 and Llama3-70B approach competitive performance. The authors propose the corpus and these results as a baseline for future work and for multilingual extension of the benchmark.
Significance. If the benchmark and the baseline numbers are reliable, this is a useful contribution to a domain that lacks public multilingual resources: the paper addresses an under-served technical area, evaluates a broad set of methods, and is honest that the annotation experiments are not conclusive at this scale. The paper's strengths include direct empirical evaluation on a newly built corpus, aligned train/validation/test splits across languages, and the comparison of two alignment-based projection methods with both monolingual fine-tuning and zero-shot LLM projection. However, the reliability of the benchmark and of the central projection-vs-monolingual comparison rests on the quality of the Italian gold annotations, which is not established in the manuscript.
major comments (4)
- [§3.0.3–3.0.4] The Italian gold annotations are produced by a single student manually transposing French labels onto machine-translated text, with no inter-annotator agreement, adjudication, or error analysis reported. Since every reported F1 in Tables 5–8 and the headline projection-vs-monolingual comparison in Section 5.2 depend on this gold standard, its reliability is load-bearing; the paper should report IAA on at least a sample, or provide a separate independently annotated Italian gold set.
- [§4.2 and §5.2] The projection evaluation is partly circular: the Italian gold was itself created by transferring French labels to Italian, so an alignment-based projection method that reproduces that transfer will match the gold even when both the transfer and the gold are semantically wrong. Section 5.1 acknowledges residual untranslated terms and inconsistent translations (e.g., 'piquages', 'camera di allaccio' versus 'scatole di derivazione'), which makes this concern concrete. The projection F1 values (0.61–0.66) should be validated against an independently constructed Italian gold or, at minimum, analyzed for how many projection errors coincide with gold-transposition artifacts.
- [§3.0.2 and Footnote 3] The starwars corpus is not released in this version, and the details of how the 65 non-MeDo French documents were collected are not given; this prevents external verification of the claimed benchmark and makes the baseline numbers uncheckable. For a benchmark paper, the corpus, annotation guidelines, and annotation project configuration should be made available, or the paper should be explicitly framed as a preliminary report.
- [§5.1] The paper states that 'this experiment is not conclusive' and that all models improve with more training data, yet the abstract and conclusions describe the results as a 'reliable baseline'. Given 110 documents per language, three random seeds, and no significance testing or confidence intervals beyond standard deviations, the reliability claim is overstated; either qualify the conclusions or add per-tag error analysis and statistical tests for the differences in Table 7.
minor comments (6)
- [Introduction] The phrase 'entity liking' appears to be a typo for 'entity linking'.
- [Tables 4–8 and §4.2–5.2] The method name is written inconsistently as 'AWESoME', 'AweSome-a', and 'A WESoME-align'; please use a single canonical name throughout.
- [Footnote 3] The note that corpus excerpts and annotation guidelines 'will be made available and cited in the final version' should be replaced with a link to the released resource, or the paper should be described as an extended abstract.
- [§4.3 and Table 8] The exclusion criteria for models and the failure modes ('Runs always fail', 'Unpredictable responses') are undefined; describe these in the text so the selection of the nine reported models is reproducible.
- [§5.1] The SemEval-2013 based analysis of missed versus incorrect annotations is said to be omitted 'due to space constraints'; since it is used to support the interpretation of the Italian results, it should be included in an appendix.
- [References] Reference [13] is marked 'Submitted to'; if it is not yet published, this should be flagged or replaced by a citable version.
Circularity Check
No significant circularity: the paper reports direct held-out empirical evaluations on a newly constructed corpus; its projection baselines are evaluated against manually checked gold annotations rather than against fitted parameters.
full rationale
The paper's claims are empirical benchmark results, not derivations. The French and Italian corpora are annotated independently enough: Italian text was machine-translated, checked by two students, and annotations manually transposed from French by a second student (Sections 3.0.3-3.0.4). The monolingual NER experiments use a fixed 70/10/20 split with aligned documents and held-out test sets across all training sizes (Section 4.1), so the reported CamemBERT, ItalianBERT, and GilBERTo F1 scores are standard held-out measurements, not fits renamed as predictions. The annotation-projection experiments compare aligner outputs against the manually produced Italian gold (Section 4.2); although the gold labels are derived from French by human transposition, the automatic aligners do not use those gold labels as training targets, and the projection task is defined precisely as reproducing the source-to-target transfer, so this is not circularity by construction. The paper is transparent about limitations: it acknowledges the need for more extensive evaluation (Section 4.3), notes residual untranslated terms and inconsistent translations (Section 5.1), and states that corpus excerpts and guidelines will be made available in the final version (footnote 3). The absence of inter-annotator agreement statistics is a correctness and reliability risk, not a circularity step. Self-citations to the MeDo project ([10], [11], [13]) provide corpus provenance and annotation-scheme inspiration but do not carry a load-bearing derivation. Overall, the benchmark and baselines are self-contained empirical contributions, so the circularity score is 0.
Assumptions & free parameters
assumptions (3)
- domain assumption The RAEPA-derived 14-tag schema is a suitable and sufficient annotation scheme for NER in the wastewater domain.
- domain assumption The machine-translated and manually corrected Italian corpus is a faithful translation that preserves sentence alignment and entity span boundaries.
- domain assumption The seqeval default evaluation configuration implementing the CoNLL scoring script is the appropriate metric for entity-level NER evaluation.
Cite this review
Pith. "Pith review of Novel Benchmark for NER in the Wastewater and Stormwater Domain." pith.science (2026). https://pith.science/paper/QNEQXND7
@misc{pith2026250601938,
author = {Pith},
title = {Pith review of: Novel Benchmark for NER in the Wastewater and Stormwater Domain},
year = {2026},
howpublished = {\url{https://pith.science/paper/QNEQXND7}},
note = {Machine review of arXiv:2506.01938}
}
read the original abstract
Effective wastewater and stormwater management is essential for urban sustainability and environmental protection. Extracting structured knowledge from reports and regulations is challenging due to domainspecific terminology and multilingual contexts. This work focuses on domain-specific Named Entity Recognition (NER) as a first step towards effective relation and information extraction to support decision making. A multilingual benchmark is crucial for evaluating these methods. This study develops a French-Italian domain-specific text corpus for wastewater management. It evaluates state-of-the-art NER methods, including LLM-based approaches, to provide a reliable baseline for future strategies and explores automated annotation projection in view of an extension of the corpus to new languages.
Reference graph
Works this paper leans on
-
[1]
BioCreative V CDR task corpus: a resource for chemical disease relation extraction
Jiao Li et al. BioCreative V CDR task corpus: a resource for chemical disease relation extraction. Database, 2016, 2016
work page 2016
-
[2]
Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. In Proc. of the 2018 Conf. on Empirical Methods in Natural Language Processing , pages 3219–3232, 2018
work page 2018
-
[3]
Nora Abdelmageed et al. BiodivNERE: Gold standard corpora for named entity recognition and relation extraction in the biodiversity domain. Biodiversity Data Journal , 10, 2022
work page 2022
-
[4]
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Jinhyuk Lee et al. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234–1240, 2019
work page 2019
-
[5]
BioBERT Based Named Entity Recog- nition in Electronic Medical Record
Xin Yu, Wenshen Hu, Sha Lu, Xiaoyan Sun, and Zhenming Yuan. BioBERT Based Named Entity Recog- nition in Electronic Medical Record. In 10th Int’l. Conf. on Information Technology in Medicine and Education (ITME), pages 49–52, 2019
work page 2019
-
[6]
Miloˇ s Koˇ sprdi´ c, Nikola Prodanovi´ c, Adela Ljaji´ c, Bojana Baˇ saragin, and Nikola Miloˇ sevi´ c. From zero to hero: Harnessing transformers for biomedical named entity recognition in zero- and few-shot contexts. Artificial Intelligence in Medicine , 156, 2024
work page 2024
-
[7]
Watergpt: Training a large language model to become a hydrology expert
Yi Ren, Tianyi Zhang, Xurong Dong, Weibin Li, Zhiyang Wang, Jie He, Hanzhi Zhang, and Licheng Jiao. Watergpt: Training a large language model to become a hydrology expert. Water, 16 (21), 2024
work page 2024
-
[8]
MultiCoNER: A Large- scale Multilingual dataset for Complex Named Entity Recognition
Shervin Malmasi, Anjie Fang, Besnik Fetahu, Sudipta Kar, and Oleg Rokhlenko. MultiCoNER: A Large- scale Multilingual dataset for Complex Named Entity Recognition. arXiv, 2022
work page 2022
Show all 26 references
-
[9]
Multilingual large language models: A systematic survey
Shaolin Zhu, Supryadi, Shaoyang Xu, Haoran Sun, Leiyu Pan, Menglong Cui, Jiangcun Du, Renren Jin, Ant´ onio Branco, and Deyi Xiong. Multilingual large language models: A systematic survey. arXiv, 2024. 9
2024
-
[10]
WEIR- P: An Information Extraction Pipeline for the Wastewater Domain
Nan´ ee Chahinian, Thierry Bonnabaud La Bruy` ere, Francesca Frontini, Carole Delenne, Julien Marin, Rachel Panckhurst, Mathieu Roche, Lucile Sautot, Laurent Deruelle, and Maguelonne Teisseire. WEIR- P: An Information Extraction Pipeline for the Wastewater Domain. In Proc. of ...
2021
-
[11]
Gold Standard du projet MeDo, 2020
Nan´ ee Chahinian et al. Gold Standard du projet MeDo, 2020
2020
-
[12]
G´ eostandard R´ eseaux d’adduction d’eau potable et d’assainissement (RAEPA) v1.2, 2019
2019
-
[13]
An ontology based data access framework for sewer network data
Batoul Haydar, Claude Pasquier, Umberto Straccia, and Nan´ ee Chahinian. An ontology based data access framework for sewer network data. Submitted to Automation in Construction , 2024
2024
-
[14]
Tjong Kim Sang and Sabine Buchholz
Erik F. Tjong Kim Sang and Sabine Buchholz. Introduction to the CoNLL-2000 Shared Task Chunking. In Fourth Conf. on Computational Natural Language Learning and the Second Learning Language in Logic Workshop, 2000
2000
-
[15]
Bert: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin et al. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proc. of the Conf. of the North American chapter of the ACL: human language technologies, Vol. 1 , pages 4171–4186, 2019
2019
-
[16]
Camembert: a tasty french language model
Louis Martin et al. Camembert: a tasty french language model. In Proc. of the 58th Annual Meeting of the ACL. ACL, 2020
2020
-
[17]
T-projection: High quality annotation projection for sequence labeling tasks
Iker Garc ´ ıa-Ferrero, Rodrigo Agerri, and German Rigau. T-projection: High quality annotation projection for sequence labeling tasks. arXiv preprint arXiv:2212.10548 , 2022
2022 arXiv
-
[18]
Simalign: High quality word alignments without parallel training data using static and contextualized embeddings
Masoud Jalili Sabet et al. Simalign: High quality word alignments without parallel training data using static and contextualized embeddings. In Findings of the ACL: EMNLP 2020 , 2020
2020
-
[19]
Word Alignment by Fine-tuning Embeddings on Parallel Corpora
Zi-Yi Dou and Graham Neubig. Word Alignment by Fine-tuning Embeddings on Parallel Corpora. In Proc. of the 16th Conf. of the European Chapter of the ACL: Main Volume , 2021
2021
-
[20]
Enriching word vectors with subword information
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. Trans. of the ACL , 5, 2017
2017
-
[21]
Unsupervised cross-lingual representation learning at scale
Alexis Conneau et al. Unsupervised cross-lingual representation learning at scale. In Proc. of the 58th Annual Meeting of the ACL , 2020
2020
-
[22]
Llama 2: Open foundation and fine-tuned chat models
Hugo Touvron et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023
2023 arXiv
-
[23]
The llama 3 herd of models
Aaron Grattafiori et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 , 2024
2024 arXiv
-
[24]
Phi-4 technical report
Marah Abdin et al. Phi-4 technical report. arXiv preprint arXiv:2412.08905 , 2024
2024 arXiv
-
[25]
seqeval: A python framework for sequence labeling evaluation, 2018
Hiroki Nakayama. seqeval: A python framework for sequence labeling evaluation, 2018
2018
-
[26]
SemEval-2013 task 9 : Extraction of drug-drug interactions from biomedical texts
Isabel Segura-Bedmar et al. SemEval-2013 task 9 : Extraction of drug-drug interactions from biomedical texts. In Proc. of the 7th International Workshop on Semantic Evaluation , pages 341–350, 2013. 10
2013
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.