{"id":"36db53f9-ce08-4f38-9ab0-9ef84ab267be","arxiv_id":"2506.01938","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new aligned French-Italian NER benchmark for the wastewater domain is introduced, with baseline fine-tuning, annotation projection, and zero-shot LLM experiments.","lead":"The paper presents a new French-Italian corpus of wastewater and stormwater documents labeled for named entities, and tests several AI models on it. It also studies whether annotations can be transferred from French to Italian automatically.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Italian gold labels are produced by one annotator transposing French labels, with no inter-annotator agreement, so every reported F1 and the key projection-versus-monolingual comparison rest on unmeasured gold reliability.","rationale":"The reader identified the same weakest assumption: unmeasured reliability of the gold annotations, particularly the Italian gold produced by machine translation and manual transposition of French labels. I agree this is the most load-bearing concern because every quantitative claim in the paper, including the central baseline comparison, is evaluated against this gold. If the Italian labels are noisy or biased toward the source-language annotation, the reported F1 scores are not trustworthy as absolute numbers, and the comparison between monolingual Italian BERT (0.53) and projection (0.65) may be inflated by the shared label-transfer path. The concern is concrete and falsifiable: an independent re-annotation study with inter-annotator agreement would either validate the gold or reveal its weaknesses. Since the reader already marked the verdict CONDITIONAL on exactly this issue, my stress-test does not move the verdict; it strengthens the justification for the condition. I do not raise ad hominem or stylistic objections; the critique targets the evidential basis of the benchmark claim. I also note that the paper has some independent support in the form of a detailed corpus-construction protocol and standard evaluation tools (seqeval), but the absence of released data and gold-quality metrics is the limiting factor.","tokens_in":9042,"tokens_out":2625,"duration_ms":26504,"concrete_test":"Run an independent re-annotation study: sample about 20 documents (roughly 500 sentences) from the Italian corpus and have two fresh annotators, blind to the French source and to the projected labels, annotate directly from the Italian text using the starwars guidelines. Compute Cohen's kappa and per-tag agreement. Then re-run the best monolingual model (Italian BERT) and the best projection method (fine-tuned AWESoME-a, 50 epochs) on the subset where both annotators agree. If the projection advantage over monolingual training shrinks or flips, the gold-transposition artifact is confirmed; if it persists, the comparison in Tables 5 and 7 is robust to gold noise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that starwars is a reliable multilingual benchmark and that the reported baselines are trustworthy. That claim depends on the quality of the Italian gold standard, which is not established. Per Section 3.0.4, the Italian annotations were not produced by independent human annotation: after machine translation and student checking, a single student manually transposed the French labels onto the Italian text. No inter-annotator agreement, adjudication procedure, or gold-error analysis is reported. Section 5.1 itself acknowledges residual untranslated terms such as 'piquages' and inconsistent translations such as 'camera di allaccio' versus 'scatole di derivazione'. Because the Italian gold is derived from the French labels through a human transposition step, the projection experiments in Section 4.2 are partly circular: an automatic aligner that reproduces the same source-to-target label-transfer behavior will score well against a gold standard produced by the same kind of transfer, even when both are semantically wrong. The headline comparison (fine-tuned AWESoME-a 0.66/0.65 vs monolingual Italian BERT 0.53/0.50) could therefore reflect transposition artifacts rather than genuine model quality. The corpus is promised only in a later version, so external verification is currently impossible.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the 'starwars' corpus, a French-Italian sentence-aligned benchmark for named entity recognition in the wastewater and stormwater management domain, with 110 documents per language and a 14-tag annotation scheme. It reports three sets of experiments: fine-tuning French and Italian BERT-family models for monolingual annotation (CamemBERT reaching 0.63 F1 on French; ItalianBERT and GilBERTo reaching 0.53 and 0.50 F1 on Italian), projecting annotations between the aligned corpora with SimAlign and AWESoME (fine-tuned AWESoME reaching 0.66 F1 French-to-Italian and 0.65 F1 Italian-to-French), and zero-shot annotation projection with decoder-only LLMs of various sizes, where only Gemma3 and Llama3-70B approach competitive performance. The authors propose the corpus and these results as a baseline for future work and for multilingual extension of the benchmark.","tokens_in":9425,"tokens_out":4768,"duration_ms":43455,"significance":"If the benchmark and the baseline numbers are reliable, this is a useful contribution to a domain that lacks public multilingual resources: the paper addresses an under-served technical area, evaluates a broad set of methods, and is honest that the annotation experiments are not conclusive at this scale. The paper's strengths include direct empirical evaluation on a newly built corpus, aligned train/validation/test splits across languages, and the comparison of two alignment-based projection methods with both monolingual fine-tuning and zero-shot LLM projection. However, the reliability of the benchmark and of the central projection-vs-monolingual comparison rests on the quality of the Italian gold annotations, which is not established in the manuscript.","major_comments":[{"comment":"The Italian gold annotations are produced by a single student manually transposing French labels onto machine-translated text, with no inter-annotator agreement, adjudication, or error analysis reported. Since every reported F1 in Tables 5–8 and the headline projection-vs-monolingual comparison in Section 5.2 depend on this gold standard, its reliability is load-bearing; the paper should report IAA on at least a sample, or provide a separate independently annotated Italian gold set.","section":"§3.0.3–3.0.4"},{"comment":"The projection evaluation is partly circular: the Italian gold was itself created by transferring French labels to Italian, so an alignment-based projection method that reproduces that transfer will match the gold even when both the transfer and the gold are semantically wrong. Section 5.1 acknowledges residual untranslated terms and inconsistent translations (e.g., 'piquages', 'camera di allaccio' versus 'scatole di derivazione'), which makes this concern concrete. The projection F1 values (0.61–0.66) should be validated against an independently constructed Italian gold or, at minimum, analyzed for how many projection errors coincide with gold-transposition artifacts.","section":"§4.2 and §5.2"},{"comment":"The starwars corpus is not released in this version, and the details of how the 65 non-MeDo French documents were collected are not given; this prevents external verification of the claimed benchmark and makes the baseline numbers uncheckable. For a benchmark paper, the corpus, annotation guidelines, and annotation project configuration should be made available, or the paper should be explicitly framed as a preliminary report.","section":"§3.0.2 and Footnote 3"},{"comment":"The paper states that 'this experiment is not conclusive' and that all models improve with more training data, yet the abstract and conclusions describe the results as a 'reliable baseline'. Given 110 documents per language, three random seeds, and no significance testing or confidence intervals beyond standard deviations, the reliability claim is overstated; either qualify the conclusions or add per-tag error analysis and statistical tests for the differences in Table 7.","section":"§5.1"}],"minor_comments":[{"comment":"The phrase 'entity liking' appears to be a typo for 'entity linking'.","section":"Introduction"},{"comment":"The method name is written inconsistently as 'AWESoME', 'AweSome-a', and 'A WESoME-align'; please use a single canonical name throughout.","section":"Tables 4–8 and §4.2–5.2"},{"comment":"The note that corpus excerpts and annotation guidelines 'will be made available and cited in the final version' should be replaced with a link to the released resource, or the paper should be described as an extended abstract.","section":"Footnote 3"},{"comment":"The exclusion criteria for models and the failure modes ('Runs always fail', 'Unpredictable responses') are undefined; describe these in the text so the selection of the nine reported models is reproducible.","section":"§4.3 and Table 8"},{"comment":"The SemEval-2013 based analysis of missed versus incorrect annotations is said to be omitted 'due to space constraints'; since it is used to support the interpretation of the Italian results, it should be included in an appendix.","section":"§5.1"},{"comment":"Reference [13] is marked 'Submitted to'; if it is not yet published, this should be flagged or replaced by a citable version.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is best treated as a promising but early-stage benchmark report. The main risk is not the small scale, which the authors acknowledge, but the unverified quality of the Italian gold standard and the partly circular evaluation of annotation projection. I would encourage the editor to request that the authors provide the corpus and either an independent Italian annotation sample or a thorough gold-error analysis before the claims can be considered reliable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The starwars corpus is a real, new resource: sentence-aligned French-Italian documents with a 14-tag scheme adapted from RAEPA/MeDo, covering wastewater and stormwater management. That alone fills a gap, since existing multilingual NER benchmarks like MultiCoNER do not touch this domain. The paper also provides honest, reproducible baselines—CamemBERT, Italian BERT, GilBERTo—plus a systematic comparison of projection methods (SimAlign, AWESoME-a) and a handful of decoder-only LLMs. The authors are candid about scale: 110 documents per language, experiments not conclusive, and the need for more data. That is credit-earning.\n\nThe soft spots are real but not fatal. The biggest one is the Italian gold standard. Per Section 3.0.4, the Italian labels were produced by one student transposing the French labels onto the Italian text after machine translation and checking. No IAA, adjudication, or gold-error analysis is reported. The paper even acknowledges residual untranslated terms like \"piquages\" and inconsistent translations. That means every Italian F1 and the key projection-vs-monolingual comparison (0.66/0.65 vs 0.53/0.50) rests on unmeasured gold quality. The stress-test's circularity worry—that automatic projection may score well because it mimics the same label-transfer process that produced the gold—is plausible, though I would not call it load-bearing. A human transposing labels by hand is not the same as an alignment algorithm, and the gold was checked by students against the French source. But without IAA, the reader cannot tell.\n\nThe second issue is that the corpus is not released, only promised in the final version. For a benchmark paper, that is a significant gap; external verification is impossible now. There are also minor internal inconsistencies in Section 5.3: the text says only Gemma3 and Mistral use the correct 14 tags \"in the HTML task,\" but the table shows Llama3 70.6B using 14 tags in the IOB2 task for French→Italian. The LLM selection process (\"excluding those that underperform or exhibit stability issues\") is also under-specified.\n\nWho is this for? Researchers working on multilingual NER in technical or environmental domains, or on annotation projection for low-resource languages. It is worth a serious referee, and I would engage with it. My recommendation: send it out, but with a clear request for the corpus release, an IAA study, and clarification of the LLM details before acceptance.","headline":"A genuinely new French-Italian NER resource for a niche technical domain, but the missing inter-annotator agreement and the unreleased corpus mean the benchmark claims are not fully supported yet.","tokens_in":9800,"tokens_out":2618,"would_cite":false,"duration_ms":24713,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces the starwars corpus, a sentence-aligned French–Italian NER benchmark for wastewater and stormwater texts, and shows that cross-lingual annotation projection with a fine-tuned word aligner reaches 0.66 F1, beating the…","keywords":["named entity recognition","wastewater management","stormwater","French-Italian corpus","annotation projection","BERT fine-tuning","large language models","cross-lingual transfer"],"falsifier":"Have two hydrology experts independently re-annotate a random sample of the Italian corpus, measure how often the experts choose the same tags, and re-run the Italian BERT and projection evaluations against a corrected gold standard; if agreement is low, or if the corrected Italian F1 moves toward the French level and the projection advantage shrinks, the benchmark's cross-lingual comparisons were partly artifacts of gold-standard noise.","tokens_in":8887,"feed_emoji":"💧","tokens_out":11822,"duration_ms":105821,"temperature":0.7,"pith_summary":"This paper introduces the starwars corpus, a sentence-aligned French–Italian benchmark for named entity recognition (NER) in the wastewater and stormwater domain, with 14 entity types such as Structure, Pipeline, Spatial, and Fault. It establishes baseline NER performance by fine-tuning French and Italian BERT-family models, reporting 0.63 F1 for CamemBERT on French and 0.53 for Italian BERT on Italian. It then shows that annotation projection—transporting French labels onto Italian sentences using word aligners—reaches 0.66 F1 with a fine-tuned aligner, outperforming the monolingual Italian models and also the French baseline. The authors argue this makes projection a viable low-resource route for extending the benchmark to new languages, and that decoder-only LLM-based projection currently fails to beat the aligner baseline. If the benchmark is sound, it gives the water-management community a shared yardstick for entity extraction and future relation extraction.","feed_headline":"Projection beats an Italian-trained model on wastewater benchmark","feed_subtitle":"New French–Italian benchmark; labels projected from French hit 0.66 F1 with zero Italian training data.","key_machinery":"The central object is the starwars corpus itself: a parallel French–Italian collection whose sentences are aligned one-to-one, with 14 entity types drawn from the RAEPA geostandard and an application ontology. The argument runs through this alignment because each French sentence has an Italian translation, so labels can be transported from source to target by word alignment. The paper's projection baselines come from SimAlign and AWESoME-align, both using mBERT contextual embeddings at the sub-word level; AWESoME-a adds fine-tuning on downstream alignment tasks and is the mechanism that pushes projection above the Italian monolingual baselines. Decoder-only LLMs are prompted to perform the same projection in IOB2 or HTML-like tag format, but most fail to respect the fixed 14-tag scheme.","core_discovery":"The central claim is that the starwars corpus is a usable, sentence-aligned French–Italian benchmark: 110 documents per language, 3,103 aligned sentences, and 5,357 French and 5,324 Italian annotated entities across 14 domain-specific tags. On this benchmark, the paper's baselines show that fine-tuned CamemBERT reaches 0.63 F1 on French, while Italian BERT reaches 0.53 and GilBERTo 0.50; the gap is attributed to smaller Italian pre-training corpora and translation-induced label noise. The main positive result is that fine-tuned AWESoME-align projection from French to Italian reaches 0.66 F1 and from Italian to French 0.65 F1, outperforming the monolingual Italian models without any Italian training data; the authors interpret this as evidence that annotation projection can substitute for costly manual annotation when extending the corpus to new languages.","pith_inferences":["Beyond the paper: the reported French–Italian performance gap may be inflated by noise in the Italian gold standard; if untranslated terms such as 'piquages' remain, both the Italian model scores and the projection scores are measured against a moving target, and true model quality could differ.","Beyond the paper: the same projection pipeline is directly testable on the planned English extension; if English projection reaches French-level F1, that would confirm alignment-based transfer, rather than monolingual pre-training size, as the bottleneck for low-resource domain NER.","Beyond the paper: the overall F1 score hides tag-specific behavior; a per-tag breakdown of projection errors would reveal whether low-frequency tags such as Shape and Fault drive the misses, and whether tag imbalance explains the LLM failures.","Beyond the paper: the near-systematic failure of decoder-only LLMs to respect the fixed 14-tag set suggests the problem is instruction following rather than missing domain knowledge; constrained decoding or output-format validation could change that picture."],"forward_implications":["French–Italian NER in this domain now has a fixed reference: any new model can be compared against CamemBERT at 0.63 F1 and Italian BERT at 0.53 F1 on this corpus.","Annotation projection with a fine-tuned aligner can create usable Italian annotations from French at 0.65–0.66 F1 without any Italian training labels, so extending the corpus to a third language appears feasible with only translation and alignment effort.","The gains from more monolingual training data are not exhausted: all fine-tuned models improve monotonically up to 100% of the training set, so collecting more annotated documents should raise the baseline scores.","Decoder-only LLMs in the tested prompting regime are not yet a reliable projection tool: most do not consistently respect the 14-tag scheme, and none beats the aligner-based baseline across both language pairs.","Because the French and Italian documents and sentences are aligned, the corpus supports future work on relation extraction and entity linking, not only NER."],"supporting_citations":[{"why":"CamemBERT is the French monolingual baseline model, reaching 0.63 F1 on the French corpus.","marker":"[16]"},{"why":"Provides the BERT architecture and the mBERT embeddings used by the Italian models and by both word aligners.","marker":"[15]"},{"why":"SimAlign supplies the embedding-based word alignment method used for annotation projection.","marker":"[18]"},{"why":"AWESoME-align, with fine-tuning, yields the best projection scores of 0.66 and 0.65 F1.","marker":"[19]"},{"why":"The RAEPA geostandard is the source of the 14-tag annotation scheme.","marker":"[12]"},{"why":"The MeDo gold standard supplies 45 French documents that are re-annotated and extended in the starwars corpus.","marker":"[11]"},{"why":"Defines the annotation projection task and the HTML-like tag format used in the LLM experiments.","marker":"[17]"},{"why":"Defines the IOB2 annotation scheme and the CoNLL evaluation convention used for the entity-level scores.","marker":"[14]"}],"fun_headline_variants":["Projection without Italian data beats trained Italian NER on new benchmark","French-Italian wastewater NER: projection outperforms monolingual Italian models","New wastewater NER benchmark: French label projection beats Italian BERT","Zero Italian data? Projection still tops Italian NER on wastewater corpus"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The gold-standard labels for Italian are assumed to be correct, but they were created by machine translation, manual correction, and hand-transposition of the French annotations, with no reported agreement between annotators; if residual translation mistakes remain, every F1 number and the comparison between languages rests on uncertain ground.","fun_headline_variants_meta":{"raw":{"variants":["Projection without Italian data beats trained Italian NER on new benchmark","French-Italian wastewater NER: projection outperforms monolingual Italian models","New wastewater NER benchmark: French label projection beats Italian BERT","Zero Italian data? Projection still tops Italian NER on wastewater corpus"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000549,"raw_usage":{"total_tokens":2565,"prompt_tokens":831,"completion_tokens":1734,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":447,"completion_tokens_details":{"reasoning_tokens":1656}},"tokens_in":447,"tokens_out":1734,"duration_ms":15237,"temperature":1.0,"reasoning_tokens":1656,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:30:14.599059+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have two hydrology experts independently re-annotate a random sample of the Italian corpus, measure how often the experts choose the same tags, and re-run the Italian BERT and projection evaluations against a corrected gold standard; if agreement is low, or if the corrected Italian F1 moves toward the French level and the projection advantage shrinks, the benchmark's cross-lingual comparisons were partly artifacts of gold-standard noise.","supporting_citations":[{"cited_title":"Camembert: a tasty french language model","cited_arxiv_id":null,"evidence_quote":"CamemBERT is the French monolingual baseline model, reaching 0.63 F1 on the French corpus."},{"cited_title":"Bert: Pre-training of deep bidirectional transformers for language understanding","cited_arxiv_id":null,"evidence_quote":"Provides the BERT architecture and the mBERT embeddings used by the Italian models and by both word aligners."},{"cited_title":"Simalign: High quality word alignments without parallel training data using static and contextualized embeddings","cited_arxiv_id":null,"evidence_quote":"SimAlign supplies the embedding-based word alignment method used for annotation projection."},{"cited_title":"Word Alignment by Fine-tuning Embeddings on Parallel Corpora","cited_arxiv_id":null,"evidence_quote":"AWESoME-align, with fine-tuning, yields the best projection scores of 0.66 and 0.65 F1."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The RAEPA geostandard is the source of the 14-tag annotation scheme."},{"cited_title":"Gold Standard du projet MeDo, 2020","cited_arxiv_id":null,"evidence_quote":"The MeDo gold standard supplies 45 French documents that are re-annotated and extended in the starwars corpus."},{"cited_title":"T-Projection: High Quality Annotation Projection for Sequence Labeling Tasks","cited_arxiv_id":"2212.10548","evidence_quote":"Defines the annotation projection task and the HTML-like tag format used in the LLM experiments."},{"cited_title":"Tjong Kim Sang and Sabine Buchholz","cited_arxiv_id":null,"evidence_quote":"Defines the IOB2 annotation scheme and the CoNLL evaluation convention used for the entity-level scores."}],"review_version":1}