{"id":"98035c2d-0d18-4d53-a647-37dc08cf36a3","arxiv_id":"2412.10918","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"Small fine-tuned NER models for de-identifying health information in eight languages are claimed to beat GPT-4o, but only the English result is benchmarked externally.","lead":"Researchers trained small AI models to remove private health information from medical text in eight languages, using large language models to help generate training data. The models score highly on their own test sets, but the non-English tests come from the same automated pipeline, so the results are not independently verified.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Multilingual F1 claims rest on a 20% split of the same self-generated translated/synthetic corpus; without external non-English ground truth, the paper's central 'across languages' result is unverified.","rationale":"The paper's central novelty is the claim of accurate multilingual de-identification; without it, Part 2 is at best an incremental English-only i2b2 result. The reader's weakest assumption correctly identifies the Achilles heel: the 80-20 split is applied to a corpus that is itself the output of the authors' translation and synthetic-generation pipeline, so the test set shares the same artifacts, formatting, label noise, and entity distribution as training. In such a setting, high F1 is expected and provides no evidence about generalization to genuine clinical documents. This is not merely a preference for external benchmarks; it is an internal validity issue, because the paper's own methods describe the evaluation as self-referential. The English i2b2 portion is more credible because the test set is external, but the paper's headline 'across languages' cannot be justified by the reported non-English evaluations. The mismatch between the abstract's 'f1-micro score averages' and the macro-avg values in Tables 5-6 is a further sign that the reported headline metrics have not been carefully tied to the results, but the decisive problem is the absence of external non-English validation. For these reasons, I agree with the reader's REJECT verdict and recommend no change.","tokens_in":15534,"tokens_out":16283,"duration_ms":142258,"concrete_test":"Construct or obtain an external set of real clinical notes in one non-English language (e.g., German or Turkish) with PHI spans annotated independently by two native clinical annotators using the paper's i2b2-style label guidelines. Once the promised codebase is released, run the corresponding small model on this held-out set and compute strict chunk-level macro-F1. If the macro-F1 differs by more than 0.05 from the paper's reported value, or falls below 0.90, the multilingual generalization claim is not supported. Repeat for at least one more language if feasible; until the authors release model and data, this check cannot be performed, which is itself a reproducibility failure.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 states that most non-English datasets were generated by translating the English corpus with the authors' own medical translation models [30] and through LLM-assisted synthetic data generation; Section 3.2.2 says the 0.20 test split is taken from that same dataset. Because training and test are random splits of the same generated distribution, the test set shares translation artifacts, synthetic templates, and LLM-labeling noise with training. The near-ceiling scores in Tables 5-6 (macro 0.922-0.963) therefore measure fit to a self-generated distribution, not performance on real German, Turkish, Arabic, or other clinical notes. The paper itself notes in Section 4 that 'it was impossible to find any benchmark tests for these languages,' so there is no external anchor. Without independent expert-annotated non-English data, the abstract's claim that the models are 'the most accurate healthcare anonymization solutions' across eight languages is not supported. As an additional internal red flag, the abstract reports 'f1-micro score averages' while Tables 5-6 report only macro-avg values, so even the headline numbers are not directly auditable from the manuscript.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes expert small NER models for PHI de-identification in eight languages, built using an LLM-in-the-loop methodology in which LLMs generate or translate synthetic training data and provide labels at intermediate stages. For English, the models are trained on a mix of open-source, synthetic, and proprietary data and evaluated on the external i2b2-2014 test set. For the seven non-English languages, training and test data are both taken as an 80-20 split of the same translated and synthetic corpus. The authors report macro-F1 scores around 0.92-0.96 on the non-English test splits and claim their models surpass prior small models and GPT-4o, with the abstract stating f1-micro averages between 0.953 and 0.978.","tokens_in":15677,"tokens_out":7678,"duration_ms":65532,"significance":"If the multilingual results were independently validated, the approach would be practically valuable because small, on-premises de-identification models avoid the privacy and API-dependency concerns of large closed models. The English evaluation is a genuine strength: the i2b2-2014 test set is an external, expert-annotated benchmark, and the strict chunk-plus-label matching is an appropriate evaluation protocol. The paper is also transparent in admitting that no non-English benchmark exists. That transparency, however, exposes a load-bearing weakness: the non-English evaluation is effectively a self-evaluation, so the headline cross-lingual claims are not currently supported.","major_comments":[{"comment":"The non-English test sets are not independent of the training data. Section 3.1 states that most non-English datasets were generated by translating the English corpus with the authors' own medical translation models [30] and by LLM-assisted synthetic data generation, and Section 3.2.2 states that a 0.20 split of the same dataset is used as the test set. Section 4 then acknowledges that no benchmark tests exist for these languages. Consequently, the macro-F1 scores in Tables 5-6 measure fit to the same generated distribution from which the training data were drawn, not performance on real clinical notes in German, Italian, French, Romanian, Turkish, Spanish, or Arabic. The abstract's claim that the models are 'the most accurate healthcare anonymization solutions' across eight languages is therefore unsupported as stated. This requires external expert-annotated non-English test data, or a clear reframing of the non-English results as an internal pilot rather than a validated cross-lingual result.","section":"Sections 3.1, 3.2.2, 4; Tables 5-6"},{"comment":"The headline numbers are not auditable. The abstract reports 'f1-micro score averages' of 0.966, 0.975, 0.976, 0.970, 0.964, 0.974, 0.978, and 0.953 for the eight languages, but Tables 5 and 6 report only macro-avg values (German 0.960, Italian 0.955, French 0.937, Turkish 0.963, Spanish 0.957, Romanian 0.930, Arabic 0.922). No micro-averaged F1 values are reported anywhere in the paper, and the abstract numbers do not match the presented macro-avg values. The authors should report the actual micro-averaged F1 used to generate the abstract, or correct the abstract.","section":"Abstract; Tables 5-6"},{"comment":"The GPT-4o comparison is internally inconsistent. Table 4 lists GPT-4o's macro-average as 0.548, but the Section 4 text says 'the macro average (0.5757)' and 'the micro average (0.5907)', while Table 7 reports a macro avg of 0.5775 and a weighted avg of 0.968. These discrepancies make it difficult to verify the claim that the proposed small models surpass GPT-4o. The paper should present one consistent evaluation of GPT-4o and report the exact numbers that support the comparison.","section":"Table 4, Section 4, Table 7"}],"minor_comments":[{"comment":"The Arabic model is included in Table 6, but Section 3.2.2 does not specify which embedding was used for Arabic, unlike the other languages. Please provide the Arabic embedding and its training details.","section":"Section 3.2.2; Table 6"},{"comment":"The text following Table 7 mentions B-EMAIL and B-FAX as classes with low precision and recall, but the table does not contain rows for these labels. The caption and the table should be consistent.","section":"Table 7"},{"comment":"The comparison with prior work in Table 4 uses a different label set: the authors include DEVICE and LOCATION-OTHER, which are absent from the comparator columns. Because macro-averaging gives equal weight to every class, the comparison is not strictly apples-to-apples; a footnote should state which labels are excluded from each comparator.","section":"Section 3.1; Table 4"},{"comment":"The paper says the codebase will be available soon, but no code, data-generation script, or annotation protocol is released. Given the central role of synthetic data and LLM-assisted labeling, a detailed data-generation protocol is needed for reproducibility.","section":"Footnote 1"}],"recommendation":"reject","confidential_remarks":"The English result on the external i2b2-2014 test set is a plausible contribution, but the paper's central multilingual claim rests on a circular evaluation: non-English training and test data are drawn from the same self-generated translated/synthetic corpus, and the authors concede no external non-English benchmark exists. The abstract also reports micro-F1 values that are not present in the tables. These are load-bearing issues that would require substantial new external evaluation, not just local revisions, to support the paper's central claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: the English de-identification result is a credible, if incremental, empirical contribution; the multilingual claim that anchors the abstract is not supported by the evidence presented. I'd treat this as a paper with one solid section and one load-bearing weakness.\n\nWhat's actually new: they train small NER models for de-identification in seven languages (German, Italian, French, Romanian, Turkish, Spanish, Arabic) where public benchmarks don't exist, and they report a macro-F1 of 0.931 on the i2b2-2014 English test set, which beats the numbers they cite from Khin et al. and Kocaman/Talby. The English evaluation uses the real i2b2 test set, so that number is meaningful, though label-set differences make direct comparison a bit apples-to-oranges.\n\nThe soft spot is the non-English evaluation. Section 3.1 says most non-English data were generated by translating English data with their own medical translation models and by LLM-assisted synthetic generation; Section 3.2.2 then takes a 20% test split from that same dataset. So the near-ceiling F1 scores (0.92-0.96) measure fit to the training distribution, not performance on real clinical text in those languages. The paper itself concedes in Section 4 that it could not find any external benchmarks, which is honest but also the point: without independent ground truth, the headline 'across multiple languages' claim is unverified. The abstract also says 'f1-micro score averages' while Tables 5-6 report macro averages, so even the headline numbers are not directly auditable.\n\nThat said, the paper is not a wreck. The English portion is a legitimate empirical result, and the LLM-in-the-loop methodology is a reasonable strategy for building small models without sending PHI to APIs. The authors appear to understand the limitation, but the abstract overstates it.\n\nWho's this for: someone working on applied de-identification who wants to see one more English result. For multilingual de-identification, this paper cannot yet be trusted. It deserves a serious referee only if the authors are willing to obtain external non-English evaluation or reframe the contribution as English-focused. As it stands, I'd recommend sending it back with the request to fix the circular evaluation and tone down the claims before considering it.","headline":"English de-identification result is credible, but the multilingual F1 scores are self-evaluations on translated/synthetic data, so the headline 'most accurate' claim overreaches.","tokens_in":16292,"tokens_out":2834,"would_cite":false,"duration_ms":25138,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuned small NER models, trained with LLMs only as data generators, outscore GPT-4o and prior systems on clinical de-identification in eight languages.","keywords":["de-identification","protected health information","named entity recognition","multilingual NER","LLM-in-the-loop","i2b2-2014","clinical NLP","synthetic data"],"falsifier":"Release the models and run them, under strict span-and-label matching, on a held-out set of authentic, manually annotated discharge and admission notes in the seven non-English languages; if macro F1 falls materially below the reported 0.92–0.98 range, the multilingual claim is refuted. A quicker check on the English side is to re-run the i2b2 benchmark with the regex rule layer disabled to see how much of the F1 depends on pattern matching.","tokens_in":15255,"feed_emoji":"🏥","tokens_out":9800,"duration_ms":74916,"temperature":0.7,"pith_summary":"This paper sets out to prove that de-identifying protected health information does not require a large language model at inference time. The authors argue that small, domain-specific NER models built through an LLM-in-the-loop pipeline, where LLMs only generate and translate training data, can match or beat general-purpose LLMs such as GPT-4o on the English i2b2-2014 benchmark while also covering seven additional languages. If the claim is correct, hospitals could anonymize records locally, keeping sensitive data off external APIs and cutting cost, without sacrificing accuracy. The reported micro-F1 averages are 0.966, 0.975, 0.976, 0.970, 0.964, 0.974, 0.978, and 0.953 for English, German, Italian, French, Romanian, Turkish, Spanish, and Arabic. A sympathetic reader would take the paper as evidence that specialized small models, not LLMs, are the right tool for this privacy-sensitive task.","feed_headline":"Fine-tuned NER models beat GPT-4o on clinical de-identification","feed_subtitle":"Fine-tuned on synthetic and translated data, they hit ~0.95-0.98 F1 in eight languages and run on-prem.","key_machinery":"The 'LLM-in-the-loop' development pipeline is the mechanism that carries the argument: LLMs are used to create training data, not to make predictions. English clinical texts are translated into German, Italian, French, Romanian, Turkish, Spanish, and Arabic using the authors' own medical translation models, and LLM-assisted synthetic data generation and labeling expand the non-English corpora. A fake-chunk augmentation step replaces entity spans with realistic alternatives in each language. The final models are fine-tuned small transformers (deberta-v3-small for English; per-language BERT-family models for the others) paired with a rule-based regex layer for structured identifiers like ZIP, PHONE, and SSN. Evaluation uses strict matching, so a prediction counts only when both the text span and the PHI label agree exactly.","core_discovery":"The central claim is that fine-tuned expert small NER models, developed with the LLM-in-the-loop methodology, are the most accurate healthcare anonymization solutions available. On the i2b2-2014 English test set, the authors' 18-label model reaches a macro F1 of 0.931, beating the best compared prior systems (0.919, 0.863, and 0.840 macro F1) and far exceeding GPT-4o, which scores 0.548 macro and 0.591 micro F1 on the same data. For the non-English languages, the per-label tables report macro-F1 averages of 0.963 for Turkish, 0.960 for German, 0.957 for Spanish, 0.955 for Italian, 0.937 for French, 0.930 for Romanian, and 0.922 for Arabic, with the abstract quoting micro-F1 averages between 0.953 and 0.978. The paper concludes that small models running on-premises can replace API-based LLMs for de-identification and that the pipeline transfers across languages and to other clinical NLP tasks.","pith_inferences":["Editorial: the non-English test sets come from the same translated and synthetic distribution as the training data, so the 0.92–0.98 macro F1 scores are best read as upper bounds until the models are evaluated on authentic clinical notes written in those languages.","Editorial: because structured labels such as ZIP, PHONE, and SSN are handled by a rule-based layer, part of the reported F1 may reflect pattern matching rather than learned NER; ablating that layer would quantify its contribution.","Editorial: the GPT-4o comparison is English-only, so the claim that small models surpass general-purpose LLMs remains unverified for the seven non-English languages.","Editorial: a direct extension would be to run the same pipeline on existing public multilingual clinical corpora, which would test whether the data-generation strategy transfers beyond the authors' own datasets."],"forward_implications":["Hospitals and research groups can de-identify records on-premises with small models, eliminating the privacy and compliance burden of sending PHI to third-party APIs.","The reported English benchmark suggests a well-trained small model can outperform a general-purpose LLM on this task, so API-based approaches are not necessary for state-of-the-art de-identification.","The LLM-in-the-loop data-generation recipe could be reused for non-English clinical NLP tasks where annotated data are scarce, including the biomedical entity and relation extraction the paper names as future work.","Multilingual de-identification at this reported accuracy would make non-English EHRs usable for research, assuming the translated and synthetic test distributions reflect real clinical text."],"supporting_citations":[{"why":"HHS guidance listing the 18 HIPAA Safe Harbor PHI types that define the label taxonomy.","marker":"[11]"},{"why":"The authors' medical translation models from Part-1, used to translate English datasets into the seven non-English languages.","marker":"[30]"},{"why":"Deep learning de-identification architecture used as the main baseline in the English F1 comparison and source of i2b2 dataset statistics.","marker":"[31]"},{"why":"Overview of the 2014 i2b2/UTHealth shared task Track 1, whose test set serves as the English evaluation benchmark.","marker":"[33]"},{"why":"Hybrid rule/ML de-identification system; one of the prior models the English results are compared against.","marker":"[39]"},{"why":"Large-scale hybrid de-identification system reported at 93% accuracy; another comparison baseline in the English table.","marker":"[45]"}],"fun_headline_variants":["Small NER models outdo GPT-4o on clinical data privacy","On-prem NER models beat GPT-4o for PHI de-identification","Tiny NER models top GPT-4o for multilingual PHI anonymization","Expert small NER models surpass GPT-4o on PHI privacy","LLM-guided NER models beat GPT-4o in 8 languages"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the non-English test sets, built by translating English records and generating synthetic text with LLMs, genuinely represent real clinical notes in those languages; if they do not, the reported near-perfect scores will not carry over to production data.","fun_headline_variants_meta":{"raw":{"variants":["Small NER models outdo GPT-4o on clinical data privacy","On-prem NER models beat GPT-4o for PHI de-identification","Tiny NER models top GPT-4o for multilingual PHI anonymization","Expert small NER models surpass GPT-4o on PHI privacy","LLM-guided NER models beat GPT-4o in 8 languages"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000953,"raw_usage":{"total_tokens":4135,"prompt_tokens":1084,"completion_tokens":3051,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":700,"completion_tokens_details":{"reasoning_tokens":2950}},"tokens_in":700,"tokens_out":3051,"duration_ms":19926,"temperature":1.0,"reasoning_tokens":2950,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:28:19.925283+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Release the models and run them, under strict span-and-label matching, on a held-out set of authentic, manually annotated discharge and admission notes in the seven non-English languages; if macro F1 falls materially below the reported 0.92–0.98 range, the multilingual claim is refuted. A quicker check on the English side is to re-run the i2b2 benchmark with the regex rule layer disabled to see how much of the F1 depends on pattern matching.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"HHS guidance listing the 18 HIPAA Safe Harbor PHI types that define the label taxonomy."},{"cited_title":"LLMs-in-the-loop Part-1: Expert Small AI Models for Bio-Medical Text Translation","cited_arxiv_id":"2407.12126","evidence_quote":"The authors' medical translation models from Part-1, used to translate English datasets into the seven non-English languages."},{"cited_title":"Burckhardt, and R","cited_arxiv_id":null,"evidence_quote":"Deep learning de-identification architecture used as the main baseline in the English F1 comparison and source of i2b2 dataset statistics."},{"cited_title":"Kotfila, and O","cited_arxiv_id":null,"evidence_quote":"Overview of the 2014 i2b2/UTHealth shared task Track 1, whose test set serves as the English evaluation benchmark."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Hybrid rule/ML de-identification system; one of the prior models the English results are compared against."},{"cited_title":"Talby, and H.U","cited_arxiv_id":null,"evidence_quote":"Large-scale hybrid de-identification system reported at 93% accuracy; another comparison baseline in the English table."}],"review_version":1}