{"id":"b8fc285e-834f-49ea-887d-bfad0e56cedb","arxiv_id":"2510.13873","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new expert-annotated French corpus of 1,301 oncology cases with 71k ICD-O normalisations and expression-level composite codes.","lead":"FRACCO is a French-language corpus of 1,301 synthetic oncology clinical cases, annotated with ICD-O tumor codes and a new expression-level normalisation layer. It gives French clinical NLP a much-needed resource for cancer entity recognition and coding.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Gold-standard claim for manual ICD-O normalisations rests on an unvalidated consensus step following only 51.5% inter-annotator agreement.","rationale":"The reader identified the same load-bearing assumption: low pre-consensus IAA (51.5%) on manual normalisations undermines the gold-standard claim because consensus accuracy is unverified. My read agrees and finds this to be the single most critical issue. The corpus is otherwise valuable: it is openly available, the NER experiments show learnable and coherent labels, and the span annotation IAA (soft F1 0.82–0.90) is more reassuring. The count inconsistencies (e.g., 71,127 normalisations vs. 71,065 annotations; 399 vs. 350 morphology codes; 272 vs. 300 topography codes) and the circular self-training validation are secondary concerns that reinforce the need for careful framing but do not independently sink the resource. The load-bearing point is that the most novel part of the dataset—manual normalisation of complex expressions into composite codes—has no demonstrated external validity. The proposed test would directly settle this: if independent coders match the consensus codes at high rates, the gold-standard claim survives; if not, the manual layer must be re-annotated or downgraded. This does not change the reader's CONDITIONAL verdict, which already calls for softening the gold-standard wording and releasing per-item confidence/review status; it sharpens the justification for that condition.","tokens_in":9385,"tokens_out":2972,"duration_ms":28527,"concrete_test":"Take the 1,000-expression manual normalisation validation sample (or a fresh random sample of ~1,000 manually normalised expressions from the released .ann files). Have two independent oncology coders, not involved in the original annotation, assign ICD-O codes to the same expressions using only the provided French text and the corpus's own guidelines. Measure (a) pairwise concordance between the two independent coders and (b) concordance of each coder with FRACCO's final consensus codes. If both independent coders agree with the FRACCO codes at or above 90%, the gold-standard claim for manual normalisations is substantially supported. If agreement falls near the original 51.5% range, the manual layer should be re-annotated, released with per-item confidence scores, or explicitly relabeled as 'expert consensus with unresolved ambiguity.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that FRACCO is a gold-standard corpus for ICD-O normalisation depends critically on the correctness of the ~15,000 manually normalised complex expressions. The paper reports only 51.5% concordance between two experts on a stratified sample of this manual subset (Technical Validation, ICD-O Normalisation Validation), versus 80.65% for the automated subset. The authors state that disagreements were resolved by consensus and propagated, but they provide no post-consensus validation against an external gold standard, no adjudication protocol, and no kappa statistic. Consensus among two annotators with such low starting agreement can institutionalize a shared bias or an arbitrary resolution rather than recover truth. Since the manually assigned codes include the large majority of the 2,043 unique composite expression_CIM codes—the dataset's most novel contribution—the 'gold-standard' label is weakest exactly where the resource claims its greatest added value. The NER fine-tuning results only demonstrate that the labels are learnable and internally consistent, not that they are externally correct. The automated subset's higher IAA does not mitigate this because those cases are dictionary-resolvable. Without independent adjudication or a second-panel re-annotation study, the safest conclusion is that the manual normalisation layer reflects expert opinion with moderate agreement, not a verified gold standard.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"FRACCO is a French-language corpus of 1,301 synthetic clinical oncology case reports, translated from CANTEMIST via FRASIMED, with three entity layers (morphologie, topographie, différenciation) and a fourth composite layer (expression_CIM). A total of 71,127 ICD-O-3.1 normalisations are claimed, produced by automated dictionary matching plus manual review. The paper reports span-level inter-annotator agreement (soft F1 0.82–0.90, hard F1 0.70–0.90, with différenciation at 0.48), normalisation concordance (80.65% on an automatic sample, 51.5% on a manual sample), and NER baselines (weighted F1 ≈ 89%). The authors position the resource as a gold standard for French oncology NER and normalisation.","tokens_in":9671,"tokens_out":5263,"duration_ms":40881,"significance":"FRACCO addresses a genuine gap: there are few French oncology corpora with ICD-O normalisation and no existing resource with expression-level composite codes. The dataset is publicly released on Zenodo with code on GitHub, and the paper includes explicit error analyses and benchmark results. If the annotation quality can be substantiated, this would be a valuable resource for French clinical NLP and for cross-lingual studies building on CANTEMIST. The main weakness is that the most novel layer — manually assigned ICD-O codes for complex expressions — rests on a consensus step after only 51.5% inter-annotator concordance, with no external validation; this weakens the 'gold-standard' designation.","major_comments":[{"comment":"The 51.5% concordance on the manual normalisation sample is a load-bearing result. The manual subset consists of the complex expressions that are the paper's most novel contribution (2,043 unique expression_CIM codes). With only two annotators, a raw concordance of 51.5%, no kappa statistic, and a consensus step whose protocol is not described, the claim that the final codes are a gold standard is not supported. Consensus can average errors rather than resolve them. I request either (a) independent adjudication by a second panel or comparison against an external gold standard on a sample of the manual codes, with kappa and per-category error analysis; or (b) a substantial softening of the gold-standard claim, describing the resource as 'expert-annotated with moderate agreement and consensus-based normalisation'.","section":"Technical Validation, ICD-O Normalisation Validation"},{"comment":"Unique-code counts are inconsistent across sections: the Abstract reports 399 morphology and 272 topography codes; the Methods reports 399 morphology and 375 topography codes; the Data Record reports 350 morphology and 300 topography codes. The document count also wavers between 1,301 (Abstract, Methods) and 1,300 (Technical Validation). Total annotations are given as 71,127, 71,065, and 'over 70,000'. These discrepancies must be reconciled; they currently prevent a precise assessment of the dataset's coverage.","section":"Abstract vs. Methods vs. Data Record"},{"comment":"The fine-tuning results show that the annotation scheme is learnable by BERT-based models, but they do not validate the correctness of the ICD-O codes. Because the test set is a random split of the same corpus, annotated using the same consensus decisions, the high F1 scores only demonstrate internal consistency. The text says these procedures 'confirm both the internal quality of the annotations and the usability of the dataset', which overstates the evidence. Please explicitly state that the NER results are not evidence for the external validity of the normalisation codes.","section":"Technical Validation, NER model fine-tuning"},{"comment":"The reconciliation rule that 'unresolved items default to the version provided by one annotator' can inject a systematic bias, especially for the différenciation category with hard F1 = 0.48. No post-reconciliation agreement is reported. Please report final agreement after reconciliation (e.g., on a held-out sample) and describe how the default choice was made.","section":"Technical Validation, Span Annotation Validation"}],"minor_comments":[{"comment":"Typographical errors: 'ditributions' (Figure 1), 'disribution' (Figure 5), 'Precisioin' (Affiliations). Also, terminology is mixed between 'histologic differentiation', 'différenciation', and 'differentiation'; please unify.","section":"Throughout"},{"comment":"The composite code 'C34.9 8010/34' is likely a typo; it should probably be 'C34.9 8010/3 4' or an explicitly defined concatenation. Please clarify the exact string representation of composite codes.","section":"Table 1"},{"comment":"The code '0* (grade 0)' is marked non-standard. Please explain why this non-standard code is included in the corpus and how it is distinguished from standard differentiation codes.","section":"Figure 5"},{"comment":"The phrase 'were flagged for manual and normalisation' appears to be missing a word; should be 'manual normalisation'.","section":"Methods, ICD-O normalisation"},{"comment":"The example filename 'case001.txt/case001.ann' is inconsistent with the actual naming convention described earlier ('cc_onco859.txt'). Please use a consistent example.","section":"Usage Notes"},{"comment":"Reference 15 (Stenetorp et al., brat) lacks full bibliographic details including publication venue and year.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper's association with FRASIMED and CANTEMIST authors is not itself disqualifying, but it raises the bar for independent validation. The inconsistent counts and the unvalidated consensus step are fixable in revision; the resource itself appears genuinely useful. Please ensure the final version does not overclaim gold-standard status and that the manual normalisation layer is either externally validated or explicitly qualified."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The FRACCO corpus fills a genuine gap: there is no other French oncology-specific corpus with this breadth of ICD-O annotations. The authors took the Spanish CANTEMIST texts, translated them, and added topography, differentiation, and a novel expression-level layer (expression_CIM) that combines multiple ICD-O components into unified concepts. That layer is the most original part of the resource, and the scale — over 71k normalisations across 1,301 documents — is substantial. The data and tools are publicly available, and the paper is honest enough to report the low agreement on the manual subset rather than bury it. For those working on French clinical NER or normalisation, this is a potentially valuable benchmark.\n\nThe soft spots are real, and they cluster around the \"gold-standard\" claim. The biggest issue is the normalisation validation. The paper reports 80.65% concordance on the automatically matched subset but only 51.5% on the manually assigned codes. That is the set that contains most of the novel expression_CIM annotations, so the reliability is weakest exactly where the added value is highest. The authors say disagreements were resolved by consensus and propagated, but there is no independent adjudication, no kappa, and no post-consensus check against an external standard. Consensus between two people with that level of starting agreement can easily bake in a shared bias. There is also a concrete data problem: the abstract says 399 morphology codes and 272 topography codes, while the Data Record says 350 and 300, and the methods text mentions 375 topography codes. These inconsistencies need to be reconciled before the corpus is trustworthy as a reference standard.\n\nThe NER experiments are fine as a learnability check, but they do not validate correctness — training and evaluating on the same corpus only shows the labels are internally coherent, which is expected. The synthetic nature of the text also means generalisation to real EHRs is untested. These are not fatal flaws, but they should be stated more carefully.\n\nMy overall sense: this deserves a serious referee, not a desk rejection. The resource is useful and the work is honest, but the presentation oversells it. I would recommend acceptance after major revision — the authors should either provide some form of external validation on the manual codes (e.g., a third annotator or a second panel on a sample), release per-item confidence or review status, and fix the count inconsistencies. They should also soften the \"gold-standard\" wording to something like \"expert-annotated reference corpus\" unless the validation can back it up.","headline":"FRACCO is a real and useful resource for French oncology NLP, but the gold-standard label is not supported by the reported validation, especially the 51.5% concordance on manually assigned normalisations.","tokens_in":10159,"tokens_out":2206,"would_cite":true,"duration_ms":20933,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FRACCO gives French oncology NLP a gold-standard ICD-O corpus","keywords":["French clinical NLP","oncology corpus","ICD-O normalisation","named entity recognition","concept normalisation","composite expressions","synthetic clinical cases","annotation benchmarking"],"falsifier":"Take a random sample of the manually normalised expressions and have them re-coded independently by a fresh panel of oncology experts; if agreement with the published codes falls to the level of the 51.5% baseline or below, the gold-standard claim for those codes is not supported.","tokens_in":9304,"feed_emoji":"🏥","tokens_out":2852,"duration_ms":27392,"temperature":0.7,"pith_summary":"The paper introduces FRACCO, a corpus of 1,301 synthetic French clinical case reports carrying 71,127 ICD-O normalisations across morphology, topography, differentiation, and a composite expression layer. The authors argue that this expert-reviewed dataset fills a gap in French-language oncology resources and provides a reference standard for training and evaluating named entity recognition and concept normalisation tools. The work matters because most French clinical NLP corpora lack oncology-specific annotations and standardised cancer terminology.","feed_headline":"French oncology corpus packs 71,000 ICD-O annotations","feed_subtitle":"Expert-validated French clinical cases give cancer-text NLP a shared reference standard.","key_machinery":"The central object is the expression_CIM layer, a composite annotation that binds a morphology code, an optional topography code, and an optional differentiation grade into a single unified clinical concept (for example, 'carcinome pulmonaire indifférencié' maps to topography C34.9, morphology 8010/3, and grade 4). This layer is what allows the corpus to represent whole oncological expressions rather than isolated terms, and it is carried by a semi-automated workflow: dictionary-based matching for straightforward spans and expert-guided reconstruction for complex or ambiguous phrases.","core_discovery":"FRACCO is presented as a gold-standard French oncology corpus built by translating a Spanish oncology corpus, projecting its annotations, manually correcting spans, and extending the annotation scheme with topography codes, differentiation grades, and a new expression-level layer. All entities were normalised to ICD-O-3.1 using automatic dictionary matching for roughly 78% of cases and expert consensus review for the remaining complex expressions. The result is 399 unique morphology codes, 272 topography codes, and 2,043 unique composite expression codes, claimed to be suitable as a benchmark for French oncology NLP.","pith_inferences":["Because the texts are translations of synthetic case reports, performance on real French electronic health records may be lower; the corpus should ideally be supplemented with authentic clinical text before being used as a sole training resource.","The low pre-consensus inter-annotator agreement on manually assigned codes (51.5%) suggests that some gold-standard codes encode consensus judgment rather than unambiguous ontology mappings, so downstream users may want to audit edge-case codes for their own applications.","The translation-and-projection workflow could be reused to create oncology corpora in other low-resource languages, though the manual correction burden appears substantial.","The expression_CIM layer could serve as training data for sequence-to-sequence normalisation models that map raw clinical phrases directly to composite ICD-O codes."],"forward_implications":["French oncology NER and normalisation models can be trained and compared on a shared benchmark with over 70,000 annotated entities.","The composite expression layer enables tasks beyond span detection, such as relation extraction and document-level classification of cancer phenotypes.","The corpus can support cross-lingual research by aligning with the Spanish source corpus, allowing direct comparison of French and Spanish clinical NLP systems.","The detailed coverage statistics (399 morphology, 272 topography codes) give a concrete picture of which oncological concepts are most represented in synthetic clinical case language."],"fun_headline_variants":["French oncology NLP gets 71K-annotated gold standard","71K ICD-O annotations for French oncology NLP","Expert-validated French oncology corpus with 71K ICD-O normalisations","French clinical corpus normalises 71K oncology terms to ICD-O","Gold-standard French oncology corpus: 71K ICD-O codes"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The corpus is called gold-standard because expert consensus resolved normalisation disagreements, but manually assigned codes agreed only 51.5% between two validating experts before discussion, so the correctness of the final codes rests on the assumption that consensus produces accurate ICD-O codes rather than merely averaging judgment.","fun_headline_variants_meta":{"raw":{"variants":["French oncology NLP gets 71K-annotated gold standard","71K ICD-O annotations for French oncology NLP","Expert-validated French oncology corpus with 71K ICD-O normalisations","French clinical corpus normalises 71K oncology terms to ICD-O","Gold-standard French oncology corpus: 71K ICD-O codes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001071,"raw_usage":{"total_tokens":4308,"prompt_tokens":714,"completion_tokens":3594,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":458,"completion_tokens_details":{"reasoning_tokens":3507}},"tokens_in":458,"tokens_out":3594,"duration_ms":22544,"temperature":1.0,"reasoning_tokens":3507,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T10:07:43.216480+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of the manually normalised expressions and have them re-coded independently by a fresh panel of oncology experts; if agreement with the published codes falls to the level of the 51.5% baseline or below, the gold-standard claim for those codes is not supported.","supporting_citations":[],"review_version":1}