REVIEW 4 major objections 5 minor 2 references
Assisted morbidity coding: the SISCO.web use case for identifying the main diagnosis in Hospital Discharge Records
T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read This paper claims that SISCO.web—a web service combining a large curated ICD-9-CM knowledge base with NLP search and a WHO-rule decision tree—suggests accurate diagnosis codes in 80% of 30 physician-tested cases and helps identify the…
desk verdict Honest system description, but the 80% figure covers only code suggestion on a leaked test set, and the main-diagnosis module has no quantitative evaluation. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying mechanism is a 322,859-term knowledge base that merges the ICD-9-CM systematic and alphabetical indexes with physician, emergency-department, rare-disease, and MeSH glossaries, all indexed for weighted full-text search. On top of it, the text encoding module (TEM) tokenizes the user's free-text diagnosis, ranks candidate leaf codes by weighted attribute matches, and offers related terms that co-occur in ICD descriptors to narrow the search. The main-condition module then feeds the selected diagnosis and procedure codes into a decision tree formalized as if-then rules in the Drools rule engine; the tree follows the WHO definition of the main condition, partitions procedures by resource consumption, and asks the physician YES/NO or code-selection questions only when the rules cannot decide alone.
What would settle it
A decisive test would be a held-out evaluation on 30 or more discharge letters from hospitals or departments that contributed no terms to the SISCO.web knowledge base, with the true main diagnosis independently adjudicated by coders not involved in the project. If exact-match code precision on that set falls well below 80%, or if the decision tree's main-diagnosis choice frequently disagrees with the panel on cases with multiple comorbidities and procedures, the central claim of generalizable coding support would not be supported.
Extended reading notes
Core claim
The central claim is that combining curated medical-language resources with explicit coding rules, rather than black-box machine learning, is enough to give physicians practically useful coding support. On the paper's own numbers, SISCO.web suggested precise ICD-9-CM codes in 80% of 30 use cases tested by three physicians, with better results when users filtered by system-suggested related terms; the same platform then ran the selected diagnosis and procedure codes through a decision tree that ends at the condition primarily responsible for the admission. The authors contrast this with recent large-language-model benchmarks reporting under 50% accuracy in medical code querying, and present the rule-driven decision tree as the part that addresses the hardest problem: identifying the main diagnosis among several coded conditions.
Load-bearing premise
The evaluation assumes the 30 test cases are an independent benchmark of coding accuracy, yet the same emergency-department database used to build the system's vocabulary also supplied those test cases, so the 80 percent figure may only reflect the system retrieving terms from its own source lexicon.
Editorial extensions
If this is right
- If the 80% suggestion accuracy holds in routine use, physicians can complete Hospital Discharge Records with fewer coding errors, improving both hospital reimbursement and epidemiological surveillance.
- The decision-tree module gives a repeatable, auditable procedure for choosing the main diagnosis, addressing the documented international variation in how the 'main condition' is defined and coded.
- Because the architecture separates the knowledge base from the rule engine, the same system can be extended to ICD-10-IM and crosswalking tables, supporting Italy's planned transition away from ICD-9-CM.
- The related-terms feature provides a concrete way for coders to refine an ambiguous free-text search without knowing the classification's exact vocabulary in advance.
Reading between the lines
- Beyond the paper, the reported 80% accuracy should be treated as an upper bound on real-world performance, because the test cases were drawn from the same SEI emergency-department database used to build the knowledge base; a fair test would use discharge letters from sources that contributed no terms.
- Beyond the paper, the most reusable contribution is likely the decision-tree formalization itself: if it were paired with a held-out gold-standard corpus of main diagnoses, it could serve as a baseline or pre-filter for machine-learning coders rather than a replacement for them.
- Beyond the paper, a practical next experiment is a randomized comparison of coding time and inter-coder agreement with and without the tool on identical discharge letters, since suggestion accuracy does not by itself prove that final coded records improve.
- Beyond the paper, adding a large-language-model term-expansion step to the existing lexical search could combine the flexibility of AI with the auditability of the rule engine, a hybrid the paper only mentions as future work.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents SISCO.web, a web-based coding support system for Italian Hospital Discharge Records (SDO). The system combines a large Italian ICD-9-CM knowledge base (about 323,000 terms) with NLP-based full-text search (the TEM module) and a Drools-based decision-tree rule engine (the IMDM module) intended to guide physicians in identifying the main pathological condition according to WHO morbidity coding rules. The authors report that on average SISCO.web provided precise ICD-9-CM code suggestions for 80% of 30 use cases evaluated by three physicians, with improved accuracy when using related terms, and they discuss limitations of the rule-engine module.
Significance. If the reported performance were robust, SISCO.web would be a useful example of an explainable, rule-based alternative to black-box AI coders, and the decision-tree formalization of WHO main-condition rules would be a valuable, reusable resource for the Italian context and beyond. The paper also contributes a detailed description of a large multilingual terminological resource and its integration with an ICD-9-CM ontology. However, the current evidence is far too weak to establish the central claims: the quantitative evaluation is limited to one module, lacks statistical grounding, and is compromised by test-set leakage from the knowledge-base source. The work would need a substantially strengthened evaluation before it can support the stated conclusions.
major comments (4)
- [Section 3 (Evaluation)] The only quantitative result is the claim that 'SISCO.web provided precise ICD-9-CM code suggestions for 80% of 30 use cases' evaluated by three physicians. No confidence interval, per-case breakdown, definition of 'precise suggestion', or inter-rater reliability measure is provided. With 30 cases and three raters, the estimate is too fragile to support the paper's conclusions, and the absence of any baseline comparator (e.g., unaided coding, an alternative system, or a simple lexical baseline) makes the 80% figure uninterpretable.
- [Section 3 and Section 2.2.1] The evaluation set is not independent of the system's construction. Section 2.2.1 states that the knowledge base was enriched with a glossary of 696 diagnoses extracted from the FVG SEI emergency-department database, and Section 3 states that the tested pathological conditions were 'extracted from the SEI database' mentioned in Section 2. This creates a direct leakage: the reported 80% accuracy may largely reflect retrieval from the system's own source lexicon rather than generalization to unseen clinical language. An independent, external test set is required.
- [Section 2.2.2 and Section 3] The paper's stated primary objective is identifying the main pathological condition through the IMDM decision-tree module, yet no quantitative evaluation of this module is reported. Section 3 explicitly says that the initial formalization 'did not yield the expected results', that the decision tree is 'not fully computerized', and that it currently requires the physician to answer a series of YES/NO questions. The central claim about main-diagnosis identification is therefore unsupported by any measured evidence; the 80% figure, even if valid, applies only to code suggestion in the TEM module.
- [Section 3 (search weights)] The search algorithm contains free parameters in the form of attribute weights (e.g., 10 for main description, 7.5 for additional terms, 2.5 for inclusion terms, 0.1 for glossary terms). The authors state that these weights were 'refined' after internal testing because the initial results appeared inconsistent. With only 30 use cases and no reported train/test split or external validation, the reported accuracy may reflect overfitting to the internal test set, and the robustness of these hand-tuned weights is not established.
minor comments (5)
- [Section 2.2.1] There are two subsections numbered 2.2.1: 'The SISCO.web Knowledge Base' and 'The Text encoding module'. Please renumber the latter as 2.2.2 (and adjust subsequent numbering) to avoid confusion.
- [Section 2.2 (Figure 3 reference)] The text refers to 'as shown in Fig. 3' when describing the main data sources of the knowledge base, but no Figure 3 appears in the manuscript. Either include the figure or correct the reference.
- [Section 3] The abbreviation for the main-diagnosis module is inconsistent: it is introduced as IMDM in Section 2.2, but Section 3 refers to it as IMCM. Please unify the abbreviation.
- [Section 4 (Related works)] The statement that 'SISCO.web performances are comparable with most of the mentioned systems' is not supported by any comparative evaluation in the paper. Please either soften this claim or provide quantitative evidence.
- [Section 5 (Conclusions)] The conclusion that the TEM module 'has reached fairly good performances' should be explicitly conditioned on the preliminary and non-independent nature of the evaluation described in Section 3, since the current wording overstates the strength of the evidence.
Circularity Check
The 80% code-suggestion accuracy is partly forced by construction: the test subset is drawn from the same SEI database whose 696-diagnosis glossary was loaded into the SISCO.web knowledge base.
-
fitted input called prediction
[Section 2.2.1 (Knowledge Base) and Section 3 (Evaluation)]
"bringing in the end to a glossary of 696 diagnoses (2,530 words) which enriched the SISCO.web KB (Section 2.2.1); 'This evaluation employed a subset of pathological conditions extracted from the SEI database mentioned in Section 2 ... On average, SISCO.web provided precise ICD-9-CM code suggestions for 80% of 30 use cases tested by physicians' (Section 3)."
The evaluation set is not independent of the system's construction. The KB was explicitly enriched with a glossary of 696 diagnoses extracted from the SEI database, and the evaluation used a subset of pathological conditions extracted from that same SEI database. Since the TEM module retrieves and suggests codes from the KB, the test terms are already present in the system's own indexed lexicon. The reported 80% 'precise ICD-9-CM code suggestions' therefore largely measures how faithfully the search engine returns its own stored source terms, not how well the system generalizes to unseen clinical language. No held-out corpus or external gold standard is used, so the headline accuracy figure is partly forced by the overlap between construction data and evaluation data.
full rationale
The paper contains one clear circular evaluation step: the 80% code-suggestion accuracy is computed on cases drawn from the same SEI database that supplied a 696-diagnosis glossary to the SISCO.web knowledge base. This is a training/evaluation leakage that makes the result partly a retrieval check on the system's own input data rather than an independent test of coding support. The paper's other central claim, about identifying the main pathological condition via the IMDM decision tree, is not backed by any quantitative evaluation; the authors themselves state that the decision tree is 'not fully computerized' and that the initial formalization 'did not yield the expected results.' That is a gap in evidence, but not by itself circularity. The self-citations (Cardillo et al. 2019, Zavaroni et al. 2018) are used descriptively and are not load-bearing in a way that forces the claimed result, since the decision tree is grounded in WHO morbidity coding rules. Overall, the partial circularity in the 80% evaluation justifies a score of 6.
Assumptions & free parameters
free parameters (1)
- TEM search attribute weights =
10, 7.5, 2.5, 0.1, 0
assumptions (3)
- domain assumption WHO ICD-10 morbidity coding rules and the WHO definition of the main condition apply to ICD-9-CM coding in the Italian SDO context.
- domain assumption The SEI emergency department database provides representative terminology for hospital discharge records.
- domain assumption The physicians' coding judgments during the usability test are a valid gold standard.
Cite this review
Pith. "Pith review of Assisted morbidity coding: the SISCO.web use case for identifying the main diagnosis in Hospital Discharge Records." pith.science (2026). https://pith.science/paper/YEONQOG4
@misc{pith2026241209651,
author = {Pith},
title = {Pith review of: Assisted morbidity coding: the SISCO.web use case for identifying the main diagnosis in Hospital Discharge Records},
year = {2026},
howpublished = {\url{https://pith.science/paper/YEONQOG4}},
note = {Machine review of arXiv:2412.09651}
}
read the original abstract
Coding morbidity data using international standard diagnostic classifications is increasingly important and still challenging. Clinical coders and physicians assign codes to patient episodes based on their interpretation of case notes or electronic patient records. Therefore, accurate coding relies on the legibility of case notes and the coders' understanding of medical terminology. During the last ten years, many studies have shown poor reproducibility of clinical coding, even recently, with the application of Artificial Intelligence-based models. Given this context, the paper aims to present the SISCO.web approach designed to support physicians in filling in Hospital Discharge Records with proper diagnoses and procedures codes using the International Classification of Diseases (9th and 10th), and, above all, in identifying the main pathological condition. The web service leverages NLP algorithms, specific coding rules, as well as ad hoc decision trees to identify the main condition, showing promising results in providing accurate ICD coding suggestions.
Figures
Reference graph
Works this paper leans on
-
[2017]
International variation in the definition of 'main condition' in ICD -coded health data
Istituto Poligrafico e Zecca dello Stato S.p.a. Roma. Proctor, M. 2012. Drools: A Rule Engine for Complex Event Processing. In: Schürr, A., Varró, D., Varró, G. (eds) Applications of Graph Transformations with Industrial Relevance. AGTIVE 2011. Lecture Notes in Computer Science, vol 7233. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-341...
arXiv 2012
-
[2018]
BERT -XML: Large scale automated ICD coding using BERT pretraining
#307 Zhang Zachariah, Liu Jingshu, and Razavian Narges. 2020. “BERT -XML: Large scale automated ICD coding using BERT pretraining”. In Proceedings of the 3rd Clinical Natural Language Processing Workshop, pages 24 –34, Online. Association for Computational Linguistics. https://doi.org/10.48550/arXiv.2006.03685
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.