{"id":"76ca5457-8361-4fa3-9233-a6b8fa45daeb","arxiv_id":"2412.09651","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"SISCO.web combines text search and a rule-based decision tree to suggest ICD-9-CM codes and identify the main diagnosis in Italian hospital discharge records, with a preliminary 80% accuracy on 30 use cases.","lead":"This paper describes SISCO.web, a web service that helps Italian physicians assign standard ICD-9-CM codes from hospital discharge records and identify the main diagnosis. Its evaluation is limited to a small qualitative test with three physicians on 30 cases, reporting an average suggestion accuracy of 80%.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 80% figure covers only code suggestion; the paper's central claim about identifying the main condition has no quantitative evaluation, and the test set leaks the KB source.","rationale":"The reader's condition—external validation on independent hospital data—is appropriate, but the single most load-bearing gap is not just corpus leakage; it is the complete absence of any numeric evaluation of the IMDM module, which is the system's defining contribution. Even a clean 80% code-suggestion result would not substantiate the paper's central claim about identifying the main pathological condition. The paper is transparent about this limitation in its conclusions, so conditional acceptance with a concrete external test is the right outcome, not rejection. The SEI benchmark leakage and absent inter-rater reliability further weaken the 80% figure, but they are secondary to the missing main-diagnosis evidence.","tokens_in":16255,"tokens_out":4441,"duration_ms":46863,"concrete_test":"Take 100 discharge letters from a hospital outside the FVG region that did not contribute to the KB. Have two expert coders independently assign the main diagnosis, with a third adjudicator resolving disagreements, as gold standard. Ask physicians to run the complete SISCO.web workflow (TEM for all diagnoses and procedures, then IMDM for main condition), and record (i) precision/recall of suggested ICD-9-CM codes against the gold standard and (ii) agreement between SISCO.web's IMDM main-condition output and the adjudicated main diagnosis, using Cohen's kappa. If main-diagnosis kappa is below 0.7 or shows no significant improvement over selecting the first-listed diagnosis, the central claim about main-diagnosis identification fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3's evaluation covers only the TEM text-encoding module. The paper's stated primary objective—identifying the main pathological condition through the IMDM decision tree—has no quantitative evaluation. Section 2.2.2 describes the rule engine, but Section 3 reports only that testing 'revealed issues' in this module, that the initial formalization 'did not yield the expected results,' and that the tree is 'not fully computerized' and requires YES/NO user answers. No accuracy, agreement, or gold-standard comparison is reported for main-diagnosis selection. Thus the 80% figure, even if valid, supports only code suggestion, not the central claim about main-diagnosis identification. The 80% figure is itself compromised by evaluation leakage: the test subset comes from the same SEI database used to build the KB glossary (Section 2.2.1), and the three physician ratings are not checked for inter-rater reliability. The load-bearing assumption is therefore not merely that 30 cases are representative; it is that the system's defining capability has any measurable evidence at all. Until an external, adjudicated test of the full TEM+IMDM workflow is performed, the central claim is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents SISCO.web, a web-based coding support system for Italian Hospital Discharge Records (SDO). The system combines a large Italian ICD-9-CM knowledge base (about 323,000 terms) with NLP-based full-text search (the TEM module) and a Drools-based decision-tree rule engine (the IMDM module) intended to guide physicians in identifying the main pathological condition according to WHO morbidity coding rules. The authors report that on average SISCO.web provided precise ICD-9-CM code suggestions for 80% of 30 use cases evaluated by three physicians, with improved accuracy when using related terms, and they discuss limitations of the rule-engine module.","tokens_in":16428,"tokens_out":3687,"duration_ms":39616,"significance":"If the reported performance were robust, SISCO.web would be a useful example of an explainable, rule-based alternative to black-box AI coders, and the decision-tree formalization of WHO main-condition rules would be a valuable, reusable resource for the Italian context and beyond. The paper also contributes a detailed description of a large multilingual terminological resource and its integration with an ICD-9-CM ontology. However, the current evidence is far too weak to establish the central claims: the quantitative evaluation is limited to one module, lacks statistical grounding, and is compromised by test-set leakage from the knowledge-base source. The work would need a substantially strengthened evaluation before it can support the stated conclusions.","major_comments":[{"comment":"The only quantitative result is the claim that 'SISCO.web provided precise ICD-9-CM code suggestions for 80% of 30 use cases' evaluated by three physicians. No confidence interval, per-case breakdown, definition of 'precise suggestion', or inter-rater reliability measure is provided. With 30 cases and three raters, the estimate is too fragile to support the paper's conclusions, and the absence of any baseline comparator (e.g., unaided coding, an alternative system, or a simple lexical baseline) makes the 80% figure uninterpretable.","section":"Section 3 (Evaluation)"},{"comment":"The evaluation set is not independent of the system's construction. Section 2.2.1 states that the knowledge base was enriched with a glossary of 696 diagnoses extracted from the FVG SEI emergency-department database, and Section 3 states that the tested pathological conditions were 'extracted from the SEI database' mentioned in Section 2. This creates a direct leakage: the reported 80% accuracy may largely reflect retrieval from the system's own source lexicon rather than generalization to unseen clinical language. An independent, external test set is required.","section":"Section 3 and Section 2.2.1"},{"comment":"The paper's stated primary objective is identifying the main pathological condition through the IMDM decision-tree module, yet no quantitative evaluation of this module is reported. Section 3 explicitly says that the initial formalization 'did not yield the expected results', that the decision tree is 'not fully computerized', and that it currently requires the physician to answer a series of YES/NO questions. The central claim about main-diagnosis identification is therefore unsupported by any measured evidence; the 80% figure, even if valid, applies only to code suggestion in the TEM module.","section":"Section 2.2.2 and Section 3"},{"comment":"The search algorithm contains free parameters in the form of attribute weights (e.g., 10 for main description, 7.5 for additional terms, 2.5 for inclusion terms, 0.1 for glossary terms). The authors state that these weights were 'refined' after internal testing because the initial results appeared inconsistent. With only 30 use cases and no reported train/test split or external validation, the reported accuracy may reflect overfitting to the internal test set, and the robustness of these hand-tuned weights is not established.","section":"Section 3 (search weights)"}],"minor_comments":[{"comment":"There are two subsections numbered 2.2.1: 'The SISCO.web Knowledge Base' and 'The Text encoding module'. Please renumber the latter as 2.2.2 (and adjust subsequent numbering) to avoid confusion.","section":"Section 2.2.1"},{"comment":"The text refers to 'as shown in Fig. 3' when describing the main data sources of the knowledge base, but no Figure 3 appears in the manuscript. Either include the figure or correct the reference.","section":"Section 2.2 (Figure 3 reference)"},{"comment":"The abbreviation for the main-diagnosis module is inconsistent: it is introduced as IMDM in Section 2.2, but Section 3 refers to it as IMCM. Please unify the abbreviation.","section":"Section 3"},{"comment":"The statement that 'SISCO.web performances are comparable with most of the mentioned systems' is not supported by any comparative evaluation in the paper. Please either soften this claim or provide quantitative evidence.","section":"Section 4 (Related works)"},{"comment":"The conclusion that the TEM module 'has reached fairly good performances' should be explicitly conditioned on the preliminary and non-independent nature of the evaluation described in Section 3, since the current wording overstates the strength of the evidence.","section":"Section 5 (Conclusions)"}],"recommendation":"major_revision","confidential_remarks":"The paper is better characterized as a system description with a preliminary usability/coding-accuracy pilot than as a fully validated study. The SEI-database leakage and the absence of any evaluation of the IMDM module are serious enough that I cannot recommend acceptance in the current form. I would advise the editor that a major revision should require an external, independent evaluation of at least the TEM module and a quantitative (or at least systematic qualitative) assessment of the IMDM decision-tree module, together with a clear statement of the relationship between the test set and the knowledge base. If the authors cannot provide external data, the paper would need to be reframed as a design and feasibility study with the accuracy claims removed or explicitly labeled as preliminary."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: what's genuinely new here is the system description, not a demonstrated result. SISCO.web is a fairly detailed account of how you can combine Lucene full-text search, a Drools rule engine, an ICD-9-CM ontology, and a WHO-derived decision tree to support Italian hospital discharge coding. It also tells you what went wrong: the decision tree is not fully computerized, the initial formalization did not yield expected results, and the authors had no LDO/SDO database, only 30 use cases. That honesty is a real strength.\n\nThe paper does several things well. The knowledge base assembly (about 323,000 terms, with sources and counts listed) is useful for anyone building similar coding support. The description of the related-terms search feature is clear. The related-work section gives a reasonable picture of why current AI/LLM coders fall short and positions SISCO.web as a rules-based alternative.\n\nThe soft spots are exactly where the stress-test note lands. The 80% figure applies only to the TEM module's code suggestions, and it comes from 30 use cases evaluated by three physicians with no inter-rater reliability, no confidence interval, no baseline comparator, and no external test set. More seriously, the test conditions were extracted from the same SEI database that supplied 696 diagnosis terms to the knowledge base. That is a leak: the number shows retrieval from the system's own source lexicon, not generalization to unseen clinical language. And the paper's central claim—identifying the main diagnosis via the IMDM decision tree—is described as not fully computerized and as not yielding expected results in the initial formalization; no accuracy or agreement is reported for that module. So if the headline is 'SISCO.web can help identify the main diagnosis,' the evidence does not support it. If the headline is 'we built this prototype and here are its components and a preliminary TEM evaluation,' the paper is honest and mostly useful.\n\nCitation pattern and related work look fine; the self-citations point to earlier descriptions of the same project and are appropriate. No code or data is released, which weakens reproducibility but is not misleading given the stated restrictions.\n\nFor a reader interested in building ICD coding support systems, this is a useful system description with a limited evaluation. It deserves a serious referee only if the venue publishes system/use-case papers and is willing to demand external validation; as a scientific evaluation of coding accuracy, it is not there yet. My recommendation: take it as a systems paper, and require an external adjudicated evaluation before any claim about accuracy or main-diagnosis identification is accepted.","headline":"Honest system description, but the 80% figure covers only code suggestion on a leaked test set, and the main-diagnosis module has no quantitative evaluation.","tokens_in":16995,"tokens_out":3557,"would_cite":false,"duration_ms":40597,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that SISCO.web—a web service combining a large curated ICD-9-CM knowledge base with NLP search and a WHO-rule decision tree—suggests accurate diagnosis codes in 80% of 30 physician-tested cases and helps identify the…","keywords":["Coding Support Systems","Hospital Discharge Records","ICD-9-CM","Morbidity coding","Coding Rules","Main diagnosis","Decision tree","NLP"],"falsifier":"A decisive test would be a held-out evaluation on 30 or more discharge letters from hospitals or departments that contributed no terms to the SISCO.web knowledge base, with the true main diagnosis independently adjudicated by coders not involved in the project. If exact-match code precision on that set falls well below 80%, or if the decision tree's main-diagnosis choice frequently disagrees with the panel on cases with multiple comorbidities and procedures, the central claim of generalizable coding support would not be supported.","tokens_in":16010,"feed_emoji":"🏥","tokens_out":8962,"duration_ms":83610,"temperature":0.7,"pith_summary":"The paper argues that a transparent coding support system can improve the accuracy of Italian hospital discharge coding, a task where manual coding is error-prone and general-purpose AI models still fall short. The system, SISCO.web, couples full-text search over a large, hand-curated knowledge base of ICD-9-CM terms with a rule-based decision tree that implements WHO morbidity-coding rules for selecting the main diagnosis. In an evaluation by three physicians on 30 use cases, the tool suggested precise ICD-9-CM codes 80% of the time, and accuracy improved when the related-terms refinement feature was used. If the claim holds, hospitals could code discharges faster and more consistently, and the approach could carry over to the Italian ICD-10-IM classification now being introduced.","feed_headline":"Coding support tool suggests right diagnosis codes in 80% of tests","feed_subtitle":"SISCO.web couples medical-language search with WHO main-diagnosis rules as a middle path between manual coding and AI.","key_machinery":"The carrying mechanism is a 322,859-term knowledge base that merges the ICD-9-CM systematic and alphabetical indexes with physician, emergency-department, rare-disease, and MeSH glossaries, all indexed for weighted full-text search. On top of it, the text encoding module (TEM) tokenizes the user's free-text diagnosis, ranks candidate leaf codes by weighted attribute matches, and offers related terms that co-occur in ICD descriptors to narrow the search. The main-condition module then feeds the selected diagnosis and procedure codes into a decision tree formalized as if-then rules in the Drools rule engine; the tree follows the WHO definition of the main condition, partitions procedures by resource consumption, and asks the physician YES/NO or code-selection questions only when the rules cannot decide alone.","core_discovery":"The central claim is that combining curated medical-language resources with explicit coding rules, rather than black-box machine learning, is enough to give physicians practically useful coding support. On the paper's own numbers, SISCO.web suggested precise ICD-9-CM codes in 80% of 30 use cases tested by three physicians, with better results when users filtered by system-suggested related terms; the same platform then ran the selected diagnosis and procedure codes through a decision tree that ends at the condition primarily responsible for the admission. The authors contrast this with recent large-language-model benchmarks reporting under 50% accuracy in medical code querying, and present the rule-driven decision tree as the part that addresses the hardest problem: identifying the main diagnosis among several coded conditions.","pith_inferences":["Beyond the paper, the reported 80% accuracy should be treated as an upper bound on real-world performance, because the test cases were drawn from the same SEI emergency-department database used to build the knowledge base; a fair test would use discharge letters from sources that contributed no terms.","Beyond the paper, the most reusable contribution is likely the decision-tree formalization itself: if it were paired with a held-out gold-standard corpus of main diagnoses, it could serve as a baseline or pre-filter for machine-learning coders rather than a replacement for them.","Beyond the paper, a practical next experiment is a randomized comparison of coding time and inter-coder agreement with and without the tool on identical discharge letters, since suggestion accuracy does not by itself prove that final coded records improve.","Beyond the paper, adding a large-language-model term-expansion step to the existing lexical search could combine the flexibility of AI with the auditability of the rule engine, a hybrid the paper only mentions as future work."],"forward_implications":["If the 80% suggestion accuracy holds in routine use, physicians can complete Hospital Discharge Records with fewer coding errors, improving both hospital reimbursement and epidemiological surveillance.","The decision-tree module gives a repeatable, auditable procedure for choosing the main diagnosis, addressing the documented international variation in how the 'main condition' is defined and coded.","Because the architecture separates the knowledge base from the rule engine, the same system can be extended to ICD-10-IM and crosswalking tables, supporting Italy's planned transition away from ICD-9-CM.","The related-terms feature provides a concrete way for coders to refine an ambiguous free-text search without knowing the classification's exact vocabulary in advance."],"supporting_citations":[{"why":"Earlier SISCO.web paper that supplies the knowledge-base design and resource counts this work builds on.","marker":"Cardillo et al. 2019"},{"why":"Defines the main condition and provides the ICD-10 morbidity coding rules the decision tree implements.","marker":"WHO, 2016"},{"why":"Provides the decision tree for identifying the main condition, which the IMDM module formalizes as rules.","marker":"Zavaroni et al. 2018"},{"why":"Documents the Drools rule engine used to execute the decision-tree logic.","marker":"Proctor 2012"},{"why":"Shows international variation in the definition of the main condition, motivating the main-diagnosis module.","marker":"Quan et al. 2014"},{"why":"Evidence that AI-based coding of discharge summaries is unreliable, used to position SISCO.web as an alternative.","marker":"Falis et al. 2024"},{"why":"Benchmark showing large language models achieve under 50% accuracy in medical code querying, the comparison baseline.","marker":"Soroush et al. 2024"},{"why":"Shared-task results, including a best mean average precision of 0.593, used to frame SISCO.web's performance against prior systems.","marker":"Miranda-Escalada et al. 2020"}],"fun_headline_variants":["SISCO.web's decision tree finds main diagnosis in 80% of tests","SISCO.web's rule tree IDs main diagnosis in 80% of cases","SISCO.web: decision tree identifies main diagnosis in 80%","80% main-diagnosis accuracy with SISCO.web's rules, not AI","SISCO.web's rule-driven code suggestions hit 80% main-diagnosis rate"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes the 30 test cases are an independent benchmark of coding accuracy, yet the same emergency-department database used to build the system's vocabulary also supplied those test cases, so the 80 percent figure may only reflect the system retrieving terms from its own source lexicon.","fun_headline_variants_meta":{"raw":{"variants":["SISCO.web's decision tree finds main diagnosis in 80% of tests","SISCO.web's rule tree IDs main diagnosis in 80% of cases","SISCO.web: decision tree identifies main diagnosis in 80%","80% main-diagnosis accuracy with SISCO.web's rules, not AI","SISCO.web's rule-driven code suggestions hit 80% main-diagnosis rate"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001,"raw_usage":{"total_tokens":4189,"prompt_tokens":857,"completion_tokens":3332,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":473,"completion_tokens_details":{"reasoning_tokens":3225}},"tokens_in":473,"tokens_out":3332,"duration_ms":24257,"temperature":1.0,"reasoning_tokens":3225,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:43:49.350387+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive test would be a held-out evaluation on 30 or more discharge letters from hospitals or departments that contributed no terms to the SISCO.web knowledge base, with the true main diagnosis independently adjudicated by coders not involved in the project. If exact-match code precision on that set falls well below 80%, or if the decision tree's main-diagnosis choice frequently disagrees with the panel on cases with multiple comorbidities and procedures, the central claim of generalizable coding support would not be supported.","supporting_citations":[],"review_version":1}