{"id":"8b0908a8-fe6c-4d83-8afd-e493a0decec4","arxiv_id":"1909.02511","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A 3D deep learning model trained on noisy text labels recognizes liver CT scan phases with a mean F1 of 0.977, correctly harvesting 92.7% of studies.","lead":"This paper trains a 3D deep learning model to automatically identify the scanning phase (non-contrast, arterial, venous, or delay) of liver CT studies in a hospital database, using noisy text labels as weak supervision. It could help researchers assemble large, correctly labeled medical imaging datasets without manual review.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported accuracy rests on unvalidated manual test labels and an unverified patient-level train/test split; either issue could inflate the headline F1 and harvest rates.","rationale":"The paper is a solid engineering contribution: the 3DSE architecture is small, the ACE loss is a correct and stable way to use \"contrast\"-only labels, and the internal consistency of the tables checks out. The main threat to the central claim is not the method but the evaluation scaffold. The reader identified manual label quality; I agree and add that the absence of an explicit patient-level split is equally load-bearing because the task is to generalize across patients in a PACS. These are both omissions rather than demonstrated errors; the numbers may well be correct. Therefore the appropriate disposition remains conditional acceptance pending the two verification checks. I do not see an internal mathematical flaw in the ACE loss or the experimental comparisons that would require rejection, and the lack of code and data is a reproducibility concern but secondary to the validity of the reported test-set numbers.","tokens_in":7797,"tokens_out":6135,"duration_ms":60488,"concrete_test":"Obtain from the authors (or from DICOM metadata if released) the unique patient IDs for the training, validation, and test studies. Verify that no patient appears in more than one split; if overlap exists, re-run the full evaluation under a strict patient-level split and report the new F1 and harvest rates. Independently, have two radiologists re-annotate a random subset of at least 100 test studies (or all 231) in a blinded fashion, and report per-phase Cohen's kappa. If kappa is below 0.90 or if any patient overlap is found, the headline 0.977 F1 and 92.7% harvest rate should be revised or presented as conditional on those caveats.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (mean F1 0.977, 92.7% study harvest) is entirely an evaluation claim, so it inherits its validity from the manual labels in the validation and test sets and from those sets being independent of training. Section 2.1 reports only that \"we then manually annotated a validation set and a test set\"; it does not state who annotated, with what instructions, or whether any consensus or inter-observer agreement was measured. Without this, the F1 scores could reflect agreement with one annotator's possibly biased phase definitions rather than clinically correct phases. Second, the paper never states that the train/validation/test split is at the patient level. The counts (3,267 + 101 + 196 = 3,564, vs. 4,666 total patients) leave a gap of ~1,102 patients that is never explained, and since patients have 1-3 studies, a study in the test set could come from a patient whose other study is in training, inflating performance through patient-specific anatomy or scanner/protocol correlation. Because the text-mining baseline and the significance tests are all evaluated against the same unvalidated labels, they do not resolve either issue. The headline numbers are therefore not yet externally grounded.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper addresses automated curation of dynamic contrast-enhanced CT liver studies from a hospital PACS. The authors use simple text-mined DICOM labels as noisy supervision, train a compact 3D squeeze-and-excitation network (3DSE), and introduce an aggregated cross-entropy (ACE) loss that treats a 'contrast' label as the union of arterial, venous, and delay phases. On a manually annotated test set of 231 studies, the model achieves a mean scan-level F1 of 0.977 and a study-level harvest accuracy of 92.7%, outperforming text mining, C3D, and 3D-ResNet. The paper includes statistical significance testing for scan-level F1.","tokens_in":8016,"tokens_out":8396,"duration_ms":83806,"significance":"If the reported results are reproducible, the paper offers a practical solution to a recognized bottleneck in medical imaging ML: extracting large labeled datasets from real-world PACS. The 3DSE model is lightweight (19.22 MB), the ACE loss is a simple and generalizable way to incorporate hierarchical weak labels, and the comparison is against a noisy baseline that mimics realistic deployment. The paper's main value is its evaluation on a large heterogeneous dataset, but the current manuscript leaves two evaluation-critical details underspecified: the exact construction and patient-level disjointness of the train/validation/test splits, and the protocol for the manual annotation serving as ground truth. These must be clarified before the headline accuracy and harvest rates can be fully trusted.","major_comments":[{"comment":"The reported counts for the training, validation, and test sets do not add up to the stated totals: the paper gives 4,666 patients, 7,680 studies, and 43,010 scans in total, while the validation and test sets account for 101 and 196 patients, 150 and 231 studies, and 801 and 1,262 scans, respectively. This leaves 4,369 patients, 7,299 studies, and 40,947 scans, but the training set is reported as 3,267 patients, 5,164 studies, and 29,891 scans, leaving roughly 1,100 patients and 2,100 studies unaccounted for. The authors must specify the exact filtering or subsampling steps and state explicitly whether the split is at the patient level. Without this, it is unclear whether any patient's studies appear in both training and test, which could inflate the reported F1 and harvest rates.","section":"Section 2.1"},{"comment":"The validation and test sets are the sole ground truth for the evaluation, but the paper provides no information about the annotation process: who annotated, what instructions or reference criteria were used, whether disagreements were resolved by consensus, or any inter-observer agreement statistic. The reported F1 of 0.977 and harvest rate of 92.7% are measures of agreement with these manual labels; without details on their reliability, the clinical significance of these numbers is unclear. Please describe the annotator qualifications, the annotation interface or instructions, and any quality assurance measures.","section":"Section 2.1"}],"minor_comments":[{"comment":"The text mining row reports 80.9% accuracy, but 186/231 equals 80.5%; please reconcile the denominator or the percentage.","section":"Table 3"},{"comment":"The paper states that the D-phase degradation 'does not meet statistical significance' but does not report p-values; providing the actual p-values or confidence intervals would allow the reader to assess the strength of the significance claims.","section":"Section 3"},{"comment":"The row label '3DSE-SE' is ambiguous; please rename it to something like '3DSE (without SE)' for clarity.","section":"Table 2"},{"comment":"The sentence 'The text-matching rules are more than sufficient to reliably extract labels based on text alone' is contradicted by the later results showing a text-mining F1 of 0.938; please rephrase to reflect the actual performance.","section":"Section 2.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid application paper with a large dataset and a clear methodological contribution. The main concern is the dataset accounting: the reported train/validation/test counts do not reconcile with the stated totals, and the paper never explicitly confirms that the splits are patient-disjoint. If the authors can clarify the split and provide the missing counts, the results are likely acceptable. I recommend requesting a point-by-point response on these issues."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nShort version: this is a useful, well-scoped paper on using visual information to curate dynamic liver CT studies from hospital PACS. The core idea—train a small 3D network on noisy DICOM text labels, add an aggregated cross-entropy loss for generic 'contrast' labels, and evaluate on manually labeled scans—is sensible, and the reported gains are large. The ACE loss is a simple logsumexp trick, clearly derived and numerically stable. The study-level harvest rate (92.7% of studies fully correct) is the metric that matters for data curation. If the results hold, this saves real radiology-lab time.\n\nWhat's genuinely new is applying visual phase recognition to PACS curation rather than text mining alone, plus the weak-label aggregation for contrast-only annotations. The 3DSE architecture is not groundbreaking, but it's appropriately small and shows that a compact model with global channel attention can beat much larger 3D nets. Citations look normal, and the comparison against text mining and larger models is fair.\n\nSoft spots, in proportion.\n\nFirst, the manual labels. Section 2.1 says 'we then manually annotated a validation set and a test set' and nothing else: no annotator expertise, no instructions, no inter-observer agreement. Since the entire evaluation is measured against these labels, this is a real gap. It's not fatal—the visual cues are standard radiology—but a revision should include at least a small consensus reading or a detailed annotation protocol.\n\nSecond, the train/validation/test split is not explicitly at the patient level. Patient counts are 3,267 + 101 + 196 = 3,564, versus 4,666 total, leaving ~1,100 patients unaccounted for. With 1–3 studies per patient, a test study could come from a patient whose other study is in training, which could inflate F1 through anatomy or scanner correlation. I suspect it's an omitted detail rather than a deliberate shortcut, but it must be stated and verified.\n\nThird, no code or data is released, and training details are sparse beyond a learning-rate sweep. For a practical pipeline paper, that limits reproducibility but is common at this venue.\n\nThe stress-test note says these issues could undermine the headline numbers. I think they are legitimate reviewer questions, but they are addressable in revision. The text-mining baseline is weak, so the absolute F1 might drop a little under a fully clean split, but the relative gains from the ACE loss and 3DSE are credible.\n\nWho should read this: people working on medical image curation, PACS mining, or weak supervision in clinical imaging. It's a solid systems paper, not a conceptual breakthrough. It deserves serious peer review; my recommendation is conditional accept pending clear confirmation of patient-level separation, annotation details, and ideally code or data.\n\nBest,\n[Your name]","headline":"A practical vision-based PACS curation pipeline with a clean weak-label loss; the headline F1 is plausible but the evaluation rests on unstated annotation and split details that need a revision.","tokens_in":8535,"tokens_out":4252,"would_cite":true,"duration_ms":40894,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A compact 3D vision model labels liver CT phases at 97.7% F1","keywords":["data curation","PACS","dynamic CT","phase recognition","liver CT","3D convolutional network","squeeze-and-excitation","weak supervision"],"falsifier":"Take a random sample of, say, 300 scans from the same hospital PACS distribution, have several board-certified radiologists independently label the phase, and measure the model's F1 against their consensus; if mean F1 falls substantially below 0.977, or if inter-radiologist agreement is itself below the assumed level, the claim of near-perfect curation fails on independent ground truth.","tokens_in":7624,"feed_emoji":"🩻","tokens_out":5582,"duration_ms":51677,"temperature":0.7,"pith_summary":"The paper asks whether visual information alone can replace unreliable text mining when harvesting medical imaging datasets from hospital PACS systems. It claims that a deliberately small 3D convolutional network with squeeze-and-excitation blocks, trained on noisy DICOM-tag labels, can recognize the scan phase of dynamic liver CT studies—non-contrast, arterial, venous, or delay—with a mean F1 of 0.977 on a held-out test set. Because many scans are only labeled as generic \"contrast\", the authors introduce an aggregated cross-entropy loss that treats contrast as the union of the three contrast phases, letting the model learn from weakly labeled data instead of discarding it. They report that this approach correctly harvests 92.7% of studies, improving on both the text-mined baseline and larger 3D architectures. If correct, the work makes large-scale, low-labor curation of clinical CT data feasible for downstream modeling.","feed_headline":"A compact 3D vision model labels liver CT phases at 97.7% F1","feed_subtitle":"Automated PACS curation harvests 92.7% of dynamic liver studies, beating text-mined DICOM tags and much larger 3D networks.","key_machinery":"The load-bearing object is the 3DSE network paired with the aggregated cross-entropy (ACE) loss. 3DSE is a streamlined 3D convolutional classifier that downsamples volumes to 128x128x32, extracts features with two 3D convolutions, recalibrates feature channels using squeeze-and-excitation blocks to inject global context, and outputs five classes (non-contrast, arterial, venous, delay, other). The ACE loss converts each weakly labeled \"contrast\" scan into a training signal by defining $p_{\\text{contrast}} = p_{\\text{arterial}} + p_{\\text{venous}} + p_{\\text{delay}}$ through the logits and using a logsumexp formulation of cross entropy that is numerically stable. This lets scans without a precise phase label still contribute gradients that sharpen the model's phase predictions, which is the mechanism behind the reported F1 and harvest-rate gains.","core_discovery":"On its own terms, the paper establishes that a compact 3DSE classifier trained with weak text-mined labels and an aggregated cross-entropy (ACE) loss can identify dynamic liver CT phases with mean F1 of 0.977 at scan level and correctly extract 92.7% of studies, meaning all scans in the study are classified correctly. The improvement over text mining is driven by visual cues that cannot always be recovered from imperfect DICOM tags: venous scans, for example, rise from F1 0.868 to 0.963. The ACE loss contributes by encoding the hierarchical relation that a scan labeled \"contrast\" is one of arterial, venous, or delay, expressed as the sum of their softmax probabilities, so weakly supervised scans help train the fine-grained outputs. The authors also report that a 19.22 MB model outperforms a 33.56 MB C3D and a 325.22 MB 3D-ResNet, attributing the latter's failure to overfitting.","pith_inferences":["The same aggregated-loss trick applies to any curation problem where a coarse label is a union of fine classes, such as \"tumor\" versus its subtypes, so ACE could be reused beyond CT phases.","Cases where text labels conflict with visual content suggest a hybrid system that lets the visual model flag or correct suspicious DICOM tags at harvest time, an extension the authors only gesture toward when listing future work on harmonizing text and vision.","A natural testable extension is to run the same 3DSE+ACE pipeline on a different organ's dynamic protocol, such as renal or pancreatic CT, and check whether the 0.977-level F1 transfers or whether new phase-specific visual cues are required.","The reported superiority of 3DSE over 3D-ResNet may reflect dataset scale as much as architecture; on larger training sets the larger model might close the gap, since the paper's training set is about 30K scans."],"forward_implications":["Hospitals can harvest dynamic liver CT studies from messy PACS archives without hand-curating every scan, using text-mined rules only to seed a visual model.","Weakly labeled scans marked only \"contrast\" no longer need to be discarded; the ACE loss turns them into usable training data for fine-grained phase outputs.","Small streamlined 3D models with SE layers are a viable alternative to much larger 3D networks for this classification task, with less overfitting and faster training.","Study-level harvest rates near 93% mean downstream analyses on multi-phase liver CT can draw on thousands of studies that would otherwise be lost or require manual review."],"supporting_citations":[{"why":"Documents that DICOM header tags are error-prone and unreliable, motivating the need for visual curation beyond text mining.","marker":"[12]"},{"why":"Squeeze-and-excitation blocks are the mechanism that injects global context into the compact 3DSE architecture.","marker":"[14]"},{"why":"C3D is the baseline 3D convolutional architecture that the streamlined 3DSE model is compared against and outperforms.","marker":"[11]"},{"why":"3D-ResNet is the larger baseline that overfits and is outperformed, framing the model-size comparison.","marker":"[13]"},{"why":"Provides the randomized statistical test used to show that F1 gains on the NC, V, and O phases are significant.","marker":"[15]"},{"why":"Holm-Bonferroni correction is applied to the multiple-comparison significance testing of phase-level F1 improvements.","marker":"[16]"}],"fun_headline_variants":["Streamlined 3D SE net labels liver CT phases at 97.7% F1","Small 3D model beats 3D-ResNet on liver CT phase labeling","Automated liver CT curation harvests 92.7% of studies","Weak DICOM tags plus compact net label liver CT phases"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire evaluation depends on the manual phase labels of the validation and test sets being correct, but the paper gives no information about how many annotators labeled them, their training, or how disagreements were resolved.","fun_headline_variants_meta":{"raw":{"variants":["Streamlined 3D SE net labels liver CT phases at 97.7% F1","Small 3D model beats 3D-ResNet on liver CT phase labeling","Automated liver CT curation harvests 92.7% of studies","Weak DICOM tags plus compact net label liver CT phases"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000795,"raw_usage":{"total_tokens":3542,"prompt_tokens":1030,"completion_tokens":2512,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":646,"completion_tokens_details":{"reasoning_tokens":2428}},"tokens_in":646,"tokens_out":2512,"duration_ms":20879,"temperature":1.0,"reasoning_tokens":2428,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:47:25.341595+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of, say, 300 scans from the same hospital PACS distribution, have several board-certified radiologists independently label the phase, and measure the model's F1 against their consensus; if mean F1 falls substantially below 0.977, or if inter-radiologist agreement is itself below the assumed level, the claim of near-perfect curation fails on independent ground truth.","supporting_citations":[{"cited_title":"Wein, H.S., Bredno, J., Lehmann, T.M.: Quality of dicom header information for image categorization","cited_arxiv_id":null,"evidence_quote":"Documents that DICOM header tags are error-prone and unreliable, motivating the need for visual curation beyond text mining."},{"cited_title":"In: IEEE conference on computer vision and pattern recognition","cited_arxiv_id":null,"evidence_quote":"Squeeze-and-excitation blocks are the mechanism that injects global context into the compact 3DSE architecture."},{"cited_title":"In: IEEE international conference on computer vision","cited_arxiv_id":null,"evidence_quote":"C3D is the baseline 3D convolutional architecture that the streamlined 3DSE model is compared against and outperforms."},{"cited_title":"In: IEEE CVPR","cited_arxiv_id":null,"evidence_quote":"3D-ResNet is the larger baseline that overfits and is outperformed, framing the model-size comparison."},{"cited_title":"In: Proceedings of the 18th Conference on Computational Linguistics - Volume 2","cited_arxiv_id":null,"evidence_quote":"Provides the randomized statistical test used to show that F1 gains on the NC, V, and O phases are significant."},{"cited_title":"Scandinavian Journal of Statistics 6 (1979) 65–70","cited_arxiv_id":null,"evidence_quote":"Holm-Bonferroni correction is applied to the multiple-comparison significance testing of phase-level F1 improvements."}],"review_version":1}