{"id":"02368571-690e-4fc0-bad5-a40156f785ac","arxiv_id":"2501.11428","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A multi-organ segmentation pipeline labels coronary calcifications per vessel on non-contrast cardiac CT and reports inter-observer-level Agatston scoring on the orCaScore benchmark.","lead":"This paper presents an automated system that segments the heart's anatomy in non-contrast CT scans to locate and label coronary artery calcifications, then computes per-vessel calcium scores. Its reported accuracy is on par with human experts on the orCaScore challenge, with interpretable color-coded maps of calcifications per vessel.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ground-truth generation for the in-house 122-scan test set may share the same distance-to-coronary-artery filtering assumption as the evaluated pipeline, which can inflate the internal inter-observer-level results; the official orCaScore comparison is independent but covers only 40 scans.","rationale":"The reader's weakest-assumption identification is essentially correct: the internal ground-truth annotation process in Section 2.1 starts from calcification proposals generated by the same kind of distance-to-CA criterion that the evaluated pipeline uses for filtering in Section 2.2.3. This is a real circularity risk for the 122-scan internal evaluation, and it directly affects the strong ICC, Pearson correlation, and per-vessel DICE numbers that support the broader claim of inter-observer-level accuracy. However, the paper also reports an evaluation against the official orCaScore challenge ground truth on the 40 challenge test cases, and that reference is externally generated and not contaminated by the authors' CA segmentation model. I therefore would not reject the paper or characterize the central claim as unsupported: the official comparison provides meaningful independent evidence for the state-of-the-art and inter-observer-level statements. But that external evidence is limited to 40 scans, and the paper's even stronger quantitative claims on 122 scans depend on internal ground truth whose independence is not established. Given that no code, data, or detailed observer-agreement statistics are provided, a conditional acceptance with a request for an independently regenerated internal ground truth is the appropriate outcome. The reader's conditional verdict should stand.","tokens_in":10064,"tokens_out":10354,"duration_ms":121093,"concrete_test":"Re-annotate a random subset of the internal test set (e.g., 30 of 122 scans) without the automated CA-distance proposal step: threshold all voxels ≥ 130 HU, have two independent experts annotate and label calcifications de novo, and reconcile disagreements. Recompute Table 1 ICC and Table 2 DICE for this subset with the current pipeline. A drop in weighted DICE of more than about 0.02 or an ICC below 0.95 would confirm that the ground-truth circularity inflates the internal metrics; statistically unchanged metrics would resolve the concern.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The least-secure assumption behind the central claim is the independence of the in-house reference standard. Section 2.1 states that calcification proposals for the ground truth were obtained by \"automated thresholding (130 HU) and the distance to CA,\" and Section 2.2.3 filters calcification candidates using the same distance-to-filtered-CA-mask rule with a 3-mm dilation. If the CA masks used in the ground-truth proposal step come from the same coronary-artery segmentation model family (Bujny et al., 2024b) that the pipeline incorporates, then the ground truth and the evaluated method share a systematic prior: lesions that the CA segmentation would place far from the predicted arteries are less likely to appear in the ground truth at all, so both false-positive and false-negative errors of the filtering module are underestimated. Expert manual correction mitigates but does not remove this bias, because the experts are correcting a proposal image rather than annotating from scratch. This threatens the 122-scan internal results (Fig. 4 rows labeled \"our GT,\" Table 1 ICC ≈ 1.00, Fig. 5 r = 0.9997, and Table 2 per-vessel DICE) that support the \"inter-observer level\" and \"best available\" statements. The official orCaScore-GT row is independent and is a genuine supporting pillar, but it covers only the 40 challenge cases and does not by itself establish the broader internal-validation claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a multi-stage, anatomy-guided pipeline for coronary artery calcium (CAC) scoring on non-contrast, ECG-gated cardiac CT. Calcification candidates are obtained by thresholding at 130 HU; nnU-Net models segment the pericardium, aorta, and coronary arteries; CNNs localize the aortic valve center and the two coronary ostia; candidates are filtered using dilated pericardium and coronary-artery masks and then labeled as aortic, RCA, LM, LAD, or LCx using a vessel-tree graph and a logistic-regression classifier; per-vessel and total Agatston scores are computed from the labeled voxels. The method is evaluated on a 534-scan multi-vendor collection (122-scan internal test set with expert ground truth), on the orCaScore Grand Challenge official test set (40 scans via an official submission), and on 24 scans where both the authors' ground truth and the challenge ground truth are available. The authors claim inter-observer-level accuracy and state-of-the-art results on orCaScore, report near-perfect agreement on the internal test set (ICC 1.00, Pearson r = 0.9997), and demonstrate rejection of pacemaker-lead, noise, and valve calcifications.","tokens_in":10415,"tokens_out":14633,"duration_ms":137826,"significance":"If the claims hold, this is a practically relevant contribution: interpretable per-vessel calcification maps and an externally, independently scored benchmark result, which is unusual and should be credited. The official orCaScore Grand Challenge submission anchors the core scoring claim in a way that purely internal evaluations cannot, and the 24-common-case comparison between two ground-truth sets is a good-faith attempt to address reference-standard differences. The multi-vendor 534-scan collection is substantially larger than the original challenge data. The main caveat is that the internal reference standard was built with the same distance-to-coronary-artery heuristic used by the pipeline itself, so the near-perfect internal metrics must be treated with caution until the provenance of the ground-truth coronary-artery masks and the extent of expert-only additions are disclosed. The lack of code or model release and the absence of an observer-variability baseline further temper the reproducibility and the precise wording of the 'inter-observer level' claim.","major_comments":[{"comment":"The in-house reference standard is at risk of sharing the method's own filtering assumption. Section 2.1 states that ground-truth calcification proposals were obtained 'using automated thresholding (130 HU) and the distance to CA' and then manually corrected by experts, while Section 2.2.3 filters calcification candidates by intersection with a 3-mm-dilated coronary-artery (CA) mask, using CA segmentations from the same model family referenced in Section 2.2.1 (Bujny et al., 2024b). If the CA masks used in the GT proposal step come from that same segmentation approach, the internal reference standard inherits the pipeline's systematic prior: lesions that the CA model places far from the predicted arteries are underrepresented in the GT, so both false positives and false negatives of the distance-filtering module are underestimated in the internal evaluation (Table 1 ICC near 1.00, Fig. 5 r = 0.9997, Table 2 per-vessel DICE, and the risk-category kappa values). Expert correction mitigates but does not eliminate this bias, because the experts edit a proposal image rather than annotating from scratch. The official orCaScore GC row in Fig. 4(a) is independent and remains a valid supporting pillar, but it covers only 40 scans and does not by itself substantiate the 'more than 500 scans' and 'inter-observer level' claims built on the 122-scan internal set. Please (i) disclose which CA masks were used in the GT proposal step, (ii) report the proportion of GT calcification voxels or lesions added manually by the experts rather than accepted from the automated proposals, and (iii) report GT-to-GT agreement (e.g., per-scan DICE between the authors' GT and the official orCaScore GT on the 24 common cases of Fig. 4(c)) as a direct bound on this bias.","section":"§2.1, §2.2.3"},{"comment":"The claim of 'inter-observer level accuracy' (abstract and Section 3) is not fully operationalized. The comparison to 'Observer 1' is a comparison to a single manual segmentation, and no observer-observer variability distribution is reported; in the standard usage, a method is at the inter-observer level when its agreement with a reference falls within the spread between pairs of human observers. On the official orCaScore data, the method's count-weighted sensitivity (0.927) is below Observer 1's (0.943), so the claim needs a precise definition and a supporting analysis. Please add a subset of scans annotated by at least two observers and report the distribution of pairwise observer agreement against which the method's agreement can be judged, or temper the claim accordingly.","section":"§3, Fig. 4(a)"},{"comment":"The statement that the technique 'outperforms [the 3rd place method] ... making it the best available' rests, on the official 40-scan test set, on a weighted DICE difference of 0.980 vs 0.979 (Fig. 4(a)), which is within rounding; the material advantages are the count-weighted sensitivity (0.927 vs 0.767) and the weighted sensitivity (0.974 vs 0.970), while the weighted specificity is slightly lower (0.986 vs 0.988). Please report per-case paired differences with bootstrap confidence intervals for the headline metrics on the 40 official scans, and state the date and ranking composition of the 'currently the best' leaderboard entry, since leaderboard claims are time-indexed. Without this, the superiority claim is not yet statistically supported.","section":"§3, Fig. 4(a)"},{"comment":"Several load-bearing parameters are hand-tuned and their influence on the final scores is not analyzed. Section 2.2.3 describes the pericardium and CA dilation sizes (1 mm and 3 mm) as 'experimentally fine-tuned', and Sections 2.2.2 and 2.2.4 add patch sizes (67×67×67 mm, 25×25×25 mm) and the features of the logistic-regression classifier for LCA subregion labeling without stating which data were used for tuning. Because the 3-mm CA dilation directly determines which calcification candidates survive to scoring, the multi-vendor robustness claim (Section 3) is under-supported: a scanner or protocol for which the CA segmentation is systematically narrower could change scores materially. Please specify the tuning protocol (training-only, with no test-set information) and provide a sensitivity analysis, e.g., sweeping the CA dilation over a plausible range and reporting ICC or per-subject error on a held-out subset.","section":"§2.2.2–§2.2.4"}],"minor_comments":[{"comment":"The abstract describes 'an open-source multi-vendor dataset'; the orCaScore data are publicly available for research but are not open-source in the software sense, so please rephrase.","section":"Abstract"},{"comment":"The term 'end-to-end' is inaccurate for a pipeline composed of separately trained segmentation networks, regression CNNs, and hand-set geometric rules; 'multi-stage' would be more precise.","section":"§1.2, §2.2"},{"comment":"The '—' entries for Observer 1 and the 3rd place method in the 'our GT' columns are unexplained; the caption should clarify that those metrics are only defined where the corresponding reference standard exists.","section":"Fig. 4(a)"},{"comment":"The scanner manufacturer 'Cannon' should be 'Canon'.","section":"§2.1"},{"comment":"The sentence 'Manual analysis of of multi-vendor CTs' contains a duplicated 'of'.","section":"§4"},{"comment":"The LM region shows markedly lower agreement (ICC 0.85, CI 0.8–0.89; mean DICE 0.730 in Table 2) than the other vessels and than the total Agatston score; the text's 'nearly perfect agreement' should acknowledge this per-vessel degradation and discuss the difficulty of ostial splitting, which is also relevant to the interpretability claim.","section":"Table 1, Table 2"},{"comment":"No data or code availability statement is provided; releasing the trained segmentation models, or at least describing the official submission route in enough detail for independent replication, would materially improve reproducibility.","section":"§3"}],"recommendation":"major_revision","confidential_remarks":"At decision time, the editor may wish to verify the 'currently the best' orCaScore leaderboard claim, including its date and the identity of the previously best entry, since the weighted-DICE margin over the prior 3rd-place method is 0.001 on the official test set. The companion works (Bujny et al., 2024a/b) appear to carry a substantial part of the segmentation machinery, so the boundary between this paper's novel contribution and the companion papers should be sharpened. The authors are from a commercial imaging company; this is not itself a concern, but a data/code availability plan would help the reproducibility assessment. The main risk is the ground-truth circularity described in Major Comment 1, which is fixable by disclosure and additional analysis rather than being fatal to the central claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid systems paper, and the official orCaScore submission is the real evidence. The internal 122-scan results are less secure than they look.\n\nWhat's new: the integrated pipeline, combining pericardium/aorta/coronary-artery segmentation, CNN-based ostia localization, graph-based vessel labeling, and distance-based calcification labeling, produces per-vessel calcium maps with a rationale you can show a clinician. The individual pieces are established, and CAC scoring is already largely automated, so this is not a fundamentally new capability. But the packaging is useful, and the interpretability claim is honest.\n\nWhat's good: they evaluated on an independent external benchmark (the official orCaScore 40-case test) and their official scores are at inter-observer level. That is strong evidence. The multi-vendor set of over 500 scans is a real dataset. The qualitative cases showing removal of pacemaker artifacts, noise, and valve calcifications are helpful. They also credit the CA segmentation model they build on rather than presenting it as new.\n\nThe main soft spot is the in-house ground truth. Section 2.1 says proposals were generated using automated thresholding at 130 HU and distance to the coronary arteries, with experts manually correcting them. Section 2.2.3 filters calcification candidates by distance to a dilated CA mask. If the CA masks used for GT proposals come from the same model family (Bujny et al. 2024b), then the GT and the method share a prior about where calcium can be, and the internal metrics (ICC 1.00, Pearson r = 0.9997) can be inflated. Manual correction reduces but does not remove this bias, because the experts are correcting a proposal image rather than annotating from scratch. This does not sink the central scoring claim, since the official orCaScore result is independent, but it does mean the 'inter-observer level' claim for the 122-scan set is not fully established.\n\nOther issues are minor: dilation sizes are hand-tuned, no code or data are released, and the comparison is to the 3rd-place orCaScore method rather than more recent state of the art. The official test set is only 40 cases, so both pillars have limits.\n\nBottom line: this deserves a serious referee. I would ask for clarity on how the CA masks for GT proposals were made, ideally with an independently annotated subset, plus code or data release and a broader comparison. It is a conditional accept with real work, not a desk reject.","headline":"A genuinely interpretable per-vessel CAC pipeline with a real external benchmark result; the internal validation is weaker than the paper claims because the ground truth and the method share a filtering assumption.","tokens_in":10972,"tokens_out":2155,"would_cite":true,"duration_ms":20772,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By segmenting the heart's anatomy before scoring calcium, the paper claims automated CAC scoring reaches the accuracy of human observers.","keywords":["coronary artery calcium scoring","Agatston score","non-contrast CT","multi-organ segmentation","coronary artery segmentation","interpretable AI","per-vessel calcium labeling","orCaScore challenge"],"falsifier":"Run the pipeline on a test set whose ground truth is produced by manual expert delineation alone, without any automated thresholding or distance-to-coronary-artery seeding during annotation, and compare total and per-vessel Agatston agreement against the paper's reported ICC 1.00 and Pearson 0.9997; a drop below inter-observer agreement would show the reported accuracy depends on the annotation protocol rather than on the anatomy-informed method itself.","tokens_in":9886,"feed_emoji":"❤️","tokens_out":6482,"duration_ms":59249,"temperature":0.7,"pith_summary":"The paper argues that the standard recipe for automated coronary artery calcium (CAC) scoring, which treats calcification detection as a direct lesion-segmentation problem, leaves accuracy and trust on the table. It proposes instead an anatomically-informed pipeline that first segments the pericardium, aorta, and coronary arteries in non-contrast cardiac CT, then uses those structures to filter, localize, and label every high-density voxel before computing the Agatston score. On a 534-scan multi-vendor dataset and the orCaScore benchmark, the authors report agreement with manual scoring at the inter-observer level (overall ICC 1.00, Pearson correlation 0.9997) and state that the approach outperforms the previous best orCaScore algorithm. If correct, the method delivers not only accurate total and per-vessel calcium scores but also segmentation maps a clinician can inspect and verify, addressing a known blind spot of black-box CAC models.","feed_headline":"Heart anatomy segmentation lifts calcium scoring to expert level","feed_subtitle":"Labels each lesion by vessel, filters artifacts, and matches manual Agatston scores almost perfectly.","key_machinery":"The load-bearing mechanism is an anatomy-first filtering and labeling cascade. A 130 HU threshold produces calcification candidates; segmentation masks of the pericardium and coronary arteries, obtained with nnU-Net models, discard candidates outside the heart or too far from an artery; CNNs localize the aortic valve center and the two coronary ostia; an Euclidean-distance separation plane splits each ostial calcification into aortic and coronary parts; and a skeletonized, graph-organized model of the left coronary tree, traversed with logistic-regression junction classification, assigns each remaining calcification to one of four territories. The coronary artery mask is the central object: both the false-positive filter and the final label assignment are defined by distance to it, so its quality bounds the interpretability and accuracy of the whole pipeline.","core_discovery":"The central discovery is that precise multi-organ segmentation of the heart, rather than end-to-end calcification detection, is sufficient to bring automated CAC scoring to the accuracy of a human observer while preserving interpretability. The authors claim their pipeline extracts calcification candidates by thresholding at 130 HU, removes false positives using dilated pericardium and coronary-artery masks, splits ostial calcifications on a separation plane between aorta and artery, and labels the remaining lesions as RCA, LM, LAD, or LCx using a skeletonized vessel-tree graph. In their experiments the total Agatston score agrees almost perfectly with expert ground truth and the segmentation metrics equal or exceed those of the reference observer; on the orCaScore challenge tests the method is the best available automated result, reaching inter-observer accuracy.","pith_inferences":["If coronary-artery segmentation quality keeps improving with more training data, the same anatomy-first cascade could plausibly transfer to non-ECG-gated or low-dose chest CT, where vessels are harder to see and current scoring is less reliable.","The ostia separation-plane trick is a template for splitting ambiguous calcium at other vessel or valve boundaries, e.g., quantifying mitral annular or aortic valve calcium separately from coronary calcium.","The pipeline's explicit dependence on a CA mask suggests a natural testable upgrade: attach an uncertainty estimate to the CA segmentation and flag cases where filtering decisions are made near its boundary, which would tell clinicians when to override the score.","A direct observer study could measure whether clinicians catch the pipeline's mistakes more readily with labeled vessel-tree maps than with black-box score outputs, testing the paper's core interpretability argument rather than just its accuracy."],"forward_implications":["Total Agatston scores can be computed automatically at near-perfect agreement with expert reading (ICC 1.00, Pearson 0.9997), making objective large-scale screening feasible.","Per-vessel labels (RCA, LM, LAD, LCx) turn a single risk number into a territory-resolved calcification map, supporting more targeted treatment decisions.","Anatomical filtering removes false positives from pacemaker leads, scanner noise, and aortic or mitral valve calcifications without explicit artifact models.","The six-minute end-to-end runtime on a single GPU is compatible with routine clinical use.","Visual maps of labeled calcifications let clinicians audit why a score was produced, addressing the opacity of end-to-end deep-learning scorers."],"supporting_citations":[{"why":"Supplies the orCaScore challenge data and evaluation protocol that the paper uses as its state-of-the-art benchmark.","marker":"Wolterink et al., 2016"},{"why":"Provides the training procedure and nnU-Net coronary-artery segmentation models that generate the central CA mask.","marker":"Bujny et al., 2024b"},{"why":"Supplies the aorta segmentation model used for ostia localization and aortic/CA separation.","marker":"Bujny et al., 2024a"},{"why":"Defines the U-Net encoder architecture that the three localization CNNs are built from.","marker":"Ronneberger et al., 2015"},{"why":"Provides the 3D skeletonization algorithm that converts the left coronary mask into the graph used for vessel labeling.","marker":"Lee et al., 1994"},{"why":"Gives the medical-standard 130 HU threshold that seeds all calcification candidates.","marker":"Hou et al., 2020"},{"why":"Defines the Agatston, volume, and mass scoring formulas that the pipeline must reproduce.","marker":"Blaha et al., 2017"}],"fun_headline_variants":["Anatomy-first CAC scoring hits inter-observer accuracy","Heart segmentation refines calcium scoring to expert level","Multi-organ segmentation boosts CAC scoring interpretability","Vessel-labeled calcifications via heart anatomy model","CAC scoring matches human experts with anatomy insight"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ground-truth labels on the authors' own test set were seeded by automated 130 HU thresholding and distance filtering to a coronary-artery mask, then manually corrected; if that seed mask came from the same kind of segmentation the pipeline itself relies on, the test metrics may partly reward the pipeline's own filtering assumptions rather than measuring it against fully independent human judgment.","fun_headline_variants_meta":{"raw":{"variants":["Anatomy-first CAC scoring hits inter-observer accuracy","Heart segmentation refines calcium scoring to expert level","Multi-organ segmentation boosts CAC scoring interpretability","Vessel-labeled calcifications via heart anatomy model","CAC scoring matches human experts with anatomy insight"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000197,"raw_usage":{"total_tokens":1336,"prompt_tokens":885,"completion_tokens":451,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":501,"completion_tokens_details":{"reasoning_tokens":380}},"tokens_in":501,"tokens_out":451,"duration_ms":5032,"temperature":1.0,"reasoning_tokens":380,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:16:04.076601+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the pipeline on a test set whose ground truth is produced by manual expert delineation alone, without any automated thresholding or distance-to-coronary-artery seeding during annotation, and compare total and per-vessel Agatston agreement against the paper's reported ICC 1.00 and Pearson 0.9997; a drop below inter-observer agreement would show the reported accuracy depends on the annotation protocol rather than on the anatomy-informed method itself.","supporting_citations":[{"cited_title":", author Leiner, T","cited_arxiv_id":null,"evidence_quote":"Supplies the orCaScore challenge data and evaluation protocol that the paper uses as its state-of-the-art benchmark."},{"cited_title":", author Fischer, P","cited_arxiv_id":null,"evidence_quote":"Defines the U-Net encoder architecture that the three localization CNNs are built from."},{"cited_title":", author Tsujioka, K","cited_arxiv_id":null,"evidence_quote":"Gives the medical-standard 130 HU threshold that seeds all calcification candidates."},{"cited_title":", author Mortensen, M.B","cited_arxiv_id":null,"evidence_quote":"Defines the Agatston, volume, and mass scoring formulas that the pipeline must reproduce."}],"review_version":1}