{"id":"23c157f2-656a-4cee-88c9-cb793b47cb85","arxiv_id":"2412.02568","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"On the public ARCADE coronary stenosis segmentation benchmark, the U-Mamba BOT model achieves an F1 score of 68.79%, the highest reported to date.","lead":"This paper applies six state-of-the-art segmentation models, mostly based on the Mamba architecture, to detect narrowed coronary arteries in X-ray angiography images. Its best model, U-Mamba BOT, scores an F1 of 68.79% on the public ARCADE dataset, beating the previous best published result by 11.8 points.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"F1 metric for U-Mamba BOT is unspecified; without the official ARCADE evaluation criterion, the cited 68.79% may not be comparable to prior SOTA.","rationale":"The reader identified comparability of prior F1 scores as the weakest assumption. I agree that the comparison to prior work is the most load-bearing element, but I refine the concern: the test split is likely the official ARCADE split (the prior works are challenge entries or explicitly use the ARCADE test set), whereas the metric definition is genuinely ambiguous. The paper does not specify how F1 is computed, and the ARCADE challenge uses a non-trivial component-based metric. If the authors used a standard per-pixel Dice/F1, their scores are not on the same scale as the cited prior numbers, which would invalidate the headline improvement. This is a concrete, checkable flaw that directly affects the central claim. The manuscript's internal precision/recall values are arithmetically consistent with the reported F1, so the concern is not about internal inconsistency but about external comparability. The appropriate verdict remains CONDITIONAL because the claim may well be correct, but it requires the authors to supply the metric definition and run the official evaluation to confirm. No code or supplementary material is provided, so independent verification is currently impossible. My proposed test directly settles whether the concern lands: recompute with the official ARCADE metric and compare.","tokens_in":7469,"tokens_out":6376,"duration_ms":59763,"concrete_test":"Obtain the official ARCADE evaluation script (or reimplement the stenosis F1 metric as specified in Popov et al., Scientific Data 11:20, 2024) and run it on the predicted masks for U-Mamba BOT from Table 1. Compare the resulting F1 to 68.79%. Also run the same script on the published predictions of SSASS, StenUNet, and Ensemble Learning (if publicly available) or recompute their F1 with the same code from their published outputs. If the official-metric F1 deviates from 68.79% by more than 1-2 points, or if the recomputed prior scores are materially different from the cited values, the SOTA claim fails. In addition, the paper should explicitly state the metric definition (per-pixel vs. component-based, matching threshold, etc.) and confirm that it matches the ARCADE challenge protocol.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that U-Mamba BOT sets a new state of the art with F1=68.79% on the ARCADE stenosis segmentation task, an 11.8-point improvement over SSASS (56.99%). For this claim to hold, the F1 reported for U-Mamba BOT must be computed with the same metric and evaluation protocol as the F1 values of Ensemble Learning, StenUNet, and SSASS cited in Table 1. The paper never defines its F1 computation. It does not state whether F1 is per-pixel, per-connected-component, or per-lesion; it does not give the matching criterion, the binarization threshold, or any post-processing steps; and it does not indicate whether the official ARCADE evaluation script was used. Prior ARCADE works (e.g., [2], [13]) report challenge scores, which are generated by the official ARCADE metric—a component/region-based F1 with a specific overlap rule, as described in the dataset paper [20]. If the authors instead computed a plain per-pixel F1/Dice, their number is not comparable, and the apparent 11.8-point gain could be a metric artifact. The paper also provides no error bars or multiple runs, so the significance of the difference is unknown. Without the metric definition, the central SOTA claim is unverifiable and the comparison to prior work is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates five Mamba-based U-Net variants (U-Mamba BOT, U-Mamba ENC, LightM-UNet, Swin-Umamba, Swin-Umamba D) and one Swin Transformer model (Swin UNetR) for coronary artery stenosis segmentation on the public ARCADE dataset. Using 1200 training images with 5-fold cross-validation and a held-out 300-image test set, the authors report that U-Mamba BOT achieves the best F1 score of 68.79%, an 11.8-point improvement over the previous SSASS result of 56.99%. The paper is an empirical benchmark comparison; it does not introduce a new architecture or release code.","tokens_in":7728,"tokens_out":6717,"duration_ms":68266,"significance":"If the comparison is valid, the paper provides useful evidence that Mamba-based U-Net backbones can improve stenosis localization in X-ray angiography on a standardized public benchmark. Strengths include the use of a public dataset, a held-out test set, and internal arithmetic consistency between the reported precision, recall, and F1 values in Table 1. The significance, however, depends critically on whether the F1 scores are computed with the same metric and evaluation protocol as the cited prior work; this is not documented in the manuscript. The apparent 11.8-point improvement over SSASS is therefore not yet established.","major_comments":[{"comment":"The F1 metric is never defined. The manuscript must state whether F1 is computed per-pixel, per-connected-component, or per-lesion; give the binarization threshold and matching/overlap rule; and specify any post-processing. The cited prior results (39.41%, 53.48%, 56.99%) are benchmark numbers from the ARCADE challenge literature and were likely produced with the official ARCADE evaluation protocol. If the authors used a plain pixel-wise F1/Dice instead of that protocol, the numbers in Table 1 are not comparable and the claimed 11.8-point improvement over SSASS could be a metric artifact. This is a load-bearing point for the central state-of-the-art claim.","section":"Section 5, Table 1"},{"comment":"Only single-run results are reported, with no confidence intervals, standard deviations, or significance tests. U-Mamba BOT's F1 of 68.79% is only 0.22 percentage points above U-Mamba ENC (68.57%) and about 2.1 points above Swin-Umamba (66.82%). Without multiple runs or a statistical comparison, the ranking of the best model and the claimed improvement over second-best are not robust.","section":"Section 5, Table 1"},{"comment":"The experimental setup is described too sparsely for reproducibility. The paper does not report the loss function, optimizer, learning rate, batch size, number of epochs, data augmentation, or the rule for selecting the final model from 5-fold cross-validation. It also does not clarify whether the 300-image test set is exactly the official ARCADE test split or whether any of the 1200 training images overlap with validation. The authors should either release code/configuration files or provide these details so that the reported F1 values can be independently checked.","section":"Sections 3 and 4"},{"comment":"The parameter counts should be verified. In particular, U-Mamba BOT is reported as 500M parameters while U-Mamba ENC is 104M; this is disproportionate and inconsistent with the public U-Mamba configurations, which typically place U-Mamba BOT far below 500M. Since Figure 4 uses parameter counts for the efficiency comparison, an erroneous count would alter the figure's message.","section":"Table 1, Figure 4"}],"minor_comments":[{"comment":"The text says 'In this figure, we can see...' when referring to Table 1; this should be corrected.","section":"Section 5"},{"comment":"The model name is written inconsistently as 'Sten UNet', 'StenUNet', and 'StenUNet' in different places; standardize the spelling.","section":"Table 1 and Section 2"},{"comment":"The phrase 'an F1-score as high as 0.5348 on a test set only 0.0005 points less than that of the 2nd place' is ambiguous; it should specify which test set and which rank is being compared.","section":"Section 2"},{"comment":"There are grammatical errors such as 'we will going to discuss' and 'The encoder expands... four parts'; these should be corrected in a polished revision.","section":"Section 4.2"},{"comment":"The caption's instruction to 'check the central point of each circle' is unclear; provide axis labels, a legend, and an explicit explanation of what the circle area represents.","section":"Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads like a workshop-style benchmark report rather than a full journal article. The core issue is that the evaluation protocol, especially the F1 definition, is unspecified, and without it the central state-of-the-art claim is unverifiable. If the authors can rerun the evaluation with the official ARCADE scoring script and provide the missing experimental details, the comparison may become publishable. The paper's novelty is limited to applying existing architectures to a public dataset, so the editor may also want to consider whether the journal's scope requires more methodological contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a systematic benchmark of six U-shaped segmentation models on the ARCADE stenosis task, and it reports a new best F1 of 68.79% for U-Mamba BOT. That is a real gain over the cited 56.99%. But the paper never defines the F1 metric, so the headline comparison rests on an unverified assumption. It deserves referee time, but not acceptance without substantial revision.\n\nWhat is actually new is the measured result, not the architecture. The paper imports six existing models, runs them on the same public dataset, and publishes the numbers. That is a legitimate, if incremental, contribution. The precision/recall/F1 arithmetic for the best model checks out (2*0.6992*0.6769/(0.6992+0.6769)=0.6879), and the use of 5-fold cross-validation on the first 1200 images with a held-out 300-image test set matches the public ARCADE split. The comparison across model sizes is also useful.\n\nThe soft spot is load-bearing: the paper says “F1 score” but never says what F1 means. The ARCADE challenge scores are region-based, computed with a specific overlap rule, not plain per-pixel Dice. If the authors used pixel-wise F1, their 68.79% is not comparable to the official ARCADE numbers they cite for Ensemble Learning, StenUNet, and SSASS. The 11.8-point gain could be a metric artifact. There are no error bars, no multiple runs, no training hyperparameters, and no code, so the reader cannot tell whether the difference is noise. There is also a small internal inconsistency: the related work says SSASS got 53.6% F1, but Table 1 lists 56.99%. The claim that U-Mamba BOT is “best” would be more credible with a significance test or at least a second run.\n\nWho is this for? Someone tracking Mamba models in medical imaging or benchmarking on ARCADE will get a quick orientation and a plausible new SOTA candidate. A serious editor should send this to peer review, because the result is new and the missing metric definition is fixable, not fatal. The referee should require the official ARCADE evaluation script or a clear statement of how F1 is computed, plus training details and code. I wouldn't cite it in my own work until those are in place.","headline":"A benchmark paper with a new top F1 on ARCADE stenosis, but the metric definition is missing, so the headline improvement may not be apples-to-apples.","tokens_in":8266,"tokens_out":1708,"would_cite":false,"duration_ms":18654,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Mamba-based U-Net reaches 68.79% F1 for coronary stenosis segmentation, an 11.8-point jump over the previous best.","keywords":["coronary artery stenosis segmentation","X-ray angiography","Mamba state space models","U-Net architecture","selective scan 2D","ARCADE dataset","medical image segmentation","F1 score"],"falsifier":"Re-run Ensemble Learning, StenUNet, and SSASS on the same 300-image ARCADE test images with identical preprocessing, metric computation, and thresholding; if any of them matches or exceeds 68.79% F1, or if U-Mamba BOT's score drops on a corrected split, the paper's central improvement claim fails. A secondary check is to evaluate U-Mamba BOT on an independent coronary angiography dataset with similar stenosis annotations.","tokens_in":7266,"feed_emoji":"🩻","tokens_out":3999,"duration_ms":34471,"temperature":0.7,"pith_summary":"This paper asks whether Mamba-based sequence models can localize coronary artery stenosis—the narrowed, often blurry sections of arteries—in X-ray angiography images. It trains six U-shaped segmentation models on the ARCADE dataset and reports that U-Mamba BOT, a U-Net with Mamba blocks only in the bottleneck, reaches an F1 score of 68.79% on the 300-image test set. That is an 11.8-point improvement over the semi-supervised SSASS method, the strongest prior result the paper quotes. The practical stake is that a fast, linear-complexity model could support automated stenosis screening as a second reader for cardiologists.","feed_headline":"Mamba model beats prior stenosis segmentation by 11.8 F1 points","feed_subtitle":"U-Mamba BOT hits 68.79% F1 on the ARCADE test set, the best among six U-shaped models.","key_machinery":"The central object is U-Mamba BOT, a U-shaped network that keeps convolution in the encoder and decoder and inserts Mamba blocks—selective structured state-space models with an input-dependent selection mechanism and scan-based computation—only at the bottleneck. The Mamba block processes image features in 1D sequences (via the 2D selective scan SS2D and cross-scan module from VMamba), giving linear scaling in sequence length rather than the quadratic attention cost of Transformers. This bottleneck placement is what the paper credits for combining global context with efficient training.","core_discovery":"On the ARCADE stenosis segmentation benchmark, U-Mamba BOT—a U-Net encoder-decoder whose encoder and decoder use convolutional residual blocks while Mamba selective state-space blocks sit in the bottleneck—obtains the best F1 (68.79%) and recall (67.69%) among the tested models, with U-Mamba ENC posting the best precision (71.13%). The paper interprets this as evidence that Mamba's linear-complexity selective scan can capture the long-range context needed to find small, blurred stenotic lesions while avoiding the quadratic cost of transformers.","pith_inferences":["If the claimed comparison is valid, Mamba-based segmentation could be paired with post-processing or ensemble steps to push stenosis localization closer to clinical usability, since even the best F1 leaves room for false positives and false negatives.","The paper's success with bottleneck-only Mamba suggests that global context matters most at the coarsest resolution; a natural test is whether adding Mamba blocks to skip connections or decoder levels improves recall further.","A direct head-to-head re-run of Ensemble Learning, StenUNet, and SSASS under identical training and evaluation settings would settle whether the 11.8-point gain reflects architecture choice rather than differences in protocol.","Because the ARCADE dataset is public, the U-Mamba BOT result is reproducible and could serve as a baseline for future stenosis segmentation work, including external validation on other angiography datasets."],"forward_implications":["U-Mamba BOT reaches 68.79% F1 for coronary stenosis segmentation, outperforming the quoted semi-supervised SSASS score of 56.99% by 11.8 points.","Among Mamba variants, Swin-UMamba D shows that putting Mamba blocks on both encoder and decoder sides yields strong performance with fewer parameters (27M versus 60M for Swin-UMamba).","The linear-complexity Mamba family is presented as a practical alternative to vision transformers for imbalanced medical segmentation tasks where lesion regions are tiny.","The models are trained with 5-fold cross-validation on 1200 ARCADE images and tested on the standardized 300-image set, with plaques defined by SYNTAX Score criteria."],"supporting_citations":[{"why":"Supplies the Mamba selective state-space model with linear-complexity scan computation that the paper adapts for segmentation.","marker":"[6]"},{"why":"Defines the U-Mamba architecture, including the ENC and BOT variants that are the paper's main test objects.","marker":"[17]"},{"why":"Provides the SS2D and cross-scan modules used to adapt Mamba from 1D sequences to 2D image features.","marker":"[15]"},{"why":"Supplies the ARCADE dataset, its 1200/300 train-test split, and the stenosis annotations used for training and evaluation.","marker":"[20]"},{"why":"Gives the SSASS semi-supervised method whose 56.99% F1 score is the main baseline the paper claims to beat by 11.8 points.","marker":"[10]"},{"why":"Provides the StenUNet architecture and its 53.48% F1 score, another prior result compared against.","marker":"[13]"},{"why":"Supplies the ensemble-learning method and its 39.41% F1 score from the ARCADE challenge, used as a lower baseline.","marker":"[2]"},{"why":"Defines the U-Net encoder-decoder structure with skip connections that every model in the study builds on.","marker":"[21]"},{"why":"Describes Swin-UMamba, the ImageNet-pretrained Mamba encoder variant whose decoder placement the paper varies.","marker":"[14]"},{"why":"Provides the Swin UNetR transformer baseline used to compare Mamba models against a shifted-window transformer.","marker":"[8]"}],"fun_headline_variants":["Mamba U-Net tops stenosis F1 at 68.79%","Stenosis F1 jumps 11.8 points with Mamba model","U-Mamba BOT scores 68.79% F1 for stenosis","Mamba segmentation lifts stenosis F1 to 68.79%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The improvement over prior work rests on the assumption that the F1 scores quoted for Ensemble Learning (39.41%), StenUNet (53.48%), and SSASS (56.99%) were computed on the same 300-image ARCADE test split with the same evaluation protocol, because the paper did not re-run those methods.","fun_headline_variants_meta":{"raw":{"variants":["Mamba U-Net tops stenosis F1 at 68.79%","Stenosis F1 jumps 11.8 points with Mamba model","U-Mamba BOT scores 68.79% F1 for stenosis","Mamba segmentation lifts stenosis F1 to 68.79%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000455,"raw_usage":{"total_tokens":2214,"prompt_tokens":804,"completion_tokens":1410,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":420,"completion_tokens_details":{"reasoning_tokens":1341}},"tokens_in":420,"tokens_out":1410,"duration_ms":10390,"temperature":1.0,"reasoning_tokens":1341,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:17:59.091906+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run Ensemble Learning, StenUNet, and SSASS on the same 300-image ARCADE test images with identical preprocessing, metric computation, and thresholding; if any of them matches or exceeds 68.79% F1, or if U-Mamba BOT's score drops on a corrected split, the paper's central improvement claim fails. A secondary check is to evaluate U-Mamba BOT on an independent coronary angiography dataset with similar stenosis annotations.","supporting_citations":[{"cited_title":"Dataset for automatic region-based coronary artery disease diagnos- tics using x-ray angiography images","cited_arxiv_id":null,"evidence_quote":"Supplies the ARCADE dataset, its 1200/300 train-test split, and the stenosis annotations used for training and evaluation."},{"cited_title":"Swin- umamba: Mamba-based unet with imagenet-based pretraining","cited_arxiv_id":null,"evidence_quote":"Describes Swin-UMamba, the ImageNet-pretrained Mamba encoder variant whose decoder placement the paper varies."},{"cited_title":"Swin unetr: Swin transformers for semantic segmentation of brain tumors in mri images","cited_arxiv_id":null,"evidence_quote":"Provides the Swin UNetR transformer baseline used to compare Mamba models against a shifted-window transformer."}],"review_version":1}