{"id":"f75f9744-2ca3-48aa-8e7f-b8c6e763d4bc","arxiv_id":"2501.10098","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"landmarker provides a modular PyTorch-based toolkit for anatomical landmark localization in 2D/3D medical images, and its included models outperform literature baselines on two benchmark datasets.","lead":"A new open-source Python toolkit, landmarker, bundles tools for finding anatomical landmarks in 2D and 3D medical images, including DICOM/NIfTI support and static or adaptive heatmap regression. The paper reports that models built with the toolkit beat published baselines on pelvis X-rays and 3D skull CT scans, though the comparisons lack error bars and identical training protocols.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark claim rests on baseline numbers that may come from a different data split than the one used for the landmarker results; the OAI table needs a protocol-equivalence check.","rationale":"The reader's weakest assumption is essentially the same: the baseline numbers in Tables 2 and 3 may have been produced under different data splits, preprocessing, and evaluation protocols. I agree with the conditional verdict. The central software contribution (a modular PyTorch/MONAI toolkit) is credible and clearly supported by the code and examples; the weakest link is the quantitative performance claim, which depends entirely on comparing against numbers from other papers without evidence that the protocols are equivalent. No error bars or retrained baselines are provided, so the claim of 'enhanced accuracy' is not fully substantiated. I would keep the verdict conditional rather than rejecting, because the concern is about the strength of a supporting claim, not about the toolkit's existence or usability. The recommended check is to reproduce the comparison with matched splits and preprocessing, which would settle whether the gap is real.","tokens_in":12702,"tokens_out":1408,"duration_ms":11842,"concrete_test":"Obtain the OAI pelvis data and the exact 5-fold split from Pei et al. 2023, train both the landmarker SCN-adaptive model and the U-Net w/ Attention baseline under identical preprocessing (image size, spacing normalization, heatmap sigma schedule, evaluation decoder), and report per-fold PE/SDR with confidence intervals. If the landmarker advantage over the re-run baseline is not reproduced at roughly the same magnitude, the paper's 'outperforms' claim should be weakened.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The strongest claim is that landmarker models outperform published baselines (PE 1.61 vs 3.14 mm on OAI pelvis; PE 1.39 vs 1.96 mm on the MML 3D skull). The reported gains are large, but Table 2 does not state what split or image resolution was used for the landmarker run, nor whether the U-Net w/ Attention baseline was evaluated on the same 5 folds as in Pei et al. 2023. The text says 'A 5-fold validation approach, as suggested by Pei et al., obtains the results,' but does not say the same fold assignments were used, nor whether the baseline numbers are copied from the original paper. If the landmarker model was evaluated on a different subset or with different preprocessing/resampling/spacing normalization, the PE gap could reflect protocol differences rather than model quality. Additionally, Table 3 compares against He et al. 2024 but the landmarker method is a one-hot ensemble from a separate paper (Jonkers et al., 2025); no details are given for how the MML split, landmarks, or evaluation metric were matched to the baseline. The package's central value as a benchmarking tool is not in question, but the 'outperforms' claim is not independently verifiable from the provided text. A concrete test is to reproduce the OAI comparison using the exact 5-fold split and preprocessing from Pei et al., and to re-run the U-Net w/ Attention baseline in the landmarker pipeline; if the gap shrinks materially, the claim needs qualification.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents landmarker, a PyTorch-based Python package for anatomical landmark localization in 2D/3D medical images. It describes the modular architecture (data handling, heatmap generation/decoding, models/losses, evaluation/visualization) and provides code listings for using the package. The authors report two benchmark experiments: an SCN with adaptive heatmaps on pelvis X-rays (PE 1.61 mm vs 3.14 mm for a U-Net with Attention) and a one-hot ensemble on a 3D skull CT dataset (test PE 1.39 mm vs 1.96 mm for Pruning-ResUNet3D). The paper also frames the toolkit as addressing gaps in general-purpose pose-estimation libraries for medical imaging.","tokens_in":13040,"tokens_out":4478,"duration_ms":38992,"significance":"If the toolkit is as functional as the code listings and public repository suggest, it provides a valuable, modular alternative to existing pose-estimation frameworks for medical landmark tasks, with support for NIfTI/DICOM, 2D/3D data, and multiple heatmap regression paradigms. The reproducibility evidence (public code, example listings) is a definite strength. However, the accuracy-enhancement claim rests on two benchmark comparisons whose protocol equivalence to the cited baselines is not established; if that is resolved, the paper would be a useful contribution to the medical imaging software ecosystem.","major_comments":[{"comment":"The claim that the landmarker SCN with adaptive heatmaps outperforms the U-Net with Attention (PE 1.61 vs 3.14 mm) is not independently verifiable without protocol-equivalence details. The text only states that 'a 5-fold validation approach, as suggested by Pei et al., obtains the results'; it does not state whether the same fold assignments, image resolution, preprocessing (e.g., resampling, contrast normalization), or evaluation code were used for both methods, nor whether the baseline numbers were copied from the original paper. Please provide the exact protocol, or re-run the baseline in the landmarker pipeline, and report variance (e.g., per-fold ranges or standard deviations).","section":"Section 4.1, Table 2"},{"comment":"The comparison against Pruning-ResUNet3D on the MML skull dataset has the same protocol-equivalence problem. The text says the dataset is 'the same as the data subset, only with complete landmarks, used as a benchmark in He et al.' but does not describe how the train/validation/test split, landmark definitions, or evaluation metric were matched. In addition, the method compared is a one-hot ensemble from a same-group preprint (Jonkers et al., 2025); since the preprint is not yet published, the method details and hyperparameters are not available in the cited literature. Please specify the split and evaluation protocol in detail, provide hyperparameters or a reference to a versioned repository, and ideally include error bars from multiple runs.","section":"Section 4.1, Table 3"},{"comment":"The broad statement in the Abstract that landmarker 'enhances the accuracy of landmark identification' is supported only by the two benchmark tables. Given the protocol-equivalence concerns above, this claim is stronger than the current evidence. Consider qualifying it (e.g., 'in the experiments presented here') or adding the missing protocol details. This is not a request to remove the benchmark, but to align the claim with the evidence.","section":"Abstract and Section 4.1"}],"minor_comments":[{"comment":"The caption contains a typo: 'pacakge' should be 'package'.","section":"Figure 2 caption"},{"comment":"The caption says 'Loading data into a LandmarkDataset' but the code calls inspection_plot; the caption should describe visual inspection instead.","section":"Listing 4 caption"},{"comment":"The caption lists '(a) Endoscopic images, (b) ISBI2015, and (c) Pelvis X-rays' but the subplot labels in the figure are '(a) ISBI2015 dataset', '(b) Pelvis X-rays', '(c) Endoscopic images'; the ordering is inconsistent.","section":"Figure 4 caption"},{"comment":"The column header '4 mmU-Net w/ Attention' is missing a space; it should be '4 mm U-Net w/ Attention'.","section":"Table 2 header"},{"comment":"The running head contains 'I MAGES' with an incorrect space; it should read 'IMAGES'.","section":"Title and running head"},{"comment":"For reproducibility, the listing could also state the image resolution, batch size (currently 1), and hardware used for training; the current listing shows only the optimizer settings.","section":"Listing 5"}],"recommendation":"major_revision","confidential_remarks":"The benchmark comparison in Table 3 relies on a same-group preprint (Jonkers et al., 2025); this is not circularity in the software claim, but it does mean the comparison method is not independently established in the peer-reviewed literature. The authors should provide sufficient detail for reproducibility. Additionally, the manuscript appears to be a published SoftwareX paper; if this submission is for a different venue, the benchmark-evidence gap should still be addressed before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is a software paper, and the software is real. The landmarker package is a modular PyTorch toolkit for anatomical landmark localization in 2D and 3D medical images, with support for NIfTI/DICOM, MONAI transforms, multiple heatmap generators/decoders, a local weighted spatial mean decoder, and implementations of published models and losses. The code is public, installable, and the examples in Section 4 look functional. If you work in medical image analysis, this will likely save you engineering time.\n\nThe second thing: the benchmark 'outperforms' claims are the weak part. The paper compares its SCN with adaptive heatmaps against U-Net with Attention on OAI pelvis X-rays and reports PE 1.61 mm vs 3.14 mm. That is a large gap, but Table 2 gives no error bars, no hyperparameters, and no statement that the same 5-fold split was used as in Pei et al. The text says 'as suggested by Pei et al.,' which suggests the split may not be identical. Table 3 has a similar problem: the comparison against He et al. 2024 uses a one-hot ensemble from the authors' own 2025 preprint, and again there is no detail on how the MML split, landmarks, or evaluation were matched. The stress-test note is right: the numbers are not independently verifiable from the text. This does not undermine the toolkit itself, but the abstract's 'enhances accuracy' is too strong given the evidence.\n\nOther soft spots are minor. Figure 4's caption looks mislabeled (panel c is called 'Endoscopic images' but is described as pelvis X-rays in the text). The taxonomy in Figure 1 is a personal take but not harmful. Some training details only appear in code listings, not as a config summary.\n\nThe package is a legitimate contribution to the medical imaging toolbox ecosystem. It deserves a serious referee, but the referee should ask for either re-run baselines under the same pipeline or a qualified wording of the accuracy claims. I would accept this for review rather than desk-reject; a revision with protocol-matched comparisons would make it solid.\n\nFor a reading group, it is a maybe: useful if you care about tooling for landmark tasks, less so if you are after algorithmic novelty.","headline":"Useful, open-source toolkit for medical landmark localization; the accuracy claims need a protocol-equivalence check before taking the gains at face value.","tokens_in":13550,"tokens_out":2561,"would_cite":true,"duration_ms":24273,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper presents landmarker, an extensible Python toolkit for anatomical landmark localization in 2D/3D medical images, and reports that models built with it outperform published baselines on pelvis X-rays and skull CT scans.","keywords":["anatomical landmark localization","keypoint detection","medical image analysis","heatmap regression","adaptive heatmap","one-hot ensemble","PyTorch toolkit","2D/3D medical imaging"],"falsifier":"Re-train the two baselines (U-Net with Attention on the OAI pelvis X-rays, Pruning-ResUNet3D on the MML 3D skull data) inside the landmarker pipeline with the same folds and preprocessing; if they achieve point errors close to landmarker's 1.61 mm and 1.39 mm respectively, the outperformance claim collapses to a pipeline artifact.","tokens_in":12526,"feed_emoji":"🩻","tokens_out":6178,"duration_ms":51448,"temperature":0.7,"pith_summary":"landmarker is a Python software package that standardizes the full pipeline for anatomical landmark localization in 2D and 3D medical images, from loading medical image formats through heatmap generation, model training, decoding, and evaluation. The paper's central claim is that this toolkit is more complete and more modular than general-purpose pose-estimation packages for medical use, and that models implemented within it match or exceed published accuracy. Concrete evidence: an adaptive-heatmap spatial configuration network reaches a mean point error of 1.61 mm on OAI pelvis X-rays versus 3.14 mm reported for an attention U-Net, and a one-hot ensemble reaches 1.39 mm on the MML 3D skull test set versus 1.96 mm for Pruning-ResUNet3D. If correct, researchers can build, reproduce, and compare landmark-localization methods with less engineering overhead while retaining state-of-the-art accuracy.","feed_headline":"Medical landmark toolkit beats baselines on 2D and 3D scans","feed_subtitle":"Open-source package cuts mean error to 1.61 mm on pelvis X-rays and 1.39 mm on skull CT.","key_machinery":"The central machinery is the modular pipeline connecting four flexible pieces: dataset classes that ingest medical formats and keep landmark coordinates aligned under affine transforms; a heatmap generator whose distribution parameters (e.g., Gaussian sigmas and rotation) can be fixed, scheduled, or learned; a decoder that turns predicted heatmaps into coordinates via argmax, weighted spatial mean, soft-argmax, or local weighted spatial mean; and interchangeable models such as the spatial configuration network and a one-hot ensemble. Adaptive heatmap regression carries the 2D benchmark: letting the Gaussian covariance be learned during training lets a standard spatial configuration network outperform a baseline that uses fixed heatmaps. The one-hot ensemble carries the 3D benchmark by framing landmark localization as per-pixel classification and combining multiple predictions.","core_discovery":"On the paper's own terms, the discovery is that a single extensible toolkit can cover the methodological diversity of anatomical landmark localization—static, adaptive, and one-hot heatmap regression—without sacrificing accuracy. The paper demonstrates this with two benchmarks: on the Osteoarthritis Initiative pelvis X-ray dataset, a spatial configuration network with learnable Gaussian heatmap parameters implemented in landmarker achieves PE 1.61 mm and SDR up to 95.85% at 4 mm, compared with PE 3.14 mm for the U-Net with Attention reported in [Pei et al., 2023]; on the MML 3D skull dataset, a one-hot ensemble achieves test PE 1.39 mm and SDR 96.31% at 4 mm, compared with 1.96 mm for Pruning-ResUNet3D in [He et al., 2024]. The implied claim is that the package's modular abstractions—datasets, heatmap generators, decoders, models, losses, metrics—make it possible to implement these leading methods quickly and to benchmark them fairly on medical formats.","pith_inferences":["If the benchmark gaps hold under identical cross-validation folds, they suggest that learnable heatmap covariance and one-hot ensembles are worth adopting as default components in medical landmark pipelines, not just package features.","The same modular separation of heatmap generator, decoder, and model could be applied to other dense prediction tasks, such as control-point detection for registration or counting, by reusing the evaluation and visualization modules.","A straightforward testable extension is to add uncertainty-aware decoding to the toolkit, which the paper names as future work, letting the reported accuracy gains be paired with confidence intervals on each landmark."],"forward_implications":["A research group can reproduce the reported pelvis and skull results using only the public package and dataset imports, without reimplementing heatmap generation or decoding.","New landmark localization methods can be dropped into the same training and evaluation harness, making comparisons across papers more apples-to-apples than ad hoc code.","Because the toolkit handles 3D volumes directly rather than reducing to slices, methods developed in it transfer to volumetric medical tasks such as skull and orthopedic landmarking.","Adaptive heatmap models implemented here could serve as stronger baselines in future medical imaging papers than the fixed-heatmap baselines currently used."],"supporting_citations":[{"why":"Supplies the OAI pelvis X-ray dataset and the U-Net with Attention baseline numbers that the 2D benchmark must beat.","marker":"[Pei et al., 2023]"},{"why":"Supplies the MML 3D skull dataset and the Pruning-ResUNet3D baseline numbers for the 3D benchmark.","marker":"[He et al., 2024]"},{"why":"Source of the SpatialConfigurationNetwork model implemented in landmarker and used in the 2D experiment.","marker":"[Payer et al., 2019]"},{"why":"Provides the adaptive Gaussian heatmap parametrization with learnable sigmas and rotation used in the 2D benchmark.","marker":"[Payer et al., 2020]"},{"why":"Extends Gaussian heatmaps to model annotation uncertainty and supplies a loss and adaptive approach included in the package.","marker":"[Thaler et al., 2021]"},{"why":"Defines the one-hot ensemble method that produces the 3D benchmark results.","marker":"[Jonkers et al., 2025]"},{"why":"Provides the Adaloss scheduler for adaptive heatmap regression, cited as a way to adjust heatmap parameters during training.","marker":"[Teixeira et al., 2019]"},{"why":"Establishes one-hot heatmaps as a categorical distribution on pixels, the conceptual basis for the one-hot ensemble.","marker":"[McCouat and Voiculescu, 2022]"}],"fun_headline_variants":["Open-source toolkit improves landmark accuracy in 2D/3D","New toolkit beats prior baselines on 2D and 3D medical scans","Landmark toolkit: 1.61mm on X-ray, 1.39mm on CT","Modular landmarker package cuts landmark error across modalities"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the baseline numbers quoted from [Pei et al., 2023] and [He et al., 2024] were produced with equivalent data splits, preprocessing, and evaluation protocol, so the reported accuracy gaps reflect the models rather than hidden pipeline differences.","fun_headline_variants_meta":{"raw":{"variants":["Open-source toolkit improves landmark accuracy in 2D/3D","New toolkit beats prior baselines on 2D and 3D medical scans","Landmark toolkit: 1.61mm on X-ray, 1.39mm on CT","Modular landmarker package cuts landmark error across modalities"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000946,"raw_usage":{"total_tokens":4033,"prompt_tokens":933,"completion_tokens":3100,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":549,"completion_tokens_details":{"reasoning_tokens":3018}},"tokens_in":549,"tokens_out":3100,"duration_ms":25357,"temperature":1.0,"reasoning_tokens":3018,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T19:23:07.169588+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-train the two baselines (U-Net with Attention on the OAI pelvis X-rays, Pruning-ResUNet3D on the MML 3D skull data) inside the landmarker pipeline with the same folds and preprocessing; if they achieve point errors close to landmarker's 1.61 mm and 1.39 mm respectively, the outperformance claim collapses to a pipeline artifact.","supporting_citations":[{"cited_title":"Learning-based landmark detection in pelvis x-rays with attention mechanism: data from the osteoarthritis initiative","cited_arxiv_id":null,"evidence_quote":"Supplies the OAI pelvis X-ray dataset and the U-Net with Attention baseline numbers that the 2D benchmark must beat."},{"cited_title":"Uncertainty Estimation in Landmark Localization Based on Gaussian Heatmaps","cited_arxiv_id":null,"evidence_quote":"Provides the adaptive Gaussian heatmap parametrization with learnable sigmas and rotation used in the 2D benchmark."},{"cited_title":"Modeling Annotation Uncertainty with Gaussian Heatmaps in Landmark Localization","cited_arxiv_id":"2109.09533","evidence_quote":"Extends Gaussian heatmaps to model annotation uncertainty and supplies a loss and adaptive approach included in the package."},{"cited_title":"Reliable uncertainty quantification for 2D/3D anatomical landmark localization using multi-output conformal prediction","cited_arxiv_id":"2503.14106","evidence_quote":"Defines the one-hot ensemble method that produces the 3D benchmark results."},{"cited_title":"Adaloss: Adaptive Loss Function for Landmark Localization","cited_arxiv_id":"1908.01070","evidence_quote":"Provides the Adaloss scheduler for adaptive heatmap regression, cited as a way to adjust heatmap parameters during training."}],"review_version":1}