{"id":"821e75f6-6591-4567-9cf1-7b195b9c3785","arxiv_id":"2411.17260","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A public micro-CT mouse femur dataset with growth plate annotations plus six deep-learning baselines, benchmarked at 1.91 mean absolute plane error.","lead":"AstraZeneca researchers ran an internal challenge in which 23 teams built deep-learning models that find the growth plate plane in 3D micro-CT scans of mouse femurs, and they plan to release the 83-scan dataset. Six submitted models reached an average error of about 1.91 image planes on 13 test scans, a level the paper calls usable, though with experts still in the loop.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ground-truth growth plate plane reliability is the load-bearing assumption; a single-expert annotation with no reported variability leaves all reported errors, rankings, and the radiologist-acceptability claim measured against an uncharacterized target.","rationale":"I agree with the reader's assessment that the paper is a useful benchmark contribution with largely internally consistent results and that the most load-bearing assumption is the correctness and reproducibility of the ground-truth growth plate plane indices. The reader's weakest_assumption identifies exactly this issue: single-expert annotation with no inter-observer or intra-observer variability. My reading of the full text confirms this is the point on which the central claim depends: all six solutions are ranked and the MAE of 1.91±0.87 is computed relative to the expert-provided GPPI (Tables 5 and 6, Section 5.1). Section 3.1 acknowledges the GPP is visually identified by image analysts and that adjacent planes are subtle; the Discussion itself concedes that 'precise acceptable error rates' still need to be established. The Eq. (1) vs Table 1 absolute-value inconsistency is real but is a presentation/implementation issue for the challenge score, not a threat to the headline MAE claim; the model-vs-model ranking is computed from the same ground truth and would shift only if labels are noisy. The public-data readiness issue is addressable and already flagged by the reader. The single expert's 20 years of experience and second-expert QC are good practices but do not quantify label stability, which is the key missing evidence. My recommended verdict remains CONDITIONAL, matching the reader, with the condition that label-variability evidence be added (or the acceptability claim be qualified). I do not see a reason to move to REJECT or ACCEPT: the central claim is plausible, the test-set computation is transparent, but the ground-truth stability question is unresolved and directly bears on the headline number.","tokens_in":16377,"tokens_out":2783,"duration_ms":20870,"concrete_test":"Have two or more blinded experts (ideally including the original annotator) independently annotate the GPPI on a subset of, say, 10-13 test scans, and compute pairwise inter-observer MAE and Bland-Altman limits of agreement, plus one repeated annotation by the original expert to estimate intra-observer variability. Compare the model MAE (1.91±0.87 overall range 1.23-3.62) against this human-level inter-observer MAE. If human inter-observer MAE is comparable to or larger than the model MAE, the 'acceptable for practical use' claim is supported; if human variability is much smaller, the claim needs qualification.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central quantitative claim is that the six solutions achieve a mean absolute error of 1.91±0.87 planes from ground truth on the test set and that this is acceptable for practical use by a radiologist. Every reported number—team MAEs, survival-function scores, team rankings, and the acceptability conclusion—is computed as a distance from the single expert's GPPI labels. Section 3.2 states that annotation was performed by one domain expert with 20 years of experience, with quality control by another expert, but no inter-observer or intra-observer variability is reported. The GPP definition itself is somewhat subtle: it is the lowest axial plane in which all four protrusions merge (Section 3.1), and adjacent planes are described as subtle and distinguishable only by experienced radiologists. If the reference plane is noisy by even ±1-2 planes, the claimed 1.91-plane MAE and the ranking of teams could change materially, and the 'acceptable for a radiologist' conclusion is not yet anchored to a measured human-variability baseline. The paper's own Discussion concedes that precise acceptable error rates still need to be established by comparing endpoints from manual and automatic segmentations. Thus the weakest link is not any single model's architecture but the stability of the target variable itself; without label-variability quantification, the benchmark's headline metric is not fully interpretable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes MiceBoneChallenge, an internal challenge organized at a pharmaceutical company to automate growth plate plane detection in micro-CT scans of mouse femurs. The authors release a dataset of 83 annotated 3D scans, present six deep learning solutions spanning 3D regression, 2.5D long-axis regression, and 2D axial classification, and report a collective mean absolute error of 1.91 ± 0.87 planes on a 13-scan test set, which they state is acceptable for practical use by a radiologist. The data, code, and trained models are promised to be made public.","tokens_in":16601,"tokens_out":6458,"duration_ms":58201,"significance":"If the dataset and annotations are released as promised, this is a useful contribution: it addresses a real gap in public preclinical micro-CT datasets for rodent bone growth plate detection and provides six independently implemented baselines with shared code. The reported performance numbers are internally consistent: the score sums in Table 6 match absolute-error scoring with a survival function, and the per-team MAEs in Table 5 are plausible. The main limitations are the absence of ground-truth label variability quantification and the small test set, both of which currently temper the strength of the accuracy and ranking claims.","major_comments":[{"comment":"The evaluation score is defined as 2Φ̄((p − t)/3) without an absolute value. As written, a prediction below the true plane (p < t) produces a negative argument to the CCDF and hence a score greater than 1; for example, an error of −3 planes gives a score of about 1.68, contradicting Table 1 and the stated [0, 1] range. The numerical results in Table 6 are consistent with 2Φ̄(|p − t|/3), so the intended metric is recoverable, but the equation and surrounding text should be corrected to avoid ambiguity in the central evaluation metric.","section":"§3.3.1, Eq. (1)"},{"comment":"The ground-truth GPPI was annotated by a single domain expert with quality control by another expert, but no inter- or intra-observer variability is reported. The GPP definition is subtle (Section 3.1, where adjacent planes are described as distinguishable only by experienced radiologists), and the paper's own motivation emphasizes eliminating observer variability. Without a repeatability study or a human-reader baseline, the reported MAE values and team rankings are measured against an uncharacterized target, and the abstract's claim that 1.91 ± 0.87 planes is 'acceptable for practical use by a radiologist' is not anchored. The Discussion already concedes that precise acceptable error rates still need to be established; please either provide label-variability data or temper the acceptability claim in the abstract and conclusions.","section":"§3.2 and §6"},{"comment":"The test set contains only 13 scans, and the differences among the top-ranked teams are small relative to the per-bone score variability. For instance, SN and BM have sum scores of 9.068 and 9.059, while the per-bone standard deviations are around 0.3, implying a standard error of roughly 0.08 on the sum; the observed difference is within noise. The paper should provide confidence intervals, bootstrap estimates, or a significance test for the team ranking, or explicitly state that the ranking is not statistically meaningful.","section":"§5.1, Tables 5–6"}],"minor_comments":[{"comment":"The abstract and conclusions state that the accuracy is 'acceptable for practical use by a radiologist,' while Section 6 refers to 'Anonymous Company's internal domain experts'; please align the terminology and avoid overclaiming.","section":"Abstract and §7"},{"comment":"There are several typographical errors: 'utilzed' in Section 4.2.2, 'utlizing' in Section 4.2, 'ensambles' in Section 4.4, 'trainiable' in Supplementary Section 10.6.2, and 'the the GPP region' in Section 6.","section":"Throughout"},{"comment":"Reference [12] (Lauterbur) is cited for µCT imaging; the appropriate citation for CT is Hounsfield [9], and the reference list should be checked for other citation mismatches.","section":"Supplementary §10.2"},{"comment":"The resize dimension for Team SN is given as 321×244×244 in Table 2, while Table 4 lists the input size as 244×244×32; the relationship between these dimensions (downsampling followed by cropping) should be clarified.","section":"Table 2 and Table 4"},{"comment":"The text states that DICOM images use an 8-bit pixel format; micro-CT data are typically 16-bit, and if the data were converted to 8-bit, the paper should note whether and how this affects Hounsfield-unit precision for downstream bone quantification.","section":"§3.2"},{"comment":"The abstract says the data 'will be shared' and the supplementary material provides an anonymized review link; for a dataset paper, a persistent DOI or a clear release license should be provided.","section":"§4.5 and Supplementary §9"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a single-company internal challenge report. The 'Anonymous Company' anonymization is unusual for a journal submission and may hinder reproducibility, but it is not a technical flaw. I would encourage the editor to verify that the data and code are actually accessible at publication time, since the abstract says they 'will be shared' and the supplementary link is anonymized. The radiologist-acceptability claim should also be checked against the company's own internal usage experience, as the paper itself acknowledges that endpoint-based validation is still pending."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe short version: this is a solid dataset-and-benchmark contribution, not a deep methods paper. What's genuinely new is the 83-scan annotated micro-CT dataset for growth plate plane detection in mouse femurs, plus six baseline solutions and code. That fills a real gap — I don't know of another public resource for this task, and the paper's own survey supports that. The experimental reporting is mostly careful: the MAE of 1.91±0.87 is consistent with the per-team numbers in Table 5, the cross-validation details are adequate, and the authors are upfront about computational costs and ensembling.\n\nThe biggest weakness is exactly what the stress-test note says: the ground truth GPPI rests on a single expert's annotations with no measured variability, and the 'acceptable for a radiologist' claim is not tied to any human-variability baseline. Adjacent planes are described as subtle, so a ±1-plane annotation uncertainty could shift team rankings and the headline MAE. The Discussion actually concedes that precise acceptable error rates still need to be established, which is the right caveat, but it should sit next to the abstract's claim. This is not fatal for a benchmark paper, but it does soften the central quantitative claim.\n\nSecond, Eq. 1 as written does not match Table 1. The complementary CDF without an absolute value gives scores greater than 1 for negative errors (up to 2), which would make a prediction below the true plane look better than a perfect prediction. Table 1 and the reported scores clearly use absolute error, so this is likely a typo, but it needs fixing.\n\nThird, the paper promises public data and code but only provides an anonymized placeholder link. For a dataset paper, that is a real reproducibility gap during review; if the link goes live on acceptance, it is fine, but it cannot be checked now.\n\nMinor points: the test set is 13 scans from one scanner, so generalizability is unproven; the teams were allowed to share code and ideas, so the six solutions are not fully independent; and the ±0.87 is across team MAEs, not across bones. None of these change the verdict.\n\nThis paper deserves a serious referee. The dataset alone is worth publishing if the annotations are usable, and the benchmark will be a reference for future work. I would ask the authors to report inter-observer variability, fix Eq. 1, and make the data accessible before acceptance.","headline":"Useful dataset-and-benchmark paper for mouse micro-CT growth plate detection; the headline MAE is internally consistent, but the single-expert ground truth and the missing absolute value in Eq. 1 weaken the 'radiologist-acceptable' claim.","tokens_in":17272,"tokens_out":3037,"would_cite":true,"duration_ms":27921,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that mouse bone growth plates can be located automatically in 3D micro-CT scans with a mean error of about two image planes, and it releases the first public annotated dataset and six deep-learning solutions to make this…","keywords":["growth plate detection","micro-CT","mouse femur","deep learning","preclinical drug development","bone quantification","public dataset","growth plate plane index"],"falsifier":"Re-annotate the 13 test scans, or a random subset of the 83, with a second independent expert and compute inter-observer plane-index disagreement. If expert-against-expert differences have a mean absolute value near 2 planes, comparable to the best models' MAE, then the reported accuracy cannot be separated from annotation noise, and the ranking among teams would be reshuffled by which expert's labels define ground truth.","tokens_in":16122,"feed_emoji":"🦴","tokens_out":7554,"duration_ms":62657,"temperature":0.7,"pith_summary":"This paper claims that automatic detection of the growth plate plane in mouse femur micro-CT scans is accurate enough for routine preclinical use, and backs that claim with the first public annotated dataset of 83 mouse bones plus six deep-learning solutions produced in an internal company challenge. On a held-out test set of 13 scans, the solutions predict the growth plate plane with a mean absolute error of $1.91 \\pm 0.87$ planes, where each plane is 10 $\\mu$m thick. The finding matters because in preclinical drug development the growth plate defines where the 1-mm volume of interest for trabecular bone measurement begins, and locating it is currently a manual, operator-dependent step that takes minutes per bone. If the claim holds, researchers gain a benchmark dataset, trained models, and evidence about which architectural families work for this task.","feed_headline":"Six AI solutions find bone growth plate within about 2 slices","feed_subtitle":"A public micro-CT dataset of 83 mouse femurs plus six trained models targets the manual step in preclinical bone measurement.","key_machinery":"The central object is the growth plate plane index (GPPI), defined as the lowest axial plane of the femoral growth plate where the four protrusions of the bone head merge; it anchors the volume of interest for trabecular bone measurement, a 1-mm cylinder starting 0.3 mm distal to the plane. The carried argument is a challenge comparison: six teams solve GPPI prediction using three architectural families: a 3D sliding-window ResNet34 with a decoupled classification and regression head, 2.5D EfficientNet regressions over stacked sagittal and coronal slices, and 2D axial-plane binary classifiers with morphological post-processing to clean the before/after decision boundary. Performance is scored with a scaled survival function, $\\mathrm{Score} = 2\\bar{\\Phi}((p-t)/3)$, which maps a zero-plane error to score 1 and decays smoothly toward 0 as the prediction-to-truth distance grows.","core_discovery":"The central claim is that bone-growth-plate identification in rodent micro-CT, a prerequisite for automated trabecular bone quantification, can be automated to an accuracy acceptable for practical use by a radiologist. The authors organize the claim around a new resource: 83 annotated 3D $\\mu$CT femur scans, with growth plate plane indices (GPPI) labeled by an expert with 20 years of experience and quality-checked by a second expert, plus the code, models, and pixel-wise bone segmentations. Six independent challenge solutions, spanning 3D sliding-window CNN regression, 2.5D long-axis regression, and 2D axial-plane classification, achieve per-team mean absolute errors from 1.23 to 3.62 planes on the test set, with a pooled mean of $1.91 \\pm 0.87$ planes. Notably, full-3D convolution did not outperform simpler 2.5D stacking approaches. The authors' intended contribution is therefore both a benchmark dataset and a demonstration that this previously manual landmark-detection task can be solved with accessible deep-learning pipelines.","pith_inferences":["Editorial inference: because annotation noise is unmeasured, the best models' MAE of 1.23 to 1.91 planes may already be at the noise floor of the ground truth; a direct test is to compare model predictions against a second expert's re-annotation of the same scans.","Editorial inference: the two-stage design in the better 2.5D solutions (coarse long-axis localization followed by fine cropped regression) suggests that a hybrid coarse-to-fine pipeline could push error below one plane, a possibility the paper notes but does not test.","Editorial inference: the claimed clinical acceptability could be verified end-to-end by computing trabecular bone volume fraction from manual versus predicted GPPI on the same scans and checking whether the derived endpoints differ by less than the $\\pm 10\\%$ drug-effect threshold the paper cites as decision-relevant.","Editorial inference: the released dataset comes from three preclinical studies on one scanner, so cross-scanner and cross-strain generalization remains open; a natural extension is fine-tuning on tibia or vertebra scans or on images from a different $\\mu$CT system."],"forward_implications":["A public benchmark for growth-plate detection in mouse femurs now exists, with 83 annotated volumes, code, and trained models released for direct comparison.","With planes 10 $\\mu$m apart, a mean absolute error of 1.91 planes moves the volume-of-interest start by roughly 19 $\\mu$m, which is small compared to the 1-mm analysis volume and supports the authors' practical-usability claim.","The observation that 2.5D and 2D approaches match full-3D accuracy means preclinical labs without large GPU budgets can adopt the cheaper pipelines.","Because pixel-wise bone segmentations are included, the dataset also supports the downstream segmentation task, enabling end-to-end automated bone quantification.","Six of the 23 challenge teams delivered final solutions, and their methods span three distinct modeling families, giving future participants a documented starting point."],"supporting_citations":[{"why":"Defines the standard endpoints and volume-of-interest protocols that make growth plate location the critical manual step.","marker":"[1]"},{"why":"Establishes established micro-CT analysis protocols for mouse long bones, vertebrae, and palms in aging mice.","marker":"[24]"},{"why":"Anchors the volume of interest to its distance from the epiphyseal growth plate, grounding the landmark definition.","marker":"[25]"},{"why":"Provides an existing public full-skeleton micro-CT dataset that the authors use to establish the novelty of their bone-growth annotation.","marker":"[23]"},{"why":"Supplies a public proximal tibia micro-CT dataset, one of the few comparable public resources at suitable resolution.","marker":"[22]"},{"why":"Offers a public lumbar vertebrae micro-CT dataset that further supports the claimed gap in growth-plate-specific annotations.","marker":"[28]"},{"why":"Introduces the decoupled detection head that Team SN adapts for sliding-window growth-plate regression.","marker":"[7]"},{"why":"Provides the ResNet backbone used by Teams SN and CW for feature extraction.","marker":"[8]"},{"why":"Supplies the EfficientNet architecture used by Team MH for 2.5D long-axis regression.","marker":"[27]"}],"fun_headline_variants":["Six AI solutions spot bone growth plate within ~2 slices","Public micro-CT dataset plus six models automate bone landmark","83 mouse scans, six AI pipelines: growth plate found accurately","AI detects bone growth plate in micro-CT with ~1.9 plane error"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the single expert's manual growth-plate labels are correct and reproducible enough to serve as ground truth; the paper reports no inter- or intra-observer variability, so every error and ranking is measured against an unexamined target.","fun_headline_variants_meta":{"raw":{"variants":["Six AI solutions spot bone growth plate within ~2 slices","Public micro-CT dataset plus six models automate bone landmark","83 mouse scans, six AI pipelines: growth plate found accurately","AI detects bone growth plate in micro-CT with ~1.9 plane error"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000286,"raw_usage":{"total_tokens":1705,"prompt_tokens":994,"completion_tokens":711,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":610,"completion_tokens_details":{"reasoning_tokens":640}},"tokens_in":610,"tokens_out":711,"duration_ms":6881,"temperature":1.0,"reasoning_tokens":640,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:20:27.407663+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate the 13 test scans, or a random subset of the 83, with a second independent expert and compute inter-observer plane-index disagreement. If expert-against-expert differences have a mean absolute value near 2 planes, comparable to the best models' MAE, then the reported accuracy cannot be separated from annotation noise, and the ranking among teams would be reshuffled by which expert's labels define ground truth.","supporting_citations":[{"cited_title":"Guidelines for assessment of bone mi- crostructure in rodents using micro-computed tomography.J Bone Miner Res., pages 1468–86, 2010","cited_arxiv_id":null,"evidence_quote":"Defines the standard endpoints and volume-of-interest protocols that make growth plate location the critical manual step."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes established micro-CT analysis protocols for mouse long bones, vertebrae, and palms in aging mice."},{"cited_title":"Micro-computed tomography assessment of bone structure in aging mice","cited_arxiv_id":null,"evidence_quote":"Anchors the volume of interest to its distance from the epiphyseal growth plate, grounding the landmark definition."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides an existing public full-skeleton micro-CT dataset that the authors use to establish the novelty of their bone-growth annotation."},{"cited_title":"Micro-computed tomography reconstructions of tibiae of stem cell transplanted osteogenesis imperfecta mice","cited_arxiv_id":null,"evidence_quote":"Supplies a public proximal tibia micro-CT dataset, one of the few comparable public resources at suitable resolution."},{"cited_title":"Zenzes and P","cited_arxiv_id":null,"evidence_quote":"Offers a public lumbar vertebrae micro-CT dataset that further supports the claimed gap in growth-plate-specific annotations."},{"cited_title":"Yolox: Exceeding yolo series in 2021, 2021","cited_arxiv_id":null,"evidence_quote":"Introduces the decoupled detection head that Team SN adapts for sliding-window growth-plate regression."},{"cited_title":"Deep residual learning for image recognition","cited_arxiv_id":null,"evidence_quote":"Provides the ResNet backbone used by Teams SN and CW for feature extraction."}],"review_version":1}