{"id":"2cda72a6-88f6-4897-9c52-a03a3fe721f2","arxiv_id":"2411.10345","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"On a public hand X-ray dataset, Xception gave the lowest bone-age prediction error (MAE 12.6 months), followed by a custom CNN (22.6 months) and VGG (34.9 months).","lead":"The authors trained three standard image-classification models, Xception, VGG, and a custom CNN, on hand X-rays to predict bone age in months, and compared their average errors. Xception had the lowest mean error at about 12.6 months, VGG the highest at about 34.9 months, and the custom CNN fell in between, so the paper offers a rough model-selection guide.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The accuracy ranking is confounded by differing input resolutions and undisclosed pretrained weights; the paper's own Methods contradict the assumption of comparable training.","rationale":"The reader's weakest-assumption analysis correctly identifies comparability as the load-bearing premise. My read agrees: the paper's own methods section reveals an explicit preprocessing difference (128x128 vs 256x256) and silence on pretraining. The strongest claim cannot be evaluated without resolving these confounds; the conditional verdict is appropriate. I additionally note a parameter-count mismatch (VGG table sums to 20.35M vs reported 17.99M) that further reduces confidence but does not alter the main condition. No independent verification, code, or error bars exist to offset these concerns. Therefore no change to the conditional verdict is warranted.","tokens_in":5383,"tokens_out":6423,"duration_ms":54515,"concrete_test":"Obtain or reconstruct the training scripts and run a controlled ablation: train all three models on the same 256x256 greyscale inputs, same optimizer/epochs/batch size, same train/validation split, and initialize Xception both from ImageNet and from scratch. If from-scratch Xception does not beat the custom CNN, the reported ranking is an artifact of transfer learning or resolution. This single experiment would settle whether the central comparison is fair.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Xception MAE 12.6 vs CNN 22.6 vs VGG 34.9) is a ranking, so it is only meaningful if differences in MAE are caused by architecture rather than by differences in how the models were prepared. The manuscript does not establish this. In Section II, preprocessing is explicitly different: VGG and CNN receive greyscale 256x256 images, while Xception receives 128x128 images followed by a Keras built-in normalization. No mention is made of whether Xception was initialized with ImageNet weights, which is the default way to use Keras's Xception; the custom CNN and VGG-like networks are described from scratch. If transfer learning was used for Xception only, the reported gap is expected and does not support a claim about architecture. The paper also does not state which split the MAE was computed on, or report repeated runs or error bars. A secondary internal inconsistency: the VGG layer table sums to 20,351,265 parameters, not the reported 17,991,457, weakening confidence that the described model is the one trained. These gaps make the stated ranking not verifiable as stated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports an empirical comparison of three deep learning models (Xception, VGG, and a custom CNN) for bone age assessment from hand X-ray images using the RSNA Bone Age dataset. The authors report mean absolute error (MAE) in months: Xception 12.6, CNN 22.6, and VGG 34.9, and they conclude that Xception is the most accurate but also the largest and most resource-intensive, while the custom CNN provides a light-weight alternative. The paper includes architecture tables with parameter counts for each model and a brief description of the data preprocessing.","tokens_in":5527,"tokens_out":3232,"duration_ms":30060,"significance":"If the reported ranking is reliable, the paper provides a practically useful comparison for clinicians and engineers choosing a bone-age model under different hardware constraints. The use of a public dataset and a uniform evaluation metric (MAE) are strengths, as are the explicitly tabulated model architectures. However, the paper currently lacks the experimental detail needed to establish that the MAE differences are attributable to model architecture rather than to training choices, input resolution, or transfer-learning status. The significance is therefore conditional on the authors providing a much more complete experimental account and resolving an internal inconsistency in the reported VGG parameter count.","major_comments":[{"comment":"The paper states that VGG and CNN receive 256x256 greyscale images, while Xception receives 128x128 images followed by a Keras built-in normalization. This means the models differ not only in architecture but also in input resolution and preprocessing. The central claim that Xception is the most accurate model is therefore confounded: the MAE gap could be due to input size or normalization rather than architecture. The authors must either use identical preprocessing for all models or run a controlled experiment that isolates the architecture effect.","section":"Section II, Methods (preprocessing)"},{"comment":"The manuscript never states whether the Xception model was initialized with ImageNet-pretrained weights, which is the default in Keras, while the custom CNN and the modified VGG are described as built from scratch. If transfer learning was used only for Xception, the observed MAE ordering is expected and does not support a conclusion about architectural superiority. This is a load-bearing omission: the authors must disclose the weight initialization and, if transfer learning was used, either compare all models under the same transfer-learning regime or clearly temper the claim.","section":"Section II, Xception Model; Section III, Results"},{"comment":"The paper reports MAE values but does not state whether these are computed on the 1425-image validation split or on a separate held-out test set. It also omits the number of epochs, optimizer, learning rate, batch size, loss function, and any repeated runs or error bars. Without these details, the reported MAE values cannot be reproduced or verified, and the reader cannot assess whether the differences among 12.6, 22.6, and 34.9 months are meaningful. The authors should specify the evaluation protocol and provide uncertainty estimates or at least state that the numbers come from a single run.","section":"Section III, Results; Section II, Dataset"},{"comment":"The VGG architecture table lists per-layer parameter counts that sum to 20,351,265, not the 17,991,457 reported in the Results and Conclusions. This is an internal inconsistency: either the table or the reported total is wrong, and it undermines the reader's confidence that the described model is the one that was actually trained. The authors must reconcile these numbers and verify that all parameter counts correspond to the trained models.","section":"Section II, VGG Model table; Section III, Results"}],"minor_comments":[{"comment":"The abstract and introduction claim the three models are 'the most widely used' for bone age assessment, but no citation or quantitative support is provided for this claim; please justify or rephrase.","section":"Section I, Background"},{"comment":"For the Xception preprocessing, the paper says 'the inbuilt Keras function to normalize the images' without naming it; please specify whether preprocess_input is used and whether the images are converted to three-channel RGB.","section":"Section II, Methods"},{"comment":"Figure 3 ('Zscore of validation and training images') is not described in the text, and its axes are not labeled; please clarify what is plotted and why it is included.","section":"Section II, Figures"},{"comment":"There are typographical issues, including 'MaxPooIing2D' in the VGG table and inconsistent wording about the VGG model's implementation ease; a careful proofread is needed.","section":"Section II and IV"},{"comment":"The conclusions state the VGG model has the highest MAE while also calling it 'the easiest to implement,' but no evidence for ease of implementation is provided beyond parameter count; please either justify or soften this claim.","section":"Section IV, Conclusions"},{"comment":"The paper does not compare the reported MAE values with published results on the RSNA Bone Age dataset (e.g., from the RSNA challenge), which would help readers calibrate the accuracy of the models; please add such context.","section":"Section III, Results"}],"recommendation":"major_revision","confidential_remarks":"This manuscript is quite thin for a journal submission and currently resembles an extended abstract. The central claim is an empirical ranking, but the missing training details and the unstated transfer-learning status are serious. If the authors can provide the full experimental protocol, clarify weight initialization, run or report a controlled preprocessing comparison, and correct the VGG parameter discrepancy, the paper could become a valid small-scale empirical study. If they cannot provide these details, reject. I recommend major_revision to give them the opportunity, but the bar for acceptance should be high."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"If you need one sentence: this is a student-level comparison of Xception, VGG, and a custom CNN on RSNA bone age data, and the headline result (Xception MAE 12.6 vs CNN 22.6 vs VGG 34.9 months) cannot be trusted as stated because the models were not trained or evaluated under comparable conditions. The paper does one thing that is marginally useful: it reports raw MAE and parameter counts for three standard architectures on a public dataset, and the numbers are internally consistent in the sense that bigger models do not automatically win. That is a legitimate small data point, and the prose is honest about the tradeoff between accuracy and model size. But that is where the credit ends.\n\nThe soft spots are substantial and load-bearing. Section II says VGG and CNN get 256x256 greyscale images, while Xception gets 128x128 with Keras built-in normalization. That alone makes the accuracy comparison meaningless as an architecture ranking. Worse, there is no mention of pretrained weights. If Xception was initialized from ImageNet (the Keras default) and the others were trained from scratch, the gap is fully explained by transfer learning, not by architecture. The paper also never states which split produced the reported MAE (validation likely, but it says \"validation images\" in Methods and never mentions a test set), reports no error bars or repeated runs, and gives no epochs, optimizer, learning rate, or batch size. On top of that, the VGG layer table sums to about 20.35 million parameters, not the reported 17.99 million, so the described model does not match the reported count. Finally, the MAEs (12.6–34.9 months) are far above published bone-age models (often 4–6 months), and the paper does not compare against any of them.\n\nWho gets value from this? Possibly a reader who wants a rough back-of-the-envelope feel for these three networks on this dataset, but not a clinician, not a researcher looking for a benchmark, and not anyone making procurement decisions. The paper is a cautionary example of how to run a model comparison without controlling for confounds.\n\nMy recommendation: desk reject. It does not deserve serious referee time in current form. If the authors release code, training protocols, error bars, consistent preprocessing, and a comparison with published baselines, then a resubmission could be evaluated on its merits. As it stands, the central claim is unverifiable and the experimental design is too weak to support any conclusion.","headline":"A naive three-model benchmark on bone age whose ranking is confounded by inconsistent preprocessing and missing training details; not ready for peer review.","tokens_in":6080,"tokens_out":2190,"would_cite":false,"duration_ms":21911,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper reports that Xception achieves the lowest mean absolute error (12.6 months) among three bone-age models, ahead of a custom CNN (22.6 months) and VGG (34.9 months).","keywords":["bone age assessment","Xception","VGG","convolutional neural network","mean absolute error","hand X-ray","deep learning","RSNA dataset"],"falsifier":"Retrain all three models from the same starting point, at the same input resolution, with the same optimizer, learning-rate schedule, and number of epochs, and evaluate them on the same held-out split; the claimed ordering fails if the CNN or VGG matches or beats Xception's 12.6-month mean absolute error.","tokens_in":5150,"feed_emoji":"🦴","tokens_out":6893,"duration_ms":56517,"temperature":0.7,"pith_summary":"This paper asks which of three deep-learning models—Xception, VGG, and a custom CNN—should be used to automate bone age assessment from hand X-rays. It reports that Xception is the most accurate, with a mean absolute error of 12.6 months, compared with 22.6 months for the CNN and 34.9 months for VGG. The practical target is to help medical institutions choose between accuracy, model size, and ease of implementation. The paper concludes that Xception wins on accuracy but is the most resource-intensive, while the CNN is the lightest and VGG the easiest to deploy.","feed_headline":"Xception posts lowest bone-age error at 12.6 months","feed_subtitle":"In the paper's comparison, Xception beats the custom CNN (22.6) and VGG (34.9) on mean absolute error.","key_machinery":"The central objects are the three architectures: a custom CNN built from alternating Conv2D, batch-normalization, and max-pooling layers with a dense regression head; VGG, a standardized deep CNN modified for single-value age output; and Xception, built from depthwise separable convolutions—a depthwise filter applied per channel followed by a 1×1 pointwise convolution—together with residual connections. The comparison is carried by mean absolute error in months, computed after Keras-based preprocessing in which Xception was trained at 128×128 resolution while the other two models used 256×256.","core_discovery":"On the RSNA Bone Age dataset, the paper reports a clear empirical accuracy ranking: Xception reaches a mean absolute error of 12.6 months, the custom CNN reaches 22.6 months, and VGG reaches 34.9 months. The paper interprets this as showing that Xception is the most accurate of the three models, while the CNN is the most lightweight at roughly 3.7 million parameters, VGG is the easiest to implement, and Xception is the largest and most resource-intensive at roughly 20.9 million parameters. The intended contribution is a practical comparison that lets radiologists and institutions pick a model based on their accuracy needs and hardware constraints.","pith_inferences":["Because Xception was preprocessed at 128×128 while the CNN and VGG used 256×256, the reported accuracy gap may partly reflect input resolution rather than architecture; an equal-resolution retraining would separate those factors.","The paper does not compare its MAE values with inter-observer variability of the Greulich–Pyle or Tanner–Whitehouse methods, so the clinical significance of a 12.6-month error is left open.","If the ranking survives matched training conditions, the natural deployment rule is a two-point trade-off: choose Xception when accuracy dominates and hardware permits, and choose the CNN when memory is the binding constraint.","The reported parameter counts and MAE values could be turned into a simple accuracy-per-parameter ratio for future model selection if the training conditions were standardized."],"forward_implications":["If the reported ordering is correct, an institution that prioritizes accuracy should choose Xception when its compute budget allows.","A resource-constrained deployment would reasonably choose the custom CNN, accepting roughly 10 months of additional mean absolute error to save about 17 million parameters.","VGG offers the simplest implementation path but the lowest accuracy of the three, making it a fallback for teams with limited deep-learning expertise.","The same MAE metric allows the three models to be compared directly, but it does not by itself establish clinical adequacy.","Larger model size does not guarantee higher accuracy in this comparison, since VGG is larger than the CNN yet less accurate."],"supporting_citations":[{"why":"Defines the Xception architecture with depthwise separable convolutions that the paper trains and reports as most accurate.","marker":"[10]"},{"why":"Defines the VGG architecture that the paper modifies and evaluates.","marker":"[8]"},{"why":"Prior deep-learning bone-age assessment work that motivates automated prediction and supplies comparison context.","marker":"[7]"},{"why":"Earlier pediatric bone-age assessment with deep CNNs, used as background for the automated approach.","marker":"[4]"},{"why":"An attention-based CNN bone-age system referenced as a prior automated alternative.","marker":"[1]"}],"fun_headline_variants":["Xception tops bone-age precision, but CNN costs less","Bone-age AI race: Xception wins, VGG lags at 34.9","Picking bone-age model? Xception for accuracy, CNN for size","Bone-age MAE: Xception 12.6, CNN 22.6, VGG 34.9","Xception beats CNN and VGG on bone-age error"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ranking depends on the three models being trained and evaluated under comparable conditions, and the paper does not state whether pretrained weights, training settings, or an unseen test set were used uniformly.","fun_headline_variants_meta":{"raw":{"variants":["Xception tops bone-age precision, but CNN costs less","Bone-age AI race: Xception wins, VGG lags at 34.9","Picking bone-age model? Xception for accuracy, CNN for size","Bone-age MAE: Xception 12.6, CNN 22.6, VGG 34.9","Xception beats CNN and VGG on bone-age error"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000656,"raw_usage":{"total_tokens":2992,"prompt_tokens":919,"completion_tokens":2073,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":535,"completion_tokens_details":{"reasoning_tokens":1967}},"tokens_in":535,"tokens_out":2073,"duration_ms":14590,"temperature":1.0,"reasoning_tokens":1967,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:43:09.631129+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain all three models from the same starting point, at the same input resolution, with the same optimizer, learning-rate schedule, and number of epochs, and evaluate them on the same held-out split; the claimed ordering fails if the CNN or VGG matches or beats Xception's 12.6-month mean absolute error.","supporting_citations":[{"cited_title":"Xception: Deep Learning with Depthwise Separable Convolutions","cited_arxiv_id":null,"evidence_quote":"Defines the Xception architecture with depthwise separable convolutions that the paper trains and reports as most accurate."},{"cited_title":"Very deep convolutional networks for large-scale image recognition","cited_arxiv_id":null,"evidence_quote":"Defines the VGG architecture that the paper modifies and evaluates."},{"cited_title":"Deep learning for automated skeletal bone age assessment in X -ray images","cited_arxiv_id":null,"evidence_quote":"Prior deep-learning bone-age assessment work that motivates automated prediction and supplies comparison context."},{"cited_title":"Paediatric bone age assessment using deep convolutional neural networks","cited_arxiv_id":null,"evidence_quote":"Earlier pediatric bone-age assessment with deep CNNs, used as background for the automated approach."},{"cited_title":"Intelligent Bone Age Assessment: An Automated System to Detect a Bone Growth Problem Using Convolutional Neural Networks with Attention Mechanism","cited_arxiv_id":null,"evidence_quote":"An attention-based CNN bone-age system referenced as a prior automated alternative."}],"review_version":1}