{"id":"d63723ab-cdfb-4a4b-8465-66a10bda1f4e","arxiv_id":"2502.00631","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A 3D ResNet with reweighted loss and logit adjustment reaches 65.38% accuracy on a private 389-patient CT dataset for three-class T-score prediction.","lead":"MedConv applies a 3D ResNet-50 with class-balanced cross-entropy and post-hoc logit adjustment to predict bone density T-scores from CT scans. The authors report it beats transformer baselines on a private 389-patient dataset, but the evaluation has serious methodological problems.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table II reports sensitivity exactly equal to accuracy and specificity exactly equal to 50% + accuracy/2 for every model, a relation no standard binary or multiclass confusion matrix can satisfy; the headline performance numbers are therefore not interpretable.","rationale":"The reader's rejection is justified, but the most load-bearing problem is not the one they emphasized. Even before considering test-set selection, Table II's numeric pattern shows the metrics cannot be standard performance measures. The identity Specificity = 50% + Accuracy/2, combined with Sensitivity = Accuracy, holds across every baseline and the proposed model, which is the signature of a formula artifact rather than real measurements. This directly attacks the strongest_claim: if the 65.38% accuracy / 79.34% AUC row is not a standard metric, the paper's central conclusion is unverified. Test-set hyperparameter selection is a real additional flaw and reinforces rejection, but the internal inconsistency is more fundamental. I agree partially with the reader's weakest_assumption: they identified a valid concern (test-set tau selection) but not the deeper metric inconsistency. I recommend no change to the reject verdict; if anything, the confidence in rejection should be high.","tokens_in":13961,"tokens_out":7258,"duration_ms":77637,"concrete_test":"Recompute Table II, row MedConv, from the actual 3-class confusion matrix on a fixed, untouched test split (with tau2 chosen on validation, not test), using standard formulas: Accuracy = (TP+TN)/N (or the explicitly stated macro-average), Sensitivity = TP/P, Specificity = TN/N. If the reported 65.38/65.38/82.69 cannot be reproduced under any standard definition, the headline result is invalid.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table II's reported metrics are internally inconsistent. For any classifier with a standard confusion matrix, if Sensitivity = Accuracy = a, then the same accuracy equation forces Specificity = a. The table instead has Specificity = 0.5*Accuracy + 0.5 in all 30+ rows (MedConv: 65.38 -> 82.69; ViT: 33.54 -> 66.77). This affine relation cannot arise from TP/TN/FP/FN counts for a nonempty negative class, so either sensitivity, specificity, and accuracy are computed with an undisclosed nonstandard aggregation or the numbers are not genuine performance estimates. The task is 3-class (Normal/Osteopenia/Osteoporosis) and the paper never defines how the single scalar sensitivity/specificity/ROC AUC are aggregated over classes; this makes the central 65.38%/79.34% claim unfalsifiable as reported. The problem is compounded by Section V-B: tau1=1 and tau2=0.5 are chosen from test-set ablations (Tables V and VI), so MedConv's row is a selection maximum rather than an independent test prediction. The correct response is to re-report all metrics from a fixed train/val/test split with standard per-class aggregation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces MedConv, a 3D ResNet-50 with a balanced cross-entropy (Bal-CE) loss and a post-hoc logit adjustment, for classifying CT-based bone density T-scores into three WHO categories (Normal, Osteopenia, Osteoporosis). On the authors' private AustinSpine dataset (389 patients), it reports 65.38% accuracy and 79.34% ROC AUC for MedConv, claimed to exceed all 2D CNN and transformer baselines, and it presents ablations over two temperature hyperparameters (tau_1, tau_2) and over two segmentation tools (CTSpine1K, TotalSegmentator). The paper's thesis is that a convolutional model with rebalancing techniques is a lower-complexity alternative to transformers for this task.","tokens_in":14223,"tokens_out":9120,"duration_ms":78606,"significance":"The question of whether simple 3D CNNs can match or surpass transformer-based models on imbalanced volumetric medical imaging is practically relevant, and the paper offers a clinically curated dataset and a head-to-head comparison of segmentation backbones. If the reported numbers were trustworthy, MedConv would serve as a useful efficient baseline for T-score screening. However, the experimental evidence in Tables II, V, and VI is unreliable due to internal metric inconsistencies and test-set-based hyperparameter selection, so the central claims cannot currently be evaluated.","major_comments":[{"comment":"In every row of Table II, Sensitivity is numerically equal to Accuracy and Specificity equals 0.5*Accuracy + 0.5 (e.g., MedConv: 65.38/65.38/82.69; ViT: 33.54/33.54/66.77). For any standard binary confusion matrix with a nonempty negative class, Sensitivity = Accuracy forces Specificity = Accuracy, so the reported affine relation is impossible. The task is three-class, and the paper never defines how scalar sensitivity/specificity are aggregated over classes. Thus the headline numbers are not genuine, interpretable performance estimates, and the claimed improvements over prior SOTA are unfalsifiable as reported. Please provide the per-class confusion matrices and the exact aggregation formulas.","section":"V-A, Table II"},{"comment":"The final MedConv configuration tau_1=1, tau_2=0.5 is selected because it maximizes Accuracy in ablations computed on the test set (Table VI shows that tau_2=0.5 gives the maximum 65.38% accuracy; the text describes tau_1=1 as 'optimal' and tau_2=0.5 as giving the best accuracy). No validation set and no nested cross-validation are described anywhere. The 65.38%/79.34% result is therefore a selection over hyperparameters, not an independent test estimate, and the comparison with baselines is biased. Moreover, the test set size implied by 65.38% is 78 patients, so the 1.28 percentage-point gap over the best non-MedConv row is about one patient; no confidence intervals or significance tests are given.","section":"V-B, Tables V and VI"},{"comment":"The post-hoc logit adjustment formula yhat_i = e^{z_i/tau_1}/(e^{z_i/tau_1}+e^{z_j/tau_2}) is a two-class softmax, with z_j and j left undefined and no normalization over the three classes of the task. As written, it cannot be applied to the reported 3-class problem. Similarly, the Bal-CE loss in Section III-B is given in binary cross-entropy form for a single sample; the paper does not state the multiclass softmax cross-entropy version with class weights. The method section is therefore incomplete for the actual experimental setup.","section":"III-C"},{"comment":"The claim that 'convolutions beat transformers' is not supported by the baseline quality: ViT+pretrain reaches only 33.54% accuracy, essentially chance for a three-class problem, while JointViT variants also stay near chance; no evidence is provided that these baselines were trained to convergence or with adequate hyperparameters. In addition, no comparison of computational cost (parameters, FLOPs, latency) is reported in Table II or elsewhere, although the abstract and conclusion advance a lower-complexity argument. A fair head-to-head would require competitive transformer training and an actual efficiency table.","section":"V-A, Table II"}],"minor_comments":[{"comment":"The heading 'Evaluation Matrices' should read 'Evaluation Metrics'.","section":"IV-B"},{"comment":"The Introduction contains the typo 'adress' for 'address'.","section":"I"},{"comment":"Several model names are corrupted timm identifiers (e.g., 'resnet18.a1in1k', 'mobilenetv2100.rain1k', 'efficientnetb0.rain1k', 'resnext5032x4d.a1hin1k'); these should be standard names such as 'resnet18.a1_in1k'.","section":"Tables II-IV"},{"comment":"The caption says the model ensures 'accurate predictions for the L1 vertebra segmentation task,' but the paper addresses T-score classification, not segmentation.","section":"Figure 2"},{"comment":"The boundary definitions overlap (Osteoporosis '-4 to -2.5' and Osteopenia '-2.5 to -1'), and the paper does not state how T-scores exactly at a boundary are assigned.","section":"Table I"},{"comment":"The dataset description provides only the total patient count (389); class counts, the train/validation/test split sizes, and the test set composition are missing, making the long-tailed distribution in Figure 4 unquantified.","section":"IV-A"},{"comment":"The abstract's claim of 'up to 21% improvement in accuracy' is not tied to a specific baseline in Table II; the closest row (resnet50.a1in1k+pretrain+balce) differs from MedConv by only 1.28 percentage points, so the magnitude of the claimed improvement should be recomputed and stated with its reference baseline.","section":"Abstract / Table II"}],"recommendation":"reject","confidential_remarks":"The pervasive equality Sensitivity=Accuracy and the affine Specificity relation in Table II suggest a systematic error in the evaluation pipeline rather than isolated typos; the authors should be asked to supply raw confusion matrices and code. Given that the main results table is invalid and that the final configuration was tuned on the test set, I recommend rejection rather than minor or major revision; a completely reworked evaluation on a fixed split with standard multiclass metrics would be needed for a new submission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Main thing to know: the headline numbers don't survive scrutiny. In Table II, every model has sensitivity exactly equal to accuracy, and specificity exactly equal to 50% + accuracy/2. That affine relation cannot come from any confusion matrix. For a 3-class problem the paper never says how sensitivity/specificity are aggregated. So the 65.38% accuracy and 79.34% AUC are not interpretable as reported.\n\nWhat's genuinely useful here is the dataset. AustinSpine, 389 CT scans with radiologist-reviewed T-scores and a long-tailed label distribution, is a real clinical resource, and comparing TotalSegmentator vs CTSpine1K input segmentation is a fair thing to test. The baseline table is broad, even if the transformer baselines look weak.\n\nThe soft spots are load-bearing. First, the metric problem above. Second, Section V-B selects tau_1 and tau_2 from test-set ablations (Tables V and VI), so the final numbers are a selected maximum, not an out-of-sample prediction. Third, no error bars or significance tests on a 389-patient dataset. Fourth, the post-hoc logit adjustment formula in Section III-C only shows two classes and is not the standard softmax temperature scaling; with three classes there is no single z_j, and the formula as written is ill-defined. Finally, there isn't much architectural novelty: a stock 3D ResNet-50 with inverse-frequency weighting and temperature scaling are known components.\n\nThe paper does cite the long-tail literature reasonably and doesn't pretend the components are new, but the claims in the abstract are not supported by the tables as they stand. If the authors re-run with a fixed train/val/test split, report standard per-class metrics and error bars, and fix the logit-adjustment description, there's a modest clinical application paper here. As submitted, I would not send it to review.","headline":"The headline accuracy is not interpretable because Table II's sensitivity and specificity are internally inconsistent, and the test-set tuning makes the result a selected maximum rather than a prediction.","tokens_in":14820,"tokens_out":2059,"would_cite":false,"duration_ms":20216,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MedConv, a 3D ResNet-50 with balanced cross-entropy loss and post-hoc logit adjustment, achieves the highest accuracy and ROC AUC on the AustinSpine CT dataset while using less compute than transformer baselines.","keywords":["bone density prediction","T-score","CT imaging","long-tailed distribution","class imbalance","3D convolutional neural network","balanced cross-entropy","post-hoc logit adjustment"],"falsifier":"Run the MedConv pipeline with $\\tau_1$ and $\\tau_2$ chosen only on a held-out validation fold, then evaluate once on the test set. If the resulting accuracy and ROC AUC do not exceed the transformer baselines in Table II, the central claim that convolutions beat transformers on this task is falsified.","tokens_in":13798,"feed_emoji":"🦴","tokens_out":8458,"duration_ms":71278,"temperature":0.7,"pith_summary":"The paper argues that a straightforward 3D convolutional network, MedConv, can outperform transformer-based models for predicting bone density (T-score) from spinal CT scans. On the AustinSpine dataset it reports 65.38% accuracy and 79.34% ROC AUC, beating all transformer and CNN baselines in its comparison while using lower computational cost. The improvement is attributed to two mechanisms: balanced cross-entropy loss that up-weights minority classes, and post-hoc logit adjustment that amplifies minority-class logits at inference. If the claim holds, resource-efficient CNN models are sufficient for CT-based bone density screening, and long-tailed class imbalance can be handled without complex architectures.","feed_headline":"Convolutions beat transformers on CT bone-density prediction","feed_subtitle":"A 3D ResNet-50 with balanced loss hits 65.38% accuracy and 79.34% AUC at lower compute cost.","key_machinery":"The load-bearing object is the 3D ResNet-50 backbone combined with two balancing mechanisms: the Bal-CE loss, which weights each training sample by the inverse frequency of its class, and a post-hoc logit adjustment that applies temperature scaling ($\\tau_1=1$ for the majority class, $\\tau_2=0.5$ for the minority class) to the final logits. The 3D convolutions capture volumetric context in CT scans, the loss weighting counters the long-tailed label distribution during training, and the logit adjustment reshapes the predicted probabilities at inference to favor minority classes. Together these parts let a standard CNN match or exceed transformer performance at a fraction of the compute.","core_discovery":"MedConv—a 3D ResNet-50 backbone fed with L1 vertebra segmentations and trained with inverse-frequency weighted cross-entropy plus a post-hoc logit adjustment ($\\tau_1=1$, $\\tau_2=0.5$)—reaches 65.38% accuracy and 79.34% ROC AUC on the AustinSpine test set, outscoring all reported transformer baselines as well as other CNN variants. The central discovery is that this conventional convolutional pipeline is enough to beat transformers on this long-tailed CT classification task, and that the class-imbalance techniques, rather than architectural novelty, account for most of the gain. The paper reports that the balanced loss raises ResNet-50 accuracy by 6.41 percentage points, and that lowering $\\tau_2$ from 1.0 to 0.5 lifts accuracy from 64.10% to 65.38%, indicating that logit rebalancing is the final performance-critical step.","pith_inferences":["Because $\\tau_1$ and $\\tau_2$ were chosen from test-set ablations, the reported 65.38% accuracy is likely an optimistic estimate of out-of-sample performance; a validation-based tuning loop would give a fairer number.","The 'convolutions beat transformers' conclusion may be sensitive to the specific transformer checkpoints and input preprocessing used, so it should be re-tested across more transformer variants and external datasets.","A direct extension would be to apply the same Bal-CE plus logit-adjustment recipe to other long-tailed 3D medical imaging tasks, such as fracture detection or liver steatosis grading, to see whether gains of this size reproduce.","Reporting confidence intervals or standard deviations across runs would make the benchmark more trustworthy, given the small dataset of 389 patients."],"forward_implications":["If the claim is right, 3D CNNs with rebalanced losses become a strong default for CT-based T-score classification, reducing the need for transformer backbones in this task.","The reported gains imply that model selection in long-tailed medical imaging should first address loss rebalancing and logit calibration before changing architecture.","The lower computational cost makes deployment on portable or edge devices feasible for bone density screening.","The AustinSpine benchmark and the TotalSegmentator-based preprocessing pipeline can serve as a standard evaluation setup for future bone density prediction models."],"supporting_citations":[{"why":"Supplies the L1 vertebra segmentations used as input to MedConv; the paper selects it over CTSpine1K after comparative experiments.","marker":"[20]"},{"why":"Provides an alternative segmentation source that is compared against TotalSegmentator to justify the input choice.","marker":"[19]"},{"why":"Defines the WHO T-score class boundaries (osteoporosis, osteopenia, normal) used for the classification labels.","marker":"[35]"},{"why":"Provides the class-balanced loss concept that the Bal-CE loss adapts for inverse-frequency weighting.","marker":"[29]"},{"why":"A prior CNN-based study predicting BMD from CT scans that MedConv extends and compares against.","marker":"[24]"},{"why":"A prior deep learning study predicting BMD from plain radiographs, used as a related state-of-the-art reference.","marker":"[23]"},{"why":"The transformer baseline (JointViT) that MedConv is directly compared with and outperforms in the experiments.","marker":"[13]"}],"fun_headline_variants":["Convolutions beat transformers on CT bone-density scans","MedConv CNN tops transformers for bone density CT","Plain 3D CNN outperforms transformers on bone density","Long-tailed bone density: convolution beats attention","CNN beats transformers at bone density with less compute"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported test accuracy is treated as an unbiased estimate of model performance, which requires that the hyperparameters $\\tau_1$ and $\\tau_2$ were not selected using the test set; the paper selects them from test-set ablations, so the final numbers are a selected maximum rather than an independent prediction.","fun_headline_variants_meta":{"raw":{"variants":["Convolutions beat transformers on CT bone-density scans","MedConv CNN tops transformers for bone density CT","Plain 3D CNN outperforms transformers on bone density","Long-tailed bone density: convolution beats attention","CNN beats transformers at bone density with less compute"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000235,"raw_usage":{"total_tokens":1475,"prompt_tokens":895,"completion_tokens":580,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":511,"completion_tokens_details":{"reasoning_tokens":507}},"tokens_in":511,"tokens_out":580,"duration_ms":6355,"temperature":1.0,"reasoning_tokens":507,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T18:14:43.981038+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the MedConv pipeline with $\\tau_1$ and $\\tau_2$ chosen only on a held-out validation fold, then evaluate once on the test set. If the resulting accuracy and ROC AUC do not exceed the transformer baselines in Table II, the central claim that convolutions beat transformers on this task is falsified.","supporting_citations":[{"cited_title":"Totalsegmentator: robust segmentation of 104 anatomic structures in ct images,","cited_arxiv_id":null,"evidence_quote":"Supplies the L1 vertebra segmentations used as input to MedConv; the paper selects it over CTSpine1K after comparative experiments."},{"cited_title":"World Health Organization, 1994","cited_arxiv_id":null,"evidence_quote":"Defines the WHO T-score class boundaries (osteoporosis, osteopenia, normal) used for the classification labels."},{"cited_title":"Class-balanced loss based on effective number of samples,","cited_arxiv_id":null,"evidence_quote":"Provides the class-balanced loss concept that the Bal-CE loss adapts for inverse-frequency weighting."},{"cited_title":"Prediction of bone mineral density from computed tomography: application of deep learning with a convolutional neural network,","cited_arxiv_id":null,"evidence_quote":"A prior CNN-based study predicting BMD from CT scans that MedConv extends and compares against."},{"cited_title":"Automated bone mineral density prediction and fracture risk assessment using plain radiographs via deep learning,","cited_arxiv_id":null,"evidence_quote":"A prior deep learning study predicting BMD from plain radiographs, used as a related state-of-the-art reference."},{"cited_title":"Jointvit: Modeling oxygen saturation levels with joint supervision on long-tailed octa,","cited_arxiv_id":null,"evidence_quote":"The transformer baseline (JointViT) that MedConv is directly compared with and outperforms in the experiments."}],"review_version":1}