{"id":"9e0bda61-abce-4a16-ad4f-c425d7a901ea","arxiv_id":"2411.13535","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A comparative study finds ResNet-50 achieves 93.06% accuracy on five-class cervical cell classification from Pap smear images, outperforming HOG-based machine learning models.","lead":"An AI benchmark compares six machine learning models and a deep neural network (ResNet-50) for classifying cervical cells in Pap smear images. ResNet-50 reaches 93.06% accuracy on the public SIPaKMeD dataset, beating the traditional models; the result is a routine benchmark with limited novelty and several methodological gaps.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 93.06% ResNet-50 accuracy is likely inflated by an image-level random split that ignores the 966-cluster structure of SIPaKMeD; a cluster-stratified test is needed before the central comparison is accepted.","rationale":"I read the paper as a routine benchmark whose central claim is the ResNet-50 test accuracy of 93.06%. The weakest point in the argument is the validity of that accuracy as a measure of generalization. Given that SIPaKMeD consists of 4049 images cropped from 966 clusters, an image-level random split risks placing images from the same cluster in both training and test sets, making the test set artificially easy. The paper does not describe any cluster-aware split or report any metric that would rule out this leakage. This is the same concern the reader identified, and it is directly load-bearing for the strongest claim. I do not see a separate internal inconsistency in the model descriptions: the ResNet-50 setup is standard, and the ML baselines are plausible, though low. The routine nature and limited novelty of the work do not by themselves invalidate the reported comparison; the correctness risk is empirical. A cluster-stratified re-run would settle whether the 93.06% figure is trustworthy. Since the reader's verdict is already CONDITIONAL and this concern does not move it in a new direction, I recommend no change to the verdict.","tokens_in":6010,"tokens_out":3818,"duration_ms":38957,"concrete_test":"Obtain the cluster identifiers for the 4049 SIPaKMeD images (available from the original dataset release), group images by cluster, and create train/validation/test splits using GroupShuffleSplit so that no cluster appears in more than one split. Retrain the same ResNet-50 pipeline (500 epochs, batch size 32, Adam learning rate 0.001) and the same ML pipelines, then report test accuracy and per-class accuracy. If the ResNet-50 accuracy drops materially (e.g., more than 3–5 percentage points) relative to 93.06%, the headline claim is inflated by data leakage and should be revised; if it remains within that range, the central claim survives this objection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that ResNet-50 achieves 93.06% test accuracy and outperforms the ML models. For this number to represent generalization to new patients or slides, the test images must be independent of the training images. Section 2.1 reports that the 4049 images were manually cropped from 966 cell clusters, and Section 2.2 describes only an 8:1:1 random split into train, validation, and test sets, with no mention of grouping by cluster. Images from the same cluster are not independent samples: they share acquisition conditions, staining, and often near-identical cell morphology, so a random image-level split can place sibling crops in both training and testing. The model can then exploit cluster-specific appearance rather than learn generalizable cell categories, inflating the reported accuracy. The claim that the model 'has potential to aid in the early diagnosis of cervical cancer' depends precisely on performance on unseen slides, so the missing cluster-level split is load-bearing. In addition, the 93.06% figure is a single point estimate with no confidence interval or repeated-split variance, but the leakage issue is the more severe threat to the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper compares traditional machine learning (Random Forest, Gradient Boosting, kNN, SVM with HOG features) and a deep learning model (ResNet-50) for classifying cervical squamous epithelial cells into five categories using the public SIPaKMeD dataset. The authors report that ResNet-50 achieves the highest test accuracy of 93.06%, outperforming the best ML model (SVM, 70.64%). They conclude that deep learning is more suitable for cell-level classification and could aid early cervical cancer diagnosis from Pap smear images.","tokens_in":6384,"tokens_out":3177,"duration_ms":32010,"significance":"If the reported result holds, the paper is a modest empirical benchmark confirming that fine-tuned deep networks outperform handcrafted-feature ML models on a standard public dataset. The use of a public dataset and the inclusion of both ML and DL baselines are positives, and the class-wise accuracies are useful for comparison. However, the central accuracy figure currently rests on a single random image-level split of images that are known to be non-independent due to cluster structure, and the lack of statistical or reproducibility details prevents the reader from assessing the reliability and generalizability of the claimed superiority.","major_comments":[{"comment":"Section 2.1 states that the 4,049 cell images were manually cropped from 966 clusters, while Section 2.2 describes only an 8:1:1 random split into train, validation, and test sets with no cluster-level grouping. Because images from the same cluster share acquisition conditions, staining, and often similar morphology, a random image-level split is likely to place sibling crops from the same cluster into both training and test sets. This can artificially inflate the reported 93.06% accuracy and the comparison between models, since the model may exploit cluster-specific appearance rather than learn generalizable cell categories. The central claim of the paper depends on performance on unseen slides, so the authors must re-run all experiments with a cluster-stratified split (e.g., splitting at the level of the 966 original clusters) and report both the accuracy and the number of clusters that span the split boundaries.","section":"§2.1 and §2.2"},{"comment":"All accuracies in Table 2 are single point estimates from one random split. The test set contains only about 405 images, so the reported differences between models are accompanied by considerable sampling variability, yet no confidence intervals, standard deviations, or repeated runs with different seeds are provided. The claim that ResNet-50 'outperformed' the ML models would be more credible if the authors reported the mean and standard deviation of accuracy over multiple independent splits, or at least a bootstrap confidence interval, to show that the ranking is stable.","section":"§3, Table 2"},{"comment":"The training and evaluation protocol is under-specified, which prevents reproducibility of the comparative analysis. For ResNet-50, the authors give the optimizer, learning rate, batch size, and epoch count, but the data augmentation operations (flipping, noise addition, contrast change) are listed without magnitudes or probabilities, and it is unclear whether the entire network is fine-tuned or only the final linear layer is trained ('feature extraction' is ambiguous). For the ML models, the HOG parameters (cell size, block size, number of orientations) and classifier hyperparameters (e.g., SVM kernel and C, k for kNN, number of trees for Random Forest, boosting hyperparameters for Gradient Boosting/XGBoost/CatBoost) are not reported. Without these settings, the comparison cannot be reproduced or independently verified.","section":"§2.2"}],"minor_comments":[{"comment":"The row numbering is inconsistent: two rows are labeled '2' (Scikit-learn Gradient Boost and XGBoost). Also, XGBoost and CatBoost appear in the results but are not described in the Methods section.","section":"Table 2"},{"comment":"The confusion matrix and ROC curves would benefit from axis labels and a legend; the current figure is difficult to read because the class names are only given in the caption, not on the axes.","section":"Figure 4"},{"comment":"The phrase 'ImageNet pre-trained weights for feature extraction' is ambiguous: clarify whether the convolutional backbone was frozen or fine-tuned. This materially affects the interpretation of the 500-epoch training procedure.","section":"§2.2"},{"comment":"The HOG feature extraction is cited to a vehicle-detection paper [21]; a more standard and complete description, such as the original Dalal and Triggs reference, would be more appropriate for this widely used method.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is a straightforward empirical comparison on a public dataset, which fits a reproducibility-oriented journal. The main blocking issue is the cluster-leakage problem; if the authors can show that results are robust under a cluster-stratified split, the paper could be suitable for publication after also adding variance estimates and full hyperparameter details. I would also encourage the editor to verify that the dataset's original description indeed supports the '966 clusters' interpretation used in this report, as the authors' own Table 1 lists 966 as the total number of 'Images' across categories."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know: the paper is a straightforward benchmark of HOG+ML and ResNet-50 on SIPaKMeD, and the headline 93.06% is probably inflated because the 8:1:1 random split ignores that the 4,049 images are crops from 966 clusters. Sibling crops leak across train and test, so the model can memorize cluster appearance. The authors describe the cluster structure in Section 2.1 but never group by it in the split (Section 2.2), so the central comparison to ML models rests on an invalid independence assumption.\n\nWhat's good: the paper is clearly written, uses a public dataset, reports per-class accuracies, and makes no parameter-fitting claims. The ML pipeline is standard and the ResNet-50 setup is transparent enough (500 epochs, batch 32, Adam LR 0.001). It does not overclaim: the discussion is moderate and the limitations of ML models are stated plainly.\n\nSoft spots: beyond the split, there are no confidence intervals or repeated runs, no code, and no direct numeric comparison with prior published results on the same dataset (e.g., CVM-Cervix, which the authors cite). The ML results are oddly low—SVM at 70%—and the paper doesn't explain why they underperform published HOG results, which suggests possible differences in preprocessing or a poorly tuned HOG. The absence of any cluster-stratified evaluation is the load-bearing flaw; everything else is fixable with a few paragraphs and a table.\n\nWho this is for: someone teaching a course on medical image classification pitfalls, or a reviewer wanting a concrete example of data leakage. As a research contribution it is incremental; it does not advance the state of the art. But it is not a waste of time—if the authors rerun with a cluster-level split and add error bars, the comparison becomes a useful sanity check for the field.\n\nMy recommendation: send it to peer review with the expectation of major revision. The leak is real but identifiable; a competent referee can force the fix. If the authors refuse to rerun, the paper should not be accepted as is.","headline":"A routine SIPaKMeD benchmark whose headline ResNet-50 accuracy is likely inflated by image-level random splitting that ignores the 966-cluster structure; fixable but not publishable as is.","tokens_in":6793,"tokens_out":1942,"would_cite":false,"duration_ms":18987,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ResNet-50 outperforms classical ML on cervical smear cells at 93%","keywords":["cervical cancer","Pap smear","squamous epithelial cells","ResNet-50","deep learning","machine learning","HOG features","cell classification"],"falsifier":"Re-run the same ResNet-50 training with an 8:1:1 split performed at the level of the 966 cell clusters rather than the 4,049 individual images; if the test accuracy falls well below 93.06%, the headline number reflects image leakage rather than generalization to new cervical samples.","tokens_in":5845,"feed_emoji":"🔬","tokens_out":8436,"duration_ms":74201,"temperature":0.7,"pith_summary":"This paper tests whether machine learning or deep learning is better at sorting cervical cells in Pap smear images into five categories: superficial-intermediate, parabasal, koilocytes, dyskeratotic, and metaplastic. Using the SIPaKMeD dataset of 4,049 cropped cell images, the authors compare handcrafted-feature classifiers (SVM, k-nearest neighbors, random forest, gradient boosting variants) against a fine-tuned ResNet-50. The central claim is that ResNet-50 reaches 93.06% test accuracy, far above the best machine-learning result of 70.64% (SVM), with per-class accuracies from 85.98% to 98.89%. The authors argue this supports deep learning as the practical choice for automated cervical cytology screening and early cervical cancer diagnosis.","feed_headline":"ResNet-50 outperforms classical ML on cervical smear cells at 93%","feed_subtitle":"It beats the best handcrafted-feature model by more than 22 points on five cervical cell types.","key_machinery":"The argument runs on two pipelines. For machine learning, the central object is the Histogram of Oriented Gradients (HOG) descriptor, which converts each cell image into a histogram of local gradient orientations that captures shape, edge direction, and texture; those feature vectors are fed into random forest, gradient boosting, XGBoost, CatBoost, k-nearest neighbors, and support vector machine classifiers. For deep learning, the central object is ResNet-50, a 50-layer convolutional neural network whose residual blocks use shortcut connections to skip layers and learn residual functions, easing the vanishing-gradient problem and enabling deeper training. The network's final layer is replaced with a five-output linear layer, weights are initialized from ImageNet pretraining, and training runs for 500 epochs with the Adam optimizer, a batch size of 32, and augmentations such as flipping, noise, and contrast changes. The data is split 8:1:1 into training, validation, and test sets.","core_discovery":"The paper's central discovery is that a ResNet-50 convolutional network fine-tuned on Pap smear cell images classifies five cervical squamous epithelial cell types with 93.06% accuracy, outperforming every handcrafted-feature machine-learning model tested. Among the classical models, the support vector machine was the best at 70.64%, while random forest, gradient boosting, XGBoost, CatBoost, and k-nearest neighbors all landed below 59%. The ResNet-50 model, trained for 500 epochs with ImageNet-pretrained weights and data augmentation, reached 98.89% accuracy on parabasal cells, 95.83% on superficial-intermediate cells, 93.27% on dyskeratotic cells, 92.52% on metaplastic cells, and 85.98% on koilocytes. The authors conclude that deep learning is the more reliable route for cell-level classification in automated cervical cancer screening, despite its higher computational cost.","pith_inferences":["The 8:1:1 split was made on the 4,049 individual images, but those images were cropped from only 966 cell clusters; if images from the same cluster appear in both training and test sets, the reported accuracy overstates how the model would perform on a new patient or slide. A cluster-level split would be the stricter test.","Because the test set shares the same staining, microscope, and acquisition setup as the training set, the model's accuracy on images from other labs or preparation protocols is an open question that the paper does not address.","The comparison could be sharpened by reporting precision, recall, and F1-score per class alongside accuracy, since the five classes are imbalanced and accuracy alone may hide systematic confusions.","Fine-tuning on a larger, multi-center cervical cytology dataset would show whether the ResNet-50 advantage persists when variability in staining and cell morphology is higher."],"forward_implications":["If 93.06% accuracy holds up, an automated ResNet-50-based screener could flag abnormal cervical cells from Pap smear images at a level of agreement that manual review struggles to match at scale.","Deep learning would be the preferred approach for clinical deployment because, as the paper notes, accuracy matters for clinical acceptance even when training is computationally expensive.","The classical machine-learning models, though cheap to train, would be too inaccurate for screening use, supporting a shift toward convolutional networks in cervical cytology.","Per-class results suggest that koilocytes, the HPV pathognomonic cells, are the hardest to recognize (85.98%), pointing future work at improving that category."],"supporting_citations":[{"why":"Supplies the dataset of 4,049 cervical cell images cropped from 966 clusters across five classes, which the whole comparison runs on.","marker":"[3]"},{"why":"Defines the random forest algorithm used as one of the machine-learning baselines.","marker":"[4]"},{"why":"Defines gradient boosting machines, the method family behind the gradient boosting baselines.","marker":"[5]"},{"why":"Describes support vector machines, the best-performing classical baseline in the study.","marker":"[7]"},{"why":"Provides the ResNet-50 architecture and residual-learning concept that carries the deep-learning pipeline.","marker":"[8]"},{"why":"Gives the HOG feature-extraction workflow used to build input vectors for the machine-learning models.","marker":"[21]"},{"why":"Earlier convolutional-neural-network work on Pap smear image classification that motivates the deep-learning approach.","marker":"[22]"},{"why":"Recent hybrid CNN-transformer framework for cervical cell classification, cited as the state of the art the paper positions itself against.","marker":"[23]"}],"fun_headline_variants":["ResNet-50 beats classical ML on cervical cell classification at 93%","Deep learning outshines handcrafted models for Pap smear cell types","AI model classifies 5 cervical cell types with 93% accuracy","Deep learning outperforms classical ML in cervical cancer screening","93% accuracy: AI model tops cervical cell classification"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The 8:1:1 split treats each of the 4,049 cropped cell images as an independent sample, but the images were cropped from only 966 cell clusters, so if images from one cluster land in both training and test sets the 93.06% accuracy overstates how well the model would generalize to a new patient or slide.","fun_headline_variants_meta":{"raw":{"variants":["ResNet-50 beats classical ML on cervical cell classification at 93%","Deep learning outshines handcrafted models for Pap smear cell types","AI model classifies 5 cervical cell types with 93% accuracy","Deep learning outperforms classical ML in cervical cancer screening","93% accuracy: AI model tops cervical cell classification"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00077,"raw_usage":{"total_tokens":3446,"prompt_tokens":1016,"completion_tokens":2430,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":632,"completion_tokens_details":{"reasoning_tokens":2341}},"tokens_in":632,"tokens_out":2430,"duration_ms":18142,"temperature":1.0,"reasoning_tokens":2341,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:17:20.520246+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same ResNet-50 training with an 8:1:1 split performed at the level of the 966 cell clusters rather than the 4,049 individual images; if the test accuracy falls well below 93.06%, the headline number reflects image leakage rather than generalization to new cervical samples.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the random forest algorithm used as one of the machine-learning baselines."},{"cited_title":"E., Dimitrakopoulos, P., Sfikas, G., Nikou, C., Krikoni, O., & Charchanti, A","cited_arxiv_id":null,"evidence_quote":"Supplies the dataset of 4,049 cervical cell images cropped from 966 clusters across five classes, which the whole comparison runs on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines gradient boosting machines, the method family behind the gradient boosting baselines."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Describes support vector machines, the best-performing classical baseline in the study."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the ResNet-50 architecture and residual-learning concept that carries the deep-learning pipeline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the HOG feature-extraction workflow used to build input vectors for the machine-learning models."},{"cited_title":"B., Kundu, M","cited_arxiv_id":null,"evidence_quote":"Earlier convolutional-neural-network work on Pap smear image classification that motivates the deep-learning approach."},{"cited_title":"M., Sun, H.,","cited_arxiv_id":null,"evidence_quote":"Recent hybrid CNN-transformer framework for cervical cell classification, cited as the state of the art the paper positions itself against."}],"review_version":1}