{"id":"ab58f8c0-8379-4afa-be64-034a92a4d74c","arxiv_id":"2508.20919","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A ConvNeXt ensemble achieves 84% balanced accuracy on MIDOG25 atypical mitotic figure classification, while a rule-based refinement module trades sensitivity for specificity.","lead":"This paper describes a deep learning ensemble that classifies atypical versus normal mitotic figures in tumor images, reaching 84% balanced accuracy on the MIDOG25 challenge test set. It also tests a rule-based refinement module that improves specificity but lowers sensitivity, showing that hand-crafted rules need better cell detectors.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unverified label harmonization across three merged datasets is the main threat to the 84.02% balanced-accuracy claim; without it, the test metric is not well-defined.","rationale":"The reader identified the label-consistency assumption as the weakest assumption. I agree that this is the most load-bearing concern: the evaluation is on a single test set whose labels come from the MIDOG25 challenge, but the training data are pooled from three datasets whose annotation guidelines are never compared. If the labels are inconsistent, the reported balanced accuracy is not a clean measure of AMF classification, and the central claim loses its validity. The reader also mentioned the absence of error bars; that is secondary because even with error bars, the metric would be ill-posed under label mismatch. The concrete test I propose directly settles the label-harmonization question. Therefore the verdict remains CONDITIONAL: the paper should be accepted only if the label harmonization can be verified or the claim is adjusted to reflect the uncertainty.","tokens_in":4225,"tokens_out":7857,"duration_ms":83786,"concrete_test":"Have an independent pathologist re-annotate 100 random mitotic figures from each source dataset (AMi-Br, MIDOG25, Octopath) following the official MIDOG25 label definition. Compute Cohen's kappa between the original dataset labels and the reference annotation. If any kappa falls below 0.8, harmonize the training labels and retrain the ensemble; if the balanced accuracy shifts by more than 2 percentage points, the original claim is compromised. A complementary computational check is to train a three-way classifier to predict the source dataset from each MF tile; source-predictive accuracy above 90% combined with source-dependent ensemble errors would signal label or domain mismatch.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section II, the paper merges AMi-Br, MIDOG25, and Octopath into a single 15,689-MF training set with no label-harmonization check. The central claim (Section III, Table I) is a balanced accuracy of 84.02% on the MIDOG25 preliminary test set. For that number to meaningfully represent 'distinguishing atypical from normal mitoses,' the label definition must be consistent across all three sources. The paper provides no inter-annotator statistics, no per-dataset label distribution, and no evaluation of cross-source transfer. If, for example, Octopath labels AMFs according to different morphologic criteria, the ensemble learns a confounded target. The reported accuracy could then reflect a mixture of true AMF recognition and dataset-specific artifacts, undermining the claim that deep ensembles perform well for AMF classification. The lack of confidence intervals further prevents us from assessing the stability of the point estimate, but the label issue is more fundamental because it challenges the validity of the metric itself.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a two-stage pipeline for classifying atypical versus normal mitotic figures (AMF/NMF) in H&E-stained histopathology images: an ensemble of three ConvNeXtBase models trained with the AUCMEDI framework, optionally combined with a rule-based refinement (RBR) module. The RBR module adjusts the deep-learning score based on StarDist cell detection and morphological features such as cell shape and orientation. On the MIDOG25 preliminary test set, the ensemble alone achieves a balanced accuracy of 84.02% (sensitivity 92.96%, specificity 75.09%, AUC 92.84); adding RBR changes these to 83.44%, 85.92%, 80.97%, and 89.17%, respectively. The authors conclude that deep ensembles provide strong baselines for AMF classification, while the current RBR approach does not improve overall performance but may improve specificity with further research.","tokens_in":4519,"tokens_out":4326,"duration_ms":45571,"significance":"If the result holds, the paper provides a strong, reproducible baseline for the MIDOG25 AMF/NMF classification task, which is of practical interest for tumor grading. The authors make their code publicly available, describe training details sufficiently for re-implementation, and honestly report that the RBR module degrades overall performance. These are strengths. The central claim, however, rests on the validity of the test metric, which depends on label consistency across three merged datasets, and the lack of uncertainty quantification weakens the strength of the conclusions.","major_comments":[{"comment":"The paper merges AMi-Br, MIDOG25, and Octopath into a single 15,689-MF training set with no label-harmonization analysis. The AMi-Br dataset is known to contain subtypes (indeed, 'AMF other' is used in the RBR rules), and if the three datasets use different morphological criteria for AMF, the binary target is ill-defined. Consequently, the balanced accuracy of 84.02% in Table I may reflect dataset-specific artifacts rather than true AMF recognition. Please report per-dataset class distributions, cite or reproduce the challenge's label definitions, and provide evidence (e.g., inter-annotator statistics or confirmation from the challenge organizers) that AMF/NMF labels are interchangeable across the three sources.","section":"Section II – Data"},{"comment":"All metrics are point estimates on a single preliminary test set, with no confidence intervals or significance tests. This is especially problematic for the RBR comparison: the specificity increase from 75.09% to 80.97% and sensitivity drop from 92.96% to 85.92% could plausibly be within sampling variability. Bootstrapped 95% confidence intervals, or at least a paired test (e.g., McNemar for the binary decisions, DeLong for AUC), are needed before asserting that 'RBR can increase specific metrics.' The current presentation does not support a statistical distinction between the two configurations.","section":"Section III – Table I"},{"comment":"The RBR modification weights (e.g., 0.6 for clear parallel orientation, 0.2 for a ring-like shape) are chosen ad hoc, and the paper acknowledges that they reflect trust in the analysis algorithms rather than measured significance. Since the RBR conclusion is based on these specific weights, a sensitivity analysis (varying the weights or showing the decision boundary) is needed to establish that the specificity improvement is robust to the choice of parameters. Without it, the statement that RBR can increase specificity should be limited to 'with these weights, on this test set.'","section":"Section II – Rule-Based Refinement"}],"minor_comments":[{"comment":"Typo: 'extend' should be 'extended.' Also, capitalization of 'Mitotic Figures' is inconsistent with 'NMFs' and 'MFs' later in the text.","section":"Abstract"},{"comment":"Typo: 'tree models' should be 'three models.' Also, there is a missing period after the torchstain reference: 'in the torchstain framework [10]. was used.'","section":"Section II – Training"},{"comment":"It would be helpful to clarify whether the final submission trained on all available data includes any images from the same patients as the preliminary test set, since the earlier patient-stratified split is abandoned. The test set is external, but potential patient overlap should be excluded or explicitly discussed.","section":"Section II – Data split"},{"comment":"Figure 2 caption is minimal; consider labeling the green path (AUCMEDI only) and the RBR path directly in the figure so the reader can follow the inference flow without switching between caption and text.","section":"Figures"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is very concise and reads like a challenge-report or short paper. The code release and honest reporting are commendable. The main gap is statistical and label-validity support for the central claim. If the target venue is a full-length journal, the paper would also need a more thorough analysis and discussion. I recommend major revision because the label-harmonization and confidence-interval issues are addressable and directly affect the paper's conclusions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis is a competent, honest challenge paper rather than a breakthrough. The genuinely new bit is the first ConvNeXtBase ensemble and a rule-based refinement (RBR) module on the MIDOG25 atypical vs. normal mitotic figure task, and it ships with code and enough detail to re-implement. The headline number, 84.02% balanced accuracy on the preliminary test set, is a respectable baseline, and the negative result on RBR is reported fairly: it raises specificity but lowers sensitivity and overall balanced accuracy. Credit where it's due: the training pipeline is described concretely, the authors openly admit the RBR was engineered without pathologists, and the GitHub makes reproduction plausible.\n\nThe soft spots are real but not disqualifying. The main one is the label-harmonization assumption. In Section II, AMi-Br, MIDOG25, and Octopath are merged into a single 15,689-MF set with no evidence that “atypical” and “normal” are defined consistently across sources. If Octopath or AMi-Br uses different morphologic criteria, the ensemble learns a confounded target and the 84.02% metric becomes a mixture of true recognition and dataset artifacts. The stress-test note makes this point, and it holds up: the paper gives no per-dataset label distributions, no inter-annotator statistics, and no cross-source transfer analysis. That is the kind of thing a referee should ask for, not a reason to desk-reject.\n\nTwo smaller issues: the point estimate comes from a single preliminary test set with no confidence intervals or bootstrap, so the 0.58 point difference between baseline and RBR is close to noise. And the RBR weights are admittedly ad hoc, chosen by the authors’ trust in their analysis modules rather than tuned or validated. That is fine for a heuristic, but it means the negative result is only suggestive.\n\nBottom line: this is a useful baseline for the MIDOG25 task and an honest evaluation of a domain-knowledge refinement idea. It deserves a serious referee and likely publication in a challenge-proceedings context after the label-consistency question is addressed. I’d bring it to a reading group only if someone here is working on mitotic figure classification; otherwise it’s a bit narrow.\n\nRecommendation: send to peer review, conditional on the authors addressing label harmonization.","headline":"A solid, honest challenge entry whose main claim rests on an unverified label-harmonization assumption across three merged datasets.","tokens_in":4878,"tokens_out":2499,"would_cite":false,"duration_ms":24531,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An ensemble of three neural networks separates atypical from normal mitotic figures with 84.02% balanced accuracy on the MIDOG25 preliminary test set; a rule-based refinement module raises specificity but lowers sensitivity and overall perf","keywords":["mitotic figures","atypical mitosis","deep ensemble learning","ConvNeXt","rule-based refinement","histopathology image classification","MIDOG25 challenge","balanced accuracy"],"falsifier":"Take the final MIDOG25 test set and rerun the exact ensemble with the same architecture, training data, and threshold; if its balanced accuracy is statistically indistinguishable from or below 75%, the 84.02% result was specific to the preliminary set. If, on the same set, the RBR with a mitosis-specific cell detector beats the ensemble, the paper's conclusion that refinement cannot improve overall performance is wrong.","tokens_in":4224,"feed_emoji":"🔬","tokens_out":9217,"duration_ms":89747,"temperature":0.7,"pith_summary":"This paper tackles a two-way classification problem in tumor pathology: deciding whether a mitotic figure, a cell caught in the act of dividing, is atypical or normal, a distinction that is labor-intensive and subjective when done by eye. The authors build an ensemble of three pretrained ConvNeXtBase image classifiers on roughly 15,700 mitotic-figure crops merged from three public datasets, and report a balanced accuracy of 84.02% on the MIDOG25 preliminary test set. They also test a rule-based refinement module that uses cell shape and orientation to adjust the ensemble's scores. The refinement raises specificity from 75.09% to 80.97% but lowers sensitivity from 92.96% to 85.92%, so the plain ensemble remains the better overall model. The paper's central claim is that deep ensembles are a strong baseline for atypical-mitosis classification, and that this refinement idea needs a more reliable cell detector before it can help.","feed_headline":"Three-network ensemble scores 84% on mitosis challenge","feed_subtitle":"Morphology-rule module gains specificity but loses sensitivity; the plain ensemble wins","key_machinery":"The mechanism is a two-branch inference pipeline. Branch one is the deep ensemble: three ConvNeXtBase networks, trained in a standardized medical-image classification framework with class-weighted focal loss and three-fold cross-validation, each output a softmax vector, and the mean vector is thresholded at 0.5. Branch two, the rule-based refinement module, is a modifier rather than a classifier: after Macenko stain normalization, a pretrained general-purpose cell detector finds cells near the image center; hand-written rules then shift the ensemble score up or down depending on how many cells are found, whether a single cell is ring-like, round, or oval, and whether two cells are parallel o","core_discovery":"At its core, the paper reports an empirical finding: on the MIDOG25 preliminary test set, an ensemble of three ConvNeXtBase networks distinguishes atypical from normal mitotic figures with 84.02% balanced accuracy, with sensitivity of 92.96% and specificity of 75.09%. When a rule-based refinement module is switched on, specificity rises to 80.97% but sensitivity falls to 85.92%, for a balanced accuracy of 83.44%. The authors therefore conclude that the deep ensemble alone is the stronger classifier, and that tying morphological heuristics to an unreliable general-purpose cell detector is what prevents the refinement from helping.","pith_inferences":["The 0.5 threshold is an unexamined dial: lowering it would shift the plain ensemble toward the same specificity as the RBR module without incurring the module's detection errors, so a threshold sweep is the cheapest test of the refinement's value.","Because the evaluation split was patient-stratified but within the same challenge datasets, the 84.02% figure likely overstates performance on a new center, stain, or scanner; the relative ranking of ensemble vs RBR could also change under domain shift.","The hand-set rule weights (0.2–0.6) encode the developers' trust in heuristics rather than measured importance; tuning these weights on a held-out set might recover some of the lost sensitivity.","The mismatch between the two stain-normalization methods (Reinhard for the ensemble, Macenko for the RBR) is never isolated; raw color differences could explain part of the sensitivity drop that is currently attributed to the rules."],"forward_implications":["A three-model ConvNeXtBase ensemble is enough to establish a strong baseline for atypical-versus-normal mitosis classification, so future challenge submissions can be compared against an 84% balanced-accuracy bar.","The reported error pattern—high sensitivity, low specificity—means the ensemble will rarely miss atypical figures but will over-flag normals; clinical deployment would need to decide which error is more costly.","The six-point specificity gain from the rule-based module shows morphological cues carry useful signal, but the module's current form is not worth its sensitivity loss.","The paper's own conclusion is that a mitosis-specific cell detector, replacing the general-purpose one, is the prerequisite for making rule-based refinement viable."],"supporting_citations":[{"why":"defines the challenge and provides the preliminary test set used for the headline evaluation.","marker":"[5]"},{"why":"supplies the ConvNeXtBase architecture that the three-model ensemble is built from.","marker":"[12]"},{"why":"provides the standardized training pipeline used to fit the ensemble.","marker":"[11]"},{"why":"supplies the class-weighted focal loss used to handle the imbalanced atypical/normal classes.","marker":"[13]"},{"why":"contributes the AMi-Br breast-cancer mitotic-figure dataset merged into the training set.","marker":"[6]"},{"why":"contributes the MIDOG25 mitotic-figure dataset.","marker":"[7]"},{"why":"contributes the Octopath atypical-mitosis dataset.","marker":"[8]"},{"why":"provides the Macenko stain normalization used only in the rule-based refinement branch.","marker":"[14]"},{"why":"supplies the pretrained cell detector whose outputs drive the morphology rules.","marker":"[15]"}],"fun_headline_variants":["Ensemble beats rule refinement in mitosis classification","Deep ensemble wins over morphology rules in MIDOG25","84% accuracy: ensemble outperforms rule-based tweaks","Rule-based refinement hurts sensitivity, ensemble better","Ensemble alone beats ensemble + rules in MIDOG25"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The test accuracy is only meaningful if all three merged datasets label 'atypical' and 'normal' the same way, so that one binary classifier has a consistent ground truth to learn and be scored against.","fun_headline_variants_meta":{"raw":{"variants":["Ensemble beats rule refinement in mitosis classification","Deep ensemble wins over morphology rules in MIDOG25","84% accuracy: ensemble outperforms rule-based tweaks","Rule-based refinement hurts sensitivity, ensemble better","Ensemble alone beats ensemble + rules in MIDOG25"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000221,"raw_usage":{"total_tokens":1228,"prompt_tokens":623,"completion_tokens":605,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":367,"completion_tokens_details":{"reasoning_tokens":531}},"tokens_in":367,"tokens_out":605,"duration_ms":5846,"temperature":1.0,"reasoning_tokens":531,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T14:41:29.516064+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the final MIDOG25 test set and rerun the exact ensemble with the same architecture, training data, and threshold; if its balanced accuracy is statistically indistinguishable from or below 75%, the 84.02% result was specific to the preliminary set. If, on the same set, the RBR with a mitosis-specific cell detector beats the ensemble, the paper's conclusion that refinement cannot improve overall performance is wrong.","supporting_citations":[{"cited_title":"Standardized Medical Image Classification across Medical Disciplines","cited_arxiv_id":"2210.11091","evidence_quote":"provides the standardized training pipeline used to fit the ensemble."},{"cited_title":"Histologic Dataset of Normal and Atypical Mitotic Figures on Human Breast Cancer (AMi-Br),","cited_arxiv_id":null,"evidence_quote":"contributes the AMi-Br breast-cancer mitotic-figure dataset merged into the training set."},{"cited_title":"OMG-Octo Atypical: A refinement of the original OMG-Octo database to incorporate atypical mitoses,","cited_arxiv_id":null,"evidence_quote":"contributes the Octopath atypical-mitosis dataset."},{"cited_title":"A method for normalizing histology slides for quantitative analysis,","cited_arxiv_id":null,"evidence_quote":"provides the Macenko stain normalization used only in the rule-based refinement branch."},{"cited_title":"Nuclei Instance Segmentation and Classification in Histopathology Images with Stardist,","cited_arxiv_id":null,"evidence_quote":"supplies the pretrained cell detector whose outputs drive the morphology rules."}],"review_version":1}