{"id":"0bff9790-63d6-43e4-8f05-eb5bfd248ff7","arxiv_id":"2504.14139","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A 4-million-parameter EfficientNetB0 pipeline with YOLOv10-based augmentation and curriculum learning reaches 89.19% macro F1 on internal thyroid FNAB tests but only 0.68 macro F1 on external validation.","lead":"This paper describes ThyroidEffi, a lightweight deep learning system that sorts thyroid biopsy images into three treatment groups, and tests it on images from two Vietnamese hospitals. The system is fast and cheap to run, but its accuracy drops sharply on the middle \"indeterminate\" category in real-world testing.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Model selection appears to have used the held-out test set (Table 2, §3.1), so the internal macro F1 of 89.19% may be an optimistically biased maximum over 12 candidates rather than an unbiased performance estimate.","rationale":"The reader's verdict is CONDITIONAL and the reader did flag in the rationale that 'the architecture selection may have used the test set,' but the reader's declared weakest assumption is the one-image-per-patient and crop-label inheritance premise. I agree that crop-label inheritance is a legitimate limitation, especially because a single FNAB field may not contain the diagnostic region and crops may not be representative. However, the test-set selection issue is more directly load-bearing for the central claim: it concerns the validity of the internal 89.19% macro F1 and 0.95-0.98 AUCs that drive the 'high-accuracy' conclusion. If Table 2 was used to choose EfficientNetB0, those numbers are not an honest out-of-sample estimate. The external validation partially mitigates this by providing independent evidence, but that evidence itself shows a substantial drop, particularly for the Indeterminate/Suspicious class. The appropriate remedy is not rejection: the paper reports a deployed system with transparent external results and a reproducible pipeline. The appropriate remedy is to require a validation-based selection protocol or, failing that, to reposition the paper's claims around the external deployment numbers and treat the internal numbers as exploratory. I therefore keep the CONDITIONAL verdict but change the binding condition from the crop-label premise to the test-set selection issue, with the crop-label concern folded into future work on site-specific and multi-field evaluation.","tokens_in":20101,"tokens_out":4579,"duration_ms":45255,"concrete_test":"Rerun the 12-model comparison in Table 2 using only the validation macro F1 for architecture selection, exactly as Appendix E.3 describes for hyperparameter selection, and keep the test set untouched until the single selected model is evaluated once. Report validation F1 for all candidates, the selected architecture, and its test macro F1 and per-class AUC. If EfficientNetB0 is no longer selected, or if its test F1 drops by more than about 0.01 from 0.8919, the internal headline claim is unsupported and the paper must be revised to present the external results as the primary evidence.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central internal-performance claim rests on §3.1, where Table 2 is explicitly labeled 'F1 Score on Test Set for Different Models and Configurations' and where EfficientNetB0 is 'designated as ThyroidEffi Basic' after achieving the highest augmented F1 (0.8919) in that same table. Section 2.2.3 (M3) describes selecting the optimal model among twelve candidates using criteria that include F1-score, and no validation-set comparison is reported for these architectures. As written, this is evidence that the test set informed architecture selection. If so, the reported 0.8919 macro F1 is the maximum of twelve test-set draws, a quantity that is expected to overstate the true performance of the chosen model even with no other errors. The external validation set was not used for this selection and therefore remains informative; however, the external macro F1 is only 0.68, with an Indeterminate/Suspicious AUC of 0.7436, which already diverges from the abstract's 'high-accuracy' characterization. The reader's crop-label inheritance concern is real but secondary: test and external evaluation are performed on full images, so noisy crop labels primarily affect training, whereas a test-set-based architecture choice directly biases the internal headline metric itself.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents ThyroidEffi, a three-class (Benign, Indeterminate/Suspicious, Malignant) deep-learning pipeline for classifying thyroid FNAB images. The pipeline combines YOLOv10-based cell-cluster detection for targeted data augmentation, a curriculum-learning training order (crop sets E, D, C, B, A), a lightweight EfficientNetB0 backbone for the Basic model, and a Transformer-inspired multi-region module for the Premium model. The authors report an internal test-set macro F1 of 89.19% with AUCs of 0.98/0.95/0.96, and an external validation on 1,015 images from a different hospital with AUCs of 0.9495/0.7436/0.8396 and a macro F1 of 0.68. The paper claims that high-accuracy, interpretable, cost-effective thyroid FNAB classification is achievable, and it provides code and deployment details.","tokens_in":20378,"tokens_out":3205,"duration_ms":28326,"significance":"If the results were unbiased, the paper would provide a useful, low-cost, deployable system for a clinically meaningful three-way triage task, with the valuable feature of an independent external validation set. The main contributions are the YOLOv10-based augmentation strategy, the curriculum ordering, the lightweight architecture choice, and the prospective external evaluation, all of which are concrete and reproducible. The availability of source code and the explicit reporting of external F1/AUC are strengths. However, the central internal-performance claim appears to be affected by test-set-based architecture selection, and the external results are substantially weaker than the abstract's 'high-accuracy' characterization, particularly for the Indeterminate/Suspicious class. The paper's significance therefore hinges on whether the authors can provide an unbiased estimate of internal performance and reframe their claims to match the external evidence.","major_comments":[{"comment":"The architecture selection appears to have used the held-out test set. Table 2 is explicitly titled 'F1 Score on Test Set for Different Models and Configurations', and EfficientNetB0 is 'designated as ThyroidEffi Basic' after achieving the highest augmented F1 (0.8919) in that same table. Since no validation-set architecture comparison is reported, the reported 0.8919 macro F1 is the maximum over 12 test-set draws and is expected to overstate the true performance of the chosen model. The authors should redo model selection using only the validation set (as described in Appendix E.3 for hyperparameters) and report the test performance of the validation-selected model, or at minimum clearly disclose this selection bias and temper the internal-performance claims accordingly.","section":"§3.1, Table 2"},{"comment":"The external validation results do not support the abstract's 'high-accuracy' phrasing for all classes. Table 4 reports external macro F1 of 0.68 and an Indeterminate/Suspicious F1 of only 0.49, and Table 5 shows that 128 of 315 Indeterminate/Suspicious cases were misclassified as Malignant. The abstract and conclusions should be revised to state class-specific limitations and to characterize the system's performance as promising but with meaningful degradation on the indeterminate/suspicious category, rather than as uniformly high accuracy.","section":"Abstract; §3.3, Table 4"},{"comment":"There is a numerical inconsistency in the reported class distribution. Section 2.1 states '871 (Malignant)' and a total of 1,804 images, but Table 1 sums to 781 Malignant (541+125+115) and a total of 1,804, and Appendix C.4 also lists 781 Malignant. Because the class balance is central to interpreting the weighted loss and the reported metrics, this discrepancy must be corrected and verified.","section":"§2.1, Table 1, Appendix C.4"},{"comment":"The crop-label inheritance assumption is load-bearing for the training procedure. Each of the 1,804 images corresponds to a unique patient and a single 1024x768 field, and every crop in sets C, D, and E is assigned the full-image diagnostic label. If a crop contains only benign material from a malignant case, or if the single chosen field misses the diagnostic region, the training labels are noisy. The paper should either provide an analysis of crop-level label quality (e.g., agreement of model predictions on crops vs. full images) or explicitly frame this as a limitation that may weaken the clinical claim.","section":"§2.2.2, §2.1"}],"minor_comments":[{"comment":"The sentence 'Detailed comparison results ... in the analysis presented in Section 3.1result on test set' contains a typo ('3.1result') that should be corrected.","section":"§3.1"},{"comment":"The caption refers to 'the first [6] and second [29]' but the surrounding text mentions two research groups; please clarify whether [6] and [29] are both from the same U.S. group and label the figure entries consistently.","section":"Figure 2"},{"comment":"The heading 'Feature research' appears to be a typo for 'Future research'; please correct it.","section":"§5.1"},{"comment":"The text reports '1000 cases in 30 seconds' while the external set contains 1,015 images; please clarify whether 'cases' here means images or patients, and state the hardware configuration more precisely.","section":"§3.4, Table 5"},{"comment":"The hyperparameter search is described as conducted on the validation set, but the architecture selection in §3.1 is reported on the test set; aligning the two selection protocols would remove ambiguity about which set informed model choices.","section":"Appendix E.3"},{"comment":"No confidence intervals or uncertainty estimates are reported for the F1 scores or AUCs; reporting these would make the magnitude of the differences between models (e.g., 0.8919 vs. 0.8690) easier to assess.","section":"Results"}],"recommendation":"major_revision","confidential_remarks":"The paper is an empirical pipeline paper with genuine external validation, which is a plus. The main concern for the editor is whether the authors can correct the test-set-based architecture selection and provide an unbiased internal estimate; if they cannot, the internal macro F1 should be downgraded in the paper and the abstract's claims tempered. The class-count inconsistency (§2.1 vs. Table 1) is a basic data-integrity issue that should be fixed before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a legitimate engineering paper, not a breakthrough. The new part is the combination: YOLOv10-based augmentation, a curriculum-learning order that feeds crops before full images, a lightweight EfficientNetB0 backbone, and a Transformer-inspired module for multi-region scoring. The three-way clinical grouping (Benign, Indeterminate/Suspicious, Malignant) maps cleanly onto Bethesda management pathways, which is genuinely useful for low-resource labs. The external validation at Hung Viet Hospital is real and the authors deserve credit for reporting it and for discussing the performance drop honestly. The code is on GitHub, which also helps. Now the soft spots. The biggest one is the model selection procedure. Table 2 is explicitly labeled \"F1 Score on Test Set for Different Models and Configurations,\" and EfficientNetB0 is designated as ThyroidEffi Basic after achieving the highest augmented F1 in that same table. No validation-set comparison is reported for the twelve architectures. That means the internal macro F1 of 89.19% is likely the maximum of twelve test-set draws, not an unbiased estimate. The external set was not used for selection, so the external numbers are the more honest estimate: macro F1 0.68, Indeterminate/Suspicious F1 0.49, AUC 0.74. Those numbers do not support the abstract's \"high-accuracy\" phrasing. The limitation section acknowledges the difficulty, but the conclusion still overstates. The one-image-per-patient assumption is a real but secondary concern. A single 1024x768 FNAB field may miss the diagnostic region, and crops inheriting the full-image label can inject training noise. Since test and external evaluation use full images, this mainly affects training rather than the headline evaluation, but it still weakens the clinical claim if the field selection is not standardized. Minor issues: no confidence intervals, no human-baseline comparison, and no explicit statement of how the representative image was chosen during acquisition. The citation pattern looks fine; the paper engages with Dov, Wang, and other relevant work without self-citation loops. Who gets value? Anyone working on cytology image classification or on deploying small models in resource-constrained clinical settings. The external validation, even with its weak middle class, is worth discussing. My recommendation: don't desk reject. Send it to peer review, but the referee should require either a validation-set-based model selection or an explicit admission that the internal number is a selected maximum, and the abstract and conclusions need to be aligned with the external results. With those revisions, it could be a useful addition to the literature.","headline":"A cost-conscious and honestly deployed thyroid FNAB system with real external validation, but the internal headline metric is suspect because architecture selection used the test set, and the external numbers are weaker than the abstract admits.","tokens_in":726,"tokens_out":2088,"would_cite":true,"duration_ms":33429,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims a three-part pipeline — YOLOv10-guided augmentation, curriculum training from crops to full images, and EfficientNetB0 — classifies thyroid FNAB images at macro F1 89.19%, with external AUCs of 0.95, 0.74, and 0.84.","keywords":["thyroid carcinoma","fine-needle aspiration biopsy","medical image classification","multi-class classification","Bethesda system","cost-effective","EfficientNetB0","curriculum learning"],"falsifier":"Have a cytopathologist reread the external cases the model labelled Malignant but whose reference diagnosis is Indeterminate (128 of 315): if most of those images contain no truly malignant cells, the crop-label-inheritance strategy is injecting label noise and the reported scores partly measure that noise. A quantitative check is to retrain the Basic model with the curriculum order reversed while keeping everything else fixed — if internal-test macro F1 stays at 89.19%, the E→D→C→B→A ordering is not carrying the gain.","tokens_in":19894,"feed_emoji":"🔬","tokens_out":13213,"duration_ms":101182,"temperature":0.7,"pith_summary":"The paper sets out to show that accurate multi-class classification of thyroid fine-needle aspiration biopsy (FNAB) images does not require a heavy deep-learning stack. Its pipeline combines three ideas — detector-guided augmentation with YOLOv10, a curriculum training order that moves from local crops to full images, and the 4-million-parameter EfficientNetB0 backbone — and targets the three categories that actually steer treatment: Benign (Bethesda II), Indeterminate/Suspicious (Bethesda I, III, IV, V), and Malignant (Bethesda VI). On the internal test set the Basic model reaches macro F1 of 89.19%, the average of the three classes' F1 scores, with per-class AUCs of 0.98, 0.95, and 0.96, and on 1,015 independent images from a second hospital it reports AUCs of 0.9495, 0.7436, and 0.8396. If true, the practical payoff is a triage tool that runs 1,000 cases in about 30 seconds on a 12-core CPU.","feed_headline":"89% macro F1 from a 4-million-parameter thyroid biopsy classifier","feed_subtitle":"Detector-guided training puts multi-class FNAB triage on a 12-core CPU; external validation confirms the transfer.","key_machinery":"The central machinery is the four-module pipeline M1–M4, whose job is to give one 1024×768 FNAB field both local and global scale information while keeping inference cheap. M1 is the augmentation engine: a YOLOv10 detector (trained on 120 annotated training images, used only during training) locates cell clusters of at least ten cells and drives five augmentation sets — the original image (A), the original overlaid with the top-eight detected clusters (B), eight dense-cluster crops (C), and twenty-four 256×256 grid crops split into sets D and E — expanding the training data by a factor of 34. M2 is the curriculum: within every epoch the augmented views are fed in the order E, D, C, B, A, so the model first learns from small noisy patches and ends on full images, exploiting the recency bias of gradient updates. M3 is the model choice: EfficientNetB0 (about 4 million parameters), selected after comparing twelve models from six families on macro F1, parameter count, and inference time. M4 is the multi-region head: a Transformer-inspired module that partitions the image into a 3×4 grid, treats each patch as a token, and initializes the classification token with the backbone's full-image output, so local details are contextualized globally in a single pass.","core_discovery":"Stated the way a sympathetic reader would state it: a deliberately cost-efficient training recipe can produce a clinically usable three-way thyroid FNAB classifier. The recipe is (1) train a YOLOv10 detector on 120 annotated training images to find dense cell clusters, then use its bounding boxes to build a ×34 augmentation — original image, an overlay of the top-eight clusters, eight high-density crops, and 24 grid crops; (2) present those views inside each epoch in the order E→D→C→B→A, from noisy local patches to the full image; (3) use EfficientNetB0 as the backbone; and (4) for the Premium variant, add a Transformer-inspired head that treats a 3×4 grid of patches as tokens and initializes the class token with the full-image encoding. The reported outcome is macro F1 of 89.19% (Basic) and 89.77% (Premium) on the internal test set, per-class AUCs of 0.98/0.95/0.96, and external AUCs of 0.9495/0.7436/0.8396 on 1,015 images collected prospectively at a second site. The authors' conclusion is that high-accuracy, interpretable thyroid FNAB classification is achievable with minimal computational demands.","pith_inferences":["A testable extension the paper leaves implicit: since the external errors concentrate on Indeterminate cases being called Malignant, feeding several FNAB fields per patient and aggregating their probabilities by average would likely lift the 0.7436 external AUC without any retraining.","Because every crop inherits the full-image label, sets C, D, and E carry deliberately noisy truth; if per-crop cytopathologist labels existed, the same pipeline would probably gain more from them than from any further architecture change — a comparison the paper does not run.","The curriculum's reliance on recency effects in gradient updates implies the E→D→C→B→A order should matter more on small datasets; the same recipe therefore ought to transfer to other scarce-data cytology tasks such as breast FNA, which the paper's own roadmap anticipates."],"forward_implications":["A triage-grade FNAB classifier can run on hardware already present in modest clinics: about 1,000 cases in 30 seconds on a 12-core CPU, with only the raw image as input at inference and no manual region extraction or ensemble.","In the paper's own comparison, the augmentation and curriculum protocol raises macro F1 for eleven of the twelve backbones tested (EfficientNetB0 from 0.8555 to 0.8919), so most of the gain sits in the training recipe rather than in any single architecture.","The three-way clinical mapping — Benign (Bethesda II), Indeterminate/Suspicious (Bethesda I, III, IV, V), Malignant (Bethesda VI) — aligns the classifier's output directly with the treatment decision, which is what lets the system act as a second reader rather than a research tool.","The accuracy claim is conditional on class and site: the external AUC holds for Benign (0.9495) and Malignant (0.8396) but drops to 0.7436 for the Indeterminate/Suspicious group, with external macro F1 at 0.68, so the deployable promise is strongest for the confident classes."],"supporting_citations":[{"why":"Supplies the YOLOv10 detector whose cell-cluster boxes drive the ×34 augmentation sets B and C.","marker":"[30]"},{"why":"Supplies the EfficientNet family whose compound-scaled EfficientNetB0 backbone anchors the whole pipeline.","marker":"[35]"},{"why":"Supplies the token/class-token design that the M4 multi-region module adapts for the Premium variant.","marker":"[36]"},{"why":"Supplies the Transformer encoder layers that refine the class token in the Premium model.","marker":"[37]"},{"why":"The main multiclass comparator (ThyroPower) whose reported per-category performance the paper argues against.","marker":"[15]"},{"why":"One of the two published multiclass WSI efforts (with [29]) that frame the comparison in Figure 2.","marker":"[6]"},{"why":"Supplies Grad-CAM, the visualization that supports the interpretability half of the central claim.","marker":"[40]"}],"fun_headline_variants":["Cost-effective thyroid FNAB AI: 89% macro F1 with 4M params","ThyroidEffi: cheap model, 89% macro F1, externally validated","Detector-guided curriculum learning yields 89% macro F1 on thyroid FNAB","Tiny EfficientNetB0 hits 89% macro F1 on thyroid FNAB","4M params, 89% macro F1: thyroid FNAB triage without the GPU"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that one 1024×768 FNAB field per patient, captured at 40× with Diff-Quick staining, is representative enough to label the patient's cytology, and that every cropped sub-region generated from that image can inherit the full-image diagnostic label as training truth.","fun_headline_variants_meta":{"raw":{"variants":["Cost-effective thyroid FNAB AI: 89% macro F1 with 4M params","ThyroidEffi: cheap model, 89% macro F1, externally validated","Detector-guided curriculum learning yields 89% macro F1 on thyroid FNAB","Tiny EfficientNetB0 hits 89% macro F1 on thyroid FNAB","4M params, 89% macro F1: thyroid FNAB triage without the GPU"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001576,"raw_usage":{"total_tokens":6407,"prompt_tokens":1178,"completion_tokens":5229,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":794,"completion_tokens_details":{"reasoning_tokens":5112}},"tokens_in":794,"tokens_out":5229,"duration_ms":36215,"temperature":1.0,"reasoning_tokens":5112,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:55:46.307629+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have a cytopathologist reread the external cases the model labelled Malignant but whose reference diagnosis is Indeterminate (128 of 315): if most of those images contain no truly malignant cells, the crop-label-inheritance strategy is injecting label noise and the reported scores partly measure that noise. A quantitative check is to retrain the Basic model with the curriculum order reversed while keeping everything else fixed — if internal-test macro F1 stays at 89.19%, the E→D→C→B→A ordering is not carrying the gain.","supporting_citations":[{"cited_title":"Deep learning models for thyroid nodules diagnosis of fine-needle aspiration biopsy: a retrospective, prospective, multicentre study in china","cited_arxiv_id":null,"evidence_quote":"The main multiclass comparator (ThyroPower) whose reported per-category performance the paper argues against."},{"cited_title":"Thy- roid cancer malignancy prediction from whole slide cytopathology images","cited_arxiv_id":null,"evidence_quote":"One of the two published multiclass WSI efforts (with [29]) that frame the comparison in Figure 2."},{"cited_title":"Grad-cam: Visual explanations from deep networks via gradient-based localization","cited_arxiv_id":null,"evidence_quote":"Supplies Grad-CAM, the visualization that supports the interpretability half of the central claim."}],"review_version":1}