{"id":"c2aae401-3bb6-467e-a951-556d63f9b1d9","arxiv_id":"2507.23648","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Confidence-based replay, a rehearsal continual learning strategy, improves cross-site accuracy and sensitivity of a YOLO malaria detector compared to single-site training.","lead":"This paper tests whether continual learning can keep a YOLO-based malaria detector accurate when it is deployed across five different clinical imaging sites. Rehearsal-based strategies, especially a confidence-based replay, improved generalization over a single-site baseline and in some metrics matched or beat joint training on all past data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing a sequential-fine-tuning (no-CL) control: reported gains could be due to training on new sites, not to continual learning.","rationale":"The reader's CONDITIONAL verdict is reasonable, and the concerns about fixed task order, small T5 test sets, and missing significance tests are valid. However, I see a more fundamental attribution problem: the study never compares against a model that is sequentially fine-tuned on each new site without any continual learning mechanism. The Baseline is frozen after D1, so all four CL methods benefit from adaptation to D2-D5; the large sensitivity gains are therefore expected and do not establish that rehearsal or regularization is responsible. This missing control is standard in continual learning evaluations and would directly test whether the CL machinery contributes beyond \"training on more data.\" The reader's task-order concern is related but secondary: it bears on generalizability of the observed ordering, whereas the missing control bears on whether the central claim is identified at all. I also note the paper otherwise has strengths: real multi-site clinical data, patient-level splitting to prevent leakage, 3-fold cross-validation, and a sensible two-detector YOLO design. My recommendation is to retain the CONDITIONAL verdict but make the added no-CL control and statistical reporting explicit conditions for acceptance.","tokens_in":7741,"tokens_out":7699,"duration_ms":80057,"concrete_test":"Add a Sequential Fine-Tuning control to the same 3-fold patient-split protocol: train YOLOv5s on D1, then for t=2,...,5 continue training on D_t only (same 50 epochs, early stopping, augmentation, and hyperparameters as the CL methods, but no memory buffer, EWC, or LWF). Compute the Table 2 metrics, BWT, and FWT for the final model on all five site test sets. If this no-CL control achieves average accuracy/sensitivity/specificity within the fold-std of Replay conf, or a non-negative BWT, then the reported gains cannot be attributed to continual learning and the central claim must be weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that \"CL, and rehearsal-based methods in particular, can significantly improve performance\" (Abstract), yet the only non-CL reference points are the frozen Baseline, trained on D1 only, and Incremental Joint Training, which is an upper bound rather than a no-CL control. No model is trained that simply fine-tunes on each new site without a memory buffer or regularization. Since every evaluated CL method is also trained on D2-D5, the improvements over Baseline (e.g., RBC sensitivity rising from 57.24% for Baseline to 72.02-84.32% for all adapting methods, Table 2a) could reflect only the benefit of seeing additional data, not the CL mechanism. Without a \"Sequential Fine-Tuning\" control, the paper cannot attribute the reported generalization gains to continual learning. The confidence-replay vs random-replay comparison partially avoids this issue, but the differences are small (RBC accuracy 98.53 vs 98.38, BWT 0.018 vs -0.006) and no significance tests or confidence intervals are reported; with only 3 folds and T5 containing 6 positive test images, these differences are within noise. Thus the paper's strongest conclusion is not yet identified: the missing control is load-bearing, and the statistical evidence compounds the problem.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes using continual learning (CL) for domain-incremental malaria diagnosis from thin blood smears, where each task is a different clinical site. A YOLOv5 detector is trained on five site datasets in sequence under four CL strategies: EWC, LwF, naive replay, and confidence-based replay. The authors compare these against a frozen baseline trained only on the first site and against incremental joint training, reporting RBC-level and image-level accuracy, sensitivity, specificity, plus backward/forward transfer. They conclude that CL, and rehearsal-based methods in particular, can significantly improve generalization, with confidence replay giving the best overall balance and even exceeding the joint-training upper bound at image level.","tokens_in":7982,"tokens_out":2879,"duration_ms":33333,"significance":"If the central claim holds, the paper would make a useful practical contribution: it would show that a malaria CAD system can be adapted to new sites without full joint retraining, using a much smaller memory footprint, and that confidence-based sample selection can outperform random replay. The study has real strengths: patient-level data splits, three-fold cross-validation, a multi-site clinical dataset with heterogeneous image quality and class imbalance, and evaluation at both RBC and image levels with BWT/FWT metrics. The rehearsal buffer design and the comparison of four CL methods on a clinically relevant object-detection task are also valuable. However, the causal attribution to continual learning is currently under-supported, because the only non-CL comparators are a never-adapted baseline and an upper-bound joint-training model; a sequential fine-tuning control is missing. In addition, the word 'significantly' is used without any statistical test or confidence interval. These issues are load-bearing for the paper's main conclusion, so the significance is conditional on additional experiments and analysis.","major_comments":[{"comment":"The experimental design lacks a sequential fine-tuning (no-CL) control. Baseline is trained only on D1 and never sees D2-D5, while every CL method is also trained on those later tasks. The large RBC-level sensitivity gain of all adapting methods over Baseline (e.g., 57.24 for Baseline vs 72.02-84.32 for CL methods) could therefore be due simply to exposure to more data, not to any continual-learning mechanism. Adding a control that fine-tunes on each new site in sequence without replay or regularization is necessary to support the abstract's claim that 'CL, and rehearsal-based methods in particular, can significantly improve performance.' Without this control, the reported improvements cannot be attributed to continual learning.","section":"Section 3, 'Reference models' and Section 4, Table 2"},{"comment":"The word 'significantly' is used repeatedly (abstract, Section 4, and the YOLOv5 vs v8/v11 comparison) but no significance tests, confidence intervals, or paired comparisons are reported. The key comparison between Replay conf and Replay naive shows RBC accuracy 98.53 vs 98.38 and image accuracy 81.95 vs 74.50, but with only three folds and, for T5, only 6 positive test images, these differences may be within noise. I recommend reporting paired tests across folds, or better, patient-level bootstrapped confidence intervals, for the main metrics and for the BWT/FWT values.","section":"Abstract and Section 4, Table 2"},{"comment":"The authors list task-order sensitivity as future work, but the fixed order D1 to D5 and the very small T5 set (16 positive training images, 6 positive test images) are load-bearing for the central claim. In domain-incremental CL, results can depend strongly on task order and on the composition of small sites. I recommend at least one additional experiment with a reversed or shuffled task order, or a clear statement that the claimed field-readiness is conditional on the observed order and site composition. If the advantage of Replay conf over Replay naive and over joint training disappears under order permutation, the main conclusion would need to be substantially weakened.","section":"Section 5, first paragraph"}],"minor_comments":[{"comment":"There are several typos that should be corrected: 'choosed' should be 'chosen', and the datasets paragraph contains 'Plamosdium' instead of 'Plasmodium'.","section":"Section 3, 'Experimental Design'"},{"comment":"For D5, only 6 positive test images are available; this should be explicitly acknowledged in the results discussion, and the reported standard deviations for any metric involving T5 should be interpreted with this in mind.","section":"Section 3, 'Datasets' and Table 1"},{"comment":"The notation is inconsistent: Figure 2 says each value represents P1,i, while Figure 3 says Pi,i except for baseline which is P1,i. Clarify the indices so that the reader can tell which performance matrix entry is plotted for each evolution curve.","section":"Figure 2 and Figure 3 captions"},{"comment":"The BWT/FWT values are computed on accuracy, but the clinical emphasis is on sensitivity and specificity; reporting BWT/FWT for sensitivity as well (or at least discussing it) would better match the paper's stated clinical priorities.","section":"Section 4, Table 2"},{"comment":"The claim that confidence-based selection outperforms the '200 random samples' approach from prior work [5] is not directly tested in the experiments; either add a comparison with that specific setting or phrase the sentence as a suggestion rather than a demonstrated result.","section":"Section 5, first paragraph"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is the first time anyone has applied continual learning to malaria CAD, and the study design has real strengths—patient-level splits, 3-fold CV, and a five-site clinical dataset that includes routine field data. The confidence-replay variant for YOLO detection is a small but genuine heuristic twist. If the results hold, rehearsal-based CL offers a practical alternative to joint retraining for field deployment. That's worth knowing.\n\nWhat it does well: the task setup is sensible (domain-incremental, five sites), the evaluation is more thorough than most—RBC and image-level metrics, BWT/FWT, and learning curves. The observation that regularization methods (EWC, LWF) lag behind rehearsal methods is a useful data point. The authors are also honest about limitations: they list task-order sensitivity and data size as future work.\n\nWhere it softens: the stress-test note is right. There is no sequential fine-tuning control—a model that simply continues training on D2-D5 with no buffer or regularization. Without it, the abstract's claim that CL 'significantly improves' performance cannot be attributed to CL. Every adapting method sees the same new data; their gains over Baseline could just be the benefit of more training data. The confidence-replay vs random-replay comparison partially addresses this, but the differences are tiny (RBC accuracy 98.53 vs 98.38, image accuracy 81.95 vs 74.50 with overlapping variability on small test sets) and no significance tests or CIs are reported. The word 'significantly' appears in the abstract and text without statistical support. T5 has 6 positive test images; 3-fold CV gives noisy estimates. The fixed task order and the chosen hyperparameters (e.g., EWC lambda=10) are not tested for sensitivity, and EWC's RBC sensitivity of 24% suggests possible implementation issues. No code or data released, which makes the numbers hard to verify.\n\nBottom line: the hypothesis is plausible and the paper is a reasonable early step. As a peer-reviewed publication, it needs the missing control, significance testing, and ideally artifact release. The current version over-states what is established.\n\nRecommendation: I would send it to peer review, but with a clear message that the central claim needs the sequential fine-tuning comparator. It's a legitimate empirical contribution for the medical imaging and CL communities, even if my own verdict is conditional.","headline":"First application of CL to malaria CAD, with a clean multi-site setup, but the missing no-CL sequential fine-tuning control leaves the central claim under-supported.","tokens_in":8599,"tokens_out":2028,"would_cite":true,"duration_ms":20249,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A continual learning approach using confidence-based replay lets a YOLO malaria detector adapt to new clinical sites without forgetting earlier ones, improving cross-site generalization.","keywords":["malaria diagnosis","continual learning","object detection","YOLO","thin blood smear","domain shift","rehearsal","confidence replay"],"falsifier":"Run the same confidence-replay and random-replay training on the same five datasets but with the site order permuted (for example, presenting the small fifth site first); if confidence replay stops beating random replay or the incremental joint-training upper bound in average accuracy, the central claim about its superiority would be refuted.","tokens_in":7546,"feed_emoji":"🔬","tokens_out":6253,"duration_ms":58018,"temperature":0.7,"pith_summary":"Malaria diagnosis from thin blood smear images works well within one clinic but degrades across sites because staining, equipment, and operators shift. This paper asks whether continual learning can make a YOLO-based detector robust to those shifts without retraining on all data at once. It sets up a domain-incremental stream of five real clinical sites and compares four continual learning methods against a no-adaptation baseline and an incremental joint-training upper bound. The authors find that rehearsal-based methods, especially replayed images selected by low YOLO confidence, significantly improve cross-site accuracy and sensitivity, reaching 98.53% red-blood-cell-level accuracy and the best backward transfer. The practical consequence is that a diagnostic tool could be updated site by site, in the field, rather than requiring a large joint retraining effort.","feed_headline":"Replaying low-confidence images lifts malaria AI to 98.5%","feed_subtitle":"Site-by-site training with confidence-based replay beats joint retraining and curbs forgetting in field microscopy.","key_machinery":"The load-bearing mechanism is a domain-incremental continual learning loop wrapped around a dual-YOLO detector: one YOLOv5 model detects all red blood cells and a second detects infected ones, and their detections are merged to label an image positive if any RBC is infected. At each new-site task, the model is trained on the new data plus a memory buffer of replayed images from previous sites; confidence replay fills that buffer with up to 125 images per site whose YOLO detections had the lowest confidence, motivated by the idea that uncertain examples carry the most information for retaining old domains. Performance is tracked with a task-to-task performance matrix and average performance, backward transfer, and forward transfer metrics. The confidence-based replay is what the paper credits for stable accuracy, high specificity, and positive backward transfer across the five sites.","core_discovery":"The paper's central claim is that applying continual learning, and in particular rehearsal with confidence-based sample selection, makes a YOLO-based malaria detector generalize across acquisition sites without catastrophic forgetting. In a fixed task stream of five sites (one public dataset and four routine clinical collections), the model trained sequentially with confidence replay achieves the highest mean RBC-level accuracy (98.53%) and specificity (99.26%), the best backward transfer (0.018, i.e., accuracy on earlier tasks improves rather than degrades), and the best image-level accuracy (81.95%), even surpassing the incremental joint-training upper bound at image level. Regularization-based methods (EWC and LWF) perform markedly worse, and a baseline with no adaptation loses sensitivity sharply from task 1 to task 2. The authors interpret this as evidence that continual learning is a practical route to field-ready malaria CAD.","pith_inferences":["Because the paper fixes the task order D1 to D5 and does not test permutations, the reported advantage of confidence replay may be order-dependent; a deployment would need to choose and test the order in which sites are onboarded.","Selecting replay samples by low-confidence detections is a general principle that could transfer to other detection-based diagnostics (e.g., tuberculosis or schistosomiasis microscopy), not just malaria; that would be a natural external test.","The smallest site has only 16 positive training and 6 positive test images, so image-level sensitivity on that site is statistically fragile; a validation set with more positive images could change which method looks best at image level.","If confidence replay holds up, it reduces the data-sharing burden in clinical partnerships: only a small buffer of low-confidence images, rather than full datasets, needs to be retained across sites."],"forward_implications":["A field-ready malaria CAD tool could be deployed at a new hospital and updated with that site's routine images alone, without gathering all previous sites' data for joint retraining.","Rehearsal methods keep earlier-site performance intact: confidence replay's positive backward transfer (0.018) means the model gets slightly better on old tasks after learning new ones, not worse.","Confidence replay reaches 81.95% average image-level accuracy, above the 78.51% of incremental joint training, suggesting that a curated small buffer can beat storing full data.","Regularization-based continual learning (EWC, LWF) underperforms replay on this data, so memory-based approaches are the more promising family for malaria CAD.","The no-adaptation baseline's sensitivity drops from near 80% to below 40% at RBC level on the second site, confirming that cross-site generalization is the binding constraint."],"supporting_citations":[{"why":"Supplies the dual-YOLO detection pipeline, the baseline showing >20% image-level accuracy drops across sites, and the previous finding that 200 random new-site samples improve generalization, which this work extends with confidence-based selection.","marker":"[5]"},{"why":"Provides the public thin-blood-smear dataset used as D1, anchoring the multi-site task stream in a known benchmark.","marker":"[9]"},{"why":"Defines the YOLOv5 model that all continual learning strategies are trained on, after v8 and v11 showed poor generalization.","marker":"[7]"},{"why":"Supplies the EWC regularization baseline, which is one of the two non-rehearsal continual learning methods compared.","marker":"[10]"},{"why":"Supplies the Learning Without Forgetting baseline, the second regularization method compared.","marker":"[14]"},{"why":"Provides the confidence-based sample selection idea that confidence replay adapts for YOLO detections.","marker":"[21]"},{"why":"Supplies the backward transfer and forward transfer metrics used to measure forgetting and generalization across tasks.","marker":"[16]"},{"why":"Frames the domain-incremental continual learning scenario and the performance-matrix evaluation approach used in the paper.","marker":"[13]"}],"fun_headline_variants":["Confidence replay boosts malaria AI to 98.5% accuracy","Continual learning nails field malaria detection at 98.5%","Confidence-based replay curbs forgetting in malaria AI","Rehearsal beats joint training for malaria CAD","Malaria AI adapts across sites with confidence replay"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results assume that the fixed order of the five sites and their very unequal sizes—especially the final site with 16 positive training images and 6 positive test images—are representative of how sites arrive in real deployment, since the paper does not test what happens when that order or composition changes.","fun_headline_variants_meta":{"raw":{"variants":["Confidence replay boosts malaria AI to 98.5% accuracy","Continual learning nails field malaria detection at 98.5%","Confidence-based replay curbs forgetting in malaria AI","Rehearsal beats joint training for malaria CAD","Malaria AI adapts across sites with confidence replay"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000516,"raw_usage":{"total_tokens":2480,"prompt_tokens":897,"completion_tokens":1583,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":513,"completion_tokens_details":{"reasoning_tokens":1501}},"tokens_in":513,"tokens_out":1583,"duration_ms":11179,"temperature":1.0,"reasoning_tokens":1501,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T10:29:28.838655+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same confidence-replay and random-replay training on the same five datasets but with the site order permuted (for example, presenting the small fifth site first); if confidence replay stops beating random replay or the incremental joint-training upper bound in average accuracy, the central claim about its superiority would be refuted.","supporting_citations":[{"cited_title":"In: Domain Adaptation and Representation Transfer","cited_arxiv_id":null,"evidence_quote":"Provides the confidence-based sample selection idea that confidence replay adapts for YOLO detections."},{"cited_title":"In: Advances in Neural Information Processing Systems","cited_arxiv_id":null,"evidence_quote":"Supplies the backward transfer and forward transfer metrics used to measure forgetting and generalization across tasks."},{"cited_title":"Continual Learning in Medical Image Analysis: A Comprehensive Review of Recent Advancements and Future Prospects","cited_arxiv_id":"2312.17004","evidence_quote":"Frames the domain-incremental continual learning scenario and the performance-matrix evaluation approach used in the paper."}],"review_version":1}