{"id":"fa2c1458-8431-45e4-a791-5f1bbc1fcbf4","arxiv_id":"1908.05489","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A heterogeneous ensemble of fine-tuned convolutional networks, selected with leave-one-dataset-out rules, outperforms individual CNNs on three plankton and two coral classification benchmarks.","lead":"This paper tests whether combining several pre-trained deep learning models improves automated classification of plankton and coral images. It reports small accuracy gains over single models on five marine image datasets and compares the combined system with earlier methods.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 6's SFFS(11) numbers may come from an ensemble selected on all five datasets, then evaluated on the same datasets; the paper's best result needs a held-out selection/evaluation check.","rationale":"The reader's CONDITIONAL verdict is appropriate. I agree that the baseline protocol mismatch weakens the state-of-the-art comparison, but the most load-bearing issue is the all-data SFFS selection. The reader noted this in the rationale but did not make it the weakest assumption, hence partial agreement. If the concern lands, the paper's best result is not an unbiased estimate; however, the simple Fus_2R+Fus_1R ensemble still improves over single DenseNet and most baselines in the tables, so a full rejection is not warranted. A conditional accept requiring a clean held-out SFFS evaluation and clarification of Table 6's protocol seems right. The lack of released code also prevents independent verification, but that is secondary to the internal ambiguity.","tokens_in":15370,"tokens_out":4200,"duration_ms":38599,"concrete_test":"Re-run SFFS selection exactly as in Fig. 2: for each target dataset, select the ensemble on the other four datasets (or on the training split only) and evaluate on the held-out target test split; report the five F-measures. Then compare to Table 6's SFFS(11). If Table 6's SFFS(11) row was produced by an all-data selection, recompute it under the leave-one-out protocol; if the honest SFFS(11) values drop below Fus_2R+Fus_1R (0.955/0.899/0.926/0.989/0.994), the headline 'lighter ensemble outperforms all other ensembles' fails. If the honest values remain above, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim needs the SFFS(11) row of Table 6 (0.958/0.900/0.927/0.990/0.995) to be a valid estimate of generalization. The paper is internally ambiguous at exactly this point. Section 2 and Fig. 2 describe SFFS selection under a leave-one-out-dataset protocol, which would be honest. But Section 3 then says: 'Finally, we ran SFFS using all the five datasets. In this way we find an ensemble of 11 CNNs...' If the 11 models listed there are the ones scored in Table 6, and if 'all the five datasets' means the same splits used for evaluation, then the ensemble members were chosen using test labels. The reported F-measures are then selection results, not predictions on unseen data. This matters because SFFS(11) is the paper's headline best result and the basis for the 'substantial performance improvement' conclusion. The protocol-mixing in Tables 7 and 8 (OPT with 75/25, EnsHC with 10-fold) is a separate weakness, but the SFFS issue is more load-bearing because it affects the paper's own best numbers, not only the comparison with external baselines.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies ensembles of ImageNet-pretrained CNN architectures, fine-tuned on five underwater image datasets (WHOI, ZooScan, Kaggle, EILAT, RSMAS), for plankton and coral classification. It compares single architectures, ensemble rules (Fus_*, DN_*), transfer learning to SVMs, and classifier-selection strategies SFFS and WS. The headline result is that a heterogeneous ensemble reduced by SFFS to 11 CNNs achieves F-measures of 0.958, 0.900, 0.927, 0.990, and 0.995 on the five datasets, outperforming the best single model (DenseNet) and previously published state-of-the-art methods. The paper also reports memory costs for the reduced ensembles and claims that the same pipeline works 'out-of-the-box' across datasets without per-dataset tuning.","tokens_in":15609,"tokens_out":2673,"duration_ms":26286,"significance":"If the reported results are valid estimates of generalization, the paper is a useful empirical contribution: it provides a broad comparison of fine-tuned CNN architectures and shows that ensembles of diverse architectures consistently improve over single models in three plankton and two coral classification tasks. The release of MATLAB source code for reproduction is a positive feature. The main value is practical, giving practitioners a concrete recipe (diverse architectures, incremental training, SFFS selection) for improving accuracy in similar small-to-medium marine image classification problems. However, the significance of the central 'substantial improvement' claim depends critically on the validity of the SFFS evaluation protocol and on the comparability of the state-of-the-art baselines, both of which are open concerns in the current manuscript.","major_comments":[{"comment":"The headline result SFFS(11) appears to be produced by the sentence 'Finally, we ran SFFS using all the five datasets. In this way we find an ensemble of 11 CNNs...' This is in direct tension with the leave-one-out-dataset protocol described for SFFS in Section 2 (Fig. 2). If the same five datasets were used both for selecting the 11-member ensemble and for evaluating the F-measures in Table 6, then the Table 6 numbers are selection results on training data, not unbiased predictions on unseen test data. Because SFFS(11) is the basis of the paper's 'substantial performance improvement' conclusion, the authors must clarify whether the 11 models were selected on held-out data, or add a truly held-out evaluation of the final ensemble (e.g., nested cross-validation or a separate split).","section":"§3, Table 6 and final paragraph of Section 3"},{"comment":"","section":"§3, Tables 7 and 8"},{"comment":"All results are reported as point estimates with no error bars, standard deviations, or repeated-run information. Since the key claimed improvements (e.g., SFFS(11) versus Fus_2R+Fus_1R, or versus the best single model) are often on the order of 0.001–0.005, the reader cannot determine whether these differences are statistically meaningful. Reporting means and standard deviations across the folds (or across multiple training runs) is necessary to support the 'substantial improvement' claim.","section":"Tables 1–6"}],"minor_comments":[{"comment":"In the backtracking step of the SFFS pseudo-code, the line 'RS:=RS+smax' appears to be a typo; after removing smin from SS, the algorithm should add smin back to RS, i.e., 'RS:=RS+smin'.","section":"§2, Fig. 2 pseudo-code"},{"comment":"The model list in Section 2 refers to 'DenseNet', but the final SFFS list includes 'densenet201_1R_Pad/Tile' and 'densenet201_2R'. Please clarify whether DenseNet in all experiments is DenseNet-201, and keep the naming consistent across tables and the final ensemble list.","section":"§2–§3, model naming"},{"comment":"The y-axis of Fig. 4 starts at 0.86, which visually exaggerates the differences between methods; consider starting at 0.80 or showing change relative to a baseline.","section":"§3, Fig. 4"},{"comment":"The formula for multi-class F-measure and accuracy is written with a per-class index c, but the averaging notation is ambiguous. Please define P_C and R_C explicitly and state whether the reported F-measure is macro-averaged (as it appears to be).","section":"§3, performance indicators"},{"comment":"The text says 'the class distribution has not been maintained when splitting the dataset' but then describes standard protocols (e.g., '2-fold cross validation', '5-fold cross validation'). Please state whether the folds were stratified or not, since this affects reproducibility.","section":"§3, Dataset split description"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The paper's practical contribution is solid in its breadth, but the validity of the headline SFFS(11) result needs to be established by a clear selection/evaluation protocol. The state-of-the-art comparison should also be made protocol-consistent. If the authors can clarify or fix the SFFS evaluation, the paper could be acceptable; as written, the central claim is not yet supported. I have not raised the fact that several baselines are from the same group as a separate issue because it is a comparison, not a premise, but the protocol inconsistency already weakens those comparisons."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a useful benchmark paper, not a breakthrough. The main empirical result—heterogeneous ensembles of fine-tuned CNNs beat the best single model on all five datasets—holds up in their own tables. The 'substantial' and 'state-of-the-art' wording oversells it, and one protocol ambiguity needs to be cleared up before the headline numbers can be trusted.\n\nWhat is actually new is the breadth: ten architectures, three fine-tuning protocols, and several ensemble selection rules tested on three plankton and two coral datasets. The finding that SFFS can shrink the ensemble from 27 classifiers to 11 while slightly improving F-measure is practically useful. The comparison of one-round, two-round, and incremental fine-tuning is also worth having, even though the differences are modest. The experiments are described clearly, the datasets are public, and the ZooScan confusion matrices give a nice sense of where the gains come from.\n\nSoft spots. First, the improvements over the next-best ensemble are tiny: SFFS(11) beats Fus_2R+Fus_1R by 0.001–0.003 F-measure. No error bars or significance tests, so those numbers are probably noise. The gains over the best single model are more solid, but 'substantial' is an overstatement. Second, the state-of-the-art comparison is not apples-to-apples: OPT uses a 75/25 split and EnsHC uses 10-fold CV, so superior F-measures against them don't mean much. Third, and most important, the SFFS protocol is ambiguous. Section 2 describes SFFS with leave-one-out-dataset selection, which is honest. But the end of Section 3 says 'we ran SFFS using all the five datasets' and lists 11 models. If those models are the ones scored in Table 6, then the ensemble was selected on the test sets and the reported F-measures are selection results, not generalization estimates. If Table 6 used the leave-one-out protocol and the all-data run is just the final deployment ensemble, the paper needs to say that explicitly. As written, it is genuinely unclear. Finally, the code link is promised but absent.\n\nWho is this for? Practitioners building automated plankton or coral monitoring systems who want a known-good recipe for ensembling off-the-shelf CNNs. It is not conceptually deep, but it is a broad empirical reference. I would not cite it for a new method; I might cite it as evidence on ensemble behavior for marine imagery if I needed a battery of results.\n\nRecommendation: send it to peer review, not desk reject. The empirical core is useful, and the ambiguity is fixable. Require the authors to clarify the SFFS selection/evaluation protocol, add error bars or soften the claims, and make the code actually available.","headline":"Useful empirical benchmark on plankton/coral classification, but the 'state-of-the-art' claim is oversold and the SFFS protocol needs clarification before the headline numbers can be trusted.","tokens_in":16135,"tokens_out":4135,"would_cite":false,"duration_ms":37161,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fusing a diverse set of fine-tuned convolutional neural networks into a score-level ensemble improves plankton and coral classification accuracy over every single model and over published baselines on five benchmark datasets.","keywords":["Convolutional Neural Network","Fine-tuning","Plankton Classification","Coral Classification","Ensemble learning","Transfer learning","Underwater image analysis","SFFS classifier selection"],"falsifier":"Re-run the baselines named in Tables 7 and 8 (for example FUS_Hand, Gaussian SVM, MKL, DeepL, and EnsHC) on the exact same dataset splits used for the proposed ensembles, with the same preprocessing. If any baseline reaches or exceeds the SFFS(11) F-measures or accuracies under identical folds, the paper's central claim of state-of-the-art performance fails.","tokens_in":15157,"feed_emoji":"🌊","tokens_out":9211,"duration_ms":76643,"temperature":0.7,"pith_summary":"This paper claims that a heterogeneous ensemble of convolutional neural networks (CNNs), combined by score-level fusion, reliably beats both the best single network and published baselines on plankton and coral classification. The authors fine-tune ten well-known CNN architectures from ImageNet weights with deliberate variations—different resizing schemes, one-round versus two-round training, incremental epoch checkpoints, and SELU activations—and test on five public datasets: WHOI, ZooScan, and Kaggle for plankton, EILAT and RSMAS for coral. Using a classifier-selection rule called SFFS, they reduce the full 27-network ensemble to 11 networks and still improve on every dataset, reaching F-measures of 0.958, 0.900, 0.927, 0.990, and 0.995 on the five benchmarks. The practical motivation is that marine experts can manually inspect only a small fraction of collected imagery, so accurate automatic classification is a bottleneck for ecosystem monitoring.","feed_headline":"Ensemble of 11 CNNs beats all prior plankton and coral classifiers","feed_subtitle":"Five public datasets, one fixed recipe: pretrained CNNs plus selected score fusion beat single models and baselines.","key_machinery":"The mechanism carrying the argument is a score-level ensemble of fine-tuned CNN models selected by Sequential Forward Floating Selection (SFFS), a greedy selection procedure adapted to classifiers: it repeatedly adds the model that improves the ensemble most and then drops the weakest member. The members are standard pretrained architectures—AlexNet, GoogleNet, InceptionV3, VGG-16/19, ResNet50/101, DenseNet, MobileNetV2, and NasNet—fine-tuned with variations that create useful disagreement, including different resizing strategies, one-round versus two-round tuning, incremental epoch snapshots, and SELU instead of ReLU activations. SFFS concentrates that diversity into a compact subset of 11 or 3 models, which is what lets the paper claim both higher accuracy and lower memory than the full fusion.","core_discovery":"On its own terms, the central discovery is that the diversity produced by small training variations is a resource worth harvesting through ensembling. Every architecture is fine-tuned from ImageNet weights under one-round, two-round, incremental, or SELU versions, and the score-level fusion of these models outperforms every single model on all five datasets. DenseNet is the best single model, with F-measures of 0.949, 0.878, 0.912, 0.969, and 0.979 on WHOI, ZooScan, Kaggle, EILAT, and RSMAS, while the selected SFFS(11) ensemble reaches 0.958, 0.900, 0.927, 0.990, and 0.995. The paper also shows that the selection step can shrink the ensemble from 27 to 11 networks, or even to 3, with memory dropping from about 5.5 GB to around 2 GB, and that the same fixed ensemble beats published handcrafted-descriptor and deep-learning results on all five benchmarks.","pith_inferences":["The paper's own tables flag that two baselines used different protocols—OPT with a 75/25 split and EnsHC with 10-fold cross-validation—so a strict head-to-head rerun under identical folds is needed before the state-of-the-art margin can be taken at face value.","Because the largest gains occur on the harder, class-imbalanced datasets, the same cheap-diversity ensemble recipe is a plausible starting point for other fine-grained marine vision tasks, such as fish species or benthic invertebrate classification.","The reported memory savings count stored weights; running all 11 networks still requires activations and forward passes, so deployment on small underwater vehicles would likely need additional compression or distillation."],"forward_implications":["A fixed recipe of pretrained CNNs, small training variations, and score fusion can reach reported state-of-the-art results on multiple plankton and coral datasets without dataset-specific preprocessing, augmentation, or architecture search.","A selected ensemble of 11 models replaces the full 27-model fusion with a much smaller memory footprint while still improving F-measure on all five datasets.","Even a single architecture such as DenseNet can form a strong ensemble from its one-round, two-round, and resizing variants, so complete architectural diversity is not strictly required.","Training SVMs on CNN-extracted features adds only negligible gains over the CNN's own scores, so the ensemble's benefit comes from combining whole models rather than from changing the final classifier."],"supporting_citations":[{"why":"Supplies the ZooScan and Kaggle plankton datasets with their evaluation protocols and the Gaussian SVM and MKL handcrafted baselines the ensembles must beat.","marker":"[7]"},{"why":"Supplies the EILAT and RSMAS coral datasets with their 5-fold protocols and the DeepL deep-learning baseline.","marker":"[22]"},{"why":"Defines the WHOI plankton dataset and the equal train/test split used in the experiments.","marker":"[34]"},{"why":"Provides the Sequential Forward Floating Selection algorithm adapted to select the compact 11- and 3-classifier ensembles.","marker":"[33]"},{"why":"Provides DenseNet, the best single model whose fine-tuned variants form the core of the stronger ensembles.","marker":"[30]"},{"why":"Provides the SELU activation used as a training variation to create diverse ensemble members.","marker":"[24]"},{"why":"Provides the FUS_Hand ensemble of handcrafted descriptors used as a plankton baseline.","marker":"[38]"},{"why":"Provides the EnsHC handcrafted-feature ensemble baseline for coral classification accuracy.","marker":"[39]"}],"fun_headline_variants":["CNN ensemble tops all plankton and coral benchmarks","Deep ensemble outguns single nets on five ocean datasets","Heterogeneous CNN ensemble wins on plankton and coral","Fusion of 11 CNNs sets new marks on five marine datasets","Pretrained CNN ensemble beats prior best on all five test sets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the published baseline results were obtained under the same training and test conditions—same folds, splits, and preprocessing—so the ensemble's advantage reflects the method and not an easier evaluation setup.","fun_headline_variants_meta":{"raw":{"variants":["CNN ensemble tops all plankton and coral benchmarks","Deep ensemble outguns single nets on five ocean datasets","Heterogeneous CNN ensemble wins on plankton and coral","Fusion of 11 CNNs sets new marks on five marine datasets","Pretrained CNN ensemble beats prior best on all five test sets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000387,"raw_usage":{"total_tokens":2109,"prompt_tokens":1077,"completion_tokens":1032,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":693,"completion_tokens_details":{"reasoning_tokens":952}},"tokens_in":693,"tokens_out":1032,"duration_ms":6936,"temperature":1.0,"reasoning_tokens":952,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:12:22.658314+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the baselines named in Tables 7 and 8 (for example FUS_Hand, Gaussian SVM, MKL, DeepL, and EnsHC) on the exact same dataset splits used for the proposed ensembles, with the same preprocessing. If any baseline reaches or exceeds the SFFS(11) F-measures or accuracies under identical folds, the paper's central claim of state-of-the-art performance fails.","supporting_citations":[{"cited_title":"Gómez-Ríos, S","cited_arxiv_id":null,"evidence_quote":"Supplies the EILAT and RSMAS coral datasets with their 5-fold protocols and the DeepL deep-learning baseline."},{"cited_title":"Sosik, R.J","cited_arxiv_id":null,"evidence_quote":"Defines the WHOI plankton dataset and the equal train/test split used in the experiments."},{"cited_title":"Pudil, J","cited_arxiv_id":null,"evidence_quote":"Provides the Sequential Forward Floating Selection algorithm adapted to select the compact 11- and 3-classifier ensembles."},{"cited_title":"Nanni, A","cited_arxiv_id":null,"evidence_quote":"Provides the FUS_Hand ensemble of handcrafted descriptors used as a plankton baseline."},{"cited_title":"Shihavuddin, N","cited_arxiv_id":null,"evidence_quote":"Provides the EnsHC handcrafted-feature ensemble baseline for coral classification accuracy."}],"review_version":1}