Hierarchical Classification for Automated Image Annotation of Coral Reef Benthic Structures

C\'elia Blondin; Guilherme Longo; Joris Gu\'erin; Kelly Inagaki; Laure Berti-\'Equille

REVIEW 2 major objections 5 minor 1 cited by

Hierarchical Classification for Automated Image Annotation of Coral Reef Benthic Structures

T0 review · 2 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A top-down hierarchical classifier beats a flat classifier for coral reef benthic image annotation, improving F1 and hierarchical F1 by about 2 percentage points.

desk verdict A clean but small empirical study; the 1–2% hierarchical classification gain is plausible but not yet solid because the test split is at patch level. read the letter →

arxiv 2412.08228 v1 pith:T6PREWJT submitted 2024-12-11 cs.CV cs.AIcs.LG

Célia Blondin , Joris Guérin , Kelly Inagaki , Guilherme Longo , Laure Berti-Équille This is my paper · ORCID

classification cs.CVcs.AIcs.LG

keywords hierarchicalclassificationbenthicimageannotationcoralreefmonitoringtop-downclassifierF1healthstatusimbalancedlabelsets

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that annotating coral reef benthic images as a hierarchical classification problem—labels organized by taxonomy and health status—works better than treating all 54 labels as a flat set. On a custom dataset from a Northeast Brazilian reef, a top-down hierarchical classifier beats a flat classifier by roughly 1 to 2 percent in both standard F1 and hierarchical F1 across different training set sizes. The authors argue this matters because ecologists already aggregate benthic labels by macro groups, and a hierarchy gives partial credit for errors that stay in the correct branch. They conclude that incorporating the hierarchy during training improves accuracy while making automated annotation more aligned with ecological monitoring objectives.

What carries the argument

The central object is the label hierarchy tree, designed jointly by an ML specialist and a marine ecologist, with 54 leaf labels covering algae, corals, substrates, and other benthic categories; coral nodes are split by genus and by health state such as bleached versus unbleached. The carrying mechanism is a top-down 'Local Classifier per Parent Node' scheme: each non-leaf node gets its own small multilayer perceptron, features come from a CNN backbone pre-trained on millions of benthic images, and a patch is classified by walking from the root to a single leaf. The evaluation is carried by hierarchical F1, a metric that gives partial credit to predictions that stay within the correct branch of the tree.

What would settle it

Run the same flat-versus-hierarchical comparison with a test split that holds out whole images, for example 10 percent of the 1,549 images, and recompute F1 and hierarchical F1; the central claim is falsified if the hierarchical classifier no longer beats the flat classifier by roughly 2 percent.

Watch

Extended reading notes

Core claim

The central claim is that the hierarchical organization of benthic labels is not just an evaluation nicety but a training signal: a top-down classifier built by training one small model at each node of an ecological label tree outperforms a flat classifier on the same 54 leaf labels. The reported gain is about 1 percent on small training sets and about 2 percent on larger sets, on both standard F1 and hierarchical F1. The authors also find the flat classifier retains some implicit hierarchical knowledge, since the hierarchical F1 gap is similar to the flat F1 gap, and they interpret this as a sign that flat models learn some class relationships without being told the tree.

Load-bearing premise

The reported gain rests on a test set built by randomly sampling patches without keeping images together, so patches from the same photo can appear in both training and test and share background and lighting; if an image-level split erases the 2 percent advantage, the central claim fails.

Editorial extensions

If this is right

If the two-percent gain holds, existing monitoring pipelines could switch from flat to hierarchical classifiers with only a change in label organization and per-node training, not new image inputs.
Cover estimates for ecologically meaningful macro groups, such as all corals or all bleached corals, could become more accurate because top-down predictions are constrained to follow the tree.
Hierarchical F1 could be adopted as a standard secondary metric for benthic annotation, making partially correct predictions visible when comparing models.
The results provide a first demonstration of top-down hierarchical classification in benthic image annotation, giving other reef datasets a baseline to test against.
Because the gain is modest, the paper's own conclusion is that deployment must weigh the extra compute of multiple classifiers against the accuracy improvement.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

An immediate testable extension is an image-level split, holding out whole images rather than random patches; if the 2 percent advantage shrinks or disappears, the reported gain is at least partly an artifact of patches from the same photo sharing lighting and background.
The paper's plan to apply the method to a larger, simpler labeled coral dataset could be made sharper by reporting per-class and per-branch gains, since the hierarchy may help most for rare leaf labels below the dominant 11 labels.
The same top-down recipe could transfer to other ecological monitoring tasks with natural label hierarchies, such as seagrass or mangrove cover classification, where flat label sets also obscure taxonomic relationships.
Automating hierarchy construction from label semantics is the paper's suggested direction; a concrete version would learn the tree from co-occurrence of labels in ecological surveys and test whether the 2 percent gain survives an automatically built tree.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a small, honest workshop paper that applies a standard top-down hierarchical classifier (Local Classifier per Parent Node via hiclass) to coral benthic patch annotation, on top of a CoralNet EfficientNet backbone. The authors report about 1–2% improvement in F1 and hF1 over a flat classifier with the same backbone and MLP head. The comparison is fairly controlled, and they are appropriately modest about the gain and the computational trade-off.

What's genuinely new is the application domain: I don't know of prior HC work on benthic image annotation, and they build a biologically informed label tree with a marine ecologist. That tree itself is a useful artifact for the coral monitoring community. They also make a nice observation that the flat classifier implicitly captures some hierarchical structure, since the hF1 gap is about the same size as the F1 gap.

The weak spot is the test split. They sampled 10% of patches independently of source image, so patches from the same image (same lighting, water column, substrate texture) can appear in both train and test. That can leak image identity and inflate the 2% gain, which is already within a small range. They acknowledge the choice but don't test its impact. An image-level split or at least a grouped split with a significance test would be needed to be confident that the benefit generalizes to new scenes. The dataset is small, heavily imbalanced, and not public, which limits independent verification. The paper ships code, which is good, but without data it's harder to reproduce fully.

There's no circularity problem: the hierarchy is used both for the method and for the hF1 metric, but the metric is applied symmetrically to both classifiers, so it doesn't favor HC by construction. The claim is empirical and clearly separated from the metric.

Bottom line: this is a credible but preliminary empirical contribution. The application is new, the comparison is clean, but the patch-level split and lack of significance testing mean the central claim is not yet fully established. Who should read it: ecologists using CoralNet workflows and researchers working on hierarchical classification for fine-grained or imbalanced image data. It deserves a serious referee, but the referee should ask for an image-level split, error analysis, and ideally a public dataset or a benchmark on an existing public dataset like Moorea Labeled Corals.

My recommendation: send it to peer review with conditions. If the authors can show the 1–2% survives an image-level split, this is a solid incremental result worth publishing.

Referee Report

2 major / 5 minor

Summary. The paper proposes using top-down hierarchical classification (a local classifier per parent node, implemented with the hiclass library and two-hidden-layer MLPs) for automated annotation of coral reef benthic image patches, using features extracted by the CoralNet EfficientNet B0 backbone. The label hierarchy is designed with marine ecologists to reflect taxonomic relationships and coral health status. The authors compare this approach against a flat MLP baseline on a custom dataset of 38,725 patches from 1,549 images from a Brazilian reef, reporting F1-score and hierarchical F1-score across increasing training-set sizes. The central claim is that the hierarchical classifier outperforms the flat classifier by approximately 1–2% in both metrics across training-data amounts, while also aligning better with ecological grouping needs. The paper concludes that these modest gains must be weighed against the added computational cost of training and running multiple local classifiers.

Significance. If the claimed improvement is genuine, the paper would provide a practical and ecologically motivated contribution to benthic image annotation, where label hierarchies are natural and where partial credit for near-correct predictions matters. The experimental design has several strengths: both classifiers use the same backbone and MLP architecture, making the comparison fair at the model level; the evaluation covers multiple training-set sizes; and the authors provide a public GitHub repository for reproducibility. However, the significance is currently limited by two load-bearing weaknesses: the test set is constructed at the patch level rather than the image level, and the reported performance differences are not accompanied by statistical significance testing. Because patches from the same image share lighting, water-column conditions, and substrate texture, a random patch-level split can leak image-identity information that may inflate the observed 1–2% advantage. The central claim is therefore plausible but not yet established.

major comments (2)

[§3.1.1] The test set is constructed by randomly selecting 10% of patches independently, without considering image-level coherence. Since each image contributes 25 patches that share lighting, background, and annotation-session bias, this split allows patches from the same image to appear in both training and test sets. A hierarchical classifier and a flat classifier may exploit these image-specific cues differently, so the reported 1–2% improvement in F1 and hF1 could be an artifact of the split rather than a genuine generalization advantage. The authors acknowledge the choice but do not evaluate its impact. To support the central claim, the experiments should be repeated with an image-level split (or a cross-validation at the image level), and the results compared with the current patch-level split.
[§3.2 and Figure 3] The claim that the hierarchical classifier 'always outperforms' the flat classifier is based on point estimates of F1 and hF1, with no significance test or confidence intervals that would let the reader assess whether the 1–2% gaps are within sampling noise. The dataset is small and highly imbalanced (11 labels account for 95% of annotations), so variance across random training-set draws could be substantial. The error bars shown in Figure 3 are standard deviations from random training-set sampling, but no test of whether the differences are statistically significant is reported. I recommend adding a formal comparison (e.g., paired bootstrap or Wilcoxon signed-rank test over repeated splits, ideally at the image level) and reporting the corresponding p-values or confidence intervals.

minor comments (5)

[Section 2.1 heading] The heading 'Building the Hierachy' contains a typo; it should be 'Hierarchy'.
[Author block] The word 'Correspondance' should be 'Correspondence'.
[Appendix A] The tree listing includes 'Mussimila Mussimila', which appears to be a typo for 'Mussismilia' based on standard coral taxonomy; this should be corrected to avoid confusion.
[§3.1.3] The hierarchical F1-score (hF1) is cited to Kosmopoulos et al. but is not defined in the text. Since hF1 is central to the evaluation, a brief formula or a precise description of how it is computed (e.g., how the hierarchy is used to give partial credit) would help readers interpret the numerical results.
[Figure 3] The figure shows error bars but the exact numerical values (means and standard deviations) are not provided in the text or tables. Reporting these values in a table would improve transparency and reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the paper is an empirical comparison and its central claim does not reduce to its inputs.

full rationale

The paper's central claim is that a top-down hierarchical classifier outperforms a flat classifier on a custom benthic image annotation dataset, improving F1 and hierarchical F1 by about 1-2% across training-set sizes (Abstract; Section 3.2; Figure 3). This is an experimental comparison, not a derivation, and no fitted parameter is relabeled as a prediction. The hierarchical tree is used both to build the HC classifier and to define the hF1 metric, but hF1 is applied symmetrically to the flat baseline as well, so the metric does not force the hierarchical classifier's advantage by construction. The only acknowledged limitation is the patch-level test split (Section 3.1.1: 'Test patches were chosen independently, without considering image-level coherence'), which is a potential data-leakage and generalization concern, not a circularity concern, and the paper does not claim a derivation that reduces to its own definitions. References to CoralNet [10] and hiclass [14] are external tools; there is no load-bearing self-citation or imported uniqueness theorem. The evaluation against a flat baseline and the honest reporting of modest gains further indicate that the claim is empirically grounded rather than circular. Therefore the circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper's central claim depends on several hand-chosen hyperparameters (MLP hidden layer sizes), an ad hoc hierarchy design, and assumptions that the labels form a valid tree, that the pretrained features transfer, and that the test split is representative. No new entities are introduced.

free parameters (3)

MLP hidden layer sizes = 200 and 100 neurons
Chosen by hand for node classifiers; affects capacity and could influence the 2% comparison, though same for both flat and hierarchical.
Benthic label hierarchy = custom tree in Appendix A
Ad hoc design by ML expert and ecologist; central to the method; no independent validation.
Test/train split proportion = 90/10
Selected without stated justification; could affect variance of reported gains.

assumptions (4)

domain assumption The 54 leaf labels can be organized into a single-path tree without overlap
The tree in Appendix A is the basis for the hierarchy; if it misrepresents biology, the method's gains may be meaningless.
domain assumption CoralNet backbone features are sufficiently transferable to this Brazilian reef
Both classifiers use the same frozen features; if the backbone is not well-suited, the comparison may not reflect the true value of HC.
domain assumption Patches from the same image are independent samples
The random patch split treats each patch as independent; correlation within images is ignored, which may inflate test performance.
domain assumption F1 and hierarchical F1 capture ecological utility
The choice of metrics reflects the authors' claim of alignment with ecological objectives, but no ecological validation is performed.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Hierarchical Classification for Automated Image Annotation of Coral Reef Benthic Structures." pith.science (2026). https://pith.science/paper/T6PREWJT

@misc{pith2026241208228,
  author       = {Pith},
  title        = {Pith review of: Hierarchical Classification for Automated Image Annotation of Coral Reef Benthic Structures},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/T6PREWJT}},
  note         = {Machine review of arXiv:2412.08228}
}

read the original abstract

Automated benthic image annotation is crucial to efficiently monitor and protect coral reefs against climate change. Current machine learning approaches fail to capture the hierarchical nature of benthic organisms covering reef substrata, i.e., coral taxonomic levels and health condition. To address this limitation, we propose to annotate benthic images using hierarchical classification. Experiments on a custom dataset from a Northeast Brazilian coral reef show that our approach outperforms flat classifiers, improving both F1 and hierarchical F1 scores by approximately 2\% across varying amounts of training data. In addition, this hierarchical method aligns more closely with ecological objectives.

Figures

Figures reproduced from arXiv: 2412.08228 by the authors.

**Figure 1.** Benthic image annotation process – A predetermined number of random points are sampled within a fixed area and annotated with the corresponding organism or substrate. The relative frequencies are used as proxies for cover proportions of benthic categories within the sampled area. investigate the application of Hierarchical Classification (HC) [11] to benthic image annotation, a novel approach in this field. This mul… view at source ↗

**Figure 2.** Hierarchical structure of benthic labels – This conceptual diagram illustrates the multilevel categories used for classifying benthic organisms and substrates in coral reef ecosystems. 3 Experiments This section presents the experiments conducted to assess the potential of HC for benthic image annotation. We provide a detailed Github repository to reproduce our results1 . 3.1 Experimental Setup 3.1.1 Dataset For th… view at source ↗

**Figure 3.** Experimental results – Comparison of flat and hierarchical classifiers for our custom benthic image annotation dataset, showing performance metrics across varying training data sizes. Error bars indicate standard deviation from random training set sampling. 3.2 Results Our experimental results are displayed in [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Number of patches corresponding to each leaf node present in our custom benthic image [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Improving Detection of Rare Nodes in Hierarchical Multi-Label Learning
cs.LG 2026-02 conditional novelty 6.0 of 10

A node-weighted loss combining inverse-frequency weighting and ensemble-uncertainty focal terms improves recall of rare classes in hierarchical multi-label models by up to ~5x.

Reference graph

Works this paper leans on

17 extracted references · 16 canonical work pages · cited by 1 Pith paper

[1]

Status of Coral Reefs of the World: 2020,

D. Souter, S. Planes, J. Wicquart, M. Logan, D. Obura, and F. Staub, “Status of Coral Reefs of the World: 2020,” 2020. [Online]. Available: https://gcrmn.net/wp-content/uploads/2022/05/ Executive-Summary-with-Forewords.pdf

work page 2020
[2]

Coral reef ecosystem services in the anthropocene,

A. J. Woodhead, C. C. Hicks, A. V . Norström, G. J. Williams, and N. A. Graham, “Coral reef ecosystem services in the anthropocene,” Functional Ecology, vol. 33, no. 6, pp. 1023–1034, 2019

work page 2019
[3]

Warm-water coral reefs and climate change,

M. D. Spalding and B. E. Brown, “Warm-water coral reefs and climate change,” Science, vol. 350, no. 6262, pp. 769–771, 2015

work page 2015
[4]

The 2014–2017 global-scale coral bleaching event: insights and impacts,

C. M. Eakin, H. P. A. Sweatman, and R. E. Brainard, “The 2014–2017 global-scale coral bleaching event: insights and impacts,” Coral Reefs, vol. 38, no. 4, pp. 539–545, Aug. 2019. [Online]. Available: https://doi.org/10.1007/s00338-019-01844-2

work page doi:10.1007/s00338-019-01844-2 2014
[5]

Systematic global assessment of reef fish communities by the reef life survey program,

G. J. Edgar and R. D. Stuart-Smith, “Systematic global assessment of reef fish communities by the reef life survey program,” Scientific Data, vol. 1, no. 1, pp. 1–8, 2014

work page 2014
[6]

The 27–year decline of coral cover on the great barrier reef and its causes,

G. De’Ath, K. E. Fabricius, H. Sweatman, and M. Puotinen, “The 27–year decline of coral cover on the great barrier reef and its causes,” Proceedings of the National Academy of Sciences, vol. 109, no. 44, pp. 17 995–17 999, 2012

work page 2012
[7]

Comparison of methods used to estimate coral cover in the hawaiian islands,

P. L. Jokiel, K. S. Rodgers, E. K. Brown, J. C. Kenyon, G. Aeby, W. R. Smith, and F. Farrell, “Comparison of methods used to estimate coral cover in the hawaiian islands,”PeerJ, vol. 3, p. e954, 2015

work page 2015
[8]

Towards automated annotation of benthic survey images: Variability of human experts and operational modes of automation,

O. Beijbom, P. J. Edmunds, C. Roelfsema, J. Smith, D. I. Kline, B. P. Neal, M. J. Dunlap, V . Moriarty, T.-Y . Fan, C.-J. Tanet al., “Towards automated annotation of benthic survey images: Variability of human experts and operational modes of automation,” PloS one, vol. 10, no. 7, p. e0130312, 2015

work page 2015

Show all 17 references

[9]

Leveraging automated image analysis tools to transform our capacity to assess status and trends of coral reefs,

I. D. Williams, C. S. Couch, O. Beijbom, T. A. Oliver, B. Vargas-Angel, B. D. Schumacher, and R. E. Brainard, “Leveraging automated image analysis tools to transform our capacity to assess status and trends of coral reefs,” Frontiers in Marine Science, vol. 6, p. 222, 2019

2019
[10]

A new deep learning engine for coralnet,

Q. Chen, O. Beijbom, S. Chan, J. Bouwmeester, and D. Kriegman, “A new deep learning engine for coralnet,” in Proc. of the IEEE/CVF international conference on computer vision, 2021, pp. 3693–3702

2021
[11]

A survey of hierarchical classification across different application domains,

C. N. Silla and A. A. Freitas, “A survey of hierarchical classification across different application domains,” Data mining and knowledge discovery, vol. 22, pp. 31–72, 2011

2011
[12]

Evaluation measures for hierarchical classification: a unified view and novel approaches,

A. Kosmopoulos, I. Partalas, E. Gaussier, G. Paliouras, and I. Androutsopoulos, “Evaluation measures for hierarchical classification: a unified view and novel approaches,” Data Mining and Knowledge Discovery, vol. 29, pp. 820–865, 2015

2015
[13]

Hierarchical multi-label classification networks,

J. Wehrmann, R. Cerri, and R. Barros, “Hierarchical multi-label classification networks,” inInternational conference on machine learning. PMLR, 2018, pp. 5075–5084

2018
[14]

HiClass: a Python Library for Local Hierarchical Classification Compatible with Scikit-learn,

F. M. Miranda, N. Köhnecke, and B. Y . Renard, “HiClass: a Python Library for Local Hierarchical Classification Compatible with Scikit-learn,” Journal of Machine Learning Research, vol. 24, no. 29, pp. 1–17, 2023. [Online]. Available: http://jmlr.org/papers/v24/21-1518.html

2023
[15]

Large-scale patterns of benthic marine communities in the brazilian province,

A. W. Aued, F. Smith, J. P. Quimbayo, D. V . Candido, G. O. Longo, C. E. Ferreira, J. D. Witman, S. R. Floeter, and B. Segal, “Large-scale patterns of benthic marine communities in the brazilian province,”PloS one, vol. 13, no. 6, p. e0198452, 2018

2018
[16]

MCR LTER: Coral Reef: Computer Vision: Moorea Labeled Corals,

Moorea Coral Reef LTER and P. Edmunds, “MCR LTER: Coral Reef: Computer Vision: Moorea Labeled Corals,” 2019. [Online]. Available: https://portal.edirepository.org/nis/mapbrowse?packageid= knb-lter-mcr.5006.3

2019
[17]

Discriminative Transfer Learning with Tree-based Priors,

N. Srivastava and R. R. Salakhutdinov, “Discriminative Transfer Learning with Tree-based Priors,” in Advances in Neural Information Processing Systems , vol. 26, 2013. [Online]. Available: https://papers.nips.cc/paper_files/paper/2013/hash/9ac403da7947a183884c18a67d3aa8de-Abst...

2013

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

Status of Coral Reefs of the World: 2020,

D. Souter, S. Planes, J. Wicquart, M. Logan, D. Obura, and F. Staub, “Status of Coral Reefs of the World: 2020,” 2020. [Online]. Available: https://gcrmn.net/wp-content/uploads/2022/05/ Executive-Summary-with-Forewords.pdf

work page 2020

[2] [2]

Coral reef ecosystem services in the anthropocene,

A. J. Woodhead, C. C. Hicks, A. V . Norström, G. J. Williams, and N. A. Graham, “Coral reef ecosystem services in the anthropocene,” Functional Ecology, vol. 33, no. 6, pp. 1023–1034, 2019

work page 2019

[3] [3]

Warm-water coral reefs and climate change,

M. D. Spalding and B. E. Brown, “Warm-water coral reefs and climate change,” Science, vol. 350, no. 6262, pp. 769–771, 2015

work page 2015

[4] [4]

The 2014–2017 global-scale coral bleaching event: insights and impacts,

C. M. Eakin, H. P. A. Sweatman, and R. E. Brainard, “The 2014–2017 global-scale coral bleaching event: insights and impacts,” Coral Reefs, vol. 38, no. 4, pp. 539–545, Aug. 2019. [Online]. Available: https://doi.org/10.1007/s00338-019-01844-2

work page doi:10.1007/s00338-019-01844-2 2014

[5] [5]

Systematic global assessment of reef fish communities by the reef life survey program,

G. J. Edgar and R. D. Stuart-Smith, “Systematic global assessment of reef fish communities by the reef life survey program,” Scientific Data, vol. 1, no. 1, pp. 1–8, 2014

work page 2014

[6] [6]

The 27–year decline of coral cover on the great barrier reef and its causes,

G. De’Ath, K. E. Fabricius, H. Sweatman, and M. Puotinen, “The 27–year decline of coral cover on the great barrier reef and its causes,” Proceedings of the National Academy of Sciences, vol. 109, no. 44, pp. 17 995–17 999, 2012

work page 2012

[7] [7]

Comparison of methods used to estimate coral cover in the hawaiian islands,

P. L. Jokiel, K. S. Rodgers, E. K. Brown, J. C. Kenyon, G. Aeby, W. R. Smith, and F. Farrell, “Comparison of methods used to estimate coral cover in the hawaiian islands,”PeerJ, vol. 3, p. e954, 2015

work page 2015

[8] [8]

Towards automated annotation of benthic survey images: Variability of human experts and operational modes of automation,

O. Beijbom, P. J. Edmunds, C. Roelfsema, J. Smith, D. I. Kline, B. P. Neal, M. J. Dunlap, V . Moriarty, T.-Y . Fan, C.-J. Tanet al., “Towards automated annotation of benthic survey images: Variability of human experts and operational modes of automation,” PloS one, vol. 10, no. 7, p. e0130312, 2015

work page 2015