{"id":"e8db3574-3f05-4299-a3ad-1868e78b5af2","arxiv_id":"2411.10074","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Rejecting low-confidence predictions from softmax classifiers raises automated herbarium label accuracy from ~86% to near-human level at reduced coverage, enabling a 600,000-specimen flowering-time analysis.","lead":"Digitized herbarium images can be labeled automatically by neural networks, but the labels are often too inaccurate to trust. This paper shows that discarding low-confidence predictions raises accuracy from roughly 86% to near-human 95-99% levels, and applies the trick to study flowering-time shifts across 600,000 New England specimens.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Validation accuracy/coverage curves are not demonstrated to transfer to the 600k target corpus, and no held-out test set exists, so the claimed research-grade accuracy at selected thresholds may not hold on deployed labels.","rationale":"I considered a second possible concern: the abstract claims \"over 99% accuracy (rejecting about 65%)\" while Table I caps at 98.6% with 70% rejection for any model, which is a genuine numerical inconsistency. However, that overclaim does not undermine the core mechanism, because 98.6% is still at or above the human-level benchmarks cited in [18]. The distribution-shift problem is more load-bearing: the entire quantitative support for \"human-level accuracy\" is validation-set accuracy/coverage curves, and the paper itself warns in Section II that confidence is only reliable in-distribution. Section V-D deploys to a far larger and more heterogeneous target corpus without an out-of-sample accuracy check. The Gallinat replication provides some external validation for the fruiting classifier on comparable New England herbaria, but it is one model and one collection context, not the four phenophase classifiers or the 600k dataset. The 15k manually annotated subset could have served as a transfer check but is analyzed only as slope error versus sample size, so the transfer assumption remains untested. This is exactly the reader's weakest_assumption, and it supports the existing CONDITIONAL verdict rather than changing it; the condition should be a held-out evaluation on the target distribution before the shared dataset's accuracy claims are accepted.","tokens_in":19064,"tokens_out":6114,"duration_ms":68267,"concrete_test":"Use the 15,000 manually annotated specimens already described in Section V-D1 as a held-out target test: first verify none of these images appear in the training or validation splits of Section V-A, then compute the accuracy and rejection rate at minimum-confidence thresholds 0.90 and 0.99 and compare directly to Table I. If, for any of the four phenophase classifiers, the accuracy at 0.99 falls below 95% or the rejection rate deviates by more than 10 percentage points from the validation value, the validation curves do not transfer to the CNH target distribution. A stronger variant is to draw a new random sample from the remaining unlabeled 600k specimens, stratified by decade and source herbarium, manually annotate it, and run the same comparison.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central load-bearing assumption is stated implicitly in Section II: softmax probabilities are \"usually only usable on in-distribution inputs.\" Section V-A trains four Xception classifiers on 47,551 NEVP images and evaluates them on a 20% validation split of that same dataset; all accuracy/coverage curves, Table I, and the threshold choices (e.g., the 0.99 threshold used in the replication study) are derived from this single validation set. The method is then deployed to roughly 600,000 Consortium of Northeastern Herbaria specimens spanning about 4,000 species after taxonomic reconciliation, with no independent evaluation of accuracy or coverage on that target distribution. If CNH images differ systematically from NEVP in collection style, image quality, preparation, or species composition, the confidence scores will not be calibrated to the same accuracy/coverage relationship as Table I, and the \"research-grade\" labels shared with the paper are not quantitatively supported. The Gallinat et al. replication is genuine external evidence for the fruiting model on similar New England herbaria, but it does not test the other three phenophase classifiers or the full 600k corpus. The 15,000 manually annotated specimens in Section V-D1 are used only for slope-error-versus-sample-size analysis (Fig. 17), not to report accuracy at the Table I thresholds, so they do not close the gap.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a confidence-based rejection pipeline for softmax classifiers: predictions whose maximum softmax probability falls below a user-chosen threshold are discarded, trading coverage for higher accuracy on the remaining labels. The authors train four binary Xception classifiers for herbarium phenophases (budding, flowering, fruiting, non-reproductive), show accuracy/coverage curves on a 20% validation split, and demonstrate that at thresholds near 0.99 the validation accuracy is close to human-level accuracy reported in the literature. They replicate a published fruiting-phenology study (Gallinat et al., 2018) and obtain a native–invasive fruiting difference of 25 days versus the original 26 days. They also apply the approach to an off-the-shelf iNaturalist classifier and report a 600k-specimen annotated herbarium dataset, used for a macrophenological analysis of flowering-time shifts. The dataset is shared on Zenodo.","tokens_in":19162,"tokens_out":4173,"duration_ms":41397,"significance":"If the claims are supported, the work has practical value for biodiversity informatics: it offers a simple, interpretable way to convert low-accuracy deep classifiers into usable annotation tools, and it provides a large publicly released trait-labeled herbarium dataset. The external replication of Gallinat et al. is a genuine strength, as is the demonstration on an unrelated multi-class model. The paper does not claim any new machine-learning method; its contribution is empirical validation and a deployed pipeline. However, the strength of the central claim is currently undermined by evaluation on the same validation set used for threshold selection, an unsupported abstract accuracy figure, and a final ecological analysis that deliberately uses the low-confidence (50%) labels, creating an apparent inconsistency with the paper's headline message.","major_comments":[{"comment":"The abstract states that a naive model with 86% initial accuracy can achieve 'over 99% accuracy (rejecting about 65%)'. Table I does not support this: at a 99% minimum confidence it reports Budding 98.3% (77% rejection), Flowering 98.6% (70%), Fruiting 98.0% (77%), and Non-Reproductive 98.6% (48%). None of these exceed 99%, and the rejection rates are not 'about 65%'. Please either correct the abstract to match the reported numbers, or provide the specific threshold and model for which the 'over 99% / about 65%' claim holds.","section":"Abstract and Table I"},{"comment":"All accuracy/coverage curves, Table I, and the thresholds used later (e.g., 0.99) are computed on the same 20% validation split, and the text in Section III-A selects thresholds from those curves and then reports accuracy on that same split. There is no held-out test set. Selecting a threshold on the validation set and reporting validation accuracy at that threshold is a form of selection on the evaluation data and is likely to yield optimistic estimates. Please add an independent test set (or cross-validation) for reporting the accuracy at selected thresholds, or use the 15,000 manually annotated specimens described in Section V-D1 to report accuracy at the deployment thresholds.","section":"Section V-A and Table I / Fig. 3"},{"comment":"The replication study provides strong external evidence for the fruiting classifier on images from similar New England herbaria, but it does not test the budding, flowering, or reproductive classifiers, nor the 600k Consortium of Northeastern Herbaria (CNH) target distribution. The 15,000 manually annotated specimens from the CNH dataset (Section V-D1) are used only for the slope-error-versus-sample-size analysis (Fig. 17), not to report classification accuracy at the thresholds used for deployment. Consequently, the claim that the 600k labels are 'research-grade' is not quantitatively supported. Please add an accuracy/coverage evaluation on a held-out labeled subset from the CNH corpus, or explicitly limit the research-grade claim to the settings that were actually validated.","section":"Sections V-B and V-D"},{"comment":"The macrophenology analysis of flowering-time shifts uses the 50% confidence threshold by design ('always use the estimate from the 50% minimum threshold'), which corresponds to the low-accuracy, zero-rejection operating point (e.g., Flowering 86.3% in Table V). This conflicts with the paper's central message that high-confidence thresholds are required for reliable annotations. The stated justification, based on Fig. 17, is that slope error depends more on sample size than on threshold; however, that analysis covers only 20 species and 15,000 manual annotations and does not show that 50%-threshold labels are research-grade. Please reconcile this choice with the paper's claims, either by presenting the 600k analysis as a study of regression robustness to label noise or by applying confidence thresholding in that analysis as well.","section":"Sections V-D1 and V-D2"}],"minor_comments":[{"comment":"The caption reads 'EXPENDED ACCURACY RESULTS'; it should be 'EXTENDED ACCURACY RESULTS'.","section":"Table II caption"},{"comment":"In the 'Underlying mechanism insights' paragraph, references to 'Fig. V-A(a)' and 'Fig. V-A(b)' should be 'Fig. 15(a)' and 'Fig. 15(b)'.","section":"Section V-A, Fig. 15 references"},{"comment":"The row for nativity reads 'non-native vs non-natives'; it should read 'non-native vs native'.","section":"Table III"},{"comment":"The text refers to 'Welsh's T-test'; the correct name is 'Welch's t-test'.","section":"Section V-D2"},{"comment":"The sentence 'at the cost of reducing the annotation coverage down to 30% (on average on our custom models)' is ambiguous: it is unclear whether coverage is reduced to 30% or rejection is 30%. Please clarify the relationship between the human-accuracy figure and the coverage/accuracy curve.","section":"Section III-A"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely to be of interest to the biodiversity-informatics and applied-ml communities, and the dataset release is valuable. The main text and abstract, however, currently overstate the validation evidence: the headline 'over 99%' number is not in Table I, and the deployment on 600k specimens lacks a held-out accuracy evaluation on that distribution. The internal inconsistency between the high-threshold message and the 50%-threshold decision in the ecological analysis should also be addressed. These are fixable within the manuscript's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this paper repackages a known confidence-threshold idea (they cite Chow and Geifman & El-Yaniv) and applies it to herbarium phenology classification. The application is new, the Gallinat replication is strong, and the dataset release is useful. But the headline numbers in the abstract overshoot the reported results, and the accuracy on the deployed 600k corpus is not measured on held-out data.\n\nWhat actually deserves credit: the four Xception classifiers for budding, flowering, fruiting, and non-reproductive; the accuracy/coverage curves; and the replication of a published fruiting-phenology study. That replication is the real evidence—using their 0.99 threshold on a separate set of New England herbarium specimens, they get a native vs. invasive fruiting date difference of 25 days against the published 26 days. That's an external check against human labels, and it lands within a day. The iNaturalist generalization, while less central, shows the mechanism isn't specific to their models.\n\nSoft spots, in proportion. First, the abstract claims \"over 99% accuracy (rejecting about 65%)\". Table I does not support that: at 99% minimum confidence, validation accuracy tops out at 98.6% (flowering, 70% rejection). The qualitative claim is fine, the specific numbers are not. Second, all accuracy/coverage curves and threshold choices come from the same single 20% validation split, and that same split is where the reported accuracy numbers come from. There is no independent test set. The method is then applied to roughly 600k Consortium of Northeastern Herbaria specimens, and the paper itself notes softmax probabilities are only reliable in-distribution. The 15,000 manually annotated specimens are used for a slope-error-versus-sample-size analysis, not to report classification accuracy at deployed thresholds. So the research-grade claim on the 600k labels rests on an unquantified distribution-transfer assumption. It's a real gap, but fixable: hold out a labeled test set from the target corpus, or at minimum add an explicit caveat about transfer. Also minor: data are shared, but no code or model weights, which limits independent reproduction.\n\nBottom line: the central empirical pattern is credible, the replication is genuine evidence, and the dataset is a gift to the community. A serious editor should send this to review. I'd ask the authors to correct the abstract numbers and either add an out-of-sample validation or soften the deployment claim. I'd bring this to a reading group, and I'd cite the dataset if I worked in herbarium phenology.","headline":"A solid applied use of a decades-old rejection trick with a genuinely strong external validation in the Gallinat replication, but the abstract overstates the tables and the deployed 600k labels lack an independent accuracy check.","tokens_in":19856,"tokens_out":2503,"would_cite":true,"duration_ms":26514,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Rejecting low-confidence labels can lift a seemingly subpar image classifier to human-level annotation accuracy.","keywords":["confidence thresholding","accuracy-coverage trade-off","selective classification","herbarium digitization","phenology annotation","deep learning for ecology","softmax confidence","rejection option"],"falsifier":"Human-annotate a random sample of specimens from the 600,000-image collection, apply the paper's chosen confidence threshold to the model's predictions on that sample, and compare the accepted labels' accuracy with the validation-set curve; if the accuracy is materially lower, the threshold does not transfer to the unlabeled collection.","tokens_in":18710,"feed_emoji":"🌿","tokens_out":6096,"duration_ms":59989,"temperature":0.7,"pith_summary":"This paper argues that a deep learning classifier's own confidence score can be used as a quality filter: reject labels below a chosen threshold and the remaining annotations become accurate enough for ecological research. On a binary herbarium-phenology model with 86 percent baseline accuracy, rejecting roughly 40 percent of labels pushes accuracy above 95 percent, and rejecting roughly 65 percent pushes it above 99 percent. The paper demonstrates the practical payoff by replicating a published, manually annotated study of fruiting phenology, recovering a 26-day native-versus-invasive difference as 25 days. It then applies the same pipeline to annotate reproductive stages for over 600,000 digitized herbarium specimens and releases the annotated dataset so ecologists can choose their own accuracy-coverage trade-off.","feed_headline":"Low-confidence label rejection lifts accuracy to human-level","feed_subtitle":"A confidence threshold turns an 86 percent model into a research-grade annotator for 600,000 herbarium specimens.","key_machinery":"The load-bearing device is the accuracy/coverage trade-off curve, a simplified risk/coverage analysis that maps each minimum softmax confidence threshold to the accuracy of accepted labels and the fraction of data retained. The softmax probability for the predicted class serves as the confidence score; predictions below a user-chosen threshold are rejected rather than labeled. This converts the classifier's top-1 score into a tunable operating point, and the paper's embedding analysis indicates that rejected low-confidence samples sit near the class boundary, so rejection effectively enlarges the decision margin.","core_discovery":"The central claim is that even a seemingly subpar softmax classifier can produce research-grade labels when low-confidence predictions are discarded. The paper validates this on four custom binary classifiers (budding, flowering, fruiting, non-reproductive) and on an off-the-shelf multi-class model, showing in each case that accuracy rises monotonically with the confidence threshold while coverage falls. The accuracy/coverage curves built on a validation set let a user pick a threshold that meets a target accuracy, and the authors show that the replication of a human-annotated phenology study becomes unreliable at the naive threshold but closely matches the original conclusions at a high threshold. This is presented as a practical, model-agnostic way to move automatic labeling from unusable to valuable for large digitized collections.","pith_inferences":["An implication the paper leaves implicit is that the validation-set accuracy/coverage curves transfer only if the unlabeled target images resemble the validation images; a collection with different preparation styles or species composition would need its own threshold calibration.","The rejection step reorders predictions but cannot correct errors the model makes confidently; a misplaced but high-confidence label passes the threshold and becomes research data.","A natural extension, consistent with the paper's pipeline, is to send the rejected low-confidence subset to human annotators, recovering full coverage while keeping the high-confidence labels automatic.","Because the 600k-specimen flowering analyses inherit the model's labeling biases, such as the fruiting-maturity mismatch the paper acknowledges in the replication, trait-level conclusions should be interpreted with that bias in mind."],"forward_implications":["A model that looks too inaccurate from its top-1 accuracy alone can still be used for research when data is abundant enough to absorb the reduced coverage.","Researchers can select a confidence threshold to match the accuracy demands of a given study, or use the pipeline to auto-label the confident fraction and hand-label only the rest.","The same thresholding procedure works for off-the-shelf multi-class classifiers, suggesting it applies across architectures and image domains.","The released 600,000-specimen dataset with per-label confidences lets ecologists perform their own analyses at any accuracy-coverage operating point."],"supporting_citations":[{"why":"Supplies the 95-98 percent human-annotation accuracy baseline that the thresholded models are measured against.","marker":"[18]"},{"why":"Defines the prior deep-learning phenology annotation accuracy of 80-87 percent that this work aims to improve on.","marker":"[6]"},{"why":"Provides the selective-classification risk/coverage formulation that the simpler accuracy/coverage curves are based on.","marker":"[23]"},{"why":"The manually annotated fruiting-phenology study whose species-level results serve as ground truth for the replication experiment.","marker":"[34]"},{"why":"The off-the-shelf multi-class model used to show the thresholding method generalizes beyond custom binary classifiers.","marker":"[35]"},{"why":"Supports the choice of four separate binary classifiers to match how phenological stages are annotated.","marker":"[17]"}],"fun_headline_variants":["Reject low confidence, hit 99% accuracy","Confidence cutoff turns 86% model into 99% annotator","Discard uncertain labels, boost accuracy to 99%","Threshold trick lifts AI labeling to human level","Skip unsure calls, get research-grade labels"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The accuracy and rejection rates measured on the validation set are assumed to hold on the far larger unlabeled target dataset, even though the target may differ in image quality, species composition, or collection style.","fun_headline_variants_meta":{"raw":{"variants":["Reject low confidence, hit 99% accuracy","Confidence cutoff turns 86% model into 99% annotator","Discard uncertain labels, boost accuracy to 99%","Threshold trick lifts AI labeling to human level","Skip unsure calls, get research-grade labels"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000623,"raw_usage":{"total_tokens":2923,"prompt_tokens":1018,"completion_tokens":1905,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":634,"completion_tokens_details":{"reasoning_tokens":1828}},"tokens_in":634,"tokens_out":1905,"duration_ms":13534,"temperature":1.0,"reasoning_tokens":1828,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:00:08.835078+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Human-annotate a random sample of specimens from the 600,000-image collection, apply the paper's chosen confidence threshold to the model's predictions on that sample, and compare the accepted labels' accuracy with the validation-set curve; if the accuracy is materially lower, the threshold does not transfer to the unlabeled collection.","supporting_citations":[{"cited_title":"Maximizing human effort for analyzing scientific images: A case study using digitized herbarium sheets,","cited_arxiv_id":null,"evidence_quote":"Supplies the 95-98 percent human-annotation accuracy baseline that the thresholded models are measured against."},{"cited_title":"Toward a large-scale and deep phenological stage annotation of herbarium specimens: Case studies from temperate, tropical, and equatorial floras,","cited_arxiv_id":null,"evidence_quote":"Defines the prior deep-learning phenology annotation accuracy of 80-87 percent that this work aims to improve on."},{"cited_title":"Selective classification for deep neural networks,","cited_arxiv_id":null,"evidence_quote":"Provides the selective-classification risk/coverage formulation that the simpler accuracy/coverage curves are based on."},{"cited_title":"Herbarium specimens show patterns of fruiting phenology in native and invasive plant species across new england,","cited_arxiv_id":null,"evidence_quote":"The manually annotated fruiting-phenology study whose species-level results serve as ground truth for the replication experiment."},{"cited_title":"Bbn network for inaturalist competition,","cited_arxiv_id":null,"evidence_quote":"The off-the-shelf multi-class model used to show the thresholding method generalizes beyond custom binary classifiers."},{"cited_title":"Phenology models using herbarium specimens are only slightly improved by using finer-scale stages of reproduction,","cited_arxiv_id":null,"evidence_quote":"Supports the choice of four separate binary classifiers to match how phenological stages are annotated."}],"review_version":1}