{"id":"b58ac690-a351-402f-97a5-ac2f84c4d76a","arxiv_id":"2412.18409","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Single-label evaluation hides multi-label recognition ability and explains much of the ImageNetV2 accuracy gap, which shrinks under multi-label-aware metrics and synthetic object-composition tests.","lead":"This paper argues that the widely reported accuracy drop on ImageNetV2 is mostly an artifact of scoring each image with only one correct label. Using multi-label evaluation on 315 models, the authors show the perceived gap narrows substantially and that models can recognize multiple objects.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline 0–6% ASMA gap is not reproducible from the paper's own metric definition: Section 3.3's Hamming-style formula cannot yield the plotted values, so the central claim is currently unverifiable.","rationale":"The reader's weakest assumption about the self-authored ImageNetV2 annotations is legitimate and is explicitly acknowledged in Section 5; if those labels are incomplete or over-inclusive, the measured gap reduction could be an artifact of the annotation protocol. However, I see the ASMA metric inconsistency as the more immediately load-bearing concern because it blocks verification of the headline quantitative claim regardless of label quality. The formula in Section 3.3, taken literally with the only described prediction mechanism (variable top-k), produces Hamming-style accuracies dominated by true negatives; such a metric would compress all differences between ImageNetV1 and ImageNetV2 to well below one percentage point, not the displayed 0–6% range. This means the reader cannot tell what was actually plotted in Figure 4c, and the central claim—that a substantial portion of the ImageNetV2 drop is an evaluation artifact—is not reproducible from the manuscript. The paper also does not specify how unannotated ImageNetV2 images (9,858 of 10,000 annotated) are treated, and no code or commit hash is provided. These issues are fixable in revision, so I would keep the reader's CONDITIONAL verdict: acceptance should require a corrected, precisely specified ASMA definition, a reproducible implementation, and ideally an independent check of the ImageNetV2 annotations. My agreement is partial because the reader emphasized the label-validation concern as the weakest assumption, whereas I would put the undefined/inconsistent metric first, with label quality as a necessary second validation.","tokens_in":17360,"tokens_out":8771,"duration_ms":83409,"concrete_test":"Implement Section 3.3's ASMA exactly as written on five representative models from Figure 4c: binarize each model's logits by setting y_hat_c=1 for the top-k_i classes (k_i = number of ground-truth labels) and 0 elsewhere, compute per-image label-wise accuracy over all 1000 classes, average within label-count subgroups, then average across subgroups. Compare the resulting V1–V2 gaps to Figure 4c. If they do not match (expected: gaps near 0.1% rather than 0–6%), the paper must specify the actual ASMA variant, release the code, and re-run the full 315-model analysis before the central claim can be evaluated.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central quantitative result—that the ImageNetV2 gap shrinks to 0–6% under ASMA (Figure 4c)—depends on a metric that is not consistently defined. Section 3.3 defines ASMA as the mean over label-count subgroups of per-datapoint label-wise accuracy (1/C) * sum_c I(y_gt = y_hat), with y_hat in {0,1}^C, but the only prediction construction given is variable top-k (Section 3.1), and no rule for converting top-k indices into the binary vector y_hat is specified. If y_hat is the natural indicator of the top-k predicted classes, then with C=1000 and k equal to the number of true labels, every image scores at least roughly 0.99 because true negatives dominate; the V1–V2 ASMA gap would be far below 1%, not the plotted 0–6% range. Thus either Figure 4c was computed with a different, unstated ASMA variant (e.g., positive-label recall), or the formula in Section 3.3 is incorrect. Until this is resolved, the reported narrowing of the generalization gap cannot be checked. Separately, Section 5 concedes that the ImageNetV2 labels from Anzaku et al. [14] may miss plausible categories; that annotation-quality concern is real, but the metric definition is the more immediate blocker because it affects the headline number even if the labels are perfect.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that the commonly reported 11–14% top-1 accuracy drop on ImageNetV2 is largely an artifact of the single-label evaluation protocol. It proposes variable top-k evaluation with k equal to the number of valid labels per image, introduces Aggregate Subgroup Model Accuracy (ASMA) as a subgroup-averaged multi-label accuracy, and presents PatchML, a synthetic dataset of composed object patches. Across 315 ImageNet-pretrained models, the authors report that the V1–V2 gap narrows to 0–6% under ASMA and that many models rank multiple correct labels highly. The paper concludes that single-label metrics understate DNN multi-label capability.","tokens_in":17641,"tokens_out":4886,"duration_ms":43478,"significance":"If the quantitative claims could be verified, the work would be valuable: it addresses a widely discussed anomaly in benchmarking, uses a broad model pool (315 TIMM models), proposes a synthetic diagnostic that removes co-occurrence cues, and ships a code repository (claimed). The reframing of the ImageNetV2 gap as partly evaluation-induced is consistent with prior work on ReaL labels and would have implications for benchmark design. However, the central quantitative claim currently rests on an ASMA definition that appears inconsistent with the reported numbers, and on self-authored ImageNetV2 annotations that lack independent validation; both need to be resolved before the conclusions can be accepted.","major_comments":[{"comment":"The ASMA definition is internally inconsistent with the reported numbers. The label-wise accuracy formula (1/C) * sum_c I(y_gt_g,i,c = y_hat_g,i,c) with C=1000 and y_hat in {0,1}^C is a Hamming-style accuracy, but no rule is given for constructing the binary prediction vector y_hat from the variable top-k softmax outputs of Section 3.1. If y_hat is the indicator of the top-k predicted classes and k equals the number of true labels g, then the per-image score is 1 - (false positives)/1000, which is at least ~0.99 for any image with all true labels in the top-k and typically not much lower otherwise. This cannot produce the subgroup accuracies near 0.2–0.7 in Figure 5b or the 0–6% ASMA gap in Figure 4c. The authors must correct the formula (e.g., to example-based accuracy |Y ∩ Yhat| / |Y ∪ Yhat|, which they claim to extend) or specify the binarization and recompute all ASMA-based figures. Because the abstract and Section 1.1 base the central 'gap shrinks to 0–6%' claim on ASMA, this inconsistency is load-bearing.","section":"Section 3.3, Eq. (Ag) and Figure 4c/Figure 5b"},{"comment":"The quantitative gap reduction on ImageNetV2 depends entirely on the multi-label annotations from Anzaku et al. [14], which are self-authored and not machine-checked, reproduced, or independently validated in this paper. The paper provides no inter-annotator agreement, error analysis, or third-party comparison for these labels. Section 5 acknowledges that the annotations 'may still miss plausible object categories,' but the main conclusion that the V2 gap is an evaluation artifact assumes the labels are complete and correct. I ask for a sensitivity analysis or an independent validation subset (e.g., labels from a different annotation protocol, or agreement rates on a sample) before the gap-reduction claim is accepted.","section":"Section 4.2.1, Table 1, Figure 4"}],"minor_comments":[{"comment":"The caption says 'all plotted against label count' but the x-axis is the index of pre-trained models; please correct the caption to match the actual axes.","section":"Figure 4 caption"},{"comment":"The ASMA sum runs over g = 0 to G-1, but no images have zero labels in Table 1; please specify how empty subgroups are handled or start the sum at g = 1.","section":"Section 3.3"},{"comment":"It is unclear whether the patch pool S is reset for each k value and for each of the five seeds; please clarify, as this affects the independence of the PatchML variants summarized in Table 2.","section":"Section 3.2, Algorithm 1"},{"comment":"The metric is called 'Aggregate Subgroup Model Accuracy' in the abstract and 'Average Subgroup Multi-Label Accuracy' in Section 3.3; please use a single consistent name throughout.","section":"Abstract and Section 3.3"}],"recommendation":"major_revision","confidential_remarks":"The main blocker is the ASMA definition; the authors should be asked to provide the exact computation code or a corrected formula, since the reported numbers cannot be reproduced from the paper as written. I also note the heavy reliance on the authors' own ImageNetV2 annotations from Anzaku et al. [14] without external validation; the editor may wish to encourage a third-party annotation check before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper asks the right question and is likely right in spirit, but the headline number does not survive contact with the paper's own metric definition. I'd send it to review, but only with the expectation of major revision.\n\nWhat's actually new: a systematic 315-model evaluation of multi-label prediction capability on ImageNetV1/V2, a variable top-k scheme, ASMA as an aggregate, and PatchML as a controlled synthetic diagnostic. The qualitative finding—that a chunk of the ImageNetV2 top-1 drop is due to single-label evaluation ignoring valid alternative labels—is consistent with ReaL, Shankar et al., and Vasudevan et al. The paper is honest about its reliance on existing re-annotation protocols, and the top-model analysis connecting VOLO's token labeling to better multi-label performance is a nice touch.\n\nNow the soft spots. The biggest one is that ASMA, as defined in Section 3.3, cannot produce the numbers in Figures 4–6. The label-wise accuracy is 1/C times the sum over 1000 classes of an indicator matching binary prediction to binary ground truth. If predictions are the obvious indicator of top-k, then every image scores at least about 0.99 because true negatives dominate, and the plotted subgroup accuracies near 0.7 are impossible. Either the formula is wrong or an unstated variant (e.g., precision/recall on positive labels) was used. This is not a nitpick; the 0–6% gap claim is the paper's central quantitative result and it is unverifiable as written.\n\nSecond, the ImageNetV2 ground truth comes from the authors' own Anzaku et al. [14] annotations, with no independent validation or human-agreement analysis. The paper's limitations section even concedes these labels may miss plausible categories, so the main measurement rests on a self-authored, unvalidated label set. The direction of the resulting bias is uncertain, but it is a genuine circularity concern.\n\nFinally, there is no code or commit hash, and the step from variable top-k indices to the binary prediction vector used in ASMA is never specified—exactly the step on which the inconsistency hinges.\n\nBottom line: the qualitative claim is probably correct and worth making, but as written the paper does not support its headline number. A revision that fixes the ASMA definition, specifies prediction binarization, releases runnable code, and addresses the label-source circularity could become a solid contribution. I would accept it for peer review, but I would flag the metric issue prominently and ask for a full rewrite of the results section.","headline":"The paper's qualitative claim is plausible and important, but the ASMA metric as defined cannot produce the reported numbers, so the headline result is currently unverifiable.","tokens_in":18188,"tokens_out":2880,"would_cite":false,"duration_ms":25657,"reading_group":"yes","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The reported ImageNetV2 accuracy drop is substantially a single-label measurement artifact.","keywords":["multi-label evaluation","ImageNetV2","accuracy gap","single-label assumption","multi-label prediction capability","variable top-k","ASMA","PatchML"],"falsifier":"Take the ImageNetV2 images the paper counts as multi-label, have independent annotators list all visible objects, and recompute ASMA and the V1-to-V2 gap; if the gap returns above the 0–6% range or the multi-label prevalence estimate changes materially, the reported artifact is an artifact of the annotation protocol.","tokens_in":17144,"feed_emoji":"🖼️","tokens_out":6168,"duration_ms":52090,"temperature":0.7,"pith_summary":"The paper tries to establish that a large part of the widely cited 11–14% top-1 accuracy drop on ImageNetV2 is a measurement artifact: single-label metrics penalize models for predicting correct secondary labels on images that genuinely contain several objects. Under the multi-label annotations used here, roughly 48% of ImageNetV2 images have more than one valid label, versus about 16% of ImageNetV1 images. When predictions are scored with a variable top-k that credits every valid label, and with the proposed ASMA metric, the average V1-to-V2 gap shrinks from 6–14% to 0–6%. The paper further introduces PatchML, a synthetic dataset that removes contextual co-occurrence cues, and reports that many single-label-trained models still rank several correct objects at the top. If this holds, standard ImageNet leaderboards have been understating the multi-label recognition ability of modern classifiers.","feed_headline":"ImageNetV2 accuracy drop shrinks to 0–6% under multi-label metrics","feed_subtitle":"A study of 315 models finds the famous 11–14 percent gap is mostly a metric artifact, not a generalization failure.","key_machinery":"Two evaluation instruments carry the argument. Variable top-k selects the top k predicted classes for each image, where k equals the number of valid labels in its ground truth, so a model is credited for each correct secondary label. ASMA divides images into subgroups by label count, computes per-label accuracy within each subgroup, and averages those subgroup accuracies, preventing the rare high-label-count images from dominating. PatchML is a synthetic diagnostic dataset built by cropping objects from ImageNet images and pasting them in random non-overlapping grid cells on black canvases, removing co-occurrence and context cues; it isolates whether multi-label recognition is genuine object-level behavior or a byproduct of scene statistics.","core_discovery":"The central discovery is that the ImageNetV2 degradation is substantially a measurement artifact: because ImageNetV2 has a higher incidence of multi-label images than ImageNetV1, top-1 accuracy penalizes models for predicting valid secondary labels. The authors quantify this by constructing variable top-k predictions, where k equals the number of ground-truth labels per image, and by introducing Aggregate Subgroup Model Accuracy (ASMA), which averages per-label accuracy across subgroups of images with the same label count. Across 315 ImageNet-pretrained models, the V1-to-V2 gap falls from 6–14% under top-1 to 0–11% under ReaL accuracy and to 0–6% under ASMA. PatchML, a synthetic dataset of randomly combined cropped objects on blank canvases, further shows that models trained with single-label supervision still rank multiple correct labels near the top, indicating latent multi-label capability beyond what standard metrics report.","pith_inferences":["A testable extension: apply the same variable top-k and ASMA protocol to other single-label benchmarks that likely contain multi-label images, such as Places or iNaturalist, to see whether their apparent domain gaps also narrow.","Beyond the paper's evaluation-only scope, the PatchML construction could be turned into a training curriculum; if models fine-tuned on multi-object composites improve ASMA on natural images, that would confirm the latent-capability story causally.","If the gap reduction is real, earlier conclusions that ImageNet classifiers fail under distribution shift may need to be re-stated as conclusions about single-label evaluation rather than about the models themselves."],"forward_implications":["The common 11–14% ImageNetV2 drop should not be read as pure generalization failure; multi-label-aware metrics cut it to 0–6%.","Models with high top-1 accuracy are not necessarily the best at ranking multiple valid labels, so top-1 leaderboards can misrank models for multi-object scenes.","Training signals that go beyond single-label targets (dense token labels, masked image modeling, distillation from web-scale teachers) predict stronger multi-label capability; scale alone is not enough.","Evaluation protocols for ImageNet should adopt multi-label-aware metrics alongside top-1 to avoid underestimating DNN capabilities."],"supporting_citations":[{"why":"Introduced the ImageNetV2 benchmark and documented the 11–14% top-1 drop that the paper re-explains as an evaluation artifact.","marker":"[4]"},{"why":"Supplied the ReaL multi-label annotations for ImageNetV1 that are used to measure multi-label prevalence and compute ReaL accuracy.","marker":"[5]"},{"why":"Supplied the ImageNetV2 multi-label annotations that underpin the variable top-k and ASMA evaluations.","marker":"[14]"},{"why":"Offered independent estimates of multi-label prevalence in ImageNetV1 and ImageNetV2 that the paper compares against its own analysis.","marker":"[6]"},{"why":"Provided a prior partial explanation of the ImageNetV2 gap via statistical bias, which the paper complements by adding the multi-label mismatch.","marker":"[16]"},{"why":"Sourced the 315 pre-trained models evaluated in the study.","marker":"[26]"}],"fun_headline_variants":["ImageNetV2 accuracy gap is a metric artifact, not real","Single-label benchmarking hides true multi-label skill of AI","Top-1 accuracy misjudges AI vision models, study finds","Multi-label metrics erase the ImageNetV2 accuracy drop","315 models show ImageNetV2 gap is mostly a label artifact"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the list of valid objects used for ImageNetV2 is complete and correct for every image; if those annotations miss valid objects or include spurious ones, the measured gap reduction is an artifact of the annotation protocol.","fun_headline_variants_meta":{"raw":{"variants":["ImageNetV2 accuracy gap is a metric artifact, not real","Single-label benchmarking hides true multi-label skill of AI","Top-1 accuracy misjudges AI vision models, study finds","Multi-label metrics erase the ImageNetV2 accuracy drop","315 models show ImageNetV2 gap is mostly a label artifact"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000451,"raw_usage":{"total_tokens":2323,"prompt_tokens":1051,"completion_tokens":1272,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":667,"completion_tokens_details":{"reasoning_tokens":1196}},"tokens_in":667,"tokens_out":1272,"duration_ms":7609,"temperature":1.0,"reasoning_tokens":1196,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:41:39.613629+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the ImageNetV2 images the paper counts as multi-label, have independent annotators list all visible objects, and recompute ASMA and the V1-to-V2 gap; if the gap returns above the 0–6% range or the multi-label prevalence estimate changes materially, the reported artifact is an artifact of the annotation protocol.","supporting_citations":[{"cited_title":"Recht, R","cited_arxiv_id":null,"evidence_quote":"Introduced the ImageNetV2 benchmark and documented the 11–14% top-1 drop that the paper re-explains as an evaluation artifact."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplied the ImageNetV2 multi-label annotations that underpin the variable top-k and ASMA evaluations."},{"cited_title":"Shankar, R","cited_arxiv_id":null,"evidence_quote":"Offered independent estimates of multi-label prevalence in ImageNetV1 and ImageNetV2 that the paper compares against its own analysis."},{"cited_title":"Engstrom, A","cited_arxiv_id":null,"evidence_quote":"Provided a prior partial explanation of the ImageNetV2 gap via statistical bias, which the paper complements by adding the multi-label mismatch."}],"review_version":1}