{"id":"55bef1e3-883c-40c1-9b68-554aad098673","arxiv_id":"2506.14418","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"Images with rare compositional attributes are classified less accurately; upweighting them during data augmentation improves accuracy on those images.","lead":"This paper proposes that images contain rare combinations of visual attributes like color, texture, and shape, and that classifiers handle images with rarer attribute combinations less accurately. The authors build a CLIP-based attribute dictionary to score each image's rarity, reweight training samples so rare images are sampled more often during augmentation, and report accuracy gains on 12 datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 2's 'all' column cannot be reproduced from the reported subset accuracies under the stated 40/30/30 partition; some rows exceed the maximum possible weighted average, so the claimed universal gains are unsupported.","rationale":"The reader recommended REJECT based on several weaknesses. I agree with rejection, but the single most load-bearing flaw is the internal inconsistency of Table 2, which is the entire quantitative basis for the method's improvement. The subset partition is explicitly defined, so the 'all' column is overdetermined; multiple rows violate the weighted-average identity, and at least two reported 'all' values exceed the maximum attainable given the subset accuracies. This makes the headline empirical claim impossible to verify from the paper as written. The dictionary-transfer concern identified by the reader is also real, but it is a threat to construct validity; the table inconsistency is a demonstrated arithmetic failure that directly invalidates the reported results. Since the reader already rejected, my verdict is unchanged. I mark partial agreement: the reader listed 'internal inconsistencies in Table 2' in the rationale but selected the unvalidated attribute dictionary as the weakest assumption; I consider the table's arithmetic inconsistencies to be more load-bearing because they are objective and fatal to the reported evidence. A single re-run or arithmetic check on the DTD row would settle whether the table is simply misprinted or the experiments were not performed as claimed.","tokens_in":15209,"tokens_out":4626,"duration_ms":42506,"concrete_test":"Reproduce Table 2 from the raw logs: for each row, verify that all equals 0.4×high + 0.3×middle + 0.3×low, or correct the partition definition in §5.3. At minimum, rerun the DTD ResNeXt50 CutMix condition and confirm whether the reported 'all' accuracy of 99.61 can be obtained from the test set; if the true weighted average is ~94.5 or lower, Section 5.5's universal-improvement claim fails on that row. Also re-extract the 'all' entries for all 36 rows and check for arithmetic consistency; any row that cannot be a weighted average of its own subsets should be flagged.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central evidence for the method is Table 2, which reports low/middle/high/all accuracies with §5.3 defining High as top 40%, Middle next 30%, Low 30%. Under that definition, the 'all' column must equal 0.4*High + 0.3*Middle + 0.3*Low. Many rows fail this identity by large margins. Example: CIFAR-100 ResNet18 CutMix gives (0.3×68.31)+(0.3×49.65)+(0.4×43.05)=52.61, but the table reports all=54.57. More decisively, DTD ResNeXt50 CutMix reports all=99.61 while its three subsets are 95.22/94.36/86.32; the maximum possible weighted average is 94.53, so 99.61 is arithmetically impossible. Similar impossible entries appear (DTD FMix ResNeXt50: all=99.56 with high=70.89; max ≈88.36). Since Section 5.5's claim that 'on all datasets, the performance improved' is derived directly from these numbers, the reported improvements—some exceeding 10 points—are not supported by the tabulated data. This is not a question of ideal experimental practice or a missing baseline; it indicates the quantitative results as printed cannot be relied on. The dictionary-validity concern is secondary: even if CAS were perfectly measured, the table itself must be internally consistent to support the claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies intra-class attribute imbalance in image classification. The authors define 20 primary attribute categories and 300+ secondary attributes, build a CLIP-based visual attribute dictionary from ImageNet-21k, and use it to annotate 12 benchmark datasets. They propose Compositional Attribute Scarcity (CAS), defined as the sum of within-class frequency ranks of an image's 20 secondary attributes, and report that higher CAS is associated with lower classification accuracy. To mitigate this, they reweight the training sampler by a power of each sample's CAS (r_i^b, with b=1.2) and combine this weighted sampler with CutMix, FMix, and SaliencyMix. Section 5 presents per-subset accuracies (low/middle/high CAS) and overall accuracy, claiming universal improvements across all 12 datasets and three augmentation methods.","tokens_in":15669,"tokens_out":10045,"duration_ms":94296,"significance":"If the reported results were correct, the paper would make a useful contribution: a scalable, model-agnostic way to characterize intra-class attribute imbalance and a zero-overhead sampling modification that consistently improves strong augmentations on a broad range of tasks. The framework is clearly motivated, the experimental scope is wide, and the idea of guiding data augmentation by compositional rarity is interesting. However, the paper's central quantitative evidence is internally inconsistent, the CLIP-based dictionary is not validated, and the method is not compared with generic reweighting baselines. The paper also provides no code, data, or error bars. As a result, the claimed empirical findings are not established in this version.","major_comments":[{"comment":"The 'all' accuracy must equal 0.4×high + 0.3×middle + 0.3×low under the stated 40/30/30 partition, but this identity fails for many rows. For example, DTD with ResNeXt-50 and CutMix reports low=95.22, middle=94.36, high=86.32, yet all=99.61, which is larger than every subset accuracy and therefore arithmetically impossible. CIFAR-100 with ResNet-18 and CutMix reports low=68.31, middle=49.65, high=43.05, giving a weighted average of 52.61, while the table says all=54.57. Because Section 5.5's conclusion that performance improved on all datasets is read directly from these numbers, the central empirical claim of the paper is not supported by the tabulated data.","section":"Table 2, §5.3"},{"comment":"The paper does not state whether the within-class frequency ranks used to define CAS are computed on the training split, the test split, or the entire dataset. If the test set contributes to the rank statistics, then both the test-set subset definition and the training sampler are informed by test-set statistics, which is a leakage concern. The authors must specify the split used and, ideally, recompute all results with ranks estimated from training data only.","section":"§3.4, §5.3"},{"comment":"The CLIP-based attribute dictionary is constructed once on ImageNet-21k and then applied unmodified to all 12 datasets, including fine-grained domains such as Stanford Cars, CUB-200-2011, Oxford-102 Flower, and DTD. No validation against human attribute labels, no per-dataset transfer check, and no measure of annotation noise are provided. Since every CAS score and every subset in Table 2 depends on this dictionary, the whole analysis is unreliable until the dictionary is shown to be accurate on the target domains.","section":"§3.2"},{"comment":"The training sampler upweights samples by the same CAS scores that define the low/middle/high test subsets, so an increase in high-CAS accuracy is partly a mechanical consequence of training more often on high-CAS samples. The paper does not compare with generic reweighting baselines such as class-balanced sampling, inverse-frequency sampling, or loss-based hard-example mining. Without such comparisons, the claimed CAS-specific benefit is not established.","section":"§4.1, §5.5"},{"comment":"The power parameter b=1.2 is selected on CIFAR-100 and ImageNet, which are also part of the main evaluation, and all results are point estimates with no standard deviations or repeated runs. Given that Table 2 already contains arithmetically impossible entries, the reported gains—up to 10.79 points on Stanford Cars—cannot be evaluated for statistical significance.","section":"§5.4, §5.5"}],"minor_comments":[{"comment":"Algorithm 1 uses r_i^β in the weight update, while Section 4.1 defines the transformation as r_i^b; the notation should be unified. Additionally, the algorithm input line says 'Combination rarity scores' rather than 'compositional attribute scarcity scores.'","section":"Algorithm 1, §4.1"},{"comment":"The text says the analysis uses ResNet-18 and ResNet-50, while the Figure 2 caption mentions ResNeXt-50; the architecture names should be made consistent throughout.","section":"§3.3, Figure 2"},{"comment":"Stanford Dogs is described as containing 20 dog breeds, but the actual dataset contains 120 breeds; this description should be corrected.","section":"§5.1"},{"comment":"The description of the horizontal axis as '10 evenly divided intervals' does not explain how CAS values are mapped to intervals, and the right vertical axis is not labeled; the figure should be self-contained.","section":"Figure 4"},{"comment":"The paper does not release the attribute dictionary, per-image CAS scores, or code, so the numbers in Figures 2–4 and Table 2 cannot be reproduced or independently checked.","section":"Reproducibility"},{"comment":"The abstract uses 'first-level and second-level attributes' while the body uses 'primary and secondary attributes'; terminology should be made consistent.","section":"Abstract and §3.1"}],"recommendation":"reject","confidential_remarks":"The arithmetic inconsistencies in Table 2 are severe enough that I cannot support major revision: the authors would need to rerun every experiment and re-verify all numbers, and the corrected results could change the paper's conclusions. If a substantially revised version is submitted, it should include the attribute dictionary and code, validation of the CLIP annotations on target domains, a clear statement of which data split is used to compute CAS ranks, and comparisons with generic reweighting baselines."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe short version: this paper has a genuinely interesting measurement idea—a CLIP-based visual attribute dictionary and a Compositional Attribute Scarcity (CAS) score that quantifies intra-class rarity—but the reported evidence for it is not internally consistent, and the central Table 2 cannot be trusted as printed. The idea is salvageable; the numbers need to be redone.\n\nWhat's new and good: Prior intra-class imbalance work is mostly qualitative or domain-specific. The paper is the first I know of to propose a general, automatic attribute taxonomy (20 primary, 300+ secondary) and to define a per-sample CAS score as the sum of within-class frequency ranks. That gives the long-tailed community a concrete per-sample difficulty signal that doesn't require a trained model's confidence, so it is more stable than confidence-based measures. The proposed fix is simple—reweight the sampler by r_i^b—and it integrates with CutMix, FMix, and SaliencyMix without new overhead. The experiments span 12 datasets and two backbones, which is a serious grid.\n\nSoft spots, in order of severity. First, the arithmetic. The paper defines the test subsets as 40/30/30 high/middle/low, so the \"all\" column should equal 0.4*high + 0.3*middle + 0.3*low. Many rows do not satisfy that identity, and some cannot possibly satisfy it—for example, DTD ResNeXt50 CutMix reports all=99.61 while the weighted maximum from its own subsets is 94.53. That is not a rounding issue. Section 5.5's claim that performance improved on all datasets is built on these numbers, so the headline result is unsupported as printed. Second, there are no error bars or repeated runs anywhere; with gains often under one point, that matters. Third, b=1.2 is selected on CIFAR-100 and ImageNet, and the same two datasets appear in the main table, so the tuning is not independent for two of the twelve datasets. Fourth, the attribute dictionary is built on ImageNet-21k and applied to fine-grained domains with no validation against human-labeled attributes; if the CLIP vocabulary is off for cars or birds, the CAS scores for those datasets are off too. On the circularity worry: reweighting by the same score used to define the test subsets is a legitimate concern, but I would call it secondary—the mechanism (sampling rare samples more often) is a plausible explanation for high-CAS gains; the table inconsistency is the disqualifying issue.\n\nBottom line: this is a paper for the long-tailed learning and data-centric AI crowd. The measurement idea deserves a serious look, but as submitted the main quantitative claim is not reproducible from its own table. I would send it to review only after the authors fix Table 2 and add proper uncertainty estimates and independent tuning—the idea is worth a referee's time, the current version is not acceptable.","headline":"The CAS attribute-scarcity measurement is a genuinely useful idea, but Table 2's internal arithmetic is broken, so the reported gains are unsupported as printed.","tokens_in":16046,"tokens_out":2469,"would_cite":false,"duration_ms":25005,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Image classifiers fail systematically on rare attribute combinations, and a scarcity-weighted sampler fixes most of the gap.","keywords":["compositional attribute imbalance","visual attribute dictionary","CLIP","attribute scarcity","data augmentation","long-tail classification","sampling strategy","image classification"],"falsifier":"Take one dataset such as CIFAR-100 or a fine-grained car dataset, have human annotators label the same 20 attributes on a sample of test images, recompute CAS from those labels, and re-plot accuracy by CAS. If the accuracy decline disappears or reverses, the central claim fails; if it survives, the CLIP-based measurement is not the source of the effect.","tokens_in":15036,"feed_emoji":"⚖️","tokens_out":9984,"duration_ms":89759,"temperature":0.7,"pith_summary":"Vision models are usually trained to balance classes, but this paper argues that a second, hidden imbalance matters more: within a single class, rare combinations of visual attributes such as a white dog or a striped car are learned much worse than common ones. To measure this, the authors use CLIP to build a visual attribute dictionary and assign each image 20 attribute labels, then define Compositional Attribute Scarcity (CAS) as the sum of the within-class frequency ranks of those attributes. Across 12 datasets, accuracy falls steadily as CAS rises, with the worst accuracy concentrated on the rarest attribute combinations. The proposed remedy is a pure sampling change: draw each image with probability proportional to its CAS raised to the power 1.2, and feed those batches into standard augmentation methods. The paper reports accuracy gains on every dataset, with the largest gains exactly on the high-CAS samples that were previously worst.","feed_headline":"Rare attributes, not rare classes, explain image accuracy gaps","feed_subtitle":"A CLIP-built attribute dictionary plus a CAS-weighted sampler lifts accuracy, especially on high-scarcity images.","key_machinery":"The load-bearing object is the Compositional Attribute Scarcity score (CAS), defined as the sum, over an image's 20 CLIP-assigned secondary attributes, of the within-class frequency ranks of those attributes (rank 1 means most frequent). Constructing it requires the paper's visual attribute dictionary: CLIP text embeddings of phrases such as \"the photo is Brown\" are matched to image embeddings to build a key-value dictionary, and each image's embedding is later matched to the nearest key to retrieve its attributes. CAS converts that dictionary into one scalar per sample, and the proposed fix then uses $p_i = r_i^b / \\sum_k r_k^b$ with $b=1.2$ as the sampling probability, feeding rare-attribute images more often into mixup-style data augmentation.","core_discovery":"The paper's central claim is that image classifiers suffer from a distinct, measurable form of data imbalance that is not captured by class counts: compositional attribute imbalance. Using a CLIP-based visual attribute dictionary, each image is annotated with 20 secondary attributes, one per primary attribute category such as color, material, or shape, and the within-class frequency rank of each attribute gives its scarcity. The Compositional Attribute Scarcity (CAS) of an image is the sum of those 20 ranks, so a higher CAS marks an image whose attribute combination is rare inside its own class. Across 12 benchmark datasets and two architectures, the paper reports that accuracy drops monotonically as CAS increases, with the steepest losses on the most scarce samples. It then shows that upsampling high-CAS images, using sampling probability proportional to $r_i^b$ with $b=1.2$ before feeding batches to CutMix, FMix, or SaliencyMix, improves accuracy on every dataset tested, with the largest gains on exactly the high-CAS samples that were previously worst.","pith_inferences":["Beyond the paper: CAS could be computed from any pretrained attribute vocabulary, so the same weighting idea could transfer to region-level tasks such as detection or segmentation if per-region attributes were available.","Beyond the paper: replacing the sampler with a CAS-weighted loss would separate the effect of seeing rare images more often from the effect of larger gradients on them.","Beyond the paper: long-tailed methods that rebalance classes could be composed with CAS rebalancing to handle two independent axes of skew.","Beyond the paper: the accuracy-CAS curve is measured with the same CLIP annotations that define CAS, so a control with randomized CAS assignments is needed to rule out annotation noise as the source of the trend."],"forward_implications":["Training pipelines that only balance classes still leave an accuracy gap on images with rare attribute combinations, so class-level balance is not sufficient for fair performance.","Swapping in the CAS-weighted sampler, with no extra parameters and no inference cost, improves overall accuracy on all 12 datasets tested and improves high-CAS accuracy the most.","The largest gains appear in fine-grained domains such as pets, cars, and flowers, where attribute combinations are the main discriminative signal.","The sampler composes with CutMix, FMix, and SaliencyMix, indicating that the benefit is not tied to one augmentation recipe.","Because CAS scores are computed offline, the method can be added to existing long-tail training frameworks without retraining an attribute model."],"supporting_citations":[{"why":"Supplies the CLIP text-image model that builds and queries the visual attribute dictionary.","marker":"Radford et al., 2021"},{"why":"One source of the 20-primary and 300-plus-secondary attribute taxonomy the dictionary encodes.","marker":"Zhong et al., 2021"},{"why":"The other source of the attribute taxonomy used to define first- and second-level attributes.","marker":"Zhang et al., 2024"},{"why":"Provides ImageNet, the corpus on which the dictionary is built and one of the 12 evaluation datasets.","marker":"Deng et al., 2009"},{"why":"CIFAR-100 is one of the 12 benchmarks and is used to tune the power parameter b.","marker":"Krizhevsky et al., 2009"},{"why":"A fine-grained benchmark where the paper reports some of its largest accuracy gains.","marker":"Khosla et al., 2011"},{"why":"A fine-grained car benchmark supporting the claim that gains are strongest where attribute combinations are diagnostic.","marker":"Krause et al., 2013"},{"why":"A fine-grained flower benchmark used to support the generalization claim.","marker":"Nilsback & Zisserman, 2008"},{"why":"Supplies the data-augmentation protocol and hyperparameter settings used across experiments.","marker":"Qin et al., 2024"},{"why":"Earlier qualitative evidence of intra-class attribute imbalance that this paper makes quantitative.","marker":"Tang et al., 2022"}],"fun_headline_variants":["Attribute scarcity, not class rarity, dictates image model accuracy","Rare attribute combos, not rare classes, cause accuracy dips","Upsampling scarce attribute images narrows the accuracy gap","CAS-weighting plus augmentations fixes hidden attribute imbalance","Attribute rarity, not category rarity, is the real long-tail culprit"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The analysis assumes the CLIP attribute dictionary labels images accurately on all twelve datasets, including fine-grained ones, yet the paper never checks those automatic labels against human judgment.","fun_headline_variants_meta":{"raw":{"variants":["Attribute scarcity, not class rarity, dictates image model accuracy","Rare attribute combos, not rare classes, cause accuracy dips","Upsampling scarce attribute images narrows the accuracy gap","CAS-weighting plus augmentations fixes hidden attribute imbalance","Attribute rarity, not category rarity, is the real long-tail culprit"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000543,"raw_usage":{"total_tokens":2587,"prompt_tokens":916,"completion_tokens":1671,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":532,"completion_tokens_details":{"reasoning_tokens":1587}},"tokens_in":532,"tokens_out":1671,"duration_ms":10961,"temperature":1.0,"reasoning_tokens":1587,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:52:07.307171+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take one dataset such as CIFAR-100 or a fine-grained car dataset, have human annotators label the same 20 attributes on a sample of test images, recompute CAS from those labels, and re-plot accuracy by CAS. If the accuracy decline disappears or reverses, the central claim fails; if it survives, the CLIP-based measurement is not the source of the effect.","supporting_citations":[{"cited_title":"Improving calibration for long-tailed recognition","cited_arxiv_id":null,"evidence_quote":"One source of the 20-primary and 300-plus-secondary attribute taxonomy the dictionary encodes."},{"cited_title":"Concept-guided prompt learning for generalization in vision-language models","cited_arxiv_id":null,"evidence_quote":"The other source of the attribute taxonomy used to define first- and second-level attributes."},{"cited_title":"Novel dataset for fine-grained image categorization: Stanford dogs","cited_arxiv_id":null,"evidence_quote":"A fine-grained benchmark where the paper reports some of its largest accuracy gains."},{"cited_title":"A., and Gao, X","cited_arxiv_id":null,"evidence_quote":"Supplies the data-augmentation protocol and hyperparameter settings used across experiments."},{"cited_title":"Invariant feature learning for generalized long-tailed classification","cited_arxiv_id":null,"evidence_quote":"Earlier qualitative evidence of intra-class attribute imbalance that this paper makes quantitative."}],"review_version":1}