{"id":"74280108-21ff-49d5-a307-c4d43df9358e","arxiv_id":"1908.08984","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A new 186,150-image clothing product taxonomy dataset is released, with ResNet-34 and attention sequence-to-sequence benchmarks reaching micro F-scores of 0.92 and 0.90.","lead":"This paper introduces Atlas, a new public dataset of 186,150 e-commerce clothing images labeled with a three-level product taxonomy of 52 leaf categories. It reports benchmark results for image classification and attention-based sequence models, with the best model reaching a micro F-score of 0.92 on the test set.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Train/test split appears image-level, not product-level; crawled listings with multiple images per product can leak the same product into both splits, inflating the reported 0.92 F-score.","rationale":"The reader identified label noise from retailer breadcrumbs and manual taxonomy mapping as the weakest assumption. That is a legitimate concern, but the paper's own text provides stronger evidence for a different data-quality problem: multiple images per product and an image-level random split. Because the dataset was collected by crawling product listings, and because Section 3.3 describes multiple images per listing, the risk of product-level leakage is concrete and grounded in the manuscript. Label noise affects absolute accuracy but may still allow meaningful relative comparison of models; leakage, by contrast, invalidates the benchmark's core purpose of predicting categories for unseen products. The proposed test is straightforward to run with the public dataset and would settle the matter. The verdict remains CONDITIONAL because the dataset may still be useful and the issue fixable, but the headline F-score should not be treated as a reference value until product-level separation is confirmed.","tokens_in":7739,"tokens_out":3411,"duration_ms":38847,"concrete_test":"Download the released Atlas data and recover the source product identity for every image (from product listing URL, image URL, or a product ID in the metadata). Group all images by product and compute how many products have images in both the training and test splits. If the overlap is nonzero, retrain the ResNet-34 benchmark on a product-disjoint split (ensuring no product appears in more than one split) and recompute the micro-F score; a drop of more than 2–3 points would confirm that the reported 0.92 is inflated by image-level leakage.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim—that Atlas is a high-quality benchmark and that ResNet-34 reaches 0.92 micro-F on its test set—requires that the test set measures generalization to new products, not just new images of already-seen products. Section 3.2 describes crawling product listings and extracting images; Section 3.3 explicitly notes that a listing often contains several images of the same product and describes only a zoomed-vs-normal filter, with no mention of keeping one image per product or recording product IDs. Section 6 says the 65/5/30 split was produced by stratified random sampling on the dataset, i.e., at the image level. If two images of the same product land in both training and test sets, the model can exploit near-duplicate appearance rather than learn category structure, so the reported F-score is not a valid estimate of performance on unseen products. The paper only claims in Figure 3 that test images are 'not seen during training or validation'; it does not claim product-level separation. Since the entire value of the benchmark depends on this split, this is a load-bearing unverified assumption that is more directly damaging than the label-noise concern: even if breadcrumb labels are correct, leakage alone can produce an inflated headline number.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Atlas, a clothing product image dataset of 186,150 images organized into a three-level taxonomy with 52 leaf category paths, collected by crawling Indian e-commerce product listings and manually mapping store breadcrumbs to a custom taxonomy. The authors describe a zoomed-vs-normal image filtering step, then present two benchmarks: a ResNet-34 image classifier and an attention-based CNN-LSTM sequence-to-sequence model that predicts category paths. On a 65/5/30 split, they report a micro F-score of 0.92 for the classifier and 0.90 for the sequence model. The dataset, code, and pre-trained models are publicly released.","tokens_in":8022,"tokens_out":4090,"duration_ms":41774,"significance":"If the reported performance is a valid estimate of generalization to new products, Atlas would be a useful, reproducible benchmark for image-based e-commerce product categorization, and the public release of data, code, and training configurations is a concrete strength. The paper also provides a plausible, extensible methodology for building similar taxonomy datasets. However, the central empirical claim depends on two load-bearing assumptions: that the train/test split prevents leakage of the same product across splits, and that the crawled breadcrumb-derived labels are accurate. These are asserted but not demonstrated, so the headline 0.92 F-score should currently be treated as unverified.","major_comments":[{"comment":"The reported benchmark numbers may not measure generalization to new products because the split appears to be image-level, not product-level. Section 6 states that the train/validation/test split was produced by stratified random sampling on the dataset, giving 119,155 training images, 11,147 validation images, and 55,848 test images; Section 3.3 explicitly notes that an e-commerce listing often contains several images of a single product. If multiple images of the same product are distributed across train and test, the model can exploit near-duplicate appearance, and the reported 0.92 micro-F becomes an overestimate of performance on unseen products. The paper does not mention product IDs, deduplication, or a product-level split. Please report whether the splits are grouped by product, and either provide a product-level split or demonstrate that no product appears in both training and test sets.","section":"Section 6, with Section 3.3"},{"comment":"The ground-truth labels are created by manually mapping each retailer's breadcrumb path to the authors' custom taxonomy, but no label-quality measurement is reported. If retailer breadcrumbs are wrong or the manual mapping is inconsistent, the labels are noisy and the F-scores do not measure what they claim. The paper should report an audit of the labeling process, such as inter-annotator agreement on a sample, a manual verification of a random subset, or a quantitative comparison against an independent label source. Without this, the 'high-quality dataset' claim is not yet supported.","section":"Section 3.2"}],"minor_comments":[{"comment":"The text says 'Our dataset contains data corresponding to 52 products' but the rest of the paper consistently describes 52 category paths (leaf nodes), not 52 products. This should be corrected to avoid confusion.","section":"Section 3, first paragraph"},{"comment":"Please define micro-F precisely: is it computed on the leaf-level category, on full path accuracy, or on per-level token predictions? For the sequence model, a path is only correct if all levels are correct, so the metric definition matters for interpreting the 0.92 versus 0.90 comparison.","section":"Section 6"},{"comment":"The results are reported as single numbers without error bars or repeated runs. Given that the two models differ by only two points, a small number of independent runs with mean and standard deviation would make the benchmark comparison more informative.","section":"Section 6"},{"comment":"The zoomed-vs-normal filter is a key data-cleaning step, but the paper does not report the size or construction of the test set used for Table 1, nor how the 6005 normal and 1054 zoomed training images were split. Please clarify whether the filter's test set is disjoint from the final Atlas train/test split.","section":"Section 3.3"},{"comment":"The beam width is said to be chosen 'based on our experiments,' but no sweep or resulting F-scores are reported. Please provide the evaluated beam widths and the corresponding performance, or state that the choice had negligible effect.","section":"Section 5.1"},{"comment":"There is a typo: 'contors' should be 'contours.'","section":"Section 3.3"}],"recommendation":"major_revision","confidential_remarks":"The decisive issue is the image-level versus product-level split. If the authors can re-run the benchmark with a product-level split or otherwise demonstrate no overlap, the paper would be a solid dataset contribution. If they cannot, the headline F-score should be substantially caveated or removed. I would also ask the editor to ensure the revised version includes the label-audit analysis, as the manual breadcrumb mapping is the other main risk to the benchmark's validity."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a useful dataset release, but the headline 0.92 micro-F should not be taken at face value until the train/test split is clarified. The paper appears to split images, not products, and it acknowledges multiple images per product. That could inflate the benchmark.\n\nWhat is actually new: the Atlas dataset—186,150 clothing images with a three-level taxonomy of 52 leaf nodes, collected from Indian e-commerce stores. The cross-retailer taxonomy mapping and the zoomed-vs-normal cleaning pipeline are real work. Public e-commerce product taxonomy datasets are rare, so this is a resource the community can use. The code and pretrained models are promised on GitHub.\n\nThe soft spots, in order. First is the split. Section 3.3 says a listing often has several images for a single product, and the cleaning step only removes zoomed images; there is no mention of keeping one image per product or recording product IDs. Section 6 describes stratified random sampling on the dataset—that is image-level. If the same product appears in both train and test, the 0.92 F-score measures near-duplicate matching, not category generalization. This is load-bearing because the dataset's value as a benchmark depends on it. It may be that the authors did split by product, but they didn't say so. Second, label quality is asserted, not measured. The breadcrumb-to-taxonomy mapping is manual, and there is no human agreement or audit. That matters because the ground truth is the entire point. Third, minor: no error bars, single runs, and the Seq2Seq baseline is trained for only 3 epochs compared to 17 for the ResNet-34. These are not fatal but should be reported.\n\nThe citation pattern looks fine; the closest work is cited. I did not verify the GitHub release.\n\nIf the split is product-level, this is a solid dataset paper. If not, the benchmark numbers are not reference values. Either way, the dataset itself may still be useful, but users should be cautious.\n\nI would send this to peer review rather than desk-reject, because the resource is real and the concerns are addressable. But acceptance should require either a product-level split or clear evidence that leakage is absent, plus some measurement of label quality.","headline":"A useful dataset release whose benchmark number is questionable until the split is clarified.","tokens_in":8484,"tokens_out":4460,"would_cite":false,"duration_ms":41931,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces Atlas, a public dataset of 186,150 clothing images with a 52-leaf taxonomy, and benchmarks image-only categorization at a 0.92 micro F-score.","keywords":["product categorization","e-commerce taxonomy","clothing images","image classification","sequence-to-sequence","attention mechanism","dataset benchmark","hierarchical classification"],"falsifier":"Take a random sample of Atlas test images, have independent annotators assign the 52 leaf categories without seeing the stored labels, and measure agreement against the dataset labels; low agreement would mean the reported F-scores partly reflect label noise rather than categorization accuracy.","tokens_in":7593,"feed_emoji":"👗","tokens_out":6063,"duration_ms":61823,"temperature":0.7,"pith_summary":"This paper introduces Atlas, a public dataset of 186,150 clothing product images labeled with category paths in a three-level taxonomy ending in 52 leaf nodes. It argues that image-based product categorization can be benchmarked reproducibly on this dataset, and reports that a ResNet-34 classifier reaches a micro F-score of 0.92 on the test split, with an attention-based sequence model at 0.90. The contribution is as much the collection methodology as the numbers: crawling retailer listings, manually mapping store breadcrumbs to a unified taxonomy, and filtering out zoomed or detail images with a small CNN trained on visually inspected examples. A sympathetic reader would care because public, real-world product taxonomy data is scarce and stores use incompatible taxonomies, so a shared benchmark lets competing approaches be compared on the same categories.","feed_headline":"New clothing dataset reaches 0.92 F-score on 52 categories","feed_subtitle":"Atlas gives e-commerce researchers a shared testbed for image-based product categorization.","key_machinery":"The load-bearing object is the Atlas dataset itself: 186,150 images with 52 category paths under a three-level taxonomy built by clustering similar retailer breadcrumbs. Two mechanisms carry the benchmark. First, a two-class CNN filter, called the Zoomed Versus Normal model, is trained on visually inspected thumbnails to remove close-up or cropped images that would otherwise corrupt the dataset. Second, the benchmark models: a ResNet-34 classifier with adaptive pooling, and an attention-based sequence-to-sequence model in which a ResNet-101 encoder produces a $14 \\times 14 \\times 2048$ feature map and an LSTM decoder with soft attention predicts the category path as a token sequence from a start marker to an end marker, using beam search at inference.","core_discovery":"The central claim is that Atlas is a clean, usable dataset for e-commerce clothing categorization and that standard visual models perform well on it: the ResNet-34 image classifier obtains a micro F-score of 0.92 over 52 leaf categories, while the attention-based encoder-decoder, which predicts the category path level by level from a ResNet-101 image encoding, obtains 0.90. The paper also claims the attention model offers an explanation for predictions by focusing on different image regions, such as the face for gender and the garment area for later levels, and that it can propose new category paths, though only 2 of 5 generated paths were valid, so manual inspection is needed. This establishes a baseline rather than a solution; the authors position Atlas as an invitation for the community to improve on these numbers.","pith_inferences":["If the labels survive independent audit, the 0.92 image-only baseline suggests the ceiling for 52 well-separated clothing categories is already close to practical accuracy, and further gains should come from difficult confusable pairs rather than model capacity.","Constraining the decoder to output only legal child categories would directly test whether invalid path generation is avoidable; this is measurable against the paper's reported 2-of-5 validity.","Because the taxonomy includes Indian ethnic-wear categories, Atlas can serve as a domain-shift test for models trained on Western fashion images.","Product titles and prices were collected alongside images but are not used by the benchmark models; a multimodal model using both could be compared against the 0.92 image-only number."],"forward_implications":["Future methods can be compared directly on fixed 65/5/30 splits, since Atlas fixes the taxonomy and the test set.","The zoomed-versus-normal filtering step shows that a small binary CNN can remove unrepresentative product images with an average F-score of 0.98, a reusable cleaning step.","The attention-based sequence model reaches a 0.90 micro F-score and produces attention maps that indicate which image region supports each category level.","The sequence model generates novel category paths, but only 2 of 5 it produced were valid, so manual review is required before enriching the taxonomy."],"supporting_citations":[{"why":"Supplies the ResNet architecture used as the backbone for both benchmark models.","marker":"[7]"},{"why":"Supplies the sequence-to-sequence formulation for category-path prediction with an LSTM decoder and beam search.","marker":"[13]"},{"why":"Provides the claim that sequence models can generate new category paths, which the paper evaluates and finds mostly invalid.","marker":"[14]"},{"why":"Provides a public product-classification challenge dataset that lacks images and uses numeric category IDs, motivating Atlas.","marker":"[15]"},{"why":"Provides a large e-commerce product dataset with text features used by earlier categorization work, setting the context for image-based categorization.","marker":"[17]"},{"why":"Supplies the train, validation, and test split proportions used for the benchmark.","marker":"[9]"},{"why":"Supplies the attention-over-image mechanism that the caption-style decoder is built from.","marker":"[22]"}],"fun_headline_variants":["Atlas: 186k images, 52 clothing categories, 0.92 Micro-F1","E-commerce clothing classification benchmark: Atlas hits 0.92 F1","Atlas dataset: open clothing taxonomy with 52 leaf nodes, 0.92 F1","0.92 Micro-F1 on 52 clothing categories: Atlas benchmark released","New Atlas dataset for clothing categorization: baselines at 0.92 F1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The dataset labels are derived from retailer website breadcrumbs that were manually mapped to the paper's taxonomy; if any mapped breadcrumb is wrong or the mapping is inconsistent across stores, the labels are noisy and the reported F-scores no longer measure what they appear to.","fun_headline_variants_meta":{"raw":{"variants":["Atlas: 186k images, 52 clothing categories, 0.92 Micro-F1","E-commerce clothing classification benchmark: Atlas hits 0.92 F1","Atlas dataset: open clothing taxonomy with 52 leaf nodes, 0.92 F1","0.92 Micro-F1 on 52 clothing categories: Atlas benchmark released","New Atlas dataset for clothing categorization: baselines at 0.92 F1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000789,"raw_usage":{"total_tokens":3463,"prompt_tokens":915,"completion_tokens":2548,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":531,"completion_tokens_details":{"reasoning_tokens":2438}},"tokens_in":531,"tokens_out":2548,"duration_ms":16998,"temperature":1.0,"reasoning_tokens":2438,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:47:03.741080+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of Atlas test images, have independent annotators assign the 52 leaf categories without seeing the stored labels, and measure agreement against the dataset labels; low agreement would mean the reported F-scores partly reflect label noise rather than categorization accuracy.","supporting_citations":[{"cited_title":"In: eCOM@SIGIR","cited_arxiv_id":null,"evidence_quote":"Supplies the sequence-to-sequence formulation for category-path prediction with an LSTM decoder and beam search."},{"cited_title":"Don't Classify, Translate: Multi-Level E-Commerce Product Categorization Via Machine Translation","cited_arxiv_id":"1812.05774","evidence_quote":"Provides the claim that sequence models can generate new category paths, which the paper evaluates and finds mostly invalid."},{"cited_title":"In: eCOM@SIGIR","cited_arxiv_id":null,"evidence_quote":"Provides a public product-classification challenge dataset that lacks images and uses numeric category IDs, motivating Atlas."},{"cited_title":"In: Proceedings of the 38th Interna- tional ACM SIGIR Conference on Research and Development in Informa- tion Retrieval","cited_arxiv_id":null,"evidence_quote":"Provides a large e-commerce product dataset with text features used by earlier categorization work, setting the context for image-based categorization."},{"cited_title":"In: Proceedings of the IEEE conference on computer vision and pattern recognition","cited_arxiv_id":null,"evidence_quote":"Supplies the train, validation, and test split proportions used for the benchmark."}],"review_version":1}