Can GPT-4o mini and Gemini 2.0 Flash Predict Fine-Grained Fashion Product Attributes? A Zero-Shot Analysis

Kunal Sonalkar; Shubham Shukla

REVIEW 4 major objections 7 minor 12 references

Can GPT-4o mini and Gemini 2.0 Flash Predict Fine-Grained Fashion Product Attributes? A Zero-Shot Analysis

T0 review · 4 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that in a zero-shot, image-only setting, Gemini 2.0 Flash outperforms GPT-4o-mini on fine-grained fashion attribute extraction by a wide margin while being cheaper and faster.

desk verdict A clean, small-scale benchmark with a plausible ranking and a real statistical soft spot: useful for practitioners, but the headline F1 gap is a single-run, single-subset measurement. read the letter →

arxiv 2507.09950 v2 pith:EJBMSZI3 submitted 2025-07-14 cs.CV cs.AI

Shubham Shukla , Kunal Sonalkar This is my paper · ORCID

classification cs.CVcs.AI

keywords zero-shotevaluationvision-languagemodelsfashionattributeextractionproductattributionDeepFashion-MultiModalGPT-4o-miniGemini2.0Flashe-commercecatalog

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether two cheap, fast vision-language models can classify fine-grained fashion attributes from a single product image without any training. It claims that Gemini 2.0 Flash is the stronger zero-shot attribute extractor, scoring 56.79% macro F1 versus 43.28% for GPT-4o-mini on a 1,000-image stratified subset of DeepFashion-MultiModal, while also costing about 12.5% less and running about 24% faster. It further claims that deterministic settings (temperature 0) improve both models by 6–7 points, and that both models handle prominent attributes like hats but miss subtle ones like necklines and waist accessories. The finding matters because it suggests lightweight multimodal LLMs can support human-in-the-loop catalog tagging in e-commerce today, without resolving the need for fine-tuning on subtle attributes.

What carries the argument

The central object is a structured output-format prompt that turns the image-understanding task into 18 independent multiclass classification problems, requiring the model to return a numerical array. The prompt, combined with the human-annotated DeepFashion-MultiModal labels, the OpenRouter API gateway, and macro-F1 evaluation, carries the experiment: it isolates visual understanding and makes model outputs directly comparable to ground-truth integers.

What would settle it

Run the identical prompt on the full DeepFashion-MultiModal dataset, or on several fresh stratified 1,000-image samples, and compare macro F1 per attribute; if the gap narrows to a few points or reverses, the subset's representativeness is the load-bearing assumption, and if it persists, the finding is stable.

Watch

Extended reading notes

Core claim

The central finding is that Gemini 2.0 Flash outperforms GPT-4o-mini as a zero-shot, image-only predictor of fine-grained fashion attributes across 18 categories, reaching a macro F1 of 56.79% versus 43.28% in the deterministic setting. The better model is also the cheaper and faster one, costing roughly 12.5% less and running about 24% faster for a batch of 1,000 images. Both models improve by 6–7 percentage points when temperature is lowered from 1 to 0 and top p from 1 to 0.3, and both succeed on visually prominent attributes such as hats while struggling with subtle details such as neckline style and waist accessories.

Load-bearing premise

The paper assumes its 1,000-image subset, stratified by primary product type, represents the full dataset's distribution of all 18 attributes; if that representation is skewed, the scores and the 13.5-point gap could shift.

Editorial extensions

If this is right

E-commerce teams using these models can expect Gemini 2.0 Flash to tag more catalog images correctly per dollar and per minute than GPT-4o-mini without any labeled training data.
Setting temperature to 0 and top p to 0.3 should be the default for structured attribute classification, since it improved F1 by 6–7 points for both models.
Prominent, well-defined attributes like hats and sleeve length are already near production-useful, while subtle attributes like neckline and waist accessories still need human review or fine-tuning.
Because the superior model is also the cheaper and faster one, there is no accuracy–cost trade-off in this model pair, removing a common deployment objection.
The most immediately deployable use is a human-in-the-loop pipeline that routes easy attributes to the model and hard ones to human annotators.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because both models are closed API systems, their rankings could shift with model versions, so this comparison should be treated as a point-in-time measurement rather than a stable property of the two model families.
The image-only design probably understates what these models would do in real e-commerce, where product text is usually available; adding text or few-shot examples could raise both scores and possibly change the gap.
A controlled test that removes easy, high-frequency categories or rebalances the 18 classes would reveal whether the overall gap is driven by a few attributes or by consistent baseline strength.
The 1,000-image subset means the reported figures are estimates; running the same protocol on the full dataset would be the direct check on stability.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a competent, honest zero-shot benchmark comparing two cheap closed VLMs on fine-grained fashion attribute extraction. The main result—Gemini 2.0 Flash beats GPT-4o-mini by about 13.5 F1 points in the deterministic setting—is plausible and useful, but the precise numbers are provisional because everything rests on one 1,000-image subset with no error bars.

What's new: a head-to-head of these specific model versions on DeepFashion-MultiModal, image-only, zero-shot, with a structured 18-attribute prompt, plus a temperature comparison and cost/latency figures. That is genuinely actionable for e-commerce teams. The paper also does several things right: image-only input, explicit 'NA' handling, per-attribute tables in the appendix, and a limitations section that acknowledges single-dataset, subset, zero-shot-only, and no fine-tuned baselines. The deterministic-vs-creative finding is sensible and probably robust.

Soft spots. The biggest is statistical. The main comparison is a single run per condition. At temperature 0 with top_p 0.3, sampling variance is low, so the gap is less likely to be noise, but it is still one sample, and the temperature=1 experiment has real variance that is not quantified. No confidence intervals, no repeated runs, no significance test, yet the paper uses the word 'significant.' The stratification was by primary product type, not by the 18 attributes, and the subset's label marginals are not compared to the full dataset. The 13.5-point gap could shift under a different sample, though I'd be surprised if the ranking reversed. A majority-class or 'always NA' baseline would help interpret the absolute F1 numbers, since some attributes are imbalanced. Cost and latency are single-batch measurements, so treat 12.5% cheaper and 24% faster as approximate. Minor issues: the DeepFashion-MultiModal citation points to Text2Human instead of the dataset paper, and there's an author name typo in the PDF (Shubbham). For a benchmark paper, citations matter.

Bottom line: this is a useful measurement, not a methodological breakthrough. It deserves referee time—fix the statistical reporting (a few repeated runs at temperature 1), add a majority-class baseline, and verify the subset distribution against the full set. With those changes it would be a solid reference for practitioners picking a low-cost model.

Referee Report

4 major / 7 minor

Summary. The paper presents a zero-shot, image-only evaluation of two cost-efficient multimodal LLMs, GPT-4o-mini and Gemini 2.0 Flash, on fine-grained fashion attribute extraction. Using a 1,000-image stratified subset of the DeepFashion-MultiModal dataset, the authors frame attribute recognition as 18 multiclass classification problems (12 shape, 3 color-pattern, 3 fabric-type) and report macro F1 scores for two settings: temperature=1/top_p=1 and temperature=0/top_p=0.3. The main claims are that Gemini 2.0 Flash outperforms GPT-4o-mini in the deterministic setting (56.79% vs. 43.28% macro F1), that deterministic settings improve both models by 6-7 percentage points, and that Gemini is roughly 12.5% cheaper and 24% faster, so there is no accuracy-cost trade-off. The paper includes per-attribute tables in the appendix, a full prompt in Table 8, and a brief limitations section.

Significance. If the central claims hold, the paper provides a useful, practically oriented benchmark for deploying lightweight vision-language models in e-commerce cataloging, with per-attribute breakdowns that show where zero-shot models succeed (Hat, Sleeve Length) and fail (Neckline, Waist Accessories). The study uses a public human-annotated dataset, reports a reproducible prompt, and makes explicit its scope and limitations. However, the quantitative conclusions rest on a single 1,000-image subset and single-run measurements, with no confidence intervals, significance tests, or majority-class baselines; these omissions make the specific numeric claims provisional rather than definitive. The methodological novelty is modest, but the comparison is a reasonable contribution for an applied venue if the statistical robustness concerns are addressed.

major comments (4)

[Section 5.2 and Tables 4-6] The central quantitative claim, a 13.5-point macro F1 gap between Gemini 2.0 Flash and GPT-4o-mini, rests on a single 1,000-image subset stratified only by primary product type. Stratification by product type does not guarantee representativeness of the 18 attribute label distributions, because attributes such as Neckline and Waist Accessories vary within product type. The paper does not report per-stratum sizes, nor does it compare the subset's attribute label marginals against the full 11,000-image dataset. An unlucky sample could shift both absolute F1 scores and the reported gap. Please report stratum sizes and label distribution checks, and provide bootstrap confidence intervals over images for the macro F1 estimates.
[Sections 6.1-6.2 and Tables 3, 6] All reported macro F1 scores come from a single run per model and configuration. At temperature=1, sampling variance is expected to be high; at temperature=0, API-level nondeterminism and subset sampling still introduce uncertainty. The claims that deterministic settings improve F1 by 6-7 points and that Gemini leads by about 13.5 points are thus presented without error bars or significance testing. Please report repeated runs for at least the stochastic setting, or bootstrap over sampled images, and state whether the observed differences are statistically distinguishable from noise.
[Section 6.3 and Table 7] The cost and latency comparison is based on a single batch measurement: 33 vs. 25 minutes and $3.20 vs. $2.80 for 1,000 images. No details are given about request concurrency, retries, API version or date, output token counts, or variability across repeated batches. Because the conclusion that there is 'no accuracy-cost trade-off' depends on these measurements, please report the measurement protocol and at least basic variability information, or temper the conclusion to reflect a single observation.
[Sections 7 and 8] The absolute performance interpretation, such as the statement that these models are 'not yet a complete replacement' for fine-tuned systems, is made without any baseline. A majority-class baseline (always predicting the most frequent label, or always predicting 'NA' for rare attributes) would contextualize whether macro F1 of 56.79% reflects genuine visual signal beyond class imbalance and label skew. This is not required for the model-versus-model ranking, but it is load-bearing for the production-readiness conclusion.

minor comments (7)

[Throughout] Model naming is inconsistent: the abstract and body use 'GPT-4o-mini', 'GPT-4o-Mini', and 'GPT-4o mini' interchangeably; please standardize to one form.
[Section 5.2] The sampling procedure is not fully reproducible: no random seed or exact sampling code is provided. Please state the seed or release the exact subset indices.
[Sections 4 and 5.3] The prompt in Table 8 asks for three separate label arrays (shape, color pattern, fabric type), but the architecture description suggests a single prediction-engine call. Clarify whether each image is sent in one API call returning all three arrays or in separate calls, and whether the cost/latency figures correspond to one call or three.
[Appendix tables 1-5] The per-attribute tables report precision, recall, and F1 but not the number of instances or the per-class support. Reporting label frequencies would help readers judge which F1 differences are stable and which may be driven by a handful of examples.
[Table 7] The column headers 'Input Tokens Cost' and 'Output Tokens Cost' list per-million-token prices, but the total cost of $3.20 and $2.80 for 1,000 images is not derived in the text; specify the actual input/output token counts used to compute these totals.
[References] Several citations are incomplete: Google 2025a/b, OpenRouter 2025, and OpenAI 2024 lack full bibliographic entries, and 'la Comble' in the text is listed as 'De la Comble' in the references; please unify and complete the reference list.
[Limitations] The limitations section is candid about dataset scope and model selection, but it does not mention the absence of repeated runs, confidence intervals, or baselines; adding these would align the stated limitations with the actual statistical rigor.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; empirical zero-shot benchmark scored against independent human labels.

full rationale

This paper is an empirical benchmark rather than a derivation. The central quantitative claim—Gemini 2.0 Flash at 56.79% macro F1 versus GPT-4o-mini at 43.28%—comes from prompting both models zero-shot on a public dataset (DeepFashion-MultiModal) and comparing their outputs to the dataset's human-annotated ground-truth labels. No parameter is fitted to the evaluation data, and no reported score is defined in terms of a fitted constant or of the other model's output. The prompt is constructed from the dataset's own label schema, which is task framing rather than circular reasoning. The cost and latency comparison is direct API measurement, and the temperature comparison is an experimental manipulation, not a circular step. The paper's stated limitations (1,000-image subset, single dataset, no fine-tuned baselines, single prompt design) are validity and generalizability concerns, not evidence of circularity, because the evaluation outcome does not feed back into the inputs that produce it. There is no load-bearing self-citation, no imported uniqueness theorem, and no known result being renamed as a new derivation. The strongest concerns about the subset's representativeness affect confidence in the magnitude of the measured gap, but they do not make any claim equivalent to its inputs by construction. Therefore, no circular steps are identified.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The evaluation rests on the accuracy of the dataset labels, the representativeness of the self-curated subset, the stability of closed API model behavior, and the fairness of a single fixed prompt. These are domain assumptions common to empirical LLM benchmarks, and the paper does not provide evidence to validate them beyond assertions.

assumptions (4)

domain assumption DeepFashion-MultiModal human annotations are accurate, complete ground truth for the 18 attributes.
The evaluation compares model outputs to these labels; if labels are noisy, F1 scores are misestimated. Section 5.1.
domain assumption The 1,000-image stratified subset is representative of the full dataset.
Results are reported on this subset; the paper does not show the subset's attribute distribution versus the full dataset. Section 5.2.
domain assumption API model behavior accessed through OpenRouter corresponds to the named models and is stable during the evaluation.
Closed-source models may be updated or routed differently; no model version snapshots are provided. Sections 3 and 4.
domain assumption The single prompt in Table 8 is a fair, equally effective probe for both models.
Prompt design can strongly affect zero-shot performance; no prompt sensitivity analysis is reported. Table 8.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Can GPT-4o mini and Gemini 2.0 Flash Predict Fine-Grained Fashion Product Attributes? A Zero-Shot Analysis." pith.science (2026). https://pith.science/paper/EJBMSZI3

@misc{pith2026250709950,
  author       = {Pith},
  title        = {Pith review of: Can GPT-4o mini and Gemini 2.0 Flash Predict Fine-Grained Fashion Product Attributes? A Zero-Shot Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EJBMSZI3}},
  note         = {Machine review of arXiv:2507.09950}
}

read the original abstract

The fashion retail business is centered around the capacity to comprehend products. Product attribution helps in comprehending products depending on the business process. Quality attribution improves the customer experience as they navigate through millions of products offered by a retail website. It leads to well-organized product catalogs. In the end, product attribution directly impacts the 'discovery experience' of the customer. Although large language models (LLMs) have shown remarkable capabilities in understanding multimodal data, their performance on fine-grained fashion attribute recognition remains under-explored. This paper presents a zero-shot evaluation of state-of-the-art LLMs that balance performance with speed and cost efficiency, mainly GPT-4o-mini and Gemini 2.0 Flash. We have used the dataset DeepFashion-MultiModal (https://github.com/yumingj/DeepFashion-MultiModal) to evaluate these models in the attribution tasks of fashion products. Our study evaluates these models across 18 categories of fashion attributes, offering insight into where these models excel. We only use images as the sole input for product information to create a constrained environment. Our analysis shows that Gemini 2.0 Flash demonstrates the strongest overall performance with a macro F1 score of 56.79% across all attributes, while GPT-4o-mini scored a macro F1 score of 43.28%. Through detailed error analysis, our findings provide practical insights for deploying these LLMs in production e-commerce product attribution-related tasks and highlight the need for domain-specific fine-tuning approaches. This work also lays the groundwork for future research in fashion AI and multimodal attribute extraction.

Figures

Figures reproduced from arXiv: 2507.09950 by the authors.

**Figure 1.** Example data point 2.0 Flash on 18 different fine-grained fashion attributes in a strictly zero-shot, image-only context. We focus on these lightweight models to assess their viability for scalable, real-world deployment where speed and cost are as important as accuracy. 3 Preliminaries To establish a clear foundation for our methodology and experimental design, we define key terms and components that are central … view at source ↗

**Figure 2.** High Level Architecture The workflow begins with an Input Image from the DeepFashion-MultiModal dataset (Jiang et al., 2022). This image serves as the sole input for the system, deliberately constraining the environment to test the models’ pure visual understanding capabilities. The image is then passed to a Prompt generation module, which formulates a query instructing the model on the task and the desired output… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 12 canonical work pages

[1]

sleeve length: 0 sleeveless, 1 short-sleeve, 2 medium-sleeve, 3 long-sleeve, 4 not long-sleeve, 5 NA

work page
[2]

lower clothing length: 0 three-point, 1 medium short, 2 three-quarter, 3 long, 4 NA

work page
[3]

socks: 0 no, 1 socks, 2 leggings, 3 NA

work page
[4]

hat: 0 no, 1 yes, 2 NA

work page
[5]

glasses: 0 no, 1 eyeglasses, 2 sunglasses, 3 have a glasses in hand or clothes, 4 NA

work page
[6]

neckwear: 0 no, 1 yes, 2 NA

work page
[7]

wrist wearing: 0 no, 1 yes, 2 NA

work page
[8]

ring: 0 no, 1 yes, 2 NA

work page

Show all 12 references

[9]

waist accessories: 0 no, 1 belt, 2 have a clothing, 3 hidden, 4 NA
[10]

neckline: 0 V-shape, 1 square, 2 round, 3 standing, 4 lapel, 5 suspenders, 6 NA
[11]

outer clothing a cardigan?: 0 yes, 1 no, 2 NA
[12]

upper clothing covering navel: 0 no, 1 yes, 2 NA Note: ’NA’ means the relevant part is not visible. Example: If you analyze an image where the sleeve length is long-sleeve, the lower clothing length is long, there are no socks, no hat, no glasses, no neckwear, no wrist wearing...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

sleeve length: 0 sleeveless, 1 short-sleeve, 2 medium-sleeve, 3 long-sleeve, 4 not long-sleeve, 5 NA

work page

[2] [2]

lower clothing length: 0 three-point, 1 medium short, 2 three-quarter, 3 long, 4 NA

work page

[3] [3]

socks: 0 no, 1 socks, 2 leggings, 3 NA

work page

[4] [4]

hat: 0 no, 1 yes, 2 NA

work page

[5] [5]

glasses: 0 no, 1 eyeglasses, 2 sunglasses, 3 have a glasses in hand or clothes, 4 NA

work page

[6] [6]

neckwear: 0 no, 1 yes, 2 NA

work page

[7] [7]

wrist wearing: 0 no, 1 yes, 2 NA

work page

[8] [8]

ring: 0 no, 1 yes, 2 NA

work page