Attribute-Conditioned Multimodal Slot Factorization for Controllable Fashion Retrieval

Evren Korpeoglu; Kannan Achan; Najmeh Forouzandehmehr; Topojoy Biswas

REVIEW 2 major objections 4 minor 18 references

Attribute-Conditioned Multimodal Slot Factorization for Controllable Fashion Retrieval

T0 review · 2 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read MM-slotgate factorizes Fashion-CLIP embeddings into four gated attribute slots and shows on H&M that this beats global fusion and text-only retrieval for constraint-satisfying fashion retrieval.

desk verdict A genuinely new per-slot modality gate, but the headline CS@10 numbers lean on an unreported cross-validated logit weight that needs an ablation before the main claim is trusted. read the letter →

arxiv 2608.12570 v1 pith:XDCG6J2W submitted 2026-08-12 cs.CV cs.IR

Najmeh Forouzandehmehr , Topojoy Biswas , Evren Korpeoglu , Kannan Achan This is my paper · ORCID

classification cs.CVcs.IR

keywords controllableretrievalmultimodalfusionslotfactorizationsemanticIDsvectorquantizationfashionattribute-conditionedConstraintSatisfied@10

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that fashion retrieval should represent an item as a set of named attribute slots, pattern, color, category, and demographic, rather than as one global embedding, and that each slot should learn its own mixture of text and image evidence. On the H&M catalog, the proposed MM-slotgate achieves 0.7566 macro ConstraintSatisfied@10, beating equal-weight multimodal fusion (0.7142) and text-only Fashion-CLIP retrieval (0.4755). The largest single gain is color, which rises from 0.321 to 0.889 as the learned color gate assigns 57.4 percent weight to image evidence. If correct, this means controllable multi-attribute fashion retrieval can be built from typed, addressable slots instead of opaque item-level codes.

What carries the argument

The central object is the per-slot scalar gate $g_s = \sigma(a_s)$, with $g_s$ as the text weight and $1-g_s$ as the image weight, in the fusion equation that mixes slot-specific text and image projections before L2 normalization and vector quantization. The gates let the model allocate modality evidence attribute-by-attribute, and they are trained end-to-end with the alignment loss on four attribute labels and the VQ commitment loss. Retrieval uses the continuous slots through a slot-weighted similarity plus the sum of the constrained slots' classifier logits, while intervention uses the quantized slot codes from each slot's codebook.

What would settle it

Run the same retrieval evaluation with the alpha term in the combined score set to zero (slot similarity only) and with the learned gates replaced by text-only; if slot-similarity-only MM-slotgate drops toward the text-only baseline of 0.4765 or no longer beats equal-weight fusion, then the claimed gating contribution is not established.

Watch

Extended reading notes

Core claim

MM-slotgate claims that supervised slot factorization plus per-slot learnable text-image gates gives better constraint-satisfying fashion retrieval than either a single global multimodal embedding or text-only retrieval. Each of the four slots receives a scalar gate $g_s = \sigma(a_s)$, initialized at 0.5 and trained end-to-end, that controls how much text versus image enters that slot's vector-quantized bottleneck. The learned gates converge without modality supervision: color is image-leaning, category is text-leaning, pattern is balanced, and demographic is slightly text-leaning. On H&M the method reaches 0.7566 macro CS@10, outperforms equal-weight fusion (0.7142) and text-only retrieval (0.4755), improves color from 0.321 to 0.889, and supports codebook-level intervention with a 15.3x lift for color while adding no measured excess cross-slot leakage beyond label correlations.

Load-bearing premise

The headline numbers come from a combined score that adds a tuned amount of the slot classifiers' confidence scores to the slot similarity score; the paper never reports how large that tuned amount is, so if it is large the gains may come from the classifiers rather than from the learned gates.

Editorial extensions

If this is right

Multi-attribute fashion queries can be satisfied without hand-tuning modality weights: each slot learns whether text or image should dominate.
Color retrieval, the weakest text-only attribute, becomes the strongest after image grounding: CS@10 for color rises from 0.321 to 0.889.
Both modalities are necessary; text-only collapses on color accuracy (0.304, near chance) and image-only degrades on category and demographic.
Quantized slots permit targeted attribute steering at retrieval time without recomputing embeddings, with color lift rising from 1.5x in the text-only encoder to 15.3x.
The learned gates are interpretable and match intuitive modality needs, and the representation adds no measured excess leakage beyond label co-occurrence.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the gate values are a function of each catalog's label-text-image correlations, the specific learned preferences (color image-leaning, category text-leaning) may not transfer to catalogs with different taxonomy or image quality; a testable extension is to re-learn gates per dataset and compare.
The combined score's alpha is tuned by cross-validation; an editorial extension would be to test whether a fixed alpha (including alpha=0) preserves the ordering, which would determine whether the slot-similarity term or the logit term drives the headline gains.
The same slot-gating mechanism could be applied to richer taxonomies such as occasion, style, or brand, though retrieval and intervention complexity would grow with the number of slots.
A direct follow-up experiment would ablate image availability per attribute class to verify that the missing-image text fallback does not systematically bias the learned gates toward text.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: the per-slot text-image gate is a real, easy-to-describe idea that I haven't seen in the semantic-ID or fashion-retrieval literature, and the paper has some careful validation (shuffled-image negative control, label-prior leakage baseline). But the headline retrieval numbers lean on a combined score whose second term is weighted by alpha*, tuned by CV and never reported, and there is no alpha=0 ablation. That is the soft spot that matters.

The architecture is straightforward: fCLIP text and image embeddings are projected per slot, mixed by a learned scalar gate, then quantized. The gate converges to sensible values (color image-leaning, category text-leaning) without modality supervision. That is a nice, clean result. The negative control -- shuffling images makes all gates text-leaning -- is the kind of check too many papers skip. And the leakage analysis, comparing off-diagonal probe AUC to the label-prior baseline, is the right way to think about disentanglement.

The retrieval evaluation is where I would want changes. Eq. (4) adds alpha times the query-constrained slot logits to the cosine term. The paper says alpha* is tuned by 5-fold CV but does not report it, and never shows the alpha=0 case. If alpha is large, the ranking reduces to item-side classifier confidence and the cosine term is decoration. The stress-test note worries this means the color gain (0.321 to 0.889) is mostly the alignment head's image classification ability, not the gate. That is a legitimate worry, but note that all methods use the same combined score, so the relative order between MM-slotgate and MM-global might survive. Still, without alpha, the absolute attribution is unclear.

The other issues are smaller: no error bars on any CS@10 number; the learned-gates-versus-fixed_half margin is 0.005 (almost certainly noise); and the abstract says fCLIP-text-only is 0.4755 while Table 1 says 0.4765. No code release either.

The paper itself is honest about the small learned-gate ablation margin, so I do not think there is misconduct; I think it is an under-reported hyperparameter that needs a simple ablation.

Who is it for? Anyone working on controllable multimodal retrieval or semantic IDs. It is a useful incremental contribution, not a field-reorganizing one. With alpha reported and an alpha=0 row added, the claims would be much easier to evaluate. I would send it to peer review and ask for that before acceptance.

Referee Report

2 major / 4 minor

Summary. MM-slotgate factorizes Fashion-CLIP text and image embeddings into four named attribute slots (pattern, color, category, demographic). Each slot has a learnable scalar gate over the two modalities, a VQ codebook, and a supervised alignment head. Retrieval uses a combined score of slot-weighted cosine similarity plus a tuned weighted sum of constrained-slot log probabilities. On a 50K-item H&M subset with a 5K held-out gallery, the paper reports macro CS@10 of 0.7566, beating MM-global fusion (0.7142) and fCLIP text-only (0.4755/0.4765), with the largest gain on color; it also reports learned gate values, a shuffled-image negative control, linear-probe leakage measurements, and codebook intervention lifts.

Significance. If the attribution is established, the paper contributes a simple, interpretable mechanism for attribute-conditioned modality selection in fashion retrieval, with a clear evaluation metric and useful diagnostics (oracle upper bound, negative control, label-prior leakage baseline). The architecture and training protocol are specified in enough detail to be reimplemented. However, the headline comparison currently depends on an unreported tuning weight, and the key learned-versus-fixed ablation lacks error bars, so the central claim is not yet fully supported.

major comments (2)

[Section 3, Eq. (4), and Section 4] The combined retrieval score is cos(w(q),w(i)) + alpha * sum_{s in C} log P(y_s = c*_s | i), with alpha tuned by 5-fold cross-validation, but the paper never reports the selected alpha* and never ablates the logit term (alpha=0). The concern that this term could dominate the ranking is well-founded: with large alpha, the top-10 items are those whose alignment heads assign high probability to the query's target classes, so the headline CS@10 gains, including color 0.321 to 0.889, could reflect the supervised alignment heads' classification accuracy rather than the learned modality gates or the slot-similarity representation. Please report alpha*, add an alpha=0 ablation, and ideally provide a per-term score breakdown so that the cosine-only and logit-only contributions can be compared.
[Tables 1-4, Sections 4-5] All results are point estimates from a single training run and a single 5K gallery, with no error bars, multiple seeds, or significance tests. The central ablation claim that learned gates beat fixed equal-weight fusion rests on a 0.005 CS@10 difference (0.7566 vs 0.7516) and on very close validation losses and color accuracies in Table 4. This margin is not interpretable without variance estimates; please provide multiple seeds or a significance test for the main comparisons, at minimum for learned versus fixed_half, MM-slotgate versus MM-global, and MM-slotgate versus fCLIP-text-only.

minor comments (4)

[Abstract, Section 1, Tables 1-2] The fCLIP-text-only macro CS@10 is reported as 0.4755 in the abstract and conclusion but as 0.4765 in Tables 1 and 2, and the relative gain in Section 1 (59.1%) is inconsistent with the 58.8% computed from the table; please reconcile all occurrences.
[Section 5, negative control] The shuffled-image negative control is reported only through validation loss and gate values; please also report CS@10 under the shuffle, since the claim that correctly aligned visual content drives the retrieval gain depends on showing that the retrieval metric degrades, not just that the model leans more on text.
[Section 7, intervention] The intervention metrics Hit@10, Null@10, and PreserveDelta are described only in prose; please give formal definitions and, if space permits, error estimates, since the color lift of 15.3x is a headline number.
[Section 3, Eq. (3)] The orthogonality penalty Lorth is mentioned but its exact form (which Gram matrix, which off-diagonal terms, how normalized) is not specified; please state it precisely.

Circularity Check

0 steps flagged · score 0.0 of 10

No material circularity: the core results rest on supervised training with held-out evaluation; the unreported alpha in Eq. (4) is an attribution concern, not a derivation-as-fit.

full rationale

The paper's derivation chain is a standard supervised learning pipeline: fCLIP embeddings are projected into four attribute slots, per-slot gates are trained end-to-end, and retrieval is evaluated on a held-out 5K split. No fitted parameter is renamed as a prediction from first principles. Equation (3) trains alignment heads on attribute labels, and Eq. (4) uses their log probabilities in the combined retrieval score; because CS@10 checks exactly those attribute labels, a large alpha could make the ranking dominated by supervised classification. However, this is a hyperparameter reporting and attribution weakness, not an identity between the method's inputs and outputs: the cosine term remains, the evaluation is held-out, and alpha is tuned by 5-fold CV rather than set to force the result. The learned gate values are reported as fitted parameters, not claimed as independent discoveries, and the negative-control shuffle experiment provides an external check that real image content matters. Comparisons to fCLIP-text, MM-global, and fixed_half are external baselines. The 'prior text-only slot encoder' appears only as a reference baseline and is not load-bearing through any self-citation chain or uniqueness theorem. No equation in the paper reduces to another by construction, so no circular step meets the evidentiary bar.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No invented entities. The four slots are re-interpretations of existing attribute labels, not new physical or theoretical objects. The central empirical claim rests mainly on pretrained fCLIP embeddings and on hyperparameters (alpha, codebook sizes, loss weights) that are tuned or hand-set but not reported in full.

free parameters (3)

retrieval logit weight alpha* = not reported
Tuned by 5-fold cross-validation in Eq. (4); controls how much the supervised slot-logit term contributes to the reported CS@10.
codebook sizes K_s = pattern=15, color=20, category=10, demographic=6
Chosen by hand; the paper gives no search or ablation for these sizes.
loss weights beta, lambda_a, lambda_perp = 0.25, 5.0, 2.0
Chosen by hand; no sensitivity analysis is reported.

assumptions (4)

domain assumption Fashion-CLIP text and image embeddings contain enough attribute-relevant signal for all four attributes.
The method adds gates and slots on top of frozen fCLIP [2] embeddings; if the backbone lacks color or pattern information, the gates cannot recover it. Linear probe diagnostics support this for color but the assumption is not proven for all attributes.
domain assumption H&M attribute labels and the 50K subset are reliable ground truth for pattern, color, category, and demographic.
The paper does not describe how the 50K subset was selected from the Kaggle dataset or how label noise was handled.
standard math The straight-through estimator and EMA codebook updates train the VQ bottleneck effectively at the chosen codebook sizes.
Standard practice in VQ-VAE-style models; the paper relies on it without proof or convergence analysis.
domain assumption The 500-query sample and 90/10 split are sufficient to estimate CS@10.
No variance or significance testing is reported, so the assumption of representativeness is load-bearing for the headline numbers.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Attribute-Conditioned Multimodal Slot Factorization for Controllable Fashion Retrieval." pith.science (2026). https://pith.science/paper/XDCG6J2W

@misc{pith2026260812570,
  author       = {Pith},
  title        = {Pith review of: Attribute-Conditioned Multimodal Slot Factorization for Controllable Fashion Retrieval},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XDCG6J2W}},
  note         = {Machine review of arXiv:2608.12570}
}

read the original abstract

Fashion retrieval often requires satisfying multiple attributes at once, such as category, color, pattern, and demographic. Monolithic embeddings mix these signals into a single vector, making attribute-specific control difficult at retrieval time. Many existing semantic-ID methods provide discrete item codes, but these codes are typically optimized as item-level or residual addresses and do not expose named, independently controllable attribute slots. We introduce MM-slotgate, a multimodal slot encoder that factorizes Fashion-CLIP text and image embeddings into four named attribute slots. Each slot learns its own text-image gate, so visually grounded attributes such as color and pattern can rely more on image evidence, while taxonomy-oriented attributes such as category and demographic can remain more text-driven. On H&M, using a combined slot-similarity and slot-logit retrieval score, MM-slotgate achieves 0.7566 macro ConstraintSatisfied@10, outperforming equal-weight multimodal fusion (0.7142) and fCLIP text-only retrieval (0.4755). The largest gain is on color, which improves from 0.321 to 0.889 (+0.568 absolute), as the learned color gate assigns 57.4% weight to image evidence. The learned gates are interpretable without modality supervision: color is image-leaning, category is text-leaning, and pattern and demographic lie near the middle. The resulting slots also remain controllable: linear probes show no measured excess leakage beyond the label-correlation baseline, and quantized slot codes support targeted intervention, including a 15.3x lift for color. These results suggest that controllable fashion retrieval benefits from typed, attribute-conditioned multimodal slots rather than either a single global embedding or opaque item-level semantic IDs.

Figures

Figures reproduced from arXiv: 2608.12570 by the authors.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 7 canonical work pages

[1]

Yoshua Bengio, Nicholas Léonard, and Aaron Courville. 2013. Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation. InarXiv preprint arXiv:1308.3432

arXiv 2013
[2]

Po-Yao Chia, Giuseppe Attanasio, Federico Bianchi, Silvia Terragni, Ana Rita Pires, Gianmarco Liu, and Ciro Legrand. 2022. Contrastive Language-Image Pre-Training for Fashion. InProceedings of the ACM Web Conference. doi:10.1145/ 3485447.3512241

arXiv 2022
[3]

Dengzhao Fang, Jingtong Gao, Chengcheng Zhu, Yu Li, Xiangyu Zhao, and Yi Chang. 2025. HiD-VAE: Interpretable Generative Recommendation via Hi- erarchical and Disentangled Semantic IDs. arXiv:2508.04618 [cs.IR] https: //arxiv.org/abs/2508.04618

arXiv 2025
[4]

Xintong Han, Zuxuan Wu, Phoenix X Huang, Xiao Zhang, Menglong Zhu, Yuan Li, Yang Zhao, and Larry S Davis. 2017. Automatic Spatially-aware Fashion Con- cept Discovery. InProceedings of the IEEE International Conference on Computer Vision

work page 2017
[5]

Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 2017. beta- VAE: Learning Basic Visual Concepts with a Constrained Variational Framework. International Conference on Learning Representations(2017)

work page 2017
[6]

H&M Group. 2022. H&M Personalized Fashion Recommendations. Kaggle com- petition dataset. https://www.kaggle.com/competitions/h-and-m-personalized- fashion-recommendations

work page 2022
[7]

Yupeng Hou, Zhankui He, Julian McAuley, and Wayne Xin Zhao. 2023. Learn- ing Vector-Quantized Item Representation for Transferable Sequential Rec- ommenders. InProceedings of the ACM Web Conference 2023. 1162–1171. arXiv:2210.12316 [cs.IR] doi:10.1145/3543507.3583434

arXiv 2023
[8]

Haohao Qu, Wenqi Fan, Zihuai Zhao, and Qing Li. 2025. TokenRec: Learning to Tokenize ID for LLM-Based Generative Recommendations.IEEE Transactions on Knowledge and Data Engineering37, 10 (2025), 6216–6231. arXiv:2406.10450 [cs.IR] doi:10.1109/TKDE.2025.3599265

arXiv 2025

Show all 18 references

[9]

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sand- hini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al
[10]

Shashank Rajput, Nikhil Mehta, Anima Singh, Raghunandan Keshavan, Trung Vu, Lukasz Heldt, Lichan Hong, Yi Tay, Vinh Q Tran, Jonah Samost, et al. 2023. Rec- ommender Systems with Generative Retrieval. InAdvances in Neural Information Processing Systems

2023
[11]

Zihua Si, Zhongxiang Sun, Jiale Chen, Guozhang Chen, Xiaoxue Zang, Kai Zheng, Yang Song, Xiao Zhang, Jun Xu, and Kun Gai. 2024. Generative Retrieval with Se- mantic Tree-Structured Identifiers and Contrastive Learning. InProceedings of the 2024 Annual International ACM SIGIR C...

2024 arXiv
[12]

Chi, and Xinyang Yi

Anima Singh, Trung Vu, Nikhil Mehta, Raghunandan Keshavan, Maheswaran Sathiamoorthy, Yilin Zheng, Lichan Hong, Lukasz Heldt, Li Wei, Devansh Tandon, Ed H. Chi, and Xinyang Yi. 2024. Better Generalization with Semantic IDs: A Case Study in Ranking for Recommendations. InProceed...

2024 arXiv
[13]

Nam Vo, Lu Jiang, Chen Sun, Kevin Murphy, Li-Jia Li, Li Fei-Fei, and James Hays. 2019. Composing Text and Image for Image Retrieval – an Empirical Odyssey. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

2019
[14]

Wenjie Wang, Honghui Bao, Xinyu Lin, Jizhi Zhang, Yongqi Li, Fuli Feng, See- Kiong Ng, and Tat-Seng Chua. 2024. Learnable Item Tokenization for Generative Recommendation. InProceedings of the 33rd ACM International Conference on Information and Knowledge Management. 2400–2409....

2024
[15]

Han Zhu, Daqing Chang, Ziru Xu, Pengye Zhang, Xiang Li, Jie He, Han Li, Jian Xu, and Kun Gai. 2019. Joint Optimization of Tree-Based Index and Deep Model for Recommender Systems. InAdvances in Neural Information Processing Systems, Vol. 32. arXiv:1902.07565 [cs.IR] https://arx...

2019 arXiv
[16]

Han Zhu, Xiang Li, Pengye Zhang, Guozheng Li, Jie He, Han Li, and Kun Gai
[2018]

InProceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining

Learning Tree-Based Deep Model for Recommender Systems. InProceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 1079–1088. arXiv:1801.02294 [cs.IR] doi:10.1145/3219819.3219826

arXiv
[2021]

InInternational Conference on Machine Learning

Learning Transferable Visual Models From Natural Language Supervision. InInternational Conference on Machine Learning

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

Yoshua Bengio, Nicholas Léonard, and Aaron Courville. 2013. Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation. InarXiv preprint arXiv:1308.3432

arXiv 2013

[2] [2]

Po-Yao Chia, Giuseppe Attanasio, Federico Bianchi, Silvia Terragni, Ana Rita Pires, Gianmarco Liu, and Ciro Legrand. 2022. Contrastive Language-Image Pre-Training for Fashion. InProceedings of the ACM Web Conference. doi:10.1145/ 3485447.3512241

arXiv 2022

[3] [3]

Dengzhao Fang, Jingtong Gao, Chengcheng Zhu, Yu Li, Xiangyu Zhao, and Yi Chang. 2025. HiD-VAE: Interpretable Generative Recommendation via Hi- erarchical and Disentangled Semantic IDs. arXiv:2508.04618 [cs.IR] https: //arxiv.org/abs/2508.04618

arXiv 2025

[4] [4]

Xintong Han, Zuxuan Wu, Phoenix X Huang, Xiao Zhang, Menglong Zhu, Yuan Li, Yang Zhao, and Larry S Davis. 2017. Automatic Spatially-aware Fashion Con- cept Discovery. InProceedings of the IEEE International Conference on Computer Vision

work page 2017

[5] [5]

Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 2017. beta- VAE: Learning Basic Visual Concepts with a Constrained Variational Framework. International Conference on Learning Representations(2017)

work page 2017

[6] [6]

H&M Group. 2022. H&M Personalized Fashion Recommendations. Kaggle com- petition dataset. https://www.kaggle.com/competitions/h-and-m-personalized- fashion-recommendations

work page 2022

[7] [7]

Yupeng Hou, Zhankui He, Julian McAuley, and Wayne Xin Zhao. 2023. Learn- ing Vector-Quantized Item Representation for Transferable Sequential Rec- ommenders. InProceedings of the ACM Web Conference 2023. 1162–1171. arXiv:2210.12316 [cs.IR] doi:10.1145/3543507.3583434

arXiv 2023

[8] [8]

Haohao Qu, Wenqi Fan, Zihuai Zhao, and Qing Li. 2025. TokenRec: Learning to Tokenize ID for LLM-Based Generative Recommendations.IEEE Transactions on Knowledge and Data Engineering37, 10 (2025), 6216–6231. arXiv:2406.10450 [cs.IR] doi:10.1109/TKDE.2025.3599265

arXiv 2025