{"id":"b2dd5cdb-63a7-4d6f-8ab0-aaf84a2284cf","arxiv_id":"2502.15696","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"FLLM fine-tunes an LLM with auto-generated fashion prompts and retrieval augmentation, reporting modestly higher outfit compatibility accuracy than prior visual models on Polyvore.","lead":"This paper proposes FLLM, a fine-tuned large language model with retrieval-augmented generation, for fashion outfit recommendations. It reports higher fill-in-the-blank accuracy than several visual models on the Polyvore dataset, plus stronger few-shot learning at low training-data ratios.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"FITB evaluation may be gamed by retrieval leakage: Section 3.3's RAG can surface candidate item text, so Table 1's accuracy may not measure fashion reasoning.","rationale":"The paper's only quantitative evidence for novelty is the accuracy comparison. The strongest claim collapses if the comparison does not isolate fashion reasoning. I focused on the least-secure premise: the inference-time retrieval described in Section 3.3 is not constrained, and the paper never states that candidate items are excluded from the vector DB. Because FITB candidates are known during evaluation, an embedding retriever could rank the true completion by text similarity, giving FLLM direct access to the answer. This is a distinct and more severe channel than the reader's fair-comparison concern, though both concern evaluation leakage; hence partial agreement. A category-label prior from text input is a second confound. The proposed test separates these channels: removing candidates from retrieval tests leakage; removing category labels tests modality. If either changes accuracy substantially, the central empirical claim is not supported. The absence of code, hyperparameters, and error bars further prevents checking this. Because the claim is currently unverified in a way that is directly testable, the reader's REJECT verdict stands; I would not change it without the ablation.","tokens_in":77,"tokens_out":4379,"duration_ms":53579,"concrete_test":"Re-run the Polyvore-disjoint FITB experiment with FLLM under three matched conditions: (a) full RAG as in the paper; (b) RAG with every candidate item's description removed from the retrieval corpus; (c) no RAG and with category names removed from all item descriptions. If condition (a) drops materially toward (c), or (b)/(c) fall below Type-Aware's 55.65, Table 1's advantage is an artifact of retrieval leakage or label prior, not domain knowledge. Report per-condition accuracy and retrieval hit-rate.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that FLLM beats image-based models in Table 1 and by ~10% in Figure 4—requires the FITB test to measure fashion compatibility. Two unguarded channels could produce the reported numbers without such reasoning. First, Section 3.3 embeds item descriptions in a vector DB and retrieves them using queries derived from the incomplete outfit; if this DB includes the candidate items (the paper never says it does not, and reports no RAG on/off ablation or retrieval hit-rate analysis), the LLM can simply adopt the top retrieved item's description. Second, FLLM receives item text while Type-Aware/CSA-Net/OutfitTransformer receive images; if that text includes category or attribute labels, a language prior can solve category-level FITB. No text-only baseline, category-masking experiment, or error bars are reported in Section 4.2. The claimed superiority is therefore not established as fashion reasoning.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FLLM, a large language model fine-tuned on fashion-specific QA data derived from Polyvore and equipped with retrieval-augmented generation at inference. The authors report fill-in-the-blank (FITB) accuracy on Polyvore and Polyvore-disjoint, few-shot learning results at reduced training-data ratios, and a qualitative visualization of recommendation diversity. The main claims are that FLLM achieves the highest FITB accuracy in Table 1, surpasses a Type-Aware baseline by about 10 percent in few-shot settings, and offers improved interpretability.","tokens_in":6337,"tokens_out":4266,"duration_ms":42615,"significance":"If the reported results were valid, the paper would make a useful contribution: a text-only LLM with retrieval augmentation outperforming established image-based compatibility models on a public benchmark would be a noteworthy result with practical implications for explainable fashion recommendation. The use of a fixed external benchmark, Polyvore, is a strength: accuracy is measured against ground-truth labels rather than the model's own outputs, so the central accuracy claim is not circular. The paper also provides a clear, if high-level, architecture for combining auto-prompt fine-tuning with RAG. However, the experimental design has load-bearing confounds that prevent the claims from being accepted as stated.","major_comments":[{"comment":"The main comparison is confounded by input modality. FLLM consumes item text while Type-Aware, CSA-Net, and OutfitTransformer consume images; if the item descriptions include category or attribute labels, the FITB task can be solved partially or entirely by applying a language prior to those labels. No text-only baseline, category-masking experiment, or input-modality ablation is provided, so the reported accuracy advantage in Table 1 cannot be attributed to fashion compatibility reasoning rather than to the information available in the text modality.","section":"Section 4.2, Table 1"},{"comment":"The retrieval-augmented inference setup creates a possible leakage channel for the FITB evaluation. The vector database described in Section 3.3 is not stated to exclude the candidate items that appear in the FITB test questions, and the paper reports no RAG on/off ablation, no retrieval hit-rate analysis, and no check for whether the retrieved item text contains the answer. If the database includes candidate item descriptions, the model can achieve high FITB accuracy simply by adopting the top retrieved item's text, which would make the numerical results in Table 1 uninformative about fashion reasoning.","section":"Section 3.3, Table 1"},{"comment":"No error bars, standard deviations, number of independent runs, or significance tests are reported for any of the accuracy numbers. The few-shot claim of 'about 10 percent' improvement appears to rest on single runs at low data ratios, and the statement that final performance is similar at the full data ratio is not enough to establish a reliable difference. Additionally, the axes of Figure 4 are internally inconsistent: the x-axis lists both 0%-80% and 10%-50%, and the y-axis is labeled 'FIIB accuracy' while the text uses FITB, making the figure difficult to interpret.","section":"Section 4.2, Table 1 and Figure 4"},{"comment":"The claimed interpretability advantage is supported only by a single qualitative visualization. There is no user study, human evaluation, or quantitative faithfulness metric for the generated explanations or recommendations, so the statement that FLLM 'surpasses existing models in accuracy, interpretability, and few-shot learning capabilities' is not demonstrated for the interpretability component.","section":"Section 4.2, Figure 5"}],"minor_comments":[{"comment":"The SCE-Net Average row cites reference [22], which is a remote sensing paper on single-view height estimation and semantic segmentation, not a fashion compatibility method; the provenance of this baseline is therefore unclear and needs to be corrected or replaced.","section":"Table 1"},{"comment":"The fine-tuning procedure omits essential reproducibility details: the base LLM is not named, and the number of training steps, learning rate, data sizes, and other hyperparameters are not reported. Section 3.3 similarly omits the sentence-transformer model and the retrieval top-k setting.","section":"Section 3.2"},{"comment":"The text uses 'FITB' while Figure 4 uses 'FIIB' for the same task; this inconsistency should be fixed, and the figure legend should clarify which curves correspond to which method.","section":"Section 4.2"},{"comment":"The manuscript retains template placeholders in the ACM reference format ('https://doi.org/XXXXXXX.XXXXXXX') and in the acknowledgments section, which should be completed before any publication.","section":"References"}],"recommendation":"reject","confidential_remarks":"The paper reads as an early-stage workshop draft. Even setting aside the missing experimental details, the central comparison is not interpretable without a text-only baseline and a RAG-leakage control; these are not local corrections but require re-running the experiments. I do not see a path to acceptance within the scope of a normal revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a plausible LLM-for-fashion pipeline, but the paper's headline accuracy claims rest on a comparison that is probably unfair, and the retrieval step may leak the answer. The core idea—fine-tune a small LLM on auto-generated QA and add RAG at inference—is legit and cheap, and the FITB numbers on Polyvore are at least reported on a standard benchmark. Credit where due: the authors clearly thought about training-data construction and they went to the trouble of evaluating on Polyvore-disjoint, which is more than many LLM papers do.\n\nThe soft spots are real and they hit the central claim. In Table 1, FLLM consumes item text while Type-Aware, CSA-Net, and OutfitTransformer consume images. If that text carries category or attribute names, a language prior can solve category-level FITB without fashion reasoning. There is no text-only baseline, no category-masking experiment, no error bars. Second, the retrieval-augmented inference in Section 3.3 embeds item descriptions and pulls them from a vector DB. Nothing in the paper says the candidate items are excluded from that DB. If they are in, the model can just copy the retrieved description. A RAG on/off ablation would settle this; none is reported. The few-shot figure also lacks axis labels and variance, and the 'interpretability advantage' is asserted but never measured.\n\nThe paper also claims there is 'no established work on using LLMs in fashion recommendation,' which is not credible—there is prior work on LLMs for outfit generation and compatibility, and the citation list doesn't engage with it.\n\nNone of this makes the method incoherent. The approach could be made to work; it's just that the evidence as presented does not support the superiority claim. As is, this reads like a workshop submission that needed one more round of honest ablation work before being put up for serious review. I would not send it to peer review in this state, but I'd tell the authors to redo the evaluation with a text-only baseline, a RAG-ablation, and variance bars; then it might become a solid domain application.","headline":"A plausible LLM-for-fashion pipeline whose accuracy claims are undercut by a modality confound and a potentially leaking retrieval step.","tokens_in":6711,"tokens_out":2517,"would_cite":false,"duration_ms":23958,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper's Fashion Large Language Model, fine-tuned with auto-prompt generation and retrieval-augmented inference, achieves the highest fill-in-the-blank accuracy on Polyvore among compared methods.","keywords":["fashion recommendation","large language models","retrieval-augmented generation","fill-in-the-blank","Polyvore","few-shot learning","auto-prompt generation","outfit compatibility"],"falsifier":"A decisive test would rerun the fill-in-the-blank evaluation on Polyvore-disjoint with a simple text-only baseline (for example, a bag-of-words or BERT classifier over item names and category labels) and with an FLLM variant whose inputs are stripped of category and attribute words; if either matches or exceeds FLLM's 62.17, the claimed advantage would be attributable to reading labels rather than to LLM-based reasoning or retrieval.","tokens_in":5847,"feed_emoji":"👗","tokens_out":5062,"duration_ms":41582,"temperature":0.7,"pith_summary":"The paper proposes Fashion Large Language Model (FLLM), an LLM fine-tuned on outfit text data with auto-prompt generation, plus retrieval-augmented inference, for fashion recommendation. It argues that this text-based generative approach outperforms established image-based compatibility models on the fill-in-the-blank task: FLLM reaches 62.17 on Polyvore-disjoint and 67.21 on Polyvore, the highest in the comparison table. The paper also claims that FLLM retains accuracy at low training-data ratios, surpassing a Type-Aware baseline by about 10 percent when data is scarce. If these results hold, they suggest that language models can substitute for visual models in fashion compatibility reasoning while adding interpretability and adaptability.","feed_headline":"Text-based LLM tops image models in fashion outfit test","feed_subtitle":"Fine-tuned fashion LLM with retrieval augmentation scores 62.17 on Polyvore-disjoint, ahead of image-based baselines.","key_machinery":"The central machinery is two-fold: (1) an auto-prompt generation training strategy in which both template QA pairs and LLM-generated, style-focused QA pairs are used to fine-tune the model on Polyvore outfits, preserving domain knowledge; and (2) a retrieval-augmented generation (RAG) inference stage that issues multiple query pathways — direct embedding queries, style- and occasion-conditioned queries, and LLM-generated dynamic questions — against a vector database of fashion documents. Together these components let the model condition its recommendations on both broad fashion knowledge and a user's stated preference context.","core_discovery":"The central claim is that a fine-tuned LLM with retrieval augmentation can solve outfit fill-in-the-blank prediction more accurately than specialized image-based models. On the Polyvore-disjoint test set, FashionLLM achieves 62.17 percent accuracy, ahead of OutfitTransformer at 59.48, CSA-Net at 59.26, Type-Aware at 55.65, and SCE-Net at 53.67; on the full Polyvore set it reaches 67.21, edging OutfitTransformer's 67.10. The paper also reports that at low training-data ratios FLLM beats the Type-Aware baseline by roughly 10 percentage points, and that its generated recommendations are interpretable and versatile across style preferences.","pith_inferences":["The comparison in Table 1 is between a text-fed LLM and image-fed baselines; if Polyvore item descriptions encode category or attribute labels, part of the observed gain may come from the model reading labels the baselines must infer from pixels. A text-only baseline or a description-stripped ablation would isolate the language-model reasoning contribution.","Because the fine-tuning uses Polyvore training outfits and the RAG database stores fashion documents, the model's 'domain knowledge' may be partly memorized from the dataset; testing on a held-out trend shift (e.g., newer outfits) would show whether retrieval actually helps adapt to distribution shift as claimed.","The few-shot claim centers on one baseline (Type-Aware); extending the low-data comparison to other image-based methods and reporting variance across seeds would strengthen the generalizability of the 10 percent figure.","A natural next step is a multimodal hybrid where the LLM fuses its text reasoning with image encoders, which the authors themselves signal in their future-work section; such a hybrid could disambiguate whether text or image contributes the compatibility signal."],"forward_implications":["If FLLM's accuracy advantage holds, outfit compatibility can be treated as a language task, enabling recommendation systems to leverage pretrained reasoning and user-facing explanation in one model.","The reported few-shot robustness implies that text-based LLM recommenders can be deployed in data-sparse settings such as new boutiques, niche styles, or emerging trends where visual training data is scarce.","The RAG-based inference with multiple query pathways suggests a modular design: updating the vector database can refresh the model's fashion knowledge without retraining the LLM itself.","The interpretability of generated outfit suggestions could support direct user interaction, allowing users to state preferences and receive style-alternative recommendations."],"supporting_citations":[{"why":"Supplies the Type-Aware embedding baseline that FLLM is compared against in Table 1 and in the few-shot experiment.","marker":"[18]"},{"why":"Supplies CSA-Net, one of the image-based compatibility baselines in the FITB accuracy comparison.","marker":"[8]"},{"why":"Supplies OutfitTransformer, the strongest image-based baseline FLLM edges out on the full Polyvore set.","marker":"[15]"},{"why":"Supplies SCE-Net, another baseline in the Table 1 comparison.","marker":"[22]"},{"why":"Provides the Polyvore and Polyvore-disjoint datasets used for training and evaluating FLLM.","marker":"[5]"}],"fun_headline_variants":["Fashion LLM outpredicts image models in outfit test","Retrieval-augmented LLM tops fashion outfit baselines","Text-based LLM wins fashion outfit accuracy contest","FashionLLM beats image models with 62% outfit accuracy","Fine-tuned fashion LLM edges out vision models in outfits"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the Table 1 comparison is fair, meaning FLLM receives item text while the baselines receive only images and the accuracy gap reflects better fashion reasoning rather than the model reading category or attribute labels from the text, a premise never tested with a text-only baseline.","fun_headline_variants_meta":{"raw":{"variants":["Fashion LLM outpredicts image models in outfit test","Retrieval-augmented LLM tops fashion outfit baselines","Text-based LLM wins fashion outfit accuracy contest","FashionLLM beats image models with 62% outfit accuracy","Fine-tuned fashion LLM edges out vision models in outfits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000321,"raw_usage":{"total_tokens":1764,"prompt_tokens":859,"completion_tokens":905,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":475,"completion_tokens_details":{"reasoning_tokens":820}},"tokens_in":475,"tokens_out":905,"duration_ms":8302,"temperature":1.0,"reasoning_tokens":820,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:14:08.170628+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive test would rerun the fill-in-the-blank evaluation on Polyvore-disjoint with a simple text-only baseline (for example, a bag-of-words or BERT classifier over item names and category labels) and with an FLLM variant whose inputs are stripped of category and attribute words; if either matches or exceeds FLLM's 62.17, the claimed advantage would be attributable to reading labels rather than to LLM-based reasoning or retrieval.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Type-Aware embedding baseline that FLLM is compared against in Table 1 and in the few-shot experiment."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies OutfitTransformer, the strongest image-based baseline FLLM edges out on the full Polyvore set."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies SCE-Net, another baseline in the Table 1 comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Polyvore and Polyvore-disjoint datasets used for training and evaluating FLLM."}],"review_version":1}