{"id":"f08ea4cd-2292-4173-9332-60098f5a8d06","arxiv_id":"2412.18416","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"MUSE is a 7,000-conversation multimodal conversational recommendation dataset synthesized by MLLM agents with scenario-grounded user profiles, including 83,148 utterances and product images.","lead":"This paper presents MUSE, a synthetic dataset of 7,000 multimodal shopping conversations in clothing, shoes, and jewelry, where images are used alongside text during recommendations. It matters because it gives conversational recommendation researchers a benchmark that includes visual product information, which text-only datasets lacked.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The usefulness of MUSE as a benchmark depends on whether the simulated users behave like real shoppers, but the generation pipeline guarantees delivery of the pre-selected target item and conditions user responses on that target, so the reported 'learnable patterns' may be artifacts of the…","rationale":"Read in good faith, the paper delivers a concrete artifact: 7,000 conversations, a documented multi-agent pipeline, release links, automatic filtering statistics, and consistent fine-tuning gains on three MLLMs. Those are real strengths. The reader's CONDITIONAL verdict already targets user-simulation realism; I agree with that direction and would sharpen it. The most damaging failure mode is not just 'gpt-4o-mini is imperfect' but a design choice that removes a core part of conversational recommendation: the assistant must discover the right item through dialogue, and the user may not accept or may leave. MUSE's fallback rule (Section 3.3.2) makes the target item available to the system at the round limit, and the user simulator knows it throughout, so every conversation is a guaranteed success trajectory. This is a label-generation scheme, not an observation of genuine recommender-user interaction. It also undercuts the interpretation of the fine-tuning experiments: models trained on guaranteed-success dialogues can learn shallow cues instead of preference inference. I did not make the 'first multimodal CR dataset' novelty dispute the headline concern, because the authors explicitly distinguish SURE and SIMMC on task formulation, and that distinction, though debatable, does not threaten the dataset's utility. I also did not hinge on the self-LLM evaluation, since manual A/B tests and the text-only ablation in Appendix B.5 provide some independent evidence. The proposed ablation of forced delivery is a single, feasible check that would separate generator-specific regularities from transferable recommendation patterns. Verdict stays CONDITIONAL pending that evidence.","tokens_in":25269,"tokens_out":5047,"duration_ms":50362,"concrete_test":"Rerun the MUSE generation pipeline on 500 randomly sampled scenario-grounded user profiles with the same prompts and settings, but remove the fallback instruction in Section 3.3.2 that provides the target product when the round limit is reached. Measure (a) how many conversations end with the user accepting a non-target item or with no acceptance, and (b) whether the Table 4 recommendation metrics and Table 5 response metrics change when fine-tuning is performed only on conversations that reached a natural, non-forced acceptance. If natural success is high and the metrics hold, the forced-target objection is weak; if success collapses or the fine-tuning gains disappear, the learnable patterns depend on the guarantee and do not transfer to unconstrained user behavior.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is that gpt-4o-mini's scenario-grounded user simulator produces realistic shopping behavior (Section 3.3.1). The strongest single threat to that assumption is not general LLM fluency but the pipeline's built-in target-item guarantee. In Section 3.3.2, 'If the round limit is reached, the Querier provides the user's target product to end the conversation.' The user simulator is given a profile that explicitly contains the target product and target requirements, and its accept/reject actions are generated by comparing each candidate against that known target. Every conversation is therefore forced to converge on a pre-selected item; there are no conversations in which the user abandons the session, settles for a different item, or reveals needs that diverge from the target. Consequently, the recommendation improvements in Table 4 and the response gains in Tables 5-6 may reflect learning structural regularities of this forced convergence (e.g., round number, target-consistent phrasing after rewrite) rather than a general ability to infer preferences from dialogue. The LLM-based conversation-level evaluation also rewards exactly the long, visually detailed, scenario-rich style that the generator was prompted and rewritten to produce, so it cannot independently validate realism. If this concern lands, MUSE remains a useful synthetic resource, but its central claim to provide learnable patterns for real multimodal conversational recommendation is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MUSE, a synthetic multimodal conversational recommendation (CR) dataset in the Clothing domain, comprising 83,148 utterances from 7,000 conversations. The dataset is generated by a three-module multi-agent framework (Scenario-Grounded User Profile Generator, Simulated Conversation Generator, Conversation Optimizer) powered by gpt-4o-mini with a Claude-3.5-haiku rewriter. The authors claim that MUSE is the first multimodal CR dataset, that its conversations are high quality (validated by LLM and human evaluations), and that fine-tuning three open-source MLLMs on MUSE yields gains in recommendation accuracy and response generation. The paper also proposes scenario-grounded user profiling as a scalable alternative to history-based profile construction.","tokens_in":25588,"tokens_out":4863,"duration_ms":40335,"significance":"If the realism and learnability claims hold, MUSE would fill a genuine gap: existing CR datasets (ReDial, TG-ReDial, PEARL, LLM-Redial) are text-only, while multimodal dialogue datasets are not CR-oriented. The scenario-grounded user-profile generator is a sensible way to escape the cold-start and privacy limitations of history-based profile synthesis, and the release of the dataset and code is commendable. The fine-tuning experiments (e.g., Qwen2-VL Recall@10 rising from 0.20 to 0.34) are concrete and reproducible in principle. However, the central claim of high-quality, realistic conversations is currently supported mainly by LLM judges scoring LLM-generated text and by a very small, same-institution human evaluation, and the pipeline has a structural property that guarantees every conversation ends with the pre-selected target item. These issues need resolution before the dataset can be trusted as a benchmark for real-world multimodal CR.","major_comments":[{"comment":"The pipeline guarantees that every conversation ends with the user receiving the pre-selected target item: 'If the round limit is reached, the Querier provides the user's target product to end the conversation' (Section 3.3.2). The user simulator also has access to the target item and target requirements, and its accept/reject decisions are generated by comparing each candidate against that known target. Consequently, 'successful recommendation' is true by construction, and there are no conversations in which the user abandons the session, settles for a non-target item, or reveals needs inconsistent with the target. This undermines the interpretation of the fine-tuning gains in Table 4: the models may be learning the structural regularity of forced convergence (e.g., the target is always delivered by the final round) rather than a general ability to infer preferences from dialogue. Please report the fraction of conversations that end via natural acceptance versus round-limit target delivery, and re-run the Table 4 experiments in a condition where the target item is not revealed in the user profile at inference time, or with a control in which a non-target item is delivered at the round limit.","section":"Section 3.3.2, Appendix A.3"},{"comment":"The conversation-level quality evaluation uses gpt-4o as the judge to score gpt-4o-mini/Claude-generated dialogues on a 0-2 scale across five dimensions. The evaluation prompt (Appendix D.4) rewards exactly the characteristics the generator was prompted to produce: long, information-rich, visually detailed, and scenario-elaborated responses. The high scores therefore do not independently validate naturalness or realism relative to human-human CR. As a minimal corrective, the authors should (a) report human ratings on the same or a subset of the 200 conversations per dataset, with inter-annotator agreement (e.g., Krippendorff's alpha); (b) include a baseline condition in which the same judge evaluates human-written CR dialogues (e.g., sampled from ReDial or SIMMC 2.0) under the identical rubric; and (c) ablate the Rewriter to show that the improved scores are not simply a reflection of the colloquial rewrite style.","section":"Section 4.2, Table 3, Appendix D.4"},{"comment":"The utterance-level human evaluation is based on six annotators from the same institution, with three generating responses and three performing the anonymous 1:1 comparisons. The paper reports only win counts (145/55, 152/48, 111/89, 119/81) without error bars, significance tests, or inter-annotator agreement. The claim that MUSE's utterances are 'superior quality compared to human-authored dialogues' is therefore not statistically supported. The Appendix B.2 replacement experiment (50 conversations, win rate 0.68) has the same limitations. Please report confidence intervals or a significance test (e.g., exact binomial test), and either expand the annotator pool or soften the claim accordingly.","section":"Sections 4.3, 5, Appendix D.5"}],"minor_comments":[{"comment":"The claim of being the 'first multimodal conversational recommendation dataset' should be qualified relative to SURE (Long et al., 2023) and SIMMC 2.0 (Kottur et al., 2021), which are also multimodal and recommendation-oriented; if the distinction is the shopping-assistant formulation of CR, state that explicitly.","section":"Abstract, Section 1"},{"comment":"The fine-tuning experiments use only 200 conversations for LoRA training; please justify this choice and show sensitivity of the results to training-set size, or at least acknowledge that the reported gains may not transfer to larger training sets.","section":"Section 4.4"},{"comment":"Several pipeline hyperparameters are not fully specified, including the BLEU deduplication thresholds, the quality screening thresholds for user-scenario-product matching, and the exact temperature schedule; a short table in the appendix would aid reproducibility beyond the released code.","section":"Sections 3.2.1, 3.2.2, Appendix A.5"},{"comment":"In Table 2, '#Users' for MUSE is reported as 7.0K, but it is unclear how distinct users are determined given that each conversation appears to receive a newly generated profile; please clarify whether 'users' is simply the number of conversations.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a multimedia or IR venue. The main risk is the circularity of the dataset construction and evaluation; if the authors address the target-item guarantee and the LLM-judge confounding convincingly, the dataset could become a useful resource. The anonymous code link should eventually be replaced with a permanent public repository."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know about MUSE: it is a real, substantial resource—7,000 image-grounded conversations, 13.7K product images, a transparent multi-agent pipeline, and reproducible fine-tuning gains on three open MLLMs (Qwen2-VL Recall@10 rises from 0.20 to 0.34). The authors also publish cost breakdowns, prompts, and a helpful diversity analysis. That is more than many dataset papers deliver.\n\nThe soft spot is structural. The user simulator is given the target item and its requirements, and the Querier can end any conversation by handing the user that target (Section 3.3.2). So every conversation is a success story by construction. There are no abandons, no substitutions, no revealed needs that diverge from the target. The fine-tuning gains may therefore reflect learning the forced-convergence schedule rather than general preference inference. I think the stress-test concern lands, but it is not fatal: the assistant still has to infer the target from dialogue, so the data does contain learnable signal—just under an optimistic success model.\n\nThe quality evidence is thinner than the claims. The headline conversation-level scores come from an LLM judging LLM-generated text against other datasets. The human A/B is six same-institution graduate students, no agreement statistics, small samples. The 'first multimodal CR dataset' claim is defensible only under their narrow definition; they address SURE/SIMMC in B.6, and the distinction is real but partly a framing choice. The citation pattern is otherwise fine.\n\nWhat would move me from conditional to confident: an ablation where the round-limit fallback is removed (conversations can end unsuccessfully), external human evaluation with agreement metrics, and sensitivity checks on generation temperatures and screening thresholds. None of these are hard, and the pipeline is documented well enough to make them feasible.\n\nNet: this deserves a serious referee. It is a new artifact with honest engineering and a clear path to strengthening. I would not desk-reject it. If I worked on CR, I would cite it with a caveat about the success guarantee.","headline":"MUSE is a genuinely new synthetic multimodal CR resource with measurable fine-tuning gains, but the design guarantees every conversation ends in successful delivery of a pre-selected target, so the realism claims need serious hedging.","tokens_in":26118,"tokens_out":4097,"would_cite":true,"duration_ms":37793,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MUSE introduces the first multimodal conversational recommendation dataset, synthesizing 7,000 clothing-domain conversations from scenario-grounded user profiles, and closes the text-only gap in conversational recommendation research.","keywords":["multimodal conversational recommendation","conversational recommendation dataset","scenario-grounded user profiles","multi-agent framework","multimodal large language models","synthetic dialogue data","clothing domain","LLM-based evaluation"],"falsifier":"Take real customer-service transcripts from an online clothing retailer, match them to MUSE conversations on the same scenarios and needs, and have blind judges rate naturalness and realism: if MUSE consistently loses, the user-simulation premise fails. Alternatively, fine-tune the same three models on MUSE and evaluate on an independent real multimodal conversational recommendation setting; if the learned patterns show no benefit over text-only training, the claim that MUSE offers learnable recommendation behavior would collapse.","tokens_in":25049,"feed_emoji":"🛍️","tokens_out":13315,"duration_ms":104894,"temperature":0.7,"pith_summary":"MUSE is the first conversational recommendation dataset whose conversations carry real images alongside text, bridging a gap between research, which has been text-only, and real shopping, where seeing the product matters. The dataset contains 83,148 utterances in 7,000 clothing-domain conversations, generated automatically by a multi-agent framework powered by multimodal large language models: user profiles are built from real-world scenarios such as attending a graduation or a book launch, each profile is matched to a real product from the Amazon Clothing, Shoes, and Jewelry collection, and simulated users and assistants then talk through acceptances, rejections, and chit-chat. The scenario grounding is the proposed answer to a scalability problem: because earlier LLM-based CR datasets build profiles from user history, they cannot cover new users, cold starts, or privacy-restricted settings, whereas scenarios are effectively unbounded. The paper reports that human and LLM judges score MUSE conversations above existing CR and multimodal dialogue datasets on naturalness, coherence, informativeness, product-context relevance, and image-text alignment, and that fine-tuning three open multimodal LLMs on MUSE improves both recommendation retrieval and response quality.","feed_headline":"7,000 AI shopper chats debut first multimodal recommendation dataset","feed_subtitle":"Profiles from real-life scenarios let AI imitate shoppers without history data, easing cold-start and privacy concerns.","key_machinery":"The load-bearing mechanism is the Scenario-Grounded User Profile Generator, which expands seed shopping situations into 593 basic scenarios, then assembles each profile from three parts—basic user information, a target product from the local multimodal product database, and a purchase backstory—while an MLLM screens each (user, scenario, product) triple for plausibility and BLEU-based deduplication removes repeats. This mechanism carries the scalability argument: user and item coverage grows from the near-infinite space of real-world situations rather than from collected history, and each scenario profile provides the conversational grounding that the Simulated Conversation Generator (user, chatter, querier, and manager agents) turns into dialogue, after which the Rewriter and Reviewer agents diversify and filter the output.","core_discovery":"The central claim, stated on the paper's own terms, is that MUSE establishes the first multimodal conversational recommendation benchmark and that its scenario-grounded synthesis method removes the history-data bottleneck limiting earlier LLM-based datasets such as Pearl and LLM-Redial. Each conversation is anchored to a real product with an image and a description, and to a scenario-grounded profile that explains why the user is shopping, so the resulting dialogues weave together visual features, scenario requirements, and natural social talk. The paper reports consistent gains from fine-tuning three open multimodal LLMs—Qwen2-VL-7B, LLaVA-Next-LLaMA-8B, and Yi-VL-6B—on MUSE, including Qwen2-VL-7B lifting Recall@10 from 0.20 to 0.34, and blind human judges preferring fine-tuned responses over zero-shot responses at an 88% win rate, which the authors read as evidence that MUSE contains learnable recommendation and response patterns.","pith_inferences":["The strongest untested implication is domain transfer: the 593 scenarios encode general situations rather than clothing-specific logic, so the same profile generator could build furniture, food, or gift recommendation conversations by swapping the product database, an extension the authors gesture at in their conclusion.","The paper's reported 21.6% profile-qualification rate implies that screening, not generation, is the dominant cost and quality lever; a cheap replication with a weaker MLLM generator would reveal whether the quality scores come from the framework's filters or from the generator's underlying conversational skill.","The appendix finding that replacing images with gpt-4o-generated captions loses human preference almost completely (97% vs 3%) implies that the dataset's value lives in the visual grounding itself, so downstream systems should preserve true image inputs rather than caption substitutes.","If the scenario-grounded premise generalizes, it offers a privacy-preserving route to personalized CR data: profiles describe situations, not identifiable individuals, sidestepping the history-data regulations the paper cites as a barrier."],"forward_implications":["Fine-tuning on MUSE lifts recommendation retrieval across all three tested MLLMs — Qwen2-VL-7B's Recall@10 rises from 0.20 to 0.34 — so the dataset carries learnable recommendation logic, not just fluent dialogue.","Blind human judges prefer responses from MUSE-fine-tuned models over zero-shot responses at an 88% versus 12% win rate, indicating the conversations encode response patterns that transfer to other models.","Because profiles derive from real-world scenarios rather than user history, the same synthesis pipeline can expand to new domains, users, and products without cold-start or privacy constraints.","The ranking of the three evaluated models on MUSE matches their ranking on a public multimodal benchmark, giving MUSE discriminative power as a benchmark dataset.","The multi-stage automatic quality screening (with manual filtering removing only a small residue) positions the pipeline itself as a reusable template for cost-controlled CR dataset construction."],"supporting_citations":[{"why":"ReDial, the crowdsourced text-only CR dataset that defines the baseline paradigm and the modality gap MUSE claims to fill.","marker":"Li et al., 2018"},{"why":"Pearl, the LLM-synthesized CR dataset built from user history; MUSE contrasts with it on scalability and adapts its target-requirement method.","marker":"Kim et al., 2024"},{"why":"LLM-Redial, which synthesizes CR conversations from user behaviors and supplies the scalability framing plus the 200-conversation fine-tuning protocol MUSE follows.","marker":"Liang et al., 2024"},{"why":"The Amazon Clothing, Shoes, and Jewelry dataset that supplies the real products, images, and descriptions anchoring every MUSE conversation.","marker":"Hou et al., 2024"},{"why":"The self-instruct method used to expand seed scenarios into the 593 basic scenarios behind the user profiles.","marker":"Wang et al., 2022b"},{"why":"LoRA, the low-rank adaptation technique used to fine-tune the three multimodal LLMs in the recommendation and response experiments.","marker":"Hu et al., 2021"}],"fun_headline_variants":["First multimodal shopping chat dataset built on scenario profiles","Muse: 7,000 scenario-driven multimodal conversational dialogues","7,000 multimodal shopper chats, scenario-grounded, no history","Muse: scenario profiles, not history, power first multimodal recommendation dataset"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The dataset's realism rests on assuming that gpt-4o-mini, given only a scenario-based profile and a pre-selected target product, can act like a genuine shopper with believable needs, rejections, acceptances, and small talk; if that role-play is not convincing, the high quality scores mainly reflect the generator's style rather than real conversational recommendation behavior.","fun_headline_variants_meta":{"raw":{"variants":["First multimodal shopping chat dataset built on scenario profiles","Muse: 7,000 scenario-driven multimodal conversational dialogues","7,000 multimodal shopper chats, scenario-grounded, no history","Muse: scenario profiles, not history, power first multimodal recommendation dataset"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000928,"raw_usage":{"total_tokens":3964,"prompt_tokens":923,"completion_tokens":3041,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":539,"completion_tokens_details":{"reasoning_tokens":2969}},"tokens_in":539,"tokens_out":3041,"duration_ms":18518,"temperature":1.0,"reasoning_tokens":2969,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:41:24.214754+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take real customer-service transcripts from an online clothing retailer, match them to MUSE conversations on the same scenarios and needs, and have blind judges rate naturalness and realism: if MUSE consistently loses, the user-simulation premise fails. Alternatively, fine-tune the same three models on MUSE and evaluate on an independent real multimodal conversational recommendation setting; if the learned patterns show no benefit over text-only training, the claim that MUSE offers learnable recommendation behavior would collapse.","supporting_citations":[],"review_version":1}