{"id":"5b8855b3-dbf7-491d-b923-d5fd5d2c2824","arxiv_id":"2507.18017","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"The paper contributes alternative-item relevance judgments for fashion CRS targets and two meta-simulators that let users switch targets, reporting that alternative-aware evaluation raises measured CRS effectiveness.","lead":"Fashion-AlterEval adds human judgments about alternative relevant items to two fashion CRS datasets, and introduces two meta-user simulators that let simulated shoppers switch targets. Tests with three CRS models suggest single-target simulators understate system effectiveness, but the speed-up is partly driven by how the simulators pick the new target.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Algorithm 1/2 choose the new target as the alternative most similar to the current top-1 item, so the reported gains may be an artifact of making the next target nearly identical to what the system already displayed.","rationale":"The reader's weakest assumption identifies the same load-bearing concern I see: the meta-simulators select the new target as the alternative most similar to the system's current top-1 item. This is the step that makes the reported improvements look almost inevitable, because the next target is chosen to be visually close to what the system just showed and because any relevant alternative counts as success. Without evidence that this selection rule models real user switching behavior, the headline claim about rapid user satisfaction is not supported by Table 3. I also considered whether the paper's comparison of metrics across different relevance sets is itself the core problem, since enlarging the relevant set can mechanically raise MRR and NDCG. That is related but less central: expanding the qrels to legitimate alternatives is the paper's stated purpose, so the key empirical question is whether the simulated user's choice of a new target reflects how real users would behave. The proposed random-alternative ablation would settle this directly by removing the similarity bias while preserving the expanded relevance set. I keep the reader's CONDITIONAL verdict rather than moving to REJECT because the dataset construction appears careful, the resource itself is reusable, and the concern is empirically testable rather than a fundamental flaw in the annotation methodology.","tokens_in":14571,"tokens_out":3606,"duration_ms":41612,"concrete_test":"Run the same Table 3 evaluation but replace Algorithm 1 line 3 and Algorithm 2 line 5 with a random draw from the relevant alternative set for that target, using the same seeds and multiple replicates. If mean MRR@10 and NDCG@10 improvements over SimBase collapse toward zero or lose statistical significance, the headline gains are caused by the similarity-based switch rather than by the availability of alternatives. A complementary check is to present human participants with the actual top-1 image at the switch turn and elicit which alternative they would adopt; low agreement with the argmax-similarity choice would directly falsify I2.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim—Table 3 reports up to +149.92% MRR@10—rests on the way alternatives become the new target. In Algorithm 1 line 3 and Algorithm 2 line 5, target = alts[arg max(sims(alts, top1k))], i.e., the simulated user switches to the relevant alternative with maximum image similarity to the item the system just ranked first. This operationalizes intuition I2 as a modeling assumption, but the paper provides no behavioral evidence that real users select substitutes this way. Because the new target is chosen to be nearly identical to the candidate the user is currently seeing, the next critique will describe only small differences, and the ranker is very likely to retrieve that target quickly; the effect is amplified because any alternative counts as a successful retrieval. The reported improvements therefore conflate 'CRS satisfies an alternative-accepting user' with 'the simulator picked a target adjacent to the system's current guess.' The dataset collection is solid and the simulators are clearly described, but the evaluation numbers cannot support the abstract's claim without validating I2 or testing robustness to alternative selection rules. Additionally, Table 3 reports the best Pswitch per cell and no variance, which further weakens the size of the reported gains.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Fashion-AlterEval, a dataset of human judgments of alternative relevant items for a sample of target items from the Shoes and FashionIQ Dresses conversational-fashion datasets. The authors also propose two meta-user simulators, MetaSimTol and MetaSimProb, that wrap an existing single-target relative-captioning simulator and allow the simulated user to switch to an alternative target after a fixed tolerance or probabilistically after a perceived loss. They evaluate three CRS models (GRU-SL, GRU-RL, EGE) with and without these simulators and report large improvements, e.g., up to +149.92% MRR@10 for EGE on Dresses, concluding that single-target evaluation underestimates CRS effectiveness and that alternative-aware simulators allow systems to satisfy users more quickly.","tokens_in":14800,"tokens_out":4147,"duration_ms":39203,"significance":"The dataset construction is a genuine contribution: it uses a power analysis, pools candidates from multiple systems, reports an assessor agreement of Cohen's kappa 0.87, and is publicly released with code. If the central claim were supported, the paper would provide a practical test-collection resource for evaluating conversational recommendation with alternative relevance. However, the main quantitative claim depends heavily on a simulator design choice that is not validated, and the reported effect sizes are likely inflated. The paper deserves credit for making the data and code available, but the experimental evidence as currently presented is not sufficient to establish that single-target evaluation underestimates real effectiveness.","major_comments":[{"comment":"In both meta-simulators, the simulated user selects as the new target the alternative with maximum image similarity to the currently displayed top-1 item (target = alts[arg max(sims(alts, top1k))]). This operationalizes intuition I2 but is not grounded in observed user behavior. Because the new target is near-identical to the item the system just ranked first, the next critique describes only small differences, and the ranker is highly likely to retrieve that near-identical alternative quickly. The reported improvements in Table 3 (up to +149.92% MRR@10) therefore conflate \"the CRS satisfies an alternative-accepting user\" with \"the simulator chose a target adjacent to the system's current guess.\" To support the central claim, the paper should include a robustness ablation that selects the new target differently: e.g., uniformly at random among judged alternatives, least similar to the displayed item, or by using human judgments of similarity rather than image-embedding similarity. Without such an ablation, the direction and size of the reported gains are not identifiable.","section":"4.1, Algorithm 1 line 3; 4.2, Algorithm 2 line 5"},{"comment":"For each tolerance level of MetaSimProb, the paper reports the best metric over the threshold grid [0.55, 0.65, 0.75, 0.85, 0.95] (Section 5.1). This amounts to selecting a free parameter per cell to maximize the reported improvement, and no variance or confidence intervals are given. Consequently, the comparison between MetaSimProb and SimBase (and between MetaSimProb and MetaSimTol) is optimistic by construction. Please report results for a single a priori threshold (e.g., 0.75, as derived from Klein et al.) or average over thresholds, and provide standard errors over the 200 target items or over multiple simulation seeds.","section":"5.3, Table 3"},{"comment":"The headline claim that \"single-target evaluation underestimates their effectiveness\" is contradicted by the GRU-RL Shoes row, where NDCG@10 drops by 13.75% while MRR@10 improves by 64.80%. The paper does not explain this divergence. The evaluation also counts a conversation as successful if any alternative is retrieved (Section 5.2), which combined with the near-identical target selection amplifies the already high MRR numbers. Please discuss this discrepancy and reconcile the metric-level conclusions with the abstract's blanket statement.","section":"5.3, Table 3"}],"minor_comments":[{"comment":"\"the realisticity of an interaction\" should be \"the realism of an interaction.\"","section":"1, paragraph 2"},{"comment":"\"making a total cost of the study was $305\" is ungrammatical; revise to \"the total cost of the study was $305.\"","section":"3.2.2"},{"comment":"The notation sims(alts, top1k) is not formally defined; specify which image embedding and similarity function are used, and whether the same representation is used for the CRS ranker's top-1 item.","section":"4.1, 4.2"},{"comment":"The sentence \"We convert Cohen's d to probability, which gives a value of 0.75\" needs a formula or reference; the conversion depends on the assumed distribution and is not self-evident.","section":"5.1"},{"comment":"Check the SimBase row for GRU-RL on Dresses; MRR@10 (0.269) being higher than NDCG@10 (0.099) is unexpected and may be a typo or misalignment.","section":"Table 3"},{"comment":"The description of training with \"all target items instead of a sample of 200\" is underspecified; clarify which items are used, how the models are trained, and why this comparison is appropriate.","section":"5.5, Figure 8"},{"comment":"The stated limitation \"our simulators do not consider multiple alternatives\" is slightly misleading; the simulators select a single alternative at a time, but the dataset provides multiple judged alternatives per target. The paper should clarify that the current implementation does not use the full alternative set as a relevance set during simulation.","section":"Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The dataset and code release are valuable and likely to be useful to the community. The central experimental claim, however, is not yet convincing because the simulator's target-selection rule appears to guarantee rapid success by construction. I would urge the editor to require a robustness analysis before accepting the evaluation conclusions. The paper is otherwise well structured and the data collection methodology is sound."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know one thing up front: the dataset work here is solid, but the main quantitative claim does not survive contact with the simulator design. The paper reports up to +149.92% MRR@10 when simulated users can switch to alternative items, and then argues that single-target evaluation underestimates CRS effectiveness. That argument is largely forced by construction. In Algorithm 1 and Algorithm 2, when the user runs out of patience, the new target is set to the alternative with maximum image similarity to the currently top-ranked item. So the critique the simulator emits after a switch describes small differences from something the system already displayed, and the ranker is almost guaranteed to retrieve that near-identical item quickly. The result measures the simulator rule, not general user behavior. The paper presents no evidence that real users pick substitutes this way (intuition I2). That is not a minor caveat; it is the load-bearing assumption behind the headline claim.\n\nWhat the paper does well is the dataset construction. The power analysis to choose target sample size, the pooling strategy from two CRS models, the attention checks, and the Cohen's kappa of 0.87 on a 40-item subset all indicate careful human annotation work. Releasing data and code on GitHub and OSF is proper and reproducible. The idea of asking assessors for alternative relevant items, and then wrapping existing single-target simulators, is a useful step toward test-collection-style evaluation for conversational recommendation. There is genuine value in the artifact itself.\n\nThe soft spots, in order of severity: (1) the target-switching rule makes the new target nearly identical to the current display, so rapid success is baked in; (2) Table 3 reports the best Pswitch per cell and gives no variance or significance testing, making the size of the reported gains unreliable; (3) the paper acknowledges it only considers switching to one alternative at a time, not multiple alternatives. None of these destroy the dataset, but together they mean the abstract's central claim about quicker satisfaction is not supported by the experiments as run.\n\nThis is a paper for CRS evaluation researchers who want reusable relevance judgments for alternative items. It deserves a serious referee: the dataset and simulator framework are worth publishing, but the evaluation section needs a reworked experimental design, ideally with a validated or at least varied user-switching model, and confidence intervals. I would send it to review, but I would expect major revisions before acceptance.","headline":"The dataset is the real contribution; the headline 'rapid satisfaction' result is an artifact of the simulator's own target-selection rule and should not be taken at face value.","tokens_in":15320,"tokens_out":1177,"would_cite":true,"duration_ms":15511,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that evaluating conversational fashion recommenders against a single target item understates their effectiveness, and that letting simulated users switch to visually similar alternatives lets systems satisfy users sooner.","keywords":["conversational recommendation","user simulation","relevance judgments","alternative items","fashion retrieval","evaluation methodology","meta-simulator","test collections"],"falsifier":"Run the same three CRS models with meta-simulators that select the new target randomly, or from human-elicited alternative preferences, instead of by maximum image similarity to the top-ranked item; if the MRR@10 gains drop far below the reported values, the effect is an artifact of the similarity-based selection rule rather than a property of alternative-aware evaluation.","tokens_in":14280,"feed_emoji":"👗","tokens_out":6200,"duration_ms":60119,"temperature":0.7,"pith_summary":"This paper argues that offline evaluation of conversational recommendation systems is too pessimistic because simulated users are assumed to want exactly one target item and to be infinitely patient. To fix this, it contributes Fashion-AlterEval, human relevance judgments marking which candidate items are acceptable alternatives for 200 target shoes and 200 target dresses from two existing fashion datasets. It wraps the standard single-target simulator with two meta-simulators: one that switches to the closest alternative once patience runs out, and one that switches probabilistically after a perceived loss. On three CRS models, using alternatives raises MRR@10 by up to 149.92% and often raises nDCG@10, leading the paper to conclude that previous evaluations underestimated how quickly systems can satisfy users who accept substitutes.","feed_headline":"Letting simulated users switch items lifts CRS scores by up to 150%","feed_subtitle":"Human judgments of substitute fashion items show single-target evaluation underrates conversational recommenders.","key_machinery":"The central object is Fashion-AlterEval, a set of relevance judgments in which human annotators marked a pool of candidate images as sufficient alternatives for 200 target items in each of two fashion categories; on average 3.5 alternatives per target were judged relevant, with high annotator agreement. The two meta-simulators are the operational mechanism: they take the existing target-based critique function $Usersim.critique(turn, top\\_ranked, target)$ and re-target it, selecting a new target $alts[\\arg\\max(sims(alts, top1k))]$ once the turn count exceeds a tolerance, or, in the probabilistic variant, only when the similarity difference $\\delta = sim(top1k, target) - sim(top1k-1, target)$ is negative and a random draw falls below $P_{switch}$. This machinery converts a single-target evaluation into a multi-target one by reusing the same learned critique model, so the observed gains come from expanding what counts as relevant rather than from retraining the recommender.","core_discovery":"The central claim is that existing single-target user simulators underestimate the effectiveness of conversational fashion recommenders. The paper proposes that relevance judgments for alternative items, collected from human assessors, can extend the target space of simulated users, and it introduces two meta-user simulators that wrap an existing relative-captioning simulator. MetaSimTol switches to the alternative most visually similar to the currently top-ranked image after a tolerance threshold of turns, and MetaSimProb adds a gain-loss heuristic so that a user who perceives a drop in similarity from the previous turn switches to an alternative with probability $P_{switch}$. Across GRU-SL, GRU-RL, and EGE on Shoes and Dresses, the alternative-aware simulators produce large gains at turn 10, up to +149.92% in MRR@10 for EGE on Dresses, which the paper interprets as evidence that single-target evaluation systematically understates how well these systems can respond to a user who is willing to change their mind.","pith_inferences":["An implication the paper leaves implicit is that the same alternative judgments could support a single-turn retrieval evaluation, in which any judged alternative counts as a hit, making offline CRS numbers directly comparable to standard image retrieval test collections.","A natural stress test is to replace the similarity-based alternative selection with alternatives drawn from human preference judgments or at random; if the gains mostly disappear, the reported improvement reflects the selection rule rather than the value of alternative-aware evaluation.","Because the new target is chosen to look almost identical to the last displayed image, the next critique describes only small visual differences, which makes rapid retrieval of that new target nearly automatic; a fairer reading of +149.92% is that it measures estimation of a broader user need, not an improvement in the recommender itself."],"forward_implications":["If the central claim is right, previously reported CRS numbers on the Shoes and FashionIQ Dresses datasets understate how well these systems serve users who accept substitute items.","The meta-simulators plug into any existing critique-based user simulator without retraining the recommender, because they only change which target image the critique call is asked about.","Switching earlier in the dialogue produces a larger difference between old and new evaluation estimates, while a more patient simulated user is already receiving better rankings before the switch.","The probabilistic gain-loss variant is consistently at least as good as the fixed-tolerance variant at tolerances 2 and 3, especially for models that attend mainly to the previous turn."],"supporting_citations":[{"why":"Provides the relative-captioning user simulator and the dialog-based interactive image retrieval framework that the meta-simulators wrap and evaluate.","marker":"[16]"},{"why":"Source of the Shoes image collection used as one of the two original fashion CRS datasets.","marker":"[2]"},{"why":"Source of the FashionIQ Dresses targets and relative captions, the other original dataset enriched with alternative judgments.","marker":"[48]"},{"why":"Defines the EGE conversational recommender, one of the three models whose evaluation changes under alternative-aware simulators.","marker":"[50]"},{"why":"Supplies the correlation effect sizes used in the power analysis that determines how many target items to annotate.","marker":"[46]"},{"why":"The deep-learning track retrieval evaluation used as evidence that deep relevance judgments can be more reliable than thousands of shallow ones, motivating the alternative-judgment dataset.","marker":"[11]"},{"why":"The gain-loss framing effect that motivates the probabilistic switching rule in MetaSimProb.","marker":"[43]"},{"why":"The large-sample replication used to set the default probability of switching to an alternative after a perceived loss.","marker":"[23]"}],"fun_headline_variants":["Simulated users who switch items expose hidden CRS gains","Fashion-AlterEval: alternative items reshape CRS evaluation","Single-target evaluation underestimates conversational recommenders","Meta-simulators with alternative items raise CRS metrics up to 150%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a user who decides to accept an alternative will pick the item that looks most like the one the system just displayed; if real users switch on other grounds, the reported gains may not transfer.","fun_headline_variants_meta":{"raw":{"variants":["Simulated users who switch items expose hidden CRS gains","Fashion-AlterEval: alternative items reshape CRS evaluation","Single-target evaluation underestimates conversational recommenders","Meta-simulators with alternative items raise CRS metrics up to 150%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000619,"raw_usage":{"total_tokens":2890,"prompt_tokens":985,"completion_tokens":1905,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":601,"completion_tokens_details":{"reasoning_tokens":1833}},"tokens_in":601,"tokens_out":1905,"duration_ms":13860,"temperature":1.0,"reasoning_tokens":1833,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T14:39:38.200576+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same three CRS models with meta-simulators that select the new target randomly, or from human-elicited alternative preferences, instead of by maximum image similarity to the top-ranked item; if the MRR@10 gains drop far below the reported values, the effect is an artifact of the similarity-based selection rule rather than a property of alternative-aware evaluation.","supporting_citations":[{"cited_title":"& Kahneman, D","cited_arxiv_id":null,"evidence_quote":"The gain-loss framing effect that motivates the probabilistic switching rule in MetaSimProb."},{"cited_title":"many labs","cited_arxiv_id":null,"evidence_quote":"The large-sample replication used to set the default probability of switching to an alternative after a perceived loss."}],"review_version":1}