{"id":"13ebc353-bc47-4cfe-9206-8f280b5c0090","arxiv_id":"2505.15128","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":7,"one_line_summary":"A pairwise relative-feedback scheme with a predictive user model that filters misaligned embedding sub-perceptions raises known-item video search rank-1 rates above PicHunter, though the evaluation uses a synthetic user.","lead":"This paper proposes a relevance feedback system for known-item video search in which users compare pairs of videos and a learned model decides which of several embedding spaces matches the user's judgment. The method improves top-rank hit rates over the classic PicHunter baseline on large video collections, but only under a simulated user model rather than real human feedback.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The robustness claim rests on a simulated user that is generated by the same majority-vote rule the model is trained to predict; real-user validation is missing, and the abstract's >40% success at depth 1000-5000 contradicts the conclusion's 6.4%.","rationale":"The paper is a competently written systems contribution with a plausible mechanism: pairwise judgments reduce cognitive load, and a predictive user model filters sub-perceptions that disagree with the observed choice. The ablations in Appendix A show that each component contributes, and the VBS t-KIS results are a genuine, if small, real-query signal. However, the central claim—robustness to imperfect, inconsistent user feedback—is supported only by simulated users generated from the same majority-vote rule that defines the training labels. This circularity means the evaluation cannot distinguish a model that truly adapts to diverse user perceptions from one that has memorized the simulator's decision rule. The fact that the abstract reports >40% success for initial ranks 1,000-5,000 while the conclusion reports 6.4% for the same regime is a separate, concrete inconsistency that further undermines confidence in the reported numbers; one of the two statements is wrong, and the headline claim depends on which one survives. The search-space pruning is not itself a fatal flaw, but it changes the semantics of 'rank 1', so the pruned results should be labeled as rank within the 5,000-candidate subset, not rank in the full collection. Because the central empirical assertion is not yet tested against real human feedback, and because a stated headline number appears to be contradicted by the paper's own conclusion, the appropriate verdict is REJECT for the paper in its current form. This does not preclude the underlying method being useful after a proper user study or after the numerical inconsistency is resolved and the claims are restated to match the evidence.","tokens_in":17871,"tokens_out":3052,"duration_ms":27194,"concrete_test":"Run a real-user study in which 10-20 human participants provide pairwise relative judgments on displays sampled by the greedy display model for a random subset of V3C2 queries. Compute human agreement with the majority-vote simulator on the same pairs. Then run the full system end-to-end using the human judgments instead of the simulator. If the predictive user model's confidence scores, trained on simulator labels, do not transfer—e.g., Recall@1 drops below PicHunter's simulator-based 0.4949, or human-simulator agreement is below 70%—the robustness claim fails. As a cheaper analytical check, recompute the depth 1000-5000 success rate from Figure 6; if it is ~6.4% rather than >40%, the abstract must be corrected before the central claim can be evaluated.","verdict_should_be":"REJECT","load_bearing_attack":"Section 4.3 defines the user simulator as majority voting over CLIP4Clip, ITV, and BLIP sub-perceptions. Section 4.5 trains the predictive user model to classify whether each sub-perception agrees with that same majority, using binary cross-entropy. Section 5.3 then evaluates the full system with the identical simulator. Consequently, test-time feedback is drawn from the exact distribution the model was optimized on, so the reported Recall@1 gains (0.5467, or 0.6384 with pruning) demonstrate robustness to this synthetic majority-vote user, not to real human inconsistency. The claim 'robust to inconsistent feedback' therefore remains untested. The search-space pruning in Section 4.4.3 also changes the evaluation: discarding all candidates initially ranked below 5,000 means 'rank 1' is relative to a 5,000-item subset, not the full dataset. Finally, the abstract's statement of >40% success for targets initially ranked 1,000-5,000 conflicts with the conclusion's 6.4% at 'depth beyond 1,000'; this internal inconsistency must be resolved before the headline numbers are relied upon.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses known-item video search with relevance feedback. It replaces PicHunter's top-k selection with pairwise relative judgments, decomposes user perception into three sub-perceptions (CLIP4Clip, ITV, and BLIP embeddings), and trains a predictive user model to assign confidence weights to these sub-perceptions before a soft Bayesian update. The system is evaluated on V3C1/V3C2 with a simulated user that selects the majority-vote choice among the three embedding-based distance comparisons. The paper reports Recall@1 improvements over PicHunter, including 0.5467 without pruning and 0.6384 with pruning after seven steps, and claims that targets initially ranked between 1,000 and 5,000 are promoted to rank 1 with success rate exceeding 40%.","tokens_in":18191,"tokens_out":5160,"duration_ms":46119,"significance":"If the claims were supported, the idea of learning to weight multiple embedding subspaces from user feedback would be a useful contribution to interactive video retrieval. The paper also provides a large-scale evaluation on V3C, an ablation study, and results on VBS t-KIS queries. However, the central claim of robustness to inconsistent human feedback is not established by the experiments: the simulated user that generates feedback at evaluation time is the same majority-vote rule that defines the training labels for the predictive model, so the reported gains may only reflect the model's ability to fit this synthetic rule. The internal contradiction between the abstract's 'success rate exceeding 40%' and the conclusion's '6.4%' for the same depth range further undermines the headline results. No real-user study is reported, despite the paper's emphasis on human inconsistency.","major_comments":[{"comment":"The evaluation is circular. Section 4.3 defines the user simulator as the majority vote of three sub-perceptions (CLIP4Clip, ITV, BLIP), and Section 4.5 trains the predictive user model with BCE labels y_i that indicate whether each feedback agrees with that same majority vote. Section 5.3 then evaluates the full system with the identical simulator. Consequently, the test-time feedback is drawn from exactly the distribution the model was optimized to predict. The reported Recall@1 gains over PicHunter therefore demonstrate that the model can learn to mimic the synthetic majority-vote rule, not that it is robust to real human inconsistency. The abstract's claim that the method works 'despite inconsistent feedback' is unsupported. A real-user study, or at minimum an independent noise model that is not derived from the training labels, is needed to substantiate the robustness claim.","section":"§4.3, §4.5, §5.3"},{"comment":"The headline numbers are internally inconsistent. The abstract states that for targets initially ranked between 1,000 and 5,000, the model 'achieves a success rate exceeding 40%' in optimizing ranks to the top. The conclusion, however, says that 'at a depth beyond 1,000, our approach can boost 6.4% of search targets to top-1.' These cannot both be accurate descriptions of the same experimental setup unless 'success rate' and 'boost' refer to different quantities, which the paper does not explain. The introduction (Section 1) also says 'our method achieves a 40% chance of reaching rank-1' for the same depth range. The authors must correct this contradiction and state precisely which metric is reported for which depth interval.","section":"Abstract vs. §6 (Conclusion)"},{"comment":"Search space pruning changes the meaning of the reported Recall@1 values. Section 4.4.3 proposes discarding all candidates outside the top 5,000 of the initial query ranking, and Section 5.4 applies this pruning before computing Recall@1. Thus 'rank 1' in the pruned results is relative to a 5,000-item subset, not to the full V3C2 collection, making the 0.6384 figure not directly comparable to the 0.5467 unpruned result. In addition, the evaluation queries are restricted to targets initially ranked no deeper than 5,000, so the method is not tested on targets below that depth even though the pruning step would exclude them from the search space entirely. The paper should report both unpruned and pruned results with the rank semantics made explicit, and discuss the coverage limitation.","section":"§4.4.3, §5.4, Table 2"},{"comment":"The description of model training is not sufficiently precise to establish what the predictor learns. The text says training uses 'simulated interactions in the perfect user setting' and then says the model is 'trained to predict the sub-perception of every user feedback at iteration t in the imperfect user setting,' but it does not define the label y_i in Eq. (7) beyond 'whether a feedback aligns with the sub-perception.' Given that Section 4.3 defines misalignment as disagreement with the majority vote, the y_i labels appear to be generated by the same rule as the test-time simulator. This should be stated explicitly, and the term 'perfect user setting' should be clarified, because it seems to contradict the construction of an imperfect user in Section 4.3.","section":"§4.5, Eq. (7)"}],"minor_comments":[{"comment":"There is a typo in Section 2: 'struggles to to maintain its effectiveness' should read 'struggles to maintain its effectiveness.'","section":"§2"},{"comment":"The notation 'EX P' appears in Eqs. (1) and (6); it should be formatted as 'exp' for readability and consistency with Eq. (4).","section":"§4.4.1, Eq. (1), Eq. (6)"},{"comment":"The pruning threshold is denoted N_prune in Section 4.4.3 but N_tprune in Section 5.1; please unify the notation.","section":"§4.4.3, §5.1"},{"comment":"Table 1 reports aggregate Recall@1 over all depths, while Figure 6 reports per-depth results. The text should clarify that the 0.5467 figure is an average across the five depth intervals, because readers may otherwise interpret it as a uniform success rate. The same applies to the pruning results in Table 2.","section":"§5.3, Table 1"},{"comment":"The formatting of Table 3, with 'Recall@1' and 'Recall@10' over three rounds, is hard to read because the column headers are not clearly separated; a layout with explicit per-round columns would improve clarity.","section":"§5.5, Table 3"},{"comment":"The paired t-test is mentioned but no standard deviations or per-depth significance values are reported; providing these would strengthen the comparison with PicHunter.","section":"§5.3"}],"recommendation":"reject","confidential_remarks":"The circular evaluation and the unresolved abstract/conclusion contradiction are load-bearing. The paper might be reconsidered if the authors add a genuine human-user study or an independently justified noise model, and if they correct the numerical inconsistency and clarify the effect of search-space pruning. As submitted, the central claim of robustness to inconsistent user feedback is not supported by the presented evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is a serious attempt to make relevance feedback for known-item video search more robust by moving from top-k judgments to pairwise comparisons, and by modeling the user as a mixture of sub-perceptions (CLIP4Clip, ITV, BLIP). The predictive user model that assigns confidence weights to each sub-perception before the Bayesian update is a genuine extension of PicHunter, and the ablation shows each component earns its keep. The authors also test across five depth ranges, two caption sources, and a small VBS query set; that is more thorough than most KIS papers.\n\nThe main soft spot is the simulated user. The user simulator is a majority vote over the same three embeddings that define the sub-perceptions, and the predictive model is trained to predict that majority. Evaluating on the same simulator therefore demonstrates that the model can learn the simulator, but not that it is robust to real human inconsistency. The paper's central claim—robustness to imperfect feedback—needs a real user study or at least a perturbation model that does not share the training labels. This is a load-bearing gap, not a cosmetic one.\n\nOne thing in the reader's report is off. The abstract's 'success rate exceeding 40%' for initial ranks 1000–5000 does not contradict the conclusion's 'boost 6.4% of search targets.' The 6.4% is clearly the improvement over PicHunter's 40.05% in the preceding sentence, so the total is around 46%. The wording is sloppy and should be fixed, but it is not an internal inconsistency.\n\nThe pruning experiment is also worth a caveat: discarding everything below rank 5,000 rewrites what 'rank 1' means. The gains with pruning are real, but they are relative to a 5,000-item subset, and the paper should say that clearly in the abstract and tables.\n\nBottom line: this is a competent systems paper with a novel combination and a credible mechanism. It deserves a serious referee who will push for external validation. I would not cite it for the robustness claim until that validation exists, but it is a useful reference for the pairwise-judgment-plus-multi-perception idea.\n\nRecommendation: send to peer review with the expectation of major revisions requiring either a human-in-the-loop experiment or a simulator that is not learned from the same labels.","headline":"A competent KIS relevance feedback paper with a genuinely novel combination, but the robustness claim is untested because the simulated user is trained and evaluated on the same majority-vote labels.","tokens_in":18660,"tokens_out":4471,"would_cite":false,"duration_ms":35526,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A system that predicts which of several similarity perceptions a user is relying on can promote most hard-to-find videos to the top rank in seven rounds of pairwise feedback.","keywords":["Known-item search","relevance feedback","video retrieval","pairwise judgment","user perception modeling","Bayesian update","interactive video search","V3C dataset"],"falsifier":"Have human users make the same pairwise relative judgments on a sample of displayed pairs from V3C2, and check whether the model's per-sub-perception confidence scores are calibrated to which embedding feature agrees with the human choice; if human judgments do not track the majority-vote simulator or the confidence scores do not predict agreement, the reported robustness collapses.","tokens_in":17663,"feed_emoji":"🎬","tokens_out":6129,"duration_ms":47371,"temperature":0.7,"pith_summary":"The paper claims that interactive known-item video search can be made robust to inconsistent user feedback by replacing top-k selections with pairwise relative judgments and by treating user perception as a mixture of several embedding-based sub-perceptions. A predictive model assigns a confidence score to each sub-perception for each feedback instance, and the Bayesian update uses those scores to down-weight judgments that are misaligned with the user's apparent choice. On the million-scale V3C collections, the method promotes more than 60% of targets with initial ranks 10-50 to the top rank within seven rounds, and more than 40% of targets initially ranked 1,000-5,000. With an explicit pruning of the initial search space to the top 5,000 candidates, overall Recall@1 reaches 0.6384. If the result holds, it would mean relevance feedback remains useful in single-target search even when users cannot reliably mirror the machine's similarity measure.","feed_headline":"Interactive video search lifts 60% of hard targets to rank 1","feed_subtitle":"Pairwise judgments plus per-feature confidence beat the PicHunter baseline on million-video collections.","key_machinery":"The load-bearing object is the pairwise relative judgment expressed as a feature-difference vector, $v_{\\text{diff}} = v_{+} - v_{-}$, where $v_{+}$ and $v_{-}$ are the representations of the chosen and unchosen videos in a pair. The paper observes that the similarity $s(v_{\\text{diff}}, v_i)$ is proportional to the term $\\left[s(v_{+}, v_i) - s(v_{-}, v_i)\\right]$ that drives PicHunter's Bayesian update, so the difference vector is a natural input for predicting which sub-perception the user is following. A transformer-based predictive user model consumes the query, the interaction history, the search state, and a learnable distance embedding of $\\|v_{\\text{diff}}\\|$ (quantized into 100 bins) to output per-sub-perception confidence scores, and a soft Bayesian update multiplies each sub-perception's evidence by the confidence score. This combination is what lets the system down-weight misaligned sub-perceptions instead of assuming the user always agrees with the machine.","core_discovery":"Known-item search has exactly one target, so standard relevance feedback's positive examples are unavailable; the paper's central claim is that the classic PicHunter Bayesian update can still be made robust by letting the user give pairwise relative judgments ('which of these two is closer to the target?') and by modeling the user's perception as a weighted combination of sub-perceptions, each living in a separate embedding space (CLIP4Clip, ITV, and BLIP). A transformer-based predictor consumes the initial query, the current search state, and the interaction history to output a confidence score for each sub-perception at each feedback, and the Bayesian update weights each sub-perception's similarity evidence by that confidence, filtering out sub-perceptions that disagree with the user's choice. The paper reports that this raises seven-step Recall@1 on V3C2 to 0.5467 from PicHunter's 0.4949, that over 60% of targets initially ranked 10-50 and over 40% of targets initially ranked 1,000-5,000 reach rank 1, and that pruning candidates below rank 5,000 lifts Recall@1 to 0.6384.","pith_inferences":["We infer that the same perception-filtering idea should transfer to ad-hoc video search with multiple relevant items, where a user's notion of relevance may also switch between sub-perceptions across a session.","We infer that the reported 'rank 1' after pruning is rank among the pruned candidates, so the gap between pruned and unpruned success suggests some targets below rank 5,000 are simply excluded rather than retrieved.","We infer a testable scaling property: the method's advantage over PicHunter should grow when the three embedding features disagree on displayed pairs and shrink when they agree, because the predictive model only has something to filter when sub-perceptions conflict.","We infer that the pairwise difference representation $v_{\\text{diff}} = v_{+} - v_{-}$ could be reused as a generic preference signal in other interactive ranking systems, independent of the Bayesian update it feeds here."],"forward_implications":["Within seven rounds of pairwise feedback, the model moves over 60% of targets with initial rank 10-50 and over 40% of targets with initial rank 1,000-5,000 to the top position.","Overall Recall@1 after seven steps is 0.5467, above PicHunter's 0.4949; with search-space pruning to the top 5,000 initial candidates it reaches 0.6384.","On the 17 VBS textual KIS queries, the model ranks 16 targets at top-1 with the full three-round query text and all 17 targets within top-10.","The choice of captioning model (BLIP2 vs LLaVA-NeXT-Video) has minimal impact on Recall@1 across all search-depth ranges.","Ablations show that removing the soft weighted update hurts most (0.5467 to 0.3321), while removing the distance embedding costs less but the gap grows over rounds."],"supporting_citations":[{"why":"PicHunter's Bayesian relevance feedback framework for known-item search; this paper extends it by replacing top-k judgments with pairwise judgments and adding perception filtering.","marker":"[2-4]"},{"why":"V3C1 dataset used to train the predictive user model.","marker":"[1]"},{"why":"V3C2 dataset used as the million-scale evaluation collection.","marker":"[17]"},{"why":"CLIP4Clip embeddings, one of the three sub-perceptions the user model filters.","marker":"[16]"},{"why":"ITV embeddings, another sub-perception used for similarity and feedback simulation.","marker":"[24]"},{"why":"BLIP embeddings, the third sub-perception; together the three define the majority-vote user simulator.","marker":"[10]"},{"why":"BLIP2 captions used to generate one set of textual queries from target videos.","marker":"[9]"},{"why":"LLaVA-NeXT-Video captions used as the other query source; caption source has little effect on results.","marker":"[29]"},{"why":"VBS textual KIS queries supply the 17-query benchmark on which the method reaches 16/17 top-1.","marker":"[12]"}],"fun_headline_variants":["Pairwise feedback beats PicHunter for one-target video search","Sub-perception model makes relevance feedback robust for single-target search","Even noisy user feedback can still find the one video, says new model","Interactive known-item search: pairwise preferences beat misaligned top-k","60% of mid-ranked targets hit rank 1 with pairwise feedback"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation and the training signal both assume that a user's inconsistent judgment is the majority vote of the three embedding features that define the sub-perceptions; if real human disagreement does not follow that rule, the claimed robustness is not tested.","fun_headline_variants_meta":{"raw":{"variants":["Pairwise feedback beats PicHunter for one-target video search","Sub-perception model makes relevance feedback robust for single-target search","Even noisy user feedback can still find the one video, says new model","Interactive known-item search: pairwise preferences beat misaligned top-k","60% of mid-ranked targets hit rank 1 with pairwise feedback"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001447,"raw_usage":{"total_tokens":5897,"prompt_tokens":1084,"completion_tokens":4813,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":700,"completion_tokens_details":{"reasoning_tokens":4724}},"tokens_in":700,"tokens_out":4813,"duration_ms":32802,"temperature":1.0,"reasoning_tokens":4724,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:23:19.427411+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have human users make the same pairwise relative judgments on a sample of displayed pairs from V3C2, and check whether the model's per-sub-perception confidence scores are calibrated to which embedding feature agrees with the human choice; if human judgments do not track the majority-vote simulator or the confidence scores do not predict agreement, the reported robustness collapses.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"V3C1 dataset used to train the predictive user model."},{"cited_title":"Multimedia Comput","cited_arxiv_id":null,"evidence_quote":"V3C2 dataset used as the million-scale evaluation collection."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"BLIP embeddings, the third sub-perception; together the three define the majority-vote user simulator."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"BLIP2 captions used to generate one set of textual queries from target videos."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"LLaVA-NeXT-Video captions used as the other query source; caption source has little effect on results."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"VBS textual KIS queries supply the 17-query benchmark on which the method reaches 16/17 top-1."}],"review_version":1}