{"id":"283579c3-c91e-445a-8f5a-2c3dbceb28b4","arxiv_id":"2508.14911","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A utility-based active sampling strategy for pairwise preference learning picks the questions that most improve expected recommendation quality, outperforming random and uncertainty-based baselines in two experiments.","lead":"This paper proposes a recommender system that learns your tastes by asking you to choose between pairs of items, and it decides which pairs to ask by estimating which answers would most improve the final recommendation. The goal is to get better recommendations from fewer questions, which matters for new users and high-stakes choices like admissions.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported active-sampling gains depend on an unspecified computation of θ|q in Eq. (9); the paper gives no implementation or complexity bound for hypothetical retraining, so the experiments may not evaluate the proposed utility gain.","rationale":"The reader's weakest assumption is exactly the missing definition of θ|q. This is load-bearing because Eq. (9) is the proposed active sampler, and every reported result (MovieLens max-rank gains, admissions Precision@10/NDCG@10) is attributed to it. The paper's only stated approximation, Monte Carlo over rankings, does not resolve the hypothetical-retraining cost. Algorithm 1 computes expected utility for a fixed θ; it says nothing about how θ updates under hypothetical query outcomes. The protocols only retrain after a query is chosen, which is not the same as evaluating θ|q for all candidate pairs. This does not disprove the method, but it leaves the experimental results under-specified at the decisive step. The reader's conditional verdict remains appropriate, and no additional concern changes it.","tokens_in":13226,"tokens_out":6992,"duration_ms":82561,"concrete_test":"Ask the authors to release the active-sampling code or run the following ablation: on a 50-user, 100-item subset of MovieLens, compute the first 5 queries by exactly evaluating Eq. (9) — full retraining from the same seed for both outcomes of every candidate pair — and compare the chosen pairs and final max-rank utility against the paper's reported selector and against a single-gradient-step approximation. If the exact selector's choices differ materially, or if exact evaluation is computationally impossible at the scale claimed, the experimental support for Eq. (9) collapses.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Eq. (9) selects the pair maximizing expected utility gain, which requires θ|q — the parameters after retraining on each possible query outcome — for every candidate pair. With n=400 applicants there are ~79,800 pairs, so one query would require ~159,600 model fits if θ|q is exact retraining; with n≈1,700 movies it is worse. The paper never states whether θ|q is full retraining, a single gradient step, a low-rank update, or approximated by using the current θ. The protocols only say that after a query the model is retrained; that does not implement Eq. (9). The Monte Carlo approximation in Algorithm 1 addresses the sum over rankings, not the hypothetical-parameter problem. If the implementation actually used a shortcut, the reported superiority over random and entropy sampling could be an artifact of that shortcut rather than of utility-based active selection. The central claim ('active utility-based sampling improves data efficiency') therefore rests on an unspecified, potentially infeasible computation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a utility-based framework for learning personalized recommendations from pairwise comparisons. A Plackett-Luce model is used to represent distributions over rankings, and recommendation menus are selected by maximizing an arbitrary expected utility (e.g., max-rank for media, top-k coverage for admissions). The central methodological contribution is an active sampling strategy, Eq. (9), which chooses the pairwise query expected to yield the greatest utility gain, approximated by Monte Carlo sampling. Two experiments are reported: a MovieLens matrix-factorization task comparing active, random, and no-sampling regimes, and a graduate-admissions neural-network task comparing active sampling with entropy, random, and cluster-based baselines. The paper claims that the proposed active sampling substantially improves data efficiency and recommendation quality.","tokens_in":13521,"tokens_out":3303,"duration_ms":35748,"significance":"The framework is appealing in principle: it cleanly separates the recommendation objective from the preference model and attempts to align active query selection with the final utility rather than generic uncertainty. The use of Plackett-Luce for sampling ranking distributions is standard and the Monte Carlo utility approximation in Algorithm 1 is sensible for a fixed parameter vector. If the active sampling step were fully specified and the experiments validated, the paper would make a useful contribution to preference elicitation for recommender systems. However, the central active-sampling computation is not actually specified: Eq. (9) requires hypothetical retrained parameters θ|q for every candidate pair and outcome, and no implementation or approximation is given. The reported experimental results therefore cannot be reproduced or even exactly interpreted. In addition, Algorithm 1, as written, computes the top-k coverage utility of Eq. (8) rather than the max-rank utility of Eq. (7) that Experiment 1 claims to use. These issues are load-bearing, so the significance of the empirical claims is not established in the current version.","major_comments":[{"comment":"The selection rule g(θ) requires θ|q, the model parameters after retraining on each hypothetical query outcome. The manuscript never states how θ|q is obtained: full retraining, one gradient step, a low-rank update, or an approximation using the current θ. This is not a minor implementational detail: with n=400 candidates there are roughly 79,800 pairs, and if exact retraining is required, one query would require on the order of 160,000 model fits; for MovieLens (n≈1,700) it is worse. The experimental protocols only say that after a real query the model is retrained; they do not indicate how the utility gain in Eq. (9) is evaluated for candidate pairs. The Monte Carlo approximation in Algorithm 1 addresses the expectation over rankings, not the hypothetical-parameter problem. Without this specification, the reported superiority over random and entropy sampling could be an artifact of wha","section":"Utility-Based Active Sampling, Eq. (9)"},{"comment":"Algorithm 1 computes the utility by counting how many of the sampled top-k items fall in the menu m (lines 13–17), which is exactly the Admissions utility of Eq. (8). Experiment 1 explicitly states that the utility-based active sampling uses the Goods and Media utility function in Eq. (7), where utility is the highest rank (best position) achieved by any recommended item. These are different objectives, and the truncation to the first k sampled items is only without loss of generality for Eq. (8), not for Eq. (7), because the max rank can lie beyond k. The implementation therefore does not match the stated active-sampling criterion in Experiment 1, and the results in Figure 1 may not reflect the claimed max-rank utility.","section":"Algorithm 1 and Experiment 1"},{"comment":"The appendix's ground-truth preference orderings are generated by sampling from a Plackett-Luce model whose parameters are the shadow ratings u_u^T v_i. The comparison-based model being tested is also a Plackett-Luce model over pairwise comparisons. This means the ground truth is drawn from the same model family as the proposed method, which structurally favors the comparison-based approach over the rating-based MF, whose objective is only indirectly related to the PL generative process. The claim that the comparison-based model fundamentally outperforms rating-based prediction is therefore not supported by this experiment. Additionally, Figure 4 compares training loss of the comparison-based model (a log-likelihood) with MSE of the rating-based model on different scales; the two curves are not directly comparable as evidence about overfitting or objective alignment.","section":"Appendix: Semi-synthetic experiment"},{"comment":"Several protocol details needed to reproduce the central results are missing. In Experiment 1, it is unclear how the active pairs are selected given Eq. (9) (see first comment), what Monte Carlo sample size R is used, how many retraining epochs are run after each query, and what 'starting from 10 users per epoch' means for the x-axis 'Number of Queries'. The statement that the no-sampling baseline has different performance for different query counts because the model is re-sampled and re-trained is confusing; if no queries are added, performance should be constant up to randomness. In Experiment 2, the latent random vector z is introduced in the model description but its role in active sampling is not explained. These details matter because the reported differences are large and the central comparison depends on the exact active-selection procedure.","section":"Experiments: Protocol and reproducibility"}],"minor_comments":[{"comment":"Typos and formatting issues: 'pertaining' should be 'pretraining'; 'V olinsky' in the references; 'F ourth' in the RecSys reference; 'Architecture' in the NeuMF description; 'Morgen & Claypool' should be 'Morgan & Claypool'.","section":"Throughout"},{"comment":"The relationship between θ_ui and s_ui is stated as s_ui = ln θ_ui, but Eq. (2) uses exp(s_ui). This is consistent, but the notation is easy to misread; a short clarification would help.","section":"Eq. (1)-(2)"},{"comment":"The Laplace smoothing formula uses θ_i without defining it in the appendix (presumably the latent score). Since the appendix is self-contained in part, please define θ_i and α.","section":"Appendix Eq. (10)"},{"comment":"The figure caption says 'average over 100 runs' but the main text does not state the number of random seeds or confidence intervals for Experiment 2. Please include this information or state that no error bars are shown.","section":"Figure 2"},{"comment":"The discussion of Mallows-based elicitation is useful, but the claim that Plackett-Luce is 'well-suited for the efficient, iterative updates needed to evaluate potential queries' is not substantiated and, given the missing θ|q implementation, appears premature.","section":"Related Work"}],"recommendation":"major_revision","confidential_remarks":"The paper presents a promising framework, but the central active-sampling mechanism is underspecified to the point that the experiments cannot be interpreted. If the authors can provide a concrete, feasible algorithm for Eq. (9) and show that the reported gains are not an artifact of an ad-hoc approximation, the paper could become a solid contribution. The appendix experiment also needs a fairer comparison. I would be willing to review a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know this paper proposes something genuinely useful on paper: a utility-driven active query selection for pairwise recommendations, where the query is chosen to maximize the expected gain in a task-specific utility (top-item rank, admissions coverage) under a Plackett-Luce model. That goes beyond the usual entropy or uncertainty sampling, and the two experimental setups show what the authors are aiming at. The writing is clear, and the related work is honestly positioned; they correctly credit Chajewska et al. and Boutilier for value-of-information.\n\nThe main problem is that the engine of the method, Eq. (9), selects the query by computing θ|q, the parameters after hypothetical retraining on each possible outcome, for every candidate pair. The paper never says how θ|q is obtained — full retraining, a single gradient step, an approximation, or nothing. With 400 items you have ~80k pairs, so exact retraining is infeasible. Without an algorithm, the reported gains over random and entropy sampling cannot be attributed to the proposed utility-gain criterion; they may come from whatever shortcut was actually taken. This is not a minor omission; it is the load-bearing step.\n\nThere is also an internal mismatch: Algorithm 1 is written for the Admissions utility (Eq. 8), but Experiment 1 claims to use the max-rank utility (Eq. 7). So the provided Monte Carlo approximation does not even match the first experiment's utility. And the appendix's semi-synthetic comparison, which generates ground truth by sampling from a Plackett-Luce model, is stacked in favor of the comparison-based model; the main real-data experiments are not circular, but that appendix does not support the fundamental-advantage claim.\n\nOn the plus side, the basic pairwise loss and the Plackett-Luce formulas are correct, and the idea of optimizing an explicit utility rather than a rating objective is the right way to frame the problem. The experiments, if the implementation were specified, would be a useful data point. As is, the paper is an interesting research proposal, not a validated method.\n\nWho should read it: researchers working on preference elicitation and active learning for recommender systems. It deserves a serious referee, but the referee will need to demand a precise description of the active sampling implementation, error bars, and code. I would not cite it until those are provided.","headline":"Utility-driven active sampling for pairwise recommenders is a sensible and well-framed idea, but the paper never specifies how Eq. (9)'s hypothetical retraining is computed, so the experimental gains rest on an unverifiable implementation and deserve a referee's scrutiny, not a desk rejection.","tokens_in":13969,"tokens_out":2317,"would_cite":false,"duration_ms":26738,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Recommenders learn faster when they ask users the pairs whose answers most improve the final ranking.","keywords":["preference elicitation","pairwise comparisons","active learning","recommender systems","Plackett-Luce model","utility-based sampling","cold start","ranking"],"falsifier":"Inspect the implementation of Eq. (9): if $\\theta|q$ was produced by full retraining for every candidate pair and both outcomes, a single query in the 400-applicant experiment would require roughly $2\\times 79{,}800$ model fits; if it was produced by one gradient step or no update, the implemented query rule is not Eq. (9). Re-running the admissions experiment with a one-gradient-step replacement for $\\theta|q$ and re-measuring the active-versus-random gap would settle whether the reported gains survive the approximation.","tokens_in":13101,"feed_emoji":"🎯","tokens_out":12000,"duration_ms":118460,"temperature":0.7,"pith_summary":"This paper claims that recommender systems learn better from a few deliberately chosen pairwise comparisons than from ratings or random questions. The authors model each user's ranking distribution with the Plackett-Luce choice model, define what makes a recommendation good through a task-specific utility function, and select the next pair to ask by estimating which answer would most improve the expected utility of the final menu. In experiments, the active rule outperforms random, entropy-based, and clustering-based querying in both a movie recommendation task and an admissions task. If the claim holds, preference elicitation becomes substantially more data-efficient, which matters when each user answer is costly.","feed_headline":"Utility-targeted queries beat random sampling in recommender tests","feed_subtitle":"Choosing which pair of items to compare next lets a system learn user tastes with far fewer questions.","key_machinery":"The load-bearing object is the utility-gain query score $g(\\theta)=\\arg\\max_{(i,j)} E_{\\theta}[U(f(\\theta|q),\\theta|q)-U(f(\\theta),\\theta|q)]$; it converts 'informative question' into 'question that moves the final recommendation metric.' It sits on top of the Plackett-Luce model, in which each item's probability of being chosen next is proportional to its latent score, allowing Monte Carlo sampling of rankings instead of enumerating $n!$ orders. Two utility functions instantiate the objective: max rank over the menu for media, and fraction of the top-$k$ ranking included for admissions. The critical connective step is $\\theta|q$, the hypothetical retrained parameters after a possible answer","core_discovery":"The central claim is that query selection should be driven by expected improvement in the utility of the final recommendation rather than by generic uncertainty. For a candidate pair $(i,j)$, the method scores the query by $E_{\\theta}[U(f(\\theta|q),\\theta|q)-U(f(\\theta),\\theta|q)]$, where $\\theta$ is the current model, $q$ ranges over the two possible answers, $\\theta|q$ is the model after retraining on that answer, and $f(\\theta)$ is the recommendation menu that maximizes expected utility; the pair with the highest score is asked next. With Plackett-Luce as the ranking distribution, the expectation is approximated by Monte Carlo sampling of the top-$k$ positions. The paper reports higher ma","pith_inferences":["The reported experiments leave open how $\\theta|q$ is obtained; if exact retraining is required, evaluating one query on the 400-applicant dataset would cost about $2\\times 79{,}800$ model fits, making the method impractical at scale without an approximation.","If the implementation instead uses one gradient step or a similar shortcut, then Eq. (9) is not being evaluated as written, and the active-versus-random gap may be an artifact of that shortcut rather than of the utility criterion.","A natural extension is to restrict the candidate-pair pool to a shortlist tied to the current recommendation menu or to high-uncertainty items; if the utility-gain ordering is preserved, the same method could scale to large catalogs.","The same active query rule could be applied to diversity-, coverage-, or fairness-based utility functions, since the mechanics only require an expected-utility score for the menu."],"forward_implications":["Starting from five random comparisons per user, a handful of deliberately chosen additional pairs can improve the max-rank utility of the final menu, so pairwise elicitation can work in cold-start conditions.","Actively optimizing final utility outperforms entropy-based uncertainty sampling, so resolving uncertainty that does not change the recommended menu is worth less than resolving task-relevant uncertainty.","Because the query rule is written against an arbitrary scoring function $f(u,i;\\Theta)$, the same active sampler transfers from matrix factorization to neural-network models.","Swapping the utility function changes what the system optimizes, so the framework can be pointed at different notions of a good outcome, such as maximizing the top item or covering as many top-$k$ candidates as possible."],"supporting_citations":[{"why":"Defines the Plackett-Luce ranking distribution in Eq. (1), the foundation for pairwise preference probabilities and for sampling rankings in the active loop.","marker":"Plackett 1975; Luce 1959"},{"why":"Supplies the MovieLens 100k dataset used in Experiment 1 to simulate a cold-start media recommendation scenario.","marker":"Harper and Konstan 2015"},{"why":"Supplies the graduate-admissions dataset whose Chance of Admission score forms the ground-truth ranking in Experiment 2.","marker":"Acharya, Armaan, and Antony 2019"},{"why":"Supplies the NeuMF neural architecture used as the base model for the admissions task.","marker":"He et al. 2017"},{"why":"Motivates the latent random vector used to quantify model uncertainty for active querying in the neural-network experiment.","marker":"Dwaracherla et al. 2024"},{"why":"Provides the standard procedure for sampling rankings from the Plackett-Luce model, used to approximate expected utilities.","marker":"Xia 2019"},{"why":"Frames learning from partial pairwise preferences and motivates the sampling-based inference the active loop builds on.","marker":"Lu and Boutilier 2011"},{"why":"Supports using top-rank accuracy rather than rating error as the user-relevant evaluation criterion.","marker":"Cremonesi, Koren, and Turrin 2010"}],"fun_headline_variants":["Utility-based query selection learns tastes with fewer questions","Pairwise active sampling for utility-maximizing recommendations","Ask the pair that most improves recommendation utility","Utility-aware active learning for pairwise preference elicitation","Choosing comparison pairs by expected utility gain"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The active-learning formula presumes the system can predict its own model parameters after each possible answer to each candidate question, and the paper does not specify how those imagined retrained parameters are obtained; the reported gains depend on that unspecified step.","fun_headline_variants_meta":{"raw":{"variants":["Utility-based query selection learns tastes with fewer questions","Pairwise active sampling for utility-maximizing recommendations","Ask the pair that most improves recommendation utility","Utility-aware active learning for pairwise preference elicitation","Choosing comparison pairs by expected utility gain"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000541,"raw_usage":{"total_tokens":2443,"prompt_tokens":768,"completion_tokens":1675,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":512,"completion_tokens_details":{"reasoning_tokens":1616}},"tokens_in":512,"tokens_out":1675,"duration_ms":11877,"temperature":1.0,"reasoning_tokens":1616,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T21:09:41.023520+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Inspect the implementation of Eq. (9): if $\\theta|q$ was produced by full retraining for every candidate pair and both outcomes, a single query in the 400-applicant experiment would require roughly $2\\times 79{,}800$ model fits; if it was produced by one gradient step or no update, the implemented query rule is not Eq. (9). Re-running the admissions experiment with a one-gradient-step replacement for $\\theta|q$ and re-measuring the active-versus-random gap would settle whether the reported gains survive the approximation.","supporting_citations":[],"review_version":1}