{"id":"36376a3e-24c8-49a8-bdcd-c4013f48e6c3","arxiv_id":"2507.16829","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A model-agnostic post-hoc pipeline uses conformal risk control to guarantee that the expected fraction of unwanted items in a recommendation list stays below a user-chosen level.","lead":"This paper proposes a post-hoc method for recommender systems that provably bounds the share of unwanted content shown to users, using conformal risk control with simple yes/no feedback. It shows on a large real-world video platform that replacing risky recommendations with previously consumed, safe videos keeps the guarantee while preserving recommendation quality.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Proposition 3's monotonicity claim is false: removing items can raise the flagged fraction, so Theorem 1 does not cover Algorithm 1 as written; the guarantee needs explicit monotonization.","rationale":"The reader's verdict is CONDITIONAL, and my analysis supports that condition rather than overturning it. The reader's weakest_assumption was Property 1, but I see a more fundamental problem: the risk function defined in Eq. (5) is not non-increasing in λ merely because the candidate pool shrinks. The fraction of flagged items can increase when high-score flagged items remain after lower-score unflagged items are filtered out. This breaks the central hypothesis of Theorem 1 for the raw risk used in Algorithm 1 and Eq. (6). The paper's Proposition 3 is therefore false as stated, even when Property 1 holds perfectly. I do not claim the whole method is unsalvageable: conformal risk control offers a standard monotonization procedure, and the paper hints at it in a footnote. If RiskControl uses the monotone envelope, the guarantee can be recovered, but then the theorem, algorithm, and experiments must be described in those terms. Since this is a serious but fixable proof gap, the conditional verdict remains appropriate. The proposed test checks the actual calibration curve and implementation to determine whether the gap is only presentational or affects the implemented guarantee.","tokens_in":17509,"tokens_out":8217,"duration_ms":105405,"concrete_test":"Run the released code on the KuaiRand calibration split and compute the empirical risk R̂(λ) = (1/n) Σ_j RH(S_λ(u_j,k)) for a fine grid of λ, exactly as used by Algorithm 1 (Replace, β=0), before any monotonization. If there exists λ1<λ2 with R̂(λ1)<R̂(λ2), the monotonicity assumption of Theorem 1 fails and Eq. (6) applied to the raw curve carries no guarantee. Then inspect RiskControl: if it actually uses the monotone envelope R̄(λ)=max_{λ'≥λ} R̂(λ'), the guarantee can be restored, but the paper must state this explicitly. Independently, the counterexample above already disproves Proposition 3 as written.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing gap is not Property 1 but the monotonicity assumption of Theorem 1. Proposition 3 (Sec. 5.1) claims RH(T_replace_λ'(U)) ≤ RH(T_replace_λ(U)) for λ<λ' because replacements have H=0. This is false for the fraction in Eq. (5). Counterexample: let T_safe={S} with H(S)=0, and let T_λ contain a flagged item F (score 10) plus an unflagged item A (score 9). At λ=5, T_replace={F,A,S}, risk=1/3; at λ=9.5, T_replace={F,S}, risk=1/2, violating monotonicity. The assertion in Sec. 4.2 that removing items 'cannot increase RH' is incorrect for proportions because numerator and denominator both shrink. The same problem applies to the returned top-k set: raising λ can remove low-score unflagged items and leave high-score flagged items, increasing risk. The footnote about monotonizing RH is the only rescue, but Algorithm 1 and Eq. (6) are stated for the raw risk; no monotonized risk is defined, computed, or used in the experiments. Hence the advertised guarantee for the final recommendation list is not established by the proof as written.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a post-hoc, model-agnostic method, Algorithm 1, that uses conformal risk control to bound the expected fraction of user-flagged ('unwanted') items in a top-k recommendation list. The method thresholds items by a score and then expands the candidate pool with previously consumed, non-flagged items whose watch time exceeds a threshold β, so as to preserve list size. The authors claim a distribution-free, finite-sample guarantee on the fraction of flagged items (Theorem 1 applied after Proposition 3), and they evaluate the approach on the KuaiRand dataset with four ranking models, comparing a 'Replace' strategy against a 'Remove' strategy and ablating the β threshold and user reporting habits.","tokens_in":1704,"tokens_out":2071,"duration_ms":80650,"significance":"The problem is timely and practically important: giving users a provable handle on unwanted content in recommendations is a valuable goal, and a simple, model-agnostic post-processing layer would be a useful contribution. The empirical study is careful, includes multiple ranking models, ablations, and a user-group analysis, and the authors release source code, which strengthens reproducibility. If the theoretical guarantee were correctly established, the method would be a strong practical tool. However, the central guarantee is currently not proven: the monotonicity assumption required by conformal risk control is violated by the proposed replacement set, and the paper's own ablation (Fig. 5a, 'None') shows the guarantee can fail. The contribution is therefore conditional on a substantive revision of the theoretical argument.","major_comments":[{"comment":"Proposition 3 is false as stated: the risk fraction in Eq. (5) is not non-increasing in λ for the set T_replace_λ defined in Eq. (8). Counterexample: let T_safe = {S} with H(S)=0, and let T_λ contain a flagged item F (score 10) and an unflagged item A (score 9). For λ=5, T_replace_λ = {F,A,S}, so R_H = 1/3; for λ=9.5, T_replace_λ = {F,S}, so R_H = 1/2. The risk increases when an unflagged low-score item is removed, because the numerator and denominator both shrink. Consequently, the assumption in Theorem 1 that R_H(S_λ(U,k)) is non-increasing in λ is not satisfied, and the expectation bound E[R_H(S_λ̂(U,k))] ≤ α does not follow for Algorithm 1 as written. The footnote in Sec. 4.2 about monotonizing R_H is not implemented: Algorithm 1 and Eq. (6) use the raw risk, and no monotone upper envelope is defined, computed, or used in the experiments. Therefore the Section 6 statement that 'the algorithm provably controls the fraction of unwanted content in the final recommendation list (cf. Eq. (5))' is not supported by the proof given.","section":"Section 5.1, Proposition 3 and Section 6"},{"comment":"The proof of Proposition 2 is incomplete because it does not specify which items are flagged. If the flagged item is D (score 5), then for k=1 the top-1 set has risk 1; but if the flagged item is A (score 1), then for k=1 the set {D} has risk 0, and the claim that only λ > 5 (empty set) achieves risk ≤ 0.1 is false. The proof needs to state the flagged status of each item and show that the impossibility holds for a fixed flagged item regardless of k, or else the proposition as stated is not established.","section":"Section 5, Proposition 2"},{"comment":"The ablation with 'None' (no watch-time filtering) shows that the empirical reduction in unwanted content falls below the target, meaning risk control fails. The paper attributes this to distribution shift, but this is also exactly the regime where Property 1 is violated and where the monotonicity needed for Theorem 1 is not guaranteed. The paper should explicitly state whether the theoretical guarantee is conditional on Property 1 holding exactly. If so, the experimental results for β>0 demonstrate behavior under an approximate satisfaction of the property, and the conditions under which the guarantee holds should be stated precisely, including what happens when Property 1 is violated.","section":"Section 7, RQ4 and Fig. 5a"}],"minor_comments":[{"comment":"The sentence 'by removing items, we cannot increase R_H(S_λ(U,k))' is incorrect for the fraction in Eq. (5); removing an unflagged item can increase the fraction. Please rephrase or qualify this statement.","section":"Section 4.2"},{"comment":"The item set is denoted I in the text and Y in lines 2-3 of Algorithm 1; please unify the notation.","section":"Algorithm 1"},{"comment":"Line 3 of Algorithm 1 uses the condition W%(U,i') > β, while Eq. (7) defines the safe pool via C(I=i') > β; the correspondence between C and W% should be made explicit.","section":"Section 5.1 and Algorithm 1"},{"comment":"The caption says 'nDGC @ 20'; this should be 'nDCG @ 20'.","section":"Figure 3 caption"},{"comment":"The report/no-report symbols in Table 2 may not render correctly in all formats; consider spelling out the four behavior combinations in text.","section":"Section 3.3, Table 2"},{"comment":"The phrase 'the set of items I ∈ {i1,...,iN}' should be 'the set of items I = {i1,...,iN}'.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript states it is accepted at RecSys 2025; this report is based on the arXiv version. The monotonicity gap in Proposition 3 is load-bearing: without a correct monotone risk or an explicit monotonization step, the advertised provable guarantee for Algorithm 1 is not established. The empirical work is solid and the fix is likely feasible (e.g., use the monotone upper envelope of R_H in Eq. (6) and Algorithm 1), so I see this as a major revision rather than a rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper does two things worth knowing. First, it moves conformal risk control to a new target—the fraction of flagged/unwanted items in a top-k list—where prior conformal recsys work aimed at nDCG or FDR. Second, it replaces filtered items with previously consumed, non-flagged items, and shows on KuaiRand that this preserves utility much better than pure removal. The data analysis (sparsity of negative feedback, repeat-viewing behavior) is honest and the code is public. Those are real contributions.\n\nThe soft spots are in the theory, and one is load-bearing. Proposition 3 claims that the replacement set T_replace_λ is non-increasing in risk as λ grows because replacements have H=0. That is false for the fraction in Eq. (5): removing an unflagged candidate from Tλ shrinks both numerator and denominator, and the proportion can rise. The stress-test counterexample (flagged high-score item + unflagged lower-score item + one safe replacement) is exactly right. The footnote in Sec. 4.2 mentions monotonizing the risk, but Algorithm 1 and Eq. (6) operate on the raw risk; no monotonized quantity is defined, computed, or tested. As written, Theorem 1 does not cover the algorithm's final recommendation list. This is fixable—define the monotonized risk and use it in calibration and evaluation—but it is not a stylistic gap.\n\nTwo smaller issues. Proposition 2's proof never says which items are flagged; as stated the impossibility argument is incomplete. And the β threshold is chosen from the same data used for the evaluation; Fig. 5a shows that without it (the None ablation) the risk guarantee fails, so the empirical success leans on that data-dependent choice. Property 1 is only approximately satisfied, which the authors acknowledge.\n\nWho gets value: practitioners who want a post-hoc, model-agnostic knob for unwanted content and want to keep list size up. The experiments are informative. The theoretical claim needs repair before I'd rely on it. I'd send it to a serious referee—the contribution is worth referee time—but the referee should push for the monotonized guarantee to be made explicit and actually used.","headline":"Useful post-hoc method for bounding unwanted content, but the proof of monotonicity for the replacement rule is wrong as written; the advertised guarantee requires the monotonized risk the paper only mentions in a footnote.","tokens_in":18280,"tokens_out":3448,"would_cite":true,"duration_ms":36583,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a post-hoc threshold-and-replace filter can provably keep the expected fraction of flagged, unwanted items in a recommendation list below a user-chosen level, without retraining the recommender.","keywords":["conformal risk control","recommender systems","unwanted content","binary negative feedback","repeat consumption","distribution-free guarantees","post-hoc filtering","KuaiRand dataset"],"falsifier":"Run Algorithm 1 with the Replace strategy on held-out data containing second exposures, and count how often a previously seen, never-reported item with first-exposure watch time above β is later flagged; if the rate is nonzero and the empirical risk R_H(S_λ(U,k)) exceeds the promised α for the chosen β, the guarantee is falsified. The paper's own no-filtering ablation in Fig. 5a already shows the risk overshoot when β filtering is removed.","tokens_in":17290,"feed_emoji":"🛡️","tokens_out":7913,"duration_ms":84038,"temperature":0.7,"pith_summary":"This paper sets out to show that a recommender system can be made safer after training, with no retraining, by post-processing the candidate list with a user-chosen risk level α. The method uses conformal risk control — a distribution-free technique that selects a threshold so that expected loss stays below a bound — to keep the expected fraction of flagged, unwanted items in the top-k list below α, using only binary negative feedback such as a \"not interested\" click. Because pure thresholding can shrink the list below k items, the paper adds replacements: previously seen items the user did not flag and watched past a watch-time threshold β, which are treated as safe to resurface. Experiments on video-sharing platform data show the procedure keeps empirical risk at or below the target across several ranking models, while preserving more ranking quality than simply removing items. If correct, the method gives users a simple, provable control over unwanted content exposure in any personalized recommender that can supply historical feedback.","feed_headline":"A post-hoc filter provably caps unwanted recommendations","feed_subtitle":"Users set a risk level; the method guarantees flagged content stays below it, without retraining.","key_machinery":"The load-bearing object is the risk function of Equation (5), the fraction of flagged items in a recommendation set, paired with the conformal risk control threshold selection of Theorem 1: choose the smallest threshold λ̂ such that (n/(n+1))R̂(λ) + 1/(n+1) ≤ α on a calibration set of size n, and the expected risk at λ̂ is provably at most α whenever the risk is non-increasing and right-continuous in λ. The replacement mechanism is the safe-item set T_safe = {previously seen, unflagged items with watch-time proportion C > β}, which is unioned with the thresholded pool so the recommender can still return up to k items. The safety of this union rests on Property 1, which asserts zero probability of a second report for items with H1st = 0 and C > β; the paper checks this approximately on Kuaishou watch-time data and explores the trade-off through the β ablation.","core_discovery":"The central claim is that the fraction of unwanted content in a personalized top-k recommendation list can be provably bounded in expectation by a post-hoc threshold-and-replace procedure. For any user-selected α, Algorithm 1 computes a threshold λ̂ from a held-out calibration set using the conformal risk control result of Theorem 1, so that the expected value of the risk R_H(S_λ(U,k)) — the fraction of flagged items in the returned set — is at most α, provided the risk is non-increasing in λ. To avoid returning fewer than k items, candidates below the threshold are replaced by items from the user's own history that were not flagged and whose first-exposure watch time exceeds a threshold β; Property 1 asserts that such items have zero probability of being flagged on a second exposure, which keeps the risk monotone and lets the guarantee survive replacement. The paper tests this by wrapping several pretrained rankers with the postprocessor on KuaiRand data and reports that the empirical reduction in unwanted content meets or exceeds the target, with nDCG and recall degrading more gracefully than under plain removal.","pith_inferences":["As an extension beyond the paper, per-user or per-group calibration with the same Theorem 1 machinery would likely reduce the conservativeness the authors observe for low-reporting users.","A further extension would replace the hard watch-time threshold β with a per-user calibrated quantile or a learned second-exposure report probability, since the data only support Property 1 approximately.","The reliance on repeated items suggests a testable boundary: on platforms where repeat exposure is rare, the method collapses to pure removal, so the promised full-list guarantee would not hold without a larger safe-item pool."],"forward_implications":["Any deployed recommender can expose a user to at most an expected α fraction of flagged items in the top-k set without retraining, as long as historical binary feedback and calibration data are available.","With a sufficient pool of safe repeated items, the user receives a full top-k list; when the pool is empty, the method degrades to pure removal and may return fewer than k items.","The number of items that must be replaced depends on the score function used for thresholding; on KuaiRand, sign-aware rankers replace more items than unsigned ones at equal risk.","Stricter safety filtering (larger β) restores the guarantee under distribution shift but shrinks the replacement pool, moving behavior closer to plain removal; no filtering can break the guarantee."],"supporting_citations":[{"why":"Supplies Theorem 1, the conformal risk control guarantee used to choose the filtering threshold λ̂.","marker":"[6]"},{"why":"Provides the thresholding framework and the earlier application of distribution-free reliability guarantees to recommenders that the paper extends with replacement.","marker":"[7]"},{"why":"Supplies the KuaiRand dataset of Kuaishou interactions with explicit negative feedback, used for analysis, calibration, and evaluation.","marker":"[30]"},{"why":"Documents repeat consumption dynamics, the behavioral premise for using previously seen items as replacements.","marker":"[3]"},{"why":"Repeat-aware recommendation method supporting the idea that repeated items can be exploited as safe alternatives.","marker":"[48]"},{"why":"LightGCL, one of the pretrained rankers used as a score function in the two-stage evaluation.","marker":"[15]"},{"why":"Sigformer, a sign-aware transformer used as a comparison ranker in the experiments.","marker":"[19]"},{"why":"Neural Collaborative Filtering model used as the watch-time re-ranker in the two-stage setup.","marker":"[35]"}],"fun_headline_variants":["Provable cap on unwanted recommendations via post-hoc filter","Guarantee: unwanted recs stay under user-set risk bound","Post-hoc conformal filter bounds unwanted recs provably","Set a risk level; keep unwanted recs below it—provably","User-chosen risk cap on bad recs, no retraining"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire argument hinges on the assumption that an item a user has already seen, never reported, and watched past a watch-time threshold will not be reported if it is shown again.","fun_headline_variants_meta":{"raw":{"variants":["Provable cap on unwanted recommendations via post-hoc filter","Guarantee: unwanted recs stay under user-set risk bound","Post-hoc conformal filter bounds unwanted recs provably","Set a risk level; keep unwanted recs below it—provably","User-chosen risk cap on bad recs, no retraining"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000297,"raw_usage":{"total_tokens":1731,"prompt_tokens":964,"completion_tokens":767,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":580,"completion_tokens_details":{"reasoning_tokens":680}},"tokens_in":580,"tokens_out":767,"duration_ms":8012,"temperature":1.0,"reasoning_tokens":680,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:44:20.778489+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Algorithm 1 with the Replace strategy on held-out data containing second exposures, and count how often a previously seen, never-reported item with first-exposure watch time above β is later flagged; if the rate is nonzero and the empirical risk R_H(S_λ(U,k)) exceeds the promised α for the chosen β, the guarantee is falsified. The paper's own no-filtering ablation in Fig. 5a already shows the risk overshoot when β filtering is removed.","supporting_citations":[{"cited_title":"Conformal risk control","cited_arxiv_id":null,"evidence_quote":"Supplies Theorem 1, the conformal risk control guarantee used to choose the filtering threshold λ̂."},{"cited_title":"Recommendation systems with distribution-free reliability guarantees","cited_arxiv_id":null,"evidence_quote":"Provides the thresholding framework and the earlier application of distribution-free reliability guarantees to recommenders that the paper extends with replacement."},{"cited_title":"The dynamics of repeat consumption","cited_arxiv_id":null,"evidence_quote":"Documents repeat consumption dynamics, the behavioral premise for using previously seen items as replacements."},{"cited_title":"LightGCL: Simple yet effective graph contrastive learning for recommendation","cited_arxiv_id":null,"evidence_quote":"LightGCL, one of the pretrained rankers used as a score function in the two-stage evaluation."},{"cited_title":"Neural collaborative filtering","cited_arxiv_id":null,"evidence_quote":"Neural Collaborative Filtering model used as the watch-time re-ranker in the two-stage setup."}],"review_version":1}