{"id":"e93d8a48-2f0b-44df-83ea-9d80ad4f8002","arxiv_id":"2507.18365","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A membership-inference-based scoring method, RecPS, quantifies privacy risk of individual interactions and users in recommender models, enabling fine-grained data removal.","lead":"This paper introduces RecPS, a way to score how risky each user-item interaction is for privacy in a recommender system. The score helps users and companies decide which data to remove when someone asks to be forgotten.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Algorithm 2 selects the per-interaction threshold by maximizing TPR/FPR on the same shadow models, so the reported scores are in-sample optima rather than unbiased estimates; the utility-preservation experiments may reflect threshold overfitting rather than true privacy risk.","rationale":"The reader's verdict is CONDITIONAL and flags threshold optimization in the rationale, but the stated weakest assumption is the shadow-model/OUT-Gaussian faithfulness. I agree that the shared-Gaussian approximation is a real calibration risk; however, the more immediately decisive weakness is the in-sample threshold selection in Algorithm 2, which can produce inflated and non-comparable scores even when the Gaussian assumption holds exactly. This is the single most load-bearing concern because the paper's central empirical claim is that interaction-level scores enable finer-grained removal that preserves utility: if the scores are overfit maxima, the removal experiments in Figures 4-5 may not generalize to an independent target model or to a fixed-threshold scoring rule. The user-level derivation flaw noted by the reader is real but secondary, since the interaction-level removal is the distinctive contribution. This concern is concrete and fixable: replacing the max-over-threshold with a fixed or cross-validated threshold would either confirm the ranking and utility result or reveal it as an artifact. Therefore the appropriate verdict remains CONDITIONAL, with the condition that the threshold-selection bias be addressed and the removal experiments rerun under an unbiased scoring rule.","tokens_in":14762,"tokens_out":7267,"duration_ms":82769,"concrete_test":"Recompute all interaction-level scores under two alternative threshold rules: (a) a fixed global threshold Lambda = 0.5, and (b) a cross-validated per-interaction threshold, where t is selected on a random half of the shadow models and TPR/FPR is evaluated on the other half. Then re-run the Figure 4-5 removal protocol with each alternative score. Additionally, report the score distribution for a set of known OUT interactions (or for a permutation of membership labels). If the fixed/cross-validated scores rank different interactions as top-sensitive, or if known-OUT interactions receive scores substantially above 0, the reported privacy-utility tradeoff is an artifact of threshold overfitting.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that the estimated interaction-level score \\hat{\\epsilon}(u,i) = ln(TPR/FPR) approximates the per-record bound in Eq. (2). Algorithm 2 undermines this: for each interaction, the candidate thresholds T are exactly the OUT shadow models' own likelihood values, and the score is the maximum of ln(TPR/FPR) evaluated on those same shadow models. With m around 500, an interaction with no real membership signal still yields an empirical ROC path whose best threshold can give FPR near 1/m and a positive ln(TPR/FPR); taking the maximum over all OUT thresholds is a selection over many noisy operating points. This upward bias is not a constant offset: it depends on m and on the empirical distribution of the interaction's Lambda values, so scores are not comparable across interactions, users, or datasets. The bias propagates directly into the headline experiments: the top-scored interactions are chosen using the same overfit statistic, so Figures 4-5 may simply demonstrate that removing the samples the biased score ranks highest removes the shadow models' overfit signal. The shared-Gaussian OUT assumption flagged by the reader is a separate calibration issue, but even if N_out were exactly correct, Algorithm 2 would still overestimate scores.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RecPS, a privacy risk scoring framework for recommender systems. It defines an interaction-level privacy score as an empirical estimate of a per-record differential-privacy-style bound, computed as ln(TPR/FPR) of a likelihood-ratio membership inference attack adapted to recommendation models (RecLiRA). The user-level score is defined as the average of the interaction-level scores. The paper evaluates RecLiRA against the existing MINER attack on three benchmark datasets and two recommender models (NCF and LightGCN), and demonstrates that interaction-level score-guided removal of sensitive records preserves recommendation utility better than full user removal. The authors claim to be the first to study privacy scoring in recommender systems.","tokens_in":15042,"tokens_out":5020,"duration_ms":46746,"significance":"If the scores are well-calibrated, the interaction-level score is a practically useful tool for data contributors and model owners, and the fine-grained unlearning scenario is timely. The paper ships reproducible code and the RecLiRA attack appears to yield strong membership inference performance. However, the user-level score derivation is mathematically unjustified, and Algorithm 2's threshold selection on the same shadow models used for evaluation introduces an in-sample optimism that undermines the validity of the reported scores and the utility-preservation experiments. These issues are central to the paper's claims and need to be addressed before the contribution can be fully assessed.","major_comments":[{"comment":"The derivation of the user-level score is invalid. The equality Pr(M_0(D_{0,u}) ∈ O) = ∏_{i∈I_u} Pr(M_0(D_{0,(u,i)}) ∈ O) assumes independence across a user's interactions in the model's output distribution, which does not hold for recommender models whose predictions for user u share the same user embedding and are therefore jointly distributed. Consequently, the claimed bound by ∑_{i∈I_u} ε_{u,i} does not follow, and the average in Eq. (3) is not a principled privacy bound; it is an ad hoc heuristic. Please either replace this with a valid composition or group-privacy argument, or present the user-level score as an empirical aggregation and remove the theoretical claims.","section":"Section 4.2, Eq. (3)"},{"comment":"The per-interaction score is computed as the maximum of ln(TPR/FPR) over thresholds T that are exactly the Λ values of the OUT shadow models, evaluated on the same shadow models. With m ≈ 500, this is a maximization over roughly 250 noisy operating points; even for an interaction with no true membership signal, the empirical ROC path will typically yield some threshold with a positive ln(TPR/FPR), so the reported score is an in-sample optimum rather than an unbiased estimate of Eq. (2). The bias is not a constant offset: it depends on m and on the empirical distribution of Λ, so scores are not comparable across interactions, users, or datasets. This directly affects the headline removal experiments (Figures 4–5), which select \"top sensitive\" interactions using this same overfit statistic. Please re-estimate scores with a fixed threshold or with thresholds selected on a set of shadow models disjoint from the evaluation shadow models, and report the bias.","section":"Section 4.3, Algorithm 2"},{"comment":"The shared Gaussian OUT distribution N_out is estimated from roughly 30 OUT samples and is assumed to hold across all models and all interactions. The paper does not test this normality or the exchangeability assumption for recommender models; if the true OUT distribution is non-Gaussian or interaction-dependent, the Λ values and hence the TPR/FPR estimates are miscalibrated. Please add a goodness-of-fit analysis and a sensitivity study over k (e.g., k = 30, 100, 500) to show that the scores and the conclusions of Figure 5 are stable.","section":"Section 4.3, Algorithm 1"},{"comment":"The \"privacy protection\" criterion used in the interaction-removal experiment is that the re-evaluated score falls below a cut-off θ derived from the same score estimator. Since the score estimator is the same biased quantity from Algorithm 2, the experiment demonstrates only that removing the interactions that the overfit statistic ranks highest reduces that same overfit statistic; it does not demonstrate a reduction in actual membership risk. Please validate the removal experiment with an independent evaluation, e.g., an attack on a held-out target model not used to select thresholds, and report the resulting TPR/FPR.","section":"Section 5.2, Figures 4–5"}],"minor_comments":[{"comment":"Equation (2) uses ε(u,i) while Section 4.3 uses \\hat{ε}(u,i) inconsistently; the hat is introduced but the notation is not applied consistently throughout the experiments.","section":"Notation"},{"comment":"The definition q = |2p - 1| and the logit transform log(q/(1-q)) are undefined at q = 0 and q = 1 (i.e., p = 0, 0.5, or 1). Please specify how these edge cases are handled in practice.","section":"Section 4.3"},{"comment":"Algorithm 1 collects φ_out for all (u,i) ∈ D \\ S_j, but the text says \"a few OUT-training samples\" and later mentions k ≈ 30. Please clarify how the k OUT samples are selected and whether the loop over all OUT samples is consistent with the stated k.","section":"Algorithm 1"},{"comment":"The caption of Figure 6 contains a typo: \"Histagram\" should be \"Histogram.\"","section":"Figure 6 caption"},{"comment":"The statement that user-level scores are \"approximately divided into three bands\" is not supported by any quantitative analysis; please either provide a statistical description or remove the claim.","section":"Section 5.2"},{"comment":"The related work discussion for privacy scores (Gu et al. [7]) is too brief; please elaborate on the differences between FT-PrivacyScore and RecPS, particularly the interaction-level granularity and the DP-based derivation.","section":"Related work"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for the journal, but the threshold overfitting issue and the invalid user-level derivation are serious. I recommend major revision rather than reject because the interaction-level score is a useful contribution if the estimation bias is addressed and the user-level claims are appropriately reframed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short take: the paper has a good idea — give individual user-item interactions a privacy score based on the TPR/FPR of a LiRA-style attack, and use those scores to guide fine-grained unlearning. RecLiRA is a sensible adaptation of LiRA to NCF and LightGCN, and the experiments are broad. The problem is in how the scores are computed. Algorithm 2 picks the per-interaction threshold by maximizing ln(TPR/FPR) on the same shadow models used to estimate those rates. With ~500 shadow models, an interaction with no real membership signal can still produce a high score because the threshold is chosen from the OUT models' own likelihood values — you're selecting the best of many noisy operating points. That biases every score upward, and the bias is not constant, so scores are not comparable across interactions, users, or datasets. The utility-preservation experiments then rank interactions by this overfit statistic, which makes the 'score-guided removal beats random removal' result look like an artifact.\n\nThe user-level score is also on thin ice. The derivation multiplies probabilities of different interactions as if they were independent, which is not true. The paper then backs off to a simple average, which is honest but makes the user-level score a heuristic, not a derived bound.\n\nWhat's genuinely new: the interaction-level privacy score definition (Eq. 2) is a reasonable DP-motivated target, and RecLiRA appears to be the first interaction-level MIA with high power for NCF and LightGCN. The code is available. The onion-effect experiments are a nice extra.\n\nI'd like to see the threshold selection fixed — use a held-out set of shadow models, or a fixed threshold, or report the ROC curve without optimizing per interaction. Also add error bars and be more careful about the MINER comparison (the modified baseline may not be faithful).\n\nOn balance, the core idea is defensible and worth engaging. It's a solid RecSys paper, not a breakthrough. Send it to reviewers, but expect them to ask for major revisions.","headline":"Useful privacy-scoring framework for RecSys with a strong interaction-level MIA, but the score estimation overfits shadow models and the user-level score is a heuristic.","tokens_in":15547,"tokens_out":4286,"would_cite":true,"duration_ms":40659,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes RecPS, a framework that assigns each user-item interaction in a recommender system a quantitative privacy risk, derived from differential privacy and estimated with a membership inference attack.","keywords":["privacy scoring","membership inference attack","recommender systems","LiRA","machine unlearning","differential privacy","privacy-utility tradeoff","RecLiRA"],"falsifier":"Train many target models on the same dataset with a known interaction $(u,i)$ included or excluded, and compare the true likelihood ratio of model outputs to RecLiRA's estimated $\\ln(\\mathrm{TPR}/\\mathrm{FPR})$. If the estimate systematically misses the measured ratio, or if the empirical OUT distribution of $\\phi(q)$ deviates materially from the Gaussian, the scoring method's calibration claim is falsified.","tokens_in":14582,"feed_emoji":"🔐","tokens_out":6223,"duration_ms":58644,"temperature":0.7,"pith_summary":"Recommender systems train on highly sensitive user-item interactions, but today a user has no quantitative way to know which interactions carry the most privacy risk. This paper proposes RecPS, a framework that assigns each interaction a privacy score derived from differential privacy: the score is $\\ln(\\mathrm{TPR}/\\mathrm{FPR})$ of a membership inference attack, measuring how easily an attacker can tell whether that interaction was in the training set. A user-level score is then the average of the interaction scores. The paper argues that these scores let users and model owners remove only the most sensitive interactions, preserving far more recommendation utility than removing entire users, and demonstrates the idea with a new interaction-level attack, RecLiRA, on NCF and LightGCN models over three benchmark datasets.","feed_headline":"Likelihood ratio gives every click its own privacy score","feed_subtitle":"A new attack-based score lets users remove only sensitive interactions, keeping recommender models useful.","key_machinery":"The load-bearing mechanism pairs a differential-privacy-derived score definition with a LiRA-style membership inference attack adapted to recommender models. RecLiRA treats a recommender's probability output $p$ for an interaction $(u,i)$ as a binary classifier confidence, forms $q=|2p-1|$, applies the logit transform $\\phi(q)=\\log(q/(1-q))$, and models the OUT-record distribution of $\\phi(q)$ as a shared Gaussian estimated from shadow models trained on random 50% subsets of the data. For each interaction it computes per-shadow-model likelihoods, probes candidate thresholds, and selects the threshold that maximizes $\\ln(\\mathrm{TPR}/\\mathrm{FPR})$, directly estimating the score. This machinery converts an abstract privacy bound into a computable per-record number that can guide data removal.","core_discovery":"The paper's central claim is that the privacy risk of a single user-item interaction in a recommender model can be quantified by the sample-specific differential-privacy bound, estimated as $\\ln(\\mathrm{TPR}/\\mathrm{FPR})$ of a membership inference attack. Equation (2) defines the interaction-level score as the supremum of a log-likelihood ratio between model outputs on datasets that differ by that interaction; a powerful attack's $\\ln(\\mathrm{TPR}/\\mathrm{FPR})$ is argued to be an empirical lower bound for this quantity. User-level risk is defined as the average of the interaction scores, avoiding overestimation for heavy users. The paper establishes that scoring is practical: the custom RecLiRA attack achieves high quality in the low-FPR region, and selectively removing top-scored interactions demotes targeted users' scores below a cutoff while causing smaller hit-rate drops than removing those users entirely.","pith_inferences":["Because the estimated score is $\\ln(\\mathrm{TPR}/\\mathrm{FPR})$, it is a lower bound on the true differential-privacy risk; any future attack that raises TPR at fixed FPR will tighten the score, so the framework improves automatically as membership inference improves.","The same score could be used proactively, before training, to flag data that should be excluded or down-weighted, not just after training for removal; nothing in the method prevents computing shadow-model scores before a production model is trained.","The RecPS framework should extend to any recommender that emits a probability-like interaction score; testing it on sequential or LLM-based recommenders, and on score-only models like matrix factorization, would show whether the Gaussian logit assumption survives outside NCF/LightGCN.","The average-based user-level score hides the worst-interaction risk; a max or quantile aggregation might better match regulatory notions of the right to erasure for a single harmful record."],"forward_implications":["Interaction-level scores enable selective unlearning: removing the top 70% of sensitive interactions for the top 5% of users on Amazon Digital Music demotes 100% of those users below the privacy cutoff while losing only 15.56% NCF utility, versus 37.05% when removing entire users.","User-level scores, computed as averages of interaction scores, differ across models (LightGCN scores run higher than NCF), suggesting the score tracks how strongly the model memorizes interactions.","Privacy scores are dynamic: removing sensitive users changes remaining users' scores by a small amount, and interaction-level removal weakens this privacy onion effect compared with user-level removal.","RecLiRA's high true-positive rates at low false-positive rates make the TPR/FPR ratio a usable score estimator on NCF and LightGCN, with AUC above 0.9 on all tested datasets."],"supporting_citations":[{"why":"Supplies the LiRA membership-inference method and the key observation that logit-transformed output scores are approximately Gaussian, which RecLiRA adapts.","marker":"[2]"},{"why":"Defines the NCF recommender model used as a target and shadow architecture in experiments.","marker":"[12]"},{"why":"Defines the LightGCN recommender model used as the other target and shadow architecture.","marker":"[11]"},{"why":"The interaction-level MIA baseline (MINER) that RecLiRA must beat in the low-FPR regime.","marker":"[36]"},{"why":"Provides the hypothesis-testing interpretation of differential privacy that grounds the TPR/FPR score definition.","marker":"[15]"},{"why":"Shows the prior use of MIA true-positive rate as a privacy score, which the paper argues is less principled than TPR/FPR.","marker":"[26]"},{"why":"User-level recommendation-unlearning approach that RecPS's selective interaction removal improves upon.","marker":"[4]"},{"why":"Prior personalized privacy scoring for LLM fine-tuning that RecPS extends to recommender systems.","marker":"[7]"}],"fun_headline_variants":["Privacy score per click: attack-based risk for recommender data","Each user click gets a privacy risk score from MIA","Likelihood ratio quantifies privacy risk of each interaction","Score your clicks: privacy risk via membership inference","Privacy-aware unlearning: target sensitive interactions, not users"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The shadow models trained on random half-subsets, together with the assumption that non-member outputs share one Gaussian distribution, must faithfully represent how the real target model behaves for member and non-member records; if the target model's training distribution or architecture differs, the estimated $\\mathrm{TPR}/\\mathrm{FPR}$ and every privacy score built from it are biased.","fun_headline_variants_meta":{"raw":{"variants":["Privacy score per click: attack-based risk for recommender data","Each user click gets a privacy risk score from MIA","Likelihood ratio quantifies privacy risk of each interaction","Score your clicks: privacy risk via membership inference","Privacy-aware unlearning: target sensitive interactions, not users"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000243,"raw_usage":{"total_tokens":1524,"prompt_tokens":934,"completion_tokens":590,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":550,"completion_tokens_details":{"reasoning_tokens":511}},"tokens_in":550,"tokens_out":590,"duration_ms":5774,"temperature":1.0,"reasoning_tokens":511,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:13:17.796670+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train many target models on the same dataset with a known interaction $(u,i)$ included or excluded, and compare the true likelihood ratio of model outputs to RecLiRA's estimated $\\ln(\\mathrm{TPR}/\\mathrm{FPR})$. If the estimate systematically misses the measured ratio, or if the empirical OUT distribution of $\\phi(q)$ deviates materially from the Gaussian, the scoring method's calibration claim is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The interaction-level MIA baseline (MINER) that RecLiRA must beat in the low-FPR regime."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the hypothesis-testing interpretation of differential privacy that grounds the TPR/FPR score definition."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows the prior use of MIA true-positive rate as a privacy score, which the paper argues is less principled than TPR/FPR."}],"review_version":2}