{"id":"8df61435-337a-4663-8889-aa912a2d6d96","arxiv_id":"2505.22057","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"SVV prunes recommender training interactions by their estimated Shapley value contribution to autoencoder loss reduction, reporting modest accuracy gains on four datasets but resting on a faulty value-function derivation.","lead":"Researchers propose SVV, which assigns each user-item interaction a Shapley value measuring how much it helps a denoising autoencoder reduce its training loss, then prunes the lowest-value interactions and retrains. On four real-world recommender datasets, SVV improves Recall and NDCG by roughly 1 to 7 percent over the base model, but the paper's proof that its score measures training loss reduction contains an algebraic error and its experimental tables are inconsistent.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 17 is not equivalent to Eq. 16, so SVV's Shapley values do not quantify training-loss contribution; the claimed mechanism is unsupported.","rationale":"The paper's contribution is a principled, model-driven valuation of interactions, and the algebraic equivalence between Eq. 16 and Eq. 17 is the step that ties the computed Shapley values to training-loss reduction. That step is incorrect: the two value functions differ by an S-dependent term, so the Shapley values are values of different games. This is not a disagreement with current consensus; it is an internal algebraic error. The paper does provide publicly available code and a simulated-noise protocol, which are creditworthy, and the experiments are extensive. However, the central theoretical claim is unsupported, and the reported accuracy gains could arise from pruning low predicted-score interactions rather than from Shapley-based loss attribution. The reader's weakest-assumption identification is correct, and the additional table inconsistency further reduces confidence in the empirical reporting, but the algebra error alone is sufficient to reject the paper's central claim as stated.","tokens_in":20871,"tokens_out":3294,"duration_ms":36517,"concrete_test":"Using the released code, take a single user (or a small user subset) from the CDs dataset, enumerate all 2^m coalitions of that user's observed interactions with the trained DAE, and compute exact Shapley values under both Eq. 16 and Eq. 17. Compare the resulting rankings via Spearman rank correlation and the Jaccard overlap of the bottom-20% interactions; if the rankings differ, the equivalence asserted in Sec. 4.2 is false.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central mechanism rests on the asserted equivalence between Eq. 16 and Eq. 17 in Sec. 4.2. Eq. 16 defines the value function as the negative normalized squared reconstruction error over observed entries, and the paper claims that for binary implicit feedback this simplifies exactly to the average predicted score in Eq. 17. Expanding Eq. 16 for observed entries gives v1(S) = (1/m)[2Σf_v - Σf_v^2] - 1, whereas Eq. 17 is v2(S) = (1/m)Σf_v, with m = 1^T r. The difference is v2 - v1 = 1 + (1/m)Σ(f_v^2 - f_v). An additive constant would not affect Shapley values, but this difference depends on S because f(r_S) changes with the coalition S. Therefore the two value functions define different cooperative games, and the Shapley values computed from Eq. 17 are not Shapley values of the training-loss game in Eq. 16. The paper's central interpretation—that low-SVV interactions contribute least to training-loss reduction—does not follow. The empirical pruning gains may still occur, but they are not evidence for the claimed mechanism.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Shapley Value-driven Valuation (SVV), a data-pruning framework for implicit-feedback recommender systems. SVV trains a denoising autoencoder (DAE), estimates Shapley values of observed user–item interactions with FastSHAP using a value function meant to measure contribution to training-loss reduction, prunes the lowest-valued interactions, and retrains the DAE on the pruned set. The authors also introduce a simulated noise injection protocol and report accuracy and robustness gains over several baselines on four datasets. The central theoretical claim is that, for binary feedback, the negative normalized squared reconstruction error in Eq. (16) is equivalent to the average predicted score in Eq. (17), so that Shapley values computed from the latter quantify each interaction's contribution to reducing training loss.","tokens_in":21012,"tokens_out":4726,"duration_ms":59315,"significance":"If the central derivation were correct, the paper would make a useful contribution: it proposes a principled, model-driven alternative to intent-based denoising, makes code available, and includes a controlled simulated-noise benchmark. The interpretability analysis and the observation that low-rated or 'noisy' interactions can aid training are also valuable. However, the load-bearing equivalence between Eqs. (16) and (17) is algebraically false, so the computed Shapley values do not measure training-loss contribution as claimed. The empirical gains may still be real, but they do not validate the paper's stated mechanism. The paper is therefore not acceptable in its current form.","major_comments":[{"comment":"The claimed equivalence between the negative normalized squared reconstruction error and the average predicted score is false for binary implicit feedback. Expanding Eq. (16) over observed entries with binary r gives v1(S) = (1/m)(2Σ f_v − Σ f_v^2) − 1, where m = 1^T r and the sums run over observed entries, whereas Eq. (17) is v2(S) = (1/m)Σ f_v. The difference is v2(S) − v1(S) = 1 + (1/m)Σ(f_v^2 − f_v). Since f(r_S) depends on the coalition S, this difference is not an additive constant; it changes the marginal contributions of individual interactions. Consequently, the two value functions define different cooperative games and yield different Shapley values. The paper's central interpretation—that low-SVV interactions contribute least to training-loss reduction—does not follow from the method as implemented.","section":"Sec. 4.2, Eqs. (16)–(17)"},{"comment":"The robustness claim in the abstract and conclusion is not supported by the results for popular-item noise. In Table 4, SVV is worse than the base model on multiple metrics under Popular injected noise, for example CDs R@20 (−3.90%), CDs R@10 (−1.42%), Games R@5 (−2.70%), and Movies R@20 (+0.11%, effectively no gain). The text acknowledges deterioration for popular items but still summarizes the method as improving 'both accuracy and robustness.' This overstates the empirical evidence and should be revised or the robustness claim qualified to the tested noise types.","section":"Sec. 5.5, Table 4"},{"comment":"There is a mismatch between the noise injection rate and the pruning ratio that weakens the interpretation of the noise-detection results. The injection protocol adds k% of each user's original interaction count as noise, so the corrupted set has |D+| + k%|D+| interactions; pruning the bottom k% of the corrupted set removes k% of the corrupted data, not k% of the original data, and the injected noise constitutes k/(100+k)% of the corrupted set. The paper sets both k=20 without explicitly addressing this distinction, so the claim that pruning exactly removes the injected noise is not internally consistent. This affects the interpretation of the 'overlap' percentages in Table 3 and the validation protocol in Sec. 5.3.","section":"Sec. 4.3, Eq. (20)"}],"minor_comments":[{"comment":"The clause 'where minimizing squared error loss closely aligns with maximizing predicted scores' should be replaced by the exact algebraic relationship, since as written it masks the incorrect equivalence in Eqs. (16)–(17).","section":"Sec. 4.2, sentence after Eq. (17)"},{"comment":"The DAE corruption (masking) ratio is not specified; the text only says the binary input vector is randomly masked using PyTorch's randint function. This is a key hyperparameter for the base model and should be reported.","section":"Sec. 5.1.4"},{"comment":"The notation 'Top↓' and 'Bottom↑' is not explained. It appears that lower Top overlap and higher Bottom overlap are desirable for detecting injected noise, but the arrow convention should be defined in the caption or text.","section":"Sec. 5.3, Table 3"},{"comment":"Figure 1 contains the typo 'Gamses' for 'Games,' and Figure 2 does not label the axes or indicate error bars, which makes the monotonicity claims in Sec. 5.4 difficult to verify.","section":"Figure 1 and Figure 2"},{"comment":"The paper claims statistical significance via a paired t-test at the 0.05 level but reports only averaged metrics over five runs; no variances, standard errors, or p-values are provided, so the significance claim cannot be checked.","section":"Sec. 5.2, Table 2"}],"recommendation":"reject","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Let me save you time. SVV uses FastSHAP to assign each observed interaction a Shapley value relative to the DAE's training loss, then prunes the bottom k% and retrains. The code is released and the simulated noise injection protocol is a genuinely useful benchmarking contribution. The experimental scope is broad: four datasets, multiple baselines, and several noise types.\n\nThe load-bearing derivation fails. In Sec. 4.2, Eq. 16 defines the value function as the negative normalized squared reconstruction error over observed entries. The paper says that for binary implicit feedback this simplifies to Eq. 17, the average predicted score. It does not. Expanding Eq. 16 over observed entries gives (1/m)∑(2f_v − f_v^2) − 1, not (1/m)∑f_v. The difference, 1 + (1/m)∑(f_v^2 − f_v), depends on the coalition S because f(r_S) changes with S. So the two value functions define different cooperative games, and the FastSHAP values computed from Eq. 17 are not Shapley values of the training-loss game. The central interpretation—low SVV means low contribution to training-loss reduction—does not follow. The pruning gains may be real, but they don't validate the mechanism.\n\nThe empirical reporting has a second problem. Table 4's Clean and Random columns don't reconcile with Table 2 for the same DAE base; the Random column in Table 4 appears to be a copy of Table 2. That's a protocol or copy error that undermines the comparisons. On top of that, Table 3 shows SVV detects injected noise less accurately than ADT and Pred, and Table 4 shows it degrades performance on popular-item noise. The robustness claim is weak on the paper's own evidence.\n\nNovelty is real but narrower than claimed. The authors cite their own prior Shapley data valuation work in recommendation ([27], [18]), so \"first attempt\" is overstated; the pruning application is incremental.\n\nThis paper deserves a serious referee because the core idea, the code, and the noise protocol are worth engaging with, and the flaws are identifiable and fixable. But as it stands, the central mechanism is unsupported and the tables need correction. I would not cite it as evidence for Shapley-based pruning. If you bring it to a reading group, it will be a useful case study in checking algebraic equivalences before trusting attribution scores.","headline":"The pruning gains may be real, but the paper's central equivalence between Eq. 16 and Eq. 17 is false, so the Shapley values do not measure training-loss contribution, and the experimental tables are internally inconsistent.","tokens_in":21649,"tokens_out":3660,"would_cite":false,"duration_ms":36222,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that an interaction's training utility can be measured by its Shapley value under a loss-reduction value function, and that pruning the bottom fifth of interactions by this value improves recommendation accuracy and…","keywords":["Shapley values","data valuation","data pruning","recommender systems","denoising","implicit feedback","denoising autoencoder","FastSHAP"],"falsifier":"Take any user vector with observed interactions and distinct predicted scores; the two expressions differ by $(1/m)\\sum_v (f_v - f_v^2) - 1$ over observed items, so the claimed equivalence can be checked numerically on a single example. Then rerun the SVV pipeline with the loss-based value function of Eq. 16 in place of the average-score form of Eq. 17 and compare the pruned sets and final metrics; if the rankings change materially, the simplified value function is carrying the result.","tokens_in":20506,"feed_emoji":"🎯","tokens_out":10739,"duration_ms":95587,"temperature":0.7,"pith_summary":"Recommender systems are trained on interactions that include accidental clicks and popularity bias, yet some interactions that look noisy still help the model learn, while some clean-looking interactions add little. This paper proposes Shapley Value-driven Valuation (SVV), which scores each observed user–item interaction by its Shapley value—the average marginal contribution of that interaction to reducing the training loss of a denoising autoencoder—and then prunes the lowest-scoring interactions before retraining. On four real-world datasets, dropping the bottom fifth of interactions by this value improves accuracy over the base model by 0.75 to 7.22 percent and outperforms existing intent-based denoising methods. The paper also introduces a simulated-noise injection protocol that provides a verifiable ground truth for measuring how well a denoiser identifies harmful interactions.","feed_headline":"Prune by Shapley value: 20% fewer clicks, up to 7.22% better accuracy","feed_subtitle":"It prunes interactions that least reduce training loss, beating intent-based denoisers on four datasets.","key_machinery":"The machinery has three parts. First, a denoising autoencoder (DAE) serves as the base model: it accepts a user's binary interaction vector as input, masks entries during training, and reconstructs the full vector, so it can score subsets of interactions without retraining. Second, FastSHAP is a real-time amortized estimator that trains a neural network to output Shapley values in a single forward pass, using a Shapley-weighted least-squares loss and an additive efficiency normalization so that the outputs converge to exact Shapley values. Third, the value function maps a subset of a user's interactions to the average predicted score over the observed interactions (Eq. 17), which the paper treats as equivalent to negative normalized squared reconstruction error (Eq. 16) for binary feedback. The computed Shapley values then feed a threshold rule that keeps the top 80 percent of interactions and drops the rest.","core_discovery":"The paper's central claim is that an interaction's training utility is a measurable, model-driven quantity: its Shapley value under a value function that reflects loss reduction, estimated for every interaction in one forward pass using a real-time estimator. For binary implicit feedback, the value function is presented in simplified form as the average predicted score over a user's observed interactions, taken to be equivalent to the normalized squared reconstruction error. SVV ranks all observed interactions by this value, removes the bottom 20 percent, retrains the base denoising autoencoder on the remainder, and reports consistent gains in Recall@K and NDCG@K across the Ta Feng, CDs, Games, and Movies datasets. The paper interprets these results as showing that pruning should follow training utility rather than assumptions about user intent, and that the computed scores are meaningful enough to identify injected noise and to expose cases where a highly rated item has low training value.","pith_inferences":["A testable extension is to recompute the Shapley values with the exact loss-based value function of Eq. 16 instead of the simplified average-score form; if the pruned sets are stable, the simplification is not the active ingredient, and if they shift, the paper's stated equivalence would need revision.","The same valuation machinery could be run iteratively, scoring interactions again after each round of pruning, which would turn SVV into an online data-selection loop rather than a one-off filter.","The paper's noise-type results suggest that SVV may systematically favor long-tail interactions, so a natural follow-up is to measure catalogue coverage or diversity after pruning and test whether the gains come partly from popularity correction.","The benchmark protocol could be reused to evaluate other value functions, for example scoring by marginal contribution to held-out ranking metrics rather than reconstruction loss, to see which definition of training utility best predicts downstream accuracy."],"forward_implications":["Data denoising in recommender systems can be reframed as utility valuation, so interactions that deviate from assumed intent but help training are retained rather than discarded.","The pruning rule needs no external labels or auxiliary signals, only the base model's training signal, so it applies to any masked-input recommender architecture.","Because Shapley values are additive and per-interaction, SVV offers an interpretable score that practitioners can inspect, such as a rating-5 item assigned a low training value.","The simulated-noise protocol provides a benchmark for comparing denoisers by known noise recovery, while the results show that higher noise recovery does not automatically mean better recommendation performance.","The method's gains depend on noise type: it is strongest for unpopular-item noise and can be negative for popular-item noise, so the type of noise determines whether pruning helps."],"supporting_citations":[{"why":"Supplies FastSHAP, the real-time amortized estimator that makes Shapley-value computation feasible for large interaction matrices, including its efficiency constraint and convergence guarantee.","marker":"[25]"},{"why":"Defines the Shapley value from cooperative game theory, which the paper uses to attribute each interaction's marginal contribution to loss reduction.","marker":"[41]"},{"why":"Provides the ADT-R and ADT-T denoising baselines and the high-loss-as-noise observation that SVV is compared against.","marker":"[45]"},{"why":"Introduces the collaborative denoising autoencoder approach that the paper adopts as its base recommendation model.","marker":"[48]"},{"why":"Supplies the autoencoder-based recommendation formulation that underlies the DAE base model.","marker":"[40]"},{"why":"LightGCN serves as a graph-based recommendation baseline; because it needs retraining for each subset, it motivates the choice of a masked-input DAE.","marker":"[19]"},{"why":"Provides the confidence-weighted square loss used to train the DAE base model on implicit feedback.","marker":"[22]"},{"why":"Inspires the simulated-noise injection protocol that creates verifiable ground-truth noise for evaluating denoisers.","marker":"[17]"},{"why":"Data debugging with Shapley importance motivates the self-validating benchmark used to assess pruning quality.","marker":"[28]"}],"fun_headline_variants":["Shapley values: Prune by training utility, not user intent","Prune by impact: Shapley data pruning beats intent-based denoising","Let Shapley values decide which clicks to keep—20% fewer, 7.22% better","Stop assuming intent—Shapley values rank interactions by learning impact"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that, for binary implicit feedback, the average predicted score over observed interactions equals the negative normalized squared reconstruction error, so that Shapley values of the simplified value function really are Shapley values of training-loss reduction.","fun_headline_variants_meta":{"raw":{"variants":["Shapley values: Prune by training utility, not user intent","Prune by impact: Shapley data pruning beats intent-based denoising","Let Shapley values decide which clicks to keep—20% fewer, 7.22% better","Stop assuming intent—Shapley values rank interactions by learning impact"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000682,"raw_usage":{"total_tokens":3097,"prompt_tokens":946,"completion_tokens":2151,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":562,"completion_tokens_details":{"reasoning_tokens":2064}},"tokens_in":562,"tokens_out":2151,"duration_ms":16484,"temperature":1.0,"reasoning_tokens":2064,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:17:19.007087+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take any user vector with observed interactions and distinct predicted scores; the two expressions differ by $(1/m)\\sum_v (f_v - f_v^2) - 1$ over observed items, so the claimed equivalence can be checked numerically on a single example. Then rerun the SVV pipeline with the loss-based value function of Eq. 16 in place of the average-score form of Eq. 17 and compare the pruned sets and final metrics; if the rankings change materially, the simplified value function is carrying the result.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies FastSHAP, the real-time amortized estimator that makes Shapley-value computation feasible for large interaction matrices, including its efficiency constraint and convergence guarantee."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the ADT-R and ADT-T denoising baselines and the high-loss-as-noise observation that SVV is compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the collaborative denoising autoencoder approach that the paper adopts as its base recommendation model."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the autoencoder-based recommendation formulation that underlies the DAE base model."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the confidence-weighted square loss used to train the DAE base model on implicit feedback."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Inspires the simulated-noise injection protocol that creates verifiable ground-truth noise for evaluating denoisers."},{"cited_title":"Data Debugging with Shapley Importance over End-to-End Machine Learning Pipelines","cited_arxiv_id":"2204.11131","evidence_quote":"Data debugging with Shapley importance motivates the self-validating benchmark used to assess pruning quality."}],"review_version":1}