{"id":"b546721d-ac13-44e0-8ed5-64365a23603d","arxiv_id":"2412.01460","paper_version":8,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A survey and benchmark that classifies Shapley value applications in data analytics, decomposes solution techniques, and validates them through the open-source SVBench framework.","lead":"This paper surveys how Shapley value (SV), a game-theory tool for distributing credit, is used across data analytics tasks, and introduces SVBench, an open-source framework for building SV applications. It decomposes existing SV methods into reusable techniques, tests them on ten datasets, and identifies trade-offs among efficiency, privacy, accuracy, and interpretability.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Efficiency claim compares algorithms at a fixed stability threshold (Δφ<0.05) rather than matched approximation error; hybrid gains may be early-stopping artifacts.","rationale":"The paper is a broad, well-organized survey with a genuinely useful modular framework and open-source code; I do not question the taxonomy or the 'first comprehensive study' framing. The soft spot is in the empirical validation that is supposed to make the framework prescriptive. The reader's conditional verdict already flags reproducibility and post-hoc choices; my concern is more specific: the efficiency comparison uses a convergence criterion that is algorithm-dependent and not tied to approximation error. Because TC, GA, and TSS are exactly the techniques most likely to create stable-but-biased estimates, the claim that hybrids 'achieve better performance in most cases' may be an artifact of the stopping rule. This does not by itself invalidate the survey or SVBench as a platform, but it means Research Direction 1 and the 'validated' guidance should be treated as conditional pending error-matched evaluation. I therefore keep the reader's CONDITIONAL verdict unchanged. I partly agree with the reader's §5.4 concern: the leave-one-out/singleton operationalization of 'impact on overall utility' is indeed not implied by the Shapley formula, and that is a related but separate logical gap. The most load-bearing issue for the paper's central quantitative claim is the §5.1 comparison, so my agreement is partial.","tokens_in":36568,"tokens_out":8836,"duration_ms":79509,"concrete_test":"Re-run the §5.1 configurations with an error budget: for each base algorithm and optimization combination, stop when ε ≤ 0.05 (and also ε ≤ 0.1) instead of when Δφ < 0.05; then compare Nuc×Tuc, Nuc, and ε across at least 10 seeds, reporting win rates. If TC+GA or TC+GA+TSS no longer beats TC alone or GA alone in a majority of the task–dataset pairs at matched ε, Research Direction 1 should be weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing quantitative assertion is Research Direction 1 (§5.1): hybrid algorithms integrating multiple efficiency techniques 'achieve better performance' than single-technique algorithms 'in most cases.' The evaluation stops every run when Δφ < τ with τ = 0.05, and reports total cost Nuc×Tuc, complexity Nuc, and approximation error ε = 1 − cosine(φ̂, φ) as separate panels (Figure 4). Comparing at a fixed self-stability threshold is not comparing at matched accuracy: TC deliberately stops adding players once U(S) is close to U(N), and GA/TSS accelerate training/inference at the cost of increased variance. Such techniques can make φ̂ stabilize before the error ε is small, so the observed Nuc and time reductions may reflect premature convergence rather than genuine efficiency–accuracy improvements. The text even concedes TC 'may enlarge the approximation error' and that GA/TSS 'may enlarge the variance,' yet no combined, error-matched, or accuracy-adjusted cost comparison is used to justify the headline recommendation. Without this, 'better performance' conflates speed with quality, and the central validation of SVBench's practical guidance is unsupported as stated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents a survey and an empirical benchmark of Shapley value (SV) techniques across the data analytics (DA) workflow. It taxonomizes SV applications into pricing, selection, weighting, and attribution; identifies four challenges (computation efficiency, approximation error, privacy preservation, interpretability); and decomposes existing methods into reusable techniques such as iteration reduction, ML speedup, variance reduction, and privacy protection. The authors implement SVBench, an open-source modular framework, and use it to evaluate hybrid algorithms, sampling strategies, privacy-protection measures, and the utility-based interpretability paradigm across four DA task types (RI, DV, DSV, FL) on ten datasets. The paper concludes with findings and seven research directions.","tokens_in":36804,"tokens_out":7316,"duration_ms":55537,"significance":"The main value of the paper is organizational: it provides a comprehensive taxonomy of SV use in DA, disentangles monolithic algorithms into modular techniques, and makes an implementable benchmark (SVBench) available. The code and data release, the breadth of cited work (183 references), and the explicit enumeration of findings and research directions are useful for practitioners and researchers. If the experimental claims are made rigorous, the framework could serve as a common testbed for comparing SV algorithms. The four-challenge decomposition and the conflict analysis (e.g., efficiency vs. error, privacy vs. effectiveness) are conceptually helpful.","major_comments":[{"comment":"The headline efficiency claim in §5.1 compares algorithms at a fixed self-stability threshold Δφ < τ with τ = 0.05 rather than at matched approximation error. Because TC truncates once U(S) is close to U(N), and GA/TSS accelerate training/evaluation at the cost of increased variance, a hybrid algorithm's φ̂ can stabilize before the approximation error ε is small. The paper reports Nuc, time, and ε as separate panels in Figure 4 but does not compare cost at a common ε tolerance or plot accuracy-adjusted cost curves. The text itself concedes that TC 'may enlarge the approximation error' and GA/TSS 'may enlarge the variance.' Consequently, the observed reductions in Nuc and time for hybrid algorithms may reflect early stopping rather than genuine efficiency–accuracy improvements, and the conclusion that hybrid algorithms 'achieve better performance' is not established. Please add error-matched comparisons (e.g., minimum cost to reach a fixed ε, or accuracy-conditioned cost curves) or qualify the claim accordingly.","section":"§5.1, Research Direction 1"},{"comment":"The interpretability experiment in §5.4 uses the single-coalition differences U(N\\{p_i}) - U(N) and U({p_i}) - U(∅) as the operationalization of a player's 'impact on the task's overall utility' (Figure 7). However, the Shapley value φ_i is the weighted average of marginal contributions over all 2^n coalitions, so there is no a priori reason that these two particular differences should track φ_i. The paper reports that the variance of marginal contributions is large (e.g., DV-Wind variance 11.83–18.00 vs. average 0.06–1.43) but does not explain why leave-one-out impact is the correct benchmark for the utility-based interpretation. The conclusion that 'the fluctuating results contradict the mainstream SV interpretations' therefore overstates the evidence; the results show only that SV does not correlate with these two specific marginal contributions. A valid test would require a more representative measure of 'impact' (e.g., average absolute marginal contribution over coalitions) or a direct correlation between SV and a task-specific notion of utility change.","section":"§5.4, Figure 7"},{"comment":"The evaluation section states that MLE is used as the base algorithm in §§5.2–5.4 because it 'generally achieves better efficiency and accuracy performance' in §5.1, and that 'varying base algorithms would not influence conclusions in those subsections.' This choice is made after observing the §5.1 results, and the invariance claim is not tested. Since §§5.2–5.4 draw conclusions about approximation-error trade-offs, privacy effectiveness, and interpretability that are intended to generalize across SV algorithms, at least one robustness check (e.g., repeating the key experiments with MC, the most widely used base algorithm) or a clear scope limitation is needed before these conclusions can be considered general.","section":"§5 (Evaluation introduction) and §§5.2–5.4"},{"comment":"No random seeds, repetitions, or dispersion measures are reported for any of the stochastic experiments in §5.1–§5.4 (MC/MLE sampling, privacy noise, attack implementations). The comparisons rely on single runs, so observed differences—especially the 'in most cases' claim in §5.1 and the AUROC/MAE differences in §5.3—could be within run-to-run variation. Please report multiple seeds with means/variances or a statistical test, or at minimum justify why the reported numbers are stable.","section":"§5.1–§5.4"}],"minor_comments":[{"comment":"The 'score of impacts' definition is written as a signed sum Σ (φ̂_i/Σφ̂_i − φ_i/Σφ_i), which can be negative and cancel out; the figure shows nonnegative values, so the formula should be clarified as, presumably, the sum of absolute differences.","section":"§5.2"},{"comment":"Finding 11 uses the abbreviation 'SMCP' for secure multiparty computation, while the correct term 'SMPC' is used elsewhere; please fix the inconsistency.","section":"§3.2.3"},{"comment":"The ♠ superscript on RE, MLE, GT, CP is mentioned in the text ('the ♠-tagged base algorithm') but not defined in the table or caption; please explain what it marks.","section":"Table 9"},{"comment":"The cell entries '/reve' are not explained in the text or caption; the reader has to infer that they indicate coverage of a given purpose or solution. Define the notation.","section":"Table 1"},{"comment":"The paper claims to be 'the first comprehensive survey of SV applied throughout the DA workflow'; given the existing surveys cited in Table 1, the novelty claim would be more precise if stated as 'first to decompose techniques into building blocks and benchmark them in a unified framework.'","section":"§1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a survey-plus-benchmark; its main value is organizational rather than theoretical. The experimental sections are the weakest part, and the major issues above are fixable in revision. The authors should also contrast their survey scope with [17, 111, 126] more explicitly, as the 'first comprehensive' claim may draw scrutiny from the community. The framework and code release are commendable, but the current evaluation depth is not yet sufficient to support the paper's broad practical recommendations."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read of Lin et al., arXiv:2412.01460. The paper earns its keep as a survey and framework: the taxonomy of players (features, tuples, datasets, derivatives), the four-challenge decomposition, and the disentangling of techniques (iteration reduction, ML speedup, variance reduction, privacy, interpretability) into building blocks is genuinely useful. SVBench is modular, open-source, and the experiments cover four DA tasks with multiple datasets. The discussion of conflicts between efficiency, error, privacy, and interpretability is the kind of consolidated guidance the field has needed.\n\nThe soft spots are concentrated in the empirical claims. The §5.1 efficiency comparison runs every variant to a fixed stability threshold (Δφ < 0.05) and then reports cost, Nuc, and approximation error in separate panels. That's not a matched-accuracy comparison. TC, GA, and TSS can make φ̂ stabilize before ε is small; the text concedes as much. So Research Direction 1 (\"hybrids do better in most cases\") is not properly supported. It may be true, but the data as presented conflates speed with quality. The authors also don't report seeds or error bars, which matters for a benchmark claiming to guide practice.\n\nThe §5.4 interpretability experiment is the weakest section. Using leave-one-out differences U(N\\{p_i})−U(N) and singleton differences U({p_i})−U(∅) as proxies for 'impact on overall utility' is questionable. A Shapley value is an average of marginal contributions over all 2^n coalitions; two specific marginal contributions need not correlate with that average, especially when variance is high. The paper's own variance numbers (e.g., DV-Wind) show the average is a noisy summary, but that does not contradict utility-based interpretations. It would be fairer to say the paradigm's practical use should be tempered by variance, not that the paradigm fails.\n\nThe 'first comprehensive study' framing is more or less defensible given the workflow coverage, and the citation practice against prior surveys is fine. This is a solid, useful paper, but the headline quantitative conclusions need revision or re-analysis. I'd send it to referees in its current state — the survey content is valuable enough to warrant the effort — but the empirical sections need a rewrite or an accuracy-matched evaluation.\n\nBottom line: recommend serious review, but with substantial revision expected.","headline":"Useful survey and modular framework, but the headline efficiency and interpretability claims rest on comparisons that don't match accuracy or the actual definition of Shapley value.","tokens_in":37314,"tokens_out":3322,"would_cite":true,"duration_ms":28572,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that Shapley value in data analytics reduces to four challenges and a small set of reusable technique blocks, and that hybrid combinations usually beat any single block.","keywords":["Shapley value","data valuation","cooperative game theory","data analytics","model interpretability","differential privacy","approximation algorithms","SVBench"],"falsifier":"Measure exact Shapley values and leave-one-out utility changes on the same task: for every player compute $\\phi_i$, $U(N\\setminus\\{p_i\\})-U(N)$, and $U(\\{p_i\\})-U(\\emptyset)$, then compute the rank correlation between $\\phi_i$ and each change across all players. The paper currently shows scatter-style plots that are read by eye; if a clean, high positive correlation appears across all four player types on, say, the DV-Wind and DSV-2Dplanes tasks the authors themselves flag as mismatched, the 'fluctuate and contradict' conclusion would be overturned.","tokens_in":36369,"feed_emoji":"🎲","tokens_out":6288,"duration_ms":52334,"temperature":0.7,"pith_summary":"Shapley value, the cooperative-game answer to 'how much did each player contribute to a joint outcome,' is now used across the whole data analytics workflow, from pricing datasets and models to selecting features, weighting federated-learning clients, and explaining predictions. This paper argues that these scattered uses share a common structure: every application is a cooperative game defined by a choice of players and a utility function, and every practical algorithm combines a small set of reusable building blocks that cut iteration count, speed up machine-learning utility calls, reduce sampling variance, or protect privacy. The authors present the first comprehensive study of Shapley value across the entire workflow, implement their analysis as an open-source modular framework called SVBench, and run experiments across four task types. Their headline empirical findings are that hybrid algorithms integrating several efficiency techniques outperform single-technique algorithms in most cases, and that the standard 'higher Shapley value means more impact on task utility' interpretation fluctuates and is contradicted by the data for all four player types.","feed_headline":"Hybrid Shapley computing beats single-method algorithms","feed_subtitle":"A modular open framework shows which Shapley-value building blocks speed pricing, selection, weighting, and attribution.","key_machinery":"The central object is the Shapley value formula itself, the weighted average over all $2^n$ coalitions of a player's marginal contribution to coalition utility (Equations 1 and 2), together with cooperative game modeling, in which the analyst decides who the players are (features, tuples, datasets, or derivatives such as trained models) and what the utility measures (goodness-of-fit scores or raw task outputs). The argument is carried by the decomposition of existing algorithms into reusable building blocks and by SVBench, a modular framework with configurable base algorithm, sampling strategy, optimization, and privacy modules that lets those blocks be recombined. What the machinery does is turn a sprawling literature into a testable design space: the paper's experiments implement dozens of combinations and adjudicate which blocks conflict and which cooperate.","core_discovery":"The core claim is that the Shapley value has become a general-purpose instrument for the whole data analytics lifecycle, and that the field is best understood not as a zoo of task-specific algorithms but as a small design space. The paper condenses four challenges—computation efficiency, approximation error, privacy preservation, and interpretability—and disentangles existing solutions into reusable techniques: iteration reduction (Monte Carlo, regression, multilinear extension, group testing, compressive permutation sampling, truncation, predictor learning), ML speedup (gradient approximation, test sample skip, model appraiser), variance reduction (stratified, antithetic, kernel-based), and privacy techniques (non-perturbation masking, homomorphic encryption, secure multiparty computation, quantization, dimension reduction, differential privacy). It then builds SVBench to assemble these blocks and evaluates them on result interpretation, data tuple valuation, dataset valuation, and federated learning. The quantitative study supports two findings the authors emphasize: hybrid algorithms that combine multiple efficiency techniques generally beat single-technique ones, and the mainstream utility-based interpretation of Shapley values—larger value means more impact on the task's overall utility—does not hold up when players are removed or added.","pith_inferences":["A direct extension the authors leave implicit: measuring impact by leave-one-out utility differences may itself be the wrong benchmark for the utility-based paradigm, since Shapley value averages over all coalitions; a fairer test would compare Shapley value against the expected marginal contribution over the actual coalition-size distribution of the task.","The variance numbers reported (e.g., marginal-contribution variance 11.83–18.00 vs average 0.06–1.43 in DV-Wind) suggest a testable redesign: a variance-aware Shapley variant that weights players not only by mean contribution but by volatility would likely align better with removal/addition impacts.","Because the evaluation covers four task types but relatively few datasets per type, the 'in most cases' conclusion could be stress-tested on large-scale, high-dimensional tasks and pre-trained models, where both utility costs and coalition structure differ.","The modular decomposition also invites a transfer result: if the building-block taxonomy is right, then any efficiency technique proven on data valuation should port to dataset pricing or model weighting with only a change of utility function; SVBench is the natural vehicle to test that."],"forward_implications":["If hybrids usually beat single techniques, then practitioners should default to combining truncation with a sampling-based base algorithm, and to adding gradient approximation and test sample skipping when utility computation involves costly model training.","If utility-based interpretation is unreliable, decisions that rank data or models by Shapley value—pricing, selection, weighting, attribution—need to report variance of marginal contributions alongside the value, not just the value.","If SVBench's modularity holds, new Shapley applications can be built by recombining existing blocks instead of designing bespoke algorithms from scratch, lowering the engineering cost of data marketplaces and federated learning incentives.","If the approximation-error results generalize, convergence thresholds should be tuned dynamically during sampling, stopping early when the downstream task (e.g., top-k selection) stops changing, rather than fixed a priori.","If privacy techniques distort Shapley rankings, then differentially private Shapley reporting needs a strength-setting guide that balances attack prevention against ranking fidelity."],"supporting_citations":[{"why":"Defines the Shapley value solution concept on which all DA applications and the paper's Equation 1 rest.","marker":"[116]"},{"why":"Supplies the DataShapley algorithm and the MC+GA utility valuation baseline, and grounds the data-tuple valuation tasks in Tables 3 and 7.","marker":"[43]"},{"why":"Supplies KernelSHAP, the canonical attribution technique and the linear/tree task-specific iteration reduction approach for RI.","marker":"[78]"},{"why":"Supplies the uniform division and influence function techniques for efficient data valuation, and the high-value/low-value data tuple interpretations.","marker":"[54]"},{"why":"Supplies GTG-Shapley, the convergence criterion and MC+TC hybrid used as comparison in federated learning efficiency experiments.","marker":"[72]"},{"why":"Defines the feature inference attack on Shapley values and the QT/DR/DP countermeasures used in Section 5.3.","marker":"[80]"},{"why":"Defines threshold KNN-Shapley and the membership inference attack plus DP defense used to evaluate privacy protection in data valuation.","marker":"[144]"},{"why":"Supplies the counterfactual interpretation paradigm and dataset pricing use case cited for explaining why one dataset has higher SV than another.","marker":"[125]"}],"fun_headline_variants":["Hybrid Shapley beats single-method in data analytics","Shapley value: universal building blocks for data analytics","SVBench: hybrid Shapley outperforms individual tricks","Shapley value impact rankings fail when players change","Modular Shapley framework tackles four data analytics challenges"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that utility-based Shapley interpretations are contradicted by the data rests on treating $U(N\\setminus\\{p_i\\})-U(N)$ and $U(\\{p_i\\})-U(\\emptyset)$ as measures of a player's impact on overall utility, even though the Shapley value averages marginal contributions over all $2^n$ coalitions and these two specific differences need not track that average.","fun_headline_variants_meta":{"raw":{"variants":["Hybrid Shapley beats single-method in data analytics","Shapley value: universal building blocks for data analytics","SVBench: hybrid Shapley outperforms individual tricks","Shapley value impact rankings fail when players change","Modular Shapley framework tackles four data analytics challenges"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000316,"raw_usage":{"total_tokens":1789,"prompt_tokens":941,"completion_tokens":848,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":557,"completion_tokens_details":{"reasoning_tokens":767}},"tokens_in":557,"tokens_out":848,"duration_ms":7182,"temperature":1.0,"reasoning_tokens":767,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:17:28.658993+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure exact Shapley values and leave-one-out utility changes on the same task: for every player compute $\\phi_i$, $U(N\\setminus\\{p_i\\})-U(N)$, and $U(\\{p_i\\})-U(\\emptyset)$, then compute the rank correlation between $\\phi_i$ and each change across all players. The paper currently shows scatter-style plots that are read by eye; if a clean, high positive correlation appears across all four player types on, say, the DV-Wind and DSV-2Dplanes tasks the authors themselves flag as mismatched, the 'fluctuate and contradict' conclusion would be overturned.","supporting_citations":[{"cited_title":"Is Shapley Value fair? Improving Client Selection for Mavericks in Federated Learning","cited_arxiv_id":"2106.10734","evidence_quote":"Supplies the uniform division and influence function techniques for efficient data valuation, and the high-value/low-value data tuple interpretations."}],"review_version":1}