{"id":"5c01664a-b625-42a4-91ba-28a556853de9","arxiv_id":"2506.12619","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Semivalue-based data valuations are shown to be highly sensitive to plausible utility-function choices and are gameable under the paper's weak definition of gameability.","lead":"This paper argues that semivalue-based data valuations, like Shapley values, are not uniquely determined by the learning task: reasonable choices about how to score models or handle small training subsets can shift which data points are valued most. The authors show these choices can be gamed by a biased valuator, so valuation outcomes should be treated as contingent rather than objective.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Gameability claim is unsupported as stated: Definitions 6-7 count the exponential semivalue computation as free, making Proposition 1 vacuous, and Algorithm 4's objective coefficient is incorrect.","rationale":"The arbitrariness half of the paper is well supported: the ambiguity-set framework (Definitions 2-5) is coherent, and the experiments show large payout and rank variation across plausible utility choices. The load-bearing weakness is in the gameability half. As the reader notes, Definitions 6-7 count cost relative to semivalue computation, which is already exponential for exact evaluation; Proposition 1 then becomes either a tautology or a violated bound depending on how 'baseline' is read. This is not a minor wording issue: the abstract's 'low-cost adversarial strategies' is the claim that makes the paper's contribution novel relative to prior arbitrariness critiques. In addition, Algorithm 4 and Proposition 2 contain a concrete coefficient error: the aggregate coefficient should be l w_{k-1} - (|P|-l) w_k, not l w_k - (k-l) w_{k-1}. This is fixable, but it means the stated algorithm does not implement the stated objective. Because the reader's conditional verdict already calls for correcting these issues and shipping code, I do not see a reason to move the verdict.","tokens_in":21088,"tokens_out":16105,"duration_ms":170629,"concrete_test":"Re-derive Definitions 6 and 7 with total end-to-end utility-evaluation count as the cost measure and recompute Proposition 1 for exact Shapley with a two-element candidate set and N=10. If the two full semivalue computations are counted as baseline, the 'additional' cost is zero and Proposition 1 is vacuous; if only one semivalue is baseline, the total is roughly 2^11 evaluations, violating the O(poly(N)) bound. Either outcome shows the 'low-cost' headline is an artifact of the current cost metric.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Definitions 6 and 7 define gameability by overhead beyond computing psi(U) for each U in the candidate set. With that metric, Algorithm 2 for a finite O(1) candidate set performs zero additional evaluations, so Proposition 1 is true by construction and does not establish that manipulation is low-cost in any end-to-end sense. If the intended baseline is one semivalue computation, as the text's 'baseline cost ... O(2^N)' suggests, then Algorithm 2 costs |U| * 2^N utility evaluations, which is not O(poly(N)) additional for exact semivalues, so Proposition 1 fails. Separately, Algorithm 4 and the proof of Proposition 2 in Appendix D.3 use the wrong coefficient for U(S) in the aggregate favorability: Definition 1 gives |S intersect P| w_{|S|-1} - (|P| - |S intersect P|) w_{|S|}, while line 14 uses (l w_k - (k-l) w_{k-1}). The algorithm can therefore return a small-coalition behavior that does not maximize aggregate value, so Proposition 2 is unproven. The arbitrariness results remain credible, but the 'gameable' contribution needs a corrected end-to-end cost definition and corrected algorithms.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript argues that semivalue-based data valuation (Shapley, Banzhaf, LOO) is fundamentally underspecified because the utility function V∘A involves modeling choices that the learning task does not determine. It formalizes three ambiguity families: small-data algorithmic behavior (Definitions 2–3), monotone score transformations (Definition 4), and cost-ratio choices (Definition 5). It introduces favorability metrics (aggregate value, rank, filter survival) and a range measure to quantify how much these choices can shift individual or group valuations. Empirically, it reports large payout and rank shifts on five datasets for both Shapley and Banzhaf values. The paper also defines exact and (ε,δ)-gameability (Definitions 6–7) and claims Propositions 1–3, with Algorithms 2–5, showing that a biased valuator can efficiently find utility specifications that favor a target group. It concludes with ethical implications and a burden-of-justification argument for semivalue-based valuation.","tokens_in":21371,"tokens_out":9520,"duration_ms":113269,"significance":"The arbitrariness analysis is a useful and timely contribution: formalizing concrete ambiguity sets and measuring their effect on payout, rank, and filter-survival metrics provides a clear framework for an often-invoked but underdeveloped critique of Shapley-style data valuation. The monotonic-transformation and small-coalition arguments are mathematically straightforward, and the empirical sensitivity results are plausible as illustrations. The gameability part is intended to sharpen the stakes, but in its current form it is not established: the exact-gameability definition trivializes constant-size candidate sets, and the small-cardinality gaming algorithm and its proof contain an incorrect coefficient. These are load-bearing issues for the 'gameable' claim in the abstract. The arbitrariness contribution is not invalidated by these problems, but the formal gameability claims need substantive correction before the paper's headline claims can be accepted.","major_comments":[{"comment":"Definition 6 counts only additional utility evaluations beyond those required to compute ψ(U) for each U∈U. For any O(1)-size candidate set, Algorithm 2 uses zero additional utility evaluations by construction, so Proposition 1 is true under Definition 6 regardless of the structure of U; it does not demonstrate that manipulation is low-cost relative to an honest evaluator computing one semivalue. If the intended baseline is a single semivalue computation, as §3.3 states, then Algorithm 2 with exact semivalues requires |U|·2^N utility evaluations, which is not polynomial additional work. The gameability definition should be reformulated in end-to-end cost terms, and Proposition 1 should be re-derived under that definition; as written, the low-cost adversarial-strategy claim is forced by the metric rather than demonstrated.","section":"§3.3, Definitions 6–7 and Proposition 1"},{"comment":"The coefficient used for U(S) in the aggregate favorability is incorrect. From Definition 1, summing ψ_j(U) over j∈P gives coefficient |S∩P|·w_{|S|-1} − (|P|−|S∩P|)·w_{|S|} on U(S). The proof of Proposition 2 instead defines α(k,l) = l·w_{k-1} + (k−l)·w_k, and Algorithm 4 line 14 uses (l·w_k − (k−l)·w_{k−1}); both differ from the correct coefficient in sign and in which weight multiplies the subtractive term. Therefore the quantity optimized by Algorithm 4 is not F_agg(ψ(U);P), and Proposition 2 is not proven as stated.","section":"§D.3 and Algorithm 4"},{"comment":"Algorithm 1 computes X^-_{i,k} and X^+_{i,k} as stratum averages (Eq. 3), but line 10 forms ψ̂_i(U,w_k) = Σ_k w_k (X^+_{i,k} − X^-_{i,k}) without the binomial coefficient binom(N−1,k). Definition 1 sums marginal contributions over all subsets, so with average estimates the correct estimator is Σ_k w_k·binom(N−1,k)·(X^+_{i,k} − X^-_{i,k}). Lemma 1's proof implicitly inserts this binomial coefficient, making the proof inconsistent with the pseudocode. Since all experiments and Algorithms 2–5 rely on Algorithm 1, the reported valuation magnitudes and range/payout figures are not reproducible from the pseudocode as written.","section":"§C, Algorithm 1, and Lemma 1"}],"minor_comments":[{"comment":"The decomposition ψ(U_{p_t}) = ψ(U_T) + (1/(1−p_t))·ψ(U_F) has the wrong sign and the following line uses U_T twice; the correct relation from Definition 5 is ψ(U_{p_t}) = ψ(U_T) − (p_t/(1−p_t))·ψ(U_F). The endpoint conclusion still follows because the coefficient of ψ(U_F) is monotone in p_t, but the displayed equations should be corrected.","section":"§D.4, proof of Proposition 3"},{"comment":"The text refers to 'Figure 4.2' where it should refer to Figure 2.","section":"§4.2"},{"comment":"The phrase 'lackspoly(N )complexity sought per 7' should read 'lacks the poly(N) complexity sought per Definition 7'.","section":"§3.3"},{"comment":"The statement mixes 'U ⊆ Usmall' with 'This candidate set is U0(k*)'; the quantifiers and notation should be aligned with the proof's construction of U^b.","section":"Proposition 2 statement"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the journal's audience and the arbitrariness contribution is worth publishing after revision. The gameability claims, which are part of the abstract and contribution list, currently rest on a definitional artifact (Definition 6) and an incorrect algebraic coefficient in Proposition 2/Algorithm 4; these need to be fixed before the paper can be accepted. I would also ask for a careful reconciliation of Algorithm 1 with Lemma 1, since the experimental results depend on that estimator."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: the arbitrariness case is real and worth taking seriously, but the gameability headline does not hold up as stated. I checked the stress-test note and it is accurate. Two problems stand out. First, Definitions 6–7 count the cost of computing semivalues as free, so Proposition 1 is true by construction: any O(1) candidate set is gameable because you were going to compute ψ(U) for each U anyway. That does not make the adversarial strategy low-cost in any end-to-end sense; for exact semivalues you pay 2^N per utility. Second, the aggregate favorability coefficient is wrong. From Definition 1, for a set S with |S|=k and |S∩P|=l, the coefficient on U(S) in Fagg is l*w_{k-1} - (|P|-l)*w_k. The proof in D.3 uses l*w_{k-1} + (k-l)*w_k, and Algorithm 4 line 14 uses l*w_k - (k-l)*w_{k-1}. Both are wrong, so Proposition 2 is unproven and the algorithm optimizes the wrong objective. That is a genuine flaw, not a nitpick.\n\nWhat is new and good: the formal ambiguity sets (small-data fallback, monotone score transformations, cost-ratio) are a useful organizing device, and the empirical sensitivity results are credible. The observation that monotone-equivalent utility functions can produce very different Shapley values, while LOO is rank-stable under them, is worth highlighting. The paper is clearly written and the ethical discussion is measured. Proposition 3 (cost-ratio) is the one gameability result that stands out as a real algorithmic insight: linearity reduces optimization over an interval to checking the boundaries.\n\nSoft spots beyond the gameability issues: no code or sampling details are shipped, so the empirical results are hard to verify independently. The variance proof in D.2 has a small union-bound issue (the bound should sum over suboptimal U', not just take one), though it is fixable. Assumption 1 is reasonable.\n\nWho this is for: anyone using Shapley or Banzhaf for data valuation in high-stakes settings. The arbitrariness message deserves to be heard. The gameability part needs a real end-to-end cost definition and corrected algorithms. My recommendation: send to peer review, but require major revision. The arbitrariness half alone justifies referee time, and the gameability claims cannot be accepted without fixing the definition and the coefficient bug.","headline":"The arbitrariness argument is solid, but the gameability results are trivial or buggy as written.","tokens_in":21864,"tokens_out":6134,"would_cite":true,"duration_ms":64513,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["91A12"],"pacs":[],"model":"deepseek-v4-flash","headline":"Semivalue-based data valuation is arbitrary and gameable.","keywords":["data valuation","semivalue","Shapley value","Data Banzhaf","utility underspecification","arbitrariness","gameability","cooperative game theory"],"falsifier":"Run a fixed benchmark where Data Shapley payouts are computed for two utilities that differ only in the small-coalition fallback threshold, say $0.05|D|$ and $0.1|D|$, and repeat across many datasets. If the median absolute payout change stays well below one average contributor and the bottom-10% filter memberships are identical, the arbitrariness claim for that ambiguity class would be contradicted.","tokens_in":20889,"feed_emoji":"🎲","tokens_out":11853,"duration_ms":130435,"temperature":0.7,"pith_summary":"Semivalue methods such as Data Shapley and Data Banzhaf promise a principled, axiom-backed way to assign credit to individual training examples. This paper argues that the promise cannot survive contact with practice: the utility function that scores every subset of data is under-specified, and small but defensible changes—how tiny training coalitions are handled, whether the score is monotonically rescaled, or how false positives are weighted—can shift individual payouts by multiple average contributors and reorder ranks enough to change which points a low-value filter drops. Because many of these utilities are observationally equivalent at the level of model selection, a bad-faith valuator can choose the one that favors a preferred group at low extra cost, while a good-faith valuator has no principled basis to single one out. The paper formalizes this as ambiguity sets and favorability metrics, proves that several ambiguity classes are gameable, and demonstrates large value and rank shifts on real benchmark datasets. If the claim holds, semivalue-based valuations are contestable instruments whose outputs should never be reported without a justification of the utility choice that produced them.","feed_headline":"Semivalue data valuations are arbitrary and gameable","feed_subtitle":"Small, defensible tweaks to the utility function shift payouts by several average units and let biased valuators favor a group.","key_machinery":"The central object is the semivalue, $\\psi_j(U,w)=\\sum_{S\\subseteq D\\setminus\\{z_j\\}} w_{|S|}[U(S\\cup\\{z_j\\})-U(S)]$, a weighted average of a datapoint's marginal contributions across all coalitions; Data Shapley, Data Banzhaf, and leave-one-out are instances that differ only in the weights $w_{|S|}$. The argument's machinery is the ambiguity set: a family of utility functions that are observationally equivalent at the level of model selection but assign different scores to coalitions. Three such sets drive the analysis—small-data algorithmic ambiguity (behavior on tiny coalitions is undefined, so fallbacks are defensible), score-transformation ambiguity (monotone rescaling preserves model choice), and cost-ratio ambiguity (net benefit with unknown false-positive versus false-negative costs). Favorability metrics (aggregate payout, rank, filter survival) convert a value vector into a concrete outcome for a preferred group, and the gameability definitions count adversarial cost in extra utility evaluations beyond one semivalue computation. Algorithms that exploit semivalue linearity and the finite range of plausible behaviors locate the most favorable utility in these ambiguity sets with only polynomial extra evaluations.","core_discovery":"On the paper's own terms, the central discovery is that every semivalue-based data valuation silently commits to a counterfactual model that the learning task does not determine. For a fixed dataset and model, three families of equally defensible choices—the fallback behavior of the learning algorithm on coalitions below a size threshold, monotone transformations of the performance score, and the false-positive cost ratio in a net-benefit metric—produce substantially different semivalue vectors. These choices are invisible to model selection, so the valuation pipeline can be driven toward a preferred group without any visible distortion of the model itself. The paper further shows that low-cost algorithms can search these ambiguity sets for the utility that maximizes a chosen group's aggregate value, rank, or filter survival, and that empirical Shapley and Banzhaf payouts on standard datasets vary by several average units under such perturbations. The conclusion is that the fairness and objectivity attributed to semivalues are contingent on utility choices that are neither fixed by the data nor settled by any axiom.","pith_inferences":["The paper's gameability cost is measured as utility evaluations beyond one semivalue computation, which is already exponential in the dataset size; under an end-to-end cost measure, the 'low-cost' claim would need a separate argument that an adversary can beat the exponential semivalue computation itself.","The arbitrariness argument transfers naturally to feature-based Shapley values and other attribution schemes: any counterfactual that is not uniquely defined creates a similar ambiguity set, and the same favorability metrics could audit interpretability claims.","A practical guardrail suggested by the structure of Proposition 3 is that for linear utility families such as net benefit, the favorability extremes over a parameter interval occur at the interval's endpoints, so an auditor can bound adversarial advantage by evaluating only two utilities.","If regulators or platforms require pre-registration of the utility function, the exploit narrows, but the epistemic problem remains because the ambiguity set is in principle unbounded when all monotone score transformations are admitted."],"forward_implications":["Data Shapley and Data Banzhaf payouts should not be reported as point estimates; a single utility choice can move an individual's payout by several average contributors, so any report should include the sensitivity range over plausible utilities.","Decisions that use semivalues for contributor compensation, dataset acquisition, or low-value filtering inherit the arbitrariness; two equally defensible specifications can disagree about whose data is kept or credited.","A bad-faith valuator can systematically favor a target group by picking a defensible utility, so transparency about the selected utility is necessary but not sufficient for fairness claims.","Leave-one-out valuation is comparatively robust to small-coalition and score-transformation ambiguities because it depends only on full and leave-one-out subsets, but it remains sensitive to the choice of performance metric.","The same gameability algorithms can be run by a good-faith evaluator as a robustness audit: report the worst- and best-case valuations over an ambiguity set alongside the chosen specification."],"supporting_citations":[{"why":"Defines the original value for cooperative games that semivalue data valuation generalizes and whose axiomatic authority the paper contests.","marker":"[Shapley, 1952]"},{"why":"Introduces Data Shapley, the primary semivalue whose payouts and rankings the paper shows are utility-sensitive.","marker":"[Ghorbani and Zou, 2019]"},{"why":"Introduces Data Banzhaf, the second semivalue used to show that smaller-coalition weights change, but do not remove, arbitrariness.","marker":"[Wang and Jia, 2023]"},{"why":"Supplies the benchmark package, baseline learners, and the small-coalition default behavior that motivates Definition 2.","marker":"[Jiang et al., 2023]"},{"why":"Provides the stratified sampling budget scheme used by Algorithm 1 for approximate semivalue computation.","marker":"[Wu et al., 2023]"},{"why":"Supplies the variance and concentration bounds that convert sampling budgets into the $(\\epsilon,\\delta)$-gameability guarantee.","marker":"[Maleki et al., 2014]"},{"why":"Grounds the claim that small-data regimes genuinely demand special modeling choices, making the small-coalition ambiguity defensible.","marker":"[Bornschein et al., 2020]"},{"why":"Motivates revenue-model and net-benefit performance metrics, from which the cost-ratio ambiguity set is built.","marker":"[Agarwal et al., 2019]"},{"why":"Motivates rank-based favorability and the contestability of absolute semivalue interpretation.","marker":"[Kwon and Zou, 2022]"}],"fun_headline_variants":["Semivalue valuations: arbitrary and gameable","Tiny utility tweaks can rig data valuations","Semivalue prices shift with arbitrary model choices","Gameable semivalues: utility ambiguity undermines data pricing","Arbitrary semivalue data valuation is gameable"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the utility functions inside each ambiguity set are all genuinely reasonable; if for a given task only one utility specification is actually correct, the valuations are no longer arbitrary and the gameability exploit disappears.","fun_headline_variants_meta":{"raw":{"variants":["Semivalue valuations: arbitrary and gameable","Tiny utility tweaks can rig data valuations","Semivalue prices shift with arbitrary model choices","Gameable semivalues: utility ambiguity undermines data pricing","Arbitrary semivalue data valuation is gameable"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000262,"raw_usage":{"total_tokens":1628,"prompt_tokens":1005,"completion_tokens":623,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":621,"completion_tokens_details":{"reasoning_tokens":547}},"tokens_in":621,"tokens_out":623,"duration_ms":8144,"temperature":1.0,"reasoning_tokens":547,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:46:24.604193+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a fixed benchmark where Data Shapley payouts are computed for two utilities that differ only in the small-coalition fallback threshold, say $0.05|D|$ and $0.1|D|$, and repeat across many datasets. If the median absolute payout change stays well below one average contributor and the bottom-10% filter memberships are identical, the arbitrariness claim for that ambiguity class would be contradicted.","supporting_citations":[],"review_version":1}