{"id":"bb0a748d-30d9-48d9-a25e-21034e25bf83","arxiv_id":"2607.03675","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"In-distribution validation noise directionally compresses KNN-Shapley values of training samples toward zero via neighborhood reshuffling, and a boundary-aware rescaling can partially restore baseline statistics.","lead":"Adding modest noise to a validation set systematically compresses training-data Shapley values toward zero under KNN-Shapley. The finding matters for data marketplaces and auditing pipelines that treat those values as stable once training data and model are fixed.","discovery_kind":"extension","skeptic_critique":{"model":"grok-4.5","headline":"The directional compression is tightly coupled to KNN neighbor ranking; the paper never shows the same shift under a non-neighbor utility, so the general-stability claim rests on an untested extrapolation.","rationale":"The reader already flags the KNN-only testbed and the clean-baseline dependence of the correction as the weakest assumption; that is precisely the load-bearing soft spot. The phenomenon is real and cleanly demonstrated inside the KNN setting, so the paper remains publishable with caveats—hence CONDITIONAL is still the right call. The concrete test above simply makes the missing transfer experiment explicit: if non-neighbor utilities exhibit the same directional compression, the claim strengthens; if they do not, the abstract and marketplace language must be narrowed. No stronger objection (internal inconsistency, numerical error, or fabricated result) appears in the manuscript.","tokens_in":9390,"tokens_out":558,"duration_ms":5415,"concrete_test":"Fix the same train/val splits used for CreditCard and Phoneme. Replace Soft-label KNN-Shapley with Monte-Carlo Data-Shapley (or Truncated Monte-Carlo) under logistic regression and a 2-layer MLP, injecting identical Gaussian feature noise into the validation set at the same σ levels. If the mean absolute Shapley value and positive-count statistics do not compress toward zero by a comparable relative amount, the neighborhood-reshuffling mechanism (and the general-stability claim) fails to transfer beyond KNN.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The strongest claim asserts that modest in-distribution validation noise produces consistent, directional compression of training-sample Shapley values via neighborhood reshuffling that flattens local validation-to-training rank order. Every empirical demonstration (Figs. 1–5, synthetic Gaussian quantiles, six real datasets) and the B/N correction (Appendix B, Eqs. 6–11) is obtained exclusively with Soft-label KNN-Shapley (K=5). For KNN the utility of a validation point is literally a function of the ordered neighbor list; any perturbation that reorders that list necessarily changes the marginal contributions, so the observed compression is almost tautological for this utility class. The paper never reports the analogous experiment for a non-neighbor model (logistic regression, small MLP, or even a fixed-kernel smoother whose utility is not defined by discrete ranks). Without that check it remains possible that the directional shift is an artifact of KNN geometry rather than a general property of Shapley-based data valuation. The marketplace-facing language in the abstract and conclusion therefore over-reaches the evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper claims that Shapley values for training data are not stable under fixed training data and model once the validation set is altered by modest in-distribution structural changes (especially additive feature noise). Using Soft-label KNN-Shapley (K=5), the authors show on a 2-D Gaussian-quantiles synthetic set and six real datasets that noise systematically compresses training-sample Shapley values toward zero, reduces their standard deviation, and lowers the count of positive values. They attribute the effect to a noise-induced neighborhood reshuffling that alters local validation-to-training rank order, with distinct roles for boundary versus non-boundary validation points (isolated via one- and two-point case studies). They propose a boundary/non-boundary (B/N) group-wise rescaling plus positive-ratio bias correction (Appendix B) that restores baseline spread and sign structure when a clean reference is available, and discuss entropy-aware normalization and boundary-aware validation design for marketplace robustness.","tokens_in":9715,"tokens_out":1254,"duration_ms":16717,"significance":"If the directional compression and its geometric mechanism hold beyond the KNN setting, the result identifies a previously under-appreciated axis of fragility in data valuation that is directly relevant to collaborative learning, federated settings, and data marketplaces where validation protocols are decentralized or noisy. The empirical consistency across synthetic and real data, the fine-grained boundary/non-boundary case analysis, and the explicit geometric story (neighbor reordering flattening marginal utilities) are genuine strengths. The work does not claim a new algorithm but a diagnostic phenomenon; that framing is useful provided the scope is accurately delimited. The B/N correction, while partly by construction, supplies a concrete, reproducible mitigation template that future work can stress-test.","major_comments":[{"comment":"Abstract, §1, and §7 frame the result as a general challenge to “the assumption of Shapley stability” and as revealing “a new axis of fragility in data valuation.” All experiments, mechanism analysis (Figs. 1–5), and the B/N correction (Appendix B, Eqs. 6–11) are obtained exclusively with Soft-label KNN-Shapley (K=5). For KNN the utility of a validation point is literally a function of the ordered neighbor list, so any perturbation that reorders that list necessarily changes marginal contributions; the observed compression is therefore almost tautological for this utility class. Without at least one non-neighbor utility (logistic regression, small MLP, or fixed-kernel smoother) the marketplace-facing generality claim over-reaches the evidence. Either restrict the claims to KNN-Shapley or add the missing experiments.","section":"Abstract, §1, §4–§5, §7"},{"comment":"The proposed correction (Eqs. 3–5 and Appendix B Eqs. 6–11) defines α_B, α_N as ratios of clean-baseline to noisy group standard deviations and chooses the bias b by quantile matching so that the positive fraction exactly recovers the clean baseline ρ_bl. Consequently the restored std and positive-count statistics are recovered largely by construction once a clean reference distribution is known. In the very settings the paper worries about (noisy or adversarially designed validation), that clean baseline is unavailable. The practical utility of the method therefore remains unclear; the manuscript should either supply a baseline-free alternative or explicitly state the strong assumption.","section":"§6, Appendix B"},{"comment":"Boundary membership is defined by an entropy-of-neighbor-labels rule on the K=5 nearest training neighbors (Appendix B), yet no numerical threshold, sensitivity analysis, or ablation on the entropy cutoff appears. Because the entire B/N decomposition and the subsequent group-wise rescaling rest on this binary mask, the load-bearing geometric story is only partially specified. A precise definition and a short sensitivity check are required for reproducibility.","section":"Appendix B, §5"}],"minor_comments":[{"comment":"Figure 1 caption and surrounding text speak of “the cost of data protection,” but the experimental noise is isotropic Gaussian feature noise; the link to privacy mechanisms is never made precise.","section":"§4, Fig. 1"},{"comment":"The abstract and conclusion mention “entropy-aware normalization” as a mitigation, yet the body only develops the B/N rescaling; either expand the former or remove the claim.","section":"Abstract, §7"},{"comment":"Table 1 and Appendix A list six real datasets; it would help readers if the main text briefly stated train/val sizes and the exact noise schedule (σ values) used for each.","section":"Appendix A, §6"},{"comment":"Typographical inconsistencies appear (“neighborhood reshuffling as the keymechanism”, “We offeractionsfor fair”, missing spaces after periods in several places).","section":"§1, §5"},{"comment":"References [13] and [11] are central to the KNN-Shapley lineage; a short comparison of how the present inflation/compression phenomenon relates to the inflation bias studied in [13] would strengthen the related-work discussion.","section":"§2"}],"recommendation":"major_revision","confidential_remarks":"The core empirical observation on KNN-Shapley is solid and the geometric case studies are carefully done; the main risk is over-claiming generality. If the authors are willing to re-scope the title/abstract/conclusion to “KNN-Shapley” or to add even a single non-KNN utility experiment, the paper becomes a clean, useful contribution. Otherwise the marketplace rhetoric will invite justified skepticism."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The thing worth knowing is that modest feature noise on the validation set systematically compresses Soft-label KNN-Shapley training values toward zero, and the paper shows this cleanly on a Gaussian synthetic setup plus six real datasets. The directional pattern is consistent, not random scatter, and the one- and two-validation-sample case studies (boundary vs non-boundary) give a usable geometric story: boundary points mainly shrink spread, non-boundary points mainly cut the positive count, both via neighbor reordering.\n\nWhat is new is the empirical regularity itself and the B/N decomposition that follows from it. Prior work already flags test-set mismatch, distributional valuation, and KNN-Shapley inflation (including the overlapping-author 2024 inflation paper), but the noise-driven compression and the boundary/non-boundary split are documented here with enough figures and tables to be useful. The correction (group-wise std matching plus positive-ratio quantile bias) is simple and visibly reduces the gap to the clean baseline on the reported plots. Citations look appropriate; math is elementary and transparent once you read Appendix B.\n\nThe soft spot is scope, not fabrication. Every demonstration and the correction itself live inside KNN utility with K=5. For KNN the utility is literally a function of the ordered neighbor list, so reordering under noise almost has to change marginals. The paper never runs the same protocol on a non-neighbor model (logistic regression, small MLP, fixed-kernel smoother). Without that check, the abstract/conclusion language about general Shapley stability and marketplace fragility over-reaches. The mitigation also assumes access to a clean baseline distribution of group moments and positive ratio, which is fine for diagnosis but less satisfying as a deployable fix. No code is shipped, which is a practical annoyance rather than a conceptual flaw.\n\nThis is for people who already work with data valuation, KNN-Shapley, or ML marketplaces and need a concrete robustness warning. It is not a foundational rewrite of Shapley theory. I would send it to peer review: the phenomenon is real enough and the evidence is sharp enough inside its stated testbed that referees should force the scope language to match the experiments rather than desk-reject it. Worth reading if you care about valuation stability; cite if you are writing on KNN-Shapley robustness; do not treat the general claim as settled until someone checks a non-neighbor utility.","headline":"Real, reproducible KNN-Shapley compression under validation noise; the general-stability claim is still an extrapolation.","tokens_in":10302,"tokens_out":584,"would_cite":true,"duration_ms":5160,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Even modest noise on the validation set compresses training-data Shapley values toward zero by reshuffling which training points count as neighbors.","keywords":["data valuation","Shapley value","KNN-Shapley","validation set","neighborhood reshuffling","boundary samples","data marketplaces","noise robustness"],"falsifier":"Run the same noise-injection protocol with a non-KNN utility (for example exact model-retraining Shapley on a small neural net or logistic regression) and check whether the same directional compression of training-point values and the same boundary/non-boundary split of effects still appear; if they do not, the claimed generality fails.","tokens_in":10261,"feed_emoji":"📊","tokens_out":669,"duration_ms":5254,"temperature":0.7,"pith_summary":"Shapley values for training data are usually treated as fixed once the training set and model are chosen. This paper shows that assumption fails under ordinary, in-distribution changes to the validation set. Adding feature noise to validation points causes the Shapley values of training points to shrink consistently toward zero. The mechanism is geometric: noise reorders the local neighbors between each validation point and the training set, which flattens the marginal-contribution landscape. Using the efficient KNN-Shapley setting, the authors demonstrate the same directional compression on synthetic Gaussians and six real datasets. They then separate validation points into boundary and non-boundary groups by neighbor-label entropy and apply a simple rescaling that restores spread and positive-count statistics. The result is a practical warning for anyone pricing or selecting data with Shapley values: the numbers are only as stable as the validation geometry that defines them.","feed_headline":"Validation noise compresses Shapley data values toward zero","feed_subtitle":"Neighbor reshuffling flattens credit for training points; a boundary-aware fix restores the baseline","key_machinery":"Noise-induced neighborhood reshuffling, observed inside Soft-label KNN-Shapley: perturbations change which training points are the K nearest neighbors of each validation point, which in turn changes every training point's average marginal contribution. Boundary versus non-boundary validation groups (defined by neighbor-label entropy) separate the two main statistical effects—lost spread and lost positive count—so each can be rescaled separately.","core_discovery":"Even modest in-distribution structural changes to the validation set, especially feature noise, produce consistent directional shifts in KNN-Shapley values of training samples, systematically compressing those values toward zero. The driver is a noise-induced neighborhood reshuffling effect that alters the local rank order between validation and training samples and thereby flattens the valuation landscape.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Validation noise compresses KNN-Shapley values toward zero","Modest validation noise flattens training data Shapley credits","Noise reshuffles neighbors and drives Shapley values to zero","Validation structure distorts Shapley data valuations systematically","Boundary noise flattens the Shapley landscape for training points"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The paper treats the KNN-Shapley geometry with a fixed small neighborhood size and an entropy-based boundary mask as representative enough of Shapley-based data valuation in general that the observed compression and the proposed correction carry over beyond this testbed.","fun_headline_variants_meta":{"raw":{"variants":["Validation noise compresses KNN-Shapley values toward zero","Modest validation noise flattens training data Shapley credits","Noise reshuffles neighbors and drives Shapley values to zero","Validation structure distorts Shapley data valuations systematically","Boundary noise flattens the Shapley landscape for training points"]},"model":"grok-4.5","effort":"low","cost_usd":0.00475,"raw_usage":{"total_tokens":1327,"prompt_tokens":707,"num_sources_used":0,"completion_tokens":84,"cost_in_usd_ticks":47500000,"prompt_tokens_details":{"text_tokens":707,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":536,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":707,"tokens_out":84,"duration_ms":5154,"temperature":1.0,"reasoning_tokens":536,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-12T00:45:29.844459+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Run the same noise-injection protocol with a non-KNN utility (for example exact model-retraining Shapley on a small neural net or logistic regression) and check whether the same directional compression of training-point values and the same boundary/non-boundary split of effects still appear; if they do not, the claimed generality fails.","supporting_citations":[],"review_version":1}