{"id":"24899010-3a45-4326-8a39-51df705fb9e6","arxiv_id":"2506.05379","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"Proposes Q-MIA and Mixed-MIA auctions to truthfully procure LLM training data with quality-based rewards, yet key budget guarantees are unproven and no experiments are reported.","lead":"This paper designs auctions and token contracts to pay data providers for LLM training data, aiming to make honest reporting a dominant strategy. It claims the mechanisms respect budgets and reward data quality, but the budget-balance proof is invalid and the claimed empirical results are missing.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Q-MIA's budget-feasibility Lemma 3 is false: with B=10 and costs (1,1,9), the critical payments are 9+9=18>B, so the central DSIC-plus-budget claim fails even under the paper's own verifiability assumptions.","rationale":"The reader's strongest claim identifies the budget constraint as load-bearing, and the reader's rationale already notes that 'budget-feasibility proof is a hand-wave that likely fails on a simple counterexample.' My check confirms this and makes it concrete: the counterexample with costs (1,1,9) and B=10 shows total critical payments of 18>B. This is stronger than the reader's stated weakest assumption about manipulability of q_i and phi_i, because the budget failure occurs even when q_i and phi_i are perfectly verifiable and fixed. The q/phi manipulability concern is real and would further undermine DSIC, but the budget counterexample is sufficient on its own to refute the paper's central claim. I therefore agree with the rejection verdict, though I would base it primarily on Lemma 3's failure rather than on the data-disclosure manipulation channel. The absence of an experiment section is secondary; the theoretical mechanism is already internally inconsistent.","tokens_in":12949,"tokens_out":7588,"duration_ms":91825,"concrete_test":"Implement Algorithm 1 exactly as written, including the critical-payment rule 'smallest c~ s.t. i would still be selected,' with B=10, n=3, q_i=phi_i=1 for all i, and reported costs (1,1,9). Print the selected set and payments. If the outputs are S*={1,2} and p=(9,9), then total payment 18>10 and Lemma 3 is falsified. A correct repair would have to change either the selection rule or the payment rule (e.g., cap payments at reported cost), but capping at reported cost breaks DSIC, so the paper must provide a new argument for budget feasibility.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—Lemma 3 and Theorem 4—is that Q-MIA and Mixed-MIA satisfy the global budget constraint while remaining DSIC. This fails even granting the paper's assumption that q_i and phi_i are verifiable and non-manipulable. Lemma 3's proof is not a proof: it states that S* is chosen subject to sum_{i in S*} c_i <= B, then asserts that critical payments 'remain within the selected budget' by appeal to unspecified 'standard results.' But the payment rule of Section 4.1/Appendix B.1 pays each selected agent its critical cost, not its reported cost, and critical costs can sum far above B.\n\nConcrete counterexample: take B=10, n=3, q_i=phi_i=1 (so virtual cost equals reported cost), and reported costs c=(1,1,9). The greedy loop selects agents 1 and 2 and stops because 1+1+9=11>10. For a selected cheap agent, the smallest report c~ that keeps it selected is 9: at c~=9 the run has cumulative cost 1+9=10, so it is selected; at c~>9 it is ordered after the cost-9 agent, and 1+9+c~ exceeds 10, so it drops out. Hence payments are p_1=p_2=9, total 18>B. This directly falsifies Lemma 3. For Mixed-MIA, Theorem 4's proof relies on 'sum p_i^Q <= B,' which is exactly the false claim; for any rho>0 with U=0, the budget violation persists. This is an internal inconsistency in the mechanism as defined, not a disagreement about modeling assumptions.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a mechanism-design framework for procuring LLM training data from strategic institutional providers, introducing three mechanisms: Q-MIA (quality-weighted greedy auction with Myerson-style critical payments), MUT (future-utility token shares), and Mixed-MIA (a convex combination of the two). The central advertised guarantees are dominant-strategy incentive compatibility (DSIC), individual rationality, and global budget feasibility, with rewards sensitive to quality and estimated marginal utility. The main theorems are Lemma 3, which asserts that Q-MIA payments never exceed the budget B, and Theorem 4, which extends this bound to Mixed-MIA. The paper also contributes a Data Share Token (DST) scoring formula, auditable quality oracles, and deployment considerations.","tokens_in":13318,"tokens_out":5968,"duration_ms":66378,"significance":"If the stated results were correct, the framework would give a practical, budget-constrained procurement mechanism for LLM training data with strong incentive guarantees. The paper has several genuine strengths: a clear formal model of the procurement problem, explicit pseudocode in Appendix B, a useful decomposition of data quality into auditable submetrics, and an honest limitations section that acknowledges estimator noise and dynamic-agent issues. However, the central budget-feasibility claim is false as stated, and the DSIC proofs rest on an unproven assumption that quality and marginal-utility signals are non-manipulable. These are load-bearing defects, not presentation issues. The theoretical contribution therefore does not currently stand.","major_comments":[{"comment":"Lemma 3 is false under the mechanism as defined. In Q-MIA, Step 5 pays each selected agent its critical cost, while Step 4 stops when cumulative *reported* costs reach B. These are different objects. Consider B=10, n=3, q_i=phi_i=1 for all i, and reported costs c=(1,1,9). The greedy loop selects agents 1 and 2 and stops because 1+1+9=11>10. For each selected cheap agent, the smallest report that keeps it selected is 9: reporting 9 gives a sorted order (1,9,9), cumulative costs 1+10=10, so the agent is selected; reporting anything above 9 places it after the cost-9 agent, and the cumulative cost exceeds 10, so it drops out. Hence p_1=p_2=9 and the total payment is 18>B. The proof's appeal to 'standard results on the budget-feasibility of VCG-type payments with capacity constraints' is inapplicable: the mechanism is not VCG, and critical payments in a greedy cost auction are not bounded by the cumulative reported costs used in the stopping rule.","section":"Section 5.3, Lemma 3; Appendix B.1"},{"comment":"The DSIC proof covers only cost misreporting and assumes that q_i and phi_i are 'verifiable and non-manipulable by agent i.' This assumption is not proven and conflicts with the paper's own stated desideratum of truthful full-dataset disclosure (Section 1.2) and its explicit list of strategic actions including 'manipulate metadata to boost perceived quality' (Section 3.4). If q_i and phi_i are computed from the submitted data, an agent can select, prune, or augment its dataset to influence these scores, and the proof gives no argument that such actions are dominated by truthful submission. Thus the paper's central DSIC claim covers only one dimension of the agents' strategy space, leaving the actual data-disclosure decision unincentivized.","section":"Section 5.1, Lemma 1; Section 3.4; Section 1.2"},{"comment":"Theorem 4's budget proof relies directly on the assertion sum p_i^Q <= B, which is exactly the false claim from Lemma 3; for any rho>0 with U=0, the counterexample above gives a Mixed-MIA payment of rho*18>B, so the advertised bound fails. The DSIC proofs for MUT (Lemma 5) and Mixed-MIA (Theorem 2) also inherit the non-manipulability gap. Moreover, Lemma 5's statement that 'misreporting or withholding data reduces phi_i' is an assertion, not a proof: a provider could in principle submit a smaller but more carefully curated dataset that raises quality and estimated marginal utility. The proof does not model the dependence of q_i and phi_i on the dataset actually submitted.","section":"Section 8.7, Theorem 4; Section 6.2, Lemma 5"}],"minor_comments":[{"comment":"The abstract claims 'theoretically and empirically' outperformance over baselines, but the manuscript contains no experiments or empirical evaluation section; either add the experiments or remove the empirical claim.","section":"Abstract and body"},{"comment":"The sentence 'Theoretical bounds on monotonicity and DSIC compatibility under psi^(eta,kappa)_i can be proven under mild regularity conditions on the estimator and cost model' is an unsupported promise of an omitted proof; either provide the proof or remove the claim.","section":"Section 4.2"},{"comment":"The related-work paragraph contains an incomplete citation: 'This conceptual foundation is extended by 2017, which shifts...' has no author or reference entry; please fix.","section":"Section 2.1"},{"comment":"Weak budget balance is defined differently in the two sections: Section 1.2 says payments must not exceed the value generated, while Section 3.2 defines budget feasibility as sum p_i <= B for a fixed budget B; these should be reconciled.","section":"Sections 1.2 and 3.2"},{"comment":"The proof of Pareto optimality is a single assertion that any improving move 'would violate feasibility'; no feasible-alternative set is defined, and linearity in rho alone does not imply Pareto optimality. This theorem needs a real proof or should be rephrased.","section":"Section 8.4, Theorem 1"}],"recommendation":"reject","confidential_remarks":"The central obstacle is a concrete falsifying counterexample to Lemma 3, not a disagreement over modeling assumptions or a matter of presentation. The mechanism as defined can pay twice its budget even in the simplest symmetric case with verifiable quality. Fixing this would require redesigning the payment rule and re-proving DSIC, IR, and budget feasibility, which is beyond a minor revision. The DSIC claim is also narrower than advertised because it assumes away manipulation of the quality and utility signals. I recommend rejection, though the authors' broader idea of combining quality-weighted virtual costs with critical payments is worth pursuing in a corrected form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's core contribution is a concrete, well-motivated package: a Myerson-style auction (Q-MIA) that ranks providers by quality-weighted virtual costs, a tokenized profit-sharing contract (MUT), and a hybrid (Mixed-MIA). The problem it targets — institutional data holders who are strategically reluctant to share with LLM trainers — is real, and the design goals (DSIC, IR, weak budget balance, quality sensitivity) are the right ones. The authors also ship real implementation detail: auditable quality oracles, influence-function-based marginal utility estimates, and a DST scoring metric with explicit weights. That is more than a sketch, and it deserves credit.\n\nThe soft spots are not minor. The load-bearing claim, Lemma 3, is that Q-MIA satisfies the global budget constraint Sum p_i <= B. The proof is a hand-wave citing 'standard results on budget-feasibility of VCG-type payments with capacity constraints,' and the claim is false as stated. Take B=10, three agents with q_i=phi_i=1 and costs (1,1,9). The greedy selection picks agents 1 and 2, stopping before the cost-9 agent. Each selected agent's critical payment is 9 (the smallest report that keeps them selected), so total payments are 18 > B. This directly falsifies Lemma 3 and, through Theorem 4, breaks Mixed-MIA's budget guarantee for any rho>0 when U=0. The DSIC proofs also assume q_i and phi_i are verifiable and non-manipulable, which removes the most strategically important choice — what data to submit — from the incentive analysis. The paper never models or proves truthful data disclosure.\n\nOther problems: Theorem 1's Pareto optimality is asserted without a real proof; the QWMP payment formula in Section 5.4 (p_i = gamma * q_i^eta * (phi_i + kappa)^gamma) is disconnected from the critical-payment rule used elsewhere; and the claimed empirical superiority over volume-based and trust-based baselines is entirely unsubstantiated — there is no experiment section. The limitations section is honest about estimator noise and one-shot assumptions, but it does not flag the budget-feasibility failure.\n\nThe stress-test counterexample is correct against the paper as written. This is an internal inconsistency in the mechanism, not a modeling disagreement.\n\nWho is this for? Researchers working on data procurement and auction design will find the framing useful, and the DST quality metric details are worth borrowing. But the central theoretical guarantees do not hold, and the paper overclaims. A serious referee should see it — the topic is timely and the flaws are fixable in principle — but the current version should not be accepted. My recommendation: send to peer review with the expectation of substantial revision, and make the budget-feasibility proof (or a corrected payment rule) the first condition.","headline":"A promising mechanism-design framing for LLM data procurement, but the central budget-feasibility and DSIC claims are not established — the budget lemma is false on a simple counterexample.","tokens_in":13906,"tokens_out":700,"would_cite":true,"duration_ms":9745,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"If data quality and marginal impact are verifiable, a quality-weighted auction can procure LLM training data truthfully while keeping total payments under budget.","keywords":["Dominant Strategy Incentive Compatibility","Strategic Data Contribution","Marginal Utility Estimation","Large Language Models Data","Tokenized Incentive Design","Trust-Minimized Mechanisms","Game-Theoretic Data Markets","Federated Data Valuation"],"falsifier":"Simulate Q-MIA with random providers, run greedy selection on reported costs, then compute each winner's critical payment; if the sum of payments exceeds the budget $B$ in a non-negligible fraction of runs, the budget-feasibility lemma fails. (The paper's proof of Lemma 3 asserts this step rather than demonstrating it.)","tokens_in":12702,"feed_emoji":"💰","tokens_out":5527,"duration_ms":53846,"temperature":0.7,"pith_summary":"The paper tries to show that a planner who needs high-quality data to train an LLM can procure it from self-interested institutions without trusting their cost reports or exceeding a fixed budget. It proposes auctions in which providers are ranked by a virtual cost—reported cost divided by verifiable quality and estimated marginal contribution—and are paid a critical value that makes truthful cost reporting a dominant strategy. The same design extends to a hybrid scheme, Mixed-MIA, that pays partly in money and partly in future utility tokens, preserving the incentive and budget guarantees. If these guarantees hold, data procurement for LLMs could be run without external subsidies, with rewards that track data quality and learning value.","feed_headline":"Auction makes LLM data buying truthful and budget-capped","feed_subtitle":"Pay providers by verifiable quality and marginal value; total spend stays under one hard cap.","key_machinery":"The virtual cost score $\\psi_i = \\hat{c}_i / (q_i^{\\eta}(\\hat{\\varphi}_i+\\kappa)^\\gamma)$ ranks providers by how much value they deliver per dollar, and the critical-value payment rule $p_i = \\inf\\{\\tilde{c}_i : i \\in S^*(\\tilde{c}_i, \\hat{c}_{-i})\\}$ makes truthful cost reporting a dominant strategy. The liquidity factor $\\rho \\in [0,1]$ in Mixed-MIA linearly blends a monetary critical payment with a normalized utility share $s_i = q_i \\hat{\\varphi}_i / \\sum_j q_j \\hat{\\varphi}_j$, which is the object that carries the MUT and DST extensions.","core_discovery":"The paper's central claim is that Q-MIA—a quality-weighted marginal-incentive auction—is dominant-strategy incentive-compatible, individually rational, and weakly budget-balanced. Providers are sorted by virtual cost $\\psi_i = \\hat{c}_i / (q_i \\hat{\\varphi}_i)$ (or a regularized variant), selected greedily until the reported-cost budget is exhausted, and paid Myerson-style critical values. The authors further claim that the Marginal Utility Token contract and the Mixed-MIA hybrid, which interpolates between cash and deferred token rewards via a liquidity factor $\\rho$, keep these properties. The anchor of the argument is that quality $q_i$ and marginal utility $\\hat{\\varphi}_i$ are verifiable and non-manipulable, so cost is the only private message.","pith_inferences":["The verifiability assumption is the load-bearing premise: if providers can influence $q_i$ or $\\hat{\\varphi}_i$ by choosing what data to submit, the DSIC guarantee covers only the cost report, not the data-submission decision.","The budget-feasibility lemma leans on an unproved step—that the sum of Myerson critical payments stays within the budget when selection is made on reported costs—so the practical claim is only as strong as that step; a stochastic simulation is the fastest check.","MUT-style tokens implicitly require agents to believe in the future success of the model, so in low-liquidity settings the effective outside option may not match the assumed utility pool $U$.","Testing how small the regularization $\\kappa$ can become before DSIC degrades under noisy marginal-utility estimates would be a natural follow-up, since the paper concedes those estimates are noisy for near-zero-impact providers."],"forward_implications":["A model developer can run a procurement auction that caps total spend at a hard budget $B$ with no external subsidies, assuming quality and marginal-contribution oracles are trustworthy.","Providers' dominant strategy is to report true costs, and reward proportions favor high-quality, high-leverage data over sheer volume.","The liquidity parameter $\\rho$ lets the planner shift compensation from immediate cash to future tokens without losing DSIC or IR, per the paper's theorems.","The Data Share Token metric offers a concrete token-allocation rule with diversity and anti-concentration protections."],"supporting_citations":[{"why":"Supplies the foundation for modeling agents that value privacy as a cost, which motivates the private-cost model in this framework.","marker":"[6]"},{"why":"Provides the template for optimal data acquisition with DSIC, individual rationality, and ex-post budget feasibility, which Q-MIA extends to quality- and utility-weighted settings.","marker":"[7]"},{"why":"Exposes the incompatibility of Shapley value with incentive-compatible design and motivates the marginal-utility-based reward structure used here.","marker":"[8]"}],"fun_headline_variants":["Truthful data auction keeps LLM budgets capped","Quality-weighted auction makes data sharing honest","Q-MIA: verifiable data, capped spend, no trust","Hybrid token auction for fair LLM data markets","Auctions solve LLM data trust and cost limits"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire construction assumes the mechanism can compute a provider's data quality and marginal contribution accurately and independently, so a provider cannot change them by what it submits.","fun_headline_variants_meta":{"raw":{"variants":["Truthful data auction keeps LLM budgets capped","Quality-weighted auction makes data sharing honest","Q-MIA: verifiable data, capped spend, no trust","Hybrid token auction for fair LLM data markets","Auctions solve LLM data trust and cost limits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000149,"raw_usage":{"total_tokens":1193,"prompt_tokens":947,"completion_tokens":246,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":563,"completion_tokens_details":{"reasoning_tokens":185}},"tokens_in":563,"tokens_out":246,"duration_ms":3512,"temperature":1.0,"reasoning_tokens":185,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:50:12.196827+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Simulate Q-MIA with random providers, run greedy selection on reported costs, then compute each winner's critical payment; if the sum of payments exceeds the budget $B$ in a non-negligible fraction of runs, the budget-feasibility lemma fails. (The paper's proof of Lemma 3 asserts this step rather than demonstrating it.)","supporting_citations":[{"cited_title":"In: ACM Conference on Economics and Computation (2015)","cited_arxiv_id":null,"evidence_quote":"Supplies the foundation for modeling agents that value privacy as a cost, which motivates the private-cost model in this framework."},{"cited_title":"In: ACM Conference on Economics and Computation (2017)","cited_arxiv_id":null,"evidence_quote":"Provides the template for optimal data acquisition with DSIC, individual rationality, and ex-post budget feasibility, which Q-MIA extends to quality- and utility-weighted settings."},{"cited_title":"ArXiv preprint (2025)","cited_arxiv_id":null,"evidence_quote":"Exposes the incompatibility of Shapley value with incentive-compatible design and motivates the marginal-utility-based reward structure used here."}],"review_version":1}