{"id":"019b2a61-237f-40c5-9a09-5d18ddcb2913","arxiv_id":"2411.11713","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A two-stage auction-based mechanism prices and selects FL clients before training using privacy-preserving aggregated data statistics, claiming >10% accuracy gains over prior client selection.","lead":"FLMarket is a system that prices and selects clients in federated learning before any training happens, using an auction and a privacy-preserving way to share data statistics. It claims to beat existing client-selection methods by over 10% accuracy and to outperform a training-time baseline by 2% while running 3x faster.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Score-function tuning on the same CIFAR-10 benchmark used for evaluation, with no error bars, leaves the headline >10% accuracy gain over pre-training baselines unsubstantiated; the gain over the best baseline (DICE) averages only ~4% across datasets.","rationale":"The reader's weakest_assumption is client dropout in PASS, but that failure mode is secondary to the central claim: the paper's headline is an accuracy improvement, and the evidence for that improvement is the empirical evaluation. The evaluation has two specific validity threats: (i) the score function is tuned on CIFAR-10, which is then used as an evaluation benchmark, and (ii) no variance information is reported for any of the 57 test cases. The paper's own reported averages versus the best baseline, DICE, are 7.08%, 1.81%, and 3.53%, yielding roughly 4% average gain, far below the 10% headline. Because the largest margins are on CIFAR-10, the headline could be an artifact of tuning. The PASS dropout issue is real but would only affect deployment reliability if clients fail; it does not compromise the accuracy comparison when the protocol runs to completion. I therefore focus the test on the evaluation loop. Agreement with the reader: disagree, because the reader's weakest_assumption focuses on dropout rather than the evaluation bias.","tokens_in":27972,"tokens_out":9155,"duration_ms":87410,"concrete_test":"Compute the average accuracy improvement of FLMarket over the four pre-training baselines separately for each dataset, then recompute the overall average excluding CIFAR-10. Also rerun the CIFAR-10 experiments across the six distributions with the alternative score functions f1, f2, f3 and constant theta from Appendix D, using at least five random seeds. If the overall average drops below 10% when CIFAR-10 is excluded, or if the CIFAR-10 advantage over DICE is not consistently above zero across seeds and f choices, the headline performance claim is not robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central accuracy claim rests on a 57-case comparison whose strongest results are obtained on CIFAR-10, the same dataset used in Appendix D to select the functional form f(x) = -ln(x) and the class-dependent coefficients theta_c. This creates a selection-on-test-data loop: the f and theta choices are not fixed before the evaluation, so CIFAR-10 accuracy numbers are optimistically biased. No error bars or multiple seeds are reported for any configuration, so we cannot separate the 10.18% average improvement from stochastic variation. The paper reports that against the strongest baseline, DICE, the average improvements are 7.08%, 1.81%, and 3.53% on the three datasets, i.e., about 4% on average, well below the headline 10%. Since CINIC-10 and DEAP improvements over DICE are small, the 10% claim may be dominated by the tuned CIFAR-10 results. This is a load-bearing weakness in the evidence for the main performance claim.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FLMarket, a framework for pre-training data pricing in federated learning. It combines a two-stage auction mechanism (winner selection and payment determination) with a privacy-preserving protocol (PASS) that aggregates clients' class distributions without revealing them. The score function values each client by a combination of data volume and global class scarcity, using a diminishing-return function. The authors prove truthfulness, individual rationality, and budget feasibility of the auction, and evaluate client selection by training on CIFAR-10, CINIC-10, and DEAP, reporting accuracy and runtime comparisons against four pre-training baselines and one in-training baseline.","tokens_in":28208,"tokens_out":9188,"duration_ms":87554,"significance":"The paper addresses a genuinely open problem—pricing data before training in FL—and the proposed mechanism is coherent: the auction is of independent interest, the score function is interpretable, and the evaluation covers a large configuration space (three datasets, multiple selection ratios and distributions). The formal statements in Section 3.3 are plausible, the PASS running example in Appendix F is helpful, and the appendix contains substantial supporting material, including a survey and a proof appendix. However, the headline performance claim is not currently supported by the evidence: the score function and its coefficients are tuned on CIFAR-10, which is also part of the main evaluation, and no result is accompanied by error bars or multiple seeds. The reported advantage over the strongest baseline (DICE) averages only about 4% across datasets, not the 10% stated in the abstract.","major_comments":[{"comment":"The functional form f(x)=-ln(x) and the class-dependent coefficients theta_c are selected on CIFAR-10 by comparing four candidate functions on the same dataset (Figure 10), and CIFAR-10 is then part of the headline evaluation. This is a selection-on-test-data loop: the CIFAR-10 results in Figure 3 and the reported 10.18% average improvement are optimistically biased. The reported improvements over the strongest baseline DICE are 7.08% on CIFAR-10 but only 1.81% on CINIC-10 and 3.53% on DEAP, so the external evidence for the headline claim is much weaker. The authors should fix f and theta using a validation split or a separate dataset, then evaluate on held-out data, or at least present the comparison with f and theta chosen without using CIFAR-10.","section":"Appendix D and Section 5.2"},{"comment":"No result is accompanied by error bars, standard deviations, or the number of seeds. Figures 3-7 and the text report single runs for each configuration. Given the high variance typical of federated training with Dirichlet-distributed non-IID data, the claimed 10.18% average improvement and the smaller differences in Appendix K cannot be distinguished from stochastic variation. Please report means and standard deviations over at least three independent runs, and where possible a paired significance test.","section":"Section 5"},{"comment":"The PASS protocol's reconstruction of the global distribution via Eq. (19) relies on receiving a masked distribution Y_e from every enrolled client. If any client drops out after the key-agreement step, the residual pairwise masks do not cancel and the server cannot compute N_s; the protocol has no dropout handling. The assumptions in Section 3.1 state that participants follow the protocol, but dropout is a separate reliability issue that is common in cross-silo FL. The authors should either extend PASS with dropout resilience (e.g., via secret sharing of the masks) or explicitly discuss this limitation in the main text; the current Discussion section only covers malicious clients and free-riding.","section":"Section 4.2 and Section 3.1"},{"comment":"The payment-determination loop (lines 15-24) is not well-defined when every client in V^{-e} satisfies the budget constraint. If the for loop completes without executing break, the variable j is incremented past the last index of V^{-e} (from j=E-1 to j=E), and line 23 reads b_j and u_j out of bounds. In this case, no critical value is computed. The proofs of Lemma 3.3 and Theorem 3.7 also assume the existence of an index \\hat{k} at which the budget constraint is first violated; for a sufficiently large budget R, such an index does not exist. The authors should handle this boundary case (e.g., by defining the critical payment using the budget constraint alone when all other clients pass) and update the proofs and worked example accordingly.","section":"Algorithm 1 (Section 3.3)"}],"minor_comments":[{"comment":"The abstract claims 'outperforms the in-training baseline with more than 2% accuracy increase', but on CIFAR-10 FLMarket's average accuracy (66.12%) is lower than S-FedAvg's (66.67%); the reported 2.1% average is only an average across datasets. Please qualify the claim.","section":"Section 5.3 and Abstract"},{"comment":"The phrase 'reminding the global data distribution' should be 'yielding the global data distribution'.","section":"Section 4, Eq. (19)"},{"comment":"'Sharply value' should be 'Shapley value'.","section":"Appendix I"},{"comment":"The axis labels in Figure 2a are garbled ('Accuracy Improvement in accuracy Number of Data Accuracy'); please clean up the figure.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable fit for the applied ML track, but the evaluation needs to be strengthened before acceptance. The auction proofs would also benefit from a careful re-examination of the boundary cases in Algorithm 1. The related-work comparison in Table 2 is self-reported and some entries may be debatable, but this is not a blocking issue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take on FLMarket. It's a clean integration of a budget-feasible auction (adapted from Zheng et al.) with a privacy-preserving distribution aggregation protocol (Diffie-Hellman key agreement plus pairwise masks) for pre-training data pricing in FL. That combination is genuinely new relative to the cited literature, and the auction proofs for truthfulness, individual rationality, and budget feasibility are plausible. The evaluation is broad: three datasets, 20 and 100 clients, 57 configurations, and the method consistently beats DICE, the strongest baseline, on most distributions. This is a real contribution to FL incentive design.\n\nBut the headline is overstated. The 'more than 10% higher accuracy' claim is an average over four baselines, two of which are weak. Against DICE, the average gains are 7.08% on CIFAR-10, 1.81% on CINIC-10, and 3.53% on DEAP — about 4% overall. Still positive, but not 10%, and the headline leans heavily on the easy comparisons.\n\nThe bigger problem is the CIFAR-10 tuning loop. The functional form f(x) = -ln(x) and the class coefficients are selected in Appendix D using CIFAR-10, then CIFAR-10 appears in the main evaluation. So the strongest results carry an optimistic bias. The CINIC-10 and DEAP results are the honest ones, and they are smaller. Also, there are no error bars or multiple seeds anywhere, so we can't separate the method's edge from stochastic variation.\n\nOne practical gap: PASS assumes every client completes the protocol. If one client drops after key agreement, the pairwise masks don't cancel and the server can't reconstruct the global distribution. The authors assume semi-honest clients but don't handle dropout, which matters in cross-silo FL. The auction itself is also not empirically stress-tested: the experiments select fixed numbers of clients but don't specify how bids and the budget are instantiated, so the evaluation mostly tests the ranking quality of the score function, not the auction's strategic properties.\n\nBottom line: solid mechanism design and a promising integration, but the evidence for the central accuracy claim is weaker than advertised. With a revised headline, error bars, held-out tuning, and a dropout-resilient PASS, this would be a good paper. As is, it deserves a serious referee, not a desk reject.","headline":"A genuine integration of auction pricing with privacy-preserving distribution aggregation for FL, but the headline accuracy gain is inflated by a CIFAR-10 tuning loop and missing error bars.","tokens_in":28711,"tokens_out":3915,"would_cite":true,"duration_ms":34386,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FLMarket prices federated clients before training via a truthful auction over secret-shared statistics, and claims over 10% higher downstream accuracy than prior pre-training selection.","keywords":["federated learning","data pricing","pre-training pricing","client selection","auction mechanism","privacy-preserving aggregation","secret sharing","budget-feasible mechanism"],"falsifier":"Make one client abort after Diffie-Hellman keys are exchanged but before it sends its masked distribution Y_e; Equation (19) can no longer cancel the pairwise PRG noises, so the server cannot recover the global distribution N_s and the pre-training price cannot be computed, directly testing the all-clients-complete premise.","tokens_in":27784,"feed_emoji":"💰","tokens_out":9371,"duration_ms":80517,"temperature":0.7,"pith_summary":"FLMarket addresses a gap in federated learning: pricing clients' data before training starts, when no model feedback exists. The paper claims that a two-stage auction—first scoring each client by data volume and class scarcity, then selecting winners under a budget with critical-value payments—picks clients whose data produces over 10% higher downstream accuracy than state-of-the-art pre-training selection, and over 2% higher accuracy with about 3x less runtime than an in-training selector that re-evaluates clients each round. To avoid leaking the class distributions that feed the scores, FLMarket adds a pairwise noise-masking protocol (PASS) whose random terms cancel when all masked distributions are summed. A sympathetic reader would care because pre-training pricing gives clients a credible reward before they commit resources, which the paper's own survey suggests is a practical obstacle to participation in FL data markets.","feed_headline":"Pre-training auction picks better FL clients: +10% accuracy","feed_subtitle":"Secret-shared statistics let buyers pay before training, beating an in-training selector with 3x less runtime.","key_machinery":"The carrying object is the two-stage, budget-constrained pricing mechanism coupled with the PASS privacy protocol. In stage one, the score function $u_e = \\sum_c \\theta_c \\phi(n^c_e)$ with $\\theta_c = 1 - n^s_c/N_s$ and $\\phi(x) = \\sum_{t=1}^{x} -\\ln(\\min(t/\\alpha, 1))$ converts each client's class-count vector into a single value that rewards data volume with diminishing returns and rewards globally scarce classes. In stage two, Algorithm 1 sorts clients by score per bid, admits winners under the budget condition $b_e \\le \\frac{R}{2}\\cdot\\frac{u_e}{U(S_k\\cup\\{e\\})}$, and pays each winner the critical value at which it would lose the auction, which makes the mechanism truthful, individually rational, and budget-feasible. PASS supplies the privacy layer: each client adds signed pseudorandom outputs derived from pairwise Diffie-Hellman seeds to its local distribution, and the sum of all masked distributions cancels the noise to recover $N_s$ without revealing any individual $N_e$.","core_discovery":"This paper claims that data for federated learning can be priced before training by viewing each client's class distribution through the score function $u_e = \\sum_c \\theta_c \\phi(n^c_e)$, where $\\phi$ is a diminishing-returns curve and $\\theta_c$ rewards classes that are globally scarce. The server never sees raw local distributions; clients mask them with pairwise Diffie-Hellman-derived random vectors, and the server sums the masks to recover only the global distribution $N_s$. A budget-feasible auction then ranks clients by score per bid, selects winners until the budget binds, and pays each winner its critical value, giving truthful, individually rational, budget-feasible prices. The paper asserts that this selection transfers to FL accuracy: across 57 test configurations on CIFAR-10, CINIC-10, and DEAP, FLMarket records the highest accuracy in 44 distributions and beats the four pre-training baselines by an average of 10.18% accuracy, while matching or exceeding the in-training baseline S-FedAvg with a 3.16x per-round speedup.","pith_inferences":["The scarcity-weighting premise could be tested outside the paper's datasets: on any benchmark where scarce classes carry noisy or mislabeled examples, $\\theta_c$ would inflate scores for clients the training would be better off ignoring, so validation-based calibration of $\\theta_c$ would be a natural extension.","PASS has no dropout path as written; in cross-device FL where clients go offline, the protocol would need threshold secret sharing or server-side handling of missing masks before it can be deployed unchanged.","The reported 10% figure is a property of client selection, not of the training algorithm; because the paper notes FLMarket can be stacked with in-training methods, the gains should be read as the value of a better starting client set."],"forward_implications":["Under the paper's evaluation, FLMarket's pre-training selection wins 44 of 57 tested configurations and raises average accuracy by 10.18% over the four pre-training baselines, with the largest gains on the most imbalanced distributions.","Because scoring happens once before training, FLMarket adds no per-round selection overhead; the paper reports reaching 0.6 accuracy on CIFAR-10 in about 130 rounds versus 170-550 rounds for the baselines.","Against the in-training S-FedAvg baseline, FLMarket achieves comparable or better final accuracy on most tested distributions with a 3.16x average per-round runtime speedup.","The payment equals the critical value, so no client can profit by bidding above its true cost, and total payments stay within the buyer's budget R.","When the client pool is scaled to 100 clients, FLMarket still shows positive average accuracy gains over the baselines, though the margin shrinks relative to the 20-client setting."],"supporting_citations":[{"why":"Supplies the data-based client evaluation baseline AUCTION that FLMarket extends; AUCTION values clients by data mislabel rate and size.","marker":"[9]"},{"why":"Provides the O(d log(2m)/m) generalization bound used to justify the logarithmic form of the score function phi.","marker":"[2]"},{"why":"Empirical scaling laws for neural language models are cited to support diminishing returns of data volume in the score function.","marker":"[17]"},{"why":"DICE is the strongest pre-training baseline and defines data quality scores from volume ratio and per-class standard deviation; FLMarket must beat it.","marker":"[39]"},{"why":"DDS is the diversity-driven pre-training selection baseline used in the comparison.","marker":"[21]"},{"why":"Supplies the budget-feasible mechanism truthfulness characterization (monotone selection plus critical-value payment) used in Theorem 3.1.","marker":"[41]"},{"why":"Contributes the budget constraint condition and critical-payment style used by Algorithm 1 for winner selection and payment determination.","marker":"[63]"},{"why":"Diffie-Hellman key agreement is the primitive that lets each client pair share a random seed for PASS noise masks.","marker":"[14]"},{"why":"S-FedAvg is the in-training Shapley-based client selection baseline against which FLMarket compares accuracy and runtime.","marker":"[33]"}],"fun_headline_variants":["Auction before FL training: +10% accuracy, 3x faster","Secret-shared stats price FL data pre-training","Pre-training auction picks winners: +10% accuracy","Masked distribution auction: better FL clients, 3x speedup","Data priced before training: FL auction with privacy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole pricing pipeline assumes every enrolled client completes the PASS masking step and submits its masked distribution; if any client drops out after key agreement, the pairwise noises do not cancel and the server cannot recover the global class distribution needed to set prices.","fun_headline_variants_meta":{"raw":{"variants":["Auction before FL training: +10% accuracy, 3x faster","Secret-shared stats price FL data pre-training","Pre-training auction picks winners: +10% accuracy","Masked distribution auction: better FL clients, 3x speedup","Data priced before training: FL auction with privacy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000212,"raw_usage":{"total_tokens":1403,"prompt_tokens":916,"completion_tokens":487,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":532,"completion_tokens_details":{"reasoning_tokens":404}},"tokens_in":532,"tokens_out":487,"duration_ms":5349,"temperature":1.0,"reasoning_tokens":404,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:13:11.522092+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Make one client abort after Diffie-Hellman keys are exchanged but before it sends its masked distribution Y_e; Equation (19) can no longer cancel the pairwise PRG noises, so the server cannot recover the global distribution N_s and the pre-training price cannot be computed, directly testing the all-clients-complete premise.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the data-based client evaluation baseline AUCTION that FLMarket extends; AUCTION values clients by data mislabel rate and size."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the O(d log(2m)/m) generalization bound used to justify the logarithmic form of the score function phi."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DICE is the strongest pre-training baseline and defines data quality scores from volume ratio and per-class standard deviation; FLMarket must beat it."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DDS is the diversity-driven pre-training selection baseline used in the comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes the budget constraint condition and critical-payment style used by Algorithm 1 for winner selection and payment determination."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Diffie-Hellman key agreement is the primitive that lets each client pair share a random seed for PASS noise masks."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"S-FedAvg is the in-training Shapley-based client selection baseline against which FLMarket compares accuracy and runtime."}],"review_version":1}