{"id":"2112dd3e-b193-4302-9ee0-2d3c0805bed9","arxiv_id":"2505.21219","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":10,"one_line_summary":"In cross-silo federated learning, combining client bids, Shapley-based reputation, and budget-constrained 0-1 selection improves global model accuracy over random selection under label noise and low-bid interference.","lead":"This paper proposes SBRO-FL, a federated learning client selection method that combines client bids, reputation scores, and Shapley-value contribution estimates to pick which data owners train the shared model under a fixed budget. The authors report accuracy gains over random selection on four image datasets, including under label noise and cheap adversarial bids.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Definition 1 (Eq. 4) omits the negative sign in the loss branch of the prospect-theory reputation score: below-threshold clients receive positive scores that increase as reputation decreases, so the score is U-shaped and the selection objective can prefer the worst clients.","rationale":"The reader identified the server-held validation set as the weakest assumption, but that is a deployment limitation that would affect any validation-based reputation method. The sign error in Eq. 4 is more load-bearing because it is an internal inconsistency in the proposed mechanism. The stated goal is to penalize low-reputation clients more heavily than high-reputation clients are rewarded (loss aversion), yet the printed score does the opposite in the loss region: the farther below the reference point, the larger the positive score. This can directly select the noisiest clients once reputations diverge, undermining the claimed 'prioritize high-quality client updates.' The numerical inconsistencies between the abstract (7.14%), Table 2 (10.3%), and Table 3 (16.7%) and the absence of multi-seed runs are additional reasons not to accept the current empirical claim, but the sign error is the decisive technical defect. As written, the method cannot implement the described reputation mechanism; therefore the conditional acceptance should be changed to rejection until the formula is corrected, the implementation is checked, and the experiments are re-reported with the corrected score. If inspection of the repository shows the intended minus sign was used in code, the verdict could become conditional on a thorough revision; but the current text alone does not support the central claim.","tokens_in":15524,"tokens_out":8947,"duration_ms":101370,"concrete_test":"Take R_th=1 and two clients with equal bid, A with R=−10 and B with R=1.5. Under the printed Eq. 4, z(A)=11^0.3≈2.05 and z(B)=0.5^0.15≈0.90, so the worst client is preferred by the selection model. Then inspect the z(R) code in the linked GitHub repository: if it contains the missing minus sign, Definition 1 and Algorithm 1 line 5 must be corrected and the reported experiments re-run; if it matches the printed formula, rerun Tables 2 and 3 with the corrected loss-averse branch −γ(R_th−R_i)^β. If SBRO-FL's margin over RS-FL disappears or reverses, the central claim fails; if it persists, the paper still needs a corrected equation and full reproducibility details.","verdict_should_be":"REJECT","load_bearing_attack":"Section 3.2, Definition 1 (Eq. 4) defines z(R_i)=γ(R_th−R_i)^β for R_i≤R_th, with α=0.15, β=0.3, γ=1. For any below-threshold client, this branch is positive, and the score increases as R_i moves further below R_th. The function is therefore minimized at R_i=R_th and rises on both sides, so a client with very negative reputation can receive a higher score than a client with reputation slightly above the threshold. The text claims this region has 'heightened negativity' and then says z_min shifts scores to be non-negative, both of which imply the intended prospect-theory form was −γ(R_th−R_i)^β. Since z(R_i) enters the selection objective (Eq. 5) and drives all reputation-weighted utility, the printed mechanism does not implement the stated loss-averse quality filtering. If the implementation follows Eq. 4, the reported gains cannot be attributed to the described reputation logic; if it adds the missing sign, the manuscript's central equation and Algorithm 1 line 5 are wrong. In either case the central claim is not supported by the text as written.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces SBRO-FL, a client-selection framework for cross-silo federated learning that combines Shapley-value-based contribution evaluation, a prospect-theory-inspired reputation update, and a 0-1 integer program for budget-constrained selection. Clients submit bids based on perceived data quality; the server computes reputation scores from historical Shapley values and bid prices, then selects clients to maximize reputation-weighted utility under a budget. Experiments on FashionMNIST, EMNIST, CIFAR-10, and SVHN under label flipping and a low-bid interference scenario compare SBRO-FL with random selection, all-client selection, and an oracle clean-subset selection, reporting consistent accuracy gains and robustness claims.","tokens_in":15782,"tokens_out":5771,"duration_ms":64152,"significance":"If the method performs as claimed, the framework is a useful step toward jointly addressing data quality, incentives, and budget constraints in cross-silo FL. The paper offers a concrete algorithmic pipeline, explicit definitions, and a GitHub link for reproducibility, and it explicitly acknowledges limitations such as Shapley computational cost and the absence of an ablation study. However, the printed reputation score in Eq. (4) has a sign inconsistency that makes the selection objective favor low-reputation clients as written, and the experimental validation lacks multiple seeds, stronger baselines, and component-level ablations. These issues must be resolved before the contribution can be assessed as reliable.","major_comments":[{"comment":"As printed, the loss branch is z(R_i)=γ(R_th−R_i)^β with γ=1 and β=0.3, which is positive and increases as R_i decreases below R_th. The score function is therefore U-shaped, minimized at R_i=R_th, and the selection objective in Eq. (5) will prefer clients with very low reputation over clients just above the threshold. This contradicts the text's claim of 'heightened negativity' and the stated loss-averse intent; the intended form appears to be −γ(R_th−R_i)^β. Since Eq. (4) drives the entire selection objective and appears again in Algorithm 1 line 5, the central claim is not supported by the manuscript as written. Please correct the equation or clarify the actual implementation and verify that the reported results follow from the corrected score.","section":"Section 3.2, Definition 1, Eq. (4)"},{"comment":"The reported variance of less than 10^-4 is computed over the final 20 training rounds of a single run, not over independent training runs with different random seeds. Without multiple seeds, the claimed accuracy improvements over RS-FL cannot be separated from initialization and sampling luck. Please report means and standard deviations across at least five independent seeds, and include statistical significance or effect sizes for the main comparisons.","section":"Section 5.1, Table 2 and Fig. 4"},{"comment":"The baseline set (RS-FL, HQRS-FL, All-FL) does not include any existing client-selection or robustness method, and Section 5.3 explicitly defers the ablation of the framework's components to future work. This leaves open which component (bidding, reputation, Shapley evaluation, budget constraint) is responsible for the gains and whether the gains are an artifact of the weak comparison. An ablation with components removed and at least one strong prior method—for example, a reputation-based client-selection baseline or a Byzantine-robust aggregation method—is needed to support the central 'unified framework' claim.","section":"Section 4.2 and Section 5.3"},{"comment":"The entire reputation signal assumes that the server holds a labeled validation dataset D_val and that exact Shapley values computed by averaging client updates and evaluating accuracy on D_val reliably measure each client's data quality. This assumption is not tested or even stated as a limitation; the paper only acknowledges Shapley's combinatorial cost in Section 6.1. In privacy-sensitive cross-silo deployments, such a validation set may be unavailable, and under non-IID or poisoned data, accuracy on averaged updates may not reflect true marginal contributions. The authors should state this assumption explicitly and evaluate a setting without D_val or with approximate Shapley values.","section":"Section 3.4, Definition 3 and Algorithm 1 line 13"}],"minor_comments":[{"comment":"The text states that SBRO-FL achieves an average improvement of 10.89% over RS-FL and 1.92% over All-FL, but Table 3's average Gain column is 16.7%, and the relative gain over All-FL from the averaged accuracies is about 17.1%. Please reconcile these numbers.","section":"Section 5.2, after Table 3"},{"comment":"The statement that SBRO-FL 'inherits similar convergence behavior under bounded-variance assumptions' is an unsupported assertion; no theorem or specific reference is provided. Either supply a convergence argument or weaken the statement to an empirical observation.","section":"Section 6.1"},{"comment":"There is a typo in the sentence 'CNN architectures were adapted to the complexity of the data et'; it should be 'data set'. Also, the prospect-theory parameters are said to be set after 'preliminary tuning' on the same benchmark suite, so the authors should discuss the risk of selection overfitting or use a separate hyperparameter split.","section":"Section 4.3"}],"recommendation":"major_revision","confidential_remarks":"The Eq. (4) sign issue is the main technical blocker. I would recommend checking the released GitHub code to determine whether the implementation matches the printed equation; if it does, the reported gains cannot be attributed to the described reputation mechanism. The paper is potentially suitable after a careful revision that corrects this, adds multi-seed experiments and stronger baselines, and clarifies the D_val assumption."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this for a quick take: SBRO-FL is a reasonable integration of known ideas—bids, Shapley-based contribution, prospect-theory-style reputation, and budgeted 0-1 selection. I have not seen that exact closed loop in the FL client-selection literature, so the assembly is new enough to be worth a look. The authors also state their limitations plainly: no ablation, exact-Shapley scaling, knapsack NP-hardness, honest-bid assumption. They point to a GitHub repository, which I could not verify from the text.\n\nThe soft spots are not minor. Definition 1/Eq. (4) has a sign error in the loss branch: γ(R_th - R_i)^β without a leading minus sign. For R_i ≤ R_th that term is positive and grows as reputation falls, so below-threshold clients can score higher than above-threshold ones. This is the opposite of the 'heightened negativity' the prose claims and makes the selection objective U-shaped. If the implementation follows the equation, the reported gains cannot be attributed to loss-averse reputation filtering. If the implementation has the intended minus sign, the manuscript's central equation and Algorithm 1 are wrong. Either way, the text as written does not support the central claim.\n\nThe experiments need strengthening too. There are no multiple seeds; the 'negligible variance' claim is not backed by reported runs. Baselines are random, all-client, and oracle clean-subset, with no state-of-the-art comparison. The ablation is explicitly left for future work. Some numbers conflict: the abstract says 7.14% average gain, Table 2 shows 10.3%, and Section 5.2's '1.92% over All-FL' does not match Table 3. Hyperparameters ω, ψ, ρ are used but never specified. The framework also assumes a server-held labeled validation set, which is not available in many privacy-sensitive FL deployments; that assumption needs at least a direct discussion.\n\nNone of this kills the underlying idea. The intended prospect-theory form is clear, and the framework becomes coherent once the sign is restored. I would not cite this in its current form, but I would send it to a serious referee with the expectation of major revision. It is the kind of paper that is useful for cross-silo FL researchers working on incentive-aware selection, provided the authors fix the equation, rerun the experiments, and add the missing statistical and ablation evidence.","headline":"The integrated SBRO-FL loop is a reasonable and novel assembly of known ideas, but the printed reputation score is missing the negative sign in the loss branch, and the experimental evidence is too thin to support the claims as written.","tokens_in":16336,"tokens_out":5420,"would_cite":false,"duration_ms":59900,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SBRO-FL, a unified client-selection framework that combines dynamic bidding, reputation scoring, and Shapley-value contribution assessment, consistently beats random selection across four datasets and stays close to an oracle that always…","keywords":["federated learning","client selection","data quality decompensation","Shapley value","reputation system","prospect theory","budget constraints","incentive mechanism"],"falsifier":"Run SBRO-FL on a non-IID partition where a client's updates improve true test accuracy but lower validation-set accuracy, and track whether that client is repeatedly excluded; if the method consistently rejects the beneficial client and final accuracy suffers, the validation-accuracy proxy at the heart of the reputation update is falsified. A cheaper check is to compute the rank correlation between the Shapley values from Eq. (8) and leave-one-out true data-quality scores on a small federation.","tokens_in":15294,"feed_emoji":"🎯","tokens_out":10537,"duration_ms":94441,"temperature":0.7,"pith_summary":"In cross-silo federated learning, participating organizations can have very different data quality, and bad updates accumulate over rounds—a problem the paper calls data quality decompensation. SBRO-FL is a single client-selection loop that simultaneously handles this degradation, the incentives that keep clients participating, and the server's limited budget. At each round, clients submit bids; the server scores their historical reliability, solves a budget-constrained 0-1 integer program to pick a subset, and later uses Shapley values on a small validation set to update each client's reputation. The paper reports that this loop consistently outperforms random selection and all-client selection on four image datasets, and that under adversarial low-bid interference it keeps most of its advantage, approaching an oracle that always picks clean clients. If the mechanism holds, a server can maintain trustworthy federated learning without knowing clients' true data quality in advance.","feed_headline":"Lifts federated-learning accuracy by 10% with a bid-reputation loop","feed_subtitle":"Server-side scoring of each client's marginal contribution filters noisy updates without knowing clean labels.","key_machinery":"The load-bearing machinery is a 0-1 integer program for selection plus two feedback signals. The selection objective maximizes $\\sum_i (z(R_i)-z_{\\min}) \\delta^{\\mathrm{count}_i} x_i$ subject to $\\sum_i B_i x_i \\le B_{\\mathrm{budget}}$, where $z(R_i)$ is a prospect-theory transformation of reputation $R_i$ that makes losses sting more than gains, and $\\delta^{\\mathrm{count}_i}$ discourages picking the same clients too often. After aggregation, each selected client's marginal contribution is computed by a Shapley value over averaged model updates evaluated on a server-held validation set $D_{\\mathrm{val}}$; positive contributors get an exponentially saturating reputation reward, while clients with non-positive Shapley values are penalized with a penalty that grows with the number of recent bad rounds. That reputation then feeds the next round's scores, closing the loop.","core_discovery":"The paper's central claim is that reputation-weighted, budget-constrained client selection can be made to work as a closed loop in which bids, Shapley-based contribution scores, and prospect-theory-adjusted reputations reinforce each other. In the experiments, SBRO-FL beats random selection on every dataset and improves average final accuracy by 10.3% relative to random selection; on the harder CIFAR-10 and SVHN tasks the relative gains are about 19%. When low-quality clients strategically lower their bids to get selected, SBRO-FL still outperforms random selection by an average of 16.7%, close to the oracle high-quality-random baseline. The paper also observes that SBRO-FL sometimes edges out the oracle, because it can still learn useful signal from mildly noisy clients rather than discarding them entirely.","pith_inferences":["A natural extension not developed in the paper is to drop the server-side validation set and replace $D_{\\mathrm{val}}$ with self-supervised or proxy metrics; the paper's own claims depend on labeled validation data, so the success of that extension is not established here.","The prospect-theory asymmetry is a design bet that inconsistency is more damaging than occasional brilliance; a symmetric reputation update would be a natural control condition to isolate the value of loss aversion.","Because Eq. (8) averages client updates rather than retraining on subsets, the 'marginal contribution' measures alignment with validation accuracy under one particular aggregation rule; under strongly non-IID partitions this proxy could mis-rank clients whose updates are useful only in combination.","Combining SBRO-FL with a Byzantine-robust aggregator could be synergistic, since the reputation filter would then only need to exclude clients whose updates are unhelpful rather than malicious by construction."],"forward_implications":["A server can protect global-model accuracy without knowing which clients have clean labels, as long as it holds a labeled validation set and can compute per-round contributions.","Low-bid interference does not break the mechanism: clients that bid cheaply but contribute negatively are penalized by the reputation update rather than rewarded by the budget constraint.","Selection diversity is maintained via the recent-history decay factor, so the system keeps exploring clients instead of locking onto a small elite subset.","The method inherits the convergence behavior of FedAvg-style training, since it changes only which clients are selected, not the local training or aggregation rule.","Because exact Shapley computation is combinatorial, deployments with many clients would need approximate contribution estimators to keep the same loop tractable."],"supporting_citations":[{"why":"Supplies FedAvg, the aggregation and the random-selection baseline that SBRO-FL is measured against.","marker":"[25]"},{"why":"Defines prospect theory, whose asymmetric value function is used to convert reputations into selection scores.","marker":"[15]"},{"why":"Introduces Data Shapley, the valuation idea from which the paper's Shapley-based contribution measurement is drawn.","marker":"[9]"},{"why":"Defines the Shapley value itself, which Eq. (7)-(8) adapt to federated client contributions.","marker":"[34]"},{"why":"Offers GTG-Shapley, cited as a scalable approximation for large federations where exact Shapley is infeasible.","marker":"[23]"},{"why":"Provides the open-source federated-learning experimentation platform on which all reported comparisons were run.","marker":"[10]"},{"why":"Shows how a single low-quality update can degrade the global model, motivating the loss-averse reputation penalty.","marker":"[7]"},{"why":"Exemplifies reputation-based client selection, the approach SBRO-FL extends with bidding and budget optimization.","marker":"[16]"}],"fun_headline_variants":["Bid-reputation loop boosts federated learning accuracy by 10%","Client bids plus Shapley scores cut noisy updates in federated learning","Reputation-weighted selection lifts FL accuracy, even when clients game bids","Shapley-bid reputation loop matches oracle despite noisy clients","Bids, Shapley values, and reputation curb data-quality decay in FL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the server holds a labeled validation set and that the Shapley value computed by averaging client updates and testing on that set reliably reflects each client's data quality, so if those labels are missing or that proxy mis-ranks clients under non-IID or poisoned data, the reputation signal loses its grounding.","fun_headline_variants_meta":{"raw":{"variants":["Bid-reputation loop boosts federated learning accuracy by 10%","Client bids plus Shapley scores cut noisy updates in federated learning","Reputation-weighted selection lifts FL accuracy, even when clients game bids","Shapley-bid reputation loop matches oracle despite noisy clients","Bids, Shapley values, and reputation curb data-quality decay in FL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000749,"raw_usage":{"total_tokens":3326,"prompt_tokens":926,"completion_tokens":2400,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":542,"completion_tokens_details":{"reasoning_tokens":2305}},"tokens_in":542,"tokens_out":2400,"duration_ms":18348,"temperature":1.0,"reasoning_tokens":2305,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:33:40.529908+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run SBRO-FL on a non-IID partition where a client's updates improve true test accuracy but lower validation-set accuracy, and track whether that client is repeatedly excluded; if the method consistently rejects the beneficial client and final accuracy suffers, the validation-accuracy proxy at the heart of the reputation update is falsified. A cheaper check is to compute the rank correlation between the Shapley values from Eq. (8) and leave-one-out true data-quality scores on a small federation.","supporting_citations":[{"cited_title":"Communication-efficient learning of deep networks from decentralized data","cited_arxiv_id":null,"evidence_quote":"Supplies FedAvg, the aggregation and the random-selection baseline that SBRO-FL is measured against."},{"cited_title":"Prospect theory: An analysis of decision under risks","cited_arxiv_id":null,"evidence_quote":"Defines prospect theory, whose asymmetric value function is used to convert reputations into selection scores."},{"cited_title":"Data shapley: Equitable valuation of data for machine learn- ing","cited_arxiv_id":null,"evidence_quote":"Introduces Data Shapley, the valuation idea from which the paper's Shapley-based contribution measurement is drawn."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Shapley value itself, which Eq. (7)-(8) adapt to federated client contributions."},{"cited_title":"Gtg-shapley: Efficient and accurate participant contribution evaluation in federated learning.ACM Transactions on intel- ligent Systems and Technology (TIST), 13(4):1–21, 2022","cited_arxiv_id":null,"evidence_quote":"Offers GTG-Shapley, cited as a scalable approximation for large federations where exact Shapley is infeasible."},{"cited_title":"Victoria Luzón","cited_arxiv_id":null,"evidence_quote":"Provides the open-source federated-learning experimentation platform on which all reported comparisons were run."},{"cited_title":"Local model poisoning attacks to byzantine-robust federated learning","cited_arxiv_id":null,"evidence_quote":"Shows how a single low-quality update can degrade the global model, motivating the loss-averse reputation penalty."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Exemplifies reputation-based client selection, the approach SBRO-FL extends with bidding and budget optimization."}],"review_version":1}