{"id":"72cf6f8c-d549-4685-bad1-8266c2f0efba","arxiv_id":"2501.11167","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"FedTest lets users evaluate each other's models with local data and aggregates models by these scores, claiming faster convergence and better robustness to malicious users.","lead":"This paper introduces FedTest, a federated learning scheme where participating users test each other's models on their own local data, and the server aggregates models using the resulting accuracy scores. The authors report faster convergence and reduced impact of malicious clients in simulated CIFAR-10 and MNIST experiments.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The FedTest score is computed from a few testers' local non-IID datasets, and the paper gives no evidence that this score tracks held-out global accuracy or that the reported convergence curves are measured on an independent test set; the main convergence and robustness claims are therefore…","rationale":"I agree with the reader's weakest_assumption and would not change the verdict. The concern is load-bearing because both promised benefits flow through the score: if tester accuracy is a biased estimate of global quality, aggregation weights are misdirected and malicious-model down-weighting is not reliable. The proposed test would settle whether the concern is real by separating the mechanism (does the score rank models correctly?) from the outcome (does convergence improve on an independent test set?). I am not claiming the method cannot work; rotation of testers and averaging over several testers could mitigate the bias if the testers collectively cover all classes, but the paper provides no such analysis or supporting experiment. The reader's CONDITIONAL verdict, with a request for a rigorous experimental study and a comparison against robust aggregation baselines, remains appropriate.","tokens_in":7672,"tokens_out":8166,"duration_ms":88330,"concrete_test":"Reproduce the CIFAR-10 setting with the missing parameters specified (K, per-user class assignment, optimizer, local epochs, seeds). For each round, compute the rank correlation between the FedTest score assigned to each client model and that model's accuracy on an independent server-held test set covering all classes, for at least three non-IID partitions (1, 2, and 5 classes per user) and for K = 1, 2, 4, 8. Then plot global accuracy versus round using only this held-out set. If the rank correlation is not positive in every tested configuration, or if the claimed 5x faster convergence disappears on the held-out curve, the central argument for FedTest fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The method's score rests on accuracies reported by K rotating testers, each computed on the tester's own local dataset (Sections III and Algorithm 1). The CIFAR-10 experiments distribute data non-IID by randomly assigning classes to users, so a tester's reported accuracy is conditional on only the classes present in that tester's data. A model that does well on those classes is scored high even if it is weak on the remaining classes, and a balanced global model can be down-weighted relative to a specialized one. The paper supplies no bound, calibration, or empirical check that the tester-average score is a reliable proxy for held-out global accuracy. It also does not state whether the convergence curves in Figures 4 and 5 are evaluated on an independent server-held test set or on the same tester datasets used to compute the aggregation scores; if the latter, the reported 5x speed-up is partly an artifact of optimizing on the evaluation data. In addition, the moving-average score combines accuracies from testers with different class coverage and different accuracy scales, with no normalization described, so the aggregation weights are not tied to any well-defined global quality estimate. This gap directly undermines both headline claims: faster convergence and mitigation of malicious users.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FedTest, a federated learning aggregation scheme in which a subset of users (testers) compute accuracies of other users' models on their local datasets; the server computes a score for each model from a moving average of these accuracies raised to the fourth power and aggregates models by weighted averaging. The paper claims that FedTest accelerates convergence by about five times and mitigates the influence of malicious users on CIFAR-10 and MNIST, relative to FedAvg and a server-side accuracy-based weighted aggregation baseline.","tokens_in":7942,"tokens_out":4232,"duration_ms":37325,"significance":"The core idea of replacing a server-held test set with distributed testers' local datasets for model-quality evaluation is a useful and privacy-friendly direction. If the claims were established with independent evaluation and full experimental detail, the contribution would be meaningful for FL systems. The paper itself acknowledges several limitations in Section V, including the ad hoc choice of the accuracy exponent and the lack of communication-cost optimization. However, these admissions, combined with missing algorithmic and experimental details, prevent the reported quantitative claims from being accepted as they currently stand.","major_comments":[{"comment":"Algorithm 1 sends only the models of the N-K non-testers to the testers (steps 6-9); the K testers' own models are sent only to the server in step 11. The server therefore has no tester-computed accuracy for K of the R models it aggregates, yet the aggregation step (step 14) uses updated scores for all models. The paper does not define what score an untested model receives, so the aggregation rule is incomplete and the experimental implementation cannot be reconstructed.","section":"Section III, Algorithm 1"},{"comment":"Each model's score is described as a weighted moving average of accuracies measured on testers' local non-IID datasets, but no formula for the moving average is given and no normalization is provided. With classes randomly assigned per user in the CIFAR-10 setup, a tester can only measure accuracy on its own subset of classes, so a specialized model can receive a high score while a balanced global model is down-weighted. The paper supplies no calibration or empirical check that the tester-average score tracks held-out global accuracy, which directly undermines the convergence and robustness claims.","section":"Section III, Figures 4-5"},{"comment":"The convergence figures do not state whether the reported accuracy is computed on an independent server-held test set or on the same tester datasets used to compute the aggregation scores. If the latter, the claimed five-fold speed-up is partly an artifact of optimizing on the evaluation data. The paper must clarify the evaluation protocol and, if an independent test set is used, describe how it was split and why it was not used for training.","section":"Section IV"},{"comment":"The paper states that 'in our simulation, we noticed that the calculated scores are better if the power is increased 4' and later suggests treating the exponent as a variable. This is an explicit post hoc selection of a free parameter on the same experiments used to report the five-fold speed-up. The moving-average formula and its historical-weight parameter are also absent. A fixed, a-priori score function with sensitivity analysis is required before the convergence improvement can be evaluated.","section":"Section V.B"},{"comment":"The experimental setup omits essential settings: the number of local epochs, batch size, learning rate, optimizer, the per-user data split (number of classes and samples per user), the number of testers K, the exact random-weight distribution used for malicious users, and the number of independent runs. Without these details, the quantitative claims of 'approximately five times' faster convergence and the accuracy comparisons are not reproducible.","section":"Section IV"}],"minor_comments":[{"comment":"The title contains a typo: 'Federating Learning' should be 'Federated Learning'.","section":"Title"},{"comment":"The phrase 'the rate of tranmission' should read 'the rate of transmission'.","section":"Section V.A"},{"comment":"Figures 4 and 5 would be easier to interpret if the number of malicious users and the dataset were stated in the figure captions or legends, rather than only in the body text.","section":"Section IV"},{"comment":"The accuracy-based baseline [2] is described only vaguely; the paper should specify whether the server uses a held-out test set and how that test set is constructed in the experiments.","section":"Section IV"},{"comment":"The rule for 'Select different K testers out of N nodes' is not specified; a random selection, a deterministic rotation, or a trust-based selection would lead to different score-accumulation dynamics and should be described.","section":"Algorithm 1, step 16"}],"recommendation":"major_revision","confidential_remarks":"The manuscript gives the impression of an early-stage draft. The algorithm description has a genuine gap (the testers' own models appear to receive no score), and the experimental validation is under-specified with an admitted post hoc tuning step. These are fixable within a revision, but the current text is not yet ready for a journal. If the authors address the algorithmic gap, provide an independent evaluation set, and include full experimental details, the paper could become a viable contribution to a workshop or conference venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, the core idea is real: instead of a server-held test set for accuracy-based aggregation (as in TiFL), FedTest has a rotating subset of users test every other user's model on their local data, and the server uses a weighted moving average of these scores to aggregate. That is a natural but distinct extension of TiFL, and it addresses a genuine bottleneck. Second, the paper's two headline claims—about five times faster convergence and mitigation of malicious users—are not backed by evidence that survives scrutiny.\n\nWhat the paper does well: it identifies a legitimate limitation of server-side testing, the algorithm is clearly described (Algorithm 1 and Fig. 2), and the experimental comparison includes FedAvg and the accuracy-based TiFL approach. The authors also admit, in Section V.B, that the exponent in the score (4) was chosen after observing improved results on the same experiments. That honesty is welcome, but it confirms that part of the reported gain is post hoc tuning.\n\nThe soft spots are substantial. The paper offers no analysis of whether a few testers' non-IID local datasets give a reliable estimate of global model quality. In the experiments, classes are randomly assigned to users, so a tester may only have a subset of classes. A model that is specialized on those classes can get a high tester score while being poor globally, and a balanced model can be penalized. The authors do not check whether the tester-average score correlates with held-out global accuracy. That is the load-bearing assumption.\n\nThe experimental section is also under-specified: no hyperparameters, no local epochs, no learning rate, no dataset split, no exact number of testers K, and no error bars. Crucially, the paper never states whether the convergence curves in Figs. 4 and 5 are evaluated on an independent server test set or on the same tester local data used for scoring. If it is the latter, the reported speed-up is partly an artifact. Finally, there is no comparison to standard robust aggregation methods (Krum, trimmed mean, median), so the robustness claim is not contextualized.\n\nThe idea is not bad, and the authors are onto something. But this version is not ready. I would send it to a serious referee, with the expectation of major revisions: full experimental detail, a clear evaluation protocol, an empirical or analytical check of the score's reliability under non-IID data, and baselines for adversarial robustness. The paper is a seed, not a finished result.","headline":"FedTest's distributed testing idea is worth a look, but this version's evidence is too thin to support the 5x speed-up and robustness claims.","tokens_in":8431,"tokens_out":3533,"would_cite":false,"duration_ms":31939,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FedTest: users grade each other's models to accelerate federated learning","keywords":["federated learning","model aggregation","convergence acceleration","adversarial attacks","malicious clients","accuracy-based weighting","non-IID data","distributed testing"],"falsifier":"Run FedTest on a 20-client CIFAR-10 split where each of the K testers holds data from only two classes, and compare global test accuracy against FedAvg over 200 rounds; if FedTest's accuracy stays below FedAvg or its convergence advantage disappears, the claim that local test scores are reliable global quality indicators fails.","tokens_in":7523,"feed_emoji":"🔄","tokens_out":6741,"duration_ms":62529,"temperature":0.7,"pith_summary":"The paper proposes FedTest, a federated learning scheme in which a subset of users, called testers, evaluates the models trained by other users on their own local data, so no central test set is required. Each tester reports the measured accuracies to the server, which combines them into a per-model score using a weighted moving average that favors recent rounds and, in the implementation, raises accuracy to the fourth power. The server aggregates models by these scores rather than by dataset size. On CIFAR-10, the paper reports that FedTest reaches the same accuracy in about 20 iterations where FedAvg and the accuracy-based approach need about 100, and that it maintains higher accuracy when malicious users send random weights. The claim is that this makes federated learning both faster and more robust to poisoned updates while keeping data where it was trained.","feed_headline":"FedTest cuts federated learning rounds fivefold by user cross-testing","feed_subtitle":"A subset of users scores every other model on local data, so the server can weight good models and ignore bad ones.","key_machinery":"The load-bearing mechanism is cross-testing with per-round accuracy scores: K users act as testers, each receives the other N-K models over orthogonal resource blocks, evaluates them on its local dataset, and sends both its own model and the accuracy vector to the server. The server forms a weighted moving-average score per model, weighting recent rounds and, in the simulation, raising accuracies to the fourth power ($a^4$) to amplify differences, then aggregates models with these scores. This replaces both sample-count weighting (FedAvg) and server-held test-set weighting (accuracy-based FL), and it is what the convergence and robustness findings are attributed to.","core_discovery":"The central discovery is that the distributed data already sitting on client devices can play the role of a held-out test set: each round, a randomly chosen set of K testers evaluates all non-tester models on its local data and sends the accuracies to the server. The server's weighted aggregation then penalizes weak or adversarial models without needing a server-side test dataset. In the paper's experiments with 20 clients, CIFAR-10 and MNIST, non-IID class assignments, and up to 4 malicious clients, this yields faster convergence and better final accuracy than FedAvg and than a server-held-test-set accuracy-based baseline.","pith_inferences":["If a tester's local data is concentrated in a few classes, a genuinely good global model can receive a low score and be down-weighted, so the scheme's robustness may degrade precisely when testers have extremely skewed data; weighting tester scores by class coverage or selecting balanced testers is a natural extension.","Because each model must reach both the server and all testers, the transmission rate is limited by the worst tester channel; joint tester selection, user scheduling, and resource-block allocation could therefore produce larger practical gains than the random tester rotation used here.","The fourth-power amplification of accuracy is heuristic, and the optimal exponent likely depends on data heterogeneity and attack strength; treating the exponent as a tunable parameter across rounds is a concrete testable improvement.","The historical score produced by the moving average could itself serve as an anomaly detector: users whose scores are persistently low or erratic can be quarantined before they can do harm, which the paper discusses only as future work."],"forward_implications":["No server-held test set is needed to evaluate local model quality, which removes a major practical bottleneck of accuracy-based aggregation.","FedTest can reach the same accuracy as FedAvg in a fraction of the rounds, roughly five times fewer on CIFAR-10, cutting communication and training cost.","Models that score poorly on tester data, including random-weight malicious updates, receive small aggregation weights and therefore have limited influence on the global model.","The method works on both CIFAR-10 and MNIST, with the MNIST gains appearing mainly when malicious users are present because that dataset does not otherwise distinguish strong from weak models.","The server can track scores across rounds and use consistently low or erratic scores as a signal for flagging suspicious users."],"supporting_citations":[{"why":"Supplies the FedAvg baseline whose convergence rate and robustness FedTest is measured against.","marker":"[1]"},{"why":"Introduces accuracy-based weighted aggregation, the approach FedTest replaces because it needs a server-held test set.","marker":"[2]"},{"why":"Supports the paper's explanation that MNIST cannot differentiate strong from weak models, contextualizing the FedTest MNIST results.","marker":"[9]"}],"fun_headline_variants":["FedTest speeds FL rounds fivefold by user cross-testing","Clients test each other's models to speed FL and foil attacks","FedTest turns local data into test sets for faster, safer FL","User cross-testing in FL: faster convergence, stronger against attacks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole scheme depends on a tester's local dataset accurately reflecting how good a model is globally; a tester with data for only a few classes can give a strong global model a low score and cause the server to down-weight it.","fun_headline_variants_meta":{"raw":{"variants":["FedTest speeds FL rounds fivefold by user cross-testing","Clients test each other's models to speed FL and foil attacks","FedTest turns local data into test sets for faster, safer FL","User cross-testing in FL: faster convergence, stronger against attacks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001147,"raw_usage":{"total_tokens":4717,"prompt_tokens":866,"completion_tokens":3851,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":482,"completion_tokens_details":{"reasoning_tokens":3777}},"tokens_in":482,"tokens_out":3851,"duration_ms":25451,"temperature":1.0,"reasoning_tokens":3777,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:33:47.552372+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run FedTest on a 20-client CIFAR-10 split where each of the K testers holds data from only two classes, and compare global test accuracy against FedAvg over 200 rounds; if FedTest's accuracy stays below FedAvg or its convergence advantage disappears, the claim that local test scores are reliable global quality indicators fails.","supporting_citations":[{"cited_title":"Tifl: A tier-based federated learning system,","cited_arxiv_id":null,"evidence_quote":"Introduces accuracy-based weighted aggregation, the approach FedTest replaces because it needs a server-held test set."}],"review_version":1}