{"id":"d58b63da-8ce8-4862-a3d6-15d6a518d47b","arxiv_id":"2507.12903","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"The authors present a new 31-class, 8-source image classification dataset for federated learning and show that Fed-Cyclic and Fed-Star beat FedAvg and RingFed on it.","lead":"The paper introduces a new dataset of 23,326 images from eight commercial image sources and proposes two federated learning algorithms, Fed-Cyclic and Fed-Star. A smart generalist might read this because it targets real-world domain shift in privacy-preserving collaborative learning, a common hurdle for federated systems.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fed-Star's pre-aggregation weights models by error on the target client, a mechanism antithetical to its stated goal and unsupported by ablation; this makes the central empirical claim unverifiable without code or a corrected formula.","rationale":"The reader's weakest assumption was the uniform transferability of frozen ImageNet features, which affects external validity. I find a more load-bearing and internal concern: the Fed-Star pre-aggregation weighting in Eq. 8 is logically inverted and not validated. If a model has low accuracy on a client's data, it receives more weight in that client's averaged model, which should degrade rather than improve the model for that client. The paper provides no ablation, no analysis of the M matrix, and no code to confirm the reported results. The frozen-feature concern is secondary: it limits generalization, whereas the weighting issue questions the soundness of Fed-Star itself. The dataset is a plausible contribution, and the cross-source FL setup is realistic, but the algorithmic claim is not assessable from the manuscript alone. The lack of error bars, single runs, and tuning of baselines on the evaluation set further undermine confidence. I therefore recommend shifting from CONDITIONAL to UNVERDICTED until the Fed-Star mechanism is clarified or the weighing is ablated, and code/data are made available.","tokens_in":11789,"tokens_out":17567,"duration_ms":195192,"concrete_test":"Re-run Fed-Star with the pre-aggregation weights changed to M(k,j)=Acc(w_j,D_k)/100 (weighting by accuracy instead of error), keeping all other settings, dataset splits, and seeds fixed. Compare the resulting accuracy with Table 5. If Fed-Star's accuracy falls below RingFed's or the margin shrinks to noise, the reported advantage is an artifact of the inverted weighting. Additionally, report means and standard deviations over at least five seeds to test whether the original 91.72% vs 89.65% gap is statistically significant.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"The central claim is that Fed-Star outperforms FedAvg and RingFed on the new dataset. The key novel step is Eq. 8: M(k,j)=1 - Acc(w_j, D_k)/100, where Acc is the training accuracy of model j on client k's data. This assigns higher weight to models that perform poorly on the target client and lower weight to models that perform well, including downweighting the client's own model when its own accuracy is high. The paper's stated rationale is to 'learn outlier features,' but low cross-domain accuracy more plausibly indicates poor transfer, and no ablation or theoretical argument is given to show that this inverted weighting is beneficial. Because no code is released, the reported 91.72% accuracy cannot be traced to the algorithm as written; if the weighting were a typo for M=Acc/100, the results might still hold, but the paper's explanation would be wrong. Since the abstract claims 'both algorithms' are better, this gap compromises the central claim.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces a new image-classification dataset for federated learning: 23,326 images from eight commercial stock-photo sources over 31 Office-31 categories, with each source treated as one client. It also proposes two federated algorithms: Fed-Cyclic, which passes a single model cyclically from client to client, and Fed-Star, in which clients share models and pre-aggregate them with accuracy-based weights before periodic server aggregation. In experiments with frozen ImageNet-pretrained VGG-19 features and three newly trained fully connected layers, the authors report that Fed-Star (91.72%) and Fed-Cyclic (91.15%) outperform RingFed (89.65%) and FedAvg (89.11%) on global test accuracy, and that Fed-Star outperforms locally trained baselines on all eight sources.","tokens_in":12035,"tokens_out":9030,"duration_ms":94154,"significance":"If the results are reproducible, the dataset is a useful real-world federated-learning benchmark with genuine domain shift, and Fed-Cyclic is a simple communication-efficient alternative to server-centric aggregation. The accuracy-weighted pre-aggregation idea in Fed-Star is also interesting because it attempts to handle statistical heterogeneity without a centralized server. The paper reports both accuracy and macro/weighted F1 scores, which is a strength. However, the contribution is currently conditional: no code or dataset is released, there are no repeated runs or error bars, the key pre-aggregation formula in Eq. (8) is not validated by any ablation, and the algorithm specifications contain ambiguities that prevent independent reproduction. The significance of the empirical claim therefore rests on the authors' ability to address these reproducibility issues.","major_comments":[{"comment":"The pre-aggregation weight M(k,j)=1 - Acc(w_j,D_k)/100 assigns the largest weight to models that perform worst on the target client's own training set. This is the opposite of the usual notion of useful transfer: low training accuracy on D_k more plausibly indicates that the model has not learned features relevant to that client, and the formula also downweights the client's own freshly trained model whenever its training accuracy is high. The text states that the goal is to 'learn outlier features,' but no ablation is provided to show that this inverted weighting is beneficial; replacing M by Acc/100, by equal weights, or by excluding j=k is not tested. Because no code is released, the reported 91.72% accuracy for Fed-Star cannot be traced to the algorithm as written, and this is load-bearing for the central claim.","section":"Section 3.3, Eq. (8) and Algorithm 2"},{"comment":"The pseudocode has specification errors that prevent reproduction. In Algorithm 1, the cyclic rule w^r_{k+1} <- w^{r+1}_k overwrites the starting weights for client k+1 within the same round, but at the end of the round only w^{r+1}_1 is set to w^{r+1}_K; the weights used by clients 2,...,K-1 at the start of the next round are not rotated, so the exact update order is ambiguous. In Algorithm 2, the server aggregation is written as w^{r+1} = (1/K) * sum_k (|D_k|/|D|) w_k^{r,P}, but the coefficients |D_k|/|D| already sum to 1, so the extra 1/K factor scales the global model down every round if taken literally. These issues, combined with the absence of code, make the experiments unreproducible from the paper alone.","section":"Algorithm 1 and Algorithm 2"},{"comment":"The RingFed baseline's hyperparameter gamma is selected by evaluating four values on the test set and keeping the best (Table 4), and the learning-rate study in Table 7 is reported without a held-out validation split. All experiments appear to be single runs with no error bars, confidence intervals, or significance tests. The claimed margins over RingFed are about 1.5-2.1 percentage points, so the central claim that the proposed algorithms are 'better than existing baselines' is not supported with statistical evidence; the difference could be due to run-to-run variance or to test-set-based hyperparameter selection.","section":"Section 5.2, Table 4, and Table 7"}],"minor_comments":[{"comment":"There are several typos, including 'converegence' in the introduction, 'anonimity' in Section 1, and 'forkth' before Eq. (9); these should be corrected.","section":"Section 1"},{"comment":"The text says that FedAvg accuracy 'steadily increases with the decrease in the learning rate,' but the reported values (89.11% at 3e-4, 91.39% at 1e-3, 91.43% at 3e-3, 91.33% at 7e-3) do not show a monotone trend; please rephrase.","section":"Section 5.3, paragraph 2"},{"comment":"The local-model baseline is labeled 'using E = 250,' but the training procedure for the local model is otherwise unspecified; please clarify whether this means 250 local epochs and which optimizer and learning rate were used.","section":"Table 6"},{"comment":"The sentence 'Fed-Star outperforms all the local models trained using traditional ML method' is inaccurate, since the local models are trained with SGD on VGG-19 features; please rephrase to something like 'locally trained models.'","section":"Section 5.2"},{"comment":"Several algorithms discussed in the related work, such as FedProx, SCAFFOLD, and FedNova, are not evaluated; including at least one strong modern baseline would strengthen the comparative claim that Fed-Cyclic and Fed-Star perform better than existing methods.","section":"Related Work and Section 5"},{"comment":"The dataset is one of the main contributions, but no download link or access mechanism is provided; please state its availability or explain any restrictions due to the commercial image sources.","section":"Section 4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be a reprint of a WACV 2023 publication posted on arXiv; the editor may wish to verify the novelty and scope relative to this venue. If code and the dataset cannot be released, the empirical claims should be treated with correspondingly less confidence, and the authors should be asked to provide at least pseudocode fixes and ablations for the problematic pre-aggregation weighting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The dataset is the genuine contribution here. 23,326 images, 31 classes, from 8 commercial stock sources with natural domain shift, and it is indeed the first FL-specific image classification dataset I know of. Fed-Cyclic is a simple sequential chain, and Fed-Star is a star topology with accuracy-weighted pre-aggregation. The reported numbers are consistent: both beat FedAvg and RingFed on global and local accuracy, and Table 6 shows Fed-Star winning every client. That is a useful benchmark and a reasonable baseline result.\n\nWhat the paper does well: the data curation is documented with per-class and per-source statistics, the evaluation uses accuracy plus F1, and the algorithms are described clearly enough to reimplement. The idea of weighting neighbor models by local accuracy is interesting, and the local personalization results in Table 6 are worth looking at.\n\nNow the soft spots. No code or data release, which is a real problem for a dataset-plus-algorithm paper. No error bars or repeated runs, so the 1-2% gains could be noise. RingFed's gamma is tuned on the test set (Table 4), and only two baselines are compared. Most tellingly, Eq. 8 is M(k,j) = 1 - Acc(w_j, D_k)/100. That gives higher weight to models that are less accurate on client k's data, including downweighting the client's own model when it is good. The paper says this is to learn outlier features, but there is no ablation, and the opposite weighting (or simple averaging) would be more natural. The stress-test note claims this makes the result unverifiable; I think that is overstated—the formula is explicit, so the implementation is clear—but the missing ablation is a genuine flaw. If the authors have code, this needs to be resolved.\n\nMinor point: the frozen VGG-19 features may transfer unevenly across sources, and there is no per-source feature analysis. The evaluation section would be stronger with a breakdown.\n\nWho is this for? Researchers building FL benchmarks with real domain shift. It deserves a serious referee because the dataset is useful and the algorithm questions are addressable, but the current analysis is too thin to trust the ranking. My recommendation: treat the dataset as plausible and the algorithm claims as tentative; if this came to us, I would send it to review but expect major revision.","headline":"The dataset is the real contribution; the algorithms are simple, the evaluation is thin, and Eq. 8's weighting rule is odd and unablated.","tokens_in":12512,"tokens_out":3031,"would_cite":false,"duration_ms":36482,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fed-Cyclic and Fed-Star, two new federated topologies, outperform FedAvg and RingFed on an eight-source commercial image dataset, with Fed-Star the most accurate.","keywords":["federated learning","image classification","domain shift","statistical heterogeneity","Fed-Cyclic","Fed-Star","commercial image sources","non-IID data"],"falsifier":"Re-run the four algorithms on the same dataset and the same splits with the VGG-19 backbone fine-tuned end-to-end, or with a randomly initialized network in place of the frozen extractor. If Fed-Star's margin over FedAvg and RingFed shrinks or disappears under full training, the claim that the algorithms themselves handle domain shift would not be established independently of the pretrained features.","tokens_in":11617,"feed_emoji":"🖼️","tokens_out":11744,"duration_ms":115002,"temperature":0.7,"pith_summary":"Federated learning (training one shared model across data owners who never share their data) normally assumes clients' local data come from similar distributions. This paper argues that in real deployments clients can be different commercial image sources, each with its own visual style, so it introduces an eight-source, 31-class dataset of 23,326 images and two algorithms built for that heterogeneity: Fed-Cyclic, which passes the model from client to client in a ring, and Fed-Star, which lets each client pre-aggregate the other clients' models weighted by how different they are from its own data. On this dataset both new algorithms beat FedAvg and RingFed, with Fed-Star reaching 91.72% global accuracy and Fed-Cyclic 91.15%, against 89.65% and 89.11% for the baselines. The paper positions the dataset as the first image-classification benchmark built specifically for federated learning rather than an existing dataset artificially split, and the claim matters because it suggests that simple changes to how weights move between clients can handle domain shift and produce models that are also better per client.","feed_headline":"Fed-Star hits 91.7% accuracy and beats FedAvg on all eight clients","feed_subtitle":"Two new federated algorithms outdo FedAvg and RingFed on an 8-source dataset with real domain shift.","key_machinery":"The load-bearing pieces are the two algorithms and the dataset. Fed-Cyclic's core identity is the client-to-client hand-off $w_{k+1}^{r} \\leftarrow w_{k}^{r+1}$ in equation (7): it makes the global model a single stream of weights that each client trains in turn, so no server-side computation is needed. Fed-Star's core machinery is the accuracy-based pre-aggregation matrix $M(k,j) = 1 - \\mathrm{Acc}(w^{r,p+1}_j, D_k)/100$ and the normalized combination $w^{r,p+1}_k = \\sum_{j} M(k,j)\\,w^{r,p+1}_j / \\sum_{j} M(k,j)$, which intentionally weights a partner's model more heavily when that model performs worse on the receiving client's local data; the stated purpose is to learn outlier features while keeping common features. The third object is the benchmark itself, where each of the eight clients holds images from one commercial image source, giving naturally heterogeneous class-conditional distributions rather than an artificial non-IID partition.","core_discovery":"The paper's central claim is that when federated clients are real commercial image sources, two proposed training topologies outperform standard federated averaging. Fed-Cyclic propagates a single global model sequentially from client to client, replacing server-side aggregation with a hand-off rule $w_{k+1}^{r} \\leftarrow w_{k}^{r+1}$. Fed-Star runs local updates in parallel and then, inside each round, has every client pre-aggregate all other clients' models using weights $M(k,j)=1-\\mathrm{Acc}(w_j, D_k)/100$, so a client leans most on the models that are least accurate on its own data; a global server then averages the resulting models at the end of the round. On the new eight-source dataset, the authors report that Fed-Star is the most accurate method overall (91.72%) and outperforms every local-only model and baseline on each of the eight clients, while Fed-Cyclic is second on six of eight clients. They conclude that both algorithms converge faster and generalize better under domain shift than FedAvg and RingFed.","pith_inferences":["An extension the paper leaves implicit: because Fed-Star's pre-aggregation weights are complements of training accuracy, the method could be tested with other dissimilarity measures, such as loss on a held-out slice, to see whether the gain comes from the accuracy weighting or from the all-to-all exchange itself.","All reported results freeze the VGG-19 feature extractor; a natural follow-up is end-to-end fine-tuning or a different backbone, which would show whether the algorithm ranking is a property of the methods or of the pretrained features.","Fed-Cyclic's sequential hand-off makes the global model behave like a single network trained on concatenated source mini-batches; that analogy suggests checking for catastrophic forgetting when one source's training dominates a round."],"forward_implications":["Fed-Star offers a personalization benefit: it is reported best on all eight local test sets, so each source gets a model suited to its own data, not just a better shared model.","Fed-Cyclic removes the server's computational role; the server, if used at all, only relays parameters, which cuts the communication bottleneck at the aggregation point.","On this benchmark the proposed methods converge in fewer global rounds than FedAvg (Fed-Star and RingFed use 50, Fed-Cyclic 150, FedAvg 250), so the accuracy gain is not bought with longer training.","The new dataset gives the field a benchmark where client heterogeneity is a real property of the data collection process, so non-IID behavior is built in rather than simulated."],"supporting_citations":[{"why":"FedAvg baseline and the weighted-averaging aggregation the paper compares against and builds its objective on.","marker":"[30]"},{"why":"RingFed baseline whose pre-aggregation scheme Fed-Star extends to all clients with accuracy weights.","marker":"[47]"},{"why":"Office-31 dataset whose categories and domain-shift setup inspire the new eight-source dataset.","marker":"[38]"},{"why":"VGG-19 pretrained on ImageNet supplies the frozen convolutional feature extractor used by every method.","marker":"[40]"},{"why":"The pretrained model library the experiments draw the VGG-19 initialization from.","marker":"[34]"},{"why":"Earlier real-world image dataset for federated learning, used to position the new dataset's novelty.","marker":"[28]"}],"fun_headline_variants":["Federated learning gets a new 8-source image dataset","Fed-Star and Fed-Cyclic beat FedAvg on commercial images","New federated topologies top baselines on domain-shifted data","Fed-Star hits 91.7% accuracy, best on all eight clients"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that ImageNet-pretrained VGG-19 features, frozen in every experiment, transfer equally well to all eight commercial sources, so if some sources are much better served by those fixed features, the reported algorithm ranking could reflect the frozen extractor rather than the federated algorithms.","fun_headline_variants_meta":{"raw":{"variants":["Federated learning gets a new 8-source image dataset","Fed-Star and Fed-Cyclic beat FedAvg on commercial images","New federated topologies top baselines on domain-shifted data","Fed-Star hits 91.7% accuracy, best on all eight clients"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000132,"raw_usage":{"total_tokens":1137,"prompt_tokens":954,"completion_tokens":183,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":570,"completion_tokens_details":{"reasoning_tokens":107}},"tokens_in":570,"tokens_out":183,"duration_ms":2635,"temperature":1.0,"reasoning_tokens":107,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:34:23.661911+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the four algorithms on the same dataset and the same splits with the VGG-19 backbone fine-tuned end-to-end, or with a randomly initialized network in place of the frozen extractor. If Fed-Star's margin over FedAvg and RingFed shrinks or disappears under full training, the claim that the algorithms themselves handle domain shift would not be established independently of the pretrained features.","supporting_citations":[{"cited_title":"RingFed: Reducing Communication Costs in Federated Learning on Non-IID Data","cited_arxiv_id":"2107.08873","evidence_quote":"RingFed baseline whose pre-aggregation scheme Fed-Star extends to all clients with accuracy weights."},{"cited_title":"Adapting visual category models to new domains","cited_arxiv_id":null,"evidence_quote":"Office-31 dataset whose categories and domain-shift setup inspire the new eight-source dataset."},{"cited_title":"Real-World Image Datasets for Federated Learning","cited_arxiv_id":"1910.11089","evidence_quote":"Earlier real-world image dataset for federated learning, used to position the new dataset's novelty."}],"review_version":1}