{"id":"2cbc9ec7-65a2-47cd-9ddb-3456015d2528","arxiv_id":"2412.15728","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"fluke is an open-source Python package that simulates federated learning locally, letting researchers prototype new FL algorithms by defining just client and server behaviors.","lead":"fluke is a new open-source Python framework that simulates federated learning on a single machine for quick algorithm prototyping. This paper argues that adding a new federated algorithm requires only a client and a server class, lowering the barrier to experimentation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'minimal overhead' claim is unquantified and the paper's own tutorial requires a third algorithm class; overhead for algorithms that deviate from the vanilla FedAvg loop is not demonstrated.","rationale":"The reader's weakest assumption focuses on the fidelity of bundled algorithm implementations, which is indeed a significant concern for the out-of-the-box benchmarking use case. My stress-tester concern is slightly different: the paper's headline promise about low overhead for adding new algorithms is not substantiated, and the walkthrough reveals that the abstraction is only 'minimal' for FedAvg-like synchronous centralized protocols. This is a load-bearing issue because it is the paper's central argument for the framework's existence. Both concerns are addressable with additional empirical evidence (overhead measurements and cross-verification against original implementations), so the conditional verdict is appropriate. I do not believe the paper should be rejected, as the open-source artifact and tutorial provide a concrete starting point, but the current evidence is too thin to justify a stronger accept.","tokens_in":8710,"tokens_out":5272,"duration_ms":45984,"concrete_test":"Re-implement SCAFFOLD (or FedProx, which requires a proximal penalty) in fluke using only the documented tutorial. Measure (a) number of classes defined, (b) number of base-class methods overridden, (c) lines of code, and (d) wall-clock time to first successful run. Compare with a from-scratch PyTorch implementation and with Flower. If the fluke version requires overrides beyond client/server classes (e.g., overriding server.fit for server-side state), the 'simply requires' claim is overstated. Separately, run the same MNIST benchmark with 10 seeds using fluke's default FedAvg config and compare the mean accuracy curve to the original FedAvg paper or official repository; a shift of more than 1-2 accuracy points would indicate the bundled implementation is not validated for out-of-the-box benchmarking.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that adding a new FL algorithm requires only defining client and/or server classes with minimal overhead. The paper's own tutorial undercuts this: 'The final class to define is the one representing the whole algorithm' (inheriting from CentralizedFL), so at least three new classes are needed, plus a YAML config and a specific file layout. More importantly, the base classes bake in a synchronous, server-broadcast global-model protocol. For algorithms that deviate (e.g., SCAFFOLD's client-side control variates, FedOpt's server-side optimizer state, personalized local models, or asynchronous updates), the user must also override server.fit, broadcast_model, receive_model, or finalize, reimplementing orchestration logic. The paper never quantifies minimal overhead (lines of code, number of overridden methods, time-to-first-experiment), and Figure 2 is a single unseeded run with default hyperparameters and no comparison to original implementations. If a researcher must rewrite the coordination loop for non-FedAvg algorithms, the framework's core value proposition is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces fluke, an open-source Python package for simulating centralized federated learning (FL) experiments on a single machine. The author argues that existing FL frameworks are too rigid or have too steep a learning curve for researchers who want to prototype new algorithms, and claims that fluke can be used out of the box for benchmarking and extended with new algorithms by defining client and/or server classes with minimal overhead. The paper describes the package architecture, the command-line interface, a tutorial for adding a new algorithm, and presents an example benchmark of several FL algorithms on MNIST. The manuscript also states that fluke explicitly does not aim to compete with full-featured frameworks and is scoped to a synchronous, centralized, simulated setting.","tokens_in":8884,"tokens_out":3517,"duration_ms":31544,"significance":"If the central usability claims are correct, fluke could be a valuable tool for FL researchers by lowering the barrier to prototyping and improving reproducibility of baseline comparisons. The paper ships open-source code, documentation, tutorials, and a set of pre-implemented algorithms and datasets, which are concrete and inspectable artifacts. However, the significance assessment is conditional because the paper does not quantify the claimed 'minimal overhead,' does not validate the faithfulness of its bundled algorithm implementations, and does not compare against existing frameworks such as Flower. These omissions leave the central value proposition unverified, although the design is plausible and the modular architecture appears sensible. The paper is likely useful as a software announcement, but its scientific claims require additional evidence.","major_comments":[{"comment":"The paper's central claim, stated in the introduction and features list, is that implementing a new algorithm 'simply requires the definition of the client and/or the server classes.' This is contradicted by the tutorial in the same section, which requires a third class inheriting from fluke.algorithms.CentralizedFL (or PersonalizedFL), plus a YAML configuration file and a specific file layout for the CLI to discover the algorithm. The paper does not provide any quantitative measure of overhead, such as lines of code, number of methods overridden, or time-to-first-experiment, nor does it compare the effort against reimplementation from scratch or against another framework. The 'minimal overhead' claim is therefore unsupported by the evidence in the manuscript.","section":"Implementing a new algorithm"},{"comment":"The benchmarking use case, illustrated in Figure 2, shows accuracy curves for FedAvg, SCAFFOLD, FedExP, FedDyn, and FedOpt run with default hyperparameters on MNIST. The figure appears to report a single unseeded run with no error bars, and the paper provides no comparison of these curves against the results reported in the original papers. Because the framework's value for benchmarking depends on the bundled implementations being faithful to their published descriptions, the absence of validation undermines the benchmarking claim. The paper should either report aggregate statistics over multiple seeds with hyperparameter details or cite external evidence that the implementations reproduce known results.","section":"Benchmarking"},{"comment":"The framework is explicitly built around a synchronous, centralized FedAvg-like protocol, as shown in Algorithm 1. The tutorial states that the server's fit method 'should not need to be overridden' only 'as long as the protocol follows the usual one in centralized FL.' For algorithms that deviate from this protocol—such as FedOpt's server-side optimizer state, SCAFFOLD's client control variates, personalized local models, or asynchronous updates—the user must reimplement orchestration logic by overriding fit, broadcast_model, receive_model, or finalize. The paper does not scope its 'minimal overhead' claim to FedAvg-conforming algorithms, and it does not discuss the additional effort for these common cases. This is a load-bearing gap because the paper motivates fluke by the need to prototype a wide range of new FL algorithms, not merely FedAvg variants.","section":"fluke and Algorithm 1"},{"comment":"The paper motivates fluke by asserting that existing frameworks are 'not easy to extend' and that Flower, while extensible, has non-trivial configuration and customization. Yet no comparative evaluation against Flower or any other framework is provided. The paper does not include a user study, code complexity metrics, or any direct comparison of the effort required to implement the same algorithm in fluke versus in an existing framework. Without such evidence, the claimed advantage over the status quo remains an assertion rather than a demonstrated property.","section":"Use cases"}],"minor_comments":[{"comment":"There are several typographical errors, including 'minimazing' (should be 'minimizing'), 'fluke raises from' (should be 'arises'), 'ant in general' (likely 'and in general'), 'learing' (should be 'learning'), and 'oof-the-shelf' (should be 'off-the-shelf'). These should be corrected throughout.","section":"Features list"},{"comment":"The code block for running a custom algorithm contains '----config' (four dashes) instead of '--config', and the word 'console' appears in multiple code blocks as if it were part of the code. The wording 'The command must be run in the directory where my algorithm.py is located' should clarify whether all custom modules must reside in the current working directory or whether Python path handling is allowed.","section":"CLI examples"},{"comment":"Figure 1 is not referenced in the text before the 'Architecture overview' section, and the module names in the figure use abbreviations ('comm','algo','eval') that are only defined later in the prose. Adding an explicit reference to Figure 1 in the text and expanding the abbreviations would improve readability.","section":"Figure 1"},{"comment":"The paper states that the tutorial for adding a custom algorithm is available online, but the manuscript itself says 'for space reasons, the following steps do not include all the necessary details.' It would be helpful to include at least a minimal code listing or pseudocode for a non-trivial custom algorithm (e.g., FedOpt) to demonstrate the claimed low overhead, rather than deferring entirely to external documentation.","section":"Documentation and tutorials"},{"comment":"The reference list contains formatting inconsistencies, such as 'Https://github.com/zalandoresearch/fashion-mnist' with a capital H, and several repository citations that list only 'et al.' without full author names. These should be normalized to the journal's style.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a short system/software paper, and its central claims are about usability and extensibility. For a journal venue, the lack of quantitative evidence for these claims is a significant concern; the paper would need to add either a user study, code metrics, or a comparison with at least one existing framework such as Flower. The example benchmark's lack of seed information and comparison to original results is also problematic if the benchmarking use case is part of the advertised contribution. I believe the issues are addressable within the scope of a revision, so I recommend major_revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this about the fluke paper: it's a real, open-source Python package for single-machine FL simulation, with a CLI, config system, a bundle of datasets, and a decent set of implemented algorithms. The paper's central claim is that adding a new algorithm 'simply requires the definition of the client and/or the server classes.' That claim is plausible, but the paper never quantifies it, and the stress-test note is right: the tutorial actually requires three new classes (client, server, and the algorithm entry point), plus a YAML config. For algorithms that deviate from the synchronous FedAvg loop—SCAFFOLD, FedOpt, personalized methods—you'll likely be overriding server.fit, broadcast_model, or receive_model, which means reimplementing orchestration logic. So the 'minimal overhead' claim is not supported as stated.\n\nWhat the paper does well: the code is public, documented, and structured to resemble the algorithms' definitions; the CLI and config separation make off-the-shelf benchmarking easy; and the author is honest about the scope—this is a utility for prototyping, not a competitor to Flower or FLARE. The paper also cites many relevant algorithms and frameworks, and the citation pattern looks fine.\n\nThe soft spots are real but addressable. There is no user study, no lines-of-code comparison against Flower, and no time-to-first-experiment measure. Figure 2 is a single unseeded run with default hyperparameters and no error bars, so it can't be independently assessed. More importantly, there is no verification that the bundled implementations match the original papers' results. The author says the algorithms are 'regularly updated' but gives no fidelity checks. These missing pieces don't sink the paper's value as a software announcement, but they do limit its usefulness as a benchmark tool.\n\nMy verdict: this is a useful resource for researchers doing FL prototyping on a single machine. It deserves a serious referee, but the paper needs revision before acceptance. I'd ask for a quantitative measure of overhead (e.g., lines of code or methods overridden for a non-FedAvg algorithm), an example with a fixed seed and error bars, and at least a sanity check that one or two bundled algorithms reproduce results near the original papers. The code is there, so these are easy asks. I personally wouldn't cite it in my own work yet, but I'd bring it to a reading group as a example of a software paper that is honest about its niche.","headline":"fluke is a clean, genuinely useful FL prototyping package with a solid algorithm zoo, but the 'minimal overhead' claim is unquantified and the paper's own tutorial shows you need more than just client/server classes.","tokens_in":9373,"tokens_out":1951,"would_cite":false,"duration_ms":18785,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding a new federated learning algorithm in fluke requires defining only the client and/or server classes.","keywords":["federated learning","software framework","prototyping","benchmarking","simulation","non-IID data","Python package","algorithm extension"],"falsifier":"Run the same algorithm in fluke and in its original implementation under identical seeds, data partitions, and hyperparameters on MNIST and compare the accuracy curves; if the curves or final values diverge substantially, the framework's benchmarking validity collapses. A cheaper check is to re-run the Figure 2 configuration with several seeds and see whether the reported ordering among methods is stable.","tokens_in":8505,"feed_emoji":"🧪","tokens_out":9652,"duration_ms":76037,"temperature":0.7,"pith_summary":"fluke is a Python package for simulating federated learning on a single machine, aimed at researchers who want to prototype and benchmark algorithms without reimplementing the federated loop. The paper's central claim is that adding a new algorithm is a small, localized coding task: a researcher defines a client class and/or a server class, and the framework supplies the rest, including model broadcast, client selection, aggregation, evaluation, and logging. Because the base classes already behave like the standard FedAvg method, only the parts of an algorithm that actually differ need to be overridden. If this works as described, fluke lowers the cost of testing new FL ideas and makes experimental setups reproducible by separating experiment configuration from algorithm configuration.","feed_headline":"fluke: new federated algorithms need only client and server classes","feed_subtitle":"Researchers can run off-the-shelf benchmarks with one command and publish their algorithm as a runnable extension.","key_machinery":"The load-bearing structure is the class hierarchy in the fluke package: fluke.algorithms.CentralizedFL (or PersonalizedFL) as the entry point, plus fluke.client.Client and fluke.server.Server, whose defaults implement a FedAvg-style protocol. A Channel class simulates the server–client communication link and logs overhead, while YAML configuration files separate experiment settings (dataset, distribution, rounds, seed) from algorithm settings (model, hyperparameters). This combination lets a novel algorithm be expressed as an override of fit, send_model, receive_model, aggregate, and related methods, with the surrounding simulation handled by the base classes.","core_discovery":"The paper's central claim is that the design of fluke reduces the work of adding a federated learning algorithm to the definition of two kinds of classes. The Client class owns local training and model exchange; the Server class owns client selection, broadcasting, and aggregation. A third class, the algorithm entry point, mostly declares which client and server classes to use. The framework simulates communication through a Channel object, and all other machinery, including data loading, non-IID splits, model definitions, evaluation, and logging, is provided. The central design bet is that FL algorithms are best expressed server-side and client-side, so researchers can focus on the learning components rather than on infrastructure.","pith_inferences":["Counting the overridden methods needed to implement a published variant, such as a proximal-term or extrapolation FL method, would provide a quantitative check of the minimal overhead claim.","A companion validation study that compares each bundled baseline against its original implementation would make fluke a reference benchmark rather than a convenience; the current paper does not include that check.","The experiment and algorithm configuration split suggests a simple batch-comparison workflow that could be extended to auto-generate benchmark tables for papers.","The framework's current single-machine, synchronous design limits its use to algorithm-level questions; topology and communication studies would need the multi-device support listed as future work."],"forward_implications":["A researcher can benchmark a new idea against several existing FL algorithms by writing one experiment config and swapping algorithm config files, without modifying code between runs.","Publishing a fluke-based algorithm gives other researchers a runnable implementation plus configuration, which supports the paper's reproducibility goal.","The built-in datasets and non-IID distribution functions remove the need to implement data partitioning, so different papers can compare under the same data conditions.","Because only client and server classes change, a new algorithm can be tested while keeping the same evaluation, logging, and channel-overhead tracking as the baselines."],"supporting_citations":[{"why":"Supplies the main comparison point: an existing extensible FL framework whose configuration is still non-trivial, motivating fluke's design.","marker":"[6]"},{"why":"Represents one of the powerful existing FL frameworks that the paper argues are not easy to extend.","marker":"[5]"},{"why":"Represents an industrial-grade FL platform that is not oriented toward fast research prototyping.","marker":"[36]"},{"why":"Represents another existing FL framework the paper lists as difficult to extend.","marker":"[44]"},{"why":"Represents an open-source FL library that the paper contrasts with fluke's prototyping focus.","marker":"[18]"},{"why":"Defines FedAvg, whose behavior the base client and server classes imitate by default.","marker":"[38]"},{"why":"Supplies the MNIST dataset used in the demonstration benchmark in Figure 2.","marker":"[27]"}],"fun_headline_variants":["fluke: FL algorithms need only client and server classes","Two classes per federated algorithm: fluke handles the rest","fluke: skip infrastructural overhead, just write client and server","fluke: prototype FL algorithms with just two classes","Client and server classes: fluke packages the rest"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that fluke's bundled implementations of published FL algorithms faithfully reproduce the original methods, so that default-settings runs yield valid comparisons; the paper does not check this against original sources.","fun_headline_variants_meta":{"raw":{"variants":["fluke: FL algorithms need only client and server classes","Two classes per federated algorithm: fluke handles the rest","fluke: skip infrastructural overhead, just write client and server","fluke: prototype FL algorithms with just two classes","Client and server classes: fluke packages the rest"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0005,"raw_usage":{"total_tokens":2376,"prompt_tokens":806,"completion_tokens":1570,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":422,"completion_tokens_details":{"reasoning_tokens":1488}},"tokens_in":422,"tokens_out":1570,"duration_ms":12153,"temperature":1.0,"reasoning_tokens":1488,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:07:54.726670+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same algorithm in fluke and in its original implementation under identical seeds, data partitions, and hyperparameters on MNIST and compare the accuracy curves; if the curves or final values diverge substantially, the framework's benchmarking validity collapses. A cheaper check is to re-run the Figure 2 configuration with several seeds and see whether the reported ordering among methods is stable.","supporting_citations":[{"cited_title":"J.; Topal, T.; Mathur, A.; Qiu, X.; Fernandez-Marques, J.; Gao, Y .; Sani, L.; Li, K","cited_arxiv_id":null,"evidence_quote":"Supplies the main comparison point: an existing extensible FL framework whose configuration is still non-trivial, motivating fluke's design."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Represents one of the powerful existing FL frameworks that the paper argues are not easy to extend."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Represents an industrial-grade FL platform that is not oriented toward fast research prototyping."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Represents another existing FL framework the paper lists as difficult to extend."},{"cited_title":"J.; Edwards, B.; Pati, S.; Riv- iera, W .; Sharma, M.; Moorthy, P","cited_arxiv_id":null,"evidence_quote":"Represents an open-source FL library that the paper contrasts with fluke's prototyping focus."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines FedAvg, whose behavior the base client and server classes imitate by default."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the MNIST dataset used in the demonstration benchmark in Figure 2."}],"review_version":1}