{"id":"818392c0-ea4b-49c0-b60f-aa0b458fbcf9","arxiv_id":"2506.03954","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"HtFLlib is a unified benchmark and library with 12 datasets, 40 heterogeneous model architectures, and systematic accuracy, convergence, and cost evaluations of 10 HtFL methods.","lead":"This paper releases HtFLlib, a benchmark and open-source library for heterogeneous federated learning that spans image, text, and sensor data. It provides 12 datasets, 40 model architectures, and standardized evaluations of 10 existing HtFL methods, aiming to make comparisons fair and reproducible.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ambiguous 'best results' reporting in Appendix A.1 could make every accuracy ranking an artifact of an unstated max-over-rounds rule; this must be settled before the benchmark's comparative claims are usable.","rationale":"The central claim is that HtFLlib is a comprehensive and robust benchmark for heterogeneous federated learning. The load-bearing condition for that claim is that the accuracy comparisons are fair and reproducible. The weakest link is the unstated reporting convention in Appendix A.1: 'We report the best results along with error bars.' If this is max-over-rounds accuracy, all rankings are potentially biased by noise and by the arbitrary stopping point, and the benchmark's headline insights could be artifacts of the reporting rule. The released code makes this concern directly checkable, which is why it is the most useful single concern to settle. The reader's flagged assumptions about K=512 and the smallest auxiliary model are also legitimate design-bias concerns, but they are stated choices with a rationale and a sensitivity study; the reporting convention is more load-bearing because it silently affects every accuracy table. I am not arguing for rejection: the library, datasets, and code are concrete contributions, and the ambiguity is fixable by reporting final-round or stable-tail accuracy together with a precise definition. Since the reader's verdict is already CONDITIONAL and my concern is one of the conditions, the verdict should remain UNCHANGED.","tokens_in":22007,"tokens_out":5752,"duration_ms":57775,"concrete_test":"Recompute Table 1 and Table 10 from the released HtFLlib code using (a) final-round test accuracy and (b) mean test accuracy over the last 10 communication rounds, instead of whatever 'best results' currently means. Also require the authors to state explicitly whether the reported value is max-over-rounds, final-round, or best-of-seeds, and how the +/- error bars are computed. If any method's rank changes by more than one position, or if the reported FedTGP/FedKTL advantages shrink below the error bars, the benchmark's qualitative insights are not robust to the reporting convention.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that HtFLlib provides a robust, standardized benchmark for heterogeneous federated learning. The load-bearing condition is that the reported accuracy numbers support fair and reproducible method comparisons. Appendix A.1 says: 'Each experiment ... is repeated three times ... We report the best results along with error bars.' This is ambiguous in a way that matters. If 'best results' means the maximum test accuracy over the 1000 communication rounds, rather than the accuracy at the final round or the mean over a stable tail, then every accuracy table in the paper reports an order statistic of a training curve. In federated learning, max-over-rounds accuracy rewards methods that spike or oscillate and penalizes stable methods that converge to a slightly lower but reproducible value. It is also sensitive to the arbitrary choice of 1000 rounds. The paper's headline insights—FedTGP 'outperforms all baselines in most cases,' prototype-sharing methods 'perform relatively poorly' on text and sensor signals, mutual distillation 'generally outperforms' partial parameter sharing—are exactly the kind of rankings that can change when the reporting convention changes. The paper never states whether the reported number is best-round accuracy, final-round accuracy, or best-of-three-seeds, nor how the error bars are computed across seeds. Without this, the reproducibility and fairness of the benchmark's comparative conclusions are not established as written.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces HtFLlib, an open-source heterogeneous federated learning (HtFL) library and benchmark. The submission contributes 12 datasets across image, text, and sensor modalities; 19 model-heterogeneity groups covering 40 architectures; 10 implemented HtFL methods organized into partial-parameter-sharing, mutual-distillation, and prototype-sharing categories; and experiments measuring accuracy, convergence, communication cost, and computation cost. The paper's central claim is that HtFLlib is the first comprehensive HtFL benchmark and that its evaluations support conclusions such as the overall strength of FedTGP on image tasks, the weakness of feature-space prototype sharing on text and sensor data, and the robustness of FedMRL to increased model heterogeneity.","tokens_in":22280,"tokens_out":8641,"duration_ms":73012,"significance":"If the quantitative comparisons are trustworthy, HtFLlib is a useful community asset: it unifies dataset/model partitions that were previously scattered across papers, spans modalities and a medical black-box fine-tuning scenario that existing HtFL evaluations ignore, and releases a modular codebase with ten methods. The cost measurements and the explicit discussion of convergence are also valuable. The main risk is that the benchmark's headline rankings are built on an under-specified accuracy-reporting convention and on default calibration choices (K=512, smallest auxiliary model) whose sensitivity is not established. Since the benchmark's purpose is fair and reproducible comparison, these issues are load-bearing rather than cosmetic.","major_comments":[{"comment":"The sentence 'Each experiment ... is repeated three times ... We report the best results along with error bars' is ambiguous in a way that affects every accuracy table and figure in the paper. If 'best results' means the maximum test accuracy over the 1000 communication rounds, the tables report order statistics of training curves, which can favor methods that spike or oscillate and penalize stable methods, and the 1000-round horizon becomes an arbitrary free parameter. If 'best results' means best-of-three seeds, then error bars computed across seeds do not describe the reported quantity. Please state precisely whether the reported number is final-round accuracy, mean over a stable tail, or best-round accuracy, how the three seeds are combined, and how error bars are computed; if max-over-rounds was used, re-analyze the comparisons with a reporting-convention-insensitive metric or demonstrate that the qualitative rankings are unchanged.","section":"Appendix A.1"},{"comment":"The default feature dimension K=512 is fixed in §3.1.2, yet the feature-dimension analysis reports that most methods improve as K grows from 64 to 256, achieve their best at K=256, and that 'a very high K may lead to a performance drop' for FD, FedProto, and FedKTL. Because all main accuracy tables use K=512, this default may systematically disadvantage exactly the prototype-sharing methods that the paper later characterizes as weak on text and sensor data. Please clarify whether K=512 was included in the Fig. 4 sweep, and either choose the default K based on a method-neutral criterion or show that the rankings in Tables 1, 2, 4, 7, and 10 are unchanged at K=256 versus K=512.","section":"§4.2 and Fig. 4"},{"comment":"Selecting the smallest model in each heterogeneous group as the auxiliary model for FML, FedKD, and FedMRL is a design choice with no sensitivity analysis. Since the auxiliary model's capacity directly affects both the accuracy and the communication cost of mutual-distillation methods, and since the smallest model can be far weaker than the largest in groups such as HtFEimg8 or HtFEtxt6, this choice can bias the category-level comparisons in §4.1.1, §4.3, and §4.4. Please justify this protocol or test at least one alternative auxiliary model to show that the reported category-level conclusions are robust.","section":"Appendix A.4"},{"comment":"Several categorical claims are made for differences that are within the reported error bars. For example, Table 1 reports FedTGP 46.94±0.12 versus FedKTL 46.94±0.23 on Cifar100 Dirichlet, and Table 4 reports FedTGP 43.17±0.23 versus FedKTL 43.16±0.82 with 50 clients, yet §4.1.1 states that FedTGP 'outperforms all baselines in most cases.' Please add pairwise significance tests or explicitly avoid ranking methods whose differences are smaller than the combined error bars; otherwise the benchmark's qualitative takeaways are not statistically supported.","section":"Tables 1 and 4 / §4.1.1"}],"minor_comments":[{"comment":"The FD row reads '8 7.73±0.17' and should read '87.73±0.17'.","section":"Table 8"},{"comment":"The text says FedProto requires 540, 533, and 570 rounds for α=0.01, 0.1, and 1, but the table lists 540, 533, 570, and 369 for α=0.01, 0.1, 0.5, and 1; please correct the sentence and include α=0.5.","section":"§4.1.4 and Table 3"},{"comment":"The dataset is listed as 'KV ASIR' in Section 3.2.2 and as 'KVASIR' in Table 6; standardize the spelling to 'Kvasir'.","section":"§3.2.2 and Table 6"},{"comment":"These figures show sweeps or training curves without error bars; if error bars were omitted for clarity, state this in the captions and point to where the full numbers are available.","section":"Figures 4, 6, 7, and 8"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the paper is a strong candidate for a benchmark-track publication if the authors can resolve the reporting-convention ambiguity and the K=512 calibration issue. I did not find evidence of deliberate bias, but the concentration of highlighted methods (FedTGP, FedKTL) from the same group makes it especially important that the comparative claims survive a convention-insensitive re-analysis. I recommend major revision rather than rejection because the assets (datasets, model zoo, code, medical scenario) are genuinely useful and the fixes are within scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know right away. HtFLlib is one of the more useful benchmark-plus-library contributions I've seen for heterogeneous federated learning, and the accuracy numbers that drive its comparative claims rest on an ambiguous 'best results' convention in Appendix A.1 that needs to be pinned down.\n\nThe things it does well are real. It unifies 12 datasets, 40 model architectures, and 10 HtFL methods across image, text, and sensor modalities in one extensible codebase, with a sensible modular API. The experiments cover label skew, feature shift, and real-world settings, and go beyond accuracy to report convergence, communication, and computation costs. The medical black-box fine-tuning experiment with pre-trained heterogeneous models is a nice addition that most prior benchmarks don't touch. Code and data are released, and the protocol of three seeds with error bars is standard and reproducible.\n\nThe soft spots are proportionate. The 'best results' ambiguity is the load-bearing one. If that phrase means max test accuracy over the 1000 communication rounds, then every table is an order statistic, and methods that oscillate or spike gain an unfair advantage. The error bars wouldn't be meaningful either. The paper doesn't say whether it's best-of-three seeds, final-round, or best-round, and the rankings of FedTGP, prototype-sharing, and mutual-distillation all depend on it. This has to be stated explicitly.\n\nSecond, there is no no-collaboration baseline in the vast majority of experiments. Only the medical table includes a pre-trained baseline. Without a 'local-only' row, the benchmark can't quantify the actual benefit of HtFL collaboration over simply training your own model. Adding that is cheap and makes the benchmark more informative.\n\nThird, the 'first HtFL benchmark' claim is slightly too strong given that FederatedScope is cited as a heterogeneity-supporting FL platform. The authors should differentiate their scope and design explicitly, not just assert novelty.\n\nMinor point: the choice of K=512 and the smallest-model-as-auxiliary rule could interact with method performance, but I don't see an obvious bias. A quick sensitivity analysis with one or two other K values would answer the question.\n\nNone of this undermines the central contribution. This is a solid benchmark paper with real practical value, and the issues are fixable. It absolutely deserves peer review rather than desk rejection, and I'd happily bring it to a reading group. Once the reporting convention is clarified and the local-only baseline is added, the comparisons become trustworthy.","headline":"A useful, well-executed HtFL benchmark whose accuracy rankings depend on an ambiguous 'best results' reporting convention that must be clarified before the comparisons are fully trustworthy.","tokens_in":22821,"tokens_out":3853,"would_cite":true,"duration_ms":36757,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces HtFLlib, a unified library and benchmark that lets heterogeneous federated learning methods be compared across 12 datasets, 40 model architectures, and 10 representative methods.","keywords":["heterogeneous federated learning","federated learning benchmark","model heterogeneity","data heterogeneity","knowledge distillation","prototype sharing","mutual distillation","partial parameter sharing"],"falsifier":"Re-run the Cifar100 Dirichlet benchmark with $K=64$ and with $K=1024$, and swap the auxiliary model from the smallest to the largest architecture in the group; if relative rankings such as FedTGP over FedKD change materially, the claimed method-neutral calibration fails.","tokens_in":21808,"feed_emoji":"📊","tokens_out":5641,"duration_ms":51258,"temperature":0.7,"pith_summary":"The paper argues that heterogeneous federated learning (HtFL), where clients train different model architectures without sharing them, has grown too fast to be evaluated fairly because each research group uses its own datasets, model splits, and implementations. It introduces HtFLlib, an open library and benchmark that standardizes these choices across image, text, and sensor-signal modalities, with 12 datasets, 40 model architectures organized into 19 heterogeneous model groups, and implementations of 10 HtFL methods. The stated goal is to make fair, reproducible comparisons possible and to reveal where each method family, namely partial parameter sharing, mutual distillation, and prototype sharing, does and does not work. If the benchmark is adopted, new HtFL methods can be validated against a common testbed rather than against self-chosen setups.","feed_headline":"HtFLlib puts 10 heterogeneous FL methods on one testbed","feed_subtitle":"Twelve datasets, 40 model architectures, and 10 methods in one unified benchmark.","key_machinery":"The load-bearing machinery is the standardized heterogeneity protocol. Each experiment assigns clients to one of 19 heterogeneous model groups denoted HtFE, HtC, or HtM with a degree $X$, where larger $X$ means greater architectural spread, and forces compatibility by inserting an average pooling layer so every model produces a shared feature dimension $K=512$ before its classifier head. Knowledge transfer is carried by lightweight carriers, namely partial model parameters, a small homogeneous auxiliary model trained by mutual distillation, or class-wise prototypes, depending on the method family. This setup is what makes the comparison fair in principle: same data partitions, same model groups, same feature dimensions, and the same training protocol for all methods.","core_discovery":"The paper's central claim is that HtFLlib is the first heterogeneous federated learning benchmark that unifies data heterogeneity, model heterogeneity, and method implementations in one extensible framework. It defines three data-heterogeneity scenarios (label skew, feature shift, and real-world), four model-heterogeneity families (heterogeneous feature extractors, heterogeneous classifiers, fully heterogeneous models, and per-modality groups), and a modular codebase that re-implements ten representative HtFL methods under identical training steps, batch sizes, and evaluation protocols. The benchmark's headline results show that prototype-sharing methods such as FedTGP lead on image tasks, mutual distillation methods such as FedKD lead on sensor signals, and generator-based knowledge transfer such as FedKTL helps under strong data heterogeneity, but these rankings vary across modalities and heterogeneity levels.","pith_inferences":["If HtFLlib becomes the default testbed, the field's implicit overfitting to favorable setups should diminish, because the benchmark fixes data partitions, model groups, and feature dimensions across all comparisons.","The benchmark's own results suggest a testable extension: methods that align in logit space degrade less on sensor signals than prototype-alignment methods, so a hybrid carrier transmitting both logits and temporally aware prototypes might combine the strengths of both families.","The medical post-training experiment could be extended to map the limits of black-box HtFL, asking at what participation ratio, round budget, and data size per client the collaboration stops improving private pre-trained models."],"forward_implications":["A new HtFL method can be positioned against ten existing baselines on a single codebase, making cross-paper accuracy claims directly comparable.","Method rankings that hold across image, text, and sensor-signal modalities, such as the strength of prototype sharing on images and its weakness on sensor signals, can guide practitioners toward the right method family for their domain.","The reproducible cost tables, in communication megabytes and server and client seconds per round, let researchers see that lightweight prototype methods are not always cheap once server-side training and server-side computation are counted.","The black-box pre-trained medical scenario shows that HtFL can improve already-trained heterogeneous models, opening a post-training use case beyond end-to-end federated training."],"supporting_citations":[{"why":"Defines FedProto, the prototype-sharing method implemented and evaluated as one of the ten benchmark baselines.","marker":"[47]"},{"why":"Defines FedGH, the partial-parameter-sharing method whose generalized global header is benchmarked here.","marker":"[57]"},{"why":"Defines FedMRL, the mutual distillation method benchmarked across image, text, and sensor-signal settings.","marker":"[58]"},{"why":"Defines FedTGP, the trainable global prototype method that the benchmark reports as the strongest image-task performer.","marker":"[62]"},{"why":"Defines FedKTL, the generator-based knowledge transfer method evaluated for image tasks and medical post-training.","marker":"[63]"},{"why":"Defines FD, the federated distillation baseline that shares logit-space knowledge in the benchmark.","marker":"[21]"},{"why":"Defines LG-FedAvg, the partial-parameter-sharing baseline that aggregates classifier head parameters.","marker":"[30]"},{"why":"Defines FedGen, the data-free knowledge distillation baseline benchmarked here.","marker":"[70]"},{"why":"Defines FML, the federated mutual learning baseline that shares output logits through an auxiliary model.","marker":"[45]"},{"why":"Defines FedKD, the communication-efficient distillation baseline that also aligns intermediate feature vectors.","marker":"[52]"}],"fun_headline_variants":["HtFLlib: first unified benchmark for heterogeneous FL","HtFLlib: 10 methods, 40 models, 12 datasets in one library","HtFLlib benchmark covers 12 datasets, 40 models, 10 methods","HtFLlib unifies heterogeneous FL benchmarking"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark assumes its standardizing choices, in particular forcing every client model to output a shared $K=512$ feature vector and choosing the smallest model in each group as the small shared distillation model, do not systematically favor or harm any method family.","fun_headline_variants_meta":{"raw":{"variants":["HtFLlib: first unified benchmark for heterogeneous FL","HtFLlib: 10 methods, 40 models, 12 datasets in one library","HtFLlib benchmark covers 12 datasets, 40 models, 10 methods","HtFLlib unifies heterogeneous FL benchmarking"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000812,"raw_usage":{"total_tokens":3599,"prompt_tokens":1025,"completion_tokens":2574,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":641,"completion_tokens_details":{"reasoning_tokens":2495}},"tokens_in":641,"tokens_out":2574,"duration_ms":18508,"temperature":1.0,"reasoning_tokens":2495,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:50:00.616659+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the Cifar100 Dirichlet benchmark with $K=64$ and with $K=1024$, and swap the auxiliary model from the smallest to the largest architecture in the group; if relative rankings such as FedTGP over FedKD change materially, the claimed method-neutral calibration fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines FedProto, the prototype-sharing method implemented and evaluated as one of the ten benchmark baselines."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines FedGH, the partial-parameter-sharing method whose generalized global header is benchmarked here."},{"cited_title":"Federated Model Heterogeneous Matryoshka Representation Learning","cited_arxiv_id":"2406.00488","evidence_quote":"Defines FedMRL, the mutual distillation method benchmarked across image, text, and sensor-signal settings."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines FedTGP, the trainable global prototype method that the benchmark reports as the strongest image-task performer."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines FedKTL, the generator-based knowledge transfer method evaluated for image tasks and medical post-training."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines FedGen, the data-free knowledge distillation baseline benchmarked here."}],"review_version":1}