{"id":"4bbeaf4a-3901-4a91-947c-ace391ec1201","arxiv_id":"2505.13576","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"FlexFed combines offline local training with performance-based retention of rare-class samples and reports reduced catastrophic forgetting in federated human activity recognition.","lead":"FlexFed is a federated learning method that lets phones train locally while disconnected and keeps only rarely seen activity samples in memory, which the authors say stops the shared model from forgetting those activities. The paper reports better accuracy and less forgetting than two existing methods in a simulated human activity recognition setting, though no code or numerical efficiency tables are included.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"FlexFed's update gate and its forgetting metric both use the same client-held test set, so the reported forgetting gains may be an artifact of the accept/reject policy rather than of better retention.","rationale":"I agree with the reader's weakest assumption that the reliance on per-client labeled test data is a serious limitation, but I would sharpen it: the same D_test_k is used both as the decision signal for accepting/rejecting local updates and as the evaluation set for the proposed forgetting metric in Eq. (5). This makes the central empirical claim less secure than the paper suggests, because the low forgetting values and high stability could partly be produced by the accept/reject policy itself. That said, I do not see a logical contradiction that proves FlexFed is wrong; the issue is that the current evaluation cannot distinguish policy-driven apparent stability from genuine knowledge retention. The authors could address this by evaluating on an independent test partition, by ablating the accept/reject gate, and by comparing against baselines that also have access to per-client test labels. Without code or numerical tables, the results remain unverifiable, but the central idea is plausible enough that a conditional acceptance with mandatory re-evaluation is the appropriate verdict. The paper would be more convincing if it also clarified the α definition contradiction in Section V-B (text says higher α means poorer performance while α = P(θ_r, D_test_k) suggests higher α means better performance) and reported statistics across seeds rather than only across model architectures.","tokens_in":11731,"tokens_out":10262,"duration_ms":114623,"concrete_test":"Re-run the FlexFed experiments with the Section V-C accept/reject gate disabled (always send the locally trained model, keeping the adaptive memory allocation and offline training unchanged), and recompute the Eq. (5) forgetting metric for FlexFed, REFL, and MIFA. As a cleaner check, hold out a per-client test partition that is never used in the accept/reject step or the α computation, and evaluate Eq. (5) on that independent partition. If the FlexFed advantage in Figures 7–8 disappears or shrinks materially, the reported forgetting mitigation is driven by the test-set-based update filter rather than by the proposed retention and offline-training mechanisms.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The FlexFed workflow in Section V-C accepts a local or offline update only when P(θ_new; D_test_k) ≥ P(θ_stored; D_test_k); otherwise the old model is sent or stored. The proposed forgetting metric in Eq. (5) is computed on the same per-client labeled test data D_test_k. This couples the mechanism to the measurement: a client can always fall back on a stored model that was selected precisely because it scored well on that test set. REFL and MIFA do not receive this labeled-test-data signal, so the comparison is not apples-to-apples. Moreover, the paper's claim of 'stability above 95%' and lower F values in Figures 7–8 may reflect the accept/reject gate filtering out any local update that lowers per-class accuracy, rather than improved representation or memory retention. The global model can still forget because discarded updates are only local contributions, but the reported metric cannot cleanly separate the policy effect from a genuine learning effect. This is compounded by the practical assumption that every client has a labeled test set every round; if D_test_k is absent, unrepresentative, or imbalanced, the α computation, the update accept/reject step, and the forgetting metric all lose their meaning.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FlexFed, a federated learning framework for human activity recognition in pervasive computing environments. FlexFed combines offline local training during idle periods without connectivity with performance-adaptive memory allocation that retains more samples from infrequent classes when the received global model performs poorly on a client-held test set. The paper also introduces a round-wise forgetting metric (Eq. 5) based on per-client per-class accuracy drops, and reports experiments on a synthetic streaming version of WISDM with six HAR model architectures, comparing FlexFed against REFL and MIFA. The main claims are that FlexFed mitigates catastrophic forgetting, improves FL efficiency by 10-15%, and achieves faster, more stable convergence, especially for under-represented classes.","tokens_in":11960,"tokens_out":5577,"duration_ms":54603,"significance":"The problem is relevant and timely: catastrophic forgetting in streaming, heterogeneous federated learning is realistic and under-addressed in pervasive computing. The proposed simulation framework with daily activity schedules is a useful contribution, and the comparison across six model architectures is broader than is typical in this area. The forgetting metric in Eq. (5) is a reasonable attempt to account for class imbalance by averaging per-class accuracy drops. However, the evidence presented does not yet support the headline claims because of the circular use of client test data, the inconsistent definition of the alpha parameter, and the absence of exact efficiency numbers. If the evaluation is repaired and the claims are substantiated with quantitative results, the approach could be a useful practical contribution to FL for HAR.","major_comments":[{"comment":"The evaluation of forgetting is coupled to the mechanism being evaluated. In Section V-C, a client sends an updated model only when P(theta; D_test_k) is at least as large as P(theta_stored; D_test_k), and the same client-held labelled test set is used in Eq. (5) to compute the forgetting metric F_r. A FlexFed client therefore always stores and sends the model that scored best on the exact data used to measure forgetting, while the baselines REFL and MIFA receive no such oracle. Consequently, the lower F_r values in Figures 7-8 and the 'stability above 95%' claim may largely reflect this accept/reject selection policy rather than superior knowledge retention. The authors should report results on an independent server-held or withheld test set, and should isolate the effect of the update gate by an ablation that disables the gate or provides the same gate to the baselines.","section":"Sections V-C and Eq. (5)"},{"comment":"Section V-B defines the memory allocation as |zeta_k| = m x (1 - alpha_k) and states that 'A lower value of alpha_k indicates better performance', while Section V-C step 6 sets alpha_k = P(theta_r, D_test_k), i.e., an accuracy in [0,1] for which higher is better. These statements cannot both be true: if alpha_k is an accuracy, then |zeta_k| = m x (1 - alpha_k) gives less memory to better-performing clients, contradicting the sentence saying lower alpha indicates better performance. The authors must define alpha_k unambiguously and ensure that the formal definition, the memory allocation formula, and the prose describe the same monotonicity.","section":"Section V-B and Section V-C step 6"},{"comment":"The abstract's claim of 'improving FL efficiency by 10 to 15%' is unsupported by the reported evidence. Section VII provides no efficiency metric such as communication rounds to a target accuracy, wall-clock time, FLOPs, or bytes transferred, no table of exact numeric results, and no repeated-seed statistics; the shaded regions in Figures 5-8 represent variation across model architectures, not across random seeds. Please add a table with mean and standard deviation over at least five seeds for accuracy, forgetting, and the chosen efficiency metric, and state precisely how 'efficiency' is measured.","section":"Abstract and Section VII"},{"comment":"The reliance on per-client labelled test data D_test_k is strong and is not stress-tested. The update gate, the memory-allocation signal alpha_k, and the forgetting metric Eq. (5) all require every client to hold labelled, representative test data each round. The paper does not specify how D_test_k is created from the sliding-window stream, its size, or its class balance, and it does not examine performance when D_test_k is absent, small, stale, or non-representative, which are likely situations in pervasive deployments. Please document the test split and run a sensitivity analysis, for example with smaller or periodically refreshed test sets, to show that the mechanism does not collapse when this signal is noisy.","section":"Section VI and Section II-C"}],"minor_comments":[{"comment":"The table header includes both WISDM and UCI HAR, but only the WISDM row is populated; either add the UCI row or remove the column.","section":"Table II"},{"comment":"The summation notation in Eq. (5) is malformed ('CX c=1', 'KX k=1'); please use standard sum notation with correct limits.","section":"Eq. (5)"},{"comment":"The notation 'S r i=0 T_i_k' in Section IV appears to denote a union of datasets, but it is not typeset or defined as such; please clarify.","section":"Section IV"},{"comment":"The text 'PyTorch v3.8.17' is ambiguous because PyTorch does not use version numbers in the 3.x range; if the authors mean Python 3.8.17, please say so explicitly.","section":"Section VI"},{"comment":"The symbol gamma is used both as the learning rate in Section VI and as the staleness scaling factor in Eq. (9); please use distinct symbols to avoid confusion.","section":"Section V-A and Section VI"},{"comment":"In the 'Brush teeth' row, the label probabilities sum to 1.1 (0.9 standing plus 0.2 walking); please correct the entries so each row sums to 1.","section":"Table III"},{"comment":"Reference [16] has an incomplete author name ('Jakub Kone' should presumably be 'Jakub Konečný'); please complete the bibliographic entry.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of the journal, but the current evidence is not yet convincing. The circularity between the update gate and the forgetting metric is the most serious concern, and the missing numeric efficiency results make the abstract's claim impossible to verify. I would ask for detailed tables, ablations, and a clear statement of the alpha definition before considering acceptance; releasing code or a detailed reproducibility appendix would also materially increase confidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read. FlexFed combines two ideas that are each not new — offline local training and memory replay — but the particular combination, with memory allocated to rare classes based on a per-client performance signal, is a real step. The simulated HAR environment with daily schedule templates is also a nice piece of evaluation infrastructure.\n\nThat said, the headline claim about catastrophic forgetting is not established by the current experiments. The core problem is the coupling between the mechanism and the metric. In Section V-C, a client sends an update only if it scores at least as well as the stored model on its own D_test_k; otherwise the stored model is sent. The forgetting metric in Eq. (5) is computed on that same D_test_k. So FlexFed is being judged on a test set it is explicitly allowed to optimize against, while REFL and MIFA never see that signal. The lower forgetting values in Figures 7–8 are partly an artifact of the accept/reject policy, not evidence of better retention. An ablation that disables the gate, or gives the same gate to the baselines, would be needed to separate the policy effect from genuine learning.\n\nThere are also smaller but real issues. The alpha parameter is defined inconsistently: Section V-B says lower alpha means better performance, but alpha is later set equal to accuracy, where higher is better. The formula m(1−α) suggests alpha is meant to be an error rate, but the text never sorts this out. The paper gives no numerical tables, no repeated-seed statistics (the shaded regions are across models, not runs), and the 10–15% efficiency claim has no supporting definition or measurement. The baseline set is also thin — only REFL and MIFA, no FedAvg, FedProx, or a streaming FL memory method like FedDS. And no code or data is released.\n\nWhat is genuinely good: the problem is real, the method is plausible, and the dynamic HAR simulation is more realistic than the static benchmarks most FL papers use. The paper also honestly states its assumption that clients have labeled test data, though it doesn't justify that assumption in a privacy-sensitive setting.\n\nIf I were the editor, I'd send this to review. The flaws are fixable, and a revised version with a fair comparison and proper statistics could be a useful contribution to the federated HAR literature. But I wouldn't cite it as-is, and I wouldn't trust the forgetting numbers until the circularity is addressed.","headline":"The paper has a plausible mechanism, but the evaluation is compromised because the same client-held test set drives both the update gate and the forgetting metric, so the central claim is not yet supported.","tokens_in":12448,"tokens_out":4030,"would_cite":false,"duration_ms":39229,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Federated learning forgets rare activities; FlexFed counters with offline training and adaptive rare-class memory.","keywords":["federated learning","catastrophic forgetting","human activity recognition","streaming data","intermittent availability","memory management","non-IID data","pervasive computing"],"falsifier":"Run the same WISDM-based benchmark but withhold labels from the clients' per-round test data, evaluating the global model only on a server-held set; if FlexFed's advantage over REFL and MIFA persists, the per-round test-data assumption is not load-bearing, and if it disappears, that assumption is confirmed as the source of the gains.","tokens_in":11552,"feed_emoji":"📱","tokens_out":6514,"duration_ms":65776,"temperature":0.7,"pith_summary":"FlexFed is a federated learning protocol aimed at a specific failure: when smartphone and IoT clients stream sensor data, hold only a small recent buffer, and connect intermittently, the global model retrains on whichever classes happen to be present and forgets rare activities. The paper argues that this catastrophic forgetting can be substantially mitigated without violating FL's no-data-sharing rule by letting clients train during offline idle periods and by sizing retained memory according to how poorly the latest global model performs on each client's own labelled test data, keeping more rare-class samples when performance is low. It also introduces a round-level forgetting metric that averages per-client, per-class accuracy drops and weights all classes equally, so that declines in infrequent classes are no longer hidden by gains in frequent ones. In a simulated HAR environment built from daily activity schedules, FlexFed is reported to converge faster and more stably than REFL and MIFA, to exceed 95% stability in most rounds, and to improve FL efficiency by 10-15%.","feed_headline":"FlexFed cuts forgetting of rare activities in federated HAR","feed_subtitle":"Offline training plus rare-class memory keeps infrequent activities accurate, with 10-15% efficiency gains.","key_machinery":"The load-bearing mechanism is the pair of client-side decisions in the FlexFed workflow: offline training with performance-gated substitution, where the offline update $\\theta^{\\text{offline}}_k$ replaces the stored model only if $P(\\theta^{\\text{offline}}_k; D^{\\text{test}}_k) \\ge P(\\theta^{\\text{stored}}_k; D^{\\text{test}}_k)$, and performance-adaptive memory, where the extra retained data consists of samples from infrequent classes only, with size $|\\zeta_k| = m(1-\\alpha_k)$ and $\\alpha_k = P(\\theta^r; D^{\\text{test}}_k)$. The proposed per-round forgetting metric, Eq. 5, $F_r = \\frac{1}{|C|} \\sum_c \\frac{1}{|K|} \\sum_k \\min(0, \\text{Acc}^r_{k,c} - \\max_{i\\le r} \\text{Acc}^i_{k,c})$, is what makes the claimed gains measurable by counting only accuracy drops and weighting every class equally.","core_discovery":"The central claim is that catastrophic forgetting in heterogeneous federated learning is not an unavoidable consequence of privacy constraints; it can be attacked by making clients active rather than passive. FlexFed does two things. First, when a client is idle and powered but lacks a connection, it continues local training and only replaces its stored model if the updated model performs at least as well on the client's test data; the better model is later uploaded. Second, memory is allocated dynamically: a client with low accuracy on the latest global model retains a larger fraction of old data, specifically samples from infrequent classes, while a high-performing client retains less. The paper also proposes replacing the standard FL forgetting metric with one that, for every round, averages only negative per-class accuracy changes across clients and classes, giving every label equal weight. Evaluated on a schedule-driven HAR benchmark derived from WISDM and the IMA availability data, FlexFed is claimed to yield faster, more stable convergence and lower per-class forgetting than REFL and MIFA, particularly for under-represented activities.","pith_inferences":["The per-round labelled test-data assumption is the natural pressure point: in deployments where clients cannot label data, the accept/reject gate and the $\\alpha_k$ signal lose power, and an unlabelled variant would need a proxy such as output confidence or a small server-held validation set.","The two mechanisms are entangled: offline training adds extra local epochs, and the memory policy changes the data mix, so part of the reported gain may come from more computation rather than from rare-class retention; an ablation that holds compute constant would separate the effects.","The Eq. 5 metric, averaged over rounds, is essentially a smoothed per-class forgetting curve and could be adapted to continual learning more broadly, where class frequencies also vary, to detect drift before aggregate accuracy falls."],"forward_implications":["If FlexFed works as reported, federated HAR models will keep rare activities like 'upstairs' accurate even when the clients carrying those labels are offline for long stretches.","Communication efficiency rises because clients train while disconnected and upload only their stored model when it improves, making the reported 10-15% efficiency gain an expected property rather than a tuning artifact.","The proposed per-class forgetting metric should become the evaluation default for streaming federated learning, since overall accuracy hides the offsetting gains and losses that the metric reveals.","The dynamic, schedule-driven HAR generator offers a reusable stress test for any FL algorithm claiming robustness to non-stationary, intermittently available clients."],"supporting_citations":[{"why":"Supplies the FedAvg baseline and the round-based local-update objective that FlexFed modifies.","marker":"[30]"},{"why":"Defines the resource-constrained client availability model and is the REFL baseline against which FlexFed is compared.","marker":"[2]"},{"why":"MIFA, the recent-update retention baseline whose memory overhead FlexFed is claimed to improve on.","marker":"[10]"},{"why":"Flashback defines catastrophic forgetting in FL and provides the forgetting formulation that FlexFed's Eq. 5 replaces.","marker":"[4]"},{"why":"Provides the backward-transfer measure that the FL forgetting metric adapts.","marker":"[5]"},{"why":"Supplies the daily-activity schedule simulation approach used to generate dynamic, non-stationary HAR data.","marker":"[12]"},{"why":"The WISDM HAR dataset from which the streaming, imbalanced client data is generated.","marker":"[17]"},{"why":"IMA availability traces used to model heterogeneous client participation.","marker":"[42]"},{"why":"Supplies the staleness-scaled aggregation rule used to combine asynchronous client updates.","marker":"[1]"}],"fun_headline_variants":["FlexFed saves rare activities in federated HAR","Federated learning that forgets less, thanks to FlexFed","FlexFed: active clients fight forgetting in HAR","Rare moves remembered: FlexFed for federated HAR","FlexFed boosts HAR accuracy on infrequent data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes every client has labelled test data for the current round and that per-class accuracy of the latest global model on that test data is a faithful signal for accepting updates, sizing rare-class memory, and measuring forgetting.","fun_headline_variants_meta":{"raw":{"variants":["FlexFed saves rare activities in federated HAR","Federated learning that forgets less, thanks to FlexFed","FlexFed: active clients fight forgetting in HAR","Rare moves remembered: FlexFed for federated HAR","FlexFed boosts HAR accuracy on infrequent data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000165,"raw_usage":{"total_tokens":1286,"prompt_tokens":1014,"completion_tokens":272,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":630,"completion_tokens_details":{"reasoning_tokens":195}},"tokens_in":630,"tokens_out":272,"duration_ms":3614,"temperature":1.0,"reasoning_tokens":195,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:17:56.726769+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same WISDM-based benchmark but withhold labels from the clients' per-round test data, evaluating the global model only on a server-held set; if FlexFed's advantage over REFL and MIFA persists, the per-round test-data assumption is not load-bearing, and if it disappears, that assumption is confirmed as the source of the gains.","supporting_citations":[{"cited_title":"Refl: Resource-efficient federated learning","cited_arxiv_id":null,"evidence_quote":"Defines the resource-constrained client availability model and is the REFL baseline against which FlexFed is compared."},{"cited_title":"Fast federated learning in the presence of arbitrary device unavailability","cited_arxiv_id":null,"evidence_quote":"MIFA, the recent-update retention baseline whose memory overhead FlexFed is claimed to improve on."},{"cited_title":"Flashback: Understanding and Mitigating Forgetting in Federated Learning","cited_arxiv_id":"2402.05558","evidence_quote":"Flashback defines catastrophic forgetting in FL and provides the forgetting formulation that FlexFed's Eq. 5 replaces."},{"cited_title":"Riemannian walk for incremental learning: Under- standing forgetting and intransigence","cited_arxiv_id":null,"evidence_quote":"Provides the backward-transfer measure that the FL forgetting metric adapts."},{"cited_title":"A framework for realistic simulation of daily human activity","cited_arxiv_id":null,"evidence_quote":"Supplies the daily-activity schedule simulation approach used to generate dynamic, non-stationary HAR data."},{"cited_title":"Real time human activity recognition using tri-axial ac- celerometers","cited_arxiv_id":null,"evidence_quote":"The WISDM HAR dataset from which the streaming, imbalanced client data is generated."},{"cited_title":"Flash: Heterogeneity-aware federated learning at scale","cited_arxiv_id":null,"evidence_quote":"IMA availability traces used to model heterogeneous client participation."},{"cited_title":"Resource-Efficient Federated Learning","cited_arxiv_id":"2111.01108","evidence_quote":"Supplies the staleness-scaled aggregation rule used to combine asynchronous client updates."}],"review_version":1}