{"id":"9a5f04b6-fbe9-41a9-b377-8f95d40cfb0c","arxiv_id":"2504.16032","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A threshold-based asynchronous federated learning strategy for fine-tuning LLMs on IoT data reports modest accuracy gains and large latency/throughput wins over FedAvg and FedOpt on the IoT-23 dataset.","lead":"This paper combines large language models with federated learning to manage IoT devices, using a threshold-based method that decides when clients upload model updates. The authors report small accuracy gains and large latency and throughput gains over standard federated baselines on the IoT-23 dataset.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The efficiency claims rest on a throughput proxy labeled 'energy efficiency' and latency numbers in incompatible units across Tables II and III; the central performance claims are not supported.","rationale":"The reader's weakest assumption identifies a genuine load-bearing flaw. I reviewed the framework and tables. Section IV-A gives the operative definitions; Section IV-B's claims rely on them. The two definitions are not commensurable with the Table III values. The accuracy part is also internally inconsistent. This is not a matter of disagreement with consensus; it is an internal validity problem. Therefore I find no reason to alter the reader's REJECT. I did not find additional support that would rescue the claim: no code, no statistical tests, no energy measurements, and no explanation of how Table III's throughput and latency relate to Table II's. The most charitable reading still requires Table III's throughput and latency to correspond to the same quantity as Table II, and they do not. Agreement: the reader pointed to the metric-proxy and unit issue; I agree, with the additional observation that the numerical magnitude of the discrepancy makes it unlikely to be a rounding or reporting artifact.","tokens_in":7804,"tokens_out":3633,"duration_ms":34793,"concrete_test":"Re-run the Section IV-B comparison on identical hardware, measuring (a) end-to-end per-request latency in the same units for standalone models and federated strategies, (b) actual energy consumption in Joules per request or per training round using hardware power meters, and (c) communication volume; then recompute Table III. Also report per-round wall-clock time and the number of uploads triggered by GSFS. If, under these measurements, GSFS no longer shows roughly 21.6%/51.5% lower latency and improved energy efficiency, the central claim fails. As a minimal first check, verify why Table II's OPT-350M throughput of 680.70 req/min becomes 1.73 req/min for the GSFS client in Table III; any explanation that is not a unit or measurement change invalidates the comparison.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section IV-A defines 'Energy Efficiency' as 'the number of requests the model can handle within one minute' and 'Response Latency' as average inference time on the test set. The headline claims of 'improved energy efficiency' and 'lower response latency' in the abstract and Section IV-B rest entirely on Table III, which reports energy efficiency in req/min and latency in seconds. Throughput is not energy: for 'energy efficiency' to be measured validly, the energy cost per request must be constant or separately measured, and no such measurement or argument is provided. The unit mismatch is concrete: Table II lists OPT-350M client latency as 88.14 ms and throughput as 680.70 req/min, while Table III reports the GSFS client model at 34.69 s latency and 1.73 req/min. If these are the same model on the same client hardware, the numbers imply a roughly 394x slowdown and 393x lower throughput than the standalone measurement, so the quantity in Table III is not the same as the quantity in Table II. There is also an internal numerical mismatch: the abstract claims 1.02% higher client accuracy versus FedAvg, but Table III gives 0.8732 vs 0.8881, a 1.49 percentage-point difference (or 1.71% relative), not 1.02. Without a consistent metric definition and a reconciliation of Tables II and III, the central efficiency and latency claims are unsupported. The accuracy claim alone is modest, but the paper's differentiator is efficiency, and that differentiator is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GSFS, an asynchronous federated strategy for fine-tuning LLMs in IoT environments. Clients upload updates only when local performance or gradient thresholds are exceeded, and the server aggregates when a threshold number of distinct clients has contributed. The framework is evaluated on the IoT-23 dataset using OPT models, comparing against FedAvg and FedOpt. The paper claims that GSFS improves model accuracy, reduces response latency, and enhances energy efficiency.","tokens_in":8090,"tokens_out":5151,"duration_ms":47596,"significance":"The motivating problem is relevant: fine-tuning LLMs under federated IoT constraints involves communication and resource bottlenecks, and a threshold-triggered asynchronous update strategy with server-side aggregation is a plausible mechanism for reducing communication. However, the empirical support is undermined by the metric definitions and internal inconsistencies described below. The paper does not provide code, configuration values, or a data availability statement, so the experiments are not reproducible. If the efficiency claims were supported by direct energy measurements and the latency units were reconciled, the contribution could be useful to the FL-for-IoT community.","major_comments":[{"comment":"The metric 'Energy Efficiency' is defined as 'the number of requests the model can handle within one minute' (req/min), which is throughput, not energy. No measurement of power or energy per request is provided anywhere in the paper. Consequently, the abstract's claim of 'enhanced energy efficiency' and Section IV-B's statement that 'GSFS achieves higher efficiency than FedAvg' are not supported. Because GSFS is designed to reduce communication and synchronization overhead, it can mechanically increase throughput by freeing resources, making the claimed improvement partly constructed by the metric definition. The authors should either measure actual energy consumption (e.g., Joules per request) or relabel the metric as throughput and remove the energy-efficiency claim.","section":"Section IV-A, Table III"},{"comment":"The latency and throughput values in Table II and Table III are not commensurable. Table II reports OPT-350M client latency as 88.14 ms and throughput as 680.70 req/min, while Table III reports the GSFS client model latency as 34.69 s and throughput as 1.73 req/min. These numbers imply a roughly 394x slowdown and 393x lower throughput if they refer to the same quantity. The paper must state the exact unit and aggregation procedure for the latency values in Table III and reconcile the throughput numbers with Table II; without this, the 'lower response latency' claim for GSFS cannot be verified.","section":"Section IV-A, Tables II and III"},{"comment":"The abstract states that GSFS achieves '1.02% higher accuracy on the client model' compared to FedAvg, but Table III reports FedAvg client accuracy as 0.8732 and GSFS client accuracy as 0.8881, which is a 1.49 percentage-point gain (1.71% relative), not 1.02%. The reported number is internally inconsistent with the results table and should be corrected, with all derived claims rechecked against the table values.","section":"Abstract and Section IV-B"},{"comment":"The title and abstract describe the framework as 'Secure IoT Management' and mention Differential Privacy and TLS/SSL, but no privacy or security evaluation is presented. The experiments measure accuracy, F1-score, latency, and throughput only. The security claims are load-bearing for the paper's framing, so the authors should either add a privacy/security evaluation (e.g., privacy budget, threat model, or attack resistance) or substantially qualify the security claims.","section":"Title/Abstract vs Section IV"}],"minor_comments":[{"comment":"The index term 'Large Learning Models' should be 'Large Language Models' for consistency with the abstract and body.","section":"Index Terms"},{"comment":"The abstract introduces 'Generative IoT (GIoT) models,' but the framework description in Section III never defines or uses this term; please clarify or remove it.","section":"Section III-A"},{"comment":"The threshold parameters δ_perf, β, α, and M from Eqs. (8), (11), and the aggregation rule are not given concrete values or a sensitivity analysis, so the reported results are not reproducible from the text.","section":"Section IV-B"},{"comment":"The caption for Figure 2 does not state which training configuration or FL strategy produced the loss curves, and the axis labels are not visible in the manuscript text; please provide this information.","section":"Figure 2"},{"comment":"The paper mentions 'Differential Privacy (DP-FedAvg)' as a secure aggregation technique but provides no DP mechanism, privacy budget, or evaluation, so this claim is currently unsupported.","section":"Section III-B"},{"comment":"The related work section cites only a few FL-LLM papers and omits the substantial literature on asynchronous and communication-efficient federated learning (e.g., FedAsync, FedProx), which would help position the contribution more accurately.","section":"References"},{"comment":"The paper does not include a data or code availability statement, which limits reproducibility and is expected for empirical work in this area.","section":"General"}],"recommendation":"reject","confidential_remarks":"To the editor: The paper's central differentiator is efficiency, and that differentiator rests on a throughput metric mislabeled as 'energy efficiency' plus latency numbers in incompatible units across Table II and Table III. These are load-bearing issues that would require new experiments (e.g., direct energy measurement, metric reconciliation) rather than local corrections. The accuracy improvement over FedAvg is small and the 1.02% vs 1.49% discrepancy suggests the reported numbers have not been carefully checked. I do not see a path to acceptance within the scope of a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the GSFS scheme is a reasonable engineering combination of existing asynchronous federated learning ideas—performance-triggered uploads, gradient-norm thresholds, and server-side weighted aggregation—applied to LLM fine-tuning for IoT. The writing is clear and the IoT-23 comparison across several LLMs is useful. But the paper's central efficiency claim rests on a metric labeled 'energy efficiency' that is actually throughput (requests/minute), and the latency numbers in Table II and Table III don't reconcile. That's a load-bearing flaw, and the accuracy gain is tiny anyway.\n\nWhat's good: the architecture description is concrete, the selection of OPT-350M as client model is justified by Table II, and the equations for GSFS are complete enough to reimplement. The idea of letting clients decide when to upload based on local change is sensible and can reduce communication. If the only claim were 'a threshold-based asynchronous FL variant that keeps accuracy while lowering communication', the paper would be a modest but acceptable contribution.\n\nWhere it falls down: the abstract reports 1.02% client accuracy improvement, but Table III shows 1.49 percentage points (0.8732 vs 0.8881). That's an internal inconsistency. More seriously, 'Energy Efficiency' is defined as requests per minute, which is throughput. Under that definition, any strategy that cuts communication can mechanically raise the number of requests handled, so the claimed energy improvement is partly baked into the metric. There is no measured energy. The latency problem: Table II reports OPT-350M at 88.14 ms and 680.70 req/min. Table III reports the GSFS client model at 34.69 s and 1.73 req/min. If these are the same model on the same client hardware, that's about a 390x slowdown, which makes no sense unless the quantities measure different things. The paper never reconciles these. There's also no code, no confidence intervals, and no comparison to existing asynchronous FL methods like FedAsync, so the novelty isn't positioned against the work that actually matters.\n\nWho it's for: someone wanting a quick survey of FL-LLM for IoT might skim it, but anyone considering GSFS needs the efficiency numbers fixed first. The core algorithm is plausible, but the evaluation doesn't establish the headline result.\n\nRecommendation: this deserves peer review, not a desk reject—the flaws are fixable and the engineering idea has some merit. A careful referee should ask for a real energy measurement (or a clear argument that throughput is a valid proxy), a reconciliation of the latency tables, corrected accuracy deltas, and a comparison to asynchronous baselines. If those come back, the paper could be salvageable.","headline":"A modest engineering tweak whose headline efficiency claim is undercut by a throughput-based 'energy' metric and inconsistent latency numbers across tables.","tokens_in":8623,"tokens_out":3174,"would_cite":false,"duration_ms":27332,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a gradient-sensing federated strategy (GSFS) lets edge devices fine-tune LLMs for IoT tasks with less communication and lower latency than standard FedAvg or FedOpt, while keeping accuracy and privacy.","keywords":["federated learning","large language models","Internet of Things","Gradient Sensing Federated Strategy","edge-cloud computing","IoT-23 dataset","communication efficiency","privacy-preserving training"],"falsifier":"Measure actual wall-clock power draw (in watts) of the central server and client GPUs across the 10 federated rounds for GSFS, FedAvg, and FedOpt, and compare energy consumed per completed task; if GSFS does not consume less energy per task, the energy-efficiency claim fails. Also re-run Table III with latency in consistent units (milliseconds) and verify whether the central latency of 37.74 seconds is test-set inference time or includes aggregation and communication overhead.","tokens_in":7587,"feed_emoji":"⚡","tokens_out":4286,"duration_ms":40214,"temperature":0.7,"pith_summary":"The paper aims to show that large language models can be fine-tuned for IoT management in a federated, privacy-preserving way without the communication and latency costs of standard federated averaging. It introduces Gradient Sensing Federated Strategy (GSFS), a client-server update rule in which edge devices upload model updates only when their local validation metric or gradient norm crosses an adaptive threshold, and the cloud aggregates only when a pool of distinct client updates reaches a threshold. On the cleaned IoT-23 dataset, the paper reports that GSFS outperforms FedAvg and FedOpt in accuracy and F1, cuts response latency, and raises throughput. A sympathetic reader would care because the mechanism addresses the main obstacle to using LLMs at the edge: high communication and compute cost.","feed_headline":"New federated strategy cuts IoT LLM latency by half","feed_subtitle":"GSFS selectively uploads model updates, lifting accuracy and throughput versus FedAvg and FedOpt in IoT-23 tests.","key_machinery":"The core mechanism is GSFS, a gradient-sensing update policy. Each client evaluates its local model on a private validation set and tracks layer-wise gradient norms, maintaining a moving mean and standard deviation. If the change in the performance metric exceeds an adaptive threshold, or if any layer's gradient norm exceeds its adaptive threshold, the client asynchronously uploads its update to the server. The server caches these updates in a pool and performs global aggregation only when the pool contains updates from a threshold fraction (default 60%) of clients, weighting updates by submission frequency. The aggregated global model is then broadcast to all clients, which reset their local reference points. This selective, asynchronous upload design is what the paper claims reduces communication overhead and latency relative to uniform full-participation rounds.","core_discovery":"The central discovery is that a federated strategy which lets clients upload model updates only when their local performance or gradient norms shift significantly—and lets the server aggregate only once enough distinct clients have reported—can match or exceed the accuracy of round-synchronous FedAvg and FedOpt while lowering response latency and raising throughput, at least on the cleaned IoT-23 dataset with OPT-family LLMs. Specifically, the paper reports central accuracy of 0.9008 against 0.8945 for FedAvg, client-side latency reduced from 71.43 s to 34.69 s, and higher throughput in requests per minute.","pith_inferences":["An extension not explored here: the same selective-upload trigger could be stress-tested on non-IID client data, where infrequent uploads may delay the server's view of local distribution shifts and potentially hurt convergence.","If the paper's 'energy efficiency' proxy is replaced with actual measured joules per task, the ranking among strategies could change; a direct power measurement is a natural test of the claimed efficiency gain.","The GSFS aggregation could be extended with staleness-aware weights, so older updates receive lower weight than fresh ones; the paper currently weights only by submission frequency.","The threshold parameters (smoothing factor and sensitivity) likely interact with client heterogeneity, so a sensitivity analysis over these hyperparameters would clarify when GSFS helps rather than hurts."],"forward_implications":["Federated fine-tuning of LLMs for IoT becomes feasible on devices with limited memory and GPU capacity, since clients upload only significant updates rather than every round.","Communication volume per round drops because not every client participates every round; the server aggregates only when enough distinct clients report.","Applications such as malware detection and anomaly detection on IoT network traffic can be updated continuously from edge data without raw data leaving the device.","The hybrid edge-cloud deployment lets small client models run on commodity GPUs while the cloud maintains a larger central model for global refinement."],"supporting_citations":[{"why":"Supplies the federated-learning problem setting and motivation (communication cost, privacy) that GSFS is designed to improve.","marker":"[7]"},{"why":"Prior LLM-IoT integration showing task-reasoning gains; the proposed framework extends this style of integration to federated fine-tuning.","marker":"[8]"},{"why":"FLoRA demonstrates federated fine-tuning of LLMs within 1.5% of centralized training, the feasibility baseline for federated LLM adaptation.","marker":"[9]"},{"why":"OpenFedLLM shows federated LLM training with under 2% accuracy gap, supporting the premise that privacy-preserving decentralized LLM training is viable.","marker":"[10]"}],"fun_headline_variants":["Selective federated LLM updates halve IoT latency, lift accuracy","GSFS: update-on-change federated LLM halves IoT latency, tops FedAvg","Federated LLM with gradient sensing cuts IoT latency by half, boosts accuracy","GSFS halves IoT LLM latency, exceeds FedAvg accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported energy and latency advantages depend on treating 'requests per minute' as a measure of energy efficiency and on comparing latency numbers collected under the same units and conditions in Tables II and III; if either is not the case, the efficiency claims collapse.","fun_headline_variants_meta":{"raw":{"variants":["Selective federated LLM updates halve IoT latency, lift accuracy","GSFS: update-on-change federated LLM halves IoT latency, tops FedAvg","Federated LLM with gradient sensing cuts IoT latency by half, boosts accuracy","GSFS halves IoT LLM latency, exceeds FedAvg accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000429,"raw_usage":{"total_tokens":2142,"prompt_tokens":845,"completion_tokens":1297,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":461,"completion_tokens_details":{"reasoning_tokens":1215}},"tokens_in":461,"tokens_out":1297,"duration_ms":11576,"temperature":1.0,"reasoning_tokens":1215,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:11:15.956023+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure actual wall-clock power draw (in watts) of the central server and client GPUs across the 10 federated rounds for GSFS, FedAvg, and FedOpt, and compare energy consumed per completed task; if GSFS does not consume less energy per task, the energy-efficiency claim fails. Also re-run Table III with latency in consistent units (milliseconds) and verify whether the central latency of 37.74 seconds is test-set inference time or includes aggregation and communication overhead.","supporting_citations":[{"cited_title":"Kairouz, H","cited_arxiv_id":null,"evidence_quote":"Supplies the federated-learning problem setting and motivation (communication cost, privacy) that GSFS is designed to improve."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"FLoRA demonstrates federated fine-tuning of LLMs within 1.5% of centralized training, the feasibility baseline for federated LLM adaptation."}],"review_version":1}