{"id":"2dd5ea45-6934-4357-89ba-c2cb3bb1698d","arxiv_id":"2505.06384","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"A federated-learning mobile app predicts sleep and distance deficits from the same features used to define those deficits, so its reported 60.71% accuracy does not demonstrate predictive efficacy.","lead":"RiM is a mobile app that uses federated learning to give students daily lifestyle advice by tracking sleep, steps, and meals. The paper reports 60.7% sign-accuracy for deficit prediction, but the prediction targets are computed directly from the input features, making the evaluation circular.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation is circular: deficit targets are deterministic functions of the input features, so accuracy/MAE do not measure predictive efficacy.","rationale":"The paper's central claim is that a FedAvg-based MLP can predict lifestyle deficits under privacy-preserving constraints. Reading in good faith, the strongest support offered is the accuracy/MAE comparison in Chapter 4. The most load-bearing problem is that the targets are deterministic functions of the input features: the deficit definitions in Section 3.2 map sleep hours and distance directly to d_sleep and d_distance. Consequently, the evaluation does not measure generalization to unseen outcomes; it measures how well the model reproduces a piecewise-linear transformation. A model that simply applies the formula would achieve perfect accuracy, so the reported numbers cannot support 'efficacy in predicting lifestyle deficits.' This is an internal validity flaw, not a disagreement with external consensus. The reader's identified weakest assumption (differential privacy) is also a serious issue: sharing model weights without clipping or noise does not guarantee DP, and the paper even cites DP-FedAvg without implementing it. Both flaws justify rejection. I partially agree with the reader: we converge on REJECT, but I view the circular evaluation as the more fundamental, load-bearing concern because it undercuts the primary efficacy claim even before considering privacy. The proposed concrete test would settle this by comparing the reported model against the trivial closed-form rule. No change to the reader's verdict is needed.","tokens_in":11271,"tokens_out":4265,"duration_ms":43523,"concrete_test":"Recompute accuracy and MAE on the same test data by applying the closed-form deficit definitions from Section 3.2 directly to the input features x (no MLP). If this rule-based baseline yields accuracy near 100% and MAE near 0, while the reported FedAvg model yields 60.71% and 0.91, then the reported metrics do not measure predictive ability; the task is deterministic and the evaluation is circular. Alternatively, evaluate on held-out future days (e.g., train on days 1-8, test on days 9-15) and compare against the same closed-form rule; if the rule still matches perfectly, the model adds no predictive value.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that RiM 'demonstrates efficacy in predicting lifestyle deficits' rests on the accuracy/MAE numbers in Chapter 4. But in Section 3.2, the targets d_sleep and d_distance are defined as piecewise-linear functions of the very same features (sleep hours, distance) that are input to the MLP: d_sleep = 7 - x_sleep when x_sleep < 7, and similarly for the other regimes. Thus the 'prediction' is a deterministic transformation of the input, not an unknown label to generalize to. A trivial rule-based model that directly applies this closed-form formula to the test inputs would achieve near-perfect sign accuracy and near-zero MAE. The reported FedAvg accuracy of 60.71% and MAE of 0.91 therefore do not demonstrate predictive skill; they indicate the MLP fails to fit a simple identity-like mapping, or that the evaluation protocol is circular. This invalidates the headline efficacy claim independently of the privacy issue. The differential-privacy claim in Sections 1.3 and 5 is also unsupported (no clipping, noise, or privacy accounting is implemented), but the invalid evaluation alone is sufficient to reject the central claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents RiM, an Android application that tracks steps, distance, sleep, and meal information, and uses a multilayer perceptron (MLP) pre-trained on simulated data and fine-tuned with federated learning (FedAvg and FedPer) on data from ten IISER Bhopal students. The model outputs sleep and distance deficits, which are then used by a rule-based recommender to show personalized wellness advice. The paper reports that FedAvg achieves 60.71% sign accuracy and 0.91 MAE, outperforming FedPer, and claims that sharing only model weights rather than raw data provides differential privacy.","tokens_in":11532,"tokens_out":4203,"duration_ms":41838,"significance":"The manuscript has clear strengths: it provides a complete mobile-sensing pipeline with two algorithmic descriptions (Algorithms 1 and 2), releases code via GitHub, and compares two federated personalization strategies on a real, albeit very small, dataset. If the evaluation were valid, a lightweight personalized recommender that keeps data local would be useful and topical. However, the headline claims are not supported: the target definitions make the main evaluation circular, and the differential-privacy guarantee is asserted without any mechanism. These issues invalidate the paper's central contributions in their current form.","major_comments":[{"comment":"The target variables d_sleep and d_distance are defined as piecewise-linear functions of the sleep and distance features that are also inputs to the MLP, since the feature vector is x = [distance, sleep, bmi, age, breakfast, meal, gender] and d_j is a function of x_j for j in {sleep, distance}. Therefore the 'prediction' task is to reproduce a known deterministic transformation of the input, not to generalize to an independent outcome. A trivial rule-based system that directly applies the closed-form deficit formula to the test features would achieve near-perfect sign accuracy and near-zero MAE, whereas the reported FedAvg accuracy of 60.71% and MAE of 0.91 are substantially worse than such a baseline. Consequently, Tables 4.1 and 4.2 do not measure predictive efficacy; they measure how well the MLP approximates a known function, and the central claim of the abstract and Chapter 5 does not follow.","section":"Section 3.2"},{"comment":"The claim that the approach 'ensures differential privacy by never transmitting raw user data' is unsupported. No clipping, noise injection, or privacy-accounting mechanism is described or implemented; Section 3.3 transmits model weights that can encode information about local data. Differential privacy is a property of the randomized algorithm's output, not of data locality, so the privacy guarantee as stated is false. Establishing a formal guarantee would require modifying the algorithm, e.g., by using DP-FedAvg-style gradient clipping and Gaussian noise with an explicit privacy budget, which is absent from the manuscript.","section":"Sections 1.3 and 5"},{"comment":"The experimental evaluation uses only ten participants, with eight days for fine-tuning and seven days for testing, and it lacks a non-trivial baseline such as the closed-form deficit formula or a constant predictor. The client-wise accuracies range from 25.18% to 81.42%, and no confidence intervals, cross-validation, or statistical tests are reported. Given the circular target definitions, the FedAvg-versus-FedPer comparison cannot support the paper's claim of demonstrated efficacy; the observed variation is consistent with noise on tiny samples and with class imbalance in the zero-deficit class.","section":"Section 4"}],"minor_comments":[{"comment":"The sentence 'We use simulated data to pre-train the model and fine-tune and test in on the collected real-world dara' contains two typos: 'in' should be removed and 'dara' should be 'data'.","section":"Section 3.5"},{"comment":"The threshold symbol τ is overloaded: it is used for step detection in Algorithm 1 and for deficit thresholds in Section 3.2, which can confuse the reader; distinct symbols should be used.","section":"Section 3.2"},{"comment":"Equation (3.3) defines accuracy as sign agreement including the zero class, but the text does not discuss how the prevalence of zero deficits affects the metric; for clients whose true deficits are mostly zero, predicting zero achieves high accuracy without any skill.","section":"Section 3.2"},{"comment":"The sentence 'FedPer algorithm is used for fine-tuning the MLP model' is inconsistent with Chapter 4, where both FedAvg and FedPer are evaluated; the phrasing should be clarified to describe the fine-tuning setup accurately.","section":"Section 2.2"},{"comment":"The age feature is listed as 'Truncated Normal (Empirical Distribution)', but the text and Figure 2.1 refer to 'an empirical distribution'; the distinction between the two descriptions should be resolved.","section":"Table 2.1"},{"comment":"The GitHub link should be cited with an access date or version, following the style used for other web resources in the bibliography.","section":"Section 3.1"}],"recommendation":"reject","confidential_remarks":"The manuscript is written as a bachelor's thesis report and would need major methodological revision before it could be considered a research contribution. The circular evaluation alone is sufficient to reject the central claim; the unsupported privacy guarantee compounds the problem."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know before you spend time on this: the evaluation is circular and the privacy claim is false. In Section 3.2 the deficit targets are defined as piecewise-linear functions of the same features the MLP takes as input (e.g., sleep deficit = 7 − sleep when sleep < 7). So 'predicting' a deficit is just reproducing a known transformation of the input. A rule-based model that applied that formula would get near-perfect accuracy; the reported 60.71% actually tells you the MLP failed to fit a simple identity-like mapping. The abstract's 'efficacy in predicting lifestyle deficits' is therefore not supported. Separately, the paper equates 'never sharing raw data' with differential privacy. That is wrong: fine-tuned weights can leak local information, and without clipping, noise, or privacy accounting there is no DP guarantee. Sections 1.3 and 5 both make this unsupported claim.\n\nWhat is worth credit: the authors built a real mobile pipeline—accelerometer step counting with debounce, sleep tracking from inactivity, meal logging, and FL fine-tuning via Flower. They also report attrition honestly (18 volunteers down to 10) and keep the system description readable. The FedAvg-vs-FedPer comparison is a routine choice but it is executed cleanly enough for an undergraduate project.\n\nThe soft spots beyond the two above: the dataset is ten participants, eight days of training, seven of test; there are no baselines beyond the two FL variants, no error bars, and no released code or data (a GitHub link appears but nothing is evaluated). These would matter less if the target were an independent outcome, but with a deterministic target they are moot. The discussion of class imbalance and personalization is reasonable interpretation, but it is interpretation of noise.\n\nWho is this for? Someone looking for a template of an FL mobile-health thesis might skim it. For a research venue, it does not deserve referee time—the central measure of success is invalid, and the advertised privacy guarantee is absent. If the authors want to salvage it, they could reframe it as a system description and drop the predictive claims, or replace the target with an actual health label independently measured. As is, my honest recommendation is to pass.","headline":"Circular evaluation and an unsupported differential-privacy claim sink an otherwise neat undergraduate FL-health thesis.","tokens_in":12003,"tokens_out":2596,"would_cite":false,"duration_ms":24829,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a federated MLP fine-tuned with FedAvg predicts student sleep and distance deficits at 60.71% sign accuracy and 0.91 MAE, outperforming FedPer, while keeping raw data on users' phones.","keywords":["federated learning","mobile health","well-being","FedAvg","FedPer","lifestyle deficit prediction","multilayer perceptron","differential privacy"],"falsifier":"Run a membership-inference attack on the shared fine-tuned weights: if an attacker can determine whether a specific day's step count or meal flag was included in a client's training set from the weight updates, the privacy claim is false. A formal DP accountant applied to the unclipped, un-noised updates would also show unbounded $\\varepsilon$.","tokens_in":11092,"feed_emoji":"🏃","tokens_out":7541,"duration_ms":72052,"temperature":0.7,"pith_summary":"RiM is a mobile app that tracks students' steps, distance, sleep, and meals, then uses a small neural network to predict which lifestyle measures fall short of ideal sleep and distance ranges. The paper tries to establish that a two-stage pipeline—pre-training a multilayer perceptron on simulated data, then fine-tuning it with federated learning on data that stays on each phone—can deliver personalized wellness recommendations without centralizing raw health records. Measured by sign agreement and mean absolute error, the FedAvg variant outperforms the FedPer variant on a ten-client real-world dataset. If correct, this offers a lightweight, privacy-preserving alternative to centralized wellness analytics for a population that rarely shares health data.","feed_headline":"Federated wellness app predicts student health deficits at 60.7%","feed_subtitle":"Weight-sharing federated model beats per-user personalization on sleep and activity advice; raw data stays local.","key_machinery":"The central object is a five-hidden-layer MLP (32-to-4 neurons) that maps a standardized feature vector $[distance, sleep, BMI, age, breakfast, meal, gender]$ to two outputs: sleep deficit and distance deficit, each defined as the signed distance outside an ideal range. It is pre-trained on simulated data whose features are sampled from stated distributions, then fine-tuned in federated rounds using FedAvg, which averages all client weights, and FedPer, which averages only shared root layers while keeping personal head layers local. A rule-based scoring layer converts predicted deficits into weighted risk scores, aggregates interaction risks, and surfaces the top two recommendation messages. The accuracy metric counts a prediction as correct when the sign of the predicted deficit matches the sign of the true deficit.","core_discovery":"The paper's central empirical claim is that the FedAvg-fine-tuned MLP outperforms the FedPer-fine-tuned variant: average accuracy 60.71% versus 46.34%, and MAE 0.91 versus 1.19. It interprets this gap as a personalization trade-off: FedAvg's global weight averaging transfers deficit-detection skill across clients, while FedPer's per-client heads preserve each user's bias toward the healthy zero-deficit pattern but fail to model rare non-ideal events. The paper further claims that the pipeline guarantees differential privacy because only model weights, not raw sensor or demographic data, leave the device.","pith_inferences":["Because the shared weight updates are not clipped or noised, the claimed differential-privacy guarantee does not follow from the algorithm as described; formal differential privacy would require calibrated noise, gradient clipping, and a privacy budget.","The accuracy metric is sign agreement, so a model that always predicts 'no deficit' would score highly on healthy clients, which likely explains part of FedPer's advantage on zero-deficit users and should be checked against a no-deficit baseline.","A direct extension would repeat the fine-tuning with calibrated noise added to the updates and measure how the 60.71% accuracy and 0.91 MAE degrade, giving an empirical privacy-utility trade-off for this setting.","Since pre-training uses synthetic distributions, the pretrained weights may not transfer to populations with different activity or meal patterns; retraining on a broader set of real cohorts is a testable next step."],"forward_implications":["A phone with only accelerometer readings can support an MLP that flags sleep and distance deficits, without requiring additional wearable sensors for the core prediction.","FedAvg's global averaging works better for students whose lifestyles deviate from ideal, while FedPer's local heads better preserve an all-healthy pattern, so the choice of aggregation scheme changes which users receive useful alerts.","The sign-accuracy measure means 60.71% reflects how often the model gets the direction of the deficit right, not the exact magnitude; MAE 0.91 quantifies the average magnitude error in the same units as the deficits.","If the privacy assumption holds, the system can be deployed without a central collection of health records, lowering the risk of data breaches and reducing the consent friction associated with sharing personal health data.","The top-two risk-scored recommendations give users actionable guidance while avoiding message overload."],"supporting_citations":[{"why":"Supplies the FedAvg algorithm that the paper fine-tunes with and compares against FedPer.","marker":"[9]"},{"why":"Introduces federated learning via model averaging, the basis for the paper's claim that sharing weights rather than raw data preserves privacy.","marker":"[13]"},{"why":"Provides the FedPer personalization-layer scheme used as the comparison baseline.","marker":"[14]"},{"why":"Describes the open-source platform used to run the federated client-server training rounds.","marker":"[12]"},{"why":"Motivates the negative-binomial distribution used to generate simulated step-count data for pre-training.","marker":"[15]"},{"why":"Motivates the log-normal distribution used for simulated distance-traveled data.","marker":"[16]"},{"why":"Motivates the normal distribution used for simulated sleep-hours data.","marker":"[17]"}],"fun_headline_variants":["FedAvg beats FedPer by 14 points in student wellness AI","Privacy-first federated app hits 60.7% accuracy on student health","Global averaging beats per-user heads in federated wellness AI","Federated learning fine-tuning boosts wellness app to 60.7% accuracy","FedAvg fine-tuning beats FedPer for private student wellness"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that only transmitting model weights, with no added noise or formal privacy accounting, yields differential privacy; if shared weights leak information about local data, the privacy guarantee collapses.","fun_headline_variants_meta":{"raw":{"variants":["FedAvg beats FedPer by 14 points in student wellness AI","Privacy-first federated app hits 60.7% accuracy on student health","Global averaging beats per-user heads in federated wellness AI","Federated learning fine-tuning boosts wellness app to 60.7% accuracy","FedAvg fine-tuning beats FedPer for private student wellness"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001156,"raw_usage":{"total_tokens":4756,"prompt_tokens":882,"completion_tokens":3874,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":498,"completion_tokens_details":{"reasoning_tokens":3782}},"tokens_in":498,"tokens_out":3874,"duration_ms":26722,"temperature":1.0,"reasoning_tokens":3782,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:44:08.232465+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a membership-inference attack on the shared fine-tuned weights: if an attacker can determine whether a specific day's step count or meal flag was included in a client's training set from the weight updates, the privacy claim is false. A formal DP accountant applied to the unclipped, un-noised updates would also show unbounded $\\varepsilon$.","supporting_citations":[{"cited_title":"How many steps/day are enough? preliminary pedometer indices for public health","cited_arxiv_id":null,"evidence_quote":"Motivates the negative-binomial distribution used to generate simulated step-count data for pre-training."},{"cited_title":"The scaling laws of human travel","cited_arxiv_id":null,"evidence_quote":"Motivates the log-normal distribution used for simulated distance-traveled data."},{"cited_title":"Self-reported and measured sleep duration: how similar are they? Epidemiology , 19(6):838--845, 2008","cited_arxiv_id":null,"evidence_quote":"Motivates the normal distribution used for simulated sleep-hours data."}],"review_version":1}