{"id":"fa717cba-63dc-4b5d-adca-c9e9db722e9e","arxiv_id":"2502.06493","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"EdgeMLBalancer uses epsilon-greedy switching among four edge object-detection models, driven by CPU usage and confidence, and reports better accuracy and fairness than two baselines in a single 30-minute smartphone trial.","lead":"This paper describes EdgeMLBalancer, an Android app that switches among four object-detection models based on real-time CPU usage and detection confidence, using an epsilon-greedy rule to avoid starving any model. It reports that on a single 30-minute traffic video, this approach beats a fixed-threshold and a round-robin baseline in accuracy and fairness, but the evaluation has no error bars and the absolute accuracy is low.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Section III-A2 score function rewards declining confidence, confirming the reader's sign-inversion concern; no change to the REJECT verdict.","rationale":"The reader's weakest assumption is exactly the load-bearing issue. The paper's central mechanism is the performance score and the arg-min selection in the Planner, and that mechanism is internally inconsistent with its stated goal: it systematically favors models whose confidence has declined relative to the historical average. The attached example in Section III-A2 makes the inversion explicit by calling a negative score better. This is not merely a stylistic or notation problem; it determines which model is selected in every exploitation step, so the accuracy advantage claimed in RQ1 cannot be attributed to the published algorithm. The paper provides no validation of the score function, no ablation, and no sensitivity analysis for the unreported epsilon value, so there is no independent support that would resolve the inconsistency. The code repository is the natural place to check whether the implementation matches the paper; if it does, the central claim is unsupported as stated. I therefore agree with the reader that the verdict should remain REJECT, and no verdict adjustment is needed.","tokens_in":13020,"tokens_out":4196,"duration_ms":38564,"concrete_test":"Inspect the Analyzer's ScoreGenerator in the linked repository (github.com/sa4s-serc/EdgeMLBalancer) and compare the implemented score and selection direction to Eq. (1) in Section III-A2. Then run Algorithm 1's exploitation branch on a two-model trace in which model A has C_i > C_avg and model B has C_i < C_avg with identical CPU terms. If the planner selects B, the sign inversion is real; if it selects A, or the code uses a different formula or arg max, the paper's formula is a typo and the reported results may still be valid.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The decision rule is internally inconsistent. Section III-A2 defines S_mi = min(U_i, U_avg) * (1 - C_avg/C_i), and Section III-A3 / Algorithm 1 selects arg min S. For any fixed CPU term, a model whose current confidence C_i is below the historical average C_avg makes the factor (1 - C_avg/C_i) negative, so it receives a lower (better) score. The stated purpose of the factor is to 'highlight models that are maintaining or improving their accuracy relative to historical performance,' but the algebra rewards the opposite. The paper's own example confirms this: the computed score -0.3627 is called better because of a 'slight decline in confidence.' Since exploitation is the dominant arm of epsilon-greedy for small epsilon, the only accuracy-aware mechanism in the Planner actively selects confidence-declining models. The reported 17.36% average accuracy in Table I therefore cannot be explained by the published algorithm; either the formula, the selection direction, or the reported evaluation is inconsistent. This is load-bearing because the entire contribution is the score-based switcher, and the reader's REJECT verdict is justified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes EdgeMLBalancer, a self-adaptive model-switching approach for object detection on resource-constrained Android devices. The system monitors per-frame CPU usage and detection confidence, computes a performance score per model, and uses an epsilon-greedy policy to select the next model, with the aim of balancing computational efficiency, accuracy, and fairness of model usage. The evaluation compares EdgeMLBalancer against a Naive threshold-based approach and a Round-Robin-with-Boosting approach on a 30-minute Indian-traffic video, reporting in Table I that EdgeMLBalancer achieves the highest average accuracy (17.36%), competitive CPU usage, and improved model-usage fairness.","tokens_in":13306,"tokens_out":4516,"duration_ms":39084,"significance":"If the results held, the contribution would be of practical interest: a lightweight, on-device, MAPE-K-based model switcher with public code, real smartphone deployment, and a fairness-oriented exploration mechanism. The authors should be credited for prototyping on a real device, releasing code and models, and addressing threats to validity explicitly. However, the central decision-making formula is internally inconsistent with its stated goal, and the empirical evidence consists of one short run per approach with no statistical support. These issues place the main claims of significant accuracy and fairness improvements on an unsupported basis.","major_comments":[{"comment":"The score function S_mi = min(U_i, U_avg) * (1 - C_avg/C_i) is sign-inverted with respect to its stated purpose. Because the planner selects the model with the lowest score, a model whose current confidence C_i is below the historical average C_avg makes the factor (1 - C_avg/C_i) negative and is rewarded, so the only accuracy-aware signal actively favors confidence-declining models. The paper's own example (S_mi = -0.3627 described as 'a better score' due to 'slight decline in confidence') confirms the inversion. This is load-bearing: with exploitation probability 1 - epsilon, the published algorithm cannot explain the reported 17.36% average accuracy in Table I unless the formula, the selection direction, or the reported evaluation is corrected.","section":"III-A2, III-A3"},{"comment":"The central comparative claim rests on a single 30-minute run per approach. No error bars, confidence intervals, or statistical tests are reported, and the manuscript's conclusion-validity paragraph concedes the limited duration. The term 'significant improvements' in the abstract and RQ1 is therefore unsupported. Please report multiple independent runs, variance, and a defined ground-truth protocol for the accuracy metric; the current statement that accuracy is 'the percentage of correctly detected objects compared to the ground truth' omits how ground truth was obtained for a 30-minute Indian-traffic video.","section":"IV-A, IV-B, Table I"},{"comment":"The frame counts in Table I (1952, 2482, and 2458) over 30 minutes correspond to roughly 1.1–1.4 frames per second, yet Section IV-A states that the input is a 30-minute recording at 60 frames per second. Either frames are dropped or sampled, or the input was not processed continuously; this must be clarified because it directly affects the accuracy and CPU measurements and the real-time claim.","section":"IV-A, Table I"},{"comment":"The reported 43.62% fairness improvement is largely by construction. Epsilon-greedy explicitly randomizes model choice with probability epsilon, so it mechanically reduces selection disparity regardless of model quality or runtime context. To support the fairness claim, the authors should compare against a pure random-selection baseline with the same average switch rate, vary epsilon, and show that the resulting distribution reflects runtime conditions rather than only forced exploration.","section":"IV-B, RQ2, Figure 5"},{"comment":"The switching-time comparison in Figure 6 and Table I is not controlled: the approaches perform different numbers of switches and process different numbers of frames, so average switching time alone does not support the conclusion that EdgeMLBalancer has 'superior decision-making'. Please report per-switch distributions, switch frequency, and the context (e.g., idle versus active inference) in which switching times were measured.","section":"IV-B, RQ3"}],"minor_comments":[{"comment":"The notation 'arg min(U_i, U_avg)' in the score formula should be 'min(U_i, U_avg)', since a minimum value, not an argument, is intended.","section":"III-A2"},{"comment":"The citation '[5]' attached to the score formula is unclear; please specify whether the score is adopted from EcoMLS or newly introduced, and clarify the relationship.","section":"III-A2"},{"comment":"The text contains typos such as 'we measured different metrics metrics' and 'optimzal performance'; a proofreading pass is needed.","section":"IV-A"},{"comment":"There are grammatical errors in phrases such as 'the existing system lack robust mechanisms' and 'leaving to inefficiencies'; these should be corrected.","section":"Abstract, I"},{"comment":"If the box plots and averages are based on a single run per approach, the captions should state this explicitly; if multiple runs were performed, the number of runs should be given.","section":"IV-B, Figures 4 and 6"}],"recommendation":"reject","confidential_remarks":"The public code repository could allow a rapid check of whether the implemented score computation matches the published formula, but the current manuscript text is internally inconsistent on a load-bearing point. The sign-inversion issue and the single-run evaluation are sufficient to reject in this round; a resubmission would need a corrected objective or a clearly documented rationale for the selection direction, plus a statistically grounded evaluation with a defined ground truth."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: the score function in Section III-A2 is backwards. S_mi = min(Ui, Uavg) * (1 - Cavg/Ci) with arg-min selection means a model whose current confidence is below its historical average gets a negative score and is preferred. The paper's own numerical example says exactly this: -0.3627 is called better because of a 'slight decline in confidence.' So the only accuracy-aware term in the planner selects for declining confidence, and the reported 17.36% average accuracy cannot be explained by the published decision rule. This is load-bearing, and the REJECT verdict holds.\n\nWhat is real: the authors built a concrete Android prototype, integrated TensorFlow Lite models, released code on GitHub, and ran a 30-minute traffic-video comparison across three policies. The fairness analysis (model selection distribution) is a reasonable thing to measure, and epsilon-greedy randomization does plausibly reduce model starvation. That part is not a mirage.\n\nWhere it falls apart: one run per approach, no error bars or statistical tests, and no description of ground truth for accuracy. Epsilon and window size are never reported. The baselines ('Naive' and 'Round Robin with Boosting') are described vaguely enough that I cannot tell what the thresholds or boosting actually do. The score formula's sign error is not a minor typo; it is the decision core. The system optimizes its own defined score, so the CPU/accuracy trade-off numbers are not independent validation anyway. The fairness improvement is partly by construction, because random exploration spreads selections out.\n\nWho this is for: someone working on runtime model switching in self-adaptive ML systems might read it as a related-work note or a cautionary example, but not as a source of validated results.\n\nRecommendation: desk reject. The artifact is nice, but the central algorithmic claim is internally inconsistent, and the evaluation would not support the headline even if the sign were fixed. No referee time needed.","headline":"Section III-A2's score function rewards declining confidence, so the reported accuracy gains cannot be explained by the published algorithm.","tokens_in":13783,"tokens_out":2196,"would_cite":false,"duration_ms":20496,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A CPU-aware epsilon-greedy switcher balances accuracy, CPU use, and fairness for on-device object detection.","keywords":["self-adaptive systems","edge computing","dynamic model switching","epsilon-greedy","CPU utilization","object detection","MAPE-K loop","real-time monitoring"],"falsifier":"Recompute the score from the paper's own example: with $U_i = 13\\%$, $U_{\\text{avg}} = 18\\%$, $C_i = 54.42\\%$, $C_{\\text{avg}} = 55.94\\%$, the score $S_{m_i} = 13 \\times (1 - 55.94/54.42) \\approx -0.36$; if $C_i$ is raised to $60\\%$ while all else stays equal, the score becomes positive, so the planner would prefer the lower-confidence model, contradicting the paper's stated goal of favoring accuracy.","tokens_in":12765,"feed_emoji":"⚖️","tokens_out":8074,"duration_ms":56567,"temperature":0.7,"pith_summary":"EdgeMLBalancer is a self-adaptive system for real-time object detection on resource-constrained edge devices such as smartphones. It continuously monitors each candidate model's CPU usage and detection confidence, computes a performance score per model, and uses an epsilon-greedy policy to decide which model runs on the next frame. The paper claims that on 30 minutes of Indian traffic video, this approach achieves the highest average detection accuracy (17.36%) among the three compared approaches, keeps average CPU usage near 19.90%, and improves the fairness of model usage by 43.62% over a naive threshold-based switcher. The significance is a lightweight, self-contained control loop that adapts to workload without relying on cloud offloading.","feed_headline":"Epsilon-greedy switching beats threshold and round-robin on edge AI","feed_subtitle":"A self-adaptive MAPE-K loop picks object-detection models from live CPU usage, improving fairness by 43.62%.","key_machinery":"The MAPE-K feedback loop with the epsilon-greedy model-selector is the load-bearing mechanism. The Monitor collects $(m_i, C_i, U_i)$ per frame; the Analyzer computes windowed averages and the score $S_{m_i}$; the Planner applies epsilon-greedy to pick the next model; the Executor swaps the active model and logs metrics; the Knowledge component stores the model repository, score table, and log registry. The epsilon-greedy parameter $\\epsilon$ controls the exploration/exploitation trade-off, and the score formula is intended to combine CPU efficiency ($\\min(U_i, U_{\\text{avg}})$) with accuracy relative to history ($1 - C_{\\text{avg}}/C_i$).","core_discovery":"EdgeMLBalancer's central claim is that a self-adaptation loop built on the MAPE-K pattern can dynamically switch among four on-device object-detection models — EfficientDet Lite0, Lite1, Lite2, and SSD MobileNet V1 — to balance computational efficiency and detection accuracy on an Android device. For each frame the monitor records CPU usage $U_i$ and a confidence score $C_i$; the analyzer aggregates these over a window to historical averages $U_{\\text{avg}}$ and $C_{\\text{avg}}$, then assigns each model a score $S_{m_i} = \\min(U_i, U_{\\text{avg}}) \\times (1 - C_{\\text{avg}}/C_i)$. The planner selects the lowest-scoring model with probability $1-\\epsilon$, and with probability $\\epsilon$ randomly explores another model to avoid starvation. The evaluation on real-time traffic video reports a 43.62% improvement in model-usage fairness over the Naive baseline, higher average accuracy than both Naive and Round Robin with Boosting, and only a modest 0.85 s average switching time.","pith_inferences":["If the score formula is read literally, lowering the current confidence $C_i$ below the historical average $C_{\\text{avg}}$ makes the score more negative and hence more attractive to the planner; the reported accuracy gains may therefore owe more to epsilon-greedy exploration than to the score's accuracy term.","The fairness metric measures how evenly frames are distributed across models, but it does not weight models by their intrinsic quality; a perfectly fair allocation could select a weak model too often on easy frames.","A natural testable extension is to vary $\\epsilon$ over time, annealing it from exploration-heavy to exploitation-heavy, to see whether the accuracy/fairness balance improves further.","The authors note future plans for LLM-based vision and hybrid edge-cloud offloading; the current MAPE-K loop would need additional monitors (e.g., network latency) to make offloading decisions safely."],"forward_implications":["A threshold-free, self-adaptive switcher can run entirely on a smartphone, removing the need for cloud offloading in real-time monitoring.","The same MAPE-K loop can be repurposed for other resource metrics (memory, battery, temperature) and other model families, not just object detectors.","Fairness through epsilon-greedy prevents a single model from dominating, which the paper argues makes the system more robust to workload shifts.","The reported switching overhead of 0.85 seconds is small enough for near-real-time operation, though it is higher than the Naive approach's 0.50 seconds."],"supporting_citations":[{"why":"Supplies the MAPE-K feedback-loop pattern that structures the approach.","marker":"[13]"},{"why":"A prior self-adaptive green ML approach whose performance-score formula is reused here.","marker":"[5]"},{"why":"Defines the EfficientDet model family used as two of the four switching candidates.","marker":"[36]"},{"why":"Benchmarks object-detection models on edge devices, motivating the accuracy/compute trade-off.","marker":"[37]"},{"why":"Provides the adaptive epsilon-greedy exploration strategy used by the planner.","marker":"[38]"},{"why":"Compares epsilon-greedy with other exploration strategies in adaptive traffic control, supporting its choice.","marker":"[39]"},{"why":"An exemplar for self-adaptive ML-enabled systems that frames the model-switching scenario.","marker":"[3]"}],"fun_headline_variants":["EdgeMLBalancer: epsilon-greedy cuts edge AI unfairness by 43.62%","Self-adaptive loop picks edge models via CPU usage, boosts fairness 43.62%","Epsilon-greedy model switching improves edge AI fairness by 43.62%","MAPE-K loop with epsilon-greedy boosts edge AI model fairness 43.62%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The decision engine assumes that the score $S_{m_i} = \\min(U_i, U_{\\text{avg}}) \\times (1 - C_{\\text{avg}}/C_i)$ correctly ranks models with the lowest score as the best trade-off, yet its confidence term makes a drop in current confidence relative to the historical average produce a lower (more favorable) score, which runs opposite to the stated goal of favoring accuracy.","fun_headline_variants_meta":{"raw":{"variants":["EdgeMLBalancer: epsilon-greedy cuts edge AI unfairness by 43.62%","Self-adaptive loop picks edge models via CPU usage, boosts fairness 43.62%","Epsilon-greedy model switching improves edge AI fairness by 43.62%","MAPE-K loop with epsilon-greedy boosts edge AI model fairness 43.62%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001358,"raw_usage":{"total_tokens":5533,"prompt_tokens":987,"completion_tokens":4546,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":603,"completion_tokens_details":{"reasoning_tokens":4449}},"tokens_in":603,"tokens_out":4546,"duration_ms":27104,"temperature":1.0,"reasoning_tokens":4449,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T15:16:42.762931+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the score from the paper's own example: with $U_i = 13\\%$, $U_{\\text{avg}} = 18\\%$, $C_i = 54.42\\%$, $C_{\\text{avg}} = 55.94\\%$, the score $S_{m_i} = 13 \\times (1 - 55.94/54.42) \\approx -0.36$; if $C_i$ is raised to $60\\%$ while all else stays equal, the score becomes positive, so the planner would prefer the lower-confidence model, contradicting the paper's stated goal of favoring accuracy.","supporting_citations":[{"cited_title":"Eryilmaz, F","cited_arxiv_id":null,"evidence_quote":"Supplies the MAPE-K feedback-loop pattern that structures the approach."},{"cited_title":"EcoMLS: A Self-Adaptation Approach for Architecting Green ML-Enabled Systems","cited_arxiv_id":"2404.11411","evidence_quote":"A prior self-adaptive green ML approach whose performance-score formula is reused here."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the EfficientDet model family used as two of the four switching candidates."},{"cited_title":"A Comprehensive Evaluation of Deep Learning Object Detection Models on Heterogeneous Edge Devices","cited_arxiv_id":"2409.16808","evidence_quote":"Benchmarks object-detection models on edge devices, motivating the accuracy/compute trade-off."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the adaptive epsilon-greedy exploration strategy used by the planner."},{"cited_title":"Thadikamalla and P","cited_arxiv_id":null,"evidence_quote":"Compares epsilon-greedy with other exploration strategies in adaptive traffic control, supporting its choice."},{"cited_title":"SWITCH: An Exemplar for Evaluating Self-Adaptive ML-Enabled Systems","cited_arxiv_id":"2402.06351","evidence_quote":"An exemplar for self-adaptive ML-enabled systems that frames the model-switching scenario."}],"review_version":1}