{"id":"e1bb38df-9fcc-433a-bc14-85e618ec1839","arxiv_id":"2505.06641","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"SneakPeek estimates each model's accuracy from live data with a fast kNN and Bayesian update, then uses those estimates in grouped scheduling and short-circuit inference to raise utility on a single GPU.","lead":"This paper presents a scheduler that chooses which AI model variant should handle each inference request on a single edge GPU, using estimates of how well each model will perform on the actual incoming data. It reports up to a 2x improvement in scheduling utility for three healthcare applications compared to existing accuracy-scaling baselines.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq 9's stable-recall assumption is untested; covariate shift within classes could eliminate SneakPeek's utility gains.","rationale":"The reader identified the same weakest assumption: Eq 9 treats per-class recalls as fixed, profiling-derived constants while only the class frequencies theta vary with the data. This is the load-bearing assumption of the entire data-awareness mechanism. If recalls are not stable under within-class covariate shift, SneakPeek's accuracy estimates are biased regardless of how well theta is estimated, and the scheduling utility gains reported in Figure 5 are not guaranteed in the intended edge deployments. The paper gives no evidence on this point: Figure 6 measures accuracy-estimation error on the same distribution used for profiling, and the scheduling evaluation does not vary the input distribution beyond class-frequency changes. A concrete covariate-shift experiment would directly test whether the assumption holds and whether the central claim generalizes. The concern does not invalidate the paper's results for the tested distribution-matched scenario, so the reader's CONDITIONAL verdict remains appropriate and no verdict change is needed.","tokens_in":19276,"tokens_out":4352,"duration_ms":45388,"concrete_test":"Construct a covariate-shifted evaluation set for each application by applying realistic within-class perturbations (e.g., Gaussian noise or lighting changes for MMAct video, babble noise for Speech Commands, and different ECG lead/amplitude scaling for MIT-BIH) while keeping class frequencies identical to the original test set. Then: (1) recompute per-class recall for every model on the shifted set and compare with the profiled recalls, and (2) rerun the Figure 5 comparison on this shifted set to see whether SneakPeek's utility advantage over LO-EDF is preserved. If per-class recall changes by more than a few percentage points or the utility advantage disappears, the stable-recall assumption is violated and the central claim's scope is limited to distribution-matched settings.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires the per-class recall terms in Eq 9 to be stable between profiling time and online operation. Section IV-A asserts this explicitly: 'While the recall for a given class depends on the trained model, the frequency of each class (theta) depends exclusively on the test data set.' This is only valid if the conditional input distribution within each class does not change. In realistic edge deployments, within-class covariate shift is likely: camera angle and lighting for fall detection, microphone placement and background noise for speech commands, and electrode placement or lead configuration for ECG monitoring. Such shifts change the confusion matrix Z and therefore the true per-class recalls, so SneakPeek's accuracy estimate is biased even with a perfect theta estimate. The paper never measures recall stability: Figure 6 evaluates only theta estimation error under the same test distribution, and the scheduling experiments keep the data distribution fixed. Consequently, the utility gains in Figure 5 could vanish or reverse under covariate shift, and the central contribution is conditional on an untested invariance.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SneakPeek, a data-aware model selection and scheduling framework for inference serving on resource-constrained edge systems. It formulates joint model selection and scheduling as a utility-maximization problem, then estimates the label distribution theta online using a k-nearest-neighbor-based SneakPeek model and a Dirichlet-Multinomial posterior. These theta estimates are plugged into a class-frequency decomposition of model accuracy (Eq. 9) to obtain sharper, data-dependent accuracy estimates for each candidate model. The scheduler combines a priority-based ordering, greedy group-level batching, and an optional short-circuit inference mechanism that uses the SneakPeek model itself when deadlines are tight. The evaluation uses three healthcare-style applications (fall detection, speech commands, ECG monitoring) on a single GPU and reports that SneakPeek achieves up to a 2x improvement in scheduling utility over a locally-optimal EDF baseline.","tokens_in":19400,"tokens_out":3776,"duration_ms":42211,"significance":"The analytical starting point is correct and clearly presented: Eq. (9) is a standard identity that decomposes profiled accuracy into class frequencies times per-class recalls, and the Dirichlet-Multinomial update is mathematically sound. If the stable-recall assumption holds, SneakPeek is an appealing, model-agnostic mechanism for adapting to label distribution shift without modifying the candidate DNNs. The paper also makes a useful pragmatic contribution by integrating batching into the scheduling decision rather than treating it as an afterthought. However, the empirical claims are not yet fully supported: the core invariance assumption is untested, the evaluation lacks repeated trials and error bars, and the reported utility may be computed from the same accuracy estimates that the method is designed to improve. These issues are fixable within the scope of the manuscript, but they are load-bearing for the paper's central claim.","major_comments":[{"comment":"The central assumption that per-class recalls are stable between profiling time and online operation is load-bearing but untested. The text in Section IV-A states that recall depends only on the trained model, but this holds only if the conditional input distribution within each class remains unchanged. Realistic edge deployments exhibit within-class covariate shift (camera angle and lighting for fall detection, microphone placement for speech commands, electrode configuration for ECG), which changes the confusion matrix and hence the recall terms. Figure 6 evaluates only the error in estimating theta under the same test distribution; it never measures recall stability. Without an experiment that perturbs the within-class conditional distributions and reports both recall drift and end-to-end schedule utility, the central contribution is conditional on an unverified invariance. Please add such an experiment (e.g., synthetic covariate shifts such as blur, noise, or sensor offset) and report whether SneakPeek's utility gains persist.","section":"Section IV-A, Eq. (9) and Section VI"},{"comment":"The empirical evaluation does not report repeated trials or error bars, and it is unclear whether the reported utility is computed from realized inference outcomes or from the model-accuracy estimates that SneakPeek produces. Equation (2) defines utility in terms of an expected accuracy term, so if Figure 5's utility is computed using SneakPeek's own estimated accuracies, then the improvement over baselines is partly by construction. Please specify exactly how each utility number is computed in Figures 5, 7, 10, and 12, and report mean and standard deviation over multiple random request sets. In addition, report the realized (actual) accuracy of the selected models, not only the estimated accuracy, to demonstrate that the scheduling choices translate into better outcomes.","section":"Section VI-B and VI-C"},{"comment":"The grouped scheduling algorithm has two free hyperparameters that are never subjected to sensitivity analysis: the brute-force threshold tau and the subgroup-splitting threshold theta_i > 0.5. The reported benefits of grouped scheduling in Figures 10-12 may depend on tau, and the splitting rule is justified only by intuition. Please report the tau values used in each experiment and test alternative split thresholds (or state that the results are insensitive to these choices). This is important because the paper's practical recommendation to adopt group-level scheduling depends on these parameters.","section":"Section V-B, Algorithm 1, and Section VI-D"}],"minor_comments":[{"comment":"The utility definition uses a scalar Accuracy(mj), while Eq. (9) defines accuracy as a sum over classes parameterized by theta; the notation should clarify that the scalar in Eq. (2) is the expected accuracy under the current theta estimate, otherwise the two notions of accuracy are inconsistent.","section":"Equation (2)"},{"comment":"The phrase \"true model accuracy\" is potentially misleading: setting theta_i = 1 for the true class label gives the model's per-request conditional accuracy, not a long-run frequency. Please state explicitly that this is a per-request oracle estimate.","section":"Section VI-C1"},{"comment":"The caption says \"SneakPeek KNN\" but the text explains that approximate nearest neighbors (Faiss) is used; please make the caption consistent and specify that k=1 and k=5 are the number of neighbors in the SneakPeek model, not in the candidate classifiers.","section":"Figure 6 caption"},{"comment":"Typo: \"relativly\" should be \"relatively\" in the discussion of the grouped scheduler's utility.","section":"Section VI-B"},{"comment":"In the brute-force branch, the algorithm returns a solution for grouped requests, but the text does not define group-level utility for a group containing multiple requests; please state whether it is the average of the request utilities or another aggregation.","section":"Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is sound and the analytical framing is clean, but the evaluation currently does not establish robustness to covariate shift, and the lack of error bars makes the headline 2x utility claim difficult to assess. I would ask the authors to add the within-class shift experiment and repeated-trial statistics before considering acceptance. No code or artifact is mentioned, which weakens reproducibility for a systems paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dan, quick read for you. The paper's real contribution is narrower than the title suggests but still genuine: it treats model accuracy as a function of the live class-frequency vector theta instead of a single profiled number, and estimates theta online with a cheap kNN-plus-Dirichlet mechanism. Equation 9 is elementary, and the authors don't overclaim it. The new part is the online estimation and its integration into grouped, deadline-aware scheduling. That mechanism, with the short-circuit option, is a legitimate and modest step for edge inference serving.\n\nWhat the paper does well: it states the assumptions clearly, the math is coherent, and the evaluation covers three datasets, several model families, deadline/arrival/penalty variations, plus a multi-worker extension. The prior analysis is honest: they show that a strongly informative prior matching the true distribution can hurt, which is a nice touch. The overhead numbers are believable. The authors also cite the obvious related work; the self-citation to their earlier CCGrid paper is justified because that earlier work introduced the multi-modal accuracy-variance context they build on.\n\nThe main soft spot, which the stress-test note correctly identifies, is the stable-recall assumption in Eq 9. Section IV-A states it explicitly—recall depends on the model, theta on the data—but that is only true if the within-class input distribution doesn't shift. In real edge deployments (camera angle, microphone, electrode placement), within-class covariate shift is the norm, and it will change the confusion matrix and thus the true recalls. The paper never tests this. Figure 6 only evaluates theta-estimation error under the same distribution, so it cannot catch recall drift. That is the load-bearing limitation, and it deserves an explicit ablation or a sensitivity analysis rather than a passing acknowledgment.\n\nThe evaluation also has a smaller gap: no error bars or multiple trials, and the utility computation could be described more precisely. The 2x claim relative to LO-EDF bundles grouped scheduling with short-circuit inference, so it does not isolate the data-awareness contribution; Figure 7 does partially untangle this, and the incremental decomposition is helpful, but the headline figure remains conflated. These are presentation and rigor issues, not fatal flaws.\n\nWho is this for? Systems people working on edge inference or model selection. It is a useful engineering contribution, not a breakthrough. I would send it to a serious referee rather than desk-reject, and if I were refereeing I would ask for the covariate-shift test and error bars, then it should be publishable as is.\n\nWould I bring it to reading group? Maybe, as an example of a data-aware scheduler done carefully. I would cite it if I worked on edge inference serving.","headline":"A solid engineering paper with a real, modest idea—online class-frequency estimation for accuracy-aware edge scheduling—whose evaluation is good enough to publish but short of airtight; the stable-recall assumption is the one soft spot worth probing.","tokens_in":19956,"tokens_out":1329,"would_cite":true,"duration_ms":12277,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SneakPeek claims that estimating model accuracy from live label frequencies beats static profiling for edge inference scheduling.","keywords":["inference serving","edge computing","model selection","scheduling","accuracy scaling","data-aware estimation","class frequency estimation","inference batching"],"falsifier":"Run SneakPeek on a stream whose class frequencies match the profiling set but whose inputs are modified within each class, such as different camera angles or added sensor noise; if observed per-class recall drops measurably, then Eq. (9)'s estimated accuracy diverges from observed accuracy, demonstrating bias under covariate shift.","tokens_in":19025,"feed_emoji":"🎯","tokens_out":4117,"duration_ms":37747,"temperature":0.7,"pith_summary":"The paper tries to establish that on a single GPU serving multiple applications, the right way to pick a model for each request is to estimate the current data's class frequencies and compute each model's expected accuracy from them, rather than trusting one profiled accuracy number. It argues that static profiled accuracy is biased toward the test set's label distribution and can mislead even an optimal scheduler. If correct, accuracy scaling at the edge becomes robust to label distribution shifts without retraining or modifying models. The authors also claim that grouping requests by application and batching them, plus a 'short-circuit' fast model, yields higher-utility schedules with few deadline violations.","feed_headline":"Live data beats profiled accuracy for edge model choice","feed_subtitle":"A data-aware estimator picks the right model per request and cuts deadline violations on a single GPU.","key_machinery":"The load-bearing identity is Eq. (9): $\\mathrm{Accuracy}(m) = \\sum_i \\theta_i \\, \\mathrm{recall}_i$, which decomposes profiled accuracy into class frequencies times per-class recall. Since recall is treated as a stable property of the trained model, the only quantity that changes out-of-sample is $\\theta$, so the paper estimates $\\theta$ in real time with a SneakPeek model: a k-nearest-neighbor lookup that produces multinomial evidence $y$, combined with a Dirichlet prior to obtain posterior SneakPeek probabilities. Those probabilities sharpen every model's accuracy estimate in one inference, and they also enable short-circuit inference and data-aware subgroup splitting. Scheduling uses the request priority $(1 + \\mathrm{Var}[\\mathrm{Accuracy}(\\mathcal{M}_{a_i})])\\,e^{-d_i}$, locally optimal model choice, and greedy application-level grouping to exploit batching.","core_discovery":"SneakPeek's central claim is that model accuracy for a classification model is not a static property but a function of the class frequencies $\\theta$ of the data being served, via $\\mathrm{Accuracy}(m) = \\sum_i \\theta_i \\, \\mathrm{recall}_i(m)$. The paper proposes estimating $\\theta$ from live data using a lightweight SneakPeek model, built from k-nearest-neighbor evidence combined with a Dirichlet-Multinomial posterior, then using these sharpened accuracy estimates inside a scheduler that orders requests by a priority combining deadline urgency and accuracy variance, selects locally optimal models, groups requests by application for batching, and optionally uses the SneakPeek model itself to answer low-priority requests. On three healthcare-style applications, the combined approach achieves the highest utility, with a reported 2x utility increase over LO-EDF and almost zero deadline violations.","pith_inferences":["The same theta decomposition should extend to other metrics such as F1 and quadratic scores, which the paper's appendix sketches, so the mechanism is not accuracy-specific.","A testable extension would be to calibrate theta estimates online with a small labeled stream and measure how much scheduling utility improves as SneakPeek estimation error falls, giving an operational target for estimator quality.","The approach may transfer to multi-GPU settings where the benefit of batching shrinks but the benefit of data-awareness grows, as the paper's preliminary multi-worker results suggest.","Since strongly informative priors can suppress the data signal, production deployments should prefer weak or uninformative priors unless the label distribution is known to be stationary."],"forward_implications":["Existing schedulers that rely on a single profiled accuracy number are systematically biased when live class frequencies differ from the test set, and they can be improved by swapping in per-class recall profiles plus a theta estimator.","Because theta is shared across all models of an application, one cheap SneakPeek inference updates accuracy estimates for every candidate model at once.","Batching requests by application and choosing one model per group reduces model-swap overhead, which matters more than per-request optimality under tight deadlines.","When deadlines are too tight for any profiled model, using the SneakPeek model itself as a zero-latency fallback salvages utility instead of returning zero."],"supporting_citations":[{"why":"Supplies the confusion-matrix accuracy formula that the paper rewrites into Eq. (9), grounding the theta decomposition.","marker":"[28]"},{"why":"Motivates per-class accuracy variation and provides multi-modal models and baselines for the healthcare edge setting.","marker":"[7]"},{"why":"Representative accuracy-scaling system that the paper extends with data-awareness and grouped scheduling.","marker":"[14]"},{"why":"Provides the deadline-aware locally optimal model selection baseline that LO-EDF builds on.","marker":"[29]"},{"why":"Baseline for opportunistic batching, which the paper contrasts with explicit grouped scheduling.","marker":"[31]"},{"why":"Establishes the model-less inference serving context and motivates dynamic model selection as a system task.","marker":"[2]"},{"why":"Supplies the MMAct dataset used for the fall detection evaluation.","marker":"[32]"},{"why":"Supplies the Speech Commands dataset for the voice command application.","marker":"[36]"},{"why":"Supplies the MIT-BIH Arrhythmia database for the heart monitoring application.","marker":"[38]"},{"why":"Supplies the approximate nearest-neighbor implementation used to generate SneakPeek evidence.","marker":"[41]"}],"fun_headline_variants":["Live data picks the right model per edge request","Adaptive accuracy estimates improve edge inference scheduling","Edge inference scheduler uses data-aware accuracy scaling","SneakPeek reduces deadline slips on single-GPU edge"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes each model's per-class recall rates are stable and known from profiling, so only class frequencies change with the data; if the live data shifts within a class, the recall values themselves drift and the estimates are biased even with a perfect frequency estimate.","fun_headline_variants_meta":{"raw":{"variants":["Live data picks the right model per edge request","Adaptive accuracy estimates improve edge inference scheduling","Edge inference scheduler uses data-aware accuracy scaling","SneakPeek reduces deadline slips on single-GPU edge"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00021,"raw_usage":{"total_tokens":1399,"prompt_tokens":924,"completion_tokens":475,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":540,"completion_tokens_details":{"reasoning_tokens":414}},"tokens_in":540,"tokens_out":475,"duration_ms":5192,"temperature":1.0,"reasoning_tokens":414,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:37:25.802102+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run SneakPeek on a stream whose class frequencies match the profiling set but whose inputs are modified within each class, such as different camera angles or added sensor noise; if observed per-class recall drops measurably, then Eq. (9)'s estimated accuracy diverges from observed accuracy, demonstrating bias under covariate shift.","supporting_citations":[{"cited_title":"Leveraging multi-modal data for efficient edge inference serving,","cited_arxiv_id":null,"evidence_quote":"Motivates per-class accuracy variation and provides multi-modal models and baselines for the healthcare edge setting."},{"cited_title":"Proteus: A high-throughput inference-serving system with accuracy scaling,","cited_arxiv_id":null,"evidence_quote":"Representative accuracy-scaling system that the paper extends with data-awareness and grouped scheduling."},{"cited_title":"Layercake: Efficient inference serving with cloud and mobile resources,","cited_arxiv_id":null,"evidence_quote":"Provides the deadline-aware locally optimal model selection baseline that LO-EDF builds on."},{"cited_title":"Clipper: A Low-Latency online prediction serving system,","cited_arxiv_id":null,"evidence_quote":"Baseline for opportunistic batching, which the paper contrasts with explicit grouped scheduling."},{"cited_title":"INFaaS: Automated model-less inference serving,","cited_arxiv_id":null,"evidence_quote":"Establishes the model-less inference serving context and motivates dynamic model selection as a system task."},{"cited_title":"Mmact: A large-scale dataset for cross modal human action under- standing,","cited_arxiv_id":null,"evidence_quote":"Supplies the MMAct dataset used for the fall detection evaluation."},{"cited_title":"Speech commands: A public dataset for single-word speech recognition","cited_arxiv_id":null,"evidence_quote":"Supplies the Speech Commands dataset for the voice command application."},{"cited_title":"PhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physiologic signals,","cited_arxiv_id":null,"evidence_quote":"Supplies the MIT-BIH Arrhythmia database for the heart monitoring application."}],"review_version":1}