{"id":"2d45666e-7780-4d65-a71e-1fd5734c82b8","arxiv_id":"2607.21602","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A gated predictor that fuses static descriptors with runtime telemetry lifts LLM latency R² on Pixel devices (prefill from −1.38 to 0.97), but cross-device gains come from calibration fitting, not from a transferable model.","lead":"This paper builds a runtime-aware predictor that estimates prefill, decode, and total latency of small LLMs on edge devices by fusing model/device descriptors with live hardware telemetry, then shifting predictions to new devices with a small calibration set. Read it to see whether cheap pre-screening of LLMs for phones and edge boards is within reach: the short answer is maybe, on evidence thinner than the tables suggest.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation feeds test-run telemetry and realized token counts into the predictor, so Tables 5–6 do not validate pre-execution screening.","rationale":"I read the paper's strongest claim as the deployment-oriented screening story: lightweight calibration should let a deployer rank candidate LLMs on an unprofiled device. For this to work, the predictor must produce latency estimates from pre-execution information only. The reader's weakest assumption identifies the crux: the dynamic path requires runtime telemetry, and the evaluation protocol randomly splits complete request records, so test-time predictions have access to the very run they are predicting. This is not just a minor leak; it directly contradicts the pre-execution screening scenario. The paper also includes realized generated-token count in the prompt-side descriptors (§4.1), which is another post-hoc signal. The static-only baseline itself may therefore be unrealistically strong for screening, and the full model's gains over it may be even less transferable. I checked whether the paper clarifies the timing anywhere: §3.6 mentions using calibration traces to estimate dynamic behavior, but the evaluation section never states that test traces are excluded. The ablation in Table 8 shows that without calibration, cross-device transfer fails badly, so the entire positive transfer result depends on how calibration is applied. Given this unresolved ambiguity, the reader's CONDITIONAL verdict is appropriate. I do not see grounds to reject outright—the framework might survive if re-evaluated with a screening-faithful protocol—but the current evidence does not establish the headline claim. Hence I keep the verdict unchanged and propose one concrete re-evaluation that would settle whether the concern lands.","tokens_in":15805,"tokens_out":5839,"duration_ms":65466,"concrete_test":"Re-run the within-device and cross-device transfer evaluations (Tables 5–6) under a screening-faithful protocol: at test time, feed the dynamic path only trace statistics derived from the calibration set (e.g., a pooled/averaged dynamic embedding from calibration runs), never the test sample's own trace; also remove generated-token count from the static features. Compare against a static-only baseline built only from pre-execution features (device, model, input length, max generation length). If the full model's R² no longer exceeds that baseline, the pre-execution screening claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the framework can screen unprofiled LLMs on a target device using only a small calibration set, before full execution. But the dynamic path (Eq. 3) consumes runtime telemetry collected during an execution, and the evaluation protocol (§4.3) randomly partitions complete request records, so each test sample carries its own hardware counters. In Tables 5–6, the full model therefore predicts a run's latency partly from that run's own telemetry—a self-predictive signal unavailable at screening time. Similarly, the static branch includes the realized generated-token count (§4.1), which is unknown before generation. The paper never states that test traces are restricted to calibration-set traces; the protocol as written suggests they are not. Consequently, the reported R² improvements do not demonstrate the advertised 'fast LLM screening before full target-device execution.' The deployment-time closed loop (§3.8) profiles selected candidates only after selection, so it cannot provide dynamic input for an unprofiled candidate. This is an internal inconsistency between the evaluation and the deployment scenario, not merely a missing baseline.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a runtime-aware latency prediction framework for screening candidate LLMs on heterogeneous edge devices. A deployment is represented as a hardware-runtime-model-prompt configuration, with a static branch for pre-execution descriptors and a dynamic branch consuming runtime telemetry; the two are combined through gated fusion, and a small target-device calibration set is used to adapt the predictor. The evaluation reports within-device and cross-device R² and MAE results on Pixel 8/Pixel 8 Pro, heterogeneous profiling on Jetson Nano, Orange Pi 5 Pro, and an RTX 3090-class GPU, plus a small deployment-screening study. The central claim is that runtime-aware prediction with lightweight calibration reduces profiling cost and supports latency-aware LLM deployment before full target-device execution.","tokens_in":15947,"tokens_out":5544,"duration_ms":56574,"significance":"If the claimed results held under a protocol that matches the screening use case, the framework would be practically valuable: it would let a deployer rank or filter candidate LLMs for an unprofiled device using only a few calibration runs, avoiding exhaustive benchmarking. The paper has notable strengths: a clean configuration formulation, explicit prefill/decode phase modeling, a multi-platform profiling pipeline, and a well-motivated calibration loop. It also makes concrete falsifiable predictions (Tables 5–6) and documents its own limitations in §5.7. However, the advertised pre-execution screening scenario is not currently validated because the evaluation protocol appears to feed test-run telemetry and realized token counts into the predictor; moreover, the reported R² values rest on very small test sets and the profiling-cost reduction is never measured. With a corrected protocol and uncertainty quantification, the approach could be a solid contribution; in its present form, the evidence does not support the abstract's headline claims.","major_comments":[{"comment":"The evaluation protocol randomly partitions complete request records into training and test sets (§4.3). The dynamic path (Eq. 3) consumes runtime telemetry from an execution, so under this protocol each test record's dynamic input is that test run's own telemetry. At screening time, however, the candidate model has not been executed and no such trace exists; §3.8 explicitly profiles selected candidates only after selection. The paper never states that test traces are restricted to calibration traces. Consequently, the full-model R² improvements (e.g., Pixel 8 Pro prefill R² −1.383→0.966) may be partially self-predictive and do not demonstrate the advertised 'fast LLM screening before full target-device execution.' Please re-evaluate with dynamic features derived only from calibration runs, or explicitly justify why the current protocol matches the deployment scenario.","section":"§4.3, Eq. (3), Tables 5–6"},{"comment":"The prompt-side descriptors recorded for each request include 'generated token count' (§4.1). If this realized output length is used as an input feature in the static path (Eq. 2 / §3.4.1), then the predictor is given the answer to part of what it must estimate for decode and total latency. §3.4.1 lists input length and maximum generation length but does not list generated token count; §4.1 does record it. Please clarify whether generated token count is a feature or only a target/statistic. If it is a feature, the within-device results (Table 5) and transfer results (Table 6) are optimistic relative to the pre-execution screening scenario.","section":"§4.1, §3.4.1"},{"comment":"The headline metrics are computed on 11–41 test samples within devices, with only 6 or 20 calibration samples in transfer. No confidence intervals, repeated random splits, or significance tests are reported. For example, Pixel 8 total R² improves from 0.953 to 0.960 with 11 test samples, which is well within sampling noise, while Pixel 8 Pro total R² actually degrades from 0.978 to 0.967 but is presented as supporting the full model. The central cross-device claims (e.g., Pixel 8 Pro→Pixel 8 decode R² −1.085→0.927) are therefore not yet statistically supported. Please add uncertainty quantification or repeated splits and interpret differences accordingly.","section":"Tables 5–6, §4.3"},{"comment":"The abstract and §1 claim the framework can 'reduce profiling cost' and 'screen candidate models before full target-device execution,' but the screening experiment in §5.6/Table 9 only reports filtering ratios on 2–3 candidate models and never measures profiling cost, wall-clock time, or savings relative to exhaustive benchmarking. The cost-reduction claim is asserted rather than measured. A simple accounting experiment—number of runs avoided, profiling time saved, or cost versus exhaustive search—would make the claim operational.","section":"Abstract, §5.6, Table 9"},{"comment":"The paper twice states that 'Qwen-family models' are excluded from the reported model pool (§5.1, §5.4), but Qwen models do not appear in Table 2 or Table 4 and no justification for the exclusion is given. If the exclusion is due to data quality or compatibility, say so explicitly; otherwise the reported model pool is a selected subset and its representativeness should be discussed.","section":"§5.1, §5.4"}],"minor_comments":[{"comment":"Please state whether the static-only baseline includes the generated-token count. This affects the interpretation of the ablation and the within-device comparisons.","section":"§5.2, Table 5"},{"comment":"The hyperparameters α, λ_pre, λ_dec, and the architecture of the gating network g(·) are not specified, and no sensitivity analysis is reported.","section":"Eqs. (7)–(9), §3.4.3"},{"comment":"The sentence 'For LLM inference, latency is shaped not only by static model and hardware properties, but also by execution structure and runtime state' appears twice in the same subsection; remove the duplicate.","section":"§6.3"},{"comment":"The calibration counts are very small (6 or 20 samples). Please describe exactly how the calibration set is constructed (number of runs, prompts, models) and how the split is performed.","section":"Table 6"},{"comment":"The normalized phase time τ is defined, but the 'state' axis, alignment procedure, and state-space smoothing method are not specified. Please clarify the reconstruction algorithm and its parameters.","section":"Figure 2"},{"comment":"Some references (e.g., [3], [4]) lack full venue or publication details; please format consistently.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern is real and should be the first issue the authors address: the evaluation protocol does not match the screening use case because test records appear to carry their own runtime telemetry and realized token counts. If the authors cannot restrict dynamic features to calibration-run traces, the central claim must be substantially weakened. The small test sets and missing cost measurements also undermine the headline R² improvements and the 'reduce profiling cost' statement. I recommend major revision rather than rejection because the underlying framework is reasonable and could become valid with a corrected evaluation and appropriate uncertainty quantification."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper's headline screening story doesn't survive a close read of the evaluation protocol, but the engineering work is honest and the profiling data has real value. The problem: §4.3 randomly partitions complete request records, so test samples carry their own runtime telemetry. The dynamic path (§3.4.2) eats that telemetry, meaning the full model is partly predicting a run from its own hardware counters. At screening time, an unprofiled candidate has no such trace. The paper never says test traces are restricted to calibration-set traces, and the protocol as written implies they aren't. So the strong R² gains — Pixel 8 Pro prefill from −1.383 to 0.966, Pixel 8 Pro→Pixel 8 total from −0.974 to 0.940 — don't demonstrate pre-execution screening. That's the load-bearing flaw.\n\nCredit where due: the authors are upfront that the non-mobile platforms are only profiling validation, and the ablation makes clear calibration is what fixes cross-device transfer. The heterogeneous results (same SmolLM2 at 8.42 tok/s on Orange Pi vs 64.38 on RTX 3090) are a useful concrete illustration. The prefill/decode split with gated fusion is a sound design, even if each piece is in the cited literature.\n\nOther soft spots, in order: Qwen-family models are excluded twice (§5.1, §5.4) with no explanation; R² values rest on 11–41 test samples with no confidence intervals; the realized generated-token count is recorded as a descriptor (§4.1) and is unknown before generation, compounding the leakage; the claimed cost reduction is never measured; no code or data released. None of these is fatal by itself — they're fixable.\n\nThis is a paper a serious editor should send to reviewers, not desk reject. The evaluation needs to be re-run so that unprofiled candidates only use static descriptors plus calibration-set traces, and the authors need to justify the Qwen exclusion and add error bars. For readers working on predictive latency for edge LLMs, the design discussion and profiling observations are worth a skim, but the headline numbers should be treated with caution until the telemetry-timing issue is resolved.","headline":"The screening claim is undercut by test-run telemetry leakage in the evaluation protocol, but the engineering is honest and the profiling data has real value; deserves serious review.","tokens_in":16585,"tokens_out":2948,"would_cite":false,"duration_ms":31336,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that LLM latency on edge devices can be predicted accurately enough for model screening by fusing static descriptors with runtime telemetry and calibrating on a handful of target-device runs.","keywords":["latency prediction","on-device LLM","edge deployment","runtime telemetry","phase-aware modeling","model screening","cross-device transfer","calibration"],"falsifier":"Run a controlled screening experiment where a candidate model and prompt that never appeared in training or calibration is predicted on a fresh target device, using only static descriptors plus a calibration set collected from unrelated requests. Compare predicted versus measured total and decode latency; if R² falls to the negative zero-shot values reported before calibration, the screening benefit is not realized. Also report bootstrap confidence intervals for R², since the headline values rest on 11–41 test samples.","tokens_in":15570,"feed_emoji":"⚡","tokens_out":6034,"duration_ms":63649,"temperature":0.7,"pith_summary":"Latency-aware deployment of LLMs on edge devices is currently expensive because every candidate model must be benchmarked on every target device. This paper argues instead that latency can be predicted from a hardware-runtime-model-prompt configuration, and that a predictor combining static descriptors with runtime telemetry—separated into prefill and decode—can transfer across devices once anchored by a small calibration set. The experiments show calibration converting cross-device decode R² from -1.085 to 0.927 and total R² from -0.974 to 0.940 on mobile phones, and within-device prefill R² from -1.383 to 0.966. If correct, the result turns exhaustive profiling into a short calibration step, making model screening practical for growing model zoos.","feed_headline":"A few calibration runs predict LLM latency on new edge devices","feed_subtitle":"Instead of benchmarking every model on every device, a short calibration anchors a transferable latency predictor.","key_machinery":"The central mechanism is a two-branch predictor with gated fusion. The static branch encodes known-before-run descriptors—device, model, runtime backend, prompt length, max generation length—into a static embedding. The dynamic branch segments the runtime trace into prefill and decode regions, aligns repeated prefill runs to normalized phase time to reconstruct sparse bursty states, and passes the sequence through a temporal encoder to produce a dynamic embedding. A gating network computes softmax weights over the two embeddings, and a multi-output regression head jointly predicts prefill, decode, and total latency. A small target-device calibration set then adjusts the predictor's scale, wh","core_discovery":"The paper's central claim is that runtime-aware prediction with lightweight calibration can replace exhaustive profiling for LLM screening on heterogeneous edge devices. It represents each inference request as a configuration of device, model, runtime, and prompt; models prefill and decode separately; and fuses static and dynamic embeddings through a gated mechanism. The load-bearing evidence is that within-device prediction is strong (decode R² 0.973 on Pixel 8, prefill R² 0.966 on Pixel 8 Pro) and that calibration pulls cross-device transfer from strongly negative to usable R² (0.927 decode, 0.940 total for Pixel 8 Pro → Pixel 8). The paper also documents that latency is highly device- and","pith_inferences":["Beyond the paper: the reported transfer experiments use only 6 calibration samples (Pixel 8 Pro → Pixel 8), so the practical recipe implied here is that a device owner can anchor an entire model zoo with a handful of representative runs; this is worth testing at zoo scale, since the screening evaluation itself only covers 2–3 candidates.","Beyond the paper: the finding that smaller models are slower than larger ones on the GPU-class platform suggests parameter count is a weak screening signal; a deployer should treat runtime and kernel effects as first-order, which strengthens the case for learned prediction over rule-of-thumb cost models.","Beyond the paper: since the dynamic traces already include power-related signals, the same architecture could be repurposed to predict energy or memory alongside latency; the paper lists this as future work.","Beyond the paper: a stricter split that isolates calibration traces from test traces would resolve whether the calibration gains are due to genuine domain alignment or to partial leakage of the target run's own telemetry."],"forward_implications":["A deployer can rank candidate LLMs on an unprofiled device using a handful of calibration runs instead of exhaustive benchmarking, cutting profiling cost.","Because prefill and decode are predicted separately, screening can enforce phase-specific budgets such as first-token latency and steady-state tokens per second.","The calibration loop is cumulative: each profiled candidate adds traces that refine future predictions on the same device.","Combining predicted latency with quality scores in a Pareto step can remove dominated candidates before full target-device execution while preserving the best feasible model.","The profiling schema extends across mobile, embedded GPU, single-board, and desktop GPU classes, so source-device data from one class can seed prediction for another."],"fun_headline_variants":["Calibrate once, predict LLM latency across edge devices","Skip full profiling: short calibration predicts edge LLM speed","Lightweight calibration transfers LLM latency prediction to new devices","Edge LLM screening: a few runs estimate latency on any device","Rapid calibration replaces exhaustive LLM benchmarking on edge"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The predictor's dynamic path uses live hardware telemetry from the execution being predicted, and the paper does not state whether test records' traces come from the test run itself; if they do, the within-device gains do not transfer to the advertised pre-execution screening use case.","fun_headline_variants_meta":{"raw":{"variants":["Calibrate once, predict LLM latency across edge devices","Skip full profiling: short calibration predicts edge LLM speed","Lightweight calibration transfers LLM latency prediction to new devices","Edge LLM screening: a few runs estimate latency on any device","Rapid calibration replaces exhaustive LLM benchmarking on edge"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000227,"raw_usage":{"total_tokens":1364,"prompt_tokens":857,"completion_tokens":507,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":601,"completion_tokens_details":{"reasoning_tokens":424}},"tokens_in":601,"tokens_out":507,"duration_ms":5305,"temperature":1.0,"reasoning_tokens":424,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T14:18:33.701612+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a controlled screening experiment where a candidate model and prompt that never appeared in training or calibration is predicted on a fresh target device, using only static descriptors plus a calibration set collected from unrelated requests. Compare predicted versus measured total and decode latency; if R² falls to the negative zero-shot values reported before calibration, the screening benefit is not realized. Also report bootstrap confidence intervals for R², since the headline values rest on 11–41 test samples.","supporting_citations":[],"review_version":1}