{"id":"1be5efa9-447c-4fd0-b63e-323e071a0fcf","arxiv_id":"2506.17538","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":5,"one_line_summary":"ConsumerBench reveals that running multiple generative AI apps concurrently on a consumer GPU causes severe starvation under greedy allocation and wasted capacity under static partitioning, driving the need for SLO-aware scheduling.","lead":"ConsumerBench is a new benchmark that measures how generative AI apps, such as chatbots, image generators, and live captioning, behave when they run at the same time on a laptop or desktop GPU. It shows that common GPU-sharing strategies cause serious slowdowns and missed response-time targets, and suggests smarter, SLO-aware scheduling as a fix.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central SLO-attainment results are threshold-sensitive: with zero-slack LiveCaptions timing and a 1s/step ImageGen SLO, a plausible re-ranging of SLOs or arrival rates could erase the reported starvation/underutilization contrast.","rationale":"The reader's weakest_assumption and my concern align: the benchmark's headline outcomes are stated relative to arbitrary SLO thresholds. This is load-bearing because all of Sections 4.2–4.4's conclusions are SLO attainment comparisons. However, I do not think this invalidates the framework; it weakens the unqualified empirical claims. The paper has independent support: open-source code, multiple hardware platforms (though with model changes on Apple), and plausible mechanisms (large kernels blocking small kernels under FCFS). The condition for acceptance should be a sensitivity analysis or explicit caveats. Therefore the reader's CONDITIONAL verdict is appropriate; no adjustment needed. Secondary issues (single primary GPU, changed models/SLOs on Apple Silicon, no error bars) reinforce the same conditional posture but are not more load-bearing than the SLO/threshold dependence.","tokens_in":16128,"tokens_out":8731,"duration_ms":99597,"concrete_test":"On the same RTX 6000 setup, sweep the parameter space for the three latency-sensitive apps: (1) LiveCaptions: fixed 2 s segments, vary inter-arrival interval over {1,2,3,4,6} s and SLO over {1,2,3,4} s; (2) ImageGen: SLO per denoising step over {0.5,1,1.5,2} s; (3) Chatbot: vary TTFT target over {0.5,1,2} s and TPOT over {0.125,0.25,0.5} s. For each configuration, run greedy allocation and static MPS partitioning, with at least three repetitions, and plot SLO attainment curves.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline findings are expressed as SLO attainment rates, but all SLO values and request timing choices in Table 1 and §3.3 are hand-picked and no sensitivity analysis is given. For LiveCaptions, the workload sends a 2 s segment every 2 s with a 2 s SLO: the deadline equals the inter-arrival period, so the configuration has zero queueing slack by construction, and the exclusive-GPU baseline itself misses 2% of segments (§4.1, Fig. 3). Under any resource-sharing strategy, such a tight periodic workload is predisposed to miss, so the reported '12.4x average latency increase / almost all requests miss SLO' is not a clean measurement of GPU scheduling unfairness; it is entangled with the zero-slack workload design. For ImageGen, the 1 s per-denoising-step SLO sits only ~1.8x above the exclusive-GPU step time (~560 ms, Fig. 4b); once MPS partitioning degrades step time past 1 s, the headline 'misses SLO despite available capacity' follows automatically. A plausible 1.5–2 s per-step SLO (in line with interactive image generation) would remove the ImageGen miss, and a 3–4 s LiveCaptions deadline would likely remove the 'almost all requests miss' result. Consequently, the qualitative contrast in §4.2–4.3 between greedy allocation and static partitioning is not established as a general property of end-user devices; it is established only for one point in the SLO/arrival parameter space. The framework itself is a reasonable contribution, but the 'reveals' conclusions require a robustness sweep before they can be stated without qualification.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents ConsumerBench, a benchmarking framework for evaluating generative AI applications on end-user devices under concurrent multi-application execution. ConsumerBench lets users define applications, models, SLOs, and workflow dependencies in YAML, then orchestrates execution under different GPU sharing strategies (greedy allocation, static MPS-based partitioning, and shared inference servers), while collecting application-level metrics (latency, SLO attainment) and system-level metrics (GPU/CPU utilization, memory, power). The evaluation on an RTX 6000 server and an Apple M1 Pro laptop covers four applications: Chatbot, DeepResearch, ImageGen, and LiveCaptions. The headline findings are that greedy GPU allocation starves lightweight latency-sensitive applications (LiveCaptions shows a 12.4x average latency increase and almost all requests miss SLO), static GPU partitioning underutilizes the GPU and causes ImageGen to miss SLOs despite available capacity, and naive model sharing via an inference server with a large KV cache degrades Chatbot SLO attainment.","tokens_in":16396,"tokens_out":5290,"duration_ms":58915,"significance":"If the findings are robust, ConsumerBench fills a real gap: existing benchmarks assume exclusive hardware access, while end-user devices run multiple GenAI applications concurrently on a single GPU. The open-sourced implementation and the custom workflow support are concrete strengths, as is the effort to connect application-level SLO misses to system-level metrics such as SM occupancy (SMACT/SMOCC). The paper makes falsifiable empirical observations and offers practical insights about kernel design and SLO-aware scheduling. However, the significance of the headline 'reveals' claims is currently tempered by the limited experimental scope and by the fact that the central SLO-attainment results depend on a single, hand-picked set of thresholds without sensitivity analysis. The framework itself is a reasonable contribution; the paper's broader conclusions about greedy vs. static partitioning are not yet established as general properties of end-user devices.","major_comments":[{"comment":"The SLO attainment results are threshold-sensitive, and no sensitivity analysis is provided. The LiveCaptions workload sends a 2-second audio segment every 2 seconds with a 2-second SLO, so the deadline equals the inter-arrival period, leaving zero queueing slack; this configuration is predisposed to SLO misses under any form of added latency, and indeed the exclusive-GPU baseline already misses 2% of segments (§4.1). The ImageGen SLO of 1 second per denoising step is only about 1.8x the exclusive-GPU step time (~560 ms, Fig. 4b), so once MPS degrades step time beyond 1 second the 'misses SLO despite available capacity' result follows automatically. The authors should either add a sensitivity analysis varying the SLO multipliers and request arrival patterns, or explicitly reframe the conclusions as point-in-parameter-space demonstrations rather than general properties of resource-sharing strategies.","section":"§4.2 and Table 1"},{"comment":"The quantitative claims appear to be based on a single run per configuration; no confidence intervals, standard deviations, or repeated trials are reported. This is load-bearing for statements such as '12.4× increase in average end-to-end request latency' and 'misses its SLO for approximately 40% of its requests', especially because the paper itself notes high variance for Chatbot-KVCache-CPU. At minimum, the authors should report the number of runs and provide mean ± std or percentile ranges for the headline latency and SLO-attainment numbers.","section":"§4.2 and §4.2.1"},{"comment":"The claim that static partitioning causes ImageGen to 'miss SLOs despite available compute capacity' rests on the stairstep SMACT/SMOCC pattern in Fig. 5a, but the paper does not quantify the idle GPU capacity available during ImageGen's execution. To make the causal link concrete, the authors should measure the fraction of time ImageGen has pending work while its MPS partition is idle and show that this idle time is directly responsible for the SLO miss, rather than relying solely on the 1-second threshold being crossed.","section":"§4.2, Static GPU Partitioning paragraph"}],"minor_comments":[{"comment":"In the second paragraph, 'credibleon-deviceinference' is missing spaces and should read 'credible on-device inference'.","section":"§2.1"},{"comment":"The SLO attainment bars in Fig. 3b are not clearly labeled for GPU vs. CPU; a legend or direct axis labels would make the 0% vs. 98% values interpretable.","section":"Figure 3"},{"comment":"The 2-second segment / 2-second SLO description leaves unclear whether the SLO includes HTTP and network overhead and whether segments are processed sequentially; please clarify the pipeline and state that this is a zero-slack workload by construction.","section":"§3.3, LiveCaptions"},{"comment":"The Apple Silicon experiments use a different image model (SD-v1-4 vs SD-3.5) and a 4-second LiveCaptions SLO; the authors should explicitly state that these results are not directly comparable with the NVIDIA experiments because the configuration is not held fixed.","section":"Appendix C"},{"comment":"The limitations section does not mention the sensitivity of the results to the chosen SLO values or the absence of repeated runs; both should be acknowledged as limitations of the current experimental study.","section":"Section 6, Limitations"}],"recommendation":"major_revision","confidential_remarks":"The paper is a systems/benchmarking contribution with a useful open-source framework, but the experimental validation is currently too thin to support the strong 'reveals' language. The SLO-sensitivity issue is likely to be raised by other reviewers and should be addressed head-on, either with a systematic sensitivity study or by substantially softening the conclusions. The lack of repeated runs is also a reproducibility concern for a benchmark paper. I see no circularity or internal inconsistency; the framework is sound as a design. With added sensitivity analysis and repeatability data, the paper could become acceptable for a journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper fills a real gap: ConsumerBench runs multiple GenAI applications concurrently on one consumer GPU and reports SLO attainment plus system-level metrics, something MLPerf, PalmBench, and MELT do not do because they assume exclusive hardware access. The DAG workflow support is a nice addition, and the open-sourced code is a concrete asset. The interference effects it surfaces are credible and well-illustrated: LiveCaptions' small decode kernels get stalled by ImageGen's large kernels, decode runs ~30x slower, and average end-to-end latency goes up 12.4x. The MPS static-partition underutilization shows up clearly in the stairstep SMACT/SMOCC traces, and the kernel-level analysis (register pressure in SD3.5 attention, Whisper decoder inefficiency) is sensible.\n\nThe soft spots are real, and the stress-test note lands. The LiveCaptions workload sends a 2-second segment every 2 seconds with a 2-second SLO, so the deadline equals the inter-arrival period—zero slack by construction. The exclusive-GPU baseline already misses 2% of segments, meaning any added delay becomes a violation. The \"almost all requests miss\" result under greedy is thus partly a workload-design artifact. Similarly, ImageGen's 1-second-per-step SLO is only about 1.8x the exclusive step time, so MPS degradation pushing step time past 1s automatically yields a miss. A robustness sweep over SLO values and arrival rates is needed before the greedy-vs-partitioning contrast can be read as a general property. The experiments also appear to be single runs with no error bars, only one NVIDIA GPU (RTX 6000) plus the Apple Silicon comparison, which changes both models and SLOs, making cross-platform claims suggestive rather than apples-to-apples.\n\nThese are fixable in revision. The core framework and the qualitative latency observations remain solid, and the \"first\" workflow-claim could be toned down but the contribution stands. This is a paper for anyone building on-device GenAI runtimes, schedulers, or kernel-level optimizations, and it deserves a serious referee. I would send it to peer review and ask for sensitivity analysis, repeated runs with variance reporting, and at least one more NVIDIA-class consumer GPU before the \"reveals\" language is fully earned.","headline":"A genuinely useful and open-sourced benchmark for concurrent on-device GenAI, with plausible headline findings that are presented without sensitivity analysis; deserves peer review after a robustness pass.","tokens_in":734,"tokens_out":755,"would_cite":true,"duration_ms":36441,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ConsumerBench argues that benchmarking generative AI on end-user devices must include concurrent multi-application interference, since exclusive-access benchmarks miss the starvation and underutilization that dominate real use.","keywords":["generative AI benchmarking","end-user devices","concurrent GPU workloads","service level objectives","GPU resource sharing","on-device inference","multi-application workflows","SLO attainment"],"falsifier":"Run the same three-application concurrent workload on the same GPU but change one plausibly realistic parameter—for example, set LiveCaptions' SLO to 4 seconds per segment (the value the paper itself uses on Apple Silicon) while keeping the 2-second arrival rate—and count SLO misses under greedy allocation; if LiveCaptions then meets most SLOs, the starvation conclusion is an artifact of the chosen threshold. Alternatively, replace ImageGen's generic attention kernel with a low-register custom kernel that raises SM occupancy and check whether LiveCaptions' decode slowdown disappears under greedy allocation.","tokens_in":15877,"feed_emoji":"📊","tokens_out":9528,"duration_ms":89330,"temperature":0.7,"pith_summary":"ConsumerBench argues that current GenAI benchmarks, which run one model alone on a dedicated GPU, do not represent the end-user device experience, where chatbots, captioning, and image generation compete for a single GPU. The paper introduces a configurable framework that turns a user's application list, request patterns, SLOs, and dependencies into a workflow DAG (a directed acyclic graph of dependent tasks), executes the applications concurrently, and reports both per-application SLO attainment and system-level utilization. In experiments on a consumer GPU, the framework finds that greedy resource allocation lets image-generation kernels starve live-captioning's small kernels, raising average end-to-end caption latency by 12.4x and causing almost all caption requests to miss their 2-second target. Static GPU partitioning removes that starvation but reserves GPU slices rigidly, underutilizing the device and causing image generation to miss its SLO even though spare compute exists. The authors conclude that on-device GenAI systems need SLO-aware scheduling, dynamic resource partitioning, and kernels designed for consumer GPU architectures.","feed_headline":"Greedy GPU sharing starves lightweight AI apps","feed_subtitle":"Benchmark of on-device GenAI shows naive sharing misses latency targets that exclusive-access tests never expose","key_machinery":"The machinery is a DAG-based execution engine: a user configuration is compiled into a directed acyclic graph whose nodes are application setup, execution, and cleanup, and whose edges encode dependencies; an orchestrator then runs the graph under a chosen resource-sharing strategy. For each application the framework measures SLO attainment on latency metrics (time to first token and time per output token for the chatbot, per-denoising-step time for image generation, per-segment time for live captions) and compares them against user-set thresholds. A system monitor records how much of the GPU is reserved (SMACT) versus actually occupied (SMOCC), which is the diagnostic that exposes low-occupancy kernels in image generation and caption decoding. The contrast between greedy allocation and static partitioning is the experiment that carries the main argument: it isolates starvation (greedy) from underutilization (static partitioning) as the two failure modes of resource sharing on end-user devices.","core_discovery":"The central claim is that the performance of GenAI on end-user devices can only be understood under concurrent, multi-application execution, and that naive resource-sharing strategies fail the SLOs users actually experience. With three latency-sensitive applications sharing one GPU, the paper shows that greedy first-come-first-served allocation is unfair: LiveCaptions' decoder phase runs 30x slower than under exclusive access because its small kernels wait behind ImageGen's large kernels, producing a 12.4x increase in average request latency and near-total SLO misses. Static GPU partitioning, by contrast, prevents the stalling but rigidly gives each app 33% of the GPU even when one app finishes, leaving ImageGen unable to use idle capacity and missing its 1-second-per-step SLO. The paper also shows that sharing one model across applications through an inference server with a static configuration (a large attention key-value cache for a background research agent) makes the latency-sensitive chatbot miss its SLO for about 40% of requests, because CPU-side attention becomes a bottleneck. The paper presents ConsumerBench as the first benchmark to include user-defined collaborative workflows and to pair application-level SLO metrics with system-level GPU/CPU utilization, memory bandwidth, and power measurements.","pith_inferences":["This suggests a direct test not run in the paper: patching only the Whisper decoder kernels to raise SM occupancy, without changing the scheduler, should reduce LiveCaptions' 30x decode slowdown under greedy allocation—if kernel occupancy, not scheduler fairness, is the main lever.","The same workflow-DAG method could be applied to integrated GPUs and phones, where memory bandwidth is shared with the CPU; the starvation-versus-underutilization trade-off may be even sharper there, but the paper does not provide data for such devices.","Because the paper reports that greedy allocation consumes more instantaneous power but finishes the workflow 45% faster, total energy per task may be lower under greedy allocation despite its unfairness; the paper hints at this but does not compute total energy.","An SLO-aware scheduler that gives LiveCaptions priority only when its small kernels are ready, while letting ImageGen fill idle capacity, would be a concrete next system; ConsumerBench's metrics (SMACT versus SMOCC) provide the monitoring hooks to build and validate it."],"forward_implications":["If ConsumerBench's results hold, any future benchmark for on-device GenAI should include concurrent workloads; exclusive-access scores will overstate both latency and SLO attainment for lightweight apps.","System designers should replace static GPU partitioning with dynamic, SLO-aware partitioning, since fixed slices both waste idle capacity and cause SLO misses after other apps finish.","Inference servers that share one model across applications need per-application configurability; a single static key-value cache setting can force one app's requests to miss their SLO nearly half the time.","Model and kernel developers should treat concurrency as a design constraint: kernels with high register or shared-memory pressure reduce SM occupancy and make applications disproportionately vulnerable to interference.","SLO-aware scheduling could recover the 45% end-to-end workflow time saved by greedy allocation while avoiding LiveCaptions starvation, because the workflow experiments show greedy is faster overall but unfair to the captioning task."],"supporting_citations":[{"why":"MLPerf Inference defines the exclusive-access, single-stream benchmark paradigm that ConsumerBench contrasts with.","marker":"[44]"},{"why":"This supplies the local inference runtime used for the chatbot and deep-research applications, including CPU-GPU co-execution.","marker":"[16]"},{"why":"This provides the static GPU partitioning mechanism used in the concurrent-execution experiments.","marker":"[11]"},{"why":"This supplies the GPU utilization and memory-bandwidth monitoring behind the system-level analysis.","marker":"[10]"},{"why":"This grounds the chatbot SLO targets of 1 second time-to-first-token and 0.25 seconds per output token in human reading speed.","marker":"[29]"},{"why":"This supplies the Llama-3.2-3B model used by both the chatbot and deep-research applications.","marker":"[2]"},{"why":"This specifies the RTX 6000 consumer GPU on which the main experiments run.","marker":"[36]"}],"fun_headline_variants":["Greedy GPU sharing makes lightweight AI apps 30x slower","Static GPU partitioning leaves idle capacity and missed SLOs","On-device GenAI benchmark exposes unfair scheduling","Shared inference server causes 40% SLO misses for chatbot"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results depend on the paper's hand-picked SLO thresholds (1s time-to-first-token and 0.25s per output token for the chatbot, 1s per denoising step for image generation, 2s per audio segment for live captions) and its request arrival pattern (a 2-second audio segment every 2 seconds); if real users' SLOs or workloads differ, the reported SLO-miss rates and the comparison between greedy allocation and partitioning could change.","fun_headline_variants_meta":{"raw":{"variants":["Greedy GPU sharing makes lightweight AI apps 30x slower","Static GPU partitioning leaves idle capacity and missed SLOs","On-device GenAI benchmark exposes unfair scheduling","Shared inference server causes 40% SLO misses for chatbot"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000307,"raw_usage":{"total_tokens":1766,"prompt_tokens":966,"completion_tokens":800,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":582,"completion_tokens_details":{"reasoning_tokens":732}},"tokens_in":582,"tokens_out":800,"duration_ms":8787,"temperature":1.0,"reasoning_tokens":732,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:29:59.375569+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same three-application concurrent workload on the same GPU but change one plausibly realistic parameter—for example, set LiveCaptions' SLO to 4 seconds per segment (the value the paper itself uses on Apple Silicon) while keeping the 2-second arrival rate—and count SLO misses under greedy allocation; if LiveCaptions then meets most SLOs, the starvation conclusion is an artifact of the chosen threshold. Alternatively, replace ImageGen's generic attention kernel with a low-register custom kernel that raises SM occupancy and check whether LiveCaptions' decode slowdown disappears under greedy allocation.","supporting_citations":[{"cited_title":"Scott Gardner, Itay Hubara, Sachin Idgunji, Thomas B","cited_arxiv_id":null,"evidence_quote":"MLPerf Inference defines the exclusive-access, single-stream benchmark paradigm that ConsumerBench contrasts with."},{"cited_title":"llama.cpp: Llm inference in c/c++, 2023","cited_arxiv_id":null,"evidence_quote":"This supplies the local inference runtime used for the chatbot and deep-research applications, including CPU-GPU co-execution."},{"cited_title":"Nvidia multi-process service (mps)","cited_arxiv_id":null,"evidence_quote":"This provides the static GPU partitioning mechanism used in the concurrent-execution experiments."},{"cited_title":"Nvidia data center gpu manager (dcgm)","cited_arxiv_id":null,"evidence_quote":"This supplies the GPU utilization and memory-bandwidth monitoring behind the system-level analysis."},{"cited_title":"Llama 3.2 3b: Multilingual large language model","cited_arxiv_id":null,"evidence_quote":"This supplies the Llama-3.2-3B model used by both the chatbot and deep-research applications."},{"cited_title":"NVIDIA, August 2018","cited_arxiv_id":null,"evidence_quote":"This specifies the RTX 6000 consumer GPU on which the main experiments run."}],"review_version":1}