{"id":"f6eae170-5f5c-405a-b03a-8bd5dd0b5795","arxiv_id":"2607.12839","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"HeteroMosaic uses micro-batching and trace-guided co-optimization to split edge LLM prefill across iGPU and NPU, achieving up to 1.73-2.05x speedups and 45.3% energy reduction on AMD Ryzen AI.","lead":"HeteroMosaic is a scheduling framework that runs edge LLM inference across both a chip's integrated GPU and its NPU at the same time, splitting work into micro-batches to keep both accelerators busy. On AMD Ryzen AI laptops it reports up to ~2x faster prompt processing and up to 45% less energy than single-accelerator baselines.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"End-to-end prefill results in Fig. 8 lack error bars and repeat-count reporting despite the paper's own microbenchmarks showing large turbo/sustained DVFS variation; the headline speedups may not be statistically robust.","rationale":"The reader's weakest_assumption (causal micro-batching exactness) identifies a real but secondary risk: chunked prefill with an incrementally extended KV cache is mathematically exact under causal attention in exact arithmetic, so the concern reduces to floating-point ordering drift. Still, the paper reports no output-equivalence or quality check, and adding one would be cheap. The more load-bearing soft spot is the absence of run-to-run statistics for the end-to-end numbers that constitute the headline, especially given the paper's own demonstration of large power-management-induced variance in Sec. 6.1. The reader's rationale already lists missing error bars as a supporting issue, so there is partial agreement, but the chosen weakest_assumption is not where the central performance claim is most likely to fail. The proposed 10-repetition interleaved protocol would settle whether the headline speedups are robust; if they survive, the remaining concerns (in-sample tuner selection, HeteroInfer re-creation) are addressable and do not invalidate the heterogeneity-first thesis. The paper deserves credit for public code, 1024-run microbenchmarks with error bars, and component ablations. No change to the reader's CONDITIONAL verdict is needed.","tokens_in":32837,"tokens_out":9848,"duration_ms":110039,"concrete_test":"Re-run every point in Fig. 8 and the current traces behind Fig. 9 at least 10 times per configuration, interleaved in randomized order (ideally across multiple days) to sample turbo and sustained power states, and report median and IQR or 95% CI instead of point estimates. Require that the minimum HeteroMosaic-over-baseline speedup across repetitions exceeds 1 for each claimed headline configuration; if the 1.73x/1.78x/2.05x ratios are not separated from baseline noise, the abstract's 'up to' claims should be revised to robust ranges.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing gap is not in the scheduling logic but in the statistical support for the headline numbers. The paper's own microbenchmarks (Sec. 6.1) report 1024-run averages with error bars that 'primarily reflect SoC power-management behavior' and explicitly mention turbo-to-sustained transitions. Yet the end-to-end prefill plots in Fig. 8, which carry the abstract's 1.73x/1.78x/2.05x and 45.3% energy claims, show only point estimates with no repeat count, error bars, or run-to-run distribution. Because a single LLM inference can straddle a turbo-to-sustained transition, the measured latency of any configuration can vary substantially from run to run. Without distributions, 'up to' claims could be best-case draws, particularly for the balanced-system numbers that are the paper's headline. This is compounded by Sec. 5.2.4's trace-guided tuner, which selects configurations by measured latency on the same evaluation configurations it tunes, making in-sample selection a further reason to require robustness reporting before the quantitative claims are accepted.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HeteroMosaic, a heterogeneity-first scheduler for edge LLM inference on AMD Ryzen AI SoCs. It contributes (i) a Gables-style heterogeneous roofline model for identifying when iGPU+NPU execution can beat single-device bounds; (ii) a causal micro-batching formulation that decomposes prefill into dependency-preserving chunks to expose cross-accelerator overlap; (iii) a trace-guided critical-interval co-optimizer that jointly tunes micro-batch schedule, tensor-split allocation, and bubbles under DVFS/NPU queueing effects; and (iv) an AMD-specific runtime with shared-memory buffers, HIP/IRON kernels, and NPU queue coalescing. The evaluation covers three Ryzen AI SKUs, four AWQ W4A16 models, prompt lengths from 1K to 16K, and compares against iGPU/NPU baselines, llama.cpp, a HeteroInfer-style baseline, and production frameworks. The central claims are up to 1.73x/1.78x/2.05x speedups on the balanced SoC and up to 45.3% energy reduction.","tokens_in":33112,"tokens_out":6816,"duration_ms":68551,"significance":"If the quantitative claims hold, this is a substantive contribution: it separates heterogeneity-first scheduling from model restructuring/sparsity/quantization changes, which are confounded in several prior edge-LLM systems, and it demonstrates consistent end-to-end gains across three balance points. Strengths include a code artifact, broad platform/model coverage, 1024-run microbenchmarks with error bars, component ablations, and an honest treatment of decode headroom. The roofline model is not circular: its efficiency factors are calibrated from separate microbenchmarks and prior GEMM studies, then compared with, not fit to, end-to-end results. However, the end-to-end experiments that carry the headline numbers currently lack the statistical support the authors themselves show is necessary, and the comparison against \"prior heterogeneous edge AI solutions\" rests on a reconstructed baseline with an artificial iGPU throttle. The central idea is defensible, but the empirical evidence needs strengthening before the specific performance claims can be accepted.","major_comments":[{"comment":"The paper's own GEMM microbenchmarks (Sec. 6.1) are averaged over 1024 runs and show error bars that \"primarily reflect SoC power-management behavior\" and turbo-to-sustained transitions. Yet the end-to-end prefill plots in Fig. 8, which carry the abstract's headline speedups, show only point estimates with no repeat count, error bars, or run-to-run distribution; the decode table (Table 2) also reports no variance. Because Algorithm 2 (Sec. 5.2.4) tunes configurations by executing them on the same model/platform/prompt ranges used in evaluation, the reported gains may reflect in-sample selection or favorable turbo draws. Please report number of repetitions, variance/confidence intervals for all end-to-end latency and energy numbers, and use a separate validation phase (e.g., held-out prompt lengths or a separate tuning run) to demonstrate that tuned schedules generalize.","section":"Sec. 6.1, 6.2; Fig. 8; Table 2"},{"comment":"The HeteroInfer comparison is a re-creation that forces a 1:10 iGPU:NPU imbalance by \"inserting bubbles on the iGPU path.\" This is not a faithful port of HeteroInfer's scheduler to AMD hardware; it is an iGPU-throttled tensor-partitioning baseline. Consequently, the abstract's claim of up to 2.35x speedup over \"prior heterogeneous edge AI solutions\" conflates heterogeneous-scheduling gains with the effect of artificially slowing the iGPU. The evaluation should either (a) compare against an unmodified tensor-partitioning baseline at the native AMD balance point, or (b) relabel the baseline as \"HeteroInfer-like under an emulated 1:10 imbalance\" and avoid presenting the speedup as a head-to-head improvement over HeteroInfer.","section":"Sec. 6.2, Footnote 4; Abstract"},{"comment":"The 45.3% energy reduction is a headline claim, but the energy methodology is underreported. Figure 9 shows 10 Hz current traces and labels \"45.3% less J\", yet the text does not state how energy was integrated from current, how many repeated runs were performed, what the variance was, or how thermal/turbo state was controlled across runs. Given the paper's own emphasis on turbo-to-sustained DVFS transitions, energy should be reported with the same statistical rigor as the latency microbenchmarks. This is load-bearing because the energy claim appears in the abstract and in the tokens-per-watt framing.","section":"Sec. 6.3; Fig. 9; Abstract"},{"comment":"Algorithm 2 accepts a proposed edit only if it improves measured end-to-end latency in the same evaluation setting (lines 13-16), with a search budget B and about eight hours of offline tuning per model/device configuration. This is an empirical minimization over a large configuration space. The paper should report the number of evaluated schedules/edits (B), show tuning curves or convergence, and evaluate the final configuration on settings not used during tuning. Without such validation, the comparison against fixed heuristic baselines is not fully controlled, and the reported \"up to\" numbers may reflect the result of exhaustive search rather than a generalizable scheduling policy.","section":"Sec. 5.2.4; Algorithm 2"}],"minor_comments":[{"comment":"The relationship between per-accelerator bandwidth B_i and the shared B_peak is not fully specified. It should be clarified whether B_i are per-device bandwidth limits to the unified memory or are already derived from B_peak; otherwise the double-counting concern in the min expression is hard to evaluate. The CPU is present in the equations but omitted from Fig. 4; justify or quantify the omission.","section":"Sec. 4, Eq. (2)-(5)"},{"comment":"With five overlapping curves, the subplots are hard to read, especially in black-and-white. Consider panel-specific markers or separating the llama.cpp curve into a distinct style, and add repeat counts or error bars as discussed in the major comments.","section":"Fig. 8"},{"comment":"The cross-framework comparison mixes numerical formats (integer, BFP16, BF16) and is appropriately labeled as coarse. It would help to state explicitly that Fig. 11 is not used to support the abstract's speedup claims, or to remove it from the narrative of the headline results.","section":"Sec. 6.5, Fig. 11"},{"comment":"The eight-hour trace-guided search cost is mentioned but no details are given for the hardware, the search budget B, or the representative prompt-length ranges. A short reproducibility note with these values would make the offline tuning step more useful to follow-on work.","section":"Sec. 5.2.4"},{"comment":"The NPU efficiency ceiling of 0.4 is taken from references [81-83], some of which share authors with this paper. A sensitivity sweep over this value would make the roofline projections more robust and avoid the appearance of calibration by fiat.","section":"Sec. 4"}],"recommendation":"major_revision","confidential_remarks":"The central scheduling idea is sound and the platform coverage is a real contribution. My main hesitation is that the headline numbers need statistical support and a more defensible prior-work baseline; both are fixable with additional runs and reframing. The NPU-efficiency calibration from co-authored references is also worth a sensitivity analysis. After a solid revision addressing these points, I would be comfortable with acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should read this if you care about edge LLM inference or heterogeneous scheduling. The core idea is that on unified-memory SoCs, you should treat prefill as a scheduling problem: decompose it into causally valid micro-batches, expose cross-accelerator overlap, and then co-optimize the schedule and device allocation around the measured critical path. That combination—causal parallel micro-batching plus trace-guided critical-interval co-optimization—is genuinely new. The paper does well to isolate heterogeneity from sparsity/quantization/outlier tricks, and the multi-platform evaluation (three Ryzen AI SoCs, several models) with ablations and public code is a cut above the usual systems submission.\n\nThe central claim is plausible: heterogeneity helps most on NPU-heavy systems, less on iGPU-heavy ones, and the decode phase is mostly memory-bound so gains are modest. The microbenchmarks are careful and the roofline model is honestly presented as an analytical target, not a predictor.\n\nSoft spots, in proportion. First, the end-to-end plots and table (Fig. 8, Fig. 11, Table 2) have no error bars or repeat counts, despite the paper's own admission that DVFS and thermal state cause substantial run-to-run variation. Given that the headline speedups are \"up to\" numbers, I don't know if those are best-of-N or medians. That's the load-bearing gap. Second, the trace-guided tuner selects configurations on the same measurements it tunes, so the reported speedups are in-sample. The paper says the search is offline and reusable, but the evaluation would be stronger with a held-out set of prompt lengths or a separate validation run. Third, the HeteroInfer comparison is a calibrated re-creation, not the original software—acknowledged, but it means the 2.35x claim rests on the author's own implementation choices. Fourth, the causal micro-batching is assumed to be mathematically exact; the paper doesn't check for numerical drift against monolithic prefill. I suspect the drift is negligible with BF16, but it's a correctness claim worth one experiment.\n\nThese are addressable, not fatal. The roofline efficiency parameters are imported from prior work, some with shared authors, but the model isn't load-bearing for the main result—it's motivation. So the argument holds up.\n\nWorth a serious peer review, and I'd want to see the missing error bars and out-of-sample tuning addressed in revision. I'd cite it for the scheduling primitive and the empirical comparison.","headline":"Solid systems paper with a plausible central claim—heterogeneity-first scheduling beats single-accelerator baselines on edge SoCs—but the headline speedups need statistical backing and the HeteroInfer comparison is a re-creation, not the original.","tokens_in":33672,"tokens_out":1160,"would_cite":true,"duration_ms":15846,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"HeteroMosaic argues that edge LLM inference on unified-memory chips should be scheduled as dependency-preserving micro-batches whose device assignment and timing are co-optimized around the measured critical path, yielding up to about 2x la","keywords":["heterogeneous scheduling","edge LLM inference","micro-batching","causal attention","roofline model","unified memory","critical-path optimization","energy efficiency"],"falsifier":"On the same SoC, take a fixed model and prompt, force every micro-batch onto the strongest single accelerator under the same DVFS and thermal state, and compare time-to-first-token, total energy, and generated-token equality against HeteroMosaic's heterogeneous schedule; if the single-accelerator run wins on latency or energy, or if the token streams diverge, the central claim fails.","tokens_in":32700,"feed_emoji":"⚡","tokens_out":6089,"duration_ms":59006,"temperature":0.7,"pith_summary":"The paper argues that on modern edge chips that pack a CPU, an integrated GPU, and a neural processor behind one memory system, the winning move is not to pick one accelerator and place every operation on it, but to treat inference as a scheduling problem: split the prompt into causally ordered chunks, run chunks that no longer depend on each other on different accelerators at once, and then tune each chunk's device assignment and timing around the measured critical path. It claims this heterogeneity-first approach, applied to unchanged off-the-shelf models, improves end-to-end time-to-first-token by up to roughly 1.7x to 2x over strong single-accelerator runtimes and cuts energy by up to about 45% on balanced systems. If true, edge devices could serve longer prompts and larger models within the same power envelope without changing model weights or quantization. The paper also shows that decode benefits are smaller because token generation is memory-bound, so the main opportunity is in prefill.","feed_headline":"Splitting prompts across GPU and NPU cuts edge LLM latency by up to 2x","feed_subtitle":"HeteroMosaic shows causal micro-batching plus critical-path tuning fits a fixed power envelope, saving up to 45% energy.","key_machinery":"The central machinery is causal parallel micro-batching: splitting prefill into sequential chunks where each chunk writes keys and values into the growing key-value cache and later chunks attend only through that cache, which is exact because attention is causal and which exposes overlap absent from the monolithic graph. A heterogeneous roofline model—a min-over-accelerators, max-over-splits expression with efficiency-scaled compute and bandwidth—bounds when that overlap is worth pursuing. Around the measured critical path, a latency-shaping policy edits each node's split ratio, accelerator assignment, or inserted bubble, accepting edits only when they improve re-measured end-to-end latency;","core_discovery":"The paper's central discovery is that heterogeneous execution becomes a practical winning strategy for edge LLM prefill when it is exposed and scheduled, not merely assigned. The key move is causal parallel micro-batching: a long prompt is split into sequential chunks, each chunk extends the key-value cache, and later chunks attend only through that cache, which makes the decomposition mathematically exact for causal attention. This restructured graph exposes overlap that is invisible in a monolithic prefill, such as the post-attention work of one chunk running concurrently with the attention of the next chunk on a different accelerator. A trace-guided critical-interval co-optimization then","pith_inferences":["Editor's inference: the reported 45% energy saving is measured as total wall-clock energy at a fixed power cap, so it partly reflects finishing sooner; battery-life gains would additionally depend on idle power draw during the saved time, which the paper does not model.","Editor's inference: a decisive testable extension is to compare logits or decoded tokens from monolithic prefill versus micro-batched prefill at 16K context; if numerical drift is negligible, the exactness claim is stronger than typical float-reordering behavior would suggest.","Editor's inference: the same micro-batching-plus-critical-path recipe may transfer to multi-stage transformer workloads such as vision-language or diffusion models, where modality-specific stages create additional independent work; the paper names this direction but leaves it open."],"forward_implications":["If the central claim holds, prefill time-to-first-token is the main win: up to 1.73x over a strong iGPU-only path, 1.78x over an NPU-only path, and 2.05x over a portable open-source runtime on a balanced SoC, all with unchanged models and standard weight-only quantization.","Energy per inference falls by up to 45.3% because HeteroMosaic completes more useful work inside roughly the same peak-power envelope on TDP-constrained chips.","Decode tokens-per-second improves only modestly (up to about 13%) and only when the iGPU does not already saturate memory bandwidth; on stronger iGPUs heterogeneous decode is neutral or slightly harmful.","The scheduling principle is intended to generalize beyond LLMs to other transformer-based pipelines, and the trace-guided co-optimization runs offline once per model/device, so deployment cost is a configuration lookup rather than per-request search.","Because micro-batching also shrinks peak memory, the method extends to very long prompts, such as 70B-parameter models at 16K tokens, where monolithic prefill fails with out-of-memory errors."],"fun_headline_variants":["HeteroMosaic: up to 2x faster edge LLM, 45% less energy","Edge LLM inference up to 2x faster using GPU and NPU together","Trace-guided micro-batching achieves 2x speedup and 45% energy cut for edge LLM","Slicing prompts across GPU and NPU cuts edge LLM latency and up to 45% energy"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that splitting a long prompt into sequential chunks and letting later chunks attend to earlier chunks through the growing key-value cache leaves the model's output numerically unchanged—if chunked attention drifts from monolithic prefill, the reported speedups come with unstated changes in model behavior.","fun_headline_variants_meta":{"raw":{"variants":["HeteroMosaic: up to 2x faster edge LLM, 45% less energy","Edge LLM inference up to 2x faster using GPU and NPU together","Trace-guided micro-batching achieves 2x speedup and 45% energy cut for edge LLM","Slicing prompts across GPU and NPU cuts edge LLM latency and up to 45% energy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001296,"raw_usage":{"total_tokens":5154,"prompt_tokens":799,"completion_tokens":4355,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":543,"completion_tokens_details":{"reasoning_tokens":4253}},"tokens_in":543,"tokens_out":4355,"duration_ms":27441,"temperature":1.0,"reasoning_tokens":4253,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T06:15:59.356653+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On the same SoC, take a fixed model and prompt, force every micro-batch onto the strongest single accelerator under the same DVFS and thermal state, and compare time-to-first-token, total energy, and generated-token equality against HeteroMosaic's heterogeneous schedule; if the single-accelerator run wins on latency or energy, or if the token streams diverge, the central claim fails.","supporting_citations":[],"review_version":2}