{"id":"ce1a4ade-c2f5-41be-9fb3-f1f27f8bc303","arxiv_id":"2504.14941","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"WindVE offloads peak vector-embedding queries to idle CPUs through a regression-calibrated queue manager, achieving up to 22.3% higher concurrency than a GPU/NPU-only baseline under the same SLOs.","lead":"WindVE is a system that sends overflow embedding queries from AI accelerators to idle server CPUs, letting RAG services handle more concurrent requests without buying hardware. It reports up to 22.3% higher concurrency than a GPU-only scheme, with smaller gains on ARM CPU-NPU servers.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 12's linear latency model is the load-bearing premise; Table III's own regression-vs-stress/fine-tuned gaps (e.g., Kunpeng 2s: 15 predicted vs 8 fine-tuned) indicate nonlinearity at the SLO boundary, leaving the 22.3%/18.6% magnitudes provisional.","rationale":"The reader's conditional verdict is well calibrated. The measured concurrency gains in Tables I and II are plausible and the qualitative ordering across SLOs and hardware pairs matches the model, so this is not a rejection. The most load-bearing point is Eq. 12: every derived benefit ratio and the queue-depth estimator relies on latency increasing linearly in concurrency up to the SLO, and Table III contains direct evidence that this extrapolation is unreliable for the Atlas/Kunpeng pair. That evidence is internal to the paper: 15 predicted CPU queue depth vs 8 fine-tuned, and 195 predicted NPU vs 172 fine-tuned under the 2s SLO. The paper's explanation ('outliers') is not a substitute for residual analysis or a nonlinearity check. The CPU/NPU interference premise is a related but secondary gap. If the proposed refit/extrapolation test passes, the estimator and the 18.6% savings become credible; if it fails, the estimator should be treated as a heuristic and the savings as specific to the measured fine-tuned operating points. Either way the central qualitative claim survives, so the correct verdict remains conditional.","tokens_in":22117,"tokens_out":7292,"duration_ms":67918,"concrete_test":"Take the underlying latency-versus-concurrency data behind Figure 4 and Table III and fit both t_proc = alpha*C + beta and t_proc = alpha*C + beta + gamma*C^2, reporting residuals and a significance test on gamma; then refit Eq. 12 using only points below 50% of the predicted maximum concurrency, extrapolate to the 1s/2s SLO, and compare with the stress-test and fine-tuned values in Table III. If the extrapolation error exceeds 10% on either device, the linear estimator is not a reliable basis for queue depth, and the cost-savings figures should be recomputed from the measured fine-tuned values.","verdict_should_be":"UNCHANGED","load_bearing_attack":"To support the central 'C_NPU + C_CPU' and cost-savings claims, it must be true that end-to-end latency is sufficiently linear in concurrency up to the SLO so that Eq. 12 extrapolation gives correct queue depths. The paper's own Table III contradicts this for the Atlas/Kunpeng pair: under the 2s SLO, linear regression predicts 195 NPU / 15 CPU queue depth, stress test gives 176 / 12, and the final fine-tuned values are 172 / 8. The Kunpeng CPU prediction is nearly double the fine-tuned value, and the NPU prediction is 13% high; the text attributes this to 'outliers,' but this is exactly the regime Eq. 12 must handle. No residuals, confidence intervals, or curvature checks are reported for Figure 4, so the estimator's extrapolation to the SLO boundary is unvalidated. A second unquantified premise is that CPU embedding does not steal memory bandwidth or cores from the NPU path; Section V-A5 only asserts this. Because the headline concurrency (96+22) and 18.6% cost saving rest on both devices meeting SLOs at the fine-tuned depths, the magnitudes are not yet established, even though the qualitative offloading effect is plausible from Table I.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents WindVE, a queue management system for vector embedding inference that offloads overflow queries from NPU/GPU accelerators to idle CPU cores. The authors motivate the work with a cost model that links deployment cost to maximum concurrency, propose a linear-regression-based queue-depth estimator (Eq. 12) for setting NPU and CPU queue lengths, and evaluate the system against FlagEmbedding and PyTorch-based baselines on two hardware platforms (Tesla V100 + Intel Xeon, Atlas 300I DUO + Kunpeng 920). The central claims are a concurrency increase of up to 22.3%, a 1.22x throughput improvement, and up to 18.6% hardware cost savings, along with secondary scalability results on query length and CPU core count.","tokens_in":22320,"tokens_out":4081,"duration_ms":36720,"significance":"If the claimed improvements hold, WindVE addresses a practical cost-performance bottleneck in RAG-based LLM inference, where vector embedding can contribute a significant fraction of latency. The system is tested on two distinct hardware stacks (x86+GPU and ARM+NPU), and the scalability experiments with query length and CPU core count are a useful contribution. The idea of using idle CPU resources for peak offloading with a simple queue-depth estimator is pragmatic and likely deployable. However, the current manuscript does not establish the precision of its headline numbers: the reported percentages are internally inconsistent, no statistical variance is given, and the linearity assumption underlying Eq. (12) is not validated at the SLO boundary where the estimator is used. The qualitative effect is plausible, but the quantitative claims are provisional pending stronger validation.","major_comments":[{"comment":"The linear-regression queue-depth estimator is the load-bearing component of WindVE, but its extrapolation to the SLO boundary is not validated. In Table III, under the 2 s limit the regression predicts a Kunpeng 920 CPU depth of 15, while the stress test gives 12 and the final fine-tuned value is 8; the NPU prediction of 195 is 13% above the fine-tuned 172. The paper attributes the discrepancy to outliers but reports no residuals, confidence intervals, or curvature checks for the fits shown in Figure 4. Because Eq. (12) also underpins the theoretical benefits in Section IV-B3, the estimator must be shown to be reliable in precisely the regime where it is used, or the headline concurrency and cost-savings magnitudes remain provisional.","section":"§V-C, Table III"},{"comment":"The headline improvement is internally inconsistent: for the Tesla V100 + Intel Xeon configuration at the 2 s limit, Table I reports baseline 96 and WindVE 96 + 22, which is a 22.9% increase, not the stated 22.3%. The same inconsistency appears between the 1.20x concurrency mentioned in the introduction and the 1.22x throughput claimed in the abstract. All derived figures (22.3%, 18.6%, 1.22x) need to be recomputed from a single consistent set of measured numbers.","section":"§V-B, Table I"},{"comment":"The experimental section reports no run counts, variance, or statistical significance for any concurrency number, including the 'less than 0.1 ms overhead' and CPU utilization figures. Furthermore, the claim that CPU offloading does not slow the NPU path is only supported by the assertion that 1/4 of the cores are reserved; no measurement of NPU throughput or latency with and without concurrent CPU embedding load is provided. Without this interference quantification, the additive C_NPU + C_CPU model in Section III-B may overestimate the achievable concurrency.","section":"§V-A5 and §V-B"},{"comment":"The 'theoretical effectiveness analysis' is presented as a prediction that the experiments validate, but it is derived from the same fitted parameters (alpha_d, beta_d) that were obtained by regressing the measured latency-concurrency data, so the agreement reported in Section V-B is in part a restatement of the fit. To make the validation non-circular, the paper should specify which claims are genuinely falsifiable (for example, the ordering of the beta values and the direction of change with the SLO) and clarify how the numerical concurrency improvements are independent of the fitting procedure.","section":"§IV-B3 and §V-B"}],"minor_comments":[{"comment":"The symbol T appears in the cost equation without a definition; 'average processing ability' should be defined precisely (for example, queries processed per second per device) so that the formula is self-contained.","section":"§III-A, Eq. (5)"},{"comment":"The testbed description names the CPU as 'Intel Xeon E5-2680', while Tables I and II and the text refer to 'Intel Xeon E5 2690'; these model numbers should be unified.","section":"§V-A1 and Table I"},{"comment":"In the sentence 'The upper boundary Cmax_NPU is determined using similar expressions as followings', the symbol should be Cmax_CPU, since Eqs. (9) and (10) define the CPU boundary; this typo should be corrected.","section":"§III-B"},{"comment":"The contribution bullet states 'we have achieved 1.20 × concurrency', while the abstract and conclusion state 22.3% and 1.22x; these numbers should be aligned to the same measured values.","section":"Introduction"},{"comment":"The fine-tuning procedure after the estimator output is described only as 'fine-tuned'; the paper should state the exact rule used (for example, incremental probing near the estimated value and the stopping criterion) so that the final queue depths can be reproduced.","section":"Table III"},{"comment":"The fitting panels would be more informative if each showed the fitted equation, R², the number of profiling points, and ideally residual plots or error bars, so that the linearity assumption can be assessed visually.","section":"Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript has a plausible qualitative result but the quantitative basis needs to be tightened before acceptance. The inconsistent headline percentages, the absence of any variance reporting, and the unvalidated linearity assumption at the SLO boundary are the main obstacles. I would encourage the editor to request a revised version in which the estimators are validated with residuals/confidence intervals and the cost-savings figures are recomputed from consistent measurements. The novelty over a straightforward two-queue scheduler with linear regression is modest, so the contribution should stand or fall on the rigor of the experimental demonstration."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's core claim is plausible: when servers have idle CPUs, offloading overflow embedding queries to them raises SLO-bounded concurrency. The authors actually built WindVE, tested it on two architectures, and honestly report that the ARM/Kunpeng gains are small. That is more than many industrial papers do. But the headline 22.3% concurrency improvement and 18.6% cost saving are not yet established. The arithmetic alone is off: 22 extra on 96 baseline is 22.9%, not 22.3%. More importantly, the linear latency model in Eq. 12 is the load-bearing piece for the queue-depth estimator, and Table III shows it misses badly at the SLO boundary for Kunpeng: predicted CPU depth 15 versus fine-tuned 8 under the 2s SLO. The paper calls this \"outliers,\" but that is exactly the regime the estimator must handle. No residuals, confidence intervals, or run counts are reported, and no code or data is released, so the magnitudes cannot be independently checked.\n\nWhat is genuinely new: this is the first application of heterogeneous CPU/accelerator offloading specifically to serving-time vector embedding with the objective of maximizing concurrency under SLO. The queue manager is simple but sensible, and the cost analysis in Section III correctly identifies concurrency as the right lever for peak-based deployment. The CPU affinity and NUMA findings for ARM are practical and useful. The experimental pattern across SLOs, query lengths, and CPU core counts is internally consistent, and the paper does not hide the weak ARM results or the memory-bandwidth bottleneck.\n\nThe main soft spot beyond reproducibility is the circularity in Section IV-B3. The \"theoretical effectiveness analysis\" derives predictions from the same fitted α and β parameters that the experiments then \"validate.\" That is a consistency check, not an independent test. The paper would be stronger if it presented the inequalities as intuition and the tables as evidence, rather than claiming validation.\n\nI would not cite this in my own work yet, but I would bring it to a reading group as a realistic example of industrial systems reporting. It deserves serious peer review: the idea is practical, the experiments are real, and the weaknesses are fixable with more data, error bars, and a humbler framing. An editor should send it out, with instructions to the referees to focus on the estimator's validation at the SLO boundary and on reproducibility.","headline":"Plausible industrial result on CPU offloading for embedding serving, but the headline numbers are provisional because of unvalidated linearity and missing reproducibility.","tokens_in":22955,"tokens_out":2110,"would_cite":false,"duration_ms":21537,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"WindVE offloads overflow embedding queries to idle CPUs, raising SLO-bounded concurrency by up to 22.3%.","keywords":["vector embedding","retrieval-augmented generation","CPU-NPU collaboration","heterogeneous computing","concurrency","queue management","linear regression","service-level objective"],"falsifier":"On the paper's primary platform (Tesla V100 plus two Intel Xeon CPUs, BGE model, 2s SLO), measure end-to-end latency at five or more concurrency levels spanning the estimated NPU limit, fit $t_{\\mathrm{proc},d}^{C_d} = \\alpha_d C_d + \\beta_d$, and test whether the measured latency at the SLO boundary deviates from the fitted line by more than 5%. A systematic upward bend would falsify the linear extrapolation that the queue depths depend on.","tokens_in":21819,"feed_emoji":"⚙️","tokens_out":8553,"duration_ms":70685,"temperature":0.7,"pith_summary":"Vector embedding is a large share of retrieval-augmented generation latency, and the paper argues that the deployment cost of an embedding service is driven chiefly by how many queries can be processed concurrently within a latency budget, not by average throughput. WindVE keeps the NPU/GPU as the primary processor and routes only overflow queries to otherwise idle CPUs, so the SLO-bounded concurrency becomes the NPU limit plus the CPU limit. A linear-regression estimator turns a handful of latency measurements into queue depths, replacing slow stress tests. On the BGE model with a Tesla V100 and two Intel Xeon CPUs, the paper reports up to 22.3% higher concurrency than the baseline embedding library, which it translates to 1.22x throughput on the same hardware or 18.6% deployment-cost savings.","feed_headline":"Idle CPUs absorb overflow embeddings, boosting concurrency 22.3%","feed_subtitle":"A queue manager with linear-regression depth estimates offloads peak vector-embedding queries to CPUs without breaking SLOs.","key_machinery":"The load-bearing mechanism is the queue manager and its queue-depth estimator. The manager gives NPU/GPU queues priority, sends overflow to CPU queues, and returns a 'busy' response only when both queues are full. Its critical parameter is device queue depth, and WindVE determines depths by fitting the linear relation $t_{\\mathrm{proc},d}^{C_d} = \\alpha_d C_d + \\beta_d$ with $\\alpha_d,\\beta_d \\ge 0$ from a few profiling points, then reading off the concurrency at the SLO. This converts a slow stress-test search into one two-coefficient fit per device type.","core_discovery":"WindVE's central claim is that the maximum concurrency under a service-level objective can be enlarged from $C_{\\mathrm{NPU}}$ to $C_{\\mathrm{NPU}} + C_{\\mathrm{CPU}}$ by offloading overflow queries to CPU embedding instances while NPUs/GPUs keep priority. The queue depths that realize this are set by fitting the linear relation $t_{\\mathrm{proc},d}^{C_d} = \\alpha_d C_d + \\beta_d$ ($d \\in \\{\\mathrm{NPU}, \\mathrm{CPU}\\}$, $\\alpha_d, \\beta_d \\ge 0$) and extrapolating to the SLO threshold. From that law the paper derives three predictions: the offloading gain is bounded by the slope ratio $\\alpha_{\\mathrm{NPU}}/\\alpha_{\\mathrm{CPU}}$, looser SLOs increase the gain, and a smaller CPU-vs-NPU performance gap is better. In experiments with the BGE model, WindVE achieved 18.2% and 22.3% higher concurrency than FlagEmbedding under 1s and 2s limits on a Tesla V100 with two Intel Xeon CPUs, raising average CPU utilization from about 8% to 80%.","pith_inferences":["If latency bends upward near the SLO boundary, the fitted straight line will overstate queue depths, so a production deployment should re-check the linear fit at the actual operating point before banking on the cost savings.","The same offload pattern should carry over to CPU-GPU servers and to any accelerator with idle host CPUs, since the model only assumes a stable latency-vs-concurrency slope difference between device types.","The reported 18.6% hardware saving is an upper-bound figure from the concurrency model rather than a measured deployment cost; actual savings depend on how often traffic actually overflows onto the CPU path.","Testing at longer sequences and with memory-bandwidth-limited CPU cores would separate graceful degradation from collapse, since the paper identifies host memory bandwidth as the eventual bottleneck."],"forward_implications":["A server's SLO-bounded concurrency becomes the sum of its NPU/GPU and CPU limits, not the NPU/GPU limit alone, whenever the CPU can finish at least one query inside the SLO.","Deploying by peak concurrency saves $C_{\\mathrm{CPU}}/(C_{\\mathrm{NPU}}+C_{\\mathrm{CPU}})$ of hardware cost; deploying by average throughput gains up to $C_{\\mathrm{CPU}}/C_{\\mathrm{NPU}}$ in throughput.","Relaxing the latency budget widens the benefit, because the CPU's fitted intercept exceeds the NPU/GPU's.","The narrower the CPU-NPU/GPU performance gap, the larger the concurrency extension, since the limit is the slope ratio $\\alpha_{\\mathrm{NPU}}/\\alpha_{\\mathrm{CPU}}$.","Longer input queries and fewer CPU cores erode the benefit; at 500-token queries with a 1s limit, the extra CPU concurrency reaches zero."],"supporting_citations":[{"why":"Defines the baseline embedding library that WindVE compares against, with its concurrency numbers appearing in Tables I and II.","marker":"[32]"},{"why":"Supplies the observed linear latency-concurrency relationship that motivates Equation 12.","marker":"[45]"},{"why":"Corroborates the linear relationship between processing latency and concurrency used for queue-depth estimation.","marker":"[46]"},{"why":"Introduces the BGE model family used as the main embedding workload in the experiments.","marker":"[43]"},{"why":"Identifies the Tesla V100 GPU used in the primary testbed.","marker":"[47]"},{"why":"Identifies the Intel Xeon CPUs whose idle capacity carries the offloaded queries.","marker":"[48]"},{"why":"Identifies the Atlas 300I DUO NPU used in the second testbed.","marker":"[50]"},{"why":"Identifies the Kunpeng 920 ARM CPUs used in the second testbed.","marker":"[51]"}],"fun_headline_variants":["CPU offload via linear-regression queues lifts concurrency 22.3%","Idle CPUs catch overflow, WindVE boosts concurrency 22.3%","WindVE offloads peak queries to CPUs, concurrency +22.3%","Linear-regression queue manager offloads CPU, lifts concurrency 22%","CPU-NPU split with smart queues yields 22.3% higher concurrency"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole queue-depth calculation rests on the assumption that processing latency rises in a straight line with concurrency on both NPUs and CPUs, and that the straight line remains valid right at the SLO boundary.","fun_headline_variants_meta":{"raw":{"variants":["CPU offload via linear-regression queues lifts concurrency 22.3%","Idle CPUs catch overflow, WindVE boosts concurrency 22.3%","WindVE offloads peak queries to CPUs, concurrency +22.3%","Linear-regression queue manager offloads CPU, lifts concurrency 22%","CPU-NPU split with smart queues yields 22.3% higher concurrency"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000532,"raw_usage":{"total_tokens":2620,"prompt_tokens":1065,"completion_tokens":1555,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":681,"completion_tokens_details":{"reasoning_tokens":1448}},"tokens_in":681,"tokens_out":1555,"duration_ms":10699,"temperature":1.0,"reasoning_tokens":1448,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:37:35.446603+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On the paper's primary platform (Tesla V100 plus two Intel Xeon CPUs, BGE model, 2s SLO), measure end-to-end latency at five or more concurrency levels spanning the estimated NPU limit, fit $t_{\\mathrm{proc},d}^{C_d} = \\alpha_d C_d + \\beta_d$, and test whether the measured latency at the SLO boundary deviates from the fitted line by more than 5%. A systematic upward bend would falsify the linear extrapolation that the queue depths depend on.","supporting_citations":[{"cited_title":"Flagembedding,","cited_arxiv_id":null,"evidence_quote":"Defines the baseline embedding library that WindVE compares against, with its concurrency numbers appearing in Tables I and II."},{"cited_title":"Nvidia tesla v100,","cited_arxiv_id":null,"evidence_quote":"Identifies the Tesla V100 GPU used in the primary testbed."},{"cited_title":"Intel® xeon® processor e5-2680,","cited_arxiv_id":null,"evidence_quote":"Identifies the Intel Xeon CPUs whose idle capacity carries the offloaded queries."},{"cited_title":"Atlas 300v pro video analysis card),","cited_arxiv_id":null,"evidence_quote":"Identifies the Atlas 300I DUO NPU used in the second testbed."},{"cited_title":"Huawei kunpeng 920 processorsunleash computing p ower,","cited_arxiv_id":null,"evidence_quote":"Identifies the Kunpeng 920 ARM CPUs used in the second testbed."}],"review_version":1}