Pith. sign in

REVIEW 4 major objections 6 minor 53 references

WindVE: Collaborative CPU-NPU Vector Embedding

T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read WindVE offloads overflow embedding queries to idle CPUs, raising SLO-bounded concurrency by up to 22.3%.

desk verdict Plausible industrial result on CPU offloading for embedding serving, but the headline numbers are provisional because of unvalidated linearity and missing reproducibility. read the letter →

arxiv 2504.14941 v4 pith:UMEYLCQY submitted 2025-04-21 cs.DC

Jinqi Huang , Xuebing Yu , Yi Xiong , Wenjie Huang , Entong Li , Li Zeng , Xin chen This is my paper · ORCID
classification cs.DC
keywords vectorembeddingretrieval-augmentedgenerationCPU-NPUcollaborationheterogeneouscomputingconcurrencyqueuemanagementlinearregressionservice-levelobjective
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Vector embedding is a large share of retrieval-augmented generation latency, and the paper argues that the deployment cost of an embedding service is driven chiefly by how many queries can be processed concurrently within a latency budget, not by average throughput. WindVE keeps the NPU/GPU as the primary processor and routes only overflow queries to otherwise idle CPUs, so the SLO-bounded concurrency becomes the NPU limit plus the CPU limit. A linear-regression estimator turns a handful of latency measurements into queue depths, replacing slow stress tests. On the BGE model with a Tesla V100 and two Intel Xeon CPUs, the paper reports up to 22.3% higher concurrency than the baseline embedding library, which it translates to 1.22x throughput on the same hardware or 18.6% deployment-cost savings.

What carries the argument

The load-bearing mechanism is the queue manager and its queue-depth estimator. The manager gives NPU/GPU queues priority, sends overflow to CPU queues, and returns a 'busy' response only when both queues are full. Its critical parameter is device queue depth, and WindVE determines depths by fitting the linear relation $t_{\mathrm{proc},d}^{C_d} = \alpha_d C_d + \beta_d$ with $\alpha_d,\beta_d \ge 0$ from a few profiling points, then reading off the concurrency at the SLO. This converts a slow stress-test search into one two-coefficient fit per device type.

What would settle it

On the paper's primary platform (Tesla V100 plus two Intel Xeon CPUs, BGE model, 2s SLO), measure end-to-end latency at five or more concurrency levels spanning the estimated NPU limit, fit $t_{\mathrm{proc},d}^{C_d} = \alpha_d C_d + \beta_d$, and test whether the measured latency at the SLO boundary deviates from the fitted line by more than 5%. A systematic upward bend would falsify the linear extrapolation that the queue depths depend on.

Watch

Extended reading notes

Core claim

WindVE's central claim is that the maximum concurrency under a service-level objective can be enlarged from $C_{\mathrm{NPU}}$ to $C_{\mathrm{NPU}} + C_{\mathrm{CPU}}$ by offloading overflow queries to CPU embedding instances while NPUs/GPUs keep priority. The queue depths that realize this are set by fitting the linear relation $t_{\mathrm{proc},d}^{C_d} = \alpha_d C_d + \beta_d$ ($d \in \{\mathrm{NPU}, \mathrm{CPU}\}$, $\alpha_d, \beta_d \ge 0$) and extrapolating to the SLO threshold. From that law the paper derives three predictions: the offloading gain is bounded by the slope ratio $\alpha_{\mathrm{NPU}}/\alpha_{\mathrm{CPU}}$, looser SLOs increase the gain, and a smaller CPU-vs-NPU performance gap is better. In experiments with the BGE model, WindVE achieved 18.2% and 22.3% higher concurrency than FlagEmbedding under 1s and 2s limits on a Tesla V100 with two Intel Xeon CPUs, raising average CPU utilization from about 8% to 80%.

Load-bearing premise

The whole queue-depth calculation rests on the assumption that processing latency rises in a straight line with concurrency on both NPUs and CPUs, and that the straight line remains valid right at the SLO boundary.

Editorial extensions

If this is right

  • A server's SLO-bounded concurrency becomes the sum of its NPU/GPU and CPU limits, not the NPU/GPU limit alone, whenever the CPU can finish at least one query inside the SLO.
  • Deploying by peak concurrency saves $C_{\mathrm{CPU}}/(C_{\mathrm{NPU}}+C_{\mathrm{CPU}})$ of hardware cost; deploying by average throughput gains up to $C_{\mathrm{CPU}}/C_{\mathrm{NPU}}$ in throughput.
  • Relaxing the latency budget widens the benefit, because the CPU's fitted intercept exceeds the NPU/GPU's.
  • The narrower the CPU-NPU/GPU performance gap, the larger the concurrency extension, since the limit is the slope ratio $\alpha_{\mathrm{NPU}}/\alpha_{\mathrm{CPU}}$.
  • Longer input queries and fewer CPU cores erode the benefit; at 500-token queries with a 1s limit, the extra CPU concurrency reaches zero.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If latency bends upward near the SLO boundary, the fitted straight line will overstate queue depths, so a production deployment should re-check the linear fit at the actual operating point before banking on the cost savings.
  • The same offload pattern should carry over to CPU-GPU servers and to any accelerator with idle host CPUs, since the model only assumes a stable latency-vs-concurrency slope difference between device types.
  • The reported 18.6% hardware saving is an upper-bound figure from the concurrency model rather than a measured deployment cost; actual savings depend on how often traffic actually overflows onto the CPU path.
  • Testing at longer sequences and with memory-bandwidth-limited CPU cores would separate graceful degradation from collapse, since the paper identifies host memory bandwidth as the eventual bottleneck.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents WindVE, a queue management system for vector embedding inference that offloads overflow queries from NPU/GPU accelerators to idle CPU cores. The authors motivate the work with a cost model that links deployment cost to maximum concurrency, propose a linear-regression-based queue-depth estimator (Eq. 12) for setting NPU and CPU queue lengths, and evaluate the system against FlagEmbedding and PyTorch-based baselines on two hardware platforms (Tesla V100 + Intel Xeon, Atlas 300I DUO + Kunpeng 920). The central claims are a concurrency increase of up to 22.3%, a 1.22x throughput improvement, and up to 18.6% hardware cost savings, along with secondary scalability results on query length and CPU core count.

Significance. If the claimed improvements hold, WindVE addresses a practical cost-performance bottleneck in RAG-based LLM inference, where vector embedding can contribute a significant fraction of latency. The system is tested on two distinct hardware stacks (x86+GPU and ARM+NPU), and the scalability experiments with query length and CPU core count are a useful contribution. The idea of using idle CPU resources for peak offloading with a simple queue-depth estimator is pragmatic and likely deployable. However, the current manuscript does not establish the precision of its headline numbers: the reported percentages are internally inconsistent, no statistical variance is given, and the linearity assumption underlying Eq. (12) is not validated at the SLO boundary where the estimator is used. The qualitative effect is plausible, but the quantitative claims are provisional pending stronger validation.

major comments (4)
  1. [§V-C, Table III] The linear-regression queue-depth estimator is the load-bearing component of WindVE, but its extrapolation to the SLO boundary is not validated. In Table III, under the 2 s limit the regression predicts a Kunpeng 920 CPU depth of 15, while the stress test gives 12 and the final fine-tuned value is 8; the NPU prediction of 195 is 13% above the fine-tuned 172. The paper attributes the discrepancy to outliers but reports no residuals, confidence intervals, or curvature checks for the fits shown in Figure 4. Because Eq. (12) also underpins the theoretical benefits in Section IV-B3, the estimator must be shown to be reliable in precisely the regime where it is used, or the headline concurrency and cost-savings magnitudes remain provisional.
  2. [§V-B, Table I] The headline improvement is internally inconsistent: for the Tesla V100 + Intel Xeon configuration at the 2 s limit, Table I reports baseline 96 and WindVE 96 + 22, which is a 22.9% increase, not the stated 22.3%. The same inconsistency appears between the 1.20x concurrency mentioned in the introduction and the 1.22x throughput claimed in the abstract. All derived figures (22.3%, 18.6%, 1.22x) need to be recomputed from a single consistent set of measured numbers.
  3. [§V-A5 and §V-B] The experimental section reports no run counts, variance, or statistical significance for any concurrency number, including the 'less than 0.1 ms overhead' and CPU utilization figures. Furthermore, the claim that CPU offloading does not slow the NPU path is only supported by the assertion that 1/4 of the cores are reserved; no measurement of NPU throughput or latency with and without concurrent CPU embedding load is provided. Without this interference quantification, the additive C_NPU + C_CPU model in Section III-B may overestimate the achievable concurrency.
  4. [§IV-B3 and §V-B] The 'theoretical effectiveness analysis' is presented as a prediction that the experiments validate, but it is derived from the same fitted parameters (alpha_d, beta_d) that were obtained by regressing the measured latency-concurrency data, so the agreement reported in Section V-B is in part a restatement of the fit. To make the validation non-circular, the paper should specify which claims are genuinely falsifiable (for example, the ordering of the beta values and the direction of change with the SLO) and clarify how the numerical concurrency improvements are independent of the fitting procedure.
minor comments (6)
  1. [§III-A, Eq. (5)] The symbol T appears in the cost equation without a definition; 'average processing ability' should be defined precisely (for example, queries processed per second per device) so that the formula is self-contained.
  2. [§V-A1 and Table I] The testbed description names the CPU as 'Intel Xeon E5-2680', while Tables I and II and the text refer to 'Intel Xeon E5 2690'; these model numbers should be unified.
  3. [§III-B] In the sentence 'The upper boundary Cmax_NPU is determined using similar expressions as followings', the symbol should be Cmax_CPU, since Eqs. (9) and (10) define the CPU boundary; this typo should be corrected.
  4. [Introduction] The contribution bullet states 'we have achieved 1.20 × concurrency', while the abstract and conclusion state 22.3% and 1.22x; these numbers should be aligned to the same measured values.
  5. [Table III] The fine-tuning procedure after the estimator output is described only as 'fine-tuned'; the paper should state the exact rule used (for example, incremental probing near the estimated value and the stopping criterion) so that the final queue depths can be reproduced.
  6. [Figure 4] The fitting panels would be more informative if each showed the fitted equation, R², the number of profiling points, and ideally residual plots or error bars, so that the linearity assumption can be assessed visually.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the headline concurrency/cost results are measured against an external baseline, and the regression estimator is a calibration tool rather than a renamed fit.

full rationale

WindVE's central claims rest on direct measurements: Table I compares WindVE's fine-tuned queue depths against FlagEmbedding on the same NPU/GPU hardware, so the 22.3% concurrency improvement and derived 18.6% cost saving are empirical outcomes, not outputs of the fitted model. The queue-depth estimator (Eq. 12) is a regression calibration: alpha and beta are fit from profiling latency-vs-concurrency data, then used to suggest a starting queue depth that is subsequently stress-tested and fine-tuned (Table III). That is inverse prediction from a fitted model, not a quantity already contained in the fit; the paper even reports cases where the regression estimate misses (Kunpeng 2s: 15 vs 8 fine-tuned), showing the estimate is not forced. The linearity premise is an assumption supported by prior work and by the paper's own fitting curves; its possible nonlinearity at the SLO boundary is a correctness/robustness risk, not a circularity. The cost formula (Eq. 6) is a modeling definition, and the 'concurrency is key' conclusion follows arithmetically, but this is motivation rather than a claimed empirical prediction. No load-bearing self-citations or imported uniqueness arguments appear: the linear-relationship citations (SLSC, Mooncake) are external and non-identical to the authors, and no derivation relies on a self-referential theorem.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central result depends on a small set of empirical assumptions: linear latency-concurrency behavior, idle CPU availability without interference, and a peak-concurrency cost model. The queue depths and slope coefficients are fitted to measured data, so the theoretical bounds are descriptive rather than first-principle predictions. No new physical entities are introduced.

free parameters (3)
  • Latency-concurrency regression coefficients α_d, β_d = β_NPU=0.27 (Tesla V100), β_CPU=0.32 (Intel Xeon E5), β_NPU=0.24 (Atlas 300I DUO), β_CPU=0.85 (Kunpeng 920)…
    Fitted to measured latency-concurrency data in Figure 4 and used in Eq. 12 to estimate queue depths and to derive the benefit bounds in Section IV-B3.
  • Queue depths C_max,NPU and C_max,CPU = Fine-tuned values from Table III: Tesla 44/96, Xeon 8/22, Atlas 84/172, Kunpeng 2/8 for 1s/2s SLOs
    These depths determine the reported concurrency values. They are obtained by linear-regression prediction followed by manual fine-tuning during CPU-NPU collaboration, so the headline concurrency is not a pure estimator output.
  • Reserved CPU core fraction (1/4) for NPU processing = 1/4 of total cores
    Chosen by hand in Section V-A5 to prevent CPU offloading from slowing NPU tasks; no sensitivity analysis is provided.
assumptions (3)
  • domain assumption Latency t_proc,d is linear in concurrency C_d for both CPU and NPU, with nonnegative α_d, β_d (Eq. 12).
    Invoked in Section IV-B2 to estimate queue depths and in Section IV-B3 to derive theoretical benefit predictions. The paper provides fitted lines but no residual analysis or nonlinearity check.
  • domain assumption CPUs in NPU/GPU servers are idle enough (under 10% utilization) to run vector-embedding instances without degrading NPU/GPU processing.
    Motivates the offloading scheme in Section III-A and the claim of no extra hardware cost. The paper reports 8-9% CPU utilization before offloading but does not measure NPU latency degradation from concurrent CPU inference.
  • domain assumption Deployment cost for SLO-sensitive inference is determined by peak concurrency via Cost = N_peak/C * D * P (Eq. 6).
    Used in Sections III-A and V-B to translate concurrency gains into 18.6% hardware cost savings. This is the paper's own cost model, not an independently validated industry metric.

how reviews work

0 comments
Cite this review

Pith. "Pith review of WindVE: Collaborative CPU-NPU Vector Embedding." pith.science (2026). https://pith.science/paper/UMEYLCQY

@misc{pith2026250414941,
  author       = {Pith},
  title        = {Pith review of: WindVE: Collaborative CPU-NPU Vector Embedding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UMEYLCQY}},
  note         = {Machine review of arXiv:2504.14941}
}
read the original abstract

Retrieval-Augmented Generation is a technology that enhances large language models by integrating information retrieval. In the industry, inference services based on LLMs are highly sensitive to cost-performance ratio, prompting the need for improving hardware resource utilization in the inference service. Specifically, vector embedding and retrieval processes take up to 20% of the total latency. Therefore, optimizing the utilization of computational resources in vector embeddings is crucial for enhancing the cost-performance ratio of inference processes, which in turn boosts their product competitiveness.In this paper, we analyze the deployment costs of vector embedding technology in inference services, propose a theoretical formula, and determine through the mathematical expression that increasing the capacity to process concurrent queries is the key to reducing the deployment costs of vector embeddings. Therefore, in this paper, we focus on improving the product's capability to process concurrent queries. To optimize concurrency without sacrificing performance, we have designed a queue manager that adeptly offloads CPU peak queries. This manager utilizes a linear regression model to ascertain the optimal queue depths, a critical parameter that significantly influences the efficacy of the system. We further develop a system named WindVE that uses a CPU-NPU heterogeneous architecture to offload peak concurrent queries, which leverages the performance differences between the two processors to effectively manage traffic surges. Through experiments, we compare WindVE to the state-of-the-art vector embedding framework FlagEmbedding, and achieve a concurrency level up to 22.3% higher than the scheme without offloading.

Figures

Figures reproduced from arXiv: 2504.14941 by the authors.

Figure 1
Figure 1. RAG workflow. NPU collaboration for vector embedding, coupled with a linear regression based estimator for ascertaining its critical parameters to maximize concurrency. • We developed WindVE, a flexible and scalable system framework for high-throughput offloading vector embed￾ding with CPU-NPU collaboration. • We demonstrated WindVE incorporating CPU-NPU het￾erogeneous computing for peak query offloading in vector e… view at source ↗
Figure 2
Figure 2. An illustration of query number changes in a day. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. WindVE System Compared to existing work. (A) shows a t [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Fitting curves and stress tests on (A) Tesla V100 GPU, [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Scalability experiment with varied input query leng [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Scalability experiment with varied CPU core numbers [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

53 extracted references · 32 canonical work pages

  1. [1]

    O3 knowledge community,

    L. Huawei Technologies, Co., “O3 knowledge community,” https://sg.o3community.huawei.com/sg/en/index/ 1783684860314767362, 2024

    work page 2024

  2. [2]

    Chatgpt,

    OpenAI, “Chatgpt,” https://openai.com/blog/chatgpt , 2024

    work page 2024

  3. [3]

    Improving real-time knowl edge retrieval in large language models with a dns-style hierarchical quer y rag,

    A. Reynolds and F. Corrigan, “Improving real-time knowl edge retrieval in large language models with a dns-style hierarchical quer y rag,” 06 2024

    work page 2024

  4. [4]

    Ret rieval augmentation reduces hallucination in conversation,

    K. Shuster, S. Poff, M. Chen, D. Kiela, and J. Weston, “Ret rieval augmentation reduces hallucination in conversation,” in Findings of the Association for Computational Linguistics: EMNLP 2021 , Nov. 2021, pp. 3784–3803. [Online]. Available: https://aclanthology.org/2021.findings-emnlp.320

    work page 2021

  5. [5]

    Bridging the preference gap between retrievers and llms,

    Z. Ke, W. Kong, C. Li, M. Zhang, Q. Mei, and M. Bendersky, “Bridging the preference gap between retrievers and llms,” 2024. [Online]. Available: https://arxiv.org/abs/2401.06954

    arXiv 2024

  6. [6]

    Search- in-the- chain: Towards the accurate, credible and traceable conten t generation for complex knowledge-intensive tasks,

    S. Xu, L. Pang, H. Shen, X. Cheng, and T.-s. Chua, “Search- in-the- chain: Towards the accurate, credible and traceable conten t generation for complex knowledge-intensive tasks,” 04 2023

    work page 2023

  7. [7]

    Paperqa: Retrieval-augmented generative agent for scientific research,

    J. L´ ala, O. O’Donoghue, A. Shtedritski, S. Cox, S. G. Rod riques, and A. D. White, “Paperqa: Retrieval-augmented generative agent for scientific research,” 2023. [Online]. Available: https://arxiv.org/abs/2312.07559

    arXiv 2023

  8. [8]

    Iag: Induction-augmented generation framework fo r answering reasoning questions,

    Z. Zhang, X. Zhang, Y . Ren, S. Shi, M. Han, Y . Wu, R. Lai, and Z. Cao, “Iag: Induction-augmented generation framework fo r answering reasoning questions,” 2023. [Online]. Available: https://arxiv.org/abs/2311.18397

    arXiv 2023

Show all 53 references
  1. [9]

    Nomiracl: Knowing when you don’t know for robust multilingual retrieval-augmented generation,

    N. Thakur, L. Bonifacio, X. Zhang, O. Ogundepo, E. Kamall oo, D. Alfonso-Hermelo, X. Li, Q. Liu, B. Chen, M. Rezagholizade h, and J. Lin, “Nomiracl: Knowing when you don’t know for robust multilingual retrieval-augmented generation,” 2024. [On line]. Available: https://arxiv.o...

    2024 arXiv

  2. [10]

    Context e mbeddings for efficient answer generation in rag,

    D. Rau, S. Wang, H. D´ ejean, and S. Clinchant, “Context e mbeddings for efficient answer generation in rag,” 2024. [Online]. Ava ilable: https://arxiv.org/abs/2407.09252

    2024 arXiv

  3. [11]

    Chat2data: An interactive d ata analysis system with rag, vector databases and llms,

    X. Zhao, X. Zhou, and G. Li, “Chat2data: An interactive d ata analysis system with rag, vector databases and llms,” Proc. VLDB Endow , 2024

    2024

  4. [12]

    V ectorsearch: Enhancing document retrieval with semantic embeddings and optimized search,

    S. S. Monir, I. Lau, S. Y ang, and D. Zhao, “V ectorsearch: Enhancing document retrieval with semantic embeddings and optimized search,”

  5. [13]

    A deep learning framewor k for enhancer prediction using word embedding and sequence gene ration,

    Q. Geng, R. Y ang, and L. Zhang, “A deep learning framewor k for enhancer prediction using word embedding and sequence gene ration,” Biophysical Chemistry , vol. 286, p. 106822, 2022. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0301462222000643

    2022

  6. [14]

    Pricing,

    O. AI, “Pricing,” 2024. [Online]. Available: https://openai.com/api/pricing/

    2024

  7. [15]

    Retrieval-augmented generation for l arge language models: A survey,

    Y . Gao, Y . Xiong, X. Gao, K. Jia, J. Pan, Y . Bi, Y . Dai, J. Su n, M. Wang, and H. Wang, “Retrieval-augmented generation for l arge language models: A survey,” 2024. [Online]. Available: https://arxiv.org/abs/2312.10997

    2024 arXiv

  8. [16]

    Fast, acc urate, and lightweight memory-enhanced embedding learning frame work for image-text retrieval,

    Z. Li, L. Zhang, K. Zhang, Y . Zhang, and Z. Mao, “Fast, acc urate, and lightweight memory-enhanced embedding learning frame work for image-text retrieval,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 34, no. 7, pp. 6542–6558, 2024

    2024

  9. [17]

    Fa st and accurate network embeddings via very sparse random project ion,

    H. Chen, S. F. Sultan, Y . Tian, M. Chen, and S. Skiena, “Fa st and accurate network embeddings via very sparse random project ion,” in Proceedings of the 28th ACM International Conference on Inf ormation and Knowledge Management , ser. CIKM ’19. New Y ork, NY , USA: Associati...

    2019

  10. [18]

    Fast and discr iminative semantic embedding,

    R. Koopman, S. Wang, and G. Englebienne, “Fast and discr iminative semantic embedding,” in Proceedings of the 13th International Conference on Computational Semantics - Long Papers , S. Dobnik, S. Chatzikyriakidis, and V . Demberg, Eds. Gothenburg, Swed en: Association for Com...

    2019

  11. [19]

    Reads2vec: Efficient embedding of raw high-throughput seq uencing reads data,

    P . Chourasia, S. Ali, S. Ciccolella, G. D. V edova, and M. Patterson, “Reads2vec: Efficient embedding of raw high-throughput seq uencing reads data,” Journal of Computational Biology , vol. 30, no. 4, pp. 469–491, 2023, pMID: 36730750. [Online]. Available: https://doi.org/10.1...

    2023

  12. [20]

    AdaEmbed: Adaptive embeddin g for Large-Scale recommendation models,

    F. Lai, W. Zhang, R. Liu, W. Tsai, X. Wei, Y . Hu, S. Devkota , J. Huang, J. Park, X. Liu, Z. Chen, E. Wen, P . Rivera, J. Y ou, C. cheng Jason Chen, and M. Chowdhury, “AdaEmbed: Adaptive embeddin g for Large-Scale recommendation models,” in 17th USENIX Symposium on Operating ...

    2023

  13. [21]

    Freddy: Fast word embeddings in database systems,

    M. G¨ unther, “Freddy: Fast word embeddings in database systems,” in Proceedings of the 2018 International Conference on Manage ment of Data , ser. SIGMOD ’18. New Y ork, NY , USA: Association for Computing Machinery, 2018, p. 1817–1819. [Online]. Availa ble: https://doi.org/1...

    2018

  14. [22]

    A fast nearest neighbor search algorithm by nonlinear embedding,

    Y . Hwang, B. Han, and H.-K. Ahn, “A fast nearest neighbor search algorithm by nonlinear embedding,” in 2012 IEEE Conference on Computer Vision and Pattern Recognition , 2012, pp. 3053–3060

    2012

  15. [23]

    Feature hashing for large scale multitask learning,

    K. Weinberger, A. Dasgupta, J. Langford, A. Smola, and J . Attenberg, “Feature hashing for large scale multitask learning,” in Proceedings of the 26th Annual International Conference on Machine Lear ning, ser. ICML ’09. New Y ork, NY , USA: Association for Computing Machinery,...

    2009

  16. [24]

    Binary code based hash embedding for web-scale applicatio ns,

    B. Y an, P . Wang, J. Liu, W. Lin, K.-C. Lee, J. Xu, and B. Zhe ng, “Binary code based hash embedding for web-scale applicatio ns,” in Proceedings of the 30th ACM International Conference on Inf ormation & Knowledge Management , ser. CIKM ’21. New Y ork, NY , USA: Association ...

    2021

  17. [25]

    Compo sitional embeddings using complementary partitions for memory-effi cient recommendation systems,

    H.-J. M. Shi, D. Mudigere, M. Naumov, and J. Y ang, “Compo sitional embeddings using complementary partitions for memory-effi cient recommendation systems,” in Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Min ing, ser. KDD ’20. New Y...

    2020

  18. [26]

    Cafe: Towards compact, adaptive, and fast embedding for la rge-scale recommendation models,

    H. Zhang, Z. Liu, B. Chen, Y . Zhao, T. Zhao, T. Y ang, and B. Cui, “Cafe: Towards compact, adaptive, and fast embedding for la rge-scale recommendation models,” vol. 2, no. 1, Mar. 2024. [Online]. Available: https://doi.org/10.1145/3639306

    2024 doi

  19. [27]

    Colossal-ai: A unified deep learning system for larg e-scale parallel training,

    S. Li, H. Liu, Z. Bian, J. Fang, H. Huang, Y . Liu, B. Wang, a nd Y . Y ou, “Colossal-ai: A unified deep learning system for larg e-scale parallel training,” 2023. [Online]. Available: https://arxiv.org/abs/2110.14883

    2023 arXiv

  20. [28]

    Efficient memory managem ent for large language model serving with pagedattention,

    W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Y u, J. E . Gonzalez, H. Zhang, and I. Stoica, “Efficient memory managem ent for large language model serving with pagedattention,” in Proceedings of 10 the ACM SIGOPS 29th Symposium on Operating Systems Principl es, 2023

    2023

  21. [29]

    Nvidia triton inference server,

    Triton, “Nvidia triton inference server,” https://developer.nvidia.com/triton-inference-server, 2024

    2024

  22. [30]

    Efficient memory managem ent for large language model serving with pagedattention,

    W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Y u, J. E . Gonzalez, H. Zhang, and I. Stoica, “Efficient memory managem ent for large language model serving with pagedattention,” 202 3. [Online]. Available: https://arxiv.org/abs/2309.06180

    arXiv

  23. [31]

    Fast inference from transformers via speculative decoding,

    Y . Leviathan, M. Kalman, and Y . Matias, “Fast inference from transformers via speculative decoding,” 2023. [Online]. A vailable: https://arxiv.org/abs/2211.17192

    2023 arXiv

  24. [32]

    Flagembedding,

    B. A. of Artificial Intelligence, “Flagembedding,” https://github.com/FlagOpen/FlagEmbedding, 2024

    2024

  25. [33]

    A rev iew of parallel heterogeneous computing algorithms in power sy stems,

    D. Rodriguez, D. Gomez, D. Alvarez, and S. Rivera, “A rev iew of parallel heterogeneous computing algorithms in power sy stems,” Algorithms, vol. 14, no. 10, 2021. [Online]. Available: https://www.mdpi.com/1999-4893/14/10/275

    2021

  26. [34]

    Heterogeneous computing (cpu–gpu) for pollution dispersion in an urban environment ,

    G. Fernandez, M. Mendina, and G. Usera, “Heterogeneous computing (cpu–gpu) for pollution dispersion in an urban environment ,” Computation, vol. 8, no. 1, 2020. [Online]. Available: https://www.mdpi.com/2079-3197/8/1/3

    2020

  27. [35]

    Query processing o n heterogeneous cpu/gpu systems,

    V . Rosenfeld, S. Breß, and V . Markl, “Query processing o n heterogeneous cpu/gpu systems,” ACM Comput. Surv. , vol. 55, no. 1, Jan. 2022. [Online]. Available: https://doi.org/10.1145/3485126

    2022 doi

  28. [36]

    Efficient processing of image proce ssing applications on cpu/gpu,

    N. Naz, A. Haseeb Malik, A. B. Khurshid, F. Aziz, B. Alouf fi, M. I. Uddin, and A. AlGhamdi, “Efficient processing of image proce ssing applications on cpu/gpu,” Mathematical Problems in Engineering , vol. 2020, no. 1, p. 4839876, 2020

    2020

  29. [37]

    Distribution of operation s in hetero- geneous computing systems for processing speech signals,

    M. Rakhimov and M. Ochilov, “Distribution of operation s in hetero- geneous computing systems for processing speech signals,” in 2021 IEEE 15th International Conference on Application of Infor mation and Communication Technologies (AICT) . IEEE, 2021, pp. 1–4

    2021

  30. [38]

    Energy-efficie nt resource management for federated edge learning with cpu-gpu hetero geneous computing,

    Q. Zeng, Y . Du, K. Huang, and K. K. Leung, “Energy-efficie nt resource management for federated edge learning with cpu-gpu hetero geneous computing,” IEEE Transactions on Wireless Communications , vol. 20, no. 12, pp. 7947–7962, 2021

    2021

  31. [39]

    Efficient pattern matching on cpu-gpu heterogeneous systems,

    V . Sanz, A. Pousa, M. Naiouf, and A. De Giusti, “Efficient pattern matching on cpu-gpu heterogeneous systems,” in Algorithms and Archi- tectures for Parallel Processing: 19th International Conf erence, ICA3PP 2019, Melbourne, VIC, Australia, December 9–11, 2019, Proc eedings, ...

    2019

  32. [40]

    Cooperative hetero geneous computing for parallel processing on cpu/gpu hybrids,

    C. Lee, W. W. Ro, and J.-L. Gaudiot, “Cooperative hetero geneous computing for parallel processing on cpu/gpu hybrids,” in 2012 16th W orkshop on Interaction between Compilers and Computer Arc hitec- tures (INTERACT), 2012, pp. 33–40

    2012

  33. [41]

    Graphvite: A high-perf ormance cpu-gpu hybrid system for node embedding,

    Z. Zhu, S. Xu, J. Tang, and M. Qu, “Graphvite: A high-perf ormance cpu-gpu hybrid system for node embedding,” in The W orld Wide W eb Conference, ser. WWW ’19. New Y ork, NY , USA: Association for Computing Machinery, 2019, p. 2494–2504. [Online]. Availa ble: https://doi.org/1...

    2019

  34. [42]

    Hetegen: Heterogeneous parallel inference for large lang uage models on resource-constrained devices,

    X. Zhao, B. Jia, H. Zhou, Z. Liu, S. Cheng, and Y . Y ou, “Hetegen: Heterogeneous parallel inference for large lang uage models on resource-constrained devices,” ArXiv, vol. abs/2403.01164, 2024. [Online]. Available: https://api.semanticscholar.org/CorpusID:268230457

    2024 arXiv

  35. [43]

    C-pack: P ackaged resources to advance general chinese embedding,

    S. Xiao, Z. Liu, P . Zhang, and N. Muennighoff, “C-pack: P ackaged resources to advance general chinese embedding,” 2023

    2023

  36. [44]

    Jina embeddings 2: 8192-token general-purpos e text embeddings for long documents,

    M. G¨ unther, J. Ong, I. Mohr, A. Abdessalem, T. Abel, M. K . Akram, S. Guzman, G. Mastrapas, S. Sturua, B. Wang, M. Werk, N. Wang, and H. Xiao, “Jina embeddings 2: 8192-token general-purpos e text embeddings for long documents,” CoRR, vol. abs/2310.19923, 2023

    arXiv

  37. [45]

    Sli ce-level scheduling for high throughput and load balanced llm servin g,

    K. Cheng, W. Hu, Z. Wang, H. Peng, J. Li, and S. Zhang, “Sli ce-level scheduling for high throughput and load balanced llm servin g,” 2024. [Online]. Available: \url{https://arxiv.org/abs/2406.13511}

    2024 arXiv

  38. [46]

    Mooncake: A kvcache-centric disaggregated architecture for llm serving,

    R. Qin, Z. Li, W. He, M. Zhang, Y . Wu, W. Zheng, and X. Xu, “Mooncake: A kvcache-centric disaggregated architecture for llm serving,” 2024. [Online]. Available: \url{https://arxiv.org/abs/2407.00079}

    2024 arXiv

  39. [47]

    Nvidia tesla v100,

    Nvidia, “Nvidia tesla v100,” 2024. [Online]. Availabl e: https://www.nvidia.com/en-gb/data-center/tesla-v100/

    2024

  40. [48]

    Intel® xeon® processor e5-2680,

    Intel, “Intel® xeon® processor e5-2680,” 2024. [Onlin e]. Available: \url{https://www.intel.com/content/www/us/en/products/sku/64583/inte l-xeon-processor-e52680-20m-cache-2-70-ghz-8-00-gts -intel-qpi/speci fications.html }

    2024

  41. [49]

    Atlas 800 inference server (model: 3000),

    Huawei, “Atlas 800 inference server (model: 3000),” 20 24. [Online]. Available: https://www.hiascend.com/en/hardware/ai-server?tag=300

  42. [50]

    Atlas 300v pro video analysis card),

    ——, “Atlas 300v pro video analysis card),” 2024. [Onlin e]. Available: https://www.hiascend.com/en/hardware/accelerator-card?tag=300I-duo

    2024

  43. [51]

    Huawei kunpeng 920 processorsunleash computing p ower,

    ——, “Huawei kunpeng 920 processorsunleash computing p ower,”

  44. [53]

    Available: https://www.hikunpeng.com/en/compute/kunpeng920 11

    [Online]. Available: https://www.hikunpeng.com/en/compute/kunpeng920 11

  45. [2024]

    Available: https://arxiv.org/abs/2409.17383

    [Online]. Available: https://arxiv.org/abs/2409.17383

    arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.