pith. sign in

arxiv: 2606.26441 · v1 · pith:QAAMOD72new · submitted 2026-06-24 · 💻 cs.IR · cs.DC

GPUSparse: GPU-Accelerated Learned Sparse Retrieval with Parallel Inverted Indices

Ashutosh Sharma This is my paper

Pith reviewed 2026-06-26 00:26 UTC · model grok-4.3

classification 💻 cs.IR cs.DC
keywords learned sparse retrievalGPU accelerationinverted indexSPLADEexact scoringTriton kernelsMS MARCOparallel algorithms
0
0 comments X

The pith

GPU kernels let sparse retrieval match CPU exact scores at 235 times the speed on 8.8 million documents.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper demonstrates that learned sparse retrieval models such as SPLADE can be scored exactly on GPUs. It builds a parallel inverted index whose posting lists are block-aligned for efficient memory access and scores hundreds of queries together with a scatter-add algorithm in fused kernels. On the full MS MARCO passage collection this produces the same MRR@10 of 0.383 and recall@1000 of at least 0.999 as CPU exact methods. Query time falls to 1.27 ms from 298 ms, removing the inference bottleneck while retaining the interpretability of sparse representations.

Core claim

Sparse scoring over an inverted index can be exactly recast as batched scatter-add on a GPU-parallel index with block-aligned, warp-coalesced posting lists; when realized in fused Triton kernels the reformulation returns scores identical to CPU traversal to three decimal places while reaching 62.6 percent of peak HBM bandwidth.

What carries the argument

GPU-parallel inverted index with block-aligned, warp-coalesced posting lists and batched scatter-add scoring in fused Triton kernels.

If this is right

  • Retrieval quality equals CPU exact scoring: MRR@10 reaches 0.383 and recall@1000 is at least 0.999.
  • Per-query latency drops to 1.27 ms from 298 ms on the 8.8 million document collection.
  • Throughput reaches 787 queries per second at batch size 500 while preserving exact scores.
  • The kernels attain 62.6 percent of H100 peak HBM bandwidth, exposing the work-efficiency versus bandwidth-efficiency tradeoff.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Real-time serving of interpretable sparse models can move entirely onto GPU hardware without CPU offload.
  • The scatter-add formulation may extend to other sparse accumulation workloads in machine learning.
  • Hybrid sparse-dense pipelines could keep both components on the same accelerator for lower latency.

Load-bearing premise

The custom GPU kernels implement posting-list traversal and score accumulation without introducing floating-point discrepancies or precision loss relative to sequential CPU execution.

What would settle it

Direct side-by-side execution of GPUSparse and Pyserini on identical SPLADE embeddings and the same MS MARCO query set, where any difference larger than 0.001 in MRR@10 or any change in the top-1000 document sets would disprove exact equivalence.

Figures

Figures reproduced from arXiv: 2606.26441 by Ashutosh Sharma.

Figure 1
Figure 1. Figure 1: Per-query latency scaling with collection size [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
read the original abstract

Learned sparse retrieval models such as SPLADE achieve retrieval quality competitive with dense models while preserving the interpretability and exact-match advantages of sparse representations. However, inference-time scoring still relies on CPU-bound inverted index traversal algorithms (WAND, Block-Max WAND), creating a fundamental bottleneck for real-time serving at scale. We present GPUSparse, a system for GPU-accelerated exact learned sparse retrieval that introduces: (1) a GPU-parallel inverted index with block-aligned, warp-coalesced posting lists; (2) a batched scatter-add scoring algorithm that processes hundreds of queries simultaneously; and (3) fused Triton kernels with an analysis of the tradeoff between work-efficiency and hardware utilization. On MS MARCO passage ranking (8.8M passages) with real SPLADE embeddings, GPUSparse matches CPU exact scoring to three decimals (MRR@10=0.383, equal to Pyserini SPLADE at this precision; Recall@1000>=0.999 vs. dense matmul, the residual from floating-point tie-breaking) while providing a 235x speedup over Pyserini CPU at 8.8M documents (1.27ms vs. 298ms per query). Compared to Seismic (the fastest CPU sparse retrieval system), which trades 25% recall for speed (R@1000=0.738 vs. 0.983 exact), GPUSparse achieves exact scoring at 787 QPS throughput (batch 500) on the full 8.8M collection, with 1.3ms per query. Our document-parallel kernel reaches 62.6% of H100 peak HBM bandwidth, revealing a fundamental work-efficiency vs. bandwidth-efficiency tradeoff in GPU sparse retrieval. The reformulation of sparse scoring as scatter-add over an inverted index is shared with SPARe's iterative mode; our contribution is its fused-kernel realization, which we measure to be 23-270x faster than a faithful SPARe iterative reimplementation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper presents GPUSparse, a GPU-accelerated system for exact learned sparse retrieval. It introduces a GPU-parallel inverted index with block-aligned, warp-coalesced posting lists, a batched scatter-add scoring algorithm for hundreds of queries, and fused Triton kernels. On the MS MARCO passage collection (8.8M passages) using real SPLADE embeddings, it claims to match CPU exact scoring to three decimal places (MRR@10=0.383, Recall@1000>=0.999) while delivering a 235x speedup over Pyserini CPU (1.27ms vs. 298ms per query) and 787 QPS throughput, outperforming Seismic in recall.

Significance. If the numerical equivalence and performance results hold under GPU arithmetic, the work would enable real-time exact sparse retrieval at scale, closing the latency gap with dense models while preserving interpretability. The reported 62.6% of H100 peak HBM bandwidth and explicit work-efficiency vs. bandwidth-efficiency tradeoff analysis provide a concrete foundation for future GPU sparse retrieval designs.

major comments (2)
  1. [Abstract, Section 3] Abstract and kernel description (referenced as Section 3): The central claim of numerical equivalence to CPU exact traversal (MRR@10 identical to three decimals and Recall@1000>=0.999) rests on the batched scatter-add producing results unaffected by re-ordered floating-point additions. The manuscript notes only a 'residual from floating-point tie-breaking' but provides no quantification of how often non-associativity alters document scores, ordering, or top-k sets on the 8.8M collection.
  2. [Abstract] Abstract: The 235x speedup and 787 QPS figures are predicated on exact equivalence being preserved at scale; without an error analysis or verification procedure for GPU floating-point effects in the fused Triton kernels, the load-bearing equivalence assertion cannot be evaluated from the reported numbers alone.
minor comments (1)
  1. [Abstract] The comparison to Seismic notes a 25% recall tradeoff but does not specify whether Seismic's approximate mode was run with identical SPLADE embeddings and the same top-k cutoff as the exact GPUSparse runs.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We appreciate the referee's thorough review and the emphasis on validating the floating-point equivalence claims. We respond to the major comments point by point below.

read point-by-point responses

  1. Referee: [Abstract, Section 3] Abstract and kernel description (referenced as Section 3): The central claim of numerical equivalence to CPU exact traversal (MRR@10 identical to three decimals and Recall@1000>=0.999) rests on the batched scatter-add producing results unaffected by re-ordered floating-point additions. The manuscript notes only a 'residual from floating-point tie-breaking' but provides no quantification of how often non-associativity alters document scores, ordering, or top-k sets on the 8.8M collection.

    Authors: We agree that quantifying the impact of floating-point non-associativity would provide stronger evidence for the equivalence claim. The current manuscript demonstrates equivalence through matching standard retrieval metrics (MRR@10 and Recall@1000) to high precision against the CPU baseline. However, we will revise the manuscript to include a quantification of score differences, such as the percentage of documents with non-zero score deltas and confirmation that top-k sets remain unchanged for the reported metrics. revision: yes

  2. Referee: [Abstract] Abstract: The 235x speedup and 787 QPS figures are predicated on exact equivalence being preserved at scale; without an error analysis or verification procedure for GPU floating-point effects in the fused Triton kernels, the load-bearing equivalence assertion cannot be evaluated from the reported numbers alone.

    Authors: The speedup and QPS figures are based on the observed metric equivalence on the full 8.8M collection. We acknowledge that an explicit error analysis procedure for the Triton kernels would help readers evaluate the claim independently. In the revision, we will add a dedicated subsection describing the verification procedure, including how CPU and GPU results were compared and the observed floating-point residuals. revision: yes

Circularity Check

0 steps flagged

No circularity; empirical implementation validated against external baselines

full rationale

The paper's core claims consist of measured speedups (235x over Pyserini, 787 QPS) and numerical equivalence (MRR@10=0.383 to three decimals, Recall@1000>=0.999) on MS MARCO with SPLADE embeddings. These rest on direct comparisons to Pyserini, Seismic, and a reimplementation of SPARe, plus hardware bandwidth utilization figures. No derivation reduces a result to its own fitted parameters or self-citations; the scatter-add reformulation is presented as shared with prior external work, with the contribution being the fused Triton kernel implementation. The paper is self-contained against external benchmarks with no load-bearing self-referential steps.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review yields no visible free parameters, axioms, or invented entities; evaluation rests on empirical performance claims.

pith-pipeline@v0.9.1-grok · 5897 in / 1053 out tokens · 22558 ms · 2026-06-26T00:26:29.979309+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

25 extracted references · 5 canonical work pages

  1. [1]

    Implementing sparse matrix-vector multipli- cation on throughput-oriented processors

    Nathan Bell and Michael Garland. Implementing sparse matrix-vector multipli- cation on throughput-oriented processors. InSC, pages 18:1–18:11, 2009

    2009

  2. [2]

    Broder, David Carmel, Michael Herscovici, Aya Soffer, and Jason Zien

    Andrei Z. Broder, David Carmel, Michael Herscovici, Aya Soffer, and Jason Zien. Efficient query evaluation using a two-level retrieval process. InCIKM, pages 426–434, 2003

    2003

  3. [3]

    Efficient inverted indexes for approximate retrieval over learned sparse repre- sentations

    Sebastian Bruch, Franco Maria Nardini, Cosimo Rulli, and Rossano Venturini. Efficient inverted indexes for approximate retrieval over learned sparse repre- sentations. InSIGIR, pages 152–162, 2024

    2024

  4. [4]

    SPARe: Supercharged lexical retrievers on GPU with sparse kernels

    Tiago Almeida and Sérgio Matos. SPARe: Supercharged lexical retrievers on GPU with sparse kernels. InECIR, pages 413–421, 2024

    2024

  5. [5]

    Dynamic superblock pruning for fast learned sparse retrieval

    Parker Carlson, Wentai Xie, Shanxiu He, and Tao Yang. Dynamic superblock pruning for fast learned sparse retrieval. InSIGIR, pages 3004–3009, 2025

    2025

  6. [6]

    Faster top-𝑘 document retrieval using block-max indexes

    Shuai Ding and Torsten Suel. Faster top-𝑘 document retrieval using block-max indexes. InSIGIR, pages 993–1002, 2011

    2011

  7. [7]

    SPLADE: Sparse lexical and expansion model for first stage ranking

    Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. SPLADE: Sparse lexical and expansion model for first stage ranking. InSIGIR, pages 2288–2292, 2021

    2021

  8. [8]

    SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval

    Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant. SPLADE v2: Sparse lexical and expansion model for information retrieval.arXiv preprint arXiv:2109.10086, 2021

    work page Pith review arXiv 2021

  9. [9]

    From distillation to hard negative sampling: Making sparse neural IR models more effective

    Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant. From distillation to hard negative sampling: Making sparse neural IR models more effective. InSIGIR, pages 2353–2359, 2022

    2022

  10. [10]

    Billion-scale similarity search with GPUs.IEEE Transactions on Big Data, 7(3):535–547, 2021

    Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs.IEEE Transactions on Big Data, 7(3):535–547, 2021

    2021

  11. [11]

    Dense passage retrieval for open-domain question answering

    Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. InEMNLP, pages 6769–6781, 2020

    2020

  12. [12]

    SPLADE-v3: New baselines for SPLADE

    Carlos Lassance, Hervé Déjean, Thibault Formal, and Stéphane Clinchant. SPLADE-v3: New baselines for SPLADE.arXiv preprint arXiv:2403.06789, 2024

    work page Pith review arXiv 2024

  13. [13]

    Pyserini: A Python toolkit for reproducible information retrieval research with sparse and dense representations

    Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. Pyserini: A Python toolkit for reproducible information retrieval research with sparse and dense representations. InSIGIR, pages 2356– 2362, 2021

    2021

  14. [14]

    A Few Brief Notes on DeepImpact, COIL, and a Conceptual Framework for Information Retrieval Techniques

    Jimmy Lin and Xueguang Ma. A few brief notes on DeepImpact, COIL, and a conceptual framework for information retrieval techniques.arXiv preprint arXiv:2106.14807, 2021

    work page Pith review arXiv 2021

  15. [15]

    MS MARCO: A human generated MAchine Reading COmprehension dataset

    Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. MS MARCO: A human generated MAchine Reading COmprehension dataset. InCoCo@NIPS, 2016

    2016

  16. [16]

    cuVS: GPU-accelerated vector search

    NVIDIA. cuVS: GPU-accelerated vector search. https://github.com/rapidsai/cuvs, 2024. 3https://huggingface.co/naver/splade-cocondenser-ensembledistil Ashutosh Sharma

    2024

  17. [17]

    Sparton: Fast and memory-efficient triton kernel for learned sparse retrieval

    Thong Nguyen, Cosimo Rulli, Franco Maria Nardini, Rossano Venturini, and Andrew Yates. Sparton: Fast and memory-efficient triton kernel for learned sparse retrieval. InSIGIR, 2026

    2026

  18. [18]

    ColBERTv2: Effective and efficient retrieval via lightweight late interaction

    Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. ColBERTv2: Effective and efficient retrieval via lightweight late interaction. InNAACL, pages 3715–3734, 2022

    2022

  19. [19]

    The probabilistic relevance framework: BM25 and beyond.Foundations and Trends in Information Retrieval, 3(4):333–389, 2009

    Stephen Robertson and Hugo Zaragoza. The probabilistic relevance framework: BM25 and beyond.Foundations and Trends in Information Retrieval, 3(4):333–389, 2009

    2009

  20. [20]

    LACONIC: Dense-level effec- tiveness for scalable sparse retrieval via a two-phase training curriculum.arXiv preprint arXiv:2601.01684, 2026

    Zhichao Xu, Shengyao Zhuang, Crystina Zhang, Xueguang Ma, Yijun Tian, Maitrey Mehta, Jimmy Lin, and Vivek Srikumar. LACONIC: Dense-level effec- tiveness for scalable sparse retrieval via a two-phase training curriculum.arXiv preprint arXiv:2601.01684, 2026

    work page arXiv 2026

  21. [21]

    GPU- accelerated decoding of integer lists

    Antonio Mallia, Michal Siedlaczek, Torsten Suel, and Mohamed Zahran. GPU- accelerated decoding of integer lists. InCIKM, pages 2193–2196, 2019

    2019

  22. [22]

    Faster learned sparse retrieval with block-max pruning

    Antonio Mallia, Torsten Suel, and Nicola Tonellotto. Faster learned sparse retrieval with block-max pruning. InSIGIR, pages 2411–2415, 2024

    2024

  23. [23]

    All-in-one graph-based indexing for hybrid search on GPUs.arXiv preprint arXiv:2511.00855, 2025

    Zhonggen Li, Yougen Li, Yifan Zhu, Congcong Ge, Zhaoqiang Chen, and Yunjun Gao. All-in-one graph-based indexing for hybrid search on GPUs.arXiv preprint arXiv:2511.00855, 2025

    work page arXiv 2025

  24. [24]

    A dense representation framework for lexical and semantic matching.ACM Transactions on Information Systems, 41(4):1–29, 2023

    Sheng-Chieh Lin and Jimmy Lin. A dense representation framework for lexical and semantic matching.ACM Transactions on Information Systems, 41(4):1–29, 2023

    2023

  25. [25]

    BEIR: A heterogeneous benchmark for zero-shot evaluation of infor- mation retrieval models

    Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. BEIR: A heterogeneous benchmark for zero-shot evaluation of infor- mation retrieval models. InNeurIPS Datasets and Benchmarks, 2021

    2021