REVIEW 9 cited by
Lance: Efficient Random Access in Columnar Storage through Adaptive Structural Encodings
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Lance: Efficient Random Access in Columnar Storage through Adaptive Structural Encodings
read the original abstract
The growing interest in artificial intelligence has created workloads that require both sequential and random access. At the same time, NVMe-backed storage solutions have emerged, providing caching capability for large columnar datasets in cloud storage. Current columnar storage libraries fall short of effectively utilizing an NVMe device's capabilities, especially when it comes to random access. Historically, this has been assumed an implicit weakness in columnar storage formats, but this has not been sufficiently explored. In this paper, we examine the effectiveness of popular columnar formats such as Apache Arrow, Apache Parquet, and Lance in both random access and full scan tasks against NVMe storage. We argue that effective encoding of a column's structure, such as the repetition and validity information, is the key to unlocking the disk's performance. We show that Parquet, when configured correctly, can achieve over 60x better random access performance than default settings. We also show that this high random access performance requires making minor trade-offs in scan performance and RAM utilization. We then describe the Lance structural encoding scheme, which alternates between two different structural encodings based on data width, and achieves better random access performance without making trade-offs in scan performance or RAM utilization.
Forward citations
Cited by 9 Pith papers
-
Homogeneous Stellar Parameters from Heterogeneous Spectra with Deep Learning
A single end-to-end Transformer model unifies stellar labels from heterogeneous spectroscopic surveys into a self-consistent scale without post-hoc recalibration.
-
LayoutBench: Performance Benchmarking of Cloud Storage Layouts for Multimedia Data
LayoutBench benchmarks three cloud storage layouts for multimedia retrieval, finding tar-based packing offers the best latency-cost tradeoff for ImageNet-scale data.
-
SilverTorch: A Unified Model-based System to Democratize Large-Scale Recommendation on GPUs
SilverTorch replaces standalone ANN indexing and filtering with a unified GPU model using a model-based Bloom index and fused Int8 ANN kernel, delivering up to 23.7x throughput and 13.35x cost efficiency gains on indu...
-
OptFSST: Optimized FSST String Compression
OptFSST and OptFSST12 raise FSST/FSST12 compression factors by 7.3%/17.0% on average via DP encoding and better symbol-table heuristics, while preserving random-access decompression.
-
Toward a Scientific Discovery Engine for Weather and Climate Data: A Visual Analytics Workbench for Embedding-Based Exploration
An open-source visual analytics workbench enables provenance-aware, interactive embedding-based retrieval of weather and climate analogs, scalable to tens of millions of embeddings on commodity hardware.
-
OptFSST: Optimized FSST String Compression
OptFSST lifts FSST's average compression factor by 7.3% and FSST12's by 17.0% across 92 string columns using DP encoding, triple counting, and pruning; it also proves the symbol-table selection problem is NP-hard when...
-
stable-worldmodel: A Platform for Reproducible World Modeling Research and Evaluation
The paper presents stable-worldmodel (swm), a platform with high-performance data layer, modern world model baselines, planning solvers, and extended environments for reproducible research and generalization evaluation.
-
Toward a Scientific Discovery Engine for Weather and Climate Data: A Visual Analytics Workbench for Embedding-Based Exploration
A visual analytics workbench enables scientists to explore, query, and verify embedding-based similarity searches on weather and climate data by tracing results back to physical evidence.
-
PystachIO: Efficient Distributed GPU Query Processing with PyTorch over Fast Networks & Fast Storage
PystachIO is a PyTorch-based distributed OLAP engine that delivers up to 3x end-to-end speedups for storage-resident queries by combining fast RDMA networks, NVMe storage, and I/O-computation overlap optimizations.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.