Pith. sign in

REVIEW 2 major objections 5 minor 2 cited by

Lance: Efficient Random Access in Columnar Storage through Adaptive Structural Encodings

T0 review · 2 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Lance's adaptive structural encoding gives fast random access without scan or RAM trade-offs.

desk verdict Genuinely new adaptive structural encoding with careful NVMe benchmarking, but the 'no RAM utilization trade-off' claim only holds for large types and the warm-only evaluation leaves it under-supported. read the letter →

arxiv 2504.15247 v1 pith:5QBHLVYQ submitted 2025-04-21 cs.DB

Weston Pace , Chang She , Lei Xu , Will Jones , Albert Lockett , Jun Wang , Raunak Shah This is my paper · ORCID
classification cs.DB
keywords columnarstoragerandomaccessstructuralencodingNVMeParquetApacheArrowLancesearchcache
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the layout of a column's structural metadata—the repetition levels, definition levels, offsets, and validity information—is what decides how many I/O operations a random lookup costs, and that this has been underappreciated relative to compression and encoding choices. It shows that a standard format like Parquet can be tuned for random access with a page offset index, but only by paying for it in scan performance or in the RAM needed to hold per-page metadata. It then presents Lance's structural encoding scheme, which alternates between two layouts: a full-zip encoding that transposes all buffers of a column into one value-by-value record for large data types, and a miniblock encoding that keeps vectorized columnar chunks for small data types. Benchmarks on NVMe storage show Lance matching or beating Parquet and Arrow-style encodings on random access while keeping full-scan throughput and a small search cache. If right, the finding matters because columnar formats are widely assumed to be inherently bad at search-oriented workloads, and it identifies a concrete, format-level reason why that assumption needs revision.

What carries the argument

The central objects are the two structural encodings and the rule that chooses between them. In full-zip encoding, the repetition and definition levels of each value are bit-packed into a one-to-four-byte control word and stored immediately before the value's data, so a column becomes a sequence of self-describing records preceded by a repetition index that maps row indices to byte offsets; this gives one or two IOPS per lookup regardless of nesting. In miniblock encoding, arrays are divided into chunks sized to one or two disk sectors (4–8 KiB), each with a small header and its own repetition/definition buffers and data buffers, and the repetition index is reduced to per-chunk counters; this keeps vectorized scans fast for small types. The adaptive rule is a threshold: data types with at least 128 bytes per value use full-zip; smaller types use miniblock. The load-bearing mechanism is the repetition index, which converts arbitrary nesting depth and variable widths into a fixed-cost lookup, and the per-chunk metadata that keeps the search cache small enough to be held in RAM.

What would settle it

Measure point-lookup throughput on a billion-row Lance file after clearing the page cache and with available RAM set below the size of the search cache, so metadata is re-read from disk on every lookup; if the row-fetch rate falls to Parquet's cold-cache level, the 'no RAM trade-off' claim is falsified.

Watch

Extended reading notes

Core claim

The paper's central claim is that the encoding of a column's structure—how repetition levels, definition levels, offsets, and validity are laid out relative to the data—determines random-access cost in columnar storage, and that this cost can be made nearly independent of nesting depth and data width by choosing the right layout. Parquet, with a page offset index and small pages, achieves strong point-lookup performance but needs about 20 bytes of in-memory search cache per page, which becomes prohibitive for wide types such as embeddings where a page may contain a single value. Arrow-style layouts avoid the cache but require one I/O per buffer, so a nested value like List<String> can cost five IOPS. Lance's scheme alternates: for values of at least 128 bytes it uses full-zip encoding, which interleaves repetition/definition control words and all data buffers into one fixed-layout stream plus a repetition index, giving one or two IOPS for any lookup; for smaller values it uses a miniblock encoding with 4–8 KiB chunks and a 2-byte on-disk chunk header, trading a small amount of read amplification for vectorized encoding and opaque compression. The benchmarks report that this combined scheme matches or exceeds Parquet's random-access rate and full-scan throughput on NVMe without the RAM cost, and avoids the need to tune row-group size.

Load-bearing premise

The load-bearing assumption is that the metadata used to locate values—page offsets in Parquet, miniblock and repetition metadata in Lance—fits in RAM and is loaded once, so its own I/O cost is ignored; cold caches or one-off searches would add IOPS that the paper does not measure.

Editorial extensions

If this is right

  • If Lance's scheme is correct, columnar file formats can serve search workloads such as vector retrieval and RAG directly from NVMe-backed object storage without a separate in-memory copy or a secondary row-oriented format.
  • Parquet's default configuration leaves a large random-access margin on the table; the paper measures over 60x improvement with correct configuration, implying that existing Parquet users can gain most of the same benefit by enabling page offset indexes and choosing small pages.
  • The 'no RAM trade-off' claim implies that search caches can be kept at roughly 24–41 bytes per chunk plus a 2-byte on-disk header, which changes the sizing rules for NVMe cache tiers in data-lake engines.
  • Because repetition and definition levels are Dremel-style, the scheme extends to arbitrary nesting and lists without additional IOPS per level, so applications with deeply nested JSON-like columns can expect point lookup cost to stop growing with nesting depth.
  • Struct packing offers a tunable knob: packing a struct into a single column trades single-field scan speed for whole-record random access, giving engines a gradient between columnar and row-based layout.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same structural-encoding lens should apply to formats beyond Lance—any format that separates structural metadata from data buffers can be analyzed by the number of dependent IOPS per lookup; the paper's methodology suggests a cost model of the form 'IOPS × (1 + nesting) + read amplification + search-cache bytes per value'.
  • Editorial inference: because the repetition index is itself a bit-packed array that can be range-read, the scheme suggests a testable extension where range scans (e.g., fetching rows 1,000 to 2,000 of nested data) cost two IOPS regardless of length, which could make secondary indexes cheaper by avoiding per-row lookups.
  • Editorial inference: the adaptive threshold of 128 bytes per value is an empirical constant measured on one NVMe drive; on storage with different IOPS/bandwidth ratios (e.g., cloud object storage with low IOPS), the crossover point between full-zip and miniblock likely moves, so a cost-aware writer that chooses the encoding per column based on device characteristics is a natural follow-up.
  • Editorial inference: the paper's warm-search assumption means files opened once and queried sparingly would pay metadata-loading IOPS not counted in the benchmarks; for such workloads, a format that inlines minimal metadata or uses a compact on-disk header (like Lance's 2-byte chunk metadata) would close the gap, but the paper does not measure this cold-start case.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper introduces 'structural encoding' as a unifying lens for understanding how columnar formats lay out nested data, and uses it to analyze Parquet, Arrow/IPC-style layouts, and the authors' Lance 2.1 format. It proposes an adaptive scheme in which large data types use a full-zip encoding with a repetition index, while small data types use a miniblock encoding with chunked opaque compression, plus optional struct packing. The experimental sections compare random access, compression, and full-scan performance on a local NVMe drive, using large datasets to reduce coalescing effects, a tuned best-case Parquet configuration, and direct file-level APIs. The central claim is that Lance achieves better random access than Parquet and Arrow-style encodings without sacrificing scan performance or RAM utilization.

Significance. If the central claims hold, the paper makes a useful design contribution: it identifies structural encoding as a first-class design dimension and shows, at least for the tested NVMe setting, that a format can combine good random access with good scan performance. The experimental design has notable strengths: it explicitly analyzes and mitigates coalescing effects (Section 5.4), benchmarks against a deliberately tuned best-case Parquet rather than a strawman default, uses direct file-level readers, and reports that reproduction scripts are available. The compression comparison across realistic ML-oriented datasets is also valuable. The most consequential claim, however, is the 'without trade-offs in RAM utilization' assertion, and that claim needs substantially stronger support before the paper's main conclusion can be accepted.

major comments (2)
  1. [Section 4.2.4, Abstract, Section 7] The claim that Lance's adaptive encoding provides random access 'without making trade-offs in ... RAM utilization' is not established for the small-data-type regime, which is exactly the regime in which the miniblock encoding is used. Section 4.2.4 states that Lance's miniblock search cache is 24 bytes per chunk without a repetition index and 41 bytes per chunk with one, while Parquet's page offset index is 20 bytes per page. With the paper's own sizes (a 4KiB Lance miniblock and an 8KiB Parquet page), a UInt64 column holds 512 values per Lance chunk and 1024 values per Parquet page, giving 24/512 = 0.047 bytes per value for Lance versus 20/1024 = 0.020 bytes per value for Parquet. Lance's search cache is therefore roughly 2.4x larger for scalar columns under the paper's own settings, and the additional 17-byte repetition index widens the gap for strings and nested types. Because Section 6 evaluates only warm searches (Section 2.3) and reports no memory-footprint measurements, the 'smaller search cache' statement in Section 7 is inverted for the small-type regime. The numerical cap in Section 4.2.4 (1.28 GiB for one billion rows) is also inconsistent with the stated 24 bytes per chunk and 32 values per chunk, since 24 x 10^9/32 = 0.70 GiB; this needs correction and direct measurement.
  2. [Section 6, Figures 10-18] The performance comparisons that carry the paper's main claims are reported without any indication of run-to-run variability. The text describes ten-second averages for random access but does not state whether multiple independent runs were performed, and the figures contain no error bars, confidence intervals, or raw throughput values. This matters because several conclusions are stated as ties or modest margins, such as Lance 'generally tie[ing] or beat[ing] Parquet' on scalar and string categories in Figure 11. Without repeated trials or a variance statement, the central empirical claim cannot be distinguished from machine noise, especially for the sub-10% differences that support the 'no trade-off' conclusion. Please report at least three runs per configuration with standard deviation, or otherwise justify why the observed margins are stable.
minor comments (5)
  1. [Section 5.3] The Arrow evaluation uses Lance 2.0 as a proxy for Arrow IPC, and Section 5.3 acknowledges one difference: Lance 2.0 uses special offsets for null lists instead of a dedicated validity bitmap. This difference can affect both the number of IOPS and the benefit of coalescing, so the conclusion that Arrow-style encodings 'require too many IOPS' should be stated as applying to the Lance 2.0 approximation, or verified against Arrow IPC directly.
  2. [Section 4.1] The 128-byte threshold for switching between full-zip and miniblock encoding is described as 'based on experimental measurements,' but no measurement or sensitivity analysis is shown. A short threshold sweep would make the adaptive scheme's robustness more convincing.
  3. [Section 2.3] The search cache is defined as aiming for 0.1% of data size, but the later arithmetic in Section 4.2.4 implies larger fractions under the stated settings: Parquet's 20-byte offset entry per 8KiB page is about 0.24% of the page data, and Lance's 24-byte entry per 4KiB miniblock is about 0.59%. Please clarify whether 0.1% is a target, a bound, or an approximate guideline.
  4. [Section 5.4] The phrase 'ascoalesced access' should read 'as coalesced access.'
  5. [Throughout] The terms 'miniblock' and 'mini-block' are used inconsistently; please choose one spelling and use it consistently.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity is present: the central claim rests on external benchmarks, while the 128-byte threshold is a design input and the paper's self-citations are contextual.

full rationale

The paper's central claim (adaptive full-zip/miniblock encoding gives better random access than Parquet and Arrow encodings without scan or RAM trade-offs) is supported by direct measurements in Section 6 against external baselines, not by deriving the conclusion from parameters fitted to the same claim. The 128-byte threshold that selects between the two Lance encodings is explicitly a design input ('We use 128 bytes per value as a threshold based on experimental measurements,' Section 4.1), so it is not a fitted quantity being relabeled as a prediction. The RAM discussion in Section 4.2.4 is an analytical size comparison (20 bytes per Parquet page versus 24 or 41 bytes per Lance miniblock chunk) that may be contestable for small scalar types, and Section 2.3 explicitly limits the evaluation to warm searches with the search cache resident; these are scope and correctness limitations, not circular reductions, because the comparison does not assume Lance's conclusion. The self-citations ([20] Lance v2 blog and [31] Rottnest) supply format context and prior vector-search motivation but are not load-bearing for the new measured results. No equation or construction in the paper makes the claimed result equivalent to its inputs, so the circularity burden is not met.

Assumptions & free parameters 1 free parameters · 4 assumptions · 1 invented entities

The central claim does not rest on fitted scientific constants; the only explicit fitted parameter is the 128-byte width threshold. Several domain assumptions about hardware behavior, warm caches, the Arrow proxy, and dataset representativeness are load-bearing.

free parameters (1)
  • full-zip threshold (data width) = 128 bytes
    Section 4.1 states the full-zip encoding is used for data types with at least 128 bytes per value, a threshold chosen from experimental measurements. This is a hand-fitted design parameter, not derived from first principles.
assumptions (4)
  • domain assumption NVMe random access performance is governed by IOPS and read amplification, with near-peak IOPS achieved at 4KiB reads.
    Section 1 and Figure 1 establish this hardware model; the entire benchmark interpretation depends on it.
  • domain assumption Warm search cache: small metadata (page offset index, miniblock metadata) is loaded once and cached, so its I/O cost is negligible.
    Section 2.3 defines the search cache and evaluates only warm searches. This assumption is load-bearing for the RAM trade-off claims.
  • domain assumption Lance 2.0 approximates Arrow's structural encoding despite using special offsets for null lists instead of a validity bitmap.
    Section 5.3 uses Lance 2.0 as a proxy for Arrow IPC. Differences could affect the Arrow comparison.
  • domain assumption The selected public datasets are representative of real ML and search workloads.
    Section 6.2 lists eight scenarios, including baby names, prompts, TPC-H dates, reviews, code, images, embeddings, and websites, assumed to be representative.
invented entities (1)
  • Lance 2.1 adaptive structural encoding (full-zip and miniblock with repetition index) independent evidence
    purpose: Achieve at most 1 IOP random access for fixed-width and 2 IOPS for variable-width columns without scan or RAM trade-offs.
    The format is implemented in the public Lance Rust crate and the paper specifies parameters and reproduction scripts, so external benchmarking is possible.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Lance: Efficient Random Access in Columnar Storage through Adaptive Structural Encodings." pith.science (2026). https://pith.science/paper/5QBHLVYQ

@misc{pith2026250415247,
  author       = {Pith},
  title        = {Pith review of: Lance: Efficient Random Access in Columnar Storage through Adaptive Structural Encodings},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5QBHLVYQ}},
  note         = {Machine review of arXiv:2504.15247}
}
read the original abstract

The growing interest in artificial intelligence has created workloads that require both sequential and random access. At the same time, NVMe-backed storage solutions have emerged, providing caching capability for large columnar datasets in cloud storage. Current columnar storage libraries fall short of effectively utilizing an NVMe device's capabilities, especially when it comes to random access. Historically, this has been assumed an implicit weakness in columnar storage formats, but this has not been sufficiently explored. In this paper, we examine the effectiveness of popular columnar formats such as Apache Arrow, Apache Parquet, and Lance in both random access and full scan tasks against NVMe storage. We argue that effective encoding of a column's structure, such as the repetition and validity information, is the key to unlocking the disk's performance. We show that Parquet, when configured correctly, can achieve over 60x better random access performance than default settings. We also show that this high random access performance requires making minor trade-offs in scan performance and RAM utilization. We then describe the Lance structural encoding scheme, which alternates between two different structural encodings based on data width, and achieves better random access performance without making trade-offs in scan performance or RAM utilization.

Figures

Figures reproduced from arXiv: 2504.15247 by the authors.

Figure 1
Figure 1. NVMe storage introduces unique performance char [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. An overview of the major components of each of the file formats [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. An interpretation of a column chunk containing [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: An example showing the layout and accesses needed [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 6
Figure 6. Figure 6: The full-zip encoding of ["AB", "C"], NULL LIST, NULL STRUCT, [NULL], [] with the data type Struct<List<String>> An example is provided in figure 6. There are 3 bits of definition and 1 bit of repetition, and so we have 1-byte control words. The least significant 3 bit…
Figure 7
Figure 7. Figure 7: A repetition index points to the control word of [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: Example miniblock chunk containing 248 nullable [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: The benefits of coalesced access diminish with [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 10
Figure 10. Figure 10: Parquet Random Access Performance As expected, the Parquet encoding scheme is highly dependent on the page size. The second chart in figure 10 shows the correlation between page size and performance. This is unsurprising, as our disk benchmarks confirmed that the size…
Figure 12
Figure 12. Figure 12: Full zip encoding is lighter weight and significantly [PITH_FULL_IMAGE:figures/full_fig_p010_12.png]
Figure 15
Figure 15. Figure 15: Row group size has significant effect on scans [PITH_FULL_IMAGE:figures/full_fig_p011_15.png]
Figure 16
Figure 16. Figure 16: Lance scan performance often exceeds Parquet [PITH_FULL_IMAGE:figures/full_fig_p011_16.png]
Figure 17
Figure 17. Figure 17: The miniblock encoding does less work during [PITH_FULL_IMAGE:figures/full_fig_p012_17.png]
Figure 18
Figure 18. Figure 18: Packing structs trade single field scan speed for [PITH_FULL_IMAGE:figures/full_fig_p012_18.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LayoutBench: Performance Benchmarking of Cloud Storage Layouts for Multimedia Data

    cs.DC 2026-07 conditional novelty 6.0 of 10

    LayoutBench benchmarks three cloud storage layouts for multimedia retrieval, finding tar-based packing offers the best latency-cost tradeoff for ImageNet-scale data.

  2. OptFSST: Optimized FSST String Compression

    cs.DB 2026-07 unverdicted novelty 5.5 of 10

    OptFSST lifts FSST's average compression factor by 7.3% and FSST12's by 17.0% across 92 string columns using DP encoding, triple counting, and pruning; it also proves the symbol-table selection problem is NP-hard when...

Reference graph

Works this paper leans on

34 extracted references · 20 canonical work pages · cited by 2 Pith papers

  1. [1]

    Daniel Abadi, Samuel Madden, and Miguel Ferreira. 2006. Integrating compres- sion and execution in column-oriented database systems. In Proceedings of the 2006 ACM SIGMOD International Conference on Management of Data (Chicago, Lance: Efficient Random Access in Columnar Storage through Adaptive Structural Encodings IL, USA) (SIGMOD ’06). Association for C...

    arXiv 2006

  2. [2]

    Azim Afroozeh and Peter Boncz. 2023. The FastLanes Compression Layout: Decoding > 100 Billion Integers per Second with Scalar Code. Proc. VLDB Endow. 16, 9 (May 2023), 2132–2144. https://doi.org/10.14778/3598581.3598587

    arXiv 2023

  3. [3]

    Kuffo, and Peter Boncz

    Azim Afroozeh, Leonardo X. Kuffo, and Peter Boncz. 2023. ALP: Adaptive Lossless floating-Point Compression. Proc. ACM Manag. Data 1, 4, Article 230 (Dec. 2023), 26 pages. https://doi.org/10.1145/3626717

    doi:10.1145/3626717 2023

  4. [4]

    AWS. [n.d.]. Best practices design patterns: optimizing Amazon S3 performance . Amazon. Retrieved February 19, 2024 from https://docs.aws.amazon.com/ AmazonS3/latest/userguide/optimizing-performance.html

    work page 2024

  5. [5]

    Census Bureau

    U.S. Census Bureau. 2021. Popular Baby Names. Retrieved February 20, 2024 from https://www.ssa.gov/oact/babynames/limits.html

    work page 2021

  6. [6]

    Biswapesh Chattopadhyay, Priyam Dutta, Weiran Liu, Ott Tinn, Andrew Mc- cormick, Aniket Mokashi, Paul Harvey, Hector Gonzalez, David Lomax, Sagar Mittal, Roee Ebenstein, Nikita Mikhaylin, Hung-ching Lee, Xiaoyan Zhao, Tony Xu, Luis Perez, Farhad Shahmohammadi, Tran Bui, Neil McKay, Selcuk Aya, Vera Lychagina, and Brett Elliott. 2019. Procella: unifying se...

    arXiv 2019

  7. [7]

    Common Crawl. 2025. Common Crawl - Open Repository of Web Crawl Data . Retrieved February 20, 2024 from https://commoncrawl.org/

    work page 2025

  8. [8]

    Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. 2023. Enhancing Chat Language Models by Scaling High-quality Instructional Conversations.arXiv preprint arXiv:2305.14233 (2023)

    arXiv 2023

Show all 34 references
  1. [9]

    Dominik Durner, Viktor Leis, and Thomas Neumann. 2023. Exploiting Cloud Object Storage for High-Performance Analytics. Proc. VLDB Endow. 16, 11 (July 2023), 2769–2782. https://doi.org/10.14778/3611479.3611486

    2023

  2. [10]

    Kira Duwe, Angelos Anadiotis, Andrew Lamb, Lucas Lersch, Boaz Leskes, Daniel Ritter, and Pınar Tözün. 2025. The Five-Minute Rule for the Cloud: Caching in Analytics Systems. In Proceedings of The Biennial Conference on Innovative Data Systems Research (CIDR)

    2025

  3. [11]

    Apache Software Foundation. [n.d.]. Apache ORC. Retrieved February 26, 2024 from https://orc.apache.org/

    2024

  4. [12]

    Apache Software Foundation. [n.d.]. Apache Parquet. Retrieved February 21, 2024 from https://parquet.apache.org/

    2024

  5. [13]

    [n.d.].Arrow Columnar Format

    Apache Software Foundation. [n.d.].Arrow Columnar Format. Retrieved February 21, 2024 from https://arrow.apache.org/docs/format/Columnar.html

    2024

  6. [14]

    GitHub. 2021. Github Activity Data . Retrieved February 27, 2024 from https://github.blog/news-insights/research/making-open-source-data-more- available/

    2021

  7. [15]

    Harby and Farhana Zulkernine

    Ahmed A. Harby and Farhana Zulkernine. 2022. From Data Warehouse to Lakehouse: A Comparative Review. In 2022 IEEE International Conference on Big Data (Big Data). 389–395. https://doi.org/10.1109/BigData55660.2022.10020719

    2022

  8. [16]

    Juicedata. [n.d.]. JuiceFS: A High-Performance, Cloud-Native, Distributed File System. Juicedata. Retrieved February 21, 2024 from https://juicefs.com

    2024

  9. [17]

    Maximilian Kuschewski, David Sauerwein, Adnan Alhomssi, and Viktor Leis

  10. [19]

    Andrew Lamb and Raphael Taylor-Davies. 2022. Querying Parquet with Millisec- ond Latency. Retrieved February 19, 2024 from https://arrow.apache.org/blog/ 2022/12/26/querying-parquet-with-millisecond-latency/

    2022

  11. [20]

    LanceDB. 2024. Lance v2: A columnar container format for modern data . LanceDB. Retrieved February 19, 2024 from https://blog.lancedb.com/lance-v2/

    2024

  12. [21]

    Gang Liao, Ye Liu, Jianjun Chen, and Daniel J Abadi. 2025. Bullion: A Col- umn Store for Machine Learning. In Proceedings of The Biennial Conference on Innovative Data Systems Research (CIDR)

    2025

  13. [22]

    Julian John McAuley and Jure Leskovec. 2013. From amateurs to connoisseurs: modeling the evolution of user expertise through online reviews. In Proceedings of the 22nd International Conference on World Wide Web (Rio de Janeiro, Brazil) (WWW ’13). Association for Computing Mach...

    2013

  14. [23]

    Sergey Melnik, Andrey Gubarev, Jing Jing Long, Geoffrey Romer, Shiva Shiv- akumar, Matt Tolton, and Theo Vassilakis. 2010. Dremel: interactive anal- ysis of web-scale datasets. Proc. VLDB Endow. 3, 1–2 (Sept. 2010), 330–339. https://doi.org/10.14778/1920841.1920886

    2010

  15. [24]

    Pedro Pedreira, Orri Erling, Konstantinos Karanasos, Scott Schneider, Wes McKin- ney, Satya R Valluri, Mohamed Zait, and Jacques Nadeau. 2023. The Composable Data Management System Manifesto. Proc. VLDB Endow. 16, 10 (June 2023), 2679–2685. https://doi.org/10.14778/3603581.3603604

    2023

  16. [25]

    Meta Platforms. [n.d.]. The Nimble File Format . Meta Platforms. Retrieved February 26, 2024 from https://github.com/facebookincubator/nimble

    2024

  17. [26]

    Martin Prammer, Xinyu Zeng, Ruijun Meng, Wes McKinney, Huanchen Zhang, Andrew Pavlo, and Jignesh Patel. 2025. Towards Functional Decomposition of Storage Formats. In CIDR 2025, Conference on Innovative Data Systems Research . https://db.cs.cmu.edu/papers/2025/p19-prammer.pdf

    2025

  18. [27]

    Tobias Schmidt, Andreas Kipf, Dominik Horn, Gaurav Saxena, and Tim Kraska

  19. [28]

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Lud- wig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. 202...

    2022

  20. [29]

    Spiral. [n.d.]. Vortex. Spiral. https://github.com/spiraldb/vortex

  21. [30]

    Frontier Research Team. 2025. image_captions. Takara.ai. Retrieved February 20, 2024 from https://huggingface.co/datasets/takara-ai/image_captions

    2025

  22. [31]

    Ziheng Wang, Sasha Krassovsky, Conor Kennedy, Alex Aiken, Weston Pace, Rain Jiang, Huayi Zhang, Chenyu Jiang, and Wei Xu. 2025. Rottnest: Indexing Data Lakes for Search. (2025). submitted for publication

    2025

  23. [32]

    Xinyu Zeng, Yulong Hui, Jiahong Shen, Andrew Pavlo, Wes McKinney, and Huanchen Zhang. 2023. An Empirical Evaluation of Columnar Storage Formats. Proc. VLDB Endow. 17, 2 (Oct. 2023), 148–161. https://doi.org/10.14778/3626292. 3626298

    2023 doi

  24. [33]

    Liran Zvibel, David Hiatt, and Barbara Murphy. 2021. WekaFS Architecture White Paper. Technical Report. Experis Technology Group

    2021

  25. [2023]

    BtrBlocks: Efficient Columnar Compression for Data Lakes. Proc. ACM Manag. Data 1, 2, Article 118 (June 2023), 26 pages. https://doi.org/10.1145/ 3589263

    2023

  26. [2024]

    In Companion of the 2024 International Conference on Management of Data (Santiago AA, Chile) (SIGMOD/PODS ’24)

    Predicate Caching: Query-Driven Secondary Indexing for Cloud Data Ware- houses. In Companion of the 2024 International Conference on Management of Data (Santiago AA, Chile) (SIGMOD/PODS ’24). Association for Computing Machinery, New York, NY, USA, 347–359. https://doi.org/10.1...

    2024

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.