Pith. sign in

REVIEW 5 major objections 6 minor 72 references

Data Formats in Analytical DBMSs: Performance Trade-offs and Future Directions

T0 review · 5 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims no single open columnar format fits all analytical DBMS workloads; ORC wins queries, Parquet wins compression, Arrow Feather wins serialization, and none fits ML embeddings.

desk verdict Useful benchmark with real new measurements on embeddings and in-memory Parquet, but the stale and mismatched software versions plus missing artifacts mean the precise numbers need a re-test before being trusted. read the letter →

arxiv 2411.14331 v1 pith:OCWKSPJM submitted 2024-11-21 cs.DB

Chunwei Liu , Anna Pavlenko , Matteo Interlandi , Brandon Haynes This is my paper · ORCID
classification cs.DB
keywords columnarstorageformatsApacheArrowParquetORCOLAPqueryperformancedataskippingcompressionandencodingvectorembeddings
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to determine whether any of the three dominant open columnar formats—Apache Arrow (Feather), Parquet, and ORC—can serve as a native format inside an analytical database management system. Through micro-benchmarks and TPC-DS query subexpressions, it finds no winner on all dimensions: ORC gives the best query performance thanks to fine-grained skipping and a dedicated in-memory representation, Parquet gives the best compression, and Arrow Feather gives the fastest serialization. It also finds that for popular machine-learning workloads on vector embeddings, all three formats compress poorly and transcode inefficiently. The paper argues that rather than picking one format, designers should co-design a unified in-memory and on-disk representation, and shows that pushing query evaluation into encoded data can be far faster than decoding first.

What carries the argument

The generic columnar format architecture that the paper abstracts from Arrow, Parquet, and ORC: tables are split into row batches, each column into chunked columns, with metadata such as zone maps, min/max values, bloom filters, and offsets stored in footers or indexes. The benchmark battery is organized around five dimensions, namely compression ratio, transcoding throughput, data access (projection and predicate evaluation), end-to-end subexpressions, and advanced features such as direct querying and SIMD. The augmented implementations, including in-memory Parquet with direct query and SIMD, and Arrow with chunk-level skipping, serve as probes to test whether pushing computation into the encoded domain can overcome the default formats' compromises.

What would settle it

Re-run the same benchmark battery with current stable releases of Arrow, Parquet, and ORC, and with per-format parameter tuning; if the ordinal rankings on the five evaluation dimensions change, or if any format achieves strong compression and fast transcoding on embedding datasets, the paper's core trade-off picture would need revision.

Watch

Extended reading notes

Core claim

Evaluated across compression ratio, transcoding throughput, projection, predicate filtering, and end-to-end select-project subexpressions, the three formats form a trade-off triangle rather than a hierarchy. ORC tends to win query-heavy subexpressions because its smaller row batches enable more skipping and its row-level index plus dedicated in-memory column vector batch make loading efficient; Parquet tends to win compression because of its dictionary, RLE, and bit-packing stack; Arrow Feather tends to win serialization and deserialization because it skips encoding. Adding direct querying in the encoded domain to an in-memory Parquet variant yields large speedups. For embedding workloads such as retrieval-augmented generation and k-nearest-neighbor search, none of the formats compresses or serializes well, and the authors suggest precision-controlled encoding as a direction.

Load-bearing premise

The measured trade-off rankings rest on the specific software versions tested (Arrow 5.0.0, ORC 1.7.2, Parquet Java 1.9.0, PyArrow 17.0.0) being representative of each format's current production performance and default settings.

Editorial extensions

If this is right

  • DBMS format choice should be workload-dependent: ORC for predicate-heavy subexpressions, Parquet for compression, Arrow Feather for serialization speed.
  • Fine-grained data skipping pays off at low selectivity, while bulk-loading APIs win at high selectivity, so a format should support both access patterns.
  • Pushing query operators into encoded data with SIMD can produce large speedups, up to roughly 100x over the Parquet baseline in this study.
  • None of the three formats is adequate for vector-embedding workloads, so compression and transcoding for embeddings need new encoding approaches.
  • A unified, co-designed in-memory and on-disk representation that avoids the current convert-to-Arrow practice is a promising direction for future formats.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Since Arrow's loading overhead in this study came largely from per-row-batch lock acquisition, implementation-level fixes or newer Arrow releases could narrow the gap with ORC and shift the trade-off surface.
  • The finding that ORC wins through smaller batch granularity suggests that workload-aware partitioning and layout learning could amplify or invert these results when applied to Parquet or Arrow.
  • The embedding compression failure points to a testable design: applying precision-controlled float encoding to nested vector columns could recover the missing gains, a direction the paper points to without implementing.
  • The paper's default cold-cache protocol likely understates Parquet's streaming advantages; a warm-cache, pipelined setting could change some ordinal rankings across the five dimensions.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper evaluates Apache Arrow/Feather, Parquet, and ORC as candidate native formats for analytical DBMSs. It organizes the comparison around compression ratio, transcoding throughput, projection and predicate data access, TPC-DS leaf subexpressions, and advanced optimizations such as direct querying, vectorization, and data skipping. The central conclusions are that Parquet offers the best compression, Arrow Feather the fastest (de)serialization, ORC the best query-subexpression performance, and that none of the formats is adequate for machine-learning embedding workloads, motivating co-design of in-memory and on-disk representations.

Significance. If the empirical findings hold, the paper is a useful reference for system builders choosing or designing columnar formats, and its embedding-workload results point to a real gap in current formats. The paper's strengths are the breadth of datasets (TPC-DS, JOB, Public BI, CodecDB, Hugging Face embeddings), the systematic dimension-by-dimension structure, and the concrete prototype variants (Arrow direct query and chunk-level skipping, in-memory Parquet with SIMD) that demonstrate possible improvements. However, the empirical basis needs to be strengthened: the benchmarked software versions are old and internally inconsistent, no error bars or repetition counts are reported, no artifact is provided, and part of the comparison conflates format properties with particular library APIs. The 'first detailed evaluation' claim also needs qualification given reference [71].

major comments (5)
  1. [Section 4, Setup] The software stack is internally inconsistent and outdated. The paper lists Apache Arrow 5.0.0, ORC 1.7.2, Parquet Java 1.9.0, and PyArrow 17.0.0 in the same setup. PyArrow 17.0.0 bundles Arrow C++ 17.0.0, not Arrow 5.0.0, so experiments that use PyArrow (the embedding datasets) and experiments that use Arrow C++ 5.0.0 (the relational tables) cannot be directly compared. Moreover, Parquet Java 1.9.0 (2016) and ORC 1.7.2 (2021) are far older than current releases, and Table 3 itself notes that Parquet's default encoding differed between the legacy Java API and the latest C++ API. Because default encodings and compression behavior determine many of the rankings in Sections 5-7, the measured results may be artifacts of old defaults. Please re-run the core experiments on current, consistently versioned libraries, or clearly explain why the old versions are representative.
  2. [Sections 5-7, all figures] No error bars, standard deviations, or repetition counts are reported. The paper states in Section 4 that numbers are reported with cold caches by default, but not how many runs were averaged. Given that several headline effects are 2-4x differences (e.g., ORC's loading advantage in Section 6, Parquet's compression advantage in Section 5.1), single-run measurements cannot establish stability. Table 6 has some AVG/STD entries, but the figures do not carry variance information. Please add repetitions, variance measures, and ideally a reproducibility artifact; without these, the quantitative rankings are not independently checkable.
  3. [Abstract and Section 1] The phrase 'first detailed, empirical evaluation' is contradicted by the paper's own reference [71], an arXiv empirical evaluation of columnar storage formats. The related-work section mentions [71] only as part of 'robust discussion' and does not explain what is different. Please qualify the novelty claim and explicitly state how the present study differs from [71] (e.g., DBMS-subsumption framing, embedding workloads, advanced-optimization experiments).
  4. [Sections 5.2.1, 6.1.1, 6.2, and 7] Several conclusions about 'formats' are actually conclusions about specific library APIs and implementations. For example, Section 5.2.1 attributes ORC's slow write time to 'better Parquet support in Arrow; both projects share the same codebase and data structures,' and Section 6.1.1 explains Arrow's poor projection performance through its bulk-loading API while praising ORC's column-level API. Similarly, Sections 6.2 and 7 compare Parquet's streaming API against ORC's and Arrow's bulk in-memory representations. These are implementation differences rather than properties of the file formats themselves. The paper should either control for this confound or consistently phrase the findings as 'with the APIs tested here' rather than as format-inherent trade-offs.
  5. [Sections 5.1.2-5.1.4 vs. 5.2.1] There is an inconsistency regarding ORC and LZ4. Section 5.2.1 states that LZ4 and Snappy bars are omitted for ORC because the Apache Arrow C++ library has limited compression support for ORC, yet Figures 10, 14, and 15 in Sections 5.1.2-5.1.4 report ORC LZ4 compression ratios. If the ORC LZ4 numbers come from a different code path or version, the cross-experiment comparability is broken; if they are from the same code path, the omission in Section 5.2.1 is unexplained. Please clarify which ORC implementation is used for each figure and make the treatment of LZ4 consistent.
minor comments (6)
  1. [Section 5.1.1] The paragraph beginning 'For floats, as we can see from Figure 8b...' and the following paragraph are nearly verbatim repeats of the earlier discussion after Figures 5 and 8; one copy should be removed or merged.
  2. [Section 8.1.2] The text says 'LL VM-based backend' and 'LL VM compiler passes'; this should be 'LLVM' throughout.
  3. [Figure 28a] The x-axis label 'Selectivty' should be 'Selectivity'. Several figure captions also contain leftover placeholder text such as 'Figure 5 legend', 'Legend removed', and 'Needs additional data points' (e.g., Figures 10, 22, 25, 29); these should be cleaned before final submission.
  4. [Section 5.1.4] The embedding dataset description does not list the 38 datasets, their download versions, or the exact embedding-generation procedure, which limits reproducibility; adding a table or a public artifact link would help.
  5. [Section 4, Setup] The statement 'we report numbers when the system caches are cold by default' is not precise: it should define how cold caches are produced (e.g., dropping OS caches, number of trials, warm-up runs). The warm/cold distinction in Section 7 is also only reported for Arrow and ORC, with the caption saying Parquet changes were negligible; please state whether Parquet was measured in both configurations.
  6. [Section 1, Contributions] The claim that the paper evaluates 'the ability for each format to take advantage of recent trends such as vectorization, query compilation, and direct querying' is overstated: Gandiva compilation is tested on only a small set of expressions, and direct querying is implemented only for custom Arrow and Parquet prototypes. Consider narrowing the wording to match the experiments.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's rankings and trade-off conclusions are empirical measurements against external datasets and baselines, not reductions to fitted inputs or self-citations.

full rationale

The paper is an empirical benchmark rather than a derivation. Its headline results—Parquet's compression advantage, ORC's query-subexpression advantage, Arrow Feather's serialization advantage, and the inadequacy of all three formats for embedding workloads—are obtained by measuring compressed sizes and runtimes over external datasets (TPC-DS, JOB, Public BI, CodecDB, Hugging Face) and against external baselines (CSV, NumPy). I looked for the listed circularity patterns and found none. There is no self-definitional step: Table 1's feature comparison is descriptive, and the optimized variants in Section 8 are implementations whose performance is measured, not consequences of the formats' definitions. There is no fitted input renamed as a prediction: no parameter is calibrated on one subset and then used to 'predict' a closely related quantity. The paper does reuse first-author prior systems (CodecDB, SBoost, BUFF) as implementation sources and as future-work suggestions, but these citations are not load-bearing for the central empirical rankings; the rankings stand on new measurements reported in Sections 5–8. Concerns such as the version mismatch between Apache Arrow 5.0.0 and PyArrow 17.0.0 and the lack of a released artifact are correctness and reproducibility risks, not circularity. No load-bearing step reduces to its own input, so the circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The benchmark rests on the assumption that default settings, public datasets, and the cited software versions yield representative comparisons. No invented entities are introduced; no numbers are fitted to data.

assumptions (3)
  • domain assumption Default format settings are representative of how real systems use these formats.
    Stated in Section 4: 'Unless stated otherwise, we use each format's default settings.' The conclusions are qualified by acknowledging tuning could change results.
  • domain assumption The benchmarked software versions are representative of current format performance.
    Section 4 lists Apache Arrow 5.0.0, ORC 1.7.2, Parquet Java 1.9.0, PyArrow 17.0.0; these are not the latest and the paper treats results as general.
  • domain assumption Column-wise serialization and aggregation by data type preserve format-level trade-offs.
    Section 5.1.1 serializes each column separately and groups by type; this may overstate metadata overhead for narrow tables.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Data Formats in Analytical DBMSs: Performance Trade-offs and Future Directions." pith.science (2026). https://pith.science/paper/OCWKSPJM

@misc{pith2026241114331,
  author       = {Pith},
  title        = {Pith review of: Data Formats in Analytical DBMSs: Performance Trade-offs and Future Directions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OCWKSPJM}},
  note         = {Machine review of arXiv:2411.14331}
}
read the original abstract

This paper evaluates the suitability of Apache Arrow, Parquet, and ORC as formats for subsumption in an analytical DBMS. We systematically identify and explore the high-level features that are important to support efficient querying in modern OLAP DBMSs and evaluate the ability of each format to support these features. We find that each format has trade-offs that make it more or less suitable for use as a format in a DBMS and identify opportunities to more holistically co-design a unified in-memory and on-disk data representation. Notably, for certain popular machine learning tasks, none of these formats perform optimally, highlighting significant opportunities for advancing format design. Our hope is that this study can be used as a guide for system developers designing and using these formats, as well as provide the community with directions to pursue for improving these common open formats.

Figures

Figures reproduced from arXiv: 2411.14331 by the authors.

Figure 1
Figure 1. Columnar format layout. best when the dataset has small cardinality and many repetitions. Queries on dictionary encoded data can be applied either on the fully decoded data or directly in the encoded domain after query rewriting using dictionary translation. Run-Length Encoding (RLE) works on data with many consecutive repetitions. It replaces a run of the same value with a pair consisting of the value and how many … view at source ↗
Figure 2
Figure 2. A Parquet row batch [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. Ratio of number of distinct values (#Distinct) to the number of rows (#Rows) in the CodecDB, Public BI and JOB datasets. The spikes for integer types near D/N = 1 in the CODEC and JOB datasets occur because of primary key columns, which contain no duplicate values. (a) Integer (Int32, Int64) (b) Float (Float, Double) (c) String (Binary) [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figures from the paper (22 more)
Figure 5
Figure 5. Figure 5: Compression ratios on the CodecDB real-world datasets with ∼18k columns. The figures show the effective compression ratio (CR) in the range [0, 1]. The CDF lines do not always reach 1.0 at CR = 1.0 because of underperforming compression on some columns [PITH_FULL_IMAG…
Figure 6
Figure 6. Figure 6: Compression ratio performance on the BI dataset over ∼13k columns. (a) Integer (int32, int64) (b) String (binary) [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Compression ratio performance on the JOB dataset with approximately ∼100 columns. (There are no floating-point columns in this dataset.). (a) Integers (b) Floats (c) Strings [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Column compression ratio CDFs over the CodecDB, BI and JOB datasets. [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Distinct value CDFs. real-world dataset. For integers in this dataset (Fig￾ure 5a), dictionary+RLE encoding in Parquet performs well for low distinctness ratios, while RLE alone in ORC works relatively well for high distinctness ratios for in￾teger columns. This is the…
Figure 10
Figure 10. Figure 10: Compression ratio (compressed size / original [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]
Figure 11
Figure 11. Figure 11: Total size on disk after compressing the numeric [PITH_FULL_IMAGE:figures/full_fig_p010_11.png]
Figure 12
Figure 12. Figure 12: Total size on disk after compressing the string [PITH_FULL_IMAGE:figures/full_fig_p010_12.png]
Figure 13
Figure 13. Figure 13: Write time from an Arrow in-memory table to each format stored either on disk or in memory. [PITH_FULL_IMAGE:figures/full_fig_p012_13.png]
Figure 16
Figure 16. Figure 16: Runtime (in seconds) for decompressing the [PITH_FULL_IMAGE:figures/full_fig_p012_16.png]
Figure 15
Figure 15. Figure 15: Total size on disk after compressing embedding datasets using the nested layout (i.e., each vector is se￾rialized as a single nested column). The “NumPy” base￾line shows the size of data serialized using the NumPy uncompressed standard binary format. 5.2.1 Compression…
Figure 17
Figure 17. Figure 17: Write time using a columnar layout (i.e., a n￾dimensional vector is represented as n columns) to each format on disk. The CSV baseline (not shown in figure) took 1,982 seconds to write. load the data and convert it into the Arrow in-memory format. The results are show…
Figure 19
Figure 19. Figure 19: Runtime (in seconds) for decompressing the table from the on-disk formats into an in-memory table using the columnar layout. NumPy Figure “vec -read” 0 60 120 None Zstd LZ4 gzip Snappy zlib Runtime (sec) Parquet Arrow Feather ORC [PITH_FULL_IMAGE:figures/full_fig_p01…
Figure 20
Figure 20. Figure 20: Runtime (in seconds) for decompressing the table from the on-disk formats into an in-memory ta￾ble using the nested layout. The “NumPy” line shows time to decompress data serialized using the NumPy uncompressed standard binary format. tently outperforms the others due…
Figure 21
Figure 21. Figure 21: Runtime (in seconds) for decompressing the [PITH_FULL_IMAGE:figures/full_fig_p014_21.png]
Figure 23
Figure 23. Figure 23: Projecting strings on the customer demographic table [PITH_FULL_IMAGE:figures/full_fig_p015_23.png]
Figure 25
Figure 25. Figure 25: Arrow serial vs parallel (default). fore projecting, Arrow Feather outperforms the other formats in this experiment. This is because by default Arrow Feather does not dictionary encode its data and is therefore able to entirely avoid the associated lookup overhead. OR…
Figure 27
Figure 27. Figure 27: Filtering strings on the customer demographic table. predicate evaluation (Sections 6.2.1 and 6.2.2 respec￾tively for numeric and string columns) and bit-vector evaluation (Section 6.2.3). 6.2.1 Numeric predicates. We evaluate two predicates over the customer sale tab…
Figure 28
Figure 28. Figure 28: Bit-vector application performance by selectiv [PITH_FULL_IMAGE:figures/full_fig_p017_28.png]
Figure 30
Figure 30. Figure 30: Table 7 runtimes for cold and warm caches on [PITH_FULL_IMAGE:figures/full_fig_p018_30.png]
Figure 31
Figure 31. Figure 31: Arrow Feather runtime with and without direct [PITH_FULL_IMAGE:figures/full_fig_p018_31.png]
Figure 32
Figure 32. Figure 32: Parquet performance with various optimizations. [PITH_FULL_IMAGE:figures/full_fig_p020_32.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

72 extracted references · 71 canonical work pages

  1. [71]

    X. Zeng, Y. Hui, J. Shen, A. Pavlo, W. McKinney, and H. Zhang. An empirical evaluation of columnar storage formats. arXiv preprint arXiv:2304.05028 , 2023

    work page Pith review arXiv 2023

  2. [1]

    en.wikipedia.org/wiki/ Comma-separated_values

    Comma-separated values. en.wikipedia.org/wiki/ Comma-separated_values. Accessed: 2024

    work page 2024

  3. [2]

    huggingface.co/docs/datasets/index

    Datasets. huggingface.co/docs/datasets/index. (Ac- cessed on 09/11/2024)

    work page 2024

  4. [3]

    www.hdfgroup.org/ solutions/hdf5/

    The hdf5® library and file format. www.hdfgroup.org/ solutions/hdf5/. (Accessed on 09/20/2024)

    work page 2024

  5. [4]

    github.com/gregrahn/ join-order-benchmark

    Join Order Benchmark (JOB). github.com/gregrahn/ join-order-benchmark . [Accessed: 2024]

    work page 2024

  6. [5]

    www.json.org/json-en

    Json (javascript object notation). www.json.org/json-en. html. Accessed: 2024

    work page 2024

  7. [6]

    github.com/lancedb/lance

    lance. github.com/lancedb/lance. (Accessed on 09/19/2024)

    work page 2024

  8. [7]

    numpy.org/devdocs/reference/generated/ numpy.lib.format.html

    Npy format. numpy.org/devdocs/reference/generated/ numpy.lib.format.html. (Accessed on 09/18/2024)

    work page 2024

Show all 72 references
  1. [8]

    docs.python.org/3/library/pickle.html

    pickle — python object serialization — python 3.12.6 doc- umentation. docs.python.org/3/library/pickle.html. (Accessed on 09/18/2024)

    2024

  2. [9]

    github.com/cwida/public_bi_ benchmark

    Public BI benchmark. github.com/cwida/public_bi_ benchmark. [Accessed: 2024]

    2024

  3. [10]

    Pyarrow - apache arrow python bindings v17.0.0. arrow. apache.org/docs/python/index.html. (Accessed on 09/20/2024)

    2024

  4. [11]

    pytorch.org/tutorials/recipes/recipes/saving_and_ loading_models_for_inference.html

    Saving and loading models for inference in pytorch. pytorch.org/tutorials/recipes/recipes/saving_and_ loading_models_for_inference.html. (Accessed on 09/18/2024)

    2024

  5. [12]

    www.dremio.com/resources/guides/apache-arrow/

    What is apache arrow? capabilities & benefits — dremio. www.dremio.com/resources/guides/apache-arrow/. Ac- cessed: 2024

    2024

  6. [13]

    stackoverflow.com/questions/53533506/ what-is-the-difference-between-apache-drills- aluevectors-and-apache-arrow

    What is the difference between apache drill’s valuevectors and apache arrow? - stack over- flow. stackoverflow.com/questions/53533506/ what-is-the-difference-between-apache-drills- aluevectors-and-apache-arrow . (Accessed on 09/18/2024)

    2024

  7. [14]

    D. Abadi. Apache Arrow vs. Parquet and ORC: Do we really need a third Apache project for columnar data representation? dbmsmusings.blogspot.com/2017/10/ apache-arrow-vs-parquet-and-orc-do-we.html . Ac- cessed: 2024

    2017

  8. [15]

    Abadi, P

    D. Abadi, P. A. Boncz, S. Harizopoulos, S. Idreos, and S. Madden. The design and implementation of modern column-oriented database systems. 5(3):197–280, 2013

    2013

  9. [16]

    Abadi, S

    D. Abadi, S. Madden, and M. Ferreira. Integrating com- pression and execution in column-oriented database sys- tems. In SIGMOD, pages 671–682, 2006. 22 Chunwei Liu et al

    2006

  10. [17]

    D. J. Abadi, S. R. Madden, and N. Hachem. Column- stores vs. row-stores: how different are they really? In SIGMOD, pages 967–980, 2008

    2008

  11. [18]

    Agarwal, A

    R. Agarwal, A. Khandelwal, and I. Stoica. Succinct: Enabling queries on compressed data. In NSDI, pages 337–350, 2015

    2015

  12. [19]

    Apache Arrow

    Apache Software Foundation. Apache Arrow. arrow. apache.org. Accessed: 2024

    2024

  13. [20]

    Apache CarbonData

    Apache Software Foundation. Apache CarbonData. carbondata.apache.org. Accessed: 2024

    2024

  14. [21]

    Apache Feather

    Apache Software Foundation. Apache Feather. arrow. apache.org/docs/python/feather.html. Accessed: 2024

    2024

  15. [22]

    Apache ORC

    Apache Software Foundation. Apache ORC. orc.apache. org. Accessed: 2024

    2024

  16. [23]

    Apache Parquet

    Apache Software Foundation. Apache Parquet. parquet. apache.org. Accessed: 2024

    2024

  17. [24]

    ColumnIndex lay- out to support page skipping

    Apache Software Foundation. ColumnIndex lay- out to support page skipping. github.com/apache/ parquet-format/blob/master/PageIndex.md. Accessed: 2024

    2024

  18. [25]

    Armbrust, A

    M. Armbrust, A. Ghodsi, R. Xin, and M. Zaharia. Lake- house: a new generation of open platforms that unify data warehousing and advanced analytics. In CIDR, 2021

    2021

  19. [26]

    Bian and A

    H. Bian and A. Ailamaki. Pixels: An efficient column store for cloud data lakes. In ICDE, pages 3078–3090, 2022

    2022

  20. [27]

    Boncz, T

    P. Boncz, T. Neumann, and V. Leis. FSST: fast random access string compression. In VLDB, volume 13, pages 2649–2661, 2020

    2020

  21. [28]

    Deutsch et al

    P. Deutsch et al. Gzip file format specification version 4.3. Technical report, RFC 1952, May, 1996

    1952

  22. [29]

    Ferrari and M

    A. Ferrari and M. Russo. The definitive guide to DAX: Business intelligence with Microsoft Excel, SQL server analysis services, and Power BI . Microsoft Press, 2015

    2015

  23. [30]

    Floratou, U

    A. Floratou, U. F. Minhas, and F.¨Ozcan. SQL-on-Hadoop: Full circle back to shared-nothing database architectures. VLDB, 7(12):1295–1306, 2014

    2014

  24. [31]

    Gailly and M

    J.-l. Gailly and M. Adler. Zlib compression library. 2004

    2004

  25. [32]

    Snappy: a fast compressor/decompressor

    Google. Snappy: a fast compressor/decompressor. google. github.io/snappy. Accessed: 2024

    2024

  26. [33]

    Gracia-Tinedo, M

    R. Gracia-Tinedo, M. Sanchez-Artigas, P. Garcia-Lopez, Y. Moatti, and F. Gluszak. Lamda-flow: Automatic push- down of dataflow operators close to the data. In CCGRID, pages 112–121, 2019

    2019

  27. [34]

    Graefe and L

    G. Graefe and L. D. Shapiro. Data compression and database performance. University of Colorado, Boulder, Department of Computer Science, 1990

    1990

  28. [35]

    Gupta, D

    A. Gupta, D. Agarwal, D. Tan, J. Kulesza, R. Pathak, S. Stefani, and V. Srinivasan. Amazon redshift and the case for simpler data warehouses. In SIGMOD, pages 1917–1923, 2015

    1917

  29. [36]

    Hentschel, M

    B. Hentschel, M. S. Kester, and S. Idreos. Column sketches: A scan accelerator for rapid and robust predicate evaluation. In SIGMOD, pages 857–872, 2018

    2018

  30. [37]

    Querying Parquet with mil- lisecond latency

    InfluxData. Querying Parquet with mil- lisecond latency. influxdata.com/blog/ querying-parquet-millisecond-latency , December

  31. [38]

    Ivanov and M

    T. Ivanov and M. Pergolesi. The impact of columnar file formats on SQL-on-Hadoop engine performance: A study on ORC and parquet. CCPE, 32(5), 2020

    2020

  32. [39]

    P. Jain, P. Kraft, C. Power, T. Das, I. Stoica, and M. Za- haria. Analyzing and comparing lakehouse storage sys- tems. In CIDR, 2023

    2023

  33. [40]

    Jiang and A

    H. Jiang and A. J. Elmore. Boosting data filtering on columnar encoding with SIMD. In DaMoN, pages 1–10, 2018

    2018

  34. [41]

    Jiang, C

    H. Jiang, C. Liu, Q. Jin, J. Paparrizos, and A. J. Elmore. PIDS: attribute decomposition for improved compres- sion and query performance in columnar storage. VLDB, 13(6):925–938, 2020

    2020

  35. [42]

    Jiang, C

    H. Jiang, C. Liu, J. Paparrizos, A. A. Chien, J. Ma, and A. J. Elmore. Good to the last bit: Data-driven encoding with CodecDB. In SIGMOD, pages 843–856, 2021

    2021

  36. [43]

    D. Kang, R. Jiang, and S. Blanas. Jigsaw: A data storage and query processing engine for irregular table partition- ing. In SIGMOD, pages 898–911, 2021

    2021

  37. [44]

    Larson, C

    P.-˚A. Larson, C. Clinciu, E. N. Hanson, A. Oks, S. L. Price, S. Rangarajan, A. Surna, and Q. Zhou. SQL Server column store indexes. In SIGMOD, pages 1177–1184, 2011

    2011

  38. [45]

    Lemire and L

    D. Lemire and L. Boytsov. Decoding billions of integers per second through vectorization. Software: Practice and Experience, 45(1):1–29, 2015

    2015

  39. [46]

    T. Li, M. Butrovich, A. Ngom, W. S. Lim, W. McKinney, and A. Pavlo. Mainlining databases: Supporting fast transactional workloads on universal columnar data file formats. VLDB, 14(4):534–546, 2020

    2020

  40. [47]

    Li and J

    X. Li and J. Li. Angle-optimized text embeddings. arXiv preprint arXiv:2309.12871, 2023

    2023 arXiv

  41. [48]

    Li and J

    Y. Li and J. M. Patel. BitWeaving: fast scans for main memory data processing. In SIGMOD, pages 289–300, 2013

    2013

  42. [49]

    C. Liu. Fast and effective compression for iot systems, 2022

    2022

  43. [50]

    C. Liu, H. Jiang, J. Paparrizos, and A. J. Elmore. Decom- posed bounded floats for fast compression and queries. VLDB, 14(11):2586–2598, 2021

    2021

  44. [51]

    C. Liu, J. Paparrizos, and A. J. Elmore. Adaedge: A dynamic compression selection framework for resource constrained devices. In 2024 IEEE 40th International Conference on Data Engineering (ICDE) , pages 1506– 1519, 2024

    2024

  45. [52]

    C. Liu, M. Umbenhower, H. Jiang, P. Subramaniam, J. Ma, and A. J. Elmore. Mostly order preserving dictio- naries. In ICDE, pages 1214–1225, 2019

    2019

  46. [53]

    Madden, J

    S. Madden, J. Ding, T. Kraska, S. Sudhir, D. Cohen, T. Mattson, and N. Tatbul. Self-organizing data contain- ers. Memory, 1:2

  47. [54]

    McKinney

    W. McKinney. Some comments to Daniel Abadi’s blog about Apache Arrow. wesmckinney.com/blog/ arrow-columnar-abadi , November 2017. Accessed: 2024

    2017

  48. [55]

    Melnik, A

    S. Melnik, A. Gubarev, J. J. Long, G. Romer, S. Shivaku- mar, M. Tolton, and T. Vassilakis. Dremel: interactive analysis of web-scale datasets. VLDB, 3(1-2):330–339, 2010

    2010

  49. [56]

    R. Meng, Y. Liu, S. R. Joty, C. Xiong, Y. Zhou, and S. Yavuz. Sfr-embedding-mistral: Enhance text retrieval with transfer learning. Salesforce AI Research Blog, 2024

    2024

  50. [57]

    R. O. Nambiar and M. Poess. The making of TPC-DS. In VLDB, pages 1049–1058, 2006

    2006

  51. [58]

    Paparrizos, C

    J. Paparrizos, C. Liu, B. Barbarioli, J. Hwang, I. Edian, A. J. Elmore, M. J. Franklin, and S. Krishnan. VergeDB: A database for IoT analytics on edge devices. In CIDR, 2021

    2021

  52. [59]

    Pedreira, O

    P. Pedreira, O. Erling, M. Basmanova, K. Wilfong, L. S. Sakka, K. Pai, W. He, and B. Chattopadhyay. Velox: Meta’s unified execution engine. VLDB, 15(12):3372– 3384, 2022

    2022

  53. [60]

    Pindikura

    R. Pindikura. Gandiva. dremio.com/blog/ announcing-gandiva-initiative-for-apache-arrow . Accessed: 2024. Data Formats in Analytical DBMSs: Performance Trade-offs and Future Directions 23

    2024

  54. [61]

    Pirzadeh, M

    P. Pirzadeh, M. Carey, and T. Westmann. A performance study of big data analytics platforms. In Big Data , pages 2911–2920, 2017

    2017

  55. [62]

    Ramakrishnan, B

    R. Ramakrishnan, B. Sridharan, J. R. Douceur, P. Kasturi, B. Krishnamachari-Sampath, K. Krishnamoorthy, P. Li, M. Manu, S. Michaylov, R. Ramos, N. Sharman, Z. Xu, Y. Barakat, C. Douglas, R. Draves, S. S. Naidu, S. Shastry, A. Sikaria, S. Sun, and R. Venkatesan. Azure data lake...

    2017

  56. [63]

    S. A. Rodriguez, J. Chackrabroty, A. Chu, I. Jimenez, J. LeFevre, C. Maltzahn, and A. Uta. Zero-cost, Arrow- enabled data interface for Apache Spark. In Big Data , pages 2400–2405, 2021

    2021

  57. [64]

    M. A. Roth and S. J. Van Horn. Database compression. SIGMOD, 22(3):31–39, 1993

    1993

  58. [65]

    J. Shi. Column partition and permutation for run length encoding in columnar databases. In SIGMOD, pages 2873–2874, 2020

    2020

  59. [66]

    Trivedi, P

    A. Trivedi, P. Stuedi, J. Pfefferle, A. Schuepbach, and B. Metzler. Albis: High-performance file format for big data systems. In USENIX, page 615–629, 2018

    2018

  60. [67]

    D. Vohra. Apache Avro. In Practical Hadoop Ecosystem, pages 303–323. 2016

    2016

  61. [68]

    Z. Wang, K. Kara, H. Zhang, G. Alonso, O. Mutlu, and C. Zhang. Accelerating generalized linear models with ML Weaving: A one-size-fits-all system for any-precision learning. VLDB, 12(7):807–821, 2019

    2019

  62. [69]

    Y. Yang, M. Youill, M. Woicik, Y. Liu, X. Yu, M. Serafini, A. Aboulnaga, and M. Stonebraker. FlexPushdownDB: Hybrid pushdown and caching in a cloud DBMS. VLDB, 14(11):2101–2113, 2021

    2021

  63. [70]

    Z. Yang, B. Chandramouli, C. Wang, J. Gehrke, Y. Li, U. F. Minhas, P.-˚A. Larson, D. Kossmann, and R. Acharya. Qd-tree: Learning data layouts for big data analytics. In SIGMOD, pages 193–208, 2020

    2020

  64. [72]

    Zhang, W

    F. Zhang, W. Wan, C. Zhang, J. Zhai, Y. Chai, H. Li, and X. Du. CompressDB: Enabling efficient compressed data direct processing for various databases. In SIGMOD, pages 1655–1669, 2022

    2022

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.