{"id":"3293a302-c95d-4a7a-84f1-86416c003409","arxiv_id":"2411.14331","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A systematic benchmark shows Arrow, Parquet, and ORC each excel in different dimensions and all compress ML embeddings poorly, motivating co-designed formats.","lead":"This paper benchmarks Apache Arrow/Feather, Parquet, and ORC as candidate native formats for analytical databases, measuring compression, transcoding, and query performance. It finds each format wins on different axes and that none handles machine-learning embedding vectors well, pointing toward co-designed in-memory and on-disk formats.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Version mismatch in the software stack may invalidate cross-format rankings; a unified re-test is needed before the trade-off conclusions can be trusted.","rationale":"The reader's weakest assumption was that the benchmarked versions are representative of current production performance. My concern sharpens this: the listed versions are internally inconsistent (Arrow 5.0.0 vs PyArrow 17.0.0), which directly threatens the validity of any cross-format comparison, not just its generalizability. This is the most load-bearing issue because the central claim is an empirical ranking, and a ranking produced by mismatched and outdated software cannot support the claimed trade-offs. However, the concern is addressable by re-running with a unified current stack, and the qualitative conclusions are plausible and consistent with prior work, so a conditional verdict remains appropriate. I do not see a stronger objection that would warrant rejection or a verdict change from the reader's conditional assessment.","tokens_in":26053,"tokens_out":3730,"duration_ms":35789,"concrete_test":"Re-run the Section 7 TPC-DS subexpression benchmarks (Table 7 queries Q1–Q5) and the Section 5.1 compression experiments on CodecDB/BI/JOB using a single, current software stack: Arrow C++/PyArrow 17.0.0 or later, ORC 2.x, and Parquet Java 1.14+ or the Arrow C++ Parquet reader, all with default settings. Execute multiple trials with warmup, report mean and standard deviation, and check whether ORC still beats Parquet on Q1–Q5 and whether Parquet still yields the best compression ratios. If either ranking changes materially, the paper's central trade-off conclusion needs substantial revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that Arrow, Parquet, and ORC have workload-dependent trade-offs, with ORC best for query subexpressions, Parquet best for compression, and Arrow Feather best for serialization. This rests on the measurements in Sections 5–7. The Setup (Section 4) lists 'Apache Arrow 5.0.0' and 'PyArrow 17.0.0' as separate tested components. These are not consistent: PyArrow 17.0.0 is the Python binding for Arrow C++ 17.0.0 (2024), while Arrow 5.0.0 is from 2021. Different sections use different Arrow versions (the C++ library for writing tables, PyArrow for embedding datasets), so cross-experiment comparability is already broken. More broadly, Parquet Java 1.9.0 (2016) and ORC 1.7.2 (2021) are far older than current releases, and their default encodings and APIs have changed. If any ranking—for instance, ORC's 2–3× faster loading than Parquet in Section 6, or Parquet's compression advantage in Section 5—is an artifact of old default behaviors, the paper's workload-dependent trade-off conclusion is unsupported. The lack of error bars or standard deviations in all reported figures makes it impossible to assess whether the observed gaps are stable or noise, and no code or artifact is provided to check this internally.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates Apache Arrow/Feather, Parquet, and ORC as candidate native formats for analytical DBMSs. It organizes the comparison around compression ratio, transcoding throughput, projection and predicate data access, TPC-DS leaf subexpressions, and advanced optimizations such as direct querying, vectorization, and data skipping. The central conclusions are that Parquet offers the best compression, Arrow Feather the fastest (de)serialization, ORC the best query-subexpression performance, and that none of the formats is adequate for machine-learning embedding workloads, motivating co-design of in-memory and on-disk representations.","tokens_in":26371,"tokens_out":6389,"duration_ms":61816,"significance":"If the empirical findings hold, the paper is a useful reference for system builders choosing or designing columnar formats, and its embedding-workload results point to a real gap in current formats. The paper's strengths are the breadth of datasets (TPC-DS, JOB, Public BI, CodecDB, Hugging Face embeddings), the systematic dimension-by-dimension structure, and the concrete prototype variants (Arrow direct query and chunk-level skipping, in-memory Parquet with SIMD) that demonstrate possible improvements. However, the empirical basis needs to be strengthened: the benchmarked software versions are old and internally inconsistent, no error bars or repetition counts are reported, no artifact is provided, and part of the comparison conflates format properties with particular library APIs. The 'first detailed evaluation' claim also needs qualification given reference [71].","major_comments":[{"comment":"The software stack is internally inconsistent and outdated. The paper lists Apache Arrow 5.0.0, ORC 1.7.2, Parquet Java 1.9.0, and PyArrow 17.0.0 in the same setup. PyArrow 17.0.0 bundles Arrow C++ 17.0.0, not Arrow 5.0.0, so experiments that use PyArrow (the embedding datasets) and experiments that use Arrow C++ 5.0.0 (the relational tables) cannot be directly compared. Moreover, Parquet Java 1.9.0 (2016) and ORC 1.7.2 (2021) are far older than current releases, and Table 3 itself notes that Parquet's default encoding differed between the legacy Java API and the latest C++ API. Because default encodings and compression behavior determine many of the rankings in Sections 5-7, the measured results may be artifacts of old defaults. Please re-run the core experiments on current, consistently versioned libraries, or clearly explain why the old versions are representative.","section":"Section 4, Setup"},{"comment":"No error bars, standard deviations, or repetition counts are reported. The paper states in Section 4 that numbers are reported with cold caches by default, but not how many runs were averaged. Given that several headline effects are 2-4x differences (e.g., ORC's loading advantage in Section 6, Parquet's compression advantage in Section 5.1), single-run measurements cannot establish stability. Table 6 has some AVG/STD entries, but the figures do not carry variance information. Please add repetitions, variance measures, and ideally a reproducibility artifact; without these, the quantitative rankings are not independently checkable.","section":"Sections 5-7, all figures"},{"comment":"The phrase 'first detailed, empirical evaluation' is contradicted by the paper's own reference [71], an arXiv empirical evaluation of columnar storage formats. The related-work section mentions [71] only as part of 'robust discussion' and does not explain what is different. Please qualify the novelty claim and explicitly state how the present study differs from [71] (e.g., DBMS-subsumption framing, embedding workloads, advanced-optimization experiments).","section":"Abstract and Section 1"},{"comment":"Several conclusions about 'formats' are actually conclusions about specific library APIs and implementations. For example, Section 5.2.1 attributes ORC's slow write time to 'better Parquet support in Arrow; both projects share the same codebase and data structures,' and Section 6.1.1 explains Arrow's poor projection performance through its bulk-loading API while praising ORC's column-level API. Similarly, Sections 6.2 and 7 compare Parquet's streaming API against ORC's and Arrow's bulk in-memory representations. These are implementation differences rather than properties of the file formats themselves. The paper should either control for this confound or consistently phrase the findings as 'with the APIs tested here' rather than as format-inherent trade-offs.","section":"Sections 5.2.1, 6.1.1, 6.2, and 7"},{"comment":"There is an inconsistency regarding ORC and LZ4. Section 5.2.1 states that LZ4 and Snappy bars are omitted for ORC because the Apache Arrow C++ library has limited compression support for ORC, yet Figures 10, 14, and 15 in Sections 5.1.2-5.1.4 report ORC LZ4 compression ratios. If the ORC LZ4 numbers come from a different code path or version, the cross-experiment comparability is broken; if they are from the same code path, the omission in Section 5.2.1 is unexplained. Please clarify which ORC implementation is used for each figure and make the treatment of LZ4 consistent.","section":"Sections 5.1.2-5.1.4 vs. 5.2.1"}],"minor_comments":[{"comment":"The paragraph beginning 'For floats, as we can see from Figure 8b...' and the following paragraph are nearly verbatim repeats of the earlier discussion after Figures 5 and 8; one copy should be removed or merged.","section":"Section 5.1.1"},{"comment":"The text says 'LL VM-based backend' and 'LL VM compiler passes'; this should be 'LLVM' throughout.","section":"Section 8.1.2"},{"comment":"The x-axis label 'Selectivty' should be 'Selectivity'. Several figure captions also contain leftover placeholder text such as 'Figure 5 legend', 'Legend removed', and 'Needs additional data points' (e.g., Figures 10, 22, 25, 29); these should be cleaned before final submission.","section":"Figure 28a"},{"comment":"The embedding dataset description does not list the 38 datasets, their download versions, or the exact embedding-generation procedure, which limits reproducibility; adding a table or a public artifact link would help.","section":"Section 5.1.4"},{"comment":"The statement 'we report numbers when the system caches are cold by default' is not precise: it should define how cold caches are produced (e.g., dropping OS caches, number of trials, warm-up runs). The warm/cold distinction in Section 7 is also only reported for Arrow and ORC, with the caption saying Parquet changes were negligible; please state whether Parquet was measured in both configurations.","section":"Section 4, Setup"},{"comment":"The claim that the paper evaluates 'the ability for each format to take advantage of recent trends such as vectorization, query compilation, and direct querying' is overstated: Gandiva compilation is tested on only a small set of expressions, and direct querying is implemented only for custom Arrow and Parquet prototypes. Consider narrowing the wording to match the experiments.","section":"Section 1, Contributions"}],"recommendation":"major_revision","confidential_remarks":"The paper's Section 8 relies heavily on the first author's prior systems (CodecDB [42], SBoost [40], and BUFF [50]) for the 'advanced optimizations' experiments. This is not circular, but the self-citation pattern should be watched, and the editor may want the final version to clarify which components are newly implemented for this paper versus reused. The version inconsistency between Arrow 5.0.0 and PyArrow 17.0.0 and the lack of any artifact are the main barriers to trust in the quantitative rankings; a major revision that re-runs on current versions with reproducibility information would substantially increase the value of this study."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look if you are choosing between Arrow, Parquet, and ORC as an internal DBMS format. The paper does not overturn the received wisdom, but it gives a systematic feature matrix and a broad set of measurements, including new compression and transcoding results on 38 embedding datasets and prototype speedups for direct querying and SIMD over in-memory Parquet. Those prototype results are the most original part: they show the order-of-magnitude potential of keeping Parquet in memory and pushing predicates into the encoded domain. The embedding finding, that none of these formats handles vector workloads well and NumPy beats them for write efficiency, is a genuinely useful data point for the ML systems crowd.\n\nThe trade-off conclusions (ORC best for query subexpressions, Parquet best for compression, Arrow Feather best for serialization) are qualitatively consistent with Abadi's 2017 analysis and with Zeng et al. 2023, which the paper itself cites. So the \"first detailed evaluation\" claim in the introduction is overcrediting itself. That is a soft spot, not a fatal one: the contribution is the breadth plus the new embedding and prototype experiments.\n\nThe bigger issue is the software stack. Section 4 lists Apache Arrow 5.0.0, ORC 1.7.2, Parquet Java 1.9.0, and PyArrow 17.0.0. Arrow 5.0.0 is from 2021; Parquet Java 1.9.0 is from 2016. Those are not representative of current production behavior, and the paper mixes Arrow C++ for some experiments and PyArrow for the embedding datasets, so cross-experiment comparability is already questionable. No error bars and no released code make the specific numbers in Sections 5-7 hard to verify. The qualitative rankings may well survive a re-run, but the burden is on the authors to show that. To their credit, the paper explicitly notes that different results could be obtained with dataset-specific tuning; that honesty helps, but it does not fix the version problem.\n\nThis is a paper for DBMS architects and query engine developers. It deserves a serious referee: the questions are right, the embedding gap is real, and the prototype section points in a productive direction. I would send it to review but ask for a re-run on current versions, error bars or repeated runs, and an artifact before treating the numbers as reliable.","headline":"Useful benchmark with real new measurements on embeddings and in-memory Parquet, but the stale and mismatched software versions plus missing artifacts mean the precise numbers need a re-test before being trusted.","tokens_in":26864,"tokens_out":2117,"would_cite":true,"duration_ms":20695,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims no single open columnar format fits all analytical DBMS workloads; ORC wins queries, Parquet wins compression, Arrow Feather wins serialization, and none fits ML embeddings.","keywords":["columnar storage formats","Apache Arrow","Parquet","ORC","OLAP query performance","data skipping","compression and encoding","vector embeddings"],"falsifier":"Re-run the same benchmark battery with current stable releases of Arrow, Parquet, and ORC, and with per-format parameter tuning; if the ordinal rankings on the five evaluation dimensions change, or if any format achieves strong compression and fast transcoding on embedding datasets, the paper's core trade-off picture would need revision.","tokens_in":25898,"feed_emoji":"📊","tokens_out":6007,"duration_ms":52878,"temperature":0.7,"pith_summary":"This paper sets out to determine whether any of the three dominant open columnar formats—Apache Arrow (Feather), Parquet, and ORC—can serve as a native format inside an analytical database management system. Through micro-benchmarks and TPC-DS query subexpressions, it finds no winner on all dimensions: ORC gives the best query performance thanks to fine-grained skipping and a dedicated in-memory representation, Parquet gives the best compression, and Arrow Feather gives the fastest serialization. It also finds that for popular machine-learning workloads on vector embeddings, all three formats compress poorly and transcode inefficiently. The paper argues that rather than picking one format, designers should co-design a unified in-memory and on-disk representation, and shows that pushing query evaluation into encoded data can be far faster than decoding first.","feed_headline":"No single data format fits every analytical DBMS workload","feed_subtitle":"A benchmark maps each format to what it does best, from serialization speed to query speed.","key_machinery":"The generic columnar format architecture that the paper abstracts from Arrow, Parquet, and ORC: tables are split into row batches, each column into chunked columns, with metadata such as zone maps, min/max values, bloom filters, and offsets stored in footers or indexes. The benchmark battery is organized around five dimensions, namely compression ratio, transcoding throughput, data access (projection and predicate evaluation), end-to-end subexpressions, and advanced features such as direct querying and SIMD. The augmented implementations, including in-memory Parquet with direct query and SIMD, and Arrow with chunk-level skipping, serve as probes to test whether pushing computation into the encoded domain can overcome the default formats' compromises.","core_discovery":"Evaluated across compression ratio, transcoding throughput, projection, predicate filtering, and end-to-end select-project subexpressions, the three formats form a trade-off triangle rather than a hierarchy. ORC tends to win query-heavy subexpressions because its smaller row batches enable more skipping and its row-level index plus dedicated in-memory column vector batch make loading efficient; Parquet tends to win compression because of its dictionary, RLE, and bit-packing stack; Arrow Feather tends to win serialization and deserialization because it skips encoding. Adding direct querying in the encoded domain to an in-memory Parquet variant yields large speedups. For embedding workloads such as retrieval-augmented generation and k-nearest-neighbor search, none of the formats compresses or serializes well, and the authors suggest precision-controlled encoding as a direction.","pith_inferences":["Since Arrow's loading overhead in this study came largely from per-row-batch lock acquisition, implementation-level fixes or newer Arrow releases could narrow the gap with ORC and shift the trade-off surface.","The finding that ORC wins through smaller batch granularity suggests that workload-aware partitioning and layout learning could amplify or invert these results when applied to Parquet or Arrow.","The embedding compression failure points to a testable design: applying precision-controlled float encoding to nested vector columns could recover the missing gains, a direction the paper points to without implementing.","The paper's default cold-cache protocol likely understates Parquet's streaming advantages; a warm-cache, pipelined setting could change some ordinal rankings across the five dimensions."],"forward_implications":["DBMS format choice should be workload-dependent: ORC for predicate-heavy subexpressions, Parquet for compression, Arrow Feather for serialization speed.","Fine-grained data skipping pays off at low selectivity, while bulk-loading APIs win at high selectivity, so a format should support both access patterns.","Pushing query operators into encoded data with SIMD can produce large speedups, up to roughly 100x over the Parquet baseline in this study.","None of the three formats is adequate for vector-embedding workloads, so compression and transcoding for embeddings need new encoding approaches.","A unified, co-designed in-memory and on-disk representation that avoids the current convert-to-Arrow practice is a promising direction for future formats."],"supporting_citations":[{"why":"Defines the Apache Arrow in-memory columnar format and IPC protocol whose serialization and loading performance is benchmarked throughout.","marker":"[19]"},{"why":"Defines the ORC storage format, its row-level index, bloom filters, and dedicated in-memory representation that drive ORC's query results.","marker":"[22]"},{"why":"Defines the Parquet storage format, its encodings, page structure, and zone maps that drive Parquet's compression and skipping behavior.","marker":"[23]"},{"why":"Supplies the lazy in-memory Parquet representation and direct-querying technique used in the augmented Parquet experiments.","marker":"[42]"},{"why":"Supplies the SIMD-accelerated filtering over encoded columnar data used in the P-IM+D+SIMD variant.","marker":"[40]"},{"why":"Supplies the Gandiva LLVM-based execution backend for Arrow that the paper benchmarks in the advanced-optimization section.","marker":"[60]"},{"why":"Supplies the Dremel nested encoding scheme that underlies Parquet's handling of nested vector columns.","marker":"[55]"},{"why":"Provides the earlier public debate about Arrow versus Parquet and ORC that this evaluation updates and tests empirically.","marker":"[14]"},{"why":"Provides a prior empirical evaluation of columnar formats that this paper contrasts with by focusing on native DBMS subsumption.","marker":"[71]"},{"why":"Supplies the community-curated vector-embedding and RAG datasets used in the compression and transcoding benchmarks.","marker":"[2]"}],"fun_headline_variants":["Arrow, Parquet, ORC: each excels but none dominates","No universal data format for analytical DBMS workloads","Analytical DBMS formats: trade-offs, not a winner","For ML workloads, Arrow, Parquet, and ORC all fall short","Pick your analytical data format by workload, not habit"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The measured trade-off rankings rest on the specific software versions tested (Arrow 5.0.0, ORC 1.7.2, Parquet Java 1.9.0, PyArrow 17.0.0) being representative of each format's current production performance and default settings.","fun_headline_variants_meta":{"raw":{"variants":["Arrow, Parquet, ORC: each excels but none dominates","No universal data format for analytical DBMS workloads","Analytical DBMS formats: trade-offs, not a winner","For ML workloads, Arrow, Parquet, and ORC all fall short","Pick your analytical data format by workload, not habit"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000537,"raw_usage":{"total_tokens":2529,"prompt_tokens":848,"completion_tokens":1681,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":464,"completion_tokens_details":{"reasoning_tokens":1596}},"tokens_in":464,"tokens_out":1681,"duration_ms":9990,"temperature":1.0,"reasoning_tokens":1596,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:16:59.781548+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same benchmark battery with current stable releases of Arrow, Parquet, and ORC, and with per-format parameter tuning; if the ordinal rankings on the five evaluation dimensions change, or if any format achieves strong compression and fast transcoding on embedding datasets, the paper's core trade-off picture would need revision.","supporting_citations":[{"cited_title":"Apache Arrow","cited_arxiv_id":null,"evidence_quote":"Defines the Apache Arrow in-memory columnar format and IPC protocol whose serialization and loading performance is benchmarked throughout."},{"cited_title":"Apache ORC","cited_arxiv_id":null,"evidence_quote":"Defines the ORC storage format, its row-level index, bloom filters, and dedicated in-memory representation that drive ORC's query results."},{"cited_title":"Apache Parquet","cited_arxiv_id":null,"evidence_quote":"Defines the Parquet storage format, its encodings, page structure, and zone maps that drive Parquet's compression and skipping behavior."},{"cited_title":"Jiang, C","cited_arxiv_id":null,"evidence_quote":"Supplies the lazy in-memory Parquet representation and direct-querying technique used in the augmented Parquet experiments."},{"cited_title":"Jiang and A","cited_arxiv_id":null,"evidence_quote":"Supplies the SIMD-accelerated filtering over encoded columnar data used in the P-IM+D+SIMD variant."},{"cited_title":"Pindikura","cited_arxiv_id":null,"evidence_quote":"Supplies the Gandiva LLVM-based execution backend for Arrow that the paper benchmarks in the advanced-optimization section."},{"cited_title":"Melnik, A","cited_arxiv_id":null,"evidence_quote":"Supplies the Dremel nested encoding scheme that underlies Parquet's handling of nested vector columns."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the earlier public debate about Arrow versus Parquet and ORC that this evaluation updates and tests empirically."},{"cited_title":"An Empirical Evaluation of Columnar Storage Formats","cited_arxiv_id":"2304.05028","evidence_quote":"Provides a prior empirical evaluation of columnar formats that this paper contrasts with by focusing on native DBMS subsumption."},{"cited_title":"huggingface.co/docs/datasets/index","cited_arxiv_id":null,"evidence_quote":"Supplies the community-curated vector-embedding and RAG datasets used in the compression and transcoding benchmarks."}],"review_version":1}