{"id":"d3c040ae-f157-45bf-8bab-15bced532b3a","arxiv_id":"2502.05311","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"ParquetDB, a lightweight Python database built on Parquet and PyArrow, outperforms SQLite and MongoDB on large-scale reads and batch updates while supporting complex nested data.","lead":"ParquetDB is a new Python library that stores and queries data in Apache Parquet files, and its authors report faster large-scale reads and updates than SQLite or MongoDB while also handling nested records. The paper benchmarks the library on synthetic data and on the Alexandria materials database, which matters for scientists and engineers choosing a storage layer for data-heavy Python workflows.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The needle-in-haystack benchmark in Sec 5.4 may use an out-of-range sentinel value, letting Parquet row-group min/max stats skip nearly all groups; with a real in-range unique key, ParquetDB likely degrades to full scans and loses to indexed SQLite/MongoDB.","rationale":"The reader's conditional verdict is already keyed to benchmark representativeness, which is the right general category; my concern is a specific, currently unstated sub-case of that: the needle benchmark's sentinel value may be out-of-range, which would make predicate pushdown trivially effective and would explain the surprisingly flat scaling in Figure 7. This is more precise than the reader's cited issues (Python-list input and omitted DuckDB) and is checkable from the public repository. I agree that the paper's headline overreaches, but I do not see grounds to move the verdict from CONDITIONAL to REJECT, because the core artifact is plausible, the read/full-scan benchmarks are directionally supportive, and the Alexandria use case is a legitimate demonstration even without cross-system comparison. I would, however, make acceptance conditional on an in-range needle re-run. I also weighed the self-admitted durability limitation in Sec 4.8: \"Quasi-Durability (ACID)\" is a real limitation, but it is not the most load-bearing issue for the abstract's performance claim, so I did not make it the headline. The code is public, so the proposed test can settle the concern without speculation.","tokens_in":37129,"tokens_out":8617,"duration_ms":94533,"concrete_test":"Re-run the Sec 5.4 benchmark on the same 100-integer-column generator, querying (a) a randomly chosen existing row value from the data range and (b) the out-of-range sentinel value used in the paper (if recoverable from the repository), at N = 10^2 through 10^6. Instrument the PyArrow dataset scan (e.g., via FragmentScanOptions or by logging row groups visited) to record bytes and row groups actually read. If (a) scales with N or is slower than indexed SQLite/MongoDB while (b) is flat, the needle-in-haystack result is an artifact of the sentinel, and the no-index query claim must be re-scoped.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.4 (\"Needle-in-a-Haystack Benchmark\") measures the time to retrieve \"an arbitrary unique value inserted into a given column\" and reports roughly constant ParquetDB query time as the dataset grows, attributing the result to predicate pushdown on Parquet statistics. The load-bearing problem is that the generated columns are random integers in [0, 1,000,000]; if the inserted unique value lies outside that interval (e.g., -1 or 1,000,001), every row group except the one holding it has a min/max interval excluding the key, so row-group statistics skip all other data and the query is effectively O(1) by construction. That is not the typical unique-ID lookup the text invokes (\"searching for unique identifiers\"): real keys fall inside the populated domain, and for random, unsorted data the row-group min/max ranges cover the whole domain, so no row groups can be skipped. Under that representative condition, ParquetDB should scan all fragments and query time should scale with N, likely falling far behind indexed SQLite and MongoDB. This directly threatens the central \"outperforms\" claim and the advertised \"reduced dependency on indexing through predicate pushdown\" (Secs 1 and 4.1). The paper does not state the inserted value's range or the row-group layout, so the constant-time curve in Figure 7 is currently uninterpretable as evidence for the no-index query advantage.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents ParquetDB, a Python database framework built on Apache Parquet and PyArrow, offering CRUD operations, nested data support, schema evolution, and predicate pushdown. It benchmarks ParquetDB against SQLite and MongoDB on synthetic integer datasets (100 columns, up to one million rows) and demonstrates a real-world application on the Alexandria 3D Materials Database. The central claims are that ParquetDB outperforms SQLite and MongoDB for large read and bulk update workloads, and that it achieves point-query performance comparable to indexed systems without maintaining explicit indexes. The paper also provides a detailed description of the API, data flow, and Parquet file format background.","tokens_in":37396,"tokens_out":4685,"duration_ms":46277,"significance":"If the performance claims are substantiated, ParquetDB offers a lightweight, open-source alternative for managing large scientific datasets, with native support for nested data and no index-maintenance overhead. The provided GitHub repository, the detailed API documentation, and the Alexandria application are valuable assets that enable reproducibility and further evaluation. However, the headline outperformance claim currently rests on benchmark methodology that is not adequately described or verified, particularly around the needle-in-a-haystack experiment, so the significance cannot be fully assessed until these issues are resolved.","major_comments":[{"comment":"The needle-in-a-haystack benchmark does not state the actual value of the inserted unique key relative to the data domain (random integers in [0, 1,000,000]). If the inserted value lies outside this range, every row group except the one containing it has min/max statistics that exclude the key, so ParquetDB's constant query time is an artifact of row-group skipping rather than a general no-index point-lookup capability. To support the claim of 'performance comparable to indexed systems', the benchmark should specify the key value, the row-group layout (rows per group and number of groups), and repeat the experiment with an in-range key drawn from the populated domain; otherwise the curve in Fig. 7 is uninterpretable as evidence for predicate-pushdown-based constant-time lookup.","section":"Sec. 5.4"},{"comment":"All reported benchmark times appear to come from a single run per configuration; no error bars, standard deviations, or repetition counts are provided. The differences in the figures (e.g., ParquetDB read time crossing SQLite around a few thousand rows, and the non-monotonic update curves) could be within run-to-run noise. The paper should report mean and standard deviation over at least 5–10 runs, describe cache/warm-up handling, and state whether timings include process startup. Without this, the headline 'outperforms' claim is not statistically supported.","section":"Sec. 5.1–5.4"},{"comment":"The text states that ParquetDB 'ultimately becomes the third most efficient system' and then, in the same section, that it 'achieves query performance comparable to the indexed versions of SQLite and MongoDB.' These two statements are inconsistent: being third of five implies slower than both indexed systems, not comparable. The authors should reconcile the wording and be precise about the ranking (e.g., third after the two indexed systems, ahead of both non-indexed ones) and about what 'comparable' means.","section":"Sec. 5.4"},{"comment":"The cross-system comparison uses Python lists as the input format for all systems, yet Sec. 5.3 demonstrates that Python lists are the slowest input format for ParquetDB (Fig. 6). While the authors justify this as a 'baseline under suboptimal conditions', the comparison is not symmetric: for SQLite and MongoDB, the Python iteration is the native driver interface, whereas ParquetDB must convert the list to a PyArrow table. This choice, combined with the abstract's qualifier 'especially when using data formats compatible with PyArrow', makes the headline 'outperforms' workload-specific. The conclusion should be scoped to the evaluated input format, or the benchmark should also include a format that is native to ParquetDB.","section":"Sec. 5.2–5.3"}],"minor_comments":[{"comment":"The record count is inconsistent: the abstract says approximately 4.8 million records, while Sec. 6 states 4.3 million unique material structures and later mentions '4.8 million structures'. Please correct the numbers to a single consistent figure.","section":"Sec. 6 vs Abstract"},{"comment":"The total JSON load time is given as 779.66 seconds at the top of the section and later as 727 seconds; the create time is 238.86 seconds initially and later '246 seconds'. Please make these numbers consistent.","section":"Sec. 6.1"},{"comment":"The sentence 'the timing was taken prior to the executemany operation' is ambiguous; please clarify whether the timer was started before or after the executemany call and what exactly is being measured.","section":"Sec. 5.2"},{"comment":"The 'Encoding Techniques' entry for SQLite is listed as 'None', but SQLite uses internal B-tree storage and record formats; please clarify what 'encoding' means in this context to avoid confusion.","section":"Sec. 3.2.5 / Table 11"},{"comment":"The paper does not mention why DuckDB or other Parquet-native query engines were excluded from the benchmark; a brief justification would help position the contribution against existing systems.","section":"Sec. 5.1"}],"recommendation":"major_revision","confidential_remarks":"The core idea is useful and the Alexandria application is a credible stress test, but the benchmark section needs substantial revision before the performance claims can be taken at face value. The needle-in-a-haystack issue is the most serious: without specifying the query key and row-group layout, the constant-time result is not evidence of a general capability. I would also encourage the authors to consult the data-management community's benchmarks for columnar formats, since the comparison set is minimal and the workload is narrow. If the authors can provide reproducible, repeated-trial benchmarks and clarify the needle experiment, the paper could become publishable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, ParquetDB itself is real and plausibly useful: a clean Python API for CRUD on Parquet files, with schema evolution, nested-structure flattening and rebuilding, and a real-world validation on the Alexandria materials database (using the public GitHub and download script). That is a legitimate contribution for the materials-informatics crowd, and the authors deserve credit for shipping code and for being upfront about limitations like quasi-durability and the cost of list-to-PyArrow conversion.\n\nSecond, the headline claim that ParquetDB 'outperforms' SQLite and MongoDB is not supported by the benchmarks as written. The needle-in-a-haystack section is the big problem. The synthetic data are random integers in [0, 1,000,000]. If the 'arbitrary unique value' inserted for lookup lies outside that range—and the paper never says—then Parquet's row-group min/max stats will skip every row group except one, and the constant query time is an artifact, not a real predicate-pushdown win. Real unique IDs fall inside the populated domain, where row-group stats can't skip anything. The paper's own text also contradicts itself: ParquetDB is called 'the third most efficient system' yet 'comparable' to indexed SQLite and MongoDB. That needs to be resolved.\n\nThere are smaller issues that pile up. No error bars or repeated runs anywhere. Record counts oscillate between 4.3 million and 4.8 million for the same database. The cross-system comparison uses Python lists—the format the paper itself identifies as worst for ParquetDB—while omitting DuckDB, the obvious Parquet-native baseline. None of these are fatal to the library's usefulness, but together they mean the empirical claims need a serious rewrite before I'd trust them.\n\nBottom line: this is a solid tool paper for readers who want a lightweight, file-based scientific database and prefer Python over SQL. The design and the Alexandria case study are worth a serious referee. The benchmark section is not acceptable in its current form; it should require major revision, not rejection. I'd send it out, with a clear request for repeated trials, a fair benchmark protocol, and an explicit statement of the lookup-key range in Section 5.4.","headline":"A genuinely useful Python library for Parquet-backed scientific data, but the benchmark claims, especially the needle-in-a-haystack result, are not yet credible as presented.","tokens_in":37932,"tokens_out":1708,"would_cite":false,"duration_ms":19277,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ParquetDB, a Python database built on Parquet, outperforms SQLite and MongoDB at scale.","keywords":["Apache Parquet","PyArrow","Python database","columnar storage","predicate pushdown","nested data","benchmark","Alexandria 3D database"],"falsifier":"Run the paper's create, read, and update benchmarks on a multi-million-row dataset containing strings and nested structures, feeding each system its natively preferred input format and including a Parquet-native query engine as an additional comparison; if ParquetDB no longer beats or ties SQLite and MongoDB on large reads and bulk updates, the generality claim fails.","tokens_in":36919,"feed_emoji":"🗄️","tokens_out":6409,"duration_ms":57605,"temperature":0.7,"pith_summary":"ParquetDB is a Python database framework that stores records as Apache Parquet files and uses PyArrow for in-memory processing. The paper claims that for large datasets it outperforms SQLite and MongoDB on reads and bulk updates, especially when data is supplied as PyArrow Tables or pandas DataFrames, and that it natively handles nested and complex records that force workarounds in relational and document databases. The authors demonstrate the design on roughly 4.8 million records from the Alexandria 3D Materials Database, showing sub-second queries over millions of rows without maintaining explicit indexes.","feed_headline":"Parquet database outperforms SQLite and MongoDB at scale","feed_subtitle":"A lightweight Python layer on Parquet and PyArrow handles millions of nested records without index maintenance.","key_machinery":"The mechanism that carries the argument is the Parquet row-group and page structure combined with PyArrow's compute and table APIs: queries use predicate pushdown and column projection to read only the needed blocks and columns, and the Parquet footer's per-column statistics let filters skip irrelevant row groups. On top of this, ParquetDB flattens nested dictionaries into dotted-column schemas (such as address.city), automatically generates integer record IDs, and offers a normalization step that rebalances rows across files for consistent performance.","core_discovery":"The central claim is that a lightweight database can be built directly on the Parquet file format rather than on a traditional storage engine, and that this design is faster for large analytic-style workloads than both a relational database (SQLite) and a document database (MongoDB) while offering schema evolution and file portability. In the benchmark, ParquetDB's read time crosses below SQLite and MongoDB once the dataset reaches a few hundred to a thousand rows, and its bulk update time becomes the best near one million rows, despite the overhead of converting Python lists into Arrow tables. The authors attribute this to Parquet's columnar layout, row-group statistics enabling predicate pushdown, and PyArrow's native handling of nested data.","pith_inferences":["If the benchmark were repeated with PyArrow Tables or pandas DataFrames as the input format instead of Python lists, ParquetDB's update and create advantages would likely grow, since the paper itself measures Python lists as the slowest input format.","The absence of in-place updates means ParquetDB is best suited to append-mostly analytic pipelines; workloads dominated by many small, scattered updates would favor a conventional row-store.","The same file-based design could be extended to object storage to give a serverless analytic database with predicate pushdown, but durability and concurrent-writer behavior would need explicit handling.","The flattening approach trades deep nesting for queryability; very deeply nested or heavily repeated sub-structures may require the rebuild_nested_struct path, which is expensive on first access."],"forward_implications":["For read-heavy, large-scale workloads, users can expect ParquetDB to load entire datasets and run filtered queries faster than SQLite or MongoDB without creating or maintaining indexes.","Nested and complex records (lists, dictionaries, arrays) can be stored and queried directly, avoiding the flattening or BLOB workarounds required by SQLite.","Because storage is a directory of Parquet files, transferring or sharing a database means copying files, and schema changes can be applied by adding fields with nulls for existing rows.","Bulk updates that rewrite many rows at once are competitive with indexed SQLite and MongoDB, while small single-record updates are much slower due to rewrite overhead.","The Alexandria 3D database case shows the design can manage roughly 4.8 million nested records, with column reads and filtered queries returning in about 0.05 to 0.1 seconds."],"supporting_citations":[{"why":"Provides the SQLite baseline that ParquetDB is benchmarked against for create, read, and update times.","marker":"[1]"},{"why":"Provides the MongoDB baseline used in the same benchmarks and feature comparison.","marker":"[2]"},{"why":"Defines the Apache Parquet columnar file format that ParquetDB builds its storage layer on.","marker":"[20]"},{"why":"Supplies PyArrow, the computational backbone for table operations, encodings, and predicate pushdown in ParquetDB.","marker":"[46]"},{"why":"Supplies the Alexandria 3D Materials Database used to validate ParquetDB on roughly 4.8 million nested records.","marker":"[47]"}],"fun_headline_variants":["Python + Parquet = faster than SQLite and MongoDB","ParquetDB: Lightweight speed for millions of nested records","Ditch SQLite: Parquet-based DB wins at scale","ParquetDB: Columnar storage without the database overhead","4.8M records? ParquetDB keeps it fast and simple"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark workload — synthetic data with 100 integer columns, bulk operations, and Python lists as the input format — represents real usage closely enough that ParquetDB's measured advantage over SQLite and MongoDB carries over to other data types and workloads.","fun_headline_variants_meta":{"raw":{"variants":["Python + Parquet = faster than SQLite and MongoDB","ParquetDB: Lightweight speed for millions of nested records","Ditch SQLite: Parquet-based DB wins at scale","ParquetDB: Columnar storage without the database overhead","4.8M records? ParquetDB keeps it fast and simple"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000604,"raw_usage":{"total_tokens":2768,"prompt_tokens":848,"completion_tokens":1920,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":464,"completion_tokens_details":{"reasoning_tokens":1834}},"tokens_in":464,"tokens_out":1920,"duration_ms":12625,"temperature":1.0,"reasoning_tokens":1834,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T19:47:41.293480+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the paper's create, read, and update benchmarks on a multi-million-row dataset containing strings and nested structures, feeding each system its natively preferred input format and including a Parquet-native query engine as an additional comparison; if ParquetDB no longer beats or ties SQLite and MongoDB on large reads and bulk updates, the generality claim fails.","supporting_citations":[{"cited_title":"https://www.sqlite.org/about.html","cited_arxiv_id":null,"evidence_quote":"Provides the SQLite baseline that ParquetDB is benchmarked against for create, read, and update times."},{"cited_title":"https://www.mongodb.com/","cited_arxiv_id":null,"evidence_quote":"Provides the MongoDB baseline used in the same benchmarks and feature comparison."},{"cited_title":"https://parquet.apache.org/","cited_arxiv_id":null,"evidence_quote":"Defines the Apache Parquet columnar file format that ParquetDB builds its storage layer on."},{"cited_title":"https://arrow.apache.org/docs/python/index.html","cited_arxiv_id":null,"evidence_quote":"Supplies PyArrow, the computational backbone for table operations, encodings, and predicate pushdown in ParquetDB."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Alexandria 3D Materials Database used to validate ParquetDB on roughly 4.8 million nested records."}],"review_version":1}