Pith. sign in

REVIEW 2 major objections 4 minor 16 references

CrypQ: A Database Benchmark Based on Dynamic, Ever-Evolving Ethereum Data

T0 review · 2 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read CrypQ turns Ethereum's public blockchain into a dynamic database benchmark that exposes real-world query optimizer failures in cardinality estimation and plan selection.

desk verdict CrypQ is a genuinely useful benchmark with real Ethereum data and fine-grained updates, but the moving-window mode has a correctness gap around eth_balance that needs fixing before the paper is fully sound. read the letter →

arxiv 2411.17913 v1 pith:2UVTOK7M submitted 2024-11-26 cs.DB

Vincent Capol , Yuxi Liu , Haibo Xiu , Jun Yang This is my paper · ORCID
classification cs.DB
keywords databasebenchmarkEthereumblockchaindynamicdataqueryoptimizationcardinalityestimationupdateworkloadPostgreSQL
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CrypQ is a database benchmark built from live Ethereum blockchain data, giving researchers a high-volume dataset that changes continuously and reflects the real skews and correlations of an active cryptocurrency market. The paper argues that existing benchmarks either generate synthetic data with fixed distributions or use static real-world snapshots, so they cannot test how cost-based query optimizers cope with data distributions that drift over time. CrypQ provides a schema, extraction scripts, a fine-grained update workload, and ten SQL queries, and the authors show that PostgreSQL's default histograms can have Q-errors above 10 and that plans chosen on one slice of Ethereum data can run over 2x slower on another. If the benchmark is adopted, it would give the query-optimization community a shared, ever-growing testbed for dynamic workloads.

What carries the argument

The load-bearing object is the Ethereum blockchain itself, accessed through Google BigQuery's public dataset: CrypQ slices it into consecutive block ranges that define initial states and generates an upsert file per block to produce a realistic, fine-grained update workload. The schema's central tables (Blocks, Transactions, Addresses, Tokens, Contracts, Token_Transactions, Withdrawals) preserve the ledger's foreign-key structure, and the query suite is designed to exercise join-heavy, skew-sensitive relational queries against this evolving data.

What would settle it

Reconstruct true historical eth_balance values for a CrypQ slice from the full Ethereum transaction history (or an archive node), then compare their distribution and the selectivity of Q1's balance predicate against CrypQ's extrapolated balances; a large divergence would show that the benchmark's balance-based queries test an artificial distribution rather than Ethereum's actual dynamics.

Watch

Extended reading notes

Core claim

The paper introduces CrypQ as a practical benchmark that exploits the public Ethereum ledger as a source of realistic, continually arriving data. CrypQ's schema captures blocks, transactions, addresses, tokens, contracts, token transfers, and withdrawals; its data-preparation pipeline extracts consecutive block slices from Google BigQuery's Ethereum dataset, converts binary identifiers to efficient byte representations, and generates batched upsert files that advance the database state block by block (with optional expiration to hold size constant). The query suite of ten queries includes joins, aggregations, subqueries, lateral joins, window functions, and recursive CTEs aimed at questions about large transfers, token supply changes, and contract usage. In the included experiments, PostgreSQL's cardinality estimates on CrypQ subqueries exceed Q-error 10 for multi-way joins in three of four slices, and a plan optimized for a low-activity slice runs up to 2.35x slower (estimated) and 1.81x slower (measured) on a high-activity slice, demonstrating that CrypQ can expose optimizer failures in realistic dynamic settings.

Load-bearing premise

The benchmark's realism rests on the assumption that extrapolating each address's end-of-slice balance yields a faithful picture of historical balances, even though the source data does not track balance history.

Editorial extensions

If this is right

  • CrypQ gives query-optimizer researchers a public benchmark where summary statistics must be refreshed or adapted as data evolves; the included Scenario 1 shows that stale histograms degrade Q-error over time.
  • The benchmark can distinguish scenarios where cardinality errors matter for plan selection from those where cached plans remain acceptable, since Scenario 1 produced no plan change while Scenario 2 produced regressions.
  • The four slices S1-S4 provide a ready-made cross-distribution testbed: same schema and queries, widely different data characteristics, enabling evaluation of estimators and optimizers across realistic distribution shifts.
  • CrypQ's update workload can be replayed at real-time speed by scheduling upsert files according to block timestamps, enabling streaming or incremental-maintenance research.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implicit consequence is that CrypQ could serve as a shared training and evaluation ground for learned cardinality estimators, since the Ethereum ledger provides an unbounded sequence of future blocks for temporal cross-validation.
  • The balance extrapolation step suggests a natural robustness test: CrypQ users could compare query results against an archive-node reconstruction of historical balances to measure how much the benchmark's selectivity depends on this approximation.
  • Because Ethereum's data grows daily, CrypQ could be extended into a continuously updated benchmark with versioned snapshots, letting the community track how optimizer performance changes as the ledger evolves over months or years.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper introduces CrypQ, a relational database benchmark built from public Ethereum blockchain data exported from Google BigQuery. It defines a schema (Blocks, Transactions, Addresses, Token_Transactions, Contracts, etc.), scripts for extracting slices and generating update workloads, and a suite of 10 SQL queries. The benchmark is intended to provide a real-world, high-volume, ever-evolving dataset with realistic skew and correlations. As an example, the authors run experiments on PostgreSQL in two scenarios: Scenario 1 applies a moving-window update workload (W1-W11), and Scenario 2 uses four temporally distant slices. They measure cardinality estimation Q-errors and query-plan regressions, showing that CrypQ can expose optimizer inaccuracies and plan-selection subtleties.

Significance. If the update workload is semantically sound, CrypQ addresses a real gap: most relational benchmarks are static or synthetic, while CrypQ provides public, reproducible artifacts and a genuinely dynamic real-world dataset. The paper is careful about the illustrative nature of the experiments and credits the plan-injection methodology to PARQO. The benchmark could be a reusable resource for future work on query optimization under data drift. However, the correctness of the moving-slice update semantics is load-bearing for the central claim that CrypQ offers realistic, ever-evolving data, and the current description has a gap that must be closed before the Scenario 1 results can be taken as demonstrating realistic dynamics rather than artifacts of the benchmark's own inconsistency.

major comments (2)
  1. [Section 3, Data Preparation and Update Workload] The optional moving-slice mode is internally inconsistent as described. The paper states that expire.sql first deletes data for the oldest blocks before each upsert batch, and that upserts 'will also update balances in Addresses as a result of the new transactions.' No compensating adjustment to eth_balance is described for the deleted (expired) transactions. Because eth_balance is a derived value that should equal the net effect of all transactions in the currently active slice, deleting transactions without reversing their balance contributions leaves Addresses with balances that do not correspond to any real Ethereum block range. This directly affects Scenario 1 (Section 4): states W2-W11 are produced exactly by this expire+upsert procedure, and the Q1 subqueries used in Figures 2 and 4 join Addresses with predicates on eth_balance. The reported Q-errors and plan regressions may therefore be driven by the benchmark's own balance inconsistency rather than by realistic Ethereum dynamics. Please specify how expire.sql or a companion step recomputes eth_balance (or otherwise maintains the invariant), or revise the benchmark to avoid the moving-slice claim until this semantics is corrected.
  2. [Section 3, Data Preparation; Section 4, Scenario 1] The extrapolation of eth_balance is not tied to a specific block height in the dynamic scenario. The paper says that because BigQuery does not track balance history, the authors 'extrapolate the eth_balance values of addresses at the end of the extracted slice.' For Scenario 1, the initial state W1 is the first 1000 blocks of a 2000-block extract, and the script gen_updates.sh is applied to create the subsequent states. If the Addresses table for W1 is loaded with balances extrapolated from the end of the full 2000-block slice, those balances are inconsistent with the transactions active in W1, and subsequently applying only the new transactions will not correct them; even W11 would then not match the stated final 1000-block window. The paper should state explicitly for which block height eth_balance is computed when generating the initial load file and each upsert batch, and should validate the extrapolation against a known historical balance (e.g., from an archive node). This matters because the filter on a.eth_balance is central to Q1 and its subqueries in both example scenarios.
minor comments (4)
  1. [Section 4, Example Use 2] There is a typo: 'cardinaility' should be 'cardinality'.
  2. [Section 4, Example Use 2] There is a typo: 'subtlties' should be 'subtleties'.
  3. [Figures 2 and 3] The Q-error labels are placed directly above the plotted lines without a separate legend; a legend or a clearer annotation style would help readers distinguish the 'refreshed estimates' and 'initial estimates' series from their Q-error labels.
  4. [Table 1] The checkmark formatting in Table 1 is inconsistent across rows; using 'Yes'/'No'/'Partial' for all cells would improve readability and avoid ambiguity about the 'periodic refreshes only' and 'monthly updates only' footnotes.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: CrypQ is built from external Ethereum data; the only self-citation is methodological and not load-bearing.

full rationale

The paper's central deliverable is a benchmark assembled from external Ethereum data via Google BigQuery. The update workload and query suite are constructed, not fitted to the experimental outcomes; cardinality estimates and latencies are measured from PostgreSQL, an external system. The only self-citation is [16] (PARQO), used instrumentally to inject cardinality estimates and query plans into the optimizer, and the paper does not rely on PARQO to establish that CrypQ is realistic or challenging. The data-preparation extrapolation of eth_balance and the moving-slice expire procedure are correctness risks that could affect whether Scenario 1 updates represent valid Ethereum states, but those are potential artifacts of the workload construction, not a case where a claimed prediction is equivalent by construction to an input. No fitted parameter is renamed as a result, no uniqueness theorem is imported from the authors, and no ansatz is smuggled via self-citation. Accordingly, no circular step is identifiable; the score of 1 reflects only a minor, non-load-bearing self-citation.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

CrypQ introduces no fitted parameters or invented entities. Its validity rests on the fidelity of external Ethereum data and on the representativeness of the hand-authored query set.

assumptions (3)
  • domain assumption Google BigQuery's public Ethereum dataset accurately represents the Ethereum blockchain and is suitable for deriving benchmark data.
    Section 3 states data is obtained from BigQuery, which 'directly obtains data from a node in the Ethereum network'. The benchmark's realism depends on this external source's fidelity.
  • domain assumption The extrapolation of eth_balance and setting block_hash to NULL for out-of-slice referenced rows preserves the realistic data characteristics CrypQ claims.
    Data Preparation section: 'we extrapolate the eth_balance values of addresses at the end of the extracted slice to be consistent with extracted data'. This is an approximation that could affect balance-dependent queries like Q1.
  • ad hoc to paper The 10 hand-written queries represent a challenging and representative workload for relational query optimizers.
    The queries are authored specifically for CrypQ (Section 3, Query Workload) and are 'work in progress'. Benchmark utility depends on their complexity being representative.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CrypQ: A Database Benchmark Based on Dynamic, Ever-Evolving Ethereum Data." pith.science (2026). https://pith.science/paper/2UVTOK7M

@misc{pith2026241117913,
  author       = {Pith},
  title        = {Pith review of: CrypQ: A Database Benchmark Based on Dynamic, Ever-Evolving Ethereum Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2UVTOK7M}},
  note         = {Machine review of arXiv:2411.17913}
}
read the original abstract

Modern database systems are expected to handle dynamic data whose characteristics may evolve over time. Many popular database benchmarks are limited in their ability to evaluate this dynamic aspect of the database systems. Those that use synthetic data generators often fail to capture the complexity and unpredictable nature of real data, while most real-world datasets are static and difficult to create high-volume, realistic updates for. This paper introduces CrypQ, a database benchmark leveraging dynamic, public Ethereum blockchain data. CrypQ offers a high-volume, ever-evolving dataset reflecting the unpredictable nature of a real and active cryptocurrency market. We detail CrypQ's schema, procedures for creating data snapshots and update sequences, and a suite of relevant SQL queries. As an example, we demonstrate CrypQ's utility in evaluating cost-based query optimizers on complex, evolving data distributions with real-world skewness and dependencies.

Figures

Figures reproduced from arXiv: 2411.17913 by the authors.

Figure 1
Figure 1. CrypQ schema.Components of the primary key for each table are shaded; UNIQUE keys are not marked. Arrows go from foreign keys to the primary keys they reference; two lines are dashed because they do not reference primary keys. Background and Database Schema [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Accuracy of cardinality estimation for Scenario 1. [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗
Figure 3
Figure 3. Accuracy of cardinality estimation for Scenario 2. [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Details of query plans. with the third table (Figure 3l). These poor estimates cause PostgreSQL to overestimate the plan cost for (tk_tx ⋊⋉ tk) ⋊⋉ tx and, consequently, the plan P(S2) = P(S3) = P(S4), which consists of this subquery; PostgreSQL hence mistakenly chooses…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 16 canonical work pages

  1. [1]

    https://github.com/dukedb-crypq

    CrypQ: A database benchmark based on dynamic, ever-evolving Ethereum data. https://github.com/dukedb-crypq

    work page

  2. [2]

    https://cloud.google.com/blog/products/data-analytics/ ethereum-bigquery-public-dataset-smart-contract-analytics

    Ethereum in BigQuery: a public dataset for smart contract ana- lytics. https://cloud.google.com/blog/products/data-analytics/ ethereum-bigquery-public-dataset-smart-contract-analytics

    work page

  3. [3]

    In: Proceedings of the Thirtieth International Conference on Very Large Data Bases- Volume 30

    Arasu, A., Cherniack, M., Galvez, E., Maier, D., Maskey, A.S., Ryvkina, E., Stone- braker, M., Tibbetts, R.: Linear road: a stream data management benchmark. In: Proceedings of the Thirtieth International Conference on Very Large Data Bases- Volume 30. pp. 480–491 (2004)

    work page 2004

  4. [4]

    In: Proceedings of the 1st ACM Symposium on Cloud Computing

    Cooper, B.F., Silberstein, A., Tam, E., Ramakrishnan, R., Sears, R.: Benchmarking cloud serving systems with ycsb. In: Proceedings of the 1st ACM Symposium on Cloud Computing. pp. 143–154 (2010)

    work page 2010

  5. [5]

    Proceedings of the VLDB Endowment 14(13), 3376–3388 (2021)

    Ding, B., Chaudhuri, S., Gehrke, J., Narasayya, V.: Dsb: A decision support bench- mark for workload-driven and traditional database systems. Proceedings of the VLDB Endowment 14(13), 3376–3388 (2021)

    work page 2021

  6. [6]

    Morgan Kaufmann Publishers Inc

    Gray, J.: Benchmark handbook: for database and transaction processing systems. Morgan Kaufmann Publishers Inc. (1992)

    work page 1992

  7. [7]

    Proceedings of the VLDB Endowment13(12), 1807– 1820 (2020)

    Gupta, P., Carey, M.J., Mehrotra, S., Yus, o.: Smartbench: A benchmark for data management in smart spaces. Proceedings of the VLDB Endowment13(12), 1807– 1820 (2020)

    work page 2020

  8. [8]

    Cardinality Estimation in DBMS: A Comprehensive Benchmark Evaluation

    Han, Y., Wu, Z., Wu, P., Zhu, R., Yang, J., Tan, L.W., Zeng, K., Cong, G., Qin, Y., Pfadler, A., et al.: Cardinality estimation in DBMS: A comprehensive benchmark evaluation. arXiv preprint arXiv:2109.05877 (2021)

    work page Pith review arXiv 2021

Show all 16 references
  1. [9]

    The Journal of Supercomputing75, 8207–8230 (2019)

    Hendawi, A., Gupta, J., Liu, J., Teredesai, A., Ramakrishnan, N., Shah, M., El- Sappagh, S., Kwak, K.S., Ali, M.: Benchmarking large-scale data management for internet of things. The Journal of Supercomputing75, 8207–8230 (2019)

    2019

  2. [10]

    In: Proceedings of the ACM/SPEC In- ternational Conference on Performance Engineering

    Hesse, G., Matthies, C., Perscheid, M., Uflacker, M., Plattner, H.: Espbench: The enterprise stream processing benchmark. In: Proceedings of the ACM/SPEC In- ternational Conference on Performance Engineering. pp. 201–212 (2021)

    2021

  3. [11]

    Leis, V., Gubichev, A., Mirchev, A., Boncz, P., Kemper, A., Neumann, T.: How good are query optimizers, really? Proceedings of the VLDB Endowment 9(3), 204–215 (2015)

    2015

  4. [12]

    In: Proceedings of the 2021 International Conference on Management of Data

    Marcus, R., Negi, P., Mao, H., Tatbul, N., Alizadeh, M., Kraska, T.: Bao: Making learned query optimization practical. In: Proceedings of the 2021 International Conference on Management of Data. pp. 1275–1288 (2021)

    2021

  5. [13]

    Proceedings of the VLDB Endowment 2(1), 982–993 (2009)

    Moerkotte, G., Neumann, T., Steidl, G.: Preventing bad plans by bounding the impact of cardinality estimation errors. Proceedings of the VLDB Endowment 2(1), 982–993 (2009)

    2009

  6. [14]

    TPC-C: https://www.tpc.org/tpcc/

  7. [15]

    TPC-DS: https://www.tpc.org/tpcds/

  8. [16]

    Proceedings of the VLDB Endowment17(13) (2024)

    Xiu, H., Agarwal, P.K., Yang, J.: PARQO: Penalty-aware robust plan selection in query optimization. Proceedings of the VLDB Endowment17(13) (2024)

    2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.