{"id":"f89b7205-594a-4581-9c6c-a7e117ce957a","arxiv_id":"2411.17913","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"CrypQ is a new relational database benchmark that uses dynamic Ethereum blockchain data to stress-test query optimizers with realistic, ever-changing data and update loads.","lead":"This paper introduces CrypQ, a database benchmark built from live Ethereum blockchain data, with realistic update workloads and ten SQL queries. It demonstrates the benchmark by showing how PostgreSQL's cardinality estimates and query plans degrade as real-world data distributions shift over time.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Moving-window update workload deletes old blocks without a described recomputation of eth_balance, so Scenario 1 states may not be valid Ethereum states and Q1-based results may be artifacts.","rationale":"The paper's central contribution is a benchmark whose update workload provides a realistic, ever-evolving dataset. Scenario 1 is the primary demonstration of this dynamic capability, and its moving-window procedure is the mechanism that supposedly keeps the database aligned with a valid Ethereum state. If expire.sql deletes old transactions without reversing their effects on eth_balance, every state W2..W11 is semantically invalid: balances reflect transactions that are no longer present. This is not merely a question of whether the extracted data is a perfect replica of Ethereum; it is an internal inconsistency in the described benchmark procedure. The inconsistency is load-bearing because the headline experimental results are obtained from exactly this procedure. Q1 and its extracted subqueries filter on a.eth_balance, so the cardinality estimates and plan choices shown in Figures 2-4 would be measuring a distribution that never existed on Ethereum. The concern is empirically testable by inspecting the scripts and comparing a moving-slice state against a fresh extraction. The reader's weakest assumption about eth_balance extrapolation is also valid and concerns static-slice realism, but the missing rollback is a sharper, procedure-level flaw that directly undermines the dynamic-update claim. I do not regard the paper as beyond repair: if the repository actually recomputes balances correctly, the concern disappears; if not, the authors can fix the scripts, rerun Scenario 1, and report whether the qualitative conclusions survive. Hence CONDITIONAL rather than REJECT: the contribution is potentially sound, but the central experimental demonstration must be verified against this consistency requirement before the benchmark's claims are accepted.","tokens_in":12311,"tokens_out":5223,"duration_ms":50490,"concrete_test":"Inspect expire.sql and gen_updates.sh in the public repository. If expire.sql contains no compensating balance updates or full recomputation of eth_balance, run the moving-slice procedure on a small slice of, say, 100 blocks with two expire+upsert batches, then compare the eth_balance values for addresses involved in expired transactions against a freshly extracted slice ending at the same final block. Any difference confirms that the moving-window state is not a valid Ethereum state at that block height.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 3 (Data Preparation and Update Workload) states that each upsert batch 'will also update balances in Addresses as a result of the new transactions,' but the optional moving-slice mode described in the same section deletes the oldest blocks (expire.sql) before applying new blocks. No compensating balance adjustment is described for these deletions. Because eth_balance is a derived value that must equal the net effect of all transactions in the current slice, deleting transactions without reversing their balance contributions leaves the database in a state that does not correspond to any real Ethereum block range. This directly affects Scenario 1 (Section 4): W2..W11 are produced exactly by this expire+upsert procedure, and the reported cardinality errors, Q-errors, and plan regressions for Q1 subqueries (Figures 2 and 4) all involve joins on Addresses with predicates on eth_balance. If balances drift artificially because expired transactions are not rolled back, the benchmark's update workload is not a faithful moving window, and the experiments may demonstrate behavior caused by the benchmark's own inconsistency rather than by realistic Ethereum dynamics. The extrapolation issue noted by the reader is related, but this is a distinct, more direct correctness gap: the paper's described procedure for the central dynamic scenario is internally inconsistent unless expire.sql also recomputes eth_balance, which is not documented.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CrypQ, a relational database benchmark built from public Ethereum blockchain data exported from Google BigQuery. It defines a schema (Blocks, Transactions, Addresses, Token_Transactions, Contracts, etc.), scripts for extracting slices and generating update workloads, and a suite of 10 SQL queries. The benchmark is intended to provide a real-world, high-volume, ever-evolving dataset with realistic skew and correlations. As an example, the authors run experiments on PostgreSQL in two scenarios: Scenario 1 applies a moving-window update workload (W1-W11), and Scenario 2 uses four temporally distant slices. They measure cardinality estimation Q-errors and query-plan regressions, showing that CrypQ can expose optimizer inaccuracies and plan-selection subtleties.","tokens_in":12600,"tokens_out":4885,"duration_ms":41824,"significance":"If the update workload is semantically sound, CrypQ addresses a real gap: most relational benchmarks are static or synthetic, while CrypQ provides public, reproducible artifacts and a genuinely dynamic real-world dataset. The paper is careful about the illustrative nature of the experiments and credits the plan-injection methodology to PARQO. The benchmark could be a reusable resource for future work on query optimization under data drift. However, the correctness of the moving-slice update semantics is load-bearing for the central claim that CrypQ offers realistic, ever-evolving data, and the current description has a gap that must be closed before the Scenario 1 results can be taken as demonstrating realistic dynamics rather than artifacts of the benchmark's own inconsistency.","major_comments":[{"comment":"The optional moving-slice mode is internally inconsistent as described. The paper states that expire.sql first deletes data for the oldest blocks before each upsert batch, and that upserts 'will also update balances in Addresses as a result of the new transactions.' No compensating adjustment to eth_balance is described for the deleted (expired) transactions. Because eth_balance is a derived value that should equal the net effect of all transactions in the currently active slice, deleting transactions without reversing their balance contributions leaves Addresses with balances that do not correspond to any real Ethereum block range. This directly affects Scenario 1 (Section 4): states W2-W11 are produced exactly by this expire+upsert procedure, and the Q1 subqueries used in Figures 2 and 4 join Addresses with predicates on eth_balance. The reported Q-errors and plan regressions may therefore be driven by the benchmark's own balance inconsistency rather than by realistic Ethereum dynamics. Please specify how expire.sql or a companion step recomputes eth_balance (or otherwise maintains the invariant), or revise the benchmark to avoid the moving-slice claim until this semantics is corrected.","section":"Section 3, Data Preparation and Update Workload"},{"comment":"The extrapolation of eth_balance is not tied to a specific block height in the dynamic scenario. The paper says that because BigQuery does not track balance history, the authors 'extrapolate the eth_balance values of addresses at the end of the extracted slice.' For Scenario 1, the initial state W1 is the first 1000 blocks of a 2000-block extract, and the script gen_updates.sh is applied to create the subsequent states. If the Addresses table for W1 is loaded with balances extrapolated from the end of the full 2000-block slice, those balances are inconsistent with the transactions active in W1, and subsequently applying only the new transactions will not correct them; even W11 would then not match the stated final 1000-block window. The paper should state explicitly for which block height eth_balance is computed when generating the initial load file and each upsert batch, and should validate the extrapolation against a known historical balance (e.g., from an archive node). This matters because the filter on a.eth_balance is central to Q1 and its subqueries in both example scenarios.","section":"Section 3, Data Preparation; Section 4, Scenario 1"}],"minor_comments":[{"comment":"There is a typo: 'cardinaility' should be 'cardinality'.","section":"Section 4, Example Use 2"},{"comment":"There is a typo: 'subtlties' should be 'subtleties'.","section":"Section 4, Example Use 2"},{"comment":"The Q-error labels are placed directly above the plotted lines without a separate legend; a legend or a clearer annotation style would help readers distinguish the 'refreshed estimates' and 'initial estimates' series from their Q-error labels.","section":"Figures 2 and 3"},{"comment":"The checkmark formatting in Table 1 is inconsistent across rows; using 'Yes'/'No'/'Partial' for all cells would improve readability and avoid ambiguity about the 'periodic refreshes only' and 'monthly updates only' footnotes.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The artifact is potentially valuable and the public GitHub repository is a strength. The main issue is a correctness gap in the moving-slice update semantics: without recomputing eth_balance after expiring blocks, the benchmark can produce states that are not valid Ethereum states, which undermines the Scenario 1 experiments. This is fixable within the scope of the paper by specifying the recomputation procedure, correcting the generation scripts, and re-running the affected experiments; I would not accept the paper in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"CrypQ is a solid benchmark paper: it fills a real gap by providing a dynamic, real-world relational dataset with fine-grained update streams, and the authors are appropriately modest about the evaluation. The schema is well designed, the extraction and update scripts are public, and the ten queries cover a useful range of SQL features. The example experiments are carefully run (median of 11, Q-errors, plan regressions) and honestly scoped as illustrative. The self-citation to PARQO is appropriate—it’s used for plan injection methodology, not to prop up the benchmark's value.\n\nThe main soft spot, which I agree with the stress-test note on, is the moving-slice update mode. The text says that when blocks are expired (expire.sql), the oldest blocks are deleted before new ones are upserted, and that balances are updated “as a result of the new transactions.” But no compensating adjustment is described for the deleted transactions. Since eth_balance is derived from the net effect of all transactions in the current slice, this procedure leaves the database in a state that does not correspond to any real Ethereum block range. That directly affects Scenario 1: W2–W11 are produced exactly this way, and the reported Q-errors for subqueries joining to Addresses could be artifacts of balance drift rather than realistic Ethereum dynamics. This is a load-bearing issue for that scenario, though not for the static slices (S1–S4) or the growing-data mode without expiry. It may be that the code actually recomputes balances in expire.sql and the paper just doesn't document it—but as written, the described procedure is inconsistent.\n\nThe other limitations are less severe: extrapolating eth_balance from the end of the slice is acknowledged and reasonable; the evaluation is only on PostgreSQL, which is fine for an illustrative paper; and the query suite is small but the authors say it is work in progress.\n\nOverall, this is a worthwhile contribution that deserves serious refereeing. The moving-window issue is fixable—either by recomputing balances on expiry or by documenting that the benchmark intentionally decouples balance history from the sliding window—but it must be addressed before the benchmark is used as-is. I would send it to review, with the request that the authors clarify or fix the update semantics, and I'd want to inspect the actual expire.sql in the repo.\n\nMy advice: accept it, but only after the balance maintenance question is resolved. The paper is honest and the artifact is genuinely useful for the adaptive-query-optimization and cardinality-estimation communities.","headline":"CrypQ is a genuinely useful benchmark with real Ethereum data and fine-grained updates, but the moving-window mode has a correctness gap around eth_balance that needs fixing before the paper is fully sound.","tokens_in":13058,"tokens_out":3040,"would_cite":false,"duration_ms":30447,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CrypQ turns Ethereum's public blockchain into a dynamic database benchmark that exposes real-world query optimizer failures in cardinality estimation and plan selection.","keywords":["database benchmark","Ethereum blockchain","dynamic data","query optimization","cardinality estimation","update workload","PostgreSQL"],"falsifier":"Reconstruct true historical eth_balance values for a CrypQ slice from the full Ethereum transaction history (or an archive node), then compare their distribution and the selectivity of Q1's balance predicate against CrypQ's extrapolated balances; a large divergence would show that the benchmark's balance-based queries test an artificial distribution rather than Ethereum's actual dynamics.","tokens_in":12150,"feed_emoji":"⛓️","tokens_out":6265,"duration_ms":49756,"temperature":0.7,"pith_summary":"CrypQ is a database benchmark built from live Ethereum blockchain data, giving researchers a high-volume dataset that changes continuously and reflects the real skews and correlations of an active cryptocurrency market. The paper argues that existing benchmarks either generate synthetic data with fixed distributions or use static real-world snapshots, so they cannot test how cost-based query optimizers cope with data distributions that drift over time. CrypQ provides a schema, extraction scripts, a fine-grained update workload, and ten SQL queries, and the authors show that PostgreSQL's default histograms can have Q-errors above 10 and that plans chosen on one slice of Ethereum data can run over 2x slower on another. If the benchmark is adopted, it would give the query-optimization community a shared, ever-growing testbed for dynamic workloads.","feed_headline":"Ethereum's ever-changing data stress-tests query optimizers","feed_subtitle":"Real Ethereum updates cause cardinality errors over 10x and plan regressions over 2x in PostgreSQL.","key_machinery":"The load-bearing object is the Ethereum blockchain itself, accessed through Google BigQuery's public dataset: CrypQ slices it into consecutive block ranges that define initial states and generates an upsert file per block to produce a realistic, fine-grained update workload. The schema's central tables (Blocks, Transactions, Addresses, Tokens, Contracts, Token_Transactions, Withdrawals) preserve the ledger's foreign-key structure, and the query suite is designed to exercise join-heavy, skew-sensitive relational queries against this evolving data.","core_discovery":"The paper introduces CrypQ as a practical benchmark that exploits the public Ethereum ledger as a source of realistic, continually arriving data. CrypQ's schema captures blocks, transactions, addresses, tokens, contracts, token transfers, and withdrawals; its data-preparation pipeline extracts consecutive block slices from Google BigQuery's Ethereum dataset, converts binary identifiers to efficient byte representations, and generates batched upsert files that advance the database state block by block (with optional expiration to hold size constant). The query suite of ten queries includes joins, aggregations, subqueries, lateral joins, window functions, and recursive CTEs aimed at questions about large transfers, token supply changes, and contract usage. In the included experiments, PostgreSQL's cardinality estimates on CrypQ subqueries exceed Q-error 10 for multi-way joins in three of four slices, and a plan optimized for a low-activity slice runs up to 2.35x slower (estimated) and 1.81x slower (measured) on a high-activity slice, demonstrating that CrypQ can expose optimizer failures in realistic dynamic settings.","pith_inferences":["An implicit consequence is that CrypQ could serve as a shared training and evaluation ground for learned cardinality estimators, since the Ethereum ledger provides an unbounded sequence of future blocks for temporal cross-validation.","The balance extrapolation step suggests a natural robustness test: CrypQ users could compare query results against an archive-node reconstruction of historical balances to measure how much the benchmark's selectivity depends on this approximation.","Because Ethereum's data grows daily, CrypQ could be extended into a continuously updated benchmark with versioned snapshots, letting the community track how optimizer performance changes as the ledger evolves over months or years."],"forward_implications":["CrypQ gives query-optimizer researchers a public benchmark where summary statistics must be refreshed or adapted as data evolves; the included Scenario 1 shows that stale histograms degrade Q-error over time.","The benchmark can distinguish scenarios where cardinality errors matter for plan selection from those where cached plans remain acceptable, since Scenario 1 produced no plan change while Scenario 2 produced regressions.","The four slices S1-S4 provide a ready-made cross-distribution testbed: same schema and queries, widely different data characteristics, enabling evaluation of estimators and optimizers across realistic distribution shifts.","CrypQ's update workload can be replayed at real-time speed by scheduling upsert files according to block timestamps, enabling streaming or incremental-maintenance research."],"supporting_citations":[{"why":"Supplies the public Ethereum dataset from which CrypQ extracts its slices and update workload.","marker":"[2]"},{"why":"Provides the methodology for injecting cardinality estimates and query plans into PostgreSQL for the experiments.","marker":"[16]"},{"why":"Defines Q-error, the metric used to measure cardinality estimation accuracy in the demonstration.","marker":"[13]"},{"why":"The static real-world benchmark that CrypQ extends by adding dynamic updates; serves as the main comparison point.","marker":"[11]"},{"why":"Another static real-world cardinality-estimation benchmark that motivates the need for a dynamic alternative.","marker":"[8]"},{"why":"Hosts the benchmark artifacts (schema, extraction scripts, and query workload) that make the benchmark usable.","marker":"[1]"}],"fun_headline_variants":["Real Ethereum updates cause 10x optimizer errors","CrypQ: Dynamic blockchain data breaks query plans","Evolving Ethereum data stumps PostgreSQL optimizers","Ever-changing Ethereum data exposes optimizer flaws"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's realism rests on the assumption that extrapolating each address's end-of-slice balance yields a faithful picture of historical balances, even though the source data does not track balance history.","fun_headline_variants_meta":{"raw":{"variants":["Real Ethereum updates cause 10x optimizer errors","CrypQ: Dynamic blockchain data breaks query plans","Evolving Ethereum data stumps PostgreSQL optimizers","Ever-changing Ethereum data exposes optimizer flaws"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000611,"raw_usage":{"total_tokens":2826,"prompt_tokens":910,"completion_tokens":1916,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":526,"completion_tokens_details":{"reasoning_tokens":1857}},"tokens_in":526,"tokens_out":1916,"duration_ms":13963,"temperature":1.0,"reasoning_tokens":1857,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:42:56.815297+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Reconstruct true historical eth_balance values for a CrypQ slice from the full Ethereum transaction history (or an archive node), then compare their distribution and the selectivity of Q1's balance predicate against CrypQ's extrapolated balances; a large divergence would show that the benchmark's balance-based queries test an artificial distribution rather than Ethereum's actual dynamics.","supporting_citations":[{"cited_title":"https://cloud.google.com/blog/products/data-analytics/ ethereum-bigquery-public-dataset-smart-contract-analytics","cited_arxiv_id":null,"evidence_quote":"Supplies the public Ethereum dataset from which CrypQ extracts its slices and update workload."},{"cited_title":"Proceedings of the VLDB Endowment17(13) (2024)","cited_arxiv_id":null,"evidence_quote":"Provides the methodology for injecting cardinality estimates and query plans into PostgreSQL for the experiments."},{"cited_title":"Proceedings of the VLDB Endowment 2(1), 982–993 (2009)","cited_arxiv_id":null,"evidence_quote":"Defines Q-error, the metric used to measure cardinality estimation accuracy in the demonstration."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The static real-world benchmark that CrypQ extends by adding dynamic updates; serves as the main comparison point."},{"cited_title":"Cardinality Estimation in DBMS: A Comprehensive Benchmark Evaluation","cited_arxiv_id":"2109.05877","evidence_quote":"Another static real-world cardinality-estimation benchmark that motivates the need for a dynamic alternative."},{"cited_title":"https://github.com/dukedb-crypq","cited_arxiv_id":null,"evidence_quote":"Hosts the benchmark artifacts (schema, extraction scripts, and query workload) that make the benchmark usable."}],"review_version":1}