Pith. sign in

REVIEW 2 major objections 1 minor 16 references

Table format metadata can satisfy roughly 60 percent of data quality rules without scanning any data.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-28 23:46 UTC pith:YTN3MRLP

load-bearing objection The paper shows a real LinkedIn deployment where Iceberg metadata covers 60% of DQ rules at zero extra cost and cuts profiling by half, with a practical proposal to add sketches via Puffin files. the 2 major comments →

arxiv 2605.30308 v1 pith:YTN3MRLP submitted 2026-05-28 cs.DB

Zero-Scan Data Quality: Leveraging Table Format Metadata for Continuous Observability at Scale

Mohit Verma , Shantanu Rawat , Christian Bush , Sumedh Sakdeo , Lokesh Amarnath Ravindranathan , Dwarak Bakshi This is my paper
classification cs.DB
keywords data qualitytable formatsIcebergmetadataobservabilityzero-scanprofilingsketches
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Modern table formats already compute and store commit timestamps, record counts, null counts, and value bounds at write time for query planning. The paper shows these statistics overlap enough with common DQ checks to support anomaly detection, drift monitoring, and null-rate tracking at zero marginal cost. In production use across more than 200,000 Iceberg tables this reuse meets about 60 percent of user-defined rules and halves profiling resource use. Adding a small set of counters and mergeable sketches for distinct values and quantiles is projected to reach nearly 90 percent coverage while preserving the store-then-aggregate pattern already used by manifests.

Core claim

Write-time metadata statistics stored by table formats such as Iceberg can be repurposed for continuous data quality observability, satisfying approximately 60 percent of production DQ rules at zero marginal compute cost and reducing profiling consumption by around 50 percent; lightweight extensions with per-file counters and sketches can raise coverage to close to 90 percent, and table formats should therefore store such sketches in Puffin sidecar files.

What carries the argument

The metadata-first approach that reuses Iceberg manifest statistics (record counts, null counts, value bounds) plus proposed lightweight counters and Theta/KLL sketches for DQ rules without data access.

Load-bearing premise

The statistics already written by table formats at write time overlap substantially with the checks needed for user-defined data quality rules.

What would settle it

A measurement on a separate large Iceberg deployment showing that the fraction of DQ rules satisfiable solely from manifest statistics is materially lower than 60 percent.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Approximately 60 percent of user-defined DQ rules become satisfiable at zero marginal compute cost.
  • Profiling resource consumption drops by around 50 percent.
  • Coverage can reach close to 90 percent once lightweight counters and mergeable sketches are added to manifests.
  • Table formats should store per-file sketches in Puffin sidecar files following the existing manifest pattern.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Data platforms could retire separate full-table profiling jobs for the covered fraction of checks.
  • The same metadata reuse pattern might apply to other open table formats that already keep column statistics.
  • Storage overhead and sketch accuracy would need direct measurement on additional production workloads beyond the validation reported.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 1 minor

Summary. The manuscript proposes repurposing write-time metadata statistics (null counts, value bounds, record counts, commit timestamps) already maintained by Apache Iceberg for continuous data-quality observability tasks including anomaly detection, drift monitoring, and null-rate tracking, without any data scans. It reports a production deployment at LinkedIn spanning 200,000+ Iceberg tables (800+ PB) that satisfies approximately 60% of user-defined DQ rules at zero marginal compute cost while cutting profiling resource consumption by around 50%. The authors further describe lightweight extensions—additional counters (sum, zero-value, boolean) and mergeable sketches (Theta for distinct counts, KLL for quantiles) stored in Puffin sidecar files—that can raise metadata-based coverage to nearly 90%, with validation of sketch accuracy, mergeability, and storage overhead performed on production data. They recommend that table formats adopt per-file sketches following the existing store-then-aggregate manifest pattern.

Significance. If the reported coverage figures and resource savings are substantiated by detailed measurement, the work offers a practical route to substantially lower the ongoing compute cost of DQ monitoring in large Iceberg-based data lakes by exploiting metadata that is already written. The concrete proposal to standardize lightweight, mergeable sketches inside Puffin files extends an established architectural pattern and could influence future table-format specifications. Validation of the sketches on real production workloads is a positive element that directly supports the feasibility of the 90% coverage target.

major comments (2)
  1. [Abstract] Abstract: the central empirical claims—60% coverage of user-defined DQ rules, 50% reduction in profiling resources, and the 90% target with extensions—are stated without any accompanying methodology, definition of the rule population, measurement protocol, baseline comparison, or error analysis. These quantities are load-bearing for the paper’s assertion of practical benefit at scale.
  2. [Abstract] Abstract and deployment description: the claim that manifest statistics “overlap substantially” with production DQ rules is quantified only by the 60% figure; no breakdown is supplied showing which rule categories are covered by existing null-count/bound statistics versus those requiring the proposed counters or sketches.
minor comments (1)
  1. The manuscript would benefit from an explicit table or section listing the exact set of Iceberg manifest fields used for each DQ rule type, together with the additional counters and sketches required to reach the 90% target.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for highlighting the need for greater transparency around our empirical claims. We address each comment below and will revise the abstract and deployment sections accordingly.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the central empirical claims—60% coverage of user-defined DQ rules, 50% reduction in profiling resources, and the 90% target with extensions—are stated without any accompanying methodology, definition of the rule population, measurement protocol, baseline comparison, or error analysis. These quantities are load-bearing for the paper’s assertion of practical benefit at scale.

    Authors: We agree the abstract should briefly reference the supporting methodology. The full manuscript (Section 4) defines the rule population as all production DQ rules across the 200k+ tables, describes the measurement protocol (rule-by-rule matching of metadata statistics against full-scan baselines on sampled tables), baseline (existing profiling jobs), and reports error analysis via sampled validation. We will revise the abstract to include a concise methodology clause and cross-reference to these details. revision: yes

  2. Referee: [Abstract] Abstract and deployment description: the claim that manifest statistics “overlap substantially” with production DQ rules is quantified only by the 60% figure; no breakdown is supplied showing which rule categories are covered by existing null-count/bound statistics versus those requiring the proposed counters or sketches.

    Authors: Section 3 of the manuscript already contains the requested category breakdown (null-rate rules covered by null counts, range/bound rules by min/max, record-count rules by file stats, etc.). We will add a compact version of this breakdown to the abstract and deployment description, explicitly separating coverage from existing manifest statistics versus the proposed counters and sketches. revision: yes

Circularity Check

0 steps flagged

No significant circularity identified

full rationale

The paper describes an engineering approach that reuses pre-existing write-time metadata statistics (null counts, bounds, record counts) already computed by Iceberg manifests for query planning, then extends them with counters and sketches stored in Puffin files. No equations, fitted parameters, predictions derived from fits, or self-citations of uniqueness theorems appear in the provided text. Coverage percentages (60%, ~90%) are presented as empirical outcomes from LinkedIn deployment and sketch validation on production data rather than any self-referential construction. The central argument rests on observed overlap between manifest statistics and DQ rules plus the store-then-aggregate pattern, which is externally verifiable and does not reduce to the paper's own inputs by definition.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on a single domain assumption about metadata-DQ overlap; no free parameters or invented entities are introduced.

axioms (1)
  • domain assumption Write-time metadata statistics accurately reflect data properties sufficiently for a substantial fraction of DQ monitoring tasks
    Invoked to justify zero-scan coverage without data access.

pith-pipeline@v0.9.1-grok · 5763 in / 1191 out tokens · 32061 ms · 2026-06-28T23:46:40.454398+00:00 · methodology

0 comments
read the original abstract

Modern table formats such as Apache Iceberg compute and store metadata-commit timestamps, record counts, and column-level statistics such as null counts and value bounds at write time as part of file writing. These statistics serve query planning, yet they overlap substantially with data quality (DQ) monitoring needs. We describe a metadata-first approach that repurposes write-time statistics for continuous DQ observability: anomaly detection, drift monitoring, null-rate tracking; without scanning any data. Deployed at LinkedIn across 200,000+ Iceberg tables (800+ PB), this approach satisfies approximately 60% of user-defined DQ rules at zero marginal compute cost and reduces profiling resource consumption by around 50%. Extending manifest statistics with lightweight counters (sum, zero-value counts, boolean counts) and incrementally mergeable sketches; Theta sketches for distinct counts, KLL sketches for quantiles; can further raise metadata-satisfiable coverage to close to 90% of production DQ rules. We validate sketch accuracy, mergeability, and storage overhead on production data and propose that table formats should store per-file sketches in Puffin sidecar files, following the same store-then-aggregate pattern used for existing manifest statistics.

Figures

Figures reproduced from arXiv: 2605.30308 by Christian Bush, Dwarak Bakshi, Lokesh Amarnath Ravindranathan, Mohit Verma, Shantanu Rawat, Sumedh Sakdeo.

Figure 1
Figure 1. Figure 1: Iceberg commit metadata flows through Kafka to [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

16 extracted references · 4 canonical work pages

  1. [1]

    Apache Iceberg Table Spec—Manifest Files

    Apache Software Foundation. Apache Iceberg Table Spec—Manifest Files. https: //iceberg.apache.org/spec/

  2. [2]

    Beyer, P

    K. Beyer, P. J. Haas, B. Reinwald, Y. Sismanis, and R. Gemulla. On Synopses for Distinct-Value Estimation Under Multiset Operations.Proc. ACM SIGMOD, 2007

    2007

  3. [3]

    Karnin, K

    Z. Karnin, K. Lang, and E. Liberty. Optimal Quantile Approximation in Streams. Proc. IEEE FOCS, 2016

    2016

  4. [4]

    Delta Lake Protocol Specification

    Delta Lake Project. Delta Lake Protocol Specification. https://github.com/delta- io/delta/blob/master/PROTOCOL.md

  5. [5]

    Puffin Spec: Statistics and Index Blobs for Iceberg Tables

    Apache Software Foundation. Puffin Spec: Statistics and Index Blobs for Iceberg Tables. https://iceberg.apache.org/puffin-spec/

  6. [6]

    Data tests

    dbt Labs. Data tests. https://web.archive.org/web/20241007092848/https://docs. getdbt.com/docs/build/data-tests

    work page arXiv

  7. [7]

    Open-source data quality framework

    Great Expectations. Open-source data quality framework. https://web.archive. org/web/20250105115124/https://greatexpectations.io/

    work page arXiv

  8. [8]

    Lakehouse Monitoring

    Databricks. Lakehouse Monitoring. https://web.archive.org/web/ 20241224095718/https://docs.databricks.com/en/lakehouse-monitoring/

  9. [9]

    Introducing New Data Observability Features in Starburst Galaxy

    Starburst. Introducing New Data Observability Features in Starburst Galaxy. Starburst Blog, 2024

    2024

  10. [10]

    Apache Hudi: Pre-commit Validators

    Apache Software Foundation. Apache Hudi: Pre-commit Validators. https://web.archive.org/web/20250122174817/https://hudi.apache.org/docs/ precommit_validator/

    work page arXiv

  11. [11]

    Support check constraint

    Apache Iceberg. Support check constraint. GitHub Issue #14906, 2025

    2025

  12. [12]

    Shankar, A

    S. Shankar, A. Parameswaran, et al. SPADE: Synthesizing Data Quality Assertions for Large Language Model Pipelines.PVLDB, 17(12), 2024

    2024

  13. [13]

    Abedjan, L

    Z. Abedjan, L. Golab, F. Naumann, and T. Papenbrock. Data Profiling. Synthesis Lectures on Data Management, Morgan & Claypool, 2018

    2018

  14. [14]

    Armbrust, A

    M. Armbrust, A. Ghodsi, R. Xin, and M. Zaharia. Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics.Proc. CIDR, 2021

    2021

  15. [15]

    The GHC Developers

    Soda. Soda Core: Open-source data quality testing framework. https://web. archive.org/web/20241201175900/https://www.soda.io/

    work page arXiv

  16. [16]

    Apache DataSketches: Stochastic Streaming Algo- rithms

    Apache Software Foundation. Apache DataSketches: Stochastic Streaming Algo- rithms. https://datasketches.apache.org/