Pith. sign in

REVIEW 3 major objections 4 minor 16 references

A single web app combines realistic error injection, conformal cleaning, and mechanism detection for tabular data.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 10:38 UTC pith:XK3WLXAG

load-bearing objection A useful integration of three prior data-cleaning tools, but the quantitative evidence in the paper doesn't yet show the integrated demo performs as claimed. the 3 major comments →

arxiv 2607.20140 v1 pith:XK3WLXAG submitted 2026-07-22 cs.LG

CURED: Creating, Understanding, and Repairing Errors Demonstrator

Nicholas Chandler , Sebastian J\"ager , Philipp Jung , Felix Bie{\ss}mann This is my paper
classification cs.LG
keywords tabular data cleaningconformal predictionerror injectionerror mechanismsdata qualityinteractive demonstratorimputationerror detection
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that the gap between theoretical research on tabular data errors and practical intuition can be closed with an interactive demonstrator. It unifies three results: an error generator that injects realistic, data-dependent errors; a conformal prediction-based cleaner that detects and corrects erroneous cells; and a mechanism detector that classifies whether errors occur completely at random, at random, or not at random. The paper reports quantitative evidence in line with the underlying methods: the cleaner detects errors with a true positive rate of about 0.65 and a false positive rate of about 0.007 across error mechanisms, and the mechanism detector reaches 0.71–0.99 accuracy depending on mechanism and column type. A sympathetic reader would care because the demonstrator makes these methods directly usable by non-specialists and offers a testbed for comparing error models and cleaning algorithms.

Core claim

The central claim is that three strands of tabular error research — realistic error generation, conformalized data cleaning, and error-mechanism detection — can be combined in a single web application without losing the individual methods' documented performance. The demonstrator implements the full loop: users upload or choose a dataset, inject errors with configurable types and mechanisms, run a cleaner that exploits conformal prediction to build per-cell confidence sets, and finally run a mechanism detector that recovers the error mechanism by comparing ML models on the error mask. The reported end-to-end tables show the cleaner detecting roughly 65% of errors at a 0.7% false positive rat

What carries the argument

The load-bearing mechanism is conformalized imputation: for each column of a table, a separate tabular ML model is trained on the remaining columns, and conformal prediction converts each model's output into a statistically valid prediction set at a user-chosen miscoverage rate α. A cell whose observed value falls outside its prediction set is flagged as an error and replaced by the model's point prediction, turning cleaning into a missing-value imputation task. The error generator decouples error type (for example, a numeric unit error or a typo) from error mechanism — completely at random, at random, or not at random — and from error rate, which enables controlled experiments. The mechanis

Load-bearing premise

The conformal cleaning guarantee holds only if the 80% training split is clean and the remaining 20% is exchangeable with it; if the uploaded data already contains correlated errors or distribution shift, the prediction sets are mis-calibrated and the reported false positive rate will not be achieved.

What would settle it

Take a dataset of 10,000 rows, apply a systematic corruption to the last 2,000 rows (say add 10 to one numeric column's values), run the conformal cleaner at α=0.001, and measure the actual coverage of the prediction sets on those corrupted rows; if coverage is substantially below 99.9%, the exchangeability assumption under the cleaning guarantee is broken.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A non-specialist can add realistic, data-dependent errors to their own CSV and immediately see which cells the conformal cleaner flags and repairs, making the abstract guarantees concrete.
  • Because the cleaning confidence α is user-adjustable, the demo doubles as an interactive illustration of the trade-off between detecting many errors and generating false positives.
  • The reported numbers provide an integrated baseline for the three methods, so a researcher can check whether a new cleaner improves on the ~0.65 TPR / ~0.007 FPR combination in the same setup.
  • The open-source code and modular design allow other error generators or cleaning back-ends to be plugged in, turning the demo into an extendable testbed.
  • The three-in-one pipeline can serve as a controlled environment for studying how the error mechanism affects cleaning performance.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A user uploading already-dirty data (rather than starting from a clean file) has no guarantee that the 80/20 split is clean; the conformal calibration would then be invalid, and the demo would benefit from warning users about this condition.
  • The noticeably lower ENAR accuracy (~0.73) suggests that errors depending on unobserved variables remain the hardest case; this points to a research gap that could be attacked with richer error models or auxiliary data.
  • The same loop could be run in reverse on a real-world dirty table to produce a diagnostic: if the mechanism detector reports ENAR, a one-pass conformal cleaning may be less reliable than if it reports ECAR/EAR, so practitioners could adapt their mitigation strategy.
  • The 10-column / 10,000-row limits are adequate for teaching and exploration but far from industrial scale; testing the pipeline on wider and longer tables would expose whether the per-column imputer approach scales.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper presents CURED, a web-based demonstrator that combines three prior research components into a single interactive pipeline: (1) tab-err for injecting realistic, data-dependent errors into tabular data, (2) Conformal Data Cleaning (CDC) for detecting and repairing errors with conformal guarantees, and (3) MechDetect for characterizing the underlying error mechanism (ECAR/EAR/ENAR). Users can upload tabular data (restricted to fewer than 10 columns, fewer than 10,000 rows, and a column named 'target'), inject errors via a UI, run conformal cleaning, and receive error-mechanism diagnostics. The paper reports component-level quantitative results in two tables: CDC detection/cleaning performance (TPR, FPR, DSI) and MechDetect classification accuracy, and claims these constitute quantitative evidence for the demonstrator.

Significance. If the integrated demonstrator works as described, it would be a useful open-source vehicle for teaching and for small-scale experimentation in tabular data cleaning and error-model research. The paper's strengths include the permissive open-source release, a live web deployment, and the reuse of components with formal underpinnings, especially conformal prediction. The significance of the specific contribution, however, is currently limited by the disconnect between the component-level evaluation and the actual demonstrator configuration. The central claim of quantitative evidence for the demonstrator is not yet supported, because the numbers in Tables 2 and 3 are not shown to be produced by the CURED pipeline under its stated UI constraints. This is a repair issue rather than a fundamental flaw, so the contribution is defensible after a major revision.

major comments (3)
  1. [Sec. 2.3 / Table 2] The abstract claims "quantitative evidence" of cleaning performance, but Table 2 does not evaluate the demonstrator as configured. It reports median TPR/FPR/DSI "follow[ing] the experimental setting of the original paper" with no dataset list, error rates, number of runs, or variance. CURED restricts users to fewer than 10 columns, fewer than 10,000 rows, a required 'target' column, and an 80/20 split, and its only preloaded dataset is an 8-column subset of OpenML 44969. Nothing in the paper shows that the reported numbers hold on the preloaded dataset or under the demo's UI-configurable error settings. This is load-bearing because it concerns the central claim of a quantitatively supported demonstrator. The authors should either run the same evaluation through the CURED default configuration and report those results, or explicitly relabel Table 2 as a component-level reproduction with r
  2. [Sec. 2.4 / Table 3] The MechDetect accuracy table has the same issue. The text says the setup was "the same as in the original paper" but with different error types (numeric: Outlier, WrongUnit, AddDelta; categorical: Typo, Replace, Extraneous), a different classifier (Catboost instead of LightGBM), and "selected datasets" (109 numeric, 17 categorical) that are never identified. No error rates, number of scenarios, or standard deviations are given, and ENAR accuracy is only about 0.71--0.73. Again, these are component-level numbers; the reader cannot verify whether a user of the CURED web app, using the preloaded 8-column dataset and available error settings, would obtain anything close to these figures. Please provide CURED-specific results or present Table 3 explicitly as a benchmark of the underlying MechDetect component, with pointers to the original paper and code.
  3. [Sec. 2.3 / Sec. 2.1] The conformal guarantee P(y_test in C(X_test)) >= 1-alpha and the 80/20 split assume that the 80% training portion is clean and that the calibration/test portion is exchangeable with it. The paper does not state that user uploads must be free of pre-existing errors; if a user uploads a table with correlated errors or distribution shift, the error mask and imputations will be miscalibrated and the interactive results could mislead. The manuscript should explicitly state this limitation, both in the text and in the UI, and clarify that the conformal guarantee applies only to the setting where the training portion is clean and the test portion is generated by tab-err from the same clean base table.
minor comments (4)
  1. [Sec. 2.3] The formal notation is inconsistent: X_c = X_{{1,...,d}\{c}} is used both as a feature matrix and as a target y_c = X_c. Clarify row/column indexing, e.g., X_{i,c} for individual cells.
  2. [Tables 2 and 3] Both tables report point estimates only. Add standard deviations or confidence intervals, and specify the number of runs and the exact dataset configurations to make the numbers reproducible.
  3. [Sec. 2.4] The p-values mentioned in the CURED interface and text are not defined. State which statistical test is used and how multiple comparisons are handled, or provide a reference.
  4. [Sec. 2.3] There is a grammatical error: "In [Jaeger et al., 2021] show that data imputation is implemented..." should be "[Jaeger et al., 2021] show that...".

Circularity Check

0 steps flagged

No significant circularity: the paper is an integration demonstrator whose only formal guarantee is standard conformal coverage.

full rationale

This is a systems/demonstrator paper, not a derivation paper. It assembles three previously published components (tab-err, CDC, MechDetect) and provides benchmark-style results. The only formal guarantee stated, P(y_test in C(X_test)) >= 1 - alpha, is the standard conformal-coverage theorem; it is not derived from the paper's own fitted values. The component results in Tables 2 and 3 are empirical evaluations against synthetic errors generated by tab-err. Although the generator and detectors share authors, the detection task is nontrivial and the accuracy numbers are not forced by construction. The paper explicitly attributes the error taxonomy to Rubin's missing-data theory and to prior work, so no known result is renamed as new. The main weakness, that component-level numbers are not shown to hold under the demo's 10-column/10,000-row/target-column constraints, is an external-validity gap, not a circularity. No step reduces to its own input.

Axiom & Free-Parameter Ledger

1 free parameters · 3 axioms · 0 invented entities

The paper contributes a software integration, not a free-standing theoretical result. The central numbers depend on the chosen conformal confidence level. The evaluation assumes the error taxonomy and conformal exchangeability from prior self-authored work, and assumes the 80% training split is clean. No new entities are introduced.

free parameters (1)
  • Conformal confidence level = 0.999 (alpha=0.001)
    Fixed in Table 2 experiments and user-adjustable in the demo; directly controls conservatism and detection TPR/FPR.
axioms (3)
  • domain assumption Conformal prediction exchangeability: P(y_test in C(X_test)) >= 1-alpha holds for user data.
    Section 2.3 relies on conformal sets; the guarantee is valid only if test data are exchangeable with calibration data. User-uploaded tables may violate this.
  • domain assumption Error mechanisms are exhaustively categorized by ECAR/EAR/ENAR.
    From Jung et al. 2025b, used by tab-err and MechDetect; if real errors lie outside this taxonomy, characterization claims fail.
  • domain assumption The 80% training split contains no errors and is representative of the data.
    Section 2.1 uses 80% to train the cleaner and downstream model, and 20% for error generation/cleaning. If the training split already has errors, the imputation models are trained on corrupted targets.

pith-pipeline@v1.3.0-alltime-deepseek · 4721 in / 10654 out tokens · 92745 ms · 2026-08-01T10:38:21.025937+00:00 · methodology

0 comments
read the original abstract

Detecting and cleaning errors in tabular data is a prerequisite for data intense software applications. Recent research at the intersection of Machine Learning (ML) and Database Management Systems (DBMS) highlights the potential of statistical learning algorithms for error detection and cleaning. This paper combines our recent work on ML-based data cleaning and error models in a unified demonstrator. The web application allows users to upload tabular data, perturb the data with realistic data dependent errors and use modern ML methods to clean and understand error mechanisms in data. Our demonstrator helps to bridge the gap between theoretical advancements and intuitive practical insights in the context of error models and data cleaning algorithms for tabular data. The demonstrator is available at https://cured.demo.calgo-lab.de/

Figures

Figures reproduced from arXiv: 2607.20140 by Felix Bie{\ss}mann, Nicholas Chandler, Philipp Jung, Sebastian J\"ager.

Figure 1
Figure 1. Figure 1: Overview of CURED. Users can upload data, perturb data [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: The task type should be specified here by the user as [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗
Figure 2
Figure 2. Figure 2: Error generation phase of the demo 2 https://github.com/calgo-lab Errors are generated using the package tab-err [Calgo￾Lab, 2026]. Following the error generation model in [Jung et al., 2025b] errors are parametrized by error type, error mech￾anism, and error rate. The error type is simply how the cell in the table is actually altered. For example a numeric cell may be multiplied by 10 if the wrong unit wa… view at source ↗
Figure 3
Figure 3. Figure 3: Error cleaning phase of the demo a binary mask, which marks incorrect cells as 1 and correct ones as 0. Replacing incorrect cells of the input data with NaNs essentially casts the cleaning problem into a missing value imputation problem. We then directly replace incorrect cells with the point predictions of the underlying ML model, which concludes the error cleaning step. In [PITH_FULL_IMAGE:figures/full_… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

16 extracted references · 1 linked inside Pith

  1. [1]

    Ilyas, Mourad Ouz- zani, Paolo Papotti, Michael Stonebraker, and Nan Tang

    [Abedjanet al., 2016 ] Ziawasch Abedjan, Xu Chu, Dong Deng, Raul Castro Fernandez, Ihab F. Ilyas, Mourad Ouz- zani, Paolo Papotti, Michael Stonebraker, and Nan Tang. Detecting data errors: Where are we and what needs to be done?Proceedings of the VLDB Endowment, 9(12):993– 1004,

    2016

  2. [4]

    tab err: A python package for detecting and correcting tabular data errors,

    [Calgo-Lab, 2026] Calgo-Lab. tab err: A python package for detecting and correcting tabular data errors,

    2026

  3. [9]

    Representation learning for tabular data: A comprehensive survey.arXiv preprint arXiv:2504.16109,

    [Jianget al., 2025 ] Jun-Peng Jiang, Si-Yang Liu, Hao-Run Cai, Qile Zhou, and Han-Jia Ye. Representation learning for tabular data: A comprehensive survey.arXiv preprint arXiv:2504.16109,

    Pith/arXiv arXiv 2025

  4. [10]

    Lightgbm: A highly efficient gradient boosting decision tree

    [Keet al., 2017 ] Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qi Ye, and Tie- Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. InAdvances in Neural Information Process- ing Systems 30 (NeurIPS), pages 3146–3154,

    2017

  5. [12]

    Impact of errors in operational spread- sheets.Decision Support Systems, 47:126–132, 05

    [Powellet al., 2009 ] Stephen Powell, Kenneth Baker, and Barry Lawson. Impact of errors in operational spread- sheets.Decision Support Systems, 47:126–132, 05

    2009

  6. [16]

    Korth, and S

    [Silberschatzet al., 2019 ] Abraham Silberschatz, Henry F. Korth, and S. Sudarshan.Database System Concepts. McGraw-Hill, New York, NY , 7th edition, 2019

    2019

  7. [1976]

    Automat- ing Large-Scale Data Quality Verification.PVLDB, 11(12):1781–1794,

    [Schelteret al., 2018 ] Sebastian Schelter, Dustin Lange, Philipp Schmidt, Meltem Celikel, Felix Biessmann, An- dreas Grafberger, and Meltem Ce-Likel. Automat- ing Large-Scale Data Quality Verification.PVLDB, 11(12):1781–1794,

    2018

  8. [2009]

    Dorogush, and An- drey Gulin

    [Prokhorenkovaet al., 2018 ] Liudmila Prokhorenkova, Gleb Gusev, Aleksandr V orobev, Anna V . Dorogush, and An- drey Gulin. Catboost: Unbiased boosting with categorical features. InProceedings of the Thirty-Second Conference on Artificial Intelligence (AAAI), pages 6639–6647,

    2018

  9. [2015]

    [J¨ager and Biessmann, 2024] Sebastian J ¨ager and Felix Biessmann

    ISO/IEC Stan- dard. [J¨ager and Biessmann, 2024] Sebastian J ¨ager and Felix Biessmann. From data imputation to data cleaning — automated cleaning of tabular data improves down- stream predictive performance. InProceedings of The 27th International Conference on Artificial Intelligence and Statistics, volume 238 ofProceedings of Machine Learning Research,...

    2024

  10. [2016]

    Automated data validation in machine learning systems.Bulletin of the IEEE Computer Society Technical Committee on Data Engineering,

    4Numeric:Outlier,WrongUnit,AddDelta Categorical:Typo,Replace,Extraneous [Biessmannet al., 2021 ] Felix Biessmann, Jacek Golebiowski, Tammo Rukat, Dustin Lange, and Philipp Schmidt. Automated data validation in machine learning systems.Bulletin of the IEEE Computer Society Technical Committee on Data Engineering,

    2021

  11. [2017]

    Openrefine: A free, open source, power tool for working with messy data, March

    [OpenRefine-Team, 2024] OpenRefine-Team. Openrefine: A free, open source, power tool for working with messy data, March

    2024

  12. [2018]

    [Rubin, 1976] Donald B. Rubin. Inference and missing data. Biometrika, 63(3):581–592,

    1976

  13. [2021]

    Mantovani, Jan N

    [Bischlet al., 2021 ] Bernd Bischl, Giuseppe Casalicchio, Matthias Feurer, Pieter Gijsbers, Frank Hutter, Michel Lang, Rafael G. Mantovani, Jan N. van Rijn, and Joaquin Vanschoren. OpenML benchmarking suites.Advances in Neural Information Processing Systems, 34:23236–23247,

    2021

  14. [2024]

    A Benchmark for Data Imputation Methods

    [J¨ageret al., 2021 ] Sebastian J¨ager, Arndt Allhorn, and Felix Bießmann. A Benchmark for Data Imputation Methods. Frontiers in Big Data, 4, July

    2021

  15. [2025]

    [ISO, 2015] ISO. Systems and software engineering — sys- tems and software quality requirements and evaluation (square) — measurement of system and software product quality — data quality measurement,

    2015

  16. [2026]

    [Gurupuret al., 2025 ] Varadraj Gurupur, Shadab Hoosh- mand, Deepthi F

    Ac- cessed: 2026-02-09. [Gurupuret al., 2025 ] Varadraj Gurupur, Shadab Hoosh- mand, Deepthi F. Prabhu, Emily Trader, and Shreya Salvi. Incompleteness of electronic health records: An im- pending process problem within healthcare.Healthcare, 13(22):2900, Nov

    2026