Pith. sign in

REVIEW 3 major objections 4 minor 82 references

Towards Next Generation Data Engineering Pipelines

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that next-generation data engineering pipelines can be made autonomous by combining optimized composition, continuous self-monitoring, and automatic adaptation.

desk verdict A coherent, honest vision paper for self-optimizing and self-adapting data pipelines, but the underspecified data-quality metric makes the central claim untestable for now. read the letter →

arxiv 2507.13892 v2 pith:ZQDOTVR6 submitted 2025-07-18 cs.DB

Kevin M. Kramer , Valerie Restat , Sebastian Strasser , Uta Störl , Meike Klettke This is my paper · ORCID
classification cs.DB
keywords dataengineeringpipelinesqualitypipelineoptimizationself-awareself-adaptiveprofilingschemaevolutionvalidation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that next-generation data engineering pipelines should be built in three escalating levels: an optimized pipeline that automatically composes cleaning operators to maximize data quality, a self-aware pipeline that continuously profiles input and intermediate data to detect changes, and a self-adapting pipeline that automatically adjusts operators and structure when significant structural or semantic change occurs. The authors claim that reaching all three levels yields an autonomous system that creates, monitors, and adapts pipelines so that high data quality is maintained even as upstream data changes schema or distribution. A sympathetic reader would care because data preparation consumes the majority of data science effort, and the paper offers a concrete architecture built on data profiles, profile diffs, pipeline profiles, and adapters for automating much of that work. The paper is explicitly a vision: each level carries open research challenges, most notably the absence of a standardized data quality metric.

What carries the argument

The central mechanism is the trio of profiles and their diffs: the data profile (a configurable set of descriptive statistics over schema, distributions, and property types), the error profile (a record of error types and positions, produced with the CheDDaR framework), and the pipeline profile (an abstract, technology-independent JSON description of operators, parameters, and ordering). Data profile diffs, error profile diffs, and pipeline profile diffs are the comparison layer that lets the system detect change, decide whether adaptation is needed, and record how the pipeline changed. The adaptation process itself follows a MAPE-K-style loop decomposed into change interpretation, adaptation analysis, and propagation and evaluation, with a schema version graph and schema modification operation inference for structural changes.

What would settle it

Construct a benchmark dataset and two cleaning pipelines that receive identical scores under the proposed data-quality metric but produce measurably different results in a downstream task, such as classification accuracy on the cleaned data; if such a pair exists, the premise that optimization for data quality is independent of the analysis objective fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that the gap between today's data engineering pipelines and fully autonomous ones can be closed by defining three levels of capability. Level 1, optimized pipelines, selects and orders cleaning operators (missing-value imputation, interval-violation repair, deduplication) from a large search space using rule-based pruning and cost-based search, with the objective of maximizing output data quality rather than fitting a downstream model. Level 2, self-aware pipelines, instruments every operator so that input, intermediate, and output data are summarized as data profiles; comparing profiles across batches yields profile diffs that expose distribution shifts, schema evolution, and new error patterns. Level 3, self-adapting pipelines, interprets those diffs as independent change steps, selects adaptation operations (reconfiguring an operator, renaming a property, replacing an algorithm, or rerunning full optimization when change is too drastic), propagates them through pipeline-profile diffs, and evaluates the result. The paper proposes a system architecture in which a profile registry and schema version graph store all metadata, and technology-specific adapters translate abstract pipeline profiles into executable pipelines.

Load-bearing premise

The whole vision rests on the existence of a data quality metric that can rank any pipeline's output independently of the downstream analysis and that can be computed over a large search space; the paper itself states that such a metric is still an open research question.

Editorial extensions

If this is right

  • If all three levels are realized, the resulting system can keep delivering high-quality data without human intervention when upstream data changes schema or value distribution.
  • Pipeline optimization independent of downstream analysis would let data-preparation teams produce reusable cleaned datasets for multiple analysis objectives.
  • Continuous profiling and diffing makes root-cause analysis of data errors and pipeline failures more targeted, and alerts data engineers to significant changes.
  • Self-adaptation, triggered by detected significant change, prevents pipeline crashes and semantically incorrect outputs caused by schema renames or distribution shifts.
  • The architecture's adapter-based design means the abstract pipeline profile can be deployed in different technologies, such as Python scripts or Airflow DAGs, without changing the monitoring or adaptation logic.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to benchmark whether a single data-quality metric can rank pipelines consistently across different downstream tasks; if two pipelines tie on data quality but differ on downstream performance, the independence premise needs refinement.
  • The three-level pyramid suggests a maturity model for data engineering tooling: existing orchestrators already provide Level 0, so vendors could incrementally add profiling and adaptation features rather than build a new system from scratch.
  • Data profile diffs could be reused beyond pipelines as a general change-detection primitive for data contracts between organizations, not just within one pipeline.
  • The use of large language models for schema-ambiguity resolution and contextualization of new properties is speculative but testable: comparing LLM-based adaptation choices against human expert choices on a benchmark of schema-evolution scenarios would give early evidence.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes a three-level vision for next-generation data engineering pipelines: (1) optimized pipelines that automatically select and order data-cleaning operators to maximize output data quality, (2) self-aware pipelines that continuously build data, error, and pipeline profiles and detect significant changes via profile diffs, and (3) self-adapting pipelines that interpret, plan, and propagate adaptations in response to structural and semantic changes such as schema renames or distribution shifts. The authors instantiate the vision with an eye-tracking running example, describe rule-based and cost-based optimization steps for Level 1, monitoring and profile abstractions for Level 2, a three-phase MAPE-K-like adaptation process for Level 3, and a system architecture centered on a profile registry and schema version graph. The paper explicitly enumerates open research challenges for each level and states that the data quality metric required for the optimization objective does not yet exist.

Significance. The vision is significant if realizable: it targets the well-documented cost of data preparation and the practical problem that pipelines degrade silently when upstream data evolves. The paper's strengths are its clean three-level taxonomy, the consistent running example, and the honest identification of the open problems, especially the absence of a data quality metric, which is correctly placed as the linchpin of both pipeline optimization and adaptation evaluation. The proposed profile/diff-based architecture provides a useful vocabulary for future work. At the same time, the paper is a position paper: no empirical or formal validation is provided, and the key feasibility determinants—a quality metric, formalized operator constraints, and semantic-correctness evaluation—are explicitly open. External evaluation of the prior components (CheDDaR, data profiles, ALPINE, SMO inference) is absent. The contribution is therefore a research agenda whose central claims can be accepted only conditionally on future results.

major comments (3)
  1. [Section 3.1.3 and 3.1.4] The paper states in Section 3.1.3 that 'A suitable metric is still an open research question' and in Section 3.1.4 that 'Such a metric does not yet exist.' This means the central objective of Level 1 is undefined. Because Level 3 also uses data quality to evaluate adaptations (Section 3.3.3) and the conclusion claims the system 'leads to the best possible data quality at all times,' the entire proposal hinges on a quantity that is not defined. Concretely, the paper should specify at least a family of usable quality metrics or a formal evaluation protocol and show how the cost-based optimizer would use it; otherwise the central claims are not falsifiable.
  2. [Section 3.1.1 and 3.1.4] The rule-based optimization step assumes that operator constraints such as 'mean value imputation → permitted data types: numerical' can be found, formalized, and efficiently linked to data profiles, but the paper itself identifies this as an open research question. This is load-bearing because the feasibility of the search-space reduction, and therefore of the subsequent cost-based optimization, depends on it. A minimal test would be a small constraint language and a benchmark showing that it prunes the search space without excluding optimal pipelines.
  3. [Section 3.3.3 and Section 4] The evaluation of self-adaptation is not well-defined. The paper states that evaluating functionality is trivial but semantic correctness is 'very hard,' and Section 4 concedes that probabilistic choices may need human judgment. Without ground truth or a probabilistic objective with explicit thresholds, the system cannot determine whether an adaptation improved data quality, making the Level 3 autonomy claim untestable. The authors should either define an evaluation protocol with injected changes and known ground truth or explicitly limit the automated claim to functional, non-semantic adaptation.
minor comments (4)
  1. [Section 3.1] The search-space formula 'PN i=0 xi!' in Section 3.1 is not readable; based on the 11! = 39,916,800 example, the intended count appears to be the number of permutations of the selected operators, so the formula should be written as (Σ x_i)! or equivalent and introduced clearly.
  2. [Section 3.3.2] The text 'as presented in Scetion 3.1.4' contains a typo and should read 'Section 3.1.4'.
  3. [Section 3.3] The phrase 'We envision diverse types offailures' should be 'We envision diverse types of failures.'
  4. [Section 3.2.2] The sentence 'An example would if a property which yielded no missing values at design time suddenly does deliver any values' is missing a word after 'would' and 'any' is ambiguous; it should likely read 'An example would be if a property ... suddenly does deliver values.'

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is a vision/position paper that openly defers its load-bearing evaluation metric and adaptation-correctness checks to future research, and no claimed prediction reduces to its inputs by construction.

full rationale

The paper contains no derivation chain whose output is equivalent to its input by construction. Its central claim is a proposed three-level roadmap (optimized, self-aware, self-adapting pipelines) with the components explicitly labeled as research challenges rather than completed results. The metric needed to rank pipelines is admittedly missing: Section 3.1.3 states 'A suitable metric is still an open research question' and Section 3.1.4 repeats that 'Such a metric does not yet exist.' Likewise, Section 3.3.3 admits that 'evaluating the semantic correctness is very hard' after an adaptation. These are openly stated limitations, not circular reductions. The paper does rely on several self-citations for component ideas (CheDDaR, data profile diffs, SMO inference, ALPINE pipeline profiles, the decision tree for schema robustness), but in each case the citation supplies a proposed building block or prior concept, not a result that is then re-labeled as a prediction of this paper. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors' prior work to forbid alternatives, and no known empirical pattern is repackaged solely under new coordinates. The absence of external benchmarks is a weakness in empirical support, but the hard rule for circularity requires exhibiting a specific reduction to the paper's own inputs, and none is present. The correct verdict is therefore no significant circularity.

Assumptions & free parameters 0 free parameters · 5 assumptions · 6 invented entities

The paper relies on several domain assumptions that are explicitly or implicitly load-bearing. No free parameters are fitted because there are no empirical analyses. The invented entities are conceptual abstractions for the proposed system; none has independent evidence because nothing is implemented or externally validated.

assumptions (5)
  • domain assumption A single data quality metric exists and can rank pipeline outputs independently of downstream analysis.
    Section 3.1.3: 'A suitable metric is still an open research question.' The optimizer's objective function depends on this metric, so the whole Level 1 approach assumes it can be defined.
  • domain assumption Operator constraints and best practices can be formalized and efficiently linked to data profiles.
    Section 3.1.4 lists this as an open research question: 'constraints such as algorithm: mean value imputation -> permitted data types: numerical must be identified' and 'an efficient way must be found to link these constraints with the data profiles'.
  • domain assumption Significant changes in data and operators are reliably detectable by comparing data profiles and diffs.
    Section 3.2 describes the monitoring approach conceptually but provides no sensitivity/specificity evidence; thresholding of 'significant change' is not specified.
  • domain assumption Automatic adaptations preserve semantic correctness.
    Section 3.3.3: 'evaluating the semantic correctness is very hard' and results are 'more or less probable'. The Level 3 vision assumes this can be made reliable.
  • domain assumption The pipeline search space can be pruned to a tractable size while still containing the global optimum.
    Sections 3.1.1-3.1.3 assume rule-based pruning, best practices, and cost-based optimization jointly make the search tractable; no completeness guarantee is given.
invented entities (6)
  • Data profile (DP)
    purpose: A configurable JSON metadata summary of a dataset (schema, types, statistics) used for optimization and monitoring.
    Introduced in Sections 3.1 and 3.2, building on prior work by the same group (Strasser and Klettke 2024). No implementation or benchmark outside the paper.
  • Error profile (EP)
    purpose: Records detected errors (type and position) used to select cleaning operators.
    Created with the authors' own CheDDaR framework (Restat et al. 2023; Diestelkämper et al. 2025); no external validation.
  • Pipeline profile (PP)
    purpose: Abstract JSON description of the selected pipeline (operators, parameters, order), enabling technology-independent deployment and adaptation.
    Proposed in the authors' ALPINE work (Restat and Störl 2025); not independently reproduced.
  • Profile diff (DPD/EPD/PPD)
    purpose: Delta between two profiles used to detect significant change and to document adaptations.
    Introduced in Strasser and Klettke 2024; no independent evaluation.
  • Schema version graph
    purpose: Tracks schema evolution across batches via inferred schema modification operations, enabling reuse or adaptation decisions.
    Builds on the authors' prior work (Klettke et al. 2017); no independent evidence of reliability.
  • Profile registry
    purpose: Central storage of all profiles and diffs to enable provenance, drift tracking, and adaptation.
    System component introduced in Section 4; not implemented.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Next Generation Data Engineering Pipelines." pith.science (2026). https://pith.science/paper/ZQDOTVR6

@misc{pith2026250713892,
  author       = {Pith},
  title        = {Pith review of: Towards Next Generation Data Engineering Pipelines},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZQDOTVR6}},
  note         = {Machine review of arXiv:2507.13892}
}
read the original abstract

Data engineering pipelines are a widespread way to provide high-quality data for all kinds of data science applications. However, numerous challenges still remain in the composition and operation of such pipelines. Data engineering pipelines do not always deliver high-quality data. By default, they are also not reactive to changes. When new data is coming in which deviates from prior data, the pipeline could crash or output undesired results. We therefore envision three levels of next generation data engineering pipelines: optimized data pipelines, self-aware data pipelines, and self-adapting data pipelines. Pipeline optimization addresses the composition of operators and their parametrization in order to achieve the highest possible data quality. Self-aware data engineering pipelines enable a continuous monitoring of its current state, notifying data engineers on significant changes. Self-adapting data engineering pipelines are then even able to automatically react to those changes. We propose approaches to achieve each of these levels.

Figures

Figures reproduced from arXiv: 2507.13892 by the authors.

Figure 1
Figure 1. Different levels of next generation data engineering pipelines [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Exemplary selection of the eye tracking data, errors (missing values and interval violations) are shaded in red [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Semantic change: The distributions of the values of [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Structural change: The properties Fixation-X and Fixation-Y are now called Fixation-Screen-X and Fixation￾Screen-Y (Section 3.2), and self-adapting (Section 3.3) data engineering pipelines can be conceptualized and implemented. We also present an architecture and a tec…
Figure 5
Figure 5. Figure 5: Overview of the proposed vision Level 2 (orange eyes): Once the optimized pipeline has been found and deployed, it must continuously monitor itself, thereby becoming self-aware. On the one hand, this concerns the changes made by the individual operators in the pipeline…
Figure 6
Figure 6. Figure 6: The dashed lines indicate the search space. The solid green line represents the best pipeline. Since an operator [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 6
Figure 6. Figure 6: Automatic composition of an optimized data engineering pipeline. Optimized refers to the pipeline resulting [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Overview of the process for determining an optimized data engineering pipeline [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Data profile for the input dataset of our running example [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 9
Figure 9. Figure 9: Integration of a data monitoring component into the pipeline. The orange symbols indicate the observation (= [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Self-adapting data engineering pipeline which automatically reacts to change over time (blue input data) by [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Overview of the proposed system and its components [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

82 extracted references · 44 canonical work pages

  1. [1]

    Four Generations in Data Engineering for Data Science

    Meike Klettke and Uta St \" o rl. Four Generations in Data Engineering for Data Science . Datenbank-Spektrum, 22 0 (1): 0 59--66, 2022. doi:10.1007/S13222-021-00399-3

    work page doi:10.1007/s13222-021-00399-3 2022

  2. [2]

    The Challenges of Data Quality and Data Quality Assessment in the Big Data Era

    Li Cai and Yangyong Zhu. The Challenges of Data Quality and Data Quality Assessment in the Big Data Era . Data Sci. J., 14: 0 2, 2015. doi:10.5334/DSJ-2015-002

    work page doi:10.5334/dsj-2015-002 2015

  3. [3]

    GouDa - generation of universal data sets: improving analysis and evaluation of data preparation pipelines

    Valerie Restat, Gerrit Boerner, Andr \' e Conrad, and Uta St \" o rl. GouDa - generation of universal data sets: improving analysis and evaluation of data preparation pipelines . In DEEM@SIGMOD, pages 2:1--2:6. ACM , 2022. doi:10.1145/3533028.3533311

    arXiv 2022

  4. [4]

    Ilyas and Xu Chu

    Ihab F. Ilyas and Xu Chu. Data Cleaning , volume 28 of ACM Books . ACM , 2019. doi:10.1145/3310205

    work page doi:10.1145/3310205 2019

  5. [5]

    DERM: A reference model for data engineering

    Daniel Tebernum, Marcel Altendeitering, and Falk Howar. DERM: A reference model for data engineering. In DATA , pages 165--175. SCITEPRESS , 2021. doi:10.5220/0010517301650175

    work page doi:10.5220/0010517301650175 2021

  6. [6]

    A Survey of Big Data Pipeline Orchestration Tools from the Perspective of the DataCloud Project

    Mihhail Matskin, Shirin Tahmasebi, Amirhossein Layegh, Amir Hossein Payberah, Aleena Thomas, Nikolay Nikolov, and Dumitru Roman. A Survey of Big Data Pipeline Orchestration Tools from the Perspective of the DataCloud Project . In DAMDID/RCDL (Supplementary Proceedings) , volume 3036 of CEUR Workshop Proceedings , pages 63--78. CEUR-WS.org, 2021. URL https...

    2021

  7. [7]

    Self-Awareness as a Prerequisite for Self-Adaptivity in Computing Systems

    Ana Petrovska. Self-Awareness as a Prerequisite for Self-Adaptivity in Computing Systems . In ACSOS-C , pages 146--149. IEEE , 2021. doi:10.1109/ACSOS-C52956.2021.00039

    arXiv 2021

  8. [8]

    Data Processing Pipeline for Eye-Tracking Analysis

    Jennifer Landes, Sonja K \" o ppl, and Meike Klettke. Data Processing Pipeline for Eye-Tracking Analysis . In GvDB, volume 3710 of CEUR Workshop Proceedings , pages 35--42. CEUR-WS.org, 2024. URL https://ceur-ws.org/Vol-3710/paper6.pdf

    2024

Show all 82 references
  1. [9]

    A unified representation and transformation of multi-model data using category theory

    Pavel Koupil and Irena Holubov \' a . A unified representation and transformation of multi-model data using category theory . J. Big Data, 9 0 (1): 0 61, 2022. doi:10.1186/S40537-022-00613-3

    2022 doi

  2. [10]

    Data Engineering for Data Science: Two Sides of the Same Coin

    Oscar Romero and Robert Wrembel. Data Engineering for Data Science: Two Sides of the Same Coin . In DaWaK, volume 12393 of Lecture Notes in Computer Science, pages 157--166. Springer, 2020. doi:10.1007/978-3-030-59065-9\_13

    2020 doi

  3. [11]

    The Art and Practice of Data Science Pipelines: A Comprehensive Study of Data Science Pipelines In Theory, In-The-Small, and In-The-Large

    Sumon Biswas, Mohammad Wardat, and Hridesh Rajan. The Art and Practice of Data Science Pipelines: A Comprehensive Study of Data Science Pipelines In Theory, In-The-Small, and In-The-Large . In ICSE , pages 2091--2103. ACM , 2022. doi:10.1145/3510003.3510057

    2022

  4. [12]

    What About the Data? A Mapping Study on Data Engineering for AI Systems

    Petra Heck. What About the Data? A Mapping Study on Data Engineering for AI Systems . In CAIN , pages 43--52. ACM , 2024. doi:10.1145/3644815.3644954

    2024

  5. [13]

    NIST Big Data Interoperability Framework: Volume 1, Definitions

    Wo Chang and Nancy Grady. NIST Big Data Interoperability Framework: Volume 1, Definitions . Technical report, National Institute of Standards and Technology, 2019

    2019

  6. [14]

    Challenging Big Data Engineering: Positioning of Current and Future Development

    Matthias Volk, Daniel Staegemann, Matthias Pohl, and Klaus Turowski. Challenging Big Data Engineering: Positioning of Current and Future Development . In IoTBDS, pages 351--358. SciTePress, 2019. doi:10.5220/0007748803510358

    2019 doi

  7. [15]

    Atkinson, Michelle Galea, Tan Fong Ang, Paul Martin, and Jano I

    Chee Sun Liew, Malcolm P. Atkinson, Michelle Galea, Tan Fong Ang, Paul Martin, and Jano I. van Hemert. Scientific Workflows: Moving Across Paradigms . ACM Comput. Surv. , 49 0 (4): 0 66:1--66:39, 2017. doi:10.1145/3012429

    2017 doi

  8. [16]

    Mahadi Hassan, Micah J

    Shubhra Kanti Karmaker Santu, Md. Mahadi Hassan, Micah J. Smith, Lei Xu, Chengxiang Zhai, and Kalyan Veeramachaneni. AutoML to Date and Beyond: Challenges and Opportunities . ACM Comput. Surv. , 54 0 (8): 0 175:1--175:36, 2022. doi:10.1145/3470918

    2022 doi

  9. [17]

    Parameswaran

    Doris Xin, Eva Yiwei Wu, Doris Jung Lin Lee, Niloufar Salehi, and Aditya G. Parameswaran. Whither AutoML ? Understanding the Role of Automation in Machine Learning Workflows . In CHI , pages 83:1--83:16. ACM , 2021. doi:10.1145/3411764.3445306

    2021

  10. [18]

    Franklin, Ken Goldberg, and Eugene Wu

    Sanjay Krishnan, Michael J. Franklin, Ken Goldberg, and Eugene Wu. BoostClean: Automated Error Detection and Repair for Machine Learning . CoRR, abs/1711.01299, 2017. URL http://arxiv.org/abs/1711.01299

    2017 arXiv

  11. [19]

    Ilyas, and Christopher R \' e

    Theodoros Rekatsinas, Xu Chu, Ihab F. Ilyas, and Christopher R \' e . HoloClean: Holistic Data Repairs with Probabilistic Inference . Proc. VLDB Endow. , 10 0 (11): 0 1190--1201, 2017. doi:10.14778/3137628.3137631

    2017

  12. [20]

    Baran: Effective Error Correction via a Unified Context Representation and Transfer Learning

    Mohammad Mahdavi and Ziawasch Abedjan. Baran: Effective Error Correction via a Unified Context Representation and Transfer Learning . Proc. VLDB Endow. , 13 0 (11): 0 1948--1961, 2020. URL http://www.vldb.org/pvldb/vol13/p1948-mahdavi.pdf

    1948

  13. [21]

    Ilyas, Mourad Ouzzani, Paolo Papotti, Nan Tang, and Yin Ye

    Xu Chu, John Morcos, Ihab F. Ilyas, Mourad Ouzzani, Paolo Papotti, Nan Tang, and Yin Ye. KATARA: A Data Cleaning System Powered by Knowledge Bases and Crowdsourcing . In SIGMOD Conference , pages 1247--1261. ACM , 2015. doi:10.1145/2723372.2749431

    2015

  14. [22]

    Data Preparation: A Survey of Commercial Tools

    Mazhar Hameed and Felix Naumann. Data Preparation: A Survey of Commercial Tools . SIGMOD Rec. , 49 0 (3): 0 18--29, 2020. doi:10.1145/3444831.3444835

    2020

  15. [23]

    Ilyas, Mourad Ouzzani, Paolo Papotti, Michael Stonebraker, and Nan Tang

    Ziawasch Abedjan, Xu Chu, Dong Deng, Raul Castro Fernandez, Ihab F. Ilyas, Mourad Ouzzani, Paolo Papotti, Michael Stonebraker, and Nan Tang. Detecting Data Errors: Where are we and what needs to be done? Proc. VLDB Endow. , 9 0 (12): 0 993--1004, 2016. doi:10.14778/2994509.2994518

    2016

  16. [24]

    SAGA: A Scalable Framework for Optimizing Data Cleaning Pipelines for Machine Learning Applications

    Shafaq Siddiqi, Roman Kern, and Matthias Boehm. SAGA: A Scalable Framework for Optimizing Data Cleaning Pipelines for Machine Learning Applications . Proc. ACM Manag. Data , 1 0 (3): 0 218:1--218:26, 2023. doi:10.1145/3617338

    2023 doi

  17. [25]

    Lindstaedt, Arnab Phani, Benjamin Rath, Berthold Reinwald, Shafaq Siddiqui, and Sebastian Benjamin Wrede

    Matthias Boehm, Iulian Antonov, Sebastian Baunsgaard, Mark Dokter, Robert Ginth \" o r, Kevin Innerebner, Florijan Klezin, Stefanie N. Lindstaedt, Arnab Phani, Benjamin Rath, Berthold Reinwald, Shafaq Siddiqui, and Sebastian Benjamin Wrede. SystemDS : A Declarative Machine Lea...

    2020

  18. [26]

    From Cleaning before ML to Cleaning for ML

    Felix Neutatz, Binger Chen, Ziawasch Abedjan, and Eugene Wu. From Cleaning before ML to Cleaning for ML . IEEE Data Eng. Bull. , 44 0 (1): 0 24--41, 2021. URL http://sites.computer.org/debull/A21mar/p24.pdf

    2021

  19. [27]

    Paritosh, and Lora Aroyo

    Nithya Sambasivan, Shivani Kapania, Hannah Highfill, Diana Akrong, Praveen K. Paritosh, and Lora Aroyo. Everyone wants to do the model work, not the data work": Data Cascades in High-Stakes AI . In CHI , pages 39:1--39:15. ACM , 2021. doi:10.1145/3411764.3445518

    2021

  20. [28]

    Wang, Henry B

    Richard Y. Wang, Henry B. Kon, and Stuart E. Madnick. Data Quality Requirements Analysis and Modeling . In ICDE , pages 670--677. IEEE Computer Society, 1993. doi:10.1109/ICDE.1993.344012

    1993

  21. [29]

    Wang, Veda C

    Richard Y. Wang, Veda C. Storey, and Christopher P. Firth. A Framework for Analysis of Data Quality Research . IEEE Trans. Knowl. Data Eng. , 7 0 (4): 0 623--640, 1995. doi:10.1109/69.404034

    1995 doi

  22. [30]

    Blake and Paul Mangiameli

    Roger H. Blake and Paul Mangiameli. The Effects and Interactions of Data Quality and Problem Complexity on Classification . ACM J. Data Inf. Qual. , 2 0 (2): 0 8:1--8:28, 2011. doi:10.1145/1891879.1891881

    2011

  23. [31]

    Requirements for Data Quality Metrics

    Bernd Heinrich, Diana Hristova, Mathias Klier, Alexander Schiller, and Michael Szubartowicz. Requirements for Data Quality Metrics . ACM J. Data Inf. Qual. , 9 0 (2): 0 12:1--12:32, 2018. doi:10.1145/3148238

    2018 doi

  24. [32]

    Automating Large-Scale Data Quality Verification

    Sebastian Schelter, Dustin Lange, Philipp Schmidt, Meltem Celikel, Felix Bie mann, and Andreas Grafberger. Automating Large-Scale Data Quality Verification . Proc. VLDB Endow. , 11 0 (12): 0 1781--1794, 2018. doi:10.14778/3229863.3229867

    2018

  25. [33]

    An Advanced Big Data Quality Framework Based on Weighted Metrics

    Widad Elouataoui, Imane El Alaoui, Saida El Mendili, and Youssef Gahi. An Advanced Big Data Quality Framework Based on Weighted Metrics . Big Data Cogn. Comput., 6 0 (4): 0 153, 2022. doi:10.3390/BDCC6040153

    2022 doi

  26. [34]

    Ilyas, and Theodoros Rekatsinas

    Alireza Heidari, Joshua McGrath, Ihab F. Ilyas, and Theodoros Rekatsinas. HoloDetect: Few-Shot Learning for Error Detection . In SIGMOD Conference , pages 829--846. ACM , 2019. doi:10.1145/3299869.3319888

    2019

  27. [35]

    Raha: A Configuration-Free Error Detection System

    Mohammad Mahdavi, Ziawasch Abedjan, Raul Castro Fernandez, Samuel Madden, Mourad Ouzzani, Michael Stonebraker, and Nan Tang. Raha: A Configuration-Free Error Detection System . In SIGMOD Conference , pages 865--882. ACM , 2019. doi:10.1145/3299869.3324956

    2019

  28. [36]

    Exploratory Training: When Annonators Learn About Data

    Rajesh Shrestha, Omeed Habibelahian, Arash Termehchy, and Paolo Papotti. Exploratory Training: When Annonators Learn About Data . Proc. ACM Manag. Data , 1 0 (2): 0 135:1--135:25, 2023. doi:10.1145/3589280

    2023 doi

  29. [37]

    Visual Interactive Creation, Customization, and Analysis of Data Quality Metrics

    Christian Bors, Theresia Gschwandtner, Simone Kriglstein, Silvia Miksch, and Margit Pohl. Visual Interactive Creation, Customization, and Analysis of Data Quality Metrics . ACM J. Data Inf. Qual. , 10 0 (1): 0 3:1--3:26, 2018. doi:10.1145/3190578

    2018 doi

  30. [38]

    o rl. FAIR is not enough - A Metrics Framework to ensure Data Quality through Data Preparation . In BTW , volume P-331 of LNI , pages 917--929. Gesellschaft f \

    Valerie Restat, Meike Klettke, and Uta St \" o rl. FAIR is not enough - A Metrics Framework to ensure Data Quality through Data Preparation . In BTW , volume P-331 of LNI , pages 917--929. Gesellschaft f \" u r Informatik e.V., 2023. doi:10.18420/BTW2023-61

    2023 doi

  31. [39]

    a mper, Ralf Diestelk \

    Indra Diestelk \" a mper, Ralf Diestelk \" a mper, and Valerie Restat. CheDDaR: Checking Data - Data Quality Report . In BTW , volume P-361 of LNI , pages 1055--1067. Gesellschaft f \" u r Informatik e.V., 2025. doi:10.18420/BTW2025-70

    2025 doi

  32. [40]

    noWorkflow: Capturing and Analyzing Provenance of Scripts

    Leonardo Murta, Vanessa Braganholo, Fernando Chirigati, David Koop, and Juliana Freire. noWorkflow: Capturing and Analyzing Provenance of Scripts . In IPAW , volume 8628 of Lecture Notes in Computer Science, pages 71--83. Springer, 2014. doi:10.1007/978-3-319-16462-5\_6

    2014 doi

  33. [41]

    Capturing and querying fine-grained provenance of preprocessing pipelines in data science

    Adriane Chapman, Paolo Missier, Giulia Simonelli, and Riccardo Torlone. Capturing and querying fine-grained provenance of preprocessing pipelines in data science. Proc. VLDB Endow. , 14 0 (4): 0 507--520, 2020. doi:10.14778/3436905.3436911

    2020

  34. [42]

    Data distribution debugging in machine learning pipelines

    Stefan Grafberger, Paul Groth, Julia Stoyanovich, and Sebastian Schelter. Data distribution debugging in machine learning pipelines . VLDB J. , 31 0 (5): 0 1103--1126, 2022. doi:10.1007/S00778-021-00726-W

    2022 doi

  35. [43]

    Fair preprocessing: towards understanding compositional fairness of data transformers in machine learning pipeline

    Sumon Biswas and Hridesh Rajan. Fair preprocessing: towards understanding compositional fairness of data transformers in machine learning pipeline . In ESEC/SIGSOFT FSE , pages 981--993. ACM , 2021. doi:10.1145/3468264.3468536

    2021

  36. [44]

    Towards Explaining the Effects of Data Preprocessing on Machine Learning

    Carlos Vladimiro Gonzalez Zelaya. Towards Explaining the Effects of Data Preprocessing on Machine Learning . In ICDE , pages 2086--2090. IEEE , 2019. doi:10.1109/ICDE.2019.00245

    2019

  37. [45]

    Dead or Alive: Continuous Data Profiling for Interactive Data Science

    Will Epperson, Vaishnavi Gorantla, Dominik Moritz, and Adam Perer. Dead or Alive: Continuous Data Profiling for Interactive Data Science . IEEE Trans. Vis. Comput. Graph. , 30 0 (1): 0 197--207, 2024. doi:10.1109/TVCG.2023.3327367

    2024

  38. [46]

    Data Profiling

    Ziawasch Abedjan. Data Profiling . In Encyclopedia of Big Data Technologies. Springer, 2019. doi:10.1007/978-3-319-63962-8\_8-1

    2019 doi

  39. [48]

    Auto-Validate by-History: Auto-Program Data Quality Constraints to Validate Recurring Data Pipelines

    Dezhan Tu, Yeye He, Weiwei Cui, Song Ge, Haidong Zhang, Shi Han, Dongmei Zhang, and Surajit Chaudhuri. Auto-Validate by-History: Auto-Program Data Quality Constraints to Validate Recurring Data Pipelines . In KDD , pages 4991--5003. ACM , 2023. doi:10.1145/3580305.3599776

    2023

  40. [49]

    Parameswaran

    Shreya Shankar, Labib Fawaz, Karl Gyllstrom, and Aditya G. Parameswaran. Automatic and Precise Data Validation for Machine Learning . In CIKM , pages 2198--2207. ACM , 2023. doi:10.1145/3583780.3614786

    2023

  41. [50]

    A DaQL to Monitor Data Quality in Machine Learning Applications

    Lisa Ehrlinger, Verena Haunschmid, Davide Palazzini, and Christian Lettner. A DaQL to Monitor Data Quality in Machine Learning Applications . In DEXA (1) , volume 11706 of Lecture Notes in Computer Science, pages 227--237. Springer, 2019. doi:10.1007/978-3-030-27615-7\_17

    2019 doi

  42. [51]

    DaQL 2.0: Measure Data Quality based on Entity Models

    Christian Lettner, Reinhard Stumptner, Werner Fragner, Franz Rauchenzauner, and Lisa Ehrlinger. DaQL 2.0: Measure Data Quality based on Entity Models . In ISM , volume 180 of Procedia Computer Science, pages 772--777. Elsevier, 2020. doi:10.1016/J.PROCS.2021.01.327

    2020 doi

  43. [52]

    Real-Time Monitoring of Data Pipelines: Exploring and Experimentally Proving that the Continuous Monitoring in Data Pipelines Reduces Cost and Elevates Quality

    Shammy Narayanan, Maheswari S, and Prisha Zephan. Real-Time Monitoring of Data Pipelines: Exploring and Experimentally Proving that the Continuous Monitoring in Data Pipelines Reduces Cost and Elevates Quality . ICST Transactions on Scalable Information Systems, 2024

    2024

  44. [53]

    Schuler, Jitin Singla, Brinda Vallat, Kate L

    Robert E. Schuler, Jitin Singla, Brinda Vallat, Kate L. White, Helen M. Berman, and Carl Kesselman. Database Evolution, by Scientists, for Scientists: A Case Study . In e-Science, pages 1--10. IEEE , 2023. doi:10.1109/E-SCIENCE58273.2023.10254872

    2023

  45. [54]

    An Empirical Study on the Design and Evolution of NoSQL Database Schemas

    Stefanie Scherzinger and Sebastian Sidortschuck. An Empirical Study on the Design and Evolution of NoSQL Database Schemas . In ER , volume 12400 of Lecture Notes in Computer Science, pages 441--455. Springer, 2020. doi:10.1007/978-3-030-62522-1\_33

    2020 doi

  46. [55]

    Schema Evolution in Wikipedia - Toward a Web Information System Benchmark

    Carlo Curino, Hyun Jin Moon, Letizia Tanca, and Carlo Zaniolo. Schema Evolution in Wikipedia - Toward a Web Information System Benchmark . In ICEIS (1) , pages 323--332, 2008

    2008

  47. [56]

    Self-healing and self-repairing technologies

    Regina Frei, Richard McWilliam, Benjamin Derrick, Alan Purvis, Asutosh Tiwari, and Giovanna Di Marzo Serugendo. Self-healing and self-repairing technologies . The International Journal of Advanced Manufacturing Technology, 69: 0 1033--1061, 2013. URL https://api.semanticschola...

    2013

  48. [57]

    DeBinelle: Semantic Patches for Coupled Database-Application Evolution

    Stefanie Scherzinger, Wolfgang Mauerer, and Haridimos Kondylakis. DeBinelle: Semantic Patches for Coupled Database-Application Evolution . In ICDE , pages 2697--2700. IEEE , 2021. doi:10.1109/ICDE51399.2021.00307

    2021

  49. [58]

    Schuler and Carl Kesselman

    Robert E. Schuler and Carl Kesselman. Managing Database-Application Co-Evolution in a Scientific Data Ecosystem . In e-Science, pages 214--224. IEEE , 2022. doi:10.1109/ESCIENCE55777.2022.00035

    2022

  50. [59]

    Understanding Uncertainty in Self-adaptive Systems

    Radu Calinescu, Raffaela Mirandola, Diego Perez - Palacin, and Danny Weyns. Understanding Uncertainty in Self-adaptive Systems . In ACSOS , pages 242--251. IEEE , 2020. doi:10.1109/ACSOS49614.2020.00047

    2020

  51. [60]

    Uncertainty in Self-Adaptive Systems: A Research Community Perspective

    Sara Mahdavi - Hezavehi, Danny Weyns, Paris Avgeriou, Radu Calinescu, Raffaela Mirandola, and Diego Perez - Palacin. Uncertainty in Self-Adaptive Systems: A Research Community Perspective . CoRR, abs/2103.02717, 2021. URL https://arxiv.org/abs/2103.02717

    2021 arXiv

  52. [61]

    o rl, Daniel M \

    Meike Klettke, Hannes Awolin, Uta St \" o rl, Daniel M \" u ller, and Stefanie Scherzinger. Uncovering the Evolution History of Data Lakes . In IEEE BigData , pages 2462--2471. IEEE Computer Society, 2017. doi:10.1109/BIGDATA.2017.8258204

    2017

  53. [62]

    Reducing Ambiguity in Json Schema Discovery

    William Spoth, Oliver Kennedy, Ying Lu, Beda Christoph Hammerschmidt, and Zhen Hua Liu. Reducing Ambiguity in Json Schema Discovery . In SIGMOD Conference , pages 1732--1744. ACM , 2021. doi:10.1145/3448016.3452801

    2021

  54. [63]

    Fu and Xuewei Chen

    Silvery D. Fu and Xuewei Chen. Compound Schema Registry . CoRR, abs/2406.11227, 2024. doi:10.48550/ARXIV.2406.11227

    2024 doi arXiv

  55. [64]

    Control-Theoretical Software Adaptation: A Systematic Literature Review

    Stepan Shevtsov, Mihaly Berekmeri, Danny Weyns, and Martina Maggio. Control-Theoretical Software Adaptation: A Systematic Literature Review . IEEE Trans. Software Eng. , 44 0 (8): 0 784--810, 2018. doi:10.1109/TSE.2017.2704579

    2018

  56. [65]

    Agent-Driven Automatic Software Improvement

    Fernando Vallecillos Ruiz. Agent-Driven Automatic Software Improvement . In EASE , pages 470--475. ACM , 2024. doi:10.1145/3661167.3661171

    2024

  57. [66]

    Fuzzy Self-Adaptation of Mission-Critical Software Under Uncertainty

    Qiliang Yang, Jian Lu, XianPing Tao, Xiaoxing Ma, Jianchun Xing, and Wei Song. Fuzzy Self-Adaptation of Mission-Critical Software Under Uncertainty . J. Comput. Sci. Technol., 28 0 (1): 0 165--187, 2013. doi:10.1007/S11390-013-1321-9

    2013 doi

  58. [67]

    Towards Evolution Capabilities in Data Pipelines

    Kevin Kramer. Towards Evolution Capabilities in Data Pipelines . In GvDB, volume 3714 of CEUR Workshop Proceedings . CEUR-WS.org, 2023. URL https://ceur-ws.org/Vol-3714/paper7.pdf

    2023

  59. [68]

    Franklin, and Eugene Wu

    Sanjay Krishnan, Daniel Haas, Michael J. Franklin, and Eugene Wu. Towards reliable interactive data cleaning: a user survey and recommendations . In HILDA@SIGMOD, page 9. ACM , 2016. doi:10.1145/2939502.2939511

    2016

  60. [69]

    What can Data-Centric AI Learn from Data and ML Engineering? CoRR, abs/2112.06439, 2021

    Neoklis Polyzotis and Matei Zaharia. What can Data-Centric AI Learn from Data and ML Engineering? CoRR, abs/2112.06439, 2021. URL https://arxiv.org/abs/2112.06439

    2021 arXiv

  61. [70]

    Johnson, Gyorgy Simon, and Constantin Aliferis

    Steven G. Johnson, Gyorgy Simon, and Constantin Aliferis. Data Preparation, Transforms, Quality, and Management, pages 377--413. Springer International Publishing, 2024. doi:10.1007/978-3-031-39355-6_8

    2024 doi

  62. [71]

    Towards an End-to-End Data Quality Optimizer

    Valerie Restat, Meike Klettke, and Uta St \" o rl. Towards an End-to-End Data Quality Optimizer . In ICDEW , pages 262--266. IEEE , 2024. doi:10.1109/ICDEW61823.2024.00039

    2024

  63. [72]

    Empirical comparison of supervised learning techniques for missing value imputation

    Chih - Fong Tsai and Ya - Han Hu. Empirical comparison of supervised learning techniques for missing value imputation . Knowl. Inf. Syst., 64 0 (4): 0 1047--1075, 2022. doi:10.1007/S10115-022-01661-0

    2022 doi

  64. [73]

    Kamrul Hasan, Md

    Md. Kamrul Hasan, Md. Ashraful Alam, Shidhartho Roy, Aishwariya Dutta, Md. Tasnim Jawad, and Sunanda Das. Missing value imputation affects the performance of machine learning: A review and analysis of the literature (2010–2021) . Informatics in Medicine Unlocked, 27: 0 100799, 2021

    2010

  65. [74]

    o rl. ALPINE: Abstract Language for Pipeline Integration and Execution . In BTW Workshops , volume P-363 of LNI , pages 207--217. Gesellschaft f \

    Valerie Restat and Uta St \" o rl. ALPINE: Abstract Language for Pipeline Integration and Execution . In BTW Workshops , volume P-363 of LNI , pages 207--217. Gesellschaft f \" u r Informatik e.V., 2025. doi:10.18420/BTW2025-125

    2025 doi

  66. [75]

    Towards machine learning-aware data validation

    Sebastian Strasser. Towards machine learning-aware data validation . In GvDB, volume 3710 of CEUR Workshop Proceedings , pages 29--34. CEUR-WS.org, 2024. URL https://ceur-ws.org/Vol-3710/paper5.pdf

    2024

  67. [76]

    Transparent Data Preprocessing for Machine Learning

    Sebastian Strasser and Meike Klettke. Transparent Data Preprocessing for Machine Learning . In HILDA@SIGMOD, pages 1--6. ACM , 2024. doi:10.1145/3665939.3665960

    2024

  68. [77]

    Breunig, Hans - Peter Kriegel, Raymond T

    Markus M. Breunig, Hans - Peter Kriegel, Raymond T. Ng, and J \" o rg Sander. LOF: Identifying Density-Based Local Outliers . In SIGMOD Conference , pages 93--104. ACM , 2000. doi:10.1145/342009.335388

  69. [78]

    Kramer, Valerie Restat, and Uta St \" o rl

    Kevin M. Kramer, Valerie Restat, and Uta St \" o rl. Evolving gracefully: Building robust and self-adaptive data cleaning pipelines for schema evolution and uncertainty. In VLDB Workshops . VLDB.org, 2025. Accepted for publication

    2025

  70. [79]

    Kephart and David M

    Jeffrey O. Kephart and David M. Chess. The Vision of Autonomic Computing . Computer, 36 0 (1): 0 41--50, 2003. doi:10.1109/MC.2003.1160055

    2003 arXiv

  71. [80]

    A Survey on Application of Knowledge Graph

    Xiaohan Zou. A Survey on Application of Knowledge Graph . Journal of Physics: Conference Series, 1487, 2020. URL https://api.semanticscholar.org/CorpusID:216342059

    2020

  72. [81]

    Michael J. Mior. Large Language Models for JSON Schema Discovery . CoRR, abs/2407.03286, 2024. doi:10.48550/ARXIV.2407.03286

    2024 doi arXiv

  73. [82]

    Towards Taming the Adaptivity Problem - Formalizing Poly-/MultiStore Topology Descriptions

    Daniel Glake, Felix Kiehn, Mareike Schmidt, and Norbert Ritter. Towards Taming the Adaptivity Problem - Formalizing Poly-/MultiStore Topology Descriptions . In SummerSOC, volume 1429 of Communications in Computer and Information Science, pages 83--99. Springer, 2021. doi:10.10...

    2021 doi

  74. [83]

    Schema Extraction and Structural Outlier Detection for JSON-based NoSQL Data Stores

    Meike Klettke, Uta St \" o rl, and Stefanie Scherzinger. Schema Extraction and Structural Outlier Detection for JSON-based NoSQL Data Stores . In BTW , volume P-241 of LNI , pages 425--444. GI , 2015. URL https://dl.gi.de/handle/20.500.12116/2420

    2015

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.