REVIEW 3 major objections 5 minor 45 references
Unfolding Data Quality Dimensions in Practice: A Survey
T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read The paper claims that the fog between data quality theory and practice can be lifted: every concrete check implemented in seven open-source data quality tools can be systematically assigned to one or more of six ISO/IEC 25012 dimensions…
desk verdict A careful survey with a genuinely useful catalog; the ISO mapping is the soft spot but it's disclosed, so worth peer review with a request to tighten the mapping methodology. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying object is the mapping table constructed through a four-stage methodology: select seven open-source tools, extract a separate functionality list from each tool's code, merge exact and near-duplicate functionalities into one lossless list with implementation variants, then assign each functionality to the ISO/IEC 25012 dimensions whose textual definition its purpose directly or indirectly aligns with. The defining rule for a connection is that functional alignment: a check such as "timestamp values are recent" is linked to currentness because both speak about the right age of data. The table also separates the semantic category of a functionality from the data granularity it operates on (value, row, column, table), which lets the same check appear at multiple levels without being fragmented.
What would settle it
Ask two independent teams to repeat the four-stage procedure on the same seven repositories using only the paper's stated alignment rule, then compare their lists and dimension assignments. If the teams disagree on even a few functionalities (for instance, whether f-01 range checks reach consistency beyond accuracy and compliance), the claimed systematicity of the mapping would not be confirmed; likewise, applying the same procedure to a fresh set of open-source tools and finding a materially different N:M structure would show the map is tool-set-dependent.
Extended reading notes
Core claim
The central claim is that the relationship between implemented data verification checks and high-level quality dimensions is systematically many-to-many, and that this relationship can be made explicit by reading tool source code against the textual definitions of ISO/IEC 25012. The paper's Table 1 records the resulting N:M mapping: 25 low-level functionalities, grouped into five semantic categories (conformance checks, distribution-based checks, volume- and cardinality-based checks, correlation-based checks, and ML-oriented checks) and annotated with the granularity level at which they operate, are each connected to the subset of six dimensions whose meaning the check's purpose aligns with. Direct assessment checks the values themselves; indirect assessment checks metadata or profiling-derived patterns. The paper presents this mapping not as a comparison of tools but as a unified picture of the fragmented error-detection landscape, and uses it to draw conclusions about how each dimension is materialized in practice, including non-obvious links between dimensions such as completeness and accessibility.
Load-bearing premise
The map stands on the authors' interpretive rule that a functionality is connected to a dimension when its purpose directly or indirectly aligns with the ISO definition, applied to seven hand-picked tools; if the standard text were read differently, or a different set of tools examined, the assignment of checks to dimensions could shift.
Editorial extensions
If this is right
- A practitioner can start from Table 1, pick the dimension a task cares about (e.g., completeness), and see which concrete checks — from row counts to missing-value probes to unique-value ratios — are available across the surveyed tools.
- A researcher can see which of the six dimensions are richly served by current tools and which are addressed only indirectly or thinly, guiding where new verification checks or metrics are most needed.
- The catalog of variants shows that the same named functionality can be tuned (freshness vs. latency, exact vs. threshold matching) so that it lands on different dimensions or different contexts of use.
- The survey's conclusions imply that terminology across tools is not standardized: the same source-code idea appears under different names, so cross-tool knowledge transfer and automated comparisons of checks require a semantic alignment like the one provided here.
- The non-intuitive dimension relationships identified, such as completeness and accessibility both flagging absent information, suggest that dimension selection alone is not enough to plan a data quality program.
Reading between the lines
- A natural extension the paper leaves implicit is to turn the catalog into a coverage profiler: given any new tool's source code, match its check names and implementation patterns against the 25 functionalities to predict which dimensions it can assess.
- Because the mapping is anchored to six inherent-quality dimensions, extending the same source-code method to system-dependent dimensions such as confidentiality, availability, or recoverability would likely require inspecting very different features (access control, audit logs, backup behavior), not just value validators.
- The acknowledged perspective-dependence suggests a stability test: if several independent teams apply the same alignment rule to the same tools, the dimensions each functionality is assigned to should converge; the paper does not provide such an inter-annotator check.
- The many-to-many view could serve as a bridge to data quality metrics research, because each dimension-functionality pair is a candidate spot to attach a quantitative metric that can be computed from the tool's output.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents a survey that connects low-level data-quality-check functionalities implemented in seven open-source data-quality tools to high-level data-quality dimensions from the ISO/IEC 25012 standard. It identifies 25 low-level functionalities, groups them into five categories, documents implementation variants, and proposes a many-to-many mapping between these functionalities and six selected ISO dimensions (accuracy, completeness, consistency, currentness, accessibility, compliance). The methodology has four stages: tool selection, independent extraction of functionalities, merging and grouping, and mapping to dimensions. The main contributions are a unified list of low-level functionalities, detailed variant catalogs, and the N:M mapping, which is intended to give practitioners and researchers a unified, actionable view of how data-quality dimensions are materialized in current tools.
Significance. If the mapping is accepted as reliable, the paper fills a real gap: prior surveys of data-quality tools either report functionality from a user perspective or compare tools, but do not systematically map concrete check-level functionalities to standardized dimensions. The paper's grounding in source-code inspection is a notable strength: Appendix A gives exact function and class names, making the factual claims checkable. The variant catalogs in Sections 3.2-3.10 are detailed and useful for practitioners selecting checks. The paper also discloses important limitations, including the perspective-dependent nature of the mapping in conclusion C3 and the re-interpretation of accessibility in Section 4.5. However, because the central contribution is a qualitative mapping, the lack of an inter-rater or external validation and a number of mismatches between Table 1 and the per-functionality justifications weaken the claim that the mapping is systematic and internally consistent.
major comments (3)
- [Table 1 vs. Sections 3.8.3, 3.8.4, 3.9.1] Table 1 assigns the compliance dimension (cmpli) to f-17 (unique elements checks), f-18 (most common elements checks), and f-21/f-22 (correlation and mutual information checks), but the corresponding 'Mapping to dimensions' paragraphs do not mention compliance at all. For f-17 the text even says 'Completeness, consistency and accessibility are the connected dimensions in descending order of connection strength', and for f-21/f-22 it names only accuracy and consistency. Since Table 1 is the central artifact of the paper, this internal discrepancy must be resolved either by removing these cells or by providing explicit justifications.
- [Section 4.5, Table 1, Appendix B] The mapping uses 'accessibility' in a way that is not the ISO/IEC 25012 definition. The ISO definition in Appendix B is specifically about access by people with disabilities and users needing supporting technology or special configuration, whereas Section 4.5 states that the paper 'primarily focuses on the first part of the definition' and redefines accessibility as whether data 'can be retrieved, interpreted, and used effectively by users or downstream tasks'. This is closer to the ISO 'availability' dimension, which the paper does not use. Because the abstract and Section 1 present the mapping as being to ISO/IEC 25012 dimensions, this operationalization needs to be either aligned with the standard or explicitly labeled as a modified notion of accessibility throughout the paper.
- [Section 2.4] The mapping rule that a connection exists 'when the purpose of a functionality directly or indirectly aligns with the textual definition of a DQ dimension' is underspecified. Since the N:M mapping is contribution (3), the paper should provide stronger evidence that the mapping is systematic rather than a single team's interpretation. As the paper itself acknowledges in C3, a different perspective or a different tool set could change the mapping materially. The authors could add an inter-rater agreement check, a comparison with ISO/IEC 25024 measurement definitions, or a robustness analysis over tool subsets. Without such evidence, the 'systematically connecting' claim in the abstract is not fully supported.
minor comments (5)
- [Appendix A] The appendix lists exact function names, which is very useful, but it does not specify the tool versions that were examined; adding version numbers or commit hashes would make the survey reproducible.
- [Table 1] The abbreviation 'cmplid' appears in the f-17 row, while the header and other rows use 'cmpli'; this appears to be a typo.
- [Section 3.8.5, Table 8] The variant f-19d is described as 'number/fraction of rows with missing values in a rows'; the trailing 'rows' is a typo.
- [Section 5.1] The sentence 'the surveys are several years old and to not reflect recent developments' contains a grammatical error ('to not' should be 'do not').
- [Section 3.2.3] The variant 'f-03c. regex to not match' is awkwardly phrased; it should be 'regex that must not match' to clarify the intended semantics.
Circularity Check
No circular derivation: the tool-to-dimension mapping is an external-standard-anchored qualitative synthesis, not a fitted or self-referential prediction.
full rationale
The paper's central contribution is an N:M mapping between low-level DQ tool functionalities and six ISO/IEC 25012 dimensions (Section 2.4, Table 1). The mapping rule—'We establish a connection when the purpose of a functionality directly or indirectly aligns with the textual definition of a DQ dimension'—is interpretive, but the anchor is the external ISO/IEC 25012 standard, whose definitions are quoted independently in Appendix B. The tool functionality lists are extracted from source code and documentation (Stage 2), and the merging criteria (MC1, MC2) are stated; neither step defines the functionalities in terms of the target dimensions. The only self-citation, [31], is an acknowledgement of a preliminary technical report and is not load-bearing: no conclusion is forced by it. Section 4 re-reads the mapping from the dimension side, but that is a summary of the same mapping, not an independent derivation that reduces to its input. The subjectivity of the 'directly or indirectly aligns' rule could affect reliability or reproducibility, but that is a soundness/validation concern, not circularity: the paper does not fit a parameter and then rename it as a prediction, and it does not invoke a self-authored uniqueness theorem. Accordingly no circular step is exhibited.
Assumptions & free parameters
assumptions (4)
- domain assumption ISO/IEC 25012 is the appropriate theoretical model for data quality dimensions.
- domain assumption The seven selected tools are representative of the open-source data quality landscape.
- domain assumption The source code and documentation inspected in Stage 2 reflect the tools' actual built-in functionality.
- ad hoc to paper A connection exists when the purpose of a functionality 'directly or indirectly aligns' with a dimension definition.
Cite this review
Pith. "Pith review of Unfolding Data Quality Dimensions in Practice: A Survey." pith.science (2026). https://pith.science/paper/BXAB5XHZ
@misc{pith2026250717507,
author = {Pith},
title = {Pith review of: Unfolding Data Quality Dimensions in Practice: A Survey},
year = {2026},
howpublished = {\url{https://pith.science/paper/BXAB5XHZ}},
note = {Machine review of arXiv:2507.17507}
}
read the original abstract
Data quality describes the degree to which data meet specific requirements and are fit for use by humans and/or downstream tasks (e.g., artificial intelligence). Data quality can be assessed across multiple high-level concepts called dimensions, such as accuracy, completeness, consistency, or timeliness. While extensive research and several attempts for standardization (e.g., ISO/IEC 25012) exist for data quality dimensions, their practical application often remains unclear. In parallel to research endeavors, a large number of tools have been developed that implement functionalities for the detection and mitigation of specific data quality issues, such as missing values or outliers. With this paper, we aim to bridge this gap between data quality theory and practice by systematically connecting low-level functionalities offered by data quality tools with high-level dimensions, revealing their many-to-many relationships. Through an examination of seven open-source data quality tools, we provide a comprehensive mapping between their functionalities and the data quality dimensions, demonstrating how individual functionalities and their variants partially contribute to the assessment of single dimensions. This systematic survey provides both practitioners and researchers with a unified view on the fragmented landscape of data quality checks, offering actionable insights for quality assessment across multiple dimensions.
Figures
Reference graph
Works this paper leans on
-
[31]
Vasileios Papastergios and Anastasios Gounaris. 2024. A survey of open-source data quality tools: shedding light on the materialization of data quality dimensions in practice. CoRR abs/2407.18649 (2024). https://doi.org/10.48550/ ARXIV.2407.18649
work page Pith review arXiv doi:10.48550/arxiv.2407.18649 2024
-
[1]
2015. ISO/IEC 25024:2015 Systems and software engineering — Systems and software Quality Requirements and Evaluation (SQuaRE) — Measurement of data quality. https://www.iso.org/standard/35749.html
work page 2015
-
[2]
Mohamed Abdelaal, Christian Hammacher, and Harald Schöning. 2023. REIN: A Comprehensive Benchmark Framework for Data Cleaning Methods in ML Pipelines. https://doi.org/10.48786/EDBT.2023.43
-
[3]
Omar Almutiry, Gary Wills, and Richard Crowder. 2015. Dimension-oriented Taxonomoy of Data Quality Problems In Electronic. Health Record. IADIS International Journal on WWW/Internet 13, 2 (2015)
work page 2015
-
[4]
Otmane Azeroual, Gunter Saake, and Eike Schallehn. 2018. Analyzing data quality issues in research information systems via data profiling. International Journal of Information Management 41 (2018), 50–56. https://doi.org/10.1016/ j.ijinfomgt.2018.02.007
work page 2018
-
[5]
José Barateiro and Helena Galhardas. 2005. A Survey of Data Quality Tools. Datenbank-Spektrum 14 (2005), 15–21
work page 2005
-
[6]
Carlo Batini, Cinzia Cappiello, Chiara Francalanci, and Andrea Maurino. 2009. Methodologies for Data Quality Assessment and Improvement. Comput. Surveys 41, 3 (2009), 16:1–16:52
work page 2009
-
[7]
Carlo Batini and Monica Scannapieco. 2016. Data and Information Quality: Dimensions, Principles and Techniques . Springer International Publishing. https://doi.org/10.1007/978-3-319-24106-7
Show all 45 references
-
[8]
Eric Breck, Marty Zinkevich, Neoklis Polyzotis, Steven Whang, and Sudip Roy. 2019. Data Validation for Machine Learning. In Proceedings of SysML. https://mlsys.org/Conferences/2019/doc/2019/167.pdf
2019
-
[9]
Lukas Budach, Moritz Feuerpfeil, Nina Ihde, Andrea Nathansen, Nele Noack, Hendrik Patzlaff, Hazar Harmouch, and Felix Naumann. 2022. The Effects of Data Quality on Machine Learning Performance. arXiv preprint arXiv:2207.14529 30 Vasileios Papastergios, Lisa Ehrlinger, and Anas...
2022 arXiv
-
[10]
Corinna Cichy and Stefan Rass. 2019. An Overview of Data Quality Frameworks. IEEE Access 7 (2019), 24634–24648
2019
-
[11]
Davenport and Rean Bean
Thomas H. Davenport and Rean Bean. 2024. Data and AI Leadership Executive Survey. (2024). https://www. randybeandata.com/s/DataAI-ExecutiveLeadershipSurveyFinalAsset.pdf
2024
-
[12]
Lisa Ehrlinger and Wolfram Wöß. 2022. A Survey of Data Quality Measurement and Monitoring Tools. Frontiers in Big Data 5 (2022). https://doi.org/10.3389/fdata.2022.850611
2022
-
[13]
Hadi Fadlallah, Rima Kilany, Houssein Dhayne, Rami El Haddad, Rafiqul Haque, Yehia Taher, and Ali Jaber. 2023. Context-aware Big Data Quality Assessment: A Scoping Review. J. Data and Information Quality 15, 3, Article 25 (Aug. 2023), 33 pages. https://doi.org/10.1145/3603707
2023 doi
-
[14]
International Organization for Standardization. 2004. ISO/IEC 13335-1: Information technology — Security techniques — Management of information and communications technology security. Part 1: Concepts and models for information and communications technology security management...
2004
-
[15]
International Organization for Standardization. 2008. ISO/IEC 25012: Software Engineering: Software Product Quality Requirements and Evaluation (SQuaRE): Data Quality Model . ISO/IEC. https://iso25000.com/index.php/en/iso-25000- standards/iso-25012
2008
-
[16]
Tom Haegemans, Monique Snoeck, and Wilfried Lemahieu. 2016. Towards a precise definition of data accuracy and a justification for its measure. In Proceedings of the International Conference on Information Quality . MIT Information Quality (MITIQ) Program, 16–16
2016
-
[17]
Karin Hartl and Olaf Jacob. 2016. The Role of Data Quality in Business Intelligence – An empirical study in German medium-sized and large companies
2016
-
[18]
Stefan Heule, Marc Nunkesser, and Alexander Hall. 2013. HyperLogLog in practice: algorithmic engineering of a state of the art cardinality estimation algorithm. In Proceedings of the 16th International Conference on Extending Database Technology (Genoa, Italy) (EDBT ’13). Asso...
2013
-
[19]
Ankush Jain and Melody Chien. 2022. Magic Quadrant for Data Quality Solutions . Technical Report. Gartner, Inc
2022
-
[20]
Rajesh Jugulum. 2016. Importance of Data Quality for Analytics . Springer International Publishing, 23–31. https: //doi.org/10.1007/978-3-319-21332-3_2
2016 doi
-
[21]
Zohar Karnin, Kevin Lang, and Edo Liberty. 2016. Optimal Quantile Approximation in Streams. In 2016 IEEE 57th Annual Symposium on Foundations of Computer Science (FOCS) . 71–78. https://doi.org/10.1109/FOCS.2016.17
2016 doi
-
[22]
Nuno Laranjeiro, Seyma Nur Soydemir, and Jorge Bernardino. 2015. A Survey on Data Quality: Classifying Poor Data. In 2015 IEEE 21st Pacific Rim International Symposium on Dependable Computing (PRDC) . 179–188. https: //doi.org/10.1109/PRDC.2015.41
2015 doi
- [23]
-
[24]
Yang Lee, Stuart E Madnick, Richard Y Wang, Forea Wang, and Hongyun Zhang. 2014. A Cubic Framework for the Chief Data Officer: Succeeding in a World of Big Data . Technical Report CISL# 2014-01. Massachusetts Institute of Technology, Cambridge, MA, USA
2014
-
[25]
Peng Li, Susie Xi Rao, Jennifer Blase, Yue Zhang, Xu Chu, and Ce Zhang. 2019. CleanML: A Benchmark for Joint Data Cleaning and Machine Learning [Experiments and Analysis]. ArXiv abs/1904.09483 (2019). https://api.semanticscholar. org/CorpusID:128316992
2019 arXiv
-
[26]
David Loshin. 2011. Business Impacts of Poor Data Quality . Elsevier, 1–16. https://doi.org/10.1016/b978-0-12-373717- 5.00001-4
2011 doi
-
[27]
Sedir Mohammed, Lou Therese Brandner, Felicia Burtscher, Sebastian Hallensleben, Hazar Harmouch, Andreas Hauschke, Jessica Heesen, Stefanie Hildebrandt, Simon David Hirsbrunner, Julia Keselj, Philipp Mahlow, Marie Massow, Felix Naumann, Frauke Rostalski, Anna Wilken, and Annik...
2024 doi
-
[29]
Sedir Mohammed, Hazar Harmouch, Felix Naumann, and Divesh Srivastava. 2024. Data Quality Assessment: Challenges and Opportunities. arXiv preprint arXiv:2403.00526 (2024)
2024 arXiv
-
[30]
Tadhg Nagle, Tom Redman, and Sammon. [n. d.]. Assessing data quality: A managerial call to action.Business Horizons 63, 3 ([n. d.]), 325–337. https://doi.org/10.1016/j.bushor.2020.01.006
2020 doi
-
[32]
Venkata Sai Venkatesh Pulla, Cihan Varol, and Murat Al. 2016. Open Source Data Quality Tools: Revisited. In Information Technology: New Generations: 13th International Conference on Information Technology , Shahram Latifi Unfolding Data Quality Dimensions in Practice: A Survey...
2016
-
[33]
Val Pushkarev, Henry Neumann, Cihan Varol, and John R Talburt. 2010. An Overview of Open Source Data Quality Tools. In Proceedings of the 2010 International Conference on Information & Knowledge Engineering, IKE 2010, July 12-15,
2010
-
[34]
Rana, Sheshadri Chatterjee, Yogesh K
Nripendra P. Rana, Sheshadri Chatterjee, Yogesh K. Dwivedi, and Shahriar Akter. 2021. Understanding dark side of artificial intelligence (AI) integrated business analytics: assessing firm’s operational inefficiency and competitiveness. European Journal of Information Systems 3...
2021
-
[35]
Thomas C. Redman. 1998. The impact of poor data quality on the typical enterprise. Commun. ACM 41, 2 (Feb. 1998), 79–82. https://doi.org/10.1145/269012.269025
1998
-
[36]
Monica Scannapieco and Tiziana Catarci. 2002. Data Quality under the Computer Science Perspective. Archivi & Computer 2 (2002), 1–15
2002
-
[37]
Sebastian Schelter, Dustin Lange, Philipp Schmidt, Meltem Celikel, Felix Biessmann, and Andreas Grafberger. 2018. Automating large-scale data quality verification. Proc. VLDB Endow. 11, 12 (Aug. 2018), 1781–1794. https://doi.org/10. 14778/3229863.3229867
2018
-
[38]
Laura Sebastian-Coleman. 2013. Measuring Data Quality for Ongoing Improvement: A Data Quality Assessment Framework. Elsevier, Waltham, MA, USA
2013
-
[39]
Flavia Serra, Verónika Peralta, Adriana Marotta, and Patrick Marcel. 2024. Use of Context in Data Quality Management: A Systematic Literature Review. J. Data and Information Quality 16, 3, Article 19 (Oct. 2024), 41 pages. https: //doi.org/10.1145/3672082
2024 doi
-
[40]
Russell Torres and Anna Sidorova. 2019. Reconceptualizing information quality as effective use in the context of business intelligence and analytics. International Journal of Information Management 49 (2019), 316–329. https: //doi.org/10.1016/j.ijinfomgt.2019.05.028
2019 doi
-
[41]
Richard Y. Wang. 1998. A Product Perspective on Total Data Quality Management.Commun. ACM 41, 2 (1998), 58–65
1998
-
[42]
Wang and Diane M
Richard Y. Wang and Diane M. Strong. 1996. Beyond Accuracy: What Data Quality Means to Data Consumers.Journal of Management Information Systems 12 (1996), 5–33. Issue 4
1996
-
[43]
Yanqing Wang. 2023. Generative AI in Operational Risk Management: Harnessing the Future of Finance. SSRN Electronic Journal (2023). https://doi.org/10.2139/ssrn.4452504
2023 doi
-
[44]
Manuel Wörsdörfer. 2023. Mitigating the adverse effects of AI with the European Union’s artificial intelligence act: Hype or hope? Global Business and Organizational Excellence 43, 3 (Nov. 2023), 106–126. https://doi.org/10.1002/joe.22238
2023 doi
-
[45]
Jan Zenisek, Florian Holzinger, and Michael Affenzeller. 2019. Machine learning based concept drift detection for predictive maintenance. Computers and Industrial Engineering 137 (Nov. 2019), 106031. https://doi.org/10.1016/j.cie. 2019.106031 A EXACT NAMES OF LOW-LEVEL FUNCTIO...
2019
-
[2010]
CSREA Press, Las Vegas, NV, USA, 370–376
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.