REVIEW 3 major objections 4 minor 56 references
A Tale of Two Models: Understanding Data Workers' Internal and External Representations of Complex Data
T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read A team of ten expert data workers who all use the same code-encoded data model carried diverse mental pictures of it, and the mismatches blocked both planning and coding analyses.
desk verdict Solid qualitative study of expert mental models of a shared reified data model, but the unreported control question leaves the central 'forgetting' claim partly unverified. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The paper's central objects are the Mental Data Model (MDM), the internal and subjective representation of data a worker carries, and the Reified Data Model (RDM), the external, shared representation encoded in code or documentation. The argument runs on the measured gap between the two, elicited through semi-structured interviews, freehand sketching, task-based recall questions, and a control question that separates genuine forgetting from inability to abstract a simple hierarchy; reflexive thematic analysis organizes the transcripts into themes. The two 'parallel hazards'—one rooted in the mental model, one in the reified model—are the mechanism that converts the observed mismatch into concrete analysis failures.
What would settle it
A replication using an interviewer with no knowledge of EnsembleAPI, who never prompts for specific components, would settle the point: if most participants then spontaneously describe the full reference model—call tree, metadata, performance and statistics tables, and their links—the reported forgetting and divergence would be shown to be partly a shared-context artifact.
Extended reading notes
Core claim
The central claim is that participants had diverse mental data models that differed from the reified data model, even among team members who had designed the model. In interviews and sketches, the performance table dominated most mental models, while the call tree and the metadata table were hazy, implicit, or missing; some participants thought hierarchically and others tabularly, and senior team members often described their data through the final charts of an analysis rather than the stored structure. The authors interpret this as evidence of two parallel hazards: a data worker with an inaccurate mental model may be unable to do the analyses they want regardless of the reified model, and a worker who knows what they want but does not understand the reified model may be unable to express it in code. From these observations they recommend design interventions—data-model reminders, visual recommendation, graphical scripting bridges, structured metadata—rather than trying to force everyone onto one canonical model.
Load-bearing premise
The load-bearing premise is that a component absent from a participant's description or drawing is genuinely missing from their mental model, rather than something they assumed the interviewer already knew.
Editorial extensions
If this is right
- Data analysis tools should surface the stored data model, for example through lightweight embedded overviews, so workers do not have to recall it from memory.
- Metadata cannot be treated as a grab bag: giving metadata machine-readable semantic types would let tools guide grouping, filtering, and display.
- Interfaces should embrace multiple coexisting mental models, including both table-centric and hierarchy-centric views, rather than forcing one canonical representation.
- Probing the intended users' mental models early, before a data model is fixed in code, may reduce later engineering debt and analysis failures.
- Visual recommender and graphical-scripting approaches that show how a chart is derived from the stored model could bridge chart-based mental models to the reified model.
Reading between the lines
- An extension the authors leave implicit: if this divergence appears within a team of ten experts who designed the model, the mismatch is likely at least as large for casual users of similar data libraries, which argues for default-on data-model views in analysis tools.
- A testable extension would be an intervention study: give analysts a lightweight always-visible schematic of the data model and measure whether task completion time and code errors drop relative to a control group.
- The findings suggest data-model recall could be improved by design, such as naming and visually distinguishing components consistently across documentation, API, and plots; the paper gestures at this but does not test it.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper reports a qualitative study of ten data workers who develop and use a shared domain-specific data analysis library (EnsembleAPI). Through semi-structured interviews, sketching tasks, and analysis-task questions, the authors elicited participants' mental data models (MDMs) and compared them with the reified data model (RDM) embodied in the library's data structures. The central finding is that participants' mental models were diverse and frequently diverged from the RDM, even for the designers of that model. The authors identify two 'parallel hazards': (1) an inaccurate mental model can prevent a worker from planning an analysis, and (2) an incomplete understanding of the reified model can prevent a worker from expressing an analysis in code. They also report themes about data sources, analysis workflows, and the particular haziness of metadata, and they propose design implications for data analysis tools and data-model design.
Significance. If the findings hold, this is a valuable empirical contribution to HCI and visualization research on mental models of data: it extends prior work such as Williams et al. [53] to a real, complex, heterogeneous data model used by experts over an extended period, and it documents divergence even among the model's designers. The study has notable strengths: it recruited nearly the entire relevant population (10 of 11 team members), combined multiple elicitation methods (description, ranking, sketching, task questions), grounds its themes in direct quotes and participant drawings, and includes an unusually candid positionality statement and limitations section. The parallel-hazards framing and the metadata findings are likely to be useful to designers of data science tools. However, the paper's load-bearing inference from omissions to forgetting rests on a control question (Q7) whose results are never reported, and several cross-participant claims are affected by protocol changes made during data collection. These issues are fixable but currently leave the central claim less strongly supported than the abstract suggests.
major comments (3)
- [Section 4.2 and Section 5.1 (T1.1, RQ1)] Section 4.2 introduces Q7 as a control 'to verify that they could abstract and describe a small call tree and its associated data, to separate "forgetting" the hierarchy versus not understanding it,' and notes it was added after the first interview. Section 5 never reports what P2-P10 drew for Q7. This is the experiment that would validate or invalidate the interpretation that omitted components reflect gaps in mental models. If all nine drew A calls B and C correctly, the interview omissions would be better explained by shared-context conversational omission or retrieval failure, and the 'parallel hazards' claim would be substantially weakened. If several failed, the finding would be strongly supported. Please report the control outcomes for all participants who received Q7, or reframe the RQ1 and T1.1 claims as being about what participants spontaneously expressed rather than what they remembered.
- [Section 4.2, Section 5, and Figure 4] The protocol was refined during the study: Q5b was changed and Q7 was added after the first interview. The central comparative evidence in Figure 4 treats recall across all ten participants, but P1 did not receive the same protocol as P2-P10. This is a potential confound for cross-participant comparisons such as the statement in Section 5.3 that P1 had 'the most sparse description and representation.' Please provide per-participant protocol-version information and a sensitivity analysis excluding P1, or hedge the cross-participant comparisons accordingly.
- [Section 5.1 and Section 6.2.1] Section 5.1 states that 'The statistics table was most forgotten' but then excludes it from further analysis 'because it exists for derived data and is not populated on collection.' Including this component in a forgetting finding is misleading, since participants were not expected to mention a table that is not part of initial data collection. Relatedly, Section 6.2.1's first hazard claims that 'sufficiently inaccurate mental models resulted in errors which would prevent practical analysis,' but the main supporting evidence is two accidental dimension-drops (P6 in their drawing and P7 in task 5a) that participants corrected during the interview. The evidence supports difficulty, slowdown, and the need for re-scaffolding, but it does not clearly support 'prevent.' Please align the hazard wording with the strength of the evidence.
minor comments (4)
- [Section 4.4] The single-coder inductive phase is justified by the reflexive thematic analysis approach, but the supplementary materials are referenced without stating how they can be accessed. Including the codebook and an excerpt of the theme-development process would strengthen the audit trail and help readers assess the trustworthiness of the inductive themes.
- [Section 5.1 and Figure 4] Figure 4's timeline would be easier to interpret if it distinguished explicit mention, implicit allusion, and inclusion in a drawing; the text describes all three modes (e.g., P6 and P10 alluding to performance data rather than naming it), but the figure appears to collapse them.
- [Figure 8] The placement of participants on the utilitarian-to-structural and seniority axes is described only as 'qualitatively assessed'; please state the criteria or coding used to place each participant so that the figure is reproducible and not read as arbitrary.
- [Throughout] There are several typographical errors: the abstract and introduction contain 'a a different mental model', RQ1 in Section 1 reads 'an known form', and Section 6.2.3 contains 'thier mental data model'. These should be corrected in revision.
Circularity Check
No significant circularity: the empirical interview findings are self-contained; coauthored prior work appears only as comparison and context.
full rationale
The paper's central claim is an empirical finding from a newly conducted qualitative study: ten EnsembleAPI data workers described and sketched their understanding of a shared reified data model, and reflexive thematic analysis of those transcripts and drawings yielded diverse mental models that diverged from the reference model. This result does not reduce to any equation, fitted parameter, or prior publication. The control question (Q7) was designed to separate 'forgetting' from 'not understanding,' and the paper's failure to report its outcomes is a reporting or validity concern, not a circularity concern. Coauthored prior work is cited—most notably Williams et al. [53], described as 'the most similar work to ours'—but it is used for comparison and related-work context, not as the source of the study's findings or as a justification that forbids alternative interpretations. Other self-citations, such as [6] and [45], appear in related-work and design-implications discussions and are not load-bearing. The 'two parallel hazards' are analytical interpretations of observed participant behavior rather than restatements of the paper's inputs. No uniqueness theorem is imported from the authors' prior work, no ansatz is smuggled in via citation, and no known result is renamed. The derivation chain is therefore self-contained with respect to the empirical contribution.
Assumptions & free parameters
assumptions (3)
- domain assumption Participants' interview responses and sketches reflect their stable mental data models rather than context-dependent or shared-knowledge omissions.
- domain assumption A single inductive coder applying reflexive thematic analysis produces themes that are intersubjectively trustworthy.
- domain assumption The reference model in Figure 1 accurately represents the EnsembleAPI reified model as used in practice.
Cite this review
Pith. "Pith review of A Tale of Two Models: Understanding Data Workers' Internal and External Representations of Complex Data." pith.science (2026). https://pith.science/paper/KGYT3YA4
@misc{pith2026250109862,
author = {Pith},
title = {Pith review of: A Tale of Two Models: Understanding Data Workers' Internal and External Representations of Complex Data},
year = {2026},
howpublished = {\url{https://pith.science/paper/KGYT3YA4}},
note = {Machine review of arXiv:2501.09862}
}
read the original abstract
Data workers may have a a different mental model of their data that the one reified in code. Understanding the organization of their data is necessary for analyzing data, be it through scripting, visualization or abstract thought. More complicated organizations, such as tables with attached hierarchies, may tax people's ability to think about and interact with data. To better understand and ultimately design for these situations, we conduct a study across a team of ten people working with the same reified data model. Through interviews and sketching, we probed their conception of the data model and developed themes through reflexive data analysis. Participants had diverse data models that differed from the reified data model, even among team members who had designed the model, resulting in parallel hazards limiting their ability to reason about the data. From these observations, we suggest potential design interventions for data analysis processes and tools.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[53]
Katy Williams, Alex Bigelow, and Katherine E Isaacs. 2023. Data Abstraction Elephants: The Initial Diversity of Data Representations and Mental Models. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–24
work page 2023
-
[1]
Robert Akscyn, Elise Yoder, and Donald McCracken. 1988. The data model is the heart of interface design. In Proceedings of the SIGCHI conference on Human factors in computing systems. 115–120
work page 1988
-
[2]
Ioannis Anagnostopoulos, Sherali Zeadally, and Ernesto Exposito. 2016. Handling big data: research challenges and future directions. The Journal of Supercomputing 72 (2016), 1494–1516
work page 2016
-
[3]
R Anderson, A Black, B Blakeley, R Bleile, JS Camier, J Ciurej, A Cook, V Dobrev, N Elliott, J Grondalski, et al. 2020. The multiphysics on advanced platforms project. Report No. LLNL-TR-815869 (2020)
work page 2020
-
[4]
Lyn Bartram, Michael Correll, and Melanie Tory. 2021. Untidy data: The unreasonable effectiveness of tables. IEEE Transactions on Visualization and Computer Graphics 28, 1 (2021), 686–696
work page 2021
-
[5]
Alex Bigelow, Steven Drucker, Danyel Fisher, and Miriah Meyer. 2014. Reflections on how designers design with data. InProceedings of the 2014 International Working Conference on Advanced Visual Interfaces. 17–24
work page 2014
-
[6]
Alex Bigelow, Katy Williams, and Katherine E Isaacs. 2020. Guidelines for pursuing and revealing data abstractions. IEEE Transactions on Visualization and Computer Graphics 27, 2 (2020), 1503–1513
work page 2020
-
[7]
Jeremy E Block and Eric D Ragan. 2020. Micro-entries: Encouraging deeper evaluation of mental models over time for interactive data systems. In 2020 IEEE Workshop on Evaluation and Beyond-Methodological Approaches to Visualization (BELIV). IEEE, 38–47
work page 2020
Show all 56 references
-
[8]
Robert Bowman, Camille Nadal, Kellie Morrissey, Anja Thieme, and Gavin Doherty. 2023. Using thematic analysis in healthcare HCI at CHI: A scoping review. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–18
2023
-
[9]
Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative research in psychology 3, 2 (2006), 77–101
2006
-
[10]
John M Carroll. 2003. HCI models, theories, and frameworks: Toward a multidisciplinary science. Elsevier
2003
-
[11]
John M Carroll and Judith Reitman Olson. 1988. Mental models in human-computer interaction. Handbook of human-computer interaction (1988), 45–65
1988
-
[12]
Ibrahim Cetin and Ed Dubinsky. 2017. Reflective abstraction in computational thinking. The Journal of Mathematical Behavior 47 (2017), 70–80
2017
-
[13]
Roger D Chamberlain. 2020. Architecturally truly diverse systems: A review. Future Generation Computer Systems 110 (2020), 33–44
2020
-
[14]
Victoria Clarke and Virginia Braun. 2013. Successful qualitative research: A practical guide for beginners. Successful qualitative research (2013), 1–400
2013
-
[15]
Victoria Clarke and Virginia Braun. 2017. Thematic analysis. The journal of positive psychology 12, 3 (2017), 297–298
2017
-
[16]
Victoria Clarke and Virginia Braun. 2021. Thematic analysis: a practical guide. SAGE Publications Ltd. Manuscript submitted to ACM 24 Scully-Allison et al
2021
-
[17]
Cooke and Anna L
Nancy J. Cooke and Anna L. Rowe. 1994. Evaluating Mental Model Elicitation Methods. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 38, 4 (1994), 261–265. https://doi.org/10.1177/154193129403800416
1994 doi
-
[18]
Juliet M Corbin and Anselm Strauss. 1990. Grounded theory research: Procedures, canons, and evaluative criteria. Qualitative sociology 13, 1 (1990), 3–21
1990
-
[19]
Anamaria Crisan, Shannah E Fisher, Jennifer L Gardy, and Tamara Munzner. 2021. Gevitrec: Data reconnaissance through recommendation using a domain-specific visualization prevalence design space. IEEE Transactions on Visualization and Computer Graphics28, 12 (2021), 4855–4872
2021
-
[20]
Anamaria Crisan and Tamara Munzner. 2019. Uncovering data landscapes through data reconnaissance and task wrangling. In2019 IEEE Visualization Conference (VIS). IEEE, 46–50
2019
-
[21]
Coral J Dando. 2013. Drawing to remember: External support of older adults’ eyewitness performance. PloS one 8, 7 (2013), e69937
2013
-
[22]
Will Epperson, Vaishnavi Gorantla, Dominik Moritz, and Adam Perer. 2023. Dead or Alive: Continuous Data Profiling for Interactive Data Science. IEEE Transactions on Visualization and Computer Graphics(2023)
2023
-
[23]
André Freitas, Edward Curry, João Gabriel Oliveira, and Sean O’Riain. 2011. Querying heterogeneous datasets on the linked data web: challenges, approaches, and trends. IEEE Internet Computing 16, 1 (2011), 24–33
2011
-
[24]
Barney G Glaser, Anselm L Strauss, and Elizabeth Strutzel. 1968. The discovery of grounded theory; strategies for qualitative research. Nursing research 17, 4 (1968), 364
1968
-
[25]
Ava Heinonen, Bettina Lehtelä, Arto Hellas, and Fabian Fagerholm. 2023. Synthesizing research on programmers’ mental models of programs, tasks and concepts—A systematic literature review. Information and Software Technology(2023), 107300
2023
-
[26]
Bill Hibbard, Michael Böttinger, Martin Schultz, and Joachim Biercamp. 2002. Visualization in earth system science. Acm Siggraph Computer Graphics 36, 4 (2002), 5–9
2002
-
[27]
Václav Jirkovsk `y and Marek Obitko. 2014. Semantic Heterogeneity Reduction for Big Data in Industrial Automation. ITAT 1214 (2014)
2014
-
[28]
Mary Beth Kery, Donghao Ren, Fred Hohman, Dominik Moritz, Kanit Wongsuphasawat, and Kayur Patel. 2020. mage: Fluid moves between code and graphical work in computational notebooks. In Proc. ACM Symp. on UI Soft. & Tech.140–151
2020
-
[29]
Gary Klein and R.R. Hoffman. 2008. Macrocognition, mental models, and cognitive task analysis methodology. Naturalistic Decision Making and Macrocognition (01 2008), 57–80
2008
-
[30]
Ganesh Kumar, Shuib Basri, Abdullahi Abubakar Imam, Sunder Ali Khowaja, Luiz Fernando Capretz, and Abdullateef Oluwagbemiga Balogun. 2021. Data harmonization for heterogeneous datasets: A systematic literature review. Applied Sciences 11, 17 (2021), 8275
2021
-
[31]
Brenda Laurel, Tim Oren, and Abbe Don. 1990. Issues in multimedia interface design: Media integration and interface agents. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 133–139
1990
-
[32]
Jiali Liu, Nadia Boukhelifa, and James R Eagan. 2019. Understanding the role of alternatives in data analysis practices. IEEE transactions on visualization and computer graphics 26, 1 (2019), 66–76
2019
-
[33]
Marina Machado, Rui Couto, and José Creisssac Campos. 2017. MODUS: Model-based user interfaces prototyping. In Proceedings of the ACM SIGCHI symposium on engineering interactive computing systems. 111–116
2017
-
[34]
Jock Mackinlay, Pat Hanrahan, and Chris Stolte. 2007. Show me: Automatic presentation for visual analysis. IEEE transactions on visualization and computer graphics 13, 6 (2007), 1137–1144
2007
-
[35]
Michelle LA Mattison, Coral J Dando, and Thomas C Ormerod. 2015. Sketching to remember: Episodic free recall task support for child witnesses and victims with autism spectrum disorder. Journal of Autism and Developmental Disorders 45 (2015), 1751–1765
2015
-
[36]
Sean McKenna, Dominika Mazur, James Agutter, and Miriah Meyer. 2014. Design activity framework for visualization design. IEEE Transactions on Visualization and Computer Graphics 20, 12 (2014), 2191–2200
2014
-
[37]
Miriah Meyer and Jason Dykes. 2019. Criteria for rigor in visualization design study. IEEE transactions on visualization and computer graphics 26, 1 (2019), 87–97
2019
-
[38]
Michael Muller, Ingrid Lange, Dakuo Wang, David Piorkowski, Jason Tsay, Q Vera Liao, Casey Dugan, and Thomas Erickson. 2019. How data science workers work with data: Discovery, capture, curation, design, creation. In Proceedings of the 2019 CHI conference on human factors in c...
2019
-
[39]
Tamara Munzner. 2009. A nested model for visualization design and validation. IEEE transactions on visualization and computer graphics 15, 6 (2009), 921–928
2009
-
[40]
Tamara Munzner. 2014. Visualization analysis and design. CRC press
2014
-
[41]
The pandas development team. 2020. pandas-dev/pandas: Pandas. https://doi.org/10.5281/zenodo.3509134
2020 doi
-
[42]
David Raneburger. 2010. Interactive model driven graphical user interface generation. In Proceedings of the 2nd ACM SIGCHI symposium on Engineering interactive computing systems. 321–324
2010
-
[43]
David C Rubin and Daniel L Greenberg. 2003. The role of narrative in recollection: A view from cognitive psychology and neuropsychology.Narrative and consciousness: Literature, psychology, and the brain(2003), 53–85
2003
-
[44]
Roger C Schank and Robert P Abelson. 2014. Knowledge and memory: The real story. In Knowledge and memory: The real story. Psychology Press, 1–85
2014
-
[45]
Connor Scully-Allison, Ian Lumsden, Katy Williams, Jesse Bartels, Michela Taufer, Stephanie Brink, Abhinav Bhatele, Olga Pearce, and Katherine E Isaacs. 2024. Design Concerns for Integrated Scripting and Interactive Visualization in Notebook Environments. IEEE Transactions on ...
2024
-
[46]
Michael Sedlmair, Miriah Meyer, and Tamara Munzner. 2012. Design study methodology: Reflections from the trenches and the stacks. IEEE transactions on visualization and computer graphics 18, 12 (2012), 2431–2440
2012
-
[47]
Jon S Simons, Maureen Ritchey, and Charles Fernyhough. 2022. Brain mechanisms underlying the subjective experience of remembering. Annual Review of Psychology 73 (2022), 159–186
2022
-
[48]
Hariharan Subramonyam, Colleen Seifert, and Eytan Adar. 2021. Protoai: Model-informed prototyping for ai-powered interfaces. In Proceedings of the 26th International Conference on Intelligent User Interfaces. 48–58
2021
-
[49]
David Thompson, Joshua A Levine, Janine C Bennett, Peer-Timo Bremer, Attila Gyulassy, Valerio Pascucci, and Philippe P Pébay. 2011. Analysis of large-scale scalar data using hixels. In 2011 IEEE Symposium on Large Data Analysis and Visualization. IEEE, 23–30
2011
-
[50]
Jagoda Walny, Christian Frisson, Mieka West, Doris Kosminsky, Søren Knudsen, Sheelagh Carpendale, and Wesley Willett. 2019. Data changes everything: Challenges and opportunities in data visualization design handoff. IEEE transactions on visualization and computer graphics 26, ...
2019
-
[51]
Junpeng Wang, Subhashis Hazarika, Cheng Li, and Han-Wei Shen. 2018. Visualization and visual analysis of ensemble data: A survey. IEEE transactions on visualization and computer graphics 25, 9 (2018), 2853–2872
2018
-
[52]
Hadley Wickham, Romain François, Lionel Henry, Kirill Müller, and Davis Vaughan. 2023. dplyr: A Grammar of Data Manipulation . https: //dplyr.tidyverse.org R package version 1.1.4, https://github.com/tidyverse/dplyr
2023
-
[54]
Kanit Wongsuphasawat, Dominik Moritz, Anushka Anand, Jock Mackinlay, Bill Howe, and Jeffrey Heer. 2015. V oyager: Exploratory analysis via faceted browsing of visualization recommendations. IEEE transactions on visualization and computer graphics 22, 1 (2015), 649–658
2015
-
[55]
Kanit Wongsuphasawat, Zening Qu, Dominik Moritz, Riley Chang, Felix Ouk, Anushka Anand, Jock Mackinlay, Bill Howe, and Jeffrey Heer. 2017. V oyager 2: Augmenting visual analysis with partial view specifications. InProceedings of the 2017 chi conference on human factors in comp...
2017
-
[56]
Yifan Wu, Joseph M Hellerstein, and Arvind Satyanarayan. 2020. B2: Bridging code and interactive visualization in computational notebooks. In Proc. Annual ACM Symp. on UI Soft. & Tech.152–165. Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009 Manuscript su...
2020
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.