{"id":"e276386b-4cb4-4ea6-97d7-826ffc032afe","arxiv_id":"1908.07894","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A qualitative interview study of 13 enterprise data analysts yields ten insights for visualization design in data preprocessing, including three not found in prior related work.","lead":"This paper interviews 13 data analysts about how they use visualization during the data preprocessing phase, and distills their responses into ten design insights. It is a useful requirements-gathering study for visualization researchers, but its small, convenience-based sample limits how far the insights generalize.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The ten-insight list is plausible but rests on a small, self-described non-representative convenience sample; the paper's own Section V limitation undercuts generalizing the list into a requirements baseline.","rationale":"The strongest claim is that the ten insights constitute a consolidated, generalizable set of requirements for visualization in preprocessing. The paper's own Section V explicitly states that the sample cannot be considered representative of all data analysts, making this the most load-bearing weakness. The reader identified the same concern, and the paper's abstract and introduction still present the list without this caveat, so the concern genuinely affects the central claim. The comparison with related works provides only partial support: it corroborates some insights but not the three unique to this study, and insight 10 comes solely from related works. A replication with a more diverse sample would directly test whether the list is stable or an artifact of the recruitment pool. Because the authors acknowledge the limitation and the study remains a useful qualitative contribution if framed as candidate requirements, the CONDITIONAL verdict remains appropriate.","tokens_in":13477,"tokens_out":4155,"duration_ms":41891,"concrete_test":"Replicate the interview protocol with 10-15 data analysts from a different national and industrial context (e.g., public sector, non-IT domains, more gender diversity), and apply the same threshold (>2 participants) to derive a new insight list. If insights 7-9 (or any of the ten) fail to reappear, or if new insights emerge, the original list is not a stable requirements baseline; if all ten reappear, the representativeness concern is mitigated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Central claim: a consolidated list of ten insights, derived from interviews with 13 enterprise professionals, can serve as a requirements baseline for visualization in preprocessing. For this claim to hold, the interview findings must be sufficiently stable across the population of enterprise data analysts. The weakest link is the sample: 13 participants, 12 male, recruited from three cities in a single country via LinkedIn/Meetup/personal networks; 12 work in private-sector IT-adjacent roles (Section III-A). The paper explicitly concedes in Section V that 'the data collected and its analysis cannot be considered a representation of all data analysts.' With n=13, each participant is ~7.7% of the sample, and the inclusion threshold 'more than two participants' admits themes mentioned by only 3/13 (~23%). Small-sample convenience themes can reflect local conditions (e.g., one national context, mostly tech industry) rather than broad enterprise preprocessing needs. If the list is intended only as candidate requirements, the limitation is survivable; but the abstract and Section I frame the list as a consolidated set of requirements for future visualization research, which invites generalization. The comparison with three related works (RW1-RW3) partially mitigates this for insights 1-6 and 10, but insights 7-9 are unique to this small sample, and insight 10 is imported from related works without interview support. Thus the load-bearing assumption is not just 'sample is representative' but 'the unique insights are not artifacts of the recruitment pool.' The paper's own limitation statement directly identifies this, but the framing in the abstract does not carry the caveat.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript reports a semi-structured interview study with thirteen enterprise data analysts, focusing on how visualization can support the preprocessing phase of data mining workflows. The authors describe the participants' profiles, their data analysis processes, prevalent preprocessing activities, data quality concerns, and their current use of visualization. From an iterative, incremental coding of the interview responses, the authors derive a consolidated list of ten insights for visualization tools, compare this list with three related works (RW1, RW2, RW3), and position the list as a requirements baseline for future visualization research. The paper concludes with acknowledged limitations, including the sample's narrow demographic and geographic representativeness.","tokens_in":13883,"tokens_out":4345,"duration_ms":44898,"significance":"If the ten-insight list is robust, it would provide a concise, actionable checklist for visualization researchers and tool builders targeting the preprocessing phase, an area that the authors argue is underserved relative to later-stage visualization. The study's strengths include its transparent description of the coding procedure, the explicit triangulation with three related interview studies, and an honest section on limitations. The paper also offers useful descriptive detail about practitioner workflows and data quality issues. However, the central contribution rests on a small, self-selected, single-country sample, and the manuscript's framing as a 'consolidated set of requirements' exceeds what the evidence can support. The insights are best regarded as candidate requirements or hypotheses for further validation, not as a settled baseline.","major_comments":[{"comment":"There is an inconsistency in the inclusion threshold that determines which items become insights. Section III-C states 'we considered the items reported by more than two participants,' which means at least three participants. In contrast, Section IV, Step 4 states 'the items that were not mentioned by at least two participants were not included,' which means at least two participants. With n=13, the difference between 2 and 3 mentions is material (15% versus 23% of the sample) and directly affects the composition of the central ten-insight list. The authors should reconcile these statements, report the exact number of participants supporting each insight, and justify the chosen threshold.","section":"Section III-C and Section IV, Step 4"},{"comment":"The abstract and introduction describe the ten insights as 'a consolidated set of requirements' for future visualization research, but Section V explicitly concedes that 'the data collected and its analysis cannot be considered a representation of all data analysts.' Given the sample of thirteen participants, mostly male and mostly working in the IT industry in one country, the requirements framing overstates the evidence. I recommend reframing the contribution as candidate insights or requirements hypotheses that need further validation, and adjusting the abstract and Section I accordingly so that the claims match the scope of the study.","section":"Abstract, Section I, and Section V"},{"comment":"The consolidated list treats all ten insights uniformly, but their provenance differs: insights 7-9 are unique to this small interview sample, while insight 10 is taken solely from related works, as the authors acknowledge. This mixing is not itself an error, but the final presentation should make the provenance of each insight explicit in the main text (not only in Figure 4) and should discuss the evidentiary status of each category. In particular, the three insights that are not corroborated by any related work should be flagged as tentative, and the manuscript should suggest what additional evidence would be needed to solidify them.","section":"Section IV and Section V"}],"minor_comments":[{"comment":"The sentence 'The same environment configuration was used for all participants, face-to-face or online conversations' is ambiguous; clarify whether the same physical setup or the same set of instructions was used across the two modes of interviewing.","section":"Section III-B"},{"comment":"There is a typo in the caption: 'insigths' should be 'insights.'","section":"Figure 5 caption"},{"comment":"The manuscript states that participants were located in three cities in the same country but does not name the country or cities. Naming the country (or at least the region) would help readers calibrate the generalizability of the findings, especially because the study's own Section V emphasizes the sample's limitations.","section":"Section III-A"},{"comment":"The paper notes that 'parts of the sessions were recorded' but does not specify which parts or how the audio was used beyond reviewing notes. A brief clarification of the recording and transcription procedure would improve methodological transparency.","section":"Section III-B"},{"comment":"The coding description says the list 'started based on the inputs received from participant one' and was then iteratively revised. This is a reasonable approach, but the paper should state whether any of the final insights were added or removed during this iterative process and how disagreements or ambiguous responses were handled.","section":"Section IV, Step 1"}],"recommendation":"major_revision","confidential_remarks":"The paper's contribution is a modest qualitative requirements list that is strongly constrained by its sample. The main technical issue I see is the threshold inconsistency and the mismatch between the 'requirements' framing and the acknowledged limitations. If the authors reposition the contribution as candidate insights and add a clear validation agenda, the paper could be acceptable for a venue that values qualitative work. As it stands, the central claim needs revision before it can support the stated contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: this is a modest but useful interview study on a genuinely underserved part of the visualization workflow. The contribution is the consolidated list of ten insights for visualization support during data preprocessing, with explicit mapping to which insights came from the authors' own interviews and which align with prior work (Batch and Elmqvist, Kandel et al., Alspaugh et al.). Three insights—work scopes, preprocessing as part of an iterative cycle, and comparison before/after transformation—are new relative to those closest works. The coding process is described step by step, and the authors separate insights they heard in interviews from the one they only found in the literature. That transparency earns credit.\n\nWhere it is soft: the sample. Thirteen participants, twelve male, from one country, mostly IT-industry, recruited via LinkedIn and personal networks. The paper acknowledges directly that this is not representative of all data analysts, and that caveat matters most for insights 7–9, which have no support in prior work—those are hypotheses, not established requirements. The 'more than two participants' inclusion rule is arbitrary and lets a theme mentioned by three people carry weight. The claim that comparing with the related works 'improves reliability' is loose; it is triangulation, not reliability. No raw data or codebook is provided, so independent re-analysis is not possible. Insight 10 is imported from related work, which is legitimate but should be labelled clearly.\n\nNone of this sinks the paper. The list is plausible and useful as a starting point, and the authors are honest about the limits. The abstract overstates a bit by calling it a 'consolidated set of requirements,' but the limitations section walks it back. The citation pattern is appropriate—the three related works are the obvious comparison set, and the authors treat them fairly.\n\nI would take this for peer review. A serious referee would ask for the raw material or a fuller codebook, a more disciplined framing of generalizability, and a clearer distinction between interview-derived and literature-derived insights. It is clearly written, the method is transparent, and the authors know the related work. If I worked on visualization for data preparation, I would cite it. For a reading group, it would spark a decent discussion about qualitative methods in vis.","headline":"Useful consolidation of preprocessing-focused visualization needs, with an honest small-sample caveat; worth refereeing.","tokens_in":14252,"tokens_out":2427,"would_cite":true,"duration_ms":24283,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Ten insights define how visualization should support data preprocessing","keywords":["visualization","preprocessing","visual data exploration","data mining","interview study","enterprise professionals","data quality","design insights"],"falsifier":"Run the same semi-structured protocol with a larger and more diverse sample, such as fifty analysts across government, healthcare, retail, and finance in more than one country; if most of the ten insights no longer reach the two-participant mention threshold, or several new high-frequency insights appear, the claim that this list is a consolidated requirements baseline for enterprise preprocessing would not hold.","tokens_in":13280,"feed_emoji":"📊","tokens_out":7406,"duration_ms":67465,"temperature":0.7,"pith_summary":"The paper claims that visualization can support the preprocessing phase of enterprise data mining in ten specific, reusable ways. Drawing on semi-structured interviews with thirteen data analysts, the authors consolidate recurring challenges and wishes into a list of ten insights, including 'keep it simple,' 'tables are OK,' 'allow comparison,' and 'capture metadata.' They then compare this list against three earlier interview studies, which ratifies most items and contributes one insight the interviews had not raised. If the list holds, it gives future visualization tool builders a straightforward requirements baseline for helping analysts during data cleaning, standardization, and feature selection rather than only when presenting final results.","feed_headline":"Ten insights define how visualization should support data preprocessing","feed_subtitle":"Interview study with 13 data analysts distills preprocessing needs into a tool checklist.","key_machinery":"The central object is the ten-insight list itself, each insight being a single recommendation for what visualization should offer during preprocessing, built through an iterative, incremental coding method. The list began from the first participant's wishlist; every later interview updated it, and after all interviews the combined entries were reviewed, labelled, ordered by frequency, and filtered so that items mentioned by fewer than two participants were dropped. A comparison with three prior interview studies merged their design implications with the list, adding the tenth insight. The list does the argument's work: it converts a heterogeneous set of interview responses into a compact, ordered set of requirements that future tool designs can be checked against.","core_discovery":"The central claim is a consolidated list of ten insights stating how visualization should support preprocessing activities from the data analyst's perspective. The insights are: keep visualization simple; keep context by remaining compatible with the programming environments analysts already use; save the analyst's time; support large-scale or Big Data scenarios; allow interaction beyond static reports; recognize tables as a legitimate visual format; pay attention to under-served work scopes such as feature creation and deep-learning interpretation; treat preprocessing as part of a back-and-forth cycle rather than a linear first phase; allow comparison of data before and after transformations; and capture the metadata or logic behind automatic transformations. The authors report that the first nine emerged from their interviews, the tenth came from the related work, and three of the insights were not present in any of the three comparison studies. Readers are asked to accept this list as a requirements baseline for new visualization efforts in visual data exploration.","pith_inferences":["The authors do not spell it out, but the ten insights also function as an evaluation rubric: a tool that violates several of them, for example by hiding transformation logic or requiring context switches, should predictably struggle to gain adoption, and that prediction could be tested in a comparative study.","Insight 8, which reframes preprocessing as a back-and-forth cycle, points toward a concrete design direction: show partial cleaning results as they become available and let the analyst keep interacting while the tool works.","Because the sample is mostly male, mostly IT-industry, and drawn from one country, the frequency ordering of the insights is the most fragile part; a testable extension is to rerun the same protocol with analysts in government, healthcare, and retail and compare which insights reach the mention threshold."],"forward_implications":["Visualization tools for data mining should run inside Python and R workflows instead of forcing analysts to switch environments and lose context.","Scalability is a precondition: analysts working with very large datasets need aggregation and density-based rendering before they can benefit from novel chart types.","Table views are a legitimate visualization; design effort should go into enhancing tables with interaction and pixel-oriented techniques rather than replacing them.","Preprocessing tooling should let analysts compare data before and after transformations and should record the logic behind automatic cleaning steps.","The ten insights can serve as a checklist for planning or evaluating new visualization solutions aimed at initial exploratory analysis."],"supporting_citations":[{"why":"This is one of the three comparison interview studies whose design implications are merged into the final insight list.","marker":"[11]"},{"why":"This is the enterprise analyst interview study that ratifies several insights and, together with the third study, contributes the metadata insight.","marker":"[12]"},{"why":"This is the third comparison study, contributing recommendations on data exploration tools that ratify the list and help add the metadata insight.","marker":"[13]"},{"why":"This sets the 10-to-15 participant target for the semi-structured interview method used to gather the interview data.","marker":"[16]"}],"fun_headline_variants":["Ten insights define visualization needs for data preprocessing","13 analysts, 10 insights: visual support for data prep","How enterprise analysts want visualization to aid preprocessing","A consolidated checklist for preprocessing visualization tools"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The study assumes that thirteen self-selected data analysts, mostly male and mostly working in one country's IT industry, can reveal enough about enterprise preprocessing practice that the ten insights generalize beyond that group.","fun_headline_variants_meta":{"raw":{"variants":["Ten insights define visualization needs for data preprocessing","13 analysts, 10 insights: visual support for data prep","How enterprise analysts want visualization to aid preprocessing","A consolidated checklist for preprocessing visualization tools"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000287,"raw_usage":{"total_tokens":1676,"prompt_tokens":927,"completion_tokens":749,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":543,"completion_tokens_details":{"reasoning_tokens":691}},"tokens_in":543,"tokens_out":749,"duration_ms":8157,"temperature":1.0,"reasoning_tokens":691,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:53:00.917546+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same semi-structured protocol with a larger and more diverse sample, such as fifty analysts across government, healthcare, retail, and finance in more than one country; if most of the ten insights no longer reach the two-participant mention threshold, or several new high-frequency insights appear, the claim that this list is a consolidated requirements baseline for enterprise preprocessing would not hold.","supporting_citations":[{"cited_title":"Research Methods in Human- Computer Interaction, 2nd Edition","cited_arxiv_id":null,"evidence_quote":"This sets the 10-to-15 participant target for the semi-structured interview method used to gather the interview data."}],"review_version":1}