Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

After two decades, mixed-initiative visual analytics still lacks a shared definition and mostly runs at low automation levels.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-04 15:23 UTC pith:JUVIMN4M

load-bearing objection A solid scoping review with a genuinely useful integrated taxonomy; the 'limited potential' finding is real but rests on a self-selected sample that the authors openly acknowledge, so treat that claim as suggestive rather than definitive. the 3 major comments →

arxiv 2509.19152 v2 pith:JUVIMN4M submitted 2025-09-23 cs.HC

A Scoping Review of Mixed Initiative Visual Analytics in the Automation Renaissance

Shayan Monadjemi , Yuhan Guo , Kai Xu , Alex Endert , Anamaria Crisan This is my paper
classification cs.HC
keywords visual analyticsmixed-initiativehuman-AI collaborationscoping reviewtaxonomyautomation levelsvisualization systemsartificial agents
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that the field of mixed-initiative visual analytics—software where a human and an automated agent jointly perform visual data analysis—has no consensus on what 'mixed-initiative' means and has only explored a narrow slice of possible human-AI collaborations. To show this, the authors reviewed 86 papers from the past twenty years and built an integrated taxonomy of 80 characteristics across 7 attributes that describes these systems. The analysis finds that most systems use low automation levels where the computer suggests a few options and the human remains in control. If true, this gives the community a shared vocabulary and a map of unexplored design choices just as AI tools are becoming mainstream.

Core claim

The paper's central discovery is that the visualization literature lacks consensus on the definition of mixed-initiative systems and explores a limited potential of the collaborative interaction landscape. The authors support this with a two-phase qualitative scoping review: a codebook developed from 60 papers and applied to 36 self-identifying mixed-initiative visual analytics systems. The resulting integrated taxonomy characterizes each system along 7 attributes—human contributions, artificial agent contributions, shared task, impact, level of automation, adherence to mixed-initiative principles, and evaluation method. The empirical pattern shows that only 5 of the 10 levels of automation

What carries the argument

The integrated taxonomy is the central mechanism: it merges top-down theoretical frameworks (Parasuraman's ten-level automation scale, Holstein's four contribution types, Horvitz's twelve mixed-initiative principles) with bottom-up codes derived from the reviewed papers. It treats a mixed-initiative system as an arrangement of human agents, artificial agents, and a shared visual analytic environment, and assigns 80 codes across 7 attributes to describe that arrangement. The taxonomy does the argumentative work of making the design space visible and comparable, so that the observed clustering at low automation levels and the gaps at high automation levels can be stated concretely.

Load-bearing premise

The Phase 2 analysis assumes that the set of papers that self-identify as 'mixed-initiative' in title or abstract fairly represents the full space of mixed-initiative visual analytics systems.

What would settle it

A complementary scoping search that adds terms such as 'human-in-the-loop,' 'guidance,' 'co-adaptive,' and 'human-AI collaboration' without requiring 'mixed-initiative' in the title/abstract, then codes the same taxonomy attributes; if this broader corpus contains many systems at automation levels 6-10, the claim that the field explores limited potential would be weakened.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Future mixed-initiative systems could deliberately occupy the unexplored levels of automation (e.g., levels 6-10) to test whether more autonomous AI benefits visual analysis.
  • The field would benefit from adopting a consensus definition of 'mixed-initiative' so results across papers can be compared and accumulated.
  • Evaluations should include baselines against an AI agent completing the task alone, not just against humans alone, to demonstrate the value of mixed-initiative interaction.
  • The taxonomy provides a scaffold for describing and positioning new systems, including those built on large language models, within the existing design space.
  • The sparse adoption of Horvitz's principles suggests specific design guidelines that are ripe for implementation as AI capabilities grow.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The self-selection filter (papers must contain 'mixed-initiative' in title/abstract) may under-count systems that use terms like 'human-in-the-loop' or 'guidance'; if those systems reach higher automation, the 'limited potential' claim could be biased, a limitation the authors explicitly acknowledge.
  • As LLM-based data agents (e.g., chat-based analysis tools) become common, they may naturally push toward higher automation levels (e.g., suggesting a single action or executing and informing), suggesting the taxonomy's upper levels may soon become populated.
  • A testable extension would be to apply the same taxonomy to a corpus assembled with broader inclusion terms and compare automation-level distributions; divergence would indicate that the field's under-exploration is partly an artifact of labeling.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper presents a two-phase scoping review of mixed-initiative visual analytics systems. In Phase 1, the authors develop an integrated taxonomy from 10 seed papers and 50 papers from a prior review on human-computer collaboration, combining bottom-up coding with existing frameworks (Holstein et al., Parasuraman et al., Horvitz, Domova and Vrotsou). In Phase 2, they apply this taxonomy to 36 papers that self-identify as mixed-initiative in title or abstract and report distributions across seven attributes: human and AI contributions, shared task, impact, level of automation, evaluation methods, and observed Horvitz principles. The main claims are that the visualization literature lacks consensus on the definition of mixed-initiative systems and that the field has explored only a limited portion of the collaborative interaction design space, concentrating on low levels of automation. The paper also contributes a publicly available web implementation of the taxonomy.

Significance. The paper addresses a timely and important question, and its strengths are real: the two-phase, top-down/bottom-up design is appropriate for a scoping review; the selection process is reported with a PRISMA-style diagram; dual coding and inter-coder reliability are reported transparently; the taxonomy is grounded in established frameworks; and the web tool is a useful community resource. The review also makes an observable, falsifiable claim—only five of ten automation levels appear in the self-identified sample—which can inform future work. However, the significance of the field-level conclusions is currently limited by two issues: the Phase 2 corpus is restricted to papers that use the exact term 'mixed-initiative,' and the pre-consensus coding reliability is modest. If these concerns are addressed, the taxonomy would be a valuable scaffold for discussing human-AI collaboration in visual analytics. As written, the headline claims outrun the evidence.

major comments (3)
  1. [§4.2, §6.3, abstract/§6.1.3] The central claim that the field 'explores a limited potential' is inferred exclusively from 36 papers that self-identify via 'mixed initiative' in the title or abstract. The Phase 1 sample (10 seed papers plus 50 papers from a prior human-computer collaboration review) includes systems that use other terminology (e.g., semantic interaction, guidance, human-AI collaboration), but those 60 papers were used only to build the codebook, not to characterize the design space or to test whether the Phase 2 distribution is representative. The limitation in §6.3 concedes this but does not qualify the abstract or §6.1.3. Because the absence of high automation levels (Table 5) is the main evidence for 'limited potential,' this is a sampling-artifact risk, not a presentation nit. I recommend either restricting field-level conclusions to the self-identified corpus or adding a robustness check, e.g.,
  2. [§4.2 and Appendix B] The mean inter-coder reliability of 0.57 (range 0.45–0.74, Jaccard) is low to moderate. The authors report this transparently, but the rest of the paper treats the consensus codes as unproblematic. No information is given about the distribution of conflict outcomes (one coder's choice vs. merged vs. new code), nor about whether consensus discussions systematically shifted codes in one direction. Table 9 shows many rows with zero agreement before consensus for the Level of Automation and MI Principles attributes, yet these attributes feed directly into Table 5 and Table 7. Pre-consensus disagreement does not automatically invalidate consensus results, but the paper should provide a sensitivity discussion or attribute-level caution. Without it, quantitative-sounding statements such as '17 of 36 papers use interviews' are presented as stable measurements of an ambiguous corpus. Please repor
  3. [§6.1.2 and abstract] The claim that the visualization literature 'lacks consensus on the definition of mixed-initiative systems' is asserted rather than demonstrated. The inclusion criterion was self-identification, not a definitional analysis, and the only direct evidence offered is the authors' difficulty aligning systems with Horvitz's principles. A reader cannot tell whether the field lacks consensus or whether the reviewed papers simply do not use the authors' chosen frameworks. Please either provide a definitional analysis (e.g., extract and compare how each Phase 2 paper defines mixed-initiative, or at least quote divergent definitions) or soften the claim to something like 'the reviewed literature rarely invokes a shared formal definition.'
minor comments (5)
  1. [Throughout] Typos: 'persepctive' (§3), 'synanymous' (§5.2), 'mixed-initative' (§5.3), 'Horvtiz' (§6.1.2), and 'In doing, so' (§6.2.3). These should be corrected before publication.
  2. [Figure 3 / Appendix A] The PRISMA diagram uses three phases (Phase 1, Phase 2, Phase 3) while the paper's method section describes only two phases. This inconsistency will confuse readers; relabel the columns to match the two-phase structure or explain the third phase explicitly.
  3. [Table 5] Dupo [52] appears under both level 3 and level 4. The text in §5.4 explains that the system has two modules, but a table footnote would make this less surprising to a reader scanning the table.
  4. [Footnote 3] The inserted question about whether an artificial agent should be able to terminate human operations interrupts the argument. Either integrate it into the main text as a design question or remove it; as a footnote it reads as an unresolved meta-comment.
  5. [§5.5] The claim that only 8 of 36 papers include empirical comparison is important but the listed references in Table 6 do not make it easy to verify. A short list of the 8 papers or a marker in the table would strengthen the point.

Circularity Check

0 steps flagged

No significant circularity: the integrated taxonomy and field-level claims are built from external frameworks plus coding of a sampled corpus; the acknowledged self-identification limitation is an external-validity caveat, not a circular step.

full rationale

This paper is a qualitative scoping review, not a predictive derivation. The derivation chain is: (1) Phase 1 builds an integrated taxonomy from 10 author-selected exemplar papers, 50 papers collected in prior author work [66], and established external frameworks (Parasuraman et al. [75], Horvitz [45], Holstein et al. [44], Domova and Vrotsou [28]); (2) Phase 2 applies that codebook to 36 papers systematically collected by self-identification as 'mixed-initiative'; (3) findings such as 'lacks consensus' and 'limited potential' are descriptive codings of that corpus using the external Parasuraman 10-level automation scale. No fitted parameter is renamed as a prediction, no equation defines the finding into existence, and no uniqueness theorem is imported from the authors' prior work. The self-citations to Monadjemi et al. [66] supply the prior 50-paper sample and an agent-based framing, but they are not load-bearing evidence for the central claims; the taxonomy's content is independently anchored by external taxonomies and bottom-up coding. The passage in Section 6.3 -- 'we limited our analysis to only papers that self-identify as such. We acknowledge that there exist papers that employ mixed-initiative approaches without the use of this specific term' -- is a genuine limitation on external validity: if non-self-identified mixed-initiative systems occupy different automation levels, the 'limited potential' claim could be an artifact of the inclusion filter. However, that is a sampling and generalization concern, not circularity, because the claim is not equivalent to the filter by construction. The paper is self-contained as a review; its claims are explicitly about the reviewed corpus, and the acknowledged limitation is appropriately disclosed rather than hidden.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

The paper is a qualitative review, so there are no fitted parameters or invented entities. The main assumptions are about the completeness of the literature sample, the validity of self-identification as a criterion, the applicability of external frameworks, and the reliability of the coding process.

axioms (4)
  • domain assumption The literature search using Semantic Scholar API and venue filters captures the relevant population of mixed-initiative visual analytics papers.
    Invoked in section 4.2; the entire Phase 2 corpus depends on this search being complete and unbiased.
  • domain assumption Self-identification as 'mixed-initiative' in title or abstract is a valid proxy for membership in the class of mixed-initiative visual analytics systems.
    Section 4.2 uses this filter; the authors acknowledge in section 6.3 that systems not using the term are excluded, which could bias findings.
  • domain assumption Existing taxonomies (Parasuraman et al. [75], Horvitz [45], Holstein et al. [44]) are valid and applicable to visual analytics systems.
    Used throughout section 5 as the top-down coding framework; the validity of these frameworks is taken as given.
  • domain assumption Qualitative coding and consensus-based conflict resolution produce accurate and reliable characterizations of the reviewed papers.
    The results depend entirely on the coding process described in section 4.2; low pre-consensus ICR values (0.45-0.74) indicate this axiom is not fully satisfied.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of A Scoping Review of Mixed Initiative Visual Analytics in the Automation Renaissance." pith.science (2026). https://pith.science/paper/JUVIMN4M

@misc{pith2026250919152,
  author       = {Pith},
  title        = {Pith review of: A Scoping Review of Mixed Initiative Visual Analytics in the Automation Renaissance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JUVIMN4M}},
  note         = {Machine review of arXiv:2509.19152}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Artificial agents are increasingly integrated into data analysis workflows, carrying out tasks that were primarily done by humans. Our research explores how the introduction of automation recalibrates the dynamic between humans and automating technology. To explore this question, we conducted a scoping review encompassing twenty years of mixed-initiative visual analytic systems. To describe and contrast the relationship between humans and automation, we developed an integrated taxonomy to delineate the objectives of these mixed-initiative visual analytics tools, how much automation they support, and the assumed roles of humans. Here, we describe our qualitative approach of integrating existing theoretical frameworks with new codes we developed. Our analysis shows that the visualization research literature lacks consensus on the definition of mixed-initiative systems and explores a limited potential of the collaborative interaction landscape between people and automation. Our research provides a scaffold to advance the discussion of human-AI collaboration during visual data analysis. Our integrated taxonomy is available in the form of a web application on https://smonadjemi.github.io/miva.

Figures

Figures reproduced from arXiv: 2509.19152 by Alex Endert, Anamaria Crisan, Kai Xu, Shayan Monadjemi, Yuhan Guo.

Figure 1
Figure 1. Figure 1: The overview of the research questions in our scoping review, where we aim to characterize [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of our two-phase methods: (1) develop the codebook, and (2) apply the codebook. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The PRISMA diagrams of the paper selection process for the three phases. [PITH_FULL_IMAGE:figures/full_fig_p025_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The distribution density (KDE) of ICR values per coded attribute. We observe a multi-modal distribution for most attributes, [PITH_FULL_IMAGE:figures/full_fig_p027_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. BONSAI: A Mixed-Initiative Workspace for Human-AI Co-Development of Visual Analytics Applications

    cs.HC 2026-04 unverdicted novelty 6.0

    BONSAI introduces a four-layer architecture and four-phase workflow for human-AI co-development of visual analytics applications, shown in case studies to enable efficient novel tool creation and reconstruction from p...

Reference graph

Works this paper leans on

139 extracted references · 14 canonical work pages · cited by 1 Pith paper

  1. [1]

    Sriram Karthik Badam, Jieqiong Zhao, Niklas Elmqvist, and David S. Ebert. 2014. TimeFork: Mixed-initiative time-series prediction. In2014 IEEE Conference on Visual Analytics Science and Technology (V AST). IEEE, Paris, France, 223–224. doi:10.1109/VAST.2014.7042501

    arXiv 2014

  2. [2]

    Leilani Battle, Remco Chang, and Michael Stonebraker. 2016. Dynamic Prefetching of Data Tiles for Interactive Visualization. InProceedings of the 2016 International Conference on Management of Data. ACM, San Francisco California USA, 1363–1375. doi:10.1145/2882903.2882919

    arXiv 2016

  3. [3]

    Jurgen Bernard, Nils Wilhelm, Bjorn Kruger, Thorsten May, Tobias Schreck, and Jorn Kohlhammer. 2013. MotionExplorer: Exploratory Search in Human Motion Capture Data Based on Hierarchical Aggregation.IEEE Trans. Visual. Comput. Graphics19, 12 (Dec. 2013), 2257–2266. doi:10.1109/TVCG.2013.178

    work page doi:10.1109/tvcg.2013.178 2013

  4. [4]

    Enrico Bertini and Denis Lalanne. 2010. Investigating and reflecting on the integration of automatic data analysis and visualization in knowledge discovery.SIGKDD Explor. Newsl.11, 2 (May 2010), 9–18. doi:10.1145/1809400.1809404

    arXiv 2010

  5. [5]

    Alexander Bock, Harish Doraiswamy, Adam Summers, and Claudio Silva. 2018. TopoAngler: Interactive Topology-Based Extraction of Fishes.IEEE Trans. Visual. Comput. Graphics24, 1 (Jan. 2018), 812–821. doi:10.1109/TVCG.2017.2743980

    arXiv 2018

  6. [6]

    Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S

    Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stef...

    Pith/arXiv arXiv doi:10.48550/arxiv.2108.07258 2022

  7. [7]

    Eli T Brown, Alvitta Ottley, Helen Zhao, Quan Lin, Richard Souvenir, Alex Endert, and Remco Chang. 2014. Finding Waldo: Learning about Users from their Interactions.IEEE Trans. Visual. Comput. Graphics20, 12 (Dec. 2014), 1663–1672. doi:10.1109/TVCG.2014.2346575

    arXiv 2014

  8. [9]

    Cai, Emily Reif, Narayan Hegde, Jason Hipp, Been Kim, Daniel Smilkov, Martin Wattenberg, Fernanda Viegas, Greg S

    Carrie J. Cai, Emily Reif, Narayan Hegde, Jason Hipp, Been Kim, Daniel Smilkov, Martin Wattenberg, Fernanda Viegas, Greg S. Corrado, Martin C. Stumpe, and Michael Terry. 2019. Human-Centered Tools for Coping with Imperfect Algorithms During Medical Decision-Making. InProceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, Glasgo...

    arXiv 2019

  9. [10]

    Davide Ceneda, Theresia Gschwandtner, Thorsten May, Silvia Miksch, Hans-Jorg Schulz, Marc Streit, and Christian Tominski. 2017. Characterizing Guidance in Visual Analytics.IEEE Trans. Visual. Comput. Graphics23, 1 (Jan. 2017), 111–120. doi:10.1109/TVCG.2016.2598468

    arXiv 2017

  10. [11]

    Hong, and Christos Faloutsos

    Duen Horng Chau, Aniket Kittur, Jason I. Hong, and Christos Faloutsos. 2011. Apolo: interactive large graph sensemaking by combining machine learning and visualization. InProceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, San Diego California USA, 739–742. doi:10.1145/2020408.2020524

    arXiv 2011

  11. [12]

    Hong, and Christos Faloutsos

    Duen Horng Chau, Aniket Kittur, Jason I. Hong, and Christos Faloutsos. 2011. Apolo: making sense of large network data by combining rich user interaction and machine learning. InProceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, Vancouver BC Canada, 167–176. doi:10.1145/1978942.1978967

    arXiv 2011

  12. [13]

    Chen Chen, Bongshin Lee, Yunhai Wang, Yunjeong Chang, and Zhicheng Liu. 2024. Mystique: Deconstructing SVG Charts for Layout Reuse.IEEE Trans. Visual. Comput. Graphics30, 1 (Jan. 2024), 447–457. doi:10.1109/TVCG.2023.3327354

    arXiv 2024

  13. [14]

    Nan-Chen Chen, Margaret Drouhard, Rafal Kocielnik, Jina Suh, and Cecilia R. Aragon. 2018. Using Machine Learning to Support Qualitative Coding in Social Science: Shifting the Focus to Ambiguity.ACM Trans. Interact. Intell. Syst.8, 2 (June 2018), 1–20. doi:10.1145/3185515

    doi:10.1145/3185515 2018

  14. [15]

    Chilton, Savvas Petridis, and Maneesh Agrawala

    Lydia B. Chilton, Savvas Petridis, and Maneesh Agrawala. 2019. VisiBlends: A Flexible Workflow for Visual Blends. InProceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, Glasgow Scotland Uk, 1–14. doi:10.1145/3290605.3300402

    arXiv 2019

  15. [16]

    Jiwon Choi and Jaemin Jo. 2022. Intentable: A Mixed-Initiative System for Intent-Based Chart Captioning. In2022 IEEE Visualization and Visual Analytics (VIS). IEEE, Oklahoma City, OK, USA, 40–44. doi:10.1109/VIS54862.2022.00017

    arXiv 2022

  16. [17]

    Christopher Collins, Natalia Andrienko, Tobias Schreck, Jing Yang, Jaegul Choo, Ulrich Engelke, Amit Jena, and Tim Dwyer. 2018. Guidance in the human–machine analytics process.Visual Informatics2, 3 (Sept. 2018), 166–180. doi:10.1016/j.visinf.2018.09.003

    work page doi:10.1016/j.visinf.2018.09.003 2018

  17. [18]

    Kristin Cook, Nick Cramer, David Israel, Michael Wolverton, Joe Bruce, Russ Burtner, and Alex Endert. 2015. Mixed-initiative visual analytics using task-driven recommendations. In2015 IEEE Conference on Visual Analytics Science and Technology (V AST). IEEE, Chicago, IL, USA, 9–16. doi:10.1109/VAST.2015.7347625

    arXiv 2015

  18. [20]

    Lafrance, Nick Cramer, Kristin Cook, and Samuel Payne

    Aritra Dasgupta, Joon-Yong Lee, Ryan Wilson, Robert A. Lafrance, Nick Cramer, Kristin Cook, and Samuel Payne. 2017. Familiarity Vs Trust: A Comparative Study of Domain Scientists’ Trust in Visual Analytics and Conventional Analysis Methods.IEEE Trans. Visual. Comput. Graphics23, 1 (Jan. 2017), 271–280. doi:10.1109/TVCG.2016.2598544

    arXiv 2017

  19. [21]

    Davis, C

    N. Davis, C. Hsiao, K. Y. Singh, B. Lin, and B. Magerko. 2017. Quantifying Collaboration with a Co-Creative Drawing Agent.ACM Trans. Interact. Intell. Syst.7, 4 (Dec. 2017), 1–25. doi:10.1145/3009981

    doi:10.1145/3009981 2017

  20. [24]

    Vaishali Dhanoa, Anton Wolter, Gabriela Molina León, Hans-Jörg Schulz, and Niklas Elmqvist. 2025. Agentic Visualization: Extracting Agent-based Design Patterns from Visualization Systems. doi:10.48550/ARXIV.2505.19101 Version Number: 2

    work page doi:10.48550/arxiv.2505.19101 2025

  21. [25]

    Rundensteiner, and Matthew O

    Di Yang, Elke A. Rundensteiner, and Matthew O. Ward. 2007. Analysis Guided Visual Exploration of Multivariate Data. In2007 IEEE Symposium on Visual Analytics Science and Technology. IEEE, Sacramento, CA, USA, 83–90. doi:10.1109/VAST.2007.4389000

    arXiv 2007

  22. [26]

    Diehl, L

    A. Diehl, L. Pelorosso, C. Delrieux, C. Saulo, J. Ruiz, M. E. Gröller, and S. Bruckner. 2015. Visual Analysis of Spatio-Temporal Data: Applications in Weather Forecasting.Computer Graphics Forum34, 3 (June 2015), 381–390. doi:10.1111/cgf.12650

    work page doi:10.1111/cgf.12650 2015

  23. [27]

    Karthik Dinakar, Jackie Chen, Henry Lieberman, Rosalind Picard, and Robert Filbin. 2015. Mixed-Initiative Real-Time Topic Modeling & Visualization for Crisis Counseling. InProceedings of the 20th International Conference on Intelligent User Interfaces. ACM, Atlanta Georgia USA, 417–426. doi:10.1145/2678025.2701395

    arXiv 2015

  24. [28]

    Veronika Domova and Katerina Vrotsou. 2023. A Model for Types and Levels of Automation in Visual Analytics: A Survey, a Taxonomy, and Examples.IEEE Trans. Visual. Comput. Graphics29, 8 (Aug. 2023), 3550–3568. doi:10.1109/TVCG.2022.3163765

    arXiv 2023

  25. [29]

    Mennatallah El-Assady, Fabian Sperrle, Oliver Deussen, Daniel Keim, and Christopher Collins. 2019. Visual Analytics for Topic Model Optimization based on User-Steerable Speculative Execution.IEEE Trans. Visual. Comput. Graphics25, 1 (Jan. 2019), 374–384. doi:10.1109/TVCG.2018.2864769

    arXiv 2019

  26. [31]

    Jinjuan Feng, Jonathan Lazar, Libby Kumin, and Ant Ozok. 2010. Computer Usage by Children with Down Syndrome: Challenges and Future Research.ACM Trans. Access. Comput.2, 3 (March 2010), 1–44. doi:10.1145/1714458.1714460

    arXiv 2010

  27. [32]

    Shi Feng and Jordan Boyd-Graber. 2019. What can AI do for me?: evaluating machine learning interpretations in cooperative play. InProceedings of the 24th International Conference on Intelligent User Interfaces. ACM, Marina del Ray California, 229–239. doi:10.1145/3301275.3302265 Manuscript submitted to ACM A Scoping Review of Mixed Initiative Visual Analy...

    arXiv 2019

  28. [35]

    Sebastian Gehrmann, Hendrik Strobelt, Robert Kruger, Hanspeter Pfister, and Alexander M. Rush. 2019. Visual Interaction with Deep Learning Models through Collaborative Semantic Inference.IEEE Trans. Visual. Comput. Graphics(2019), 1–1. doi:10.1109/TVCG.2019.2934595

    arXiv 2019

  29. [36]

    1967.The Discovery of Grounded Theory

    Barney G Glaser and Anselm L Strauss. 1967.The Discovery of Grounded Theory. Aldine, Chicago

    1967

  30. [37]

    Dorota Glowacka, Tuukka Ruotsalo, Ksenia Konuyshkova, Kumaripaba Athukorala, Samuel Kaski, and Giulio Jacucci. 2013. Directing exploratory search: reinforcement learning from user interactions with keywords. InProceedings of the 2013 international conference on Intelligent user interfaces. ACM, Santa Monica California USA, 117–128. doi:10.1145/2449396.2449413

    arXiv 2013

  31. [38]

    David Gotz, Shun Sun, and Nan Cao. 2016. Adaptive Contextualization: Combating Bias During High-Dimensional Visualization and Data Selection. InProceedings of the 21st International Conference on Intelligent User Interfaces. ACM, Sonoma California USA, 85–95. doi:10.1145/2856767.2856779

    arXiv 2016

  32. [39]

    David Gotz and Zhen Wen. 2009. Behavior-driven visualization recommendation. InProceedings of the 14th international conference on Intelligent user interfaces. ACM, Sanibel Island Florida USA, 315–324. doi:10.1145/1502650.1502695

    arXiv 2009

  33. [40]

    Spence Green, Jason Chuang, Jeffrey Heer, and Christopher D. Manning. 2014. Predictive translation memory: a mixed-initiative system for human language translation. InProceedings of the 27th annual ACM symposium on User interface software and technology. ACM, Honolulu Hawaii USA, 177–187. doi:10.1145/2642918.2647408

    arXiv 2014

  34. [41]

    Ken Gu, Madeleine Grunde-McLaughlin, Andrew McNutt, Jeffrey Heer, and Tim Althoff. 2024. How Do Data Analysts Respond to AI Assistance? A Wizard-of-Oz Study. InProceedings of the CHI Conference on Human Factors in Computing Systems. ACM, Honolulu HI USA, 1–22. doi:10.1145/ 3613904.3641891

    arXiv 2024

  35. [42]

    Healey, S

    C.G. Healey, S. Kocherlakota, V. Rao, R. Mehta, and R. St. Amant. 2008. Visual Perception and Mixed-Initiative Interaction for Assisted Visualization Design.IEEE Trans. Visual. Comput. Graphics14, 2 (March 2008), 396–411. doi:10.1109/TVCG.2007.70436

    arXiv 2008

  36. [43]

    Healey and Brent M

    Christopher G. Healey and Brent M. Dennis. 2012. Interest Driven Navigation in Visualization.IEEE Trans. Visual. Comput. Graphics18, 10 (Oct. 2012), 1744–1756. doi:10.1109/TVCG.2012.23

    work page doi:10.1109/tvcg.2012.23 2012

  37. [44]

    Kenneth Holstein, Vincent Aleven, and Nikol Rummel. 2020. A Conceptual Framework for Human–AI Hybrid Adaptivity in Education. InArtificial Intelligence in Education, Ig Ibert Bittencourt, Mutlu Cukurova, Kasia Muldner, Rose Luckin, and Eva Millán (Eds.). Vol. 12163. Springer International Publishing, Cham, 240–254. doi:10.1007/978-3-030-52237-7_20 Series ...

    work page doi:10.1007/978-3-030-52237-7_20 2020

  38. [45]

    Eric Horvitz. 1999. Principles of mixed-initiative user interfaces. InProceedings of the SIGCHI conference on Human factors in computing systems the CHI is the limit - CHI ’99. ACM Press, Pittsburgh, Pennsylvania, United States, 159–166. doi:10.1145/302979.303030

    arXiv 1999

  39. [46]

    Fahd Husain, Pascale Proulx, Meng-Wei Chang, Rosa Romero-Gomez, and Holland Vasquez. 2021. A Mixed-Initiative Visual Analytics Approach for Qualitative Causal Modeling. In2021 IEEE Visualization Conference (VIS). IEEE, New Orleans, LA, USA, 121–125. doi:10.1109/VIS49827.2021.9623318

    arXiv 2021

  40. [47]

    Shichao Jia, Zeyu Li, Nuo Chen, and Jiawan Zhang. 2022. Towards Visual Explainable Active Learning for Zero-Shot Classification.IEEE Trans. Visual. Comput. Graphics28, 1 (Jan. 2022), 791–801. doi:10.1109/TVCG.2021.3114793

    arXiv 2022

  41. [48]

    Seokweon Jung, Kiroong Choe, Seokhyeon Park, Hyung-Kwon Ko, Youngtaek Kim, and Jinwook Seo. 2021. Mixed-Initiative Approach to Extract Data from Pictures of Medical Invoice. In2021 IEEE 14th Pacific Visualization Symposium (PacificVis). IEEE, Tianjin, China, 111–115. doi:10.1109/PacificVis52677.2021.00022

    arXiv 2021

  42. [49]

    Sean Kandel, Andreas Paepcke, Joseph Hellerstein, and Jeffrey Heer. 2011. Wrangler: interactive visual specification of data transformation scripts. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, Vancouver BC Canada, 3363–3372. doi:10.1145/1978942.1979444

    arXiv 2011

  43. [50]

    Daniel Keim, Gennady Andrienko, Jean-Daniel Fekete, Carsten Görg, Jörn Kohlhammer, and Guy Melançon. 2008. Visual Analytics: Definition, Process, and Challenges. InInformation Visualization, Andreas Kerren, John T. Stasko, Jean-Daniel Fekete, and Chris North (Eds.). Vol. 4950. Springer Berlin Heidelberg, Berlin, Heidelberg, 154–175. doi:10.1007/978-3-540-...

    doi:10.1007/978-3-540-70956-5_7 2008

  44. [51]

    Hannah Kim, Dongjin Choi, Barry Drake, Alex Endert, and Haesun Park. 2019. TopicSifter: Interactive Search Space Reduction through Targeted Topic Modeling. In2019 IEEE Conference on Visual Analytics Science and Technology (V AST). IEEE, Vancouver, BC, Canada, 35–45. doi:10.1109/ VAST47406.2019.8986922

    arXiv 2019

  45. [52]

    Hyeok Kim, Ryan Rossi, Jessica Hullman, and Jane Hoffswell. 2023. Dupo: A Mixed-Initiative Authoring Tool for Responsive Visualization.IEEE Trans. Visual. Comput. Graphics(2023), 1–10. doi:10.1109/TVCG.2023.3326583

    arXiv 2023

  46. [53]

    Nicholas Kong, Ben Hanrahan, Thiébaud Weksteen, Gregorio Convertino, and Ed H. Chi. 2011. VisualWikiCurator: human and machine intelligencefor organizing wiki content. InProceedings of the 16th international conference on Intelligent user interfaces. ACM, Palo Alto CA USA, 367–370. doi:10.1145/1943403.1943467

    arXiv 2011

  47. [54]

    Kuno Kurzhals, Brian Fisher, Michael Burch, and Daniel Weiskopf. 2016. Eye tracking evaluation of visual analytics.Information Visualization15, 4 (Oct. 2016), 340–358. doi:10.1177/1473871615609787

    work page doi:10.1177/1473871615609787 2016

  48. [55]

    Lam, Janice Teoh, James A

    Michelle S. Lam, Janice Teoh, James A. Landay, Jeffrey Heer, and Michael S. Bernstein. 2024. Concept Induction: Analyzing Unstructured Text with High-Level Concepts Using LLooM. InProceedings of the CHI Conference on Human Factors in Computing Systems. ACM, Honolulu HI USA, 1–28. doi:10.1145/3613904.3642830 Manuscript submitted to ACM 22 Monadjemi et al

    arXiv 2024

  49. [57]

    Jonathan Lazar, Jinjuan Heidi Feng, and Harry Hochheiser. 2017. Analyzing qualitative data. InResearch Methods in Human Computer Interaction. Elsevier, 299–327. doi:10.1016/B978-0-12-805390-4.00011-X

    work page doi:10.1016/b978-0-12-805390-4.00011-x 2017

  50. [58]

    Doris Jung-Lin Lee, Himel Dev, Huizi Hu, Hazem Elmeleegy, and Aditya Parameswaran. 2019. Avoiding drill-down fallacies withVisPilot: assisted exploration of data subsets. InProceedings of the 24th International Conference on Intelligent User Interfaces. ACM, Marina del Ray California, 186–196. doi:10.1145/3301275.3302307

    arXiv 2019

  51. [59]

    Yanna Lin, Haotian Li, Leni Yang, Aoyu Wu, and Huamin Qu. 2023. InkSight: Leveraging Sketch Interaction for Documenting Chart Findings in Computational Notebooks.IEEE Trans. Visual. Comput. Graphics(2023), 1–11. doi:10.1109/TVCG.2023.3327170

    arXiv 2023

  52. [60]

    Martin Lindvall, Claes Lundström, and Jonas Löwgren. 2021. Rapid Assisted Visual Search: Supporting Digital Pathologists with Imperfect AI. In 26th International Conference on Intelligent User Interfaces. ACM, College Station TX USA, 504–513. doi:10.1145/3397481.3450681

    arXiv 2021

  53. [62]

    Derek Lomas, Jodi Forlizzi, Nikhil Poonwala, Nirmal Patel, Sharan Shodhan, Kishan Patel, Ken Koedinger, and Emma Brunskill

    J. Derek Lomas, Jodi Forlizzi, Nikhil Poonwala, Nirmal Patel, Sharan Shodhan, Kishan Patel, Ken Koedinger, and Emma Brunskill. 2016. Interface Design Optimization as a Multi-Armed Bandit Problem. InProceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, San Jose California USA, 4142–4153. doi:10.1145/2858036.2858425

    arXiv 2016

  54. [63]

    1998.The future of air traffic control: Human operators and automation

    James P McGee, Raja Parasuraman, Anne S Mavor, and Christopher D Wickens. 1998.The future of air traffic control: Human operators and automation. National Academies Press

    1998

  55. [64]

    Fabio Miranda, Maryam Hosseini, Marcos Lage, Harish Doraiswamy, Graham Dove, and Cláudio T. Silva. 2020. Urban Mosaic: Visual Exploration of Streetscapes Using Large-Scale Image Data. InProceedings of the 2020 CHI Conference on Human Factors in Computing Systems. ACM, Honolulu HI USA, 1–15. doi:10.1145/3313831.3376399

    arXiv 2020

  56. [66]

    Shayan Monadjemi, Mengtian Guo, David Gotz, Roman Garnett, and Alvitta Ottley. 2023. Human–Computer Collaboration for Visual Analytics: an Agent-based Framework.Computer Graphics Forum42, 3 (June 2023), 199–210. doi:10.1111/cgf.14823

    work page doi:10.1111/cgf.14823 2023

  57. [67]

    Shayan Monadjemi, Sunwoo Ha, Quan Nguyen, Henry Chai, Roman Garnett, and Alvitta Ottley. 2022. Guided Data Discovery in Interactive Visualizations via Active Search. In2022 IEEE Visualization and Visual Analytics (VIS). IEEE, Oklahoma City, OK, USA, 70–74. doi:10.1109/VIS54862. 2022.00023

    arXiv 2022

  58. [68]

    Thomas Muhlbacher, Lorenz Linhardt, Torsten Moller, and Harald Piringer. 2018. TreePOD: Sensitivity-Aware Selection of Pareto-Optimal Decision Trees.IEEE Trans. Visual. Comput. Graphics24, 1 (Jan. 2018), 174–183. doi:10.1109/TVCG.2017.2745158

    arXiv 2018

  59. [69]

    Zachary Munn, Micah D. J. Peters, Cindy Stern, Catalin Tufanaru, Alexa McArthur, and Edoardo Aromataris. 2018. Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach.BMC Med Res Methodol18, 1 (Dec. 2018), 143. doi:10.1186/s12874-018-0611-x

    doi:10.1186/s12874-018-0611-x 2018

  60. [71]

    Alvitta Ottley, Roman Garnett, and Ran Wan. 2019. Follow The Clicks: Learning and Anticipating Mouse Interactions During Exploratory Data Analysis.Computer Graphics Forum38, 3 (June 2019), 41–52. doi:10.1111/cgf.13670

    doi:10.1111/cgf.13670 2019

  61. [73]

    Aditeya Pandey, Arjun Srinivasan, and Vidya Setlur. 2023. MEDLEY: Intent-based Recommendations to Support Dashboard Composition.IEEE Trans. Visual. Comput. Graphics29, 1 (Jan. 2023), 1135–1145. doi:10.1109/TVCG.2022.3209421

    arXiv 2023

  62. [74]

    Prateek Panwar and Christopher M. Collins. 2018. Detecting Negative Emotion for Mixed Initiative Visual Analytics. InExtended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, Montreal QC Canada, 1–6. doi:10.1145/3170427.3188664

    arXiv 2018

  63. [75]

    Parasuraman, T.B

    R. Parasuraman, T.B. Sheridan, and C.D. Wickens. 2000. A model for types and levels of human interaction with automation.IEEE Trans. Syst., Man, Cybern. A30, 3 (May 2000), 286–297. doi:10.1109/3468.844354

    arXiv 2000

  64. [76]

    Deokgun Park, Seungyeon Kim, Jurim Lee, Jaegul Choo, Nicholas Diakopoulos, and Niklas Elmqvist. 2018. ConceptVector: Text Visual Analytics via Interactive Lexicon Building Using Word Embedding.IEEE Trans. Visual. Comput. Graphics24, 1 (Jan. 2018), 361–370. doi:10.1109/TVCG.2017.2744478

    arXiv 2018

  65. [77]

    Alexis Pister, Paolo Buono, Jean-Daniel Fekete, Catherine Plaisant, and Paola Valdivia. 2021. Integrating Prior Knowledge in Mixed-Initiative Social Network Clustering.IEEE Trans. Visual. Comput. Graphics27, 2 (Feb. 2021), 1775–1785. doi:10.1109/TVCG.2020.3030347

    arXiv 2021

  66. [78]

    Pu and D

    P. Pu and D. Lalanne. 2000. Interactive problem solving via algorithm visualization. InIEEE Symposium on Information Visualization 2000. INFOVIS

    2000

  67. [79]

    Pearl Pu and Denis Lalanne. 2002. Design visual thinking tools for mixed initiative systems. InProceedings of the 7th international conference on Intelligent user interfaces. ACM, San Francisco California USA, 119–126. doi:10.1145/502716.502736

    arXiv 2002

  68. [80]

    Pérez-Messina, D

    I. Pérez-Messina, D. Ceneda, M. El-Assady, S. Miksch, and F. Sperrle. 2022. A Typology of Guidance Tasks in Mixed-Initiative Visual Analytics Environments.Computer Graphics Forum41, 3 (June 2022), 465–476. doi:10.1111/cgf.14555 Manuscript submitted to ACM A Scoping Review of Mixed Initiative Visual Analytics in the Automation Renaissance 23

    doi:10.1111/cgf.14555 2022

  69. [81]

    Pernilla Qvarfordt and Shumin Zhai. 2005. Conversing with the user based on eye-gaze patterns. InProceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, Portland Oregon USA, 221–230. doi:10.1145/1054972.1055004

    arXiv 2005

  70. [82]

    Ragan, Alex Endert, Jibonananda Sanyal, and Jian Chen

    Eric D. Ragan, Alex Endert, Jibonananda Sanyal, and Jian Chen. 2016. Characterizing Provenance in Visualization and Data Analysis: An Organizational Framework of Provenance Types and Purposes.IEEE Trans. Visual. Comput. Graphics22, 1 (Jan. 2016), 31–40. doi:10.1109/TVCG. 2015.2467551

    arXiv 2016

  71. [83]

    Victor Riley. 1989. A General Model of Mixed-Initiative Human-Machine Systems.Proceedings of the Human Factors Society Annual Meeting33, 2 (Oct. 1989), 124–128. doi:10.1177/154193128903300227

    work page doi:10.1177/154193128903300227 1989

  72. [84]

    Brown, and Alex Endert

    Bahador Saket, Hannah Kim, Eli T. Brown, and Alex Endert. 2017. Visualization by Demonstration: An Interaction Paradigm for Visual Data Exploration.IEEE Trans. Visual. Comput. Graphics23, 1 (Jan. 2017), 331–340. doi:10.1109/TVCG.2016.2598839

    arXiv 2017

  73. [85]

    Arvind Satyanarayan and Graham M. Jones. 2024. Intelligence as Agency: Evaluating the Capacity of Generative AI to Empower or Constrain Human Action.An MIT Exploration of Generative AI(mar 27 2024). https://mit-genai.pubpub.org/pub/94y6e0f8

    2024

  74. [86]

    Ellingson, David Nuckley, John Carlis, and Daniel F Keefe

    David Schroeder, Fedor Korsakov, Carissa Mai-Ping Knipe, Lauren Thorson, Arin M. Ellingson, David Nuckley, John Carlis, and Daniel F Keefe

  75. [87]

    Thomas B Sheridan, William L Verplank, and TL Brooks. 1978. Human/computer control of undersea teleoperators. InNASA. Ames Res. Center The 14th Ann. Conf. on Manual Control

    1978

  76. [88]

    Ben Shneiderman and Pattie Maes. 1997. Direct manipulation vs. interface agents.interactions4, 6 (Nov. 1997), 42–61. doi:10.1145/267505.267514

    arXiv 1997

  77. [89]

    Monika Simmler and Ruth Frischknecht. 2021. A taxonomy of human–machine collaboration: capturing automation and technical autonomy.AI & Soc36, 1 (March 2021), 239–250. doi:10.1007/s00146-020-01004-z

    work page doi:10.1007/s00146-020-01004-z 2021

  78. [90]

    Fabian Sperrle, Davide Ceneda, and Mennatallah El-Assady. 2022. Lotse: A Practical Framework for Guidance in Visual Analytics.IEEE Trans. Visual. Comput. Graphics(2022), 1–11. doi:10.1109/TVCG.2022.3209393

    arXiv 2022

  79. [91]

    Fabian Sperrle, Astrik Jeitler, Jürgen Bernard, Daniel Keim, and Mennatallah El-Assady. 2021. Co-adaptive visual data analysis and guidance processes.Computers & Graphics100 (Nov. 2021), 93–105. doi:10.1016/j.cag.2021.06.016

    doi:10.1016/j.cag.2021.06.016 2021

  80. [92]

    Sperrle, H

    F. Sperrle, H. Schäfer, D. Keim, and M. El-Assady. 2021. Learning Contextualized User Preferences for Co-Adaptive Guidance in Mixed-Initiative Topic Model Refinement.Computer Graphics Forum40, 3 (June 2021), 215–226. doi:10.1111/cgf.14301

    doi:10.1111/cgf.14301 2021

Showing first 80 references.

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.