{"id":"1d1f0455-2863-4cd2-a8c0-6c8504f4fee3","arxiv_id":"2412.14481","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Five qualitative researchers' interviews and prototype feedback suggest that designing for personal agency (action, choice, narrative, space) can make computational thematic analysis tools more acceptable to domain experts.","lead":"Researchers who do qualitative analysis often distrust AI tools that might take control of their work. This paper uses interviews and a paper prototype to propose that a theory of personal agency can guide the design of more acceptable data visualization tools.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Deductive coding with Eichner's categories (§4.3) makes the reported fit between participant concerns and the four agency dimensions partly an artifact of the method, so the 'design lens' claim needs an independent re-analysis.","rationale":"The reader's weakest-assumption analysis focused on the small n=5 sample and on the deductive coding, and I agree that both matter. I would elevate the deductive-coding circularity as the single most load-bearing concern: Section 4.3 states the analysis was deductive and the agency codes are exactly Eichner's four dimensions, so the central observation that participant experiences align with those dimensions is partially guaranteed by the method. The paper does contain useful qualitative detail and a plausible conceptual framing; the DSR process and the prototype feedback are real evidence, and the authors are appropriately tentative in places. However, the headline claim that 'personal agency can be used as a design lens' could survive with weaker evidence, because the lens was never compared against an alternative framework or an open-coding baseline. The proposed re-coding test would not invalidate the paper's exploratory value, but it would show whether the four dimensions capture the data better than an unconstrained reading would. Since the reader already assigned CONDITIONAL and my concern points to the same kind of condition, the verdict should remain unchanged.","tokens_in":18466,"tokens_out":3353,"duration_ms":30367,"concrete_test":"Have two independent coders re-analyze the anonymized interview transcripts: Coder A applies Eichner's four-dimension codebook; Coder B performs open inductive coding without exposure to Eichner or the paper's research questions. Then compare: (1) Does Coder B's open coding converge on the same four dimensions, or does it produce alternative themes (e.g., trust, ownership, workload) that cut across or outside them? (2) Do two coders using Eichner reach acceptable inter-rater reliability (e.g., Cohen's kappa ≥ 0.6) on the four categories? (3) Can the design guidelines in §5 be derived from Coder B's themes alone? If open coding yields substantially different categories or poor reliability with the Eichner codebook, the claim that personal agency is a useful design lens loses its empirical support and becomes a purely conceptual proposal.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that Eichner's four dimensions of personal agency can serve as a design lens for computational thematic analysis interfaces. The decisive evidence is the mapping between participant concerns and these dimensions reported in §7. But the analysis in §4.3 was explicitly deductive: 'We conducted the thematic analysis deductively with the theories of fostering of agency,' and the agency codes were defined as 'the four kinds of mastery, choice, narrative, action, and space.' Any interview statement touching on control, options, movement, or storytelling would be assigned to one of these buckets, making the reported fit a product of the method rather than an empirical discovery. The prototype was then built on guidelines 'grounded in the concepts of personal agency' (§5), and the feedback session asked participants to react to that prototype; a positive reaction shows the features are liked, but not that the four-dimension lens is what produced them. The design features (categories, editing tools, transparency) are generic and could just as plausibly have been derived from self-determination theory or prior HCI work on qualitative tools. With n=5 healthcare researchers, the study also cannot rule out that a different lens would have been equally or more productive. The paper's central claim is therefore underdetermined: it demonstrates that Eichner's vocabulary can be imposed on qualitative-analysis interface design, not that it is a practically useful lens for generating better designs.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a design science research study in which five healthcare-focused qualitative researchers were interviewed about their analysis processes and visualization practices, a low-fidelity prototype for computational thematic analysis was developed, and the same participants provided feedback on the prototype. The interviews were analyzed deductively using Eichner's four dimensions of personal agency—action, choice, narrative, and space—and the authors claim that personal agency can serve as a design lens for software interfaces that support qualitative analysis with AI. The paper also reports that participants valued transparency, guidance, and editing capabilities, and that their openness to AI delegation increased over the course of the co-design process.","tokens_in":18677,"tokens_out":2841,"duration_ms":26738,"significance":"If the central claim were well supported, the paper would make a useful contribution by translating an agency framework from media theory into concrete interface design considerations for human-AI collaboration in qualitative analysis. The work is especially timely given recent HCI findings that loss of agency is a barrier to adoption of automated tools. The paper has notable strengths: participant sketches are reported in detail, the DSR process is made explicit, the four-phase structure is easy to follow, and the participants' own words are used throughout to ground the reported themes. The connection to prior work on trust and delegation is also appropriate. However, because the interview analysis was deductively coded with the very framework being claimed as a design lens, and because the evaluation uses the same participants who helped shape the prototype, the paper currently demonstrates that Eichner's vocabulary can be mapped onto qualitative-analysis interface concerns rather than showing that this lens is uniquely or practically valuable for design.","major_comments":[{"comment":"The thematic analysis was conducted deductively with Eichner's agency framework, and the coding scheme in §4.4 defines agency sub-codes as 'mastery, choice, narrative, action, and space.' This makes the reported fit between participant concerns and these dimensions partly an artifact of the method: any statement about control, options, movement, or storytelling would be assigned to one of these buckets. The central claim in §1 and §7 that 'personal agency can be used as a design lens' therefore needs independent evidence that the framework captured participant concerns rather than imposing them. I recommend adding an inductive coding pass or a disconfirming-case analysis, or explicitly reframing the contribution as an exploratory mapping rather than a validated design lens.","section":"§4.3 and §4.4"},{"comment":"The prototype was built on principles 'grounded in the concepts of personal agency' (§5), and the feedback session in §6 asked the same participants who had helped generate those requirements to react to the resulting prototype. Positive reactions show that the features (categories, editing tools, transparency) are acceptable and liked, but they do not show that the four-dimension agency lens is what produced those features. The features are also consistent with self-determination theory and with prior HCI work on transparency and trust. To support the central claim, the paper should either compare the design process against a plausible alternative lens or substantially weaken the claim to compatibility rather than causal or practical superiority.","section":"§5 and §6"},{"comment":"The study is based on five snowball-sampled healthcare researchers, yet the Discussion and Implications sections generalize to 'qualitative researchers' as a broader population. The authors invoke Malterud et al.'s information-power argument, but they do not explain why this particular sample has sufficient information power for the breadth of the claims made. The scope should be explicitly limited to the studied population and experience levels, with the broader design-lens claim presented as a hypothesis for future work.","section":"§4.1"},{"comment":"The Discussion repeatedly refers to 'the sample analysis' (e.g., §7.1, §7.2, §7.4) as a source of evidence about how existing software fosters or hinders agency, but the Methods section only mentions a close reading of similar software in Phase 1 (§3) and does not describe the selection, procedure, or analytic treatment of that software sample. Without a description of this analysis, these claims are not verifiable. Please either report the sample-analysis method and results or remove these references from the Discussion.","section":"§7"}],"minor_comments":[{"comment":"The sentence about controversy in HCI contains an unresolved citation placeholder 'noted by ? ]', and reference [24] appears as '[24? ]' in the text; these must be completed.","section":"§2"},{"comment":"Reference [42] contains the placeholder DOI 'https://doi.org/10.1145/nnnnnnn.nnnnnnn' and should be replaced with the correct bibliographic information.","section":"References"},{"comment":"The description of the DSR phases lists '3) specifying needs and requirements, 3) development of prototype, and 5)' without a clear fourth item; the numbering should be corrected.","section":"§3"},{"comment":"There is a typo in 'agency was fostered thorough the researchers’ interactions' — it should be 'through.'","section":"§7.1"},{"comment":"The manuscript text around Figure 2 contains duplicated paragraphs and internal figure labels such as 'Figure 27' and 'Figure 29' that do not match the published numbering; these should be cleaned up.","section":"Figures"}],"recommendation":"major_revision","confidential_remarks":"The paper has a useful exploratory contribution, but the central claim is currently stronger than the evidence supports. The deductive coding and same-participant evaluation are the main load-bearing issues; I would expect the authors to either add analytical safeguards that address circularity or carefully reframe the contribution. The topic is well within CHI's scope, and the design-process material is likely to be of interest to the community if the claims are calibrated to the actual evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a decent, modest design-science case study, and the core claim survives if you read it as 'Eichner's four dimensions give you a workable vocabulary for designing qualitative-analysis tools.' It does not survive as evidence that this lens is better than alternatives.\n\nWhat's actually new: applying Eichner's personal-agency framework (action, choice, narrative, space) to interface design for computational thematic analysis. Prior HCI work on agency in qualitative tools exists (Jiang et al., Feuston & Brubaker), but the explicit four-dimension mapping and the derived design guidelines are a real addition. The paper also does some things well: the DSR process is clearly laid out, the participant sketches are a nice touch, and the prototype guidelines (categories, guidance, editing, transparency) follow sensibly from the interviews. The discussion of trust, delegation, and transparency aligns with prior work and reads as honest. Citation pattern is fine—the self-cites are to the group's own toolkit papers, which are directly relevant.\n\nSoft spots, in proportion. The biggest is the circularity burden: §4.3 says the thematic analysis was done deductively using Eichner's agency types, so the finding that participant concerns map onto the four dimensions is partly a product of the coding scheme. The authors ground each dimension in quotes, which helps, but the mapping evidence is weaker than the text sometimes implies. Second, n=5 health researchers via snowball sampling is thin for the generalizing language ('qualitative researchers' broadly). The information-power argument is fine for a design study, but the claims should be scoped accordingly. Third, the prototype feedback is uniformly positive from participants who helped shape the requirements; that's expected and doesn't validate the lens as such. Finally, the manuscript has editing artifacts: duplicated blocks in Figure 2, unresolved citation placeholders ('? ]', '24? ]'), and a 2018 copyright/format date. These are fixable but need attention.\n\nNone of this sinks the paper. The contribution is a useful design lens and a set of guidelines, not a controlled experiment. If the authors temper the generalization and clean up the artifacts, this is a reasonable CHI-scale paper.\n\nRecommendation: send it to review, but with a clear request to address the deductive-coding limitation and the editing artifacts. I'd cite it in work on agency and human-AI collaboration.","headline":"A modest but useful design-lens case study; the deductive coding and n=5 limit the evidence, but the framework and guidelines are worth engaging.","tokens_in":19227,"tokens_out":4142,"would_cite":true,"duration_ms":33495,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that personal agency, as conceptualized by Eichner's four dimensions of action, choice, narrative, and space, can be used as a design lens for software interfaces supporting computational thematic analysis.","keywords":["personal agency","qualitative data analysis","computational thematic analysis","human-AI collaboration","data visualization","design science research","researcher autonomy"],"falsifier":"A study with qualitative researchers from other disciplines and varied programming backgrounds could test whether their stated concerns still fall naturally into action, choice, narrative, and space, or whether new dimensions emerge. A sharper test would compare two versions of the same visualization tool, one built with the agency lens and one without, and measure whether users report more agency, trust, and willingness to delegate in the agency-lens version.","tokens_in":18248,"feed_emoji":"🧭","tokens_out":4572,"duration_ms":36909,"temperature":0.7,"pith_summary":"The paper sets out to show that personal agency, defined through Eichner's four dimensions of action, choice, narrative, and space, can serve as a practical design lens for software that helps qualitative researchers analyze large text corpora with machine learning. Through a four-phase design science study with five health-care qualitative researchers, the authors develop a low-fidelity data-visualization prototype and test it in co-design feedback sessions. They report that participants value choices, transparency, editing control, and guidance, and that these features map onto the four dimensions of personal agency. The claim, on the authors' terms, is that the four dimensions translate abstract worries about autonomy into concrete interface features for human-AI collaboration.","feed_headline":"Four design moves keep qualitative researchers in control of AI analysis","feed_subtitle":"A media-theory framework turns autonomy worries into concrete interface features for AI-assisted thematic analysis.","key_machinery":"The central object is Eichner's personal agency framework, which decomposes agency into mastery of action, choice, narrative, and space, and which the paper applies deductively as both an analytic coding scheme and a generative design tool. The framework comes from media reception theory, where agency is felt when a user's actions visibly change an environment, when structured rules make choices meaningful, when recognizable patterns let a user predict and follow a story, and when navigation is seamless enough to move without friction. In this paper the framework organizes the interview coding into the categories of agency, cognitive assistance, and visual literacy, and it generates the prototype's guiding principles, such as grouping visualizations by researcher task, offering guidance without removing final choice, and providing editing tools and transparency.","core_discovery":"The central claim is that Eichner's four dimensions of personal agency, action, choice, narrative, and space, can be used as a design lens for interfaces supporting computational thematic analysis. The authors argue that when researchers can act and see results quickly, choose among guided options rather than face blank canvases or full automation, follow and shape a visual story in their data, and move through a familiar unobstructed interface, they experience ownership over AI-assisted analysis. The interview and prototype-feedback phases indicate that transparency about how AI reaches results, editing tools, and category-based guidance foster agency, while opaque black-box automation and hidden navigation diminish it. If the claim is right, the four dimensions give designers a concrete checklist for building human-AI tools that qualitative researchers will trust and adopt.","pith_inferences":["If the agency lens generalizes, it may also apply to other expert-AI collaborations where users fear replacement, such as journalism, medicine, or law, where the same four dimensions could guide interfaces for delegating screening, summarization, or evidence review.","The paper's finding that guidance plus choice fosters agency suggests a testable design principle: an interface that reveals a curated set of options, then lets the user reject or modify them, may outperform both full automation and open-ended tools, and this could be measured with task-based studies.","The observed shift in participants' attitudes during co-design hints that the design process itself is an intervention; future work could test whether simply walking researchers through a prototype's transparency features, without co-design, produces the same increase in openness to AI."],"forward_implications":["The four dimensions can be used as a checklist for designers: each interface feature can be classified by which form of agency it fosters, and features that impede action, choice, narrative, or space can be flagged early.","For AI-assisted qualitative analysis, transparency about how results are reached is not an optional extra; participants treated it as a condition for delegating tasks and feeling in control.","Tools should offer guided categories and suggestions rather than either blank canvases or full automation, since participants reported that excessive freedom and complete automation both reduce agency.","Editing tools that let researchers adjust visualizations support agency through action and narrative, and fast feedback on changes strengthens the sense of influence.","Co-design itself appeared to shift skeptical participants toward accepting AI assistance, suggesting that involving researchers in prototype development can build trust alongside the artifact."],"supporting_citations":[{"why":"Supplies the four-dimension personal agency framework that the paper applies as its design lens.","marker":"[25]"},{"why":"Defines agency as action that visibly changes an environment, grounding the action and space dimensions.","marker":"[46]"},{"why":"Provides the underlying theory of human agency with intention, forethought, and self-reflection.","marker":"[5]"},{"why":"Introduces commitment to meaning and creative agency, which the paper draws on for narrative and co-creation.","marker":"[55]"},{"why":"Offers prior evidence that qualitative researchers resist tools that reduce ownership and control, motivating the agency focus.","marker":"[27]"},{"why":"Shows that loss of agency is a primary concern in human-AI qualitative analysis and that AI should collaborate rather than replace.","marker":"[36]"},{"why":"Provides the information-power rationale the paper uses to justify its five-participant sample.","marker":"[44]"},{"why":"Presents the Computational Thematic Analysis Toolkit, one of the sample tools analyzed in the paper's navigation and guidance discussion.","marker":"[31]"}],"fun_headline_variants":["A four-part agency lens for trustworthy AI-assisted analysis","Designing AI analysis that puts researchers in the driver's seat","From autonomy worries to concrete interface features for AI analysis","Eichner's four dimensions: a design checklist for human-AI tools"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the four dimensions of personal agency, applied deductively to interview data from five health-care researchers, actually capture how qualitative researchers in general experience agency, even though the analysis method assumes the framework fits before testing it.","fun_headline_variants_meta":{"raw":{"variants":["A four-part agency lens for trustworthy AI-assisted analysis","Designing AI analysis that puts researchers in the driver's seat","From autonomy worries to concrete interface features for AI analysis","Eichner's four dimensions: a design checklist for human-AI tools"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000654,"raw_usage":{"total_tokens":2950,"prompt_tokens":852,"completion_tokens":2098,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":468,"completion_tokens_details":{"reasoning_tokens":2028}},"tokens_in":468,"tokens_out":2098,"duration_ms":12810,"temperature":1.0,"reasoning_tokens":2028,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:10:34.165761+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A study with qualitative researchers from other disciplines and varied programming backgrounds could test whether their stated concerns still fall naturally into action, choice, narrative, and space, or whether new dimensions emerge. A sharper test would compare two versions of the same visualization tool, one built with the agency lens and one without, and measure whether users report more agency, trust, and willingness to delegate in the agency-lens version.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the four-dimension personal agency framework that the paper applies as its design lens."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines agency as action that visibly changes an environment, grounding the action and space dimensions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the underlying theory of human agency with intention, forethought, and self-reflection."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces commitment to meaning and creative agency, which the paper draws on for narrative and co-creation."},{"cited_title":"Brubaker","cited_arxiv_id":null,"evidence_quote":"Shows that loss of agency is a primary concern in human-AI qualitative analysis and that AI should collaborate rather than replace."},{"cited_title":"Gauthier and James R","cited_arxiv_id":null,"evidence_quote":"Presents the Computational Thematic Analysis Toolkit, one of the sample tools analyzed in the paper's navigation and guidance discussion."}],"review_version":1}