pith. sign in

arxiv: 2605.30800 · v1 · pith:FKLYTCSTnew · submitted 2026-05-29 · 💻 cs.HC

Computer-Aided Tagging on Wikimedia Commons: Designing for Human-AI Collaboration in Open Knowledge Work

Yihan Yu , David W. McDonald This is my paper

Pith reviewed 2026-06-28 21:22 UTC · model grok-4.3

classification 💻 cs.HC
keywords Human-AI collaborationWikimedia CommonsAI-assisted taggingOpen knowledge workQualitative analysisCommunity feedbackTool deactivationVolunteer platforms
0
0 comments X

The pith

Analysis of community comments and interviews identifies seven issues that led to deactivation of the Computer-Aided Tagging tool on Wikimedia Commons.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper examines Wikimedia Commons contributors' experiences with an AI-assisted image tagging system intended to boost discoverability, searchability, accessibility, and multilingual support. Through thematic analysis of 595 related comments across 11 wiki pages plus 16 interviews, it isolates seven issues behind the tool's mixed reception and eventual shutdown. The authors also compile community suggestions for improvement and consider what these experiences imply for human-AI collaboration in volunteer-driven open knowledge projects. A reader would care because the case shows how AI features must mesh with contributor norms if they are to last on non-corporate platforms.

Core claim

The authors establish that seven specific issues identified in the qualitative data on the Computer-Aided Tagging tool produced its mixed reception and led to deactivation; they further supply community-informed recommendations for redesign and reflect on the broader requirements for effective human-AI collaboration in open knowledge work beyond corporate, text-centric settings.

What carries the argument

Thematic analysis of 595 CAT-related community comments from 11 wiki pages and 16 in-depth interviews that isolates seven key issues.

If this is right

  • Future AI tools on Commons must address the seven issues to gain sustained use.
  • Design of human-AI systems in open knowledge settings requires explicit attention to volunteer control and workflow fit.
  • Suggestions derived from community input can guide revisions that improve tool acceptance.
  • The case extends HCI understanding of human-AI collaboration to multilingual, volunteer platforms.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Similar issues may appear when AI tools are introduced on other volunteer platforms such as Wikipedia editing interfaces.
  • Transparent explanations of AI decisions could mitigate some of the identified issues in future deployments.
  • Longitudinal tracking of contributor retention after tool changes would test whether addressing the seven issues improves participation.

Load-bearing premise

The sampled comments and interviews represent the wider contributor population and the thematic analysis correctly isolates the main reasons for deactivation.

What would settle it

A larger-scale survey or archival record showing that deactivation occurred primarily for reasons other than the seven identified issues.

Figures

Figures reproduced from arXiv: 2605.30800 by David W. McDonald, Yihan Yu.

Figure 1
Figure 1. Figure 1: A side-by-side comparison of a file’s traditional, unstructured File information view (right) and its [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The CAT interface displaying unverified suggested tags for an example image file. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The Structured data tab showing Depicts statements (left) alongside traditional category tags stored [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Screenshots from an animation demonstrating multilingual structured data statements. The example [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
read the original abstract

This study investigates Wikimedia Commons contributors' lived experiences with the Computer-Aided Tagging (CAT) tool, an AI-assisted image tagging system designed to improve Commons' discoverability, searchability, accessibility, and multilingual support. Using a qualitative analysis of 595 CAT-related community comments from 11 wiki pages and 16 in-depth interviews, we identify seven key issues that contributed to CAT's mixed reception and eventual deactivation. We also offer community-informed suggestions for improving the tool. We reflect on the implications for designing human-AI collaboration on Commons and for developing AI-assisted tools that support open knowledge work. This work contributes to HCI and CSCW research by extending the understanding of human-AI collaboration beyond Anglophone, text-centric, corporate platforms.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript reports a qualitative study of Wikimedia Commons contributors' experiences with the Computer-Aided Tagging (CAT) AI tool. Using thematic analysis of 595 CAT-related comments drawn from 11 wiki pages plus 16 in-depth interviews, the authors identify seven issues that contributed to the tool's mixed reception and eventual deactivation. They also provide community-informed design suggestions and reflect on implications for human-AI collaboration in open, multilingual knowledge platforms, positioning the work as an extension of HCI/CSCW research beyond corporate, text-centric settings.

Significance. If the methodological gaps are resolved, the study would provide grounded, community-sourced evidence on the practical challenges of AI-assisted tagging in volunteer open-knowledge projects. Strengths include the direct use of public wiki comments and interviews to surface issues such as trust, accuracy, and workflow fit, which are rarely examined outside corporate platforms. This could inform design of future tools in similar decentralized settings and contribute to broader discussions of human-AI collaboration in non-Anglophone, non-commercial contexts.

major comments (2)
  1. [Methods] Methods section (data collection): The selection of the 11 wiki pages and the process for identifying and filtering the 595 comments are described only at a high level, with no explicit inclusion/exclusion criteria, time bounds, or justification for the convenience sample. This directly affects the central claim that the seven issues 'contributed to' deactivation, as self-selected talk-page comments may over-represent engaged users and miss developer or admin rationales.
  2. [Methods] Methods section (analysis): The thematic analysis process for deriving the seven issues from comments and interviews lacks detail on coding procedures, inter-rater reliability, saturation criteria for the 16 interviews, or negative-case analysis. Without these, the mapping from raw data to the reported issues remains difficult to evaluate for researcher dependence, which is load-bearing for the paper's primary findings.
minor comments (2)
  1. [Abstract] The abstract lists seven issues but does not name them; a brief enumeration or table in the results section would improve readability.
  2. [Discussion] Consider adding a limitations subsection that explicitly discusses the self-selected nature of wiki comments and the small interview sample.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for these constructive comments on our methods. We agree that greater transparency is needed and will revise the Methods section accordingly. Below we respond point by point.

read point-by-point responses

  1. Referee: [Methods] Methods section (data collection): The selection of the 11 wiki pages and the process for identifying and filtering the 595 comments are described only at a high level, with no explicit inclusion/exclusion criteria, time bounds, or justification for the convenience sample. This directly affects the central claim that the seven issues 'contributed to' deactivation, as self-selected talk-page comments may over-represent engaged users and miss developer or admin rationales.

    Authors: We accept that the current description is insufficiently detailed. In the revision we will add: (1) explicit inclusion/exclusion criteria (e.g., comments must explicitly reference the CAT tool or its output; non-English comments were translated only when machine translation was reliable); (2) the time window (comments collected from tool deployment through deactivation announcement); and (3) justification for the 11 pages (they comprise the primary public discussion venues identified via Commons search and category links). We will also qualify the causal language around deactivation, noting that the seven issues reflect prominent community concerns surfaced in these sources rather than a comprehensive account of all decision-making rationales. Developer and admin perspectives are acknowledged as outside the scope of the collected data. revision: yes

  2. Referee: [Methods] Methods section (analysis): The thematic analysis process for deriving the seven issues from comments and interviews lacks detail on coding procedures, inter-rater reliability, saturation criteria for the 16 interviews, or negative-case analysis. Without these, the mapping from raw data to the reported issues remains difficult to evaluate for researcher dependence, which is load-bearing for the paper's primary findings.

    Authors: We agree that additional procedural detail is required. The revised Methods section will describe: the two-phase coding process (initial open coding of a 20% subset followed by iterative refinement across the full corpus); that the first author performed primary coding with regular discussion of emerging themes with the second author; the absence of formal inter-rater reliability metrics (as is common in interpretive thematic analysis) but the use of consensus-building meetings; saturation criteria (no new themes after the 12th interview); and explicit negative-case analysis (instances where contributors expressed positive views of CAT accuracy or workflow fit were retained and used to refine theme boundaries). These additions will make the analytic path more traceable. revision: yes

Circularity Check

0 steps flagged

No circularity; findings derived from independent qualitative data sources

full rationale

The paper's central claim rests on thematic analysis of externally generated community comments (595 from 11 wiki pages) and interviews (16), which are independent inputs not defined in terms of the seven issues or fitted to produce them by construction. No equations, parameter fitting, self-citation chains, or ansatzes are present that would reduce the identified issues to the analysis inputs. The derivation is self-contained empirical interpretation of separate data.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The study rests on the standard domain assumption of qualitative HCI research that thematic analysis of comments and interviews can surface load-bearing user issues; no free parameters or invented entities are introduced.

axioms (1)
  • domain assumption Thematic analysis of community comments and interviews reliably identifies the primary issues affecting tool reception.
    Core premise of the qualitative method used.

pith-pipeline@v0.9.1-grok · 5656 in / 1076 out tokens · 30722 ms · 2026-06-28T21:22:11.128644+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

36 extracted references · 4 canonical work pages

  1. [1]

    B Thomas Adler, Luca De Alfaro, Santiago M Mola-Velasco, Paolo Rosso, and Andrew G West. 2011. Wikipedia vandalism detection: Combining natural language, metadata, and reputation features. InComputational Linguistics and Intelligent Text Processing: 12th International Conference, CICLing 2011, Tokyo, Japan, February 20-26, 2011. Proceedings, Part II 12. S...

    2011

  2. [2]

    Judit Bar-Ilan, Snunith Shoham, Asher Idan, Yitzchak Miller, and Aviv Shachak. 2008. Structured versus unstructured tagging: a case study.Online Information Review32, 5 (2008), 635–647

    2008

  3. [3]

    Abeba Birhane, Vinay Prabhu, Sang Han, and Vishnu Naresh Boddeti. 2023. On hate scaling laws for data-swamps. arXiv preprint arXiv:2306.13141(2023)

    work page arXiv 2023

  4. [4]

    Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology.Qualitative research in psychology3, 2 (2006), 77–101

    2006

  5. [5]

    Dan Cosley, Dan Frankowski, Loren Terveen, and John Riedl. 2007. SuggestBot: using intelligent task routing to help people find work in wikipedia. InProceedings of the 12th international conference on Intelligent user interfaces. 32–41

    2007

  6. [6]

    Andrea Forte, Vanesa Larco, and Amy Bruckman. 2009. Decentralization in Wikipedia governance.Journal of Management Information Systems26, 1 (2009), 49–72

    2009

  7. [7]

    2016.Managing digital cultural objects: analysis, discovery and retrieval

    Allen Foster and Pauline Rafferty. 2016.Managing digital cultural objects: analysis, discovery and retrieval. Facet Publishing

    2016

  8. [8]

    Tarleton Gillespie. 2010. The politics of ‘platforms’.New media & society12, 3 (2010), 347–364

    2010

  9. [9]

    Justin Grandinetti. 2023. Examining embedded apparatuses of AI in Facebook and TikTok.Ai & Society(2023), 1–14

    2023

  10. [10]

    Aaron Halfaker and R Stuart Geiger. 2020. Ores: Lowering barriers with participatory machine learning in wikipedia. Proceedings of the ACM on Human-Computer Interaction4, CSCW2 (2020), 1–37

    2020

  11. [11]

    Jane Im, Amy X Zhang, Christopher J Schilling, and David Karger. 2018. Deliberation and resolution on wikipedia: A case study of requests for comments.Proceedings of the ACM on Human-Computer Interaction2, CSCW (2018), 1–24

    2018

  12. [12]

    Suchae Jeong, Inseong Choi, Youngsik Yun, and Jihie Kim. 2025. Culture-TRIP: Culturally-Aware Text-to-Image Generation with Iterative Prompt Refinement.arXiv preprint arXiv:2502.16902(2025)

    work page arXiv 2025

  13. [13]

    Hyunjin Kang and Chen Lou. 2022. AI agency vs. human agency: understanding human–AI interactions on TikTok and their implications for user engagement.Journal of Computer-Mediated Communication27, 5 (2022), zmac014

    2022

  14. [14]

    Lyndon S Kennedy, Shih-Fu Chang, and Igor V Kozintsev. 2006. To search or to label? Predicting the performance of search-based automatic image classifiers. InProceedings of the 8th ACM international workshop on Multimedia information retrieval. 249–258

    2006

  15. [15]

    Aniket Kittur, Bongwon Suh, Bryan A Pendleton, and Ed H Chi. 2007. He says, she says: conflict and coordination in Wikipedia. InProceedings of the SIGCHI conference on Human factors in computing systems. 453–462

    2007

  16. [16]

    Travis Kriplean, Ivan Beschastnikh, David W McDonald, and Scott A Golder. 2007. Community, consensus, coercion, control: cs* w or how policy mediates mass participation. InProceedings of the 2007 ACM International Conference on Supporting Group Work. 167–176

    2007

  17. [17]

    Travis Kriplean, Michael Toomim, Jonathan Morgan, Alan Borning, and Amy J Ko. 2012. Is this what you meant? Promoting listening on the web with reflect. Inproceedings of the SIGCHI conference on human factors in computing systems. 1559–1568

    2012

  18. [18]

    Tina Kuo, Alicia Hernani, and Jens Grossklags. 2023. The unsung heroes of facebook groups moderation: A case study of moderation practices and tools.Proceedings of the ACM on Human-Computer Interaction7, CSCW1 (2023), 1–38

    2023

  19. [19]

    Dong Liu, Xian-Sheng Hua, and Hong-Jiang Zhang. 2011. Content-based tag processing for internet social images. Multimedia Tools and Applications51 (2011), 723–738

    2011

  20. [20]

    Shudong Liu, Yiqiao Jin, Cheng Li, Derek F Wong, Qingsong Wen, Lichao Sun, Haipeng Chen, Xing Xie, and Jindong Wang. 2025. Culturevlm: Characterizing and improving cultural understanding of vision-language models for over 100 countries.arXiv preprint arXiv:2501.01282(2025)

    work page arXiv 2025

  21. [21]

    Sasha Luccioni, Christopher Akiki, Margaret Mitchell, and Yacine Jernite. 2023. Stable bias: Evaluating societal representations in diffusion models.Advances in Neural Information Processing Systems36 (2023), 56338–56351

    2023

  22. [22]

    Amanda Menking and David W McDonald. 2020. Image Wishlist: Context and Images in Commons-Based Peer Production Communities.Proceedings of the ACM on Human-Computer Interaction4, CSCW2 (2020), 1–21

    2020

  23. [23]

    Marc Miquel-Ribé and David Laniado. 2018. Wikipedia culture gap: quantifying content imbalances across 40 language editions.Frontiers in physics6 (2018), 54

    2018

  24. [24]

    Shilad Sen, Shyong K Lam, Al Mamunur Rashid, Dan Cosley, Dan Frankowski, Jeremy Osterhouse, F Maxwell Harper, and John Riedl. 2006. Tagging, communities, vocabulary, evolution. InProceedings of the 2006 20th anniversary conference on Computer supported cooperative work. 181–190

    2006

  25. [25]

    Sara Shatford. 1986. Analyzing the subject of a picture: a theoretical approach.Cataloging & classification quarterly6, 3 (1986), 39–62. , Vol. 1, No. 1, Article . Publication date: June 2018. Computer-Aided Tagging on Wikimedia Commons: Designing for Human–AI Collaboration in Open Knowledge Work27

    1986

  26. [26]

    Börkur Sigurbjörnsson and Roelof Van Zwol. 2008. Flickr tag recommendation based on collective knowledge. In Proceedings of the 17th international conference on World Wide Web. 327–336

    2008

  27. [27]

    C Estelle Smith, Bowen Yu, Anjali Srivastava, Aaron Halfaker, Loren Terveen, and Haiyi Zhu. 2020. Keeping community in the loop: Understanding wikipedia stakeholder values for machine learning-based systems. InProceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–14

    2020

  28. [28]

    Katherine Thornton and David W McDonald. 2012. Tagging Wikipedia: collaboratively creating a category system. In Proceedings of the 2012 ACM International Conference on Supporting Group Work. 219–228

    2012

  29. [29]

    Fernanda B Viégas, Martin Wattenberg, Jesse Kriss, and Frank Van Ham. 2007. Talk before you type: Coordination in Wikipedia. In2007 40th Annual Hawaii International Conference on System Sciences (HICSS’07). IEEE, 78–78

    2007

  30. [30]

    Meng Wang, Bingbing Ni, Xian-Sheng Hua, and Tat-Seng Chua. 2012. Assistive tagging: A survey of multimedia tagging with human-computer joint exploration.ACM computing surveys (CSUR)44, 4 (2012), 1–24

    2012

  31. [31]

    Sitong Wang, Samia Menon, Tao Long, Keren Henderson, Dingzeyu Li, Kevin Crowston, Mark Hansen, Jeffrey V Nickerson, and Lydia B Chilton. 2024. ReelFramer: Human-AI co-creation for news-to-video translation. InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems. 1–20

    2024

  32. [32]

    I agree with you, bot!

    Magdalena Wischnewski, Thao Ngo, Rebecca Bernemann, Martin Jansen, and Nicole Krämer. 2024. “I agree with you, bot!” How users (dis) engage with social bots on Twitter.New Media & Society26, 3 (2024), 1505–1526

    2024

  33. [33]

    Yihan Yu and David W McDonald. 2022. Unpacking Stitching between Wikipedia and Wikimedia Commons: Barriers to Cross-Platform Collaboration.Proceedings of the ACM on Human-Computer Interaction6, CSCW2 (2022), 1–35

    2022

  34. [34]

    Why do you need 400 photographs of 400 different Lockheed Constellation?

    Yihan Yu and David W McDonald. 2023. " Why do you need 400 photographs of 400 different Lockheed Constellation?": Value Expressions by Contributors and Users of Wikimedia Commons.Proceedings of the ACM on Human-Computer Interaction7, CSCW2 (2023), 1–34

    2023

  35. [35]

    Youngsik Yun and Jihie Kim. 2024. Cic: A framework for culturally-aware image captioning.arXiv preprint arXiv:2402.05374(2024)

    work page arXiv 2024

  36. [36]

    Amy X Zhang, Lea Verou, and David Karger. 2017. Wikum: Bridging discussion forums and wikis using recursive summarization. InProceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. 2082–2096. A Interview Protocol A.1 Opening Script Hi! I’m <First Author>. Today, I’d like to talk with you about your experiences w...

    2017