Pith. sign in

REVIEW 2 major objections 5 minor 23 references

RelAItionship Building: Analyzing Recruitment Strategies for Participatory AI

T0 review · 2 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that recruitment—how the people who shape an AI system get found and brought in—determines who benefits from participatory AI, yet it is treated as peripheral and left largely undocumented.

desk verdict A modest, honest empirical start on recruitment in participatory AI; the sample is thin but the claims are proportioned to it. read the letter →

arxiv 2508.20176 v1 pith:5JVW3S5Z submitted 2025-08-27 cs.CY cs.AI

Eugene Kim , Vaibhav Balloli , Berelian Karimian , Elizabeth Bondi-Kelly , Benjamin Fish This is my paper · ORCID
classification cs.CYcs.AI
keywords participatoryAIrecruitmentmethodologystakeholderengagementforsocialimpactqualitativeanalysisreflexivedocumentationcommunitypartnershipsdesign
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that recruitment—how the people affected by an AI system are identified, contacted, and drawn into its development—is a decisive but neglected part of Participatory AI. Analyzing a 37-project corpus and interviewing five AI researchers, the authors find that recruitment methods are frequently undocumented, and that who ends up participating is shaped less by formal plans than by funding and time constraints, researchers' career incentives, and pre-existing relationships with community organizations. The authors recommend treating recruitment as a first-class methodological problem: build relationships with communities before the project starts, and document the 'backstage' work—personal networks, institutional support, refusals and dead ends—that actually determines who participates. If they are right, improving recruitment would change not just who is consulted in AI development, but which systems get built and whom they end up serving.

What carries the argument

The load-bearing mechanism is the two-part empirical design: inductive coding of 37 participatory AI projects across categories (who is recruited, who recruits, strategy used, empowerment level, documentation specificity), plus five semi-structured interviews with AI researchers that surface what the published papers omit. The corpus yields the taxonomy of current practice—organizational, infrastructural, personal-network, and event recruitment—while the interviews supply the causal story that structural constraints, researcher goals, and relationships drive outcomes. The paper also introduces ParticipAIte, a structured documentation schema (stakeholder roles, initiator, strategy, call for p

What would settle it

A concrete test would be a larger, independently sampled study of participatory AI projects—including community-led projects and Global South contexts—coded with the same scheme, plus a broader survey of practitioners beyond the five interviewed. If that study found recruitment routinely well documented in some subfields, or found many projects succeeding through cold outreach to strangers with no prior relationship, the paper's central claims would not generalize. Conversely, if a systematic analysis found no association between pre-existing researcher–community relationships and outcomes (pa

Watch

Extended reading notes

Core claim

The paper's central claim: recruitment methodology is the neglected hinge of Participatory AI—who gets approached, through which channels, and on the basis of which prior relationships determines who takes part, and so whose values the AI reflects. Two findings carry the argument, from a coded 37-project corpus and five researcher interviews. First, recruitment is poorly documented: across the corpus, strategies for at least one stakeholder group were often missing, so projects cannot be transparently compared or reproduced. Second, outcomes are shaped less by formal methods than by structural conditions (funding, time, logistics, trust deficits), researchers' goals and expectations, and the

Load-bearing premise

The load-bearing premise is that the 37 projects chosen for the corpus and the five interviewees—recruited through the authors' personal networks—faithfully represent the range of recruitment practice in participatory AI; if projects that fail at recruitment or recruit in fundamentally different ways are missing from this sample, the themes and recommendations overgeneralize. The paper acknowledges this dependence in Sections 3.3 and 7.

Editorial extensions

If this is right

  • Documented recruitment would make participatory AI projects comparable and reproducible: published accounts would have to name who initiated contact, through which organizations or networks, and who was excluded and why.
  • Funding bodies and universities would need to count relationship-building as legitimate research labor with dedicated time and money, since the study finds cold-start recruitment without prior relationships rarely succeeds.
  • Evaluation of participatory projects would shift from whether participants were involved to how they were recruited, because recruitment fixes who is in the room from the start.
  • Researchers new to participatory methods would get concrete guidance: work through community organizations, embed in local networks before the project starts, and expect representativeness to require more than a sampling plan.
  • A shared documentation schema such as ParticipAIte could turn one-off reviews like this corpus into an accumulating, comparable database of recruitment practice.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable asymmetry follows: if relationship-based recruitment is as dominant as reported, projects whose researchers had no prior community ties should show lower participation and shorter-lived deployments; a larger, quantitative companion to this corpus could check that directly.
  • The paper's data also support a reading it does not fully state: since the 'researcher initiates' pattern is near-universal, and since interviewees describe communities approaching them after reputations form, recruitment may reflect whose hands hold funding and institutional access as much as it reflects method choice.
  • The ParticipAIte schema could be carried further than the paper takes it: AI venues could adopt recruitment documentation as a required reporting artifact, alongside the documentation sheets already expected for datasets, turning this one-off review into an accumulating practice.
  • The sharpest test is named, not run, in the paper's own limitations: the five interviewees came from the authors' networks and therefore mostly represent researchers who already succeeded at relationship building; a sample that includes researchers who abandoned or failed at recruitment would test whether 'relationships are everything' generalizes or is a survivor's story.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper investigates recruitment methodology in Participatory AI through two complementary data sources: a corpus of 37 AI projects (coded for stakeholder types, recruitment initiators, strategies, documentation quality, and empowerment levels) and semi-structured interviews with five AI researchers. The authors report that recruitment methods are frequently not documented in enough detail for reproducibility, and that recruitment outcomes are shaped by structural constraints, researchers' goals, and relationships between researchers and communities. Based on these findings, they propose relationship-forward recruitment strategies and reflexive documentation practices, and introduce a preliminary database called ParticipAIte.

Significance. If the findings hold, the paper makes a valuable contribution by foregrounding an under-studied step in the Participatory AI pipeline and offering concrete, actionable recommendations. Strengths include a transparent qualitative coding process grounded in existing frameworks (Corbett et al. 2023; Delgado et al. 2023), a publicly available database artifact, explicit acknowledgment of the authors' positionality, and practical suggestions such as documenting refusals and institutional support. The paper is honest about its limitations, which enhances its credibility. However, the breadth of the central claims currently outstrips the evidence base, particularly with respect to sample representativeness.

major comments (2)
  1. [Section 3.1 / Section 4.1] The corpus is assembled from existing reviews, selected venues, and Google Scholar searches without a reported search protocol, screening criteria, or counts; saturation is declared when no new recruitment categories appeared. Because Section 4.1's claim that recruitment methods are 'often not documented sufficiently' and the frequency patterns in Figs. 1-3 are general statements about practice, the lack of a systematic sampling frame leaves the central claim dependent on the sample's representativeness. I recommend either adding a systematic literature-search procedure (databases, screening, inclusion/exclusion, counts) or explicitly reframing the findings as describing the sampled projects rather than the population.
  2. [Section 3.3 / Section 5] The interview component is based on five researchers recruited via the authors' personal networks, with no community-group perspectives. The paper acknowledges this in Section 7, but Section 5 nevertheless draws general conclusions, e.g., 'outcomes of participatory projects are fundamentally based on the relationships built by the researchers and participants' (Section 5.3). With n=5 and a network-recruited sample, such universal claims are not supported. Please rephrase these as emergent themes/hypotheses from a small, self-selected sample, and strengthen the positionality/limitations discussion accordingly.
minor comments (5)
  1. [Section 3.1] The coding procedure (two coders, consensus via weekly meetings) is described, but no inter-rater reliability metric is reported. Since the documentation-gap finding rests on these codes, adding a kappa/agreement measure or a justification for its omission would improve transparency and reproducibility.
  2. [Table 8 / Section 4] Several papers appear in multiple empowerment categories (e.g., Bakker et al. 2022 under both 'Consult' and 'Include'). Please clarify whether a project can receive multiple codes (e.g., for different stakeholder groups) or whether this reflects an inconsistency.
  3. [Table 4 vs. Table 7] Table 4 lists only 5 papers as 'Not Documented for at least one participant group,' while Table 7 lists 20 papers as having 'Not all stakeholders have recruitment strategies documented.' The relationship between these two categorizations is unclear; please explain the difference.
  4. [Section 3.1] Minor typo: 'Appendix1 Tables 10, 11, and 12' should have a space after 'Appendix.'
  5. [Section 3.2] ParticipAIte is described as 'preliminary' and a 'first step,' but it is later used to structure interviews. A brief statement on its status as a proof-of-concept (not a validated tool) would set expectations appropriately.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's claims are empirical generalizations from an externally grounded corpus and open-ended interviews, not predictions derived from its own definitions.

full rationale

The paper's central claims are empirical and grounded in external sources, not defined into existence. The corpus is assembled from prior reviews, venue-specific searches, and Google Scholar (Section 3.1), and the coding uses external frameworks such as Delgado et al. (2023)'s empowerment scale and Corbett et al. (2023)'s participation entry points. The documentation-gap finding (Section 4.1) is a measurement against the authors' own ParticipAIte form, but it is a descriptive characterization of the sampled papers, not a prediction forced by the form. The interview themes in Section 5—structural constraints, researcher goals and expectations, and relationships—emerged from open-ended narratives rather than from the ParticipAIte database fields, which only encode stakeholder types, initiator, strategy, lessons, and call type. The ParticipAIte interview prompt does create a mild self-referential loop: the authors coded the corpus into ParticipAIte, then used those author-generated entries as prompts for the same projects, and later note that the interviews 'align' with the corpus categories. However, this is a methodological coherence issue, not a circular derivation; the main qualitative findings do not reduce to the database entries by construction, and the paper does not claim to 'predict' anything from the database. Sections 3.3 and 7 explicitly acknowledge sample, network, and positionality biases, which affect generalizability rather than circularity. The self-citations present (Bondi et al. 2021; Barocas et al. 2020) are background support for values and refusal, not load-bearing premises. No uniqueness theorems, ansatz-smuggling, or renaming of known results were found. The derivation chain is therefore self-contained with respect to the paper's conclusions.

Assumptions & free parameters 0 free parameters · 5 assumptions · 1 invented entities

No numerical free parameters are fitted, consistent with the qualitative design. The operative constructs are the hand-chosen analytic categories: stakeholder roles from Corbett et al. (2023), the empowerment scale from Delgado et al. (2023), and the four recruitment strategy codes (organizational, infrastructural, personal networks, events) derived from the authors' own coding. The paper's generality rests on the representativeness of its corpus and interview sample plus the reliability of two-coder consensus, all acknowledged as limited by the authors.

assumptions (5)
  • domain assumption The 37-project corpus assembled from named reviews, venues, and Scholar searches is sufficiently diverse to characterize recruitment practice in participatory AI.
    Section 3.1 describes assembling the corpus from existing reviews, FAccT/EAAMO/CHI/AIES venues, and Google Scholar searches. The generality of the corpus findings depends on this selection being representative.
  • ad hoc to paper Saturation, declared when no new recruitment categories appeared, is a valid stopping criterion for corpus assembly.
    Section 3.1 states: 'We determined saturation was reached and concluded the literature review when novel recruitment categories in AI projects were no longer identified.' No formal protocol or audit trail for this determination is given.
  • domain assumption Two-coder inductive coding with weekly consensus meetings yields sufficiently reliable categories.
    Section 3.1 reports two authors (EK, BK) coded the papers and focused on consensus via weekly meetings. No inter-rater reliability statistic is reported, so coding reliability is asserted rather than measured.
  • domain assumption Five interviewees selected from the authors' professional networks suffice to surface the themes that shape recruitment outcomes.
    Section 3.3 describes contacting ten known researchers and interviewing five, prioritizing varied experiences. The paper acknowledges the selection was biased by personal networks, which bounds generalizability.
  • domain assumption The Delgado et al. (2023) empowerment scale can be applied to recruitment as a valid proxy for participant empowerment outcomes.
    Section 3.1 and Appendix Table 8 apply the empowerment scale (Consult, Include, Collaborate, Own) to corpus projects. The scale was designed for participation modes generally, and its application to recruitment specifically is an interpretive choice.
invented entities (1)
  • ParticipAIte database independent evidence
    purpose: A public, form-based database for documenting recruitment strategies, stakeholder roles, who initiated recruitment, calls for participation, and lessons learned in participatory AI projects.
    ParticipAIte is a concrete artifact with a public URL (realize-lab.github.io/participaite) that others can inspect and use. It is not a postulated theoretical entity, but the paper presents it as a deliverable and partially justifies its design through the paper's own interview and corpus findings.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RelAItionship Building: Analyzing Recruitment Strategies for Participatory AI." pith.science (2026). https://pith.science/paper/5JVW3S5Z

@misc{pith2026250820176,
  author       = {Pith},
  title        = {Pith review of: RelAItionship Building: Analyzing Recruitment Strategies for Participatory AI},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5JVW3S5Z}},
  note         = {Machine review of arXiv:2508.20176}
}
read the original abstract

Participatory AI, in which impacted community members and other stakeholders are involved in the design and development of AI systems, holds promise as a way to ensure AI is developed to meet their needs and reflect their values. However, the process of identifying, reaching out, and engaging with all relevant stakeholder groups, which we refer to as recruitment methodology, is still a practical challenge in AI projects striving to adopt participatory practices. In this paper, we investigate the challenges that researchers face when designing and executing recruitment methodology for Participatory AI projects, and the implications of current recruitment practice for Participatory AI. First, we describe the recruitment methodologies used in AI projects using a corpus of 37 projects to capture the diversity of practices in the field and perform an initial analysis on the documentation of recruitment practices, as well as specific strategies that researchers use to meet goals of equity and empowerment. To complement this analysis, we interview five AI researchers to learn about the outcomes of recruitment methodologies. We find that these outcomes are shaped by structural conditions of their work, researchers' own goals and expectations, and the relationships built from the recruitment methodology and subsequent collaboration. Based on these analyses, we provide recommendations for designing and executing relationship-forward recruitment methods, as well as reflexive recruitment documentation practices for Participatory AI researchers.

Figures

Figures reproduced from arXiv: 2508.20176 by the authors.

Figure 1
Figure 1. “Those recruited” papers (Appendix Table 6) [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 3
Figure 3. “Recruitment strategy” papers (Appendix Table 4) [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 20 canonical work pages

  1. [1]

    Organizational

    We contacted and worked with an NGO, university, advocacy group, union, etc. to recruit these stakehold- ers. This corresponds with the “Organizational” code. • If appropriate to disclose: Which Organization? How did you know about this organization?

    work page

  2. [2]

    Infrastructural

    I advertised the opportunity directly on a mailing list, crowdworker platform, on the wall or train, on a web- site, etc. This corresponds to the “Infrastructural” code. • On what mode of communication did you adver- tise?

    work page

  3. [3]

    Personal Networks

    I leveraged existing personal networks as a recruiting strategy. This corresponds with the “Personal Networks” code. • Describe how the personal network was used in re- cruiting and what the personal connection was

    work page

  4. [4]

    This corresponds to the “Event” code

    I met the participant at an event, such as a conference, a civic advocacy meeting, etc. This corresponds to the “Event” code. • Which event? • What important lessons did you learn from recruit- ing this stakeholder? This question is added to encour- age reflection on the recruitment process and to share any special learnings about the recruitment strategy...

    work page 2023

  5. [5]

    Rapid Damage Assessment Using Social Media Images by Combining Human and Machine Intelligence

    Rapid damage assessment using social media im- ages by combining human and machine intelligence. arXiv preprint arXiv:2004.06675. Irani, L.; Vertesi, J.; Dourish, P.; Philip, K.; and Grinter, R. E. 2010. Postcolonial computing: a lens on design and development. In Proceedings of the SIGCHI conference on human factors in computing systems, 1311–1320. Irani...

    work page Pith review arXiv 2004

  6. [11]

    Science, 366(6464): 447–453

    Dissecting racial bias in an algorithm used to man- age the health of populations. Science, 366(6464): 447–453. Publisher: American Association for the Advancement of Science. O’Brien, J. E.; Brewer, K. B.; Jones, L. M.; Corkhum, J.; and Rizo, C. F. 2022. Rigor and respect: recruitment strategies for engaging vulnerable populations in research. Journal of...

    work page 2022

  7. [12]

    In 2023 ACM Conference on Fairness, Account- ability, and Transparency , 1882–1895

    Queer In AI: A Case Study in Community-Led Partic- ipatory AI. In 2023 ACM Conference on Fairness, Account- ability, and Transparency , 1882–1895. Chicago IL USA: ACM. ISBN 9798400701924. Roscoe, R. 2021. Designing for diversity: Inclusive Sam- pling. Ergodesign & HCI, 9. Sendak, M.; Elish, M. C.; Gao, M.; Futoma, J.; Ratliff, W.; Nichols, M.; Bedoya, A.;...

    work page 2023

  8. [13]

    In Proceedings of the 2020 conference on fairness, accountability, and trans- parency, 99–109

    ” The human body is a black box” supporting clin- ical decision-making with deep learning. In Proceedings of the 2020 conference on fairness, accountability, and trans- parency, 99–109. Sharp, A.; and Anderson, K. 2010. Online citizen panels as an advance in research and consultation: A Review of pilot results. Commonwealth Journal of Local Governance , (...

    work page 2020

Show all 23 references
  1. [14]

    In Proceedings of the AAAI Conference on Arti- ficial Intelligence, volume 34, 13369–13375

    Improving efficiency of volunteer-based food rescue operations. In Proceedings of the AAAI Conference on Arti- ficial Intelligence, volume 34, 13369–13375. Issue: 08. Sinders, C. 2020. Feminist data set. Clinic for Open Source Arts. https://carolinesinders. com/wp-con tent/upl...

    2020

  2. [16]

    In Proceedings of the 2022 ACM Conference on Fair- ness, Accountability, and Transparency, 667–678

    Towards intersectional feminist and participatory ML: A case study in supporting Feminicide Counterdata Collec- tion. In Proceedings of the 2022 ACM Conference on Fair- ness, Accountability, and Transparency, 667–678. Swanson, A.; Kosmala, M.; Lintott, C.; Simpson, R.; Smith, ...

    2022

  3. [23]

    (2018) https://www.nature.com/articles/s41586-018-0637-6 Bakker et al

    (10) Table 9: How specific are the institutions, platforms, and or- ganizations? Country Number Botswana 1 Ethiopia 1 Global 10 Netherlands 1 New Zealand 1 South Africa 1 Tanzania 1 Uganda 1 United Kingdom 6 United States 17 Uruguay 1 Zimbabwe 2 Table 10: Table of Numbers by C...

    2018

  4. [101]

    Bødker, S.; and Kyng, M

    Publisher: Taylor & Francis. Bødker, S.; and Kyng, M. 2018. Participatory design that matters—Facing the big issues. ACM Transactions on Computer-Human Interaction (TOCHI), 25(1): 1–31. Pub- lisher: ACM New York, NY , USA. Calacci, D. 2022. Organizing in the end of employment:...

    2018 arXiv

  5. [112]

    ISBN 9798400704505

    New York, NY , USA: Association for Computing Ma- chinery. ISBN 9798400704505. Kulynych, B.; Madras, D.; Milli, S.; Raji, I. D.; Zhou, A.; and Zemel, R. 2020. Participatory Approaches to Machine Learning. Published: International Conference on Machine Learning Workshop. Kuo, T...

    2020

  6. [476]

    Gilman, M

    Publisher: Sage Publications. Gilman, M. E. 2023. Democratizing AI: Principles for Meaningful Public Participation. Data & Society. Groves, L.; Peppin, A.; Strait, A.; and Brennan, J. 2023. Going public: the role of public participation approaches in commercial AI labs. In Pro...

    2023

  7. [517]

    Sloane, M.; Moss, E.; Awomolo, O.; and Forlano, L

    pd f. Sloane, M.; Moss, E.; Awomolo, O.; and Forlano, L. 2022. Participation is not a design fix for machine learning. InPro- ceedings of the 2nd ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, 1–6. Small, M. L.; and Calarco, J. M. 2022. Qualit...

    2022

  8. [2011]

    Is it OK for me to hit the record button now?

    eBird: engaging birders in science and conservation. PLoS biology, 9(12): e1001220. Publisher: Public Library of Science San Francisco, USA. Young, M.; Magassa, L.; and Friedman, B. 2019. Toward inclusive tech policy design: a method for underrepresented voices to strengthen t...

    2019

  9. [2013]

    In Proceedings of the SIGCHI Conference on Hu- man Factors in Computing Systems, 429–438

    Configuring participation: on how we involve people in design. In Proceedings of the SIGCHI Conference on Hu- man Factors in Computing Systems, 429–438. Paris France: ACM. ISBN 978-1-4503-1899-0. Wilder, B.; Onasch-Vera, L.; Diguiseppi, G.; Petering, R.; Hill, C.; Yadav, A.; R...

    2021

  10. [2019]

    Research involvement and engagement, 5: 1–9

    Recruitment of caregivers into health services re- search: lessons from a user-centred design study. Research involvement and engagement, 5: 1–9. Publisher: Springer. Li, R.; Kingsley, S.; Fan, C.; Sinha, P.; Wai, N.; Lee, J.; Shen, H.; Eslami, M.; and Hong, J. 2023. Participa...

    2023 arXiv

  11. [2020]

    In Proceed- ings of the 2020 Conference on Fairness, Accountability, and Transparency, 695–695

    When not to design, build, or deploy. In Proceed- ings of the 2020 Conference on Fairness, Accountability, and Transparency, 695–695. Benjamin, R. 2016. Informed refusal: Toward a justice- based bioethics. Science, Technology, & Human Values , 41(6): 967–990. Publisher: SAGE P...

    2020

  12. [2021]

    Public health nursing (Boston, Mass.), 38(6): 1102–1115

    Recruitment and Retention of Underrepresented and Vulnerable Populations to Research. Public health nursing (Boston, Mass.), 38(6): 1102–1115. Le Dantec, C. A.; and Fox, S. 2015. Strangers at the gate: Gaining access, building rapport, and co-constructing community-based resea...

    2015

  13. [2022]

    International Journal of Qualitative Meth- ods, 21: 16094069221119576

    Successful recruitment to qualitative research: A crit- ical reflection. International Journal of Qualitative Meth- ods, 21: 16094069221119576. Publisher: SAGE Publica- tions Sage CA: Los Angeles, CA. Nekoto, W.; Marivate, V .; Matsila, T.; Fasubaa, T.; Ko- lawole, T.; Fagbohu...

    2020 arXiv

  14. [2023]

    Proceedings of the ACM on Human-Computer In- teraction, 7(CSCW2): 1–26

    Public health calls for/with AI: an ethnographic per- spective. Proceedings of the ACM on Human-Computer In- teraction, 7(CSCW2): 1–26. Publisher: ACM New York, NY , USA. Kallina, E.; Bohn ´e, T.; and Singh, J. 2025. Stakeholder Participation for Responsible AI Development: Di...

    2025

  15. [2025]

    Proceedings of the ACM on Human-Computer Interaction, 9(2): 1–31

    Towards Equitable Community-Industry Collabora- tions: Understanding the Experiences of Nonprofits’ Collab- orations with Tech Companies. Proceedings of the ACM on Human-Computer Interaction, 9(2): 1–31. Publisher: ACM New York, NY , USA. Erete, S.; Rankin, Y . A.; and Thomas,...

    2021

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.