REVIEW 3 major objections 5 minor 50 references
Making Transparency Advocates: An Educational Approach Towards Better Algorithmic Transparency in Practice
T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read A two-hour educational workshop can turn professionals into algorithmic transparency advocates, with participants reporting real advocacy actions such as speaking up at an AI strategy meeting.
desk verdict Honest small pilot on teaching transparency advocacy; useful taxonomy, but effectiveness claim overstates self-report evidence. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the two-hour workshop, built around five modules covering transparency definitions, tools (model cards, datasheets, explainer dashboards, Shapley values), the stakeholder-first Transparency Playbook, a role-playing breakout activity where participants argue for and against disclosure at a fictional startup, and common objections to transparency with rebuttals. The role-play is central because it rehearses the tensions participants will meet as advocates and equips them with counterarguments. A pre/post survey adapted from prior responsible data-science teaching measures six constructs, and semi-structured interviews capture reported actions.
What would settle it
A randomized evaluation in which employees within the same organizations are assigned to the workshop or to a placebo training, with advocacy actions measured from verifiable records such as meeting minutes, adoption of model cards or datasheets, emails, or project artifacts, would settle whether the workshop causes advocacy rather than merely accompanying it.
Extended reading notes
Core claim
The paper claims that education can translate algorithmic transparency research into practice by equipping motivated individuals inside organizations with knowledge, tools, and argumentative strategies to advocate for transparency. The authors delivered a two-hour, open-source workshop built on a stakeholder-first transparency playbook, and their qualitative interviews and pre/post surveys indicate gains in both transparency literacy and willingness to advocate. The most notable reported outcome is a participant who raised transparency concerns at an organization-wide AI strategy meeting days after the workshop, directly applying the workshop's lessons. The paper also claims that advocacy has three levels—conversational, implementational, and influential—and that news and media professionals tend to be more willing but less equipped to act, while startup professionals have more technical means but less organizational room to prioritize transparency.
Load-bearing premise
The conclusions rest on participants' own unverified reports, in surveys and unrecorded interviews, that their understanding and advocacy increased, without a control group or external confirmation of the reported actions.
Editorial extensions
If this is right
- Organizations can cultivate bottom-up pressure for algorithmic transparency through short, low-cost workshops, even in the absence of strong regulation.
- Advocacy is multi-level: education should prepare people for conversational, implementational, and influential actions, since each requires different skills and authority.
- Professional domain shapes advocacy, so training and support must be tailored: news and media professionals need tools, while startup professionals need resources and prioritization.
- A single advocate can bring transparency onto a high-stakes organizational agenda, and the reported AI strategy meeting episode suggests a possible ripple effect on company practices.
- Open-source workshop materials make the intervention replicable, allowing other organizations and educators to test and adapt it.
Reading between the lines
- If the effects replicate with larger samples, workshop-based advocate cultivation could complement regulation by creating internal demand for transparency that outlasts individual workshops.
- A natural extension is a longitudinal study tracking whether reported advocacy translates into auditable organizational artifacts, such as published model cards or disclosure policies, months after training.
- The level taxonomy suggests a testable prediction: training that matches advocacy level to role, such as engineers to implementational actions and managers to influential ones, will produce more sustained change than generic training.
- Domain differences imply that startup-focused interventions may need to bundle transparency with business value, such as risk reduction or investor due diligence, rather than relying on ethical appeals alone.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper reports on a workshop-based educational intervention designed to create "transparency advocates" who can drive bottom-up organizational change toward algorithmic transparency. The authors delivered a two-hour workshop, developed over several years and based on their prior stakeholder-first transparency playbook, to two groups of professionals: 15 news/media professionals and 12 technology startup professionals. Data include pre/post workshop surveys (n=15 matched responses) and semi-structured follow-up interviews with 7 participants, which were not recorded but captured via detailed notes. The paper claims that the workshop increased participants' algorithmic transparency literacy and their willingness to advocate for transparency in their professional lives, and it proposes a three-level taxonomy of advocacy (conversational, implementational, influential) plus domain-specific differences in advocacy barriers. The authors openly acknowledge the small, self-selected sample and the absence of statistical testing, and they make all workshop materials publicly available.
Significance. If the central claims were established, this work would make a valuable contribution to responsible AI education and AI governance by offering a concrete, replicable, bottom-up mechanism for translating XAI research into organizational practice. The paper's strengths include a freely available, open-source workshop curriculum, a two-domain comparison, and a honest acknowledgment of key limitations (small sample, self-selection, descriptive statistics only). The proposed taxonomy of advocacy levels is a useful conceptual contribution, and the reported real-world actions (e.g., a participant speaking up at an organization-wide AI strategy meeting) are motivating examples. However, the evidentiary basis for the causal claim that the workshop increased literacy and advocacy willingness is fragile, resting primarily on self-report measures and unrecorded interviews; the paper's significance therefore hinges on how these evidential limitations are treated in the framing of the conclusions.
major comments (3)
- [Methods (Data Collection and Analysis); Table 3] The claim that the workshop increased algorithmic transparency literacy is not established by the quantitative evidence. The survey's literacy measure (Q5) is a self-rating of understanding on a 1-10 scale, not an objective knowledge test. The observed increase from 5.00 to 7.79 could reflect increased confidence or a shift in self-perception rather than actual knowledge gain. This ambiguity is especially acute because the qualitative results ("Uncovering knowledge gaps") indicate that participants realized they "didn't know what they didn't know," which should, if anything, bias post-workshop self-ratings downward; the paper does not address this tension. The authors should either temper the literacy claim to "perceived understanding" or supplement the self-report with an objective measure (e.g., a short knowledge quiz) in future iterations, and they should explicitly discuss the direction of self-report bias in the current study.
- [Results ("Taking action"); Discussion ("Levels of Advocacy")] The attribution of participants' advocacy actions to the workshop is not sufficiently supported. P2 is quoted as saying "I always would've advocated for transparency anyway," and the paper acknowledges that actions "appeared to be motivated, at least in part, by the workshop." This "at least in part" rests entirely on participants' own judgments, which are vulnerable to social desirability bias and hindsight bias. Without a counterfactual, a control group, or independent verification of the reported actions, the paper cannot rule out the possibility that participants were already inclined to advocate and would have acted identically without the workshop. The authors should either substantially soften the causal language in the abstract and conclusion, or provide a more rigorous analysis of the specific workshop elements that plausibly changed behavior (e.g., the role-playing activity's direct influence on P4's strategy-meeting intervention).
- [Methods (Data Collection and Analysis); Results (Thematic Analysis Findings)] The reliability of the qualitative findings is compromised by the decision not to record interviews. The paper states that "we chose not to record the interviews" and instead "took detailed notes," but it does not describe how the notes were verified (e.g., by member checking, participant review, or second-coder agreement). Because the thematic analysis was conducted by the authors on their own notes, there is a risk of selective note-taking and confirmatory interpretation. The paper should either provide evidence of note reliability or explicitly acknowledge this as a threat to the trustworthiness of the central qualitative claims, and it should explain how the 33 codes and 6 themes were audited beyond "two separate working sessions."
minor comments (5)
- [Discussion ("The Importance of Domain-of-use")] There is a typo: "we found vast differences in the attitudes towards algorithmic transparency in new and media vs. technology startups" should read "news and media."
- [Introduction; References] The name "Myerson" appears in the text, but the reference is to "Meyerson (2003)"; please use the correct spelling consistently throughout.
- [References] The reference for Covert et al. begins with "DBLP:journals/corr/abs-2004-00668" which appears to be a leftover identifier; this should be cleaned up.
- [Appendix (Figure 2)] The figure is referred to as "Appendix Figure 2" in the Methods section but is simply "Figure 2" in the appendix; please number it consistently.
- [Methods (Pre- and post-workshop surveys)] The paper states that the survey was "adapted from previous work by Lewis and Stoyanovich (2021)" but does not give details on which items were changed or validated for this context; adding this information would improve replicability.
Circularity Check
No circularity: the workshop-effectiveness claims are empirical evaluations, not derivations that reduce to their inputs.
full rationale
This paper contains no derivation chain, equation, fitted parameter, or uniqueness theorem, so the enumerated circularity patterns do not apply. The central claims—that the workshop increased algorithmic transparency literacy and willingness to advocate—are empirical evaluations supported by pre/post surveys and semi-structured interviews. The survey was 'adapted from previous work by Lewis and Stoyanovich (2021)', a self-citation, but that is instrument provenance rather than a load-bearing result: the outcome of the workshop is not entailed by the survey's origin. Similarly, the workshop content draws on the authors' prior playbook, but testing one's own educational material is not circular; the evaluation could have failed, and the paper reports both positive and mixed participant responses. The acknowledged limitations—small sample size, optional participation bias, self-reported outcomes, and no control group—affect evidentiary strength and internal validity, not circularity. No quantity is defined in terms of another, no prediction is statistically forced by a fitted input, and no external result is imported via self-citation to forbid alternatives. The finding is therefore not circular.
Assumptions & free parameters
assumptions (3)
- domain assumption Self-reported survey responses and interview accounts accurately reflect participants' transparency literacy and advocacy behavior.
- domain assumption Changes in self-reported outcomes are attributable to the workshop rather than to other factors.
- domain assumption The authors' thematic analysis fairly represents participants' experiences.
Cite this review
Pith. "Pith review of Making Transparency Advocates: An Educational Approach Towards Better Algorithmic Transparency in Practice." pith.science (2026). https://pith.science/paper/WBICAEMX
@misc{pith2026241215363,
author = {Pith},
title = {Pith review of: Making Transparency Advocates: An Educational Approach Towards Better Algorithmic Transparency in Practice},
year = {2026},
howpublished = {\url{https://pith.science/paper/WBICAEMX}},
note = {Machine review of arXiv:2412.15363}
}
read the original abstract
Concerns about the risks and harms posed by artificial intelligence (AI) have resulted in significant study into algorithmic transparency, giving rise to a sub-field known as Explainable AI (XAI). Unfortunately, despite a decade of development in XAI, an existential challenge remains: progress in research has not been fully translated into the actual implementation of algorithmic transparency by organizations. In this work, we test an approach for addressing the challenge by creating transparency advocates, or motivated individuals within organizations who drive a ground-up cultural shift towards improved algorithmic transparency. Over several years, we created an open-source educational workshop on algorithmic transparency and advocacy. We delivered the workshop to professionals across two separate domains to improve their algorithmic transparency literacy and willingness to advocate for change. In the weeks following the workshop, participants applied what they learned, such as speaking up for algorithmic transparency at an organization-wide AI strategy meeting. We also make two broader observations: first, advocacy is not a monolith and can be broken down into different levels. Second, individuals' willingness for advocacy is affected by their professional field. For example, news and media professionals may be more likely to advocate for algorithmic transparency than those working at technology start-ups.
Figures
Reference graph
Works this paper leans on
-
[1]
, " * write output.state after.block = add.period write newline
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Abdul, A.; von der Weth, C.; Kankanhalli, M.; and Lim, B. Y. 2020. COGAM: Measuring and Moderating Cognitive Load in Machine Learning Model Explanations. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 1--14
work page 2020
-
[4]
Beattie, L.; Taber, D.; and Cramer, H. 2022. Challenges in Translating Research to Practice for Evaluating Fairness and Bias in Recommendation Systems. In Proceedings of the 16th ACM Conference on Recommender Systems, 528--530
work page 2022
-
[5]
Bell, A.; Nov, O.; and Stoyanovich, J. 2023. The Algorithmic Transparency Playbook: A Stakeholder-first Approach to Creating Transparency for Your Organization’s Algorithms. In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems, 1--4
work page 2023
-
[6]
Bell, A.; Solano-Kamaiko, I.; Nov, O.; and Stoyanovich, J. 2022. It’s just not that simple: an empirical study of the accuracy-explainability trade-off in machine learning for public policy. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, 248--266
work page 2022
-
[7]
Braun, V.; and Clarke, V. 2006. Using thematic analysis in psychology. Qualitative research in psychology, 3(2): 77--101
2006
-
[8]
Covert, I.; Lundberg, S. M.; and Lee, S. 2020. DBLP:journals/corr/abs-2004-00668 Feature Contributions Through Additive Importance Measures. CoRR, abs/2004.00668
arXiv 2020
Show all 50 references
-
[9]
Dastin, J. 2022. Amazon scraps secret AI recruiting tool that showed bias against women. In Ethics of data and analytics, 296--299. Auerbach Publications
2022
-
[10]
Datta, A.; Sen, S.; and Zick, Y. 2016. Algorithmic transparency via quantitative input influence: Theory and experiments with learning systems. In 2016 IEEE symposium on security and privacy (SP), 598--617. IEEE
2016
-
[11]
de Laat, P. B. 2022. Algorithmic decision-making employing profiling: will trade secrecy protection render the right to explanation toothless? Ethics and Information Technology, 24(2): 17
2022
-
[12]
Dom \' nguez Figaredo, D.; and Stoyanovich, J. 2023. Responsible AI literacy: A stakeholder-first approach. Big Data & Society, 10(2): 20539517231219958
2023
-
[13]
Doshi-Velez, F.; Kortz, M.; Budish, R.; Bavitz, C.; Gershman, S.; O'Brien, D.; Scott, K.; Schieber, S.; Waldo, J.; Weinberger, D.; et al. 2017. Accountability of AI under the law: The role of explanation. arXiv preprint arXiv:1711.01134
2017 arXiv
-
[14]
Gasser, U.; and Almeida, V. A. F. 2017. A Layered Model for AI Governance. IEEE Internet Comput. , 21(6): 58--62
2017
-
[15]
W.; Wallach, H
Gebru, T.; Morgenstern, J.; Vecchione, B.; Vaughan, J. W.; Wallach, H. M.; III, H. D.; and Crawford, K. 2021. Datasheets for datasets. Commun. ACM , 64(12): 86--92
2021
-
[16]
Griffiths, C.; Pio, E.; and McGhee, P. 2022. Tempered radicals in manufacturing: Invisible champions of inclusion. Journal of Management & Organization, 1--22
2022
-
[17]
Hill, K. 2022. The secretive company that might end privacy as we know it. In Ethics of Data and Analytics, 170--177. Auerbach Publications
2022
-
[18]
Holstein, K.; Wortman Vaughan, J.; Daum \'e III, H.; Dudik, M.; and Wallach, H. 2019. Improving fairness in machine learning systems: What do industry practitioners need? In Proceedings of the 2019 CHI conference on human factors in computing systems, 1--16
2019
-
[19]
u ller, H. 2020. Measuring the quality of explanations: the system causability scale (SCS). KI-K \
Holzinger, A.; Carrington, A.; and M \"u ller, H. 2020. Measuring the quality of explanations: the system causability scale (SCS). KI-K \"u nstliche Intelligenz , 1--6
2020
-
[20]
Hu, Q.; and Rangwala, H. 2020. Towards Fair Educational Data Mining: A Case Study on Detecting At-Risk Students
2020
-
[21]
D.; and Jacobsen, K
Jacobsen, G. D.; and Jacobsen, K. H. 2011. Health awareness campaigns and diagnosis rates: evidence from National Breast Cancer Awareness Month. Journal of health economics, 30(1): 55--61
2011
-
[22]
E.; and Kohn, C
Jacoby, J.; Speller, D. E.; and Kohn, C. A. 1974. Brand choice behavior as a function of information load. Journal of marketing research, 11(1): 63--69
1974
-
[23]
Jobin, A.; Ienca, M.; and Vayena, E. 2019. Artificial Intelligence: the global landscape of ethics guidelines. CoRR, abs/1906.11668
2019 arXiv
-
[24]
S.; Samadi, M.; and Tuzun, T
Kirilenko, A.; Kyle, A. S.; Samadi, M.; and Tuzun, T. 2017. The flash crash: High-frequency trading in an electronic market. The Journal of Finance, 72(3): 967--998
2017
-
[25]
Kirton, G.; Greene, A.-M.; and Dean, D. 2007. British diversity professionals as change agents--radicals, tempered radicals or liberal reformers? The International Journal of Human Resource Management, 18(11): 1979--1994
2007
-
[26]
Lewis, A.; and Stoyanovich, J. 2021. Teaching responsible data science: Charting new pedagogical territory. International Journal of Artificial Intelligence in Education, 1--25
2021
-
[27]
Loi, M.; and Spielkamp, M. 2021. Towards accountability in the use of artificial intelligence for public administrations. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, 757--766
2021
-
[28]
M.; and Lee, S
Lundberg, S. M.; and Lee, S. 2017. A Unified Approach to Interpreting Model Predictions. In Guyon, I.; von Luxburg, U.; Bengio, S.; Wallach, H. M.; Fergus, R.; Vishwanathan, S. V. N.; and Garnett, R., eds., Advances in Neural Information Processing Systems 30: Annual Conferenc...
2017
-
[29]
A.; Stark, L.; Wortman Vaughan, J.; and Wallach, H
Madaio, M. A.; Stark, L.; Wortman Vaughan, J.; and Wallach, H. 2020. Co-designing checklists to understand organizational challenges and opportunities around fairness in AI. In Proceedings of the 2020 CHI conference on human factors in computing systems, 1--14
2020
-
[30]
Metcalf, J.; Moss, E.; et al. 2019. Owning ethics: Corporate logics, silicon valley, and the institutionalization of ethics. Social Research: An International Quarterly, 86(2): 449--476
2019
-
[31]
Meyerson, D. 2003. Tempered Radicals: how everyday leaders inspire change at work Boston. MA: Harvard Business School Press, xi--xii, 41: 59--60
2003
-
[32]
Meyerson, D.; and Tompkins, M. 2007. Tempered radicals as institutional change agents: The case of advancing gender equity at the University of Michigan. Harv. JL & Gender, 30: 303
2007
-
[33]
D.; and Gebru, T
Mitchell, M.; Wu, S.; Zaldivar, A.; Barnes, P.; Vasserman, L.; Hutchinson, B.; Spitzer, E.; Raji, I. D.; and Gebru, T. 2019. Model Cards for Model Reporting. In danah boyd; and Morgenstern, J. H., eds., Proceedings of the Conference on Fairness, Accountability, and Transparenc...
2019
-
[34]
Munn, L. 2023. The uselessness of AI ethics. AI and Ethics, 3(3): 869--877
2023
-
[35]
W.; Gramby-Sobukwe, S.; and Williams-Gegner, K
Ngunjiri, F. W.; Gramby-Sobukwe, S.; and Williams-Gegner, K. 2012. Tempered radicals: Black women’s leadership in the church and community. Journal of Pan African Studies, 5(2): 84--109
2012
-
[36]
Obermeyer, Z.; Powers, B.; Vogeli, C.; and Mullainathan, S. 2019. Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464): 447--453
2019
-
[37]
OECD.AI. 2021. Database of national AI policies
2021
-
[38]
Phadke, S.; Samory, M.; and Mitra, T. 2022. Pathways through conspiracy: the evolution of conspiracy radicalization through engagement in online conspiracy discussions. In Proceedings of the International AAAI Conference on Web and Social Media, volume 16, 770--781
2022
-
[39]
D.; Smart, A.; White, R
Raji, I. D.; Smart, A.; White, R. N.; Mitchell, M.; Gebru, T.; Hutchinson, B.; Smith-Loud, J.; Theron, D.; and Barnes, P. 2020. Closing the AI accountability gap: Defining an end-to-end framework for internal algorithmic auditing. In Proceedings of the 2020 conference on fairn...
2020
-
[40]
Rakova, B.; Yang, J.; Cramer, H.; and Chowdhury, R. 2021. Where responsible AI meets reality: Practitioner perspectives on enablers for shifting organizational practices. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW1): 1--23
2021
-
[41]
T.; Singh, S.; and Guestrin, C
Ribeiro, M. T.; Singh, S.; and Guestrin, C. 2016. ``Why should i trust you?'' Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, 1135--1144
2016
-
[42]
Rudin, C. 2019. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature machine intelligence, 1(5): 206--215
2019
-
[43]
Sapiezynski, P.; Kassarnig, V.; and Wilson, C. 2017. Academic performance prediction in a gender-imbalanced environment. In Proceedings of FATREC Workshop on Responsible Recommendation at ACM RecSys
2017
-
[44]
Selbst, A.; and Powles, J. 2018. ``Meaningful Information''' and the Right to Explanation. In Friedler, S. A.; and Wilson, C., eds., Conference on Fairness, Accountability and Transparency, FAT 2018, 23-24 February 2018, New York, NY, USA , volume 81 of Proceedings of Machine ...
2018
-
[45]
Stoyanovich, J.; and Howe, B. 2019. Nutritional Labels for Data and Models. IEEE Data Eng. Bull. , 42(3): 13--23
2019
-
[46]
The White House . 2023. FACT SHEET: President Biden Issues Executive Order on Safe, Secure, and Trustworthy Artificial Intelligence
2023
-
[47]
UNICRI. 2020. Towards Responsible Artificial Intelligence Innovation. European Comission
2020
-
[48]
Walton, S.; and Kirkwood, J. 2013. Tempered radicals! Ecopreneurs as change agents for sustainability--an exploratory study. International Journal of Social Entrepreneurship and Innovation, 2(5): 461--475
2013
-
[49]
Williams, R. 2021. How to train your robot: project-based ai and ethics education for middle school classrooms. In Proceedings of the 52nd ACM Technical Symposium on Computer Science Education, 1382--1382
2021
-
[50]
Yang, Y.; Kandogan, E.; Li, Y.; Sen, P.; and Lasecki, W. S. 2019. A study on interaction in human-in-the-loop machine learning for text analytics. In IUI Workshops
2019
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.